A Hybrid LSTM-GNN-Q-Learning Model for Zero-Day Attack Detection: Evaluation on CICIDS2017 with Simulated Zero-Day Setting
DOI:
https://doi.org/10.63158/journalisi.v8i3.1670Keywords:
Zero-day attack detection, Hybrid deep learning, Intrusion detection system, Graph Neural Network, Reinforcement learningAbstract
Zero-day attacks exploit previously unseen vulnerabilities, making them difficult to identify using signature-based approaches. Their ability to bypass conventional detection mechanisms can result in significant financial losses, system compromise, and data breaches. To address this challenge, this study proposes a Hybrid Predictive Deep Learning (HPDL) model that integrates the Long Short-Term Memory (LSTM) network for modelling temporal relationships, Graph Neural Networks (GNN) for structural relationship modelling, and Q-Learning for feature weighting and adaptive decision making. The model was evaluated on CICIDS2017 dataset under a simulated zero-day setting by holding out four attack types (Brute Force, SQL Injection, XSS, and Infiltration), totaling 2,179 zero-day samples deliberately excluded from training and validation and used only for final testing. Experimental results show that the proposed HPDL model achieved a zero-day attack detection accuracy of 99.63% and F1-score of 0.9970, outperforming LSTM-only and GNN-only baseline models, which achieved accuracies of 98.5% and 85.0%, respectively. These results indicate that integrating temporal, structural, and reinforcement learning paradigms provides an effective approach for zero-day attack detection.
Downloads
References
[1] M. Gracy, B. R. Jeyavadhanam, P. K. Babu, S. H. Karthick, and R. Chandru, “Growing Threats Of Cyber Security: Protecting Yourself In A Digital World,” in 2023 International Conference on Networking and Communications (ICNWC), 2023, pp. 1–5. doi: 10.1109/ICNWC57852.2023.10127398.
[2] M. Inzimam, C. Yongle, and Z. Zhang, “An Efficient Approach towards Assessment of Zero-day Attacks,” Int. J. Comput. Appl., vol. 177, no. 26, pp. 34–39, Dec. 2019, doi: 10.5120/IJCA2019919742.
[3] C. A. Teodorescu, “Perspectives and Reviews in the Development and Evolution of the Zero-Day Attacks,” Informatica Economica, vol. 26, no. 2/2022, pp. 46–56, Jun. 2022, doi: 10.24818/issn14531305/26.2.2022.05.
[4] D. Muktadir-Al-Mukit and M. H. Ali, “The Dynamics of Stock Market Responses Following the Cyber-Attacks News: Evidence from Event Study,” Information Systems Frontiers, 2025, doi: 10.1007/s10796-025-10639-6.
[5] M. A. Mohamed Mohideen et al., “Behind the Code: Identifying Zero-Day Exploits in WordPress,” Future Internet, vol. 16, no. 7, p. 256, Jul. 2024, doi: 10.3390/FI16070256.
[6] Y. Guo, “A review of Machine Learning-based zero-day attack detection: Challenges and future directions,” Comput. Commun., vol. 198, pp. 175–185, Jan. 2023, doi: 10.1016/J.COMCOM.2022.11.001.
[7] D. Georgoulias, R. Yaben, and E. Vasilomanolakis, “Cheaper than you thought? A dive into the darkweb market of cyber-crime products,” in ACM International Conference Proceeding Series, ACM, Aug. 2023. doi: 10.1145/3600160.3605012.
[8] W. Wang, L. Chen, L. Han, Z. Zhou, Z. Xia, and X. Chen, “Vulnerability Assessment for ICS system Based on Zero-day Attack Graph,” Proceedings - 2020 International Conference on Intelligent Computing, Automation and Systems, ICICAS 2020, pp. 1–5, Dec. 2020, doi: 10.1109/ICICAS51530.2020.00009.
[9] A. Armijos and E. Cuenca, “Zero-day attacks: review of the methods used based on intrusion detection and prevention systems,” in 2023 IEEE Colombian Caribbean Conference (C3), 2023, pp. 1–6. doi: 10.1109/C358072.2023.10436218.
[10] T. Ohtani, R. Yamamoto, and S. Ohzahata, “Detecting Zero-Day Attack with Federated Learning Using Autonomously Extracted Anomalies in IoT,” in 2024 IEEE 21st Consumer Communications & Networking Conference (CCNC), 2024, pp. 356–359. doi: 10.1109/CCNC51664.2024.10454669.
[11] M. A. Shyaa, N. F. Ibrahim, Z. Zainol, R. Abdullah, M. Anbar, and L. Alzubaidi, “Evolving cybersecurity frontiers: A comprehensive survey on concept drift and feature dynamics aware machine and deep learning in intrusion detection systems,” Eng. Appl. Artif. Intell., vol. 137, p. 109143, Nov. 2024, doi: 10.1016/J.ENGAPPAI.2024.109143.
[12] D. Han et al., “Evaluating and Improving Adversarial Robustness of Machine Learning-Based Network Intrusion Detectors,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 8, pp. 2632–2647, Aug. 2021, doi: 10.1109/JSAC.2021.3087242.
[13] H. Hindy, R. Atkinson, C. Tachtatzis, J. N. Colin, E. Bayne, and X. Bellekens, “Utilising deep learning techniques for effective zero-day attack detection,” Electronics (Basel)., vol. 9, no. 10, pp. 16–84, Oct. 2020, doi: 10.3390/electronics9101684.
[14] A. A. Korba, A. Boualouache, and Y. Ghamri-Doudane, “Zero-X: A Blockchain-Enabled Open-Set Federated Learning Framework for Zero-Day Attack Detection in IoV,” IEEE Trans. Veh. Technol., vol. 73, no. 9, pp. 12399–12414, 2024, doi: 10.1109/TVT.2024.3385916.
[15] D. A. Ammara, J. Ding, and K. Tutschku, “Architectural Selection Framework for Synthetic Network Traffic: Quantifying the Fidelity–Utility Trade-off,” IEEE Access, vol. 14, pp. 468–484, 2026, doi: 10.1109/ACCESS.2025.3646769.
[16] X. Yuan, J. Wan, D. An, and H. Pei, “A novel encrypted traffic detection model based on detachable convolutional GCN-LSTM,” Sci. Rep., vol. 15, no. 1, p. 27705, Jul. 2025, doi: 10.1038/s41598-025-13397-2.
[17] Y. Zhang, S. Chen, C. Zhang, J. Zhao, K. Zhang, and Z. Lu, “Power information network attack chain identification and disaster recovery early warning mechanism based on graph neural network,” International Journal of Intelligent Information and Database Systems, vol. 18, no. 6, pp. 1–38, 2026, doi: 10.1504/IJIIDS.2026.153373.
[18] P. Zhang et al., “From Prediction to Planning: A Spectral-Temporal GNN and Bi-Directional Decoding RL Framework,” Signals, vol. 7, no. 3, pp. 1–37, May 2026, doi: 10.3390/signals7030047.
[19] Z. Utic and A. Oyemaja, “Q-Learning Approach Applied to Network Security,” Electronics (Switzerland), vol. 14, no. 10, May 2025, doi: 10.3390/electronics14101996.
[20] R. Ahmad, I. Alsmadi, W. Alhamdani, and L. Tawalbeh, “Zero-day attack detection: a systematic literature review,” Artif. Intell. Rev., vol. 56, no. 10, pp. 10733–10811, Feb. 2023, doi: 10.1007/S10462-023-10437-Z.
[21] R. M. Al-Khatib, L. Heilat, W. Qudah, S. Alhatamleh, and A. Al-Khateeb, “A novel improved deep learning model based on Bi-LSTM algorithm for intrusion detection in WSN,” Networks and Heterogeneous Media, vol. 20, no. 2, pp. 532–565, 2025, doi: 10.3934/nhm.2025024.
[22] H. R. Sayegh, W. Dong, and A. M. Al-madani, “Enhanced Intrusion Detection with LSTM-Based Model, Feature Selection, and SMOTE for Imbalanced Data,” Applied Sciences (Switzerland), vol. 14, no. 2, Jan. 2024, doi: 10.3390/app14020479.
[23] T. Bui, M. Tran, D. Tran, and L. G. Nguyen, “Real-time Android malware detection using Graph Isomorphism Network and statistical network traffic features,” Journal of Cyber Security Technology, 2025, doi: 10.1080/23742917.2025.2553924.
[24] B. Khemani, S. Patil, K. Kotecha, and S. Tanwar, “A review of graph neural networks: concepts, architectures, techniques, challenges, datasets, applications, and future directions,” J. Big Data, vol. 11, no. 1, pp. 1–43, Dec. 2024, doi: 10.1186/S40537-023-00876-4/TABLES/13.
[25] Y. Li, “GAGA-Net: A GAN and GNN Hybrid Model for Enhanced Network Anomaly Detection in Cybersecurity,” Informatica, vol. 49, no. 36, Dec. 2025, doi: 10.31449/INF.V49I36.9768.
[26] D. O. Oyewola, S. A. Akinwunmi, and T. O. Omotehinwa, “Deep LSTM and LSTM-Attention Q-learning based reinforcement learning in oil and gas sector prediction,” Knowl. Based. Syst., vol. 284, p. 111290, Jan. 2024, doi: 10.1016/J.KNOSYS.2023.111290.
[27] C. J. C. H. Watkins and P. Dayan, “Technical Note: Q-Learning,” Mach. Learn., vol. 8, no. 3, pp. 279–292, 1992, doi: 10.1023/A:1022676722315.
[28] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, Feb. 2015, doi: 10.1038/nature14236.
[29] T. T. Nguyen and V. J. Reddi, “Deep Reinforcement Learning for Cyber Security,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 8, pp. 3779–3795, Aug. 2023, doi: 10.1109/TNNLS.2021.3121870.
[30] M. Alazab, S. Venkatraman, P. Watters, and M. Alazab, “Zero-day malware detection based on supervised learning algorithms of API call signatures,” in Proceedings of the Ninth Australasian Data Mining Conference (AusDM), Ballarat: Australian Computer Society, Dec. 2011, pp. 171–182. doi: 10.5555/2483628.2483648.
[31] C. Redino et al., “Zero Day Threat Detection Using Graph and Flow Based Security Telemetry,” 3rd IEEE 2022 International Conference on Computing, Communication, and Intelligent Systems, ICCCIS 2022, pp. 655–662, 2022, doi: 10.1109/ICCCIS56430.2022.10037596.
[32] Y. Wu, Y. Hu, J. Wang, M. Feng, A. Dong, and Y. Yang, “An active learning framework using deep Q-network for zero-day attack detection,” Comput. Secur., vol. 139, p. 103713, Apr. 2024, doi: 10.1016/J.COSE.2024.103713.
[33] J. F. Cevallos M., A. Rizzardi, S. Sicari, and A. C. Porisini, “NERO: NEural algorithmic reasoning for zeRO-day attack detection in the IoT: A hybrid approach,” Comput. Secur., vol. 142, Jul. 2024, doi: 10.1016/j.cose.2024.103898.
[34] R. Ranpara, S. K. Patel, O. P. Kumar, and F. A. Al-Zahrani, “Scalable architecture for autonomous malware detection and defense in software-defined networks using federated learning approaches,” Sci. Rep., vol. 15, no. 1, p. 30190, Aug. 2025, doi: 10.1038/s41598-025-14512-z.
[35] J. Wang et al., “Self-learning model fusion for network anomaly detection: A hybrid CNN-LSTM-transformer framework,” PLoS One, vol. 20, no. 10, p. e0332502, Oct. 2025, doi: 10.1371/JOURNAL.PONE.0332502.
[36] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in ICISSP 2018 - Proceedings of the 4th International Conference on Information Systems Security and Privacy, SciTePress, 2018, pp. 108–116. doi: 10.5220/0006639801080116.
[37] Z. Zhang, Y. Zhang, D. Guo, and M. Song, “A scalable network intrusion detection system towards detecting, discovering, and learning unknown attacks,” International Journal of Machine Learning and Cybernetics, vol. 12, no. 6, pp. 1649–1665, 2021, doi: 10.1007/s13042-020-01264-7.
[38] Z. Cang, A. Mahanti, R. Naha, and S. K. Battula, “Double DQN-GAMO: A Cyber Threat Detection Framework for Zero-Day Attacks,” IEEE Conference on Local Computer Networks, pp. 1–9, Sep. 2025, doi: 10.1109/LCN65610.2025.11146309.
[39] S. Das, M. Ashrafuzzaman, F. T. Sheldon, and S. Shiva, “Ensembling supervised and unsupervised machine learning algorithms for detecting distributed denial of service attacks,” Algorithms, vol. 17, no. 3, p. 99, Feb. 2024, doi: 10.3390/a17030099.
[40] R. Perumal, T. Karuppiah, U. Panneerselvam, V. Annamalai, and P. Kaliyaperumal, “Enhancing network security using unsupervised learning approach to combat zero-day attack,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 36, no. 2, pp. 1284–1293, Nov. 2024, doi: 10.11591/IJEECS.V36.I2.PP1284-1293.
[41] C. S. Shieh, F. A. Ho, M. F. Horng, T. T. Nguyen, and P. Chakrabarti, “Open-Set Recognition in Unknown DDoS Attacks Detection With Reciprocal Points Learning,” IEEE Access, vol. 12, pp. 56461–56476, 2024, doi: 10.1109/ACCESS.2024.3388149.
[42] J. Fang and C. Xie, “Unknown intrusion traffic detection method based on unsupervised learning and open-set recognition,” Sci. Rep., vol. 15, no. 1, p. 17001, May 2025, doi: 10.1038/s41598-025-01084-1.
[43] G. Engelen, V. Rimmer, and W. Joosen, “Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study,” 2021. doi: 10.1109/SPW53761.2021.00009.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
