A Systematic Review of Agentic AI for Threat Detection and Mitigation in 5G Networks
DOI:
https://doi.org/10.63158/journalisi.v8i1.1382Keywords:
Agentic Artificial Intelligence, 5G Networks, Threat Detection, Autonomous Agents, Reinforcement LearningAbstract
Fifth-generation (5G) networks face escalating security challenges driven by decentralised architectures, stringent ultra-low-latency requirements, and rapidly evolving threat landscapes. Agentic Artificial Intelligence (agentic AI) autonomous systems that perceive network conditions, decide on countermeasures, and act in real time offers a promising route toward adaptive defence. This systematic review examines how agentic AI is being applied to detect and mitigate threats within 5G networks. Following PRISMA 2009 guidelines, four databases (IEEE Xplore, ACM Digital Library, SpringerLink, and ScienceDirect) were searched, yielding 22 eligible peer-reviewed studies published between 2020 and 2025, selected for explicit 5G relevance and empirical evaluation. The reviewed evidence clusters into four primary security areas: anomaly detection, DDoS mitigation, network slicing security, and intrusion detection. Across these domains, approaches based on federated learning, deep reinforcement learning, and multi-agent systems generally report stronger detection performance and/or more adaptive response behaviour than conventional, reactive baselines, while supporting privacy-preserving intelligence at the edge. However, key deployment barriers remain: 86% of studies rely on simulation-based validation, scalability beyond 100 nodes is insufficiently characterised, and reported coordination delays (120–180 ms) may conflict with 5G latency constraints in time-critical settings. To consolidate findings, this review proposes a Perception–Decision–Action–Feedback conceptual framework and highlights priorities for real-world validation and deployment-oriented evaluation.
Downloads
References
[1] S. Sharma, I. You, and M. Atiquzzaman, “Security, Privacy and Reliability in 5G Networks,” IEEE Commun. Surv. Tutorials, 2020, doi: 10.1109/COMST.2020.2991754.
[2] Z. Zhang et al., “6G Wireless Networks: Vision, Requirements, Architecture, and Key Technologies,” IEEE Veh. Technol. Mag., 2020, doi: 10.1109/MVT.2019.2952109.
[3] N. Panwar and S. Sharma, “5G Enabled Internet of Things: Security and Privacy Issues,” IEEE Sens. J., 2021, doi: 10.1109/JSEN.2021.3050832.
[4] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Pearson, 2020.
[5] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 2018.
[6] M. Wooldridge, An Introduction to MultiAgent Systems. Wiley, 2009.
[7] I. B. M. Research, “Agentic AI and Autonomous Systems White Paper,” 2025.
[8] M. Eriksson and J. Holmberg, “Towards Autonomous Networks with Agentic AI,” Ericsson Technol. Rev., 2025.
[9] S. Chishakwe, N. Moyo, B. M. Ndlovu, and S. Dube, “Intrusion Detection System for IoT environments using Machine Learning Techniques,” 2022 1st Zimbabwe Conf. Inf. Commun. Technol. ZCICT 2022, pp. 1–7, 2022, doi: 10.1109/ZCICT55726.2022.10045992.
[10] S. Kaur, N. Gupta, and R. Kumar, “A Systematic Review of AI and ML Techniques for Cybersecurity,” Comput. Secur., 2023.
[11] L. U. Khan, I. Yaqoob, M. Imran, and Z. Khan, “AI and Machine Learning for 5G Security: A Comprehensive Survey,” IEEE Commun. Surv. Tutorials, 2023.
[12] M. Nasralla and R. Hassan, “A Systematic Literature Review on 5G Security Challenges and Machine Learning-Based Solutions,” IEEE Access, 2022.
[13] S. Hussain, M. H. Rehman, and R. Sana, “Deep Learning in Intrusion Detection Systems: A Systematic Review,” ACM Comput. Surv., 2023.
[14] A. Ferdowsi and W. Saad, “A Comprehensive Survey on Machine Learning for Wireless Security,” IEEE Commun. Surv. Tutorials, 2019.
[15] S. Alghamdi, “Security Challenges in 5G Networks: A Survey,” Sensors, vol. 21, no. 24, 2021.
[16] M.-T. Nguyen, “A Survey of Reinforcement Learning Applications in Communication Networks,” IEEE Access, 2020.
[17] N. Kaur, N. Gupta, and R. Kumar, “A Systematic Review of AI and Machine Learning Techniques for Cybersecurity,” Comput. Secur., 2023, doi: 10.1016/j.cose.2023.103248.
[18] D. Moher, A. Liberati, J. Tetzlaff, and D. G. Altman, “Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement,” BMJ, vol. 339, no. 7716, pp. 332–336, 2009, doi: 10.1136/bmj.b2535.
[19] D. and L. Moher Alessandro and Tetzlaff, Jennifer and Altman, Douglas G., “Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement,” PLoS Med., vol. 6, no. 7, p. e1000097, 2009.
[20] J. McGowan, M. Sampson, and D. M. Salzwedel, “PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement,” J. Clin. Epidemiol., vol. 75, pp. 40–46, 2016, doi: 10.1016/j.jclinepi.2016.01.021.
[21] B. Kitchenham, D. Budgen, and P. Brereton, Evidence-Based Software Engineering and Systematic Reviews. Boca Raton, FL: CRC Press, 2015.
[22] A. Liberati et al., “The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration,” BMJ, vol. 339, p. b2700, 2009, doi: 10.1136/bmj.b2700.
[23] P. F. Saura, J. M. Bernabé Murcia, E. G. de la Calera Molina, A. Molina Zarca, J. Bernal Bernabé, and A. F. Skarmeta Gómez, “Federated Network Intelligence Orchestration for Scalable and Automated FL-Based Anomaly Detection in B5G Networks,” Comput. Mater. Contin., vol. 80, no. 1, pp. 163–193, 2024, doi: https://doi.org/10.32604/cmc.2024.051307.
[24] R. Moreira et al., “An intelligent native network slicing security architecture empowered by federated learning,” Futur. Gener. Comput. Syst., vol. 163, p. 107537, 2025, doi: https://doi.org/10.1016/j.future.2024.107537.
[25] J. Á. Fernández-Carrasco, L. Segurola-Gil, F. Zola, and R. Orduna-Urrutia, “Security and 5G: Attack mitigation using Reinforcement Learning in SDN networks,” in 2022 IEEE Future Networks World Forum (FNWF), 2022, pp. 622–627, doi: 10.1109/FNWF55208.2022.00114.
[26] A. Arun, C. H. Basha, M. J. Rani, V. Jamuna, P. Vijayakumar, and K. J. Kumar, “Deep Reinforcement Learning-Based Intrusion Detection System for Next-Generation Wireless Networks,” in 2025 6th International Conference on Inventive Research in Computing Applications (ICIRCA), 2025, pp. 283–291, doi: 10.1109/ICIRCA65293.2025.11089597.
[27] F. Alalyan, M. Awad, W. Jaafar, and R. Langar, “Secure Distributed Federated Learning for Cyberattacks Detection in B5G Open Radio,” vol. 6, no. November 2024, pp. 3067–3081, 2025.
[28] A. Kaur, “Intrusion Detection Approach for Industrial Internet of Things Traffic Using Deep Recurrent,” IEEE Trans. Artif. Intell., vol. 6, no. 1, pp. 37–50, 2025, doi: 10.1109/TAI.2024.3443787.
[29] H. Sharma, N. Kumar, and R. Tekchandani, “Mitigating Jamming Attack in 5G Heterogeneous Networks : A Federated Deep Reinforcement Learning Approach,” IEEE Trans. Veh. Technol., vol. 72, no. 2, pp. 2439–2452, 2023, doi: 10.1109/TVT.2022.3212966.
[30] K. Bedda, Z. Fadlullah, and M. M. Fouda, “Efficient Wireless Network Slicing in 5G Networks : An Asynchronous Federated Learning Approach,” 2022 IEEE Int. Conf. Internet Things Intell. Syst., pp. 285–289, 2022, doi: 10.1109/IoTaIS56727.2022.9976007.
[31] A. Thantharate, R. Paropkari, V. Walunj, C. Beard, and P. Kankariya, “Secure5G : A Deep Learning Framework Towards a Secure Network Slicing in 5G and Beyond,” 2020 10th Annu. Comput. Commun. Work. Conf., pp. 852–857, 2020, doi: 10.1109/CCWC47524.2020.9031158.
[32] E. Ossongo, M. Esseghir, and L. M. Ieee, “A multi-agent Federated Reinforcement Learning-based optimization of quality of service in various LoRa network slices,” 2023.
[33] L. Pu et al., “Federated Learning-based Heterogeneous Load Prediction and Slicing for 5G Systems and Beyond,” in GLOBECOM 2022 - 2022 IEEE Global Communications Conference, 2022, pp. 166–172, doi: 10.1109/GLOBECOM48099.2022.10000870.
[34] S. Wijethilaka and M. Liyanage, “A Federated Learning Approach for Improving Security in Network Slicing.”
[35] Y. Cui, H. Shi, R. Wang, and P. He, “Multi-Agent Reinforcement Learning for Slicing Resource Allocation in Vehicular Networks,” IEEE Trans. Intell. Transp. Syst., vol. 25, no. 2, pp. 2005–2016, 2024, doi: 10.1109/TITS.2023.3314929.
[36] D. K. Dake, “DDoS and Flash Event Detection in Higher Bandwidth SDN-IoT using Multiagent Reinforcement Learning,” pp. 16–20, 2021, doi: 10.1109/ICCMA53594.2021.00011.
[37] F. Alalyan, B. Bousalem, W. Jaafar, and R. Langar, “Secure peer-to-peer federated learning for efficient cyberattacks detection in 5G and beyond networks,” ICC 2024 - IEEE International Conference on Communications. IEEE, pp. 1752–1757, 2024.
[38] H. Sedjelmaci and A. Boualouache, “When Two-Layer Federated Learning and Mean-Field Game Meet 5G and Beyond Security : Cooperative Defense Systems for 5G and Beyond Network Slicing.”
[39] M. Kiely et al., “Exploring the Efficacy of Multi-Agent Reinforcement Learning for Autonomous Cyber Defence : A CAGE Challenge 4 Perspective,” no. 2024, 2025.
[40] P. Radoglou-Grammatikis et al., “Strategic Honeypot Deployment in Ultra-Dense Beyond 5G Networks: A Reinforcement Learning Approach,” IEEE Trans. Emerg. Top. Comput., vol. 12, no. 2, pp. 643–655, 2024, doi: 10.1109/TETC.2022.3184112.
[41] A. Villafranca, K. M. Thant, and I. Tasic, “AI-Enabled IoT Intrusion Detection : Unified Conceptual Framework and Research Roadmap,” pp. 1–38, 2025.
[42] D. Cajaraville-aboy et al., “CO-DEFEND : Continuous Decentralized Federated Learning for Secure DoH-Based Threat Detection,” pp. 1–17.
[43] S. Khozam, G. Blanc, S. Tixeuil, and E. Totel, “DDoS Mitigation while Preserving QoS: A Deep Reinforcement Learning-Based Approach,” in 2024 IEEE 10th International Conference on Network Softwarization (NetSoft), 2024, pp. 369–374, doi: 10.1109/NetSoft60951.2024.10588889.
[44] B. Bousalem, M. A. Sakka, V. F. Silva, W. Jaafar, A. Ben Letaifa, and R. Langar, “DDoS Attacks Mitigation in 5G-V2X Networks: A Reinforcement Learning-Based Approach,” in 2023 19th International Conference on Network and Service Management (CNSM), 2023, pp. 1–5, doi: 10.23919/CNSM59352.2023.10327917.
[45] B. Ndlovu and K. Maguraushe, “Balancing ethics and privacy in the use of artificial intelligence in institutions of higher learning: A framework for responsive AI systems,” Indones. J. Informatics Educ., vol. 9, no. 1, 2025, doi: 10.20961/ijie.v9i1.100723.
[46] J. Mutengeni, A. Musasa, and B. Mutunhu, “Local Area Network Based Collaboration Using Distributed Computing,” 2022 1st Zimbabwe Conf. Inf. Commun. Technol. ZCICT 2022, pp. 1–7, 2022, doi: 10.1109/ZCICT55726.2022.10045858.
[47] P. Vareta, H. Muzenda, T. Nyamupaguma, Y. Dube, and B. Ndlovu, “The Rise of Quantum Computing and Its Impact on Cybersecurity,” Indones. J. Comput. Sci., vol. 14, no. 6, 2025, doi: 10.33022/ijcs.v14i6.5040.
[48] P. Maitireni, V. Ncube, B. Ndlovu, and T. Sibanda, “Quantum Computing Cryptography : A Systematic Review of Innovations , Applications , Challenges , and Algorithms,” vol. 7, no. 4, pp. 3668–3710, 2025, doi: 10.63158/journalisi.v7i4.1331.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
