AN AI BENCHMARK SELECTION FRAMEWORK FOR SUSTAINABLE CYBERSECURITY: COMPARATIVE CHARACTERIZATION OF CIC-IDS2017, UNSW-NB15, AND IOT-23
Article Sidebar
Main Article Content
Abstract
AI is not just a tool for digital transformation, but a vital component in safeguarding the critical digital infrastructure and facilitating sustainable digital transformation. But the choice of inappropriate benchmark datasets can make AI-driven IDS less reliable, transparent, and reproducible, ultimately undermining their role in supporting resilient cybersecurity ecosystems aligned with the SDGs, especially SDG 9 (Industry, Innovation, and Infrastructure) and SDG 16 (Peace, Justice and Strong Institutions). This work proposed four complementary analytical dimensions that were comparatively analysed for three widely used benchmark datasets: dataset characterisation, attack diversity, Mutual Information (MI)-based feature importance, and feature correlation analysis. The findings show significant inter-dataset differences. There are 71,984,818 network records in the IoT-23, much larger than those of CIC-IDS2017 (2,830,743) and UNSW-NB15 (2,540,047), which makes it more suitable for large-scale deep learning research. In addition, CIC-IDS2017 offers the highest feature representation (80 features) and attack diversity (15 attack categories). In comparison, UNSW-NB15 is a good benchmark for feature representation (50 features) and attack diversity (9 attack categories) after class harmonization. The results of the feature importance analysis also reveal significant differences across datasets: statistical flow features are the most important in CIC-IDS2017, communication endpoint features are the most informative in IoT-23, and protocol-related features are the most important in UNSW-NB15. Based on these results, this study proposes a Benchmark Selection Matrix and a Benchmark Selection Framework to help translate the comparative analysis of datasets into an evidence-based decision-making process for specific AI research scenarios.
JEL Classification Codes: C6, C8, C9.
Downloads
Article Details
Section
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
References
Arul, M., Kishore Kumar, R., Santhosh, S., Boomika, M., Bhavana Shree, J., & Shree Durga, K. (2026). A multi-model deep learning approach for advanced cybersecurity threat detection. In Proceedings of the 2026 Second International Conference on Multi-Agent Systems for Collaborative Intelligence (ICMSCI). https://doi.org/10.1109/ICMSCI67830.2026.11469498
Abdo, S., Brits, J., & Akpinar, K. O. (2025). A comparative study on IoT attack detection. In Proceedings of the 2025 2nd International Conference on Artificial Intelligence, Metaverse, and Cybersecurity (ICAMAC). https://doi.org/10.1109/ICAMAC67779.2025.11398518
Alhassan, S., Abdul-Salaam, G., Michael, A., Missah, Y., Ganaa, E., & Shirazu, A. S. (2024). CFS-AE: Correlation-based feature selection and autoencoder for improved intrusion detection system performance. Journal of Internet Services and Information Security, 14(1), 104-120. https://doi.org/10.58346/jisis.2024.i1.007 DOI: https://doi.org/10.58346/JISIS.2024.I1.007
AlMohamad, J. A. (2026). Synergizing explainable AI and federated learning for proactive information security: A novel framework for zero-day threat detection. ECOSOCIAL Studies: Banking, Finance and Cybersecurity, 1(1), 1–13. https://doi.org/10.56334/ecosbankfincyber/8.1.1
Assudani, P., Kumar, N., Mohanambal, K., & Chitra, R. (2025). Explainable artificial intelligence-driven intrusion detection system for enhancing reliability and interpretability in IoT-based network security solutions. Journal of Intelligent Systems and Internet of Things, 17(1), 219-238. https://doi.org/10.54216/jisiot.170116 DOI: https://doi.org/10.54216/JISIoT.170116
Balega, M., Farag, W., Wu, X.-W., Ezekiel, S., & Good, Z. (2024). Enhancing IoT security: Optimizing anomaly detection through machine learning. Electronics, 13(11), 2148. https://doi.org/10.3390/electronics13112148 DOI: https://doi.org/10.3390/electronics13112148
Chinnasamy, R., & Subramanian, M. (2025). An explainable intrusion detection system using novel Indian millipede optimization and WGAN-GP with a dynamic attention-based ensemble model. PeerJ Computer Science, 11, e3278.https://doi.org/10.7717/peerj-cs.3278 DOI: https://doi.org/10.7717/peerj-cs.3278
Chizari, M., Alam, A., Ali Mirza, Q. K., & Chizari, H. (2026). A Tri-Axis Systematic Literature Review of AI-Powered Cyber Defense: ATT&CK-Aligned Analysis of Cyberattacks, Machine Learning Methods, and Datasets. Electronics, 15(13), 2804. https://doi.org/10.3390/electronics15132804
Darwish, R., & Roy, K. (2025). Comparative analysis of federated learning, deep learning, and traditional machine learning techniques for IoT malware detection. In Proceedings of the International Conference on Applied Informatics and Communication (ICAIC). https://doi.org/10.1109/ICAIC63015.2025.10849203 DOI: https://doi.org/10.1109/ICAIC63015.2025.10849203
Dash, N., Chakravarty, S., & Rath, A. (2024). Deep learning model for elevating Internet of Things intrusion detection. International Journal of Electrical and Computer Engineering (IJECE), 14(5), 5874-5883. https://doi.org/10.11591/ijece.v14i5.pp5874-5883 DOI: https://doi.org/10.11591/ijece.v14i5.pp5874-5883
Dhote, S., & Agrawal, D. D. (2026). Machine Learning and Deep Learning-Based Intrusion Detection Systems: A Comprehensive Review of Datasets, Algorithms, Challenges, Explainability, and Future Research Directions. Interdisciplinary Journal of AI, Machine Learning & Data Science, 1(3), e004. https://doi.org/10.66261/tg4asf94
Dubey, A., Pandey, V. K., Shukla, A., Sahu, A., & Prakash, S. (2025). Deep learning-based intrusion detection framework for healthcare IoT networks. In Proceedings of the 2025 International Conference on Decision Aid Sciences and Applications (DASA). https://doi.org/10.1109/DASA68193.2025.11499071
García, P., Curtò, J., & Zarzà, I. D. (2025). Foundation models for tabular intrusion detection: Evaluating TabPFN and LLM few-shot classification on IoT network security. In Proceedings of the 2025 3rd International Conference on Foundation and Large Language Models (FLLM). https://doi.org/10.1109/FLLM67465.2025.11391169
He, M., Wang, X., Wei, P., Yang, L., Teng, Y., & Lyu, R. (2024). Reinforcement learning meets network intrusion detection: A transferable and adaptable framework for anomaly behavior identification. IEEE Transactions on Network and Service Management, 21(2), 2477-2492. https://doi.org/10.1109/TNSM.2024.3352586 DOI: https://doi.org/10.1109/TNSM.2024.3352586
Hejazi, S. M., Alshalabi, A. Y., Hatamleh, M., & Albaroudi, E. (2025). A lightweight hybrid deep learning-based intrusion detection system for detecting botnet attacks in IoT networks. Journal of Scientific Research and Reports, 31(11), 97-120. https://doi.org/10.9734/jsrr/2025/v31i113654 DOI: https://doi.org/10.9734/jsrr/2025/v31i113654
Hleha, K., & Hol, V. (2025). XAI optimization for low-latency neural-based intrusion detection systems in network environments. Bulletin of V. N. Karazin Kharkiv National University, Series "Mathematical Modeling. Information Technology. Automated Control Systems", 66, 19–36. https://doi.org/10.26565/2304-6201-2025-66-02 DOI: https://doi.org/10.26565/2304-6201-2025-66-02
Hoa, N. T. (2025). Intrusion detection in network systems using transformer-based approach. Vinh University Journal of Science ,54(4A), 59-72. https://doi.org/10.56824/vujs.2025a0117a DOI: https://doi.org/10.56824/vujs.2025a0117a
Huang, W., Tian, H., Wang, S., Zhang, C., & Zhang, X. (2024). Integration of simulated annealing into pigeon inspired optimizer algorithm for feature selection in network intrusion detection systems. PeerJ Computer Science, 10, e2176.. https://doi.org/10.7717/peerj-cs.2176 DOI: https://doi.org/10.7717/peerj-cs.2176
Hussein, Z. N., Hammood, D., & Al-Abbasi, Z. (2025). DeepCyber-IDS: A deep learning-based intrusion detection system. In Proceedings of the 2025 VI International Conference on Neural Networks and Neurotechnologies (NeuroNT). https://doi.org/10.1109/NeuroNT66873.2025.11049980 DOI: https://doi.org/10.1109/NeuroNT66873.2025.11049980
Kaliyaperumal, P., Periyasamy, S., Manikandan, T., Balusamy, B., & Benedetto, F. (2024). A novel hybrid unsupervised learning approach for enhanced cybersecurity in the IoT. Future Internet, 16(7), 253. https://doi.org/10.3390/fi16070253 DOI: https://doi.org/10.3390/fi16070253
Kalyana, K. K. (2026). AI-powered real-time CNN-LSTM intrusion detection: From streaming traffic to actionable alerts. International Journal of Scientific Research in Engineering & Technology, 6(2), 123-128. https://doi.org/10.59256/ijsreat.20260602018
Kamal, H., & Mashaly, M. (2025). Enhanced Hybrid Deep Learning Models-Based Anomaly Detection Method for Two-Stage Binary and Multi-Class Classification of Attacks in Intrusion Detection Systems. Algorithms, 18(2), 69. https://doi.org/10.3390/a18020069 DOI: https://doi.org/10.3390/a18020069
Kodete, C. S., Raju, K. B., Karmakonda, K., Sikindar, S., Ramesh, J. V. N., & Tirumanadham, N. K. M. K. (2025). Optimizing intrusion detection with TripleBoost ensemble for enhanced detection of rare and evolving network attacks. International Journal of Electrical and Electronic Engineering & Telecommunications, 14(3), 115-129. https://doi.org/10.18178/ijeetc.14.3.115-129 DOI: https://doi.org/10.18178/ijeetc.14.3.115-129
Kwubeghari, A., & Ezeji, N. G. (2025). Designing an explainable intrusion detection system (X-Ids) using machine learning: a framework for transparency and trust. ABUAD Journal of Engineering Research and Development (AJERD), 8(2), 319-328. https://doi.org/10.53982/ajerd.2025.0802.32-j DOI: https://doi.org/10.53982/ajerd.2025.0802.32-j
Li, L., Zhang, Y., Wang, J., & Ke, X. (2024). Deep learning-based network traffic anomaly detection: A study in IoT environments. World Journal of Innovative Modern Technology, 7(6), 13-26. https://doi.org/10.53469/WJIMT.2024.07(06).03 DOI: https://doi.org/10.53469/wjimt.2024.07(06).03
Mada, Y. M., Bello-Salau, H., Yusuf, S. M., Ahmad, B., Adekale, A., & Dauda, A. (2024). Deep learning-based network intrusion detection system using predator optimization algorithm for feature selection. In Proceedings of the 2024 IEEE 5th International Conference on Electro-Computing Technologies for Humanity (NIGERCON). https://doi.org/10.1109/NIGERCON62786.2024.10927361 DOI: https://doi.org/10.1109/NIGERCON62786.2024.10927361
Mittal, S., & Rajvanshi, P. (2025). Towards a lightweight hybrid deep learning approach for malware detection enhancement in IoT-based systems. In Proceedings of the 2025 8th International Conference on Electronics, Materials Engineering and Nano-Technology (IEMENTech). https://doi.org/10.1109/IEMENTech65115.2025.10959623 DOI: https://doi.org/10.1109/IEMENTech65115.2025.10959623
Memmesheimer, P., Machmeier, S., & Heuveline, V. (2024). Increasing detection rate for imbalanced malicious traffic using generative adversarial networks. In Proceedings of the European Interdisciplinary Cybersecurity Conference. https://doi.org/10.1145/3655693.3655703 DOI: https://doi.org/10.1145/3655693.3655703
Mohamed, M. A., Emary, E., & Attalla, M. A. (2025). Improved IoT anomaly detection through hybrid machine learning and deep learning approaches using the IoT-23 dataset. In Proceedings of the International Conference on Computing Advancements (ICCA). https://doi.org/10.1109/ICCA66035.2025.11430814
Prajwalasimha, S. N., Shelke, N., Saini, D. K. J. B., Pimpalkar, A., Tadkal, S., & Balla, R. (2025). Hybrid transformer-CNN neuro-symbolic explainable AI for cyber threat intelligence: Advancing transparency and adversarial robustness. In Proceedings of the 2025 3rd International Conference on Intelligent Cyber Physical Systems and Internet of Things (ICoICI). https://doi.org/10.1109/ICoICI65217.2025.11254796 DOI: https://doi.org/10.1109/ICoICI65217.2025.11254796
Nugroho, K. A., Hariguna, T., & Barkah, A. S. (2025). Optimizing Early Network Intrusion Detection: A Comparison of LSTM and LinearSVC with SMOTE on Imbalanced Data. Jurnal Teknik Informatika (Jutif), 6(6), 5349-5370. https://doi.org/10.52436/1.jutif.2025.6.6.4672 DOI: https://doi.org/10.52436/1.jutif.2025.6.6.4672
Nzuva, S. M., Nder, L., & Mwalili, T. (2024). A novel bagging-XGBoost ensemble model for attaining high accuracy and computational efficiency in network intrusion detection. E3S Web of Conferences . https://doi.org/10.1051/e3sconf/202450101007 DOI: https://doi.org/10.2139/ssrn.4944137
Oziegbe, T. E., Edje, A. E., & Akazue, M. (2026). DEEP LEARNING–BASED INTRUSION DETECTION IN VEHICULAR NETWORKS: A REVIEW OF GATED RECURRENT UNIT APPROACHES. Science World Journal, 21(1), 264-272. https://doi.org/10.4314/swj.v21i1.37
Pinto, D., Vitorino, J., Maia, E., Amorim, I., & Praça, I. (2024). Flow exporter impact on intelligent intrusion detection systems. In Proceedings of the International Conference on Information Systems Security and Privacy. https://doi.org/10.48550/arXiv.2412.14021 DOI: https://doi.org/10.5220/0013131900003899
Rajasa, M. C., Rahma, F., Rachmadi, R. F., Pratomo, B., & Purnomo, M. (2023). A review of imbalanced datasets and resampling techniques in network intrusion detection systems. In Proceedings of the 2023 8th International Conference on Information Technology and Digital Applications (ICITDA). https://doi.org/10.1109/ICITDA60835.2023.10427217 DOI: https://doi.org/10.1109/ICITDA60835.2023.10427217
Ren, Y. (2026). Data science and machine learning for cyber intrusion detection: a systematic review. Mach Learn Res, 11(1), 8-21. https://doi.org/10.11648/j.mlr.20261101.12
Rai, I. N. A. S., Heryadi, D., Yani, Y. M., & Nashir, A. K. (2025). RETHINKING EUROPEAN CYBERSECURITY INTEGRATION THROUGH LIBERAL INTERGOVERNMENTAL POLITICS IN GENERAL DATA PROTECTION REGULATION ENFORCEMENT STUDY. Bangladesh Journal of Multidisciplinary Scientific Research, 11(1), 23-34. https://doi.org/10.46281/bjmsr.v11i1.2638 DOI: https://doi.org/10.46281/bjmsr.v11i1.2638
Sagaran, E., Spier, J., & Hasan, R. (2025). SpiCAE: Spiking contrastive learning and autoencoder for network intrusion detection. In Proceedings of the IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON). https://doi.org/10.1109/IEMCON67450.2025.11381151 DOI: https://doi.org/10.1109/IEMCON67450.2025.11381151
Sah, G., Singh, S., & Banerjee, S. (2024). Intrusion detection system using classification algorithms with feature selection mechanism over real-time data traffic. China Communications, 21(9), 292-320. https://doi.org/10.23919/JCC.fa.2022-0076.202409 DOI: https://doi.org/10.23919/JCC.fa.2022-0076.202409
Saini, N., Bhat Kasaragod, V., Prakasha, K., & Das, A. K. (2023). A hybrid ensemble machine learning model for detecting APT attacks based on network behavior anomaly detection. Concurrency and Computation: Practice and Experience, 35(28), e7865. https://doi.org/10.1002/cpe.7865 DOI: https://doi.org/10.1002/cpe.7865
Sharma, V., & Kumar, M. (2025). Improving intrusion detection with hybrid deep learning models: A study on CIC-IDS2017, UNSW-NB15, and KDD CUP 99. Journal of Information Systems Engineering and Management, 10(11S), 633-650. https://doi.org/10.52783/jisem.v10i11s.1665 DOI: https://doi.org/10.52783/jisem.v10i11s.1665
Sravani, A., Sri, M. R., & Sanjana, C. (2026). Evaluating single-model and ensemble-based intrusion detection on the CIC-IDS2017 dataset. International Scientific Journal of Engineering & Management, 5(3), 1-6. https://doi.org/10.55041/isjem05901
Sreerenjith, P., & Benitta, D. (2026). Cloud-native intrusion detection and DDoS attack mitigation using federated deep learning on AWS Free Tier. In Proceedings of the 2026 3rd International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE). https://doi.org/10.1109/RMKMATE69073.2026.11519014 DOI: https://doi.org/10.1109/RMKMATE69073.2026.11519014
Tamuka, N., Mathonsi, T., Olwal, T., Maswikaneng, S., Muchenje, T., & Tshilongamulenzhe, T. (2026). Intrusion detection in fog computing: A systematic review of security advances and challenges. Computers, 15(3), 169. https://doi.org/10.3390/computers15030169
Thomas, R., Chaturvedi, A., & Goswami, D. N. (2025). The role of AI-enabled optimization in network traffic management. International Journal for Sciences and Technology ,16(2), 1-15. https://doi.org/10.71097/ijsat.v16.i2.3428 DOI: https://doi.org/10.71097/IJSAT.v16.i2.3428
Tian, W., Shen, Y., Guo, N., Yuan, J., & Yang, Y. (2024). VAE-WACGAN: An Improved Data Augmentation Method Based on VAEGAN for Intrusion Detection. Sensors, 24(18), 6035. https://doi.org/10.3390/s24186035 DOI: https://doi.org/10.3390/s24186035
Xie, H., Shao, Y., Li, Z., Alomari, Z., & Makanju, A. (2025). Optimization of class imbalance techniques in machine learning models for network intrusion detection. In Proceedings of the International Conference on Cryptography, Security and Privacy. https://doi.org/10.1109/CSP66295.2025.00025 DOI: https://doi.org/10.1109/CSP66295.2025.00025
Xu, L., Wang, L., & Jiang, Y. (2026). Optimized Deep Learning-Based Intrusion Detection System Using SMOTE and Genetic Algorithms. International Journal of Pattern Recognition and Artificial Intelligence, 40(3), 2552032. https://doi.org/10.1142/S0218001425520329 DOI: https://doi.org/10.1142/S0218001425520329
Zhang, C., Li, J., Wang, N., & Zhang, D. (2025). Research on intrusion detection methods based on transformer and CNN-BiLSTM in the Internet of Things. Sensors, 25(9), 2725. https://doi.org/10.3390/s25092725 DOI: https://doi.org/10.3390/s25092725
Zhang, X., & Tuo, J. (2025). Research on network intrusion detection of automation systems based on machine learning. In Proceedings of the 2025 10th International Conference on Electronic Technology and Information Science (ICETIS). https://doi.org/10.1109/ICETIS66286.2025.11144041 DOI: https://doi.org/10.1109/ICETIS66286.2025.11144041