Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems

Ahmed Abusnaina, Afsah Anwar, Sultan Alshamrani, Abdulrahman Alabduljabbar, Rhong Ho Jang, Dae Hun Nyang, David Mohaisen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


The rapid growth of the Internet of Things (IoT) devices is paralleled by them being on the front-line of malicious attacks. This has led to an explosion in the number of IoT malware, with continued mutations, evolution, and sophistication. Malware samples are detected using machine learning (ML) algorithms alongside the traditional signature-based methods. Although ML-based detectors improve the detection performance, they are susceptible to malware evolution and sophistication, making them limited to the patterns that they have been trained upon. This continuous trend motivates large body of literature on malware analysis and detection research, with many systems emerging constantly, outperforming their predecessors. In this paper, we systematically examine the state-of-the-art malware detection approaches, that utilize various representation and learning techniques, under a range of adversarial settings. Our analyses highlight the instability of the proposed detectors in learning patterns that distinguish the benign from the malicious software. The results exhibit that software mutations with functionality-preserving operations, such as stripping and padding, significantly deteriorate the accuracy of such detectors. Additionally, our analysis of the industry-standard malware detectors shows their instability to the malware mutations. Through extensive experiments, we highlight the gap between the capabilities of the adversary and that of the existing malware detectors. The evaluations and analyses show that the optimal malware detection system is nowhere near and calls for the community to streamline their efforts towards testing the robustness of malware detectors to different manipulation techniques.

Original languageEnglish
Title of host publicationProceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022
PublisherAssociation for Computing Machinery
Number of pages13
ISBN (Electronic)9781450397049
StatePublished - 26 Oct 2022
Event25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022 - Limassol, Cyprus
Duration: 26 Oct 202228 Oct 2022

Publication series

NameACM International Conference Proceeding Series


Conference25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022

Bibliographical note

Funding Information:
Our efforts show that malware detectors proposed in the literature are vulnerable to adversarial perturbation and binary manipulation attacks. Similarly, industry-standard malware detectors are prone to such attacks. Our efforts also unveil the status-quo of the existing detectors and bring forward various insights to consider when proposing detection systems. Particularly, in addition to optimizing baseline malware detection accuracy, researchers should consider the robustness of the proposed systems under adversarial capabilities. Investigating the adversarial settings is crucial to understand the drawbacks of implemented malware detection models. In the literature, it has been discussed that incorporating adversarial examples within the training process may increase the model’s robustness. While this is true to some extent, we argue that training on specific adversarial settings and configuration does not guarantee the robustness under different adversarial attacks, nor same attack with different configurations. Due to the large space of adversarial perturbation, it is infeasible to train malware detectors on large set of adversarial attacks. This eventually results in decreased performance, while still vulnerable to various adversarial settings. We note that adversarial attacks exploit poor design choices, obligating for a deep understanding of the underlying learning algorithms and data representations. Acknowledgement. The authors would like to thank anonymous reviewers of RAID’22 for their valuable suggestions and Erman Ayday for shepherding this work. This work was supported by the Global Research Lab (GRL) Program of the National Research Foundation (NRF) funded by the Ministry of Science, Information, and Communication Technologies (ICT), Future Planning (NRF-2016K1A1A2912757), and a seed grant from CyberFlorida. The work was additionally supported by the NRF grant funded by the Korea government (MSIT) (NRF-2020R1A2C2009372).

Publisher Copyright:
© 2022 ACM.


  • Adversarial Machine Learning
  • Robust Malware Detection


Dive into the research topics of 'Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems'. Together they form a unique fingerprint.

Cite this