Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems

Ahmed Abusnaina; Afsah Anwar; Sultan Alshamrani; Abdulrahman Alabduljabbar; Rhong Ho Jang; Dae Hun Nyang; David Mohaisen

doi:10.1145/3545948.3545960

Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems

Ahmed Abusnaina, Afsah Anwar, Sultan Alshamrani, Abdulrahman Alabduljabbar, Rhong Ho Jang, Dae Hun Nyang, David Mohaisen

Department of Cyber Security

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The rapid growth of the Internet of Things (IoT) devices is paralleled by them being on the front-line of malicious attacks. This has led to an explosion in the number of IoT malware, with continued mutations, evolution, and sophistication. Malware samples are detected using machine learning (ML) algorithms alongside the traditional signature-based methods. Although ML-based detectors improve the detection performance, they are susceptible to malware evolution and sophistication, making them limited to the patterns that they have been trained upon. This continuous trend motivates large body of literature on malware analysis and detection research, with many systems emerging constantly, outperforming their predecessors. In this paper, we systematically examine the state-of-the-art malware detection approaches, that utilize various representation and learning techniques, under a range of adversarial settings. Our analyses highlight the instability of the proposed detectors in learning patterns that distinguish the benign from the malicious software. The results exhibit that software mutations with functionality-preserving operations, such as stripping and padding, significantly deteriorate the accuracy of such detectors. Additionally, our analysis of the industry-standard malware detectors shows their instability to the malware mutations. Through extensive experiments, we highlight the gap between the capabilities of the adversary and that of the existing malware detectors. The evaluations and analyses show that the optimal malware detection system is nowhere near and calls for the community to streamline their efforts towards testing the robustness of malware detectors to different manipulation techniques.

Original language	English
Title of host publication	Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022
Publisher	Association for Computing Machinery
Pages	308-320
Number of pages	13
ISBN (Electronic)	9781450397049
DOIs	https://doi.org/10.1145/3545948.3545960
State	Published - 26 Oct 2022
Event	25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022 - Limassol, Cyprus Duration: 26 Oct 2022 → 28 Oct 2022

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022
Country/Territory	Cyprus
City	Limassol
Period	26/10/22 → 28/10/22

Bibliographical note

Funding Information:
Our efforts show that malware detectors proposed in the literature are vulnerable to adversarial perturbation and binary manipulation attacks. Similarly, industry-standard malware detectors are prone to such attacks. Our efforts also unveil the status-quo of the existing detectors and bring forward various insights to consider when proposing detection systems. Particularly, in addition to optimizing baseline malware detection accuracy, researchers should consider the robustness of the proposed systems under adversarial capabilities. Investigating the adversarial settings is crucial to understand the drawbacks of implemented malware detection models. In the literature, it has been discussed that incorporating adversarial examples within the training process may increase the model’s robustness. While this is true to some extent, we argue that training on specific adversarial settings and configuration does not guarantee the robustness under different adversarial attacks, nor same attack with different configurations. Due to the large space of adversarial perturbation, it is infeasible to train malware detectors on large set of adversarial attacks. This eventually results in decreased performance, while still vulnerable to various adversarial settings. We note that adversarial attacks exploit poor design choices, obligating for a deep understanding of the underlying learning algorithms and data representations. Acknowledgement. The authors would like to thank anonymous reviewers of RAID’22 for their valuable suggestions and Erman Ayday for shepherding this work. This work was supported by the Global Research Lab (GRL) Program of the National Research Foundation (NRF) funded by the Ministry of Science, Information, and Communication Technologies (ICT), Future Planning (NRF-2016K1A1A2912757), and a seed grant from CyberFlorida. The work was additionally supported by the NRF grant funded by the Korea government (MSIT) (NRF-2020R1A2C2009372).

Publisher Copyright:
© 2022 ACM.

Keywords

Adversarial Machine Learning
Robust Malware Detection

Access to Document

10.1145/3545948.3545960

Cite this

Abusnaina, A., Anwar, A., Alshamrani, S., Alabduljabbar, A., Jang, R. H., Nyang, D. H., & Mohaisen, D. (2022). Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems. In Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022 (pp. 308-320). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3545948.3545960

@inproceedings{4e1d3a97be58470cbc57bf9861b2f3b4,

title = "Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems",

abstract = "The rapid growth of the Internet of Things (IoT) devices is paralleled by them being on the front-line of malicious attacks. This has led to an explosion in the number of IoT malware, with continued mutations, evolution, and sophistication. Malware samples are detected using machine learning (ML) algorithms alongside the traditional signature-based methods. Although ML-based detectors improve the detection performance, they are susceptible to malware evolution and sophistication, making them limited to the patterns that they have been trained upon. This continuous trend motivates large body of literature on malware analysis and detection research, with many systems emerging constantly, outperforming their predecessors. In this paper, we systematically examine the state-of-the-art malware detection approaches, that utilize various representation and learning techniques, under a range of adversarial settings. Our analyses highlight the instability of the proposed detectors in learning patterns that distinguish the benign from the malicious software. The results exhibit that software mutations with functionality-preserving operations, such as stripping and padding, significantly deteriorate the accuracy of such detectors. Additionally, our analysis of the industry-standard malware detectors shows their instability to the malware mutations. Through extensive experiments, we highlight the gap between the capabilities of the adversary and that of the existing malware detectors. The evaluations and analyses show that the optimal malware detection system is nowhere near and calls for the community to streamline their efforts towards testing the robustness of malware detectors to different manipulation techniques.",

keywords = "Adversarial Machine Learning, Robust Malware Detection",

author = "Ahmed Abusnaina and Afsah Anwar and Sultan Alshamrani and Abdulrahman Alabduljabbar and Jang, {Rhong Ho} and Nyang, {Dae Hun} and David Mohaisen",

note = "Funding Information: Our efforts show that malware detectors proposed in the literature are vulnerable to adversarial perturbation and binary manipulation attacks. Similarly, industry-standard malware detectors are prone to such attacks. Our efforts also unveil the status-quo of the existing detectors and bring forward various insights to consider when proposing detection systems. Particularly, in addition to optimizing baseline malware detection accuracy, researchers should consider the robustness of the proposed systems under adversarial capabilities. Investigating the adversarial settings is crucial to understand the drawbacks of implemented malware detection models. In the literature, it has been discussed that incorporating adversarial examples within the training process may increase the model{\textquoteright}s robustness. While this is true to some extent, we argue that training on specific adversarial settings and configuration does not guarantee the robustness under different adversarial attacks, nor same attack with different configurations. Due to the large space of adversarial perturbation, it is infeasible to train malware detectors on large set of adversarial attacks. This eventually results in decreased performance, while still vulnerable to various adversarial settings. We note that adversarial attacks exploit poor design choices, obligating for a deep understanding of the underlying learning algorithms and data representations. Acknowledgement. The authors would like to thank anonymous reviewers of RAID{\textquoteright}22 for their valuable suggestions and Erman Ayday for shepherding this work. This work was supported by the Global Research Lab (GRL) Program of the National Research Foundation (NRF) funded by the Ministry of Science, Information, and Communication Technologies (ICT), Future Planning (NRF-2016K1A1A2912757), and a seed grant from CyberFlorida. The work was additionally supported by the NRF grant funded by the Korea government (MSIT) (NRF-2020R1A2C2009372). Publisher Copyright: {\textcopyright} 2022 ACM.; 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022 ; Conference date: 26-10-2022 Through 28-10-2022",

year = "2022",

month = oct,

day = "26",

doi = "10.1145/3545948.3545960",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "308--320",

booktitle = "Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022",

}

Abusnaina, A, Anwar, A, Alshamrani, S, Alabduljabbar, A, Jang, RH, Nyang, DH & Mohaisen, D 2022, Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems. in Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 308-320, 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022, Limassol, Cyprus, 26/10/22. https://doi.org/10.1145/3545948.3545960

Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems. / Abusnaina, Ahmed; Anwar, Afsah; Alshamrani, Sultan et al.
Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022. Association for Computing Machinery, 2022. p. 308-320 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems

AU - Abusnaina, Ahmed

AU - Anwar, Afsah

AU - Alshamrani, Sultan

AU - Alabduljabbar, Abdulrahman

AU - Jang, Rhong Ho

AU - Nyang, Dae Hun

AU - Mohaisen, David

N1 - Funding Information: Our efforts show that malware detectors proposed in the literature are vulnerable to adversarial perturbation and binary manipulation attacks. Similarly, industry-standard malware detectors are prone to such attacks. Our efforts also unveil the status-quo of the existing detectors and bring forward various insights to consider when proposing detection systems. Particularly, in addition to optimizing baseline malware detection accuracy, researchers should consider the robustness of the proposed systems under adversarial capabilities. Investigating the adversarial settings is crucial to understand the drawbacks of implemented malware detection models. In the literature, it has been discussed that incorporating adversarial examples within the training process may increase the model’s robustness. While this is true to some extent, we argue that training on specific adversarial settings and configuration does not guarantee the robustness under different adversarial attacks, nor same attack with different configurations. Due to the large space of adversarial perturbation, it is infeasible to train malware detectors on large set of adversarial attacks. This eventually results in decreased performance, while still vulnerable to various adversarial settings. We note that adversarial attacks exploit poor design choices, obligating for a deep understanding of the underlying learning algorithms and data representations. Acknowledgement. The authors would like to thank anonymous reviewers of RAID’22 for their valuable suggestions and Erman Ayday for shepherding this work. This work was supported by the Global Research Lab (GRL) Program of the National Research Foundation (NRF) funded by the Ministry of Science, Information, and Communication Technologies (ICT), Future Planning (NRF-2016K1A1A2912757), and a seed grant from CyberFlorida. The work was additionally supported by the NRF grant funded by the Korea government (MSIT) (NRF-2020R1A2C2009372). Publisher Copyright: © 2022 ACM.

PY - 2022/10/26

Y1 - 2022/10/26

N2 - The rapid growth of the Internet of Things (IoT) devices is paralleled by them being on the front-line of malicious attacks. This has led to an explosion in the number of IoT malware, with continued mutations, evolution, and sophistication. Malware samples are detected using machine learning (ML) algorithms alongside the traditional signature-based methods. Although ML-based detectors improve the detection performance, they are susceptible to malware evolution and sophistication, making them limited to the patterns that they have been trained upon. This continuous trend motivates large body of literature on malware analysis and detection research, with many systems emerging constantly, outperforming their predecessors. In this paper, we systematically examine the state-of-the-art malware detection approaches, that utilize various representation and learning techniques, under a range of adversarial settings. Our analyses highlight the instability of the proposed detectors in learning patterns that distinguish the benign from the malicious software. The results exhibit that software mutations with functionality-preserving operations, such as stripping and padding, significantly deteriorate the accuracy of such detectors. Additionally, our analysis of the industry-standard malware detectors shows their instability to the malware mutations. Through extensive experiments, we highlight the gap between the capabilities of the adversary and that of the existing malware detectors. The evaluations and analyses show that the optimal malware detection system is nowhere near and calls for the community to streamline their efforts towards testing the robustness of malware detectors to different manipulation techniques.

AB - The rapid growth of the Internet of Things (IoT) devices is paralleled by them being on the front-line of malicious attacks. This has led to an explosion in the number of IoT malware, with continued mutations, evolution, and sophistication. Malware samples are detected using machine learning (ML) algorithms alongside the traditional signature-based methods. Although ML-based detectors improve the detection performance, they are susceptible to malware evolution and sophistication, making them limited to the patterns that they have been trained upon. This continuous trend motivates large body of literature on malware analysis and detection research, with many systems emerging constantly, outperforming their predecessors. In this paper, we systematically examine the state-of-the-art malware detection approaches, that utilize various representation and learning techniques, under a range of adversarial settings. Our analyses highlight the instability of the proposed detectors in learning patterns that distinguish the benign from the malicious software. The results exhibit that software mutations with functionality-preserving operations, such as stripping and padding, significantly deteriorate the accuracy of such detectors. Additionally, our analysis of the industry-standard malware detectors shows their instability to the malware mutations. Through extensive experiments, we highlight the gap between the capabilities of the adversary and that of the existing malware detectors. The evaluations and analyses show that the optimal malware detection system is nowhere near and calls for the community to streamline their efforts towards testing the robustness of malware detectors to different manipulation techniques.

KW - Adversarial Machine Learning

KW - Robust Malware Detection

UR - http://www.scopus.com/inward/record.url?scp=85142502223&partnerID=8YFLogxK

U2 - 10.1145/3545948.3545960

DO - 10.1145/3545948.3545960

M3 - Conference contribution

AN - SCOPUS:85142502223

T3 - ACM International Conference Proceeding Series

SP - 308

EP - 320

BT - Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022

PB - Association for Computing Machinery

T2 - 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022

Y2 - 26 October 2022 through 28 October 2022

ER -

Abusnaina A, Anwar A, Alshamrani S, Alabduljabbar A, Jang RH, Nyang DH et al. Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems. In Proceedings of 25th International Symposium on Researchin Attacks, Intrusions and Defenses, RAID 2022. Association for Computing Machinery. 2022. p. 308-320. (ACM International Conference Proceeding Series). doi: 10.1145/3545948.3545960

Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this