Malware Detection Using Deep Learning Approach

Abstract

The rapid growth of malicious software is one of the major challenges in modern cybersecurity.Traditional signature-based antivirus systems are efficient, but they struggle to detect new, obfuscated, or polymorphic malware variants. This thesis investigates the application of deep learning to the automated detection and classification of malware targeting the Windows Portable Executable (PE) file format across three representation modalities and four benchmark datasets. Three deep learning architectures are proposed and evaluated: a ResNet-SE Convolutional Neural Network applied to greyscale byte-visualisation images of PE binaries on the Malimg dataset; a Deep Residual DNN trained on the pre-vectorised structural feature set of the EMBER 2018 benchmark; and a CNN-BiLSTM with Multi-Head Attention processing dynamic API call sequences on the APIMDS and Mal-API-2019 datasets. All deep learning models are systematically benchmarked against classical machine learning baselines — Random Forest, SVM, and XGBoost — under consistent experimental protocols. Experimental results demonstrate that the ResNet-SE CNN achieves 99.07% classification accuracy across 25 malware families on the Malimg corpus (MCC = 0.9891), the DNN ResidualBN attains 96.85% accuracy with a ROC-AUC of 0.9930 on EMBER 2018 binary detection, and the CNN-BiLSTM + Attention model reaches 55.00% accuracy (Macro F1 = 0.5716) on the 8-class Mal-API-2019 task — surpassing the published baseline of Çatak et al. (47%) with statistical significance (p = 0.0037). Critical engineering contributions include the identification of Log1P compression as a prerequisite for stable deep network training on heavy-tailed tabular features, and a class weight calibration methodology for extreme-imbalance binary detection. The thesis further analyses the semantic equivalence ceiling in API-based family classification, the inductive bias advantage of tree models on fixed-length positional sequence data, and six directions for future research.