Evolving Evasive Malware: Using Genetic Algorithms to Test and Improve Machine Learning Detectors
Aim
To develop a proof-of-concept framework that uses a genetic algorithm to generate simple evasive variants of a known malware family and evaluate how these variants affect the performance of a standard machine learning classifier, thereby demonstrating the principles of adversarial learning.
Objectives
1. Prepare a dataset and train a baseline classifier by extracting features from labelled benign and malicious executable files from a single, well-documented malware family, and training a machine learning model (e.g., Random Forest) to distinguish between them. 2. Design and implement a genetic algorithm that applies basic, predefined "mutations" (e.g., inserting dead code, reordering instructions) to a malware sample's feature representation with the goal of evading the baseline classifier. 3. Generate and analyse evasive variants by running the genetic algorithm to produce a set of successful mutants, and analyse which types of mutations are most effective at causing misclassification. 4. Evaluate adversarial training by augmenting the original training data with successful evasive variants, retraining the classifier, and comparing its robustness against the baseline when tested on new evolved variants.
Deliverables
a. A working Python-based framework integrating a machine learning classifier with a genetic algorithm for generating evasive malware variants. b. A baseline malware detection classifier with documented performance metrics. c. An analysis report detailing the types of mutations generated by the GA and their effectiveness. d. A comparative evaluation showing the performance of the original classifier versus the retrained (adversarially-trained) classifier. e. Final project dissertation and presentation.
Research Questions
A. Can a simple genetic algorithm, operating on a limited feature set, reliably generate variants of a known malware family that evade a standard machine learning detector? B. What types of feature mutations (e.g., adding specific API calls, inserting sequences of dead code) are most effective at causing misclassification? C. To what extent does retraining the classifier with evolved variants improve its robustness against subsequent, unseen evolved variants?
Methodology
The project will follow an experimental, iterative design methodology. Literature Review: The student will begin by reviewing foundational papers on malware detection, adversarial machine learning, and evolutionary computing, including the supervisor's prior work. Dataset Preparation (Objective 1): A suitable, publicly available dataset will be selected. Static analysis tools (e.g., pefile for Windows executables) will extract a defined set of features (e.g., imported DLLs, function calls, section entropy) to create numerical feature vectors. The dataset will be split into training, validation, and testing sets. Baseline Classifier Development (Objective 1): A machine learning classifier (likely Random Forest or a simple neural network) will be implemented using scikit-learn or TensorFlow/PyTorch. Its performance on the clean test set will be established as a baseline. Genetic Algorithm Design (Objective 2): A genetic algorithm will be implemented. A "genome" will represent a set of allowable mutations to the feature vector. The fitness function will be the classifier's probability that the mutated sample is benign (i.e., the goal is to maximise this probability for a malicious sample). The GA will evolve populations of mutants from a small set of seed malware files. Experimentation and Analysis (Objectives 3 & 4): The most successful evasive variants from the GA will be analysed to identify effective mutation types. These variants will then be used to augment the training data, and the classifier will be retrained. Evaluation: The original and retrained classifiers will be tested against a new, unseen set of evolved variants. Metrics will include detection accuracy, evasion rate, and false positive rate, directly addressing the research questions.
Evaluation
The project's success will be evaluated against the research questions using the following criteria: 1) Functional Framework: Does the implemented GA successfully generate variants that are classified differently by the target model? (Addresses RQ1) 2) Evasion Analysis: What percentage of evolved variants successfully evade the baseline classifier, and which mutation types are most successful? (Addresses RQ2) 3) Robustness Improvement: Does the adversarially retrained classifier show a statistically significant decrease in evasion rate compared to the baseline? (Addresses RQ3) 4) Critical Reflection: Does the student critically evaluate the limitations of their approach (e.g., working with features, not real binaries; using a single malware family) and propose directions for future work?
Prerequisites
1. Programming: Strong proficiency in Python. 2. Machine Learning: Familiarity with core ML concepts and experience with libraries like scikit-learn. 3. Cybersecurity: A foundational understanding of malware types and basic static analysis concepts. 4. Mathematics: Basic understanding of probability and linear algebra. 5. Independent Working: Ability to manage a project timeline, troubleshoot code independently, and communicate progress effectively.
References
1. Babaagba, K. O., Tan, Z., & Hart, E. (2020). Automatic Generation of Adversarial Metamorphic Malware Using MAP-Elites. European Conference on the Applications of Evolutionary Computation. 2. Babaagba, K. O., Tan, Z., & Hart, E. (2020). Improving Classification of Metamorphic Malware by Augmenting Training Data with a Diverse Set of Evolved Mutant Samples. IEEE Congress on Evolutionary Computation (CEC). 3. Wang, X., & Miikkulainen, R. (2020). MDEA: Malware Detection with Evolutionary Adversarial Learning. IEEE Congress on Evolutionary Computation. 4. Do, J. (2020). A NEAT Approach to Malware Classification. Master's Projects, San José State University.
Supervisor
Babaagba, Kehinde
Contact SupervisorRequired Skills
Project Information
- Academic Year 2026/27
- Published 11 Mar 2026
- Last Updated 11 Mar 2026