TL;DR — Deployed ML/DL models for real-time IoT intrusion detection on a Xilinx Kria KV260 FPGA, achieving 0.18ms per-sample inference latency and 99.9% binary F1, with full benchmarking against Random Forest and XGBoost baselines.
Investigating ML vs. DL-based intrusion detection systems (IDS) for IoT networks, with edge deployment on an FPGA.
A practical comparison demonstrating:
- RT-IoT2022 usage for binary (malicious or benign) and multi-class network traffic classification
- ML models (Random Forest, XGBoost) for training and comparison
- DL models (CNN with and without self-attention) for training and comparison
- Edge deployment — baseline CNN quantized via Vitis AI and accelerated on a Kria KV260 FPGA
Separate dataset download required — https://doi.org/10.24432/C5P338
- Overview
- Thesis
- Features
- Project Structure
- Installation
- Usage
- Results
- Limitations & Future Work
- Contributing
- License
- References
The Internet of Things (IoT) connects billions of task-specific devices across healthcare, industrial automation, smart homes, and more. This diversity makes it nearly impossible to define a single security solution, and the resource constraints of IoT hardware make traditional security approaches impractical. This project investigates the design of a real-time intrusion detection system (IDS) targeting network-layer attacks on IoT systems, and optimizes it for deployment at the network edge to act as a guardian of the IoT gateway.
Both ML and DL approaches are trained and benchmarked on the RT-IoT2022 dataset for two tasks: binary classification (benign vs. malicious) and multi-class classification (12 specific traffic/attack types). The models evaluated are Random Forest (RF), XGBoost, a baseline CNN, and a CNN with self-attention. The most deployable model is then quantized via Vitis AI and accelerated on a Kria KV260 FPGA, demonstrating feasibility for real-time edge inference with minimal latency.
Main contributions:
- Training and evaluating RF, XGBoost, and CNN (± self-attention) on a recent IoT IDS dataset across both binary and multi-class objectives
- Quantitative comparison of ML vs. DL trade-offs in accuracy, F1, MCC, and computational overhead — relevant to resource-constrained IoT environments
- Proof-of-concept edge deployment of a quantized CNN on a Kria KV260 FPGA, demonstrating real-time inference capability at the network gateway
This repository originated as a group capstone project.
The accompanying thesis was independently authored as part of the Oakland University Young's Honors College and expands on the ideas explored here.
- Author: Willow Connelly
- Title: Real-Time Intrusion Detection for IoT Networks: Evaluating ML/DL Models and Feasibility of Edge Deployment on the Kria KV260
- Type: Undergraduate Honors Thesis (Awarded: Thesis of Distinction)
- Scope: Independent work. Contributions include data preprocessing, significant ML/DL modeling, and all edge deployment work.
- Binary Classification — Classify network traffic as Benign or Malicious
- Multi-Class Classification — Identify 12 specific traffic/attack types (ARP poisoning, DDoS Slowloris, DOS SYN, MQTT, five NMAP scan variants, Metasploit SSH brute force, ThingSpeak, Wipro Bulb)
- ML Baselines — Random Forest and XGBoost with hyperparameter tuning
- 1D/2D CNN Models — Convolutional networks treating features as 1D sequences or reshaped 2D grids
- Self-Attention Variants — CNN architectures with an added self-attention module; notably improves minority class discrimination (NMAP subtype separation)
- Class Imbalance Handling — SMOTE oversampling applied during training
- Edge Deployment — Baseline CNN quantized to a fixed-point
.xmodelvia Vitis AI and deployed on a Xilinx Kria KV260 DPU for accelerated real-time inference - Evaluation Suite — Classification reports (accuracy, F1, MCC, ROC AUC), confusion matrices, and per-sample latency benchmarks saved per model
.
├── dataPreprocessScript.py # Loads and splits RT-IoT2022 (80/20 train/test)
├── dataAnalysisVisualization.ipynb # Dataset analysis and class distribution visualization
│
├── ml_binary_trainingValidation.ipynb # ML binary: XGBoost, RF: trains, tunes, validates, exports
├── ml_multi_trainingValidation.ipynb # ML multi-class: XGBoost, RF, CatBoost: trains, tunes, validates, exports
│
├── 1dcnn_binary.ipynb # 1D CNN binary classification
├── 1dcnn_binary selfattn.ipynb # 1D CNN + self-attention binary classification
├── 1dcnn_multi.ipynb # 1D CNN multi-class classification
├── 1dcnn_multi_selfattn.ipynb # 1D CNN + self-attention multi-class classification
│
├── latencyTiming.ipynb # Per-sample latency benchmarking across models and hardware
│
├── models/ # Exported model checkpoints
├── classification_reports/ # CSV evaluation metrics per model
├── confusionmatrices/ # Confusion matrix PNGs per model
│
├── edge/ # Edge deployment (Vitis AI / Kria KV260)
│ ├── {per model training}.ipynb # Model training notebooks
│ ├── BinaryCNN_classFile.py # Model class definition for binary 2D CNN
│ ├── MultiCNN_classFile.py # Model class definition for multi-class 2D CNN
│ ├── binaryQuantizer.py # Quantizes binary CNN model (Vitis AI)
│ ├── multiQuantizer.py # Quantizes multi-class CNN model (Vitis AI)
│ ├── binaryTestLoader.py # DataLoader for binary calibration/eval
│ ├── multiTestLoader.py # DataLoader for multi-class calibration/eval
│ ├── edgeDataPreparer.py # Prepares .npy data arrays for edge use
│ ├── inference.py # On-device inference script
│ ├── inference.ipynb # Multi-class inference notebook
│ ├── cap_binaryCNN_first.xmodel # Quantized binary CNN (Vitis AI xmodel)
│ ├── cap_multiCNN_first.xmodel # Quantized multi-class CNN (Vitis AI xmodel)
│ └── cnn_compile.sh # Vitis AI compile script
│
├── webdemo/ # Project website
│
├── environment.yml # Conda environment definition
└── requirements.txt # pip dependencies
- Install Anaconda/Miniconda if you haven't already.
- Create a new environment:
conda create -n iot_ids python=3.12
- Activate it:
conda activate iot_ids
Within your activated conda environment:
conda env create -f environment.ymlOr, if you prefer pip:
pip install -r requirements.txtKey dependencies:
- Python 3.12, PyTorch 2.x
- scikit-learn, imbalanced-learn (SMOTE)
- XGBoost, CatBoost
- Pandas, Matplotlib
Manual download required from the UCI ML Repository: https://doi.org/10.24432/C5P338
After downloading, run dataPreprocessScript.py to split the dataset into train and test sets (80/20 split).
python dataPreprocessScript.pyOptionally, open dataAnalysisVisualization.ipynb to explore class distributions and feature statistics before training.
Each notebook is self-contained — open and run top-to-bottom in Jupyter:
| Task | Notebook |
|---|---|
| ML binary (XGBoost, RF) | ml_binary_trainingValidation.ipynb |
| ML multi-class (XGBoost, RF, CatBoost) | ml_multi_trainingValidation.ipynb |
| 1D CNN binary | 1dcnn_binary.ipynb |
| 1D CNN binary + self-attention | 1dcnn_binary selfattn.ipynb |
| 1D CNN multi-class | 1dcnn_multi.ipynb |
| 1D CNN multi-class + self-attention | 1dcnn_multi_selfattn.ipynb |
| 2D CNN binary (edge-targeted) | edge/2dcnn_binary.ipynb |
| 2D CNN binary + self-attention | edge/2dcnn_binary selfattn.ipynb |
| 2D CNN multi-class (edge-targeted) | edge/2dcnn_multi.ipynb |
| 2D CNN multi-class + self-attention | edge/2dcnn_multi_selfattn.ipynb |
Trained model files are saved to models/ (.pth for PyTorch, .pkl for scikit-learn/XGBoost).
Classification reports and confusion matrices are automatically saved to classification_reports/ and confusionmatrices/ when running each training notebook. For latency benchmarking across hardware, see latencyTiming.ipynb.
For edge inference, see edge/inference.py or edge/inference.ipynb.
| Model | Accuracy | Precision | Recall | F1 | MCC |
|---|---|---|---|---|---|
| Random Forest | 0.9984 | 0.9988 | 0.9994 | 0.9991 | 0.9913 |
| XGBoost | 0.9983 | 0.9994 | 0.9987 | 0.9991 | 0.9909 |
| CNN (baseline) | 0.9913 | 0.9940 | 0.9964 | 0.9952 | 0.9522 |
| CNN + Self-Attention | 0.9900 | 0.9911 | 0.9978 | 0.9945 | 0.9443 |
ML ensemble models achieved the strongest binary performance. The baseline CNN, while slightly behind, reached a competitive F1 of 99.52% and, notably, is the only architecture compatible with full quantization and DPU deployment on the target hardware given the scope of the project.
| Model | Accuracy | Weighted F1 | Macro F1 | MCC |
|---|---|---|---|---|
| XGBoost | 0.9981 | 0.9980 | 0.9792 | 0.9951 |
| Random Forest | 0.9980 | 0.9980 | 0.9569 | 0.9949 |
| CNN (baseline) | 0.9493 | 0.9406 | 0.5324 | 0.8753 |
| CNN + Self-Attention | 0.9905 | 0.9912 | 0.8686 | 0.9762 |
The baseline CNN struggled with the five NMAP scan subclasses, collapsing them into a single predicted class. The self-attention variant resolved this by learning to weight the discriminative features between scan types, improving macro F1 by over 30 percentage points. XGBoost achieved the best overall multi-class performance.
Only the baseline CNN models were quantized and deployed to the Kria KV260 DPU (the attention variant was not compatible with the target DPU configuration given scope).
Binary — per-sample inference latency:
| Model | Hardware | Latency |
|---|---|---|
| CNN (baseline) | Kria KV260 DPU | 0.18 ms |
| CNN (baseline) | Intel i7 CPU | ~0.45 ms |
| XGBoost | Intel i7 CPU | — |
| Random Forest | Intel i7 CPU | 10.73 ms (59× slower than DPU) |
Multi-class — per-sample inference latency:
| Model | Hardware | Latency |
|---|---|---|
| CNN (baseline) | Kria KV260 DPU | 0.21 ms |
| CNN (baseline) | Intel i7 CPU | 0.71 ms (3.25× slower) |
| XGBoost | Intel i7 CPU | ~7.4 ms (35× slower than DPU) |
Post-quantization performance:
- Binary: F1 decreased marginally to 99.14%, MCC dropped 3.27 points to 91.95% — trade-off is minimal
- Multi-class: Macro F1 increased from 53.24% to 83.23% after quantization — suggesting the original model was overfit and quantization acted as a regularizer
The successful deployment demonstrates that edge-accelerated CNNs can provide near real-time threat detection at the IoT gateway with latency well under 1 ms per sample, while ML models — despite higher classification accuracy — are orders of magnitude slower and not quantization-compatible with the target hardware.
Limitations:
- The RT-IoT2022 dataset lacks temporal/sequential information, which limits the theoretical benefit of convolutional operators (features have no inherent spatial ordering)
- Class imbalance remains a challenge for rare attack types even with SMOTE; the NMAP subclasses have very few samples
- Only baseline CNN architectures were tested on the DPU due to time constraints — attention variants and other architectures were not evaluated on hardware
Future directions:
- Evaluate on more realistic, temporally-rich IoT network datasets to better leverage CNN sequence-learning capability
- Assess quantization-aware training (QAT) to reduce the performance gap introduced by post-training quantization
- Compare alternative edge hardware platforms beyond the Kria KV260
- Design a lightweight data ingestion and transformation pipeline suitable for live edge environments
- Explore hybrid DL architectures (e.g., CNN + LSTM, transformer-based) for improved minority class detection
- Fork this repository.
- Create a new branch for your feature/fix:
git checkout -b feature-my-improvement
- Commit your changes and push to your fork:
git commit -m "Add my new feature" git push origin feature-my-improvement - Open a Pull Request into the main branch.
This project is licensed under the MIT License.
- RT-IoT2022 Dataset — B. S. and R. Nagapadma, "RT-IoT2022," UCI Machine Learning Repository, 2023. DOI: https://doi.org/10.24432/C5P338.
Thank you for visiting IoT-AI-IDSs-edge-testing! If you have any questions or issues, feel free to open an issue or reach out.