Detecting APT Lateral Movement in Kubernetes with Lightweight ML
My MSc research: building a fintech microservices testbed, emulating APT lateral movement, and training lightweight ML models on network-flow features to detect an attacker moving between pods — evaluated against a signature-based IDS baseline.
This is a write-up of my MSc research project, Detecting APT Lateral Movement in Kubernetes-Hosted Microservices Using Lightweight Machine Learning on Network Traffic. It is longer and more technical than my usual posts, because the point is not a single trick — it is an end-to-end methodology: a realistic testbed, an emulated adversary, a labelled dataset built from first principles, two trained models, and an honest comparison against the tool this is supposed to improve on.
I have organised it around four things: the problem, the system architecture I built to study it, the machine-learning pipeline, and the findings — including the parts that should make you cautious.
The problem
Advanced Persistent Threats (APTs) are defined less by how they get in than by what they do once inside: they establish a foothold, then move laterally — from the initially compromised workload toward higher-value targets — often over weeks, deliberately blending into normal activity.
Kubernetes makes that lateral phase unusually comfortable for an attacker. The default network posture is allow-all: any pod can reach any other pod's service. Workloads authenticate users at the edge but frequently trust each other implicitly inside the cluster. So once an adversary controls one pod — via a vulnerable dependency, a leaked token, an over-permissive service account — their next moves look like ordinary east-west traffic. Reusing a valid JWT. Enumerating service ports over cluster DNS. Reading a ConfigMap with the pod's own credentials. There is no malicious payload for a scanner to match; the bytes are legitimate.
Signature-based intrusion detection is structurally poorly suited to this. A signature can only match a known-bad pattern, and there is no CVE-shaped payload here — only behaviour that is anomalous in context. That framing produced three research questions:
- RQ1 — Can lightweight machine learning, using only statistical features of network flows (no payload inspection), detect APT lateral movement between microservices in a Kubernetes cluster?
- RQ2 — How do two candidate classifiers — Random Forest and XGBoost — compare on detection quality and deployment cost (latency, memory, model size)?
- RQ3 — How does the ML approach compare against a production signature-based IDS (Suricata) run over the same traffic?
"Lightweight" is a first-class constraint, not an afterthought. The detector has to be cheap enough to run beside the workload it protects, on the same constrained infrastructure — so model size, inference latency, and memory footprint are evaluation criteria, not footnotes.
The system architecture
There is no public dataset of APT lateral movement inside a fintech Kubernetes cluster, so the first contribution is the testbed that produces one. It has three layers: the application, the service mesh, and the traffic generators (benign and adversarial).
The application: eight microservices
The workload is a deliberately simple but realistic fintech backend — eight services in a fintech namespace on a single-node minikube cluster. Five are FastAPI (Python), three are Go using the standard-library net/http. State is in-memory except the audit sink, which uses SQLite.
| # | Service | Lang | Port | Role |
|---|---|---|---|---|
| 1 | auth-service | FastAPI | 8001 | Login, JWT issuance, token introspection |
| 2 | user-service | FastAPI | 8002 | User profiles + credential verification |
| 3 | payment-service | Go | 8003 | Payment orchestration |
| 4 | account-service | Go | 8004 | Balances + ledger |
| 5 | notification-service | FastAPI | 8005 | Simulated dispatch |
| 6 | audit-service | FastAPI | 8006 | Append-only audit sink (SQLite) |
| 7 | config-service | Go | 8007 | Non-secret config from a ConfigMap |
| 8 | report-service | FastAPI | 8008 | Admin compliance/summary reports |
The design principle throughout: realism lives in the inter-service call patterns, not in business completeness. The app exists to generate believable east-west traffic. Those calls form a graph — the exact traffic that gets captured later:
client ──login──▶ auth-service ──▶ user-service (verify-credentials)
└─────────▶ audit-service (login events)
client ──▶ payment-service ──▶ account-service (debit / credit ledger)
└──────────────▶ audit-service (payment_processed)
└──────────────▶ notification-service (payment_confirmation)
account-service ──▶ audit-service (ledger_applied)
account-service ──▶ payment-service (/payments/{ref}/callback)
report-service ──▶ account-service + audit-service (admin reports)
auth, payment, report ──▶ config-service (GET /config, on startup + every ~30s)
One decision is threaded through every layer and is worth stating loudly: internal pod-to-pod calls have no authentication. Users authenticate at the edge with a JWT, but services trust each other implicitly inside the cluster. That is not a bug I neglected to fix — it is a faithful model of the default Kubernetes trust boundary, and it is precisely the gap the research studies. "Hardening" it would delete the phenomenon under investigation.
The mesh: Istio with permissive mTLS
The services run under Istio 1.19.9 with PeerAuthentication in PERMISSIVE mTLS mode, with Prometheus, Grafana, and Kiali for observability. Permissive mode matters for two reasons. First, it is what real clusters run during migration, so it is realistic. Second, and more importantly for the method: the traffic is encrypted, but the detector never looks inside a packet. It works entirely on flow-level metadata — sizes, counts, timing, TCP window values — which survives encryption. The approach does not depend on decrypting anything, and that is a deliberate property, not a limitation.
The traffic: benign baseline and emulated adversary
Two generators run against the mesh:
Benign traffic comes from Locust, deployed in a separate load-gen namespace (no sidecar, so its own traffic is distinguishable). It runs two personas — a majority of CustomerUsers and a few AdminUsers — under a diurnal load shape that varies concurrency by the real hour of day, so the baseline breathes like a real system rather than droning at constant RPS.
Adversarial traffic comes from a compromised pod inside the fintech namespace, running five MITRE ATT&CK lateral-movement techniques:
- T1046 — Network Service Discovery (scanning the mesh for reachable services)
- T1550.001 — Use Alternate Authentication Material: Application Access Token (replaying a harvested JWT)
- T1210 — Exploitation of Remote Services (IDOR-style object enumeration + a malformed request)
- T1083 — File and Directory Discovery, adapted to ConfigMap/Secret enumeration via the pod's service account
- T1557 — Adversary-in-the-Middle
A note on method here: the original plan used MITRE Caldera as the C2 orchestrator. After several build cycles fighting an end-of-life base image, a Python version that had removed a module Caldera depended on, and an API returning silent 500s, I judged the C2 server to be incidental complexity — the techniques themselves are just commands run from the compromised pod. I replaced Caldera with a ~200-line runner that produces identical on-the-wire traffic and, crucially, writes a structured ground-truth log for every operation: technique ID, start/finish timestamps, source pod, and destination. That log is what makes the dataset labellable at all. The pivot is documented as a methodological limitation rather than hidden.
The machine-learning pipeline
The data path turns raw packets into a trained, evaluated classifier in five stages. Each stage hides a correctness problem that took real effort to get right.
tcpdump (CNI bridge) → CICFlowMeter → flow-tuple labelling → feature engineering → RF / XGBoost
raw packets ~78 features/flow ±2s vs ground truth variance→corr→RFECV train & evaluate
1. Capture. A hostNetwork tcpdump pod captures packets off the CNI bridge across the pod CIDR. This is metadata-only by design — no payload is retained or inspected.
2. Flow extraction. CICFlowMeter converts packets into per-flow records — roughly 78 statistical features per bidirectional flow (packet-length distributions, inter-arrival timing, byte and packet counts, TCP window sizes, flags). A flow, not a packet, is the unit of classification.
3. Labelling. This is where correctness actually lives. A flow is labelled malicious only if the attacker pod is an endpoint, its start time falls within ±2 seconds of a logged attack operation, and its destination matches that operation's target. Everything else — including legitimate traffic from the attacker pod outside attack windows — is benign. Every bug I hit at this stage was an alignment bug: flow timestamps in local time while ground truth was UTC; then, once fixed, sub-second attack windows (one technique's window was 0.384s) collapsing into each other under one-second timestamp resolution. The fix was microsecond timestamps plus a best-match rule that prefers the tightest-fitting window, and resolving destinations by both pod IP and ClusterIP.
4. Feature engineering. Feature selection was data-driven, not hand-picked: drop near-zero-variance features, remove one of each highly-correlated pair (keeping the higher Random-Forest-importance member), then run recursive feature elimination with cross-validation (RFECV). That reduced ~78 candidates to 12 features:
fwd_pkt_len_max fwd_pkt_len_mean pkt_len_min pkt_len_mean
fwd_header_len fwd_act_data_pkts flow_iat_std init_fwd_win_byts
init_bwd_win_byts subflow_fwd_byts protocol dst_port
Two of these (protocol, dst_port) are categorical, which matters for the balancing step below.
5. Dataset construction and training. A 3-hour collection yielded 100,131 benign flows and 420 malicious (0.42%) — a real, severe class imbalance. I used a stratified 70/15/15 split and then made two disciplined choices:
- StandardScaler was fit on the training split only, then applied to validation and test — no test statistics leak into preprocessing.
- SMOTENC oversampling was applied only to the training set, balancing it to 70,091 / 70,091. Validation and test were left at their true, ugly, 0.42% distribution. Balancing the test set would have been measuring a fiction.
Both models — Random Forest and XGBoost — were tuned with Optuna (60 trials, TPE sampler, validation-F1 objective), then evaluated on the held-out imbalanced test set, with 5-fold cross-validation and SHAP for interpretability.
The findings
RQ2 — Random Forest vs XGBoost
Both models detect the attacks; Random Forest edged it on the imbalanced test set.
| Metric | Random Forest | XGBoost |
|---|---|---|
| Precision | 0.984 | 0.969 |
| Recall | 1.000 | 1.000 |
| F1 | 0.992 | 0.984 |
| False-positive rate | 0.00007 | 0.00007 |
| ROC-AUC | 1.000 | 1.000 |
| Inference | 0.0015 ms/sample | 0.0005 ms/sample |
| Model size on disk | 0.20 MB | 0.40 MB |
| Peak inference memory | 4.5 MB | — |
The Random Forest confusion matrix on the test set was TP 63, FN 0, FP 1, TN 15,019 — every malicious flow caught, a single benign flow misclassified. Per-technique recall was 1.0 for T1046 (37 flows), T1550.001 (13), T1210 (10), and T1083 (3). T1557 produced too few labelled flows in the test split to report — a real limitation of a short collection against a rare technique.
Both models answer RQ1 affirmatively and satisfy the lightweight constraint: sub-millisecond inference, sub-megabyte on disk, a few megabytes of inference working set. Random Forest was selected as the deployed model for its slightly higher F1 and its more interpretable structure.
RQ3 — the signature baseline
I ran Suricata 8.0.6 with the Emerging Threats Open ruleset over the same captured packets. It detected zero of the five techniques — recall 0.00. Its only output was ~1.6 million "invalid checksum" decoder warnings, which are an artifact of capturing on a virtual bridge (checksum offload), not detections; I had to filter them out and re-run with checksum validation disabled to confirm there was genuinely nothing.
This is the crux of the comparison, and I want to be precise about what it shows. It is not "my model beats Suricata at Suricata's job." It is that signature detection is structurally blind to novel, Kubernetes-native lateral movement, because these techniques have no payload signature to match — which is exactly the gap the ML approach is meant to fill.
Was the result too good?
An F1 of 0.992 and an AUC of 1.0 are a smell, not a trophy. There are two ways to earn a score like that dishonestly — leaking a feature, or a trivially separable task — and I tested for both.
Leakage. My concern was dst_port: if the model just learned "traffic to this port is bad," that is an address lookup, not detection. So I ran an ablation, retraining with suspect features removed:
full feature set F1 = 0.9921
minus dst_port F1 = 0.9921 ← unchanged
minus dst_port + both TCP-window features F1 = 0.9844 ← barely moved
Removing dst_port changed the score by exactly zero. The signal is distributed across the packet-length and timing features — it is behavioural, not an addressing artifact. That is the reassuring result.
Separability — the honest caveat. The task is genuinely easy in this testbed, and that is the single most important sentence in the whole project. The emulated attacks are burst-style: rapid, uniform, machine-generated — twenty calls in four seconds, a port scan that fans out at once. Against varied, human-paced benign traffic, that stands out sharply. A real APT moving low and slow — deliberately pacing actions to blend into the baseline — is a materially harder problem, and this dataset does not demonstrate the model would catch it. The near-perfect numbers are a property of the adversary emulation as much as the classifier, and they belong in the limitations chapter, stated plainly, not buried.
Deployment cost
I also deployed the Random Forest as an in-cluster FastAPI inference service in a security-monitoring namespace to measure real overhead. Server-side inference held around 0.004 ms/sample. One finding is worth surfacing: the pod would not start in the 256 MB the methodology originally budgeted — not because the model is heavy (it is under a megabyte) but because the Python + scikit-learn serving runtime has a ~175 MB resident floor. The model is lightweight; a Python process serving it is less so. That distinction — model cost vs. serving-runtime cost — is a genuine deployment insight, and I raised the limit to 512 MB with a startup probe to accommodate it.
Limitations
Being explicit about these is part of the contribution:
- Single node. Everything ran on one minikube node, which saturated under combined load + capture and truncated some collection windows. A multi-node cluster would exercise cross-node traffic paths this setup cannot.
- Loud adversary. As above, burst-style emulation makes the task easier than a patient real APT would.
- Short collection. Three hours gave a usable but small malicious class; T1557 in particular was under-represented.
- Emulated, not real. Techniques are faithful ATT&CK emulations, not captured real-world APT traffic — the perennial constraint of this research area.
Conclusion
The research answers its questions. Lightweight ML on flow-level metadata can detect emulated APT lateral movement between Kubernetes microservices (RQ1); Random Forest slightly outperforms XGBoost while both meet the lightweight constraint (RQ2); and a mature signature IDS detects none of the same techniques (RQ3) — evidence that behavioural ML addresses a gap signatures cannot.
The two results have to be held together honestly. The satisfying one: a sub-megabyte model, reading only encryption-safe flow features, caught every technique a production IDS missed, cheaply enough to run inside the cluster it protects. The sobering one: the emulated attacks were loud, and the real test is a quiet adversary I have not yet built.
For me the most valuable output was not the 0.992. It was the reproducible pipeline — capture, label, select, train, and interrogate your own results — and a methodology honest enough to name why its best number might not survive contact with a patient attacker. That is the next piece of work: a low-and-slow adversary model, on a multi-node cluster, over a much longer collection.