How AI Detects Suspicious Network Traffic

There is a specific kind of exhaustion that comes from maintaining a rule-based detection system. You write a rule. You catch a threat. Three weeks later, a variant of that same threat slips through because it changed one header value. You write another rule. Repeat forever.

I know this exhaustion well. Before we built the detection engine at ColadAI, I lived it.

The Way We Used to Do It

The old approach was straightforward in theory. You define what bad traffic looks like, you write a signature for it, and you flag anything that matches. Port scanning looks like this. A brute force attempt looks like that. A known malware beacon has this user-agent string. Rules for all of it.

It worked. Until it didn't.

The problem was not the rules themselves. The problem was that the ruleset was a museum of past attacks. Everything in it was something we had already seen. It told us nothing about what we had not seen yet.

On top of that, every new threat vector meant a new feature request. "Can we add detection for X?" Sure. We'd pull the logs, identify the pattern, write the rule, test it, deploy it. That whole cycle took days. Sometimes a week. And in that window, you were blind.

We also had false positives. A lot of them. The rules were blunt. A signature that caught malicious traffic also caught legitimate traffic doing something slightly unusual. Security engineers would spend hours triaging alerts, most of which turned out to be nothing. The human cost of that was enormous, and it scaled badly. More traffic meant more alerts, which meant more triage, which meant more headcount.

It was not sustainable.

What Made Us Change

Honestly it was not a single dramatic incident. It was the accumulation of small ones.

We kept getting feature requests that all had the same shape: "We are seeing this new pattern, can you add a rule for it?" And each time, I would think about how long it would take, and how the moment we shipped it, the attacker could trivially change the pattern and we'd be starting over.

At some point I stopped thinking about individual rules and started thinking about the underlying question. We were not trying to match patterns. We were trying to answer: is this traffic behaving normally for this network? That is a different question. And it is one that machine learning is genuinely good at.

That was the starting point for what became the detection engine inside ColadAI.

How the AI Approach Works

The core shift was moving from "does this traffic match a known bad pattern" to "does this traffic look like normal traffic for this environment."

Every network has a baseline. The times of day traffic spikes. The internal services that talk to each other. The external destinations that get hit regularly. The typical volume and size of requests between certain hosts. This baseline is not static and it is not the same for every network. It is specific.

The ML model learns this baseline continuously. Not from a labelled dataset of good and bad traffic, but from the live behaviour of the network it is watching. It builds a probabilistic model of what normal looks like here, for this customer, at this time.

When something deviates from that baseline, it gets flagged. Not because it matched a rule we wrote, but because it looks statistically anomalous relative to established behaviour.

A few things this catches that rules never could:

Slow and low attacks. An attacker who is being careful will not trigger a brute force rule because they are not brute forcing anything. They might try one login every six hours from a rotating set of IPs. No rule catches that. A behavioural model does, because over time, those failed attempts accumulate into an anomaly.

Lateral movement. Once an attacker is inside a network, the interesting signal is not what they are doing in isolation but how traffic between internal hosts is changing. A server that has never talked to another server suddenly initiating connections is suspicious. Rules would need to know in advance which pairs of hosts should never talk. The model just knows whether it has seen this before.

Data exfiltration in disguise. Large volumes of outbound data to a cloud storage endpoint is something many legitimate applications do. But if that volume triples at 2am to a destination the network has never used before, that is a different story. The model sees the combination of factors. A rule would have to anticipate every possible combination in advance.

What the Human Still Does

This is the part I always want to be honest about, because the "AI replaces everything" framing is both wrong and counterproductive in security.

The model flags anomalies. It does not make decisions. A human, or in our case a tiered system where lower-confidence flags go to a queue and high-confidence ones trigger automated responses, still reviews what matters.

What changed is the quality of what humans are looking at. Instead of triaging a hundred alerts where ninety are false positives, a security engineer is looking at ten flags that all have context attached. Here is the anomaly. Here is what normal looks like for this host. Here is what changed and when. Here are the other events in the same time window that might be related.

That is a very different job. It is faster. It is less draining. And it catches things that the old process missed entirely because the analyst never got to them under the pile of noise.

What Building This Taught Me

The hardest part was not the model. The hardest part was convincing myself and the team that the goal was not to eliminate false positives but to make each alert worth a human's attention.

Rule-based systems give you a false sense of precision. You wrote the rule, you know exactly why it fired, you can defend it. Anomaly detection feels fuzzier. The model says this is unusual. You have to trust it and build enough context around the flag to make that trust actionable.

That trust is earned over time. The model gets better as it sees more of the network. The thresholds get tuned as you learn where it is oversensitive. The workflow gets refined as analysts give feedback on what is useful and what is noise.

It is a living system, not a rulebook you write once and forget.

We are still building it. But the difference between where we are now and where we were two years ago, trying to outrun attackers with a spreadsheet of signatures, is not incremental. It is fundamental.

This detection engine is part of our Network Monitoring tool at ColadAI. We built it for our own use, but we are now offering it as a service. If your team is dealing with the same rule-maintenance nightmare we were, or you just want something smarter watching your network, drop me a text and we can talk about what that looks like for you.