SIEM validation

SIEM validation is the continuous testing of whether the detection rules you have configured actually catch the attack techniques they are meant to catch. Not whether the system is running, not whether logs are coming in, not whether the dashboard lights up: whether a simulated attack technique run against your own environment produces an alert that an analyst can act on. You run a technique, you check whether the rule fires, and if it does not fire you know where the gap is. That is the whole of it. Everything around it, frameworks, scenarios, cadence, reporting, is only relevant as long as that core holds.

This piece is about a term that in practice means two things at once, and that causes confusion. For some, SIEM validation is a one-off tuning exercise after a new rule set has been implemented. For others, it is a continuous discipline that is as routine as the patching process. The two are not the same, and the difference determines whether you actually see anything at the moment of a real attack.

What is SIEM validation, exactly?

A SIEM is a platform that collects logs and events from your environment and applies rules to them that generate alerts when a pattern matches something suspicious. Whether those rules recognise the right pattern is not a matter of belief, it is a matter of testing. SIEM validation is the practice of carrying out that test in a structured way: you release an attack technique into your own environment, you record what the SIEM sees of it, and you compare that with what it should have seen.

The reason for this distinction is familiar to anyone who experiences the work at close range. A SIEM implementation is delivered, the vendor demonstrates a set of use cases that work "out of the box", the team sees alerts appear, and on that basis the system is considered operational. What is missing in that setup is the question of whether the alerts you see are also the alerts you need. The absence of an alert is tacitly read as the absence of an attack, when it can equally mean the absence of a working rule. Validation takes that distinction out of the realm of assumption and returns it to the test.

A common language for describing attack techniques systematically is the MITRE ATT&CK framework. It catalogues techniques that attackers use in practice, ordered by phase of an attack, and gives the detection engineer a vocabulary in which to express their coverage. An open project such as Atomic Red Team adds ready-made scenarios to this: small, reproducible scripts that mimic a specific ATT&CK technique, intended to be released against your own telemetry in a controlled environment. The combination of the two, a taxonomy and executable scenarios, is what turns validation from a well-meant intention into a workable process.

The question SIEM validation answers is therefore sharper than "is our SIEM running". It is: for which attack techniques do we have a rule that demonstrably picks it up, for which do we have no rule, and for which do we have a rule that does not fire when you run the technique. Only with those three columns on the table do you know where you stand.

Why is "it runs, so it works" not the same as detection?

In many organisations the health of the detection platform is measured by what the system itself displays: the number of events per second, the retention of the logs, the uptime of the collectors, the volume of alerts per week. These are useful operational indicators, but they measure the wrong thing. They measure whether the platform works, not whether the detection works. The difference is large enough to build an entire discipline around it.

A SIEM can function technically perfectly and at the same time be blind to the techniques that actually matter in your environment. This happens more often than it should. A rule that once worked well breaks after an upgrade of the source component it depends on, a log source stops delivering unnoticed because a credential has expired, an ATT&CK technique is adapted by attackers and the rule that used to catch it no longer catches the new variant. In all of those cases the dashboard looks fine. The health of the platform is reported as green, the health of the detection is not.

The Dutch National Cyber Security Centre stresses in its guidance on detection and response that detection capability is an established function, not a by-product of a tool purchase. The NIST Cybersecurity Framework places Detect and Respond as functions in their own right alongside Protect, with explicit requirements for continuously improving and testing detection mechanisms. Both put the emphasis where it belongs: not on the presence of a SIEM, but on the demonstrable functioning of what detects within it. An organisation that reports platform metrics alone answers neither framework on the point that truly matters to them.

Validation is how you make that difference visible. It is not an audit instrument for proving something to a third party, it is a working method for the detection engineer and the SOC lead to see for themselves where their system shows silences that cannot be explained by the absence of a threat. Silence in itself says nothing. Silence plus an executed technique that produced no alert says everything.

How do you organise validation without buying a second SIEM?

The practical concern that often follows here is predictable: this sounds like yet another tool, yet another project, yet another budget line. It need not be. Validation is a habit, not a platform. What it does require is that a few things are in place and that they recur on a fixed cadence.

A workable setup consists of four components:

An agreed coverage list. Which ATT&CK techniques are relevant for your environment, based on your threat picture, your business model and the log sources you have. Not the full catalogue, that is an illusion. A bounded list that you maintain explicitly, with a reason per technique for why it is or is not on it.

Reproducible scenarios. For each technique on the list an executable scenario that mimics it in a controlled test setup or, by arrangement, in production. Open material such as Atomic Red Team offers a starting point; your own scenarios fill the gaps where your environment differs.

A fixed cadence. Validation that takes place only after incidents or before an audit is not validation, it is firefighting. A monthly or quarterly pace in which a fixed portion of the coverage list is worked through produces comparable measurement points and makes deterioration visible before it hurts.

A feedback loop into detection engineering. Every missed technique leads to a concrete action: adjust a rule, add a log source, sharpen a scenario, or consciously accept it with a reasoned justification. Validation without a feedback loop produces lists; validation with a feedback loop produces better detection.

What is not part of this is a second SIEM, a breach-and-attack-simulation platform or a separate test environment that is a copy of production. Those things can add value, but they are not the precondition. The precondition is that someone is accountable, that the cadence is in place, and that the outcomes land where they need to land. Without those three the most expensive platform does not help; with those three a simple setup already delivers visible results within a quarter.

What this means for the choice you make

For the CISO opening the internal conversation on this, the usable summary is that SIEM validation is not a new procurement topic but a working habit. The question to your detection team is not "is the SIEM running" but "when did we last demonstrably test whether technique X leads to an alert, and what was the outcome". The question to your vendor is not "which use cases are included" but "how will I still know next quarter that they work". The question to your board, if it ever asks, is not "do we have a SIEM" but "which attack techniques do we demonstrably have in view, which do we not, and how does that change over time".

That is a different conversation from the one held in most organisations. It shifts the measurement from platform health to detection health, and it turns the absence of an alert into a question rather than a reassurance. It also brings with it a test that is rarely spoken aloud: who keeps ownership of the outcome. A validation that implicitly leaves the rules, the scenarios and the reporting with the vendor delivers less than a validation whose coverage list, measurement points and follow-up choices remain yours. A party that conveniently always arrives at expanding its own platform is not carrying out independent validation. It is conducting a sales conversation with a report wrapped around it.

The logical next step, then, is not a request for a validation product, but an honest stocktake of where you stand now: do you have an agreed coverage list, do you have a cadence, do you have a feedback loop into detection engineering. If one of those three is missing, that is the conversation to have first, internally and possibly with someone who looks at it with you. Only once those three are in place is it worthwhile to talk about scenarios, tooling and reporting. Anyone who reverses that order buys a dashboard and keeps a blind spot.