A security test that actually delivers

A security test that delivers is one that establishes what needs to happen, and that makes it possible for follow-through to be assigned, carried out and evidenced. It is a diagnostic instrument, not an end in itself. A test that leaves nothing behind but a list of findings has delivered nothing, however thorough it was. The value is not in the finding, but in what moves afterwards. That is the difference between a test as a box-ticking moment and a test as the starting point of work.

This piece is about a word that in practice means two things at once and creates confusion as a result. For one person, "security test" is an annual penetration test with a fixed scope; for another, it is the continuous question of whether the security actually does what it is meant to do. The two are not the same, and the difference determines whether anything changes after the test or not.

What does a penetration test actually solve?

A penetration test is a test in which specialists simulate an attack, find vulnerabilities and report them. The classic form is a scheduled test, often annual, with a defined scope. That is useful work. But it pays to be precise about what it does and does not solve.

A penetration test does one thing above all: it makes visible what was wrong on the days tested, within the scope tested. That is a diagnosis. A valuable one, provided it is accurate and provided the scope covered the right things. But a diagnosis is not a treatment. The penetration test changes nothing in the environment. It describes it.

What a penetration test does not solve is everything that comes afterwards. It closes no gap. It assigns no ownership. It does not ensure that last year's finding is not back on the list this year. And it does not cover the time between two tests. An annual test tells you the security posture on the days testing took place. The eleven months in between are not a proven safe period. They are months the penetration test never looked at. In an environment that changes continuously, with new systems, new integrations and new misconfigurations, that unexamined window is precisely where things can go wrong.

The image that goes with this is familiar to anyone on the shop floor: the penetration-test PDF that disappears into a drawer. Not because no one took it seriously, but because the report was the end point instead of the starting point. The test was seen as the achievement. The achievement is what happens afterwards.

Why an annual penetration test isn't what you think it is

If you buy an annual penetration test, this is the question to ask beforehand. The assumption beneath an annual test is that a single snapshot is representative of the whole year. It rarely is. Between two tests the environment changes constantly. Code goes live, a supplier alters an integration, someone opens a port temporarily and forgets to close it. The January test does not see the misconfiguration that arose in June.

This is not a plea to run a full penetration test every month. It is a plea to be precise about what a test gives you. Three things determine whether a security test delivers:

Coverage over time. Not just "was it tested", but "how much of the year was actually in view". A single annual snapshot leaves the greater part of the time unexamined. Whoever fails to name the risk of that window sells a partial answer as a complete one.

Workability of the outcome. A report with fifty findings, without prioritisation, without an owner and without any relation to what the organisation can absorb, is not a diagnosis but a new pile of work. An outcome you cannot process has delivered you nothing.

Demonstrability. After the test, can you show what was validated, which gap is closed, and what remains open with a reason why? Demonstrability is what lets a test be turned into an argument upwards, to a CISO or a budget holder. Without it, it stays a feeling.

These three are the real test. Not "did we have our annual penetration test", but "what has demonstrably changed and what does it not cover". The Dutch National Cyber Security Centre describes security testing as part of risk-driven security management, not as a standalone annual event. In its work on the threat landscape, ENISA points out that the attack surface moves with the organisation. A testing cadence that does not move with it falls behind the moment the ink is dry.

The difference between a test as a snapshot and a test as a diagnosis

Here lies the original point of this piece. There is a notion that a security test is an achievement: an exam you pass or fail. That notion is the core of why so many tests deliver nothing. A test is not an exam. A test is a diagnosis, and a diagnosis only has value once a treatment follows that fits what the organisation can absorb.

That changes what you may expect from a test and how you buy it. A test set up as a diagnosis does not ask "which boxes are still unticked", but "what actually affects this specific environment in its own working process". That is a different question, and it yields a different kind of outcome: fewer findings, more sharply prioritised, tied to what has to happen next. A list of fifty vulnerabilities is easier to produce than a diagnosis that says: these are the three things that matter for you, in this order, for this reason.

Hanging off this distinction is one more question, and it is rarely asked out loud: whose does the outcome remain? A test that implicitly leaves governance and tool choice with the testing party delivers less than a test whose conclusion and follow-up stay with you. The check you can apply here is behavioural and simple: does the party running the test happen to locate the problem at the solution it sells itself? A test whose recommendation happens to always land on what the tester supplies is not an independent diagnosis. It is a quote with a report wrapped around it. A security test that delivers keeps the outcome, the prioritisation and the next choice with you, and makes explicit what interest the testing party does or does not have in a particular outcome.

This is also where the line sits between this story and the practical request "I want a penetration test carried out now". That request is legitimate and has its own place: a concrete engagement with a date, a scope and a price. But it is a different question from the one this piece addresses. Whoever buys only a penetration test buys a snapshot. Whoever wants a security test that delivers buys a diagnosis plus the work and the demonstrability that follow from it. It is wise to know which of the two you need before you sign, because the wrong answer to that question is a year-long false sense of certainty.

What does a test that does deliver look like?

Concretely, and without pitching a service here: a security test that delivers has four properties you can insist on beforehand.

The first is that the scope is derived from what actually affects the organisation, not from what is easy to test. The second is that the outcome is prioritised according to what the working process can absorb, not according to the number of findings. The third is that an owner and a follow-up are attached to every finding before the report is delivered, so that nothing lands in a drawer by default. The fourth is that the result is demonstrable: not just "we tested", but "this was validated, this is closed, this is still open and here is why".

A standard such as ISO/IEC 27001 helps here, provided you read it as intended. ISO/IEC 27001 does not prescribe an annual penetration test as a standalone box to tick. The standard calls for risk assessment, risk treatment and demonstrable management within an ISMS. A test is an instrument within that, not an end. Whoever uses the standard as an argument for one test a year uses it as a checklist and misses exactly the point. Whoever uses it as an argument for a continuous, demonstrable testing process that moves with the environment uses it as it was intended.

For anyone who has to raise this upwards, to a CISO or a budget holder, this is the usable frame. The conversation is not about "more testing is better". It is about coverage over time, about the workability of the outcome, and about the unexamined risk of the months between tests. Those are arguments that hold up under scrutiny. "We already run an annual penetration test" is not an argument that holds up, because it answers none of the three questions.

The next step, then, is not a request for a test, but an honest conversation about which of the two you need: a snapshot with a date, or a diagnosis with a follow-up. Only once that question is answered does it make sense to talk about scope, cadence and price. Whoever reverses that order buys another report for the drawer.