top of page

Penetration Testing: Why Annual Tests Are Not Enough

Writer: PAGO Networks
PAGO Networks
5 hours ago
4 min read

A penetration test can establish whether an attacker could reach critical systems under the conditions examined during the assessment. Those conditions can change as soon as a new application goes live, a network rule changes or a service account receives additional permissions. Security decisions made weeks later may therefore depend on evidence collected from a different environment.


At PAGO Security Summit 2026, one question was about the continuity of penetration testing conducted once a year or once a quarter. The practical concern for CISOs is how to maintain useful evidence about exposure throughout the year as systems, access and attack techniques change.

Why annual and quarterly penetration tests leave gaps

Consider an illustrative example. A company completes a penetration test in March and resolves the reported findings. In April, a support project introduces remote access. In May, an application migration changes connectivity between two environments. The March assessment remains a valid record of its scope and testing period. Each subsequent change creates a reason to revisit the relevant conclusions.


Increasing the number of scheduled tests shortens the interval between assessments. Organizations also need a way to identify which changes require validation before the next scheduled engagement. Otherwise, the security team may have little evidence about whether recently introduced access creates a path to sensitive systems.


The age of a report is only part of this problem. A recent test can still leave a critical application, identity system or business service outside its scope. CISOs should examine when testing occurred, what it covered and which material changes followed it.


What continuous security validation should mean

Continuous security validation is a recurring process of testing whether relevant exposures can be exploited, recording the results and checking that corrective action works. A program operating 365 days a year should be able to respond to changes throughout the year, with test frequency matched to business risk and operational constraints.


That does not require every exploit to run against every asset every day. Discovery may run frequently, selected checks may follow deployments, and intrusive testing may need a planned window. Each result should state its scope, starting conditions, method, timestamp and limitations. Untested assets and inconclusive results should remain visible in reporting.


In his Summit presentation, Siwoo Lee placed validation within the 5 stages of Continuous Threat Exposure Management, or CTEM: scoping, discovery, prioritization, validation and mobilization. This connects the evidence from testing to decisions about what the organization should address.


Validate paths to business impact

A vulnerability finding becomes more useful when the team understands what it enables. Can an exposed application provide initial access? Can that access reach another system? Do excessive permissions allow a compromised identity to reach a critical service?


The Summit session described collecting evidence across exploitation, credential acquisition, lateral movement and privilege escalation. This helps establish how individual weaknesses combine into an attack path. For a CISO, that evidence supports a specific decision: which change would interrupt access to a business asset that matters?


For example, validation might show that a reachable server and reused administrative credentials create access to a finance system. The resulting action plan could involve patching the server, changing credential practices and restricting connectivity. Follow up testing should examine whether an equivalent route remains available.


Keep remediation open until the result is verified

One of the most useful details in the Summit presentation came from a lab demonstration. During a repeat assessment after corrective action, major threats had generally decreased, while additional assets and vulnerabilities were identified. The speaker used the example to explain why validation must repeat as the environment changes.


Operationally, each validated finding needs an owner, an agreed completion date and a defined retest. When immediate remediation would disrupt a service, the team should document the temporary mitigation, the remaining exposure and when the decision will be reviewed.


Closure should include evidence that the intended attack path has been interrupted. A focused retest can verify the specific correction. A broader reassessment can examine connected systems and changes made since the previous test. Keeping both options available helps prevent a narrow fix from leaving related exposure unresolved.


Choose test triggers and observe the SOC response

Useful triggers include a new external service, a material identity or network change, relevant information about active exploitation, and the completion of remediation. The organization should define which changes require testing, who initiates it and how quickly it must occur for each class of business service.

Repeated testing also needs agreed operating boundaries. NIST SP 800 115 discusses the availability and data risks of intrusive assessments and the need to choose appropriate techniques and environments. Define permitted actions, sensitive systems, stop conditions and accountable contacts before making tests recurring.


Where detection and response are part of the exercise, record whether the expected telemetry arrived, whether an alert reached an analyst and whether the response occurred as planned. A test that demonstrates access to a critical system should also help the SOC identify any missing visibility. Document test specific exclusions so they do not distort conclusions about defensive performance.


Measure coverage and verified closure

CISOs can use four measures to assess whether the program is keeping evidence current. These are proposed management measures, with targets to be set according to the organization’s services and testing constraints.

Measure

What to track

Validation coverage

Share of critical services with relevant, current test evidence; show exclusions separately.

Time since relevant validation

Elapsed time since testing that reflects the latest material change.

Time to verified closure

Elapsed time from confirmed exposure to a retest showing the path is interrupted.

Exposure recurrence

Previously resolved conditions that return, with the configuration or process responsible.

Keep specialist penetration testing in the program

Automated validation has limits determined by its scope, credentials, supported techniques and access to the environment. Specialist testers remain valuable for examining application business logic, unusual attack chains and assumptions that predefined tests may miss. A failed exploitation attempt also needs interpretation: missing prerequisites or limited test coverage can affect the result.


A practical starting point is one critical business service and its dependencies. Establish the current scope, validate relevant attack paths, assign corrective work and retest the outcome. Then connect validation to the changes that could make that evidence stale. Expanding this process across services gives CISOs a more current basis for security decisions throughout the year.


Sources

bottom of page