Control Reliability Engineering (CRE): Applying SRE Principles to Cybersecurity Controls

Security breaches are often not the result of awesome attacker capabilities or the sudden emergence of sophisticated zero-day exploits. Instead, what we usually find are the controls designed to stop the attack were believed to be operational but were actually broken or misconfigured at the moment when they were needed. Sometimes they were never fully in place to meet the security team's original intent.  

So, continuous control monitoring is needed to counter the natural decay that occurs to any type of control, not just security. More than just the monitoring itself, we need to apply a wider operational discipline to design controls that are able to be monitored, to look at root cause analysis in the face of detected failures, and to apply engineering principles to the overall control environment. In short, we need Control Reliability Engineering (CRE) as a direct application of Site Reliability Engineering (SRE) approaches to cybersecurity controls. 

What is Site Reliability Engineering (SRE)?

Site Reliability Engineering is a discipline that applies engineering principles directly to the entire IT lifecycle. There are vast and wonderful resources about all things SRE here. 

The core tenet of SRE is that reliability is the most fundamental feature of any product. A system is functionally useless if it remains unavailable to its users. By treating operations as a software engineering problem, SRE structures the entire lifecycle of a service, from inception and deployment through operation and eventual decommissioning, around engineered predictability rather than simply the hope of actual or near 100% uptime.

SRE balances the necessity of rapid innovation with the requirement for service stability through several foundational pillars, including:

• SLIs and SLOs (Service Level Indicators & Objectives): SLIs are specific quantitative measures of a service's performance (such as uptime, success rate or latency), while SLOs are the targeted values or ranges set for those metrics. This pairing establishes a technical contract of success that clearly defines what constitutes a healthy service.

• Error Budgets: This is the difference between 100% reliability and the defined SLO, for example, a 99.9% SLO leaves a 0.1% error budget. Failures eat into that budget. The error budget acts as a buffer that directs risk-taking. It defines a permissible level of failure. If a team exhausts their budget, feature innovation temporarily stops so engineering capacity can focus strictly on stability, so as to not breach or lessen the breach of the SLO.

• Toil Elimination: The aggressive automation of repetitive, manual operational tasks that scale linearly with service size and lack long-term value. Eliminating toil reduces operational entropy and prevents "toil-debt" from overwhelming engineering teams, freeing them to focus on scaling, design, security and reliability.

Within this structured framework, the Production Readiness Review (PRR) serves as the ultimate gatekeeper. No service is officially onboarded for operational support until it meets strict standards for instrumentation, monitoring, and structural soundness. Once a system is live, maintaining reliability requires continuous learning, which SRE achieves through thorough root-cause analysis and blameless post-mortems. This practice ensures that failures are treated as opportunities to harden the system rather than occasions to assign blame. Ultimately, these infrastructure-centric principles do more than just safeguard uptime, they provide the logical blueprint for managing risk across other domains, including security.

The Path to Control Reliability Engineering (CRE)

Control Reliability Engineering (CRE) addresses the central problem that broken controls can often look identical to functioning ones until an incident reveals it not to be so. Like a radiation detector that reads zero might mean the environment is safe, or it might mean the sensor is faulty (which is why many such detectors have a small harmless source so they never should read zero). 

CRE is the application of monitoring, management, and incident learning to the specification and execution of security controls. By treating a Control Incident (a failed control whether or not it led to an actual security incident) with the same gravity as a Security Incident (a breach or other compromise), we force ourselves to remediate structural weaknesses before they become a major issue or catastrophe. Some examples:

• Ticket-Based Response to Metric-Driven Control Health: Instead of reacting to individual alerts, CRE monitors the request success rate of the controls themselves (e.g. policy evaluation success).

• Subjective Risk to Objective SLIs: Security health is measured by quantitative indicators, such as the ratio of policy-evaluated flows to total flows, control coverage vs. risk universe, and so on. 100% completion of a control project might mean nothing if success was framed as only covering 20% of the environment. 

• Manual Tuning to Standardized Frameworks: Security is inherited through SRE-validated platform infrastructure rather than being retrofitted via custom, one-off deployments. This capitalizes on SRE approaches to solve security operational effectiveness, ensuring that defensive systems converge to their intended policy.

The 10 Essential Elements of Control Reliability Engineering

The following list are the essential elements for a mature CRE program. 

• Control Ontology & Cataloging. Building a catalog of key controls using a well-formed ontology to treat controls as identifiable, measurable assets. This ensures that security is not a vague concept but a collection of discrete, engineered objects that can be tracked for drift and entropy.

• Controls as Code. Treating security controls as automation/code with versioning, peer review, and CI/CD pipelines. This eliminates "manual-only" configurations and ensures that every change to a control is testable, auditable, and rollback-ready.

• Control SLIs and SLOs. Provide an engineering-standard metric to judge if a control is reliable enough to meet the organization’s risk tolerance.

• Continuous Control Monitoring (CCM). Implement systems that monitor control health in real-time to minimize the window between failure and detection. This shifts the approach from periodic auditing to runtime verification, effectively reducing the window of opportunity for attackers or other threats to impact the environment.

• Synthetic Event Injection. Actively probe the environment with harmless synthetic events injected into a control to verify it is not "reading zero" because it is broken.

• Control Readiness Reviews (CRR). Adapting the PRR to ensure a control meets operational standards (instrumentation, playbooks, recovery plans) before being onboarded into production. This helps prevent the deployment of fragile security tools that add more toil than protection.

• Safe Changes. Gradual deployment of control changes to a subset of the environment or traffic flow before full rollout. This helps prevent global lockouts, where a bad configuration change prevents all users (and administrators) from accessing the system. The best thing about building confidence in change is the minimal recurrence of such reliability knock-on effects reduces the resistance to new controls which speeds up the security program.  

• Control Incident Management. Formalize control incident declarations for failed defenses, followed by a blameless postmortem to identify root causes. Treat close-calls with the same intensity as breaches, fostering a culture of continuous improvement and engineering rigor.

• Error Budgets for Security Controls. Defining a specific threshold for unplanned control downtime that, once exhausted, halts new deployments to focus on control stability. This balances the speed of implementing new security features against the risk of destabilizing the current environment.

The Future of Defensive Engineering

Controls inevitably degrade unless that force is counteracted with disciplined systems. SRE was successfully built, first at Google, and then in many other organizations to do this for reliability. Adapting SRE principles for controls gives us Control Reliability Engineering (CRE). This acknowledges that in a complex environment, failure is inevitable, and therefore our defensive systems must be engineered for security and reliability. By treating security controls as high-scale engineering objects, we move from troubleshooting to resilience. As the SRE maxim reminds us: hope is not a strategy.  

Bottom line: The future of defensive engineering belongs to the past. We must apply the rigorous reliability lessons learned over the last decades of high-scale SRE to current and future security challenges.