The Vulnpocalypse - Good News and Bad News
tl;dr
The bad news is we are inundated with new vulnerabilities due to the increased volumes discovered by AI tooling.
The good news is many organizations are starting to handle this with increased speed, prioritization, and existing methods of defense in depth.
The bad news is there are also many organizations not doing this, and worse, are sitting on a whole ton of software, configuration, and structural/policy vulnerabilities from prior years.
The good news is many of these targets have not been exploited due to attacker’s resource constraints.
The bad news is that this might not hold as attackers industrialize their capabilities to turn more targets of opportunity into actual victims.
The good news is, apart from scams, crypto/Web3, and some parts of the ransomware kill chain, it appears that such industrialization is not helping monetization (yet).
The bad news is attackers are an innovative bunch and not likely to give up and go set up a bakery, or go back to drug dealing. Besides, chaos may still ensue as agent swarms simply exploit vulnerabilities anyway.
The good news is defenders can use all this AI capability for defense. We are seeing the potential that many significant code bases will have dramatically lower vulnerabilities levels than ever before due to the “deep cleaning” (h/t Heather Adkins) with the very same tools we are concerned attackers will use.
The bad news is getting a code base “clean” doesn’t mean the updated packaged software will arrive quickly in all the places where attacks will target.
The good news is……..?
We are inundated with as much commentary about the "vulnapocalypse" as there are vulnerabilities themselves. Some say this is a crisis. It’s certainly not the opposite of a crisis and there is a clear increase in workload associated with triaging the massive uptick in discovered vulnerabilities due to the incredible capabilities of our latest AI models.
As I’ve talked about here, the risk (of vulnerabilities being exploited to create impact) isn’t about the raw numbers of vulnerabilities. It’s not even always about the criticality, reachability or exploitability of such vulnerabilities. Instead it is often a function of the industrialized capability of attackers to use existing techniques, increasingly fueled by AI to actually exploit those reachable and exploitable vulnerabilities in a context that achieves an action on an objective. For most criminal threat actors this is about monetization. As it always has been. The difference now is that we are dealing with massive shifts in each of these stages from vulnerability to actual attack. I’ve covered this wider dynamic (itself a variant of “attackers have bosses and budget too”) in other blogs. For this post I want to narrow in on the possibilities of fixing more vulnerabilities than ever before and what we might need to do to achieve the actual risk reduction from that. This is a good news and bad news story as well.
In my spirit of short term pessimism and long term optimism I do think we are, as Heather Adkins and others have said, in a "deep clean" phase of working through many code bases to find and fix long-standing latent vulnerabilities as shown in the chart below.
But, of course, we also need to account for an additional graph of deploying the fixes. This is larger and, simultaneously, fatter tailed.
There are some organizations (Google being an amazingly good, if not simply the best, example) where finding a vulnerability and fixing that in the code base results in that fix being rapidly promulgated to all the places where it needs to be.
But for many other organizations finding the vulnerability and fixing the issue in source code are the “easy” bits. The hard bits are actually getting the updates out into the field to the points where attacks meet the vulnerability. This is heavily constrained by everything from cloud or on-premise server dependencies, edge or device specific dependencies, device / equipment recertification needs, intermittent connectivity, or just simply the scale and complexity of pushing to thousands or millions of semi-managed devices. And, this isn’t just about software vulnerabilities, the same applies to other weaknesses like configuration vulnerabilities, firmware, hardware and more.
So, it’s helpful to remind ourselves of the total effort of vulnerability mitigation. There are, perhaps in excessively simple terms, 4 steps:
Vulnerability discovery.
Vulnerability fix development.
Vulnerability fix testing and staging.
Vulnerability fix deployment.
There’s also a stage 0 which is vulnerability creation. This might seem odd to include it, and I didn’t so as to not distract from the main point, but over time this will be something we need to pay more attention to. We can reduce the risk of vulnerability creation with secure coding built into human or AI code generation. This could be through the use of standardized heavily vetted libraries or frameworks to implement critical code or other elements that are prone to vulnerabilities, the use of safe(r) programming languages, compiler flags and other capabilities to reduce impact from common issues like memory safety bugs, and pre-defined well architected system configuration or hardened builds. But, in time we will also have to concern ourselves more with deliberately introduced or induced vulnerabilities. If we really do deep clean our code bases, and attacks want to happen through software vulnerabilities then attackers will need to create (not just find) the vulnerabilities. Except for some special cases, many undisclosed, this has not needed to happen given vulnerabilities have been relatively easy to discover without the need for attackers to explicitly create them. But this will change.
There’s also a stage 5 which is “oh crap, the patch broke some stuff”. Naturally, this is what the testing and staging is designed to avoid. But the necessary pace of change and jagged reality of most organization’s infrastructure and software base means that breaks will inevitably happen and an approach to make the 2, 3 and 4 stages iterate quickly is always important.
Anyway, back to the 4 elements of dealing with vulnerabilities. In my experience, and from seeing this in many other places the effort to do this is essentially shown below:
Yes, the plural of anecdote is not data and the actual data for this is all over the place depending on how you measure it for each particular situation. It’s also situationally dependent, both in whether the patch can hit the targets timely enough. For example, getting a server patch to well managed infrastructure (the cattle not the pets) can be very speedy. But, getting something out to millions of telecom edge nodes, medical devices, artisanally managed infrastructure, industrial platforms, weapons, or even satellites is an entirely different proposition. So, whether you agree with this characterization precisely, I think we can agree that getting fixes deployed for a vast majority of software is very hard, usually harder than the other stages.
Our goal is to optimize each stage. To appeal to Goldratt’s The Goal and its cyber-incarnation in The Phoenix Project and the wider theory of constraints, we need to see where our bottlenecks are at all stages. Indeed, the speeding up of the vulnerability discovery and fix (of the code base) process further exacerbates the bottleneck that is in deployment.
Let’s look at each stage:
1.Vulnerability Discovery
There is clearly massive innovation here and we will likely see ever more innovation not just in the code base but also in the wider systems and business processes, driven by the use of threat models and other situational context.
2.Vulnerability Fix Development
Similarly, there is also much innovation on auto-generating code fixes and in many cases some effective tight loops between auto-discovery and auto-fixing.
However, this is a space where much additional innovation is needed and happening on not just dealing with the finding and fixing of a source code vulnerability but also handling configuration vulnerabilities and fixes. This can include adjusting or providing surrounding controls that mitigate the potential impact of the vulnerability. There are at least 3 areas of focus here:
Setting Policy Intent. Even if we have no software vulnerabilities we will have plenty of other exploitable paths. I’m excluding here outright missing controls (say, if an organization decides to not implement strong MFA and just live with passwords then there’s little any of this other stuff that can help with that) and instead focusing on the myriad of configuration issues from open ports, broken IAM, ineffective segmentation, and so on. This is the sort of stuff that is found on-premise and in the cloud with so-called posture management tooling. The next approach here, alongside hardened images and protected supply chains, is to move up a level of abstraction and set a policy intent that is auto-enforced by an AI-driven configuration system. Such policy intent might be “don’t let my workloads be accessible except through a context-aware access gateway”. The policy intent system takes care of the implementation detail in multiple contexts and at the same time sweeps away large amounts of micro-issues from the posture management tooling.
Auto-Adjusting Compensating Controls. This is where we can impact the reachability and exploitability of vulnerabilities where a patch or other type of fix has yet to land. These could be tools that understand vulnerabilities and their mitigation and, importantly, the wider organization context to know what tactical adjustment is needed on a WAF, IPS, firewall, or many other active controls.
Run Time Mitigation. This is kernel level, or otherwise, mitigation to actively intervene in an attempted exploit or to pre-emptively constrain an attack path.
All of these inherently reduce situational vulnerability, reduce reachability and exploitability and ultimately limit your attack surface. Even if not wholly effective they can still buy you time to get the actual patch or fix into the places where it matters (not just the source code).
3.Vulnerability Fix Testing and Staging
It’s still important to get the patches or fixes out as fast as you can. We know we can’t just rely on patching speed given we’re already in the “patchless window” and in negative time to exploit territory. Other controls and the approaches we just discussed are essential, but in many cases they don’t obviate the need to also get most, if not all, patches out at some point. There’s a few reasons for this. First, you do need to keep somewhat current on updates. For example, what happens when there’s a vulnerability that needs fixing that can’t be mitigated with compensating controls, run-time or otherwise, and you can’t apply that fix without first applying some or all of the prior fixes. You’ll now wish you’d done all the prior updates. Second, the compensating controls might also need continuous adjustment in light of new means of reachability or exploitability. Finally, in the spirit of defense in depth, an environment solely dependent on compensating controls is inherently brittle and if there’s a break in a compensating control then there’s no further defense. That is unless there’s compensating controls for your compensating controls (“turtles all the way down”).
There’s a couple of paths of innovation here:
Focus. You can prioritize the most critical vulnerabilities, or chains of lower criticality vulnerabilities that together represent a critical risk. But this is still a large and possibly overwhelming number for many organizations. Instead you can further focus (working up the criminal kill chain to monetization) and hyper-focus on the smaller number of vulnerabilities that are actually being exploited.
Patch Intelligence. Another approach to work the other side of the “balance sheet” is to get so good at understanding the interplay of fixes and your environment, with precision, that you move faster with less effort on testing because you know what is safe to push quickly, and you are forewarned of what might break in what way so you can preemptively resolve that. This is an approach that can remove the friction and reliability risk that would otherwise forestall rapid fixing.
4.Vulnerability Fix Deployment
Finally, you get to push the updates. There are lots of workable approaches here that also are adept at high speed, federated/distributed, deployment in relatively well managed infrastructures. But, that’s not yet representative and so more work needs to be done here that could include:
Fast Roll Back. Software distribution and managed infrastructure that supports very fast and reliable reversion. As we’ve all seen in many cases, a staged deployment of 1% and then 5%, if sufficiently sampled, can give you a high confidence that 100% deployment won’t carry a lot of reliability risk. But it’s not uncommon that stuff breaks. This could be because the sample of the organization in the 5% deployment is not a true sample of the organization. Often IT teams deliberately avoid putting critical systems in the 5%, for obvious reasons. But sometimes those critical systems have some idiosyncrasies that in bypassing such testing reduce the chance that potential breaks are caught and handled. There’s also the risk of race conditions where other updates (non-security) are introduced in between the time of the 5% test and full deployment. So, instead of going from 5% to 100% as the final step of deployment you could go from 5% to 20%, 40%, 60%, 80% and then 100% very quickly and pausing and rolling back at each stage if significant issues occur.
Patchable Architectures. Similarly, designing our infrastructure for software and configuration reproducibility inevitably means we can take more risk in pushing updates because the test environment is inherently more representative of the full deployment environment.
Faster Certification. Some environments have the ability to push software but can’t because that would require a full environment re-certification, for example medical devices, certain OT environments and more. In these cases, innovation is needed both in certification practices to move quicker and to more deliberately separate function and security updates into differently streamed processes. At the same time, architectural design can help isolate functional impact (the purpose of the certification) from holding back the security update.
Bottom line: Remediating vulnerabilities is more than finding and fixing the vulnerability in a code base. That’s becoming the “easy” bit. The hard bit for the vast majority of enterprises is actually getting the update actually deployed to the places that will face the exposure. This requires many adjustments in both compensating controls but also in ways we can manage the reliability risks of updates so we can move quicker (move fast but don’t break things). We’re coming to a world where vulnerability management transitions to end to end vulnerability operations.
Comments