Vulnerability Finding: Two Inflection Points
Understanding the numbers from Anthropic and the system that surrounds Glasswing gives us new possibilities for effective defense.
In February, before project Glasswing, I wrote about An Inflection Point in Vulnerability Finding. Friday, Anthropic released a long post, Project Glasswing: An initial update. That post has ... a lot of numbers, and frankly, they're hard to make sense of. To help, I've put the numbers, as best I can extract them, into a the Sankey map shown above.
What stands out most when re-visualizing the data, is the 1,006 not-yet-patched vulnerabilities, and the huge amount of human work both there on the left, in security firm review, and on the right, in writing effective patches. Anthropic frames this as:
The relative ease of finding vulnerabilities compared with the difficulty of fixing them amounts to a major challenge for cybersecurity.
The second thing which stands out is the true positive rate, which seems to be about 17%. (3900/23019, based on the statement “at our current post-triage true-positive rates, it’s on track to have surfaced nearly 3,900 high- or critical-severity vulnerabilities in open-source code.”) This is important because that cost is being handed to humans, and if you apply Mythos to your code, you’ll need to either triage six vulns for each one you fix, or apply 5 needless fixes to your code because triage requires skills which are less available than writing code. Don’t forget that every code change has some chance of introducing a problem.
Today I’m expanding my comment on inflection points. We have two: vulnerability finding, which is growing very quickly, and vulnerability fixing, which is not. You might ask, how is that an inflection point if it’s not changing? It is, as covered below in discussion of Claude Enterprise, but it’s not growing as quickly or having as wide an impact. The public, defenders, and policymakers all need numbers on vulnerabilities per watt on some standard chipset, to help us understand the ways in which new models are working. (Given how well Anthropic seems to be turning Glasswing token spend into marketing, it’s conceivable that the inflection point is not model efficiency, but dollars, and thus watts, spent on scaling.)
Vulnerability fixing + software development
However, while Anthropic describes this as a major challenge for cybersecurity, there’s a more important point: This is a major challenge for LLM-oriented software development. If LLMs can be entrusted with software development, then they ought to be writing patches that work.
They’re not.
The contrast between the breathless blog posts from commercial entities and ... 97 findings patched in the open source world is really shocking. Anthropic does contrast open source to closed:
In the three weeks since launch, Claude Opus 4.7 has been used to patch over 2,100 vulnerabilities. (This is faster than the open-source patching described above in large part because enterprises are fixing their own code, whereas open-source fixes usually require volunteer maintainers who work through coordinated disclosure.)
Indeed, several maintainers have told us (Anthropic) they’re currently severely capacity constrained, and some have even asked us to slow down our rate of our disclosures because they need more time to design patches. (On average, a high- or critical-severity bug found by Mythos Preview takes two weeks to patch.)
Anthropic talks about how they’re making some of these skills available, to paying customers in Claude Enterprise edition, but I don’t see the denominator anywhere. Is that 2,100 Mythos-found vulns, or does it include some set that were known and not patched? That is, how many vulns were found by Mythos, and what fraction does that 2,100 represent?
Defensive design
I’d also like to go back to my February post, where I talked about the value of micro-segmentation, and triple down. As the rate of vulnerability finding grows, so does the importance of seeing vulnerabilities as a part of a system, and seeing exploitation as a phase in an attack lifecycles.
All technical systems have flaws, some of which are vulnerabilities that are deemed to be worth patching. Good system design includes small packages which limit blast radius, least-privilege implementation that make next steps needed and harder, and safer platforms which make exploitation harder.
A crucial value we get from threat modeling is seeing these possibilities as we consider “what can go wrong” and “what are we going to do about it?” (This is in contrast to the narrow, post-facto threat models that Claude derives from code.) And these architectural defenses take more time to implement, and the importance of re-factoring will continue to grow as vulnerabilities are found faster.
The point about micro-segmentation, least-privilege, and safer platforms are often overlooked, and at ThreatModCon Vienna, I'll be using a “small group mastermind” session to talk through “Layering defenses: A new hope?” a new way of thinking about defenses to get people to that sort of systems thinking.
Applying systems thinking lets us look at the whole set of numbers from new perspectives, rather than trying to optimize a single, hard-to-optimize element.
I used Flourish.studio to visualize the data and then annotated the image above by hand. The link should take you to the data, which is reproduced below.
Source,Dest,Value,Step from,Step to Mythos,sec firms,6202,0,1 sec firms,true positives,1752,1,2 true positives,crit or high,1094,2,3 crit or high,advisories,88,3,4 crit or high,not yet patched,1006,3,4 sec firms,Not rated,4450,1,2 true positives,medium,658,2,3 Mythos,medium (Mythos-rated),16817,0,1 medium (Mythos-rated),TBD,16817,1,0