Vulnerability Finding: An Inflection Point
LLM-driven vuln finding has reached an inflection
A spate of recent stories show an inflection point in vulnerability discovery. They include:
- Anthropic's new model is a pro at finding security flaws “Anthropic's latest AI model has found more than 500 previously unknown high-severity security flaws in open-source libraries with little to no prompting.” (Axios, Feb 5), and the Anthropic blog.
- What AI Security Research Looks Like When It Works from AISLE.
- Introducing Trusted Access for Cyber from OpenAI.
This comes together to show that LLMs are getting decent at finding vulns, and that means vuln finding in semi-hard targets like Ghostscript is increasingly feasible. (I say semi-hard because they don’t reveal finds in Chrome or OpenSSH, and what they do find includes... strcpy.) But: that’s most software. Most software hasn’t gotten the same attention as the premier projects, and even some of those which have (like MS Office) continue to have vulns because they include piles of old code.
By my count, seven of the top ten collectives on HackerOne are now AI. (Flysec, ZDI, and eternalmoon are not.) Relatedly, Nico Waisman had a post, The road to Top 1: How XBOW did it
We can consider two main cases: Code you create and code you rely on. The code you create may be open or closed source, and may even never leave your servers. I don’t think closed source matters very much here: LLMs will do just fine with binary analysis, possibly even better. Code only running on your systems make it possible for you to detect the probes, but you need both processes to detect and respond.
Code you rely on (open source of commercial), like the code you distribute, is available to a wide range of attackers. They can now make economic decisions about how much they’re able to spend per zero day. We’ll also see a dramatic uptick in the availability of exploit code.
So: more vulns, faster.
You have to assume zero days have been discovered in your code. The expensive and time consuming apparatus to do “risk assessment” of reported vulns is going to break in several ways, at once. Those include:
- ”Vulns will be reported by security researchers looking for fame” will be replaced by private pools of vulns.
- ”Most vulns have CVEs” will be replaced by most vulns won’t. (Some would argue that this ain’t new. I guess they’ve won by ... attrition.)
- The lack of publication, CVEs, and CVSS or EPSS scores mean there’s no sane time to start an “SLA clock” for the vuln.
A primary new line of defense will be boundaries that surround smaller and smaller areas of impact. Here, Docker and Kubernetes containers are at the large end of blast radius. Patterns like sshd’s multi-process model are derived from an understanding that splitting out a listener and session binary reduce the blast radius of an attack; they also mean that each may have fewer libraries linked into it. Going further into these patterns are sandboxes like RLBox, designed to sandbox third party C libraries. You may be thinking that’s somewhat extreme. I’ll just say I’m not the one decorating my apartment with couches I pick up on the sidewalk.
More seriously, open source code is often high-quality, and using it for the undifferentiated parts of your systems makes a lot of sense. And like all software, it literally says “No Warranties express or implied. Use at your own risk.” There are limits to what digging into the maintainers, the update rates, the security badges, or the SBOM can do for you. Those limits are compounded and emphasized by the ability of LLMs to reliably and scalably find vulns.
This is a place where the attacker has a tremendous advantage: They care less about false positives than your developers do, and so the success of LLMs at finding vulns doesn’t mean they’ll replace your SAST tooling. It’s ok for attackers to get a low hit rate because their job includes finding the working exploit. (The new “when it works” post doesn’t change what I wrote here, but adds color in that 5 of the 12 OpenSSL vulns included a workable patch.
Your job is starting to include the assumption that working exploits exist in each of your binaries, and to design engineering and deployment patterns that isolate those binaries and limit blast radius. Threat modeling is the best way to identify potential boundaries and ensure your teams are talking about those boundaries.
That discussion needs to include both where those boundaries are, and where they could be. The latter part can take time to achieve. Like planting a tree, the second best time is now.
Coda: As I was finalizing this essay, Bruce Schneier linked to the “when it works” post now in the list at the top, which found 12 vulns in OpenSSL. It breathlessly describes them as “zero-days” which is ... entirely meaningless marketing copy in the sense it’s being used: All vulns are discovered at some point, and are thus “zero days.” The articles don’t mention the token cost, but do mention the multi-month timelines, so we can assume that it makes the most sense when venture capitalists are buying tokens for you. We can reasonably expect the cost to fall quickly.
Image by midjourney, which, relevant to this post, missed half the prompt: “a photograph of a row of robots standing behind a conveyor belt. The robots are doing quality assurance. The belt is full of unique rube goldberg machines. The floor is littered with broken machines. Marketing photo, high production.”