Focus on high priority threats(?)
It’s easy to think prioritization is an easy problem, but it’s one deserving careful consideration.
“We need to focus on high priority threats!”
We hear this all the time. In fact, it was practically a refrain at a recent National Academies Forum on Cyber Resilience meeting on Securing AI systems. And it’s obvious. Given the challenges and the practical speed at which companies are deploying AI systems, we need to move quickly. Oh, and I also think it’s... worrisome, and quite possibly dangerous.
It’s worrisome because we might encounter one or more of:
- The threats prioritized are those that come to mind first.
- The search is stopped when a few apparently high-priority items have been found, or the “fix budget” seems full.
- Threats are implicitly prioritized, and likelihood or impact is mis-understood. Ignored threats are never revisited.
One of the advantages to modern threat modeling (compared to brainstorming, abuse cases or even attack trees†) is that tools let us be structured in our analyses. We’re not relying on a glance and an instinctual claim of “these are the issues,” we’re walking through (for example) a DFD and using STRIDE to perform an analysis, and we can expect a certain number of issues to emerge. There are also deeper analysis methods like STPASec or using the Berryville Institute’s LLM ARA.
If we separate the questions of “what can go wrong” and “what are we going to do about it,” then we can include a prioritization step within “what are we going to do about it.”
Structuring our approach let us be confident we’re really getting the high priority threats. What’s more, it’s important to specify and understand the prioritization scheme that’s in use. A great deal of academic work on LLM-security focused on model theft, a priority threat to those creating models, but perhaps less important than helping plan mass shootings. There are at least questions of ‘who’s hurt’ and ‘how the various harms relate to the motivations of AI tool creators.’
We should evaluate our prioritization approaches. This can be tricky, because the issues which we prioritize may be prevented by defenses we build. But in a research setting, we can evaluate aspects like intra-rater consistency, and even same-rater consistency (how self-consistent people are over time).
We can also evaluate threat discovery techniques. We can ask:
- Does this tool give consistent answers?
- How long does this tool take to use?
- What skills are needed to use this tool?
Research questions that I believe a National Academies Panel should consider include:
- What are the threat discovery techniques and what are their properties?
- How do “fast and cheap” methods relate to more structured methods? What is the “effort” to “do more” and what does that “more” get us? For example, STRIDE is very general and may not provide LLM-specific analytic capabilities any better than brainstorming about LLMs. In contrast, the Berryville LLM ARA should provide both more specific threats. (This question is about threats, in contrast to the next, which is about defenses.)
- What’s the relationship of specific analysis structures to defenses? Do the more specific threats lead to more specific defenses, or is the current set of defenses small enough that there’s limited benefit?
- What are the prioritization schemes in use, how consistent are they and what biases might they encode?
- How can we measure the reliability of prioritization methods?
† Some might be surprised by the claim about attack trees. To be more specific, by themselves, trees provide a way to record an analysis, but no guidance that leads to discovering “there’s another way to achieve this goal.” Attack trees are frequently used in conjunction with other techniques that provide more structure.
Image Credit: Midjourney, “a photograph of an army of robots attacking and someone trying to prioritize”