Defenses: Protect, Detect, Respond and much more (Threat Model Thursday)
How do you choose defenses when you can’t cover everything and useability is something you shouldn't trade away?
A major theme of the new edition of Threat Modeling is that security is subject to engineering tradeoffs. Those tradeoffs happen both at a macro level (“How much security do we need?”) and at a micro-level (“how should we address this threat?”).
Security purists may wish this were not the case, but engineering always has the iron triangle of “fast, cheap, good: pick any two.” Not engaging with that means security gets bolted on, poorly. This also informs the frequently invoked “we can’t fix everything” objection. The rest of engineering knows that tradeoffs will be made. One of the primary ways to manage that is “wait and see,” which requires a way to see, and imposes observability requirements.
Engineers know they’re always making tradeoffs, and Chapter 9 covers a set of important technical properties for the defenses we’re incorporating into our delivery. The list comes after discussions of customer-appropriateness and security versus compliance, and includes:
- Goal: protect, detect, respond
- Goal: reduce likelihood or impact
- Simple to complex
- Reliable or probabilistic
- Strong or weak (passwords vs. auth apps)
- Chess move or checkmate
- Proven or innovative (and of course, disproven)
- Fail open or closed
- Build or buy
- One or many
- Automated or manual
- Effort to design, implement, maintain, or operate
This list evolved as I worked on the book, and I’ve noticed that the choices associated with LLM defenses are informative. LLM guardrails are typically probalistic, weak, chess moves, “innovative” in the sense of unproven, fail-open, and high effort to maintain and operate.
While the list wasn’t focused on LLMs, the description that emerges of guardrails is useful, and captures much of the discomfort that I hear people talking about.
Being able to characterize emergent systems is useful in a list like this, and while this edition is going to the printer soon, all models are wrong, and some are useful. How can we make this list more useful?
Note that can should trade
off. An unusable system is an invitation
for people to stop using it, go elsewhere,
or work around it. (Which is why the book
continues to have an entire chapter on
Human Factors and Usability, even as other
chapters were retired to the website.)
Image by midjourney: A watercolour painting of a friendly, simple robot kneeling beside open carryon luggage, with an overly large pile of items that won't all fit. The robot holds an iteme in one hand and rests the other hand on its chin, deciding what to pack. Use deep indigo purple, lime green, bright orange, and red-orange. Calm, thoughtful mood.