Lessons from Threat Modeling Intensive With AI
Actionable lessons from delivering Threat Modeling with AI, and using AI more generally.
Obviously LLMs are changing our world. We aspire to respond with more nuance than “skeptic” or “booster.” We make practical use of these tools as we run a growing company. And we’re not only thinking about how we use them, we’re having lots of conversations with clients and prospects about how they’re using LLMs and how they hope to be able to use them. All of which leads directly to us experimenting with new courses.
At the last OWASP Global Appsec conference, we delivered the first version of our new “Threat Modeling Intensive with AI” course. This is part of how we’re exploring LLMs and their impact on our work, and so I want to start with lessons from that delivery:
“Before this course, I would have accepted this as a good threat model!”
- First and foremost, we learned that LLMs are unreliable when you vibe threat model. The tasks are too broad for asking ChatGPT for a threat model. One of our students fed the same exact prompt to ChatGPT three times, and got 20 threats on one run, and 44 on another.
- We saw that no one wants to review the output. In some cases, people were getting upwards of 50 pages out of the chatbots. Some folks learned to tune the output with prompt elements like “be concise,” “provide a bullet list” or even “few-shot prompting” with examples of the sort of output they wanted. (They also got a lot fewer imaginary risk tables!) Mike Ensign even built and shared rulesets as part of the course (and follow-up afterwards!)
- When people did review the output, one of the most fascinating things we heard was “Before this course, I would have accepted this as a good threat model!”
- We saw that people enjoyed the experience of using LLMs to threat model. That’s important, and worth teasing apart because people like doing fun things. (That’s both obvious, and obviously not obvious to the makers of most enterprise software.) That’s probably a future post.
Our experiments aren't limited to that course, or even one course. We’re constantly experimenting to see if we can improve customer experiences by using LLMs. Our experiments lead us to be solidly in the “measure outcomes, not use” camp. What we find, consistently, includes:
- Rewriting tricky emails? Awesome.
- Models vary dramatically in their ability to accomplish simple tasks like “list action items from a meeting.” Yesterday, one of them built four separate bullets for a single “make an introduction” task.
- Accidental focus lock continues to be a problem. Look closely at what happens after “Jira” in meeting minutes.
- The bigger the task, the less reliable the results.
- LLMs behave strangely, and even though the prompt for this post’s header image moved from post-its to whiteboards, we got post-its.
This lines up pretty closely with what we see in the training results.
In that spirit of experimentation, we’ve built and delivered Threat Modeling Intensive with AI, a three-day course focused on how to threat model and how to leverage LLMs as you get more proficient and as you scale. We’ve built Threat Modeling AI Systems and are delivering it next month. That’s a two-day course for people who know how to threat model and want to dive deep into the specifics of how to apply that knowledge to AI systems. And we’re working on a “complete AI” edition for Blackhat, with baseline threat modeling skills from our Intensive, buttressed with using LLMs to threat model, and threat modeling LLM systems. (BlackHat's early pricing is available on that course until May 22.)
All of these new courses are part of how we’re learning and sharing our journey in these disruptive times.
Image by Midjourney: “adult students working with laptops and whiteboards. There are robots in the classroom, distracting learners. clean workspace, watercolor paint style, vibrant cool tones, minimal distraction, modern educational setting.”