Appsec Roundup - March 2025
Big news for LLMs in threat modeling!
Threat Modeling
- Matthew Adams introduced TM-Bench “The World's First LLM Threat Modeling Benchmark.” I’m glad to see this, testing and evaluation is important.
- Tony Lee has released DeepTM, a tool for chaining threat models. (Tony was nice enough to help me find the core code for the agents.)
As a general comment on these systems, LLMs are tremendously reactive to very small wording changes. A useful example is in Adversarial Examples for Evaluating Reading Comprehension Systems, where the authors show that ‘distractor sentences’ dramatically alter reading comprehension. Those distractors are intentional, but spotlight a deep challenge with building or evaluating tools that work on natural language input. Don’t misread ‘this LLM or that one is “better”’ as “this chatbot will do better with the random questions you send. (This is one of the reasons that I’m excited to be working with the team at Irius on Jeff and Bex: the continuous improvement they’re investing in makes a huge practical difference.)
Appsec
- The International Obfuscated C Code Contest has announced The 40th anniversary of the IOCCC, IOCCC28, (are) open for submissions from 2025-03-05 23:19:17.131107 UTC to 2025-06-05 04:03:02.010099 UTC.
Shostack + Associates updates
- We’re sponsoring the Threat Modeling Connect #hackathon, going on now.
- Adam will be keynoting BSides Seattle (April 18/19, Seattle).
- Adam will be co-presenting with Tanya Janca at RSA: Red Teaming AI: 50 Years of Failure, But This Time, For Sure! - [IAIS-R03].
- There's a few slots open if you want to meet with Adam at RSA.
- Adam and Erik will be training at OWASP Global Appsec Barcelona (May 27-28), and seats are roughly half-gone.
Image by Midjourney: “a photograph of a robot, sitting in a library, working on a jigsaw puzzle”