Base Rate & Infosec
At SOURCE Seattle, I had the pleasure of seeing Jeff Lowder and Patrick Florer present on “The Base Rate Fallacy.” The talk was excellent, lining up the idea of the base rate fallacy, how and why it matters to infosec. What really struck me about this talk was that about a week before, I had read a presentation of the fallacy with exactly the same example in Kahneman’s “Thinking, Fast and Slow.” The problem is you have a witness who’s 80% accurate, describing a taxi as orange; what are the odds she’s right, given certain facts about the distribution of taxis in the city?
I had just read the discussion. I recognized the problem. I recognized that the numbers were the same. I recalled the answer. I couldn’t remember how to derive it, and got the damn thing wrong.
Well played, sirs! Game to Jeff and Patrick.
Beyond that, there’s an important general lesson in the talk. It’s easy to make mistakes. Even experts, primed for the problems, fall into traps and make mistakes. If we publish only our analysis (or worse, engage in information sharing), then others can’t see what mistakes we might have made along the way.
This problem is exacerbated in a great deal of work by a lack of a methodology section, or a lack of clear definitions.
The more we publish, the more people can catch one anothers errors, and the more the field can advance.
Adam,
Thank you for your kind words.
You bring up two very important points:
1) As we discussed in Seattle, you have to use these concepts in order to master them – practice makes more perfect.
2) It is vital that we learn to embrace our errors and failures as much as our successes. Yes, it’s important to succeed, and I wouldn’t recommend that anyone actively seek out failure. But when failures occur, we can learn a great deal.
Best wishes,
Patrick