Science isn't about Checklists
Over at Zero in a Bit, Chris Eng has a post, “Art vs. Science“:
A client chastised me once for making a statement that penetration testing is a mixture of art and science. He wanted to believe that it was completely scientific and could be distilled down to a checklist type approach. I explained that while much of it can be done methodically, there is a certain amount of skill and intuition that only comes from practical experience. You learn to recognize that “gut feel” when something is amiss. He became rather incensed and, in effect, told me I was full of it. This customer went on to institute a rigid, mechanical internal process for web app pen testing that was highly inefficient and, ultimately, still relied mostly on a couple bright people on the team who were in tune with both the art and the science.
Certifications only test the science.
I want to disagree strongly. Science isn’t about checklists. It’s about forming and testing hypothesis. In the case of pen tests, you have an overarching hypothesis, “this thing is secure.” You conduct experiments which demonstrate that hypothesis to be false. (Lather, rinse, repeat, you can’t test security in.)
The design of good experiments is an art. Some people are better at it than others. Great science is driven by a small number of great scientists who have both a comprehension that something is wrong with today’s theories, and a flair for great experiments which illuminate those issues.
The problem isn’t science versus art, the problem is checklist and bureaucracy versus skilled professional.
Halvar recently posted some reflections on intuition and experience.
You’re right there’s nothing unscientific about it.
Exactly. That’s why the “Risk Management: More Art Than Science” meme really ticks me off. Yeah, that art and science thing? They’re not opposites.
“The problem isn’t science versus art, the problem is checklist and bureaucracy versus skilled professional.”
Wonderful quote. I’m going to have to borrow it.
I agree absolutely. Science is a process. I am therefore reasonably happy with the adverb “scientifically”, but I usually find that the adjective “scientific” is used by people who don’t understand science. As in “there is no scientific evidence” or “this is uncertain therefore unscientific”.
I think the part that always gets under our skins when discussing the “art of testing” vs the “science of testing” is that most people believe science follows a rigorous format of testing hypothesis. There’s little art involved in verifying the effect of solution Ab-18215 on test subject T9946. You do the test, review the results and check accordingly.
But the truth is there is an art to taking the next step in the testing process. Unfortunately the typical penetration test time-lines are rarely as long as we’d like. Following every avenue is difficult because the project team has a deadline and they’re waiting for the “rubber stamp” from InfoSec to go live. That’s where the art and prior knowledge has to combine with the “science checklist” to ensure something doesn’t go live without any serious deficiencies.
It’s like chocolate and peanut butter, living together in harmony! Except not as tasty.
Arguing about whether penetration testing is an art or a science insults both art and science. Art is about creating beauty, science is about discovering universal patterns. A penetration tester who spends time trying to find a beautiful intrusion path rather than paths that work is wasting his client’s money.
If penetration testing were a science, people would be publishing articles that compare theories of defect origination and relate bug sources to penetration methods. This is a level of sophistication vastly beyond any pentest practice. Again, a pen tester who spends time evaluating different methods of discovering penetration paths to the same target instead of moving on to the next vulnerability is wasting his client’s money, possibly even unethically using that time for his own purposes instead of for the client’s purposes.
Many clients would not mind their data being used for research purposes if proper privacy was preserved, but this should be explicitly stated in the terms of engagement.
You go to a doctor or a lawyer, and they don’t fill out a checklist then give it to you to make your own decision. They use their professional experience to make a call and provide you with risks and recommendations.
If one could just fill out a health check list, we wouldn’t need doctors, we could just write software to be administered by clerks, but we don’t because a software checklist does not provide any assurance or value. Sure, it covers someones arse so that they can say, “we followed our process, and since we have met our commitment to follow the process, we are not responsible for anything else, other than following our process”, but unless a credible person brings to bear the weight of their professional opinion about risk and security, a checklist assessment is meaningless.
I work in both security and health care, and processes are valuable for getting repeatable results and distributing accountability away from individuals, but they do not provide quality or assurance. Only a professional opinion can provide that.
In one sense, penetration testing is the opposite of science–the results aren’t really replicable. Instead, three different skilled attackers will typically find different attacks, and even very good attackers/testers will miss some stuff that others find.