CSI/FBI Survey considered harmful
The latest 2006 CSI-FBI Computer Crime and Security Survey has been released.
Already, it is making waves, as it does each year.
I want to simply state that there is no reason to give this survey any credence.
The survey instrument is sent only to CSI members. This time, it was sent to 5,000 of them. There is no reason AT ALL to think that these people are a representative sample of infosec practitioners, or that their employers are representative of employers generally. Think of every infosec practitioner you know. Now think of that person’s boss…and that person’s. When you reach the C-suite, stop. Is any of those people a CSI member? I didn’t think so.
The overall response rate for this survey was just over 12% (616 of 5000). Were the 12% who did answer different in any other way from the 88% who did not? We do not know, because the report doesn’t tell us.
Of course, professionals, notably physicians, are difficult to survey effectively. Despite this, real survey research [link to http://www.norc.uchicago.edu/studies/health.htm no longer works] is actually done. As an example drawn from the link above, one survey of US physicians which (like CSI’s) was done through the mail, got a response rate of 62%. That isn’t perfect, but the survey concerned “physicians’ attitudes and behavior concerning physician-assisted suicide and voluntary active euthanasia”.
That’s right — it asked doctors if they had killed any patients on purpose — and they got a 62% response rate. CSI, surveying its own membership — 12%.
Enough, already.
A computer scientist, an economist, and a survey researcher need to gang up on this. The economist and CS guy can get the NSF money, and the survey researcher can spend it the right way: on a statistically valid sample and techniques proven to increase response rate. They all can put together a decent instrument, the survey researcher can quantify the extent to which their conclusions might be tainted by non-response bias or sampling error, and later the economist can have some fun with Stata and grab some headlines for the group. They could even make the data [link to http://www.icpsr.org/org/index.html no longer works] available for the rest of us to work with.
[stupid math error — 616/5000 != .3 — corrected]
I get the survey in the mail every year and always fill it out truthfully for my home network…