Shostack + Friends Blog Archive

 

Why I Don't Like CRISC, Day Two

Yesterday, I offered up a little challenge to suggest that we aren’t ready for a certification around understanding information risk.  Today I want to mention why I think this CRISCy stuff is dangerous.

What if how we’re approaching the subject is wrong?  What if it’s mostly wrong and horribly expensive?

I’m going to offer that we’re still too early on to know the answers to these questions (an offer that if correct, would also serve to prove my point yesterday about CRISC).  But if it turns out that we are doing things incorrectly (and really, what’s the probability that we are doing risk management correctly) – does something like CRISC make it easier or more difficult to change to something more effective?

Obviously, you don’t have to have a degree in Organizational Behavior to identify the problem here. If our approach to risk management is wrong, then CRISC is only going to serve to ensure that we are set in our incorrect ways.

Now where this should *really* upset you, my dear reader, is if you subscribe to various theories about how sciences progress.  If you believe that sciences progress by sporadic, somewhat instantaneous little revolutions – then we’re totally screwing ourselves by creating a bureaucracy that makes it more difficult for the next revolution to take place.  And believe me, as I’ve found out over the past 4 years, creating that revolution in risk management is hard enough already.

14 comments on "Why I Don't Like CRISC, Day Two"

  • Meh. Please denote the sarcasm below and I promise to avoid a rant.

    “Compliance”, “Risk Management”, “whatever-else” … Same thing! Different terminology; Different year — MAY-BE — but the words mean the same thing each time.

    I’m a security {product|solution|service} vendor and I want to sell the same things I’ve been selling for almost 15 years. Get out of my way!

  • Sorry in advance for my rambling here…

    Regarding Andre’s comment, I disagree. Compliance != Security != Risk Management.

    Compliance is an external entity saying we must be this tall to ride. It’s a component of that entity’s risk management, perhaps, but not our own. Instead it presents additional risk–any effective compliance regime must have consequences for failing to meet requirements.

    Security sometimes feeds into the implementation of compliance, but I as a security guy don’t spend all of my time focusing solely on compliance. Security is also an implementation of parts of risk management, which I think is where the “I Don’t Like CRISC” line is coming from. But Security does not encompass Risk Management.

    Risk Management is a practice that is well established in other areas, but security and compliance people don’t really understand that, because they don’t work in that realm. Classic risk management (tongue-in-cheek) as far as I understand it pertains more to insurance and mitigating financial loss than anything else.

    But perhaps that’s our disconnect. I say “our” because I’m not sold that a certification from ISACA is the right thing. We’re not good at estimating costs of IT failures. That’s not our job! But if you find the right actuary and have her sit down and calculate the cost of losing three file cabinets full of medical records, they’re probably going to come up with a number based in reality. Perhaps the CRISC has a place, but perhaps it would be a stronger certification if it were more explaining technology stuff to experienced risk management folks rather than teaching geeks to be risk managers.

    @jth

  • Alex says:

    @John

    “But Security does not encompass Risk Management.”

    Rather security is a subset of risk management – the side of the equation focused on frequency of incident – models and data that happen independent of the cost of an incident.

    Speaking of which: “We’re not good at estimating costs of IT failures. That’s not our job! But if you find the right actuary and have her sit down and calculate the cost of losing three file cabinets full of medical records, they’re probably going to come up with a number based in reality”

    It’s been my experience (small sample size warning) that once a process is established, this (impact) is the easy part of the equation.

    “Perhaps the CRISC has a place, but perhaps it would be a stronger certification if it were more explaining technology stuff to experienced risk management folks rather than teaching geeks to be risk managers.”

    You’re stealing the thunder of my final “I don’t like” post 🙂

  • Jay Jacobs says:

    Without knowing squat about the CRISC program, what if they weren’t doing it wrong? What if exam consisted of questions like “Please discuss the arguements for and against quantitative analysis methods in 500 words or less.” What if they attempt to align terminology so when someone says “threat” everyone nods with the same understanding?
    I guess my question (to turn it back around) is: is there something that could be done, perhaps through certification, to advance the evolution of risk management as a field of research?

  • Alex says:

    @Jay – Laughing – OK, I think that could happen, but I have on good inside authority that CRISC isn’t that way.

    I should probably *do* a post on what a CRISC could be in my little opinion.

  • Jay Jacobs says:

    I did some quick reading on it:
    Urs Fischer, chair of ISACA’s CRISC Task Force: “We conducted global research and found that enterprises are becoming more risk-aware and are looking to identify professionals who possess the skills to help them protect their assets and enhance their businesses. CRISC fills a gap that exists in the marketplace.”
    (http://www.net-security.org/secworld.php?id=8706)

    If that’s the motivation, then I couldn’t agree more with your posts. They want to make it marketable and palatable to the majority of businesses… which means status quo of perception-based-risk-management-selling-techniques.

    And I’d be interested in your little opinion on what it could be!

  • Oliver says:

    I did the CISA and now I am applying for CRSIC. I do not understand the grumbling voices. I have worked my whole life on working on an internal control framework for a big comp, introducing COSO (I read it) and had the SOX hype behind me. And boy, there are way too much people who say they understand RiskManagement. (also in those posts)

    CRISC should certify that you can apply a framework to an IT organisation, design execute, test an Internal Control Framework and you have the managerial skills to get people to work in a risk-aware culture (stop the tweaks which collaps the whole production but are necessary because it’s an easy fix for pilot fish customer). As this experience is (and always will be) hard to measure I am optimistic that the certification will lead to a constructive discussion. From a business side I still think IT has not yet achieved the state of entrepreneurship as there are too much decisions without a solid business case.

    • Oliver says:

      And one central thing you completely forgot:

      Proper Risk Management is: Balancing Control Costs to Business Benefit. So if you are talking about bureaucracy you either have someone who liked SOX or des NOT understand RM.

  • Alex says:

    O-

    Rather, *proper* risk management is the act of aligning the capability to manage risk, and the exposure to risk, to the risk tolerance of the data owner (yet another reason NOT to like CRISC, we can’t even agree on terminology, so let’s enforce a bad one!)

    • Oliver says:

      Who is interested in word gymnastics if you are knowledgeable to save the company money by having it’s risk managed by minimizing cost of the processes used to manage it’s risk.

      How much money does this (which is one! of many) terminology save the company? Right, nothing. And people do not understand it. If you got the idea behind it you can leave the word gymnastics to the risk theorists.

  • Mark says:

    Alex, great analysis on the CRISC certification. One of the unfortunate things about IT risk management is that every client/person has a different (non-standard) perspective on what risk management means to them in order to make risk-based decisions.

    Let me try use an over-simplified financial investment risks as an example and try to compare that to IT risk and then try to make some conclusions about IT risk management.

    If I’m an investment analyst, before I financially commit to a fund, I will perform risk analysis. This is likely modeled based on a risk formula that is devised to identify those factors that are most likely to represent a significant change in fund value (general market conditions, price to earnings ratio, market cap, etc, etc, etc…)

    Most successful financial risk analysts (that make the most money) carefully guard these models and formulas as they would have no advantage if they were to become public knowledge.

    The problems we have in IT risk management include;

    a) the complexity of risk factors. Each IT environment is unique in some way, and a different set of risk factors exists for each organization. Information assets, uses of the assets, business processes, technical implementation, geo-political factors, etc, etc. This prevents/impedes standardization of threats, the PCI council’s guidance is an attempt to standardize this for a specific business issue (primary account numbers) where it is easier.

    b) the will and desire to manage risk. Each organization that I’ve had experience in has a slightly different motivation for managing risk. Some its public perception, others its compliance/fines, others it’s legal losses. And even when two organizations seem to have similar desires to manage risk, there are different opinions on what is acceptable. “we can accept the loss of X customer records every year”. This difference in opinions prevents the standardization of IT risk acceptance and management decisions. If you want to have fun, take one of your internal auditors out for coffee and talk about IT risk management.

    These are just two problems that show in comparison with financial risk management IT risk management is a much more complicated problem. I would expect that if you compare IT risk management to any other risk management discipline (aerospace, medical, personal, military, etc) you’d find IT to be the most complicated.

    The problem I see with this is that the business/political impacts of IT risks (due to our crazy dependence on IT systems) are quickly catching up with those others.

    I would also love an open debate/discussion on the needs to improve IT risk management. I offer a few suggestions.

    – Standardization of IT risk factors (are there really any common ones?)
    – Standardization of IT risk impacts
    – Public tracking and reporting of loss (how can we ever hope to determine probability based on individual experience?)

    Thanks!

  • Isaac says:

    Sorry, it’s a bit of a ramble.
    From a verdict perspective I’m still out on the CRISC. The problems highlighted around the certification, motivation of ISACA behind the certification and the lack of a defined, concrete body of knowledge (that’s agreed upon within IS especially) is not endemic to risk management.

    For those of us in regulated industries the mandates for risk management modeled on, among other things, NIST special publications, law & regulatory agencies are growing and these mandates provide some definition around the expectations for performing risk assessments.

    The concept of Information Assurance as a whole suffers from a lack of a “body of knowledge” that is agreed upon and as we enter the realm of enterprises jumping on the cloud wave it may well come to harm us in the near future. As a practitioner in a regulated industry (Insurance) I need to know where my data is, that my data is purged when I said purge it and that I can answer audit, eDiscovery and investigatory requests truthfully. All I can say about SaaS, IaaS and PaaS is really read your contract before you tell me that you absolutely know where your digital assets are. (If you use Google I can give you the answer to that now but you won’t like it.)

    I’m a bit leery of any new certification because it will be only time that tells us if it’s a bastion for “paper tigers” or actually demonstrates a fundamental understanding of a functional area/product.

    The point of all of this is risk management needs to begin to coalesce around a set of common goals and expectations using quantitative metrics we can talk with the business about. While you may think it’s not your job to do this for your business leadership you will regret not doing it. If you can’t demonstrate the value of your services & why they may need mitigation planning to protect them against foreseeable disasters you will ultimately be far less effective than you could have been.

    As to the CRISC being fatally flawed, it’s too early to tell but I wouldn’t put it as a mandate for a corporate risk specialist yet either.

  • Alex says:

    Isaac,

    Thanks for the thoughtful post.

    “The point of all of this is risk management needs to begin to coalesce around a set of common goals and expectations using quantitative metrics we can talk with the business about”

    My problem with this is that we’re in a much more difficult position than natural sciences or financial services in terms of the breadth and width of modeling necessary – *if we continue with the same approach to risk management* (which of course, ISACA will probably cement by minting out your “paper tigers”.

    If, on the other hand, we let 100 flowers bloom in terms of modeling and measurement and approaches to decision making – well, we might even come across an approach or two that actually, you know, doesn’t involve multiplying ordinal rankings.

    As long as coalescence doesn’t negate innovation, we’re good. Once coalescence begets bureaucracy, we’re screwed.

    The question is, then, which will the CRISC from ISACA drive, innovation or bureaucracy?

  • Bill Clancy says:

    I think risk is often talked about, but seldom undrestood. (Myself included). I’m as guilty of collecting certs as the next guy, but I figure if industry likes a string of letters after my name, I’ll play… especially if work is willing to finance my effort.
    As far as the CRISC itself is concerned, I’m glad someone is paying attention to the science of risk. Places I work (DOD installations) always talk about risk, but are mostly clueless when actually trying to quantify it. Time will tell if the CRISC can mature past it’s current infant state.

Comments are closed.