Why “The AI Explained It” Isn't Good Enough: Introducing the SCORE Framework
Exploring what it means for an AI to explain itself, and why “it gave a reason” is not the same as accountability.
By Michael Novack, AI Security & Safety Engineer at Cranium
Trainers at Shostack + Associates, like Michael, are experienced security practitioners working in the field. We're excited to see his talk in Vienna and are thankful to Cranium for allowing us to highlight the work he'll be presenting there.
A medical AI system recommends denying a patient treatment. The doctors hesitate, something feels off, but they can't articulate why. Later, an audit reveals what the model had quietly learned: zip codes correlate with health outcomes. The system had been penalizing people from lower-income neighborhoods, and nobody caught it because nobody could see inside.
This isn't a hypothetical. It's the kind of failure that happens when we treat AI explanation as an afterthought rather than a requirement. And it's exactly why explainability needs a structured standard, not just a shrug and a "the AI said so."
Explainability Isn't Optional Anymore
The pressure to explain AI decisions is coming from multiple directions at once. The EU AI Act, GDPR, and the U.S. AI Bill of Rights all treat transparency and traceability as foundational, not optional. Auditors and regulators want to know how a decision was made. End users want to understand why. And developers need to see which inputs drive outputs in order to catch errors before they cause harm.
But beyond compliance, there's a simpler reason explainability matters: if you can't explain what your AI is doing, it's running your business, not you.
Introducing the SCORE Framework
The AI Explainability Scorecard is a structured way to evaluate how effectively any AI model communicates its reasoning. It rates models across five criteria, each on a 1-5 scale, with the final score as the average across all five.
The five criteria form an acronym: SCORE
The framework is built to answer two questions: Can you trust the results of AI explainability? And does that explainability actually drive decisions? Sound and Consistent determine whether you can trust it. Optimize, Readable, and Easy determine whether it drives decisions.
S - Sound
Is this explanation actually how the model made its decision?
Soundness asks whether the explanation reflects the model's real logic or is just a plausible story built after the fact. A chain-of-thought response from an LLM might read as coherent while having little connection to the actual computation behind it. A high soundness score means the explanation matches what the model actually did.
C - Consistent
Would the model explain similar decisions the same way?
If the same input produces different explanations on different runs, the system can't be reliably audited or debugged. Consistency requires that explanations are stable across identical or similar inputs. Deterministic models tend to score well here; probabilistic generators often don't.
O - Optimize
Does this provide actionable insights to improve the AI system?
An explanation that clarifies a decision but offers nothing to act on has limited value. Optimize asks whether the explanation surfaces information you can use: adjusting training data, correcting biased correlations, or tuning model behavior. High scores here mean explainability feeds improvement, not just understanding.
R - Readable
Does this help a non-technical expert understand and act?
Doctors, judges, loan officers, and compliance teams all interact with AI outputs. Readability asks whether the explanation works for them, translating model behavior into terms that inform real-world judgment. A technically accurate explanation that only ML engineers can parse scores low. One a domain expert can act on scores high.
E - Easy
How costly is it to access the model's explainability?
Some techniques are computationally expensive, require specialized infrastructure, or are too complex to run in production. If explainability is theoretically possible but practically out of reach, it won't get used when it matters. Ease of access determines whether explainability is a real tool or just a checkbox.
Why This Framework Matters
The SCORE framework makes a few things possible that vague assertions about "transparent AI" don't.
It makes explainability comparable. Rather than debating whether one system is "more transparent" based on intuition, teams can score models against the same criteria and compare results objectively.
It makes explainability appropriate to context. Not every AI use case needs the same level of transparency. A high-stakes medical decision system needs to score very differently than a low-risk content recommendation engine. SCORE helps organizations calibrate expectations to actual risk.
And it exposes the gaps between explanation and reasoning. The framework makes it much harder to paper over a model's opacity with a convincing narrative. If the explanation isn't sound and consistent, the score reflects that, regardless of how readable the output happens to be.
Hear More at OWASP Global AppSec EU
The AI Explainability Scorecard and the full model comparison analysis behind it will be presented at OWASP Global AppSec EU Vienna 2025.
The session goes deeper on how the SCORE framework applies across model types, from simple K-Nearest Neighbor classifiers to neural networks and large language models, and explores advanced techniques for making LLM usage more explainable.
If you're working on AI security, governance, or responsible deployment, this is the conversation you want to be in.
Register and add the session to your schedule!
Michael Novack is an AI Security & Safety Engineer at Cranium. His work focuses on making AI systems safer, more accountable, and easier to understand for the people who build them and the people affected by them. Michael is also an course designer and instructor at Shostack + Associates.
Image by midjourney: "A glowing lattice of interconnected token nodes and attention layers, forming a vast digital architecture, overlaid with a scorecard grid, dark cyberpunk control room background, faint human silhouette at center. Cool blue and amber tones, cinematic lighting."