How Do You Grade AI Work That Has No Right Answer?

Imagine that you’re a business leader about to sign a deal for an AI tool that drafts business cases. Ask it a specific question - “Should we enter this market?” and it evaluates the problem within the context of your company's situation and its industry and comes back with a structured recommendation: the case for, the case against, the risk involved and a final call.

The memo is well written, in a way that would make any business school graduate proud. Which is great, but here’s the problem: how do you know that it’s a good recommendation? Unlike a university class, there is no answer key. You could of course try out the tool directly, once, twice, even three times. But even if the output seems on track, how do you know that this will work equally well over time? Under different conditions?

Right now, most companies answer that question the only way they can: on faith. They like the demo, they trust the vendor, so they pull the trigger and deploy, hoping for the best.

This is not a flaw in the tool. The problem is that AI now produces a new kind of work that's open-ended, with no single right answer and no unit test that goes red or green. The natural instinct is to shrug and call it subjective. That’s tempting, but incomplete: "no right answer" does not mean "no standard for quality”, it means that the standard isn't sitting in an answer key waiting to be looked up.

Capture The Judgment, Don't Invent It

So you can't just “take a look”. Eyeballing doesn't scale, for two reasons:

• Every output is fluent and well-written, so flaws get hidden.
• Every output is different: unlike ordinary software, a language model can return a different answer every time you ask, even with identical inputs.

How do we handle this in our daily work life? An experienced person already has a standard; they know a market-entry case needs sound reasoning, a fair hearing of the downside, and evidence where evidence is possible. That judgment is usually tacit, living in one person's head.

So the move for evaluating the AI output is to capture that judgment. Write it down as a rubric: a set of named axes, each defining what strong versus weak looks like. Then the AI applies it at scale, consistently, the same way every time. It doesn't invent what "good" means for each iteration; you do, only once.

One note of caution: the axis scores are meant to be a diagnosis, not a formula. You don't average four numbers into a verdict. A case can score well on most axes and still be wrong where it counts most - the scores tell you where to look, not what to conclude. It's how you would grade a capable new hire's first memo: not a thumbs-up, but "the logic holds, the downside is thin, the evidence is missing”, judging it implicitly against a standard you already carry.

One Standard, Three Jobs

That one rubric then pays off three ways. It tells you whether today's memo is good enough to act on (accept), whether a change to the prompt or model quietly dropped quality (regression), and where the work is weakest, so you know what to fix next (improvement). And that last job is the real payoff: it isn’t just an acceptance gate, it’s how you make the tool itself better over time.

Setting Limits

Two limits for this approach keep it effective:

1. Grade the work, not the outcome. A sound market-entry case can still lose money - say, the market shifts or a rival moves first - so the result can't tell you whether the reasoning was sound. We want to grade the thinking that was knowable at decision time, not the verdict history eventually delivers.
2. Use it where quality can’t be counted: The more a judgment collapses toward something countable - is the clause in the contract, is this supplier cheaper - the less you need any of this; a simple checklist does the job. Reach for the heavier, multi-axis approach when quality is real but not measurable.

None of this needs an answer key. Instead, make "good" explicit and capture it as a repeatable rubric - one standard doing three jobs. It's the same move as reading your codebase once and projecting many documents from it: do the expensive thinking once, then reuse it many ways.

This is one part of the solution. What we’ve bought with this approach is a consistent standard - the same work judged the same way every time; but by itself that’s not sufficient. It is not the same as being right. The rubric is a proxy, and the AI applying it is itself a fallible judge that can be confidently wrong.

So who checks the checker? That's the subject of the next post.