How We Fixed an AI Eval That Said 50% When the Model Was 97% Accurate

One Sunday morning, our weekly accuracy check said our document classifier had fallen to 50%. Nobody had touched the model, the prompt or the code that calls it. The model was fine. The test was broken.

You've probably had a version of this. A test suite that was green on Friday comes back half red on Monday. You open the failures expecting a real bug, and every one of them says the same thing: record not found. The report counts them as failures, the chart shows a cliff, and for the first ten minutes you're debugging the wrong thing.

Multibagg AI is an AI-powered stock research platform for Indian investors. Every document that reaches us is sorted by type before search and alerts use it, so a weekly score that cries wolf either sends us chasing a healthy model or teaches us to ignore the alarm.

Here's how a database change I had shipped three days earlier quietly broke half of our test set, how we got the references back without relabelling anything, and what the cleanup found hiding in the labels.

The score halved, and the model never moved

The classifier is a single AI model call. It reads a document and returns a confidence flag and one of four types: an annual report, a call transcript, a presentation or a general announcement. A teammate built it. I built the weekly check around it.

That check is an eval: a script that runs the model on documents where we already know the right answer, and reports how often the model agrees. The documents and their right answers are the golden set. Ours has two parts: a short list of hand-picked hard cases that are always included, and a large pool of labelled documents. The script tops each type up to 125 from the pool, with a sample seeded by the date. Four types at 125 each makes 500 items. The job runs every Sunday, posts the result to our team chat and flags anything under 95%.

That Sunday the message said 50% accuracy, 250 of 500, with 257 documents classified and 243 failed. Here are the numbers next to the re-run after the fix:

The model's accuracy on the documents it actually read was 97.3% both times. What changed was how many documents it got to read.

The bottom row is the whole story. On the 257 documents the model read, it got 250 right. The other 243 never reached the model at all.

How 243 failed lookups became 243 wrong answers

The eval works in two steps. First it looks up each golden document by its ID, downloads the file and asks the model for a type. Each outcome goes into a results file as one line: the predicted type, or an error if anything failed along the way. Then a scorer compares the results with the golden labels.

The scorer skipped error lines when it loaded the results, then walked the whole golden set and counted the correct answers. A golden item with no result simply wasn't correct. In generic form, it looked like this:

interface Prediction {
  id: string;
  label?: string;
  error?: string;
}

export function accuracy(golden: Map<string, string>, predictions: Prediction[]): number {
  const answered = new Map<string, string>();
  for (const prediction of predictions) {
    if (prediction.error === undefined && prediction.label !== undefined) {
      answered.set(prediction.id, prediction.label);
    }
  }
  let correct = 0;
  for (const [id, expected] of golden) {
    if (answered.(id) === expected) correct++;
  }
   golden. ? correct / golden. : ;
}

Dividing by the size of the golden set, not by the number of answers, is a fair choice. If the model starts failing on half the documents, you want the score to fall, and that's exactly why the alarm went off. The cost is that the number can't tell "the model got it wrong" from "the model never saw it", and those two need very different fixes.

flowchart TD
    A["Weekly eval samples 500 golden items"] --> B["Look up each document by its ID"]
    B -->|"ID found in the new table"| C["Model reads the file and returns a type"]
    B -->|"ID only in the retired table"| D["Error line: not found"]
    C --> E["Compare with the golden label"]
    D --> F["Scorer skips error lines"]
    F --> G["No answer, so the item counts as wrong"]
    E --> H["Accuracy = correct / 500"]
    G --> H

The message showed the failed count, but not why those items had failed. The reasons lived only in the results file on the server, and the next weekly run wipes that file. So I asked the AI coding assistant I work with to re-run the eval locally on the same day's sample and group the errors by reason. Every one of the 243 had the same reason: not found. The local run matched production almost exactly, with 252 right, 5 wrong and the same 243 failures.

I had moved the table out from under my own test set

Three days earlier, I had shipped a change that merged several of our document tables into one new table and switched every reader over to it. The lookup the eval uses was one of those readers. Before the change, it checked the old table and the new one. After it, only the new one.

Most documents kept their IDs in the move. Some didn't: they landed in the new table under a different ID. The golden set still held the old ones. A read-only check in the database settled it. All 243 failed IDs existed only in the retired table, and none were in the new one. Across the whole golden source, not just that week's sample, 1,009 of 1,973 records pointed at retired IDs. That's 51%, which is why a sample of 500 lost almost half its items.

The error message didn't help. It still said the document was missing from both tables, though the code now checked only one. The message described the code as it was before my change.

Remap by fingerprint, don't relabel

We had two obvious options, and I didn't like either. Copying the 1,009 rows back into the new table would undo part of a migration for the sake of a test. Deleting them from the golden set would throw away half of the labelled data, much of it labelled by hand.

A third option needed no new labels. Every document row stores a content hash, a fingerprint of the file's bytes, along with the ID of the company that published it. Call that the owner. So for every golden ID missing from the new table, we could read its owner and hash from the retired table, find the live row with the same owner and hash, and swap in that row's ID. The labels stay exactly as they were.

Before writing any code, I ran a read-only check: all 243 failed IDs had a hash, and each one matched exactly one live row with the same owner. The coding assistant wrote the script. Its first version handled five kinds of edge case and logged every swap. I told it to keep the code simple, and it came down to about 50 lines. The core, in generic form:

interface StoredDocument {
  id: string;
  ownerId: string;
  fileHash: string | null;
}

const fileKey = (doc: StoredDocument): string => `${doc.ownerId}:${doc.fileHash}`;

export function remapIds(
  goldenIds: string[],
  liveIds: Set<string>,
  retired: StoredDocument[],
  live: StoredDocument[],
): { remap: Map<string, string>; unmapped: string[] } {
  const liveIdByKey = new Map(live.((doc): [, ] => [(doc), doc.]));
   remap =  <, >();
   ( doc  retired) {
     liveId = doc. ? liveIdByKey.((doc)) : ;
     (liveId) remap.(doc., liveId);
  }
   unmapped = goldenIds.( !liveIds.(id) && !remap.(id));
   { remap, unmapped };
}

Matching on the owner as well as the hash matters. Two companies can publish the same file, and a hash alone would happily point one company's document at the other's. The simple version has one weak spot: if two live rows shared an owner and a hash, the map would keep the last one without saying so. I re-checked that while writing this. Each of the 1,009 retired IDs matches exactly one live row, so the script never had to choose.

We ran it over the whole golden source, not just the sample. It rewrote 1,009 IDs and left none unmapped. The diff touched nothing but IDs. The same-day re-run had zero failures and scored 473 of 486, or 97.3%.

The remap created duplicates, and two of them disagreed

486, not 500. The remap had created 54 duplicate IDs. Different old IDs that carried the same file now pointed at the same live document. Every one of them had a hand-picked copy and at least one pooled copy of the same document. The sampler counted both copies toward its 125 per type, then the loader kept only one, so the run shrank to 486 without an error. The hardest-hit type got 119 items instead of 125.

Two of the duplicates were worse: their copies carried different labels.

The first was one file attached to three separate filings by the same company. It went out once under a title that said "annual report", then twice more, four minutes later, attached to a meeting notice. A title rule had labelled the first copy an annual report. A person had labelled the other two a general announcement. The second was a single sustainability report listed twice: a rule (over 100 pages, with "report" in the title) called it an annual report, and a person called it a general announcement.

Before the remap, each disagreement was invisible, because each copy had its own ID, though at least one copy of each was already mislabelled. After it, the scorer graded against whichever row happened to load last. One of the 13 wrong answers in the 486-item run was the first file: the model called it an annual report, and the row that loaded last said general announcement.

The coding assistant downloaded every copy and hashed the bytes: identical files, every time. Then I opened them myself. The classifier's stored answer for each document was right there in the data, and I told the assistant to ignore it. A label you pick by looking at the model's answer just teaches the test to agree with the model. The first file is an annual report. The second is a general announcement. We kept one row for each.

The other 52 duplicates had matching labels, so the fix was mechanical: delete the pooled copy, keep the hand-picked one. That removed 56 rows, because one ID had five pooled copies. A check like this one found both kinds of collision:

interface GoldenRow {
  id: string;
  label: string;
  source: "pooled" | "handPicked";
}

export function findCollisions(rows: GoldenRow[]): { duplicates: string[]; conflicts: string[] } {
  const byId = new Map<string, GoldenRow[]>();
  for (const row of rows) {
    byId.set(row.id, [...(byId.get(row.id) ?? []), row]);
  }
  const duplicates: string[] = [];
  const conflicts: string[] = [];
  for (const [id, group] of byId) {
    if (group. < ) ;
     ( (group.( row.)). > ) conflicts.(id);
     duplicates.(id);
  }
   { duplicates, conflicts };
}

flowchart TD
    A["Golden ID missing from the new table"] --> B["Read owner and content hash from the retired table"]
    B --> C["Find the live row with the same owner and hash"]
    C --> D["Swap in the live ID, keep the label"]
    D --> E{"Is that ID already in the golden set?"}
    E -->|"No"| F["Done"]
    E -->|"Yes, same label"| G["Delete the pooled copy, keep the hand-picked one"]
    E -->|"Yes, different label"| H["Open the file and label it by hand, without the model's answer"]

The golden source went from 1,973 records to 1,914, every ID unique. That's 3% smaller. Replaying the sampler on the cleaned data fills all four types back to 125, so the weekly run is back to 500 distinct documents.

Failures now arrive with their reasons

The first diagnosis needed a local re-run only because the reasons were stuck in a file on the server. So the fix also writes a small list of failed items (ID, expected label and error), and the weekly job uploads it together with the full results to file storage and links both in the chat message. A week with no failures uploads only the results.

My first design let an upload failure fail the whole job. An AI code review I ran on the change flagged that: one storage hiccup would hide the week's accuracy. I agreed. Each upload now catches its own error and logs it, the message says "upload failed" in place of the link, and the score always goes out.

What the numbers say now

• The headline. The reported accuracy went from 50.0% (250 of 500) to 97.3% (473 of 486), with failures going from 243 to 0. The model's accuracy on the documents it actually read was 97.3% before and after. Scored against the corrected label for the first conflict file, the re-run is 474 of 486.
• Every reference resolves. While writing this, I queried the database again (read-only). All 1,914 golden IDs exist in the live table, and none are missing. Before the fix, 1,009 of 1,973 were.
• A second measurement is consistent. Production stores the type the classifier gave each document when it first arrived. For the 1,914 golden documents, that stored type matches the golden label 1,889 times, or 98.7%. It's a different measurement from the eval (older prompts, no fresh model call), and it lands in the same range, 98.7% against 97.3%.
• The real misses are visible now. All 13 wrong answers in the re-run were general announcements the model called an annual report (8) or a presentation (5), and it marked all but one of the 13 as low confidence. One of the 13 was the first conflict file, where the model's answer turned out to be right, so 12 are real misses. That's actual model work, and the eval can finally see it.

What the fix costs

The run costs more. Model spend per run went from $0.42 to $0.82, because the model now reads every document instead of half of them. The cheaper run was the broken one.

The golden set is 3% smaller, and every removed row was a duplicate. No category lost coverage, because the sampler refills from the pool.

The remap trusts the stored hash. I checked the bytes by hand for the two conflicts. The other matches rely on the hash the pipeline computed when it saved each file.

The score still divides by every item. That's deliberate. A blended number that crashes is what made us look, and the failed count and the failure list sit right next to it now.

This wasn't the eval's first production-style bug

This was the third time the "test script" behaved like production code. On its first Sunday on the server, the eval failed with "no such file or directory" and named a file that looked like one of its inputs. The real cause was an output folder. The script writes its sampled golden set into a folder that git ignores, so the folder didn't exist on a fresh deploy, and the error named the first file it tried to write. It read like a missing input. The fix was to create the folder first.

Before that, a teammate's review caught a field-name mismatch: the job read the run's cost in camelCase while the classify step returned it in snake_case. A double type cast had hidden it from the compiler, and it would have failed every Sunday. The fix was to type the result properly and let the compiler check it.

And this time it was a migration, made harder to spot by an error message that still described two tables. Three times, the eval hit the same things that break production code: deploys, types and migrations.

Lessons for anyone running scheduled evals

1. Before you trust a sudden eval accuracy drop, count what couldn't be scored. If a big share of items has no answer, the model isn't your first suspect. Group the failures by reason before you open the prompt.
2. Keep failures in the denominator, but print the scored-only number next to it. The blended score is the alarm. The scored-only score is the diagnosis. Here's the shape of report I'd want from any eval:

interface Outcome {
  id: string;
  correct?: boolean;
  error?: string;
}

interface EvalReport {
  total: number;
  scored: number;
  correct: number;
  blendedAccuracy: number;
  scoredAccuracy: number;
  failuresByReason: Map<string, number>;
}

export function summarize(outcomes: Outcome[]): EvalReport {
  const failuresByReason = new Map<string, number>();
  let scored = ;
   correct = ;
   ( outcome  outcomes) {
     (outcome. !== ) {
      failuresByReason.(outcome., (failuresByReason.(outcome.) ?? ) + );
      ;
    }
    scored++;
     (outcome.) correct++;
  }
   total = outcomes.;
   {
    total,
    scored,
    correct,
    : total ? correct / total : ,
    : scored ? correct / scored : ,
    failuresByReason,
  };
}

3. A golden set is data that points at other data. Any migration that changes IDs is a migration of your test set too. I wrote the migration and the eval, and I still didn't connect them. Put "what do our tests reference?" on the migration checklist.
4. When you remap, look for collisions. Merging records merges their labels. Plain duplicates shrink your test set without an error, and conflicting labels make the score depend on file order.
5. Settle a disputed label from the source, not from the model. If the model's answer is in front of you, you'll drift toward it, and the test stops measuring anything.
6. Error messages age. Ours described a lookup that no longer existed. When you change what a function does, read its error strings again.

The takeaway, and what's still open

When an eval score falls off a cliff, check the eval before the model. Ours said 50% while the model was at 97.3% on everything it read. The data under the test had moved, and nobody had told the test. Fixing it took a 50-line remap, two decisions only a person could make and a cleanup that made the golden set 3% smaller and clean.

A few things are still open:

• Golden items are pinned to database IDs. Storing each item's owner and content hash next to its ID would let the eval find a document even after the next ID change.
• The script that assembles the golden set only warns about duplicates inside the pooled set. Making it fail on any duplicate or conflicting ID, across both parts, would stop this class of drift at the source.
• The chat message shows the blended score and the failed count. A separate scored-only accuracy line, like the report above, is the next small change.
• The 12 real misses are the real work. General announcements that look like reports or presentations are where the model is weakest, and now that the eval measures the model again, that's where the next prompt change goes.