How We Tag New Documents Within Minutes Using an LLM and a Fail-Open Validator

We use one AI model to tag documents and a second one to argue with it. Adding the second model was easy. Deciding what should happen when it breaks took a lot longer.

You've probably clicked a tag on a blog or a docs site, say "machine learning", and found that half the posts under it mention the idea once, in passing. Something tagged every post that used the words, and nothing pushed back. The opposite failure is quieter. The tagging step chokes on a post, the post ends up with no tags at all, and it never shows up under the tag that fits it best.

Multibagg AI is an AI-powered stock research platform for Indian investors. People browse our theme lists to find companies by what they actually do, so a wrong tag sends them to a company that mentioned an idea once, and a missing tag hides one that's built a business on it.

Here's how we split tagging, where each label is a theme from a fixed list, into a propose step and a check step, why the check step fails open, and what changed when we moved from one daily batch to one job per document.

Before is read from the old job's code. After is measured in our production database over the 24 days since the change, including a one-off reclassification of older documents. The wait row covers only documents whose text became ready after the change.

The old job tagged once a day, and one bad vote lost everything

The documents that matter for themes are long: slide decks and meeting transcripts, often dozens of pages each. Once a day, a batch job grouped the newest deck and transcript for each company, sent their text to a model and asked it for two things in one answer: a written summary of the company, and which themes from a fixed list the company belongs to, each with a reason. The theme list is short and lives in code, so the model can only pick names from it. A JSON schema on the response enforces that.

The first model was generous. Its prompt asks for themes the company fits "strongly and materially", but a model reading 40 pages of a company talking about itself finds a lot of things that sound strategic. So a few months after the first prompt, I added a second step. A different model, from another provider, gets each proposed theme, the theme's description and the first model's reason, and answers one question per theme: keep or drop. Its instructions set a higher bar on purpose. Mere mentions don't count; the company has to show clear intent, real scale or a recent big change in that area.

That design held up. The failure handling around it didn't. In the old job, the check step's output went through a strict schema parse, and nothing caught the error if the parse failed:

1. A malformed vote threw. The exception escaped the check step and landed in the per-company catch block. The company was counted as a failure, so its themes weren't updated, and neither was its summary, which had been fine.
2. A failure never marked the documents as done. The next day's run picked the same company up again and paid for both model calls again. It kept doing that every day until the quarter ended and the documents dropped out of the job's window.
3. A deck with no transcript waited. To avoid tagging a company on half its story, the job skipped a deck less than 15 days old if no transcript had arrived yet.
4. Labels lived on the theme list, not on the document. The job worked out which theme lists to add the company to or remove it from, and wrote those changes. Why that was a problem in its own right is a story for another post.

flowchart TD
    A["Daily run: each company with documents filed this quarter"] --> B["First model: summary plus themes, each with a reason"]
    B --> C["Second model votes keep or drop on each theme"]
    C -->|"valid vote"| D["Add or remove the company on each theme list"]
    C -->|"malformed vote"| E["Exception: the company fails, summary lost too"]
    E --> F["Documents not marked done"]
    F -->|"next day"| A

The worst part is the trade it made without anyone choosing it. The check step exists to remove a few weak themes. When it broke, it threw away all of them, and the new summary too.

Propose, then check: what each model sees

I kept the two-step idea and changed almost everything around it. Each step has one narrow job.

The first model reads one document and proposes zero or more themes from the fixed list. Each theme comes with a list of short reasons, taken from that document. Zero is a valid answer, and a common one.

The second model never sees the document. It gets a small JSON list: the theme name, the theme's description and the first model's reasons. It votes keep or drop on each. A theme it doesn't vote on counts as a drop.

Keeping the document away from the checker was deliberate. Its input is a few hundred words instead of a few dozen pages, so the second call costs a small fraction of the first. It also forces the checker to judge the argument, not to redo the reading. If the reason for a theme is "mentions the idea in its outlook section", the checker can see that's weak without opening the deck. The cost is that it can't catch a reason that misquotes the document. I come back to that below.

Here's a generic version of the check step, with synthetic label names. The schemas are the real shape, and the schema for the vote only allows label names from the list, so a model can't invent a new one:

import { z } from "zod";

const LABELS = ["machine-learning", "databases", "security"] as const;
type Label = (typeof LABELS)[number];

const DESCRIPTIONS: Record<Label, > = {
  : ,
  : ,
  : ,
};

  = z.({
  : z.(z.({ : z.(), : z.(z.()) })),
});
  = z.< >[][];

  = z.({
  : z.(z.({ : z.(), : z.() })),
});

  =
  ;

  =  < | >;

  = {
  : ;
  : ;
  :  ;
  :  <>;
};

   (): <[]> {
   (proposed. === )  [];
   input = proposed.( ({ ...label, : [label.] }));
   raw =  ctx.(, .(input));
   (!raw) {
    ctx.();
     proposed;
  }
   {
     { votes } = .(.(raw));
     kept =  (votes.( vote.).( vote.));
     proposed.( kept.(label.));
  }  {
    ctx.();
     (ctx.)  ctx.();
     proposed;
  }
}

When the checker breaks, keep the labels

The two return proposed lines are the decision this post is about. If the checker returns nothing, or returns something the schema rejects, the job keeps every label the first model proposed. That's failing open.

I chose it by comparing the two worst cases. Failing closed, as the old job did by accident, means a broken checker wipes out a good proposal, and the document's new themes never show up. Failing open means a broken checker lets through the first model's full list. That's the same trust the system placed in the first model before the checker existed, and every one of those labels still carries a reason a person can read. One of those failures is invisible to users. The other is a few weak tags. I'd take the weak tags.

The second rule is about noise. A broken vote never throws, so it never makes the job retry or fail. It logs a warning, and a vote the schema rejects raises an alert only when it happens on the job's last attempt. A schema mismatch from a model is usually a one-off, and an alert that fires on every blip teaches people to ignore alerts.

Why not throw on early attempts, the way the first step does? Because a retry runs the whole job again, and that means paying the first model to read the whole document again just to get a second opinion on a few lines.

That rule has a cost. The checker only runs after the first model has succeeded, which is almost always on the first attempt, so in practice a broken vote shows up only as a warning in the job log. And because we store the labels that were kept, not the vote that kept them, the stored data looks the same either way.

Fail open also has an edge. It covers a checker that answers badly, not one that doesn't answer. The call to the checker sits outside the try, as in the sketch above, so if the request itself errors, say a timeout or the provider being down, the whole job fails and the queue retries it, first model and all. In the sketch, askChecker returns null only for an empty reply; a failed request throws.

A failed document has to finish too

The new job runs once per document, as soon as the document's text is ready. A queue event starts it, and the queue runs a failed job up to three times in total, with growing delays between tries. A sweep twice a day catches documents from the last seven days that never got a result, in case an event went missing. The seven-day floor came from a teammate's review, which pointed out that without it the sweep would keep scanning old documents forever.

That combination brings back the old trap in a new form. If a document fails every time, the sweep finds it again and pays for it again. So every path where a model answers ends in a final state:

1. If the first model returns nothing, its output fails the schema, or the document's text is empty, the job throws on attempts one and two, so the queue retries it.
2. On the final attempt, the job stamps the document as classified, leaves it with no label list and sends an alert. The sweep stops picking it up.

type Attempt = { attemptsMade: number; maxAttempts: number };

const isFinalAttempt = (attempt: Attempt): boolean =>
  attempt.attemptsMade >= attempt.maxAttempts - 1;

type LabelStore = {
  saveLabels: (docId: string, labels: Proposal[]) => Promise<void>;
  settleWithoutLabels: (docId: string) => Promise<void>;
};

export async function retryOrSettle(
  attempt: Attempt,
  docId: ,
  : ,
  : ,
  : (message: ) => <>,
): <> {
   (!(attempt)) {
      ();
  }
   store.(docId);
   ();
}

Settling with no label list needed care. In our storage, a settled failure has no label list at all (null), while a document the model looked at and found no themes for has an empty list. Those mean different things. An AI code review I ran on the rewrite caught where that mattered. Each screen builds a company's themes from its newest classified documents, so a failed document stamped as classified could look newer than a good one and hide its labels. The fix was one rule in the shared query every screen uses: a document with no label list is ignored, and a document with an empty list counts as "no themes". A failure can never replace a real answer.

flowchart TD
    A["Document text is ready"] --> B["First model proposes labels, each with reasons"]
    B -->|"valid proposal"| C["Checker votes keep or drop on each label"]
    B -->|"no output, bad shape or empty text"| R{"Final attempt?"}
    R -->|"no"| T["Throw, the queue retries with a delay"]
    R -->|"yes"| S["Settle: mark done, no label list, alert"]
    C -->|"valid vote"| K["Store the kept labels"]
    C -->|"no output or bad shape"| ALL["Store every proposed label, log a warning"]

I directed an AI coding assistant through this rewrite. The decisions above were mine; the review findings were not, and I'm glad I asked for them. The first version of the new job had the same trap as the old one: on empty text it returned early without stamping anything, so the sweep would have re-queued that document forever. It was caught and fixed before the change merged.

Every label keeps its reason and its source

Each label is stored on the document that produced it, with the reasons the first model gave. So when someone opens a theme and sees a company, the screen can show why it's there and which document said so. The source is simply the document the label lives on.

That last part used to be broken in a way nobody noticed. In the old setup, the list of sources shown next to a theme went through a filter that only allowed the old storage paths. Documents had moved to a new path, so the filter rejected every source, and the list was always empty. No error, just a blank field. Storing the label on the document removed the need for that list, and the filter with it.

What it costs, and what it doesn't catch

The per-document design is better at most things, but not free, and the data shows a few gaps.

• More model calls. The old job made two calls per company in each run. The new one makes up to two calls per document; the checker is skipped when nothing is proposed. A company that files a deck and a transcript now pays for two pairs of calls, and each document is judged on its own instead of side by side.
• The checker can't see the document. It judges the argument it's given. A confident reason that misreads the document passes.
• Failing open is mostly silent. As above, a broken vote almost always leaves only a log line.
• Request errors aren't covered. A model request that errors, in either step, fails the job instead of settling it, and a document that runs out of tries is left for the sweep to pick up again while it's inside the seven-day window. So far that's been enough: every document filed since the change whose text became ready has been classified.
• A valid vote can still drop everything. Fail open covers a vote that's missing or malformed. A vote that parses but leaves labels out drops them, and an empty vote list drops all of them.
• Repeated labels. 2.2% of documents came back with the same theme more than once in their label list, and those repeats are 7.5% of all stored labels. Neither step removes duplicates. One document came back with 239 labels covering every theme in the list, which also means the checker kept all of them, or never got a valid vote in.
• Labels without a reason. 2.6% of labels have an empty reason list. The schema asks for a list of reasons, and an empty list is still a valid list.

What the numbers look like after the change

Since the change, about 7,800 slide decks and transcripts have been classified, including a one-off batch of older documents run through the same job.

• Speed. For documents whose text became ready after the change, labels arrived 3.9 minutes later at p95 and 18.7 minutes later at p99. For documents filed after the change, measured from when each was filed, it's 28.6 hours at p95 and 196.5 hours at p99. Most of the wait now sits upstream, in getting the file and turning it into text.
• Failures. 0 documents ended up settled as failures. Every document classified in those 24 days got a stored result, and no document filed since the change is still waiting for labels once its text is ready.
• Empty answers. 19.2% of documents got no themes at all: 25.1% of slide decks and 5.3% of transcripts. For those documents, the pipeline found no theme with a strong enough case. That's a valid answer, not an error.
• Labels per document. Among documents with at least one theme, the median is 2 distinct themes, the p95 is 5 and the p99 is 7.

Lessons for any LLM classification validator

1. Give the checker a different job, not the same job twice. Our checker sees the claim and the rule, not the source, and holds a stricter bar. I think a second model that rereads the document with the same prompt would mostly repeat the first.
2. Decide what a broken validator means before it breaks. A filter that fails closed turns a small quality tool into a way to lose data. If the validator is there to trim, failing open usually costs less. If it's there to block something harmful, the answer flips.
3. Every failure path needs a final state. Retries plus a sweep will pay for a hopeless document forever unless the last attempt writes something that says "done, and failed".
4. Keep "failed" and "found nothing" apart in storage. Null and empty list look alike in code review and mean opposite things to a reader.
5. Store the reason with the label. It costs a few lines of text, and it's the only way a person can check the model without rerunning it.

The takeaway, and what's still open

I think an LLM classification validator is worth its cost when it has a narrow job and a clear rule for its own failure. Ours votes on arguments, not documents, and when it breaks we keep the proposal, because a few weak labels with reasons beat a document that disappears from every list.

Some things are still open:

• How often the checker drops a label. We store the labels that survive, not the proposal and the vote. Logging both counts per document would give us the drop rate and the fail-open rate.
• Whether the checker improves precision. That needs a labelled sample of documents scored with and without the check step.
• Duplicates. Removing repeated labels before they're stored is a one-line change; the open question is whether the 239-label document points to a deeper problem in the proposal step.
• Checker outages. Moving the checker call inside the fallback would make an outage behave like a bad answer: keep the labels, log it, move on.
• Reasonless labels. The schema could require at least one reason, which would turn those documents into retries instead.

What's settled is the part that used to hurt. A new document gets its labels within minutes of its text being ready, a malformed vote can't erase a good answer, and in 24 days not a single document failed for good.