How We Test Whether Our AI Search Finds the Right Page

Our search test said 67.5%. After I fixed the test, it said 56.1%. Nothing in the search had changed.

You've probably felt the problem this test is built to catch. You search a long report for a number you know is in there. The right file comes up first, and you still scroll through forty pages to find the line. Search "worked", and it didn't really help you.

Multibagg AI is an AI-powered stock research platform for Indian investors. Our AI answers questions from company documents, so if search hands it the wrong page, the answer comes out wrong or empty, however good the model is.

Here's how our retrieval test works, the three flaws I found in my own question set, and why the score went down, up and down again before I trusted it.

Same search, four versions of the test

I ran the test four times while I fixed it, over a few days. The search code and settings stayed the same, and the index only grew by the new documents that arrived in between. Only the questions changed.

Search didn't change between these runs. Only the questions did. Each fix made the test fairer, and the score only settled once all three were in.

Each jump in page accuracy between versions is 10 to 20 points, far above the 2 points that repeat runs on the same questions can drift. The first number flattered us. The last one is lower, and it's the first one I trust. The rest of this post is about why.

What one test question checks

Our AI search is retrieval-augmented generation (RAG). Before the model answers, a search step pulls the most relevant chunks, small pieces of document pages, from our index. The model only reads what search hands it. If the right page isn't in that list, no prompt can fix the answer.

The search is hybrid. One half compares meaning, using embeddings (number vectors that capture what a text is about). The other half matches keywords with BM25, a classic ranking formula. The two result lists are merged with reciprocal rank fusion (RRF), which rewards chunks that rank high in either list. Every search is filtered to one owner's documents, because a question is always about one company. In production, search returns 30 chunks and a reranker, a second model that re-scores those 30 against the question, keeps the best 15 for the model.

The standard tool for RAG retrieval evaluation is a golden set: a fixed list of questions where you already know the right answer. In our case, each question is tied to the exact document and page that holds the answer. For each question, the test runs the same kind of hybrid query production runs and checks two things:

1. Document hit. The right document appears anywhere in the top K results (K is how many results we ask for, 30 or 50).
2. Page hit. The right page of the right document appears in the top K. This is the stricter one, and the one that matters, because the model reads pages, not document titles.

There's no language model judging the results and no reranker in the path. Same index, same question, same result. The only drift comes from new documents landing in the live index, which moves the total by about 2 points. It's cheap too: the only paid call is embedding each question. A scheduled job runs it every morning and posts the numbers to our team chat. A full run of 196 questions takes about six minutes.

Here's a generic version of the core, using the Qdrant client:

import { QdrantClient } from "@qdrant/js-client-rest"

type GoldenCase = {
    id: string
    question: string
    ownerId: string
    documentId: string
    pageNo: number
    category: string
}

type  = { : ; :  |  }
  = { :  | ; :  |  }

 client =  ({ :  })

  (): <[]> {
     filter = { : [{ : , : { : ownerId } }] }
     { points } =  client.(, {
        : [
            { : vector, : , : k, filter },
            { : { : question, :  }, : , : k, filter },
        ],
        : { :  },
        : k,
        : ,
    })
     points.( {
         page = point.?.
         {
            : (point.?. ?? ),
            :  page ===  ? page : ,
        }
    })
}

 ():  {
     doc = hits.( h. === golden.)
     page = hits.( h. === golden. && h. === golden.)
     { : doc === - ?  : doc + , : page === - ?  : page +  }
}

   (): <<, >> {
     vector =  (golden.)
     : <, > = {}
     ( k  depths) {
        byDepth[k] = ( (vector, golden., golden., k), golden)
    }
     byDepth
}

One detail in there cost me a wrong assumption. I wanted the top 30 and the top 50 scored in one run and reported separately. The cheap way is to search once for 50 and cut the list at 30. That gives the wrong top 30. K sets the limit of both prefetch lists, not just the final cut, so a search for 50 fuses a different pool and ranks it differently. On all 196 questions, the top-30 order differed from the first 30 results of the top-50 search. So each depth is a real, separate search.

A few guards keep the numbers honest. A question that errors, say a timeout from the index, is counted separately and not as a miss, because an infrastructure error isn't evidence that search failed. A run where nothing scores fails loudly instead of posting a cheerful 0%. A run that hits its 10-minute limit writes nothing, so it can't pollute the numbers.

Flaw one: one kind of document made up 70% of the test

I built the first question set with an AI coding assistant. It wrote the scripts, generated candidate questions and ran the test. The calls on what a fair test means were mine, and the first one came from a single count.

We search four kinds of documents: short notices of one or two pages, long yearly reports that run to hundreds of pages, slide decks, and transcripts of calls. The first set had 200 questions, and 141 of them, 70.5%, came from short notices. Call transcripts had 24, slide decks 22, yearly reports 13.

So the headline score was really the short-notice score. A 10-point swing on yearly reports moved the total by 0.65 points. The cause was a quota bug, not bad luck. The authoring script had one global target and took turns across 14 document sub-types, and 11 of those were kinds of short notice. When it hit the target, it simply stopped, so whatever came last got cut. One sub-type ended up with 2 questions.

I asked for equal weight per document type, with every sub-type covered. The new set had about 50 questions per type, at most 3 per document, and a floor of 4 for each short-notice sub-type.

Page accuracy fell from 67.5% to 47.2%. Yearly reports, slide decks and call transcripts each scored lower than before, now on two to four times as many questions, so their old numbers came from small samples and didn't hold up on a larger, more even set. Short notices scored lower too, on fewer questions, because the new set spread them evenly across sub-types instead of piling onto the easy ones. Search hadn't changed.

Flaw two: many questions had no single right answer

Then I read the misses. A lot of them looked like this:

• "What is the company's registration code?"
• "On what date was this document issued?"
• "What is the second email address shown on this page?"

Every document a company publishes carries its registration code in the header. Ask search for it, and any of that company's documents is an equally good answer. The test marks it a miss unless search happens to return the one document we had in mind. The test was punishing search for failing an impossible task. In the first 200-question set, 25.5% of the questions were this kind of boilerplate, and the balanced set still had plenty.

My call was that questions must be genuine and unique per document, so that the test is fair to search. Prompt wording alone doesn't hold that line, so the coding assistant turned it into checks in code. A language model reads a page, writes a question and quotes the answer from the page word for word. Then the candidate has to pass these checks (this is the final version, including the meeting-time rule from the next section):

type Candidate = {
    question: string
    answer: string
    ownerId: string
    documentId: string
    pageNo: number
}

type Page = { documentId: string; pageNo: number; text: string }

const BOILERPLATE = /\b(registration (code|number)|ticker|email|phone|address|issued|issue date|start time|end time|commenced|concluded)\b/i

export function rejectReason(candidate: Candidate, ownDocument: Page[], ownerOtherDocuments: []):  |  {
     answer = candidate..()
      = () => page..().(answer)
     pagesWithAnswer = ownDocument.(contains)
     (pagesWithAnswer. !==  || pagesWithAnswer[]. !== candidate.) {
         
    }
     (ownerOtherDocuments.(contains))  
     (.(candidate.))  
     (candidate..().(answer))  
     
}

The boilerplate pattern is deliberately blunt. It also throws out some good questions, like one about shares issued, and that's fine: losing a good candidate costs one more model call, while keeping a bad one skews the test. The second check does the heavy lifting. If the answer text shows up in another document from the same owner, the question has no single right target, so it's out. That check has to run against the full document store, not the sample of documents the questions are drawn from. In our sample, only 3 of 204 owners had more than one document, so a sample-only check would have caught almost nothing. The questions also have to name their subject, the way a real user types them, instead of saying "this document".

Document accuracy went from 65.6% to 87.4%, and page accuracy from 47.2% to 66.8%. Boilerplate questions fell from 25.5% in the first set to 2.0%. Questions that name a company went from 48.0% to 79.9%.

That jump looked great. It wasn't the end.

Flaw three: page one was too easy and too common

The page sampler took the first, middle and last page of each document. For a long document, page one is a cover, a letterhead or a title slide. Those are easy to find, because the company name and the title sit right there.

83 of the 199 questions, 42%, pointed at page one. Search found the right page for 88.0% of those, against 51.7% for deeper pages. That's a 36-point gap, and with 42% of the set on page one, it was worth about 15 points on the total. A second pocket was questions about when a meeting started or ended. There were 26 of them, and they scored 84.6% on the page, against 64.2% for everything else.

I asked for a more trustworthy test over a higher score, knowing the number would drop. The fix changed how pages are picked:

export function samplePages(pageCount: number): number[] {
    const pages = Array.from({ length: pageCount }, (_, i) => i + 1)
    if (pageCount <= 2) return pages
    const body = pages.slice(1)
    const picks = [1 / 6, 1 / 2, 5 / 6].map((f) => body[Math.floor(body.length * f)])
    return picks.filter((page, i) => picks.indexOf(page) === i)
}

A document with three or more pages loses its first page, and the picks spread across the body. There's no randomness, so a rebuild picks the same pages. Meeting-time questions joined the rejection list. Page one's share fell from 42% to 9.7%, and every remaining page-one question comes from a short notice of one or two pages, where page one is most of the document. The deepest question now sits on page 751.

Page accuracy fell from 66.8% to 56.1%. That's the number I trust.

flowchart TD
    A["Pick pages: skip the cover, spread picks across the body"] --> B["A language model writes a question and quotes the answer"]
    B --> C{"Answer on exactly one page of this document?"}
    C -->|"no"| X["Reject"]
    C -->|"yes"| D{"Answer in another document from the same owner?"}
    D -->|"yes"| X
    D -->|"no"| E{"Boilerplate, meeting time, or answer inside the question?"}
    E -->|"yes"| X
    E -->|"no"| F["Keep, until its document type has about 50 questions"]

Two depths tell a ranking problem from a finding problem

Here's where the final test started paying for itself. This is page accuracy per document type, at both depths:

Going from 30 to 50 results finds 5 more slide-deck pages and 1 more transcript page. Same test, two different problems.

No page that was found in the top 30 was lost in the top 50, so each gain is pages found, not a net of wins and losses. Read the gain column, not just the score:

1. A big gain points at ranking. Slide decks gain 10.2 points, 5 of 49 questions. For those, search does find the right page, just not high enough. Part of the slide-deck misses is a ranking problem, and better ordering or a few more candidates is a cheap win there. The other 18 misses are still missing at 50.
2. A tiny gain means search isn't finding the page at all. Call transcripts gain only 2.0 points, a single question, and 29 of 49 still miss at 50. More results won't help. My leading suspect is how transcripts are split into chunks; the test can't prove that, but it can judge a chunking change.
3. A gap between document and page means the right file, wrong page. Yearly reports found the right document 96.0% of the time and the right page 56.0% of the time. That's the scroll-through-forty-pages problem from the start of this post, measured.

The ranks back this up. When search finds the right page, its median position is 3 out of 30, and 90% of those hits sit in the top 17. Search mostly misses pages outright. It rarely buries them.

Code review caught one real bug and one wrong fix

An AI code review I ran on the change found a real bug. I wanted the retrieval test to run after our other model evaluation job, and the first version did that by making it the other job's parent, since in our job queue a parent runs after its children. That coupled two jobs that don't depend on each other: the other job lost its place as a standalone job, and the retrieval test had to wait for it to finish. Nothing in the retrieval test needs the other job's result, so the ordering wasn't worth the coupling. Both now run as separate jobs.

The same review also flagged something I kept. Production search adds a window of recent years to every query, and the test leaves it out, so the review called that drift that inflates the score. I kept it out on purpose. The test asks whether search can find a known page in a known document. With a date window, questions would start failing as their documents aged out of it, for a reason that has nothing to do with search. The golden set would rot on a calendar.

A teammate's review made a fairer point: "the same search production runs" overstated it. That one goes in the limits below.

What the test costs, and what it can't see

The costs are real but small:

• Building the set got slower. Checking each candidate against a sample of up to 21 other documents from the same owner took authoring from about 4 minutes to about 18, with paid model calls for every candidate.
• Two searches per question per run. Scoring both depths doubles the queries to the index.
• About 2 points of noise. Two runs on the identical 196 questions gave 57.7% and 55.6%. The index is live, so new documents shift the keyword statistics. Differences under 2 points mean nothing, which is why the test reports numbers and doesn't alert on a threshold yet.
• Thin sub-types. With about 50 questions per type, a short-notice sub-type has 3 to 6 questions. Those scores are directional only. Two sub-types finished at 3, below the floor of 4, because I wouldn't loosen a check just to fill a quota.
• The repo holds only the test and the questions. I kept only the test script and the golden file in the codebase and moved the authoring scripts out. The coding assistant warned that the set can't then be rebuilt from the repo. I chose that knowingly and saved the scripts to my notes.

And here's what it can't see:

flowchart LR
    Q["Question"] --> E["Embed the question"]
    E --> H["Hybrid search over one owner's documents"]
    H --> S["Test: score document hit and page hit at 30 and 50"]
    H --> R["Production: reranker keeps 15 of 30"]
    R --> T["Expand table chunks"]
    T --> M["Model writes the answer"]

1. It measures candidates, not what the model reads. Production reranks 30 chunks down to 15 and expands table chunks before the model sees anything. A page that ranks 25th counts as a hit here and may never reach the model.
2. It searches with the test question. In production, the model writes the search query itself, so real queries are worded differently.
3. It's a copy of the production query, not a call to it. The shared search function returns formatted text without ranks, so the test rebuilds the query. Nothing forces the two to stay the same, and the copy already lacks production's fallback to meaning-only search when the hybrid query fails.
4. Retrieval isn't the answer. A page hit means the model had a chance. It doesn't mean the answer was right.

Lessons for anyone testing a search step

1. Check the mix before you trust the score. Our first headline was one document type wearing a total's clothes. Count questions per type, and report per type, before you read the total.
2. Every question needs exactly one right answer, and code has to enforce it. If the answer text appears in two documents, the question tests luck. A prompt that says "be specific" didn't stop that. A check against the owner's other documents did.
3. Score the hard part on its own. Document hits flatter you, so score pages too. And check whether easy cases, like cover pages, are inflating your set.
4. A score that drops after you fix the test is information, not a regression. Keep the old run, compare, and write down why it moved, so nobody reads the drop as search getting worse.
5. Leave out filters that make the test age. A date window belongs in production. In a test, it slowly turns valid questions into failures.

The takeaway, and what's still open

A retrieval score is a claim about your questions as much as about your search. Ours fell by 11 points while the search stood still, and the lower number was the useful one. It told us that part of the slide-deck misses are a ranking problem, most transcript misses aren't found even in the top 50, and yearly reports find the right file but the wrong page. That's three different fixes, and the first trustworthy test is what told them apart.

Some things are still open:

• Measure what the model actually reads. The next step is scoring after the reranker keeps 15, so the number maps to the model's real input.
• One shared query. Moving the raw hybrid query into a function that both production and the test call would end the copy problem.
• Fix transcripts, then let the test judge. Chunking is the leading suspect for transcripts. The next change to how they're split gets judged by this same set.
• A trend before an alarm. With 2 points of noise, a few weeks of runs should come before any alert threshold.