How We Stopped PDF Crashes From Taking Down Our Python Workers

Our PDF workers kept dying a few minutes after we started them. There was no error and no stack trace. We restarted them, and they died again.

If you run background workers, you've probably seen a version of this. You start a process, it picks up a few jobs, and a while later it's simply gone from the process list. The last log line is an ordinary progress message. You search for the exception that killed it, and there isn't one.

Multibagg AI is an AI-powered stock research platform for Indian investors. Every company document that reaches us has to become searchable text before our AI can answer questions from it, so a stalled converter means answers that miss the latest documents.

Here's how a segfault (a hard crash in compiled code) in an OCR library took our workers down, the three theories we got wrong first, and the small change that lets one bad document fail without taking the worker with it.

Our workers died, and nothing recorded why

The setup is simple. A queue holds documents waiting for conversion. A Python worker takes one message, converts the PDF to markdown with Docling (an open-source document converter), stores the result and moves on. If a page is an image rather than text, Docling runs OCR (optical character recognition) to read it. A sweep job runs twice a day and re-queues any document that has no "attempted" mark yet, so nothing gets forgotten.

The old worker had three safety nets, and on paper they covered every failure:

1. A time limit. Before converting, the code set a SIGALRM alarm for 600 seconds, and the alarm handler raised a TimeoutError.
2. A fallback. A teammate had added a retry without OCR for errors that looked like OCR problems.
3. Bookkeeping in a finally block. Whatever happened, the handler marked the document as attempted, and a failed document went to a failed-documents queue. A reclaim job, also a teammate's work, retried those up to three times and then set the document aside with an alert.

In generic form, with the pieces from three files folded into one function, the old path looked like this:

import signal


def on_alarm(signum, frame):
    raise TimeoutError("conversion timed out")


def handle(message):
    queue.delete(message)
    signal.signal(signal.SIGALRM, on_alarm)
    signal.alarm(600)
    try:
        markdown = convert_pdf(message.pdf_url, use_ocr=True)
        store(message.id, markdown)
    except Exception as error:
        failed_queue.send(message, reason=str(error))
    finally:
        signal.alarm(0)
        mark_attempted(message.id)

Every one of those nets assumes the Python process is still alive when something goes wrong. That assumption was the bug. The message was already deleted, so when the process died mid-conversion, nothing sent it to the failed queue. The finally block never ran, so the document never got its "attempted" mark. At the next sweep it went straight back into the queue and killed whichever worker picked it up. Nothing restarted the workers either. We started them by hand.

flowchart TD
    A["Sweep re-queues documents with no attempted mark"] --> B["Worker takes a message and deletes it"]
    B --> C["OCR runs inside the worker process"]
    C -->|"normal Python error"| F["finally marks the attempt, message goes to the failed queue"]
    C -->|"segfault in native code"| D["Worker process dies"]
    D --> E["No failed-queue entry, no attempted mark"]
    E --> A

Three reasonable theories, all wrong

I worked through the investigation with an AI coding assistant. It read the code and logs and proposed causes, and I checked each one against what the server actually showed. Three theories sounded right and didn't survive.

The kernel log settled it: a segfault in the Python process. A segfault (segmentation fault) means native code touched memory it had no right to, and the operating system ended the process on the spot. The crash always came during the OCR step.

The second finding was quieter. onnxruntime, the inference runtime that the RapidOCR engine uses by default, wasn't installed on the server. When Docling picks an OCR engine automatically and can't import onnxruntime, it moves down its list and can end up running RapidOCR on PyTorch instead. Nothing warned us. Every crash we saw happened inside that fallback path.

I want to be careful here. The crashes lined up with the fallback, but we never reproduced one with a core dump, so "the torch backend crashed" is a correlation, not a proven root cause. What it did teach me: a missing optional dependency doesn't always fail. Sometimes it quietly swaps in a different code path, and now you're running software you never tested.

Can you catch a segfault in Python?

No. A segfault isn't a Python exception. It's a signal from the operating system, and its default action ends the process. No except block sees it and no finally block runs, because there's no Python left to run them. The Python docs even say it makes little sense to handle SIGSEGV yourself: Python returns to the C code that faulted, which is likely to fault again. The faulthandler module can print a Python traceback when a segfault happens, which helps with diagnosis, but the process still dies.

That also explains why our 600-second alarm was the wrong tool. Python runs signal handlers only in the main thread, and only once control comes back to the interpreter. A long call into native code can run for any length of time, whatever signals arrive, and the handler waits until the call returns. So the alarm couldn't stop a hung OCR call, and it could do nothing about a crashed one. On top of that, signal.signal can only be called from the main thread, so the pattern breaks the day conversions move to a thread pool.

So the question changes. You can't make a crash survivable inside the process where it happens. You can make sure the process that crashes isn't the one you care about.

A crash costs one document when it happens in another process

The fix runs every conversion in a short-lived child process. The worker becomes a parent: it starts the child, waits for a result with a time limit, and cleans up. If the child segfaults, the parent sees an ordinary failure and carries on. Here's a generic version of the code:

import multiprocessing as mp


class ChildCrashed(Exception):
    pass


def _convert_in_child(sender, pdf_url, use_ocr):
    try:
        result = (True, convert_pdf(pdf_url, use_ocr=use_ocr))
    except Exception as error:
        result = (False, str(error))
    sender.send(result)
    sender.close()


def convert_isolated(pdf_url: str, use_ocr: bool, timeout: float | None) -> str:
    ctx = mp.get_context("spawn")
    receiver, sender = ctx.Pipe(duplex=False)
    child = ctx.Process(target=_convert_in_child, args=(sender, pdf_url, use_ocr))
    child.start()
    sender.close()
    :
          receiver.poll(timeout):
             TimeoutError()
        :
            ok, payload = receiver.recv()
         EOFError:
            child.join()
            code = child.exitcode
            how =   code <   
             ChildCrashed()
          ok:
             RuntimeError(payload)
         payload
    :
        receiver.close()
         child.is_alive():
            child.kill()
        child.join()

Five details in there matter more than they look:

1. spawn, not fork. The spawn start method runs the child in a fresh Python interpreter, so it doesn't inherit the parent's threads, locks or half-initialised native libraries. It's slower to start, which comes back later. On Linux, Python's default was fork up to 3.13, and 3.14 changed it to forkserver, so ask for spawn explicitly. The child's target function has to live at module level so spawn can import it, and the worker's entry point needs an if __name__ == "__main__": guard, because spawn re-imports the main module in the child.
2. The parent closes its copy of the send end. Right after start(), the parent calls sender.close(), so the child holds the only send end. If the child dies, the pipe closes, poll() wakes up and recv() raises EOFError. Skip that one line and a crash looks exactly like a timeout, because the pipe stays open for as long as the parent holds a send end.
3. EOF means the child died. A healthy child always sends something: a result or an error message. If the pipe closes with nothing in it, the child crashed. The exit code says how. A negative exit code -N means the child was killed by signal N, so a segfault shows up as "killed by signal 11".
4. Read before you join. The parent reads the result before it waits for the child to exit. A child sending a large result can block until someone reads the other end, so joining first risks two processes waiting on each other forever. One of our tests pushes a 5 MB result through the pipe to make sure that doesn't happen.
5. Always reap. The finally block closes the pipe, kills the child if it's still running (the timeout case) and joins it. No orphaned OCR process keeps burning CPU after the parent has given up.

sequenceDiagram
    participant W as Worker
    participant C as Child process
    W->>C: start with the PDF and OCR on
    W->>W: close own copy of the send end
    alt a result arrives
        C-->>W: markdown, or an error message
        W->>W: return the markdown or raise a normal error
    else the pipe closes with nothing in it
        C--xW: child dies, the parent reads EOF
        W->>W: decode the exit code, start a fresh child with OCR off
    else nothing before the time limit
        W->>C: kill
        W->>W: raise a timeout error
    end
    W->>C: kill if still alive, then join

Retry once, with OCR off

My first reaction to the child-process idea was that it wasn't enough. If the OCR step crashes on a PDF, a child process saves the worker, but that document still never converts. I wanted both: the worker survives and the document gets converted.

So a crash with OCR on gets exactly one retry, in a fresh child, with OCR off:

def convert(pdf_url: str, timeout: float | None = 1200) -> str:
    try:
        return convert_isolated(pdf_url, use_ocr=True, timeout=timeout)
    except ChildCrashed as crash:
        try:
            return convert_isolated(pdf_url, use_ocr=False, timeout=timeout)
        except Exception as retry_error:
            raise RuntimeError(f"{crash}; retry without OCR failed: {retry_error}")

Without OCR, the crashing code never runs, and Docling reads the PDF's own text layer. Most of our documents have one, so the output is still good. A scanned PDF with no text layer can come out empty, and the worker already treats empty output as a failure, so it doesn't slip through as a success. Retrying with OCR on again would only repeat the crash, so there's one retry and no more. A timeout isn't retried at all, because a second attempt would most likely be just as slow.

If the retry fails too, the error carries both causes, and the document takes the normal failure path: marked as attempted, sent to the failed queue, retried by the reclaim job up to three times, then set aside with an alert. I had asked for exactly that alert. It turned out my teammate's reclaim job already did it, as long as a crash became a normal failure. That's the part of this fix I like most. Nothing in the queue, the sweep or the reclaim job changed. They had been right all along. They just never got the chance to run.

Pin the stack

The same change pinned every Python dependency to the versions running in production and added onnxruntime, so Docling's automatic OCR choice lands on the onnxruntime backend instead of PyTorch. Before that, every reinstall pulled whatever was newest, which is how the native libraries had drifted in the first place.

What isolation costs

The child process isn't free, and two costs showed up quickly.

First, every document now starts a fresh Python interpreter and loads the conversion models again. That adds a few seconds per document. We never measured it precisely, so all I can say is that it's small next to a long conversion and large next to a tiny one.

Second, that overhead met CPU contention. In about ten days of worker logs before the change, the slowest successful conversion took a little over nine minutes, just under the 10-minute limit. In roughly two days after the change, about a hundred conversions hit the limit. We didn't count the total for that window, so I can't give a rate. They clustered in the busiest hours and right after the reclaim runs, when the most documents arrive at once, and the server's load average was more than double its core count. A few seconds of startup can't push a nine-minute conversion past ten minutes on its own. Most likely it was the mix: each child loading its models fresh, and workers that used to die now all staying up and competing for the same cores. The limit never had much headroom, and that contention used it up.

The follow-up was boring on purpose. I doubled the per-child limit to 20 minutes and wrote down a sizing rule: workers times conversion threads should stay within the CPU cores. The trade-off is plain. A hung conversion now holds a worker for the full 20 minutes, and a crash followed by a hung retry can hold it for 40 before the document is given up on.

There were smaller costs too. Docling's own log lines from the child now go to the worker's standard error instead of our log files, though errors and "converted" lines still land where they did. And the fix wasn't run on the production Linux server before it merged. The real-PDF checks ran on my machine, and the first Linux run was the deploy itself.

What changed after the fix

Before merging, I asked the coding assistant whether it had actually run the flow and converted a PDF itself, or used markdown that already existed. That question mattered, because the next step was a real run: real Docling on a real PDF through the new path. The normal run took 84 seconds. With the child killed mid-conversion, the retry without OCR returned the markdown in 38 seconds. That's one document, so read it as "the OCR run took about twice as long as the retry", not as a benchmark. The automated tests cover a child crash, a timeout, the retry without OCR after a crash and the 5 MB result.

In production, the log now shows lines like "child process killed by signal 11, retrying without OCR", and the worker keeps going. When a retry also failed, the document was marked as attempted and sent to the failed queue like any other failure, instead of coming back at the next sweep to kill another worker. The workers stayed up through the sweeps that used to take them down.

What this doesn't prove is the root cause. PDFs still crashed in the OCR step after the deploy. I didn't confirm which OCR backend those later crashes ran on, so the missing onnxruntime explains some of the crashes at best, and maybe none. The fix contains crashes. It doesn't prevent them.

Lessons for any worker that runs native code

1. Draw the crash boundary around the native code, not around the worker. try/except protects you from Python errors. A process boundary protects you from everything else. If a library is mostly C, C++ or Rust and it handles input you don't control, assume it can take the process down.
2. Bookkeeping in finally only runs if the process survives. Our marking, failed queue and retry logic were all correct and all useless against a crash, because they lived in the process that died. Either do the bookkeeping before the risky work, or make sure the risky work can't kill the process doing the bookkeeping. We chose the second, because it needed no change to the queue or the sweep.
3. Put the time limit outside the thing you're limiting. An alarm inside the process can't interrupt native code. A parent with poll(timeout) and kill() can stop anything.
4. Pin native dependencies, and make silent fallbacks loud. A missing package didn't break the install. It changed which engine ran. If a library can quietly pick another backend, log the one it picked at startup and check it.
5. Ask how a result was produced. "Have you run it?" is a cheap question, and it's the one that turned a claim into a measured run here.

The takeaway, and what's still open

You can't catch a segfault in Python, so stop trying to handle it in the process where it happens. Put the risky native call in a child process, treat a closed pipe as a crash, retry once down a safer path, and let the failure handling you already have do its job.

Some things are still open:

• The root cause. We know where the crashes happen, not why. A core dump from one crash would tell us more than any amount of reasoning about versions.
• Restarting the parent. The workers survive a crashed conversion, but if a worker dies for any other reason, nothing restarts it yet. A process supervisor would close that gap.
• The cost per document. A fresh interpreter for every PDF is the simplest safe option. A pool of reusable children, replaced after a crash, would cut the startup cost, but we haven't measured whether it's worth the extra code.
• The time limit. Twenty minutes gives headroom today. It's a number tuned to the current load, not a guarantee.