The Blind LLM Problem: Why I Ditched Multimodal AI for YOLOv9 on Checkbox Detection

This is a story about a system that looked simple on paper and turned out to be one of the more humbling engineering problems I've worked on.

The System

We were building an automated QC pipeline for support ticket images. Field engineers submit evidence when closing a ticket: photos of devices, installation checklists, sign-off forms. The pipeline's job was to validate these images automatically before a ticket could be marked resolved.

One of the core validation requirements: detect whether specific checkboxes on a checklist form were ticked or not. A boolean signal per checkbox. Sounds trivial. It was not.

First Attempt: Qwen3-VL 27B

We were already running Qwen3-VL 27B (8-bit quantized) on our inference stack for other semantic validation tasks, so the natural first move was to throw the checkbox problem at it too.

The prompt was straightforward — here's the image, tell me which checkboxes are ticked. The model responded with confidence. It was also wrong. Consistently, structurally wrong.

The hallucination pattern was specific: the model would look at a partially filled form and infer checkbox state from surrounding context rather than the actual visual mark. If the text next to a checkbox said "installation complete", the model assumed it was ticked. If a checkbox was near the bottom of a crowded form, it got skipped entirely. The model was reading the document semantically and guessing — not doing pixel-level visual detection.

This is the blind LLM problem. The model doesn't admit it can't see something. It fills the gap with plausible inference, returns a confident answer, and you only find out it was wrong downstream. Research published in 2025 gave this a name — perceptual hallucination: when vision-language models generate information "as if perceived, despite absent or damaged visual evidence." The ACL 2026 DocHallu benchmark^[1] found this occurs across all models, with hallucination rates higher for precise visual elements than for textual content. The vision encoder introduces the error; the language decoder amplifies it.

Trying to Fix It with Prompting

The next move was prompt engineering. We tried:

• Explicit instructions: "do not infer checkbox state from surrounding text, only look at the visual mark inside the box"
• Chain-of-thought: asking the model to describe what it sees in each checkbox region before giving a boolean
• Coordinate-based prompting: splitting the image into regions and asking about each one individually
• Few-shot examples with ticked and unticked checkboxes labelled

Some of these reduced the hallucination rate. None of them eliminated it. The model improved from confidently wrong to inconsistently right — which in a production QC pipeline is arguably worse, because you can't predict where it will fail.

This tracks with what researchers have found about spatial reasoning in VLMs. The "Mind the Gap" benchmark (2025)^[2] found that models' apparent competence decreases dramatically under tasks requiring precise spatial localization — with accuracy across models approximating random chance in the hardest cases. Prompting cannot fix a representational gap: the physical world is geometric and continuous, but LLMs learn spatial concepts as discrete statistical patterns in text.

Escalating to Top-Tier Models

At this point the question was: is this a Qwen limitation, or is checkbox detection fundamentally hard for multimodal LLMs?

We ran evals against stronger models. The results were the same story at a higher confidence level.

It turns out this is a known, documented failure. The FormFactory benchmark (2026)^[3] evaluated GPT-4o, Gemini 2.5 Pro, Claude Sonnet 3.7, Qwen-VL-Max, and Grok 3 on form field detection — including checkboxes and radio buttons. No model surpassed 5% accuracy on atomic form field detection. Not a small gap. A fundamental one.

This isn't a capability problem. It's a training objective problem. VLMs are optimised to understand and generate language grounded in visual context. They are not trained to do precise binary spatial classification on small visual regions in low-quality document scans. As the "Vision Language Models Are Blind" paper (2025)^[4] puts it bluntly: these models can describe a scene without actually seeing it.

The honest conclusion from our evals: no prompt, no model, no chain-of-thought was going to reliably produce a boolean from a checkbox.

Switching to YOLOv9

The right tool for checkbox detection is object detection. We switched to YOLOv9.

The approach was straightforward once the decision was made:

1. Collect and annotate examples. We manually annotated checkbox regions from real ticket images — ticked and unticked — building a labelled dataset from the actual artifacts the system would encounter in production. Custom training data built on your own domain always outperforms a generic model on a specific task.
2. Fine-tune YOLOv9. Standard fine-tuning on our annotated dataset. The model learned to locate checkbox regions and classify each as ticked or unticked. YOLO's architecture — a unified regression framework that predicts bounding boxes and class labels in a single forward pass — is exactly suited to this: fast, local, spatially precise.
3. Output a boolean. For each detected checkbox, the model returns a confidence score and a binary state. We threshold the confidence and pass a clean boolean to the downstream pipeline. No ambiguity, no hallucination, fully auditable.
4. Plug into Holmes. The YOLOv9 output feeds directly into the same validation script that Qwen3-VL handles the semantic checks — each model doing the job it's actually good at.

Results

Checkbox detection went from unreliable to production-grade. The boolean output was clean, consistent, and deterministic. The pipeline now runs both models together — YOLOv9 for spatial binary detection, Qwen3-VL for semantic content validation — covering the full QC surface across 5,000+ ticket images per day.

The Takeaway

LLMs are powerful and I use them heavily across production systems. But they have a failure mode that's worse than being wrong: being wrong with confidence. Checkbox detection exposed this clearly because the ground truth is binary — a box is ticked or it isn't — and the hallucinations were easy to audit.

The research backs this up. VLMs systematically fail at tasks that require them to count, locate, or classify small precise visual elements — not because they lack intelligence, but because that's not what they were built to do. The "Can Vision-Language Models Count?" paper (2025)^[5] identified what it calls "enumerative binding failure" — models fail to count objects they can perceptually see. Every major model has a distinct failure signature: Claude under-counts, ChatGPT massively over-counts, Gemini template-hallucinates.

The fix wasn't a better prompt. It was choosing the right class of model for the problem. Computer vision problems need computer vision solutions. The moment we stopped trying to prompt our way around a fundamental limitation and trained a detector on our actual domain data, the problem was solved.

Know your tools. Know their failure modes. And when a foundation model fails at something a purpose-built model does easily, build the purpose-built model.