It was my first few weeks at the firm. I'd just joined, I didn't know anyone well enough to push back on assignments yet, and someone handed me a stack of image files — scanned documents, screenshots, mixed-format files — with a task: extract the data and organize it. Manually. The estimate was about a month of work.
I stared at the first image for maybe fifteen minutes. Then I opened a new Python file.
Two days later, the entire backlog was done. This is the technical story of how that pipeline worked — the OCR layer, the structural parsing problem, the database schema, and the parts that were harder than they looked.
The Problem, Precisely Stated
The images weren't random photographs. They were structured documents — forms, invoices, tables, ledger entries — that had been scanned or screenshotted and saved as PNGs and JPEGs. The task was to extract the text content and store it in a structured format that could be queried and reported on.
Doing this manually means: open image, read it, type the values into a spreadsheet, repeat. For a few dozen images, that's an afternoon. For hundreds, it's a month.
Automating it means solving three distinct problems in sequence:
- OCR: Extract raw text from the image pixels.
- Structural parsing: Understand what the text means — which values are field labels, which are data, where the logical boundaries are.
- Storage: Persist the extracted, structured data in a form that's queryable without additional transformation.
Each of these looks straightforward until you start implementing it.
The OCR Layer: EasyOCR
OCR (Optical Character Recognition) is a solved problem in the sense that good libraries exist. Choosing the right one matters. The two main Python options I evaluated were Tesseract (via pytesseract) and EasyOCR.
Tesseract is older, faster on clean inputs, and widely used. EasyOCR is a deep learning-based approach — it runs a CRAFT (Character Region Awareness For Text detection) model for text detection followed by a CRNN (Convolutional Recurrent Neural Network) for recognition[1]. The distinction matters for document images specifically: Tesseract assumes relatively clean, typed text on white backgrounds. The documents I was working with had varying scan quality, occasional handwritten annotations, mixed fonts, and tables with thin ruling lines. EasyOCR's detection model handles irregular layouts more robustly.
The EasyOCR API is straightforward:
import easyocr
reader = easyocr.Reader(['en'], gpu=False)
results = reader.readtext('document.png')
reader.readtext() returns a list of tuples — one per detected text region — each containing three elements: a bounding box (four corner coordinates), the detected text string, and a confidence score between 0 and 1:
[ ([[12, 18], [210, 18], [210, 38], [12, 38]], 'Invoice No:', 0.97), ([[215, 18], [310, 18], [310, 38], [215, 38]], 'INV-20221104', 0.94), ([[12, 52], [180, 52], [180, 72], [12, 72]], 'Date:', 0.99), ([[185, 52], [280, 52], [280, 72], [185, 72]], '04/11/2022', 0.91), ... ]
The raw output gives you text with spatial position. That's the foundation — but spatial position alone doesn't tell you that "Invoice No:" is a label and "INV-20221104" is its value, or that these two form a key-value pair that should land in the same database row.
The Hard Part: Structural Parsing
This is where most OCR tutorials stop, and where the actual engineering starts.
A scanned document has implicit structure that humans read effortlessly: labels are visually paired with their values, table headers span columns, rows belong to sections, totals appear at the bottom of groups. None of this structure is encoded in the pixels. EasyOCR gives you text fragments with bounding boxes. Turning those fragments into structured records requires reasoning about spatial relationships.
I built the parser around two observations:
1. Horizontal proximity on the same line is key-value pairing. If two text regions share approximately the same vertical center (within a tolerance, since scan alignment is imperfect) and are horizontally adjacent with no other text in between, they're probably a label-value pair. The rule:
def same_line(box_a, box_b, y_tolerance=8):
center_a = (box_a[0][1] + box_a[2][1]) / 2
center_b = (box_b[0][1] + box_b[2][1]) / 2
return abs(center_a - center_b) < y_tolerance
The Y-tolerance of 8 pixels handled the slight baseline drift from scanning. Too tight and you'd miss pairs on slightly misaligned scans; too loose and you'd pair text from adjacent rows.
2. Labels end with a colon. This was the heuristic that made the key-value detection reliable on these specific documents. Fields like "Invoice No:", "Date:", "Amount:", "Vendor:" all terminated with a colon. If a text region ends with ":" and is on the same line as a subsequent region that doesn't, the first is the key and the second is the value. It sounds simple — it was. But it worked on 94% of the documents without exception handling.
The structural parsing pipeline:
- Sort all OCR results by vertical position (Y-coordinate) to process top-to-bottom.
- Group results into "lines" by clustering on Y-center with the tolerance above.
- Within each line, sort by X-coordinate (left to right).
- Scan each line for colon-terminated tokens; pair them with the immediately following token on the same line.
- Collect all key-value pairs from the document into a flat dictionary.
def extract_kv_pairs(ocr_results, y_tolerance=8):
# Sort by vertical position
sorted_results = sorted(ocr_results, key=lambda r: (r[0][0][1] + r[0][2][1]) / 2)
# Group into lines
lines = []
current_line = [sorted_results[0]]
for result in sorted_results[1:]:
if same_line(current_line[-1][0], result[0], y_tolerance):
current_line.append(result)
else:
lines.append(sorted(current_line, key=lambda r: r[0][0][0]))
current_line = [result]
lines.append(current_line)
# Extract key-value pairs
pairs = {}
for line in lines:
for i, (box, text, conf) in enumerate(line):
if text.strip().endswith(':') and i + 1 < len(line):
key = text.strip().rstrip(':').strip()
value = line[i + 1][1].strip()
pairs[key] = value
return pairs
For a typical invoice image, this produced a dictionary like:
{
"Invoice No": "INV-20221104",
"Date": "04/11/2022",
"Vendor": "Acme Supplies Pvt Ltd",
"Amount": "₹48,200.00",
"GST": "₹8,676.00",
"Total": "₹56,876.00"
}
Table Detection: The Harder Case
Not all documents were forms. Some were tables — grids of data with headers and rows. These required a different parsing strategy because the colon heuristic doesn't apply and the horizontal grouping is two-dimensional.
For tables, I used a different approach: detect the header row (usually the topmost line with multiple tokens, often with bold or distinct styling that EasyOCR picks up as higher-confidence detections), then treat each subsequent row as a record keyed against those column headers.
The column alignment challenge: in a scanned table, the X-coordinates of values in a column aren't exactly aligned — there's pixel-level variation. I resolved this by building a column boundary map from the header row. For each header token, I recorded its X-range (left edge to right edge). Values in subsequent rows were assigned to columns by checking which header's X-range their X-center fell within, with a tolerance for slight misalignment:
def assign_to_column(token_box, column_ranges, x_tolerance=15):
token_center_x = (token_box[0][0] + token_box[1][0]) / 2
for col_name, (col_left, col_right) in column_ranges.items():
if col_left - x_tolerance <= token_center_x <= col_right + x_tolerance:
return col_name
return None # token falls outside all known columns
This worked reliably on clean tables. For tables with merged cells or spanning headers, I fell back to manual flagging — the pipeline wrote those images to an ambiguous/ folder for human review rather than silently producing wrong output.
Storage: SQLite with Dynamic Schema
The storage layer had one interesting constraint: the documents didn't all have the same fields. Different document types had different key sets. A purchase order had fields an invoice didn't; a receipt had fields neither had. I needed a schema that could accommodate this without requiring a different table per document type.
I used SQLite for two reasons: zero infrastructure (it's a file, no server), and the fact that the downstream consumer of this data was going to query it with SQL anyway. No point adding a translation layer.
The schema:
CREATE TABLE IF NOT EXISTS documents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
filename TEXT NOT NULL,
doc_type TEXT,
processed_at TEXT DEFAULT (datetime('now')),
confidence_avg REAL
);
CREATE TABLE IF NOT EXISTS fields (
id INTEGER PRIMARY KEY AUTOINCREMENT,
document_id INTEGER REFERENCES documents(id),
field_key TEXT NOT NULL,
field_value TEXT,
confidence REAL
);
CREATE INDEX IF NOT EXISTS idx_fields_key ON fields(field_key);
CREATE INDEX IF NOT EXISTS idx_fields_doc ON fields(document_id);
Every document gets a row in documents. Every extracted key-value pair gets a row in fields, referencing its parent document. The field_key index makes querying across documents for a specific field fast — SELECT d.filename, f.field_value FROM documents d JOIN fields f ON f.document_id = d.id WHERE f.field_key = 'Invoice No' runs in milliseconds even with thousands of documents because the index covers the predicate.
I also stored the OCR confidence score per field and the average per document. This turned out to be useful immediately — documents with low average confidence (below 0.75) were flagged for review, and the threshold caught every genuinely bad extraction in the test set.
The Orchestrator
The top-level pipeline tied it together:
import os, sqlite3, easyocr
reader = easyocr.Reader(['en'], gpu=False)
conn = sqlite3.connect('documents.db')
init_schema(conn)
image_dir = 'inputs/'
for fname in os.listdir(image_dir):
if not fname.lower().endswith(('.png', '.jpg', '.jpeg')):
continue
path = os.path.join(image_dir, fname)
ocr_results = reader.readtext(path)
if is_table_document(ocr_results):
records = extract_table_records(ocr_results)
store_table_records(conn, fname, records)
else:
kv_pairs = extract_kv_pairs(ocr_results)
avg_conf = sum(r[2] for r in ocr_results) / len(ocr_results)
store_kv_document(conn, fname, kv_pairs, avg_conf)
conn.close()
The is_table_document() heuristic was simple: if the topmost detected line had more than 3 tokens and none ended with a colon, it was a table header. That covered every table in the dataset without a single misclassification.
EasyOCR Initialization: The One Footgun
One thing worth calling out explicitly: easyocr.Reader downloads model weights on first instantiation and loads them into memory — this takes 10-30 seconds and several hundred MB of RAM. Initialize the reader once at startup, never inside a loop. The code above does this correctly, but it's easy to miss if you're iterating quickly, and the performance penalty of re-initializing per file is severe enough to make the pipeline feel broken.
Numbers
The full dataset was close to 3,000 image files. Processing time end-to-end was about 36 hours on CPU (no GPU) — I kicked it off overnight and let it run. The EasyOCR model is the bottleneck: each image takes 1-4 seconds depending on content density. With GPU inference that drops by roughly 5x, but the machine I was running this on didn't have a GPU and it didn't matter: 36 hours of unattended compute to do a month of manual work is an excellent trade.
Accuracy: 94% of key-value pairs extracted correctly on the first pass. The remaining 6% fell into two buckets — genuinely poor scan quality (blurry or skewed images where EasyOCR confidence was already low and flagging worked correctly) and a handful of two-line field values that the single-line parser missed. I fixed the two-line case in about 20 minutes after reviewing the flagged outputs.
The SQLite database ended up at around 18 MB for ~3,000 documents and ~42,000 extracted fields. Query time for any field lookup: under 5ms.
What I'd Do Differently
The colon-heuristic for label detection is brittle outside the specific document types I was working with. A more robust approach would be to use a layout analysis model — LayoutLM[2] or a similar document understanding model — which reasons about document structure semantically rather than with geometric heuristics. For the scope of this project, the heuristic was the right call: it took an hour to implement and worked. A transformer-based layout model would have taken days and delivered marginal accuracy improvement on these specific documents.
I'd also separate the confidence-based flagging threshold into a config parameter exposed at the CLI level. Hardcoding 0.75 worked here, but different document types have different baseline confidence distributions, and making this tunable without touching code would make the tool more reusable.
The schema is also intentionally simple. For a production system with many document types, you'd want a document_types table and a way to define expected fields per type — so you can detect when a supposedly standard document is missing required fields, rather than silently storing an incomplete record.
The Actual Lesson
The technical decisions here were all reasonable, but the more interesting thing is what the exercise revealed about the nature of "manual" knowledge work. The person who assigned me this task had done the same type of image-to-spreadsheet work before, by hand. It had taken roughly a month. The expectation was that it would take me a month too — that the work was inherently proportional to the number of files.
It isn't. Most structured manual data extraction is pattern-following. If it's pattern-following, it can be described as rules. If it can be described as rules, it can be automated. The limiting factor is rarely the complexity of the automation — the EasyOCR + SQLite pipeline is maybe 200 lines of Python — it's whether you recognize that the pattern exists and whether you have the tools to exploit it.
The boredom was useful. If the task had been interesting, I might have just done it manually.