OCR Text Extraction Pipeline

This project was updated after the internship concluded.

A general-purpose OCR tool: give it any image and it extracts the text. It also has an optional mode for pulling structured fields (name, date, ID number, etc.) out of documents that have a label/value layout, like ID cards or receipts.

What it can do

• Extract text from any image — photos, screenshots, scanned documents, receipts. Not limited to any one document type.
• Optional deskewing — estimates rotation from text-focused image components and applies a correction only when alignment evidence passes safety checks. Deskewing is disabled in the selected configuration.
• Configurable image scaling — supports original-size and 2× images. The selected configuration uses 2× scaling, gentle denoising, no thresholding, and Tesseract segmentation mode 6; settings are saved in ocr_settings.json.
• Pull out structured fields — for documents with a label/value shape ("NAME" / "Ruhaan Sahi", or "TOTAL AMOUNT $117.00"), it returns them as clean key-value pairs instead of a wall of text. Works generically — it isn't hardcoded to ID cards specifically, and correctly returns nothing when an image has no such structure.
• Report a confidence score — every result comes with a 0-100 confidence number, computed from Tesseract's own per-word confidence values. This is the engine's confidence, not measured transcription accuracy or CER.
• Generate synthetic test data — a script produces fake ID-card images with known correct answers, at three difficulty levels (clean, slightly rotated, rotated + blurred), so the pipeline can be tested without ever touching real personal data.
• Measure its own accuracy — an evaluation script runs the full pipeline against the synthetic dataset and reports character error rate, word error rate, and per-field accuracy, broken down by difficulty tier.
• Command-line interface — one entry point (main.py) ties everything together, with plain-text or JSON output.

Evaluation

7.48% CER across all 50 official FUNSD test forms. All 50 forms completed without processing failures.

Character error rate (CER) measures the number of character insertions, deletions, and substitutions needed to turn recognized text into reference text, divided by the number of reference characters. Lower is better. Unlike confidence, CER requires correct reference text.

• evaluate.py evaluates synthetic ID images against the fixed reference text constructed from their CSV labels. It reports mean per-image CER and WER, plus exact field-match accuracy, by difficulty and overall.
• benchmark_funsd.py loads the FUNSD dataset ZIP using Python's zipfile, pairs each scanned PNG with its corresponding JSON annotation, and extracts the images locally.
• It reads annotated words and bounding boxes from the JSON files, groups vertically overlapping words into rows, sorts rows top-to-bottom and words within each row left-to-right, and assembles the reference text. Annotation boxes are used to construct references, not supplied to the OCR engine.
• It selects preprocessing using the first 20 sorted training forms and confirms the selected configuration on another 20 training forms before freezing settings for the test run. No model training is performed.
• Each test image is converted to grayscale with OpenCV, gently denoised, and scaled to 2× size. Tesseract 5.5.3 is called through pytesseract.image_to_data with segmentation mode 6. Deskewing and thresholding are disabled in the tested configuration.
• Recognized words are assembled into text using Tesseract's line identifiers. Both recognized and reference text are casefolded and whitespace-normalized before RapidFuzz's Levenshtein distance counts character insertions, deletions, and substitutions. Aggregate CER is total character edits divided by total reference characters; processing failures remain included with empty predictions.
• Per-form predictions, errors, timings, configuration, and source hashes are saved locally under the ignored benchmark_results/ directory. Source and configuration hashes remained unchanged during the test run.

The test forms have been inspected previously, so this is a repeat evaluation. The reconstructed reference reading order can differ from Tesseract's order on tables or columns and increase CER. The score measures transcription under this protocol, not identity validation or universal document accuracy. Confidence is a separate engine-reported measure.

Tech stack

• Python
• Tesseract OCR — recognition engine (system binary)
• pytesseract — Python wrapper for Tesseract
• OpenCV — preprocessing (grayscale, denoising, optional deskewing, scaling, and thresholding)
• NumPy — array operations underpinning OpenCV
• Pillow — synthetic test data generation
• RapidFuzz — edit-distance calculations for the FUNSD benchmark

Tools used

Tesseract OCR — the underlying open-source OCR engine that does the actual recognition. It's a separate system binary (installed via Homebrew on Mac), not a Python package — pytesseract just calls out to it.

Bundled fonts (DejaVu Sans) — shipped in assets/fonts/ and used by the synthetic data generator instead of relying on whatever font happens to be installed on a given machine. Testing showed that different default fonts (Arial on Mac vs. DejaVu on Linux) produced meaningfully different OCR accuracy on what was supposed to be the same benchmark — bundling the font makes results reproducible across machines.

Git — version control.

Limitations

• Fails on stylized signage and scene text. Tested against a real billboard photo with large 3D-perspective lettering over a busy background — Tesseract returned garbled fragments, not real text. It's built for flat printed or scanned text, not text photographed at an angle in a natural scene. This would need a dedicated scene-text detection model to handle properly.
• Blur destroys alphanumeric field accuracy badly. On the synthetic "hard" difficulty tier (rotation + blur), ID number extraction accuracy drops to ~2%, even though names and dates hold up much better on the same images. A single misread character breaks an exact-match field like an ID number, while it's more forgivable in a name or date. Upscaling and sharpening were tested as fixes and only produced modest gains (~5-7 percentage points).
• No text localization stage. The tool assumes the image is already roughly text-bearing; it doesn't find and crop text regions out of a larger, busier photo first.
• Field extraction is rule-based, not learned. extract_fields.py uses regex and formatting heuristics (ALL CAPS labels, colon separators) to find label/value pairs. It works well on documents shaped like the ones tested, but isn't a trained model and won't generalize to arbitrarily formatted documents.
• No handwriting support. Tesseract is built for printed text; handwriting accuracy is poor.
• Limited file format support. OpenCV can't read HEIC or AVIF images directly — they need to be converted to PNG/JPG first (e.g. with macOS's sips command).
• Binarization is off by default. An Otsu binarization step exists in preprocess.py but isn't applied by default, since testing showed it reduces accuracy on rotated or blurred text more than it helps — it destroys antialiasing detail Tesseract's own internal thresholding relies on.

Planned future work

• Train a custom Keras/TensorFlow model (CNN + RNN with CTC loss) for character recognition, aimed specifically at improving accuracy on blurred images, where Tesseract currently struggles most.
• Evaluate keras-ocr as a pretrained alternative for the scene-text/signage case that Tesseract currently fails on.
• Character-segmentation pipeline for reading full handwritten words (printed/block letters first), built on top of a trained single-character classifier.