Vision-LLM OCR over Harvest receipt scans — unlock mileage-over-time + service substrate (parked 2026-05-24)

DARE.CO.UK · PARKED SKETCH · 2026-05-26

Mirrored from ~/.claude/.../memory/parked_sketch_receipt_vision_ocr_vehicle_history_2026-05-24.md. This is a design sketch parked for future build — read for context, not as a current deliverable.

Harvest “notes” fields are summary-only (“LR4”, “Replacement Wing Mirror LR4 via eBay”); the actual maintenance data (odometer, line items, fluids, wear measurements) lives INSIDE the receipt scan images. A vision-LLM batch pass over the 5,607 Harvest receipts costs ~$4 and reconstructs the entire vehicle-service substrate as structured JSON. First consumer: real mileage-over-time charts per vehicle. Second: cost-per-mile by window, service-interval drift detection, parts catalog by model.

Dan 2026-05-24: “the real catch, is extracting from Harvest scans, data inside receipts, that holds the key the milage-over-time unlock”

The problem this solves

Asked LR4 for its mileage. Harvest has 145 LR4-tagged receipts. Grep returned 0 with mileage readings — because the notes field is a 2-word summary, not the receipt content. The odometer is inscribed on the service invoice image, never typed into Harvest’s note field.

Result: today we ask Dan (“125,000ish? Actually 129,050. Wait, that’s the F-250.”), the truth lives inside images, and we can’t run any time-series analysis (cost-per-mile, service-interval drift, mileage acceleration) until the substrate moves from images to JSON.

The mechanism

Vision-LLM pass over receipt JPG/PDF scans. Per receipt, extract structured JSON:

{
  "vendor": "A&G Customs",
  "service_date": "2023-02-14",
  "odometer_mi": 128986,
  "vehicle_hint": "F-250",          // VIN snippet or plate or written model
  "line_items": [
    {"description": "Exhaust manifolds replacement", "amount": 1240.00},
    {"description": "Tie rod replacement", "amount": 420.50},
    {"description": "Brake fluid flush", "amount": 89.00},
    {"description": "Front shocks (pair)", "amount": 285.00},
    {"description": "2x Hankook Dynapro AT2 RF11 tires", "amount": 412.00},
    {"description": "PA state inspection", "amount": 42.00}
  ],
  "total": 3228.15,
  "fluids_noted": ["brake fluid flush"],
  "wear_measurements": {"front_brakes_mm": null, "tire_tread_mm": null},
  "next_service_due": null,
  "tech_notes": "spark plugs original — danger zone above 120k"
}

Cost (Haiku 4.5 Batches with vision)

Per receipt: ~500-1500 input tokens (image) + ~300 output tokens. At Haiku Batches rates (no caching applicable per feedback_micro_gestures_pattern.md cost analysis) ≈ $0.0007 per receipt.

Handwriting is included — same model

Dan 2026-05-24 follow-up: “Add to the list, hand-written invoices from Jorge - will need a hand-writing-OCR-capability to infer and capture”

No special handwriting-OCR tooling needed. Haiku 4.5 vision handles printed receipts and handwritten invoices through the same API call. Legibility is the only constraint — clearly-written contractor receipts (Jorge garden work, Jose electrics, A&G Customs hand-noted services) extract just as well as printed POS receipts. Where handwriting is borderline, the prompt can ask the model to flag low-confidence fields.

What the substrate unlocks (downstream)

1. Mileage-over-time chart per vehicle — actual odometer history with service-date x-axis. Not Dan-estimates.
2. Real miles/year by year — was the BMW driven 17k in 2024 vs 8k in 2025? Now answerable.
3. Cost-per-mile by window — service-cost density when accumulating miles vs parked.
4. Service-interval drift detection — “last oil change 8k mi ago vs 5k recommended” alerts.
5. Parts catalog per vehicle — which model has eaten the most brake pads / spark plugs / tires.
6. Replacement-anniversary tracking — “tires due Feb 2027 based on 2023 install + 50k mi life”.
7. Pre-2017 mileage gaps — even one old service receipt with odometer pegs the timeline.
8. Future insurance/sale ammo — full service history reproducible from substrate for sale listing or claim packet.

Why now isn’t urgent (but soon is right)

Wait condition: nothing depends on the vehicle history TODAY. But the moment any of these surfaces: - Selling a vehicle (need maintenance history packet) - Insurance review (need service substantiation) - Deciding LR4 keep-vs-sell (cost-per-mile vs replacement) - BMW high-mileage service planning (when’s the next big spend predicted?)

…this becomes the load-bearing data. Build before the question is asked, not after.

Build approach

Build trigger

Build when ANY: - Vehicle sale/keep decision is on the table - Insurance review window - Dan asks “what did the LR4 cost per mile in 2024?” - After the unified-toolkit-scaffold build completes (sibling priority)

Cross-references

• parked_sketch_amazon_5pct_cashback_calculator_2026-05-24.md — sibling substrate-payoff sketch (cash-back vs vehicle cost-per-mile)
• feedback_one_batch_llm_beats_hand_iteration.md — the LLM-batch-as-substrate-extractor pattern
• feedback_vendor_lookup_layer2_2026-05-23.md — vendor-name disambiguation via LLM; sibling layer-2 work
• feedback_substrate_recategorization_assisted.md — same recipe (rule sweep + LLM residue + human review)
• pa/_data/vehicles.json — consumer of the mileage data
• pa/properties/new-hope/expenses/_data/expenses.jsonl — Harvest substrate (5,607 receipts)

Validation run — 2026-05-24 sample results

Ran pa_harvest_receipt_vision_batch.py on the 138-receipt service-shaped subset (vehicle services + handyman invoices, filtered via filename regex). Findings:

Reconstructed mileage timeline from receipts alone:

Data-quality issues to address before charting:

• ~25% of odometer captures pulled the wrong number (Costco gas transaction code, Amex statement reference, $0 line items where “mileage” was a quote not a reading). Filter rule: require co-occurring service-vendor name + non-zero total.
• 4 receipts had vehicle=”?” — filename lacks vehicle keyword. Re-attribute via Harvest expense ID join.
• Vendor names sometimes drift between OCR runs (A&C Customs vs A&G Customs). Could canonicalize via per-vendor fuzzy-match pass.
• Rate-limit errors recover cleanly on --force rerun.

Two-layer architecture validated by Dan 2026-05-24

Dan: “OCR gets you 70% of the way, with JSON. Then level 2 clean-up gets you to 95%.”

The clean substrate that enables cross-vendor / cross-time product-level queries (“coffee in 2020 vs 2025 at Giant vs Trader Joe’s“) needs both layers:

Without L2, line-item search is grep-and-hope across receipt-printer abbreviations. With L2, it’s a real database.

Reusable pipeline-pattern emerging — vision-OCR ingest

This is the second “vision-LLM-as-substrate-extractor” pattern instance (Amazon emails were the first via Chrome screenshot → email-evidence). Generalizes to:

Worth a sibling memory: feedback_vision_ocr_ingest_pipeline.md capturing the generic pattern (filter by filename → preprocess via sips → vision-extract → JSON sidecar → optional L2 normalize → query substrate).

Primary target — audreyinc datasets (not pa vehicle receipts)

Dan 2026-05-24 follow-up: “Mostly, it’s for audreyinc - processing her datasets”

The vehicle-receipt sample run was the validation case. The real build is processing Audrey’s business data substrate. Vehicle reactivation only happens if a specific pa-side question lands; the bigger ROI is audreyinc-side.

Likely audreyinc image substrates worth processing (pending Dan/Audrey-confirmed inventory):

This connects to existing audreyinc work in memory:

• parked_sketch_shopify_full_lifecycle_2026-05-23.md — Shopify side of the audreyinc surface; vision-OCR feeds the pre/parallel-Shopify paper trail into the same substrate
• feedback_audrey_eras_few_clicks_deep.md — archive eras snapshot project; OCR of physical archive feeds the same era-by-era narrative
• feedback_email_ingest_pattern_extends_to_ebay.md — sibling extraction pattern for digital substrates; vision-OCR is the analog-document counterpart

Vehicle receipts — validation scope only

The 138-receipt run above was the proof-of-mechanism. The actual vehicle-side work (cleanup, charting on pa.gf.cx/vehicles/) only proceeds if a specific pa question lands:

1. Selling a vehicle (need maintenance history packet)
2. Insurance review window
3. “What did the LR4 cost per mile in 2024?” or sibling question
4. Kitchen-reno / claim-packet build needs the receipt substrate

Resume / build trigger

Status: PARKED post-validation 2026-05-24. Sample run proved the mechanism works at acceptable quality and cost on vehicle receipts. Wait condition before unparking:

1. audreyinc data substrates inventoried — Dan/Audrey identify which boxes of paper/scanned material are worth processing first
2. A specific audreyinc question that needs the substrate (“trade-show contacts from 2024 we never followed up with”, “wholesale sell-through by retailer”, “vendor cost per fabric category”)
3. pa-side trigger (insurance / sell / claim) as above

When unparked for audreyinc: scope and cost depend on inventory size + which substrates (vendor invoices alone is a different size than full archive). The pipeline pattern (filter → preprocess → vision L1 → optional L2 → query) is the same; the per-substrate cost scales with image count.

The aphorism

The data isn’t missing. It’s just inside the image. Vision-LLM is the can-opener. The second pass is the categorizer that turns the can’s contents into ingredients.