From be422e81c662c8dbad6b9c97b141f3c0c127c4bb Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Sat, 11 Jul 2026 10:49:21 -0700 Subject: [PATCH 1/7] Add 'Fast LLM + Human-in-the-Loop' post; refresh featured post and series New post 'How a Three-Person Team Used Cheap, Fast LLMs + Human-in-the-Loop to Classify Hundreds of Millions of Documents in ~2 Months', tagged both 'featured' (becomes the featured post) and 'content-trust-series' (joins the featured series). Removes 'content-trust-series' from the PhotoDNA post so the series now shows Dual-Embedding Trust Scoring + this new post. Co-Authored-By: Claude Opus 4.8 (1M context) --- _posts/2026-01-20-photodna-csam-detection.md | 1 - ...st-llm-human-in-the-loop-classification.md | 297 ++++++++++++++++++ 2 files changed, 297 insertions(+), 1 deletion(-) create mode 100644 _posts/2026-07-11-fast-llm-human-in-the-loop-classification.md diff --git a/_posts/2026-01-20-photodna-csam-detection.md b/_posts/2026-01-20-photodna-csam-detection.md index bcca299..5b8a412 100644 --- a/_posts/2026-01-20-photodna-csam-detection.md +++ b/_posts/2026-01-20-photodna-csam-detection.md @@ -6,7 +6,6 @@ tags: - aws - lambda - databricks -- content-trust-series team: ML Data Engineering author: anishk123 --- diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md new file mode 100644 index 0000000..31f6c95 --- /dev/null +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -0,0 +1,297 @@ +--- +layout: post +title: "How a Three-Person Team Used Cheap, Fast LLMs + Human-in-the-Loop to Classify Hundreds of Millions of Documents in ~2 Months" +tags: +- featured +- content-trust-series +- machinelearning +- llm +- databricks +- scribd +team: ML Data Engineering +author: anishk123 +--- + +*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools.* + +## TL;DR — Key Learnings + +LLM models change. Prompt quality changes. Cost changes. We assumed that from day one. + +So instead of building a one-off system tied to a specific model or a single "perfect prompt," we focused on a **repeatable process** that we successfully applied to two large-scale projects (Slideshare and Scribd). That repeatable process is what we're sharing. + +**Key learnings:** + +1. **Fast models give you breadth.** A simple prompt on a fast, low-cost model can label tens of thousands of documents quickly. The results won't be perfect — and that's okay. It turns Subject Matter Expert (SME) work from "label everything from scratch" into "review, correct, and curate." +2. **Batch inference makes scaling simpler.** For large backfills and corpus-scale labeling, asynchronous batch inference is usually both operationally simpler than building realtime request pipelines and meaningfully cheaper. +3. **"Fast model → Judge model" creates a scalable quality loop.** Use a higher-capability model as a judge to agree/disagree with the fast model's output. Use disagreements (and a small sample of agreements) as the highest-leverage items for SME review and prompt iteration. +4. **Validate beyond your small golden set.** Engineer prompts on a small golden dataset, then check generalization using 1% of the corpus, and run judge review on a 0.1% random sample (plus targeted slices when needed). +5. **Use familiar tools to keep a small team moving fast.** Databricks for data sourcing and orchestration → batch LLM calls → Google Sheets for SME validation → ingest back into Databricks → repeat. + +Repeatable mantra (we used it constantly): + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## What We Achieved (With a Small Team) + +This process enabled projects that normally require far more people and time: + +- **Slideshare:** scanned tens of millions of slideshows, classifying each against adult-related content categories (e.g., sexually explicit content, drugs, alcohol, tobacco) and flagging the small fraction that matched. +- **Scribd:** scanned hundreds of millions of documents with a small cross-functional team — 1 engineer, 1 SME (annotations), 1 data scientist — in about two months. + +The specific labels and thresholds vary by project and policy requirements. The important part is that the **process** was portable across both efforts. + +## The Core Point + +LLM models, prompt quality, and cost will change over time. The durable asset is not a single prompt or model — it's a **repeatable human-in-the-loop process**. + +That process is: + +1. Use a fast model to generate candidate labels and short rationales at scale +2. Use a judge model to flag disagreements and ambiguous cases +3. Focus SME review only on the highest-value slices +4. Ingest SME corrections as a growing golden dataset +5. Iterate prompts and re-run only what you need (slices / disagreements) + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Workflow Overview + +Here's the end-to-end loop we repeated: + +1. **Source documents in Databricks** +2. **Standardize a document PDF** and metadata +3. **Build JSONL batches** for LLM inference +4. **Run batch inference** with a fast model (candidate label + short rationale + context) +5. **Store outputs** with prompt/model versioning +6. **Validate** on 1% of the corpus; run judge on 0.1% random sample +7. **Export disagreements** (and a small sample of agreements) to Google Sheets +8. **SMEs review and correct labels** quickly +9. **Ingest corrections back into Databricks** as the next golden dataset version +10. **Improve prompts** and repeat the loop + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Step-by-Step + +### Step 1 — Standardize Inputs (Before You Touch Prompts) + +The easiest way to make prompt work repeatable is to standardize what you feed the model. We used a consistent document "snippet" concept so every workflow starts the same way. + +**Example fields:** + +- `doc_id` +- `title` (if available) +- `pdf` +- `language` (if known) +- optional metadata + +**Why this matters:** + +- Prompts are simpler and more stable +- Batch sizing is predictable +- You avoid rewriting "how do we feed docs to the model?" for every project + +### Step 2 — Fast-Model Bulk Pass (Breadth First) + +**Goal:** create candidate labels at scale, cheaply and quickly. + +We asked the fast model for: + +- **label** +- **short rationale** + +Illustrative prompt (example only; actual production prompts were tuned and different): + +``` +Classify this document as ADULT, NOT_ADULT, or UNKNOWN. +Provide a 1–2 sentence rationale + +Document: + +``` + +**Why ask for rationales:** + +- Makes SME review faster +- Makes judge evaluation more reliable +- Makes debugging prompt failures dramatically easier + +### Step 3 — Batch Inference (Keep It Boring) + +For very large datasets, asynchronous batch inference is the "boring" choice that unlocks speed. You submit a JSONL file of requests, let it run, then download results later. + +**Why it works well:** + +- Generally 50% cheaper than realtime inference +- Fewer moving parts than realtime queueing systems +- Fewer timeout/retry headaches +- Easier to reproduce runs +- Easier to parallelize and monitor at the batch level + +### Step 4 — Store Everything with Versioning (So You Can Iterate) + +Store both raw and parsed outputs with run metadata: + +- prompt version +- model name / model version +- run timestamp +- `doc_id` +- label / rationale + +This turns iteration into something you can measure and regress-test, not a "vibes-based" exercise. + +### Step 5 — Judge Model Pass (Quality Signal at Scale) + +The judge model is a higher-capability model used to review a sample and decide whether it agrees with the fast model. It should explicitly: + +- **agree or disagree** +- provide a short rationale +- provide a corrected label when it disagrees + +Illustrative judge prompt (example only): + +``` +Here is the document and the fast model's label + rationale. +Do you agree? YES/NO +If NO, correct the label. +Explain briefly. +``` + +The judge is not a replacement for SMEs — it's a way to: + +- concentrate SME attention on the highest-value cases +- surface failure patterns faster +- create measurable disagreement signals for iteration + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +### Step 6 — SME Loop via Google Sheets (Focused Human Time) + +We exported: + +- **judge disagreements** (high priority) +- **a small random sample of agreements** (spot checks) + +SMEs reviewed the sheet and made edits. This is where human-in-the-loop scales: you're not asking SMEs to label everything — you're asking them to validate a small, high-value slice. + +Then we ingested the SME edits back into Databricks and versioned the golden dataset. + +## Sampling Strategy: 1% Validation + 0.1% Judge Sample + +Why not just rely on the small golden dataset? Because small golden datasets tend to be: + +- cleaner than the real corpus +- missing long-tail edge cases +- biased toward examples you already understand + +We used: + +- **1% of the corpus** as a validation slice for sanity checks +- **0.1% random sample** for judge review (plus targeted slices when needed) + +**What this catches:** + +- rare content patterns +- languages or formats underrepresented in golden data +- prompt ambiguity that doesn't show up in curated examples +- unexpected "UNKNOWN" spikes due to drift + +If random sampling isn't enough, stratify by patterns visible in the rationale and disagreements such as: + +- document type/format +- document length +- language + +## Prompt Engineering: Make It Iterative, Not Fragile + +A scalable, repeatable pattern: + +1. fast model outputs label + rationale +2. judge model explicitly agrees/disagrees +3. use disagreements as "training data" for prompt improvements: + - clarify definitions + - add a few targeted examples (carefully) + - tighten output formatting rules + - reduce ambiguity and edge-case confusion + +You should see: + +- disagreement rate drop over time +- SME correction rate drop over time +- fewer surprise failures on the 1% validation slice + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Metrics That Matter (Simple but Effective) + +You do not need an overly complex evaluation harness to start. Track: + +- disagreement rate (fast vs judge) +- SME correction rate (on judge disagreements) +- throughput (docs processed per job/day) +- iteration trend (disagreement should fall as prompts improve) + +The point is not "perfect metrics." The point is to make iteration measurable. + +## Common Failure Modes (And How We Handled Them) + +1. **Overconfident wrong answers** + Fix: capture context and prioritize big disagreements for SME review. +2. **Google Sheets becomes a bottleneck** + Fix: keep the sheet schema strict; export smaller batches; parallelize review across multiple sheets. +3. **Batch retries / partial failures** + Fix: keep batch requests idempotent; retry only failed batches; store intermediate results per batch. + +## What To Copy If You Want to Replicate This + +1. Standardize document inputs (`doc_id` + pdf or excerpt + minimal metadata) +2. Run a small fast-model probe (100–1k annotated docs) to sanity check outputs +3. Run a fast-model batch warmup for a larger slice +4. Run a judge model on 0.1% random sample +5. Export disagreements to Google Sheets and have SMEs review +6. Ingest edits as a versioned golden dataset +7. Iterate prompts, validate on 1%, then scale up + +Repeatable mantra (again): + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Summary + +LLM models, prompt quality, and cost will change over time. That's why we built a repeatable process — not a one-off prompt — and validated it across two large real-world projects. + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +That repeatability is what made it possible for a small cross-functional team to classify ~25M slideshows and ~400M documents in ~2 months — and it's the process we hope others can reuse and adapt as models and pricing evolve. + +## Appendix + +### A) Suggested Google Sheets Columns for SME Review + +| doc_id | snippet | fast_label | fast_rationale | judge_agree | judge_label | judge_rationale | sme_label | sme_notes | reviewer | review_ts | +|--------|---------|------------|----------------|-------------|-------------|-----------------|-----------|-----------|----------|-----------| + +### B) Mermaid Diagram + +```mermaid +flowchart LR + A[1) Source data in Databricks] --> B[2) Standardize doc snippets + metadata] + B --> C[3) Build JSONL batches (idempotent)] + C --> D[4) Batch inference: fast model] + D --> E[5) Store candidate label + rationale] + E --> F[6) Validate 1% slice; judge 0.1% sample] + F --> G[7) SME reviews disagreements in Sheets] + G --> H[8) Ingest edits → grow golden set → iterate] + H --> C +``` + +### C) One-Page Checklist + +- Standardized doc snippet schema +- JSONL batch builder +- Fast-model batch runner + parser +- Judge-model batch runner + disagreement computation +- Sheets export/import +- Golden dataset versioning +- Dashboard: disagreement rate + SME correction rate + unknown rate From b159a27a915d0614ee630a09aa36ee533f27958e Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Sat, 11 Jul 2026 11:11:52 -0700 Subject: [PATCH 2/7] Address review: add tag page, fix Mermaid, add workflow visuals - Add tag/llm/index.md so the new `llm` tag no longer 404s (Cursor Bugbot; matches the CONTRIBUTING generate-tags output format). - Fix the Appendix B Mermaid: quote node labels containing parentheses so it renders (previously broken syntax). Heading and text otherwise unchanged from the source doc. - Add three on-brand SVG diagrams (Scribd palette) grounded in the post text: - Workflow Overview: the repeatable Fast Model -> Judge -> SME -> Ingest -> Iterate loop. - Sampling Strategy: breadth-to-precision funnel (100% -> 1% -> 0.1% -> SME reviews the disagreements). - The Core Point: human-in-the-loop (model proposes labels, humans have the final say, corrections grow a versioned golden set, feeding back). - Add horizontal-rule dividers between top-level sections for readability (mirrors the section breaks in the source doc). Co-Authored-By: Claude Opus 4.8 (1M context) --- ...st-llm-human-in-the-loop-classification.md | 44 +++++++++++--- .../2026-fast-llm-hitl/human-in-the-loop.svg | 58 +++++++++++++++++++ .../2026-fast-llm-hitl/repeatable-loop.svg | 56 ++++++++++++++++++ .../2026-fast-llm-hitl/sampling-funnel.svg | 46 +++++++++++++++ tag/llm/index.md | 6 ++ 5 files changed, 202 insertions(+), 8 deletions(-) create mode 100644 post-images/2026-fast-llm-hitl/human-in-the-loop.svg create mode 100644 post-images/2026-fast-llm-hitl/repeatable-loop.svg create mode 100644 post-images/2026-fast-llm-hitl/sampling-funnel.svg create mode 100644 tag/llm/index.md diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md index 31f6c95..0ca2bcc 100644 --- a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -12,7 +12,7 @@ team: ML Data Engineering author: anishk123 --- -*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools.* +*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools* ## TL;DR — Key Learnings @@ -32,6 +32,8 @@ Repeatable mantra (we used it constantly): > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## What We Achieved (With a Small Team) This process enabled projects that normally require far more people and time: @@ -41,10 +43,14 @@ This process enabled projects that normally require far more people and time: The specific labels and thresholds vary by project and policy requirements. The important part is that the **process** was portable across both efforts. +--- + ## The Core Point LLM models, prompt quality, and cost will change over time. The durable asset is not a single prompt or model — it's a **repeatable human-in-the-loop process**. +![Humans stay in the loop: the model proposes labels at scale, humans have the final say on the disagreements, and corrections grow a versioned golden set that feeds back into the model.](/post-images/2026-fast-llm-hitl/human-in-the-loop.svg) + That process is: 1. Use a fast model to generate candidate labels and short rationales at scale @@ -55,10 +61,14 @@ That process is: > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Workflow Overview Here's the end-to-end loop we repeated: +![The repeatable loop: a fast model labels for breadth, a judge model checks a sample, SMEs make the final call, corrections grow the golden set, and the prompt is iterated — then the loop repeats.](/post-images/2026-fast-llm-hitl/repeatable-loop.svg) + 1. **Source documents in Databricks** 2. **Standardize a document PDF** and metadata 3. **Build JSONL batches** for LLM inference @@ -72,6 +82,8 @@ Here's the end-to-end loop we repeated: > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Step-by-Step ### Step 1 — Standardize Inputs (Before You Touch Prompts) @@ -177,8 +189,12 @@ SMEs reviewed the sheet and made edits. This is where human-in-the-loop scales: Then we ingested the SME edits back into Databricks and versioned the golden dataset. +--- + ## Sampling Strategy: 1% Validation + 0.1% Judge Sample +![Breadth first, then concentrate human time: the fast model labels the full corpus, a 1% slice validates generalization, a judge model re-checks a 0.1% sample, and SMEs review only the disagreements.](/post-images/2026-fast-llm-hitl/sampling-funnel.svg) + Why not just rely on the small golden dataset? Because small golden datasets tend to be: - cleaner than the real corpus @@ -203,6 +219,8 @@ If random sampling isn't enough, stratify by patterns visible in the rationale a - document length - language +--- + ## Prompt Engineering: Make It Iterative, Not Fragile A scalable, repeatable pattern: @@ -223,6 +241,8 @@ You should see: > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Metrics That Matter (Simple but Effective) You do not need an overly complex evaluation harness to start. Track: @@ -234,6 +254,8 @@ You do not need an overly complex evaluation harness to start. Track: The point is not "perfect metrics." The point is to make iteration measurable. +--- + ## Common Failure Modes (And How We Handled Them) 1. **Overconfident wrong answers** @@ -243,6 +265,8 @@ The point is not "perfect metrics." The point is to make iteration measurable. 3. **Batch retries / partial failures** Fix: keep batch requests idempotent; retry only failed batches; store intermediate results per batch. +--- + ## What To Copy If You Want to Replicate This 1. Standardize document inputs (`doc_id` + pdf or excerpt + minimal metadata) @@ -257,6 +281,8 @@ Repeatable mantra (again): > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Summary LLM models, prompt quality, and cost will change over time. That's why we built a repeatable process — not a one-off prompt — and validated it across two large real-world projects. @@ -265,6 +291,8 @@ LLM models, prompt quality, and cost will change over time. That's why we built That repeatability is what made it possible for a small cross-functional team to classify ~25M slideshows and ~400M documents in ~2 months — and it's the process we hope others can reuse and adapt as models and pricing evolve. +--- + ## Appendix ### A) Suggested Google Sheets Columns for SME Review @@ -276,13 +304,13 @@ That repeatability is what made it possible for a small cross-functional team to ```mermaid flowchart LR - A[1) Source data in Databricks] --> B[2) Standardize doc snippets + metadata] - B --> C[3) Build JSONL batches (idempotent)] - C --> D[4) Batch inference: fast model] - D --> E[5) Store candidate label + rationale] - E --> F[6) Validate 1% slice; judge 0.1% sample] - F --> G[7) SME reviews disagreements in Sheets] - G --> H[8) Ingest edits → grow golden set → iterate] + A["1) Source data in Databricks"] --> B["2) Standardize doc snippets + metadata"] + B --> C["3) Build JSONL batches (idempotent)"] + C --> D["4) Batch inference: fast model"] + D --> E["5) Store candidate label + rationale"] + E --> F["6) Validate 1% slice; judge 0.1% sample"] + F --> G["7) SME reviews disagreements in Sheets"] + G --> H["8) Ingest edits → grow golden set → iterate"] H --> C ``` diff --git a/post-images/2026-fast-llm-hitl/human-in-the-loop.svg b/post-images/2026-fast-llm-hitl/human-in-the-loop.svg new file mode 100644 index 0000000..a6aa2ed --- /dev/null +++ b/post-images/2026-fast-llm-hitl/human-in-the-loop.svg @@ -0,0 +1,58 @@ + + + + + + + + + + + + + + + + Humans stay in the loop + The model proposes labels at scale; humans have the final say; corrections grow a versioned golden set. + + + + + + + + + + the golden set & improved prompts feed back into the model + + + + Model proposes + + Fast model + candidate label + rationale, at scale + + Judge model + flags disagreements on a sample + + + + Humans decide + + SME review + the final say on the label + + Focus on disagreements + not every document + + + + Ingest & iterate + + Golden set + corrections captured, versioned + + Measurable + disagreement & correction rates + diff --git a/post-images/2026-fast-llm-hitl/repeatable-loop.svg b/post-images/2026-fast-llm-hitl/repeatable-loop.svg new file mode 100644 index 0000000..9e274b0 --- /dev/null +++ b/post-images/2026-fast-llm-hitl/repeatable-loop.svg @@ -0,0 +1,56 @@ + + + + + + + + + + + + + + + + The repeatable loop + Breadth first with a fast model — then concentrate scarce human time where it matters. + + + + Iterate: improve prompts, then re-run only the slices / disagreements that changed + + + + + + + + + + + + + Fast Model + batch: label + rationale + + + + Judge Model + 0.1% sample, agree/disagree + + + + SME Review + Google Sheets, final say + + + + Ingest + grow the golden set + + + + Iterate + versioned, measurable + diff --git a/post-images/2026-fast-llm-hitl/sampling-funnel.svg b/post-images/2026-fast-llm-hitl/sampling-funnel.svg new file mode 100644 index 0000000..8be8f9e --- /dev/null +++ b/post-images/2026-fast-llm-hitl/sampling-funnel.svg @@ -0,0 +1,46 @@ + + + + Breadth first, then concentrate human time + The fast model labels everything; the judge and SMEs only ever touch a sample. + + + + + + + + + + + + Full corpus · 100% + fast model labels every document + + + Validation · 1% + sanity-check generalization + + + Judge · 0.1% + stronger model agrees / disagrees + + + SME review + review the disagreements + + + + One cheap batch pass + ~400M docs (Scribd), ~25M slideshows + + Beyond the golden set + catches drift & long-tail edge cases + + Quality signal at scale + measurable disagreement rate + + Highest-leverage human time + review disagreements, not everything + + diff --git a/tag/llm/index.md b/tag/llm/index.md new file mode 100644 index 0000000..a455404 --- /dev/null +++ b/tag/llm/index.md @@ -0,0 +1,6 @@ +--- +layout: tag_page +title: "Tag: llm" +tag: llm +robots: noindex +--- From 8344f1a9066e1bc2365548892120bf80464c146f Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Mon, 13 Jul 2026 11:49:57 -0700 Subject: [PATCH 3/7] Address review: keep photodna in the series, shorten the title MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Revert removing `content-trust-series` from the PhotoDNA post (per @rtyler). The Featured Series again shows PhotoDNA + Dual-Embedding Trust Scoring; the new post remains the Featured Post at the top of the home page. - Shorten the title to "Stop Building Models. Start Building Systems." — pithy, and it matches the title of Jeanie Lam's TrustCon 2026 presentation that this post accompanies. The longer framing lives in the subtitle. Co-Authored-By: Claude Opus 4.8 (1M context) --- _posts/2026-01-20-photodna-csam-detection.md | 1 + _posts/2026-07-11-fast-llm-human-in-the-loop-classification.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/_posts/2026-01-20-photodna-csam-detection.md b/_posts/2026-01-20-photodna-csam-detection.md index 5b8a412..bcca299 100644 --- a/_posts/2026-01-20-photodna-csam-detection.md +++ b/_posts/2026-01-20-photodna-csam-detection.md @@ -6,6 +6,7 @@ tags: - aws - lambda - databricks +- content-trust-series team: ML Data Engineering author: anishk123 --- diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md index 0ca2bcc..75b1522 100644 --- a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -1,6 +1,6 @@ --- layout: post -title: "How a Three-Person Team Used Cheap, Fast LLMs + Human-in-the-Loop to Classify Hundreds of Millions of Documents in ~2 Months" +title: "Stop Building Models. Start Building Systems." tags: - featured - content-trust-series From 3006d323896d4682a2abc6f87af1fe1658dc00a1 Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Mon, 13 Jul 2026 11:56:01 -0700 Subject: [PATCH 4/7] Address review: restructure the intro, relocate Key Learnings (per @rtyler) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per Tyler's note on article flow: drop the "TL;DR" heading and the up-front bullet dump so the opening reads as prose (headline -> gist -> mantra hook) that flows straight into the concrete results in "What We Achieved". Move the five key-learnings down into a "Key Learnings" recap just before the Summary. No article wording changed — the body has legal approval, so this only removes the "TL;DR" heading + redundant "Key learnings:" label and reorders blocks (verified via a token-multiset diff: zero words added, only those labels removed). Co-Authored-By: Claude Opus 4.8 (1M context) --- ...st-llm-human-in-the-loop-classification.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md index 75b1522..7a17080 100644 --- a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -14,20 +14,10 @@ author: anishk123 *A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools* -## TL;DR — Key Learnings - LLM models change. Prompt quality changes. Cost changes. We assumed that from day one. So instead of building a one-off system tied to a specific model or a single "perfect prompt," we focused on a **repeatable process** that we successfully applied to two large-scale projects (Slideshare and Scribd). That repeatable process is what we're sharing. -**Key learnings:** - -1. **Fast models give you breadth.** A simple prompt on a fast, low-cost model can label tens of thousands of documents quickly. The results won't be perfect — and that's okay. It turns Subject Matter Expert (SME) work from "label everything from scratch" into "review, correct, and curate." -2. **Batch inference makes scaling simpler.** For large backfills and corpus-scale labeling, asynchronous batch inference is usually both operationally simpler than building realtime request pipelines and meaningfully cheaper. -3. **"Fast model → Judge model" creates a scalable quality loop.** Use a higher-capability model as a judge to agree/disagree with the fast model's output. Use disagreements (and a small sample of agreements) as the highest-leverage items for SME review and prompt iteration. -4. **Validate beyond your small golden set.** Engineer prompts on a small golden dataset, then check generalization using 1% of the corpus, and run judge review on a 0.1% random sample (plus targeted slices when needed). -5. **Use familiar tools to keep a small team moving fast.** Databricks for data sourcing and orchestration → batch LLM calls → Google Sheets for SME validation → ingest back into Databricks → repeat. - Repeatable mantra (we used it constantly): > **Fast Model → Judge Model → SME Review → Ingest → Iterate** @@ -283,6 +273,16 @@ Repeatable mantra (again): --- +## Key Learnings + +1. **Fast models give you breadth.** A simple prompt on a fast, low-cost model can label tens of thousands of documents quickly. The results won't be perfect — and that's okay. It turns Subject Matter Expert (SME) work from "label everything from scratch" into "review, correct, and curate." +2. **Batch inference makes scaling simpler.** For large backfills and corpus-scale labeling, asynchronous batch inference is usually both operationally simpler than building realtime request pipelines and meaningfully cheaper. +3. **"Fast model → Judge model" creates a scalable quality loop.** Use a higher-capability model as a judge to agree/disagree with the fast model's output. Use disagreements (and a small sample of agreements) as the highest-leverage items for SME review and prompt iteration. +4. **Validate beyond your small golden set.** Engineer prompts on a small golden dataset, then check generalization using 1% of the corpus, and run judge review on a 0.1% random sample (plus targeted slices when needed). +5. **Use familiar tools to keep a small team moving fast.** Databricks for data sourcing and orchestration → batch LLM calls → Google Sheets for SME validation → ingest back into Databricks → repeat. + +--- + ## Summary LLM models, prompt quality, and cost will change over time. That's why we built a repeatable process — not a one-off prompt — and validated it across two large real-world projects. From 3669e4daee2cb028895f4e2e27cea0e8dbc06c5a Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Mon, 13 Jul 2026 12:48:00 -0700 Subject: [PATCH 5/7] Restructure per review: Hypothesis framing, Workflow Overview to the end MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename "The Core Point" -> "Hypothesis" so the section reads as the idea we set out to test. - Move the whole "Workflow Overview" section down to sit after Key Learnings, just before the Summary — it's reference material that broke the narrative. - Drop the redundant intro mantra restatement (the subtitle already states the Fast Model -> Judge -> SME -> Iterate loop). No approved body copy reworded (verified via token-multiset diff): only the "TL;DR"/mantra labels removed and the "Hypothesis" heading added; every sentence is verbatim and Workflow Overview moved intact. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...st-llm-human-in-the-loop-classification.md | 48 +++++++++---------- 1 file changed, 22 insertions(+), 26 deletions(-) diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md index 7a17080..88373ad 100644 --- a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -18,10 +18,6 @@ LLM models change. Prompt quality changes. Cost changes. We assumed that from da So instead of building a one-off system tied to a specific model or a single "perfect prompt," we focused on a **repeatable process** that we successfully applied to two large-scale projects (Slideshare and Scribd). That repeatable process is what we're sharing. -Repeatable mantra (we used it constantly): - -> **Fast Model → Judge Model → SME Review → Ingest → Iterate** - --- ## What We Achieved (With a Small Team) @@ -35,7 +31,7 @@ The specific labels and thresholds vary by project and policy requirements. The --- -## The Core Point +## Hypothesis LLM models, prompt quality, and cost will change over time. The durable asset is not a single prompt or model — it's a **repeatable human-in-the-loop process**. @@ -53,27 +49,6 @@ That process is: --- -## Workflow Overview - -Here's the end-to-end loop we repeated: - -![The repeatable loop: a fast model labels for breadth, a judge model checks a sample, SMEs make the final call, corrections grow the golden set, and the prompt is iterated — then the loop repeats.](/post-images/2026-fast-llm-hitl/repeatable-loop.svg) - -1. **Source documents in Databricks** -2. **Standardize a document PDF** and metadata -3. **Build JSONL batches** for LLM inference -4. **Run batch inference** with a fast model (candidate label + short rationale + context) -5. **Store outputs** with prompt/model versioning -6. **Validate** on 1% of the corpus; run judge on 0.1% random sample -7. **Export disagreements** (and a small sample of agreements) to Google Sheets -8. **SMEs review and correct labels** quickly -9. **Ingest corrections back into Databricks** as the next golden dataset version -10. **Improve prompts** and repeat the loop - -> **Fast Model → Judge Model → SME Review → Ingest → Iterate** - ---- - ## Step-by-Step ### Step 1 — Standardize Inputs (Before You Touch Prompts) @@ -283,6 +258,27 @@ Repeatable mantra (again): --- +## Workflow Overview + +Here's the end-to-end loop we repeated: + +![The repeatable loop: a fast model labels for breadth, a judge model checks a sample, SMEs make the final call, corrections grow the golden set, and the prompt is iterated — then the loop repeats.](/post-images/2026-fast-llm-hitl/repeatable-loop.svg) + +1. **Source documents in Databricks** +2. **Standardize a document PDF** and metadata +3. **Build JSONL batches** for LLM inference +4. **Run batch inference** with a fast model (candidate label + short rationale + context) +5. **Store outputs** with prompt/model versioning +6. **Validate** on 1% of the corpus; run judge on 0.1% random sample +7. **Export disagreements** (and a small sample of agreements) to Google Sheets +8. **SMEs review and correct labels** quickly +9. **Ingest corrections back into Databricks** as the next golden dataset version +10. **Improve prompts** and repeat the loop + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +--- + ## Summary LLM models, prompt quality, and cost will change over time. That's why we built a repeatable process — not a one-off prompt — and validated it across two large real-world projects. From 47e463e400b03a0f90b9189fe7f8fd68369d3aa4 Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Mon, 13 Jul 2026 13:02:34 -0700 Subject: [PATCH 6/7] Sharpen the narrative: drop subtitle, tighten headings, trim the mantra refrain - Remove the orphaned subtitle line (Tyler r3572461161); the post now opens on the "LLM models change..." hook. - "That process is:" -> "This idea is:" under Hypothesis (frames the steps as the idea we set out to test). - Heading beats: "What We Achieved (With a Small Team)" -> "What We Achieved"; "Step-by-Step" -> "How the loop works"; "Workflow Overview" -> "The workflow, end to end". - Trim the "Fast Model -> Judge -> SME -> Ingest -> Iterate" refrain from 6 to 2 (kept at Hypothesis + Summary as a bookend); removes the dangling "Repeatable mantra (again):". No substantive copy reworded (token-multiset diff: only new heading words and "This idea" added; subtitle, mantra repeats, and old heading labels removed). Co-Authored-By: Claude Opus 4.8 (1M context) --- ...st-llm-human-in-the-loop-classification.md | 20 ++++--------------- 1 file changed, 4 insertions(+), 16 deletions(-) diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md index 88373ad..2db8f6e 100644 --- a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -12,15 +12,13 @@ team: ML Data Engineering author: anishk123 --- -*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools* - LLM models change. Prompt quality changes. Cost changes. We assumed that from day one. So instead of building a one-off system tied to a specific model or a single "perfect prompt," we focused on a **repeatable process** that we successfully applied to two large-scale projects (Slideshare and Scribd). That repeatable process is what we're sharing. --- -## What We Achieved (With a Small Team) +## What We Achieved This process enabled projects that normally require far more people and time: @@ -37,7 +35,7 @@ LLM models, prompt quality, and cost will change over time. The durable asset is ![Humans stay in the loop: the model proposes labels at scale, humans have the final say on the disagreements, and corrections grow a versioned golden set that feeds back into the model.](/post-images/2026-fast-llm-hitl/human-in-the-loop.svg) -That process is: +This idea is: 1. Use a fast model to generate candidate labels and short rationales at scale 2. Use a judge model to flag disagreements and ambiguous cases @@ -49,7 +47,7 @@ That process is: --- -## Step-by-Step +## How the loop works ### Step 1 — Standardize Inputs (Before You Touch Prompts) @@ -141,8 +139,6 @@ The judge is not a replacement for SMEs — it's a way to: - surface failure patterns faster - create measurable disagreement signals for iteration -> **Fast Model → Judge Model → SME Review → Ingest → Iterate** - ### Step 6 — SME Loop via Google Sheets (Focused Human Time) We exported: @@ -204,8 +200,6 @@ You should see: - SME correction rate drop over time - fewer surprise failures on the 1% validation slice -> **Fast Model → Judge Model → SME Review → Ingest → Iterate** - --- ## Metrics That Matter (Simple but Effective) @@ -242,10 +236,6 @@ The point is not "perfect metrics." The point is to make iteration measurable. 6. Ingest edits as a versioned golden dataset 7. Iterate prompts, validate on 1%, then scale up -Repeatable mantra (again): - -> **Fast Model → Judge Model → SME Review → Ingest → Iterate** - --- ## Key Learnings @@ -258,7 +248,7 @@ Repeatable mantra (again): --- -## Workflow Overview +## The workflow, end to end Here's the end-to-end loop we repeated: @@ -275,8 +265,6 @@ Here's the end-to-end loop we repeated: 9. **Ingest corrections back into Databricks** as the next golden dataset version 10. **Improve prompts** and repeat the loop -> **Fast Model → Judge Model → SME Review → Ingest → Iterate** - --- ## Summary From 1966641686ce9512b660a282100aa78f056c1637 Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Mon, 13 Jul 2026 14:51:21 -0700 Subject: [PATCH 7/7] Fix Featured Series to show the most-recent series posts (Cursor r3573323856) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Featured Series block sorted ascending and took the first two, so it always showed the two OLDEST `content-trust-series` posts (PhotoDNA + Dual-Embedding) and never surfaced newer entries. Add `| reverse` so it shows the two most recent series posts — consistent with the Featured Post logic just above, and with @rtyler's expectation that "the later post will appear on the home page." PhotoDNA keeps its `content-trust-series` tag (per @rtyler); it simply ages off the two-item display now that a newer series post exists. Co-Authored-By: Claude Opus 4.8 (1M context) --- _includes/featured-post-hero.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_includes/featured-post-hero.html b/_includes/featured-post-hero.html index de1dcaa..e5a0cea 100644 --- a/_includes/featured-post-hero.html +++ b/_includes/featured-post-hero.html @@ -71,7 +71,7 @@

Featured Series

    - {% assign featured-series = site.tags[site.featured_series] | sort: 'date' %} + {% assign featured-series = site.tags[site.featured_series] | sort: 'date' | reverse %} {% for post in featured-series limit: 2 %} {% include post-list-item.html %}