Add Workglow Eval: CLI harness for model evaluation on HuggingFace datasets#636
Merged
Conversation
…er HF datasets New examples/eval package (@workglow/eval): a general-purpose model-comparison harness modeled on the sec repo's extraction eval, built on Workglow storage. - dataset pull: fetch HuggingFace dataset splits into SqliteTabularStorage, via the datasets viewer API with a direct hub-file fallback (parquet via hyparquet incl. ClassLabel footer metadata, jsonl(.gz), ndjson, json) - run-classify: per-row StructuredGenerationTask workflow with an enum- constrained label schema, scored by accuracy against the gold label - run-similarity: per-row TextEmbeddingTask workflow over sentence pairs, cosine similarity scored by Pearson/Spearman against gold ratings - model ids resolve to inline ModelConfigs by shape (claude-*/gpt-*/gemini-*/ grok-*/org-name ONNX paths); embedding dimensions read from hub config.json - runs and per-row results persist to storage; report re-scores any stored run and ranks models by quality with latency tie-break - unit tests for scorers, parsers, model resolution, and storage round-trip, plus a live .e2e.test.ts covering pull -> run -> score end to end Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
Correctness: - reference dist-safe ./worker_hft.js in the HFT worker URL (bun build does not rewrite worker specifiers, so the built bin could never load the .ts path; a .js specifier resolves to the .ts source when running unbuilt) - map integer gold labels through the --labels override when the dataset declares no ClassLabel names (previously expected stayed "0"/"1" and the documented --labels path silently scored 0 accuracy) - deep-sanitize parquet rows: nested int64 BigInts (list/struct columns) and Date timestamps are now JSON-safe instead of crashing the pull - validate --limit/--offset as integers with clear errors (NaN limit used to wipe the stored split, store 0 rows, and report success) - run pull's delete+insert inside withTransaction, and make --offset pulls extend the split instead of destroying previously pulled rows - exclude NaN from correlation scoring (NaN passed the null filter, zeroing Pearson and silently corrupting Spearman ranks) and reject non-numeric gold scores per row in the similarity executor - record a per-row failure for unparseable stored rows instead of aborting the sweep, and parse each row once instead of once per model - route org/name model ids to local ONNX before cloud-prefix matching (gpt-omni/mini-omni previously misrouted to OpenAI) - validate columns in the classify executor (parity with similarity) and raise structured-generation maxTokens for long label sets - persist the datasets-server-resolved config name; warn when the viewer truncated cells; validate stored run kinds in report; fix sort comparators - add a package-local vitest config so `bun run test` works from the package directory (root config's setupFiles path broke it) Cleanup: lazy provider registration (read-only commands skip model-stack startup), shared HF auth-header helper, per-kind CLI flags only on their command, failed-outcome helper in the runner, drop unused @workglow/tasks dependency and dead exports/fields, remove redundant PK tuple spreads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
…NNX-ready) Bonsai 27B (released 2026-07, Qwen3.6-27B base) ships GGUF/MLX/AWQ conversions; no ONNX conversion exists upstream yet (onnx-community stops at the 8B sizes), so the GGUF is wired now and the ONNX id slots in with zero code once published. Test model registry (packages/test): - new samples registrar LlamaCppModelSamples.ts with Bonsai-27B-gguf Q1_0 and Ternary-Bonsai-27B-gguf Q2_0 records (LOCAL_LLAMACPP, json-mode via llama.cpp grammar constraints), alongside the existing ONNX registrar - ModelSamples.test.ts covers both registrars Eval CLI (examples/eval) gains a GGUF model path: - gguf:org/repo:Quant / gguf:hf:… / gguf:*.gguf ids resolve to a node-llama-cpp inline config (models_dir under the eval home, embedding mode on similarity evals) - LOCAL_LLAMACPP registered via a worker (worker_llamacpp.ts, built like the HFT worker); GGUF weights fetched by an explicit ModelDownloadTask step before the sweep since llama.cpp run-fns load from disk only - README documents the gguf id shape and a Bonsai-27B vs cloud example in both formats, noting the ONNX 27B is pending upstream - unit tests for gguf id resolution; ggufSmoke.e2e.test.ts runs a live hf: download + similarity eval through the llama.cpp provider Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
bun build does not rewrite worker new URL() specifiers, so the published bin (dist/workglow.js) kept the "./worker_hft.ts" literal while dist only contains worker_hft.js — every local-model run from the built CLI failed with ModuleNotFound. A ".js" specifier works in both modes: bun resolves it to worker_hft.ts when running from source, and the file exists as-is in dist. Same fix as examples/eval. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Coverage Report
File CoverageNo changed files found. |
…ing request URLs Dataset ids and model paths come from CLI flags; they are now checked against the hub id alphabet (name or org/name, no traversal or reserved URL characters) and rebuilt from URL-encoded segments before being interpolated into huggingface.co URLs — a crafted id like "org/name/../../resolve/main/x" can no longer steer the request path. Repo-relative file paths from tree listings get the same treatment (empty/./.. segments rejected, segments encoded). The datasets-server path validates too, keeping one error message for bad ids everywhere. WORKGLOW_EVAL_HOME is normalized to an absolute path; the env override is the invoking user's own machine-level setting (same trust domain as HOME) and remains supported as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
Generalizes the SEC repo's extraction eval to any HuggingFace dataset: each row runs a StructuredGenerationTask with an items-array schema built from the gold rows' fields, and candidates are aligned to gold rows by a key field (case/punctuation-insensitive) and scored on three axes, micro-averaged across the split: - score: field-level agreement over matched rows (ranking metric) - found: entity recall - prec: precision over distinct candidate rows (duplicates are over-production, not hallucination) The gold column accepts real arrays of objects (parquet struct/list, datasets-server JSON) or JSON strings. CLI: run-extract with --expected-column/--key-field/--fields/--instruction; report renders score/found/prec and re-scores stored row JSON using the key/field options persisted on the run. Kind dispatch is now table-driven (executors, ranking metrics, report columns) so a missed site fails the type check instead of silently falling into another kind's branch. Covered by scorer/prompt/aggregation unit tests (56 total now) and a live extractSmoke e2e (skipped without ANTHROPIC_API_KEY) that scored recall 1 / precision 1 against claude-haiku in verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new @workglow/eval example package: a CLI harness that pulls HuggingFace datasets into tabular storage, runs per-row Workglow workflows across multiple model backends (cloud + local ONNX/GGUF), and scores/aggregates results (accuracy, correlations, extraction scoring) with reporting and tests.
Changes:
- Introduces the
examples/evalpackage (CLI commands, storage schemas, HF dataset fetchers, eval runners, scorers, reporting). - Adds worker-based provider bootstrapping for local ONNX (HFT) and GGUF (llama.cpp) execution, plus model-id shape routing.
- Adds unit + (excluded-by-default) e2e tests for dataset file selection/sanitization, scoring math, storage round-trips, and full eval loops.
Reviewed changes
Copilot reviewed 43 out of 44 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/test/src/test/ai/ModelSamples.test.ts | Tests sample model registrars against an in-memory model repository. |
| packages/test/src/samples/LlamaCppModelSamples.ts | Adds sample GGUF model records (Bonsai 27B) to the global model repo. |
| examples/eval/vitest.config.ts | Vitest config for the eval example package (excludes *.e2e.test.ts). |
| examples/eval/tsconfig.json | TS project config + references for the eval example package. |
| examples/eval/src/workglow-eval.ts | Main CLI entrypoint wiring dataset/run/report commands and lazy init. |
| examples/eval/src/worker_llamacpp.ts | Worker entry for llama.cpp provider registration. |
| examples/eval/src/worker_hft.ts | Worker entry for HuggingFace Transformers worker registration + cache dir. |
| examples/eval/src/util.ts | Shared CLI formatting + flag parsing helpers. |
| examples/eval/src/test/storage.test.ts | Unit tests for in-memory storage schemas and keying behavior. |
| examples/eval/src/test/score.test.ts | Unit tests for label normalization, classification scoring, and correlation math. |
| examples/eval/src/test/models.test.ts | Unit tests for model-id routing to provider configs and model list parsing. |
| examples/eval/src/test/liveSimilarity.e2e.test.ts | Live e2e: pulls STS-B, runs similarity workflow, checks correlation. |
| examples/eval/src/test/ids.test.ts | Unit tests for HuggingFace repo/file path sanitization. |
| examples/eval/src/test/hubFiles.test.ts | Unit tests for hub-file split selection and dataset file parsing/sanitization. |
| examples/eval/src/test/ggufSmoke.e2e.test.ts | Live e2e: downloads GGUF and runs a similarity eval smoke check. |
| examples/eval/src/test/extractSmoke.e2e.test.ts | Live e2e: Anthropic-backed extraction eval (skipped without API key). |
| examples/eval/src/test/extraction.test.ts | Unit tests for extraction prompt building, parsing, scoring, and aggregation. |
| examples/eval/src/test/aggregate.test.ts | Unit tests for per-model aggregation, ranking, and NaN filtering. |
| examples/eval/src/storage.ts | Defines SQLite + in-memory tabular schemas for datasets/runs/results. |
| examples/eval/src/score/extraction.ts | Implements extraction alignment + micro-averaged scoring. |
| examples/eval/src/score/correlation.ts | Implements Pearson/Spearman correlation and fractional ranks. |
| examples/eval/src/score/classification.ts | Implements normalized-label accuracy scoring. |
| examples/eval/src/report/aggregate.ts | Aggregates stored per-row results into ranked per-model reports. |
| examples/eval/src/providers.ts | Registers tasks + providers defensively, wiring worker-based local providers. |
| examples/eval/src/models.ts | Maps model-id “shapes” to inline ModelConfig and handles downloads/dimensions. |
| examples/eval/src/hf/types.ts | Types for fetched dataset rows, label names, and fetch options. |
| examples/eval/src/hf/pullDataset.ts | Fetches datasets (viewer API with fallback) and persists to tabular storage. |
| examples/eval/src/hf/ids.ts | Sanitizers to prevent HF repo/file path traversal/host injection. |
| examples/eval/src/hf/hubFiles.ts | Hub fallback: tree listing, split file selection, downloads, parsing/parquet support. |
| examples/eval/src/hf/datasetsServer.ts | Viewer API client with pagination + ClassLabel extraction. |
| examples/eval/src/hf/auth.ts | Builds HF bearer-token headers (HF_TOKEN). |
| examples/eval/src/evals/types.ts | Shared eval runner types (columns/options/context/executor). |
| examples/eval/src/evals/similarity.ts | Similarity eval workflow (embedding + cosine similarity). |
| examples/eval/src/evals/runner.ts | Sweep runner: iterates models × rows, records successes/failures + latency. |
| examples/eval/src/evals/extract.ts | Extraction eval workflow (structured generation + JSON item array). |
| examples/eval/src/evals/classify.ts | Classification eval workflow (structured generation constrained to label enum). |
| examples/eval/src/config.ts | Eval home/config resolution and on-disk directory creation. |
| examples/eval/src/commands/run.ts | CLI run-* commands, column/model parsing, and sweep execution. |
| examples/eval/src/commands/report.ts | CLI run listing + report rendering (table/json). |
| examples/eval/src/commands/dataset.ts | CLI dataset pull/list/show commands. |
| examples/eval/README.md | Documentation for the eval harness usage, workflows, and storage layout. |
| examples/eval/package.json | New workspace package definition + build/test scripts + deps. |
| examples/cli/src/workglow.ts | Aligns HFT worker URL resolution to .js to work from source and dist. |
| bun.lock | Adds the new workspace package + new deps (hyparquet, compressors, fzstd, etc.). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+81
to
+84
| rows.sort((a, b) => a.row_index - b.row_index); | ||
| for (const row of rows.slice(0, Number(opts.limit))) { | ||
| console.log(row.data); | ||
| } |
Comment on lines
+113
to
+123
| const columns: ColumnOptions = { | ||
| textColumn: flags.textColumn ?? DEFAULT_TEXT_COLUMN[kind], | ||
| labelColumn: flags.labelColumn ?? "label", | ||
| labels: flags.labels ? flags.labels.split(",").map((l) => l.trim()) : undefined, | ||
| pairColumn: flags.pairColumn ?? "sentence2", | ||
| scoreColumn: flags.scoreColumn ?? "score", | ||
| expectedColumn: flags.expectedColumn ?? "expected", | ||
| keyField: flags.keyField ?? "name", | ||
| fields: flags.fields ? flags.fields.split(",").map((f) => f.trim()) : undefined, | ||
| instruction: flags.instruction, | ||
| }; |
Comment on lines
+11
to
+20
| export interface EvalConfig { | ||
| /** Directory holding the SQLite database and model cache. */ | ||
| readonly home: string; | ||
| /** SQLite database file with dataset rows, runs, and results. */ | ||
| readonly dbPath: string; | ||
| /** ONNX model cache directory for the HuggingFace Transformers worker. */ | ||
| readonly modelCache: string; | ||
| /** GGUF weights directory for the node-llama-cpp worker. */ | ||
| readonly ggufCache: string; | ||
| } |
Comment on lines
+70
to
+80
| function sanitizeValue(value: unknown): unknown { | ||
| if (typeof value === "bigint") return Number(value); | ||
| if (value instanceof Date) return value.toISOString(); | ||
| if (Array.isArray(value)) return value.map(sanitizeValue); | ||
| if (value !== null && typeof value === "object" && value.constructor === Object) { | ||
| const out: Record<string, unknown> = {}; | ||
| for (const [key, inner] of Object.entries(value)) out[key] = sanitizeValue(inner); | ||
| return out; | ||
| } | ||
| return value; | ||
| } |
- await worker registration in the HFT worker entries (eval + cli) so a failed SDK load rejects loudly instead of leaving a worker that looks started with no run-fns - share the GGUF cache path via ggufCacheDir() so EvalConfig.ggufCache and the model resolver cannot diverge - validate dataset show --limit with parseIntFlag (and wrap the action in the same clean error handling as its siblings) - drop empty names from --labels/--fields so stray commas cannot put into the label enum or the extraction field list - preserve int64 values beyond the safe-integer range as decimal strings instead of silently losing precision in Number() Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
Code-review fixes across the eval example: Scoring correctness: - scoreExtraction no longer scores the key field itself — a matched pair agrees on it by construction, so including it inflated every score toward 1 and let a key-only extraction score a perfect 1.0 - normalizeValue compares object/array field values by JSON content instead of collapsing them all to "[object Object]" - normalizeLabel is Unicode-aware; accented/CJK labels no longer all normalize to the same empty key - extract ranking falls back to entity recall when field agreement is NaN, so a model that found entities outranks one that found nothing - aggregateResults gates accuracy to classify and correlations to similarity runs Extraction robustness: - non-key schema fields are nullable — grammar-constrained local backends emit every declared property regardless of `required`, and null is an honest value for fields the text never states (prompt wording updated to match) - parseExpectedRows rejects array elements, not just non-objects - prototype-polluting field names (__proto__ etc.) are filtered from schema/scored fields; unsafe --key-field values are rejected - shared fenceText helper grows the prompt fence when the row text itself contains the delimiter (classify + extract) - explicit --fields hoists the schema build out of the per-row loop CLI/report hardening: - comma-list flags dedupe and error when passed but empty; --key-field must be non-empty; commander defaults that duplicated the ?? fallbacks moved into the option descriptions - report parses run.options only for extract runs and falls back to defaults on malformed JSON instead of crashing the report Cleanup: - hub id segments may start with "_" or "-" (both exist on the hub); removed the unreachable dot-only check - removed the dead EvalConfig.ggufCache field - LlamaCppModelSamples JSDoc no longer overstates where downloaded weights land Tests updated for the key-field exclusion (agreement counts drop the key), plus new coverage: Unicode labels, fence collisions, structured field values, prototype-pollution filtering, recall-fallback ranking, and underscore hub ids. Live-verified: LFM2.5-350M extracts the nested items schema cleanly (found=1, prec=1), claude-haiku e2e passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
A computed ["__proto__"] key in an object literal does create an own property, but JSON.parse is unambiguously prototype-safe and mirrors how a hostile gold row actually arrives from stored dataset JSON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4
sroussey
added a commit
that referenced
this pull request
Jul 15, 2026
## @workglow/ai ### Bug Fixes #### ai - key OpenAIShapedResponses tool-call accumulator on stable item id #### xai,openrouter - guard chunk.choices access and downshift strict schema ## workglow ### Features #### storage - add DuckDB tabular storage backend (@workglow/duckdb) (#635) ### Chores #### deps - update @cloudflare/workers-types to 5.x and tslog to 5.x ### Updated Dependencies - `tslog`: ^5.0.0 ## @workglow/storage ### Features #### storage - add DuckDB tabular storage backend (@workglow/duckdb) (#635) ## @workglow/test ### Features - add Workglow Eval: CLI harness for model evaluation on HuggingFace datasets (#636) #### storage - add DuckDB tabular storage backend (@workglow/duckdb) (#635) ### Bug Fixes #### node-llama-cpp,huggingface-transformers - concurrency + sequence-leak + bounded pipeline cache (#634) #### node-llama-cpp - eviction disposes embedding, broaden isVramError, retry embedding create - address review feedback on VRAM/LRU/sequence handling #### huggingface-inference - guard chunk.choices[0] in streaming run-fns #### ai - key OpenAIShapedResponses tool-call accumulator on stable item id #### xai,openrouter - guard chunk.choices access and downshift strict schema ### Tests #### huggingface-transformers - regressions for H1 (refcount race) and H2 (AbortController key) ### Chores - update deps #### deps - update @cloudflare/workers-types to 5.x and tslog to 5.x ### Updated Dependencies - `@aws-sdk/client-sqs`: ^3.1088.0 - `@cloudflare/workers-types`: ^5.20260715.1 - `miniflare`: ^4.20260710.0 ## @workglow/openai ### Bug Fixes #### xai,openrouter - guard chunk.choices access and downshift strict schema ## @workglow/node-llama-cpp ### Bug Fixes #### node-llama-cpp - address structured generation review feedback - fresh context per structured generation - eviction disposes embedding, broaden isVramError, retry embedding create - address review feedback on VRAM/LRU/sequence handling - close sequence-reclaim race and auto-evict on VRAM pressure #### node-llama-cpp,huggingface-transformers - concurrency + sequence-leak + bounded pipeline cache (#634) ### Chores #### node-llama-cpp - clarify allocation and sequence error matching constants ## @workglow/aws ### Chores - update deps ### Updated Dependencies - `@aws-sdk/client-sqs`: ^3.1088.0 ## @workglow/duckdb ### Features #### storage - add DuckDB tabular storage backend (@workglow/duckdb) (#635) ## @workglow/xai ### Bug Fixes #### xai,openrouter - guard chunk.choices access and downshift strict schema ## @workglow/cloudflare ### Chores #### deps - update @cloudflare/workers-types to 5.x and tslog to 5.x ### Updated Dependencies - `@cloudflare/workers-types`: ^5.20260715.1 ## @workglow/huggingface-transformers ### Bug Fixes #### huggingface-transformers - key model AbortControllers per-controller to survive concurrent loads - hold pipeline refcount before getPipeline lookup #### node-llama-cpp,huggingface-transformers - concurrency + sequence-leak + bounded pipeline cache (#634) ## @workglow/openrouter ### Bug Fixes #### xai,openrouter - guard chunk.choices access and downshift strict schema ## @workglow/huggingface-inference ### Bug Fixes #### huggingface-inference - guard chunk.choices[0] in streaming run-fns ## @workglow/eval ### Features - add Workglow Eval: CLI harness for model evaluation on HuggingFace datasets (#636) ## @workglow/cli ### Features - add Workglow Eval: CLI harness for model evaluation on HuggingFace datasets (#636) ## @workglow/web ### Chores - update deps ### Updated Dependencies - `react-resizable-panels`: ^4.12.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a comprehensive command-line evaluation framework (
@workglow/eval) for comparing AI models on Workglow task workflows. The harness pulls HuggingFace datasets into SQLite storage, runs each row through eval workflows across multiple models, and scores the results with quality and latency metrics.Key Changes
Dataset integration: Pull HuggingFace dataset splits via the datasets-server API (with fallback to direct hub file download). Supports parquet, JSONL, and JSON formats with automatic split detection.
Eval workflows: Two built-in eval kinds:
run-classify: StructuredGenerationTask with enum label output; scored by accuracy (case/punctuation-insensitive)run-similarity: TextEmbeddingTask for sentence pairs; scored by Pearson/Spearman correlation against gold scoresStorage layer: Three tabular schemas (DatasetMeta, DatasetRow, EvalRun, EvalResult) persisted to SQLite, enabling repeatable runs, result inspection, and re-scoring without re-execution.
Model routing: Inline ModelConfig resolution for any model shape without a registry:
gguf:…→ node-llama-cpp (local GGUF)org/name→ HuggingFace Transformers ONNX (local)claude-*,gpt-*,gemini-*,grok-*→ cloud providers (Anthropic, OpenAI, Google, xAI)Provider registration: Defensive registration of all AI providers (cloud + local) with environment-based credential loading.
Scoring & reporting: Per-model aggregation with accuracy/correlation metrics, latency tracking, and ranked output (table or JSON).
Test coverage: Unit tests for file selection, label normalization, correlation math, and storage round-trips; end-to-end tests for live dataset pulls and embedding workflows.
Implementation Details
HuggingFace file handling:
scoreFileForSplit()ranks data files by match quality (exact name > shard prefix > directory match) to select the best split files. Supports gzip decompression and parquet metadata extraction via hyparquet.Row execution:
runSweep()parses stored rows once, then runs each through the eval workflow per model. Failures are recorded (not aborting) so one bad model or row doesn't lose the rest of the run.Label mapping: Integer-coded labels (HuggingFace ClassLabel) are mapped through dataset-declared names or explicit
--labelsflag; string labels pass through unchanged.Correlation math: Pearson and Spearman implementations handle degenerate cases (constant series, too few points) by returning NaN.
CLI structure: Commander.js with subcommands for dataset pull/list, eval runs, and result reporting. Config stored under
~/.workglow/eval(overridable viaWORKGLOW_EVAL_HOME).Files Added
workglow-eval.ts,config.ts,providers.ts,models.ts,storage.ts,util.tshf/types.ts,hf/auth.ts,hf/hubFiles.ts,hf/datasetsServer.ts,hf/pullDataset.tsevals/types.ts,evals/runner.ts,evals/classify.ts,evals/similarity.tsscore/classification.ts,score/correlation.tsreport/aggregate.tscommands/dataset.ts,commands/run.ts,commands/report.tstest/hubFiles.test.ts,test/models.test.ts,test/score.test.ts,test/aggregate.test.ts,test/storage.test.ts,test/liveSimilarity.e2e.test.ts, `test/ggufSmhttps://claude.ai/code/session_01LiEn5HpZm9sVkmF38b1Js4