Repository navigation
Measure Jev-relative quality latency and billed cost on frozen requests #7
Description
Gap and evidence
Layev validation has no authorized Jev run. API-shape compatibility and synthetic fixture scores do not establish Jev-relative quality, latency or cost. Jev version, access, data permission and paid budget are explicit prerequisites rather than assumptions.
Depends on representative-quality protocol #5 and native/context proof #4; the current interface gate is #6.
Acceptance
- Use an approved, frozen request set with identical state/questions, response schema, scoring and declared budget for Layev and an identified Jev model/version.
- Obtain existing authorization for Jev access, data transfer and paid use before calls; if unavailable, record the exact blocker and leave Jev parity unknown.
- Record permitted request/response hashes, sample sizes, per-slice quality, calibration, latency distributions and actual billed cost, with hardware and software identities.
- Report limitations and failures; do not infer proprietary architecture, RLCD implementation or equal quality from the public interface.
Activity
CompleteDotTech commentedon Sep 28, 2026
#7 preparation only: signed PR #33 adds an offline paired scorer and CLI for frozen test transcripts. It verifies identical state/questions across Layev and a version-pinned Jev request, checks common Choice/Score/Noul distributions, and applies the same quality metrics and slices. It never contacts either service and marks latency, billed cost, data-transfer authorization and Jev parity unverified.
Local staged source audit, compilation and critical Ruff passed; Windows full suite: 816 passed, 16 skipped, including an end-to-end synthetic transcript test. The five required hosted jobs in run 36380435279 are still queued. PR #33 is not merged yet.
The actual #7 comparison remains dependent on the licensed/reviewed frozen data and explicit TypeSafe transfer permission, account access, and #4/#5 native/quality evidence. No paid call or representative quality result is claimed.
CompleteDotTech commentedon Sep 28, 2026
After #34 advanced protected main, PR #33 was refreshed by signed merge commit
2a8a8570d7c996268233cebdffb1c45a6318e876against main821ffea. Combined local source passed 818 tests/16 skips, compile, indexed audit and critical Ruff. PR head/base read back to those SHAs; five new required checks are queued. This offline scorer preparation makes no Jev remote call or comparison claim; #7 stays open.CompleteDotTech commentedon Sep 28, 2026
Offline paired scorer PR #33 passed all five protected checks on signed head
2a8a8570d7c996268233cebdffb1c45a6318e876and merged normally at verifieda6926f7b4585e840b36f1d9157c3af3907210ef5. Primary main is clean/synchronized 0/0. Exact-merge postmerge run36384521208is queued. This delivers scoring software only; no authorized live Jev request, billed cost, representative quality or parity result was measured. #7 remains open.CompleteDotTech commentedon Sep 28, 2026
TypeSafe's official model list checked 2026-09-28 identifies one current versioned Jev model,
jev-1.13.0;jev-latestandjev-previewboth resolve to it at this read. The official API reference specifiesPOST https://api.typesafe.ai/v1/systemone. The eventual frozen paired comparison should requestjev-1.13.0and retain the response model ID and billed usage rather than relying on a moving alias.No
TYPESAFE_API_KEYis present in the current process, and the real agentic deployment episode source, data-use review, explicit TypeSafe transfer permission and frozen representative split have not been provided. No paid request or Jev-relative result has been produced. The user approved TypeSafe spend without a cap, but that does not resolve those access/data prerequisites. Keep #7 open.Sources: https://docs.typesafe.ai/models and https://docs.typesafe.ai/api