Add PSAI Auto Router submission

[subhajeet-sapient]

Add SapientAI Auto Router submission

Router: SapientAI Auto Router
Organisation: Publicis Sapient — proprietary, closed-source
Author: Subhajeet Dey (@subhajeet-sapient)
Split evaluated: full (8,400 queries) + robustness (420 queries)

---

Summary

SapientAI Auto Router is Publicis Sapient's production LLM router. It is a
proprietary, privately hosted service: the routing decision is produced by an
internal endpoint, and neither the classifier weights nor the serving code are
included in this pull request. This submission therefore contains prediction
artefacts only — we want to include this submission to the official leaderboard.

All five models in the pool already have entries in model_cost/model_cost.json
and universal_model_names.py, so this PR touches nothing outside its own three
files.

Results

Metric	Value
Arena Score	75.49
Accuracy	77.99%
Cost	$0.4345 / 1K queries ($3.65 total)
Robustness	73.33

Methodology

1. Complexity and Boosting classifier

The routing signal is a DeBERTa-v3-base sequence classifier fine-tuned on
Publicis Sapient's own production traffic — real customer prompts accumulated
across live engagements over an extended period, with outcome labels derived from
observed model sufficiency in production.

The classifier emits a single scalar complexity score in [0, 1] per prompt

2. Cost-versus-complexity analysis

Each candidate model is profiled offline on two axes: realised unit cost
(blended input/output token price at production traffic ratios) and capability
ceiling (the complexity level beyond which the model's answer quality degrades).

This produces a cost/capability frontier. Models on the frontier form an ordered
ladder of tiers; models dominated on both axes are excluded from serving. The
router's task reduces to selecting the cheapest tier whose capability ceiling
clears the query's predicted complexity.

3. Proprietary tier boosting

A pure threshold cut on the complexity score under-serves queries near a tier
boundary, where classifier uncertainty is highest and the cost of an
under-powered answer is asymmetric with the cost of a stronger model.

The tier boosting layer promotes a query above its base tier when boundary
conditions warrant it, trading a bounded cost increase for the reduction in
failure probability. Boosting is applied per-query at decision time based on
routing and token-length signals, and domain/task priors.

4. Serving pool

Five models, spanning roughly a 10× spread in output-token price:

Tier	Model
Cheap	gpt-5-nano
Mid	google/gemini-3.1-flash-lite
Strong	gemini-3-flash-preview
Elite	deepseek/deepseek-v4-flash
Premium	Qwen/Qwen3-Coder-Next

Reproducibility

The routing endpoint is private infrastructure and cannot be called by the
evaluation workflow. The submitted generated_result fields are the verbatim
outputs from the run described above, and the evaluation harness recomputes
accuracy and cost from them using the repository's own scorers and
model_cost.json prices.

Compliance

The complexity classifier was trained exclusively on Publicis Sapient production
data. No RouterArena query, label, or split file was used to train or fit the
classifier.

Files

• router_inference/config/PSAI-Auto-Router.json
• router_inference/predictions/PSAI-Auto-Router.json
• router_inference/predictions/PSAI-Auto-Router-robustness.json

[subhajeet-sapient]

/evaluate

[github-actions]

Router Evaluation Results

Router: PSAI-Auto-Router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7541
Accuracy	77.89%
Total Cost	$3.649615
Avg Cost per Query	$0.000434
Avg Cost per 1K Queries	$0.4345
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7333

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.0593
Opt.Cost (Cost Efficiency)	0.1999
Opt.Acc (Accuracy vs Optimal)	0.9200

---

Evaluation completed by RouterArena automated workflow

[subhajeet-sapient]

@yl231 Can we get this on the leaderboard ?