English | 简体中文
LLM API evaluation, model fingerprinting, and token billing audits—with anonymized real-world results.
A practical evaluation framework for AI procurement, third-party LLM API onboarding, and vendor management. It covers pre-onboarding evaluation → post-onboarding checks → ongoing quality monitoring.
Detailed protocols, question banks, and case-study notes are currently primarily in Chinese. This README provides an English entry point.
| Area | Questions to investigate | Materials |
|---|---|---|
| Capability and model consistency | Capability differences across providers, behavioral drift, and routing changes | 60-question comparison bank, behavioral and probability fingerprint protocols |
| Token billing | Unnecessary output, accounting differences, duplicate charges, and cost fluctuations | Dedicated token billing audit protocol |
| Service reliability | Latency, rate limits, peak-load behavior, and changes over time | Retesting and monitoring procedures |
- Read the overall testing plan and define your evaluation stage and objectives.
- Choose the vendor comparison bank, behavioral fingerprint protocol, probability fingerprint protocol, or token billing audit.
- Freeze the dataset version, parameters, model identifiers, and scoring rules. Retain original requests, responses, and billing evidence.
- Document findings and human review using the report template and billing record fields.
| Path | Contents |
|---|---|
| docs/ | Overall plan, four detailed protocols, references, and known limitations |
| examples/ | Maintained reference scripts, original behavioral attachments, and offline tests |
| data/probability/ | 743 original prompts and two example ground-truth whitelist entries |
| case-studies/ | Anonymized internal evaluation results and review records |
| templates/ | Blank report and billing record templates |
| LICENSES/ | Upstream behavioral fingerprinting copyright and license notice |
Real API testing completed internally · Anonymized results published · Further improvements underway
The framework has been used in an internal enterprise evaluation of model onboarding and acceptance. The published anonymized case study covers 6 test configurations, 4,758 primary test records, and 28 worksheets, including scores, usage measurements, and anomaly reviews.
- Test coverage: Each configuration includes 21 behavioral probes, 743 probability-bank calls, and 29 token-usage tests. Selected configurations also have tool-use and LLMmap fingerprint records.
- Recorded results: Behavioral scores, request status, token usage, latency, anomalies, and retest conclusions. Company, provider, and internal identifiers have been anonymized.
- Acceptance progress: Initial evaluations and reviews are complete. Some configurations remain conditionally accepted or await end-to-end validation; consult the final review records for their scope.
- Ongoing work: Repeated probability sampling, consistent tool testing across configurations, threshold calibration, and resolution of accounting anomalies.
Public code scope: The reference scoring and collection flows now include bug fixes, an explicitly configured Chat Completions adapter, and offline regression tests. Collection and scoring share an 11-question subset. Full answer whitelists, the remaining rubrics, and the original behavioral scripts still require completion or adaptation; the updated adapter has not been tested against live services. See the code notes for implementation status and the case study for the internal execution records.
Interpret results alongside the known limitations. Fingerprints, capability scores, and token-count differences are signals for further investigation; they do not independently prove model identity or improper billing. See the source inventory for included materials.
Python 3.10+ is recommended. The updated vendor/probability flows and offline tests use only the standard library.
python -m unittest discover -s tests -v
python examples/scoring_pipeline.py --demoThe demo is explicitly labeled and uses simulated inputs. For real collection, configure LLM_API_URL, LLM_API_KEY, and LLM_MODEL; configure the equivalent JUDGE_ variables for the judge, then follow the run instructions. Real API commands can incur costs. Missing rubrics, GT, or dimensions are not replaced with fabricated results. Probability scoring now uses normalized whole-answer matching and Wilson intervals; rebuild baselines before comparing with earlier scores.
The historical evaluation results include per-question metrics, acceptance decisions, and anomaly reviews across 28 worksheets and 6 test configurations. Identifying information has been anonymized, and original free-text payloads have been withheld from the public export.
The results document the author's internal testing. They do not establish that the public reference scripts reproduce the entire internal workflow end to end; the documented code limitations still apply.
Issues and pull requests are welcome for completing rubrics, fixing reference scripts, and improving baselines and statistical methods. When changing prompts, ground truth, parameters, or scoring rules, record the version and its effect on existing baselines. Use anonymized examples and keep API keys and unauthorized business data out of submissions.
Framework author: Mao Youqi (毛友琦) · maoyuki31-dev
Original project documentation, question banks, and code are released under the MIT License. Third-party research, probes, and derived material retain their respective rights and licenses; see third-party notices. Research paper PDFs are not redistributed in this repository.