Optional EvalPort adapter for TextGenerationRequest/Response

[adhabnr-ux]

Hi — I maintain EvalPort, an open, framework-agnostic JSON Schema spec for portable LLM evaluation data (TestCase/Suite/Grader/Result/ResultSet), with Python/TS SDKs that validate against the schemas. Early-stage project (~35 shipped adapters, no big star count) — not overselling traction, just proposing a fit.

No CONTRIBUTING.md in this repo, so filing as an issue as the safe default before writing any code.

I read the real dataclasses in deepinfra/types/text_generation/ rather than guessing:

# request.py
@dataclass
class TextGenerationRequest:
    input: str
    stream: Optional[bool] = None
    max_new_tokens: Optional[int] = None
    temperature: Optional[float] = None
    top_p: Optional[float] = None
    stop: Optional[List[str]] = None
    ...

# response.py
@dataclass
class GeneratedText:
    generated_text: str

@dataclass
class TextGenerationResponse:
    inference_status: InferenceStatus
    results: List[GeneratedText]
    num_tokens: int
    num_input_tokens: int

This is about as direct a fit as it gets: TextGenerationRequest.input → EvalPort TestCase.input, TextGenerationRequest.stop/temperature/max_new_tokens → the test case's optional provider overrides, and TextGenerationResponse.results[].generated_text → the EvalPort Result.output, with num_tokens/num_input_tokens carried into metadata for cost/usage tracking.

Proposed as a standalone, optional adapter, zero footprint on this repo:

from deepinfra.clients.deepinfra import DeepInfra
from deepinfra_openeval_adapter import to_openeval, from_openeval
from openeval.validate import validate_suite, validate_result_set

client = DeepInfra(api_key=...)
req = TextGenerationRequest(input="Explain photosynthesis.")
resp = client.text_generation.generate(model="...", request=req)

test_case = to_openeval(req, test_case_id="tc1")
result = from_openeval  # (or the reverse: response_to_openeval(resp, test_case_id="tc1"))

Concrete use case: someone using deepinfra-python for text generation across many models wanting to snapshot prompt/response pairs as a portable regression suite instead of a one-off script's output.

Happy to build this as a standalone package in EvalPort's own adapters/ directory (zero footprint here), or as a PR into this repo alongside deepinfra/types/ if you'd rather — your call, no pressure either way.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

Posting as Sahi, independent contributor (not affiliated with DeepInfra).