feat(evaluators): Add evaluator for LLM

[wrisa]

Summary

• Added a new galileo.llm evaluator that invokes Galileo's LLM-as-judge scorer API directly from Agent Control. The evaluator follows the same architecture as the existing galileo.luna evaluator — it uses an internal JWT for authentication, supports all the same comparison operators and threshold types, and is registered as a plugin entry point. Unlike Luna, galileo.llm requires both a runtime step and authenticated execution context (supplied via evaluate_with_extensions), as Orbit uses execution_context to fetch LLM credentials with no legacy inputs-only fallback.

Intentional divergence from luna evaluator introduced in this PR

• Feature flag gates the entire evaluator (is_available()), not just runtime records

  Luna's GALILEO_FEATURE_FLAG_SCORER_INVOKE_RUNTIME flag only suppresses the structured record and execution_context fields — the evaluator still loads and invokes the scorer using legacy inputs.

  The LLM evaluator has no legacy fallback: Orbit requires execution_context to fetch LLM credentials, so there is no inputs-only path that can succeed. GALILEO_FEATURE_FLAG_LLM_INVOKE_RUNTIME=enabled gates is_available() itself — when the flag is absent the evaluator is not loaded at all, rather than silently degrading.
  Note: GALILEO_FEATURE_FLAG_LLM_INVOKE_RUNTIME is designed as a deploy-time feature gate (to not ship the evaluator to stacks that don't have Orbit execution_context support yet), not a runtime toggle. To unload galileo.llm, you need a pod restart with the flag absent or not "enabled".

• step and execution_context are required, not optional

Luna proceeds with inputs-only when step=None or execution_context=None. The LLM evaluator fails fast in _evaluate() with a clear error result for either condition, and client.invoke() raises ValueError if either is absent. The engine always supplies both via evaluate_with_extensions, so this only affects out-of-engine direct usage.

• llm/config.py — ScorerInvokeConfig is leaner than Luna's

  Luna's ScorerInvokeConfig carries two legacy fields for backward compatibility with existing Orbit callers:

threshold: float | None = None        # legacy Orbit field
score_threshold: float | None = None  # legacy Orbit field

The LLM evaluator was written after these were identified as legacy, so galileo.llm's ScorerInvokeConfig omits them entirely. galileo.luna retains them to avoid breaking existing deployments.

• llm/client.py — invoke() takes config: ScorerInvokeConfig | None, not ScorerInvokeConfig | JSONObject | None

Luna's invoke() still accepts a raw dict for config (with a model_validate compat branch) to support callers that were passing dicts before ScorerInvokeConfig existed. The LLM evaluator has no such callers, so it accepts only the typed model.

• Made scorer_version_id mandatory end-to-end: required on LlmEvaluatorConfig, ScorerInvokeRequest, and client.invoke()

• Removed ScorerInvokeInputs and inputs from ScorerInvokeRequest; made execution_context required on ScorerInvokeRequest

Scope

User-facing / API changes:

• New evaluator type galileo.llm available when agent-control-evaluator-galileo is installed. Configured with scorer_id, threshold, operator, scorer_version_id and optional scorer_label, scorer_config, payload_field, timeout_ms.
• from agent_control.evaluators import LlmEvaluator, LlmEvaluatorConfig now available with galileo extras.
• New entry point galileo.llm registered in agent_control.evaluators.

Internal changes:

• llm/client.py — GalileoLLMClient with connection pooling, JWT internal auth, CA file support, and the same connection tuning env vars as Luna (reusing GALILEO_LUNA_* env var names).
• llm/config.py — LlmEvaluatorConfig with identical schema to LunaEvaluatorConfig.
• llm/evaluator.py — LlmEvaluator with evaluate, evaluate_with_context, evaluate_with_extensions, and _execution_context_from_extensions.
• evaluators/__init__.py in the SDK updated to optionally re-export LLM types.
• llm/config.py — GALILEO_FEATURE_FLAG_LLM_INVOKE_RUNTIME env var and llm_invoke_runtime_enabled() function gate is_available().
• llm/evaluator.py — LlmEvaluator with evaluate_with_extensions as the primary entry point (engine-facing); evaluate and evaluate_with_context fail fast since step and execution_context are both required. is_available() checks the feature flag in addition to import availability.

Out of scope:

• Shared base class / deduplication of Luna and LLM code (intentionally kept as independent implementations to avoid coupling).

Risk and Rollout

Risk level: low

• The LLM evaluator is opt-in — it only activates when a control is explicitly configured with type: galileo.llm. No existing Luna controls are affected.
• The new entry point is registered alongside the existing galileo.luna entry point in the same package; if the package isn't installed the import silently falls back.

Rollback plan:

• Remove or disable any controls using type: galileo.llm. No schema migrations or persistent state changes involved.

Testing

• Added or updated automated tests
• test_llm_evaluator.py — integration-style tests for execution context, extensions, org ID validation, and feature flag (GALILEO_FEATURE_FLAG_LLM_INVOKE_RUNTIME) gating is_available().
• test_llm_coverage_gaps.py — 94 tests covering all utility helpers, operator branches, payload routing, connection config validation, JWT auth, CA file fallback, client pool rotation, HTTP/request error propagation, config threshold validation, step/execution_context required guards, and factory-based record construction. Brings llm/client.py to 97%, llm/evaluator.py to 97%, llm/config.py to 98% patch coverage.
• Ran make check
• lint and typecheck pass.
• Manually verified behavior
• scorer invocation tested against a live Galileo environment via ace-demo and setup_controls.py with control configured with a galileo.llm prompt injection scorer.

GALILEO_FEATURE_FLAG_LLM_EVALUATOR=enabled - evaluatorgalileo.llm found

GALILEO_FEATURE_FLAG_SCORER_INVOKE_RUNTIME=enabled - orbit's scorer_runtime enabled
The control with LLM evaluator is triggered.

GALILEO_FEATURE_FLAG_LLM_EVALUATOR=enabled - evaluator galileo.llm found

GALILEO_FEATURE_FLAG_SCORER_INVOKE_RUNTIME=disabled - orbit's scorer_runtime disabled

Control Span is sent with output:
{
"action":"deny"
"matched":false
"confidence":0
"error_message":"Evaluation failed due to an internal evaluator error."
}

GALILEO_FEATURE_FLAG_LLM_EVALUATOR=disabled - evaluator galileo.llm not found

Agent Control error:

  {                                                                                                                                                                                                              
    "message": "Evaluator error for control 'banking-multilevel-deny-trace-sensitive-final-llm' (evaluator: galileo.llm): ValueError: Evaluator 'galileo.llm' not found. Available evaluators: json, list, regex,
   sql, galileo.luna",                                                                                                                                                                                           
    "exc_info": "Traceback (most recent call last):\n  File \".../agent_control_engine/core.py\", line 321, in _evaluate_leaf\n    evaluator = get_evaluator_instance(evaluator_spec)\n  File                    
  \".../agent_control_evaluators/_factory.py\", line 85, in get_evaluator_instance\n    raise ValueError(...)\nValueError: Evaluator 'galileo.llm' not found...",                                                
    "level": "ERROR",                                                                                                                                                                                            
    "logger": "agent_control_engine.core",                                                                                                                                                                       
    "filename": "core.py",                                                                                                                                                                                       
    "lineno": "342"                                                                                                                                                                                              
  } 

Control Span is sent with output:
{
"action":"deny"
"matched":false
"confidence":0
"error_message":"Evaluation failed due to an internal evaluator error."
}

Checklist

• Linked issue/spec
• SAO-16858
• evaluators/contrib/README.md, sdks/python/ARCHITECTURE.md, SDK __init__.py docstring updated to reference galileo.llm. SDK __init__.py and GalileoLLMClient env var docstring updated to document the GALILEO_FEATURE_FLAG_LLM_INVOKE_RUNTIME requirement.
• Included any required follow-up tasks
• LLM and Luna share ~80% identical code; the remaining divergence is intentional (feature flag scope, required vs optional step/execution_context, leaner ScorerInvokeConfig). A future refactor to extract a shared base (_shared/) is tracked separately and should account for these differences.

[codecov]

Codecov Report

❌ Patch coverage is 94.27481% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../src/agent_control_evaluator_galileo/llm/client.py	95.88%	11 Missing ⚠️
...c/agent_control_evaluator_galileo/llm/evaluator.py	94.44%	11 Missing ⚠️
...eo/src/agent_control_evaluator_galileo/__init__.py	25.00%	6 Missing ⚠️
.../src/agent_control_evaluator_galileo/llm/config.py	97.61%	1 Missing ⚠️
...ks/python/src/agent_control/evaluators/__init__.py	80.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[jasmine-ab-tea]

🤖 This review was generated by the Astra agent (claude-opus-5-5). It may contain mistakes.

Verdict: request_changes — The new engine skip path lets deny controls fail open for any unavailable evaluator, and the PR description does not mention this change.

General Comments

• 🟠 major (design): The PR description has gaps and does not match the code.
• It does not mention the change to engine/core.py or _discovery.py. That change alters fail-closed semantics for every evaluator, not only galileo.llm.
• "Risk level: low" and "No existing Luna controls are affected" are not accurate after this change.
• The description names the flag GALILEO_FEATURE_FLAG_LLM_INVOKE_RUNTIME and the function llm_invoke_runtime_enabled(). The code uses GALILEO_FEATURE_FLAG_LLM_EVALUATOR and llm_evaluator_enabled().
• The "disabled" test evidence shows a "not found" error. Commit f9f5643 replaced that behavior with a silent skip, so that evidence is outdated.
  Please update the description so the human reviewer can approve the engine semantics change on purpose.
• 🟡 minor (question): Do you want a disabled galileo.llm deny control to block requests or to let them through? A middle option is to keep the error result but tag it, for example with error_type=evaluator_disabled. Consumers can then filter the span without the engine failing open. Please confirm the intended behavior with the engine owners.

Follow-ups

Suggested follow-up work that could be tracked as Jira tickets:

• evaluators/contrib/galileo/src/agent_control_evaluator_galileo/llm/client.py:1-621: About 80% of llm/client.py is the same as luna/client.py, including JWT signing, pooling, and env parsing. The LLM client also reuses the GALILEO_LUNA_* env var names. Extract a shared _shared/ transport module, as the description already plans.