perf(powerbi): M-Query performance improvements - #70
Open
MrBlack1995 wants to merge 38 commits into
Open
Conversation
Adds a prompt-optimization stack built on MLflow GenAI + GEPA (reflective prompt evolution) that works end-to-end inside Kasal: Template optimization (Configuration -> Prompts): - Six generation templates (detect_intent, generate_agent, generate_task, generate_crew, generate_crew_plan, generate_job_name) get a per-template Optimize action that runs GEPA against real usage data mined from the LLM interaction logs, with junk filtering and format+correctness scoring. - Single consolidated "Prompts" surface; optimization opens in a scoped dialog next to each template. Crew optimization (crew catalog -> Optimize Prompts): - Full crew-in-the-loop GEPA: every metric call executes the crew for real and judges score the actual deliverable. Agent role/goal/backstory and task description/expected_output are evolved together via a labeled crew-document serialization with a strict parser (malformed candidates score 0 without spending an execution). - Hard execution cap: the budget the user picks is a promise; the predict_fn refuses executions beyond the cap even when GEPA evaluates candidates in parallel. Stop button cancels between iterations. Human judgment loop (all inside Kasal, persisted as MLflow assessments): - Grade past evaluation answers 0-10 with per-judge attribution, comments, and ground-truth Expectations (log_feedback + log_expectation). - Create custom LLM judges (make_judge) in-app; judges live in a library and are assigned per crew via a crew-scoped registered copy; built-in graded 0-10 quality judge always on. - Harvested crew thumbs and human assessments feed both the judge rubric and the GEPA reflection objective, so non-expert feedback autonomously steers prompt evolution. Infrastructure: - MLflow registry/tracking handling that respects the launch MLFLOW_TRACKING_URI (preserved as KASAL_LAUNCH_MLFLOW_TRACKING_URI before main.py forces "databricks"), local OSS registry gated behind MCP_SERVER_ENABLED, databricks-uc otherwise. - Cross-loop safety: worker threads reach the app DB only via run_coroutine_threadsafe onto the main loop with UserContext re-established; crew execution polling uses fresh sessions. - Autolog import-hook deadlock avoidance during optimization spans; reflection-model preflight so a dead provider fails fast instead of burning budget. - New dependency: gepa (required by MLflow's GepaPromptOptimizer). uv.lock churn beyond the gepa addition is uv re-canonicalization only; no other package versions changed. 25 unit tests cover doc serialization round-trips, judge value grading, budget accounting, registry gating, and apply/cancel flows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…make human feedback reach the mutator
Live diagnosis of flat runs (score X -> X, "baseline won") on a real crew
with 12 human grade-0 notes exposed four compounding causes:
1. Budget burned re-measuring the baseline. GEPA re-evaluates the SAME
candidate doc repeatedly (upfront smoke test, baseline valset pass, and
a fresh reflective-minibatch pass every iteration). Each re-run cost a
real crew execution, so a 4-execution run bought exactly ONE distinct
candidate (observed: total_metric_calls=7, candidates=1). Fix: cache
deliverables by candidate-doc hash — each DISTINCT candidate costs one
execution; re-evaluations are free and post-cap re-evals of the
baseline stay truthful.
2. Judge noise made acceptance a coin flip. The same prompts drew grades
of 0.0 and then 4/10 two minutes apart; the one candidate that DID
incorporate the human requirements lost to a lucky baseline draw.
Fix: cache the judge verdict by deliverable hash — within a run,
identical outputs always score identically, so candidate-vs-baseline
comparisons are stable.
3. Reflection was blind to WHY candidates failed. Scorers returned bare
floats, and MLflow only forwards textual rationales to GEPA's
reflective dataset from Feedback objects — so the mutator saw "0.0"
but never "wrong region, rentals instead of sales". The judge was even
instructed to reply with only the number. Fix: the judge now critiques
first (quoting the violated requirement) and grades on the last line;
output_correct returns Feedback(value, rationale) including registered
judges' rationales; aggregation unwraps Feedback values. Harvested
human Expectations additionally ride the train row's expectations
channel, which the reflective dataset surfaces explicitly.
4. Invisible feedback usage + misleading timestamps. Users could not see
that their grades were harvested ("we are not using the grading"), and
naive-UTC created_at rendered a 01:20 local run as "11:20 PM". Fix:
runs now report human_feedback_count ("guided by N human notes" chip)
and candidates_tried ("N variants tried" chip); timestamps are
timezone-aware so browsers render local time. Harvest also iterates
traces oldest-first so the keep-last-12 slice keeps the NEWEST notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…single-example datasets
A live run WITH the previous fixes still returned "baseline won" while
producing a candidate that fully incorporated the human requirements.
proposals.json showed why: minibatch score 0.9 vs 0.9 — a TIE, rejected
by GEPA's strict-improvement acceptance. Three compounding causes, each
verified against gepa 0.1.4 source and live A/B judge experiments:
1. No gradient: the judge graded EVERY real deliverable 0/10. Feeding
the raw harvest ("human_grade: 0.0 ..." x13) anchored it — the same
compliant answer scored 0/10 under the litany rubric and 6/10 under a
requirements checklist. And raw complaint sentences as checklist
items made a 30B judge fail every mark by quoting the requirement
itself as evidence. Now: harvested notes are LLM-distilled once per
run into <=5 testable imperatives; the judge marks each R<n>
PASS/FAIL and must quote the violating passage VERBATIM from the
answer (cannot quote -> PASS), plus a Q quality mark. The grade is
COMPUTED from the marks (0.8 x pass-fraction + 0.2 x Q), never taken
from the model's arithmetic — a judge writing "40" previously
clamped to a perfect 10. Validated live: compliant answer 0.96,
Geneva-containing answer 0.60 with correct verbatim quotes.
2. gepa defaults hostile to 1-example datasets: reflection_minibatch_size
defaults to 3, so each proposal step evaluated the SAME single example
three times — three concurrent predict calls raced past the result
cache (two crew executions finished the same second) and burned 3
executions per candidate. Now reflection_minibatch_size=1,
acceptance_criterion="improvement_or_equal" (lateral moves survive
flat regions), cache_evaluation=True (gepa-side result cache skips
repeat metric calls entirely), and a lock serializes
check-then-execute so concurrent identical calls cannot double-run
the crew.
3. Metric budget conflated with execution budget: cached re-evaluations
still consumed GEPA's metric-call budget, so a 10-execution run
stopped after 4 executions with budget left. The optimizer now gets
metric headroom (2x executions + 3) while the user's number remains
a hard cap on real crew executions.
Also: judge context no longer includes the crew objective line (the
task text said "cities like Zurich, Geneva" while the human demanded
German-side only — the contradiction primed hallucinated FAILs), and
eval-trace logging failures are logged at warning (a baseline eval
vanished silently, leaving nothing to grade).
11 new unit tests: requirement distillation/parsing and checklist-mark
grading (including the model-arithmetic and duplicate-mark hazards).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ic reflection dead-loops the search With everything else fixed, a live run still returned "baseline won" after spending only 2 of 10 executions: proposals.json showed ELEVEN byte-identical proposals (same 2268 chars, same 0.571 -> 0.356 scores), each a free cache-hit rejection until the metric budget drained. The reflection prompt is identical every iteration in this regime — same parent (the baseline never gets displaced), same single training example, same cached judge rationale — so a reflection endpoint with a deterministic default (temperature ~0) regenerates the exact same candidate forever. The result caches introduced earlier correctly made those repeats free, which turned determinism into a zero-exploration loop instead of a budget fire. gepa's LM wrapper forwards reflection_lm_kwargs to litellm.completion; an explicit temperature=1.0 makes each iteration propose a genuinely different variant, so the execution budget is spent on NEW candidates and a good draw can beat the baseline. (Judge and distillation calls already run at temperature 0 — grading stays deterministic.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion and pin the mutation output format
Two more live-run killers found after the temperature change shipped:
1. Reflection was served from Kasal's own cache. llm_manager enables a
process-global litellm disk cache at import time, and gepa's LM
wrapper rides the same litellm in the same process — so the identical
per-iteration reflection prompt got the identical cached response
forever (observed live: duration=0.00s, byte-identical proposals at
temperature 1.0). Reflection calls now send cache={"no-cache": True}.
2. gepa's default proposal template says "write a new instruction ...
within ``` blocks", which invited the reflection model to restructure:
it returned {"instruction": "..."} JSON blobs that lost the
[AGENT]/[TASK] document structure, so every proposal free-rejected
without ever executing (observed live: 11/11 malformed, run ended
after 1 execution). A custom reflection_prompt_template now pins the
output contract to the crew-doc format. Validated offline against the
live reflection endpoint: with the fenced-output instruction 2 of 3
samples degenerated to a bare "```"; with the no-fence contract at
temperature 0.8, 4 of 4 samples parsed, were distinct, and carried
the human requirements into the mutated fields.
Also: _parse_crew_doc strips surviving markdown fences before parsing
(a recoverable wrapper should not cost a candidate), with tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing panel Crew deliverables are mostly GFM tables; the read-only TextField showed them as raw pipe soup. The grading panel now renders them with react-markdown + remark-gfm (the Documentation viewer's stack), with compact table styling, scroll containment, and external links opening in a new tab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…elete from library, fix cross-crew visibility
Judges were create-only from Kasal. Now:
- Edit: assigned judge chips and library rows open an edit dialog for
instructions and/or model. Updating registers a new version under the
same registry name (MLflow scorers are versioned; latest wins —
verified live against the local registry). Editing an assigned copy
changes what that crew's runs use; editing a library judge leaves
already-assigned snapshots untouched, and the dialog says so.
list_judges now returns full instructions (4000 chars, was a 500-char
preview) so the edit round-trip cannot truncate-corrupt a judge.
- Delete: library judges get a delete action with a confirm dialog
(assigned copies survive; the chip's unassign already covered those).
- Cross-crew visibility fix: creating a judge from a crew's dialog
registered ONLY the crew-scoped copy — no library original existed,
so the judge never appeared in any other crew's Assign menu (observed
live). Creation now registers the shared library original AND the
crew-scoped copy.
- Experiment pinning: scorers are per-experiment, and none of the judge
CRUD paths pinned one — a fresh worker's active experiment is
Default/0, so judges could register into or list from the wrong
experiment and silently vanish. All five judge bodies now pin the
launch experiment ('kasal' fallback), matching the optimization runs.
Verified end-to-end against the live API: create-from-crew registers
both names, update re-versions the target only, delete cleans up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ce, backend and frontend
Backend — 93 tests:
- service (67): crew-doc serialize/parse round-trip incl. multiline
continuation and malformed-candidate rejection (the GEPA mutation
contract), judge verdict normalization (numeric/bool/categorical),
requirement distillation + checklist-mark grading (incl. the
model-arithmetic and duplicate-mark hazards), judge lifecycle against
a faked mlflow registry (dual library+scoped registration, {{ outputs }}
autofix, update keeps omitted fields, assign snapshots, delete all
versions, per-operation experiment pinning, local-mode gating), run
registry behaviors (public-field allowlist never leaks task handles,
timezone-aware sort, cancel transitions/guards, prune spares active
runs), plus the existing mining/scorer/registry/apply coverage
- router (26): every endpoint's handler — ValueError → 400 for client
input, missing runs → 404, response-schema wrapping incl. the
progress-chip fields, judge CRUD passthrough, eval-feedback numeric
coercion, user-context publication, and the request-schema bounds
(budget 4..40 default 10; template name is a closed set)
Frontend — 46 tests:
- PromptOptimizationService (19): every method's endpoint, payload
undefined-stripping, judge-name URL encoding, zero-grade preservation,
list unwrapping and boolean coercion
- CrewOptimizeDialog (10): judge scoping by crew registry prefix
(assigned chips vs library menu; other crews' judges invisible;
same-name library duplicate excluded), edit-by-full-name, delete
confirm, create auto-assign, honest-progress chips, budget-as-cap
start request, ungraded-first eval ordering with counts, markdown
answer rendering, grade submission
- PromptConfiguration (5): Optimize action gated to optimizable
templates and to an onOptimize handler; edit dialog
- PromptOptimization (4): fixedTemplate hides the picker and filters
runs; start payload; backend rejection surfaced
- Prompts (3): dialog opens scoped to the chosen template and closes
- optimizableTemplates (2): mirrors the backend TEMPLATE_TASKS set
- CrewFlowDialog optimize entry (3): source-level guards in the
established CrewFeedback.test.tsx style (heavy dialog): click
isolation and dialog wiring
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…any explicit temperature A crew run on kimi-k2.7-code-highspeed "completed" after a single execution: gepa made 10 reflection attempts and produced ZERO proposals (no proposals.json artifact, no subsample scores). Direct reproduction against Moonshot with gepa's exact request showed why — every call 400'd with "invalid temperature: only 1 is allowed for this model"; the same call without a temperature returns a clean, parseable crew doc (thinking rides the separate reasoning_content field). _resolve_reflection_model now returns (uri, env, provider) and the crew body builds reflection_lm_kwargs per provider: Kimi gets only the litellm cache bypass (its forced default sampling is already diverse); everyone else keeps temperature 0.8. The template-mode optimizer gains the same cache bypass (it never sent a temperature, so it was Kimi-safe but cache-exposed). New test pins the kimi resolver path (key lookup, URI, provider tag); resolver tests updated for the 3-tuple. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…provider plumbing outside the manager The optimization service had grown its own provider layer for the calls MLflow/gepa make directly (reflection, registered judges, preflight): a URI resolver, a provider→env-key table, per-tenant API keys written into the shared process env, and request quirks re-implemented per provider. Each of those independently produced live failures (retired DeepSeek names, Kimi temperature 400s, key-lookup drift) that LLMManager already handles centrally for the rest of Kasal. Now every LLM call goes through LLMManager.completion: - GEPA reflection: gepa accepts a LanguageModel callable, but MLflow's GepaPromptOptimizer pins reflection_lm to a litellm string. A one-time idempotent patch on gepa.optimize swaps in a per-worker-thread callable (each run owns one thread) backed by _sync_llm_completion. The callable adds a unique system-line cache-buster (the process- global litellm cache replayed one proposal forever for the identical per-iteration prompt) and passes temperature=0.8 — the MANAGER drops it for providers that reject it (Kimi), so the provider-aware kwargs from the previous commit are deleted again. - Registered judges: rendered and invoked HERE via LLMManager instead of mlflow's own model client. Judges store a plain Kasal model key (wrapped as 'openai:/<key>' only to satisfy make_judge's URI shape); legacy URI-storing judges are stripped back to their best key. - Preflight: a manager-routed ping instead of a raw litellm call with env juggling. Deleted: _resolve_reflection_model, _REFLECTION_KEY_ENV, reflection env application/restore in both sync bodies, reflection_lm_kwargs, and the reflection_provider threading. Security side effect: per-tenant API keys are no longer written into os.environ. Tests: resolver suite replaced with coverage of the new pure helpers (key stripping, grade parsing, bridge install/override/idempotency, preflight error wrapping, reflection message shaping incl. distinct cache-busters); judge lifecycle updated for key storage. 95 backend tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
npm install re-resolves and rewrites package-lock.json on every run (optional/peer dependency metadata shifts with npm/registry state), causing spurious lockfile churn on every local build and deploy. npm ci installs strictly from the lockfile and never modifies it. Also resyncs package-lock.json, which had drifted out of sync with package.json (npm ci was failing on missing optional platform deps) independent of this change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Maximilian Kohnen <57352546+mckohnen@users.noreply.github.com>
…T/DCC) Consolidated digest of three CCHBC dataset runs and the repo changes they imply: - SC: success baseline — 30 views, 476/476 measures exact-match to ground truth (100%), +41 vs the prior run; 74% coverage of all source measures. - PAAT: report skin, no M-Query source tables; 46 measures resolvable → ~10-19% best-effort ceiling once physical source names are supplied. - DCC: KPI/data-quality report; 40% measure-on-measure DAX, 0 resolvable → ~0-5% ceiling, needs composition + table-expr translators. Actionable repo changes: fact_source_map + allow_best_effort config keys, a best-effort view generator reusing measure_resolutions, not-emitted docs reusing the SC machinery, and evaluate_ucmv.py as the CI gate (correctness + coverage never-regress). Report-layer limits documented (selectors/disconnected tables never translate).
Most 'no UC Metric Views' outcomes (PAAT/DCC) are thin reports on a separate/
upstream semantic model, not conversion failures — the model with the M-Queries and
physical tables is a different dataset than the one extracted. Document this as the
primary fix:
- NEW customer-facing guide thin-report-and-source-resolution.md (three model kinds,
how to check via lineage / REST GET /reports/{id} -> datasetId, and what to do).
Registered in the in-app Documentation viewer (Power BI migration section).
- Roadmap doc: add 'resolve the upstream model' as the preferred fix (Kasal already
extracts by dataset_id; automate thin-report detection + follow the upstream link);
fact_source_map is the fallback-of-the-fallback for unreachable models.
Clearly flags the best-effort limits: tables not mapped are skipped, selector/
measure-on-measure DAX may not convert, every best-effort measure is TODO: verify —
tables/measures may be missing.
When the normal path produces 0 views (thin report — no M-Query source tables), allow drafting thin metric views IF the user opts in (allow_best_effort) AND supplies physical sources (fact_source_map: pbi_table -> catalog.schema.table). This is the fallback-of-the-fallback; the preferred fix remains converting the upstream semantic model (see thin-report-and-source-resolution.md). - New config keys allow_best_effort + fact_source_map (forwarded generically via tool_configs; no tool_factory allowlist change needed). - _build_best_effort_views: emits one view per supplied source from measures that ALREADY resolved to real aggregatable SQL in config.measure_resolutions (reusing base_expr + base_filters). NEVER emits TODO/selector/empty resolutions, so the 'no silently-wrong SQL' contract holds. Every measure flagged 'TODO: verify'. - Returns a best_effort_report making the GAP explicit (skipped tables/measures, 'may be MISSING' warning) for the UI to surface. - Gated: fires only when yaml_output is empty AND both keys present. Default path unchanged. - 9 unit tests (gate, never-emit-unresolved, filters, missing-table reporting); existing 28 generator tests still green. Co-authored-by: Isaac
…ention) public/docs/*.md are tracked in this repo (not gitignored build artifacts), so the in-app Documentation viewer serves them without a build step. Add the committed copy of thin-report-and-source-resolution.md alongside the other powerbi docs so the sidebar entry loads. Co-authored-by: Isaac
A thin-report run used to yield a silent fallback JSON with no explanation. Add _diagnose_zero_views: on any empty run, classify WHY and tell the user what to do, from two deterministic signals (source tables present? measures resolvable?): - thin_report_no_source_tables + resolvable>0 → 'point at the upstream model, or enable allow_best_effort + fact_source_map (tables/measures may be missing)' - thin_report + resolvable==0 (the DCC signature) → 'these measures cannot be converted; re-run against the upstream model if one exists' - sources_present_no_views → 'translation issue, no source mapping needed' Surfaced as output.zero_view_diagnosis (+ a warning log). Turns the silent empty result into an actionable next step pointing at the real fix (upstream model). 4 new tests; 13 total in the file, all green. Co-authored-by: Isaac
Concrete code-change plan (companion to the HTTP-header serving-cost path in the
kasal_consumption_tracking repo): stamp custom_tags={'app':'kasal'} at resource
CREATE time so system.billing.usage can attribute cost. Covers the resources Kasal
provisions — Lakebase instance (lakebase_service.create_instance ~L359), VS endpoint
(databricks_vector_endpoint_repository.create_endpoint payload), the App (budget
policy / app.yaml). Exact sites + SDK-field caveats + verification query + rollout
order. Honestly scoped: shared serving endpoints and connect-to-existing resources
are NOT tag-attributable (the header token->$ path handles serving; volume is the
ceiling for the rest).
Co-authored-by: Isaac
|
|
Flow runs logged "InvalidRequestError: This session is provisioning a new
connection; concurrent operations are not permitted" from
execution_name_service, which then fell back to a timestamp run name
("Execution-20260805_141402") instead of the LLM-generated one.
Root cause: request_scoped_session() intentionally REUSES the request-scoped
session when the `_request_session` ContextVar is set. Name generation runs on
a task that inherited the HTTP request's context, so every nested "open my own
session" call — notably the model-config read in
LLMManager.configure_crewai_llm — got that shared session and ran concurrently
with the in-flight request. AsyncSession is not concurrency-safe.
Fixes:
- generate_execution_name: in standalone mode (session=None) call
detach_request_session() first, so nested reads open private sessions. This
is the documented remedy (src/db/session.py) and was previously only used by
crew_generation_service.
- _get_name_template / _log_llm_interaction: use async_session_factory (a
private session) rather than request_scoped_session, so the naming step can
never join — and so never contend with — the request transaction.
- crewai_flow_service: build ExecutionNameService with session=None instead of
self.session (the request session).
Two existing tests caught real bugs in the first cut (unsafe _session access;
patch targets that no longer matched) — code fixed, patches updated to the
symbols the code now calls. Added regression tests asserting standalone mode
detaches and injected-session mode does NOT (detaching there would break the
caller's transaction).
Verified: 11/11 name-service, 390 service, 2396 engine/flow/UCMV tests pass;
a real flow run now gets its LLM-generated name with no traceback.
Co-authored-by: Isaac
MrBlack1995
marked this pull request as ready for review
August 5, 2026 13:08
… it declined
Two related defects, both surfaced by running the re-evaluation sweep against
real CCHBC data (104 recorded non-transpiled measures).
1. Stale skip_reason (dax_llm_fallback.py) — a CORRECTNESS bug in what we show
customers. The regex fast path sets
"routed to LLM (fast-path dropped a DAX component)" to mean "hand this to the
LLM". When the LLM subsequently ALSO declined, the code recorded dax_class but
left that routing text in place, so an INTERMEDIATE state was reported as the
final one: the "Not transpiled" panel and the re-evaluation report both implied
the LLM never got a turn, for measures it had already rejected. Now the decline
path writes a terminal verdict naming the decider and the reason, e.g.
"LLM declined [unsupported] — no UC metric-view equivalent". Fixed on both the
live and the cached-decline path (the cache had the same gap).
2. dax_class gate (reevaluate.py) — the sweep retried measures the LLM had
already classified as untranslatable, so it looked busy while proposing
nothing. dax_class in {unsupported, display_layer, out_of_scope,
architecture_change} now means "already declined at this capability level" and
is skipped by default (opt-in via include_impossible). Checked BEFORE the
free-text markers because the LLM's own verdict is the more reliable signal.
Effect on the real Aug-5 run: 73 -> 5 retried per dataset (321 -> 15 overall).
The 5 survivors are the genuinely actionable ones — blocked by the silent-wrong
guard, not by LLM refusal (unresolved measure ref; additive term dropped; ratio
denominator dropped), two of them classed composed / translatable_direct, i.e.
recoverable once composition/guard improves.
Tests: +7 re-evaluation (LLM-declined skipped, opt-in override, guard-blocked
still retried), +3 for the terminal-reason helper. 27 re-evaluation, 28
LLM-fallback, 995 metric_view_utils all pass. The 3 failures in the wider tools
suite (llm_model_default) pre-exist on HEAD and are unrelated.
Co-authored-by: Isaac
Direct-embedded-SQL resolution missed reference-following (staging query passthrough) and parameter-driven Value.NativeQuery (string- concatenation) cases, so DCC/PAAT-scale reports with parameterized M queries silently lost source SQL. Adds mquery_let_evaluator.py to evaluate M `let` blocks narrowly enough to resolve both patterns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Maximilian Kohnen <57352546+mckohnen@users.noreply.github.com>
Crew 1 (Pipeline Config Generator) and Crew 2 (UC Metric View Generator) run in separate OS subprocesses, so a /tmp handoff never worked (confirmed empirically). Crew 2's own independent extraction was structurally weaker (no reference-following / parameter- substitution resolution). Persists Crew 1's admin_tables + expressions to the powerbi_extraction table, keyed by job_id, and has Crew 2 look it up and rebuild mquery_json/measures_json via the same static methods Crew 1 used — so quality matches instead of degrading. Also: - Tier-gates Crew 1's fallback extraction (Fabric TMDL, SP-retry Admin Scanner) on EITHER admin_tables OR expressions being empty, each tier only filling what's still missing. - Adds admin_client_id/admin_client_secret to Crew 2's schema/UI as an optional SP-retry fallback tier. - Adds per-table wall-clock logging through the fact-table processing loop (pipeline.py) and the DAX-vs-SQL validation loop (uc_metric_view_generator_tool.py) — needed to diagnose a large- report timeout (see the flow-timeout/logging-visibility commit). - Adds _diagnostics to the tool's output (db_fallback_fired_for, preinject_*_json_chars, etc.) so the handoff is inspectable from execution_trace without re-deriving it from logs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Maximilian Kohnen <57352546+mckohnen@users.noreply.github.com>
run_schema_self_heal shared one connection/transaction across all healer steps. On Postgres, one failed statement aborts the entire transaction, so every step after the first failure silently no-opped (observed in production against Lakebase). A per-step SAVEPOINT (conn.begin_nested()) didn't fully fix it either: healers already swallow their own errors internally, so the savepoint wrapper never saw an exception to react to — it only discovered the poisoned transaction when it tried to auto-commit, and by then a plain ROLLBACK TO SAVEPOINT didn't reliably leave the connection usable for the next step's own SAVEPOINT statement (confirmed in production: every later step failed too, each at its own SAVEPOINT line). Switches run_schema_self_heal to take an AsyncEngine and open a fresh connection + transaction per step, so there's no shared transaction state left for one step's failure to poison. Updated all 4 call sites (init_db, main.py's Lakebase hot-swap, and both lakebase_service.py connect/expand-schema paths) accordingly, and fixed the powerbi_extraction.expressions column healer's is_sqlite detection (was checking the app's global DATABASE_URI instead of the actual connection's dialect) along with 6 other _ensure_* functions that had the same bug. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Maximilian Kohnen <57352546+mckohnen@users.noreply.github.com>
completion_with_usage calls litellm.completion() directly and only copied api_key/api_base off the configured LLM object, dropping the 297s/300s timeout that configure_crewai_llm already computes and that completion()'s CrewAI LLM.call() path already respects. Callers on this path (e.g. dax_llm_fallback.py's per-measure DAX translation) could hang indefinitely on one stuck/throttled call — confirmed in production as a 50+ minute silent stall with zero further LLM spans, on a report whose DAX-LLM fallback was hitting sustained REQUEST_LIMIT_EXCEEDED throttling from the Databricks serving endpoint. Forwarding "timeout" alongside api_key/api_base closes the gap; a throttled call now fails within the existing bound instead of hanging, and dax_llm_fallback.py's existing fail-open fallback path takes over immediately. Also bumps the flow's per-crew kickoff timeout from 20 to 55 minutes (flow_methods.py) — MetricViewPipeline processes fact tables sequentially, and a report with many of them can legitimately exceed 20 minutes even with calls now bounded. Leaves a 5 min margin under process_flow_executor's outer 1-hour subprocess watchdog. Adds the UCMV custom-tool packages (uc_metric_view_generator_tool, pipeline_config_generator_tool, metric_view_utils, metric_view_validation_utils) to the subprocess logging allowlist (logging_config.py). Their loggers were never routed to flow.log/ execution_logs, so none of their output — including the per-table timing added in the DB-handoff commit — was ever visible; this is what let the throttling and the subsequent hang actually be diagnosed from execution_logs instead of guesswork. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Maximilian Kohnen <57352546+mckohnen@users.noreply.github.com>
Merges the admin_client_id/admin_client_secret keys from this branch with allow_best_effort/fact_source_map from the colleague's thin- report best-effort work — both are additive, opt-in config fields on the same tuple. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Maximilian Kohnen <57352546+mckohnen@users.noreply.github.com>
Fix/ucmv pipeline mquery handoff
MrBlack1995
force-pushed
the
feat/pbi-mquery-performance
branch
from
August 17, 2026 07:24
6870de1 to
220785c
Compare
MrBlack1995
changed the base branch from
prompt-optimization
to
pbi-optimization
August 18, 2026 11:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #67 (
prompt-optimization). Scoped to PowerBI M-Query pipeline performance.Status
Draft — opening early to track. First commit is the UCMV coverage evaluation +
best-effort roadmap doc (SC/PAAT/DCC analysis); the M-Query performance changes
land in follow-up commits.
Included so far
docs(powerbi): UCMV coverage evaluation + best-effort roadmap — consolidatesthe SC (100% measure match, 74% coverage), PAAT (~10-19% ceiling) and DCC
(~0-5% ceiling) dataset analyses and the repo changes they imply.
Planned (M-Query performance)
Base
Targets
prompt-optimization(PR #67), notmain, so the diff shows only theM-Query work stacked on top of the prompt-optimization changes.
This pull request and its description were written by Isaac.