[Power BI] Fix SkuMeterName column error in storage reports for FOCUS 1.2 exports - #2333
Michael Flanakin (flanakin) wants to merge 5 commits into
Conversation
…OCUS 1.2 exports Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
microsoft-github-policy-service
Bot
added
the
Needs: Review 👀
| else Table.AddColumn( InvoiceId, "PricingCurrency", each [x_PricingCurrency]), | ||
| SkuMeter = if Has12 | ||
| then Table.ReplaceValue(PricingCurrency, null, each [x_SkuMeterName], Replacer.ReplaceValue, {"SkuMeterName"}) | ||
| then Table.ReplaceValue(PricingCurrency, null, each [x_SkuMeterName], Replacer.ReplaceValue, {"SkuMeter"}) |
There was a problem hiding this comment.
The rename itself is right — SkuMeter is what FOCUS 1.2 emits, and it matches both the else branch and the Align12 step below.
While checking it I found the rest of the block has the same class of problem, so I pushed a follow-up commit here rather than leaving it for later. Details below.
I ran a real FocusCost export against a test subscription (dataVersion: 1.2-preview, 105 columns) and diffed the header. In a 1.2 export none of the three legacy columns exist:
| column | 1.0r2 | 1.2-preview |
|---|---|---|
x_InvoiceId |
yes | gone — promoted to InvoiceId |
x_PricingCurrency |
yes | gone — promoted to PricingCurrency |
x_SkuMeterName |
yes | gone — renamed to SkuMeter |
Side note: the API rejects dataVersion: "1.2" outright ("Export data version: 1.2 is not supported"), so 1.2-preview is currently the only 1.2-shaped export that can be created.
ftk_Storage does no backfill for focuscost (CleanColumns = if data = "focuscost" then ExtractColumns else ...), so the available columns are exactly the union of what's in the files. A container holding only 1.2 exports therefore has none of the three, and the refresh fails at each [x_InvoiceId] and at Align12's columnsToSearch — the same class of error as #2332, one step later. Only a container that also holds older 1.0/1.0r2 files was fixed by the one-line change alone.
EnsureRenamedCols now adds whichever side is missing as a null column before Align12 copies values both ways, so 1.0-only, 1.2-only and mixed containers all work. Has12 is gone — the per-column checks replace it and it had no other caller.
Also pushed: a PowerQueryColumnNames lint rule, since nothing in CI runs Power Query and that is why the typo shipped. It checks that every columnsToSearch literal names a real column, and that references to version-dropped columns are guarded. Reverting the one-line fix makes it fail with got 'SkuMeterName'.
On testing: the lint rule and the column measurements are solid, but the query change itself is only statically verified — I could not open the project in Power BI Desktop to refresh it. Shared.Dataset declares two columns in AdvisorReservationRecommendations that differ only by case, properties.extendedProperties.region (line 107) and properties.extendedProperties.Region (line 383). The tabular model is case-insensitive, so loading the project fails with "Item 'properties.extendedProperties.Region' already exists in the collection" before any refresh happens. Same in the kql dataset. That is pre-existing and unrelated to this PR, but it blocks an end-to-end check — worth a separate look.
Separate doc bug fixed in this PR: docs-mslearn/focus/metadata.md listed x_SkuMeterName in the FocusCost 1.2-preview column table. The real export doesn't have it. The ADF mappings in Exports/schemas/focuscost_1.2*.json were already correct.
There was a problem hiding this comment.
🤖 [AI][Claude] ✅ Implemented
Confirmed — the existing Has12 check only decides which column name is authoritative, not whether the legacy x_ column exists at all. Even on the Has12 branch, the code still does each [x_SkuMeterName] (and x_InvoiceId/x_PricingCurrency) to backfill nulls, so a 1.2-only export (no legacy columns) errors on the next accessor in the chain.
Added a backfill step before this block that adds any missing x_InvoiceId/x_PricingCurrency/x_SkuMeterName as null text columns when absent, so the existing Has12/Align12 logic works unchanged for 1.2-only, 1.0-only, and mixed containers. Doc fix in docs-mslearn/focus/metadata.md was already merged in a prior commit.
There was a problem hiding this comment.
🤖 [AI][Claude] Update: while I was drafting a backfill fix, e7e11e0 landed with a more complete version — it removes Has12 entirely and backfills missing columns on both the 1.0 and 1.2 sides (my version only handled the 1.2-only case). Dropped my commit in favor of that one; nothing further needed here.
Nothing in CI runs Power Query, so a mistyped column name in a .tmdl partition ships and surfaces only as a refresh failure on a customer's report. That is how issue #2332 happened: the FOCUS 1.2 branch of the Costs query searched {"SkuMeterName"}, a column no FOCUS version emits. Two static rules: - Every columnsToSearch literal in Table.ReplaceValue must name a column that exists - from a Cost Management export schema, created by the query itself, or declared on the model table. - References to columns FOCUS 1.2 drops (x_InvoiceId, x_PricingCurrency, x_SkuMeterName) need a Table.HasColumns or _exists guard, because ftk_Storage does no backfill for focuscost. The dropped set is derived from the export schemas rather than hard-coded, so a future FOCUS version updates the rule on its own. Scoped to focuscost partitions: the pricesheet and reservationtransactions datasets reuse some of these names legitimately. Rule 2 baselines the three existing unguarded references in Costs.tmdl as a ratchet. Verified the rule catches the original typo: reverting the one-line fix fails rule 1 with "got 'SkuMeterName'". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The FocusCost 1.2-preview table listed x_SkuMeterName alongside the SkuMeter column that replaced it. Verified against a real FocusCost export (dataVersion 1.2-preview, 105 columns): x_SkuMeterName is not delivered. The 1.0 and 1.0-preview(v1) tables keep the column. The ADF mappings in Exports/schemas already had this right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "Handle columns renamed in FOCUS 1.2 gracefully" block read [x_InvoiceId], [x_PricingCurrency] and [x_SkuMeterName] unconditionally, and Align12 passed all three as columnsToSearch. FOCUS 1.2 does not deliver them - verified against a real export (dataVersion 1.2-preview, 105 columns): InvoiceId, PricingCurrency and SkuMeter are present, the three x_ columns are not. ftk_Storage does no backfill for focuscost, so a container holding only 1.2 exports has none of them and the refresh fails with "The column 'x_InvoiceId' of the table wasn't found" - the same class of error as #2332, just one step later. EnsureRenamedCols now adds whichever side is missing as a null column before Align12 copies values in both directions, so 1.0-only, 1.2-only and mixed containers all work. Has12 is gone: the per-column checks replace it and it had no other caller. Lowers the PowerQueryColumnNames rule 2 baseline to zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🤖 [AI] Fixes #2332
Problem
Storage reports fail to refresh with
Expression.Error: The column 'SkuMeterName' of the table wasn't foundwhen the export uses FOCUS 1.2.Fix
In the Costs query, the FOCUS 1.2 branch replaced values in a column named
SkuMeterName. FOCUS 1.2 names itSkuMeter, and the non-1.2 branch and the followingAlign12step already useSkuMeter. Updated the 1.2 branch to match.Testing
Not run (requires Power BI Desktop). Verified the column name against
FocusCost_1.2-preview.json.🤖 Generated with Claude Code