Skip to content

Harden Cost Management export retries for HTTP 429 #2257

Description

@MSBrett

🐛 Problem

Start-FinOpsCostExport can fail instead of retrying when the Cost Management Run API returns HTTP 429. Invoke-Rest emits Write-Error for the response before the caller can reliably inspect Throttled, and the loop condition does not repeat a single-export run even when throttling is returned non-terminatingly.

The current unit test mocks Invoke-Rest returning a throttled response directly, so it does not cover the terminating error behavior observed in CI.

👣 Repro steps

  1. Run New-FinOpsCostExport -Execute or Start-FinOpsCostExport in a context where PowerShell treats Write-Error as terminating.
  2. Have the Cost Management export Run API return HTTP 429 with a Retry-After value.
  3. Observe that the command exits with an error instead of waiting and retrying.

🤔 Expected

  • HTTP 429 responses are returned to Start-FinOpsCostExport in a form it can handle.
  • The command honors the server-provided Retry-After value, with a safe fallback when absent or invalid.
  • Both single-export and multi-month runs retry the same request after throttling.
  • Retries are bounded to prevent an infinite loop.
  • Unit tests reproduce the real Invoke-Rest/Write-Error behavior and cover single-export, multi-month, retry-delay, retry-limit, and eventual-success cases.

📷 Screenshots

N/A

🔧 Environment

ℹ️ Additional context

Observed in the Hubs (manual) check for PR #2187: deployment completed, then the initial export run failed with Too many requests. Please retry after 60 seconds. (Code: 429).

Failed run: https://github.com/microsoft/finops-toolkit/actions/runs/31678558903

The two immediately preceding PR deployment runs succeeded, indicating the service-side 429 was transient while the retry-handling failure is deterministic.

🙋‍♀️ Ask for the community

We could use your help:

  1. Please vote this issue up (👍) to prioritize it.
  2. Share other Cost Management throttling responses, especially variations in status codes or retry headers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Tool: PowerShellPowerShell scripts and automation

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions