fix(dal): retry transient Pebble lock opens

[shivanshgupta365]

Summary

• Retry Pebble startup only when the wrapped error is syscall.EAGAIN.
• Use ten capped exponential-backoff waits from 100ms to 5s, for a 26.3s total retry budget.
• Preserve immediate failure for permission, corruption, configuration, and every other non-lock error.
• Cover transient success, non-retryable failure, retry exhaustion, and real cross-process Pebble lock contention.

Fixes #1522

Validation

• go test -race ./internal/storage/dal -count=1 -timeout 10m
• go vet ./internal/storage/dal
• go build ./...
• go mod tidy in the root and operator modules
• golangci-lint 2.12.2 on ./internal/storage/dal/... with all project build tags: 0 issues

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[NumaryBot]

✅ Approve — automated review

The retry logic is narrowly scoped to EAGAIN-shaped Pebble open failures, preserves immediate failure for other errors, and the added tests cover the main success, non-retry, exhaustion, and real lock-contention cases. I did not find a discrete correctness issue introduced by this diff.

No findings.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 725b86bd-68c7-4cd4-9d3c-23baf3221f8a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[flemzord]

Revalidated live head 86d568d2. The startup retry is restricted to wrapped EAGAIN lock-contention errors, remains bounded, and preserves immediate failure for every other Pebble open error. The focused tests cover transient success, exhaustion, non-retryable errors, and real cross-process lock contention; the available CI and review checks are green. I found no correctness/runtime blocker on this head. GitHub approval is intentionally disabled for this cron task.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.79%. Comparing base (c3ddbad) to head (86d568d).
⚠️ Report is 2 commits behind head on release/v3.0.

Additional details and impacted files

@@              Coverage Diff              @@
##           release/v3.0    #1664   +/-   ##
=============================================
  Coverage         74.78%   74.79%           
=============================================
  Files               447      447           
  Lines             47595    47612   +17     
=============================================
+ Hits              35596    35610   +14     
- Misses             8796     8801    +5     
+ Partials           3203     3201    -2     

Flag	Coverage Δ
e2e	74.79% <100.00%> (+<0.01%)	⬆️
scenario	74.79% <100.00%> (+<0.01%)	⬆️
unit	74.79% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.