Skip to content

Fix: cache Status.TimeLocation() to stop HTTP request storm in EventStableChecker#350

Open
royteeuwen wants to merge 1 commit into
wttech:mainfrom
royteeuwen:fix/event-stable-checker-request-storm
Open

Fix: cache Status.TimeLocation() to stop HTTP request storm in EventStableChecker#350
royteeuwen wants to merge 1 commit into
wttech:mainfrom
royteeuwen:fix/event-stable-checker-request-storm

Conversation

@royteeuwen

Copy link
Copy Markdown
Contributor

Fixes #349

Problem

EventStableChecker.Check() resolves the instance's timezone via
instance.Time()Status.TimeLocation() once per OSGi event being
evaluated, and TimeLocation() previously issued an uncached HTTP GET to
/system/console/status-System%20Properties.json on every call. A single
event_stable check with N events therefore issues N HTTP requests, and
this repeats on every poll while aemc waits for instance stability
(instance await, instance start, package deploy, etc.), which can
add up to thousands of requests per minute against a single instance.

Fix

  • Added sync.Once-based caching to Status.TimeLocation(), so the
    underlying HTTP request happens at most once per Status object (i.e.
    once per CLI invocation, per instance) instead of once per OSGi event
    per poll. This is safe because the instance's user.timezone JVM
    system property is fixed at instance startup and cannot change during
    a single aemc process run.
  • Converted all Status methods (SystemProps, SlingProps,
    SlingSettings, parseProperties, TimeLocation, RunModes,
    AemVersion) from value receivers to pointer receivers. This is
    required because a struct containing a sync.Once must never be
    copied by value (go vet flags "passes lock by value"); Status is
    always accessed via a *Status field on Instance already, so this
    is a non-behavioral change.
  • As a side effect, this also fixes a latent bug: previously, if
    time.LoadLocation failed, the function returned (timeLocation, nil)
    — i.e. a nil location with a nil error — which could cause a
    nil-pointer panic in time.Time.In(nil) downstream. The fix properly
    propagates the error so the existing caller fallback
    (instance.TimeLocation() returning time.Now().Location() on error)
    actually kicks in.

Verification

  • go build ./... — clean.
  • go vet ./pkg/... — clean (no more "passes lock by value" warnings).
  • go test ./... — all existing tests pass.
  • Manually verified end-to-end against a running local AEM instance by
    building both a baseline (pre-fix) and fixed binary and running
    instance await with each, diffing the AEM request log for hits to
    status-System%20Properties.json over the same ~30s / 4-poll-round
    window:
    • Baseline: 1506 requests.
    • Fixed: 1 request.
    • Both runs reported identical instance health check results
      (checked (4/4), all checks passing), confirming no behavioral
      regression.

Notes for reviewers

No test coverage currently exists for pkg/status.go or pkg/check.go.
Happy to add a unit test for Status.TimeLocation() (e.g. via
httptest.Server, asserting the endpoint is hit exactly once across
multiple calls) if that's wanted before merge — let me know.

…hecker

EventStableChecker.Check() calls instance.Time() once per OSGi event to
determine event age, which resolves the instance timezone via
Status.TimeLocation() on every call. Since TimeLocation() previously issued
an uncached HTTP GET to /system/console/status-System%20Properties.json,
a single event_stable check with N events triggered N HTTP requests, and
this repeats on every poll while aemc awaits instance stability - easily
producing thousands of requests per minute against a single instance.

The instance's user.timezone JVM system property cannot change while a
single aemc process is running, so it's safe to resolve it once per
Status instance using sync.Once.

Status methods are switched from value to pointer receivers since a
struct containing a sync.Once must not be copied by value. Status is
always accessed via a *Status field on Instance already, so this is a
non-behavioral change.

As a side effect this also fixes a latent bug where a time.LoadLocation
failure returned (nil, nil) instead of propagating the error, which could
cause a nil-pointer panic downstream in time.Time.In(nil).
@krystian-panek-vmltech

Copy link
Copy Markdown
Contributor

Tbh I have never seen any problems related to it. What's your motivation to fix this?

@royteeuwen

royteeuwen commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

We are noticing 1000's of http requests going to the instance, just to get the timezone again and again, which doesnt change? Inefficient and it pollutes the logs, why wouldnt you want to fix it if this can easily be cached? Is there a reason not to do it?

@krystian-panek-vmltech

Copy link
Copy Markdown
Contributor

Just asking 🙂 you are right. Caching might be sometimes problematic. When to cache and when to revalidate... Etc. I will probably merge but give me some time for tests.

@royteeuwen

Copy link
Copy Markdown
Contributor Author

Super, let me know if you need any changes ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EventStableChecker triggers an HTTP request storm against status-System Properties.json

2 participants