Skip to content

Resume broker schema collection when the monitor toggle is switched back on - #8997

Open
cstns wants to merge 2 commits into
mainfrom
8995-broker-schema-idempotent-start
Open

cstns wants to merge 2 commits into
mainfrom
8995-broker-schema-idempotent-start

Conversation

@cstns

@cstns cstns commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

When schema collection stops, either through the API or when the agent hits its own 24 hour auto-shutdown, the agent pauses but the platform still has it recorded as running. Switching the "Monitor Broker for Smart Schema Suggestions" toggle back on after that did nothing: the start path saw "already running" and skipped sending the agent a start command, so it was never told to resume. The toggle looked on, nothing collected.

The team-broker start handler now always sends the agent a start command when it exists and isn't mid-start or mid-suspend, instead of no-oping when it looks running. The agent treats a repeat start as a reconnect, so it's safe when it was already collecting, and it resumes collection otherwise. No new agent states or migration, so the agent lifecycle is untouched, and it covers the 24 hour auto-stop case too.

Checked locally against the team broker: start then stop leaves the agent connected=false, and before this a second start left it that way; now the second start brings connected back to true.

Closes #8995

…ack on

When schema collection stopped, either through the API or the agent's own 24 hour auto-shutdown, the agent paused but was still recorded as running, so switching the monitor toggle on again hit the "already running" branch and never told the agent to start. The toggle looked on but nothing collected.

The start handler now always sends the agent a start command when it exists and isn't mid-start or mid-suspend, instead of no-oping when it looks running. The agent treats a repeat start as a reconnect, so this is safe when it was already collecting, and it resumes collection otherwise.
@cstns cstns self-assigned this Oct 20, 2026
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.46%. Comparing base (790d2f2) to head (bb509c5).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8997      +/-   ##
==========================================
+ Coverage   79.44%   79.46%   +0.01%     
==========================================
  Files         475      475              
  Lines       26017    26018       +1     
  Branches     6934     6934              
==========================================
+ Hits        20670    20674       +4     
+ Misses       5347     5344       -3     
Flag Coverage Δ
backend 79.46% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andypalmi
andypalmi self-requested a review October 20, 2026 16:36

@andypalmi andypalmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Costin, I traced the agent code and I don't think we can merge this as is.

  1. A repeat start is a full teardown, not a harmless reconnect. The route sends start to the agent (3rdPartyBroker.js L450-L457), which hits /commands/start and runs Agent.start(). That calls stop() (ends the MQTT client, uploads, clears the report interval, sets connected=false), then primeCache() and connect(), which re-fetches credentials and opens a new client. Calling start on an agent that is already collecting takes the broker offline for that whole window and drops any messages in it.

  2. This now hits every caller. The route used to be a safe no-op for a running agent. With this change every API caller, including the broker lifecycle tool, takes that hit, not just the toggle.

  3. Concurrent starts can leak a client. start() and connect() are not serialized. Two overlapping starts both see this.client, both stop, then both connect. The second overwrites this.client and this.reportInterval, so the first client and its interval are never cleaned up. The route doesn't dedupe, so a double click or two users can trigger it.

  4. Failures look like success. Both drivers swallow errors from the start command (driver-docker, driver-k8s), so the route replies 200 even if the agent never resumed.

  5. Suggestion. The no-op is right when the agent is actually collecting, and the bug is that running on the row doesn't tell us that. Could the route ask the agent for its real state and only send start when it isn't connected? The agent's /status returns connected, and both drivers expose it (docker, k8s) through app.containers.getBrokerAgentState. Then toggling on after a stop or the 24h auto-shutdown resumes collection, and a start on a healthy agent stays a no-op.

Sending start to an agent that is already collecting tears it down and reconnects, taking the broker offline for that window. Instead of always re-sending start for a running agent, ask the agent for its real state and only send start when it isn't connected. A stopped or auto-shut-down agent resumes; a healthy one stays a no-op.
@cstns

cstns commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Good catch, you're right, start() tears down and reconnects. Reworked to your suggestion: the handler checks the agent's real state and only sends start when it isn't connected, so a stopped agent resumes and a healthy one stays a no-op. Tests for both.

The concurrent-start leak and swallowed driver errors are in the agent/driver repos, so I'll leave those as separate follow-ups.

This branch was successfully deployed

1 active deployment
staging — bb509c5b Deployed Oct 20, 2026 by cstns via Deploy application #12632
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broker schema monitoring doesn't restart when the toggle is switched back on

2 participants