Repository navigation
Conversation
…ack on When schema collection stopped, either through the API or the agent's own 24 hour auto-shutdown, the agent paused but was still recorded as running, so switching the monitor toggle on again hit the "already running" branch and never told the agent to start. The toggle looked on but nothing collected. The start handler now always sends the agent a start command when it exists and isn't mid-start or mid-suspend, instead of no-oping when it looks running. The agent treats a repeat start as a reconnect, so this is safe when it was already collecting, and it resumes collection otherwise.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8997 +/- ##
==========================================
+ Coverage 79.44% 79.46% +0.01%
==========================================
Files 475 475
Lines 26017 26018 +1
Branches 6934 6934
==========================================
+ Hits 20670 20674 +4
+ Misses 5347 5344 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
andypalmi
left a comment
There was a problem hiding this comment.
Thanks Costin, I traced the agent code and I don't think we can merge this as is.
-
A repeat start is a full teardown, not a harmless reconnect. The route sends
startto the agent (3rdPartyBroker.js L450-L457), which hits/commands/startand runsAgent.start(). That callsstop()(ends the MQTT client, uploads, clears the report interval, setsconnected=false), thenprimeCache()andconnect(), which re-fetches credentials and opens a new client. Calling start on an agent that is already collecting takes the broker offline for that whole window and drops any messages in it. -
This now hits every caller. The route used to be a safe no-op for a running agent. With this change every API caller, including the broker lifecycle tool, takes that hit, not just the toggle.
-
Concurrent starts can leak a client.
start()andconnect()are not serialized. Two overlapping starts both seethis.client, both stop, then both connect. The second overwritesthis.clientandthis.reportInterval, so the first client and its interval are never cleaned up. The route doesn't dedupe, so a double click or two users can trigger it. -
Failures look like success. Both drivers swallow errors from the start command (driver-docker, driver-k8s), so the route replies 200 even if the agent never resumed.
-
Suggestion. The no-op is right when the agent is actually collecting, and the bug is that
runningon the row doesn't tell us that. Could the route ask the agent for its real state and only sendstartwhen it isn't connected? The agent's/statusreturnsconnected, and both drivers expose it (docker, k8s) throughapp.containers.getBrokerAgentState. Then toggling on after a stop or the 24h auto-shutdown resumes collection, and a start on a healthy agent stays a no-op.
Sending start to an agent that is already collecting tears it down and reconnects, taking the broker offline for that window. Instead of always re-sending start for a running agent, ask the agent for its real state and only send start when it isn't connected. A stopped or auto-shut-down agent resumes; a healthy one stays a no-op.
|
Good catch, you're right, The concurrent-start leak and swallowed driver errors are in the agent/driver repos, so I'll leave those as separate follow-ups. |
When schema collection stops, either through the API or when the agent hits its own 24 hour auto-shutdown, the agent pauses but the platform still has it recorded as running. Switching the "Monitor Broker for Smart Schema Suggestions" toggle back on after that did nothing: the start path saw "already running" and skipped sending the agent a start command, so it was never told to resume. The toggle looked on, nothing collected.
The team-broker start handler now always sends the agent a start command when it exists and isn't mid-start or mid-suspend, instead of no-oping when it looks running. The agent treats a repeat start as a reconnect, so it's safe when it was already collecting, and it resumes collection otherwise. No new agent states or migration, so the agent lifecycle is untouched, and it covers the 24 hour auto-stop case too.
Checked locally against the team broker: start then stop leaves the agent connected=false, and before this a second start left it that way; now the second start brings connected back to true.
Closes #8995