Run the out-of-band management power-state task in a managed context to fix a connection leak - #14090
nagaboinaramgopal wants to merge 1 commit into
Conversation
… context The per-host power-state sync submitted to the background executor was a plain Runnable, so the database connection its DB work acquires on the worker thread was never released back to the pool. Over time one leaked connection per configured host per run exhausted the pool (HikariPool active reaching maxActive), and the management server stopped serving requests. Make PowerOperationTask a ManagedContextRunnable, like the poll task that submits it, so the managed context releases the connection when each run finishes. Fixes: apache#13382
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #14090 +/- ##
============================================
- Coverage 19.78% 19.78% -0.01%
+ Complexity 19995 19992 -3
============================================
Files 6371 6371
Lines 575909 575909
Branches 70509 70509
============================================
- Hits 113950 113933 -17
- Misses 449526 449545 +19
+ Partials 12433 12431 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@nagaboinaramgopal did you test this before and after with an environment? The issue should have been already fixed by one of the commits from 4.22.1.1 release. Can your please share how you reproduced the issue in a real environment? |
You are right, thanks. This is already fixed on main by the fix for the connection leak in scoped config retrievals, which covers the OOBM background task's per-run scoped config reads. Closing this as redundant. |
2 participants
Description
The out-of-band management power-state poll task submits a per-host power status task to a background executor. That task, PowerOperationTask, was a plain Runnable, while the poll task that submits it is a ManagedContextRunnable. Running the per-host work outside a managed context means the database connection its DB work acquires on the worker thread is never released back to the pool. Over time this leaks one connection per configured host on every run, and once the pool reaches maxActive the management server stops serving requests with "Connection is not available, request timed out".
This makes PowerOperationTask a ManagedContextRunnable, matching the poll task that submits it, so the managed context releases the connection when each run finishes.
Fixes: #13382
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
N/A
How Has This Been Tested?
Added a unit test that runs the task and checks it is a ManagedContextRunnable and that running it still performs the power operation, so the per-host work now runs inside a managed context and the delegation is unchanged.
How did you try to break this feature and the system with this change?
The change only wraps the existing per-host work in a managed context and does not change what it does. The sibling poll task that submits this task already uses the same managed-context base, so the two now behave consistently.