Summary
On an install with a few hundred accumulated sessions, hapi hub pegs one core at ~100% CPU indefinitely and every HTTP request queues behind the blocked event loop. Clients (phones, other devices, and the local CLI alike) see connect-but-never-respond timeouts.
The hub is not hung — it is re-scanning the entire sessions table several times per second. Cost scales linearly with session count, not with load, so an install slowly walks itself into a wedged hub as sessions accumulate. Restarting does not help: the new process resumes scanning the same rows within seconds.
Pruning old sessions is the only thing that fixed it, and the improvement is proportional to the row count removed — which I think identifies the hot path fairly precisely.
Environment
- hapi
v0.29.0 (latest at time of writing), installed via npm, @twsxtd/hapi-linux-x64
- Node v20.20.0, Linux 6.8.0, 16GB box
- hub run under pm2, ~30 local agent sessions (claude/codex) polling it
~/.hapi/hapi.db 2.46 GB, 521 sessions, 480k messages
Symptoms
TCP accepts fine, the application never answers:
connect=0.00027s <- TCP handshake OK
start=0.000000 <- no application response
code=000 <- 10s timeout
Meanwhile the hub's own access log shows requests being served in ~1ms:
--> GET /cli/sessions/<id>/messages?afterSeq=7454&limit=200 200 1ms
--> GET / 200 1ms
So requests are not slow — they are queued. ss confirms it: 73 established connections, nearly all with a non-empty Recv-Q (816, 1027, 1231, 7096 bytes …) — inbound request bytes sitting in kernel buffers with nobody calling read(), because the event loop never yields.
Process state: main thread state=R, 100–107% CPU, sustained for days. usage_events had stopped growing ~24h before I looked, while the process kept burning a full core.
Evidence
1. It is all SQLite page reads, not socket work:
$ strace -c -p <hub pid>
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
92.82 0.239998 7 33591 pread64
4.40 0.011371 16 690 madvise
1.87 0.004828 13 351 8 futex
0.42 0.001090 33 33 sched_yield
Sustained throughput measured from /proc/<pid>/io: ~129 MB/s of rchar, ~32k reads/sec, indefinitely, with zero rows written.
2. The reads are a small page set, read over and over:
Capturing pread64 offsets for 4 seconds and converting to page numbers (offset / page_size + 1):
total reads: 22068 distinct pages: 2122 page range: 1 - 600364
~10x re-read of the same 2122 pages in 4 seconds.
3. Mapping those pages to tables via dbstat:
sessions 14567 reads <- 66%
messages 7061 reads
idx_messages_session 208 reads
sqlite_schema 205 reads
sqlite_autoindex_sessions_1 27 reads
sessions is 919 pages / 3.8 MB / 521 rows. Being credited 14567 reads in 4 seconds means the whole table is being scanned several times per second (and this is under strace, which slows the process down — unthrottled frequency is higher). Each pass pulls every row's metadata / agent_state JSON to be parsed.
The messages reads look like the normal afterSeq polling — those go through idx_messages_session and complete in ~1ms. The sessions scans are the ones eating the core.
Scaling proof
Deleting sessions untouched for >14 days (nothing else changed — same version, same clients, same box):
|
before |
after |
| sessions |
521 |
214 |
| messages |
480,375 |
309,356 |
| hub CPU |
107% |
~50% |
| read rate |
129 MB/s |
54 MB/s |
TTFB on GET / |
5–12 s |
0.0025 s |
| hapi.db |
2.46 GB |
1.05 GB |
CPU and read rate both fell to ~41% of their prior value, matching the row-count ratio (214/521 = 41%) almost exactly. That linearity is what makes me confident the hot path is a full scan of sessions rather than anything data-dependent.
A ~3000x latency improvement from deleting rows nobody was querying suggests the scan is unconditional — not driven by what clients actually request.
Why restart doesn't help
Restarting the hub gives ~20 seconds of responsiveness, then it returns to 100% CPU as the fleet reconnects. The trigger is the row count in the database, so it survives process restarts. This makes the failure look like a hang to operators, and pm2 list happily reports online throughout.
Suggested directions
- Cache the session list in memory and invalidate on write, instead of re-reading
sessions per request/event.
- Avoid selecting the heavy JSON columns (
metadata, agent_state, todos, team_state) for list-shaped queries — a list view rarely needs them.
- Consider a retention/archival story for old sessions. There is currently no documented way to prune; I had to delete rows directly with
ON DELETE CASCADE.
Side note
sessions.active appears unmaintained: the table had 0 rows with active=1 while 30 agent processes were live against that hub. That made it useless as a safety predicate when deciding what was prunable (I fell back to updated_at). This looks related to #842.
Summary
On an install with a few hundred accumulated sessions,
hapi hubpegs one core at ~100% CPU indefinitely and every HTTP request queues behind the blocked event loop. Clients (phones, other devices, and the local CLI alike) see connect-but-never-respond timeouts.The hub is not hung — it is re-scanning the entire
sessionstable several times per second. Cost scales linearly with session count, not with load, so an install slowly walks itself into a wedged hub as sessions accumulate. Restarting does not help: the new process resumes scanning the same rows within seconds.Pruning old sessions is the only thing that fixed it, and the improvement is proportional to the row count removed — which I think identifies the hot path fairly precisely.
Environment
v0.29.0(latest at time of writing), installed via npm,@twsxtd/hapi-linux-x64~/.hapi/hapi.db2.46 GB, 521 sessions, 480k messagesSymptoms
TCP accepts fine, the application never answers:
Meanwhile the hub's own access log shows requests being served in ~1ms:
So requests are not slow — they are queued.
ssconfirms it: 73 established connections, nearly all with a non-emptyRecv-Q(816, 1027, 1231, 7096 bytes …) — inbound request bytes sitting in kernel buffers with nobody callingread(), because the event loop never yields.Process state: main thread
state=R, 100–107% CPU, sustained for days.usage_eventshad stopped growing ~24h before I looked, while the process kept burning a full core.Evidence
1. It is all SQLite page reads, not socket work:
Sustained throughput measured from
/proc/<pid>/io: ~129 MB/s ofrchar, ~32k reads/sec, indefinitely, with zero rows written.2. The reads are a small page set, read over and over:
Capturing
pread64offsets for 4 seconds and converting to page numbers (offset / page_size + 1):~10x re-read of the same 2122 pages in 4 seconds.
3. Mapping those pages to tables via
dbstat:sessionsis 919 pages / 3.8 MB / 521 rows. Being credited 14567 reads in 4 seconds means the whole table is being scanned several times per second (and this is under strace, which slows the process down — unthrottled frequency is higher). Each pass pulls every row'smetadata/agent_stateJSON to be parsed.The
messagesreads look like the normalafterSeqpolling — those go throughidx_messages_sessionand complete in ~1ms. Thesessionsscans are the ones eating the core.Scaling proof
Deleting sessions untouched for >14 days (nothing else changed — same version, same clients, same box):
GET /CPU and read rate both fell to ~41% of their prior value, matching the row-count ratio (214/521 = 41%) almost exactly. That linearity is what makes me confident the hot path is a full scan of
sessionsrather than anything data-dependent.A ~3000x latency improvement from deleting rows nobody was querying suggests the scan is unconditional — not driven by what clients actually request.
Why restart doesn't help
Restarting the hub gives ~20 seconds of responsiveness, then it returns to 100% CPU as the fleet reconnects. The trigger is the row count in the database, so it survives process restarts. This makes the failure look like a hang to operators, and
pm2 listhappily reportsonlinethroughout.Suggested directions
sessionsper request/event.metadata,agent_state,todos,team_state) for list-shaped queries — a list view rarely needs them.ON DELETE CASCADE.Side note
sessions.activeappears unmaintained: the table had 0 rows withactive=1while 30 agent processes were live against that hub. That made it useless as a safety predicate when deciding what was prunable (I fell back toupdated_at). This looks related to #842.