Stackchunk reuse old gen - #228
Conversation
|
👋 Welcome back franz1981! A progress list of the required criteria for merging this PR into |
|
❗ This change is not yet ready to be integrated. |
|
@fisk hi! Let me know wdyt of this change 🙏 |
Glancing over this, my main question isn't really if we can make stack chunks further reusable for Serial/Parallel (of course we can), but rather whether we should do that or not. We had more of this kind of opportunistic GC-specific shenanigans earlier on, including allowing G1 to do some more reuse in situations when it's safe. The risk is that you end up without a clear model for when stack chunks are reused and end up with different models for different GCs. So we sort of walked away from that in favour of simplifying the code. It's also worth mentioning that now G1 will be the default, even in small environments (cf. https://openjdk.org/jeps/523). That means we have an optimization for users that explicitly select Serial/Parallel in constrained environments. And adds back different modes to an area that is already rather complicated, which we have previously walked away from. On a tangentially related note, we are currently playing around with adding a form of reference counting to the old generation of ZGC and should be able to eagerly reclaim old stack chunks easily without the churn typically induced by major collections. Sounds like it would help with the problem you described. Just saying! |
|
Hi @fisk, thanks a lot for taking the time to review this and for the deep context 🙏. Answering with quotes:
As a JDK maintainer (which I'm not, but I can certainly imagine...), I agree that baking GC-specific paths into the freeze/thaw core is not ideal. The parallel/serial focus was a limit on my side—specifically my GC knowledge—to make this patch work safely, not a declaration that these are the only collectors that need it. In fact, if you read the blog post, you'll see that my first attempt failed spectacularly 😅 (I hope you enjoy the read!).
I actually had this same conversation with Alan this morning, so I am fully aware—and honestly a bit scared. The problem isn't just small heaps, it's small CPU time (real CPU time, which is not the same as shared CPU quotas). The stark backpressure of STW pauses from Serial/Parallel is dreadful for tail latencies, yet it is remarkably effective at keeping memory tight and predictable on these broadly deployed, resource-constrained environments.
The ZGC reference counting news is extremely exciting! But the cost/benefit analysis here isn't just about allocation. Pooling/reusing chunks offers memory warmth: it reduces zeroing costs (where applicable), reduces cache misses, and limits the card-marking overhead of constantly replacing old chunks with new ones. To summarize my point:
|
|
Thinking about it more carefully I think that the issue on https://quarkus.io/blog/to-cache-or-not-to-cache-virtual-threads/ is relevant for jdk mainline as well without the changes on this PR nor any pooled VT i.e.
In short, the same heap footprint issue due to long running VT and tiered compilation, but caused by abundance of heap capacity |
|
Hi @franz1981, I took a look at the patch and have a few comments/questions, thanks. I read the pooling VTs experiment but I’m a bit confused by the explanation of the results that motivated this patch. The attributed reason for the two order of magnitude difference in the number of full GCs between the non-pooled and pooled versions in the 1GB heap case, is that there is severe stackChunk churn: in each GC cycle current stackChunks are promoted, later triggering new allocations and abandoning of the chunks in the old-gen. But if that’s the case, I would have expected the number of full GCs to decrease significantly with this patch applied. Instead, they only fall from 461 to 435. So seems that we are in a case where there are just too many live objects in the old-gen, regardless of reusing stackChunks or not. That matches the observation that live heap in the old-gen after each full GC was ~700MB for both baseline and with the reuse patch. Are those ~700MB stackChunk objects too or is the majority some other pool/application state? I see that you found reused stackChunks were bigger than what was needed, and that caused wasted space in old-gen of about 90MB. But that still wouldn’t explain the big difference in number of full GCs. In general we want to avoid re using chunks in the old-gen because that forces both freeze and thaw into their slow paths. In particular, it means we have to transform the chunk during freeze, and apply gc barriers to each thawed frame. It’s true that we could avoid the allocation, but most of the times we should be allocating from the TLAB which is very fast and should be cache friendly. Also note that zeroing doesn’t include the stack space. About extending the patch to cover other GCs, I’m not sure how easy it is to do that with concurrent collectors. One thing that we started prototyping some time ago and it’s in our list is to compress stackChunks in the old-gen. Compiled methods use fixed-size frames, but many frame slots may be dead or unused at a particular freeze point. It will also help with cases where the size of the chunk is bigger than what is actually stored, either because it was reused previously (as you run into) or because frames were thawed and fewer frames remain. |
|
Hi @pchilano, thanks for looking. To be clear on framing: this is an experimental patch, not a proposal — the article's conclusion is that it made things worse. The point of having it here is that you can run it and do your own math on it. On the Full GC count: agreed, churn can't be the whole story given 461→435. I don't know what the ~700MB live set is, and I don't have the no-pooling number either. I'll take a live histogram after Full GC in both modes plus -Xlog:gc* and report back. On the regression: the headroom explanation doesn't hold given Full GCs went down. I did profile with async-profiler and the slow path wasn't visible either, so I don't have an explanation for the 4%. Point taken on zeroing; I withdraw the warmth argument too. The case I still care about is my Aug 27 comment: a VT on a long-running HTTP request, no pooling. Once its chunk is promoted, each freeze allocates a new one and leaves the old in old gen; if it was frozen C1-sized, that's locked in until replaced. Compression covers the sizing part — does anything cover the replacement side, or is Full GC reclaiming those considered fine? |
For the long running virtual thread case, I would expect first a young gc cycle that transforms and moves the oversized chunk to the survivor space (note that we also transform chunks in this case, so gc_mode does not imply it’s in the old-gen). The next freeze will allocate a new chunk with the right size for the c2 frames. Then the next young gc cycle should reclaim the discarded chunk. |
|
thanks @pchilano for reviewing the numbers, again. |
Great, glad you got to the bottom of it. Sure, it would be interesting to see the data, so feel free to share it here or just ping me when you update the article and I’ll take a look. |
The post's original explanation of why pooling virtual threads hurt on a small heap was wrong. This reworks it around what pooling actually buys and costs, measured end to end. Pooling allocates about 8% less per request - real in every run, and it moves nothing: young collections, Full GCs, pause time and throughput all sit inside the run-to-run spread. It retains about 60 MB of live heap, roughly 6 KB per pooled thread, for the life of the pool. That retained heap is one promoted StackChunk per pooled thread, minted during JIT warm-up when the request path was still interpreted or C1 and so froze about four times as many stack words. A chunk's capacity is fixed when it is allocated; a promoted chunk is never written into again, so later freezes go into a small tail in front of it; and only the tail is ever thawed from, so the parent keeps the frames the JVM pushed to start the thread. A pooled worker never returns out of its loop, so those frames never leave. An unpooled thread unwinds all the way out and its chunk goes with it. A diagram shows the split. Adds a note on what we saw at 900 MB: a third of pooled runs and no unpooled run fell into hundreds of Full GCs, with the collector logging that predicted promotion had exceeded free old-generation space. Reported without an explanation, because we do not have one. Credits Patricio Chilano Mateo, whose review of the original findings on openjdk/loom#228 is what led here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FsJiSTC5SqmiaLSx1qoGvh
The post's original explanation of why pooling virtual threads hurt on a small heap was wrong. This reworks it around what pooling actually buys and costs, measured end to end. Pooling allocates about 8% less per request - real in every run, and it moves nothing: young collections, Full GCs, pause time and throughput all sit inside the run-to-run spread. It retains about 60 MB of live heap, roughly 6 KB per pooled thread, for the life of the pool. That retained heap is one promoted StackChunk per pooled thread, minted during JIT warm-up when the request path was still interpreted or C1 and so froze about four times as many stack words. A chunk's capacity is fixed when it is allocated; a promoted chunk is never written into again, so later freezes go into a small tail in front of it; and only the tail is ever thawed from, so the parent keeps the frames the JVM pushed to start the thread. A pooled worker never returns out of its loop, so those frames never leave. An unpooled thread unwinds all the way out and its chunk goes with it. A diagram shows the split. Adds a note on what we saw at 900 MB: a third of pooled runs and no unpooled run fell into hundreds of Full GCs, with the collector logging that predicted promotion had exceeded free old-generation space. Reported without an explanation, because we do not have one. Credits Patricio Chilano Mateo, whose review of the original findings on openjdk/loom#228 is what led here.
The post's original explanation of why pooling virtual threads hurt on a small heap was wrong. This reworks it around what pooling actually buys and costs, measured end to end. Pooling allocates about 8% less per request - real in every run, and it moves nothing: young collections, Full GCs, pause time and throughput all sit inside the run-to-run spread. It retains about 60 MB of live heap, roughly 6 KB per pooled thread, for the life of the pool. That retained heap is one promoted StackChunk per pooled thread, minted during JIT warm-up when the request path was still interpreted or C1 and so froze about four times as many stack words. A chunk's capacity is fixed when it is allocated; a promoted chunk is never written into again, so later freezes go into a small tail in front of it; and only the tail is ever thawed from, so the parent keeps the frames the JVM pushed to start the thread. A pooled worker never returns out of its loop, so those frames never leave. An unpooled thread unwinds all the way out and its chunk goes with it. A diagram shows the split. Adds a note on what we saw at 900 MB: a third of pooled runs and no unpooled run fell into hundreds of Full GCs, with the collector logging that predicted promotion had exceeded free old-generation space. Reported without an explanation, because we do not have one. Credits Patricio Chilano Mateo, whose review of the original findings on openjdk/loom#228 is what led here.
|
@pchilano FYI quarkusio/quarkusio.github.io#2996 To consume the article with images etc etc, go https://quarkus-website-pr-2996-preview.surge.sh/blog/to-cache-or-not-to-cache-virtual-threads/ Feel free to review the adoc content - I've tried to keep the content readable to "common" users (fingers crossed) for divulgation purposes but I have created a JDK patch which can inspect the content of the stack chunks frames to make sure of what I wrote there - including detecting the compilation level. But adding such to a divulgative article was looking like an overkill to me, with the risk to just further confuse people... |
The post's original explanation of why pooling virtual threads hurt on a small heap was wrong. This reworks it around what pooling actually buys and costs, measured end to end. Pooling allocates about 8% less per request - real in every run, and it moves nothing: young collections, Full GCs, pause time and throughput all sit inside the run-to-run spread. It retains about 60 MB of live heap, roughly 6 KB per pooled thread, for the life of the pool. That retained heap is one promoted StackChunk per pooled thread, minted during JIT warm-up when the request path was still interpreted or C1 and so froze about four times as many stack words. A chunk's capacity is fixed when it is allocated; a promoted chunk is never written into again, so later freezes go into a small tail in front of it; and only the tail is ever thawed from, so the parent keeps the frames the JVM pushed to start the thread. A pooled worker never returns out of its loop, so those frames never leave. An unpooled thread unwinds all the way out and its chunk goes with it. A diagram shows the split. Adds a note on what we saw at 900 MB: a third of pooled runs and no unpooled run fell into hundreds of Full GCs, with the collector logging that predicted promotion had exceeded free old-generation space. Reported without an explanation, because we do not have one. Credits Patricio Chilano Mateo, whose review of the original findings on openjdk/loom#228 is what led here.
The post's original explanation of why pooling virtual threads hurt on a small heap was wrong: the benchmark it rested on served JSON through Jackson's default ThreadLocal buffer recycling, which pinned a buffer set per virtual thread and inflated heap occupancy and GC work in precisely the arm under test. Every figure here was re-measured with a non-ThreadLocal recycler. The article now opens with a revision notice: what was wrong, why it was revisited, what exactly changed, and a permalink to the superseded version. What the corrected measurements show. Pooling allocates about 8% less per request - real in every run, and it moves nothing: young collections, Full GCs, pause time and throughput all sit inside the run-to-run spread. It retains about 60 MB of live heap, roughly 6 KB per pooled thread, for the life of the pool. That retained heap is one promoted StackChunk per pooled thread, minted during JIT warm-up when the request path was still interpreted or C1 and so froze about four times as many stack words. A chunk's capacity is fixed when it is allocated; a promoted chunk is never written into again, so later freezes go into a small tail in front of it; and only the tail is ever thawed from, so the parent keeps the frames the JVM pushed to start the thread. A pooled worker never returns out of its loop, so those frames never leave. An unpooled thread unwinds all the way out and its chunk goes with it. A diagram shows the split. The throughput collapse the original attributed to StackChunk churn is now reported as an observation without an explanation, because we do not have one: at 900 MB a third of pooled runs and no unpooled run fell into hundreds of Full GCs, with the collector logging that predicted promotion had exceeded free old-generation space. Credits Patricio Chilano Mateo, whose review of the original findings on openjdk/loom#228 is what led here.
3 participants
This is an experimental patch, used to write https://quarkus.io/blog/to-cache-or-not-to-cache-virtual-threads/
The purpose of the patch is to enable stack chunk reuse for collectors which have not concurrent phases and are usually suitable for "small heaps".
The patch doesn't try to be smart, as it doesn't have any heuristic to decide if is worthy to reuse only C2 compiled stack frames nor any average mean exponential decay (or similar) algorithm to decide IF is better to stick with a specific stack chunk capacity, and allow detachment to happen.
Which means that, similarly to heuristics built for native allocators, which have the same exact pooling problem vs unknown user-driven lifecycle usage, maybe there's a way to make it right.
I'm not (at all!) a GC expert, but I hope the article help to clarify what's the intent 🙏
That said, pooling (as caching), is one (if not THE) most complex CS problem - since none knows the future - and specifically for FJP, it introduces a "stealthy" scheduling advantage to pooled VTs which I haven't (on purpose) mentioned in the article, as unparking a VT from a carrier can enable local (with signaling) submission, which was the primary reason I was playing with fire trying to understand and dissect scientifically the pros/cons of the known rule "never pool virtual threads".
@theRealAph @tstuefe this is the patch I've mentioned. If it's ugly, it's all my fault :P
Progress
Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.org/loom.git pull/228/head:pull/228$ git checkout pull/228Update a local copy of the PR:
$ git checkout pull/228$ git pull https://git.openjdk.org/loom.git pull/228/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 228View PR using the GUI difftool:
$ git pr show -t 228Using diff file
Download this PR as a diff file:
https://git.openjdk.org/loom/pull/228.diff