Repository navigation
ShapeId exhaustion aborts the process after ~475 TypeScript transpiles (~2.26M ids minted per transpileModule, ~45 per AST node) #10868
Description
Activity
Census: why those 2.26 M ids are minted
Instrument:
PERRY_SHAPE_MINT_DIAG=<path|1>(object/shape_mint_census.rs, default-off, one relaxed atomic load when unset, dumped atatexit). It labels every mint by which ofshape_descriptor_ensure_with_holes' six identity facts moved against the family already indexed under the same keys-array address, and separately records the ordered key-name list's content hash — so "a new layout" and "the same layout at a new array address" are different rows.The headline
ts.transpileModuleof a 1,201-line generated source, typescript 5.8.2 compiled from source by perry, output byte-identical to node:1 transpile 2 transpiles ensure()calls2,574,498 5,116,060 memo hits 286,629 571,769 MINTS 2,287,869 4,544,291 distinct key-NAME lists 10,867 10,867 distinct keys-array ADDRESSES 1,116,873 2,189,037 mints per distinct key-name list 210.5 418.2 identical(afacts_keymemo failure)0 0 Slope 2,256,422 ids per transpile, and the number of distinct layouts is constant at 10,867 while mints grow linearly.
Perry mints 2.26 M ShapeIds per transpile to describe 10,867 layouts.
That reframes this from a budget problem into a defect.
What it is not
- Not a
facts_keymemo failure.identicalis 0 on every workload measured. - Not genuinely new layouts.
fresh_keys_new_listis 0.1 % of mints, and the layout count does not grow with the work. - Not unavoidable churn from short-lived objects — see the
poolcontrol below, where an identical creation pattern costs zero ids per object.
What it is: a direct-mapped cache thrashing, at two ids per miss
TRANSITION_CACHE_SIZE = 16384, direct-mapped (transition_cache_slot(prev_shape_id, key_id) & 16383, one edge per slot):1 transpile 2 transpiles lookups 1,399,817 2,780,162 hits 117,788 (8.4 %) 235,227 (8.5 %) miss_COLLIDE(a different live edge holds the slot)1,221,842 2,450,003 miss_empty(cold)60,186 94,931 inserts 1,137,572 2,265,528 evicting inserts 1,093,019 (96.1 %) 2,193,725 (96.8 %) 87 % of all lookups collide with a different live edge; only 4 % are cold. The edges are not new — they are evicted and re-learned.
Every transition-cache miss costs exactly TWO ShapeIds, and the arithmetic closes:
2 × 1,137,572 = 2,275,144against 2,287,869 mints, the remainder being the 2.4 % of other causes. The two are the top two call sites, one line each:object/field_set_by_name/tail.rs:1022— 1,077,000 mints, allfresh_keys_known_list. This is the copy-on-write clone: a shared keys array is cloned before the add, andset_object_keys_array(obj, cloned)publishes a shape for the clone whose ordered key list is identical to its predecessor's. An intermediate state with no consumer.object/field_set_by_name/tail.rs:1102— 1,073,301 mints, allkey_count. The append that follows.
Cause split overall:
fresh_keys_known_list49.6 % +key_count47.8 % = 97.4 %;gen_unique2.2 %,gen_deterministic0.2 %,fresh_keys_new_list0.1 %. Retirement confirms the waste: 2,237,158 retirements against 2,287,869 mints, 1,094,525 of them within 2 mints of birth.The controls — the sharing machinery works; tsc's working set does not fit
Six fixtures, each output byte-identical to node, same instrument:
fixture what it does mints sameN objects, 8 adds, statically foldable into one anon-shape class 1, at N = 1 k and 10 k poolN objects, the same 8 field NAMES, added through a DYNAMIC key so HIR cannot fold — the AST-node-factory shape 4,534 / 4,536 / 4,564 at N = 1 k / 10 k / 100 k — constant; cache hits 84.6 % → 98.2 % → 99.8 % deldelete+ add per object5,516 → 14,516 = 1 id/object, linear accdObject.defineProperty, data descriptor3,002 → 30,002 = 3 ids/object, linear accObject.defineProperty, accessor4,002 → 40,002 = 4 ids/object, linear diff8 adds, names unique per object 22,518 → 184,516 = 18.45 ids/object; miss_COLLIDE8,624 → 199,949, evictions 31.6 % → 82.0 %poolis the existence proof: the same creation pattern as an AST node factory — same field names, dynamic key, unfoldable — costs zero ids per object once warm. The machinery is capable. tsc's working set simply does not fit in 16,384 direct-mapped slots.A second, independent defect: the descriptor key-add path never publishes an edge
On
acc/accdthe transition cache reports 1,000 lookups, 0 hits, 0 inserts. The descriptor key-add path consults the cache and never populates it, so no object can ever reuse another object's edge. That is whyObject.definePropertycosts a flat 3–4 ids per object forever, and it reads like an oversight rather than a trade.The fork cascade — why this is geometric rather than merely wasteful
One fork is not one wasted id.
deterministic_semantic_generation(#10287) is a pure function of (predecessor ShapeId, key, attrs). Once an object has been forked onto a private keys array by the copy-on-write clone, its predecessor ShapeId is private to it — so the deterministic generation is now deterministic in a value no other object shares, and every subsequent descriptor transition on that object mints too. This is visible onacc: of its 4 ids per object, 2 aregen_deterministic, i.e. #10287's shared-generation path minting per object because the fork upstream already made sharing impossible.One fork makes every later transition on that object private.
Two levers, neither proposed here
- The transition cache's size and associativity — the direct cause of the 96 % eviction rate. Note perf: one Object.defineProperty sends every later store on that object to the slow path (zod v4 schemas 29× bun; top OpenCode startup cost) #10287 measured a 262,144-entry table as a negative (it made zod worse), so this needs the harder argument.
- The copy-on-write clone's intermediate publish at
tail.rs:1022— half of every miss's cost, publishing a layout identical to the one it replaces.
Each needs its own before/after on tsc and on all six controls, with the layout count (10,867) held as an invariant: if that number moves, shape identity has been altered rather than mint volume.
Reproduction
Workloads and fixtures on perrymaster:
/root/fr/work/tscwork.ts(typescript 5.8.2 from source),/root/fr/mx/{same,pool,del,accd,acc,diff}.ts. Compile withPERRY_NO_CACHE=1 PERRY_NO_AUTO_OPTIMIZE=1 PERRY_RUNTIME_DIR=<worktree>/target/release, then run withPERRY_SHAPE_MINT_DIAG=1.Also measured, and worth recording because it removes a false urgency: zod does not burn the band at all. 23,370 mints at 200, 1,000 and 2,000 iterations — identical, a one-time warm-up cost.
- Not a
Correction to the census: the descriptor key-add path DOES publish and replay edges
The "second, independent defect" above is withdrawn. The
acc/accdreceiver is{ a: i, b: i + 1 }— two fields — andINLINE_SLOT_FLOOR = 2, so the literal allocates exactly two inline slots andObject.defineProperty(o, "c", …)landscat index 2: an overflow target.ensure_key_in_keys_arraydeclines overflow targets on both sides by design (slot_idx < alloc_limiton the probe,new_index < inline_capacityon the teach — the comment there says why: a keys-only install writes no value, so the overflow entry an overflow edge implies would never be created). The 1,000 lookups / 0 hits / 0 inserts were that exclusion firing, not a path that never publishes.Same two fixtures with a one-field receiver
{ a: i }, soclands inside the floor. Same instrument, same binary, output identical to node:fixture N mints lookups / hits / inserts acc1(accessor)1,000 7 1,000 / 999 / 1 acc110,000 7 10,000 / 9,999 / 1 accd1(data descriptor)1,000 6 1,000 / 999 / 1 accd110,000 6 10,000 / 9,999 / 1 Constant in N. #10287's descriptor edge works, including its deterministic generation (
memo hits1,999 at N = 1,000 onacc1— two per object, both served). What the original rows measured is the overflow-target exclusion, a deliberate trade, and it is not material to this issue: the whole descriptor family is ~0.6 % of tsc's mints (keys_array.rs:2635,367 +descriptor_state.rs:664/6707,401 of 2,287,869). No PR for it. Whether overflow-target edges should be taught the way the[[Set]]tail already teaches them is a separate question with its own risk (the value / keys-only asymmetry that comment names), and nothing here turns on it.The fork cascade stands, with the example read correctly: on
acc(two fields) the twogen_deterministicmints per object are per object because the copy-on-write fork gave that receiver a private predecessor id; onacc1the replayed edge keeps the predecessor shared and the same two generations are memo hits. The cascade is a consequence of the fork, and a fork happens on every transition-cache miss. Lever (ii) halves what a fork costs; it does not prevent one.Lever map and who is on what
Three parallel sessions have now touched this issue, so here is the split, in the one place every session reads. A marker file inside a worktree cannot defend a worktree — this can at least make the race visible.
lever what it is status instrument PERRY_SHAPE_MINT_DIAG— the per-mint cause/key-list/call-site/cache-outcome census the numbers above come fromPR #10885, ready for review. Lands first so every fix's before/after uses the same instrument (i) transition-cache size / associativity 16384 entries, direct-mapped, 96.1 % evicting inserts on tsc — the direct cause of the eviction rate free. Note #10287 measured a 262,144-entry table as a negative (it made zod worse), so this needs the harder argument, not the obvious one (ii) the copy-on-write clone's intermediate publish tail.rs:1022, 1,077,000 mints on one transpile, every onefresh_keys_known_list— a shape published for a clone whose ordered key list is identical to its predecessor'sclaimed, branch feat/cow-single-publish(iii) the descriptor key-add path never publishes a transition edge acc/accd: 1,000 / 10,000 cache lookups, 0 hits, 0 inserts; 3–4 ids per object, linear and unboundedclaimed by me, taking it next Also mine, and unrelated to the mint rate:
feat/field-representation(object-model design step 5). Its build is suspended pending this issue; only its measurement commits exist.(ii) and (iii) are genuinely independent, which is why they can run in parallel. (ii) is about a path that does use the transition cache and pays two ids per miss; (iii) is about a path that consults the cache and never populates it, so no object can ever reuse another object's edge and no change to the cache's size, associativity or publication discipline can help it. A path that never publishes an edge reads like an oversight rather than a trade.
Why any of these is worth more than its own line count — the fork cascade.
deterministic_semantic_generation(#10287) is a pure function of (predecessor ShapeId, key, attrs). Once an object has been forked onto a private keys array, its predecessor ShapeId is private to it, so the deterministic generation is now deterministic in a value no other object shares and every subsequent descriptor transition on that object mints too. Onacc, 2 of the 4 ids per object aregen_deterministic— #10287's shared-generation path minting per object because a fork upstream had already made sharing impossible. One fork makes every later transition on that object private, which is why the rate is geometric in practice rather than merely wasteful.Invariant for every fix here: the distinct key-name list count must not move. It is 10,867 on one
transpileModuleand 10,867 on two. If a change moves it, that change altered shape identity, not mint volume, and the mint delta means nothing.And the acceptance set is the six controls plus tsc, all on the same instrument:
same(1 mint),pool(constant — the existence proof that the machinery works),del(1/object),accd(3/object),acc(4/object),diff(18.45/object). A change that helps tsc and regressespoolordiffis not a fix.- added a commit that references this issue
on Sep 21, 2026 Lever (iii) root-caused: one predicate, and a footprint dial that governs shape publishing
Measured rather than read: I instrumented every decline reason in
define_append_transition_eligibleand at the edge-publication site. Onaccd(a data-descriptor install per object), N = 1,000, output byte-identical to node:defineProperty key-add, edge outcomes: no publish: new_index >= inline_capacity 1000 eligible 1000The receiver is eligible and the cache lookup does run. Exactly one predicate refuses, at
object_ops/keys_array.rs:306:new_index < inline_capacity.Why it refuses every time — and the finding that outlives the fix
INLINE_SLOT_FLOOR = 2(object/mod.rs:49). #7916 moved it 8 → 4 → 2, and its own doc comment is explicit about what it is for:the floor is purely a growth-headroom dial for objects that gain properties by name after birth.
So a two-field literal
{a, b}allocates exactly two slots and has zero growth headroom. The first key added after birth lands atnew_index == inline_capacity == 2, spills to overflow, and the transition edge is never published. Every receiver forks onto a private keys array, and from there the fork cascade does the rest.A dial tuned purely for footprint — documented as having nothing to do with shapes — silently governs whether transition edges get published at all. Neither subsystem's author could have seen that from their own side: #7916 was reasoning about bytes per object and correctly proved
alloc_limitis a fixed point of the allocation; the transition cache was reasoning about edges and reusedalloc_limitas a proxy for "can this key be shared". The coupling is invisible from both ends. Worth more to the next reader than the fix, and worth writing down before the next person tunes a footprint dial.Do not raise
INLINE_SLOT_FLOORIt is the obvious fix — the refusal is
new_index < inline_capacity, so 4 or 8 makes the symptom vanish — and it is wrong three times over:- It trades footprint for shapes. This campaign runs under a standing ≤ +10 % peak-RSS budget, and perf: object representation is now the binding constraint on the retain cluster (72 bytes per 2-field literal; 216 MB written to store 48 MB) #7916's move was a measured footprint decision. Reversing it to fix a shape-publishing bug spends their win to buy ours.
- It moves the cliff, it does not remove it. At a floor of 4, a four-field literal has zero headroom and forks on its first added key. The predicate is still wrong; it just refuses later.
- The correct behaviour already exists next door.
field_set_by_name/tail.rs:1078— the[[Set]]tail's overflow arm — publishes the edge with no inline-capacity gate at all, and its comment gives the reason in the words the fix needs: "so the next object with this exact predecessor ShapeId that adds the same key hits the fast path". One path handles this situation; the other refuses it. That is missing wiring, not a missing capability.
It also explains the
poolcontrol:poolgrows by[[Set]], so its overflow adds publish edges and its mints stay constant at 4,534 however much work it does. Same object shape, same overflow, opposite outcome — decided only by which path did the add.It is two changes, and it must be predicted as two
The adopter gate refuses overflow-located edges as well (
keys_array.rs:~200and the first-key arm at~66):// An overflow target stays on the private path below: a keys-only // install (an accessor claiming its slot) writes no value, so the // overflow entry such an edge implies would never be created. if next_keys != 0 && slot_idx < alloc_limit && cached_target_fits(target_shape_id, alloc_limit)
That reason is correct for accessors and over-broad for data. An accessor install claims a keys slot and writes no value, so adopting an edge whose slot lives in overflow would imply an overflow entry that never gets created. A data descriptor does write the value, so the same objection does not apply to it.
So lever (iii) is: publish the edge when the install writes a value (mirroring
tail.rs:1078), and let the adopter take an overflow-located edge in that same case. Publishing alone changes nothing, because nobody can adopt — it would measure as a wash and read as "the idea didn't work", when half the mechanism simply wasn't built.cached_target_fitsstays either way: it guards a different hazard (a target whose live inline bound exceeds this receiver's allocation, which would have the collector trace past the end of the object).The acceptance criterion is differential, not directional
accd(data descriptor, 3 ids/object, linear) must collapse to constant.acc(accessor, 4 ids/object) must NOT improve. The accessor keeps the private path by design.
A change that improves both has broken the keys-only-install argument rather than fixed the data one, and that is the result to watch for. Plus the usual: all six controls and tsc on the same instrument, and the distinct key-name list count held at 10,867.
Step 2.5 — canonical shape identity: what it requires
Charter home:
OBJECT_MODEL_SINGLE_PATH_DESIGN_2026-09-20.md§L8.1. (The section numbers cited below — §4, §7 — are the original doc's and are stable.)10,867 distinct key-name lists. 1,116,873 distinct keys-array addresses. One
transpileModule.That is #10868 in one line, and it is the whole of it.
What is actually broken
There is an authoritative store and it never fails.
ShapeTableInner::by_factsis an unboundedHashMap<facts_key, IdList>, consulted on everyshape_descriptor_ensure_with_holes. The census'sidenticalrow fires exactly when a mint happens despite an exact-facts sibling already existing; it reads 0 on zod, on tsc, and on all six controls. Same facts ⇒ same ShapeId already holds, always.So the defect is not a missing store. It is that the authoritative table dedups on a key that a lossy cache produces:
facts_keyincludes the keys-array address, and two objects share a ShapeId only if they share an address.Three facts from source, all of which belong in the charter:
-
ShapeRecordhas no transitions field.keys, semantic_generation, logical_key_count, live_inline_slot_count, hole_count, flags, _pad— 32 bytes, const-asserted.ShapeTableInnerisindices | by_facts | families. The design doc §4's "transitions out of this shape (add key / delete key / descriptor change → next shape)" was never built; the 16,384-entry direct-mapped cache is the only transition structure that exists. -
Exactly three sites produce a shared keys array, and they are not equally safe:
- Birth (
object/mod.rs:724) — objects born with their key list. A direct-mapped inline cache with an unboundedshape_cache_overflowHashMap behind it. Two-tier, lossless. Evidence:samemints 1 id;poolmints a constant 4,534 at any N. - Growth (
object/mod.rs:1113, reachable only fromtransition_cache_lookup/_insert) — objects grown by name. 16,384 direct-mapped entries, nothing behind them. Evidence: 96.1 % evicting inserts, 87 % of lookups colliding with a different live edge. for_in_stable.rs:212— a for-in stability path, not a general producer.
So born-with-keys identity is canonical by construction; grown-by-name identity is cache-determined. The two-tier pattern the growth path needs already exists in this codebase, on the birth path.
- Birth (
-
deterministic_semantic_generationinherits forks, it does not cause them. It is a pure function of (prev ShapeId, key, attrs) with no capacity — but keyed on the predecessor, so once the predecessor is private the generation is deterministic in a value no other object shares. That is the cascade, and it is why an accessor install costs 4 ids and not 1.
The two options against the exit sentence
The criterion is "same layout + same facts ⇒ same ShapeId, however the object was built". The last clause is the One Path part: a shape compare replaces per-site proofs only if two objects that are the same layout compare the same — born from literal site A, literal site B, or grown by name.
Option 1 — an overflow store behind the growth cache. Stops eviction forks. It does not merge a born
{a,b}with a grown{a,b}, nor two literal sites whose static keys arrays differ but whose key lists are equal: identity stays address-keyed, just losslessly. So it meets the criterion only if address-distinct-but-layout-equal arrays are rare in practice — which is a measurement, below.Option 2 — a content-canonical key. Meets the criterion by construction. It is also what the design doc's own table rule asks for — "no table keyed by an object's address" — and there is a second, independent argument for it that has nothing to do with identity:
move_shape_family: "The accelerator was keyed with the OLD address; the other five facts never change under the collector."Every GC that moves a keys array must remove each id from
by_factsunder the old facts key and re-insert it under the new one, andscan_shape_table_rekey_mutwalks the families to do it. A content key is GC-invariant, so option 2 deletes that entire rekey pass.facts_keyon an address is precisely the derived-structure-under-a-moving-collector hazard the rule exists for.The two design points, confirmed from source
- An incremental content key is feasible, and the shape exists twice already.
deterministic_semantic_generationis already a SplitMix64 fold of (prev shape id, key bytes, attrs); the census'skey_list_content_hashis already a position-sensitive FNV fold. Making the layout key a rolling function of (predecessor's layout key, added key) is a re-association, not new machinery. Then a transition-cache miss costs oneby_factsprobe with an O(1)-computed key — not a mint and not a key-list walk — which demotes the 16,384-entry cache to a true accelerator whose evictions fork nothing. - One canonical keys array per ShapeId changes three things, and one of them is a real cost.
GC_FLAG_SHAPE_SHAREDdegenerates: every keys array a shape record owns is shared, so clone-before-mutate becomes unconditional and the flag stops being a distinction. Simpler, not harder.retire_owned_shape_siblingsbecomes wrong, not merely unnecessary: today an owned array's predecessors are retired on every same-address publish; under a canonical array the predecessors are other objects' layouts and must be retained.- The cost: today an owned array appends in place, O(1). Under a canonical array every growth resolves the successor layout, and a genuinely new layout costs a copy of length k. That is once per layout, not once per object — which is the win — but an object with a unique key list makes it O(k²) over k appends. That is the pathological case and it is what dictionary mode is for; §7 already calls dictionary mode required rather than optional, and this is the argument that makes it load-bearing rather than defensive.
- Holes are in
facts_key, so the canonical key must be over the ordered list including tombstone positions — otherwise a delete could collide two different arrangements. Key order is observable in JS, so the key is over the ordered list:{a,b}and{b,a}must stay distinct. The census hash is already position-sensitive (it multiplies after each fold), and that property needs a test that fails without it.
How option 1 gets its number — and the trap in getting it
The question is: of the 1,116,873 addresses, how many survive under lossless edges?
The naive measurement is wrong and would flatter option 1's opposite. Counting today's distinct
(prev_shape, key)edges gives a number keyed on predecessors that have already forked — 1.1 M of them — so it reports the fork, not the fix, and would say "option 1 leaves ~1.1 M" as an artifact. The error is recursive: fewer forks upstream means fewer distinct predecessors means fewer distinct edges.The honest instrument is a shadow simulation: run the real program, and alongside it maintain a parallel id space under each option's rules — birth arrays as roots,
lossless_edge[(shadow_prev, key)] → shadow_successorfor option 1, andintern(content_key, counts, generation, kind, holes)for option 2 with the generation recomputed over shadow predecessors. Count the distinct shadow ids. That is a faithful simulation of each option and it costs no correctness risk, because nothing branches on it.Predictions, committed before building
For option 2, on one
ts.transpileModule:- distinct shadow layout ids: 10,867 – 15,000. Above the bare key-list count because
facts_keyalso carries hole counts, kinds, live-slot bounds and descriptor generations, and a layout can legitimately appear with more than one of those; below 15,000 because the census already showsgen_deterministicat 0.2 % andgen_uniqueat 2.2 % of mints, and both collapse when their predecessors merge. - distinct keys-array addresses: ≈ the same number, one canonical array per layout.
- mints: ≈ distinct layouts, i.e. the exit criterion, against 2,287,800 today.
For option 1: I will not guess. The simulation decides it, and the decision rule is stated in advance — if option 1's shadow count is within ~2× of option 2's, the options converge and cost decides; if it is materially more, option 1 is a lever and not the step.
Before any of it: the semantics that must not move
Key order is observable. Differential tests against node for
Object.keys,for-inandJSON.stringifyorder across born-vs-grown objects of the same layout go in before the change and must be sabotage-proven — a canonicalisation that merged two orders would be a silent wrong answer in every program, not a slow one.-
Step 2.5 recommendation: option 2 (content-canonical key)
Charter: §L8.2. Facts and measurement trap: §L8.1 / comment 5763294802.
Ralph's constraint — avoid side tables — decides the architecture ahead of the counts, because the two options move in opposite directions against it. The design doc's own §4 rule says the same thing: no table keyed by an object's address; what remains is keyed by shape id or code site.
- Option 1 adds a table (an unbounded overflow store behind the growth cache) and keeps the address-keyed dedup and its per-GC rekey pass. It buys losslessness for evictions only.
- Option 2 removes tables, and the ones it removes are exactly the address-keyed ones.
The shadow simulation still runs — the 10,867–15,000 prediction needs confirming, and option 1's number tells us how much of #10868 was eviction versus address-keying, which is worth knowing whichever way it lands. But the recommendation does not hang on it.
What option 2 deletes
Every item here exists because identity is keyed on a moving address.
- The
by_factsrekey pass.scan_shape_table_rekey_mutwalks the families on every moving collection; for each idmove_shape_familydoesfacts_remove(facts_key_with_keys(old))+facts_push_back(facts_key_with_keys(new)). Its own comment states the premise — "the other five facts never change under the collector" — so a content key makes the whole pass a no-op. Deleted. - The transition cache's GC surface, entirely. Today an entry holds
next_keys(a heap address) andkey_ptr, so it needsscan_transition_cache_roots_mut, theTRANSITION_CACHE_YOUNGlog,arm_transition_cache_young,transition_entry_is_minor_relevant, and the young-log rule-1 ordering discipline. Under a content key the entry is (prev layout id, key id) → successor layout id — u32s, no addresses — and the successor's array is read from its own shape record. Root scanner, young log and arming discipline all deleted; the cache becomes a pure accelerator whose loss costs a hash probe and forks nothing. GC_FLAG_SHAPE_SHAREDand its clone-decision machinery. Three stamp sites,transition_cache_stamp_shape_shared, and thekeys_sharedbranch that every growth path opens with (tail.rs,keys_array.rs,delete_rest.rs). With one canonical array per layout, a shape-owned array is always shared and clone-before-mutate is unconditional. The flag stops being a distinction. Degenerates.retire_owned_shape_siblings' role. It exists to stop an owned array's per-append prefix descriptors accumulating. There are no owned arrays under option 2; predecessors are other objects' layouts and must be retained. Deleted (and keeping it would be wrong, not merely unnecessary).- ~1.1 M private keys arrays per transpile. Not a table, but each is a traced GC object with a header and an all-pointer payload, and they are the memory half of the same defect. One canonical array per layout.
- Two indexes become shape-keyed rather than address-keyed.
families(keys addr → ids) collapses to ~one id per array and can be a back-pointer in the record;indices(keys addr → key→slot accelerator) can key on the layout id. Neither needs the address any more.
Net: fewer tables than today, and none of the survivors keyed by an address.
The one real cost, and where dictionary mode's storage lives
An owned array appends in place, O(1). Under a canonical array, growth resolves the successor layout and a genuinely new layout costs a copy of length k. That is once per layout — the win — but an object whose key list is unique to it makes every append a new layout: O(k²) over k appends. That is the pathological case, and it is what makes §7's "dictionary mode is required, not optional" load-bearing rather than defensive.
Dictionary mode must respect the same rule. A dictionary-mode object's keys live in the object, behind the ext pointer —
ObjectMeta, which already exists and already carriesprototypeandspill(#6812) — not in a global map keyed by the object's address. That is §4's storage model as written, and it means dictionary mode adds no table: it moves a per-object key list from a shared canonical array into the per-object record it already has.The entry condition wants to be a measured latch, not a guess, and it must be proved to fire (§17: a check that cannot FIRE). The census already distinguishes the two populations —
fresh_keys_new_listis 0.1 % of mints today, so genuinely-unique key lists are rare, which is the same evidence that says the canonical path will hold for almost everything.What still has to be true before any of it
Key order is observable. The canonical key is over the ordered list including tombstone positions,
{a,b}and{b,a}stay distinct, and the differentialObject.keys/for-in/JSON.stringifyorder tests across born-vs-grown objects of the same layout go in before the change, sabotage-proven. A canonicalisation that merged two orders is a silent wrong answer in every program, not a slow one.Predictions, unchanged and still committed
Option 2, one
ts.transpileModule: distinct shadow layout ids 10,867–15,000; distinct keys-array addresses ≈ the same; mints ≈ distinct layouts, against 2,287,800 today. Option 1: no guess, and the decision rule stays — within ~2× of option 2 the counts converge and cost decides; materially more and it is a lever, not the step. The side-table criterion now says that even convergence would not make option 1 the recommendation.Shadow simulation: option 2 confirmed, option 1 is ~0, and a fourth fork source
One
ts.transpileModule, output identical to node, instrument
PERRY_SHAPE_MINT_DIAGwith the option-1 edge replay done at dump time (no
per-object state, no address keying). Charter: §L8.1–L8.3.shapes ×today today 2,288,773 1.000 option 1 (lossless edges) 2,283,638 0.998 option 2, structural (content key, no generation) 12,846 0.006 option 2, keeping today's generation values 1,193,019 0.521 distinct key-name lists 10,867 Edges replayed 1,138,062. Births 1,150,711.
Option 1 saves 0.2 %, and the reason is the births
Over half of all minted shapes are births — shapes that are never the target
of a transition edge — and option 1 cannot merge a birth. They are almost
entirely the copy-on-write clone intermediates:tail.rs:1022alone accounts
for 1,077,458 mints, every onefresh_keys_known_list, and a clone publish is
not an edge, so a lossless edge store never sees it.One caveat I cannot measure away: option 1's value is entangled with lever
(ii). With the clone intermediate removed, those 1.15 M births largely
disappear and option 1's number would change. The replay reproduces today's
code faithfully and cannot separate the two. So the honest claim is "option 1
is worth 0.2 % on the tree as it stands", not "option 1 is worthless in
principle". It does not change the recommendation, which was decided by the
side-table rule rather than by counts.Option 2: my predicted band was right about the part I predicted, and I missed a component
12,846 against a predicted 10,867–15,000 — inside the band, and ×1.18 of the
distinct-layout count, which meets the exit criterion on the structural
axis.But that is the structural number, and the total is outside my band, for a
reason worth naming rather than smoothing over.transition_object_shape_semantics
allocates its generation from the monotonicSHAPE_SEMANTIC_NEXTcounter,
not from a pure function — so option 2 cannot canonicalise those shapes at
all. #10287 made the data-descriptor path deterministic and left this one.On tsc that is 50,083 mints (2.2 %), of which 48,197 come from a single site:
prototype_chain.rs:473, theprototype_diverged → transition_object_shape_semantics(obj)
call. Adding them back:- option 2 without fixing it: up to 62,929 shapes = ×5.8 of distinct
layouts — does not meet the exit criterion. - option 2 with it fixed: 12,846 = ×1.18 — meets it.
So this is a fourth fork source, it is one call site, and it is the only
thing standing between option 2 and mints ≈ distinct layouts. The fix has a
precedent to copy exactly: make the generation a pure function of
(predecessor layout, the prototype identity, the link kind) the way
deterministic_semantic_generationalready is for descriptor installs. Two
receivers that diverge their prototype the same way over the same predecessor
should land on the same successor.Adding it to the lever map as (iv), claimed by me, sequenced inside step 2.5
rather than before it — it is only worth doing once identity is canonical,
because today its predecessors have already forked and merging its generations
would merge nothing.Why the "upper" row is an artifact and not a bound to plan against
1,193,019 keeps today's generation values, which are keyed on predecessors
that have already forked — the same recursion I flagged before running this.
It is reported for completeness and it is not the number to design against; the
structural row plus a countedgen_uniqueresidue is.- option 2 without fixing it: up to 62,929 shapes = ×5.8 of distinct
Landed in merge train 253 (v0.5.1633): the instrument and lever (iii)
Verified by content on
origin/main@0fa3915293— both PRs show closed because the merge train rewrites SHAs, so the check is the code itself:shape_mint_census::classify_mint,ensure_key_in_keys_array_for_value, and theshape-mint-diagfeature are all present.- perf(runtime): a default-off census that says WHY each ShapeId was minted (#10868) #10885 — the census.
PERRY_SHAPE_MINT_DIAG, every call site behind the non-defaultshape-mint-diagfeature, so the shipped runtime is unchanged (off-cost measured at 0.00–0.03 % on all six controls, i.e. within the method's resolution). This is now the instrument every later slice of step 2.5 is measured with. - perf(runtime): publish a transition edge when a descriptor writes a value (#10868) #10901 — lever (iii). A data-descriptor key-add now publishes and adopts an overflow-located transition edge.
The record, one
ts.transpileModule, typescript 5.8.2 from source, output identical to nodebefore #10901 after #10901 MINTS 2,288,352 2,287,800 distinct key-name lists 10,867 10,867 transition-cache hits 8.4 % 8.4 % evicting inserts 96.1 % 96.1 % tsc is flat, as predicted on the record before measuring — its mints are 97.4 %
fresh_keys_known_list+key_countfrom the[[Set]]copy-on-write path, a different fork source. The requirement for lever (iii) was must not regress, and the layout-count invariant held: identity did not move, only mint volume.Where lever (iii) does move things, it moves them to constant:
accd(a data-descriptor install per object) 3,002 / 30,002 → 5 / 5 at N = 1 k / 10 k, with −22.5 % instructions.acc(accessor) unchanged to the id, which is the row that makes it sound — an accessor is a keys-only claim whose implied overflow entry is never created, so it must keep the private path.What this does not do
Neither change makes identity canonical. #10868's exit criterion — mints ≈ distinct layouts; same layout + same facts ⇒ same ShapeId, however built — is step 2.5's job. Its plan is §L8.3; Stage 0 (the order/identity differential tests) is #10919, ready for review off
main; the shadow simulation puts the content key at 12,846 shapes against today's 2,288,773 (×1.18 of the 10,867 layouts), with lever (iv) as the one remaining fork source standing between it and the criterion.- perf(runtime): a default-off census that says WHY each ShapeId was minted (#10868) #10885 — the census.
Birth sizing coverage — what fraction of objects can be born the right size statically?
For lane 8's
L8.3.7precondition (every object of a layout has the same inline capacity).
Diagnosis only. Source onupstream/main; counts from uprobes onjs_*allocation entry
points ofPERRY_KEEP_SYMBOLS=1builds of/root/fr/work/{tscwork,zodwork}.ts.1. By site — the premise is mostly already true, from source
The compiler already sizes every construction whose key set it can see:
construction allocator capacity at birth right size statically? closed object literal {a, b}new __AnonShape_…→object_alloc_class_inline_keys_implexact key count yes other object literal js_object_alloc(0, N)—expr/object_literal.rs:561, N = the literal's key countmax(N, 2)yes, for the keys at the site new C(class)inline bump, max(field_count, INLINE_SLOT_FLOOR)—lower_call/new_alloc.rs:436declared fields yes, when the ctor's this.x =stores are unconditional (#10503's conditional add is the exception)new F()(function ctor)js_object_alloc(cid, learned_inline_field_count(cid))—class_registry/construct.rs:1111learned high-water mark runtime slack tracking — already exists Object.create(P)js_object_alloc(class_id, 0)—object_ops/prototype.rs2 no So "can the compiler know the final key set?" is the wrong question for literals and classes —
it already knows and already passes it. The real gap is growth after birth, and that is what
breaks the invariant: a 5-key literal (capacity 5) and a{}grown to the same 5 keys (capacity
2) share a layout and not a capacity.Slack tracking already exists — with two holes
Every spill records the class's true width —
spill.rscallsnote_learned_inline_fields(obj, class_id, field_index + 1)onoverflow_setandspill_set_slow, commented "Learn the class's
true width so FUTURE instances allocate it inline" — andlearned_inline_field_countfeeds it
back at allocation. Two things stop it covering the growth population:learned_inline_field_countreturns 0 whenclass_id == 0(spill.rs:462). Object
literals are allocated with class id 0, so a literal that grows after birth never learns —
every instance spills, forever.Object.createcallsalloc_synthetic_class_id()on every call (object_ops/prototype.rs),
so each object gets a fresh class id. It records learning against an id no other object
will ever use, and allocates with field count0rather than consuming the learned value. So
the obvious one-line fix — passlearned_inline_field_count(class_id)at that call — would
not help: the id has to be cached per prototype first. (A fresh class id perObject.create
also means every such object is a distinct class for shape identity, which is ShapeId exhaustion aborts the process after ~475 TypeScript transpiles (~2.26M ids minted per transpileModule, ~45 per AST node) #10868's problem
too.)
2. By object at run time
Uprobe call counts per allocation entry point.
clsinandstampedare the same births
(…_inline_keys_stampedcalls…_inline_keys_impl; the counts match exactly). With LTO some
js_object_alloccalls are inlined into runtime callers, solitcounts only calls made from
compiled user code — i.e. literal sites — which is the population wanted here.zod, per iteration (marginal, 40 vs 10 iterations — module init cancels):
entry point per iteration sizing object_alloc_class_inline_keys_impl(anon-shape literals, class instances)112.0 exact js_object_allocfrom compiled code (object literals)19.0 exact for site keys js_object_alloc_with_parent(generic allocator)79.0 as called js_arguments_object_alloc6.0 — js_new_function_construct0 learned js_object_create0 — overflow_set(a value stored to a spilled slot)22.0 — ensure_key_in_keys_array(descriptor key claims)0 (all 286 at init) — tsc, one iteration (
transpileModuleof the 400-declaration source; the second point is
still running under uprobes, so this includes module init):entry point count sizing js_new_function_construct(new F()— TypeScript's AST node constructors)67,768 learned js_object_alloc_with_parent(generic allocator)164,539 as called object_alloc_class_inline_keys_impl31,419 exact js_object_allocfrom compiled code61 exact js_object_create0 — overflow_set8,519 — ensure_key_in_keys_array5,505 — 3. Concentrated or diffuse?
By allocation position (
--opt-report's unbound-site table): two positions dominate —
call argument + return are 257 of 322 sites on zod (80%) and 1,689 of 2,698 on tsc (63%).
Growth is small against births on zod: 22 spilled stores per iteration against ~130 births,
and descriptor key claims happen entirely at module init. That is slack tracking's easy case.What this means for
L8.3.7Object.createis absent from both real workloads. perf: an Object.create() receiver costs 3-4x a literal or class receiver on every operation, including OWN reads and writes (322 vs 112 read, 288 vs 69 write) — a receiver-kind cost, not a prototype-chain cost #10905 is a real defect but fixing its
sizing buys nothing on tsc or zod.- tsc's dominant construction is
new F(), which is already learned-sized — and that is the
sharpest risk to the precondition, not the solution to it.learned_inline_field_countis a
monotonic high-water mark, so instances born before the width was learned have a smaller
inline capacity than instances of the same layout born after. Learned sizing, as built,
produces exactly the same-layout / different-capacity split canonical identity has to rule out.
V8 closes the same gap by finishing slack tracking after N instances and then fixing the map's
size; perry's learning never finishes. - The class-id-0 hole is the one that matters for literals: they are exactly sized at birth,
so they only diverge by growing, and when they grow they cannot learn.
Posted on #10868 and in
secret-tests/matrix/FINDINGS-2026-09-21.md.Follow-up: tsc's per-iteration marginal is in — and it confirms the learning-phase split
The tsc table above was a single-iteration snapshot including module init. The second point
(3 iterations) has landed, so here is the true per-iteration marginal,(3 − 1) / 2:entry point iteration 1 3 iterations per iteration js_new_function_construct(new F(), AST nodes)67,768 203,160 67,696 js_object_alloc_with_parent(generic allocator)164,539 492,233 163,847 object_alloc_class_inline_keys_impl31,419 88,397 28,489 js_object_allocfrom compiled code61 71 5 js_object_create0 0 0 overflow_set(a value stored to a spilled slot)8,519 8,997 239 ensure_key_in_keys_array5,505 5,519 7 97% of tsc's spills happen in the first iteration and then stop. 8,519 spilled stores in
iteration 1, 239 per iteration after that, while function-constructor births hold constant at
~67,700 per iteration. That is slack tracking working —new F()learns its width from the early
instances that spill, and every later instance is born wide enough not to.It is also the concrete form of the risk raised above. The early instances were born narrow and
spilled; the later ones are born wide. Same layout, two inline capacities, and the population
of narrow ones is exactly the ~8,300 objects allocated before the high-water mark settled. For
L8.3.7that pins the size of the problem on this workload: not "every object", but a bounded
learning-phase cohort per constructor — which is precisely the cohort V8 handles by finishing
slack tracking after N instances and then fixing the map's size. perry's learning has no finish
point, so the cohort is never reconciled with the steady-state layout.One correction to the zod reading: its spill rate (22 per iteration against ~130 births) is also
steady-state, since zod's marginal already excluded init. zod has no function constructors, so its
residual spills come from class-id-0 growth — the hole that cannot learn at all.Lever (iv) measured, and the capacity collision resolved
Charter: §L8.3.10b and §L8.3.11. One
transpileModule, output identical to node.Lever (iv) pays now — and the argument that put it later was wrong
prototype divergences distinct predecessors distinct (predecessor, prototype) pairs to NULL 48,197 78 ≤ 97 (upper bound) 0 The predecessors are shared, not forked. I had argued lever (iv) belonged after the content key because "its predecessors have already forked, so earlier merges nothing". I committed a two-branch prediction before measuring precisely because I was not sure — and the measurement picked the other branch. A generation keyed on (predecessor, prototype identity, link kind) collapses this site's 48,197 mints to ≤ 97: about −48,100 on tsc, standalone, now, and the one fork source between option 2 and the exit criterion.
Prototype identity is a serial in the prototype's own
ObjectMeta, not a unique prototype ShapeId: a prototype's ShapeId changes when it gains a method (that is why #10842's validity hook lives in the stamp funnel), so keying on it would fork receivers that diverge before vs after the mutation. The serial's cost is now measured, not assumed: 112,058 of 112,426 meta records are prototype marks, so it is ~0.9 MB, and "every meta" vs "only prototypes" is the same population on tsc.Capacity is a shape fact — lane 13's collision resolved as (b)
Lane 13 found 97 % of tsc's spills come from objects built before the size was learned: same key list, smaller capacity. From source, the learned size lives in
LEARNED_INLINE_FIELDS(object/spill.rs), a 1,024-entry direct-mapped per-thread table keyed by class id that:- opens with
if class_id == 0 … return— object literals never learn; - on a slot collision evicts another class's learned size — the same defect class as the transition cache;
- cannot carry across
Object.create's fresh synthetic class ids.
The choice is (b), and it is principled, not a concession: §3 requires the shape to determine key → location, capacity determines location, so capacity is a shape fact. Two objects with the same key list and different capacities are different facts, and same layout + same facts ⇒ same ShapeId still holds.
It is already priced. The option-2 simulation keyed on
live_inline_slot_count, so 12,846 already includes capacity in the key — it is option (b)'s number. Against 10,867 key lists, capacity diversity plus kind and hole variation costs at most 1,979 shapes (×1.18), which meets the criterion. On tsc: 0 dictionary-mode objects from this cause, RSS neutral, mints ≤ 12,846.(a) makes early spills permanent; (c) is the right goal but needs shrinking to be a new layout, which is (b); (d) would put 8,519 objects in dictionary mode in tsc's first iteration — a cliff for a problem (b) handles at ≤ 1,979 shapes.
The follow-up that cuts both spills and shapes: under option 2 the learned size keys on the layout, not the class id — which closes the literal-0 and
Object.creategaps at once — held as a per-layout fact in the shape record, not in another direct-mapped table that evicts.- opens with
Live-set composition: 82.6% of what every collection traces is keys arrays
Measured with the runtime's own
PERRY_GC_CENSUS, which classifies the heap at the mark-complete
point of a synchronous full collection. Nothing was built for this beyond a copy of
tscwork.tsthat callsgc()from a TypeScriptbeforetransformer — so the census lands
mid-transpile, with the AST fully live — and once between transpiles. Output is
typescript 5.8.2 193520, byte-identical tonode --expose-gc.Mid-transpile (AST live) — live heap 130.8 MiB, 870,974 objects
type objects live bytes share of live array 703,549 109.92 MiB 84.1% ↳ of which shared keys arrays 682,146 107.99 MiB 82.6% object 49,681 10.01 MiB 7.7% object_meta 41,353 5.36 MiB 4.1% string 50,112 2.91 MiB 2.2% closure 20,336 2.24 MiB 1.7% The program's genuine live data — AST nodes, strings, closures — is about 15 MiB. Everything
else the collector traces is object-model bookkeeping, and almost all of it is one kind of object.Between transpiles — live heap 15.6 MiB
Keys arrays are 10.62 MiB of 15.6 MiB = 67.9%, and 50,687 of them persist across the call.
So this is not only a transient: the steady-state retained heap is mostly keys arrays too.Shape records are NOT in the traced heap — settled from source
ShapeSlab { pages: Vec<Option<Page>>, len }(object/shapes_store.rs:269) is a RustVec
through the global allocator, andShapeTableInner'sindices/by_facts/familiesare
PtrHashMaps. The census reports all of them as side tables, not arena types. So the 2.26 M
minted ShapeIds are an RSS cost, not a GC-tracing cost — which is the answer to "is the slab
inside the collector's heap at all": no.Their Rust-heap footprint is nevertheless large, and larger than the live GC heap:
side table (Rust heap, not traced) entries bytes shapes.descriptors712,577 51.85 MiB shapes.families682,223 25.69 MiB shapes.by_facts711,204 25.00 MiB shape table total ≈102.5 MiB gc.layout_slot_masks792,056 18.13 MiB side tables, all 154.2 MiB The shape table's GC cost is not tracing but the per-collection rekey walk, which I measured in
the tsc profile at 0.95% of the run (scan_shape_table_rekey_mut5,971 samples,
ShapeTableInner::facts_remove1,155,shape_keys_entry_is_minor_relevant660).Why this makes step 2.5 a GC fix
shapes.familiesis keyed by keys-array address, and there are 682,223 families to 682,146
shared keys arrays — one family per array. The keys-array population is the distinct-shape
population. Canonical identity that collapsed ~2.26 M mints toward the ~12,846 distinct layouts
#10868 counts would collapse the family count with it, and therefore the arrays — which are
82.6% of everything every one of the 244 full collections traces.So the two levers multiply, and the composition says which is which:
- pacing (perf(gc): 88% of a real TypeScript transpile is GC — 244 full collections per transpileModule, because #7937's absolute old-reclaim arm re-arms after every collection (a ~42 MiB live set parked just under the 48 MiB threshold) #10928) divides the number of collections;
- canonical identity (ShapeId exhaustion aborts the process after ~475 TypeScript transpiles (~2.26M ids minted per transpileModule, ~45 per AST node) #10868) divides the size of each one;
- they are independent, and 82.6% is the size factor's ceiling on this workload.
Note the direction of the surprise: private keys arrays are not the live population here.
682,146 of the 703,549 live arrays carryGC_FLAG_SHAPE_SHARED; the private ones are minted and
die, leaving ~21,400 arrays (1.93 MiB) at any instant. It is the shared arrays that accumulate.Scope and confidence
- High confidence: the type breakdown and the keys-array share. This is the collector's own
classification of its own marked set, not a sample. - The moment matters: this census is taken at an explicit full collection mid-emit, which is
not the same instant as the 244OldReclaim-triggered collections, and it covers the whole heap
(nursery included) whileold_reclaimableis old-gen only. The live totals therefore differ from
the trigger telemetry; the composition ratio is the finding, and it is corroborated at two
very different moments (82.6% and 67.9%). shape_keys_arrayscounts arrays flaggedGC_FLAG_SHAPE_SHARED. Private keys arrays are counted
as ordinary arrays, so the true keys-array share is at least what is quoted.
Artefacts on perrymaster:
/root/matrix/census.jsonl(4 censuses),/root/fr/work/tscensus.ts.64 remaining items
- added 15 commits that reference this issue
on Sep 23, 2026
Summary
Perry mints ~2,257,000 ShapeIds per
ts.transpileModule()— roughly 45 per AST node. The ShapeId band is 2^30 ids. At that rate a process aborts after about 475 transpiles:Exhaustion is not a graceful degradation.
alloc_shape_id_fromparks at the end and returnsErr(ShapeIdExhausted);shape_descriptor_error_abortturns that into:The refusal is correct — publishing an untracked shape would be a silent wrong value, and never wrapping is right because a wrapped id could alias a live shape. The problem is upstream of it: the consumption rate makes a bounded, non-recycled 2^30 space a hard runtime limit on process lifetime.
Why this is a real-world limit, not a synthetic one
475 transpiles is not a stress test. It is a few minutes of ordinary work for:
All of these are the intended workloads for a TypeScript-to-native compiler's own runtime. A process that aborts after a few hundred compilations cannot host them.
Shapes are minted but never reclaimed
Two independent facts combine into the limit:
So the fix is not "recycle ids" — it is to stop minting one per operation. Directions worth evaluating, none of them scoped here:
shape_descriptor_ensure_with_holesalready de-duplicates byfacts_key; a 45-per-node rate suggests many mints are semantically identical shapes that miss the memo. The 2026-09-15 work ondeterministic_semantic_generation(perf: one Object.defineProperty sends every later store on that object to the slow path (zod v4 schemas 29× bun; top OpenCode startup cost) #10287) found exactly this shape of defect — a memo table clearing at 8192 entries re-minted ~57k generations and forked every dependent receiver. A census of why each id is minted, by call site, would say whether this is the same class of problem.+4layout rule (fix(runtime): close rule 3 — no non-object cell can hold a live ShapeId at payload +4 #10828) and every range test; it buys a constant factor, not a fix.Measurement
typescript5.8.2, onets.transpileModule(), output byte-identical to node. Counted via a default-off diagnostic added while designing field representation in shapes; azod3.23.8 parse loop shows a much lower slope (1.32 ids/iteration), so the rate is workload-dependent and the compiler case is the severe one.Relationship to work in flight
Found while measuring the shape-count cost of adding field representation to shapes (design step 5 of the object-model rework). That design would add ~12% to the mint rate, which is a rounding error against this: the budget is already two orders of magnitude short of a real
tscrun. A parallel change assigning some ShapeIds at link time proposes splitting this band in two, which would halve the runtime half to ~237 transpiles.Both of those are downstream of this. Whatever consumes 45 ids per AST node should be understood before anything else is designed against the remaining budget.