Skip to content

Support per-layer cache configuration and attention-mask selection - #48178

Open
eladsegal wants to merge 12 commits into
huggingface:mainfrom
eladsegal:cache-heterogeneity-compat
Open

Support per-layer cache configuration and attention-mask selection#48178
eladsegal wants to merge 12 commits into
huggingface:mainfrom
eladsegal:cache-heterogeneity-compat

Conversation

@eladsegal

@eladsegal eladsegal commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

CPU CI GPU run-slow

What does this PR do?

Updates cache construction and attention-mask selection to support models whose layers use different cache settings.

Previously, DynamicCache and StaticCache derived one set of initialization arguments from the top-level config and reused it for every layer. They now initialize each cache layer from its corresponding layer config, preserving per-layer sliding_window, attention_chunk_size, and number_of_conv_states values.

Cache and mask helpers also inferred cache metadata from a fixed layer, which could select a linear-attention layer or an unpopulated KV layer instead of one representing the requested attention pattern. When no explicit layer_idx is provided, they now select a representative KV layer: dynamic caches prefer a populated matching layer, while static caches use the first matching KV layer in their fixed layout.

Who can review?

@ArthurZucker
@Cyrilvallez

(Re-opening of #47901)

@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 33315074346:2
Result: failure | Jobs: 16 | Tests: 137,956 | Failures: 0 | Duration: 12h 6m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant