Skip to content

Speed up lbuild with template caching - #87

Merged
salkinium merged 6 commits into
modm-io:developfrom
salkinium:feature/speedups
Sep 29, 2026
Merged

salkinium merged 6 commits into
modm-io:developfrom
salkinium:feature/speedups

Conversation

@salkinium

@salkinium salkinium commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

I tried to make lbuild faster with Python 3.14's free-threading, which kinda worked (2x for a single build), but in CI we already run one lbuild per CPU, so it didn't help at all, lol. The profiling on the way was a lot more useful though: more than 40% of the runtime was just Jinja2 compiling the same templates over and over again, since every module gets its own Jinja2 environment.

  • Cache compiled Jinja2 templates in a FileSystemBytecodeCache shared by all modules and lbuild runs. It's stored in the per-user temp folder ($TMPDIR/_jinja2-cache-$UID), keyed by template path and source checksum, and the Jinja2 and lbuild versions are part of the filename so an upgrade starts fresh. If the temp folder is not usable, lbuild just compiles without the cache.
  • Cache node name lookups until the tree changes. Every env[…], env.query(…) and env.has_module(…) did a full anytree glob from the root, ~1000-2000 times per build.
  • Match the module filename before the ignore patterns in add_modules_recursive, which skips ~96k fnmatch calls.
  • Make the minimum Python version 3.11, since 3.8 and 3.9 are EOL and 3.10 will be next month (and the cache uses functools.cache anyway). CI now tests 3.11 to 3.14 with the current GitHub Actions versions.
  • Unit tests pass.
  • Generated output is identical to before for modm (test/all/cortex-m.xml and almost all of modm:**) on STM32H743.

With modm on STM32H743ZIT6:

before after (warm cache) after (cold cache)
test/all/cortex-m.xml 0.94s 0.37s ~0.65s
almost all of modm:** 1.59s 0.95s ~1.25s

The cache works across targets too: after building the H743, an H723 compiled zero new templates and an F469 only 23. So the CI jobs should benefit a lot.

There's a follow-up that returns copies of cached query results (feature/query-copy), since modules mutating the shared results is a real bug. But it breaks older modm versions on F7/H7, so it has to wait until modm-io/modm has the fix in a release.

@salkinium
salkinium merged commit 58b8a99 into modm-io:develop Sep 29, 2026
4 checks passed
@salkinium
salkinium deleted the feature/speedups branch September 29, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant