External 4230 cache dir fix#4378
Open
rasapala wants to merge 12 commits into
Open
Conversation
The continuous batching servable initializer constructs the GenAI ContinuousBatchingPipeline directly and never applied the server-level --cache_dir (ServerSettings.cacheDir). Unlike the non-CB path, which applies it via ModelInstance::setCacheOptions, the CB path left model compilation caching disabled unless the user duplicated the value into the node's plugin_config as CACHE_DIR. As a result, .blob/.cl_cache artifacts were never persisted and every restart fully recompiled the model. Inject the global cache_dir into the pipeline plugin config before constructing the pipeline. An explicit CACHE_DIR in the node's plugin_config remains authoritative. Adds a regression test (LLMNodeOptionsCacheDirPropagation) covering both propagation of the global value and precedence of an explicit node value. Fixes #4230 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LLMNodeOptionsCacheDirPropagation only asserts that --cache_dir lands in properties->pluginConfig, which doesn't prove OpenVINO Core actually persists compiled-model cache artifacts -- the actual symptom in #4230 (log said "cache enabled", nothing was ever written to disk). Addresses feedback on the issue: #4230 (comment) LLMNodeOptionsCacheDirWritesCacheArtifacts constructs a real ContinuousBatchingPipeline against a temp --cache_dir and asserts at least one cache file actually lands there. Verified locally on Windows (MSVC) against facebook/opt-125m: [ OK ] LLMOptionsHttpTest.LLMNodeOptionsCacheDirWritesCacheArtifacts (707 ms)
Extract the global --cache_dir propagation into a shared GenAiServableInitializer::applyGlobalCacheDir helper and call it from all GenAI initializers. The continuous batching path already applied it (and VLM_CB shares that same initializer), but the legacy LM and legacy VLM paths construct their pipelines directly and never applied the server-level cache_dir. Routing every path through one helper covers all four pipeline types (LM/VLM x CB/legacy) and removes the duplicated inline block from the CB initializer. In the cache_dir tests, replace the manual end-of-test Config restore with a GlobalCacheDirGuard RAII helper so a failed ASSERT mid-test can no longer leak the modified --cache_dir singleton into subsequent tests in the suite. The guard also removes the temporary cache directory on scope exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a regression where the server-level --cache_dir was not propagated into OpenVINO GenAI pipeline plugin configuration for LLM/VLM servables, preventing compiled-model cache artifacts from being persisted across restarts (issue #4230).
Changes:
- Introduces a shared
GenAiServableInitializer::applyGlobalCacheDir()helper to inject the global--cache_dirintopluginConfigwhen the node doesn’t explicitly setCACHE_DIR. - Applies this helper in GenAI servable initializers (continuous batching + legacy).
- Adds/extends unit tests and updates model cache documentation.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/llm/llmnode_test.cpp | Adds regression tests for --cache_dir propagation and attempts an end-to-end cache artifact check. |
| src/llm/servable_initializer.hpp | Declares applyGlobalCacheDir() helper for GenAI initializers. |
| src/llm/servable_initializer.cpp | Implements applyGlobalCacheDir() and adds required includes. |
| src/llm/language_model/continuous_batching/servable_initializer.cpp | Calls applyGlobalCacheDir() before constructing the CB pipeline. |
| src/llm/language_model/legacy/servable_initializer.cpp | Calls applyGlobalCacheDir() before constructing the legacy pipeline. |
| src/llm/visual_language_model/legacy/servable_initializer.cpp | Calls applyGlobalCacheDir() before constructing the legacy VLM pipeline. |
| docs/model_cache.md | Documents how --cache_dir interacts with GenAI/LLM node plugin_config overrides. |
Collaborator
|
I think we are missing handling at least for image gen cases: model_server/src/image_gen/imagegen_init.cpp Line 280 in 0447498 What about audio? |
atobiszei
reviewed
Jul 17, 2026
| #pragma GCC diagnostic pop | ||
| #pragma warning(pop) | ||
|
|
||
| #include "../config.hpp" |
Collaborator
There was a problem hiding this comment.
Suggested change
| #include "../config.hpp" | |
| #include "src/config.hpp" |
Collaborator
There was a problem hiding this comment.
And thats probably why bazel didnt catch this when theres no change in BUILD file.
Collaborator
Author
There was a problem hiding this comment.
Lets leave it this way.
mzegla
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary
Issue #4230
Align cache_dir behavior across classic models and GenAI pipelines, and document the effective precedence that is now covered by tests.
Classic ModelInstance-based models use the ModelManager cache directory unconditionally. The global cache_dir value applied by the manager overrides any per-model cache_dir that may come from config loading.
GenAI graph nodes keep an explicit CACHE_DIR from graph.pbtxt plugin_config. That node-level value wins over the global cache_dir.
GenAI graph nodes without CACHE_DIR in plugin_config inherit the global cache_dir from CLI or server settings as a fallback.
The config.json path for multiple GenAI models is covered as well: one graph with explicit CACHE_DIR keeps its own value, while another graph in the same config without CACHE_DIR receives the global fallback.
The classic config.json model-loading path is covered separately to confirm that the manager-level cache_dir is propagated to loaded ModelInstance configs.
Tests cover both ModelManager cache_dir propagation and GenAI precedence for single-node, multi-node, and multi-model config.json scenarios.
🧪 Checklist
``