vm: keep qcow2 overlays buffered so a shared base stays in the page cache - #194
Open
CMGS wants to merge 2 commits into
Open
vm: keep qcow2 overlays buffered so a shared base stays in the page cache#194CMGS wants to merge 2 commits into
CMGS wants to merge 2 commits into
Conversation
CMGS
force-pushed
the
fix/qcow2-overlay-buffered-backing
branch
from
August 12, 2026 05:38
f8dba13 to
6a27a92
Compare
CMGS
force-pushed
the
fix/qcow2-overlay-buffered-backing
branch
from
August 12, 2026 06:59
6a27a92 to
c23b130
Compare
…ache Cloud Hypervisor used to open a qcow2 backing file with buffered I/O no matter what the disk asked for, so the shared base image behind every cloudimg and Windows VM was served from one host page-cache copy. It now opens the backing file with the disk's own direct flag, and cocoon sets direct=on for every writable disk, so each VM started reading the base image straight from storage instead. Default direct=off for a qcow2 that layers over a base image. The overlay itself loses O_DIRECT, so its writes land in the host page cache as well as the guest one. That memory is reclaimable, and the residual cost is one extra copy per written byte, which is the cheaper half of this trade: without it every VM refetches the shared base from storage. Raw COW disks, read-only layers and data disks are unchanged, and an explicit direct_io on a data disk still wins. Expressing the split the old Cloud Hypervisor gave us by accident, a direct overlay above a buffered base, needs a per-backing-file cache policy that no released version has. That is requested upstream in cloud-hypervisor/cloud-hypervisor#8718. Measured on a 16-core host with a 700 MiB base image and 8 VMs reading 1 GiB each: base image resident in the page cache 310 MiB with the fix against 0 MiB without it, aggregate read 1.86s against 2.47-2.67s.
asl flags RefuseManifest for splitting the Store type from its first method. It is a standalone function, so it belongs with the other utilities below the method sets, exported ones first.
CMGS
force-pushed
the
fix/qcow2-overlay-buffered-backing
branch
from
August 12, 2026 08:27
c23b130 to
886e829
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cloud Hypervisor used to open a qcow2 backing file with buffered I/O regardless of what the disk asked for. Every cloudimg and Windows VM layers a per-VM qcow2 overlay over one shared base blob (
prepareCloudimgrunsqemu-img create -f qcow2 -F qcow2 -b <base> <overlay>and attaches only the overlay), so that base was served from a single host page-cache copy no matter how many VMs ran on it.Upstream now opens the backing file with the disk's own direct flag, and cocoon sets
direct=onfor every writable disk, so each VM reads the shared base straight from storage instead.Measurement
16-core host, 700 MiB base image, qcow2 overlays, guest reads 1 GiB from
/dev/vda. Page-cache residency of the base measured withmincore, files evicted withposix_fadvise(DONTNEED)before each cell.direct8 VMs reading 1 GiB each, 3 repetitions:
direct=on2.47-2.67 s againstdirect=off1.86 s.This is not a host-memory regression — O_DIRECT uses less host RAM. The cost is read amplification: N VMs on one base each fetch it from storage instead of one fetch plus N-1 cache hits. A second pass over the same blocks was not faster in either configuration on this NVMe, so the measured time difference is O_DIRECT losing readahead rather than cache reuse.
Change
Default
direct=offfor a qcow2 disk that layers over a base image, ineffectiveDirectIO. Both the launch path and the restore-time config patch derivedirectfrom that function, so they stay in agreement.The overlay itself loses O_DIRECT, so its writes land in the host page cache as well as the guest one. That memory is reclaimable and the residual cost is one extra copy per written byte — the cheaper half of this trade, since without it every VM refetches the shared base from storage.
Unchanged: raw COW disks, read-only layers, data disks, and an explicit
direct_ioon a data disk.Why not keep O_DIRECT on the overlay
Expressing the split the old Cloud Hypervisor gave us by accident — a direct overlay above a buffered base — needs a per-backing-file cache policy that no released version has. It is requested upstream as cloud-hypervisor/cloud-hypervisor#8718, with an implementation parked on the fork branch
qcow-backing-direct. This PR deliberately takes the approach that works against stock Cloud Hypervisor, so cocoon does not become dependent on a fork-only--diskkey.Second commit
aslflagsRefuseManifestfor splitting theStoretype from its first method. Layout-only move, no behavior change. It was already failing on master.Verification
go build ./...go test -race -count=1 ./...make lint— 0 issues on GOOS=linux and GOOS=darwingolangci-lint fmt --diff— cleanasl ./...— clean on both GOOS (was 1 finding on master, fixed here)