Skip to content

vm: keep qcow2 overlays buffered so a shared base stays in the page cache - #194

Open
CMGS wants to merge 2 commits into
masterfrom
fix/qcow2-overlay-buffered-backing
Open

vm: keep qcow2 overlays buffered so a shared base stays in the page cache#194
CMGS wants to merge 2 commits into
masterfrom
fix/qcow2-overlay-buffered-backing

Conversation

@CMGS

@CMGS CMGS commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Problem

Cloud Hypervisor used to open a qcow2 backing file with buffered I/O regardless of what the disk asked for. Every cloudimg and Windows VM layers a per-VM qcow2 overlay over one shared base blob (prepareCloudimg runs qemu-img create -f qcow2 -F qcow2 -b <base> <overlay> and attaches only the overlay), so that base was served from a single host page-cache copy no matter how many VMs ran on it.

Upstream now opens the backing file with the disk's own direct flag, and cocoon sets direct=on for every writable disk, so each VM reads the shared base straight from storage instead.

Measurement

16-core host, 700 MiB base image, qcow2 overlays, guest reads 1 GiB from /dev/vda. Page-cache residency of the base measured with mincore, files evicted with posix_fadvise(DONTNEED) before each cell.

cloud-hypervisor direct base resident after read
before the upstream change on 310.1 MiB (44.3%)
before off 310.1 MiB
after on 0.0 MiB
after off 310.1 MiB

8 VMs reading 1 GiB each, 3 repetitions: direct=on 2.47-2.67 s against direct=off 1.86 s.

This is not a host-memory regression — O_DIRECT uses less host RAM. The cost is read amplification: N VMs on one base each fetch it from storage instead of one fetch plus N-1 cache hits. A second pass over the same blocks was not faster in either configuration on this NVMe, so the measured time difference is O_DIRECT losing readahead rather than cache reuse.

Change

Default direct=off for a qcow2 disk that layers over a base image, in effectiveDirectIO. Both the launch path and the restore-time config patch derive direct from that function, so they stay in agreement.

The overlay itself loses O_DIRECT, so its writes land in the host page cache as well as the guest one. That memory is reclaimable and the residual cost is one extra copy per written byte — the cheaper half of this trade, since without it every VM refetches the shared base from storage.

Unchanged: raw COW disks, read-only layers, data disks, and an explicit direct_io on a data disk.

Why not keep O_DIRECT on the overlay

Expressing the split the old Cloud Hypervisor gave us by accident — a direct overlay above a buffered base — needs a per-backing-file cache policy that no released version has. It is requested upstream as cloud-hypervisor/cloud-hypervisor#8718, with an implementation parked on the fork branch qcow-backing-direct. This PR deliberately takes the approach that works against stock Cloud Hypervisor, so cocoon does not become dependent on a fork-only --disk key.

Second commit

asl flags RefuseManifest for splitting the Store type from its first method. Layout-only move, no behavior change. It was already failing on master.

Verification

  • go build ./...
  • go test -race -count=1 ./...
  • make lint — 0 issues on GOOS=linux and GOOS=darwin
  • golangci-lint fmt --diff — clean
  • asl ./... — clean on both GOOS (was 1 finding on master, fixed here)

@CMGS
CMGS force-pushed the fix/qcow2-overlay-buffered-backing branch from f8dba13 to 6a27a92 Compare August 12, 2026 05:38
@CMGS CMGS changed the title vm: keep qcow2 overlays buffered so a shared base stays in the page cache vm: keep the shared qcow2 base buffered with backing_direct Aug 12, 2026
@CMGS
CMGS force-pushed the fix/qcow2-overlay-buffered-backing branch from 6a27a92 to c23b130 Compare August 12, 2026 06:59
CMGS added 2 commits August 12, 2026 16:24
…ache

Cloud Hypervisor used to open a qcow2 backing file with buffered I/O no
matter what the disk asked for, so the shared base image behind every
cloudimg and Windows VM was served from one host page-cache copy. It now
opens the backing file with the disk's own direct flag, and cocoon sets
direct=on for every writable disk, so each VM started reading the base
image straight from storage instead.

Default direct=off for a qcow2 that layers over a base image. The
overlay itself loses O_DIRECT, so its writes land in the host page cache
as well as the guest one. That memory is reclaimable, and the residual
cost is one extra copy per written byte, which is the cheaper half of
this trade: without it every VM refetches the shared base from storage.
Raw COW disks, read-only layers and data disks are unchanged, and an
explicit direct_io on a data disk still wins.

Expressing the split the old Cloud Hypervisor gave us by accident, a
direct overlay above a buffered base, needs a per-backing-file cache
policy that no released version has. That is requested upstream in
cloud-hypervisor/cloud-hypervisor#8718.

Measured on a 16-core host with a 700 MiB base image and 8 VMs reading
1 GiB each: base image resident in the page cache 310 MiB with the fix
against 0 MiB without it, aggregate read 1.86s against 2.47-2.67s.
asl flags RefuseManifest for splitting the Store type from its first
method. It is a standalone function, so it belongs with the other
utilities below the method sets, exported ones first.
@CMGS
CMGS force-pushed the fix/qcow2-overlay-buffered-backing branch from c23b130 to 886e829 Compare August 12, 2026 08:27
@CMGS CMGS changed the title vm: keep the shared qcow2 base buffered with backing_direct vm: keep qcow2 overlays buffered so a shared base stays in the page cache Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant