Build the gfxstream host backend on macOS with CI - #165
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
Hi @ericcurtin, it based on your previous work, so I add you as the co-authored-by, could you pass the CLA if you want to keep it? |
082ffe3 to
7d4b723
Compare
|
@gurchetansingh @jmacnak PTAL. In my fork branch, all CI passed including new added macOS jobs. |
|
@utzcoz I will test and review once, but ultimately it's the maintainers that got to approve and merge. You haven't signed the Google CLA, that will likely be a blocker see the failed build. |
|
@ericcurtin It's not me, the job said your docker email(I added it as co-authored-by) doesn't sign the CLA. I have sent multiple PRs recently, my sign is okay. See https://github.com/google/gfxstream/pull/165/checks?check_run_id=89684625026. |
|
Try this one @utzcoz I think it's signed with this one: If that fails, just remove my email and name, it's fine. |
ericcurtin
left a comment
ericcurtin
left a comment
There was a problem hiding this comment.
Tested this locally on real Apple Silicon hardware (M4 Max, macOS 26.5, MoltenVK 1.4.1 / vulkan-loader 1.4.350.1 via Homebrew) and everything the PR describes checks out:
- Meson (
meson setup build && meson compile -C build): clean build, only pre-existing/benign deprecation warnings (NSOpenGLView, C-linkageunique_ptrreturn, etc.), no errors. - CMake (
cmake -S . -B cmake-build -G Ninja && ninja -C cmake-build): clean build, same class of warnings as Meson, no errors. Confirmed theVulkan_unittestsdiscover_tests()reordering (host/CMakeLists.txt) is a genuine bug fix — previouslydiscover_tests()ran between theif (APPLE)block and theelseif (QNX)/(UNIX)chain, so the QNX/XCB compile definitions were applied after test discovery; now it correctly happens after the whole platform chain. - Bazel host backend (
//host:gfxstream_backend_static //host:gfxstream_backend_shared //host:gfxstream_backend,--graphics_drivers=host): builds cleanly, including registering the Apple CC toolchain viaapple_support. - Bazel host Vulkan tests on MoltenVK:
vk_common_operations_tests,gfxstream_compositorvk_tests,gfxstream_vsyncthread_tests,vk_format_utils_tests,gfxstream_emulatedphysicalmemory_tests,gfxstream_emulatedphysicalqueue_tests,gfxstream_displayvk_tests— all pass (24/24) against the real MoltenVK/Vulkan-loader Homebrew install, including the newVK_KHR_portability_subset/VK_KHR_PORTABILITY_ENUMERATIONhandling incompositor_vk_unittest.cpp. - ANGLE + MoltenVK GLES (
gles_angle_vulkan_moltenvkenv,//host:gfxstream_framebuffer_tests): ANGLE's newangle_appleobjc_library builds (Metal/IOSurface WSI,libEGL.dylib/libGLESv2.dylib), and all 30 framebuffer tests pass with the driver staging workaround ingraphics_test_environment.cppworking as advertised (confirmed the ANGLE+loader staging into a writable temp dir actually gets used — log showsGraphics Adapter ... ANGLE (Apple, Vulkan 1.1.334 (Apple M4 Max), MoltenVK-0.2.2209)). - end2end guest→host (
//tests/end2end:gfxstream_end2end_tests,*CompositionTest*OverVirtioGpuAsg*): builds the full mesa guest driver + rutabaga + kumquat Rust stack and all 18 composition tests pass, including the YV12/planar-YUV cases that specifically exercise theMVK_CONFIG_USE_METAL_ARGUMENT_BUFFERS=0workaround invulkan_dispatch.cppfor the immutable-YCbCr-sampler SPIR-V-to-MSL failure described in the commit message.
Great, thorough piece of work — nice attention to detail on the platform quirks (BSD accept() inheriting O_NONBLOCK, SO_NOSIGPIPE, the virtgpu_kumquat_finish null-pointer panic-on-abort, kill(0) on process group 0, etc). I left a couple of minor, non-blocking notes inline on the new macOS Rust sys layer.
1bd3ecb to
14929a5
Compare
|
Yeah, looks like ericcurtin17@gmail.com worked |
|
@ericcurtin yeah. You gmail works with CLA checking. I tried to upstream safe patches to Mesa now, and https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43217 is the first one. |
|
Wow, thanks for the patches. I think would help to avoid downstream Mesa patches though for maintainence reasons. I think the strategy here should be:
So I do recommend focusing this MR on (1), and having follow up MRs for the rest. |
|
@gurchetansingh No problem. I will backup existing work with a new branch and then change this PR or start a new PR for 1) this week. |
26e82b9 to
8df0743
Compare
utzcoz
changed the title
c59a1f2 to
57f30f7
Compare
Get the host backend compiling on macOS with all three build systems and add
CI jobs so that it keeps compiling.
Meson:
- Enable the objc and objcpp languages on Darwin, so that meson can configure at
all with mac_native.m and native_sub_window_cocoa.mm in the tree. Declaring
them in project() instead would make meson probe for an Objective-C compiler
everywhere, which fails on toolchains built without one; every Objective-C
source here is already behind a Darwin guard.
- Give the Vulkan-only build the GL include paths it needs: the Vulkan server
includes the GLES dispatch headers regardless of which decoders are enabled.
- host/iostream and host/snapshot are static libraries with no sources, which
BSD ar refuses to archive. Give them a stub translation unit on Darwin.
- Link the Apple frameworks the host backend needs, build the Objective-C
sources, and require xcb only off-Apple.
- Fix leftover includes of "GlesCompat.h" from before the rename.
CMake:
- Same framework, Objective-C and NEON ISA handling as the Meson build.
Bazel:
- Register the Apple CC (Xcode) toolchain via apple_support. The hermetic LLVM
toolchain registered next to it cannot compile Objective-C, which the macOS
window and context sources are written in. The Apple toolchain is constrained
to Apple platforms, so toolchain resolution elsewhere is unaffected.
- Raise the deployment target to 11.0. The default of 10.11 is no longer
supported by the current SDK's libc++, which surfaces as a -Werror failure.
- Compile the Objective-C sources with -fno-objc-arc. They use manual reference
counting, and objc_library turns ARC on by default.
- Give the platform selects in glslang and swiftshader a macOS arm so that the
target graph resolves on Darwin.
Host:
- EGLNativeWindowType was typedef'd as unsigned int in gles_compat.h, which
truncates pointers on 64-bit; native_sub_window_cocoa.mm returns an NSView*
through it.
- Create the display surface through VK_USE_PLATFORM_METAL_EXT rather than
VK_USE_PLATFORM_MACOS_MVK. vkCreateMetalSurfaceEXT is the entry point the
surrounding code already calls, and it is the one MoltenVK exposes.
- Normalize shm_open() names to be POSIX-compliant (leading slash) on all
platforms, and use shm_open() on macOS, which has no memfd_create().
- Propagate the return value of stream_renderer_create_blob() instead of
always reporting success.
CI:
- New Bazel, CMake and Meson jobs build the host backend on macOS.
- The Bazel job names the host backend libraries explicitly instead of building
everything: macOS cannot build the full target graph, because the guest stack
needs libdrm and swiftshader bundles an LLVM with no macOS configuration.
- No tests run on macOS yet. That needs a graphics driver environment, which is
left for a follow-up along with the guest stack and end-to-end coverage.
- Satisfy the workflow security scan:
- Pin every action reference to a commit hash, with the version it stands for
in a trailing comment.
- Pass the bazel target and argument inputs to the build and test steps
through the environment instead of expanding them into the script, so that
their contents can never be parsed as shell code. The asan job's argument
value loses its quotes as a result: expanding a variable does not re-run
quote removal, and the value has no spaces to protect.
- Check out without persisting credentials. No step needs the token once the
tree is on disk.
- Give both workflows a read-only token instead of the broad default.
Only the bazel target input is new here; every other finding predates this
branch.
Test (macOS on Apple Silicon):
bazel build --graphics_drivers=gles_angle_vulkan_swiftshader \
//host:gfxstream_backend_static \
//host:gfxstream_backend_shared \
//host:gfxstream_backend
meson setup -Ddefault_library=static -Dgfxstream-build=host build
meson compile -C build
cmake -S . -B cmake-build -G Ninja
ninja -C cmake-build
Co-Authored-By: Eric Curtin <ericcurtin17@gmail.com>
|
@jmacnak @gurchetansingh PTAL, all CI passed on my fork branch and without third_party dependencies changes in this PR. |
3 participants
Port gfxstream to macOS across all three build systems and wire up CI.
Build systems and CI:
ANGLE and MoltenVK:
Guest stack without libdrm:
end2end on macOS: