[maleadt]

A round of load-time and first-use latency fixes, from profiling using Metal and the first operations of a fresh session on an M1 (Julia 1.12 and 1.13).

• Metal.rand(Float32, ...) called rand! on an abstractly typed array, because Julia doesn't specialize on a T::Type argument that is only passed on. Verifying that edge against GPUArrays' AnyGPUArray unions took about 40 ms of every using Metal (Metal's image: 389 → 348 ms on 1.13, 504 → 466 ms on 1.12).
• Several host functions involved in compiling and launching a kernel were specialized on its type, so every new kernel compiled them again: the closure passed to GPUCompiler.cached_lookup, calling a HostKernel, and device()'s first-use path, inlined into code specialized per kernel. A callable struct, an unspecialized launch_kernel and an out-of-line device() initialization bring a second new kernel from 38 to 27 ms. The launch specialization test now covers these. Launch overhead is unchanged.
• Labelling every pipeline with its kernel's Julia signature compiled printing code on the first launch of a session (~15 ms: first launch 39 → 25 ms). Signatures are only computed when label_resources is enabled now; otherwise pipelines are labelled with the generated function's name.
• invoke_frozen hid the result types of the GPUCompiler calls in compile_to_metallib, so the code using them was dynamically dispatched and its callees compiled in every session; asserting the types (and typing the intrinsics Dict and bind_arguments' parameters) saves ~25 ms on the first kernel (300 → 270 ms).
• The workload now also displays arrays (first show 81 → 8 ms; REPL display 105 → 15 ms, through a precompile statement since a terminal can't be created while precompiling), runs Metal.zeros/fill (103 → 28 ms), a scalar multiplication (108 → 1 ms) and a kernel with an atomic (185 → 58 ms), for 0.6 MiB more image and ~4 ms more load time.
• can_precompile_launches() (internal) combines the checks that decide whether Metal's workload can launch kernels, so that downstream workloads can use them too: AcceleratedKernels' Metal extension needs it to precompile its tuned reductions and scans, which Metal's own workload can't (Precompile the Metal operations that depend on the extension's tunings AcceleratedKernels.jl#200).

Related, in other packages: GPUCompiler's workload now compiles a representative kernel and avoids dynamic dispatch in back-end IR code (JuliaGPU/GPUCompiler.jl#1070, #1071), and ObjectiveC.jl's generated property methods shrink (JuliaInterop/ObjectiveC.jl#92, #93), which together take another ~100 ms off loading Metal and ~60-100 ms off the first kernel. Most of what's left of the first kernel's host compilation is JuliaLang/julia#58465.

[github-actions]

Documentation preview: https://metal.juliagpu.org/previews/PR1124/

[codecov]

Codecov Report

❌ Patch coverage is 80.00000% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.93%. Comparing base (c0a323a) to head (bfa8a6e).

Files with missing lines	Patch %	Lines
src/random.jl	16.66%	5 Missing ⚠️
src/precompile.jl	70.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1124      +/-   ##
==========================================
- Coverage   88.94%   88.93%   -0.01%     
==========================================
  Files         105      105              
  Lines        9362     9291      -71     
==========================================
- Hits         8327     8263      -64     
+ Misses       1035     1028       -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Benchmark results

Compared with c0a323ab3, the merge base.
The environment differs from the baseline's: GPUArrays 12.1.1 → 12.2.0, GPUArraysCore 0.2.1 → 0.2.2, SPIRVIntrinsics 1.5.1 → 1.6.0.

Times are the minimum time of every benchmark. A change is unusual when it exceeds both three times the benchmark's typical change between consecutive runs on the default branch and a minimum per category (performance 3%, latency 5%, applications 3%, other 3%). A benchmark that starts or stops allocating is listed however few the allocations. A single run can still be a fluke, so rerun before drawing conclusions.

	Geometric mean	Unusual: slower	Unusual: faster	Compared
Performance	−0.0%	1	1	74
Latency	−1.3%	0	0	4
Applications	−0.2%	0	0	5

Benchmark	Baseline	This PR	Change
Performance
array/shared/iteration/findfirst/int	282 µs	255 µs	−9.7%	🟢 faster
kernel/metalcall	825 ns	875 ns	+6.1%	🔴 slower

All benchmarks

Benchmark	Baseline	This PR	Change
Performance
array/accumulate/Float32/1d	392 µs	393 µs	+0.2%
array/accumulate/Float32/dims=1	369 µs	365 µs	−1.1%
array/accumulate/Float32/dims=1L	8.86 ms	8.86 ms	−0.1%
array/accumulate/Float32/dims=2	435 µs	434 µs	−0.2%
array/accumulate/Float32/dims=2L	2.72 ms	2.76 ms	+1.6%
array/accumulate/Int64/1d	520 µs	523 µs	+0.6%
array/accumulate/Int64/dims=1	523 µs	518 µs	−0.8%
array/accumulate/Int64/dims=1L	1.64 ms	1.64 ms	−0.4%
array/accumulate/Int64/dims=2	541 µs	527 µs	−2.6%
array/accumulate/Int64/dims=2L	5.44 ms	5.44 ms	+0.0%
array/broadcast	237 µs	221 µs	−6.9%
array/construct	2.21 µs	2.13 µs	−3.8%
array/permutedims/2d	339 µs	338 µs	−0.4%
array/permutedims/3d	460 µs	460 µs	−0.0%
array/permutedims/4d	419 µs	419 µs	+0.1%
array/private/copy	224 µs	219 µs	−1.9%
array/private/copyto!/cpu_to_gpu	221 µs	217 µs	−1.5%
array/private/copyto!/gpu_to_cpu	217 µs	219 µs	+0.9%
array/private/copyto!/gpu_to_gpu	217 µs	219 µs	+0.6%
array/private/iteration/findall/bool	352 µs	351 µs	−0.4%
array/private/iteration/findall/int	422 µs	423 µs	+0.4%
array/private/iteration/findfirst/bool	279 µs	282 µs	+1.1%
array/private/iteration/findfirst/int	280 µs	282 µs	+0.7%
array/private/iteration/findmin/1d	347 µs	348 µs	+0.5%
array/private/iteration/findmin/2d	390 µs	393 µs	+0.8%
array/private/iteration/logical	358 µs	362 µs	+1.3%
array/private/iteration/scalar	1.37 ms	1.36 ms	−0.4%
array/random/rand!/Float32	421 µs	425 µs	+1.1%
array/random/rand!/Int64	444 µs	439 µs	−1.3%
array/random/rand/Float32	437 µs	438 µs	+0.1%
array/random/rand/Int64	514 µs	514 µs	+0.0%
array/random/randn!/Float32	352 µs	347 µs	−1.3%
array/random/randn/Float32	366 µs	365 µs	−0.1%
array/reductions/mapreduce/Float32/1d	210 µs	215 µs	+2.6%
array/reductions/mapreduce/Float32/dims=1	214 µs	212 µs	−0.9%
array/reductions/mapreduce/Float32/dims=1L	579 µs	581 µs	+0.4%
array/reductions/mapreduce/Float32/dims=2	253 µs	250 µs	−1.2%
array/reductions/mapreduce/Float32/dims=2L	422 µs	416 µs	−1.4%
array/reductions/mapreduce/Int64/1d	221 µs	217 µs	−2.0%
array/reductions/mapreduce/Int64/dims=1	228 µs	223 µs	−2.1%
array/reductions/mapreduce/Int64/dims=1L	995 µs	994 µs	−0.2%
array/reductions/mapreduce/Int64/dims=2	242 µs	256 µs	+5.8%
array/reductions/mapreduce/Int64/dims=2L	602 µs	600 µs	−0.3%
array/reductions/reduce/Float32/1d	216 µs	217 µs	+0.1%
array/reductions/reduce/Float32/dims=1	207 µs	213 µs	+2.9%
array/reductions/reduce/Float32/dims=1L	578 µs	578 µs	−0.0%
array/reductions/reduce/Float32/dims=2	255 µs	249 µs	−2.2%
array/reductions/reduce/Float32/dims=2L	421 µs	423 µs	+0.5%
array/reductions/reduce/Int64/1d	234 µs	231 µs	−1.3%
array/reductions/reduce/Int64/dims=1	210 µs	215 µs	+2.3%
array/reductions/reduce/Int64/dims=1L	992 µs	995 µs	+0.3%
array/reductions/reduce/Int64/dims=2	255 µs	255 µs	+0.3%
array/reductions/reduce/Int64/dims=2L	599 µs	592 µs	−1.2%
array/shared/copy	127 µs	127 µs	+0.4%
array/shared/copyto!/cpu_to_gpu	36.2 µs	36.3 µs	+0.3%
array/shared/copyto!/gpu_to_cpu	36.2 µs	37.6 µs	+3.8%
array/shared/copyto!/gpu_to_gpu	36.8 µs	38.1 µs	+3.6%
array/shared/iteration/findall/bool	353 µs	357 µs	+1.1%
array/shared/iteration/findall/int	423 µs	423 µs	+0.2%
array/shared/iteration/findfirst/bool	280 µs	282 µs	+0.8%
array/shared/iteration/findfirst/int	282 µs	255 µs	−9.7%	🟢 faster
array/shared/iteration/findmin/1d	344 µs	345 µs	+0.3%
array/shared/iteration/findmin/2d	394 µs	387 µs	−1.9%
array/shared/iteration/logical	364 µs	362 µs	−0.6%
array/shared/iteration/scalar	75.3 ns	84.6 ns	+12.4%
array/sorting/1d	1.92 ms	1.91 ms	−0.1%
array/sorting/2d	8.37 ms	8.33 ms	−0.4%
kernel/indexing	209 µs	215 µs	+2.6%
kernel/indexing_checked	391 µs	391 µs	+0.0%
kernel/launch	763 ns	767 ns	+0.6%
kernel/metalcall	825 ns	875 ns	+6.1%	🔴 slower
kernel/rand	407 µs	403 µs	−1.0%
metal/synchronization/context	557 ns	551 ns	−1.1%
metal/synchronization/stream	363 ns	345 ns	−4.9%
Latency
compilation	1.77 s	1.74 s	−1.3%
import	1.47 s	1.44 s	−1.8%
launch	1.58 s	1.53 s	−3.1%	noise unknown
precompile	35.0 s	35.4 s	+1.2%
Applications
byval/reference	1.10 ms	1.12 ms	+1.3%
byval/slices=1	1.12 ms	1.10 ms	−1.6%
byval/slices=2	2.01 ms	2.01 ms	−0.1%
byval/slices=3	6.85 ms	6.85 ms	+0.0%
metaldevrt	392 µs	389 µs	−0.6%

History: https://juliagpu.org/perf/?pkg=metal