[smoogipoo]

At some point I'll make a blog post to go through all of my findings, but here is a short(er) version.

First of all I'll say that most of the below is based on my conjecturing or supported by anecdotal evidence. I may not be using the most accurate terms, but there is seldom documentation on this - especially from Apple.

VBLANK

There are three processes that are important to us: VBLANK, Scanout, and Flip.

• VBLANK is a pause duration at the start of the frame before Scanout.
• Scanout is the duration to transmit data to the screen's internal framebuffer. This is both dependent on cable bandwidth and any internal processing delays of the screen itself.
• Flip is the duration for the screen to adjust the colour of every pixel. Historically, CRT screens used this period to irradiate each phosphor for some period of time, but this is almost instantaneous for LCD screens (save for some muxing delays).

Importantly, VBLANK is the only parameter that can be adjusted, and is done so as to meet the physical characteristics of the display panel (e.g. pixel response time) as well delays imposed by Scanout + Flip. There is a particular deadline that the display has to hit, and VBLANK is the mechanism to control that.

For internal displays or displays where high bandwidth is guaranteed, Scanout takes a very short amount of time, and as mentioned above Flip is basically instant. VBLANK becomes the dominating factor and it is now important to remember that it is just an artificial delay imposed by the screen.

Anecdotal testing puts a MacBook's internal ProDisplay at ~5ms VBLANK.

There are two peculiarities with how macOS handles this:

1. You are allowed to render as late as you want during the VBLANK interval. It's possible to get <1 frame latency by delaying your rendering as long as you don't miss the VBlank period - if you miss it then you get shoved into the next Scanout which is up to 2 frames away. That said, it is not easy to do this - macOS provides no mechanism that tells you exactly how long VBLANK is.
2. Metal is locked into a FIFO pipeline that does not provide support for a "mailbox" mechanism. Importantly, this means that the only effect of triple buffering is to delay your frame by one full refresh cycle, which has its use cases but cannot be used as a mechanism to bypass the limitations imposed by VBLANK.

Display Links

Where do CADisplayLink, CVDisplayLink, and CAMetalDisplayLink fit in all of this? Does it handle this for us?

No.

As far as I can surmise - Apple's docs on this subject are especially poor - display links are simply a VBLANK timer. They callback to your app at the start of one VBLANK and provide a targetTimestamp value that appears to indicate the start of the next VBLANK.
They also perform a second function, which is to control when your app applies backpressure.

Backpressure

Now for a slight tangent onto backpressure. A typical (Metal) gameplay loop looks like the following:

while (running):
    sampleInput();
    update();
    draw():
        METAL_nextDrawable();
        ... (submit rendering commands)
    METAL_presentDrawable();

In modern APIs, "swap buffer" doesn't exist. It's an async process that occurs when you call presentDrawable() (in Metal). This allows you to push many frames as fast as you want, but eventually your GPU will give up and block your execution at nextDrawable(). But in the above loop, this occurs somewhere in the middle of the frame after input has been sampled and would be detrimental to input latency.

The original (removed) code in this PR uses CVDisplayLink, but a more fitting construct would be CAMetalDisplayLink. It doesn't change much - it's still just a VBLANK timer, but it provides a CAMetalDrawable inside its callback. This allows the above loop to be rewritten as:

CAMetalDisplayLink.callback = onCallback;

onCallback(drawable):
    sampleInput();
    update();
    draw():
        ... (submit rendering commands)
    METAL_presentDrawable();

while (running):
    sleep(1);

It applies backpressure in a way that you're both synchronised to VBLANK and also the GPU's queues are empty for the next frame. It caps the number of in-flight frames to 1.

For osu! this is irrelevant because we already separate our input, update, and draw threads. It would be useful to use this for single threaded execution mode, but the way it is implemented right now is not correct - we really need to run the entire loop through the display link callback rather than blocking at some point inside DrawFrame().

For osu!, there is another mechanism that provides us exactly the same type of backpressure we're attempting to apply here - vsync.

Changes in this PR

According to the above, this PR applies two changes.

1. Remove triple buffering

macOS triple buffers by default. This does nothing for us other than increasing FPS (making numbers bigger), it does not lower input latency - it actually increases it.

2. Remove display link

As I hoped to explain above, the display link is only bloating our code without providing any functional benefit.

Results

The most impactful change is the removal of triple buffering. When vsync is enabled, it will reduce frame latency by around 50-75%. If the scene is fast enough to fit within the VBLANK window, then you will see 1 frame latency when in current builds it incurs a full refresh cycle (2 frames).

This change comes at a cost. FPS will be decreased which means that single threaded execution is impacted. I'd point out again that single threaded execution is not really being done correctly either way. In particular, this may negatively impact iOS devices which run in STE mode by default.

Scenes that have high load, like song select, may miss the VBLANK window and degrade to 60fps on ProMotion displays. Gameplay should run fast enough to meet the VBLANK deadline in general, and experience the most benefit from the reduction in input latency.

I believe in this PR as a stepping stone and that we can work on iOS separately. Either implementing CAMetalDisplayLink properly or passing the buffer count through SwapchainDescription (it doesn't have support for this right now).

Further work

This PR directly leads to a re-evaluation of the D3D11 pipeline, where we use waitable swapchains in a similar effort to display links expecting that they'll magically reduce input latency to as low as possible. And I'm not being hyperbolic in saying that, when the official documentation lists them for this purpose.

In reality, waitable swapchain is just a mechanism to introduce backpressure such that the GPU queue is limited to 1 frame at a time.

Furthermore, I mentioned above that you can delay your rendering to as close to the end of the VBLANK period as possible, and that this is not easily doable. This is somewhat untrue because NVIDIA Reflex, AMD Anti-Lag, and Intel XeLL all provide the mechanism to do exactly this.

[peppy]

Have you tested the impact on iOS? I'm very hesitant to roll this out without having some kind of confirmation it isn't going to bone older / slower devices, especially since we're looking to push an iOS release out soon.

[smoogipoo]

Valid concern. I haven't tested iOS because testing on that platform is time-prohibitive. If you're okay with the tradeoff for macOS, my best path of resolution would be to continue to enforce triple buffering for iOS.

The thing is, for macOS there really is no other option than vsyncing. Vsync-off seems to be fundamentally broken at an OS level, to the point that the simplest Objetive-C app rendering nothing (literally just clearing the screen in a loop) is unable to keep a stable framerate.

So by double buffering we're making a tradeoff between lowest input latency and stable framerates, and I think for osu!'s case, if we're saying that vsync is the only way to go, then we care more about input latency during gameplay where we're generally also going to be meeting the frame budget for ProMotion displays.

---

As an aside, I find it very concerning that we're so extremely GPU bound, enough so for this matter. Here's a little excerpt taken on my MBP:

Left half is in main menu, right half is in song select. In this case green represents CPU work, red represents true GPU-bound work, and I believe blue to represent the remaining portion of the vblank region + the "scanout + flip" portions of my explanation.

We are barely meeting (or, in song select missing) frame budget deadlines from GPU work alone. I'll have to look into it again, because I thought we were in a better state than this...

---

Something to consider is that we have been double buffering on D3D11 all this time:

veldrid/src/Veldrid/D3D11/D3D11Swapchain.cs

Line 283 in 3cdad6c

BufferCount = 2,

I believe all OpenGL drivers also default to double buffering by default.

In the case of D3D11, my plan is to actually go in the opposite direction. The presentation model we use (FLIP_DISCARD) should be equivalent to a mailbox mode, which theoretically combined with a triple buffer means we should get the best of both words of reducing latency + stable framerates, at the cost of rendering at a higher rate.

We'll have to discuss this later, but I think the end goal for us should be to decompose the single "frame limiter" moniker into separate concepts for "fps", "vsync", and "latency mode".