Measuring keyboard-to-photon latency with a light sensor

https://thume.ca/2020/05/20/making-a-latency-tester/

by chmaynard • 6 years ago

118 46 6 years ago

What's going on with those two editors that the input latency is two frames on average? I would have expected it to be somewhere around 1 frame just because the editors aren't doing any extra work like auto-completion. Two whole frames sounds pretty bad.

raphlinus• 6 years ago

A large part of that is the extra frame of latency added by the compositor. One experiment that I'd very much like to do is try to get Windows to promote a window to using hardware overlays (this is sometimes called "DirectFlip" but the public documentation on it is very sparse) and see how much that helps.

lukevp• 6 years ago

This is really interesting. I use vs code on Windows on a 144 hz gaming monitor with an nvidia gpu and gsync, and it “feels” like it has the lowest input latency of any editor I use (out of notepad, notepad++, and Visual Studio).

I’m curious what impact hardware acceleration and gsync would have on this. I would guess Electron is using a hardware accelerated surface in my scenario, but I don’t imagine gsync is doing much on the desktop since it is synchronized to the entire OS and not a specific application.

zamadatix• 6 years ago

If gsync is enabled for windowed applications in the nvidia control panel it actually synchronizes to the active window not the rest of the OS. This can actually be pretty problematic if your not a single app at a time user which is why it defaults to turning on just fullscreen gsync when you toggle it.

Hardware acceleration or not probably doesn't play into it as much as how abstract are the layers i.e. are there 4 layers of hardware abstraction library to make the app work everywhere which use it's own buffering to stay smooth regardless of the underlaying system and then hands that to the OS which then buffers it as part of the compositor to be displayed or does something just blit "n" to the display framebuffer as the latter is likely to have less delay/jitter.

ubercow136• 6 years ago

Comparing 'Chrome html input' to VS Code in the article, it seems like Chrome is capable of rendering typed text with low latency. This seems to imply that the high input latency in VS Code can't be due to low level GPU and rendering pipeline issues brought about by electron, and must instead be due to processing VS Code is itself doing every time you type (syntax highlighting, semantic things, layout?). If this is true it seems unlikely monitor refresh rate or gsync could help.

londons_explore• 6 years ago

Chrome has effectively a 1 frame latency. First the event fires for a keypress, and later the compositor decides it's time for a new frame, which will include the effects of that keypress.

jart2• 6 years ago

Try DOS w/ XT keyboard.

anewdirection• 6 years ago

Amatuer. Any intterupt-based (at/xt/ps2) passive matrix keyboard, realtime kernel, and a CRT is the way of the low latency wizard.

akiselev• 6 years ago

VScode has its own renderer for drawing code tabs and the sidebar components along with custom input handling, bypassing chrome engine, I believe. They're not webviews like extension tabs and are much faster.

sime2009• 6 years ago

Sorry, but that is just nonsense. VSCode is running on plain Electron in this case. There is no "bypasses chrome" magic going on here.

hasperdi• 6 years ago

The terminal latency test more or less confirms previous tests done (https://danluu.com/term-latency/).

Alacritty is still claiming that it is the fastest terminal around, even though that claim has been proven false several times.

trishume6• 6 years ago

Alacritty makes claims about throughput, which is different than latency, and then uses the word "fast" which can be confused to mean either or both. Alacritty's benchmarks are based on how much text it can process being output to the terminal per second, which it is much better at than most terminals. However separately I think it might be true that kitty also has better throughput, while having better latency and many more features, this is just based on talking to a friend though I haven't tested this.

chmln• 6 years ago

Sadly, alacritty feels like such a waste of effort. The input lag is pretty bad, and text processing performance is a non-issue with most terminals. I've used xterm for years and never felt it was too slow at that.

Latency makes much more difference in how fast a terminal feels. This is where kitty does a fantastic job, even letting me trade some CPU cycles for really low latency with its options. I can strongly recommend it to anyone who values low latency.

KarlKemp2• 6 years ago

Throughput used to be a problem (looking at you, Hype), but the differences have become smaller as everyone upped the9r game.

This is obviously subjective to some degree. But I used to frequently get annoyed at various terminals until two or so years ago, and have just forgotten about the issue since.

riquito2• 6 years ago

3 years old article [1] on a 3 year old software that gets daily commits. Is it really relevant?

[1] there's no date, had to check Internet Archive, and they refer to 3 years old OSes

edit: I mean, so much has changed that the performance could easily be way better or way worse, I don't think it's representative anymore (usuful for future benchmarks, sure, but relevant not so much)

ubercow136• 6 years ago

Except that, like they say, the OP article from this month confirms that nothing has changed

KarlKemp2• 6 years ago

I'm not really sensitive about this issue, but recently had an experience that has raised my interest: after making some regretful mistake, I had to use an an UEFI shell to shuffle some boot settings around (the one that comes with the Clover bootload).

And typing was..noticeably more "immediate". It was bewildering, sort of like wearing prescription glasses for the first time.

I like all the extra stuff editors and the OS do these days and don't wish to turn the time back, and I doubt it's realistic to achieve both. But maybe someone like Apple, being fully integrated, should take a shot at it. It'd be the sort of polish that makes using a computer enjoyable.

trishume6• 6 years ago

In this case what you were experiencing might have been the lack of compositor, which adds a 16ms delay, or just using a slow editor. The theoretically best possible average latency under a compositor model at the top of the screen is 16+(16/2)=24ms which isn't that far from the 30ms I observe at best on macOS. Cutting the compositor would still halve average latency.

Unfortunately Apple shows no interest in getting rid of that delay, while Microsoft is moving towards sometimes using hardware overlays to avoid that (see raphlinus's comment about DirectFlip), and Linux lets you disable it at the cost of tons of artifacts. Apple only recently tried to add compositor bypass for fullscreen Metal apps but the way they did it is super opaque and I have never succeeded at getting it to trigger and they have no example program that they say definitely does it.

Flow• 6 years ago

Thanks for reminding me. Years ago, on another iMac, I did this to turn off VSync. I recall it did work. I don't know if it does work on recent macOS's though.

https://stackoverflow.com/questions/12345730/how-to-disable-...

zozbot234• 6 years ago

Do the artifacts really matter for terminal use? AIUI the main drawback of not using a compositing layer is screen tearing - which is something that's only really apparent wrt. video or animation that takes up a large fraction of the screen. It would be nice if we could be more selective about when to enable compositing, especially since even Wayland now has support for bypassing the compositing step in full-screen apps.

TheTon• 6 years ago

A single fullscreen window where the content view is either a Metal or GL layer, and nothing else in the layer hierarchy should be sufficient.

Using Instruments’s Metal and Display instruments you can see the drawables and the vsync they present on. Bring up the app switcher with cmd-tab to trigger the compositor and you can see the difference in Instruments.

One weird thing is, you can configure a 2 or 3 drawables swapchain on the Metal layer. Using 2 doesn’t work for me when the compositor bypass enables - it always locks to 30Hz, even when doing nothing but drawing a solid color. When the compositor is enabled, the 2 buffer swapchain runs at full speed, presumably because it’s effectively triple buffered. But, in the 3 buffer case, in fullscreen, you can see that you will just use 2 buffers provided you can render fast enough, so it seems like there’s no latency cost to the 3 drawable config in the ideal case.

jart2• 6 years ago

h/t Tristan Hume https://gist.github.com/jart/49cfc20d21f619c2547016be3038e89...

CyberDildonics• 6 years ago

I think it would be worthwhile to test gvim (which is known to have low latency), windows running at a higher refresh rate and a bios shell. I think the USB keyboard input standard itself is not great for latency unfortunately, but it would be good to know where the lower bound is.

trishume6• 6 years ago

I unfortunately don't have a native Linux install currently, just a VM, and my Windows NUC can only connect to my Z27.

There's some good info here on minimal USB latency, it's less than 1ms and I don't think mine is an issue although the macOS event processing stack might introduce some latency: https://michael.stapelberg.ch/posts/2018-04-17-kinx-latency-...

solarkraft• 6 years ago

Are there common approaches for measuring touchpad lag?

trishume6• 6 years ago

I wouldn't say anybody measuring input latency at all is common, but if I were doing it I'd use https://isitsnappy.com/ or some other high speed camera frame counting and check the time between when your finger starts moving and the screen changes.

ilrwbwrkhv• 6 years ago

sublime text again proving to be the fastest

snarfy• 6 years ago

If you want low latency you really need to use USB3/USB-c. USB3 doesn't use polling like USB2. USB2 latency is measured in milliseconds. USB3 latency is measured in microseconds.

Zolomon• 6 years ago

First I thought of the message polling in Win32, then I realized you were talking about the hardware. Would have been fun otherwise. :)

trishume6• 6 years ago

If you use 1000hz polling like my latency tester and keyboard do the latency becomes less than 1ms which is pretty small compared to the other factors. See https://michael.stapelberg.ch/posts/2018-04-17-kinx-latency-...

I'd optimize lots of other things like trying to avoid the compositor, higher refresh rate monitors and lower travel keys before I'd look at USB3.

zokier• 6 years ago

Huh, the results seem surprisingly uniform. The difference between different compared editors/terminals is basically one frame, and the standard deviations don't seem remarkable either. Maybe the biggest observation here is that the buffer state can significantly impact the latency in some cases (the added 10ms on vscode "aa" test), but even that was a sidenote here.

It would definitely be interesting to have more software and hardware tested, I would imagine there is lot more variance out there.

pbhjpbhj• 6 years ago

Presumably pro esports teams are measuring latency of their rigs to get the fastest gear where ms count?

If typical gamers reaction time is O(300ms) and a keyboard, change can save 40ms, or a screen can add 80ms (extreme cases) seems like they'd be wise to be testing and verifying things.

Anyone links to details of how esports teams are doing things here?

I'm curious about the light-detection - do screens have a very tight synchronisation between different colours. Could the sensor be matched in some way with the specifics of the screen?

mfalce• 6 years ago

Hey, it reminds me of what we did at my former research lab. It was a Computer Human Research team, and we were focusing on ways to measure the interaction lag when using a mouse, a keyboard or a touch device: http://mjolnir.lille.inria.fr/turbotouch/lagmeter/

I participated in writing 2 papers. In the first one, we used an optical mouse on a screen displaying a special texture to measure the end-to-end latency. In the second one, we used an external Arduino (quite similar to what the author did here) connected to a photodiode.

For what I remember, the main factor on the latency is the screen, use a gamer screen to improve the delay (CRT screens are told to be quite fast also, but we hadn't access to one to test :'( ).

gorgoiler• 6 years ago

Oh dear.

100-133ms for my keyboard to display USB hub, to laptop, and back to display.

66-83ms for my laptops own keyboard and display.

Something about my computer has always felt a bit off. For a long time I thought it was a commonly reported vim + tmux bug but it never felt like it sped up very much.

Now I just want a 386 in VGA mode again. I’m sure things were better in the good old days (said with some sense of irony.)

(Measured with 5 samples on each.)

swingbyte• 6 years ago

HI, We applied for a patent back in 1996 for a system that measured simulator to human latency through various means. The basic concept is to instrument the simulator image generator to activate a trigger i.e. a section of the display that changes in response to human input. This was monitored by a light sensor on the screen and a circuit that monitored that area of the video signal going into the display device. A timer circuit started counting when the button was pushed and stopped when the change to the display was detected. We also automated out the human to collect mass data on the simulator latency. This worked very well and gave us both the delay to video and to display times from a defined input through the complete simulator pipeline.

mncharity• 6 years ago

You can see that some combination of the light sensor and the monitor take nearly 100ms to fully transition.

That's the sensor. It's not a wonderful match for this job.

The linked[1] light sensor module looks like a CdS photoresistor. In its Amazon Q&A, the seller claims "The response time: Rise 30ms Decline 30ms", under unspecified conditions.

Better to use a phototransistor like [2], with a rise time of microseconds. 4 orders of magnitude faster. It's easy - one resistor, and an analog input on the Teensy.

Deja vu. For 3D UI prototyping on my laptop, I diy'ed LCD shutter glasses driven by an arduino with a photosensor reading a white/black square rendered flashing in a screen corner. IIRC, I actually tried a photoresistor first, but the plotted screen response curve times weren't plausible. Only then did I look for a datasheet, and had an "okaaay, that's not the right thing" moment. :/

[1] https://www.amazon.com/gp/product/B01N1FKS4L/ [2] https://www.adafruit.com/product/2831

brokenmachine• 6 years ago

An LDR like he is using has a typical response time of up to 18ms. Better to use a phototransistor which will respond in the order of μs. [1] is $0.95.

http://lednique.com/opto-isolators-2/light-dependent-resisto... https://www.adafruit.com/product/2831

Measuring keyboard-to-photon latency with a light sensor

Related Stories