Web Latency Benchmark: A new kind of browser benchmark

http://google.github.io/latency-benchmark/

by ggurgone • 13 years ago

102 54 13 years ago

google.github.io

Low latency is more important than overall throughput when it comes to UX, so that is a measurement that makes a lot of sense. Has anyone experience with it?

109876• 13 years ago

1.1 out of 10 running the latest stable version of Chrome on a 2010 13" MacBook Pro with a 2.40GHz Core 2 Duo, 8GB of RAM, and a Samsung 840 SSD. Geez... Is there anything I can do to improve the score?

hnriot• 13 years ago

odd, i have much the same mac, same cpu/ram with a regular 7200rpm disk, same chrome and i get 2.5

dmourati2• 13 years ago

I learned a new definition for jank today!

3.0/10 on MPBR early 2013 w 16GB and SSD.

Authors should consider a way to collect and present the data.

cheald• 13 years ago

3.3 on a beefy Windows 8 machine running Chrome 31.0.1650.39 beta-m. Interestingly, running an Incognito tab with no extensions allowed to run in incognito results in a 2.7.

Huh.

dmourati2• 13 years ago

Lower scores are better since we are measuring latency?

gjm11• 13 years ago

Looking at the source, higher scores are better. There are a bunch of tests, and each one produces a score from 0 (bad) to w (good) for some w. The total score is the total of all these.

Given that all the other numbers on the page are ones for which smaller is better, I think the main score itself should be accompanied by a note saying explicitly that higher scores are better.

lucideer2• 13 years ago

keep on getting https://github.com/google/latency-benchmark/blob/master/src/... on the first test... not sure why.

modeless15• 13 years ago

Interesting, what platform and browser?

lucideer2• 13 years ago

Vista all attempted browsers (tried FF, Chrome, Opera)

benologist• 13 years ago

I can't help but think this should be a test for websites, not computers. It's individual sites full of 3rd party shit that are slow and you can see the performance penalty by installing Ghostery.

wtallis• 13 years ago

There are already good tools for measuring network delays. Firefox's web inspector does a beautiful job of graphing how the various requests contribute to overall page load time. This tool is about measuring how responsive the UI is, even when you're still fetching more data.

modeless15• 13 years ago

It's more of a test for browsers than computers. You can compare Chrome vs. Firefox vs. IE vs. Safari.

Measuring the input latency of individual websites would be interesting too. I'd love to incorporate that into the benchmark if I could figure out a way to do it.

Zikes2• 13 years ago

1.8/10 on a Dell Latitude laptop.

Did anyone else have difficulty getting it to work on a multi-monitor setup? It gave me an error about trying to find the reference image and suggested I switch monitors, but that didn't work. I finally had to disconnect my secondary monitor altogether before it would work.

modeless15• 13 years ago

(benchmark author here) Interesting. I will have to do more testing on Windows multi-monitor setups. I would not be surprised to learn that it's broken.

Zikes2• 13 years ago

If it helps at all, I'm on Windows 7 with my laptop screen as my primary monitor and an LCD screen to the left as my secondary.

I was initially in Win7's "enhanced graphics" mode (can't remember actual feature name) but when I tried to run the benchmark I was thrown into Basic mode. I had a fair amount of other applications running, though, so I may have just run out of memory.

euroclydon• 13 years ago

Multiple monitors. Windows 7. Just got a black screen in both IE9 and Chrome Canary.

madh• 13 years ago

2.8 / 10.0 on a 1 week old 15" MBP (high-end) running latest stable Chrome.

k3n2• 13 years ago

Keeps failing for my Win7x64/30.0.1599.101 m.

Failed to find test pattern on screen. Ensure that your browser's zoom level is set to "100%", and the top-left corner of the window is visible. If you have multiple displays, try moving the browser window to the main display.

* Zoom 100%: check.

* Main monitor (of 2): check.

* Top-left visible: check.

I give up.

modeless15• 13 years ago

Sorry to hear that. Due you have the Chrome flag for "Experimental Web Platform features" enabled? I recently learned that there's an issue with the test related to this flag.

Edit: It's more likely that it's just buggy support for multiple monitors. I will look into fixing it; for now the only workaround would be disabling your second monitor.

k3n2• 13 years ago

Sorry for the late reply; I did notice a Github issue for it so I commented there. Thanks!

andrethegiant• 13 years ago

Has anyone else heard the term jank before? It's the first time I've seen it.

mcpherrinm• 13 years ago

The Firefox team definitely used the term with this usage, since at least 2011.

Pwntastic• 13 years ago

I use/have used the term 'janky' before to describe similar kinds of things or things that seem off or strange.

Grew up in the midwest fwiw

azinman22• 13 years ago

We (at Empirical) use janky to describe animations that aren't butter as this author uses it. Not sure how standard but I suspect many people (native speakers) will understand when in context.

makepanic• 13 years ago

There is a website [1] with more informations and talks about "jank".

[1] http://jankfree.org/

rmccue• 13 years ago

(I'd guess it's originally related to the concept of "jerk" in physics. http://en.wikipedia.org/wiki/Jerk_%28physics%29 )

It's been used in this context since at least 2010 by the Chrome team: http://en.wiktionary.org/wiki/Talk:janky

codexon• 13 years ago

No, usually the term I have heard used for this is jitter.

sp332• 13 years ago

The windows version disables the "Aero Glass" transparency effects on my Win7. A first glance looks like it might not be intentional, but it triggers some kind of compatibility mode in the OS. Not sure why a headless daemon would mess with transparency effects.

modeless15• 13 years ago

Benchmark author here. This is unfortunately intentional due to the way the benchmark works. Measuring latency requires taking screenshots really fast, and on Windows 7 it is very difficult to impossible to take fast screenshots with Aero enabled. In Windows 8 they added a new DirectX screenshot API[1] that makes this unnecessary (which is good because it is impossible to disable Aero on Windows 8).

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/hh40...

azinman22• 13 years ago

Has anyone assembled benchmarks across the different browsers yet? I like the aims of this benchmark a lot -- I've recently switched away from Firefox to Chrome simply because it feels less janky.

modeless15• 13 years ago

(benchmark author here) I don't have comprehensive numbers compiled yet, but I can give you a general ranking of browsers. From best score to worst, the current ranking is roughly:

  Safari
  Firefox (except Mac)
  IE 11
  Chrome and Mac Firefox

Semaphor3• 13 years ago

I wonder if I'm using my browser differently than others or if there is something wrong with my machine. Whenever I try to switch from Chrome to FF it seems way behind Chrome in terms of speed and snappiness.

edit:

Windows 8.1, SSD, i3, 16GB RAM (63% used)

Chrome 30.0.1599.101 m: 3.4/10

Chrome Incognito: 4/10

Firefox 25: 5.4/10

FF Private Browsing 3/10 O.o

All tests done at least twice with similar numbers.

But Chrome has been running for a few days while FF was just started. Guess I'll leave FF running a bit and see if it changes anything.

Also interesting, FF takes around 2-3 times as long to run the tests than Chrome.

The reason for the low Chrome score is this:

JavaScript jank Tests responsiveness during JavaScript execution. CSS: 1.0 frames jank, Scrolling: 170.1 frames jank

rMBP2• 13 years ago

Does not work in Safari on Mavericks using a rMBP @1680x1050 retina resolution. It suggests browser zooming be set to 100% which it is.

modeless15• 13 years ago

Thanks for the report; I see the problem. I'll look into fixing it, but in the meantime you can get it running by enabling WebGL in Safari (under the Develop menu, and if you don't have a Develop menu you can enable it in Preferences->Advanced).

Spoiler: Safari actually gets a very good score (relative to other browsers).

rMBP2• 13 years ago

Enabling WebGL works. My three tests scored 8.7, 6.7 and 6.3 respectively. I guess Safari, like me, hates repetitive tasks. :)

selectodude• 13 years ago

I do have WebGL enabled, and it can't pick up the test pattern.

aaron42net• 13 years ago

I also get "Failed to find test pattern on screen." in Safari with WebGL enabled on a Mid 2011 Mac Mini running 10.9 with dual monitors. View is set to "Actual Size", and I've tried moving the Safari window between monitors.

Other scores - Chrome 30: 2.4/10. Firefox 24 and 25: 1.2/10.

nej• 13 years ago

Getting different scores when the test is ran at different browser sizes.

3.4 when the window is sized to it's smallest size.

2.5 when it's at maximum resolution (1600x1200).

Running on OSX 10.9.

modeless15• 13 years ago

Yes, window size can definitely affect latency and jank. The benchmark page has a hidden full-window-sized layer that the browser must paint every frame, and a large browser window requires more CPU and GPU work to paint.

carlob• 13 years ago

I managed to get a 6.5 on Safari on Mavericks on a 3.5 year old MBP. All the other browser I've tried were below 2, except for Opera Next (2.2).

nrp• 13 years ago

Awesome to see a non-Virtual Reality use of our Latency Tester.

modeless15• 13 years ago

It's an awesome piece of hardware, thank you for making it! I'm currently brainstorming ways to use it more extensively in the tests.

This benchmark does have a connection to virtual reality, as part of my inspiration to make it came from my experience using vr.js[1] and WebGL for virtual reality in browsers[2]. The Web isn't yet ready for VR due to latency issues, but I hope to change that!

On that note, are you guys interested in browser support for the Rift? I believe the latency issues are surmountable with effort and some cooperation from browser vendors, and I'd love to see something official in that direction (in fact, I'd love to work on something official).

[1] https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=7...

[2] This is what I made: http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s

ygra• 13 years ago

Somehow the second test always timeouts in IE 11 on Windows 8.1 for me.

Web Latency Benchmark: A new kind of browser benchmark

Related Stories