Investigating the impact of HTTP3 on network latency for search

https://dropbox.tech/frontend/investigating-the-impact-of-http3-on-network-latency-for-search

by kiyanwang • 3 years ago

154 62 3 years ago

dropbox.tech

giuliomagnifico• 3 years ago

Very interesting reading, thanks. I just wonder why this geographic differences from one region to another.

DamonHD• 3 years ago

RTT to the US servers will be most of it, IMHO.

abujazar• 3 years ago

tl;dr; HTTP3 provides a little improvement, but optimizing search speed is much more important in this case.

Thiez• 3 years ago

Although HTTP3 was a significant win for people with lossy connections and high latency.

abwizz• 3 years ago

like mobile

x-complexity• 3 years ago

tl;dr; HTTP3 provides a little improvement, but optimizing search speed is much more important in this case.

The article itself shows otherwise:

https://dropbox.tech/frontend/investigating-the-impact-of-ht...

------------------------------------------------------------------------------------

|p25 | -3.20ms / -6% | -2.34ms / -2% | -3.73ms / -2% |

|p50 | -4.21ms / -5% | -3.84ms / -3% | -5.12ms / -2% |

|p75 | -9.03ms / -8% | -11.1ms / -6% | -15.0ms / -4% |

|p90 | -44.9ms / -17% | -47.3ms / -13% | -77.3ms / -14% |

|p95 | -118ms / -22% | -141ms / -21% | -200ms / -22% |

------------------------------------------------------------------------------------

While the average case was negligible, the edge cases fared significantly better because of HTTP3's removal of the head-of-line blocking in TCP.

clementmas• 3 years ago

Can we use HTTP3 today? Is it widely supported?

unmole• 3 years ago

Yes. Google and Facebook have deployed HTTP/3 across their properties for quite some time now. Support in popular servers is not mature but if you're using a CDN like Cloudflare, it's trivial to enable HTTP/3 for your users.

noway4212• 3 years ago

Wondering the same- how’s the compatibility with very restrictive networks? Say a university/corporate network that only allows port 443 and DPI’d 80 egress. QUIC won’t pass through that since it’s UDP. Can server and client reliably negotiate a fallback to http/2 in such a case?

oefrha• 3 years ago

HTTP/1.1 isn’t going anywhere in any popular server, so yeah, HTTP/2 fallback isn’t a problem. (HTTP/1.1 is basically required even if you just want to serve an https redirect on port 80, since h2 requires TLS. The clear text h2c protocol has no adoption AFAICT.)

ignoramous• 3 years ago

Yep, just like IPv4 didn't, HTTP/1 and H2 aren't going anywhere anytime soon.

supriyo-biswas2• 3 years ago

Yes. Browsers typically initiate a TLS (TCP) connection to port 443 for HTTP1/2, and then upgrade to HTTP/3 based on the Alt-Svc header.

paulddraper• 3 years ago

IIRC Chrome has whitelisted some domains, but requires a flag to enable it generally.

fridek3• 3 years ago

QUIC is enabled by default for all domains across most major browsers: https://caniuse.com/http3

For Cloudflare HTTP/3 accounts for 28% of the traffic (https://radar.cloudflare.com/)

agildehaus2• 3 years ago

It remains missing from Safari. It used to be offered as an experimental feature, but I don't seem to have it anymore on Safari 16.5.

jeroenhd2• 3 years ago

You can enable http/3 just fine even if you also serve clients that are falling behind on web standards, as long as you also serve http/2 as a fallback.

Safari not supporting isn't a problem unless your business only and specifically targets Safari, and even then it's a relatively painless way to help the few customers who run preview versions get a nicer experience.

JimDabell• 3 years ago

Safari added support for HTTP/3 in version 14, released in September 2020. That’s why it isn’t listed as an experimental feature any more.

https://developer.apple.com/documentation/safari-release-not...

You can check with Cloudflare’s test page here:

https://cloudflare-quic.com

Alternatively, open up developer tools, go to the network panel, right-click on the headings and enable the protocol field. It will tell you which protocol was used for each request made by your browser. Obviously the server also needs to support HTTP/3 for a request to use HTTP/3.

shpx• 3 years ago

It is supported and randomly enabled for ~50% of Safari 16 users, according to https://github.com/Fyrd/caniuse/pull/6664#issuecomment-14934...

petecooper2• 3 years ago

Nginx 1.25 has HTTP/3 in its roadmap, first cut from that branch is scheduled to land tomorrow:

https://trac.nginx.org/nginx/roadmap

BucketsMcG• 3 years ago

We use it on our rinky-dink ecommerce site, and it represents 40% of requests at the moment. HTTP/1 and HTTP/2 are at 30% each. Our users skew a bit older, and definitely aren't techno-savvy (one scrolled past all the products and used the contact us page to ask for a printed product catalogue and order form...).

esprehn• 3 years ago

HTTP/1 is surprising, what browser is that coming from?

Wagthesam2• 3 years ago

One thing I'm wondering about is that http3 seems to be from end user to search, which is a general serving problem and not really about search. Do dropbox clients call the search service directly?

Wagthesam2• 3 years ago

I have wondered recently about head of line blocking's impact to latency at AZ level within AWS. How reliable are TCP connections within the same datacenter and do these issues noticeably impact performance at the tail end?

diroussel• 3 years ago

Maybe there are some answers to that here

https://aws.amazon.com/blogs/hpc/in-the-search-for-performan...

Where AWS explain why TVP isn’t good enough for them and why they are developing a replacement.

rimtutituki3• 3 years ago

Just wandering, what http3 solutions are on Linux side. Can IPVS be http3 compatible at all?

jeroenhd2• 3 years ago

Why not? As long as you can send and receive UDP traffic, there's no reason why you can't do http/3.

Caddy and HAProxy both support the protocol. nginx doesn't have it in the public stable version yet, though there are preview packages available for modern server platforms.

rimtutituki3• 3 years ago

Yeah. Should test it, would be interesting to see if DR is working with UDP balancing (via IPVS).

petecooper2• 3 years ago

nginx doesn't have it in the public stable version yet

It's near, roadmapped for 1.25:

https://trac.nginx.org/nginx/roadmap

bbss2• 3 years ago

HTTP/3/QUIC supports migrating connections between two networks, such as if a user switches from WIFI to LTE. IPVS or any UDP load balancer won't handle this scenario properly since it doesn't introspect the QUIC header and load balance based on the QUIC connection ID. This QUIC connection ID allows for a stable connection when the device needs to switch networks. If operators have any sort load balancer (like IPVS) between the client and the point the HTTP/3 connection is terminated, they will need to ensure that it has proper support for QUIC. One example is Katran[1] which has support for this method of load balancing.

[1] https://github.com/facebookincubator/katran

why_only_15• 3 years ago

This is a nice post, but it feels a little weird because while HTTP/3 gets them 10ms on average, they could get 30ms on average by reducing Asia latencies (15% of traffic, 300ms) to North America latencies (100ms). I'm sure they have some internal constraint that makes that difficult, but just stuck out at me.

supriyo-biswas2• 3 years ago

That might require them to put replicas of the search service in Asia, and operating a distributed system over distant geographies is a hard operational problem.

jefftk2• 3 years ago

Not necessarily: they could terminate the initial connection close to the user and then run a new connection. While this sounds like it shouldn't help, because the second connection is fully under their control they can make it highly reliable and very fast. Additionally, any recovery from dropped packets (on the user's home network) only has the latency of an in-region connection.

This also sets them up for ruining a light cache in-region, and potentially scaling that up to handle more kinds of API calls.

Neil44• 3 years ago

Yes I'm sure they've considered doing that as it seems so obvious. I would also think that the use of any one account was very region specific so could you actually base their whole data out near them. Be interesting to hear how they've ended up at the compromise that they have.

jupp0r2• 3 years ago

Or they could place the to-be-searched data closer to their users and get rid of traffic to North America entirely for most users.

chrisweekly2• 3 years ago

Related tangent: highest possible recommendation for https://hpbn.co/ (High-Performance Browser Networking), an extraordinarily helpful book and website.

mablopoule• 3 years ago

Didn't knew about this book, it looks extremely interesting, thank you :).

pksohn• 3 years ago

I second this recommendation, but note that this book predates and does not cover HTTP3/QUIC.

chrisweekly2• 3 years ago

Good disclaimer. HPBN is still recommended reading (almost prereq?) before tackling H3/QUIC, though.

londons_explore• 3 years ago

It would be nice to see that 99th percentile, the 99.9th percentile, and the 99.99th percentile.

My browser has made over 100,000 network requests so far today, and I expect yours has too. I'm sure at least some of those will be in the 99.99th percentile.

endorphine• 3 years ago

It would be interesting to see similar experiments published from big edge networks like Cloudflare and Fastly. Do they have something similar published?

asymptotic2• 3 years ago

While HTTP3 does provide some improvements, it's clear that further optimizations are needed, particularly for users in Europe and Asia.

One potential solution that hasn't been discussed much here is leveraging AWS managed services like AWS CloudFront or AWS Global Accelerator. It's worth noting that Dropbox's website already uses AWS CloudFront, so they are already leveraging AWS in some capacity. Based on my cost calculations, using AWS CloudFront would cost around $40k a month, while AWS Global Accelerator would be around $22k a month.

As of August 22, 2022, AWS CloudFront supports terminating HTTP/3 in a Point of Presence (POP), which could potentially help with the latency issues Dropbox is facing. AWS Global Accelerator, on the other hand, is designed to improve the performance of applications by terminating UDP/TCP as close to users as possible then routing user traffic through the AWS global network infrastructure. This could help reduce latency by ensuring that user traffic is routed through the most optimal path, even if the user is located far from a Dropbox data center.

It's hard to estimate what the potential latency reduction of using e.g. AWS Global Accelerator is, especially at higher percentiles. However, using https://speedtest.globalaccelerator.aws/, and assuming symmetry, my connections to Asia are 35-40% lower latency.

Of course, there are trade-offs to consider when using managed services like AWS CloudFront and AWS Global Accelerator. While they can provide significant performance improvements, they also come with additional costs and potential vendor lock-in. However, given the scale of Dropbox's operations and the importance of providing a fast, reliable search experience for their users, it may be worth exploring these options further.

---

Cost estimates

Assumptions:

    1. Dropbox's peak traffic is 1,500 queries per second (QPS).
    2. Average data transfer per query is 100 KB.
    3. 50% of the traffic comes from North America, 25% from Europe, and 15% from Asia (remaining 10% from other regions, for pricing purposes put it into Asia's calculations).

---

1) AWS CloudFront cost estimation:

    Data transfer:
    - North America: 1,500 QPS * 0.5 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 194.4 TB  
    - Europe: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB  
    - Asia: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB

    Data transfer cost:
    - North America: 194.4 TB * $0.085/GB = $16,524
    - Europe: 97.2 TB * $0.085/GB = $8,262
    - Asia: 97.2 TB * $0.120/GB = $11,664

    Total data transfer cost: $16,524 + $8,262 + $7,006 = $36,200

    HTTP requests:
    Total requests: 1,500 QPS * 60 seconds * 60 minutes * 24 hours * 30 days = 3,888,000,000

    Using the updated AWS CloudFront pricing (as of May 22, 2023):
    - HTTP requests cost: 3,888,000,000 * $0.0075/10,000 = $2,916

Total estimated monthly cost for AWS CloudFront: $36,200 (data transfer) + $2,916 (HTTP requests) = $40k

---

2) AWS Global Accelerator cost estimation:

    Data transfer:
    - Total data transfer: 194.4 TB (NA) + 97.2 TB (EU) + 97.2 TB (Asia) = 388.8 TB

    Using the updated AWS Global Accelerator pricing (as of May 22, 2023):
    - Data transfer cost (averaged across regions): 388.8 TB * $0.035/GB = $13,608
    - (Also need to add EC2 egress cost, 388.8 TB * $0.02/GB = $7,776
    - Total data transfer cost = $21,384

    Accelerator:
    - Assuming 1 accelerator with 2 endpoints (1 for HTTP/2 and 1 for HTTP/3)
    - Accelerator cost: 1 * $18/accelerator/day * 30 days = $540

Total estimated monthly cost for AWS Global Accelerator: $21,384 (data transfer) + $540 (accelerator) = $22k

blerb795• 3 years ago

Former Dropbox employee, just correcting one assumption:

Dropbox's peak traffic is 1,500 queries per second (QPS).

I can't speak to search QPS directly, but most individual serving hosts for file sync/retrieval were receiving tens of thousands of QPS. The overall edge QPS peaked at several hundreds of thousands QPS every day across all the hosts. So I'd guess that even just search is an order of magnitude higher than 1,500 :)

asymptotic2• 3 years ago

I misread the article when it stated:

Traffic regularly exceeded 1,500 queries per second (QPS) at peak times

It may be quite cost prohibitive to use a managed service like AWS Global Accelerator.

jupp0r2• 3 years ago

Note that some of the users on the worst networks are also users that won't be able to use HTTP3 due to corporate firewall restrictions that block UDP and downgrade TLS.

Source: I've worked on network protocols for video conferencing and have seen my share of corporate network horrors.

cbsmith• 3 years ago

Hopefully, as HTTP3 picks up steam, this will stop happening.

But yeah, there's no amount of innovation that can't be undone by a motivated adversary. ;-)

Investigating the impact of HTTP3 on network latency for search

Related Stories