talawahtech

Thanks for posting this. I just made a donation to the PSF.

Amazon Benchmarking | L5 SDEs | Seattle, WA | Full-time | https://www.amazon.jobs/en/jobs/2933471/software-development...

Help make AWS faster, more efficient, and cheaper for our customers!

Our team is hiring multiple SDEs with experience in distributed system performance and strong problem-solving/detective skills.

Feel free to reach out to the hiring manager on LinkedIn with any questions: https://www.linkedin.com/feed/update/urn:li:activity:7324200...

Cloud specific features and optimizations, particularly geared toward AWS hardware. Also a combination of a 2 year release cadence and 5 years of Amazon backed of support for each release.

Performance Optimizations: https://docs.aws.amazon.com/linux/al2023/ug/performance-opti...

Release Cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence....

Kernel Live Patching: https://docs.aws.amazon.com/linux/al2023/ug/live-patching.ht...

Cloud Init: https://docs.aws.amazon.com/linux/al2023/ug/cloud-init.html

Ships with kernel 6.1. Nice! Also has the most recent releases of bpftrace and bcc.

Full package list: https://docs.aws.amazon.com/linux/al2023/release-notes/all-p...

It's not actually an HTTP server though...

Correct, it is a fake HTTP server, serving a real HTTP workload. This post is about comparing two different networking stacks (kernel vs DPDK) to see how they handle a specific (and extreme) HTTP workload. From the perspective of the networking stack, the networking hardware, and the AWS networking fabric between instances, these are real HTTP requests and responses.

I'd really love to see this adapted to do actual webserver work and see what the difference is.

Take a look at my previous article[1]. It is still an extreme/synthetic benchmark, but libreactor was able to hit 1.2M req/s while fully parsing the HTTP requests using picohttpparser[3].

From what I recall, when I played with disabling HTTP parsing in libreactor, the performance improvement was only about 5%.

1. https://talawah.io/blog/extreme-http-performance-tuning-one-...

Yea, it is definitely a fake HTTP server which I acknowledge in the article [1]. However based on the size of the requests, and my observation of the number of packets per second in/out being symmetrical at the network interface level, I didn't have a concern about doubled responses.

Skipping the parsing of the HTTP requests definitely gives a performance boost, but for this comparison both sides got the same boost, so I didn't mind being less strict. Seastar's HTTP parser was being finicky, so I chose the easy route and just removed it from the equation.

For reference though, in my previous post[2] libreactor was able to hit 1.2M req/s while fully parsing the HTTP requests using picohttpparser[3]. But that is still a very simple and highly optimized implementation. FYI, from what I recall, when I played with disabling HTTP parsing in libreactor, I got a performance boost of about 5%.

1. https://talawah.io/blog/linux-kernel-vs-dpdk-http-performanc...

2. https://talawah.io/blog/extreme-http-performance-tuning-one-...

3. https://github.com/h2o/picohttpparser

Very cool! Most of my bpftrace usage has been on the kernel side, but out of habit I often fall back to the debugger (or printf) in userland, which can seriously distort analysis of low latency functions, especially if polling or timers are involved.

I am going to try this out next week.

Wordle was acquired for an undisclosed price in the low-seven figures.

Sure, I was just pointing out that their current rates are cheaper than gandi, especially for renewals. So if someone wants to renew for multiple years it may make sense to switch.

No, I confirmed that it is for all years. They deliberately don't make money on domain name registration. The use it to attract/retain business for their other services.

FYI Cloudflare currently charges $30 per year for .io domains.

AWS re:Invent is this week. This was announced as part of the CEO's keynote. I am sure we will get more details throughout the week in some of the more technical sessions.

I was wondering how they were going to manage the fact that AMDs Zen3 based instances would likely be faster than Graviton2. Color me impressed. AWS' pace of innovation is blistering.

I wear bluetooth headphones which are connected to my Android TV player (Nvidia Shield).

I don't go flat-out, but it is definitely high-intensity. I am always out of breath at the end, but not collapsed on the floor.

When I started I was doing around 2:05/500m. Now I am down to around 1:56/1:57, but some days I will go slower if I didn't sleep great or something. It is more about the consistency than the pace.

It is less than 8 mins total so it is enough to wake me up and get my heart pumping, but also allow me to recover quickly and feel ready to start the day.

I've been rowing twice a day, every day since the pandemic and it has been a game changer. A high intensity 2k (while listening to music) in the mornings before I start my day, and a low intensity 5k (while watching tv) at night.

It is the most consistent I have ever been with at-home exercise and it has been perfect for managing stress and getting good sleep.

1.2M req/s means 2.4M (send/recv) syscalls per second. I definitely think io_uring will make a difference. Just not sure if it will be 5% or 25%.

The answer to that question is not quite as straight-forward as you might think. In many ways, this experiment/post is about figuring out the answer to the question of "what is the best the hardware can do".

I originally started running these tests using the c5.xlarge (not c5n.xlarge) instance type, which is capable of a maximum 1M packets per second. That is an artificial limit set by AWS at the network hardware level. Now mind you, it is not an arbitrary limit, I am sure they used several factors to decide what limits make the most sense based on the instance size, customer use cases, and overall network health. If I had to hazard a guess I would say that 99% of AWS customers don't even begin to approach that limit, and those that do are probably doing high speed routing and/or using UDP.

Virtually no-one would have been hitting 1M req/s with 4 vCPUs doing synchronous HTTP request/response over TCP. Those that did would have been using a kernel bypass solution like DPDK. So this blog post is actually about trying to find "the limit", which is in quotes because it is qualified with multiple conditions: (1) TCP (2) request/response (3) Standard kernel TCP/IP stack.

While working on the post, I actively tried to find a network performance testing tool that would let me determine the upper limit for this TCP request/response use case. I looked at netperf, sockperf and uperf (iPerf doesn't do req/resp). For the TCP request/response case they were *all slower* than wrk+libreactor. So it was up to me to find the limit.

When I realized that I might hit the 1M req/s limit I switched to the c5n.xlarge whose hardware limit is 1.8M pps. Again, this is just a limit set by AWS.

Future tests using a Graviton2 instance + io_uring + recompiling the kernel using profile-guided optimizations might allow us to push past the 1.8M pps limit. Future instances from AWS may just raise the pps limit again...

Either way, it should be fun to find out.

Hard to say exactly. I have been working on this in my spare time, but pretty consistently since covid-19 started. A lot of this was new to me, so it wasn't all as straight-forward as it seems in the blog.

As a ballpark I would say I invested hundreds of hours in this experiment. Lots of sidetracks and dead ends along the way, but also an amazing learning experience.

The MTU is 9001 within the VPC, but the packets are less than 250 bytes so the MTU doesn't really come into play.

This test is more about packets/s than bytes/s.

I didn't make any coordinated omission changes (I really didn't make many changes in general), so twrk does what wrk does. It attempts to correct it after the fact by looking for requests that took twice as long as average and doing some backfilling[1].

I am no expert where coordinated omission is concerned, but my understanding is that it is most problematic in scenarios where your p90+ latency is high. Looking at the results for the 1.2M req/s test you have the following latencies:

  p50     203.00us
  p90     236.00us
  p99     265.00us
  p99.99  317.00us
  pMAX    626.00us

I am not claiming that wrk's hack is "correct", just that I don't think coordinated omission is a major concern for *this specific workload/environment*

1. https://github.com/wg/wrk/blob/a211dd5a7050b1f9e8a9870b95513...

That is an area I am curious about as well, especially if you throw io_uring into the mix. I think most kernel bypass solutions get some of their gains by just forcing you to use the same strategies covered in the perfect locality section. It doesn't all just come from the "bypass" part.

Even if isn't quite as fast as DPDK and co, it might be close enough for some people to start opting to stick with the tried and true kernel stack instead of the more exotic alternatives.

Thank you for sharing all your amazing tools and resources brendangregg! I wouldn't have been able to do most of these optimizations without FlameGraph and bpftrace.

I actually did the same thing and hacked up the perl code to generate the my custom palette.map

Thanks for the tip re: the disconnected kernel stacks. They actually kinda started to grow on me for this experiment, especially since most of the work was on the kernel side.

Recompiling the kernel using profile guided optimizations[1] is yet another thing on the (never-ending) to-do list.

1. https://lwn.net/Articles/830300/

I would also love to see that dhclient issue resolved upstream, or at least a cleaner way to work around it. But we should also be mindful that for most workloads the impact is probably way, way less.

Some of these things really only show up when you push things to their extremes, so it probably just wasn't on the developer's radar before.

Thanks, that actually means a lot. It took a lot of work, not just on the server/code, but also the writing. I asked a lot of people to review it (some multiple times) and made a ton of changes/edits over the last couple months.

Thanks again to my reviewers!

It is a statically generated site created with vitepress[1] and hosted on Cloudflare Pages[2]. The only dynamic functionality is the contact form which sends a JSON request to a Cloudflare Worker[3], which in turn dispatches the message to me via SNS[4].

It is modeled off of the code used to generate Vue blog[5], but I made a ton of little modifications, including some changes directly to vitepress.

Keep in mind that vitepress is very much an early work in progress and the blog functionality is just kinda tacked on, the default use case is documentation. It also definitely has bugs and is under heavy development so wouldn't recommend it quite yet unless you are actually interested in getting your handa dirty with Vue 3. I am glad I used it because it gave me an excuse to start learning Vue, but unless you are just using the default theme to create a documentation site, it will require some work.

1. https://vitepress.vuejs.org/

2. https://pages.cloudflare.com/

3. https://workers.cloudflare.com/

4. https://aws.amazon.com/sns/

3. https://github.com/vuejs/blog

This comprehensive level of extreme tuning is not going to be directly useful to most people; but there are a few things in there like SO_ATTACH_REUSEPORT_CBPF that I hope to see more servers and frameworks adopt. Similarly I think it is good to be aware of the adaptive interrupt capabilities of AWS instances, and the impacts of speculative execution mitigations, even if you stick to the defaults.

More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.

There is no published limit, but based on my tests the network device for the c5n.xlarge has a hard limit of 1.8M pps (which translates directly to req/s for small requests without pipelining).

There is also a quota system in place, so even though that is the hard limit, you can only operate at those speeds for a short time before you start getting rate-limited.

Yea, I looked it wrk2 but it was a no-go right out the gate. From what I recall the changes to handle coordinated omission use a timer that has a 1ms resolution. So basically things broke immediately because all requests were under 1ms.