The database servers powering Let's Encrypt (2021)

https://letsencrypt.org/2021/01/21/next-gen-database-servers.html

by alexzeitler • 3 years ago

205 119 3 years ago

letsencrypt.org

tyingq2• 3 years ago

They were issuing 2M certs/day when this happened in 2021, now up to 3.4M/day.[2]

Looks like it scaled pretty well so far.

[2] https://letsencrypt.org/stats/

jimaek• 3 years ago

Refreshing to see big projects running bare metal rather than overpaying for AWS. Especially one relying on donations and sponsors.

sho• 3 years ago

I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...

d_silin2• 3 years ago

Absolutely.

Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.

civilitty• 3 years ago

It's gotten absolutely bonkers with all the demand for GPUs and the cloud companies are just raking it in hand over fist.

On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.

qwertox• 3 years ago

If you don't need to host something big, where you can get away with one good VPS, the cloud has the benefit of offering cheap bandwidth.

Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host.

Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.

landemva• 3 years ago

In my experience, in-house groups racking and configuring and maintaining boxes often become a priesthood of negative value bickering over pet machine names.

While cloud overcharges for McD product, I know what I get.

dharmab3• 3 years ago

What you pay for is the ability to provision hardware for new R&D and projects in minutes rather than days, weeks or months. Companies are willing to spend millions a month in cloud fees to accelerate hundreds of millions in revenue.

nunez3• 3 years ago

Because there's no such thing as a free lunch, running your own datacenter (or managing datacenter managed services) is work, and the clouds are better at providing exactly the compute you need when you need it than buying a ton of excess hardware you might or might not use (or can't scale quickly enough if you _do_ use it all)

A datacenter makes sense if your usage profile is steady state and hardly ever changes, or if growth rate is predictable and capacity can be procured in advance. Any other use case is better suited for the cloud, IMO.

YetAnotherNick• 3 years ago

Because cloud infatuation is largely a myth. Every middle or large organization does periodic cost analysis and select cloud based on that analysis not on convenience. e.g. See this: https://aws.amazon.com/solutions/case-studies/emirates-case-....

louwrentius• 3 years ago

I think there are ton of great use cases for the cloud, but people should try and think for themselves and decide if their circumstances and workload is really a good fit.

A ton of people forget that a bunch of servers across a few colocations can pay for itself in months, especially if you go for second-hand gear that is dirt cheap.

Again, going (back) to colocating hardware could not be good fit. But with modern management tools and datacenter services like 'remote hands' I think people should not reject it upfront.

pid-1• 3 years ago

I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage.

AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.

However, storage costs alone would be astronomic. Like > 100,000 USD / month.

I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense.

OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but:

I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month

I would not take the other side on that bet

dharmab3• 3 years ago

I'd also bet money that this has higher disk I/O performance than the rough equivalent on Amazon.

10000truths2• 3 years ago

$225K for this sounds bonkers:

  24x 6.4TB Intel P4610 NVMe SSD = 24 x $310 = $7440
  2x AMD EPYC 7542 = 2 x $1300 = $2600
  2 TB DDR4 ECC RAM ~ $13700 (estimate from a couple of Google results)

Those add up to something like $25K. Sure, there's also the price of the motherboard, chassis, maybe some other peripherals like external network cards, assembly + support + warranty etc. but that doesn't explain an 800% markup.

CathalMullan• 3 years ago

These are expensive servers, crossing into six digits, but not $200k.

https://news.ycombinator.com/item?id=25865967

Spooky23• 3 years ago

Nobody is paying base price for a box like that. I’d probably bid them against HPE and pay ~90-100k.

If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.

riku_iki10• 3 years ago

AWS does not have a 100% similar VM

does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..

ttegloma• 3 years ago

AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.

Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month.

For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example.

Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.

jaas• 3 years ago

This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR.

- ED of ISRG / Let's Encrypt

alberth2• 3 years ago

It’s closer to $80k, if you went via a VAR.

I spec’d a current gen server with:

- 2x more cores

- 50% more RAM

- 50% more storage

For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage.

It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old).

https://www.siliconmechanics.com/system/rackform-a335.v9

gloyoyo• 3 years ago

Always nice to get an upgrade.

mark242• 3 years ago

Someone reading the headline of this might think "oh, they're using Cockroach, or Fauna, or Planetscale" -- nope, this is about next-gen hardware powering their single-write (with a number of read replicas) MariaDB instance.

esafak• 3 years ago

Indeed what I expected. It should have been titled "The Next Gen Servers Powering Let's Encrypt's Database"

riku_iki10• 3 years ago

they probably started building before Cocroach and others became solid choice, and now would need some big migration project to switch.

PaywallBuster• 3 years ago

Is there an update to this article?

Since the article was written: - AMD has gen4 available - Genoa - there's NVMe drives 15tb capacity each

rz2k• 3 years ago

And, it looks like with PCIe 5.0 x4 performance exceeds 2,500,000 IOPS reads, and 14 GBps sequential read rate.

tyingq2• 3 years ago

I would guess they try to get 3-5 years out of a new server, and this was done in 2021...so still a bit early for a refresh. The servers prior to this one had E5-2650 processors, so they would have been bought sometime between 2013-2016. Meaning their last refresh cycle was at least 5 years.

wmf• 3 years ago

Those 2021 servers will probably last them 4-5 years so they haven't upgraded yet.

mcpherrinm3• 3 years ago

The same hardware is still in use today, and I expect will continue to be used for some time.

The biggest thing that’s happened in the meantime is reducing load on the databases by pushing some of the OCSP load out to redis caches: https://letsencrypt.org/2022/12/15/ocspcaching.html By not storing OCSP in the MariaDBs, we reduced write volume and storage size significantly.

The next big thing is going to be sharding the database so multiple servers can share the write loads, and some queries moved out to redis caches.

(I work for Let’s Encrypt)

efxhoy• 3 years ago

Why add caching and sharding? Is the current architecture not going to keep up with demand?

candiddevmike• 3 years ago

Doesn't mention if they're running Linux/what flavor they're running. I personally would be wary of running OpenZFS in production on Linux, especially ZFS on root. It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable.

Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?

shrubble• 3 years ago

ZFS has been very reliable for me since 2006 or so when I first started using it on SPARC hardware with the Solaris 10 beta; I assume that since they have a backup server and a primary server, they don't update and reboot them both at the same time.

pmarreck5• 3 years ago

I personally would be wary of running OpenZFS in production on Linux

A ton of people in the enterprise have been doing this for years without issue; https://openzfs.org/wiki/Companies

especially ZFS on root

I've been running ZFS on root on NixOS since this excellent guide (https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...) for about 2 years. Zero issues. (Actually, I see they've updated it, I need to look at that. Also, they default to encrypted root. I turn it off, because the slight hit to performance and extra risk of recovery impossibility is not worth it to me.)

It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable

Well, I think you've found your problem, then. (That also might be why FreeNAS, which I also have running as my NAS, switched to Linux from Debian when they re-branded as TrueNAS Core/Enterprise.) Come over to NixOS, where you can simply reboot into any of N previous instances after an update that borks something (which almost never happens, anyway, because you can actually specify, in your machine configuration, "use the latest kernel that is compatible with ZFS"). No USB boot key needed. Here's the magic line from my own declarative configuration:

    kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

Aaaaand... DONE. ;)

Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF?

You're assuming they're not doing it, without evidence. Also, if they're already managing the security around their certs and cert generation properly, they might not need FDE. FDE is overrated IMHO, frankly, and also incurs a performance cost, as well as an extra risk cost (try recovering an encrypted drive to know what I mean). In short, religions are bad, even in technological choices; there is no single technological configuration choice that is 100% better than all possible alternative configurations.

That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?

Multiple layers of encryption incur an extra performance cost with almost no gain in extra security.

DistractionRect• 3 years ago

Usually you aren't updating production servers unless it's a security patch, fixes a problem you have, or adds a feature you want/need. Even then, usually you have a test environment to verify the upgrade won't bork the system

sofixa3• 3 years ago

Disagree, you have to continuously update production servers (not daily, but ~weekly/monthly). The more often you do it, the more automated and risk-free it is, and the smaller the version gap for when that critical security vulnerability hits that needs to be patched ASAP.

Caligatio• 3 years ago

Given that they're using a HSM (actually several), there's really not much that needs protection via FDE. The certs are obviously public and the domains are in the transparency log.

On the ZFS note: it's been rock solid for me with Ubuntu but a living nightmare with Arch. My Arch update would upgrade the kernel but then OpenZFS would semi-routinely incompatible resulting in an unbootable system.

pmarreck5• 3 years ago

I had the same issue. I went from Ubuntu -> Arch -> NixOS looking for a distro with well-supported ZFS. Finally found one (the last one).

This is the magic line from my declarative configuration that ensures I never get a kernel update that is incompatible with my ZFS-on-root config:

kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

Been running it for 2 years now; quite happy with it, and this is probably my "final distro hop". Once you go declarative (and climb the learning curve), you are done with everything else I guess.

Put it this way, I can switch to an entirely different window manager by changing 1 configuration line. And it works every time. I've tried that on other Linux distros, and it always borks something or other. I can also change my GRUB theme and boot animation via Plymouth, something I would have NEVER risked on ANY other Linux distro due to the risk of modifying boot configuration... but since it's declarative (and validated) on NixOS, I've had no issues (just tweaking). If I manage to bork something, which is rare, I just reboot into a previous update of my OS and fix it and then try again.

curt15• 3 years ago

ZFS on Linux is commonly used in HPC these days. E.g. https://computing.llnl.gov/projects/openzfs

vorpalhex• 3 years ago

You can always boot off ext4 and then just run data off OpenZFS pools. The benefits of booting off ZFS are extremely minimal compared to having your working data on ZFS.

KronisLV• 3 years ago

We currently use MariaDB, with the InnoDB database engine.

Oh hey, it's not often that you hear about MySQL/MariaDB on HN, so this is a nice change. For what it's worth, it's a pretty decent database for getting things done, even if not as advanced as PostgreSQL in some respects.

hyc_symas4• 3 years ago

Still an order of magnitude slower / less efficient than OpenLDAP / LMDB.

berkle4455• 3 years ago

Would their needs be met by a simple key-value store?

WJW3• 3 years ago

Still completely sufficient for the workload, apparently.

lelouch11• 3 years ago

The same argument came up in a big company, incidentally for the same use case, certificate store in a CA. This led to benchmarks, where OpenLDAP was significantly slower, like three orders of magnitudes. Databases have gotten really faster in the last couple of years while OpenLDAP has stagnated.

turtles3• 3 years ago

I think it is also important to acknowledge that there are things innodb does better than Postgres, eg. For most workloads an undo log is a far better data structure than implementing MVCC by duplicating rows. Autovacuum and vacuum can be an absolute nightmare, plus the extra disk traffic the duplication generates. Maybe one day OrioleDb will bring this to postgres too.

linsomniac• 3 years ago

I have had a MariaDB + Galera multi-master 3-node cluster running for several years, and prior to that Percona+Galera for ~5 years, and it's been just great. It stores lookup tables for our postfix mail server, fairly small databases and low hits, but it's great to just be able to reboot nodes for updates or migration to other VMs without having to do any of the old clustering gyrations.

I almost switched to CockroachDB in the last refresh, until I found that Postfix required Latin-1 encoding or something, and CockroachDB only supported UTF8. Postfix has more recently gotten a config option for changing that.

d_silin2• 3 years ago

Shows how much bang for buck you can get when hosting your own hardware.

tda• 3 years ago

I recently bought my first server, just about the cheapest Dell had available (still €2000). I was completely undewhelmed by the specs, it even has spinning rust. The upgrade costs to SSD's were like 250 per disk. So reading these specs, how much does such a server cost? Or do you put the disks in yourself? Can you negotiate a 50% discount? So many questions I have as a newbie in the entireprise server world.

milesdyson_phd• 3 years ago

If you are an enterprise (or just big enough) the prices on Dell’s site are meaningless, they are just conversation starters if that.

Dachande663• 3 years ago

From old knowledge, you’re out by about 2 orders of magnitude.

barkingcat2• 3 years ago

No one buys dell servers at list price.

Neil44• 3 years ago

Where possible I go third party on disks and ram.

dang• 3 years ago

Discussed at the time:

The database servers powering Let's Encrypt - https://news.ycombinator.com/item?id=25861422 - Jan 2021 (226 comments)

nunez3• 3 years ago

These performance improvements are absolutely insane, and churning out certs at speed is the _perfect_ use case for these CPUs.

Kudos the LE team!

hu3• 3 years ago

Indeed. If I had to bet, I would say doubling the memory allowed some/all indexes to fit in RAM. RAM is usually a big improvement when it comes to Relational Database Management Systems (RMDBS).

nologic01• 3 years ago

I bet if one could develop an index of useful computation per kwh (or $) this project would be near the top.

Imagine if that kind of hyperleveraged impact was the norm rather than the exception.

dmurray• 3 years ago

It's kind of hard for it to be. If the only software projects that existed were the elite hyperefficient ones, the ten guys who ran them could just meet up every year for a key signing party - no need for a scalable service to automatically issue them security certificates.

Computers being commoditized means the median project is very low-value, but it also means there are millions of cool projects out there.

distract89013• 3 years ago

Interesting that their previous server is just one or two models up from the server I recently grabbed foe $150 on eBay.

I'm just now getting serious about a homelab and it's shocking how much compute you can get for peanuts from enterprise surplus. I've had this machine (1U poweredge r620) for about a year and I'm already itching for an upgrade. Hopefully something with a lot of 3.5" drive bays for long-term data hoarding.

dorfsmay2• 3 years ago

It's because if you don't use that compute power, it's just an expensive loud heating system that's going to run up your electricity bill!

sgarland6• 3 years ago

I have three R620s. They are shockingly quiet after being booted, even under pretty decent load. If you tune C-states for energy conservation in the BIOS, they idle at just under 100W each. With moderate load (like you'd have in a homelab), around 125W. According to iDRAC, the peak they've seen is about 350W.

For better apples-to-apples, each of them has 2x E5-2650v2, 3x SATA SSDs, 8x 16 GiB PC3-12800R sticks, and 1x NVMe SSD (mounted in a PCIe adapter).

As a comparison, I also have a Supermicro X11SSH-F with a bunch (9) of spinning disks. It has a single E3-1220 v5, 1x NVMe SSD, 2x 32 GiB PC4-17000U, and averages 125W. Given the relative power consumption of HDDs, that's a pretty impressive difference IMO.

My entire rack (the aforementioned servers, and some Unifi networking gear) pulls around 500-600W in total. When the backup server (Supermicro X9DRi-LN4F+; dual-socket something-or-other Xeons, some DDR3 ram, and 8x spinning drives) kicks on daily to ingest ZFS snapshots, the total consumption is around 700W, I think. Maybe 750W.

distract89013• 3 years ago

The absolute maximum I've ever seen it hit is 400W. At my electricity rate, the theoretical maximum is something like $15/month. I'm perfectly fine with that, it's well within my hobby budget.

Really, that's what it is to me, a hobby. It doesn't have to be useful or profitable, just an interesting way to spend my free time tinkering

sgarland6• 3 years ago

We increase target & max IOPS above the defaults. We still use conservative values to avoid excessive SSD wear, but the defaults were tuned for spinning disks: innodb_io_capacity=1000, innodb_io_capacity_max=2500.

I'd be interested to see if they actually needed this. Those parameters affect the baseline and burst IOPS InnoDB is allowed to use for background tasks like flushing logs, respectively, and generally you don't need to raise them. They default to 200 and 2000; those defaults been perfectly adequate for me on a MySQL 8.x instance serving 120K+ QPS.

FpUser• 3 years ago

What? No k8, no microservices, no cloud ? What is the world coming to?

The database servers powering Let's Encrypt (2021)

Related Stories