Let's Encrypt had a higher error rate for 90 minutes today

https://letsencrypt.status.io/#2026

by widdakay • 1 month ago

161 106 1 month ago

letsencrypt.status.io

drsalt• 1 month ago

thats too bad

hermeticlock• 1 month ago

saagarjha2• 1 month ago

Seems not ideal for an entity who seems to be pushing for shorter expiration periods all the time

tonyhart7• 1 month ago

isn't this the other way around ??? because shorter expiration time resulting on more issuing cert and therefore make it more prone to downtime

RetroTechie• 1 month ago

And perhaps more opportunities to insert bad certificate somehow.

xp84• 1 month ago

I think it’s mostly Apple and maybe Google who have the hard-ons for the shortest expiries possible.

fragmede2• 1 month ago

To be fair, if someone managed to steal a set of keys to Gmail.com and icloud.com, I would want them to expire as short a time as possible too.

Dylan16807• 1 month ago

If it goes past 24 hours, that becomes a real worry.

If anyone is renewing certificates with less than a day remaining, that's an issue on their end far more than anything else.

Kesseki6• 1 month ago

To be clear, “Degraded Performance” means just that, not “down.” Let’s Encrypt’s issuance is mostly working fine.

widdakay6• 1 month ago

I have tried many times to renew my certs and have had 0 successes throughout today. It seems to be 100% degraded to me.

Kesseki6• 1 month ago

That’s unexpected. Please post details on the “Help” topic of the Let’s Encrypt community forum so that folks can take a look.

saagarjha2• 1 month ago

I see you are unfamiliar with status page-ese. “Degraded performance” is a term which means some form of “the entire datacenter is probably on fire”.

Kesseki6• 1 month ago

Although I only post here personally, I work for Let’s Encrypt.

AceJohnny22• 1 month ago

I thought it meant "electricity has ceased to be a physical phenomenon in the general vicinity of our servers"

AceJohnny22• 1 month ago

A common confusion; this interpretation only applies to OVH.

ref: https://www.reuters.com/article/world/millions-of-websites-o...

xarope• 1 month ago

That would a Microsoft'ese, "Some regions are encountering issues" => "The entire world is down, but our status page is working"

gib444• 1 month ago

What % of requests succeeded vs failed? How many certificates were issued during the outage vs the average? That might actually clear things up

greatgib• 1 month ago

They claim "Degraded Performance", but 400 and 500 error responses is a non fully working service and not a performance that is just "less good".

Some clients may encounter 400 and 500 error responses.

pibaker• 1 month ago

What are the viable alternatives to LE? And in case none exists, what does it take to build one?

Requirements: free, available to everyone, automation friendly, issues certificates that are actually considered trustworthy by other parties.

evbogue• 1 month ago

Like peers could sign sites?

treesknees• 1 month ago

ZeroSSL – free 90-day certs via ACME, also has a web UI for cert management

Google Trust Services – free ACME certs, requires a Google account for registration

SSL.com Free DV SSL – offers free 90-day certs through ACME

polpo• 1 month ago

I use acme.sh for certs on my personal server and was a little surprised when it started using ZeroSSL by default. Despite being more "corporate" I decided to roll with it and it's worked just fine.

otabdeveloper42• 1 month ago

What are the viable alternatives to LE?

None. Big tech intentionally made Let's Encrypt a single point of giant failure.

And in case none exists, what does it take to build one?

A new Internet and Web standards stack. The whole problem is self-imposed -- we could have published self-signed Ed25519 keys on the DNS instead, and the result would be more secure than whatever it is we have now.

icedchai2• 1 month ago

Do you remember the early days of SSL certificates? It took an act of god just to get a certificate: verification rituals like faxing corporate paper work, phone calls, manually reissuing certs because someone forgot the "www", forgotten renewals...

Let's Encrypt is incredible.

dlcarrier4• 1 month ago

This video explores a little on how certificate authorities were given their authority and a lot on how it can fail: https://www.youtube.com/watch?v=M1si1y5lvkk

It's a bit mathy, but if you can make it through that, I highly recommend watching the whole video, especially if you like dad jokes.

JumpCrisscross• 1 month ago

Have the EU or Canada pushed to launch an analog of their own?

It seems a bit silly that a service that could be forced by EO to revoke foreign certificates is the backbone of so much of the internet.

nubinetwork• 1 month ago

It's a good thing that acme clients try to renew early, rather than leaving it to the last minute...

ardeaver• 1 month ago

I realize this is very much not the point, but the fact that the "Active Incident" banner is green is upsetting.

NewJazz2• 1 month ago

We're operating normally, but with reduced redundancy. We continue to work with our upstream ISP to identify and resolve the issue.

Kesseki6• 1 month ago

The banner's colour is based on the "Incident Status;" it's green because services are currently operational. It would be yellow or red if the impact were more severe.

dxdm2• 1 month ago

Using only color to communicate the status is confusing. If you want to communicate something, it's often best to just say it. The color can be a visual reinforcement of that. Then your explanation would not be needed.

dlcarrier4• 1 month ago

Their monitors don't seem to be detecting the outage. Sometimes they run directly on the server, and aren't able to detect routing or DNS problems.

dlcarrier4• 1 month ago

That explains why one of my IoT vendors is using an expired certificate.

I wish Firefox would just give a mild warning for a recently expired certificate, instead of treating it the same as a true man-in-the-middle attach. It's not like someone who couldn't factor the private key in 200 days could in 201 days or even 300 days.

I'm convinced that we'd have better security, if we didn't have so much security theater. You'd think TLS is useless, from the warning my phone gives if I connected to a public Wi-Fi AP, but then again there's nothing in TLS (or WPA) that prevents it from being used in a way that is completely useless: https://www.youtube.com/watch?v=M1si1y5lvkk

jaas7• 1 month ago

That explains why one of my IoT vendors is using an expired certificate.

I don't think so. There was a dip in success rates for 90 minutes today, but nobody should be renewing their certificate within 90 minutes of expiration. If you're at that point, something went wrong weeks ago.

LtWorf• 1 month ago

weeks ago

How long do you think a certificate lives?

mannyv2• 1 month ago

"nobody should be renewing their certificate within 90 minutes of expiration"

You obviously haven't worked with hardware guys.

"I mean, what's the point of those last 30 days if you need to renew it 30 days before expiration? Why not just renew it before it expires? If I'm required to renew it 30 days before the expiration date then the expiration date is a lie, isn't it?"

dingaling• 1 month ago

I wish Firefox would just give a mild warning for a recently expired certificate

Nope, if the SSL industry continues to insist on increasingly short cert lifetimes then I want Firefox to give no quarter when a cert expires.

Play by their rules and fall by their rules too.

MobiusHorizons• 1 month ago

How does that help? Seems like mostly the end user suffers.

mannyv2• 1 month ago

Certificate expiry is less severe than an untrusted issuer or a host mismatch.

The former is most likely an administrative error (ie: someone forgot to renew, or the auto-renew is failing). The latter is more likely to be an MTM attack.

I'm not sure how you would use an expired cert as an attack vector. By loading in an old cert into an expired domain so you could spoof older content?

fragmede2• 1 month ago

omg new tom7!

bruce511• 1 month ago

But it's only the extreme warning that alerts the website (usually via a customer complaining) that the cert hasn't been renewed. Having the lesser warning just kicks the can down the road.

The IoT should have updated the certs weeks in advance. If they haven't done it by day 0 then their process is broken and delaying the scary warning to say day +5 won't solve anything.

tgsovlerkhgsel2• 1 month ago

A warning with a clear clickthrough button would work for alerting - the default TLS warnings are designed to be somewhat hard to bypass to make people think twice.

lambdaone• 1 month ago

What might be better is to, in addition to failing hard when the certificate expires, web browsers were to give a 'soft' click-through user warning if the certificate on the site - while still within its validity period - has less than say 7 days to go before expiry.

That's probably long enough for most companies to be alerted to the problem in time and to get their act together to fix the problem.

hannob• 1 month ago

There are reasons browsers do things the way they do.

Experience and user studies have shown that users have a hard time decoding what error messages mean. "This certificate is expired, but only for a little while" isn't meaningful for people who don't have a mental model of what a certificate is.

Furthermore, "downgrading" warnings increases the incentive to ignore issues, potentially causing more problems down the line.

bluesign• 1 month ago

What you want is warning when certificate expiry in next 7 days, then everyone would update before the warning.

jaas7• 1 month ago

Let's Encrypt has been working normally for most of the day. There was a ~90 minute period during which some of our users would have received a higher error rate due to upstream networking issues, but the majority of requests were successful even during that period.

It seems our status.io notes are being misinterpreted as much more severe than they were intended to reflect.

Edit: Note that this was written in response to a previous submission title implying that Let's Encrypt was entirely down most of the day.

widdakay6• 1 month ago

I'm not sure if your higher error rate is sticky per user or something, but I've tried 10+ times throughout the day and have had 0 successes. They all come back as internal server error. That's why I eventually posted.

jaas7• 1 month ago

It would not have been sticky for the entire day. If it was sticky at all, it would have been only during the 90 minute period I referenced. It's most likely that there is some other issue with how you're requesting the cert. Folks can help debug at: https://community.letsencrypt.org/

teekert• 1 month ago

Why are you trying? Doesn’t Caddy (or something) just takes care of this well in advance and should have no issues with one or several days of my service at all at any time?

Edit: my bad. I’ve tried as well recently, when you’re rushing to get your new domain up of course…

po1nt4• 1 month ago

Let's encrypt is a single point of failure for a large percentage of the internet.

gsliepen2• 1 month ago

No, it's not. You can always switch to a different SSL provider. There are other free ones (as mentioned in other comments).

However, thinking about how to make your own setup more robust without having to manually change configuration when one SSL provider stops working is a good exercise. I wonder if you can just get your server's private key signed by multiple SSL providers, and serve multiple certificates to clients, and whether all browsers handle that correctly.

doublerabbit• 1 month ago

Nothing is a point of failure if you can switch but that's not really true unless you have fail-over.

If LE was to go nope right now, how fast could you move your stack from LE?

You can't use multiple SSL certificates as redundancy. You could probably create something bespoke with a Load Balancer and SSL offloading but that's just more overhead for really nothing.

po1nt4• 1 month ago

If you couldn't switch, that would be a monopoly. But single point of failure is when you put all your fruit in one basket. Airplanes have redundant systems, even though you can always buy new components. But it's much harder to change them mid-flight.

anal_reactor2• 1 month ago

Hot take, but in general single points of failure are less of an issue than it seems because usually outages simply aren't that common. Meanwhile maintaining whole infrastructure to avoid single point of failure is often very expensive.

po1nt4• 1 month ago

In theory this sounds great, but you only realize how much do you rely on a single point of failure, once it fails. Just see github outages or even electricity outages at your home.

Let's Encrypt had a higher error rate for 90 minutes today

Related Stories