Revoking certain certificates on March 4

https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864

by teddyh • 6 years ago

364 147 6 years ago

community.letsencrypt.org

shakna• 6 years ago

Thankfully it's fairly painless to see if you're affected:

    curl -XPOST -d 'fqdn=example.com' https://unboundtest.com/caaproblem/checkhost

Replace example.com with your Fully Qualified Domain.

MrStonedOne2• 6 years ago

That can only test port 443, and not say, sstp on port 8384, or soap/https on port 3443 or other rando ports for various internal https layered applications

rswail4• 6 years ago

It can only test hosts that it has access to. Otherwise you can download the file and check your serial number against the list.

Ayesh2• 6 years ago

All my recent certificates are affected. I think it is the same case for the majority of us.

vesinisa• 6 years ago

I use DNS validation to obtain one certificate with both domain.tld and *.domain.tld wildcard. My certificates seem not to be affected. My certs last auto-renewed on 2020-02-20.

tialaramex12• 6 years ago

The most likely cause will be that you issued similar certificates a few days (more than eight hours) earlier with the same Let's Encrypt account.

Suppose on Wednesday you get a cert for example.com and www.example.com, and then on Thursday you realise you also need images.example.com - you use the same ACME account (if you run Certbot this will happen by default if you use the same machine and user account, it silently makes you a free account if you don't have one already) and so Let's Encrypt can see that this account showed proof-of-control for two of these names on Wednesday, so only fresh proof-of-control of images.example.com is needed. Unfortunately this bug means Let's Encrypt forgot to re-check CAA for the old names, and so there's a risk they technically were no longer authorised to issue for these names and shouldn't have given you the Thursday certificate.

Rather than try to argue about whether it's appropriate to disregard this check, Let's Encrypt decided to revoke all ~3 million affected certificates. That's maybe 2-3 days worth of normal issuance in the last 90 days, so lots but hardly "the majority".

owenmarshall• 6 years ago

We confirmed the bug at 2020-02-29 03:08 UTC, and halted issuance at 03:10. We deployed a fix at 05:22 UTC and then re-enabled issuance.

Wow. That is a truly impressive way to handle a security bug and I know it’s not the first time Let’s Encrypt has responded extremely quickly.

I would love to hear how their engineering practices make this possible.

d33• 6 years ago

And I'm curious about the work culture park. They're a small organisation in terms of workforce, but somehow managed to respond within minutes on Saturday. How does that work? Do they have shifts or somehow the workers are so devoid of private life that they have to respond during early morning on weekend?

icebraining• 6 years ago

In California, it was Friday evening still.

m_eiman• 6 years ago

I assume they probably have 24h support, but also consider time zones: 3 AM Saturday in UTC is Friday evening in San Fransisco:

https://duckduckgo.com/?q=03%3A00+utc+to+pst&ia=answer

michaelt• 6 years ago

The blog post only says they halted issuance within minutes of confirming the bug - not that they confirmed the bug within minutes of receiving a bug report.

lidHanteyk• 6 years ago

They don't have many incidents or outages. As a result, it's much easier to respond to the incidents that do occur.

tialaramex12• 6 years ago

For me at least 24/7 incident response is completely acceptable in a properly compensated role so long as it's accompanied by the culture that says preventing such incidents in the first place is Job #1

That is, I'm OK with being woken at 0200 to try to understand and if appropriate fix or recover from a disaster only so long as if I'd suspected this might happen the people expecting me to be awake at 0200 would have given me the resource (money, people, whatever) to fix it. If I feel like I don't have that support, I'll only start looking at your disaster during my working day.

My impression is that ISRG pays a lot of attention to preventing disasters, so if I worked for ISRG (not very practical since they're based on the US West Coast and I live in England) I'd be comfortable taking a call in the middle of the night to fix things.

dboreham• 6 years ago

Out of hours response to a critical problem is standard. You can achieve it in various ways but they all boil down to people who know what they're doing having a professional ethic. Typically it isn't possible to have shifts of engineers with deep understanding of the code on call so ultimately you need to wake someone up. So remember to keep a note (up to date) of key staff home phone numbers, their home addresses.

pgporada3• 6 years ago

Speaking for myself here.

The workforce is spread across the states. When you're drawn to the mission like I am, late nights here and there don't matter at all. I communicated with my wife and I'm sure others informed their significant others what was going on etc and why Friday, Saturday, Sunday, Monday, and Tuesday would be thrown out of wack. Members of the team put in much more hours than I did and that is truly impressive. It takes all of us with our different specialties to make an accurate and effective response.

Some of the things we do are internal post mortems and find ways to prevent the issue from happening again by either improving alerting/monitoring, writing a runbook, fixing code, and fixing misconceptions about a part of the entire system. We do weekly readings of various RFCs, the Baseline Requirements, and other CAs CP and CPS documents to again better understand our system and Web PKI as a whole. This is an understatement, but we heavily rely on automation. From the moment the call was made to stop issuance, an SRE was ready to run the code that disables the issuance pipeline.

The biggest takeaway is that communication and leadership makes all the difference.

I have to go, there's work to be done.

jaas• 6 years ago

Head of Let's Encrypt here.

We have an on-call rotation and a system for getting others notified and online quickly when necessary. We make sure not to bring too many people online so that some people are fresh and can rotate in later if the incident lasts longer.

It's not often that staff have to put in time at night or on weekends, and when it happens we work hard to make sure the problem doesn't happen again.

MrStonedOne2• 6 years ago

CAA

What is a CAA? Letsencrypt: please dont use initialisms in customer facing blog posts without using the FULL name at the first use. Makes things more learnable and googleable.

anderskaseorg• 6 years ago

https://en.wikipedia.org/wiki/DNS_Certification_Authority_Au...

tyingq• 6 years ago

Confused me too. They even already have a page they could have linked to. https://letsencrypt.org/docs/caa/

Ayesh2• 6 years ago

Certificate Authority Authorization.

It's a fairly common initialism in CA/TLS world (heh). The DNS record is also named "CAA".

rswail4• 6 years ago

It's a free service. I understand your frustration, but CAA is a well known DNS record type. If you read the description of the problem, it explains it.

djsumdog3• 6 years ago

I've been doing development and devops for years and have never heard of a CAA. Then again, I use http-01 and not dns validation, so that's probably why. Before LetsEncrypt I'd buy certs and it involved a lot of copying/pasting CSRs into web interfaces.

lvh• 6 years ago

CAA is a term of art for a standard encoded in DNS. It stands for "Certification Authority Authorization", but most people who know what a CAA record is probably do not recognize it written as words. (I know I would need to read it several times to know that's what they meant, and I do this for a living.)

tialaramex12• 6 years ago

Expanding other DNS record names isn't very helpful either. I know what a Canonical Name for something is, but CNAME seems clearer, Mail Exchanger definitely isn't a helpful way to think about what MX records are for...

PTR and TXT aren't initials they're just short for "pointer" and "text" neither of which is much help divining what they're actually used for, and presumably AAAA doesn't actually stand for anything at all (?) other than it's four times bigger than the A record.

deweller• 6 years ago

Will a nightly certbot invocation will replace these revoked certificates without manual intervention?

londons_explore3• 6 years ago

I don't believe so.

I hope they add support for that soon.

schoen3• 6 years ago

It was included in the Certbot 1.3.0 release today.

https://community.letsencrypt.org/t/certbot-1-3-0-release/11...

(only certbot-auto users are likely to get this release immediately)

nitely• 6 years ago

Run `certbot renew --force-renewal`. That's what it says in the email they sent me. But if you didn't get an email, then your domains should not be affected.

Tomte2• 6 years ago

My cert is affected, and I didn't get an email.

Check your domain using the linked online tool.

gindely4• 6 years ago

I didn't get the email, and my domains were affected. (Two of them are in the list, one of them current.)

I do get "you didn't renew your certificate" messages on a semi regular basis (domains that have passed out of my control) so I know they have my details.

simias• 6 years ago

Apparently you have to manually use --force-renewal for certbot to regenerate new certificates (I just did it just in case, even though I'm 99% sure that I'm not concerned).

I assume that by default certbot only checks the expiration date of local certificates against the system clock, it doesn't ping any external resources so it can't be aware that the certificate might have been revoked even though it hasn't expired.

I agree that it would be nice if there was such an option, although I assume that it would increase the server load significantly if certbot connected to letsencrypt's servers at every invocation so maybe that's why they didn't do it.

tehlike• 6 years ago

Actually i think it would not be a significant issue. At least for a range of situations. A static file with time stamp ranges in case of an issue could be resource-efficiently served to signal clients.

lightswitch05• 6 years ago

I assume that by default certbot only checks the expiration date of local certificates against the system clock, it doesn't ping any external resources so it can't be aware that the certificate might have been revoked even though it hasn't expired.

I think the actual issue here is that the certificates have not been revoked yet. We know that they will be revoked, which is why we have to run with --force-renewal, but there is no process for certbot to know that a certificate, although not revoked, will soon become revoked. I would expect certbot to automatically renew the next time its ran post-revocation.

schoen3• 6 years ago

Prior to today's 1.3.0 release, no; in versions of Certbot ≥ 1.3.0, yes.

ge0rg• 6 years ago

There is a list of all affected certificates posted under https://letsencrypt.org/caaproblem/ - and it looks like they are also leaking the account IDs from the list, so now you can map different domains/certificates to the account that got them issued.

tialaramex12• 6 years ago

Yeah, it does seem like it'd have been sensible not to list the account ID in this file. It's convenient if you know your account ID and want to pull out just your certs, but for most people this associates all their certificates together.

If you own both https://www.happy-rainbow-nursery.example/ and https://hardcore.bdsm-videos.example/ you probably go to some lengths to avoid visitors realising the connection. Nothing you're doing is illegal or even unethical - but it's obviously going to cause uncomfortable conversations so why not avoid that altogether. Let's Encrypt aren't doing you a favour if tomorrow a mom at nursery says now she knows why you sound so much like Masked Mistress Martha...

djsumdog3• 6 years ago

Yea, this is really bad. I've done some searching of the data. Sometimes it doesn't matter. It looks like whoever is currently running gab.com is probably a big consulting company with like 100 other clients, so there's no big relation there. But if you run a small personal blog and use the same e-mail address for maybe more controversial sites that are hosted on different IPs, now you could get doxed.

I'm guessing customer IDs are associated with e-mail addresses? This seems like a good case of using different e-mails for ever cert. There are open source tools like anonaddy.com you can host yourself or buy from them (they have a decent free tier).

I feel like this list seriously needs to be pulled. There is some serious lack of oversight here.

appleflaxen• 6 years ago

There are several links in this thread, but the following page allows you to enter your hostname and check online.

(no SSH, terminal access, etc) and it's from the letsencrypt team (linked in blog post).

https://unboundtest.com/caaproblem.html

zimpenfish2• 6 years ago

Alas, only works for things on port 443 which is a bit of a problem for most of my certificates...

b4d• 6 years ago

If anybody needs a bulk check:

for domain in $(cat domains.txt); do printf "$domain :" && curl -XPOST -d "fqdn=$domain" https://unboundtest.com/caaproblem/checkhost; done

karimmaassen• 6 years ago

Yes, let's. https://www.reddit.com/r/ProgrammerHumor/comments/7x2ugb/let...

32gbsd2• 6 years ago

So it starts

GuyPostington• 6 years ago

So what starts?

32gbsd2• 6 years ago

random bugs

tialaramex12• 6 years ago

I suppose this is one way to answer my question

I asked (on m.d.s.policy) on the 29th how many issuances were affected, Jacob replied saying they intended to spend that day figuring out the answer, but then there was nothing further from him. The incident doesn't seem drastic enough to prompt urgent answers so I intended to revisit later this week if I heard nothing further.

Now we have a complete list of affected certificates instead, (the answer to my original question is about 3 million)

I was sort of hoping the answer was going to be like five thousand or something manageable. Alas. In hindsight I guess this was to be expected.

thenewnewguy4• 6 years ago

Well, from my understanding, they have no idea of whether or not a given certificate was misissued as long as it meets the baseline criteria for triggering the bug (which seems to be just "issued after X date with more than 1 domain").

So while the number of certificates that should not have been issued due to a blocking CAA record is likely small (or possibly even 0), they have to revoke every cert that could have triggered the bug, as they have no way to travel back in time and find out what the CAA records they didn't check would have been.

tialaramex12• 6 years ago

The criteria also require that the challenge answers used to validate control were old (more than eight hours old).

If all proof-of-controls were fresh the CAA checks are also fresh for those proof-of-controls so there's no bug. That's why the big list is "only" about three million certificates.

Suppose you own example.com, example.org and example.net and all you do is every 60 days or so you spin up Certbot once to get a certificate for six names, the three domains and the associated www FQDNs - that won't trigger this bug because each time your old proof of control have expired and new fresh ones will be used, triggering fresh CAA checks.

You're right though that it's likely the number of truly mis-issued certificates may be zero because the most common way to have a CAA record deliberately changed to forbid Let's Encrypt after having successfully done a proof-of-control (the scenario that would trigger their bug) is researchers looking for bugs in CAA checking, and of course such researchers would have reported this to Let's Encrypt triggering exactly the same incident but probably at a more friendly time like a Monday morning.

IceWreck• 6 years ago

Yes, I just got their mail. Four certs out of six, all issued at the same time were affected. The other two were not.

rswail4• 6 years ago

This bug only affects you if you got domain validation (eg by dns-01) but didn't immediately issue a certificate.

Letsencrypt validates the domain ownership for 30 days, so the bug allows you to issue a certificate within that window, even if you added a CAA record after validation that says "don't allow issue by letsencrypt, or only allow issue by MyCA.example.com".

But if you have everything automated, you're checking for renewal and issuing every day and probably validating as part of that, so unlikely to encounter the bug, unless you validate in one step and then sometime 8h+ later, issue a certificate.

JeanMarcS• 6 years ago

Thank you. I was just about to run tests on all my clients certs but as I’ve never done that it seems it’s not usefull.

(Will still have a look but less stressfully)

kn1002• 6 years ago

I don't think this is true - I don't recall using domain validation. I think it's more related to multi domain certs.

saluki• 6 years ago

y, I use an automated tool to issue/renew them and all of mine that I checked were listed as OK.

wbond6• 6 years ago

Emailing users and giving them only 24 hours before their certs are revoked seems very unreasonable. Say you are down and out with a stomach bug or on holiday for a day or two.

My understanding is that the 90 day lifetime is largely because revocation can be thwarted. Thus the practical difference between 24 hours and one week is meaningful for server admins, but inconsequential if someone is staging an attack.

thdrdt• 6 years ago

Well it is very inconvenient, but isn't this the strength of certificates?

When something is wrong you can revoke them immediately.

Why leave a potential vulnerability open for more than 24 hours?

wbond6• 6 years ago

Because revoking them will cause interruption to legitimate users, but doesn’t stop an attack.

I’m just starting to think LE is more aimed at large organizations than people running smaller configurations. Which is fine, thankfully we still have traditional CAs. I just hope we don’t devolve into a monoculture of ACME-only SSL.

pfg6• 6 years ago

The Baseline Requirements for publicly-trusted CAs (section 4.9.1.1) require timely revocation of mis-issued certificates - either 24 hours or 5 days depending on the reason. I'm not entirely certain which is applicable here, but I'd assume Let's Encrypt's hands are tied in this case.

wbond6• 6 years ago

That is a very useful bit of info. I guess if the mis-issuance happened on Friday evening PT, then fives days is March 4th.

djsumdog3• 6 years ago

I'm glad they decided on the 24 hours, unlike CAs like Comodo which really shouldn't still be a CA after all their fuckups.

tialaramex12• 6 years ago

Not thwarted exactly, but the problem is that the question "Is this certificate still good?" has three possible answers:

1. "Yes, it's still good"

2. "No, it's revoked"

3. "There was a network problem so I'm not sure"

Of course bad guys who know you'd get answer 2 can most likely ensure you have answer 3 instead. So the only safe thing to do is treat 2 and 3 the same. If we're not sure this certificate is fine then it's not fine. But in practice answer 3 is common anyway. For some users it may happen essentially all the time. So browser vendors don't like to treat 2 and 3 the same, even though that's the only safe option and that can thwart the effectiveness of revocation.

There's definitely further opportunity for improved tooling here. Perhaps this incident will drive it (Let's Encrypt's sheer volume can help in this way).

willglynn• 6 years ago

OCSP is a request/response protocol intended to answer certificate validity questions. It works as you describe, and failures cannot be treated as errors. An attacker who stole a certificate can use it even after revocation by blocking access the relevant OCSP responder.

https://tools.ietf.org/html/rfc2560

OCSP stapling is a mechanism by which a TLS server can make OCSP requests ahead of time and serve the response in-band. TLS clients get a certificate signed by the CA as usual, as well as a recent OCSP response signed by the CA attesting to its continued validity. OCSP stapling allows TLS clients like browsers to know a certificate's revocation status without having to make an extra request, but it changes nothing for an attacker who stole a certificate since they can simply not use it.

https://tools.ietf.org/html/rfc6066#section-8

OCSP Must Staple is an option that can be included on a certificate stating "I promise to use OCSP stapling". An attacker who stole a "must staple" certificate can either include an OCSP response indicating the certificate is revoked, or they can omit an OCSP response which the TLS client will treat as a hard error.

https://tools.ietf.org/html/rfc7633

In short, RFC 7633 makes certificate revocation work. Web browsers and web servers support this today. If you use Let's Encrypt's `certbot`, pass it `--must-staple`.

wowaname5• 6 years ago

The problem is LE had five days between reporting the bug and revoking as per Baseline Requirements, but they decided to wait until the last day to send out E-mails. This is not only inconsiderate to customers but also downright reckless to sensitive deployments involving LE.

rswail4• 6 years ago

It also only affects you if you are issuing the certificate for more than one domain name if I'm reading it right.

What's supposed to happen:

  For each fqdn in the request
    if challenge succeeds (eg dns-01)
      check whether caa record exists
      if (it doesn't) or (it does and allows issue)
        issue certificate

In the step on "check whether caa record exists", instead of using the domain name that is being issued in this loop, it uses the first one it found (or one of them, it's unclear which one). So theoretically, if you wanted a cert for:

  domain1.example.com, domain2.example.com

and you had a CAA record for domain1 that allowed letsencrypt but then a different CAA record was added between the CAA check on domain1 and the CAA check on domain2 (which wouldn't happen because of the bug) you could get a cert for domain2 that the CAA record said not to issue.

captncraig2• 6 years ago

Kinda frustrating having certs revoked when there have never been CAA records for any names involved. I know they can't know that to be true historically, but I wish they could do some additional filtering.

terom4• 6 years ago

Here's some quick&dirty stats from the list of revoked certificates: https://gist.github.com/SpComb/6338facd12e020ec4fe561ca91f32...

There's 3M "missing CAA checking results" in total, of which 2M are dated from 2020 and 1M from last month. FWIW the only certs of mine affected were old certs from 2019-12 which had since already been renewed in Feb, and the renewed certs are not affected?

The largest account has 445k certs revoked, and the most revoked certs from last month (most likely to still be in active use?) is 43k for a single account. I hope your rate-limits are in order if you're going to start reissuing all of those before midnight :/

BTW account number 131 at the top of the file seems to mostly be akamaiedge.net sites :)

terom4• 6 years ago

For context in terms of what caused this, here's the PR which I assume fixed the bug in question: https://github.com/letsencrypt/boulder/pull/4690

It looks like a nasty and subtle pass-by-reference of a for-range local variable, although I'm having trouble figuring out where the reference is stored: https://github.com/letsencrypt/boulder/blob/542cb6d2e06e756a...

I've spent plenty of time hunting down similar bizarre bugs in Go code as well, where the called function ~implicitly~ takes a pointer to the iteration variable and stores it somewhere. Each iteration of the for loop updates the stack-local in-place, and later reads of the stored reference will not read the original value. It's hard to spot from the actual call site :/

EDIT: This was an explicitly taken `&v` reference, but the same thing can also happen implicitly, if you call a `func (x *T) ...` method on the variable.

heavenlyblue• 6 years ago

People say Rust’s borrowing rules are only useful for multi-threaded environments. This is one of those issues that they are supposed to solve.

Thaxll• 6 years ago

I'm not sure I understand, it's a business logic bug, would have happen in any language.

rcaught• 6 years ago

Is it just me or is the PR and the associated linking really lacking? The PR doesn't have a description and neither it or the commits link back to the original communication (or vise versa).

terom4• 6 years ago

For even more context, this seems to have been on a Friday night (assuming US West coast) with production down: https://letsencrypt.status.io/pages/incident/55957a99e800baa...

I'll cut the LE team some slack on this one :) the PR does have tests

gwd2• 6 years ago

What's particularly unfortunate about this is the comment just above the call:

    // Make a copy of k because it will be reassigned with each loop.

But v is reassigned with each loop too.

The real question is why there's so much pass-by-reference in the first place. K looks to be a domain name string -- it's almost certainly faster to copy it than to dereference it everywhere.

thenewnewguy4• 6 years ago

The real question is why there's so much pass-by-reference in the first place. K looks to be a domain name string -- it's almost certainly faster to copy it than to dereference it everywhere.

I don't program in rust, so my knowledge here is limited to what these words mean in C/C++ - however shouldn't making a copy still require dereferencing the copy?

terom4• 6 years ago

LE just posted their own (excellent!) incident report of this on the mozilla bugtracker, including the discovery timeline, analysis of the bug, and follow-up steps: https://bugzilla.mozilla.org/show_bug.cgi?id=1619047#c1

The original bug report, which was initially diagnosed as only affecting the error messages, not the actual CAA re-checking: https://community.letsencrypt.org/t/rechecking-caa-fails-wit...

Brief discussion on revocation exemption requests: https://bugzilla.mozilla.org/show_bug.cgi?id=1619179

Tomorrow will tell if granting a revocation exemption might have been a good idea in hindsight.

kn1002• 6 years ago

Yeah my domain was affected. Renewed, that would have sucked if I hadn't seen this! Also, I didn't get an email and I'm pretty sure my certs were generated with my email!

rb808• 6 years ago

Certificates are such huge a maintenance problem. So many mines waiting to blow up. We really need something better.

talkingtab• 6 years ago

So lets see, the deal is that I get free certificates on any and all of my domains. I get an easy way to install and update my certificates that works with my nginx services. I can move the service to a new address and instantly get a new certificate. The certificates are universally trusted. I get notified by email when there is a problem along with a way to detect and fix the problem available to me.

I'm speechless. I used to pay real money to get certs without half the service I get now for free.

Thanks letsencrypt.

wowaname5• 6 years ago

I get an easy way to install and update my certificates

You're lucky, because it took me forever to devise a renewal system that met my requirements of being able to renew certificates for domains that are used across multiple machines, both physical and virtual, many of the virtual machines which share IPv4 behind NAT thus cannot sanely use HTTP-01 renewal. Fine, use DNS-01, but: having to deal with my strict DNSSEC setup where I sign zones at home on a trusted device, keeping my keys off live servers, ensuring keys don't end up on untrusted hardware, and wanting my TLS private keys only on the servers they're being used on—all that suddenly makes my deployment more complex.

What I'd rather do is, since my nameserver runs a cronjob to pull signed zones from my home device (unauthenticated because the signed zonefiles are public information), I wanted to incorporate that into my DNS-01 ACME verification too. But all the existing tools (certbot, dehydrated, acme.sh, et al) seem not to support this style of setup well, where I can batch my DNS challenges, wait for my nameserver to pick them up, and then verify the challenges en-masse an hour or a day later (LE challenges last for seven days, so this is acceptable delay for me).

But I'm stuck with the CNAME/alternate NS approach where I have to run yet another network-facing service from home, for the duration of renewals, because I simply do not have time nor patience to sift through ACME's specification and implement a better tool suited for my needs.

I get notified by email when there is a problem

LE has a mailing list for critical announcements in the interest of current customers and interested prospectors? Do tell, because all I know about is their blog and the Discourse forum, neither of which are solely for critical announcements.

ferzul• 6 years ago

LE has a mailing list for critical announcements in the interest of current customers and interested prospectors? Do tell, because all I know about is their blog and the Discourse forum, neither of which are solely for critical announcements.

I think they plan to improve their communication after this mishap.

But you can use an rss reader to subscribe to the incidents category (search incidents.rss in this page), which is very low traffic. It's not “action required” level only, but with only two or three a year it may be suitable.

qXlihgad7n• 6 years ago

Is there an RSS feed that you can subscribe to for alerts like this? I can't see one.

thenewnewguy4• 6 years ago

https://community.letsencrypt.org/c/incidents.rss perhaps?

(Pro tip: you can append .rss to many pages on discourse to get an RSS feed)

low_key2• 6 years ago

Looks like LE will be adding to the billion certs they've issued!

https://news.ycombinator.com/item?id=22434466

low_key2• 6 years ago

PSA: The unbounded checker doesn't seem to work if you have certificates issued for both ECC and RSA keys. For some of mine, it passes the check with status "OK" and shows the serial number of the certificate for the ECC key. The certificate that is going to be revoked is not shown.

tialaramex12• 6 years ago

If you have more than one certificate in use, regardless of what flavour, they only see one and assess that. Maybe the checker should emphasise that. For small users they probably only have one certificate in use, so this avoids some problems.

The issue would probably also affect people who have geographically separate certificates e.g. if you have two servers in different regions and decided rather than make things more complicated for key distribution you'll just have them each get their own certificates for the same name - that's totally fine with Let's Encrypt (it doesn't scale, but if you had 500 servers not 2 you'd probably redesign everything) but obviously this test only sees one of those servers and won't check the other certificate.

There's no way to know, given that two (or more) valid certificates exist for a name, and seeing one of them, whether the others are still actively used anywhere.

It would obviously be pretty easy to build a web form where you can type in an FQDN and get told if any certificates matching that name will be revoked, but then you get false positives where it says yes, this certificate for some.name.example will be revoked, you rush to replace your certificate for some.name.example but maybe actually the one that will be revoked is from 20 December 2019, and you already got a newer one which was unaffected in February.

namibj2• 6 years ago

I wish they'd issue short-term scoped CAs under the same criteria as they currently use for wildcards.

No significant load on their infrastructure, and you'd not have to break the "private keys don't move over _any_ network" rule.

bobmaxup2• 6 years ago

Only 13 out of ~3,500 of certificates I manage required renewal

vermontdevil• 6 years ago

3,500? wow. What are you managing if I may ask?

bobmaxup2• 6 years ago

A CMS platform with customer provided domains.

donatj• 6 years ago

Can we get a “(some)” for the title?

mholt• 6 years ago

FWIW, I believe any Caddy sites will not be affected by this since caddy does not manage multi SAN certificates. Even if it did, Caddy will immediately replace a certificate when it sees a Revoked OCSP response. So, if you're using Caddy, there's probably nothing you need to do. But if you are a caddy user and are impacted, let me know.

Tomte2• 6 years ago

I haven't got a mail (I think), and I don't see that on their web site or on their blog.

Is wading through Discourse threads now the new minimum requirement for using services?

gindely4• 6 years ago

No, you can get your information from Hacker News.

(Do check your domain even if you didn't get an email, since they have not delivered emails to everyone who is affected.)

gindely4• 6 years ago

If like me you have several hundred certificates to check, please do something like this:

cd somewhere-nice

wget https://d4twhgtvn0ff5.cloudfront.net/caa-rechecking-incident... gunzip caa-rechecking-incident-affected-serials.txt.gz

cat serials/* | tr -d " " | sort | uniq > serials.collate

grep $( cat serials.collated | head -c-1 | tr "\n" "|" | sed -e 's/|/\\|/g' ) ../caa-rechecking-incident-affected-serials.txt

It will take a moment and then it may tell you that letsencrypt misspoke when they said they sent emails to everyone whose contact details they have.

Operyl2• 6 years ago

I thought I was in the minority there! We have 45 certificates (of many more) that were affected, and our account id was listed, and it has an email contact associated. I got no email whatsoever, but I'm glad I had the foresight to check anyway.

gindely4• 6 years ago

I just noticed I got an email at 1949 UTC. I guess they're still sending them out. Presumably some people will receive their emails after the revocation.

paulfurtado• 6 years ago

Does anyone know of a generic way to detect that letsencrypt will revoke a certificates soon?

The goal would be to have our automation automatically rotate the certificates when similar issues occur in the future.

pfg6• 6 years ago

To my knowledge, there's no such mechanism in any of the relevant protocols (i.e. ACME and OCSP).

ck2• 6 years ago

That thread enlightened me to a great trick to force cert renewal even if it's been done too recently: add a second (sub)domain and make a new cert with both

geocrasher• 6 years ago

Here's a bash one-liner to check all the domains you have:

for domain in $(cat list-of-domains.txt); do curl -s -X POST -F "fqdn=$x" https://unboundtest.com/caaproblem/checkhost ;done | sed '/is OK./d'

TwoNineFive• 6 years ago

They have known for over 30 days and they didn't notify me until about 10 hours ago that several of my domains were affected.

That's not okay. That's serious negligence.

EDIT: My bad. They have known about this for maybe a week, and they did a blog post 4 days ago. I was basing my thought that they had known for 30 days based on the title of the post and in the URL: https://community.letsencrypt.org/t/2020-02-29-caa-recheckin...

forbiddenlake• 6 years ago

I'm confused about what you mean; 2020-02-29 was 4 days ago, not 30.

They didn't know the extent of the bug until the 29th of February: https://bugzilla.mozilla.org/show_bug.cgi?id=1619047#c1

chrisMyzel• 6 years ago

Oh Let's Encrypt - what a chaos tuesday this was for me - I was informed yesterday, 19:00 (UTC+7) and it was a fun evening - many befriended companies and their clients where not informed at all.

Funny to find out the otherwise awesome traefik has no force-renewal other than messing around with acme.json until it renews

robjmills2• 6 years ago

Is there anything to suggest if this relates to Certbot or API provisioning only? or both? We've checked a whole bunch of API v1 provisioned certs against their tool and nothing has been listed so far.

robjmills2• 6 years ago

FWIW I think the reason we're unaffected (as far as we can tell so far) is because we're not re-issuing certs within a short time period. The bug their end was to do with checking CAA records, if you re-issued the cert for a multi-domain cert within a short period of time after the initial provision then it wouldn't re-check the CAA records. This meant that subsequent CAA changes wouldn't be checked and theoretically a cert could be re-issued despite a CAA record being added to prevent this. As i'm reading it, if you didn't re-issue within this timeframe then your cert can be assumed to be correct as the original CAA check wasn't a problem.

Revoking certain certificates on March 4

Related Stories