Running a Certificate Transparency log

https://words.filippo.io/run-sunlight/

by Metalnem • 1 year ago

160 60 1 year ago

words.filippo.io

agwa2• 1 year ago

Sunlight and static-ct-api are a breath of fresh air in the CT log space. Traditional CT log implementations were built on databases (because that's the easiest way to implement the old API) and were over-complicated due to a misplaced desire for high write availability. This made operating a CT log difficult and expensive (some operators were spending upwards of $100k/year). Consequentially, there have been a rash of CT log failures and few organizations willing to run logs. I'm extremely excited by how Sunlight and static-ct-api are changing this.

eddythompson80• 1 year ago

I wonder if this is the solution something like SponsorBlock is looking for[1][2]. They have a similar-ish problem. How to replicate crowdsourced data that trickles in slowly, but ideally you want replicated quickly.

WAL replication, rsync, bittorrent, etc all things that don't quite work as needed.

[1] https://github.com/mchangrh/sb-mirror/blob/main/docs/breakdo...

[2] https://github.com/ajayyy/SponsorBlock/issues/1570

tonymet4• 1 year ago

Is any amateur or professional auditing done on the CA system? Something akin to amateur radio auditing?

Consumers and publishers take certificates and certs for granted. I see many broken certs, or brands using the wrong certs and domains for their services.

SSL/TLS has done well to prevent eavesdropping, but it hasn't done well to establish trust and identity.

sleevi• 1 year ago

All the time. Many CA distrust events involved some degree of “amateurs” reporting issues. While I hesitate to call commenters like agwa an amateur, it certainly was not professionally sponsored work by root programs or CAs. This is a key thing that Certificate Transparency enables: amateurs, academics, and the public at large to report CA issues.

At the same time, it sounds like the issues you describe aren’t CA/issuance issues, but rather, simple misconfigurations. Those aren’t incidents for the ecosystem, although definitely can be disruptive to the site, but I also wouldn’t expect them to call trust or identity into disrepute. That’d be like arguing my drivers license is invalid if I handed you my passport; giving you the wrong doc doesn’t invalidate the claims of either, just doesn’t address your need.

tonymet4• 1 year ago

it seems more ad-hoc, bounty-driven , rather than systematic. is that a fair perspective?

Spivak• 1 year ago

I think over the years trust and identity have gone out of scope for TLS—I think for the better. Your identity is your domain and it's not TLS's problem to connect that identity to any real life person or legal entity. I'm sure you still can buy EV certs but no one really cares about them anymore. Certainly browsers no longer care about them. And TLS makes no claim on the trustworthiness of the site you're connecting to, just that the owner of the cert proved control of the domain and that your connection is encrypted.

I can't even imagine how much a pain it would be to try and moderate certs based on some consistent international notion of trustworthiness. I think the best you could hope to do is have 3rd parties like the BBB sign your cert as a way of them "vouching" for you.

NovemberWhiskey• 1 year ago

Meet the QWAC.

https://en.m.wikipedia.org/wiki/Qualified_website_authentica...

oasisbob• 1 year ago

Yup, it happens. There was a case I remember where a CA was issuing certs using the .int TLD for their own internal use, which it should not be doing.

Happened to see it in the CT logs, and when that CA next came up for discussion on the Mozilla dev security policy list, their failure to address and disclose the misissuance in a timely manner was enough to stop the process to approve their request for EV recognition, and it ended in a denial from Mozilla.

dlgeek• 1 year ago

Yes. All CAs trusted by browsers have to go through WebTRUST or ETSI audits by accredited auditors.

See https://www.mozilla.org/en-US/about/governance/policies/secu... and https://www.ccadb.org/auditors and https://www.ccadb.org/policy#51-audit-statement-content

tptacek5• 1 year ago

As I understand them, these are accounting audits, similar (if perhaps more detail) to a SOC2. The real thing keeping CAs from being gravely insecure is the CA death penalty Google will inflict if a CA suffers a security breach that results in any kind of misissuance.

tonymet4• 1 year ago

that's good news about the CA's , but how about the publisher certificates that are in use?

torbid4• 1 year ago

These sound like good improvements but I still don't really get why the ct log server is responsible for storage at all (as a 3rd party entity)..

Couldn't it just be responsible for its own key and signing incremental advances to a log that all publishers are responsible for storing up to their latest submission to it?

If it needed to restart and some last publisher couldn't give it its latest entries, well they would deserve that rollback to the last publish from a good publisher..

michaelt2• 1 year ago

The point of CT logging is to ensure a person can ask "What certificates were issued for example.com?" or "What certificates were issued by Example CA?" and get an answer that's correct - even if the website or CA fucked up or got hacked and certificates are in the hands of people who've tried to cover their tracks.

This requires the logs be held by independent parties, and retained forever.

torbid4• 1 year ago

I understand that. But..

If 12 CAs send to the same log and all have to save up to their latest entry not to be declared incompetent to be CAs, how would all 12 possibly do a worse job of providing that log on demand than a random 3rd party who has no particular investment at risk?

(Every other CA in a log is a 3rd party with respect to any other, but they are one who can actually be told to keep something indefinitely because they would also need to return it for legitimizing their own issuance.)

singron• 1 year ago

The publishers can't entirely do the storage themselves since the whole point of CT is that they can't retract anything. If they did their own storage, they could rollback any change. Even if the log forms a verification chain, they could do a rollback shortly after issuing a certificate without arousing too much suspicion.

Maybe there is an acceptable way to shift long-term storage to CAs while using CT verifiers only for short term storage? E.g. they keep track of their last 30 days of signatures for a CA, which can then get cross-verified by other verifiers in that timeframe.

The storage requirements don't seem that bad though and it might not be worth any reduced redundancy and increased complexity for a different storage scheme. E.g. what keeps me from doing this is the >1Gbps and >1 pager requirements.

torbid4• 1 year ago

If CAs have to share CTs and have to save everything the CT would save to their last submission then no CA can destroy the log without colluding with other CAs.

(I.e. your log ends abruptly but polling any other CA that published to the same CT shows there is more including reasons to shut you down.)

I don't see how a scheme where the CT signer has this responsibility makes any sense. If they stop operating because they are sick of it, all the CAs involved have a somewhat suspicious looking CT history on things already issued that has to be explained instead of having always had the responsibility to provide the history up to anything they have signed whether or not some CT goes away.

NoahZuniga• 1 year ago

Even if the log forms a verification chain, they could do a rollback shortly after issuing a certificate without arousing too much suspicion.

This is not true. A rollback is instantly noticeable (because the consistency of Signed True Heads can not be demonstrated) and is a very large failure of the log. What could happen is that a log issues a Signed Certificate Timestamp that can be used to show browsers that the cert is in the log, but never incorporating said cert in the log. This is less obvious, but doing this maliciously isn't really going to achieve much because all certs have to be logged in at least 2 logs to be accepted by browsers.

Maybe there is an acceptable way to shift long-term storage to CAs while using CT verifiers only for short term storage? E.g. they keep track of their last 30 days of signatures for a CA, which can then get cross-verified by other verifiers in that timeframe.

An important source of stress in the PKI community is that there are many CAs, and a significant portion of them don't really want the system to be secure. (Their processes are of course perfect, so all this certificate logging is just them being pestered). Browser operators (and other cert users) do want the system to be secure.

An important design goal for CT was that it would require very little extra effort from CAs (and this drove many compromises). Google and other members of the CA/Browser would rather spend their goodwill on things that make the system more secure (ie shorter certificate lifetimes) than on getting CAs to pay for operating costs of CT logs. The cost for google to host a CT log is very little.

dboreham• 1 year ago

Add an incentive mechanism to motivate runn a server, and hey it's a blockchain. But those have no practical application so it must not be a blockchain..

schoen• 1 year ago

There is some historical connection between CT and blockchains.

http://www.aaronsw.com/weblog/squarezooko

Ben Laurie read this post by Aaron Swartz while thinking about how a certificate transparency mechanism could work. (I think Peter Eckersley may have told him about it!) The existence proof confirmed that we sort of knew how to make useful append-only data structures with anonymous writers.

CT dropped the incentive mechanism and the distributed log updates in favor of more centralized log operation, federated logging, and out-of-band audits of identified log operators' behavior. This mostly means that CT lacks the censorship resistance of a blockchain. It also means that someone has to directly pay to operate it, without recouping the expenses of maintaining the log via block rewards. And browser developers have to manually confirm logs' availability properties in order to decide which logs to trust (with -- returning to the censorship resistance property -- no theoretical guarantee that there will always be suitable logs available in the future).

This has worked really well so far, but everyone is clear on the trade-offs, I think.

Dylan16807• 1 year ago

Yes, that is correct. (Other than the word "must"? I'm not entirely sure your intent there.) This is close to a blockchain in some ways, but a blockchain-style incentive mechanism would be a net negative, so it doesn't have that.

If you figure out a good way to involve an incentive structure like that, let us know!

some_random• 1 year ago

I'm happy to offer an incentive of 100 Cert Points (issued by me, redeemable with me at my discretion) to anyone running CT /s

In all seriousness, the incentive is primarily in having the data imo

gslin3• 1 year ago

The original article seems deleted, so https://archive.ph/TTXnK this.

FiloSottile3• 1 year ago

My bad! This is what I get for doing a deploy to fix the layout while the post is on HN. Back up now.

ysnp• 1 year ago

https://words.filippo.io/passkey-encryption/ also seems to be gone?

gucci-on-fleek• 1 year ago

https://web.archive.org/web/20250707205158/https://words.fil...

gslin3• 1 year ago

You Should Run a Certificate Transparency Log

And:

Bandwidth: 2 – 3 Gbps outbound.

I am not sure if this is correct, is 2-3Gbps really required for CT?

remus• 1 year ago

It seems like Fillipo has been working quite closely with people running existing ct logs to try and reduce the requirements for running a log, so I'd assume he has a fairly realistic handle on the requirements.

Do you have a reason to think his number is off?

ApeWithCompiler• 1 year ago

or an engineer looking to justify an overprovisioned homelab

In Germany 2 – 3 Gbps outbound is a milestone, even for enterprises. As a individual I am privileged to have 250Mbs down/50Mbs up.

So it`s at least off by what any individual in this country could imagine.

gslin3• 1 year ago

Let's Encrypt issues 9M certs per day (https://letsencrypt.org/stats/), and its market share is 50%+ (https://w3techs.com/technologies/overview/ssl_certificate), so I assume there are <20M certs issued per day.

If all certs are sent to just one CT log server, and each cert generates ~10KBytes outbound traffic, it's ~200GB/day, or ~20Mbps (full & even traffic), not in the same ballpark (2-3Gbps).

So I guess there are something I don't understnad?

xiconfjs• 1 year ago

So we are talking about 650TB+ traffic per month or $700 per month just for bandwith…so surr not a one-man-project

dilyevsky• 1 year ago

If you’re paying metered you’re off by an order of magnitude - much more expensive. Even bandwidth based transit will be more expensive than that at most colos

dpifke• 1 year ago

I pay roughly $800/mo each for two 10 Gbps transit connections (including cross-connect fees), plus $150/mo for another 10 Gbps peering connection to my local IX. 2-3 Gbps works out to less than $200/mo. (This is at a colo in Denver for my one-man LLC.)

nomaxx117• 1 year ago

I wonder how much putting a CDN in front of this would reduce this.

According to the readme, it seems like the bulk of the traffic is highly cacheable, so presumably you could park something a CDN in front and substantially reduce the bandwidth requirements.

mcpherrinm• 1 year ago

Yes, the static-ct api is designed to be highly cacheable by a CDN.

That is one of the primary motivations of its design over the previous CT API, which had some relatively flexible requests that led to less good caching.

ncrmro• 1 year ago

Seems like something that might be useful to store on Arweave a block chain for storage. Fees go to an endowment that’s has been calculated to far exceed the cost of growing storage

cypherpunks01• 1 year ago

How does the CT system generally accomplish the goal of append-only entries, with public transparency of when entries were made?

Is this actually a good use case for (gasp) blockchains? Or would it be too much data?

udev40963• 1 year ago

Instead of mainstreaming DANE, you want me to help a bunch of centralized CAs? No thanks. DANE is the future, it will happen

jcgl4• 1 year ago

I like the idea and I like DNSSEC too (well, well enough at least—lots of tooling could be better), but DANE can’t catch on faster than DNSSEC does. And DNSSEC isn’t exactly taking the world by storm.

udev40963• 1 year ago

Even if everyone catches up on DANE, we still have ICANN controlling the registrars and root nameservers. It's absolutely crazy how everyone (incl. me, unfortunately) gave in to the centralized DNS model

baobun• 1 year ago

For the downstream side Mozilla has done some great progress with CRLite, which I think hasn't gotten enough attention.

https://youtube.com/watch?v=gnB76DQI1GE&t=19517s

https://research.mozilla.org/files/2025/04/clubcards_for_the...

1vuio0pswjnm7• 1 year ago

Original HN title: "You Should Run a Certificiate Transparency Log"

Running a Certificate Transparency log

Related Stories