A brief history of code signing at Mozilla

https://hearsum.ca/posts/history-of-code-signing-at-mozilla/

by todsacerdoti • 1 year ago

246 88 1 year ago

SilasX• 1 year ago

Wow. No mention in this history/hagiography of the 2019 global add-on outage where (after banning unsigned add-ons even if you enabled them in about:config) they forgot to update their certs?

Some of the HN threads:

https://news.ycombinator.com/item?id=20421948

https://news.ycombinator.com/item?id=19871989

Edit: removed speculation if consequences of privacy loss, but come on … that’s still pretty serious.

jvehent4• 1 year ago

Yeah. Sorry about that...

We didn't think that the intermediate CA expiring would break the signatures, because code signing generally doesn't care about expiration, but we never tested the code path until the intermediate expired and the signatures broke. That was a hard lesson to learn...

extraduder_ire• 1 year ago

I've seen the idea floated for combatting non signing related time-based bugs, but I'm a firm believer in having at least one machine run tests with its time set artificially far in the future (e.g. 1 year) to catch these ahead of time where possible.

peterldowns• 1 year ago

The one that probably got people killed

Got any more info on this? Seems like an absurd exaggeration.

alexsereno• 1 year ago

It’s such an insane take that proof would be amazing

hansvm• 1 year ago

I doubt there's proof (much like how China harvesting organs from certain races and religions didn't have solid proof till recently), but it's in the right ballpark. 50+ journalists are killed (and documented as such) each year, for their journalism, despite their attempts to maintain privacy. Privacy measures being suddenly dropped wouldn't make those numbers better.

Toss in another 150+ human rights activist murders per year, dozens of whistleblowers, 60+ transgender people, and any number of other at-risk groups who tend to rely on technical measures to preserve their privacy and safety abroad.

Did that info being leaked definitely get them killed? No clue. Did it "probably [get] people killed"? I don't know how those people are typically uncovered, so it's hard to say if a few privacy extensions actually mattered. It doesn't seem like an outlandish claim though (and with a prior that those extensions do matter, it's a likely claim).

scosman4• 1 year ago

Does anyone have a windows code signing system they like? Something on GitHub actions ideally.

I’m distributing unsigned binaries not because I care about the $200 for a cert, but I don’t want to spend hours figuring out a signing flow.

landr0id• 1 year ago

I haven't tried it but have been looking into the same problem. This is probably the best bet (does require an Azure account though): https://github.com/Azure/trusted-signing-action

scosman4• 1 year ago

Azure looked promising, but their trusted signing service won't issue a cert unless your corp is 3 years old. Doesn't look like you can bring your own keys here.

donmcronald• 1 year ago

I thought that if you want a code signing certificate that isn't useless it has to be an EV certificate and those require an HSM.

How would that ever get worked into GitHub Actions?

scosman4• 1 year ago

I thought OV still bypassed smart screen and virus false positives from Windows defender?

Although I think even OV require HSM now for local, with some options for hosted services like Azure Trusted Signing.

But I really don't know. I read details only to end up not certain.

weinzierl• 1 year ago

Isn't Microsoft switching to Trusted Signing[1] and you can get or you'll have to get the cert from them directly in the future?

Seems not to make the process easier, because apparently your organization has to be more than 3 years old[2] to be eligible.

For the traditional way with a third-party cert provider someone on Reddit made a comparison table[3]. The whole thread is interesting.

[1] https://learn.microsoft.com/en-us/azure/trusted-signing/over...

[2] https://learn.microsoft.com/en-us/answers/questions/2082252/...

[3] https://www.reddit.com/r/electronjs/comments/17sizjf/a_guide...

jagged-chisel• 1 year ago

As of November 27, 2024:

We are working on making the functionality available for organizations that were incorporated less than 3 years ago. We don't have an ETA yet to share.

From your link at [2]

paulddraper2• 1 year ago

You have two options:

1. Get an EV certificate from a trusted CA [1]. That will run $400-800.

2. Use managed Trusted Signing from Azure. [2] Identity validation takes ~a week. Signing can be done via web or CLI. This does require a tax history of 3+ years. [3]

[1] https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

[2] https://learn.microsoft.com/en-us/azure/trusted-signing/quic...

[3] https://learn.microsoft.com/en-us/azure/trusted-signing/quic...

CaliforniaKarl• 1 year ago

It's not using GitHub Actions, but it is possible to use a YubiKey 5 FIPS model as an HSM for the purpose of getting a code-signing certificate.

https://support.sectigo.com/IS_KnowledgeDetailPage?Id=kA03l0... is an example explaining how to get both a CSR and an attestation certificate from a YubiKey 5 FIPS, on Windows.

https://support.yubico.com/hc/en-us/articles/360016614840-Co... explains how to use Windows' signtool with a Yubikey.

Using a YubiKey does require that you provide your PIN every time you want to do a signature, which limits how much you can automate things. A YubiHSM would remove that requirement, and might be able to work with a self-hosted GitHub Actions Runner, but it's more expensive, and you'd want confirmation from your CA (Sectigo, for example) that a YubiHSM is OK.

Stratoscope• 1 year ago

Using a YubiKey does require that you provide your PIN every time you want to do a signature, which limits how much you can automate things.

AutoHotkey is your friend!

When I set up our code signing machine at AltspaceVR ten years ago, I wrote a simple little AutoHotkey script that watched for the signtool PIN popup and typed in the PIN. It was maybe 15-20 lines of code.

Problem solved.

likeabatterycar2• 1 year ago

What is wrong with Microsoft's free signtool.exe?

bagels3• 1 year ago

Okay, you need an EV certificate (required to be on a hardware token), and next you need to integrate it in to a CICD pipeline.

eXpl0it3r• 1 year ago

Haven't used them, but for open source, SignPath might be interesting, they do require transparent builds, but do support GitHub Actions: https://about.signpath.io/product/open-source

Some other signing that works in the cloud and has support for GitHub Actions would be DigiCert's KeyLocker (note: every signed binary is counted and by default the subscription only contains 1000 signings): https://www.digicert.com/signing/code-signing-certificates#c...

bagels3• 1 year ago

I just went through this process. It's a racket. It's really poorly documented. I rolled my own. It's a lot of work. It almost inspired me to make a product, but it's not a business I want to be in, especially if Microsoft does eventually relax their 3 year old business requirement for their Trusted Signing product.

My formula was: Github actions, Sectigo usb key, physical windows machine in the office that runs a self-hosted action runner that does the signing step.

MrAlex94• 1 year ago

I use a Azure Key Vault[1], and jsign[2] to do this.

You can execute it on any platform that supports Java, so I have Linux builds cross compiling to windows with clang and then sign with jsign: https://github.com/BrowserWorks/Waterfox/blob/7eda3b998a56ad...

[1] https://trustzone.com/knowledge-base/purchasing-an-ev-code-s... [2] https://trustzone.com/knowledge-base/purchasing-an-ev-code-s...

tonyedgecombe• 1 year ago

but I don’t want to spend hours figuring out a signing flow.

That part is trivial, the real pain comes from dealing with the certificate providers.

mscrivo• 1 year ago

I've actually got an action for this (based on fork, so can't take full credit): https://github.com/mscrivo/signtool-code-sign

you can see a usage example here: https://github.com/mscrivo/OotD/blob/main/.github/workflows/...

tfolbrecht• 1 year ago

I use jsign and AWS KMS. It's a lot less expensive than a full on HSM or "cloud HSM" service.

There's a lot to it, and expect a ton of a back and forth emails with a CA to get an EV Cert.

Here's some example code for generating the CSR (Certificate signing request) to submit to a CA.

https://github.com/aws-samples/csr-builder-for-kms

electroly• 1 year ago

I ultimately concluded the automation was too much work and I didn't want to figure it out. I bought the Sectigo cert (they will sell code signing certs to individuals) with the USB dongle and installed their crappy software (SafeNet Authentication Client), and I manually sign builds from GitHub Actions on my workstation with "signtool."

scosman4• 1 year ago

Based on comments here, Azure Trusted Signing with a personal certificate and their GitHub action worked.

This blog post below is great. It works through the process, including the truly weird/awful UX choices Azure makes. I archived it on wayback in case it ever disappears.

https://www.hendrik-erz.de/post/code-signing-with-azure-trus...

piker• 1 year ago

https://melatonin.dev/blog/code-signing-on-windows-with-azur... is the way forward if you haven't found a solution already.

jvehent4• 1 year ago

Hey Ben!

(I made and used to maintain Autograph at Mozilla)

bhearsum_• 1 year ago

Hey Julien, long time! We should catch up!

kangs• 1 year ago

hey Mozillians (:

jvehent4• 1 year ago

Ping me on signal when you get a chance!

Vinnl• 1 year ago

One thing I appreciate about being at Mozilla is that there are quite a few old-timers around, and that so much work always happens in public. A lot of institutional knowledge gets preserved that way!

RunSet• 1 year ago

You might ask the old-timers whether any of them knows why this bug is unfixed since 2009.

https://bugzilla.mozilla.org/show_bug.cgi?id=505521

Surely there is some rich lore to be mined.

jaharios3• 1 year ago

Does everything need to be signed? Use a tool to much and you end up becoming it's slave.

For a long time a had problems with some websites not loading videos when opened in background because I had an addon which disabled the page visibility api. So I download the source of the addon ( straight from the addons store right click the install button and save as..) changed the manifest to exclude the said sites, zip it back and install it again?

Well your addon needs to be signed to be installed permanent even when you manualy install it. I understand that this is to keep tech-illiterate users safe from malicious addons. So is there a flag/about:config setting for advanced users to enable which skips the check? Nope, there was one but not anymore. So what options do I have? Either use beta version of Firefox as my main browser (no thanks). Or create an account as a developer, upload my addon and wait for Mozzila to check and sign my addon (that I created for myself).

braiamp• 1 year ago

The reason for that is like you expect, someone abused that, and we do not have nice things anymore. That's the same reason why apt-get doesn't allow you to remove essential packages on Debian, without modifying a configuration file.

bravetraveler• 1 year ago

DNF has a similar mechanism, I appreciate it when poorly-written third party package manifests try to remove/deprecate more than they should

megous7• 1 year ago

I can not run my own extension in Firefox by modifying a config file. It's not possible. Not even if I don't let dishonest actors anywhere near my Firefox install.

I can murder some trees and poison the environment for all of us, to do pointless mutli-hour re-builds of Firefox for each release and point release to have it accept my add-ons, though.

I've also never seen a reason, why I can't at least place my CA into Firefox /usr/lib/firefox folder or /etc/firefox and have it be respected. Or just place local extensions there and have firefox not require signatures for them, because there's no way these can be installed accidentally from web by clicking some link.

And if someone can trick me into modifying /usr/lib, they can just trick me into replacing Firefox completly with their malwared build, so signing will not save me anyway.

eadmund2• 1 year ago

Honestly, that trade-off is not worth it, particularly with browsers. People who do ignorant things are ignorant (note that I did not say stupid: they may be quite intelligent in other areas of life). Tools should do what their owners direct them to do, even if it is a mistake.

With apt, there is a bit of method to the madness: if one imagines that apt’s job is to do the thing the user requested and then deliver a correct system, then clearly essential packages can’t be removed.

But there is not a reason to prevent the user of the software from using the software in the way he wishes on his own machine.

solarkraft• 1 year ago

The biggest browser by far, Chrome, can somehow afford allowing people to install unsigned extensions, so Firefox ought to be able to as well.

umvi• 1 year ago

Does everything need to be signed? Use a tool to much and you end up becoming it's slave.

In a trusted society full of honest people, no.

In an untrusted society with dishonest people, the amount of resources (money, time, cpu, memory) required to protect yourself from abusers will only ever keep growing.

megous7• 1 year ago

No. For SW distribution, yeah sign the software in transit to my disk or whatnot. That's good so I know who I'm getting it from.

But requiring it for running the code, that's just anti-software-freedom behavior.

userbinator2• 1 year ago

Or just patch the binary to remove the check, like what real users did when they encountered software they didn't like 100% the behaviour of, whether or not they had the source.

(Yes, I know Firefox is open-source and you can theoretically compile it with the check removed, but that is going to take far more time and effort than finding and NOP'ing a jump.)

SunlitCat5• 1 year ago

Yeah that one! I wonder when the first installer comes around patching known versions of Firefox to accept unsigned addons to dump their awesome toolbar onto the user. (Or just outright overwrite the users Firefox with an own version of Firefox, not including said check)

prerok• 1 year ago

Yeah, and you have to do that for every update. No, thanks.

Ayesh• 1 year ago

You can still run unsigned extensions on Firefox Developer Edition (FF beta with some developer tooling) with an about:config tweak.

SunlitCat5• 1 year ago

The privacy notice[0] says:

In addition to the data collection described in this Privacy Notice, these versions by default may send certain types of web activity and crash data to Mozilla and in some cases to our partners.

Is there any way to switch off that additional snooping around in stuff they shouldn't do because it's not their business what I do on the web with the only version allowing me to run unsigned addons?

[0] https://www.mozilla.org/en-US/privacy/firefox/#pre-release

SunlitCat5• 1 year ago

The required signing of addons (even local installed ones) is the main reason i won't use firefox anymore. And no, using a developer build (which might work or not) isn't fixing it.

Even Edge allows me to run unsigned addons (even if it's with a warning, but for my use case I am fine with that one).

UltraSane• 1 year ago

just patch the binary to remove the check

mike_hearn2• 1 year ago

Code signing is a trip alright. I make a build tool called Conveyor that simplifies all this way down for developers [1] as part of making shipping desktop apps a single command, and damn has it been a lot of work.

First challenge: the tools provided by the OS developers only run on their OS and are of questionable quality, with bad error messages, poor documentation and unintuitive defaults. So I used a mix of open source and custom code to make signing work for every OS on any OS with good errors, docs and sample apps. The open source libs out there are slowly catching up with this, but they don't solve every part of the task and aren't integrated with everything else you need to ship apps (like laying out a working app tree, adding a software update engine etc). It's nice because it lets you ship from cheap Linux CI workers.

Second challenge: you get a proliferation of private keys. Keys to sign your Mac and Windows binaries, keys to sign your Mac update feeds, keys to sign apt repositories. So, Conveyor can derive them all from a single root key that you can back up on paper with a pen. Except for Windows where this worked up until Microsoft started requiring all keys to be held in an HSM.

Third challenge: said HSMs. They're a pain. As the Mozilla blog alludes, some tools like to require things to be done with a GUI. So, Conveyor can detect PKCS#11 drivers and drive the signing process itself using passphrases supplied via CI secrets or similar. It can also store the root key in the Mac keychain if you want to use that, which is a pretty nice way to keep it secure as macOS will ensure that only Conveyor can read it.

Fourth challenge: Windows CAs decided to squeeze extra revenue from devs by exploiting the new HSM requirement's incompatibility with cloud CI. They started offering "cloud HSMs" that can be used without any hardware, just a textual credential again (so removing much of the security an HSM is supposed to create!) and these are popular because everyone wants to ship from CI. Unfortunately these services all charge extremely high prices per signature once you go over a fairly small monthly allowance. If you want to ship builds on a daily basis, for instance, you can easily blow hundreds or even thousands of dollars on signatures alone. So Conveyor has a disk cache that stores signatures for binaries, and separates them from the underlying DLLs/EXEs so the cache doesn't fill up too fast even if you're shipping very large binaries, and a GitHub Action to integrate that cache with GitHub's own. All this saves a lot of signatures and therefore money.

Fifth challenge: Windows chooses to identify apps by the hash of their X.500 name as decided by their CA, but CAs constantly change their mind on how to format this name and change it for pointless reasons. Because the Windows ecosystem hates developers this can easily break app reputations/updates that are checking code signatures, and nobody at the CAs or Microsoft cares. So Conveyor has a whole system that recognizes when this has occurred and can drive an automatic reinstallation to repair things. The user doesn't notice except for a short delay on app startup (and seeing a progress bar).

Sixth challenge: platforms like the JVM that like to hide native code inside ZIP files, which may also contain native libs for the wrong CPU architecture or platform. Conveyor finds them, deletes the irrelevant binaries, signs the ones that remain and puts them back (or extracts them into the app directory and ensures they get found).

Seventh challenge: enterprises that have custom internal signing servers. So you can write a little script that submits binaries to the server and gets them back again. This works with all the above stuff too, when relevant.

Eighth challenge: MSIX. Microsoft's only non-deprecated packaging tech has a lot of benefits, and allows non-admin users to install packages, but the signature format is pure madness. By far the craziest scheme I ever saw. Reverse engineered and implemented.

Next challenge on the list: Azure Trusted Signing, Microsoft's new CA. It's marginally cheaper than the others, so some users want to use it, but it also doesn't provide a standard driver or use standard protocols. It'll require custom support.

Looking to the future, the path forward is to use kernel sandboxing tech to eliminate signing requirements for our users when possible. For apps that can fit inside the sandbox it should be possible for us to sign and distribute their apps for them, with full code signing only required if the app has special needs. That would give a much more web-like experience for non-web apps, where a developer can just create a scaffolded app with one command, ship it with a second and not think about any of the platform specific details or code signing at all. Unfortunately this is a way off yet.

[1] https://hydraulic.dev/

jonathantf2• 1 year ago

Slightly off-topic but my god that font is hard to read.

tingletech• 1 year ago

Once upon a time circa 1997 I made a javascript based windows shell for kiosk mode public terminal across 15 odd campus libraries. I had to buy a signing certificate from some CA to use the API that let me get rid of the netscape navigator chrome. All you could do on the computers was run netscape navigator, and but there was a menu that popped up on the left that had a menu of different library related web pages, and you could not tell that was really a browser window. I also had over 100 plugins we needed to install for all the licensed databases. The terminals ran Novell Netware on NT. I forget what software we used now, but they booted from a read only partition, and we created a new boot image when we shipped updates and someone would have to walk around with CDs and upgrade each workstation. The department that ran the public computer labs on campus had to sign off on the security before they would let me put them on the campus network.

I also had a shrinkwrapped netscape enterprise server with its server side javascript, and bought a shrinkwrapped sybase -- both at the same time I bought a Sun desktop computer. I bought all 3 at the campus bookstore and paid with a G/L code. I learned solaris, web stuff, javascript, and databases on that job. The novell netware was a directory server based on LDAP or x.509.

rednafi• 1 year ago

Interesting history. Signing software is neat, but I wish there were a single standard like HTTP. Instead, we have a fractal mess of GPG, Sigstore, Cosign, and package-specific solutions (npm, PyPI, Maven, etc.), all doing their own thing.

This makes me doubt how often these signatures are actually verified on client machines. For example, I've never done it manually because the process is so tedious. Some tools automatically check signatures, but even that is fragmented.

mike_hearn2• 1 year ago

In this context code signing mostly means Windows and macOS signatures, which are definitely checked by various things in the OS and third party tooling.

causality0• 1 year ago

You're going to write a history of code signing at Mozilla and not even mention the time they fucked their users by negligently letting a certificate expire and forcibly disabling every extension on every Firefox instance worldwide?

A brief history of code signing at Mozilla

Related Stories