Behind GitHub’s new authentication token formats

https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/

by todsacerdoti • 5 years ago

244 79 5 years ago

github.blog

ramses0• 5 years ago

https://tools.ietf.org/html/rfc8959 - "secret-token:E92FB7EB-D882-47A4-A265-A0B6135DC842%20foo"

RyJones2• 5 years ago

ok, great, when will I be able to scope tokens to acting on only one repo, or only one org, or in any meaningful way?

nathanaldensr• 5 years ago

I completely agree. This change is interesting and all, but GitHub (Enterprise, in my case) tokens aren't granular enough. There's a lot more benefit to be had in fixing that issue.

chatmasta7• 5 years ago

I definitely agree this is needed, but I have to imagine it's quite a complex change on GitHub's side. It probably entails changing a lot of their authentication architecture, in terms of what's stateless and stateful and what requires a trip to the DB to check (i.e., it's hard to encode a list of which resources you should have access to in a single token). I'm sure they're aware this is a problem though. Maybe this recent change sets the groundwork for fixing it.

RyJones2• 5 years ago

That the only method I have to scope tokens is to break the TOS for GitHub - by creating single-use accounts - is super lame.

threeseed5• 5 years ago

GitHub for me is running out of excuses for why the fundamentals of their platform is so poor. And why compared to Gitlab they deliver improvements at such an anaemic pace. They act like a company with 15 employees let alone 1,500+.

Everything from Security, Actions, Containers, Packages, Terraform, JIRA Integration etc is either completely broken or has major outstanding issues that haven't been fixed for years.

tptacek3• 5 years ago

I'm sure this is a valid complaint about Github, but it has nothing to do with the article, which is a bit annoying since there's some cleverness in that article (checksumming tokens, for instance) that we could be talking about stealing, rather than turning this thread into a generic referendum on whether Github is good.

chatmasta7• 5 years ago

The article is about GitHub's authentication tokens. It seems relevant to bring up a complaint about their scopes.

threeseed5• 5 years ago

a) I don't see what is particularly clever about their token algorithm.

b) Talking about fundamental flaws in their token implementation seems relevant in a discussion about their token implementation.

c) Public discussions about flaws are often the best way to educate others and make the company aware of them.

prepend• 5 years ago

It’s valid because GitHub is using cycles on format when the functionality is more important.

I think these new formats are nice, but don’t care give how hard to use their token scheme is. Just some mention of other stuff coming would be nice.

The problem I have is I’m not even sure if GitHub recognizes this as a problem (I have to grant access to every repo I can access to every script) and it’s been broken for years. Would be nice to know what they’re working on.

paxys3• 5 years ago

You can already do that today. Create a Github App and add it only to a single repo.

Serow225• 5 years ago

I'd be happy to have a repo read-only scope to start with, it seems a little nutty that in the year 2021 the only repo scope available is read-write https://docs.github.com/en/developers/apps/scopes-for-oauth-...

mperham• 5 years ago

The key insight here is that random tokens should be self-describing, so you know their intended use and therefore can make decisions and take action when one is detected.

If a script sees "ABC123" in a code commit, that's meaningless. If you see "secret-token:ABC123", now you can fail the commit with an error message: "Secret token detected in public commit, aborting."

echelon• 5 years ago

One of the biggest learnings of our org was to prefix tokens with the entity type. It's helped immensely.

entity-type:RANDOM_TOKEN

* Helps in migrations, especially complex ones where you split up entity types.

* Identities what tokens are so people can look them up if they see them in logs.

* Polymorphic relationships can delegate to the appropriate owning service easily without additional bookkeeping.

You can also encode other stuff in the token entropy, too, such as the author DC/region for active-active setups where you need to forward the request to the source of truth in the brief window where the other regions don't know about it yet.

11235813213455• 5 years ago

that's how Stripe prefixes their IDs too, depending on what type of entity it is. Makes debugging, docs, .. easier

fiddlerwoaroof• 5 years ago

I’ve always thought that a Java-style reverse domain name format (or, perhaps, URLs) is a great way to encode IDs: com.foo.bar.Person:0000-11111-22222-33333 or whatever. That way, any code that logs IDs or transfers IDs across the network gets tracing “for free” and, when you see an ID in a bug report, you can use it to help focus the investigation.

xPaw• 5 years ago

For those that haven't seen it, "secret-token:" is an RFC. I've started using it at work.

https://tools.ietf.org/html/rfc8959

nine_k• 5 years ago

An unencrypted version marker would be pretty useful, too. If anything is long-term, you can safely bet in that it'll need to evolve.

cornstalks• 5 years ago

Note that the RFC's category is "informational", which doesn't give it as much weight as something that is "standards track". Usually the important RFCs are "standards track" though there are some "informational" RFCs that are also important.

From Wikipedia[1]:

An informational RFC can be nearly anything from April 1 jokes to widely recognized essential RFCs like Domain Name System Structure and Delegation (RFC 1591). Some informational RFCs formed the FYI sub-series.

[1] https://en.wikipedia.org/wiki/Request_for_Comments#Informati...

staticassertion• 5 years ago

FWIW it is still very much worth reading the article, since they talk about how they implement that approach. They bring up why they use an underscore, checksum'ing, entropy, etc.

revicon• 5 years ago

The ID prefixing is cool from an identification point of view, but we've been using UUIDs for tokens and if we implemented this we wouldn't be able to use the UUID optimized datatype field in Postgres.

marksomnian• 5 years ago

Surely you can just strip off the prefix in the application layer before sending it to Postgres? You still get the benefits, while being able to use the native query.

orf• 5 years ago

Why not? You don’t have to store the prefix and the UUID in the same column?

asimpletune• 5 years ago

Can’t you just add a column to your schema with the prefix?

vsareto• 5 years ago

One other neat thing about _ is it will reliably select the whole token when you double click on it

Shout out and kudos to whomever brought that up

chatmasta7• 5 years ago

Frankly the fact that this doesn't happen with `-` in a `<code>` block should be considered a browser bug.

williamdclt2• 5 years ago

Well it doesn't happen in IDEs either (by default, at least)

atq2119• 5 years ago

'-' is a minus sign. It will appear in code blocks much more often in contexts where you do want it to act as a word delimiter.

systemvoltage2• 5 years ago

Side rant: Github devs, if you're listening, please give us REST API. Not everyone knows GraphQL or has the motivation to do so. The industry standard is REST for public facing APIs, including companies such as Stripe (widely considered to be the gold standard for public API design and documentation). You can use GraphQL internally.

minitech• 5 years ago

https://docs.github.com/en/rest

systemvoltage2• 5 years ago

I remember having to use GraphQL to delete a Docker image that was stuck in my private repo and there was no GUI to clear it. Wasted a couple of hours trying to send a GraphQL query which would have been a 2 minute jobbie using cURL. Github's public REST API didn't have this feature.

azimuth11• 5 years ago

A lot of REST APIs are just as hard to grok as GraphQL is as a whole. Companies often lack schemas and documentation, which GraphQL helps with out of the box.

seoaeu• 5 years ago

It seems weird that in a blog post about a new format for tokens, there isn't a single example of what a GitHub token now looks like.

abrookewood• 5 years ago

From the opening paragraph: "old authentication token formats are hex-encoded 40 character strings"

sneak2• 5 years ago

description != example

Wowfunhappy• 5 years ago

Does anyone know if these new tokens are backwards compatible with software that used the old tokens? By which I mean, I’m using a version of Git Tower from before they switched to a subscription model, and I’m wondering whether regenerating my tokens will make me unable to log in.

sneak2• 5 years ago

I imagine they should be, they're still just strings.

It's possible that Git Tower does some local validation that might complain, but I think they are in the same set of characters.

You could always write a crack for the local validation, too.

greysteil• 5 years ago

The reason we (GitHub) kept the new tokens to 40 characters, matching the length of our old tokens, was to make this change as backwards compatible as possible. We’ve never documented or committed to our tokens being 40 characters long, but they’ve been that way a long time and there may be software out there storing them in a fixed length database columns.

If you use a service you’re worried the change may break I’d recommend minting a new token and trying it, perhaps on a new account on that service, before revoking your old one.

(We do plan to increase the length of our tokens in future, but not before July 2021 at the earliest.)

gkop2• 5 years ago

Would someone comment on this idea in context of JWTs? Not trolling, just curious as I use JWTs and embed this kind of metadata as a custom claim, which accomplishes some but far from all of what GitHub accomplishes here, but then I have no need for the easy scanning. So seeking wisdom from anyone who has thought carefully about whether or not to prefix their JWTs in this way.

paxys3• 5 years ago

A big motivation for such token formats is to quickly and easily identity when they are shared somewhere they shouldn't be. JWTs aren't helpful in that regard, since they always present themselves as a base64 encoded blob.

gkop2• 5 years ago

Totally, but JWT-like blobs can be detected (see sibling comment), and parsing attempted, so for the automated scanning use case, if I understand correctly, it can be done with perfect accuracy, just at a larger computation expense and worse security exposure due to the complexity of the scanning and the need to parse.

The more interesting side to me is the benefit to humans, from the prefix technique.

threeseed5• 5 years ago

The question is why you couldn't just have "${prefix}-${JWT}" as your format.

Then you can just strip the prefix before parsing. Which means don't need to worry about checksumming or entropy and you have the ability to embed large amounts of data as well as plenty of client support and libraries.

Would be curious if this implementation is somehow more performant.

parhamn• 5 years ago

It’s pretty grepable because {“ (json opener) always encode to “ey”. So a base64 that starts with “ey” and has 3 dot separated sections is a good start for a regex. I’m sure you can go further by looking at the spec.

benatkin3• 5 years ago

JWTs purposely contain information in plain text (unencrypted and not stored in a database), however it is in base64 so you don't need to worry about url encoding issues and so it looks like a token.

You could add a prefix to a jwt. That would make it a token that contains a jwt.

I don't think the tiny prefix is what they want to obscure. So it wouldn't go against the design of JWT to add one.

I would do it. I don't see any issues with it.

It would be something like:

BA_<base64>.<base64>.base64

If you wanted to be able to double click to copy and paste, which I don't think is a huge usability improvement, you could replace the . with _, and I think a lot of devs would be able to figure out that it's a representation of a JWT.

Mandatum• 5 years ago

I wonder why they went with 2 rather than 3 or 4 for company identifier. Stock ticker for instance would make sense. Not really practical.

paxys3• 5 years ago

This isn't meant to be a standard, just something they picked for themselves. And it doesn't even need to be a company identifier. Slack tokens are prefixed with "xox<token type>-", for example.

chrismorgan• 5 years ago

Thus, we are adding a separator: _. An underscore is not a Base64 character which helps ensure that our tokens cannot be accidentally duplicated by randomly generated strings like SHAs.

This is just a little bit misleading. Base64 isn’t a single neat and tidy thing: there are several alternatives for the encoding characters 62 and 63, padding, line break behaviour and one or two more things; see https://en.wikipedia.org/wiki/Base64#Variants_summary_table. When you’re talking about Base64 on the web, you’ll very commonly be talking about base64url, the URL- and filename-safe variant, rather than what’s most commonly called base64 and typically the default. But base64url is in widespread use, and has _ as character 63.

Also “randomly generated strings like SHAs” aren’t typically doing Base64 anyway,but rather hexadecimal encoding.

jridgewell• 5 years ago

The post isn't super clear, but I think they're describing base62 here. It says base62 later in text when talking about the checksum, and the "Our implementation for OAuth access tokens are now 178" section uses `a-zA-Z0-9` which is base62.

AlexC04• 5 years ago

That was a really well thought out article.

I loved the use of underscore and the "it'll reliably double click"

I liked that they looked to other companies out in the sold and acknowledged that they're learning from others.

I didn't know what to expect when I opened the link, but I'm glad I read it!

Behind GitHub’s new authentication token formats

Related Stories