API Tokens: A Tedious Survey

https://fly.io/blog/api-tokens-a-tedious-survey/

by enobrev • 5 years ago

387 120 5 years ago

tptacek23• 5 years ago

The one thing I'm not super comfortable about here is my PASETO take. My attitude going in was that PASETO has a lot of boosters and not a lot of critical takes. I can beat up on Macaroons because we're using them, and I'm going to follow up with a post about what our Macaroons like like. I'm not doing that with PASETO. So, like, I stand by it, but take it for what it's worth.

Later

I turned the post into a handy chart. Let me know if you want a poster of it.

https://twitter.com/tqbf/status/1430303808945524739

nightpool• 5 years ago

In your article, you say "For reasons I will never understand, the PASETO authors submitted it to CFRG for consideration. Never do this". Is issue here the implied design-by-committee that led to JWT being a bad standard, or something else? It does seem like it found some valid issues in PASETO, at least, so I had trouble understanding your point.

tptacek23• 5 years ago

CFRG's response to PASETO was tepid. The few people who participated in the thread identified three issues:

* The v1 local tokens used a novel nonce construction (I'm doing this from memory) and CFRG's take was "standard constructions or GTFO".

* The HMAC/RSA thing, which PASETO noted and documented but didn't fix.

* The fact that PASETO is basically a restricted profile of JWTs, begging the question of why it didn't just specify a restricted JWT profile.

I don't think this feedback was especially valuable.

I think there are subjects on which CFRG discussions shed a fair bit of light, when they're high-profile enough to drag academic cryptographers into the fray. But the other thing that happens in CFRG is that bad stuff (like the Dragonfly PAKE) gets blessed (because there's no outcome besides "this is fine" and "this is trivially broken" --- and even "this is trivially broken" can get laundered back to "this is fine" if the threads get tedious enough).

In the worst case, you get people proposing bikeshed changes to constructions that are already de facto standards, which (if I remember right) happened with Curve25519, though thankfully not successfully.

I think the whole practice of standards based cryptography is mostly discredited at this point. Signal Protocol isn't a standard despite being the reference model for most secure messaging systems. WireGuard isn't a standard either. The original ethos of the IETF was that things get popular, and then they get standardized. IETF does a lot of stuff de novo now, which is how we end up with stuff like Heartbleed and JWT.

tingletech• 5 years ago

is there a key to decode the scores on the chart?

tptacek23• 5 years ago

This seemed funny until I realized everyone else isn't 40+ years old and former Consumer Reports subscribers. The red-with-white-dot thing is "best".

WatchDog• 5 years ago

Could do with adding a key to the chart.

er4hn2• 5 years ago

I have no idea if I am reading the chart right, but it looks like biscuits are the worst for scalability. Why is that?

I would imagine that if the entire policy is encoded in the biscuit, it is very easy to evaluate without needing to call external services. And it can be extended like macaroons without needing a central authority, assuming I groked your blog post correctly. The only issue I can see is revocation.

tptacek23• 5 years ago

Ok so first of all let me just say the chart was a joke I wrote for Twitter. Then Joël Franusic suggested I add a bunch of meta tags to the post so that Twitter would show the chart as the "card" for the post on Twitter. To make that work I had to pull the chart into the actual site (I'd just posted it to Twitter originally), so I figured, what the hell, might as well slap it on the end of the post. I don't even know if the ratings I came up with make sense! I docked Biscuits because they're chained public key verification on an per-API-request basis, but who knows? I haven't used them! The chart isn't serious! And I feel like it's all I'm talking about now!

To top it all off, my dumb meta tags didn't even work; they needed to be in the <head> of the page, and I'll be damned if I'm going to figure out how to do that in our static site generator configuration.

I just wanted the Carl Yastrzemski with the big sideburns.

zrail• 5 years ago

There's an additional nuance to opaque random-ish tokens that can be helpful in high-traffic situations. You can essentially encode some, for lack of a better word, "routing" information (shard, region, etc) into the token when you generate it. It's still random, you still verify the whole token with your database, but you can extract the routing info and pass it to the correct backend from a mostly-stateless frontend.

mooreds• 5 years ago

This was great. A really fair survey of various token methods. Plus plenty of liveliness, not boring at all. Thanks, OP!

One thing that I wish was addressed more was language/library support. It gets casual references a couple of times, but for an average developer (as I consider myself) a set of robust, supported open source libraries that help me use a token is so important (not write an implementation, but use in a project that just wants to use the tokens safely).

I don't have anything but anecdata, but I feel like most software is going to be in the 'just want to use it' category, rather than the 'need to implement it'.

This is where the standards like OAuth and JWT win right now. That doesn't mean they always will, but in my experience, that's the current situation.

CiPHPerCoder4• 5 years ago

For PASETO, the quick guide to library support is https://paseto.io

simonw7• 5 years ago

This is great.

One thing that's worth remembering about randomly generated tokens is that it's important to always use safe comparison methods when comparing them to the stored one - otherwise you could be vulnerable to timing attacks.

In Python you can use secrets.compare_digest(a, b) for this: https://docs.python.org/3/library/secrets.html#secrets.compa...

csnweb• 5 years ago

I am wondering how that relates to searching for the token in a database (index). Does it still matter? I would assume the time might depend on where or even wether the token is in the index or not, not sure how much though and what to do about it.

ezekg8• 5 years ago

Run the token through a secure HMAC function before storing it in the DB. Problem solved. KISS. :)

simonw7• 5 years ago

Yes, it matters - timing attacks could absolutely work against an indexed column in a database.

The trick I've used for this is to have tokens that look like this:

"234523:7a561002f780e23853bdfbd89ae79bf2"

Then you have database entries like this:

id = 234523 token = 7a561002f780e23853bdfbd89ae79bf2

When a token comes in you use the bit before the : to look up the database record by its indexed ID, then perform a safe comparison between the rest of the token and the value you retrieved from the database.

some_furry2• 5 years ago

I am wondering how that relates to searching for the token in a database (index). Does it still matter?

It can, but the practicality of exploiting this timing leak isn't at all a settled issue.

Previously, https://soatok.blog/2021/08/20/lobste-rs-password-reset-vuln...

SilverRed• 5 years ago

Could you elaborate on what you mean by timing attacks? What would go wrong if you did a basic params[:token] == dbvalue ?

pests• 5 years ago

Most naive string comparison methods (including language expressions like above) will usually compare one letter at a time until a mismatch is found. This allows an attacker to build up the correct password one character at a time. The correct letter will take longer to check then the incorrect.

simonw7• 5 years ago

This post reminded me a little of https://blog.thea.codes/building-a-stateless-api-proxy/ from 2019, an absolutely brilliant hack that attempted to make up for GitHub's lack of finely-grained API tokens.

The key idea there is to build your own custom proxy for the GitHub API, then issue tokens for it which are actually encrypted bundles of the full-permission API token plus a set of rules about what the proxy should allow it to do - only allow a GET to paths that match "/gists/.*" for example.

It's somewhat similar to Biscuits storing a Datalog program "to evaluate whether a token allows an operation."

greysteil4• 5 years ago

Since the author is on here - reckon you could beat up on simple random tokens a _little_ bit more? In particular how easy they are to identify and prevent leaking (easily fixed by adding a prefix).

I work on secret scanning at GitHub. When token issuers use easily identifiable formats for their tokens we can easily spot them when they're accidentally committed. We can then work with the token issuer to automatically alert them of those leaks. A good example is AWS - if you commit an AWS key and secret to a public GitHub repo we will tell AWS about it and they will tell you about it (and quarantine the exposed keys) within a few seconds. We work with dozens of other token issuers too, though - some of the latest we added were Linear, PlanetScale and Ionic.

The above relies on tokens being identifiable - we can't send hundreds of partners everything that looks like 32 hex chars. In future we want to be able to do even more sophisticated things, like ask users for confirmation before the push code that contains secrets. We recently changed our own token pattern for that reason.

GitHub secret scanning program: https://docs.github.com/en/developers/overview/secret-scanni...

GitHub's updated token format: https://github.blog/2021-04-05-behind-githubs-new-authentica...

tptacek23• 5 years ago

I agree with you. Someone brought this up on Twitter and I'm kicking myself for not remembering to include the notion of adding identifiable markings to sensitive tokens (I'd do it not, but I'd feel like I was plagiarizing).

And it's a noodly and somewhat incoherent notion of "safety" I'm using here, because of course, random tokens are unconstrained bearer tokens --- authenticated requests, CATs, Macaroons, and Biscuits all address that weakness. I'm biased by my concern over cryptographic implementation mistakes.

It's a neat property of Macaroons (and maybe Biscuits) that you can come up with sane configurations where checking a Macaroon into source code can be, if not totally safe, at least not a major incident. I wish I'd thought of that, too, since I think "checking the token into source control" is a more vivid example than "emailing tokens" or "passing them around".

tialaramex2• 5 years ago

Surely you needn't feel like you're plagiarizing, just give credit. The credit won't be any less deserved next week or next year, even if you "Know" you'd definitely have fixed this without prompting, nobody else knows that, and so it looks like you avoided giving credit where it was due, so, don't do that.

Nobody's asking for royalties, so "Shout out to @SomeoneOnTwitter for reminding me" is enough.

ithkuil• 5 years ago

Macaroons are great. I was using them successfully in a (now dead) product. Very easy to reason about when you get the basics. Unfortunately the ecosystem is small and the effort spent of getting colleagues on board (i.e. convincing them you're not using some fringe thing) was substantial and continuative.

remram3• 5 years ago

I bring this up every time this is mentioned, but I really wish the API token format included a domain to notify in case of leaks. Having services register with your in-house secret scanning system works very well if you're GitHub, but otherwise it's a very closed mechanism.

If sendgrid tokens were `secret:sendgrid.com/91on9SIkbUfSs` instead of `SG.91on9SIkbUfSs`, or Amazon keys looked like `amazon.com/JGUIERHT` instead of `AKIAJGUIERHT`, we wouldn't need a database of regexes and endpoints to report secret leaks.

See also the last time I ranted about this: https://news.ycombinator.com/item?id=26568651

greysteil4• 5 years ago

Appreciate your passion here Remi. I don't think a full standardisation of API token formats is ever likely, but I do think there's value in nudging things in that direction.

One big challenge is that it's hard to get service providers to change their token formats. Very few have this at the top of their priority list - they're busy with other things. Here's an example playing out in OSS that is pretty typical: I tried to persuade the (excellent) team at Sentry to update their format, and they essentially told me "we have other priorities" https://github.com/getsentry/sentry/pull/26313. And that's a relatively simple changes, not the adoption of a whole standard.

In addition, as Thomas points out in this article, there are a lot of different token types for someone thinking about minting API tokens to choose between. They might rationally have different preferences over them. A standard that is prescriptive of format and approach is likely to struggle given that diversity.

With that said, I do see an opportunity here for a more modest standard targeted at service providers that already use JWTs or Macaroons. Generic tokens of those types are relatively easy for scanning providers to identify, and it's easy (and hopefully uncontroversial) for service providers to encode more information in them, like an "if found" link. I think a standard that defines the attribute name there, and the API for reporting / responding, would be a good start that might see adoption.

sudhirj3• 5 years ago

Taken a step further, the secret should just be a URL that revokes itself - Like https://revoke.sendgrid.com/91on9SIkbUfSs. Github should then just make a get request to every URL (can whittle abuse down a bit by requiring https://revoke.sendgrid.com/robots.txt to have a Revoke: YES` section). They and anyone could maintain an allowlist of revocation URLs to pattern match as well. This makes a global registry unnecessary, and standardises the act of revocation.

MaxGabriel• 5 years ago

In future we want to be able to do even more sophisticated things, like ask users for confirmation before the push code that contains secrets

If you all have thought about it, do you imagine you'd only warn in the presence of some generic token identifier, like `secret-token` a la https://datatracker.ietf.org/doc/html/rfc8959 ? Or, would you be able to warn on everything that matches the regular expressions your partners give you to identify their API tokens?

greysteil4• 5 years ago

The latter. Our objective for secret scanning is to prevent as many serious secret leaks as possible. Where a service already has a token format that is highly identifiable we want to take advantage of that, rather than rely on the adoption of generic token identifiers.

kdeldycke• 5 years ago

That's for the exact same reason (obvious identification and easy scanning) that I proposed to prefix all Scaleway's access keys with `SCW`.

See: https://blog.scaleway.com/strengthening-scaleway-token-secur...

tialaramex2• 5 years ago

As a feedback on the visuals, I'm sure you know what this scorecard means. I'm sure I could figure out what this scorecard means if I put a bunch of effort in. The fact that "figuring out what it means" is a problem worth talking about is all you need to know.

If the idea was "LOL, joke scorecard" then, I guess joke is on me, otherwise, if you revise this I'd recommend to decide what one thing the scorecard is supposed to communicate, then visualise only that and accept that people will need to read the rest of the text to know more.

tptacek23• 5 years ago

It's like 90% a joke (it's the old Consumer Reports scale, where the red-white-dot is "best" and black-white-dot is "worst"). It's definitely not important to the content in the post.

staticassertion2• 5 years ago

I also was unclear on what the scorecard was. From elsewhere in this thread it's something to do with "Consumer Reports", which I've never heard of. Wikipedia helped with the rest.

"Consumer Reports graphs formerly used a modified form of Harvey balls for qualitative comparison. The round ideograms were arranged from best to worst. On the left of the diagram, the red circle indicated the highest rating, the half red and white circle was the second highest rating, the white circle was neutral, the half black circle was the second-lowest rating, and the entirely black circle was the lowest rating possible"

jopsen• 5 years ago

Why all the hate for JWTs?

Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use.

(It's just a more convenient way of rolling your own scheme)

That said: random tokens have a lot going for them :)

CiPHPerCoder4• 5 years ago

Why all the hate for JWTs?
Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use.

That's not what JWT is, but I can understand why someone would be misled into believing that.

JWT isn't just an encoding format, it also includes a crypto algorithm negotiation protocol that lets the attacker choose the algorithm. Even if you strictly allow-list which algorithm you want to support, you can accidentally bypass this control in many libraries if you support the `kid` (key ID) header. [1]

It also allows attackers to completely strip the security. [2] [3]

Put shortly, JWT is a gun aimed directly at your foot. That's why there's so much hate for JWTs.

[1] https://github.com/firebase/php-jwt/issues/351

[2] https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-ba...

[3] https://www.howmanydayssinceajwtalgnonevuln.com/

daigoba66• 5 years ago

crypto algorithm negotiation

If you control both sides, then you can ignore this part or do it out-of-band.

Though, if you control both sides, then you can use literally anything else too.

benatkin2• 5 years ago

Even if you strictly allow-list which algorithm you want to support

This is the norm.

As for the key ID attack, this sounds like it's just a trick to know where the private key is located? It shouldn't be publicly accessible.

https://blog.pentesteracademy.com/hacking-jwt-tokens-kid-cla...

simonw7• 5 years ago

There's a lot of material out there explaining what's wrong with JWTs. Two of my recent favourites:

https://groups.google.com/g/django-developers/c/6oS9R2GwO4k/... - on the Django mailing list

https://www.zofrex.com/blog/2020/10/20/alg-none-jwt-nhs-cont... - where the punchline is "Writing the code to sign data with a private key and verify it with a public key would have been easier to get correct than correctly invoking the JWT library. In fact, the iOS app (which gets this right) doesn’t use a JWT library at all, but manages to verify using a public key in fewer lines of code than the Android app takes to incorrectly use a JWT library!"

progx• 5 years ago

The main argument is not against JWT, it is against the libraries and that people use them, without knowing what they are doing.

betaby• 5 years ago

Simple random tokens are essentially passwords. Thus one should store them 'salted' in the authentication database. Is my understanding correct?

simonw7• 5 years ago

I don't see the need to salt them in most cases.

The reason you hash and store passwords with a salt is that there's a very good chance that a user will have used that same password somewhere else. As such, it's important that you make it as hard as possible for you - or a co-worker - or someone who gets hold of a dump of your database - to gain access to the original password.

Random tokens were generated by you. They are only valid against your own service. If someone bad gets hold of them, all they can do is make API calls against your service until that token is revoked. So salting and hashing them doesn't win you much.

That said, if a lot of people at your company have unfettered access to run queries against your database it may be a good idea to store your random tokens in a way that prevents people from copying them out and abusing them. But if you're storing any private data at all and you have that kind of a culture you have bigger problems you need to solve.

tptacek23• 5 years ago

If you're worried about database timing attacks (I'm mostly not) then there's a good thread about that here:

https://twitter.com/colmmacc/status/1430282652821585920

keithwinstein• 5 years ago

I'd love to hear tptacek's view on the pros and cons of "authenticated connections," where the client uses asymmetric crypto to establish an authenticated-as-the-principal bytestream, and then issues a bunch of unsigned requests over that bytestream.

In practice, I'm talking about (1) "SSH public-key authentication to GitHub" (or to an EC2 or GCE instance), or (2) using FIDO2/WebAuthn (or TLS client certs...) to authenticate an HTTP connection or browsing session.

One major pro is that this is a lot more widely implemented and standardized than "authenticated requests" -- there are lots of reasonable implementations of SSH pubkey authentication or FIDO2/WebAuthn, compared with trying to sign an individual HTTP request. And that maturity brings robustness and security, e.g., I like that I can have my SSH private key (or my FIDO2 private key) on a tamper-resistant hardware device and not worry about some rogue software stealing my private key, whereas trying to find a hardware device to hold my AWS credentials and do the AWSv4 signature is a different story. Any bearer-token-based scheme seems... a lot scarier in that sense.

I'm less clear on the practical cons (assuming "authenticated requests" is not available). With SSH keys or TLS client certs, I could imagine it's annoying because only the terminus of the secure connection can verify that the client is who it claims to be, and maybe it's annoying to have your SSH/TLS termination component directly talk to a user database to authenticate individual clients and then attest that it did so to the rest of the backend. But... somehow GitHub/GitLab/Bitbucket all manage to do this at scale, and with WebAuthn/FIDO2, any component can be the one to make the challenge. (I suppose only recently can you make a "user presence not required" challenge, so maybe that's why nobody uses this in the non-presence setting...?)

tptacek23• 5 years ago

I have actually a lot to say about this, in the context of mTLS; it's a post I'm working on.

I am generally a fan of the mTLS approach for very simple topologies, like "central Consul cluster nobody can talk to without the right client certificate". I am much less a fan of it for the general inter-service authentication problem.

FreakLegion• 5 years ago

I get the criticism of ECDH-ES with JWT in vivo, but not the linked tweets' broadside against it in vitro. Are we not just talking about ECIES?

tptacek23• 5 years ago

It's all the things that had to go wrong all at once. P-curves, not Curve25519 (where curve point validation is less important). Static-ephemeral ECDH, so there's a key to target. Long-term durable keys. I'm not making an argument against ECIES (though I guess I'd push not to use the P-curves).

catchmeifyoucan• 5 years ago

What about revokable JWT tokens? Isn't that similar to the random token approach, except each token holds some additional meaning, and revokable.

tptacek23• 5 years ago

Yes. Revocation is painful with all of the stateless approaches; it's not on its own a reason to avoid JWT. On the other hand, the folkloric draw of JWT is that it's stateless, and they're only stateless if you can revoke them without issuing SQL queries.

initplus• 5 years ago

A downside of stateful auth is the extra DB round trip on every request.

Different revocation techniques like periodically distributing a revocation list to your auth services can resolve that part of the issue.

RustyRussell• 5 years ago

Because standards are always improved by creating new options, I should note runes here (simplified macaroons with Python implementation): https://pypi.org/project/runes/

Disclosure: I'm the author

Scaevolus• 5 years ago

Google's internal mutual authentication/encryption might be of interest. https://cloud.google.com/security/encryption-in-transit/appl...

They show how far you can drive Protocol Buffer Tokens.

monocasa2• 5 years ago

I'm curious where kerberos tickets fit into this. A little out of vogue, but it'd be neat to hear how exactly they're long in the tooth.

tptacek23• 5 years ago

Nobody sane uses Kerberos to do IAM for public APIs. But people use them for inter-service authentication, as the post mentions. It links to an article I wrote a couple years ago that considers Kerberos in the context of a variety of other inter-service security tools.

monocasa2• 5 years ago

Nobody sane uses Kerberos to do IAM for public APIs.

Why if you don't mind? I'm actually looking at a major proprietary protocol used by hundreds of millions of users that use Kerberos as the fundamental cryptographic attestation of identity and roles. Like clients all connect to a VPN who's bridge uses the Kerberos ticket to whitelist backend services accessible from that connection. They're basically being used the same as if an API gateway understood oath claims and could stop whole classes of client calls at the perimeter.

ignoramous2• 5 years ago

Very consistent with what tptacek has been advocating for on news.yc over the years.

The authn mechanism we use is closer to Keybase's NIST (non-interactive session tokens)[0] that are a mix of AWS-style Bearer Tokens and the usual Random Tokens. Of course, the problems around "logistics" (public-key cryptography)[1] are a real nightmare as the post points out.

We exchange these tokens between devices (if needed) over password-authenticated channels (using CPACE [2]).

[0] https://keybase.io/docs/api/1.0/nist

[1] As examples, see what goes on when a Keybase user associates a new device: https://book.keybase.io/docs/crypto/key-exchange or when SQRL user revokes compromised keys: https://www.grc.com/sqrl/idlock.htm

[2] https://github.com/jedisct1/cpace

cormacrelf• 5 years ago

Is NIST NIST-compliant?

ignoramous2• 5 years ago

Nothing to do with NIST.gov, that's just unfortunate naming coincidence. Re: Compliance: Keybase, if am not wrong, has pretty much rolled out their own crypto here. At least in one instance they were subject to criticism for rolling out a key-wrapping scheme viz. TripleSec: https://news.ycombinator.com/item?id=9655245

alex-olivier2• 5 years ago

I tend to fall in the trap of once you have authenticated the person and passing tokens around that can hold claims/role for authorization, you will eventually reach the point where the tokens are getting so bloated due to the complexity of access controls.

AuthN is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform

https://cerbos.dev/blog/the-hidden-costs-of-user-authorizati...

alex-olivier2• 5 years ago

...and by that I mean authorization is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform

tptacek23• 5 years ago

Authorization is authz, for what it's worth. "authn" means authentication.

traceroute66• 5 years ago

You know, what I'd like to see more than standardised API tokens to make scanning easier is actual addressing of the underlying problem.

For example, we had a pentest done on a website and the pentester got all excited because they found some AWS tokens.

Trouble is, they would be worthless to anyone external because we were making good use of AWS IAM to lock them down with ACLs, Roles etc.

So it was effectively a non-event.

What happened to the old concept of layered security ? Why should discovery or leakage of an API key automatically give the attacker all the keys to the castle ?

In my ideal world, all cloud and API service operators would have the equivalent to AWS IAM and preferably would enforce its usage (i.e. "here's your API key, but it won't work until you set some layered security")

innomatics2• 5 years ago

This very good practice and I am pleased that the one time I got the dreaded message from GitHub and AWS, that I'd done the unthinkable (on a public repo), the keys were only for accessing a single junk dev S3 bucket - phew!

But no amount of layering makes the problem go away. Sometimes god-like keys are unavoidable. Those needed by Terraform etc are an example that comes to mind.

mwcampbell• 5 years ago

I continue to believe that boring, trustworthy random tokens are underrated, and that people burn a lot of complexity chasing statelessness they can't achieve and won’t need, because token databases for most systems outside of Facebook aren’t hard to scale.

Then why did you decide to go with macaroons rather than random tokens? Do you know for sure that fly.io needs that statelessness?

mrkurt• 5 years ago

We need statelessness for service reliability more than we need it for scale. There are a number of services that people use within our infrastructure that need to continue working even when a centralized DB is unavailable.

Logs are a good example. We expose logs as a NATS service, people auth and subscribe to their logs. It's useful to terminate auth right there both because it's faster and because it continues to work when the internet burps.

tptacek23• 5 years ago

I think this is a good question. I'm writing up our Macaroon implementation as I roll it out --- it'll be a few weeks before I publish --- and, if I've done it right, I'll have answered that question clearly.

sudhirj3• 5 years ago

Ironically my main takeaway is that given the pros and cons, JWT seems like the best balance of security and convenience, assuming you always hardcode/verify the algo. Lots of library support, allows Macaroon type policy setup using the JSON (but no chaining). It's not recommended in the article, but it seems like knowing what's out there has made JWT's usefulness clearer by making its downsides more obvious.

API Tokens: A Tedious Survey

Related Stories