Tokenized Tokens

by darthShadow • 3 years ago

190 30 3 years ago

tptacek6• 3 years ago

Psyched to get this out (I mean, I didn't do any of the work, but I'm still psyched it's getting out). I'm also a little bit surprised Vault doesn't already have a feature like this as a Secrets Engine.

JoachimSchipper• 3 years ago

Yes, this is cool - thanks to fly.io for documenting and open-sourcing it!

disintegrator• 3 years ago

Great project and write up!

Did the team consider developing a custom secret engine [1] for Vault? or is it that the specific dances between Rails, tokenizer, ssokenizer cannot be accommodated by a secret engine?

[1] https://developer.hashicorp.com/vault/tutorials/custom-secre...

tptacek6• 3 years ago

We're really picky about how our existing Vault clusters are exposed to applications, so building a Secret Engine for this would have required us to run an entire new Vault cluster. Moreover, we're moving away from Vault for a bunch of use cases (not all of them! we'll be running Vault indefinitely) --- not because of any failing of Vault, but because at this point we understand our needs very well, and operational legibility has become a really big priority. This also has a clearer path to integrating with our internal Macaroon tokens.

I have to imagine somebody is going to build a Secret Engine that does this.

mike_d2• 3 years ago

This is really cool, but I think it ends up just being a complicated decryption oracle?

If I compromise your rails app, and (hypothetically) Stripe allows me to specify the message as it appears on a users credit card statement, could I not just ask it to insert the API key in that field as well and then check my bank statement? This gets easier if there is something where a value gets reflected back to the user, say an SSO error message.

My apology if there is already a protection for this, but I didn't see any obvious use restrictions in the Github README example.

filleokus• 3 years ago

As I read the docs, I think the specification for how the secret should be injected lives "statically" within the Tokenizer service, and is not configurable by the internal service making the request.

I.e, if you only get control of the Rails app would need to find an api.stripe.com endpoint that reflects back the authentication header.

---- EDIT: No, I misunderstood it completely, you are right. But hmm. One way I can think of solving what you mentioned is if the token itself contains the processor parameters. That way it wouldn't be possible to change how the templating works after the secret have been tokenised (i.e by an attacker)

mike_d2• 3 years ago

Self-reply since I can't edit my comment anymore. It looks like they are fixing this:

https://github.com/superfly/tokenizer/pull/9

evisdrenova• 3 years ago

This is pretty cool and I think the right direction. Would like to see more companies do this. I've built out systems like this in the past that essentially only handle tokens instead of sensitive data whether it's secrets, PII, etc.

You can also take this a step further and do mathematical operations on encrypted data using homomorphic encryption without ever having to decrypt the data.

Just one small nitpick (mainly because I worked in this space for a few years) is that tokens and encrypted values are different. Tokens aren't encrypted and instead randomly generated using a KV pair look up table so that an attacker could never reverse engineer them. Whereas encrypted values obviously use a key (whether symmetric or asymmetric) and could theoretically (although pretty much never practically if you're using something like AES256) be hacked if someone got the key.

ShaneCurran• 3 years ago

Shameless plug, but we have a product at Evervault[0] called Outbound Relay[1] that’s basically a hosted, managed version of this flow. Secrets can be encrypted anywhere you collect them (either from third-party APIs or directly from the user in their browser) and then used any time you send a request to a downstream API provider.

[0] https://evervault.com

[1] https://docs.evervault.com/products/outbound-relay

salgorithm• 3 years ago

I could imagine developers using something like this to get credentials off laptops.

js2• 3 years ago

Summary: fly.io wrote two new programs ("some little security thingies") to reduce the attack surface of their codebase.

1. "Tokenizer is an HTTP proxy that injects third party authentication credentials into requests. Clients encrypt third party secrets using the proxy's public key. When the client wants to send a request to the third party service, it does so via the proxy, sending along the encrypted secret in the Proxy-Tokenizer header. The proxy decrypts the secret and injects it into the client's request. To ensure that encrypted secrets can only be used by authorized clients, the encrypted data also includes instructions on authenticating the client."

https://github.com/superfly/tokenizer

2. "Ssokenizer provides a layer of abstraction for applications wanting to authenticate users and access 3rd party APIs via OAuth, but not wanting to directly handle users' API tokens. Ssokenizer is responsible for performing the OAuth dance, obtaining the user's OAuth access token. The token is then encrypted for use with the tokenizer HTTP proxy. By delegating OAuth authentication to ssokenizer and access token usage to tokenizer, applications limit the risk of tokens being lost, stolen, or misused."

https://github.com/superfly/ssokenizer/

If these sound interesting to you, click the submitted link for the "big long essay about how the thingies came to be."

ec1096853• 3 years ago

To be fair to the author of that blog post, it was a pretty good non top much fluff read.

agwa• 3 years ago

Props to Fly for open sourcing this!

Another thing you can do with a proxy like this is limit the types of requests which can be made to the 3rd party service. This is very useful if the 3rd party service doesn't support fine-grained permissions for its API keys and you don't want your application to have full access to the 3rd party service.

(I implemented something very similar for my company, which I described here in case anyone is curious: https://sslmate.com/resources/protecting_customer_credential...)

justin_oaks• 3 years ago

Good point about limiting request types.

I created a simple proxy application at my job to handle authenticating to various Atlassian services, like Jira, using OAuth 1.1. I used the proxy to not only perform the OAuth operations on behalf of a client, I also used it to limit operations to read-only. It works great, and was pretty easy to make.

As the article says, creating these proxies allows for a much smaller attack surface area. The proxy itself is relatively simple.

As an aside, the whole reason I needed the proxy I made was because Atlassian uses OAuth 1.1 instead of OAuth 2, or at least they did in the past with their server products. The cloud products may do something different now.

The problem with OAuth 1.1 is that it requires clients to perform cryptographic operations. These tend to be complicated and tricky to implement. OAuth 2 fixed that by eliminating the need for cryptographic operations or pushing them to the server instead of the client.

ozfive3• 3 years ago

Private Key is held in an environment variable called 'OPEN_KEY'. Wouldn't they be better served retrieving the key from Vault or KMS for revocation purposes?

tptacek6• 3 years ago

In practice, we do store things in Vault, but we expose that to VMs (where this runs in our architecture) via environment variables injected into the entrypoint process by our init through our orchestrator. You could write direct Vault integration, or do what most people probably do and just use Vault to set an environment variable.

ozfive3• 3 years ago

Hey, thanks for clarifying that for me!

twodave• 3 years ago

First read this as “Tokenized Tolkiens”. What a let-down.

itake• 3 years ago

South Park did an episode about this

ozfive3• 3 years ago

Ahhh just for fun... https://www.youtube.com/watch?v=L9fyB6iJvTU

ec1096853• 3 years ago

I wish more things / ecosystems supported mTLS. Then you aren’t storing any long lived tokens. You exchange per instance metadata with key material to generate client tls certs that are then used to auth server calls. The certs have a short lifetime.

csears• 3 years ago

The same thing occurred to me while reading this.

I suspect mTLS adoption has been slow because it’s easier to reason about authentication when the mechanics are “closer” to your application code. The mental model of bearer tokens in HTTP headers is pretty easy. Using mTLS requires understanding a lot more moving parts, and TLS still feels like a magical black box in many ways.

Are there any libraries you would recommend that provide a good developer experience around using mTLS?

ec1096853• 3 years ago

We’ve used Istio for some and a open source system called AthenZ to manage the key material.

lll-o-lll2• 3 years ago

It’s cool, but the http connection to the proxy makes this only viable with the “vpn” tunnel they use. Otherwise MITM becomes a thing.

Also, you’ve gotten the secret off the client machine, but the attacker can still do anything the secret can do by using the proxy? Perhaps I’m missing something.

tptacek6• 3 years ago

We don't need it, nor does anyone who can set up a WireGuard tunnel between their app instance and their proxy instance, or any other IP-level secure channel, but if you really cared you could write the TLS termination logic to make this work --- every MITM testing proxy has it, and there's Go code to do it. It's just pointless in our environment.

The attacker can currently do anything with the secret by interacting with the sites allowlisted for that secret, but they can't exfiltrate the secret, which is the goal of this security control. You can do better, if you like, by further locking down which endpoints they can call, but the wins past "log carefully and no exfiltration" get smaller and smaller, and at some point you're burning time that can be spent more productively on unrelated controls.

If you get what it's doing, you get it. :)

lll-o-lll2• 3 years ago

I think I get what you’re doing, I’m just struggling to see the significance. The OAuth2 token is supposed to only provide the access the client needs. So you don’t get the raw token and instead need to interact via the proxy, so what? What have you blocked the attacker from doing? [edit] I do see intrinsic value in being able to prevent direct access to the tokens and secrets, it just doesn’t look the game changer implied here. Not to me at least.

e12e• 3 years ago

This looks great. A little unfortunate that this effectively requires plaintext http/1.1 (and won't work with "standard" encrypted http/2) - but that's not fly.io's fault.

Tokenized Tokens

Related Stories