How not to sign a JSON object

https://latacora.micro.blog/2019/07/24/how-not-to.html

by LaSombra • 7 years ago

268 151 7 years ago

latacora.micro.blog

nemo1618• 7 years ago

I recall this plaguing the Secure Scuttlebutt project -- they were signing JSON objects that were encoded however Node saw fit to encode them, which meant that in order to write an alternative implementation of the protocol, you had to encode all the various objects exactly how Node encodes them, which is (unsurprisingly) non-trivial. I wonder if they addressed this in a later version.

Vendan6• 7 years ago

As one of the people that built a alternative implementation, this is completely true, but also much worse, cause there were 2 things around crypto, signing and hashing, and one encoded it one way, and the other encoded it a different way (encoding as far as utf16 and such) which meant that for ascii, everything just worked fine, but as soon as you started using non-ascii, everything got 2x as complex.

tracker110• 7 years ago

If you're serializing the JSON, that is UTF8. You're signing UTF8 bytes... don't mess with them, and the signature will be the same... also, for JWT, the UTF8 is then converted to base64 representation and tethered to the signature. You're signing the UTF8 bytes, not JSON. It doesn't matter how it's serialized, if it isn't UTF8, you're doing it wrong. The order or properties doesn't matter, the signature is on the bytes.

Vendan6• 7 years ago

SSB signs objects like so:

    {
      "previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256",
      "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
      "sequence": 2,
      "timestamp": 1514517078157,
      "hash": "sha256",
      "content": {
        "type": "post",
        "text": "Second post!"
      }
    }

gets signed like

    {
      "previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256",
      "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
      "sequence": 2,
      "timestamp": 1514517078157,
      "hash": "sha256",
      "content": {
        "type": "post",
        "text": "Second post!"
      },
      "signature": "z7W1ERg9UYZjNfE72ZwEuJF79khG+eOHWFp6iF+KLuSrw8Lqa6
                    IousK4cCn9T5qFa8E14GVek4cAMmMbjqDnAg==.sig.ed25519"
    }

which means that you have to be careful about how you remove the signature in order to verify the original. The "main" node implementation does this by parsing the json, removing the field, and then re serializing, forcing any alternate implementation to exactly match the node serialization in order to be compatible

lvh41• 7 years ago

That's true if you have the luxury of a traditional tag on the outside, but falls apart when you have systems where signatures are in-band, like SAML, which is where canonicalization shows up. (Both for UTF-8 itself, which isn't canonical, but especially for your serialization format e.g. XML/JSON, which is usually far hairier.)

steveklabnik• 7 years ago

JSON is not always guaranteed to be UTF-8. It wasn't updated to mandate that until 2017 https://www.tbray.org/ongoing/When/201x/2017/12/14/RFC-8259-...

That said, I certainly hope that it is, generally speaking.

andrewstuart2• 7 years ago

An easy enough mantra: sign bytes, not semantics. I've trudged through the bowels of SAML long enough to know that canonicalization is thoroughly tricky and annoying to deal with.

I don't care what you throw in the bytes; I can figure that out easily enough. I want to know who generated those bytes and that they haven't changed.

For this reason also, I think the "reach for symmetric first" is bad advice. The primary benefit I can think of for HMAC is if you want to store client-side state for more stateless browser-targeted services. It makes it easy to throw an obscure cookie to your client and then validate it on the way back. It seriously complicates things if you want to allow other people to validate provenance of those bytes. With symmetric crypto, as soon as you can validate you can also generate. That's not the system we want.

lvh41• 7 years ago

You say that's not the system you want but that's based on a pretty big "if": you need "someone else" to validate something, and those people can't talk to you directly and ask you if it's valid or not. That's not in practice how we see this being used.

If you do need third parties to validate, a much simpler API is to just ask the thing that holds the HMAC credential over TLS, and have it return true/false. That makes it harder to demonstrate non-repudiation, but I'm not convinced that's a property you generally care about. Even in OIDC (which mandates a JWT, but for reasons that defy understanding doesn't mandate cryptographic domain separation between (IdP, RP) pairs), leading OIDC providers have long recommended you just talk to them over TLS to get userinfo. (GSuite has recently muddled this in their docs, which irks me.)

(Disclaimer: I'm the author.)

ysleepy2• 7 years ago

But what about myself wanting to validate the payload on a system I don't necessarily trust? In the symmetric case I would grant that system the ability to sign.

hirsin3• 7 years ago

Eh? Got additional details on which OIDC providers aren't using a proper userinfo + AT? Not that that shouldn't include TLS...

We don't do token introspection (basically what you're referring to) and instead use JWT/JWE(soon) to reduce round trips for the RP.

Could you expand on the OIDC issue with cryptographic domain separation? Not sure I fully understand what benefits you're looking for there.

DenisM2• 7 years ago

holds the HMAC credential over TLS, and have it return true/false

And if you're connecting to the original party anyway might as well not even use HMAC or any crypto at all. When "signed" data needs to be sent to the client don't send it at all, instead store the data locally under a GUID and send that GUID to the client. When client takes their GUID to another party, that other party connects to the original party and retrieves perfectly authentic data. Free bonus - faster client performance.

Best crypto is absence of crypto.

SlowRobotAhead3• 7 years ago

Sign bytes makes a lot of sense. But you’re still on your own for how that “should” be done.

Obviously you need to build and hash the pairs and not just key values. But the order is obviously important too.

Is there some standard for this I don’t know of?

zdkl• 7 years ago

Sort the keys alphabetically usually comes up.

blattimwind• 7 years ago

You generally don't need canonical representation.

sk5t• 7 years ago

It's a good mantra--just not a wish that is applicable to the foundation of XML as a tree/document thing that self-describes its encoding and is capable of including its own signature within itself. I mean, it gets crazy complex, but the alternatives sort of require going on to some message format that isn't XML. One of the design goals of XML is that you could zip an XML document along through different systems, one of which supports only UTF-16LE, the next demands UTF-8, the next demands ISO-8859-1, and each could do the right thing. I mean, it's almost inconceivable that all the systems involved in such a quagmire would actually do the right thing, but they could, in theory. I'm sure you're already dreadfully familiar with this having looked at XML-DSIG.

OTOH: base64(jsonString(envelope)) '.' base64(jsonString(claims)) '.' base64(sig) works fine for JWT, because JSON has a very much limited choice of encoding (one option, zero alternatives!) and a JWT has no requirement to still be JSON on both sides of the transformation. It's just an opaque 7-bit-safe string to any party that doesn't know what the contents are.

munchbunny• 7 years ago

I don't care what you throw in the bytes;

I do think whoever is coming up with the scheme for assembling those bytes should care a lot. Specifically, whenever you're signing something, you have to pay attention to whether that something still has exploitable ambiguities. Most of those come from how you delineate the fields in the thing you're signing, which is a classic "you find out what you missed when someone sends you a CVE" type of problem. So you're right back to the problem of "if not JSON or XML then you're still rolling your own."

I agree with "reach for symmetric" in this sense: what are you trying to accomplish? Prove the data came from a specific source or that they are who they claim to be? Can you just use TLS with mutual auth? Prove the data came from yourself? Symmetric is probably okay.

When you need verifiable attestations of data from parties that don't talk to each other... welcome to the jungle.

rstuart4133• 7 years ago

canonicalization is thoroughly tricky and annoying to deal with.

The bitcoin malleability problem was the same issue, just with ASN.1 instead of JSON.

kerblang3• 7 years ago

Maybe you don’t need request signing? A bearer token header is fine

If you're going to insist on symmetric key signatures, yeah. Otherwise a symmetric signature would be the same as using symmetric encryption to store user passwords, wouldn't it? You have to have the secret key to verify the user's signature.

lvh41• 7 years ago

I don't understand your analogy. Can you elaborate? I don't see the parallel between (a)symmetric signatures and encrypting passwords.

kerblang3• 7 years ago

The article appears to say that you must sign your requests with a symmetric key. To verify the request, the server needs the same key, right? So the server needs a copy of every single user's encryption key.

The server _could_ just store the keys plaintext in a database but I'm assuming we can agree that's a horrible idea. The best it can do is encrypt them symmetrically before storing them, using an extra-special secret key that needs to be protected very carefully.

With username/password authentication, we don't store symmetrically encrypted passwords, do we? We store a one-way hash of the passwords instead, because symmetric is deemed not-good-enough.

lvh41• 7 years ago

A super-awesome related project I didn't get a chance to work into this post (because it's entirely deserving of its own): https://github.com/benlaurie/objecthash

To be clear: I think that's a niche use case and while I think ObjectHash does a great job of exploring it, I don't expect the median startup to need an ObjectHash implementation.

(Disclaimer: I'm the author.)

tialaramex3• 7 years ago

Why is it _better_ to do tricky message processing before we achieve confidence that the message is legitimate/ genuine?

Or is the argument that even though this is worse it's so useful we might really want to do it anyway?

lvh41• 7 years ago

Definitely authenticate first!

If there's anything I said that made you think otherwise, let me know: I would like to amend that so no-one else thinks I could possibly mean that. The initially recommended (unless you can do otherwise) approach in the blog post is clearly "tag at the end" and every other approach also validates first. If you're referring to ObjectHash: like I said, it's a very niche application, I don't expect people to use it, and yeah, it enables new use cases.

(I expect you'd still really be authenticating the ObjectHash somehow -- e.g. by sending it over TLS -- but that's out of scope for ObjectHash itself.)

mixedCase2• 7 years ago

Anyway, no, because you need to parse a header to read the JWT, so you inherit all of the problems that stem from that.

Can anyone enlighten me what point is the author trying to make? JWT is pretty damn standard so it's my go-to for signing objects.

lvh41• 7 years ago

We should write the "The JWT Problem" post one day, but this post isn't that, so I really don't want to derail the discussion.

The short version is that there are flaws in the JWT specification that make certain bugs likely. A classic example of the "you have to parse a header to use the JWT" problem is the HS256 vs RS256 confusion bug, where your JWT library would interpret an allegedly-HS256 (HMAC) JWT using RS256 (RSA) key material. The JWT would get validated using the public key of the RSA pair, interpreted as an HMAC key. But the public key is, you know, public! So the impact of the bug is that everyone can forge JWTs. That is not a problem that can happen in well-designed schemes.

We do have a blog post from last year that tells you what we think you should do if you want to be safe and you know what kind of abstract thing (e.g. signing, MACing, etc) you need: https://latacora.micro.blog/2018/04/03/cryptographic-right-a...

(Disclaimer: I'm the author.)

mixedCase2• 7 years ago

Well, it does makes sense to me that if JWT allows for both symmetric and asymmetric tokens that's a complexity detail that can be gotten wrong when implementing.

Thankfully, JWT being quite standard and having momentum behind it I think there's a lower risk in recommending someone to pick a popular JWT library than telling them to roll his own simple scheme on top of HMAC (which is the article's recommendation and what JWT will end up doing regardless), specially when scale is considered.

I know, I know I'm arguing for "worse is better", but I honestly can't imagine a clean solution to this that would be so good it justifies dropping a seemingly decent standard, leaving mountains of legacy and requiring the entire developer world to learn about yet another crypto scheme. But then again I'm no expert in that area, and I would love to read a post about it from someone who is.

thegagne2• 7 years ago

I have a need to provide an update service where client software (that I control) can request download files, but any other clients cannot download the files.

Currently using a terrible method with hashed query strings based on date, path, and a secret, which are then validated and have an expiration. Also, HTTP, so yeah it’s broken, but it at least prevents drive by scraping.

At this point we have no need for assymetric (don’t need to identify the requester, just need to prevent spoofing). What method of securing would you recommend?

jcranmer• 7 years ago

We do have a blog post from last year that tells you what we think you should do if you want to be safe and you know what kind of abstract thing (e.g. signing, MACing, etc) you need

A little bit off-topic, but what is your opinion on NSS versus OpenSSL? Or LibreSSL and BoringSSL?

merlincorey2• 7 years ago

I would be interested to hear your thoughts on PASETO, which I increasingly see as a replacement for JWT -- I just haven't had a chance to use it in production yet, myself.

merb2• 7 years ago

actually you can just never send the header and always assume that the header is `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9` { "alg": "HS256", "typ": "JWT" }

(or whatever algorithm you wish) of course only "internal" clients would understand the jwt that isn't according to the spec.

random023987• 7 years ago

Assuming a JWT implementation accepts only a fixed header (all header fields must be present and match, no additional fields can be present), are there any other issues with "just use jwt"?

SlowRobotAhead3• 7 years ago

Not for nothing, but JWT support in C is really bad. I have an embedded device that I considered JWT, but it didn’t look great for supported libs in C.

So knowing JWT exists, this is still of interest to me.

Boulth• 7 years ago

Looks good. I'm missing ACME request signing in comparison.

baybal22• 7 years ago

More about use case than how to do it: client side certificate authentication is not a rocket science.

Unless you 100% need to use signing for the use case like a client side ACL, there is genuinely no need to overengineer your web app with your own authentication scheme.

But if your scale already mandates doing operations without server side ACL lookup for performance reasons, doing it over HTTP and web stack might already be inefficient by itself for the task.

geofft7• 7 years ago

Client certs are great if you control the client. If, say, Slack told me that to use the Slack API I'd need to get a client cert, I would laugh.

One thing that comes to mind is that either you have to check revocation on the server side, or you have to re-provision client certs frequently, and in most client libraries that's actually difficult / annoying. If the server just sends me an updated token, I can just put that in a local variable and call it done.

There's a reason most APIs have moved to "you can just put the token in the request payload". It's hard enough to ask people to set HTTP headers, asking them to set client certs will be super complicated.

baybal22• 7 years ago

in most client libraries that's actually difficult / annoying.

You can add a client cert with just 2 clicks on Windows. To my experience, that's way easier than most API authentication schemes

EGreg• 7 years ago

Seems that you just need to go ahead and build a canonical JSON encoder in JS. It would be slower but hey, you need the same consistent algorithm for signing. The encoder would be part of the spec.

lifthrasiir• 7 years ago

The OP notes that there are multiple competing (well, probably?) specifications for a canonical JSON and they are annoying to implement. JSON is neither well-designed nor well-specified [1] and it is likely that we will never see the official canonical JSON format anyway.

[1] http://seriot.ch/parsing_json.php

tracker110• 7 years ago

JSON is UTF8, it's already canonical in that regard, if you don't f*ck with the serialized bytes tethered to the signature, you'll be absolutely fine.

lvh41• 7 years ago

Not in the sense that the term of art "canonical", as in canonicalization, is used. UTF-8 is not canonical: different byte sequences can map to identical meanings, as covered in the post.

beefhash• 7 years ago

I'd go one step further: If you're in the situation that allows for HMAC over asymmetric signatures, you might as well go all the way with authenticated encryption instead of purely a MAC. Even if you have TLS, you still gain a minor benefit of having black box tokens to everything outside the system, in particular if you have to let untrusted or semi-trusted clients hold on to signed data that they needn't know the contents of. Yes, security through obscurity is not sufficient on its own, but can help stall analysis and exploitation efforts,

lvh41• 7 years ago

I don't think that addresses the problem set out in the post. Other, non-cooperative third-party systems need to be able to parse the JSON blob as is.

notJim• 7 years ago

I'm sort of skeptical of the idea that a bunch of people hand-rolling HMAC stuff is going to contain fewer security issues in the long term than widely-used, tested JWT libraries. Sure if they do exactly what you say here, and nothing else, that might work. But what happens when they realize they need some minor tweak to it…

tracker110• 7 years ago

Beyond that, hand-rolling JWT isn't so hard at all. When I read about some of the exploits in certain libraries, I didn't have to worry about it, as I only supported the known authority's public key in the system I was working on.

Of course, that makes the header effectively useless in practice.

lvh41• 7 years ago

If you write your own library to do it, there are strictly more things that can go wrong because at some point you're implementing a HMAC validation thingy. If you use a third party library, you're exposed to bugs that stem from JWT's design flaws even if you only allow a specific key, like the RS256 v HS256 bug I mentioned.

You are definitely right that if for some reason you must do JWT, the way to do that is to strip as much of it away as possible. In particular, if you wanted to do safe HS256-only, you'd ignore the header, decode the body and tag, and validate the tag.

tptacek2• 7 years ago

There are two obvious problems with this approach to engineering:

1. You don't know what you don't know, and there is a lot to know about cryptography beyond the minimum needed to interoperate with other systems.

2. If you're engineering seriously, you know people are going to inherit your code and your design down the road, and if you're relying solely on a minimal feature set without a coherent, informed design, those people are building on sand.

Rolling your own JWT is a bad plan.

lvh41• 7 years ago

That is a healthy skepticism! I definitely would not counsel people to do it with primitives in general: it just turns out that a PRF doesn't have a complicated API and HMAC is bulletproof. More or less the only design criterium that comes to mind that is critical is "don't do anything before you validate the MAC", and that design guideline is hard to break when you use the suggested approach.

minitech• 7 years ago

HMAC is bulletproof

… with a well-typed library. Otherwise a lot of people end up comparing it in a timing-unsafe way. :(

mikepurvis• 7 years ago

It's not JSON, but I had a scenario like this where I wanted an in-band checksum on an archive of files. In the end it was indeed a file with the signature included in the archive itself, and the formula for computing it was basically the shasum of the shasums of `file .` sorted alphabetically, with the signature file itself excluded.

That worked out just fine, but I can see the argument that it's much harder to get to a canonical JSON representation than it is to get to a canonical "tree of files" representation. Indeed, it was easy enough in my case that the repo contained a shell script one-liner that would compute it, and that was the reference against which the "real" python implementation was validated.

lvh41• 7 years ago

That sounds like you found an alternative format where finding something canonical to sign is easy (files, the shasum CLI) and punted on canonicalizing the in-band signature (which is good!).

tracker110• 7 years ago

I actually prefer JWT with an asymmetric key. Anyone can confirm the payload's providence. HTTPS takes care of encryption for the payload. Of course with JWT, only allow trusted keys or signed keys from a trusted CA.

There have been some poor implementations, but the method is pretty sound.

dev_dull• 7 years ago

It's amazing to me that the whole article glosses over issues around key storage for symmetrical encryption, not to mention now you have an O(n) liability issue with all of your readers.

tracker110• 7 years ago

+1 on this. With asymmetric signing, you don't have to spread a secret far and wide. You could go a step farther with client certs, but still would need a more heavy management interface and a CA setup against your applications which means quite a bit more complexity.

lvh41• 7 years ago

The set of readers is of cardinality one in the scenario described.

geofft7• 7 years ago

I think the scenario they're envisioning is one where the server generates an object, and some time later, wants to make sure it's getting back an object that the server has previously generated. HTTP cookies are a good example of this. Anything where you'd reach for JWT is probably also this use case.

his post is mostly about authenticating consumers to an API. ... you’re trying to differentiate between a legitimate user and an attacker, usually by getting the legitimate user to prove that they know a credential that the attacker doesn’t.

The recipient of the API key doesn't need to verify their object. There's no attack from being able to give someone a fake API key - any attacker in a position to modify the API key in transit, which would just be a DoS, is also in a position to drop the connection, which is also a DoS. Such an attacker is probably also in a position to steal the API key silently, which is a bigger problem. (If a client is really curious whether they have a valid API key, they can just make an API call with it and see if it works, they still don't need to actually check the signature.)

closeparen• 7 years ago

I don’t think that’s a fair assumption. Google IAP puts JWTs in headers to attest the end user’s identity in a way you can’t just forge from inside the firewall. Other use cases have you obtain a JWT from a dedicated auth service, maybe even a 3rd party provider, so that other random services don’t have to know about passwords and 2FA. In both cases it would be prohibitively expensive to call back to the auth service to verify each token. The point of using a standard like JWT is to make it intelligible in other codebases, otherwise you could make your own format and serialization.

jiveturkey• 7 years ago

A fatally flawed post that dooms it to the category of SEO rather than what should be useful information. I did especially like:

Canonicalization is a quagnet, which is a term of art in vulnerability research meaning quagmire and vulnerability magnet. You can tell it’s bad just by how hard it is to type ‘canonicalization’.

The flaw?

There's nothing here not already well known, so this isn't an insightful piece for those already well versed. Therefore, this is a piece written for those that aren't so practiced, for the sake of discussion: junior dev or devops, or non-security devs.

The content itself is a good discussion, and digestible (lol).

But, as a piece that junior folks are expected to get a takeaway from, the introduction is a disaster:

This post is mostly about authenticating consumers to an API.

ie, not service-to-service auth.

Unless you have a good reason why you need an (asymmetric) signature, you want a MAC.

A MAC/HMAC requires the signer and all verifiers to have the key. As stated just prior, this is about "frontend" signing. A novice reader might not realize they have to guard the key very well, and might even send it to the client browser. "Unless you have a good reason" is not a sufficiently strong warning for a post that is written as an instructable, more or less.

More architectural introduction (bonus with diagrams) is required. As is, this post is a footgun.

tastroder5• 7 years ago

Not sure why you are downvoted, about 10% of the article addresses it's headline and I frankly do not get the point either. The proposed solution is in the first numbered list and between that and the conclusion list is a bunch of prose on non-JSON payload.

The only slightly JSON related content in there is a constructed scenario for 'in-band' signatures (the regex thing) which can just as well be achieved by a bit of string processing. Any JSON object will start with {, end with } and have some more information between it. Replacing the initial curly brace with '{"hmac": "foo",' gives you a valid JSON document. You can remove that easily before parsing and place no restrictions on the object's keys. You can handle edge cases like JSON literals or arrays by wrapping the whole thing in {"hmac": "foo", "payload": yourstring} if you feel like it.

lvh41• 7 years ago

Not sure why you are downvoted, about 10% of the article addresses it's headline and I frankly do not get the point either. The proposed solution is in the first numbered list and between that and the conclusion list is a bunch of prose on non-JSON payload.

The proposed solution is not in the numbered list. The numbered list describes how to sign a JSON blob from the outside. The rest of the doucment describes what you do if that's not an option, and you need to sign the blob in-line. The very next paragraph after said numbered list describes how to do that.

(I'm the author.)

HONEST_ANNIE• 7 years ago

Howto prove you are a God:

Create a json file

    {"msg":"I am a God. My name is Bob.",
     "sha256":"78A873E..."}

where the hash is the checksum of the file including the checksum. It would be equal to finding fixed point in cryptographic hash function that happens to be checksum to your message.

geofft7• 7 years ago

That's pretty easy to do with MD5 :) https://twitter.com/i/moments/838685002703466497

dmitrygr• 7 years ago

Canonicalization is fiendishly difficult.

1000 times this! If you think you need canonicalization, always remember: "no you do not!" It is not a hill you want to die on.

neilv• 7 years ago

While integrating with a large company's SSO protocol (with a cleanroom implementation), I found that the off-the-shelf "standard" XML DSIG canonicalization code they were using (from a major vendor) actually was not compliant with the W3C spec. That was unpleasant to have to discover and then explain.

k__• 7 years ago

What's the issue with it? I read, some GraphQL caching implementations relon canonicalization.

eridius• 7 years ago

The "regex bait and switch trick" doesn't work if any of the intermediate processors might decode and re-encode the JSON.

lvh41• 7 years ago

That's an excellent point, you're of course right and the blog post could do a better job explaining that. I'll amend.

forgotmypw3• 7 years ago

Canonicalization is a quagnet, which is a term of art in vulnerability research meaning quagmire and vulnerability magnet. You can tell it’s bad just by how hard it is to type ‘canonicalization’.

The funniest line from this article and my word of the day.

lvh41• 7 years ago

That one is all 'tptacek.

timemachine• 7 years ago

A fun read and primer on the subject. However, I would like to see more emphasis on including a time stamp as part of the signed payload. The reason is multifold: (a) Most importantly prevents ‘replay attacks’ [1]. Systems that need to send send the same command will do so with a unique time stamp and, therefore, a unique signature. (b) The time stamp + signature is an idempotency key. Your key cache’s TTL is the same as the timeout for your time stamp rejections. (c) To a lesser extent bolsters the system against a brute force length extension attack (see the Flickr API flaw in the reading) by reducing the time the middle-man has to correctly calculate the glue bytes.

1: https://en.wikipedia.org/wiki/Replay_attack

How not to sign a JSON object

Related Stories