You Are Doing JSON APIs Wrong (“foo”:{}, not “type”: “foo”)

https://gist.github.com/FeepingCreature/d2fd982f485973a154abcaf0ccb4003c

46 34 4 years ago

gist.github.com

this is how I usually write JSON. { “type”: foo, … } is more verbose and can lead to issues if you forget to check the type, and moreover it makes it easier to generate some complex types.

nly2• 4 years ago

The type of every JSON object field should be uniquely determined by its field name.

Fields of variant/oneof/union type are really useful, so I don't agree with this. The problem regarding JSON serialization of type-tagged fields and streaming parsers is very well observed though.

I suppose one nice thing about adding a sibling "type" key, is it allows you to have the option to make a non-variant field a variant later on (by allowing the type field to have a default if it's missing).

FeepingCreature5• 4 years ago

You can do union types with the same mechanism: `{"foo": FooTypedValue}` or `{"bar": BarTypedValue}`. No variant, admittedly, but there's no making open-ended JSON objects fast no matter what you do. The problematic thing with `"type"` is that it forces the parser to treat your entire message as variant.

nly2• 4 years ago

That's not a variant, it's just 2 optional fields: one called "foo" and one called "bar". A validating parser then has to know that they are semantically linked. Admittedly this is what protocol buffers does in the binary wire format, because oneof's are just a frontend hack (iirc the latest value decoded gets kept), but it's not the same.

hyperhello• 4 years ago

It sounds like the point is to type members before their data in the stream. But that can’t be a huge drain on serializers. JSON isn’t supposed to be incredibly performant, it’s supposed to be human readable. And with multi core I would expect serializers to read ahead anyway.

There was a thing called Reverse Hungarian that put an abbreviation for the type in the variable name. That had some value to work on very small screens without much scrolling.

FeepingCreature5• 4 years ago

It's even more human-readable this way though (IMO), because it saves you from having to pull a type field from somewhere of a soup of a log message. With the key-identified format, the type is guaranteed to be directly before the data.

eatonphil2• 4 years ago

Pretty sure TypeScript discriminated unions don't work unless you have a type field you can switch on.

hyuuu• 4 years ago

why cant you have:

type Message { contentA?: ContentAStructure, contentB?: ContentBStructure, }

on read, you are already accessing the property data as typed right?

matt_kantor• 4 years ago

That's not a union. Both `{}` and `{ contentA: a, contentB: b }` inhabit that type, so it doesn't represent "a value of type `Message` is either a value of `ContentAStructure` or a value of `ContentBStructure`".

eatonphil2• 4 years ago

You can but that's not type safe in TypeScript.

stickfigure4• 4 years ago

This is premature optimization. A very small percentage of APIs return so much data that you need to worry about stream parsing. The vast majority of APIs in the world, including probably yours, returns structures that are easy to manage in memory.

If you render this `{"MyMessageType":{...}}` structure, people who live in statically typed languages will hate you. At best you've created a union type with a truly vast number of possible cases. In practice you've forced this to be read as `Map<String, JsonNode>` and dynamically interpreted.

Unless your problem domain routinely streams gigabytes of data across an API, don't do this. And if it does stream gigabytes of data across an API, figure out how to chunk the data and still don't do this.

Spivak• 4 years ago

How does { "type": "huge union of types" } help you here? You still have to read the whole thing as a generic Map just to find the type key and then dynamically unmarshall to your real object.

Like what even is the return type for parseJSONBlobToOneOfFiftyMessageTypes(String json)? Guess you could return a generic APIMessage and a type object so the caller can cast it but now you're back to playing dynamic games.

agency• 4 years ago

Coming from TypeScript, having an explicit discriminator field (like "type") is a lot easier to deal with[1] (which to be fair is compatible with the last example given in the OP where they have a "type" field and a generic "data" payload field).

[1] https://www.typescriptlang.org/docs/handbook/2/narrowing.htm...

stickfigure4• 4 years ago

In Javaland, Jackson and GSON both have easy ways of deserializing polymorphically based on a type property.

FeepingCreature5• 4 years ago

I've ran into business-relevant slowdowns due to this issue. I live in a statically typed language. Your "at best" is actually very straightforward. Also, sum types exist.

The problem is that by the time you stream gigabytes of data, if you have done it the wrong way, you now have to add a new API revision for the right format, update your whole ecosystem - if the protocol is even in your control. If you did (or specced) it right from the start, you'd only have to adjust the clients at that point, and it would have been just as easy up until then.

Zababa• 4 years ago

How is this premature optimization? Since it's equivalent to the alternative, it should be known that this way has an advantage.

stickfigure4• 4 years ago

Adding a ton of structural noise is not "equivalent". This pattern will be repeated throughout your hierarchy, and your clients will have to deal with this:

    GET /assets/123
    {
        "Asset": {
            "owner": {
                "Person": {
                    "name": "Bob",
                    "contact": {
                        "Address": {
                            "street1": "blah",
                            ...
                        }
                    }
                }
            }
        }
    }

On the other hand, the traditional approach:

    GET /assets/123
    {
        "owner": {
            "type": "Person",
            "name": "Bob",
            "contact": {
                "type": "Address",
                "street1": "blah",
                ...
                }
            }
        }
    }

This is much more sane to navigate. And when you end up making some previously-static node polymorphic (as happens), the structure does not change.

kevincox• 4 years ago

I don't know about other languages but this is perfectly fine in Rust. It is actually the default formatting for enums (unions) and does streaming parsing.

    #[derive(Debug,serde::Deserialize)]
    struct MyMessage {
        data: i32,
    }
    
    #[derive(Debug,serde::Deserialize)]
    enum Example {
        MyMessageType(MyMessage)
    }
    
    fn main() {
        dbg!(serde_json::from_str::<Example>(r#"{"MyMessageType":{"data":5}}"#));
    }

Running:

    [src/main.rs:12] serde_json::from_str::<Example>(r#"{"MyMessageType":{"data":5}}"#) = Ok(
        MyMessageType(
            MyMessage {
                data: 5,
            },
        ),
    )

https://play.rust-lang.org/?version=stable&mode=debug&editio...

I'm not sure why you are forced to read it into a Map<String, JsonNode> unless you are doing your parsing without type information in which case any solution is going to require a map.

selfhoster113• 4 years ago

1. I live in a statically typed language, and honestly what's purposed in the article is not such a burden. It'd much prefer to receive the object type data via that Content-Type header or equivalent, but this will do as well.

2. If you're sending me gigabytes of JSON, something has gone horribly wrong. Split it up, or send me the result in a more suitable format.

vba616• 4 years ago

If you're sending me gigabytes of JSON, something has gone horribly wrong. Split it up, or send me the result in a more suitable format.

Gigabytes of XML, coming right up!

henrydark• 4 years ago

I came in knowing I'd be against it, and came out convinced. The argument is just solid: it's easier to make things correct as well as faster this way.

__ryan__2• 4 years ago

While it's easier to use discriminated unions with some "type" field (easier to use switch/case) here's the reality:

1. There's many ways for the compiler to narrow a union type [0]

2. There's a better solution than depending on "easy" narrowing for excessively large union types: type guards, callback registry, etc.

[0] https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgA...

jurschreuder• 4 years ago

I may be weird but I do it like this:

{ "type": "a", "version": "1", "payloadJson": "{\"a\": 10}" }

__ryan__2• 4 years ago

Why?

Otek• 4 years ago

That is kinda weird, why would you store json in a json as a string?

arinlen• 4 years ago

{ "type": "a", "version": "1", "payloadJson": "{\"a\": 10}" }

What's wrong with REST and leaving stuff like versioning and resource type to HATEOAS?

selfhoster113• 4 years ago

It is weird. There aren't many reasons that I can think of for encoding things this way.

remram• 4 years ago

It sounds like you are reinventing HTTP headers and trying to put them in the body for whatever reason.

I've also seen this before, but I can't say I didn't laugh:

    HTTP/1.1 200 Ok
    Content-type: application/json

    {"status": 404, "type": "xml", "body": "<entry><name>...</name><author>...</author></entry>"}

benibela• 4 years ago

I am doing ["foo", {} ]

You Are Doing JSON APIs Wrong (“foo”:{}, not “type”: “foo”)

Related Stories