Ultra-minimal JSON schemas with TypeScript inference

https://github.com/ar-nelson/spartan-schema

103 44 4 years ago

metadat4• 4 years ago

Unfortunately excessively verbose and non-portable to backends other than Node.js.

This is a good start, and what problems does this solve IRL? As in, actual problems that happen? (Not just theoretically possible classes of problems.) I get the narrowest use-case, and then beyond that this seems otherwise doomed to become irrelevant over time rather than gain momentum and become something meaningful in any lasting sense.

Think bigger.

Then the meta solution will become self-evident.

skywal_l• 4 years ago

What do you mean by excessively verbose? They seem to make a point at being minimalist and the comparison with JSON schema in the README page seems to attest to it.

metadat4• 4 years ago

Compared to a struct in C, Rust, Golang, or Java it takes more key presses on behalf of the programmer and only works in (/is optimized for) a narrowly specific set of circumstances.

Convenient in some ways (e.g. if you are only comfortable in JS), and does nothing useful in other contexts.

Nothing inherently wrong with it, other than it seems like a missed opportunity.

Is this more than a subset of Wordnik's Swagger?

https://swagger.io/

the_gipsy• 4 years ago

doomed to become irrelevant over time rather than gain momentum and become something meaningful in any lasting sense.

You seem to be jumping from topic to topic without being able to express your thoughts very well.

varanauskas• 4 years ago

For those wanting to infer Typescript types from JSONSchema itself there is: https://github.com/lbguilherme/as-typed

evv3• 4 years ago

Nice! I have been happily using json-schema-to-ts for this exact thing.

And if you're doing code generation, there is also json-schema-to-typescript

vlovich1233• 4 years ago

I don’t know. Zod feels more reasonable path forward to accomplish the same thing (granted it’s not using the language of JSON schema if you have a dependency on that for some reason)

girvo• 4 years ago

Zod is fantastic. Our whole front-end is built around it, with some React hooks that wrap up a `createApi` call around Zod decoders and expose them to components/containers. It's lovely. Only downside is those decoders can't be re-used across other languages, whereas technically this could be, I guess?

evv3• 4 years ago

Zod is great, but JSON-schema is my pick because it is easily serialized. So a client can query for the schema of the data it should send or receive, including descriptions/metadata that is useful for presenting a UI.

vlovich1233• 4 years ago

Zod to JSON schema is a thing so I think that solves your problem, no?

https://www.npmjs.com/package/zod-to-json-schema

dgb232• 4 years ago

Does zod let me walk/parse the definitions as data?

vlovich1233• 4 years ago

I can’t recall if it’s particularly ergonomic, but technically I believe it’s possible.

jitl6• 4 years ago

The last time I tried this, I remember feeling sad and needing to wrap Zod in my own layer of stuff.

brodo4• 4 years ago

I like the idea. Writing JSON schema by hand is cumbersome. I would like to able to generate JSON schema from this.

dgb232• 4 years ago

Just to be sure: You still find it cumbersome with an editor like VSCode where you get autocompletion for pretty much all the stuff json-schema can do?

brodo4• 4 years ago

Yea. I now use JSONNET to generate my schemas because of it oftentimes.

wryun• 4 years ago

Even more minimal one I made years ago:

https://wryun.github.io/yajsonschema/

mirekrusin3• 4 years ago

I'm fan of point-free combinators [0] [1] which compose in intuitive way.

[0] https://github.com/appliedblockchain/assert-combinators

[1] https://github.com/preludejs/refute/

lifthrasiir• 4 years ago

I made a similar thing back in time, unfortunately not a public code though. The idea itself is simple, it was approximately something like this:

    const kind = Symbol();
    type Spec = "unknown" | "string" | "number" | ... |
        { [kind]: "optional"; spec: Spec } |
        { [kind]: "array"; element: Spec } |
        { [P in string | number | symbol]: Spec } | ...;

    function optional<S extends Spec>(spec: S): { [kind]: "optional"; spec: S } { ... }
    function arrayOf<E extends Spec>(element: E): { [kind]: "array"; element: E } { ... }
    // and so on, you get the idea. these helper functions exist so that
    // you can write `optional(...)` or `arrayOf(...)` instead of the full specification

    type Validated<S> =
        S extends "unknown" ? unknown :
        ...
        S extends { [kind]: "array"; element: infer E } ? Array<Validated<E>> :
        S extends { [P in string | number | symbol]: Spec } ? { [P in keyof S]: Validated<S[P]> } :
        ...;

    function validate<S extends Spec>(value: unknown, spec: S): asserts value is Validated<S> {
        if (typeof spec === "string") {
            switch (spec) { ... }
        } else {
            switch (spec[kind]) { ... }
        }
    }

The actual implementation of course had to take care of implicit nulls and other caveats of the TypeScript type system.

iainmerrick• 4 years ago

This looks really nice as an alternative to JSON schema, which is horribly verbose. Like using RELAX NG compact syntax for XML schemas back in the day, versus horrible verbose DTDs or XSDs.

But I think the most natural way to express JSON schemas would be TypeScript type declarations. Is there a project that can take a TS type and generate a runtime parser/validator for it?

I thought maybe Quicktype (https://quicktype.io/) was that, but it looks like it only takes JSON as input, not TS types.

xmonkee• 4 years ago

YES! I've introduced ts-to-zod into my codebase only yesterday, and it's served my needs exactly. It doesn't handle generics or mapped types, but that's a fair price to pay.

It also works the other way, you can define a zod schema and you can infer types for your data from it.

https://github.com/fabien0102/ts-to-zod

jitl6• 4 years ago

After some frustration with the TypeScript schema library ecosystem, I've decided that I'd prefer to declare my types using TypeScript's excellent type syntax, so I can take advantage of generics, mapped types, etc. Then, I'll take those TypeScript types and compile them to whatever alternative schema format I need.

There are many libraries that claim to convert your Typescript types to other formats, such as ts-json-schema-generator, ts-to-zod, or typeconv/core-types-ts. These libraries work by interpreting the Typescript AST, essentially re-implementing a bare-bones type system from scratch. Most do not support advanced Typescript features like generic application, mapped types, or string literal types. So, what's the point? To avoid those limitations, I use Typescript's first-party ts.TypeChecker API to analyze types, and an existing library called ts-simple-type (which I've forked) to convert from ts.Type to a more usable intermediate format. Then, I recurse over the intermediate format and emit "AST nodes". It's pretty simple, but seems promising.

So far, I have compilers from TypeScript type to Python 3 and Thrift. But I plan to add OpenAPI/JSONSchema, Protobuf (Proto3), Kotlin, Swift, and maybe Zod and Avro. Each target language is around ~300 LoC so they're pretty easy to put together.

My ultimate goal is to use this toolkit with Notion's internal types, which are quite a bit more complex than something like ts-to-zod can handle.

Repo: https://github.com/justjake/ts-simple-type

Compiler input and output: https://github.com/justjake/ts-simple-type/blob/jake--compil...

Thrift compiler: https://github.com/justjake/ts-simple-type/blob/jake--compil...

Python compiler: https://github.com/justjake/ts-simple-type/blob/jake--compil...

eropple• 4 years ago

Most do not support advanced Typescript features like generic application, mapped types, or string literal types. So, what's the point?

I use @sinclair/typebox and the point is that I can get a JSON Schema out the other side, usable in-process and as a communicative description of what my code expects, just by evaluating my code and printing out an object.

It does support string literal types, FWIW. I'm good with it not supporting generics, because the places where I need an interchange format generally don't. Mapped types would be nice but aren't really critical to where I need to generate interchange formats; dependent types that result from transforming these can use mapped types, so it's close enough.

For my money, typebox has far-and-away the best user experience of any of the flavor of libraries you described, and IMO is probably worth studying.

jitl6• 4 years ago

I agree that the runtime-validador-with-type-inference library genre is generally enough for API surface area. We have a similar internal library, including JSON schema output, and it’s serviceable for checking our public & private API incoming requests.

It does come with some costs. First, the type inference using these large mapped types (at least in our implementation and/or scale) noticeably slows down our Typescript type checking; the public api files with our runtime type declarations always top our build profiling. Second, it’s annoying to need to re-implement an existing type as a runtime-type if you start to need it in an API specification. There’s a tension in the codebase between using the easily-available, language native type system, or making do with the runtime type stuff which increases verbosity/noise and decreases expressiveness.

We have thousands of lines of types, including many types inferred from `as const` literals, written in Typescript. Rewriting those types in another schema language, such as typebox or Protobuf, would take a bunch of time and spread that tension from an isolated spot (backend API handlers) into every part of the codebase. It imposes new constraints, and requires devs to learn more things.

The libraries I listed and the compiler I’m building try to address this issue by supporting the native typescript declaration syntax. No more red types and blue types, much better gradual adoption pathway, and ideally one less thingy to learn for most developers.

brodo4• 4 years ago

Looks promising. Thanks for sharing. This also needs a CLI. I was thinking about building something like this in Rust using swc.

jitl6• 4 years ago

Swc will give you an AST, but you’ll still need to re-implement mapped types, type inference, template string types, etc - a large part of the TS type checker. Although, the author of swc is supposedly building a full type checker.

This also needs a CLI.

I’m not planning on implementing one at the moment. The project goal is to greatly expand the use-cases for Typescript types by making it much easier to build custom compile targets.

A CLI means directing much more of my energy to configuration logic and file system IO. As a library, it’s clear how users should “configure” behavior - write or subclass a compiler target and implement whatever you want!

gavinray• 4 years ago

Wow, holy smokes. This is incredible, thanks for posting.

hebrox• 4 years ago

We're using TypeBox [0]: you use TypeScript to create JSON Schema's and it gives you a compile-time type.

From the documentation:

    const T = Type.String()     // const T = { type: 'string' }
    type T = Static<typeof T>   // type T = string

The main thing we're using it for now, is defining an OpenAPI schema and using the types in the front-end and backend, and using the schema to validate requests and responses, both client-side as well as server-side. Validation is done using avj [1].

We used Joi before and want to replace that code with JSON Schema, but that is quite the hassle. I like that TypeBox uses JSON Schema underneath, so migrating to another library should be easy.

[0] https://github.com/sinclairzx81/typebox

[1] https://ajv.js.org/

Yaina• 4 years ago

I recently used a package that converts typescript to JSON Schema, which I thought was pretty nice.

I get the legibility (and IntelliSense) of typescript interfaces but can make use of the all the validation libraries that use JSON Schema.

seanwilson• 4 years ago

I wish there would be an official TypeScript solution for this. I know there's a TypeScript goal that TypeScript types should not impact runtime behaviour, but I think converting arbitrary JSON from a file or network to a typed TypeScript object is such a common use case it needs a standardised solution. There's tens of libraries that try to solve this now in different ways, all working around this area where TypeScript doesn't try to help.

jitl6• 4 years ago

Deepkit is quite interesting in this area: https://deepkit.io/

It reimplements the type system as… a stack based VM that expands the TS typedefs if needed at runtime?

I would use it if the risk factor wasn’t so high. I’m too scared of needing to maintain their crazy cool stuff to use it.

I would love Microsoft to build those APIs first party and officially support them.

EDIT:

I know there's a TypeScript goal that TypeScript types should not impact runtime behaviour

This is why I’m rolling up my sleeves to build my own typescript-to-X tooling - it seems like no one is gonna do it for me, just the way I like — so I should make it easy for everyone to do it the way they like.

mirekrusin3• 4 years ago

To be honest it doesn't even have to do anything during runtime per se.

It just needs to have nice macro system.

It would solve this problem and others like ie. pattern matching.

jitl6• 4 years ago

there's always https://www.sweetjs.org/ that you can plug into your Babel pipeline

Ultra-minimal JSON schemas with TypeScript inference

Related Stories