Introducing Conjure, Palantir’s Toolchain for HTTP/JSON APIs

https://medium.com/palantir/introducing-conjure-palantirs-toolchain-for-http-json-apis-2175ec172d32

by iamdanfox • 8 years ago

61 10 8 years ago

medium.com

markelliot4• 8 years ago

I'm one of the authors of this project -- happy to answer questions!

brokensegue• 8 years ago

Python bindings?

jdm2212• 8 years ago

Yup! https://github.com/palantir/conjure-python

0l0rin2• 8 years ago

At a first glance it seems like a step backwards from already established solutions like Thrift or Protobufs? What are some of the reasons you decided to go this direction?

markelliot4• 8 years ago

Two big motivations for us: (1) we had a large footprint of JAX-RS annotated Java services and a correspondingly large footprint of Typescript frontends communicating with those services, both hand-maintained; (2) we wanted something that felt just as native and ergonomic in browsers as it did on the backend.

Migrating from that setup required first having a declarative API format to translate through, and Conjure was our answer (starting in 2016) to generate human-quality code that would drop-in-replace our hand-maintained, language-specific client and server definitions.

That said, we're fans of gRPC/Protobuf and heavy Cassandra users (so also have a good breadth of experience with Thrift), and gave both serious looks before getting going, and again before deciding to open source our work today.

When we started on this tooling there really weren't great gRPC options for the browser, and the balance of our developer pain was around frontend/backend rather than backend/backend RPC. We also took a long look at Swagger/OpenAPI, but ultimately moved on because it focused more on the full coverage of any kind of HTTP/JSON API and as a result was too general to end up with consistent APIs across many services.

Over the last two years of development (and conversion of all our hand-maintained clients) we found that Conjure held a lot of value as an easy-to-adapt declarative definition format, that it applied strong enough constraints so as to make API development focused on the semantics and behaviors rather than the syntax or specification. And, we thought that that had sufficient value we should open it up to others.

Beyond that, we've got some work underway to use protobufs as the wire format, and enough flexibility built into the framework that we can use that or other non-JSON wire formats along-side JSON with the same client and server interfaces and code implementations.

0l0rin2• 8 years ago

Thanks for your answer. I think I initially misunderstood the intended purpose of this tool (probably because of how many times the word RPC appears in the text) and assumed it was used for backend-to-backend communication, for which HTTP/JSON seemed extremely sub-optimal given the existing open-source solutions.

While it does make a lot more sense in the context of FE<->BE, the second part of your answer, where you speak about consistency of APIs across many services is still a little confusing, as it suggests BE<->BE communication again. Unless you have a huge, monolithic, internet/browser-facing service (which sounds rather undesirable) it's hard to imagine how browser<->middleware API could get out of hand to the point where it requires a dedicated unification framework.

Rephrasing my question - even though Conjure is not a commercial product (respect to Palantir for contributing to opensource) you must have had a target audience in mind for it: who is it? What exactly is the problem that Conjure is solving better than its existing alternatives?

konradb• 8 years ago

This looks interesting. How do you (if at all) tackle API versioning, or do you see that as orthogonal?

iamdanfox• 8 years ago

We require all teams to use semver and usually define the Conjure API in the same repo as the server that implements it (so they are versioned together). Most of the time teams want to preserve wire-format back compat (i.e. only add optional fields to requests and don’t remove things from responses). Devs are pretty good at this, but we actually run a conjure-backcompat checker as part of CI to catch any unintended breaks they might have missed.

When it comes to intentional breaks, teams save up a bunch of carefully documented changes and then tag the next semver major version (i.e. 2.0.0). This is safe to do because all microservices and frontends we build actually contain embedded minimum and maximum version constraints on the backend services they require. Our deployment infrastructure knows about all these constraints and can then ensure that new major releases will only be rolled out in production when all their callers have explicitly declared support.

markelliot4• 8 years ago

The deployment infra constraints mentioned in parent post are generally managed by tooling we also publish[1]. The encoding ends up similar to Helm’s, but happens through this tooling just by virtue of taking a dependency on a published jar, npm package or conda package, and that removes a lot of programmer error/guess work/maintenance. (I think we’d be open to also emitting Helm’s formats in our packaging tools, if that’s interesting to someone reading this feel free to open an issue on our repo and reference my post.)

[1] https://github.com/palantir/sls-packaging

Introducing Conjure, Palantir’s Toolchain for HTTP/JSON APIs

Related Stories