The Future is Hypermedia APIs

http://www.emergentone.com/blog/the-future-is-hypermedia/

by Milagre • 14 years ago

36 58 14 years ago

I think these ideas would get a lot more traction if they were presented in terms of something like "Semantic REST" (S-REST?).

Basically, I'm thinking that the ultimate benefit to having somewhat standardized interactions with our web APIs (with more included metadata) is that they can be mined, discovered, and manipulated intelligently by algorithms.

It helps people, sure, but just like using contact microformats helps give the user a phone number they can click to call, it's equally if not more important that it gives a crawler an easy way to say "hey look, a phone number!"

This is the main criticism I have with DHHs comments on hypermedia: they seem to focus too much on the human consumer (and often the programmer) and not enough on the algorithmic potential.

steveklabnik16• 14 years ago

"Hypermedia" was chosen at a term because it gets right to the core of what "REST" (made popular by Rails and other frameworks) misses from REST: the hypermedia constraint.

notJim3• 14 years ago

The HATEOAS wonks are crazy, IMO. Most people who consume your API will have no idea what HATEOAS is and they sure as hell won't care. What will they care about? The URL to hit to get the data they want. They will hardcode that URL in their app, too, so expect any changes to it to require a great deal of fanfare to re-educate your users.

In theory, the URN solution could work, but it's far more complex and requires more maintenance than simply having URLs that can be relied upon. His solution to the problem of breaking bookmarks is also a lot of work for consumers of his API--they have to two codepaths: one for when the URL works like it should, and another for when it changes and they have to use hypermedia to find it again. Again, most consumers of the API aren't going to give a shit. They are going to hardcode URLs and complain when you break them.

Not only that, but consider the implications of an API that requires you to start from the beginning each time for mobile traffic. Each request over a mobile connection takes an eternity due to high latency. You're seriously telling me that you want to force users of your API to make N * depth of resource requests, instead of just going straight there, when each request takes 100 ms or more? Okay, so maybe you can cache it, but that just means the user's first impression of the app is that it takes 500 ms just to do the first thing. Pretty shitty first impression.

The comparison to a browser doesn't hold much water for me either. Users (and search engines) will save your URLs, so if you break them, you will lose traffic. If I try to go to my bookmark and get a 404, you really think I'm going to take the time to find your page again? Maybe, but it's much more likely I'm going to shake my head and close the tab. Of course Google will eventually find your page again, but it will have lost the SEO points it gained, which means you'll have to start rebuilding your SEO from scratch.

The other issue with the browse as hypermedia engine that he leaves out is the human brain. The human brain is excellent at extracting information from text data, so there doesn't need to be any kind of common standard or format to browse a webpage. The brain just figures it out that when you say click: [here], [here] is a link that goes to whatever you were just talking about. That dramatically reduces the burden on webpage authors to conform to any kind format. His theoretical (well, implementations certainly exist) universal REST client only works when every API you want to consume standardizes to presenting the information in the same format.

I drank the HATEOAS Kool-Aid when I first learned about REST too, and then I built a beautiful API that used it. What did the developers ask for? A list of URLs.

The future is almost never $elegantly_designed_complex_system. It's almost always a pile of garbage, with a few humans sitting around sorting through the garbage to find the thing they want.

steveklabnik16• 14 years ago

> Most people who consume your API will have no idea what HATEOAS is and they sure as hell won't care. What will they care about? The URL to hit to get the data they want.

A few years ago, before Rails made "REST" popular, this exact same statement was made. "Nobody is going to want to learn about PUT and DELETE. They just want to do everything over POST."

> They will hardcode that URL in their app, too,

This is an education problem. We're still in the early days of this stuff.

> One for when the URL works like it should, and another for when it changes and they have to use hypermedia to find it again.

This is a tooling problem.

> You're seriously telling me that you want to force users of your API to make N * depth of resource requests, instead of just going straight there, when each request takes 100 ms or more?

One thing that's nice about hypermedia is that you can lead clients to wherever you want them to go. You're free to change things at any time. Have a mobile app? Make less features, and make the request paths shorter. Have a desktop app? Make them longer, add more stuff. It's up to you. Nobody says that you have to complete N steps to do anything unless your server does.

> Users (and search engines) will save your URLs, so if you break them, you will lose traffic.

Right. So what do we do on the web? We 301 redirect until we don't have enough traffic to care anymore. Same thing with your API.

> The future is almost never $elegantly_designed_complex_system. It's almost always a pile of garbage, with a few humans sitting around sorting through the garbage to find the thing they want.

Luckily, Hypermedia is the _opposite_ of Big Design Up Front. It lets you evolve over time. Partial application still yields benefits.

Please see my comment over here: http://news.ycombinator.com/item?id=4949311 GitHub, BalancedPayments(YC W11), and others are finding real benefits today.

notJim3• 14 years ago

You might be right, as you've clearly thought about this more than I have. I have a couple of thoughts though.

> A few years ago, before Rails made "REST" popular, this exact same statement was made. "Nobody is going to want to learn about PUT and DELETE. They just want to do everything over POST."

> This is an education problem. We're still in the early days of this stuff.

The developers I was working with also had a lot of trouble with REST. I had to explain repeatedly what PUT, DELETE, GET and POST were (they weren't really familiar with HTTP at all), and that they weren't doing RPC calls. On the one hand, I feel REST is a simple and elegant concept and its not much to ask someone to understand, but on the other hand, when I build an API, I want it to be easy to use and accessible to developers of wide skillsets and levels of experience. To me, sacrificing some of my desire for elegance is worth it if it reduces the level of “education” required.

> One thing that's nice about hypermedia is that you can lead clients to wherever you want them to go. You're free to change things at any time. Have a mobile app? Make less features, and make the request paths shorter. Have a desktop app? Make them longer, add more stuff. It's up to you. Nobody says that you have to complete N steps to do anything unless your server does.

Either I'm not understanding you, or this seems totally backwards to the way APIs should work. You build an API so that users can do things with their data/your service that you haven't thought about, or cover edge cases that aren't interesting to you. I feel like my API should enable people to build the things they want, not limit and encumber them so that I can have my pretty hypermedia. In other words, I'm not interested in leading clients places. I'm interested in where they lead me.

Milagre6• 14 years ago

OP here.

I'm not strictly advocating HATEOS. Hypermedia and HATEOS are related but not the same concept. I am very strongly advocating hypermedia, as I believe it it greatly reduces the complexity of interacting with an API.

Never having to build a URL by hand is great! Being able to just grab a field from a resource, and go do a GET on its value is very easy to code, even if that process involves finding a link with a certain relation. You speak of complexity, and I'm speaking of convenience.

With that convenience also comes the ability to automatically crawl the data with good solid knowledge of how each resource relates to others, enabling the ability to create massive graphs of information. Like wikipedia has for humans to read, but structured and standard for computers to parse.

> His theoretical (well, implementations certainly exist) universal REST client only works when every API you want to consume standardizes to presenting the information in the same format.

That sounds like a great day to me.

notJim3• 14 years ago

> Never having to build a URL by hand is great! Being able to just grab a field from a resource, and go do a GET on its value is very easy to code, even if that process involves finding a link with a certain relation. You speak of complexity, and I'm speaking of convenience.

This kind of thing sounds great for exploratory development, but undesirable (for the reasons I outlined previously) for an actual production app.

> That sounds like a great day to me.

I hope you're used to waiting :).

technoweenie2• 14 years ago

> I drank the HATEOAS Kool-Aid when I first learned about REST too, and then I built a beautiful API that used it. What did the developers ask for? A list of URLs.

Maybe you should've built them a client too? https://www.braintreepayments.com/braintrust/when-rest-isnt-...

I've heard great things from people updating their API clients to use our new hypermedia stuff. So we're pushing forward with our hypermedia experiments at GitHub. People are still free to optimize for speed (reduce the number of HTTP calls in environments where it really matters) or even development speed (hardcoding a few API calls instead of dropping in a full client library) if they like.

thibauts2• 14 years ago

> Hint: Your browser is an API client

Apart from parsing, I am the API client in this instance. I'm the one making decisions from understanding what services are about.

The universal API client seems to be a kind of silver-bullet-IA in many people's mind. I feel it's both impossible and not necessary to accomplish that. The complexity we have to handle to use APIs is already minimal.

What makes hypermedia interesting IMHO is (minor point) easier intra-linking within a webservice, and (major point) the ability within a service to point to resources from another one.

This will enable the distributed web we are all waiting for.

Milagre6• 14 years ago

> Apart from parsing, I am the API client in this instance. I'm the one making decisions from understanding what services are about.

Your browser parses, yes, but then it goes and fetches images, css, javascript, iframes, etc. Following the analogy, these are 'api calls'. It knows to make these calls because the HTML defined the links, and for the HTML to be rendered the contents of the links are required.

I maintain the browser is the client - it responds to your actions, but it does so with a lot more activity than just parsing.

>What makes hypermedia interesting IMHO is (minor point) easier intra-linking within a webservice, and (major point) the ability within a service to point to resources from another one. > >This will enable the distributed web we are all waiting for.

Couldn't agree more. But without some standards, those links to other services may not be as useful as they could be.

thibauts2• 14 years ago

Sure css, js etc are fetched, but this is hard-coded behaviour, there is nothing dynamic or discovered about it. Or I fail to understand it.

This may well be the problem then, as we can see here and elsewhere. Many people like me don't understand. The claims are too vague. We need real-world exemples of how this can be applied with clear benefits. Mere transposition of the hypertext qualities to a medium intended for machine consumption will not do it for most of us.

paul_odin• 14 years ago

Is there a list of Hypermedia-esque APIs in use today (apart from sitemap)? I can find lots of explanations of the concept but it's hard to find examples.

The closest I can think of is the way some APIs handle pagination (GitHub and Recurly for example) by using Link: rel=next/prev/start headers rather than through GET parameters, which in principle would allow a generic REST client to iterate through results on its own.

steveklabnik16• 14 years ago

GitHub does it for more than pagination: http://developer.github.com/v3/pulls/

Please see my comment here: http://news.ycombinator.com/item?id=4949311

Also, Balanced (YC W11): https://www.balancedpayments.com/docs/overview#storing-the-u...

technoweenie2• 14 years ago

$ curl https://api.github.com

We're experimenting with more :)

gcampbell• 14 years ago

What are the advantages/disadvantages of serializing links in the body JSON (i.e. HAL) vs. using the Link header?

mjallday• 14 years ago

Like steveklabnik mentioned, take a look at the Balanced API, here's an example you can run yourself using a test marketplace:

    curl https://api.balancedpayments.com/v1/marketplaces/TEST-MP6IEymJ6ynwnSoqJQnUTacN -u 7b7a51ccb10c11e19c0a026ba7e239a9:

To keep this thread free of tons of code samples I've included the result @ https://gist.github.com/4350290

You can see in our python client how we then parse - https://github.com/balanced/balanced-python/blob/master/bala... - the result of this which then consumes these URIs and turns them into dynamic resources attached to whatever object you're looking at.

This means you can do this:

    debit = balanced.Debit.find(uri)
    debit.account.cards.all()  # get a list of all cards associated with the account that created this debit

And the client does not have to know anything about the fact that debits have an account property, or that accounts have a cards resource underneath them.

mahmoudimus2• 14 years ago

I think we should probably write about https://github.com/bninja/wac and how our Balanced Python client influences it. Gone are the days when your internal services need to sync on some kind of schema.

We make changes to our internal services all the time and HATEOAS allows us to ensure our services are properly functioning without breaking contractual API obligations.

I can see why most developers resist the urge to try hypermedia APIs and it's honestly because the tooling to build services and clients just aren't there.

This is why I'm a big fan of the https://github.com/rails-api/rails-api project. It's essentially accepting this deficiency, has some brilliant minds behind it and as a consequence, I'm sure the tooling to be built around it will be amazing; it's definitely one to follow.

Some more resources:

* https://github.com/rails-api/rails-api

* http://django-rest-framework.org/

mjs6• 14 years ago

A few thoughts:

* Mike Kelly (the author of the HAL spec) is not aiming for a universal API client http://news.ycombinator.com/item?id=4949357 (and also seems to be under the impression that no-one else is either).

* A universal client seems impossible because APIs will typically need to use service-specific link relations for all "interesting" relations (e.g. product, author). You might be able to share generic link relations (like those in IANA's table: http://www.iana.org/assignments/link-relations/link-relation...), but sharing only these doesn't seem to be worth the effort.

* Hypermedia-powered experimentation is kinda cool, but you're still going to need something to drive it (see e.g. Kelly's HAL browser http://haltalk.herokuapp.com/explorer/hal_browser.html#/), at which point you might as well write a full-fledged API dashboard.

steveklabnik16• 14 years ago

We already have a 'universal API client' called the browser. Therefore, implementing a NEW one isn't very interesting.

mjs6• 14 years ago

Sorry, I don't get what you're responding to. Browsers don't understand hypermedia APIs (or even HAL)--they only grok HTML.

AffableSpatula3• 14 years ago

Why would you want to write a dashboard from scratch every time? Foxycart used HAL for their API, and when they'd finished they pointed the browser at it and instantly had a browsable API.. for free.

mjs6• 14 years ago

Well, companies seem to like to brand their dashboards, emphasise different aspects and so on. But still, I agree it would be really cool if you could use the same API browser everywhere, to avoid crappy dashboards, and that HAL would could enable that.

How can I point the HAL browser at foxycart's API? Or even see Foxycart's browser? I can only find standard hand-written documentation.

pdonis2• 14 years ago

I'm confused about something. The author's "simple and easy solution" to the problem of people saving API URLs is to have a "bookmark relation" link; but all that means is assigning a unique, permanent name to the resource so people can find it again. Why wouldn't you just design your API so the API URL is that unique, permanent name? The "bookmark" link "identifier" ends up being a URL anyway; why not make it the API URL to start with?

Milagre6• 14 years ago

This is a fair point!

Mostly, I think that would produce an ugly API.

The reason I've been leaning towards that solution is that it assigns the responsibility of locating any other resource, in one request, with full backwards compatibility, to exactly one location, freeing other locations to change in a more fluid and graceful way without the cruft.

Edit: Oh and because other URLs can have the context of their parent paths deliver some value or information. But then again URLs should be opaque =). Definitely worth thinking more about!

pdonis2• 14 years ago

Mostly, I think that would produce an ugly API.

I don't see why the API would have to be ugly. The examples of "bookmark" link URLs given in the article are perhaps ugly (using query strings, etc.), but there's no reason why such URLs couldn't be "nice" ones. The article talks about things like "what if I want a URL structure like /User/projects/5 instead of /projects/5?", but that's a red herring: you could just as easily say "what if I want a bookmark query string that looks like ?user=User&project=5 instead of ?project=5?".

The bottom line is that as soon as you have a public API, you have identifiers that can't be easily changed because people need to be able to bookmark them; and those identifiers can't be too ugly because they are visible to users. So they should work OK as API URLs.

sopooneo2• 14 years ago

I am having deja-vu. When I was just starting, everyone screamed about pure "semantic" markup and how all css class names should describe what was contained in a node, and have nothing to do with how it was presented. People bent over backwards to comply. I couldn't make it work, but assumed I was just far behind the curve. A few years later and that roar is gone. Bootstrap.css for everything, with class names that describe structure. I can't say I'm hurt by this as it is what I had resorted to all along. But the absolute 180 without even a hint of the old battle cry in the air is startling.

I feel like that is again happening here. I can not image how fully compliant REST api's would work in the real world. And when I ask for an example? Twitter, check the Twitter API! But no, that breaks the rules all over the place in the name of practicality. Oh yeah? You want a real example? How about the whole web? The whole web is your example!. That example is so far outside of instructive or helpful I don't even know where to start. I stand agape, unable to even speak. The way clients access my blog system should be based on the whole web? Even though it itself is on the web?

I'm done chasing prophets. In five years I do not believe this REST mania will exist anymore, and not because it is so ubiquitous as to go unstated. It will have died or changed drastically in the name of actually working. Bootstrap.rest. Either I am too dumb, or the movement is crazy, or both. But either way, I'm done trying to make things in a way I don't understand. The great and holy can thump their Fielding Bibles with faith that they will be taken home soon. Me, I've given up on salvation, and find the road to hell a much less exhausting one.

steveklabnik16• 14 years ago

> I can not image how fully compliant REST api's would work in the real world.

Have you ever used GitHub's API?

    $ curl https://api.github.com/

sopooneo2• 14 years ago

I have not. But I will play with it. Thank you for the suggestion.

mjs6• 14 years ago

Oh! If that's the sort of thing we're talking about, then I'm all for it. The proponents of Hypermedia APIs have been saying a lot of vague things about how it will enable automated clients, and I assumed that the clients they had in mind would be able to do something useful with arbitrary APIs.

But it's clear from GitHub's API that it will do nothing of the sort: those link relations (and even concepts) are very much GitHub specific, and no machine will ever be able to figure out what to do with the them, what methods or media types the endpoints support, and so on.

(I'm not sure that it adds very much over a text document that that gives the relations and how to generate endpoints, but at least it's cheap to produce, and not actively harmful.)

laughinghan• 14 years ago

When I was just starting, everyone screamed about pure "semantic" markup and how all css class names should describe what was contained in a node, and have nothing to do with how it was presented. [...] A few years later and that roar is gone. Bootstrap.css for everything, with class names that describe structure. [...] the absolute 180 without even a hint of the old battle cry in the air is startling.

I don't think a single advocate of semantic CSS class names has changed their mind and now says there are no problems with using Bootstrap presentational CSS class names in your HTML. Just yesterday, on the front page of Hacker News, there was an article about the pain caused by overuse of Boostrap's unsemantic CSS class names: http://blog.pamelafox.org/2012/12/a-tale-of-two-bootstraps-l...

I don't know why you ever thought people were "screaming" about it, but if you're hearing about it less than about Bootstrap now, it's definitely from different people.

There are objective benefits for maintainability by designing your HTTP APIs to be RESTful, just as there are objective benefits for maintainability by choosing your CSS class names to be semantic, and they have been espoused by the creators of HTTP and CSS since their creation. It is neither "mania" nor "crazy".

Your child-like conversation with an unhelpful strawman notwithstanding, it is true that in the real world, software architecture always has to balance long-term maintainability against short-term ease of implementation. That doesn't change the fact that semantic CSS class names and RESTful APIs improve maintainability, always have, always will, people always have said they do, and people always will say they do.

jsankey3• 14 years ago

“Proper” use of a REST or hypermedia API generally requires you only ever initiate communications with the API from the root (https://my.api.example/ versus https://my.api.example/some/other/url). From there, every response from the API will then contains links to other resources, often with a relation to explain how each resource relates to the next. In this way, you never guess a URL - you are always given it.

Speaking from practical experience using an API like this: it can result in a lot of extra round trips to get to your final destination. The indirection is nice purity-wise, but in practice do you really want to make 3-4 times the round trips to the server for every API interaction? Especially when you might be on a mobile network? All for the sake of flexibility you may never need?

I'd say supporting the discoverability for learning, and consistency etc is nice. But in practice concerns like this (and the DHH example, and others already in comments here) will crop up and clients will start building their own direct URLs. So the idea that you'll be able to change the URL structure without affecting clients is a pipe dream.

steveklabnik16• 14 years ago

> it can result in a lot of extra round trips to get to your final destination.

You can usually get around this. Many people make their resources have too much hierarchy; there's nothing inherently unRESTful about having a flat one.

You should _not_ be having 3-4 round trips for every API interaction. Each request is an interaction. Hypermedia APIs expose a workflow, not a data model.

Milagre6• 14 years ago

> Hypermedia APIs expose a workflow, not a data model.

I would say this is true of REST APIs. Not necessarily hypermedia APIs. An API that directly exposed a graph database as an HTTP-based API using hypertext/links within resources would still need to be classified as a hypermedia API, but would fail to adhere to the HATEOAS principle.

Edit: More specifically - a graph database that exposed itself with links for every edge and a resource for every node.

jsankey3• 14 years ago

First, are there then multiple roots? If so, you've already sacrificed one level of indirection, moving knowledge to the clients. If not, you're adding at least one request.

Secondly, when your data is hierarchical, it's nice for other reasons to reflect this in your URI structure. It's intuitive (and thus discoverable in its own way) and can make for human-friendly URLs (especially when your datatypes have natural identifiers).

robconery• 14 years ago

I dunno. It seems like the future is arguing about Hypermedia. If the idea doesn't take hold ... is that our fault as developers? It reminds me of the OpenID hype - engineers solving a problem that doesn't exist or, worse yet, solving a future problem that has yet to materialize.

If the answer to this is "educate people" then that's the wrong answer. Or maybe it's the right answer if you're an IT type that loathes their users. I know I might sound crusty about all of this but how many years are we going to chase an idea that only a few people seem to understand?

Can we just admit it, right here, right now: all of this is academic. We should be focused on extracting patterns and practices that provide value and cycling on that.

The rest is noise.

The Future is Hypermedia APIs

Related Stories