Don't design your own REST back end

https://plus.google.com/u/0/+ÓlafurArason/posts/fCmaosPgmwN

by rograndom • 12 years ago

135 75 12 years ago

plus.google.com

bitcrusher• 12 years ago

Still digesting this, but it doesn't pass the smell test. There are too many assumptions that are just plain... wrong.

For example:

There seems to be an implicit assumption that all of the REST endpoints are just maps to CRUD operations. Perhaps for the most barebones, basic API this is true, but service APIs are more than just CRUD ops. They're abstracted interfaces to a complex series of tasks (for the sake of argument, we'll call them 'orchestrations' ). Just because you're doing a "POST" to /user/foo doesn't mean that the service is only doing a simple insert. It might be looking up similar users, sending out notifications, validating data, de-duping, etc.

The more I think about this the less sense it makes. Perhaps I'm missing something.

wisty• 12 years ago

You can do this by adding all the logic to the database. The author is a CouchDB fan. There's pros and cons to this approach. It's not a new idea - see http://stackoverflow.com/questions/1473624/business-logic-in...

staticvar• 12 years ago

His post does focus mostly on CRUD. Personally I usually proxy CouchDB to a path in my REST API for handling CRUD and then proxy in the rest of my services at other paths. In the case of doing a write and then some other operation, the CouchDB folks usually write a process that listens to the changes API which then triggers the desired other operations.

platz• 12 years ago

Perhaps the client would poll a response if everything was truly async?

ollysb2• 12 years ago

Couchdb lets you listen to changes[1], if you want to do anything in addition to the basic CRUD you can do it there. This does limit you to performing asynchronous operations but most of the time that's good enough. You do still end up with a scaling issue but because the users aren't hitting that part of your system directly it's easier to spread the load over time.

[1] http://guide.couchdb.org/draft/notifications.html

batbomb• 12 years ago

I'm not sure if the author is even aware of the 202 status code, which seems to solve this and other asynchronous issues.

protonfish3• 12 years ago

It's clear the author doesn't understand REST - he thinks that it is a dumb wrapper over a relational database. It's not surprising that straw man burns so well.

BenoitP• 12 years ago

I have yet to see a crystal clear definition of REST. Where would be the best one you have found?

analyst74• 12 years ago

If you keep the data on client side for longer than trivial amount of time, wouldn't you have to deal with conflict resolution? This could be a big problem for many use cases.

mirsadm• 12 years ago

Syncing and resolving conflicts is one of those things that everyone thinks is initially easy then realizes how difficult it is.

MetaCosm2• 12 years ago

Then they punt and centralize.

collyw• 12 years ago

All I saw was the problems when he talked about that. The data will be a mess (if his database is in any way similar to the one I work on).

vidarh2• 12 years ago

Yes, it is incredibly hard. But on the other hand, the problem of clients losing connection does not go away just because syncing is hard. If you're targeting mobile users, you have the choice between your app being unavailable a lot of the time to a lot of users, or handling syncing.

tantalor• 12 years ago

You hopefully care about how the URL schema looks like

That's not REST, that's something else. Bit of a strawman.

So then you don't have beautiful URLs right.

You never needed them.

phamilton• 12 years ago

This is a key point. HTTP states that URLs are opaque. Understanding that is important. Trying to leverage URL conventions is just asking for trouble. SEO complicates this a bit for crawled sites, but for web services it is a very different situation.

rhizome314• 12 years ago

Beautiful URLs are convenient though. Not every API call is going to follow a succession of links from the main entry point. Do you really design opaque URLs on purpose? Does it work well for your clients? This is a genuine question, I'm quite seduced by the hypermedia story, but still wondering how it works in practice. It seems that for the most part, we don't really know yet.

notastartup• 12 years ago

so how does using couchdb solve some of the pain points? by providing a user authenticated REST api out of the box? what if you need to write some server side business logic?

I read the article, it definitely sounds exactly like what I've been through. What I didn't get from the article is, what to do next? What is the right step to take here?

AdrianRossouw2• 12 years ago

I love CouchDB so much.

90% of the time I end up just proxying connections to couch and doing sanitizing/filtering on the data on the fly.

The views are bit annoying, but I just plug in elasticsearch once it becomes even moderately complex. It just automatically indexes the data by listening to the _changes feed.

emsy3• 12 years ago

What if someday, you decide that you want to change the database because CouchDB was bought by Oracle? You'll have to clone Couch's REST API or you may break your clients.

MetaCosm2• 12 years ago

Yep. There is a reason people abstract things. Sometimes people get a new toy and just go a bit overboard, want that one tool to solve all their problems.

Don't get me wrong, CouchDB is awesome, but it isn't the answer to everything.

janl2• 12 years ago

CouchDB can’t be bought by Oracle or anyone, since it is run as an independent Open Source project and under an organisation (the Apache Software Foundation) that allows and supports commercial entities contributing, but single vendors can never control the project in a way that a single vendor product could.

You are still vulnerable to CouchDB as a project folding, but since it is all Open Source and open discussions, that one is easy to find early on.

And finally, you could just use one of the other implementations of the CouchDB API available :)

emsy3• 12 years ago

Of course the "Oracle buys CouchDB" scenario wasn't meant serious :). But the point still stands. There may be much more trivial cases which can drive an user to switch from CouchDB and CouchDB derivatives to another DB. E.g. the user figured that a relational DB suits his use case better after all.

Lazare• 12 years ago

As others have noted, as an Apache project, the CouchDB project can't be "purchased by Oracle". Obviously you meant that somewhat metaphorically (and as a reference to MySQL), but it's worth noting:

MySQL was the product of a private company. It was always possible for it to be purchased, and that eventually happened. The same is true for many modern databases, including most of the major NoSQL databases. CouchDB cannot be. Yes, there's risk of the project could die in a variety of ways, but there is an entire class of risks that CouchDB is not subject to, when most of its competitors are. If you really want to avoid having your DB purhcased and killed by Oracle, you'd actually prefer CouchDB over, eg, MongoDB.

Beyond that, you say: "You'll have to clone Couch's REST API". That's actually the other half of what makes CouchDB a safer bet: The HTTP protocol is itself open source, well documented, and has already been cloned repeatedly. It's used by quite a few different projects in different ways, including PouchDB, CouchDB, TouchDB, Hoodie.io, BigCouch, and CouchBase Lite.

If you're curious about the topic, I recommend this blog post: http://caolanmcmahon.com/posts/couchdb_is_not_a_database/

emsy3• 12 years ago

Thanks for the link. CouchDB really sounds more like a platform/framework. As for the points you mentioned, refer to my answer to janl.

mplewis• 12 years ago

What if you want to plug in another DB for testing? When I use Peewee ORM with Python, I use Postgres in production and SQLite in-memory for testing. Is there a good way to do this when you put logic in the DB?

janl2• 12 years ago

it’s not quite there yet, but pouchdb.com with pouchdb-server (npm, gh) is the “SQLite to Postgres” in CouchDB terms :)

sgonyea• 12 years ago

Can't read easily on an ipad. Back button pressed.

contrahax• 12 years ago

This article is really making a lot of broad assumptions about what people are making APIs for and how people write code.

"Now you hit an API that has to do some async behaviour and now your screwed. There are some solutions for that but they make the code really complex, even Node.js."

What does that even mean?

daGrevis• 12 years ago

I don't know what he meant, but we have a situation where our REST API needs to reply with “I see you want to get X. Come back and ask for X again when it's ready.“. In our backend, we just add the task in message queue.

k__• 12 years ago

I never had a problem with this.

The protocol I wrote just sent a job-ID to the client and the client started polling for the result.

Since AJAX is already asynchronous I could write a JavaScript library for the browser, which made the whole thing transparent for the client.

The only problem I ran into were multiple long running jobs, which caused parallel polling. But since the communication between server and client was hidden behind the client-lib, I could change the protocol later, to merge multiple polling requests in one etc.

thibauts• 12 years ago

HTTP 202 Accepted [1] does the job.

[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10...

spacemanmatt5• 12 years ago

It means he had a problem and so will you. I'll stop short of calling it a profoundly disturbing assumption, but it pervades the entire article.

room271• 12 years ago

The main benefit of having an actual API layer is decoupling - between your front end clients and you back end service(s). This article seems to forget this major point.

Decoupling has loads of benefits - for example, it gives separate development cycles, and in this case provides a point of scalability too - not all REST dbs cache brilliantly or aggressively enough for all use cases.

The point about using delayed syncing between client and server is separate. I can see how it can help in certain use cases (Google docs doesn't auto-save every single letter you type in for example I suspect, but probably bulks them up into a save every x seconds) but at other times this kind of strategy sounds like it will add unnecessary complication.

vidarh2• 12 years ago

The syncing adds unnecessary complication for desktop apps. For apps that needs to be accessible via mobile devices, it means the difference between whether or not your app is usable at all for a lot people. About 30%-50% of my commute is "dead zones" where I'm lucky to get GPRS speeds, and often have no signal at all... This is in one of the busiest commuting corridors in London....

Getting syncing right is tricky, though, because you do need to think through your possible conflict resolution issues in case the user moves from device to device, potentially while one or both devices lack a connection.

orthecreedence• 12 years ago

I write my own REST APIs and I don't really run into problems like this. Removing REST isn't really the answer to the problem "what happens if clients disconnect?" You can write a layer in the client to deal with this gracefully and still use REST. I don't really get how the two concepts relate.

For anyone interested, I recently wrote a library called Hustle (https://github.com/orthecreedence/hustle) which implements a beanstalk-like queuing system on top of indexeddb, allowing you to queue up writes to your local data to be synced remotely.

spacemanmatt5• 12 years ago

I also don't understand why any of this (OP's problems with REST) is the fault of REST or a database. To me this is 100% architecture woes. Lets of people use REST successfully.

rhizome314• 12 years ago

Despite the slightly ranty tone, this was an interesting read. The sync-first approach presented here may certainly have value in some contexts. Also interesting to hear of someone implementing backends with just CouchDB. Again one size does not fit all but it's always interesting to read another developer articulating his thoughts on the way he works.

neverminder2• 12 years ago

I don't know about everyone else, but the problems described in the article I encounter every day and so far I managed to beat it with solid Typesafe stack. Main components of my stack - Scala/Play/Akka/Slick/PostgreSQL. By changing few configs I can scale from 1 server to 10 just like that thanks to Akka.

egeozcan2• 12 years ago

Do you know of some tutorials or even better some examples for getting started with this stack? I know of this JHipster [1], but it's Java and doesn't use Akka AFAIK.

[1] http://jhipster.github.io/presentation/

neverminder2• 12 years ago

On http://typesafe.com you'll find plenty examples using the whole stack. When I was learning Scala I found http://twitter.github.io/scala_school/ best source to break into it. http://playframework.com is a good source of tutorials and docs for Play. http://akka.io is the home of Akka and http://slick.typesafe.com is Slick's website.

adamors3• 12 years ago

Explicitly linking database operations to API calls is idiotic for multiple reasons. First it creates tight coupling, second if you want to do multiple operations at once you have to make multiple calls to the API. That doesn't scale at all.

Everyone should be writing their REST back end and the API should be as specialized as possible. One HTTP call should give you back everything you need for a page/view w/e.

drawkbox2• 12 years ago

I agree the article is missing the abstraction all public apis need to offer non changing services.

Behind that public front everything can change and routes can be added, if things change majorly you version it. But never is it a good idea to map the db operations directly to the rest urls, so much more has to happen in production.

The public api layer is sometimes mapped more manual as auto generation rest apis rarely aren't leaky abstractions. ORMs, data layers, authentication etc behind the scenes may be generated or flexible to change.

Public apis might even have public objects that aren't objects from the database in full or at all i.e. meta/tracking fields, private info.

Public apis offer multiple levels of service: direct data/json, mvvm views, tools, syncing, per-view state (multiple objects), authentication etc. The term REST has colluded it a bit but ultimately REST is representational state transfer, it doesn't have to be web to db without an application layer.

An API is a public definition that should be as concrete as possible for the service use cases and public apis provide a flexible abstraction to expose a service to other consumers. So I will keep writing my public apis, I'll generate boilerplate behind that but I won't be locked to any implementation other than a uniquely designed api for my service, features needed and use case.

adamors3• 12 years ago

it doesn't have to be web to db without an application layer

I don't even think it's possible to do that if you have a bit more complicated application. Because every API that does more than the very basic CRUD needs something else besides interfacing with the database.

spacemanmatt5• 12 years ago

That mega-JOIN that brings back the entire client context is more awesome for db throughput, than plethoras of single-object calls. THIS IS WHY you build an API, rather than exposing CRUD. CRUD is not an API.

icebraining3• 12 years ago

A Remote Facade, as Martin Fowler would put it: http://martinfowler.com/eaaCatalog/remoteFacade.html

junto2• 12 years ago

For those of you using .NET I highly recommend ServiceStack:

https://github.com/ServiceStack/ServiceStack

https://servicestack.net/features

Doesn't use ASP.NET. Runs on Mono.

Don't bother with WebAPI.

manuletroll• 12 years ago

ServiceStack is great but the recent licensing change is irritating to say the least.

junto2• 12 years ago

You can still use the v3 version for free.

I agree that for a hobbyist the new license fees are too high. For a company those costs are a no-brainer though.

What Demis has done with ServiceStack (and now having made it his full-time job after leaving Stack Exchange) deserves remuneration in my opinion.

ermintrude• 12 years ago

There are problems with your APIs losing data because of errors or downtime. A lot of the Internet is going over unreliable wireless technologies. So your beautiful REST calls are now riddled with exception handling, because there are so many ways of things going wrong.

Couldn't this be easily solved by: a) putting the server behind nginx to terminate slow connections after several seconds, and b) adding retry logic to the client?

This isn't a rhetorical question - would this approach actually work if a client was part way through sending data when it lost network connectivity, or am I missing something?

skrebbel• 12 years ago

There is a core of truth to this article in that we're all reinventing the wheel trying to sync a client-side model with a server-side model (assuming your (web) app has client-side state at all, of course).

Indeed, CouchDB/PouchDB already solved this problem, and redoing it can be considered waste. I think the author cut a bit too many corners for the argument to stick, but in principle it's entirely true.

The "sad" thing is that if I'd use Couch for this, I'd get more than what I asked for: I also get a database, document-oriented, with a big map-reduce component, difficult to add custom queries, difficult to extract management information from, limited admin tooling. It might fit certain problems well, but it also fits many problems badly.

I'd love for a Couch/Pouch kind of tool for replication/sync that somehow allows me to plug my own datastore on the backend and my own model structure on the frontend. I'm not sure what that would look like - there's a good reason data storage and sync are so tied together in Couch. But still, "you shouldn't design your own API and the only way not to is to use this particular database here" just doesn't feel like we're done with this debate.

AdrianRossouw2• 12 years ago

I just use elasticsearch to index couch, but that's also stricly on the server side.

there's rarely enough data in pouch for me to have to index that way though.

mikegioia• 12 years ago

Aren't you severely limited in storage with pouch? I think best case with browsers is like 50Mb with the average being about 10Mb. I can't envision a situation where you could ever index pouch.

ollysb2• 12 years ago

I've got a low traffic project I'm working on at the moment that uses pouchdb as an offline data store for a tablet app. I'm currently playing with implementing the couchdb protocol in rails. It seemed like a crazy idea to start with but with a little redis sprinkled in to handle information that wouldn't normally in active record models it seems to coming together better than expected.

e12e• 12 years ago

   > A lot of the Internet is going over unreliable wireless 
   > technologies. So your beautiful REST calls are now riddled 
   > with exception handling, because there are so many ways of
   > things going wrong.

Let me get this straight, you want do to representational state transfer (REST[1]), that is, keep all long-lived state on the server, and all (single) state changes confined to a single request - and you're worried about your clients intermittent internet connection?

You might have a problem with your architecture, but your architecture doesn't appear to be a REST architecture.

[1] https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arc...

cardamomo• 12 years ago

nobackend.org

spacemanmatt5• 12 years ago

This is a meandering tour of one developer's trials, not a solid discussion of REST at all.

I withdrew after "Whether that it's tried and tested of MySQL or shiny and new of MongoDB. This choice is probably going to affect how you scale." It was not worth continuing after that.

spacemanmatt5• 12 years ago

If you have already written a competent relational schema, written a SQL API on top of that, and mapped that API to a RESTful service successfully, this article is not for you.

rubiquity• 12 years ago

I'm a huge fan of CouchDB, but I think that CouchDB is the answer to his problem just as much as using some other database like Postgres is the problem to begin with. The problem with "REST" (oh yeah, baby, I'm getting the air quotes out) APIs to begin with is most of the time we heavily couple them to the fact that they will be consumed by a browser client. Design your REST APIs to stand by themselves and design them to be consumed by any client. There is no doubt that web browsers have special needs, address those special needs with a (hopefully) thin layer between your REST API and the browser.

findjashua• 12 years ago

Does couchdb have something that provides fine-grained control over the db models, like flask-restless does(https://flask-restless.readthedocs.org/en/latest/customizing...)? I didn't find anything in their documentation or on google.

Don't design your own REST back end

Related Stories