Why We Chose API First Development at Our Startup

http://blog.pop.co/post/67465239611/why-we-chose-api-first-development

by jqueryin • 13 years ago

97 73 13 years ago

blog.pop.co

girvo• 13 years ago

In the past 6 months, at the urging of my friend Dave (sup!), who suggested this style of development, it's all I've done, even for small personal projects. I sort of take it further, eschewing frameworks somewhat, relying on libraries around actual "Business Logic" to deliver stuff as needed to outside clients.

It's been an eye opener. Sometimes it requires a little more upfront thought, but I end up with a much more maintainable project at the end of it. TDD is your new best friend if you go down this road, and the best part is: it's easier to do test driven development when things are nicely separated like this!

It's basically SRP writ large, and something tells me it's another one of those "ideas" that the grey-beards learned a long time ago that we are only re-learning now. Although REST does kick the crap out of SOAP...

jqueryin8• 13 years ago

I agree on all fronts here. I should note that we support all modern RESTful request methods for CRUD operations: GET, POST, PUT, and DELETE. Internally it simplifies our routing as each corresponds very well to the type of action we're performing.

macspoofing6• 13 years ago

Two problems:

1) Search engine indexing is broken with client-side rendering

2) Page takes longer to load, and so you will want some server-side rendering to take place (even if to seed your initial models, so they don't have to call back immediately).

robzienert• 13 years ago

You posted the same two problems in two different areas of the thread. As mentioned here (https://news.ycombinator.com/item?id=6763780), your first assertion is incorrect. Page loads do take longer, but you also win being able to support multiple frontends (consumer facing web, backend management web, mobile apps, desktop apps, other services, etc) much easier than if you had to maintain server-side rendering as well as an API. After the first page load, however, you're transferring far less data and will be loading pages much quicker. Sure, if you have a mostly static site, it's not worthwile, but if your app is highly personalized, this is a great direction to go.

reverius42• 13 years ago

Nobody said doing an API-based design requires you to (purely) client-side render. Lots of people are consuming APIs and rendering (initially) on the server, then consuming the same or similar APIs to do updates on the client. Best of both worlds -- this approach has neither downside you mention (even though the downsides are not as bad as you suggest even if you do pure client-side rendering).

napoleond3• 13 years ago

1) Search engine indexing is broken with client-side rendering

There is no rule stating that an API must be consumed client-side, and obviously a great many can't be (depending on how auth is set up, etc). For projects where SEO matters but where an API will also be useful (or if you've just gotten into the habit of building user-facing bits around an API) you can render everything server-side, or first loads server-side and subsequent loads client-side. Airbnb's Rendr (https://github.com/airbnb/rendr) is neat for this, although there are a million other ways to do it.

If you combine this approach with something like CouchDB, your API can basically just be your DB interface and you can focus on building the user-facing bits.

EDIT: Sorry, everyone wrote the same thing at the same time :)

iLoch• 13 years ago

1.) I'm using Prerender.io which targets bots and renders the page for them using PhantomJS - it's really neat! The only drawback is the response time will increase a bit for the bots, which may affect SEO.

ds_4• 13 years ago

I recently launched a service which addresses (1) [http://crawlspa.com]. An example page usually rendered with javascript: http://crawlspa.com/?_escaped_fragment_=dashboard.

Rabidgremlin• 13 years ago

How do you handle authentication and sessions? Just curious on the approach you took.

jqueryin8• 13 years ago

We use CORS, which unfortunately is more of a headache than it's worth and lacks IE <10 support. Our backend proxies and handles session management via a secure, hashed cookie helper library which maintains expirys among other things. We take advantage of shared cookies on a global cookie domain . There's some ugly handling for IE10 regarding CORS and how it handles cookies which I won't get into. Authentication state is managed via the session, which we actually query our backend for on page requests. It's a bit of overhead to have to always be querying for the user/auth object to check login status, but it's easily cacheable (and bustable).

normalfaults2• 13 years ago

Are there some go to standards for API design that are followed?

iaskwhy• 13 years ago

I use the following sources as standards:

- Vinay Sahni's Best Practices for Designing a Pragmatic RESTful API: http://www.vinaysahni.com/best-practices-for-a-pragmatic-res...

- HN discussion on Vinay's post: https://news.ycombinator.com/item?id=5819231

- Twitter's Error Codes & Responses: https://dev.twitter.com/docs/error-codes-responses

normalfaults2• 13 years ago

Thanks these are great!

johns6• 13 years ago

If I can throw my own in: http://john-sheehan.com/blog/dont-build-the-best-rest-api-bu...

icedog• 13 years ago

Apigee's free ebook is pretty nice. http://pages.apigee.com/web-api-design-ebook.html

Kerrick• 13 years ago

http://jsonapi.org/ is an emerging standard.

http://dev.billysbilling.com/blog/How-to-make-your-API-bette... is a great blog post on API design.

steveklabnik• 13 years ago

Coauthor of JSON API here, expect to hear lots of neat stuff soon. ;)

johns6• 13 years ago

A new site was just launched to help with this: apicommons.org

My general advice is if it's internal, build what feels right to your team. If it's external, research the APIs you like or are popular (Twilio, Stripe, etc.) and don't be clever.

grosskur• 13 years ago

"Designing REST + JSON APIs" by Les Hazlewood is a good starting point:

http://www.stormpath.com/blog/designing-rest-json-apis

jwingy• 13 years ago

You missed another benefit towards API first development: it makes it much easier to get a head start on mobile development as you can reuse many of your endpoints for common operations.

jqueryin8• 13 years ago

You bring up a great point. I didn't cover mobile development specifically as we hadn't had a use case for a mobile app yet as the site is full responsive. I opted for more of a blanketed statement in the section on "Developer Liberation" as to how the API covers all client bases equally well.

Nilzor• 13 years ago

This merely sounds like they're adhering to the buzzword of the year 2008 - SOA. I'm I wrong? (Not that there's anything wrong in that, just nothing new)

johns6• 13 years ago

It's a lighter weight form of the same idea. SOA had a lot of great benefits for its time. This revision of those ideas also has a lot of merit.

dreamfactory4• 13 years ago

From what I've seen in enterprise java, SOA tends to just refer to internal services at a very low level of granularity - basically seems to be OOP with classes described as services. (This may well be a mangling of a good concept.)

RobinUS2• 13 years ago

Great initiative, I personally prefer to refer to this as "Eat your own sh*t".

vcherubini• 13 years ago

I did this when I first started writing REST APIs. It felt natural and the "right" way to do it. However, it didn't scale well.

And I'm not talking about website scalability with traffic.

It eventually made development more difficult than it was worth. For example, doing things like enabling/disabling an object is a whole API call rather than a simple UPDATE statement. Either you have to set up an API endpoint to handle toggling the status of that object, or you have to send a whole (sparse) request just to flip a flag. Stuff like that can be tedious.

I love REST APIs and I loved what they have done for the web, but I've changed my mind over the last few months that this separation of design is always necessary.

mason55• 13 years ago

The other mistake that I've seen people make is to try to build their application completely on top of a public API. Sometimes you want to do things in your application that, either for security or scaling reasons, you don't want someone consuming the public API to be able to do (bulk operations, administrative functions, cross-account updates, etc).

You either need private API functions or you need to allow actions that aren't triggered by an API call.

jqueryin8• 13 years ago

I didn't discuss this in the article, but I should note we do indeed have a fully private API that functions independent of the other and is utilized by our administrative management tools.

johns6• 13 years ago

Our public API is a front end for our other services with auth added on and better guarantees against breaking. We don't depend on it directly for anything public facing and I wouldn't suggest anyone completely rely the same API they offer partners/openly to avoid getting into a tough spot with managing changes.

jqueryin8• 13 years ago

You're 100% correct that it's not the absolute quickest or easiest method for implementing what would be deemed a quick change. It is, however, a great way to reduce risk. It's my personal opinion that something even as simple as an UPDATE statement comes at a great cost on your production environment. It should be tested all the way up the chain of environments, backups should be issued on your database, etc, etc. It's really not easy unless you're whole-heartedly okay with the potential ramifications of "cowboy coding" so to speak. I myself have been bitten by performing live query UPDATEs/DELETEs very early on in my career, and API first development is designed to protect people like myself from bad things happening.

debaserab2• 13 years ago

Yes, separating this logic is tedious, but it's core to the concept and it scales just fine.

If every time you want to enable/disable an object you simply write an SQL update query, if you ever want to add any other requirement when an enable/disable occurs (example: you want to record the reason why an object is enabled/disabled) you've now cornered yourself into a situation where you need to identify all the code where you've written the SQL update query.

A RESTful API makes a lot of sense when dealing with plain data, but starts making less sense when you're attempting handle full fledged use cases.

However, your core API logic should be separated from whatever network layer exposure you're giving your API (and ideally should handle endpoint negotiation for you)

danenania• 13 years ago

A thin layer that maps methods to api requests should take care of this issue and be simple to write. This is exactly what database drivers and service sdk libs do, abstracting connection details into an easily consumable interface.

rubiquity• 13 years ago

I love this development and it's something I fight to try to get places that I work to adopt. Building your service through the eyes of a client gives so much clarity about both the what and the how. It pairs really nicely with TDD as well. You also end up getting an API for free out of it and have an easy transition to Service Oriented Architecture if you so choose.

jemeshsu• 13 years ago

I wonder if API first approach is suitable for all type of app. Is there anyone taking this approach for a CMS site, a eCommerce site or a stock trading site?

macspoofing6• 13 years ago

This approach is very attractive to developers because it makes things so neat and tidy and perfect, but there are two problems that you quickly run into:

1) Search engines can't index your page properly. If you rely on this to drive traffic, you'll have a problem

2) Your page takes a longer load (even more noticeable on mobile), so you will inevitably move some work back to server, whether to render some or all parts of the page, or seed your initial models with current values so that they don't have to immediately call back.

kaolinite3• 13 years ago

1. Not really true nowadays. Google can trawl Javascript sites. Other search engines may be able to as well.

2. Initial page load will be slightly higher (and subsequent pages will be much faster) but I can't imagine it'll be too noticeable. People get away with a lot - even on mobile - loading numerous JS libraries, tonnes of CSS, etc. As long as you go easy in other regards, this shouldn't be a problem.

There's no reason that API driven design has to involve client-side rendering. You can always consume your API on the backend and render on the server.

macspoofing6• 13 years ago

There's no reason that API driven design has to involve client-side rendering. You can always consume your API on the backend and render on the server.

I'll give you that I suppose, though I understand it as a lightweight server, and heavier client. On the other hand, splitting up your components into API and page-rendering servers (in addition to your database, caching, messaging servers and your reverse proxies and load-balancers) complicates your deployment and maintenance and you pay for that - so it isn't quite the panacea and now you really have to decide whether it makes sense to go down that route, that early.

napoleond3• 13 years ago

1. Not really true nowadays. Google can trawl Javascript sites. Other search engines may be able to as well.

People have been saying this for a long time, but as of 6 months ago my experience didn't support it (Javascript sites get indexed, but not ranked as well as traditional sites). I haven't tried recently--does anyone have up-to-date information about the effect of client-rendered sites on SEO?

jqueryin8• 13 years ago

I should note that our logged in application utilizes the API whereas our frontend site is primarily static content for purposes of page speed and to lighten server load.

macspoofing6• 13 years ago

whereas our frontend site is primarily static content for purposes of page speed and to lighten server load.

Yeah, that's absolutely the right way to do it. So it's less "API-first", and more "API-first-unless-we-want-it-fast". =)

pyrrhotech• 13 years ago

I thought all modern web/mobile apps were architected this way? Is this novel information?

jcampbell1• 13 years ago

REST APIs are nice, right up to the point you need a transaction... three things need to happen in the correct order, and all succeed or are rolled back. If the API is too granular, then you end up pushing this logic to the client, and that turns into a disaster.

I'd personally rather go back and extract the smallest possible functional API, rather than build a massive API upfront. You don't really know where the transaction boundaries are as you are developing, and it is easy to expose an API that not really possible.

johns6• 13 years ago

The way we handle this (we have 18 internal services) is that processes requiring transactions are formalized into a endpoint/resource. The actual transaction becomes tangible (e.g. you can cancel them by DELETE-ing on the resource) instead of just being documented. Only logic the client needs is the data to kick off the process. Async or long-running processes get a little tricker, but polling or webhook callbacks work fine for us.

dreamfactory4• 13 years ago

A transaction should be the lowest level by definition e.g. it's not an API if you are just proxying CRUD on your db tables. Agree on smallest API possible (otherwise you will die in maintenance hell after you pick up a few clients) - but your API should be the only point of contact for your site and other clients.

jrgnsd• 13 years ago

Some interesting thoughts on doing financial transactions using a REST API here: http://jpos.org/blog/2013/11/w55y/

Kiro• 13 years ago

How do you handle authentication?

jqueryin8• 13 years ago

Vaguely answered here: https://news.ycombinator.com/item?id=6763221

mkramlich2• 13 years ago

I'll just leave this here:

CLIFMO: CLI's First, Maybe Only

life will be better. trust me. it's a beautiful thing.

(Also a big fan of RDD or README Driven Development. I didn't coin that term but I was doing it long before I finally heard somebody coin it. RDD and CLIFMO FTW. I have only a love/hate relationship for TDD, in comparison -- it's a mixed bag, advantages and disadvantages, awesome and terrible, depending -- but only pure love for RDD and CLIFMO.)

machielo• 13 years ago

You, Sir, are a gentleman and a scholar. I'm a believer and I'll be using the CLIFMO and RDD acronyms from now on!

Yhippa• 13 years ago

CLI = Command Line Interface? Can you expand upon this a bit more? I can't find any other references to this term.

danudey2• 13 years ago

Our entire software deployment system across all of our servers is built on systems built on systems built on a command-line tool I wrote in my first month.

It's really useful because anything that you can do from our deployment panel you can do by hand by logging into the server (and actually, even more stuff which isn't exposed).

The one thing I haven't gotten around to doing is revamping the tool to properly support JSON output for all commands. It might require a new logging module to handle errors though…

mkramlich2• 13 years ago

nice! that sounds like CLIFMO to me. yeah it's amazing how much easier it makes so many things. development-wise it yields so many freebies that pay off later.

sbov• 13 years ago

I find the biggest win about API first development is it forces discipline upon you. I find I can get most of the benefits if I code in a disciplined fashion. It's usually not easy though.

In many ways, I find API first development most useful in languages such as python, which lacks the idea of an interface. Similarly, I find interfaces not much more than yet another tool that forces discipline upon you.

It seems like as we develop these abstractions which increasingly force more discipline upon us we forget that classes and functions are supposed to be abstractions themselves. We are supposed to be relying on the interface of those classes and functions, not the actual implementations, even if the class itself doesn't use the interface keyword.

One of the oft maligned issues of c++ is the .h file. I rather like them though. They're a constant reminder that "this is all you should know". It becomes obvious when you add cruft onto it because you don't have that cruft obscured by hundreds of lines of implementation. It's much more difficult to notice it in languages where the implementation of a class or module is present alongside its interface.

rpedela• 13 years ago

Yep I agree. APIs are often much easier to test as well. Proper testing is part of being disciplined.

sebkomianos• 13 years ago

A "How We Developed POP.co Following the API Driven Development Approach" would find a lot of people quite interested too, I think.

X4• 13 years ago

Perfect timing!! I was looking for it. Do you know any opensource or free tool that can generate the commented code + API Documentation required, from the API spec?

Only found: http://apiary.io/ but it doesn't generate any code.

Why We Chose API First Development at Our Startup

Related Stories