Building Clojure services at scale

http://blog.josephwilk.net/clojure/building-clojure-services-at-scale.html

by erkose • 12 years ago

137 21 12 years ago

blog.josephwilk.net

arohner2• 12 years ago

Great article, thanks.

[from TFA]

    > This however does come with a cost:
    > - We cannot use Clojure’s concurrency primitives (futures/promises/agents).

Running my own large-ish Clojure service (CircleCI), I'm pretty convinced that futures are an anti-pattern in many situations. If your future is for side-effects, rather than for computation, the future call should probably be going on a durable queue being processed by a cluster of machines.

yayitswei• 12 years ago

What are you guys using for durable queues? I'm currently evaluating Redis and RabbitMQ.

coolsunglasses2• 12 years ago

I wouldn't use either of those if you need durability.

lgierth• 12 years ago

Pairs of RabbitMQ brokers, used in a Publish-One-Subscribe-Many fashion for availability.

Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105

Edit: usually with durable queues on SSDs

Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...

Scuds• 12 years ago

and here I am, staring down the barrel of Erlang. again

smrtinsert• 12 years ago

That's a very interesting way of looking at it.

efuquen• 12 years ago

I don't quite get what you mean. The author's example is a GET request on some web service, are you arguing we should always be using queues instead of a request/response models when talking to external services?

arohner2• 12 years ago

No. My stereotypical use case is say, responding to an HTTP request that involves side-effects:

    (post "/user/foo" [request]
      (update-db)
      (future (send-user-email))
      {:status 200
       :body (render-page)})

What happens if email delivery fails? What happens if this box powers off? etc. Queues don't solve all your failure scenarios here, but they're a great first step.

lkrubner• 12 years ago

I have not worked at the scale discussed by Joseph Wilk, but I am pleased so far with my ability to build out micro-services with Clojure. The eco-system around Clojure seems to get stronger every day. I'll mention a few of the libraries that I'm using.

For building out a RESTful API, I found Liberator to be very clever and original:

http://clojure-liberator.github.io/liberator/

For handling internal queues, I use Lamina:

https://github.com/ztellman/lamina

For instance, when saving something to the database, I usually put the data in a closure, put the closure in a Lamina "channel" and let some workers execute the closure when the thread-pool and IO constraints allow.

I just started experimenting with the Ribol conditional restart system, for error handling:

http://docs.caudate.me/ribol/

For the integration of frontend with backend (MongoDB) we were originally going to use DerbyJS, but I talked the team into trying Sente instead:

https://github.com/ptaoussanis/sente

We are not done, so the jury is still out, but I am very impressed, so far, with Sente.

For dealing with MongoDB, we use Monger, a library that just keeps getting better and better:

http://clojuremongodb.info/

In terms of my personal development, I have struggled to find a workflow that allows me to be efficient at the REPL. I am just now starting to use Daniel Szmulewicz "system" which I am hopeful about:

https://github.com/danielsz/system

Rapzid• 12 years ago

How are people getting along with core.async? I'm a bit surprised it wasn't mentioned in the article.

catlover312• 12 years ago

1) Give a kiss to your hand (left hand) 2) Say your crushes name. 3) Close your hand. 4) Say the name of a weekday. 5) Say your name.

6) Open your hand. 7) Paste this to 15 comments n the day you said In step 4 he/she will tell you they like you. If you don't do this you will have bad luck in the next hour.

Building Clojure services at scale

Related Stories