andma

Hey I work as an engineer for a web startup in palo alto pretty close to stanford. I have worked across the entire stack. Honestly, I don't think any particular choice for a framework/language is going to matter much. You should choose whatever you feel the most comfortable with or have the most experience with. Also, why do you feel compelled to rewrite it? The time you spend rewriting it might be better spent on customer development, marketing, etc. You can contact me at andmahn1@g​mail.com if you want to meet up in palo alto.

You hit the nail on the head. If you don't need strong consistency guarantees any sort of message queue will do. That is a solved problem.

There are two key quotes from the article:

1. "... if our queue begins dropping messages, we run the risk of silently corrupting data"

The two statements that they are executing "app.update" (modifying the database) and "enqueue" (sending to message queue") are not atomic. They can make it atomic using something like 2 phase commit but implementing that will be more trouble than it's worth. Additionally, there will be performance (latency) implications since it is a synchronous operation.

2. "Notice how both of our writes are inside of a local database transaction"

Now they have atomicity very easily!

I agree with the grandparent that using SQL as a message queue is generally a bad idea. However the pitfalls are well known and can be engineered around. See this article: http://www.engineyard.com/blog/2011/5-subtle-ways-youre-usin...

I am actually working on something very similar to this at my day job. The "buffer processor" that the article is describing, we implemented using a scala process. Perhaps what's most interesting about the entire system as mentioned at the end of the article is maintaining the large amount of state for durability (we use hbase) and deliverability (we wrote a custom scala server).