werner

You are right, it was a typo. It should have been 10.0.0.0/8. Updated that in the blog, also resolved a confusion about the memory in a P3dn.24xlarge.

Thanks for your critical reading!

Amazon has been building technology based on ML&DL for over 20 years and has developed several frameworks. You must have missed the announcement of this open source framework earlier in the year: https://github.com/amznlabs/amazon-dsstne.

Can you elaborate on the intra- inter-* latencies your are seeing? for AZ independent services this should single digits.

No mesos under the hood. You can bring mesos as your own scheduler.

You can choose eventual or fully consistent in DynamoDB. Given that full consistency comes at a higher cost (read from a quorum of replicas) we expose that cost to you.

BTW nobody wants eventual consistency, it is a fact of live among many trade-offs. I would rather not expose it but it comes with other advantages ...

Nope, we have built a new scheduler for you that will allow placement over multiple AZ's, replace failed containers, allow them to connect to ELB's, etc.

Luke, drop me a note at werner [at] amazon with a link to the support ticket you created, and we'll see what we can do.

Yep, I thought it would be of interest to the HN community, but most importantly I was interested about the comments/feedback/criticism of those who would potentially be using it.

And if you are looking for really cool jobs in which you can apply all your distributed systems and cloud skills on amazingly interesting products, beyond just video streaming, check out Amazon.com (75+ pages of 20 jobs per page in software development alone)...

http://www.amazon.com/gp/jobs/ref=j_sq_btn?keywords=&cat...

(sorry Jeremy, coudln't resist) :-)

5000 reads per sec of 64Kb items, would make you stream 2.5 Gbits/sec using consistent reads and 1Gbits/sec writes, moving close to 1.5TB each hour. At the end of the month you have read well over 800 TB and updates 160 TB... That is a substantial application you have in mind... :-)

noted. (both requests that is).

The amount of consumed read units by a query is not necessarily proportional to the # of items. It is equal to the cumulative size of processed items, rounded up to the next kilobyte increment. For example if you have a query returning 1,500 items of 64 bytes each, then you’ll consume 94 read units, not 1,500.

you can choose whether queries are consistent or eventually consistent.

It is on the requested features list

Can do me a favor and drop that in the AWS DynamoDB Forum so folks can look at it? https://forums.aws.amazon.com/forum.jspa?forumID=131

As with all the other AWS services you can have your limits lifted upon request.

Actually I did do some coding for the conversion, etc. :-) But I like it when doing something new to be able to look at how other people solved similar problems. And it is a bit early for Cactus in that respect, and Liquid feels much simpler than Django templates.

The extension and plugin mechanisms will make it easier for me to start adding my own code without having to modify the core framework. But it is always more fun to add these kind of things if there is a community to give you feedback.

Actually that was my fault. I had just switched off the redirect of allthingsdistributed.com to www.allthingsdistributed.com and as such you ended up at the old MT installation. That is now corrected.

You are correct; to map to an S3 bucket you need a CNAME. But DNS doesn't allow the apex to be a CNAME so you will need to redirect that. Route53 solves that for EC2 with the help of ELB. But there is no such solution for S3 (yet).

I am using the www subdomain as much as possible, so the redirect only happens if a visitor actually types in the apex name, in all other cases they will get where they need to be directly. But I agree that it would be better to solve this at a different level.

Launch and then iterate...

My blogpost w background information: http://wv.ly/iLDaqu

The data that goes with the research can be found at: http://faultlinux.lip6.fr/

For a presentation I needed an additional voice to convince the audience that building a sustainable business was preferable over striving for a massive exit. A short compilation of David Heinemeier Hansson talk at 2008 Startup School did the trick. With all the increased bubble talk it is worth watching again. You can find the full talk at http://www.justin.tv/hackertv/b/259414909

Absolutely. Next steps already defined :-)

Actually you could run wordpress with WP Super Cache enabled to generate static files.

Moveable Type has support for multi-site deployment through rsync (see the MT docs on rsync target, etc). After that the job queue finished MT will call rsync to upload the files to remote locations.

Didn't take that much work to make it work with rs3sync instead (see http://rs3sync.net).

You guys are important customers...

Dynamo is still in use, but as with all technologies that have to operate at Amazon's scale, the systems evolve rapidly. The storage systems in use now no longer look like the ones from 5 years ago when Dynamo was developed.

The Dynamo SOSP paper had two goals: 1) show how systems are a composition of techniques and how all of these need to work together build a production system 2) given that it was based on a variety of research results it was intended to give feedback to the academic community about the difficulties of moving from research results to production, and what matters in real-life vs production.

The paper was never intended to be a complete blueprint for easy design of follow-up systems.There just isn't enough room in an academic paper to do justice to that. If it was going to have any role in that the best we could hope for was as a collection of points you would have to think hard about and make decisions about when you were going to design your own storage engine.

That doesn't means I agree with the conclusions of the analysis, on the contrary, I think they are seriously flawed as well (as with everything else that is not absolutely perfect :-)). But I can understand that when people look for the dynamo paper to be a blueprint that solves all their storage needs and provides a perfect available service under all failure scenarios (and solves world peace), that they may be left with a few questions afterwards.

I always thought that the real contribution of the paper was that it made you think hard about the trade-offs you are faced with when you have to design high-available, ultra-scalable systems that are cost-effective and provide guaranteed performance. 5 years later we know a lot more but this stuff is still hard, and we still need to balance rigorous principles with production magic to make it work. But you do need to fully understand the principles before you can make the production trade offs.

Caveat: some of these remarks were tongue-in-cheek fun; I leave it to the reader which ones :-)

This might help: http://www.ntia.doc.gov/osmhome/allochrt.pdf