jrosoff

oh man! You had a tape drive for your vic 20? when i bought games they came as a book and i had to type the assembly instructions in for 3 hours if i wanted to play!

The look and feel of your site is great! Nice job! I agree with other commenters about the need for richer data. For example, just picking on the "top performing" site in your list (skysheet.com), here's the yottaa report on that site:

http://www.yottaa.com/url/skysheet-com-4d2ca049038ade0c05000...

Yottaa notes a similar reachability time as stella does in the "Reachability > Washington" metric. But response time performance is significantly worse at other locations.

Also, it's important to track not just response time of the server, but actual browser performance. If you look at the Page Load Time metric, you'll see that even when the server responds pretty quickly, the actual browser experience is significantly worse than a simple HTTP Client.

Overall, I like the site a lot. If you could get some more accurate metrics, it'd be great. Keep going!

BTW> Here at Yottaa we're beta-ing an API that would let you run tests and access all of our browser and low level data. Would you be interested in getting some better data from us? Contact me : jrosoff AT yottaa DOT com

Great writeup! Couple questions:

- I'm curious why querying before a write makes such a big difference. I would have guessed that updating a document that's not in RAM would first load it into RAM, then perform the update. Does the write get applied to disk without loading the page into RAM first? If you do an update to a document that is not in RAM, is it in RAM following the update?

- Can you elaborate on the corruption that occurred to both the master & the slave during a DAS failure? We have seen something similar in our deployment (high write volume leading to corruption in both master & slave. required repair to recover. ran on a partially functioning slave during the repair), but were unable to identify the root cause.

Looks like this was a result of old data.. The score of 78 was measured on October 2nd. I hit the "Click here to re-check now" button and the score is now listed as 88. We're looking at the sub-scores now to see exactly what's different.

What site are you measuring? I'll go check it out. Feel free to follow up via jrosoff AT yottaa DOT com

Cool! There are a bunch of external monitoring services that are worth checking out in addition to this..

- http://www.yottaa.com (shameless plug for my own company) - http://www.webpagetest.org - http://www.zoompf.com - http://www.showslow.com

Yes definitely some issues with measuring page load time this way. It is useful data to get started with. The web timing API will make this data much better, but it's still early (it's in IE9 and some early developer builds of chrome and firefox). Once this API is more widely deployed, this is the right way to measure performance as it allows you to take into account navigation time as well as rendering time.

Google Webmaster Tools does indeed give you a good perspective on performance. I believe this data comes from the google toolbar (can someone confirm this?). My problem with the data reported by webmaster tools is that it's an average. It doesn't tell me what the _worst_ page load time of the _best_ page load time my users are experiencing.

As a shameless plug, you can also check out our own tools from yottaa: http://www.yottaa.com that give you a bunch of detailed information about your websites performance. I think tools like these complement the approach detailed in the post.

Thanks for the feedback!

You're correct about the issues with "page load" time. The approach in the post is really measuring the amount of time that the browser spends processing the page. However, for many modern web apps, most of the time is spent in these internal aspects of your page such as loading CSS, running javascripts, fetching images, etc...

The web timing API (there's another post about that here: http://blog.yottaa.com/2010/10/using-web-timing-api-to-measu...) we can get more detailed timers that count not just the browser time, but actually the full amount of time between typing in the URL into the browser (or clicking a link) and finishing the load of the page.

The linear trend line was created by excel automatically so I can't vouch for its accuracy other than my implicit trust of that feature.

Hummingbird is awesome and both as a tool and a case study. We learned a lot from Hummingbird that we incorporated into the design of our system.

We have hit 1000's of updates per second on our current system during some high load periods and did not see any problems. Our steady state is 100's per second, but it bursts to 1000's for extended durations about once per week if not more often.

10 reports per second is actually not that much load and has almost no impact on writers. We have an alerting system that runs while data is input to the system. It effectively loads a report for each metric reported in the input and decides whether or not to send an alert. That system generates queries about 50 reports per second on an ongoing basis and does not impact the writers. Our read volume in steady state is about 2x our write volume.

We have not seen any queueing problems on writes and the lock ratio in mongodb is typically in the 0.01 - 0.005 range.

We have found that we can break this by running lots of map-reduce jobs simultaneously while processing high write volume but that's a whole other ball of wax.

Our data access patterns very easily accomodate sharding. Both reads and writes are pretty even distributed across the set of URL's we track. By activating sharding using URL as shard key, we feel we can handle scaling several orders of magnitude beyond where we are now without anything more than additional hardware (or virtual machines).

I'd love to hear how your system scaling goes. Feel free to hit me up via email if you want to discuss (jrosoff AT yottaa.com)

Yeah slide 11 depicted what we thought would be a great solution before we started investigating MongoDB. MongoDB effectively replaced all those other systems for us and was _significantly_ easier to set up and develop against.

A few people have mentioned HBase as an alternative. We did not consider HBase at the time we were making our architecture choices, however if we were starting today, we'd probably have looked at it too. My first impressions of HBase are that it lacks the level of documentation & community support behind MongoDB. I am definitely going to dig in some more to see how it would compare. That being said, we're totally happy with our choice of MongoDB and would recommend it to anybody considering HBase.

We do not take into account CDN's with custom CNAMEs at the moment. I responded to a comment above that addresses this. See above.

yeah we have issued a smack down to the guy who set up our wordpress for not putting links in there. sorry about that. and we do need to buy some of the variations of our domain name. but hey, we're a startup!

regarding the CDN. Totally agree, you can't just assume it's a CDN based on CNAME. we're working on some smarter ways of identifying the CDN you're using based on a few things. we walk the dns resolution tree. so we'll see, for example, that img.yoursite.com is cname'd to xxxxxx.cloudfront.net. we're building a database of known cdn's so we can easily identify what CDN you're on. we also want to let site owners tag their own domains to self-declare which things are cdn's and which are self-hosted. all of these will help to give us better data.

oh and we love feedback, so please tell us if you've got better ways!

A few minutes is definitely too long. Should be more like 30 seconds.

Try refreshing the page or searching for the URL again.

What URL were you querying?

Email me if you still have problems (jrosoff AT yottaa.com)

If you use Google Analytics, you can create an account at www.yottaa.com and we'll chart visitor count, bounce rate, and a few other metrics against your websites performance. If your site is new to us, it will take a few days before there's enough data to be interesting.

We have a lot of very cool features planned to extend this as well.

We're also working on putting together some aggregated statistics about these metrics for a future blog post.

Check out our startup -- http://www.yottaa.com

We are cranking up our intervals and locations every day. And it's free! Would love to know if it solves your problem and if not, what you would need from it.

What's your sales pitch for mongrel2 against things like Unicorn & Passenger. The new features of mongrel2 seem awesome (particularly multiple protocols over the same socket) and given the success of mongrel, I think we all have high hopes for mongrel2. But I'm curious what your thoughts are on what the "killer apps" are for mongrel2 that make it the superior solution to the alternatives.