jehiah

With NSQ there is a built in utility nsq_to_file which just becomes one additional consumer you'd use to archive each message topic to disk. It provides dead simple archiving of messages, but doesn't provide any native replay ability.

ditto. https://github.com/tornadoweb/tornado/tree/master/tornado

All Bitly links are no longer blocked by Google Safebrowsing in Chrome or Firefox.

It was inspired largely by http://bl.ocks.org/mbostock/4061502 so credit where credit is due, Mike Bostock probably deserves most of it.

thats funny. Almost added a diapers spending category

thanks! Most comes from credit card, and foursquare but some various other places. strava (bike ride timings) starbucks card (some coffee timings), scripted photo snapshots, scripted computer activity snapshots etc.

I've also posted some of the scripts on github like this one to parse out iMessage/sms logs from an iphone backup. https://gist.github.com/jehiah/4437883

Great writeup of some fun unicode/font details. I've had to use the trick of limiting font files to specific unicode ranges while working on Amharic support for itypeamharic.com (Amharic/Ge'ez glyphs are also typically not present in any fonts on most devices)

best-idea-ever

Is there an official way of measuring beard length?

I've dumped the cron job i use for data collection here https://gist.github.com/4477372

My rough approach to post processing this data was to group by 6 minute intervals, and to drop any records where the computer was inactive for more than 120 seconds.

The bitly office moved, so my morning coffee habit had to switch slightly. It's for the better thought; I <3 Bourbon Coffee.

thanks dude!

yup. I actually managed to travel without my laptop for a few weeks!

I've not heard of whatpulse, but looks cool. I rolled my own data collection for keyboard stats last year.

I've highlighted some last.fm data in a few of my previous annual reports, but it didn't make it this year (I Wasn't quite into new music enough recently for it to be interesting to me).

I was most surprised by the sheer amount of time I spend on the computer; clearly I need to unplug more often. (Though you can tell I did drop off the grid for a few weeks when my daughter was born, and again for a vacation in August. At least that's a start).

I used isightcapture (The original source is offline, but i think this is the same thing http://distfiles.macports.org/isightcapture/)

https://gist.github.com/4477040 is the cron script I use on a 10 min interval.

There might be other ways to do this now days, but i've used this successfully for a few years.

doh. That's probably due to loading a full years worth of images as png's (I've added support for webp images when browsing w/ Chrome but that isn't going to help the iPad). Sorry i didn't get to do more than minimal iOS testing.

glad you enjoyed the report.

lies, damned lies, and statistics. Unfortunately i didn't have time data for my other trips to Starbucks so that dataset is partial (I mostly visit Starbucks on the weekend/holidays).

The transition from Carte Blanche to Burbon Coffee in the middle of the year is when @bitly's office moved, and my coffee habit followed.

coffee data is more driven from foursquare checkins, and only partially credit card transactions.

The motivation for me is a way to learn about my year (not directly to change my actions), and part a fun exercise to get better at data processing, javascript, and UI Interaction/charting. I like the hard deadline that a personal annual report creates naturally (ie: it's no good if i release it in june).

The script I used for extracting sms/iMessages is here https://gist.github.com/4437883

Some of the other scripts just download data from foursquare's API, and post processing transaction download from Mint. Hopefully I'll have time to clean and post those scripts.

a lot of it is just simple python scripts to download/convert data to csv files, and do some simple post processing.

for the computer activity i took a snapshot every 30seconds of what program was active, and how long my laptop was active.

Thanks for the reminder. Getting binary builds up is on our to-do list, so check back soon for those.

In the mean time, installing Go is pretty easy (you can use brew, or the official OSX package (assuming you are on OSX) http://golang.org/doc/install) and we've tried to leave clear steps for building NSQ here https://github.com/bitly/nsq/blob/master/INSTALLING.md (there really are very few dependencies other than go itself)

Good idea, we will try to put together some numbers with regards to message size.

Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq

In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.

The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.

If there are any specific numbers you are curious about, please ask.