How We Work on Queries at GitHub

http://samlambert.com/posts/how-we-work-on-queries-at-gitHub/

by samlambert • 12 years ago

82 29 12 years ago

Now, does anyone have the time to figure out how many users Github has in their users table? :) http://dheera.net/projects/blur

petercooper• 12 years ago

The mosaic effect they used doesn't appear to be real, as no number has a horizontal gap between two parts.

However, GitHub user IDs are/always used to be sequential, create a new account and see what its user ID is. It's around 8.6 million at the moment.

minimaxir• 12 years ago

Atleast 3,815,207 users. (who have performed some meaningful GitHub action)

Via BigQuery:

   SELECT COUNT(DISTINCT actor) FROM [githubarchive:github.timeline];

ckluis• 12 years ago

Awesome. I’m going to share this to see if we can do something similar where I work. Neat idea.

blaincate• 12 years ago

from : http://ghtorrent.org/downloads.html

atleast :

   4,151,457 repos   

   2,480,478 users

popular repos :

twbs/bootstrap 40662

jquery/jquery 34633

joyent/node 34522

mbostock/d3 30247

h5bp/html5-boilerplate 28736

popular users:

visionmedia 10712

torvalds 9984

paulirish 5885

schacon 4431

mattt 4053

pjhyett 3732

src code: https://github.com/akuchlous/githublike

simonw• 12 years ago

"Once we have decided if we want to modify our schema we can perform an incremental rollout across our cluster. I will cover this more in another post." - looking forward to that.

techdebt5112• 12 years ago

my guess is a thin wrapper around PTOSC.

famousactress• 12 years ago

So, maybe a toy example but I can see the query included a join. Curious how you guys clone enough tables (and their keys) to troubleshoot things like that? Seems like it gets a lot more complex than the example suggests pretty quickly. Wondering if you have neat tools for that.

[Edit: Just noticed poster is author. Hi Sam and welcome to HN :)]

samlambert7• 12 years ago

Basically you can hit /mysql clone for any table and it will make its way to an isolated db for the user running the command.

It would be cool to be able to pass the script a query and have it clone all the tables.

samlambert7• 12 years ago

Hey! Thank you for the welcome :)

wldlyinaccurate• 12 years ago

I really wish this went into more detail. You are notified when a query is slow. You can EXPLAIN it in chat so everybody can see. What happens next? Are slow queries treated as high priority? Is there any tooling around debugging complex queries? Basically, what makes GitHub's process different to the decades-old "grep the slow query log and run an EXPLAIN"?

technoweenie2• 12 years ago

Basically, what makes GitHub's process different to the decades-old "grep the slow query log and run an EXPLAIN"?

Sam's chatops tools opens this process up to a lot more people that otherwise wouldn't be comfortable logging on to servers to access the slow query logs (assuming that they even have access to the servers). It's a great way for app developers to level up their sql skills from other more experienced coworkers.

The impact of slow queries determines their priority. How frequent are the slow queries? Is it from a background job, or does it cause exceptions on important pages or API calls?

I don't know if this process is streets ahead of what other companies have, but it's made a hugely positive impact on our MySQL infrastructure.

samlambert7• 12 years ago

If we get a spike in slow queries we get alerted via pager etc. If a query is slow enough to be killed Hubot tells us https://twitter.com/isamlambert/status/502818333914566656

I am working on some query linting as a side project.

joshmn2• 12 years ago

Nice read Sam, thanks for the writeup.

Thought I'd let you know you have an error in your json - section.name (http://i.imgur.com/CoHol0f.png)

;]

samlambert7• 12 years ago

haha thank you :)

albertoleal• 12 years ago

Is haystack open sourced anywhere?

samlambert7• 12 years ago

Unfortunately not. It is so closely tied to our applications.

possibilistic• 12 years ago

Thanks for posting this! I like hearing about internal tooling. Is there more on the query tagging? How do you guys bubble query annotations through the stack?

I don't know if you're at liberty to discuss further, but how has it been scaling a giant Rails app? Have there been any pushes to break it up into smaller components? Ie. fast moving stuff stays Rails, core infra moves to something statically typed?

yRetsyM• 12 years ago

Are all github internal services a dark colour scheme? good way of determining internal vs external I suppose

Caged• 12 years ago

There's no hard rules on it. When I started working on it a couple of years ago, I was fond of dark color schemes for monitoring interfaces.

samlambert7• 12 years ago

I'm not sure tbh, I think it makes it easier for graphs to stand out. I do love the interface of Haystack.

revskill• 12 years ago

Why doesn't Github open source their products/libraries regularly ? There is too few open source projects from Github on Github.

holman• 12 years ago

We do!

https://github.com/github https://github.com/libgit2 https://github.com/boxen

When it's easy to extract, well-documented, and has a clear team of maintainers, we try to open source. Sometimes it's difficult to nail one or all of those bullet points, though.

nacs• 12 years ago

You forgot https://github.com/atom

joshmn2• 12 years ago

Haystack looks interesting ;)

How We Work on Queries at GitHub

Related Stories