How GitHub No Longer Works

http://zachholman.com/talk/how-github-no-longer-works/

by bado • 13 years ago

223 132 13 years ago

I'm a big fan of Zach Holman's talks and slides. He's a great speaker on these topics. Looking forward to seeing the video up soon.

bronson• 13 years ago

Seems like 20% of that slide deck is about chat rooms... GitHub still uses Campfire for chat?

I tried Campfire and Hipchat a couple years ago, neither stuck. Not sure why, maybe poor offline notifications and logging? Or maybe our team didn't have enough timezone overlap or things to chat about? Hard to say.

Is strong chat as important to Github as Zach implies?

holman2• 13 years ago

Absolutely.

As I mentioned, we have 150+ rooms that we chill in, and most of our day-to-day work practices are tied into chat. Mention someone's name in a room they're not present in and we send push notifications to their phone and desktop- works great to pull people in as needed, without needing to distract them with the noise of being in the room 24/7.

Our ops team is particularly in deep with chat. Instead of siloing everyone's process off individually in SSH sessions, many operations happen in chat, so everyone can learn and help out when diagnosing problems. They've built some really fascinating tooling around the problems they face- if you're interested, check out @jnewland's talk on ChatOps: http://www.youtube.com/watch?v=NST3u-GjjFw

DanielRibeiro• 13 years ago

Great post Zach!

Tom Preson Warner, GitHubs founder and CEO (and creator of TOML[1]), also gave a very good overview of how GitHub works: http://www.youtube.com/watch?feature=player_detailpage&v=Ms-...

Tim Berglund recently talked in details about the PRPs and the product development process: http://www.infoq.com/interviews/berglund-product-owners

[1] https://github.com/mojombo/toml

trumbitta2• 13 years ago

I still don't get how do you manage to do security-sensitive things in Campfire via Hubot :-/

kawsper• 13 years ago

All the teams I have been part of have used chat together with Github. First team used Campfire, on the second team we started out with Hipchat, but we decided to give Flowdock a try, and now we can't live without the chat threads that Flowdock provides.

jcutrell3• 13 years ago

We've been on HipChat for a bit now. Campfire seemed a bit ugly to our tastes.

We don't use chat so much for the Hubot-ish stuff, but I have enjoyed using Robut[0] as a culture tool. I usually serve it directly from my computer, which is fun.

https://github.com/justinweiss/robut

mnutt• 13 years ago

My team uses Slack (http://slack.com) and is very happy with it. It's in active development, and has good cross-platform clients. (Flint for Campfire is great, but last time I asked they didn't have any way to do volume licensing)

msutherl• 13 years ago

This probably works great for all-hacker teams, but I think it's worth pointing out that some people don't like typing or otherwise aren't comfortable expressing themselves in text. I work with some people who simply refuse to use online collaboration tools. Were I starting a company, I probably wouldn't hire them, but I wonder if there's something to be done about this.

speg• 13 years ago

How GitHub works is one of my favourite slides, looking forward see how things have changed.

carbon82• 13 years ago

I've found it interesting how many people in the Bay Area, particularly veterans of the 90s boom, discount a lot of the ideas pushed by GitHub and 37signals. Anedotally, one of the common responses whenever I reference either company is, "Yeah, but how big is [GitHub|37signals], really?" It's actually somewhat surprising to me how consistently I've encountered this response (so consistent, it feels as if it was distributed like a political talking point). As mentioned in the slides, publicly announced funding definitely serves as a signal, and I haven't heard anyone say that about GitHub since then, though someone said it to me regarding 37signals within the past few months.

What I think it really has to do with is that GitHub seems to make an effort to hire doers, whereas many companies develop processes to deal with non-doers, eg, having staging servers so non-technical staff can check the work of "their" developers. Remote work also doesn't serve the interests of non-doers, since they want to Have Meetings and Make Decisions.

runako• 13 years ago

> staging servers so non-technical staff can check the work of "their" developers

Surely staging servers have some other benefits? Maybe some teams consider running code on the production setup for the first time to be risky to their customers? Perhaps some teams have encountered differences between dev laptops and production servers, which run on different OS/memory/etc. combinations?

carbon82• 13 years ago

Short-lived staging environments (eg, a temporary clone of a production environment) certainly have a place, such as when making architectural changes, but these kinds of changes are generally not happening on a regular basis if you are making incremental changes and doing continuous deployment.

I'm sure that there are companies that have a valid engineering need for perpetual staging environments rather than feature flags, but I've seen absolutely no evidence that staging servers are commonly engineering driven. Certainly in every part of the web startup world I've had contact with or heard about, staging environments have consistently been for product QA in organizations with heavy-handed processes and/or product management by non-technical stakeholders.

Edit for the people responding: This is not about haphazardly pushing to production. You should familiarize yourselves with continuous integration (http://en.wikipedia.org/wiki/Continuous_integration) and the various deployment strategies of major web companies.

justincormack• 13 years ago

well I heard New Relic today saying they run their own production monitoring on their staging.dogfood it first...

trustfundbaby• 13 years ago

I've found it interesting how many people in the Bay Area, particularly veterans of the 90s boom, discount a lot of the ideas pushed by GitHub and 37signals

Then you can imagine how many people outside of the Bay area do the same thing. I've worked at places where I (stupidly) suggested some of their principles as solutions to systemic problems that came up, assuming that everyone would be on the same page (I mean their ideas just make sense right?) ... no way. I've gotten laughed out of the room a couple of times, wised up and am now resigned to the fact that unless I start my own company or join the founding core of a startup, I'll never get to apply those kinds of ideas unless I work in the Bay

eru• 13 years ago

Don't give up so quickly. Sneaking in good ideas into companies is a skill that takes time to learn, and even when mastered takes patiences to exercise.

dasil003• 13 years ago

What really sets GitHub and to a lesser extent 37signals apart is that they are making a product for themselves. That's how they manage to only hire doers. They don't need marketing or usability testing or operations or acquisitions beyond what the doers provide because the feedback loop to the target market is near direct. That's not to say it's easy to do what they've done, but rather it's much less likely those principles could be held as the staple of companies in other markets where engineers and designers will not be as naturally adept at nailing product/market fit by themselves.

jcutrell3• 13 years ago

Love these talks from Zach Holman.

Brilliant work happening at GitHub, of course. Seeing the rise of the "primarily responsible person" is huge for us in our small company - we've found it's natural to work this way, without a label.

Keep on putting out these fantastic slides, Zach.

ddoolin• 13 years ago

Seriously. I almost never just flip through slides without a video but these were engaging and to the point, as a slide show should be.

...and now I wanna work at GitHub. :)

Keep it up!

skrebbel• 13 years ago

Admittedly I'm cynical, but how is a "primarily responsible person" different from a lead developer? It feels a bit like you're just calling things different to avoid manager-like names.

jbarnette2• 13 years ago

We use PRP in any situation where someone explicitly takes responsibility for an outcome, no matter whether the outcome is some software or a clean office or an accurate tax return.

jcutrell3• 13 years ago

Sometimes the primarily responsible person isn't a developer on a given team. Especially at smaller firms, hats of responsibility are passed around to multiple members of the team, regardless of that member's particular talents/core competencies.

We call them project champions. I work in an agency model, so we have many clients who have many projects. The project champion usually knows the client, their wishes, etc - acts like a product owner (if you're familiar with the term from agile methods).

gaius• 13 years ago

I am reminded of the Syndicalists in 1930s Spain. After their mini-revolution, they abolished money, hooray! The next day they introduced "tokens" so they could get anything actually done...

eknkc• 13 years ago

I love almost eveything GitHub does, and the way they do them. I'm a GitHub fanboy. There it is.

I had a similar crush on Google, maybe 6-7 years ago, they lost me somewhere on the road.

reustle• 13 years ago

I think they lost a lot of us as soon as the G+ train started rolling.

kmfrk• 13 years ago

Speaking of the Google analogy, I hope they do more with their acquisitions of Speaker Deck and gaug.es, as I can't recall a single feature being added or improved since their acquisitions.

sillysaurus2• 13 years ago

The question is, do we matter anymore?

pekk• 13 years ago

The problems Google has have nothing to do with Facebook being better, or G+ being empty or whatever. Facebook is terrible.

RandallBrown• 13 years ago

Back in college around '08 my friend and I always used to joke that Google would buy a startup or company, then make their product free and better.

That doesn't seem to happen much anymore.

ldng• 13 years ago

Nop, it does not. They're walling their garden. Many thanks Mr Job.

Youtube comment only available if you're logged in for instance.

Actually, nothing that new here. They wanted to build the Google distributed "computer" since early on. Except some of us thought it would be open like the PC. It's now all closed like a Mac

yapcguy• 13 years ago

What's the big deal about Github? Too much hype.

Whenever you hear about the number of repos, most of them are just forks. In fact, I've created many by accident because Github changed their UI so clicking the icon instantly starts a fork instead of asking for confirmation first. I guess it's good for bumping the numbers up when getting private equity investors interested (cynical, I know).

Security wise, meh. Why would anybody pay to put their closed-source proprietary code onto Github? More holes than a sieve with their Rails setup.

Ok, you want somewhere to stash some private repos, nothing confidential. So why would you pay for Github when you can use Bitbucket for unlimited private repos? Plenty of other providers too.

Their enterprise product? Expensive meh. Atlassian's enterprise Stash is shared/open source when you get a license. You can also just install Gitlab, Gitorious, etc. for free on your own servers.

Github fanboys get excited whenever a big name dumps some code onto Github, like when Linus copied the kernel code over, but it was just a dumping ground while they sorted out server problems of their own. Ubuntu uses Bazaar and Trac, Linux Kernel development uses git on their own servers, KDE is over at Gitorious. Heck, there are some people still using Darcs and SVN.

Github is popular but it's not essential unless crave social karma points and your name/handle in lights.

Sorry for the rant like writing, lots of thoughts above, but I just never saw what the big deal about Github was. A good business and a slick offering but it's not going to suddenly transform your software projects or your abilities as a programmer.

jmathai2• 13 years ago

A good business and a slick offering but it's not going to suddenly transform your software projects or your abilities as a programmer.

I'd wager a pretty penny that Github has exposed a few orders of magnitude more people to the actual source of open source software than its predecessor, SourceForge.

Edit: grammar

techpeace• 13 years ago

The big deal about GitHub is that they were in large part responsible for the current Open Source Renaissance we currently find ourselves in. Do you remember the world before? Where you would hunt for someone's private SVN repo on their personal site and email them patches, only to hope that something comes of your effort? Compare that with the world of today, where, if I'm using GitHub, I can submit a fix to a project I happen upon entirely from my web browser, if I wish. The maintainer is notified immediately, we can have a line-by-line discussion about the change, and then it can be merged - also entirely from the browser, if that's what you're into.

Introducing DVCS and the pull request model to the wider development world is the reason that the pace of OS software development has accelerated so much in the past half decade. Of course, GitHub did not invent git, but they made it easy to host your repositories, and once Rails migrated to GitHub, the rest of the Ruby world came with it - followed shortly thereafter by a sizable chunk of the wider web development community. If you use one of their many competitors, that's fine - but they wouldn't exist had it not been for GitHub. Remember that Bitbucket started out as a GitHub clone for Mercurial. Gitlab is unashamedly an OS clone of GitHub.

More holes than a sieve with their Rails setup.

This is just unnecessary FUD. The GitHub and Rails teams have both been fast to deal with security issues as they are raised. Just because a security issue is well publicized, it does not mean that every other piece of software ever written does not have its own security issues.

namuol4• 13 years ago

You're misunderstanding what "problem" GitHub is trying to solve.

Remember their slogan? "Social Coding." No other platform out there has the userbase, let alone the sheer usability of GitHub.

There's a reason GitHub fanboys exist: it's a service that's delightful to use.

willejs• 13 years ago

Dear github, please sort out your: - Intermittent angry unicorns - Slow data transfer (150kb/sec?!) - Poor support - Expensive enterprise licence, why don't you host a better service and i will pay for it?

absconditus• 13 years ago

Their enterprise license is pretty comparable in price to something like Perforce, which does not include hosting. Do you also find Atlassian's Stash pricing to be high?

WestCoastJustin• 13 years ago

If you are so fed up with them, why not host an internal Git service with gitolite for user access rules [1]. You could even add something like gitweb as a web interface for git/gitolite [2]. Or, better yet, use something like GitLab [3].

[1] http://sysadmincasts.com/episodes/11-internal-git-server-wit...

[2] https://git.wiki.kernel.org/index.php/Gitweb

[3] http://gitlab.org/

RyJones• 13 years ago

We used gerrit at amazon and my current employer and it seems to have plenty of enterprise features out-of-the-box.

https://code.google.com/p/gerrit/

__chrismc• 13 years ago

Why do you rate GitHub's support as "poor"?

voltagex_• 13 years ago

Few things to help diagnose the slow transfer:

* What protocol are you cloning over?

* What connection are you cloning over?

* Where is the connection that you are cloning over?

grandalf• 13 years ago

I can't wait to try Github/REDACTED

Raphael• 13 years ago

Which one? It's confusing to have 2 teams with the same name, github:[redacted].

adamb_• 13 years ago

Please remove "[video]" from the title.

As of right now there's no video (there's a placeholder saying it's coming soon.)

fn• 13 years ago

That's funny. I didn't click on the link cause I thought it was a video, until I saw this.

nocman• 13 years ago

Actually, I'm kind of glad that the link said "video". Because I might not have clicked it if it didn't, and I almost just closed the tab when I saw the video wasn't available.

Amusingly, I'm also glad that the video wasn't available yet, because I clicked through the slides, and I suspect I got the gist of the talk at just about the right density for my interest (and it took very little time).

Funny how things work out like that sometimes.

bado• 13 years ago

When I submitted the link, the "[video]" wasn't in there; it was added by someone else later (when it hit the main HN page I presume).

serf• 13 years ago

[Video] coming soon.

So.. the topic is wrong?

Maro• 13 years ago

I was at the talk. It was great. Good job!

eliben• 13 years ago

Nice presentation, thanks for sharing!

Favorite quote: "your product should be cutting edge, not your tech"

lucasnemeth• 13 years ago

That guy knows how to make good looking slides.

rurounijones• 13 years ago

The one thing that would make it better would be to black-line border the text I think so that it stands out on the black AND white parts of the slide.

cenhyperion2• 13 years ago

Wow, I'm taking notes on the slide design. Looking forward to the video.

z999• 13 years ago

He has an earlier blog post on how he designs his slides. Not extremely informative but csn give you couple of tips on how to get that zach holman look. Iirc it was in response to the how github uses github to build github talk.

cenhyperion2• 13 years ago

Here's the blog post I think you're referring to.

http://zachholman.com/posts/slide-design-for-developers/

canistr• 13 years ago

I'm always skeptical when companies say they have "no managers" and are a completely flat organization with 100+ employees. I don't pretend to fully understand organizational theory, but I take it that at some point, there will be someone doing the "management" work. Whether that's someone with the title of ninja/hacker/product guy/project manager/director/CEO/BDFL

mpyne• 13 years ago

If nothing else Dunbar's number will creep in somewhere. And of course the whole reason we don't use flat hierarchies unless the company is really small is because of the explosive growth in the company communications graph.

People could form ad hoc teams to reduce the exponential growth (instead of having such teams pushed on them by management), but then it's not a flat hierarchy anymore. Instead it's a self-organizing construct which may or may not be better than actually designing a proper org. structure.

bct• 13 years ago

Somebody needs to do the management work, but they don't need to be "higher" in the org chart than the people they're working with.

The best managers I've had are more like facilitators than bosses.

ben_straub5• 13 years ago

(Disclaimer: Hubber.)

Here's the thing about "management work": it turns out you don't necessarily need management to do it.

Setting priorities? This can be done through consensus.

Making sure schedules are met? Don't have schedules. (1)

Hiring? Have everybody do it.

Giving out raises? Set up a deterministic system and forget about it.

Giving feedback? Everybody can do this.

Running meetings? Don't have meetings.

Communicating with the other parts of the company? We have technology for this.

1) EDIT: forgot my footnote. Of course, we're in sort of a fortunate situation here, and not every company can just not have a schedule.

cpeterso• 13 years ago

Making sure schedules are met? Don't have schedules. (1)

Did you forgot a footnote (1)? I'm curious to read more.

techscruggs• 13 years ago

What is your deterministic system for determining salary?

lttlrck2• 13 years ago

Works well until sh*t hits the fan and you need to cull teams or have a very angry customer. The buck really does need to stop somewhere, it cannot be distributed.

brown9-2• 13 years ago

I'm skeptical when people claim that their lack of managers mean they don't have a hierarchy. If there isn't an official hierarchy there will still be an unofficial one in some way - founders and early employees with more esteem/authority, etc.

ben_straub5• 13 years ago

Spot on. What you're describing is leadership, which is pretty deeply wired into human nature. We don't try to avoid this; rather, we want to recognize it as a discipline, and help people become better at it.

Authority, however, is totally different. It's the power to make someone do something they don't want to, and we try our darnedest to avoid that kind of situation. (Also note that "having authority" is different from "being authoritative".)

wffurr• 13 years ago

Anybody else made sad by

Open positions: Technical Account Manager

or just me?

Or it's like my company, and they don't bother to list positions for software engineers because they're hiring those all the time.

timc3• 13 years ago

What I get from these really great presentations presentations/posts/slidehows that Zach creates is that obviously GitHub is an interesting place right now, and Zach has got some really good insight but I can't help feeling that Zach really needs to get some experience with working at few more companies for some of the points he makes. Companies that succeed in different ways, companies that get bought out, companies that fail.

From the outside GitHub is in a really interesting space in that they can dog-food their own product very successfully, they have a huge market and intellectual share, and he writes as if this is the way that all companies could operate - but they simply can't.

For instances some companies have to play safe with what they say and do because they are in regulated industries, some just operate by people that have other values or outlets for their own time. I personally think it's fine if a company that I use has given up blogging or whatever to spend more time on creating a great product, or doing that while enjoying friends and family. I will leave/stop buying when the company no longer creates value for me, not when they stop talking at events or the original founders cash out because some people are born to create new interesting companies but are not suited to the 100+ person growth.

Zach if you are reading this, keep up the good work, but damn your writing is going to be interesting in 10 years time with a few more companies under your belt.

d0m• 13 years ago

Funny, couple days ago I was wondering why zach stopped talking about the github internal.. which I found illuminating and highly entertaining, and then, a couple days later, boom an amazing presentation.

If I may add a question: How do you manage to keep a consistent design across all teams considering that there's no manager and that most teams have different designers?

holman2• 13 years ago

We keep a styleguide (which is actually public: https://github.com/styleguide) that we loosely use across many products and apps. Beyond that, we have two libraries (one CSS, one JavaScript) that we pull into projects that help us maintain a general feeling of consistency.

Beyond that, designers get passed around the company a lot. Typically they can implement the frontend quicker than the backend can be finished. This has an interesting and unintended side effect of getting designers working all across the company. I think this plays a large role in getting a lot of the consistent feel across different projects: there's a lot of transfer of taste across the company.

rschmitty• 13 years ago

"Our tech stack shrinks as we age, fewer trendy languages and databases, your product should be cutting edge, not your tech"

I'm glad this was said. I find it hard to quell the excitement about new FOTM language/framework/database in our internal team and always wanting to use the hot new stuff that HN is raving about.

northisup• 13 years ago

Not actually a video.

fragmede• 13 years ago

So... you guys are relying on a bot on a 3rd party chat system to do live-deploys?

Does that scare the crap out of anyone else?

How GitHub No Longer Works

Related Stories