Github and Engineyard part ways

http://www.engineyard.com/blog/2009/engine-yard-and-github-transition/

by jcapote • 17 years ago

157 95 17 years ago

mitchellh• 17 years ago

Does anyone have any idea what host github is moving to?

jherdman• 17 years ago

It doesn't actually say Github is moving. It just says that Github isn't going to be supported by EngineYard in the same way. It's entirely feasible that Github remain at EngineYard.

Edit Woops. Looks like I missed the key part of a sentence there. Sorry folks.

judofyr2• 17 years ago

Accordingly, we are currently working directly with their next provider to maintain smooth service as they cut over.

jcapote2• 17 years ago

"Accordingly, we are currently working directly with their next provider to maintain smooth service as they cut over."

"Provider" really sounds like they are talking about infrastructure

teej• 17 years ago

That's exactly what the article says:

"However, in the end, the best arrangement for them was moving."

pjhyett2• 17 years ago

We're moving to Rackspace.

judofyr2• 17 years ago

Cloud or dedicated/managed?

datums2• 17 years ago

that was my guess, you guys have a big dev following, rackspace has marketing money that can be reallocated to hosting github.

good luck

lsb3• 17 years ago

What FS will you use? SSDs?

jasonjwwilliams2• 17 years ago

That'll be great...and hopefully address one of concerns about having our private repos sitting on ATAoE storage.

seiji2• 17 years ago

I think they could have worded this better: "GitHub offers the largest free storage quota among the big SCM hosters, and we came to the conclusion that we didn’t want to subsidize that quota for non-Ruby developers."

It sounds odd Engine Yard hosts an entire for-profit company for free just in exchange for some complimentary accounts.

My imagination tells me GitHub is tired of being slow and Engine Yard couldn't do anything more to help them. Instead of the news being "GitHub Leaves Engine Yard Because It's Too Slow" the news is "Engine Yard Kicks GitHub To The Curb Because GitHub has Non-Ruby Repos."

I don't see either headline being positive towards Engine Yard.

jeremymcanally• 17 years ago

> It sounds odd Engine Yard hosts an entire for-profit company for free just in exchange for some complimentary accounts.

Well, sort of. It's really a value trade-off. EY offers its customers something that would cost them $50 a month, so it's costing Github (customers) * $50 to offer that. They exchange the lost value for added value in hosting (i.e., they're "losing" a few thousand a month in free accounts for our customers, so we'll offer them free hosting to continue to offer that).

It's simple barter.

jhancock3• 17 years ago

Very excited to see GitHub continuing to grow and EngineYard and GitHub handling their barter deals so cordially. Both companies are class acts!!!

jcapote2• 17 years ago

That assumes that Github is slow because of their hosting (Engineyard) and not because of how it's written.

I dunno, it could go either way for me: "Engineyard drops Github for being too costly" vs "Github needs better hosting, so we're dropping Engineyard"

jhancock3• 17 years ago

Its probably more about costs for both sides. I doubt that EY is _slow_ without regard to cost. Just that instead of rewritting GitHub's web app to deal with inherent performance issues, it makes more sense to throw more hardware at the problem. There does come a point where owning more of your server stack makes more financial sense.

conoryoung• 17 years ago

It's like when your boss let's you "resign" so you don't have to tell a future employer you were fired. EY was fired.

fnid4• 17 years ago

How can github fire EY when EY was giving them free stuff. Why doesn't github just pay EY for the services they are using?

This is another failure of Freemium. GH is only able to provide such large repos because they get free infrastructure.

I can't really blame EY for bailing. The partnership is no longer equitable. GH is getting something for nothing. I shouldn't say nothing, it's something, but it's not enough.

fizx• 17 years ago

Well, EY could make Github fast again by providing more resources. But then, someone would have to pay for the additional resources. EY doesn't think it's worth it, and Github can get the additional resources elsewhere for cheaper (EY is expensive, because you pay for premium support).

It seems like it's more about the fit between the companies, than any particular performance issues.

noodle• 17 years ago

> It sounds odd Engine Yard hosts an entire for-profit company for free just in exchange for some complimentary accounts.

why? its fairly high quality, targeted advertising with a built-in demonstration.

anonysock• 17 years ago

Ok, as far as I can tell, the real story is that Engine Yard relies on a GFS/SAN setup that doesn't scale in the unique way that Github needs.

If you think about it, Github is one of the few sites that actually directly uses the filesystem heavily. Everyone else hits scaling issues on the DB first.

  The sad thing of all of this is it's not really a matter  
  of scaling, and it never has been. Our bottleneck has 
  always been the file system. GFS just... sucks. I'm sorry, 
  but I have to say it. Case in point, your graph. The first 
  rebuild I ran timed out because of GFS. The second one ran 
  fine, took maybe a minute to process, if that. GFS impacts 
  everything... gem build failures due to cloning... GFS. 
  Network graphs taking long time to build... GFS. Caching 
  jobs not completing... GFS. I think you see where I'm 
  going here. There's no plans to deploy the new code to the 
  live servers, and I think the reason is that we're afraid 
  it'll make GFS performance worse, not better. But on the 
  new servers where we don't have to fight GFS, it's 
  amazing.

ezmobius• 17 years ago

Funny thing is that we told github that gfs would not scale for them over a year ago, we also outlined how to move to a shared nothing chunk server architecture. They didn't take our advice so it's mostly their own architecture decisions that were holding them back with regards to gfs.

Anyway there seems to be plenty of airchair quarterback on this one. The real story is that we can't afford to host them for free anymore.

durin42• 17 years ago

It's also inaccurate: GitHub doesn't offer the largest free storage quota. That's still Google Code, which offers 1024 mb/project, rather than 300 mb/user.

antonovka2• 17 years ago

Google is also happy to bump up your quotas if you have a genuine need.

omouse2• 17 years ago

Don't offer a free storage quota or limit how many projects you can host. There, problem solved. The people who leave will no longer be a drain on your servers and the people who stay will be paying for the maintenance of those servers.

matthewcford• 17 years ago

That would kill the majority of open source projects hosted on github, free hosting for public repos is one of the reasons it's so great. I'm willing to pay for private repos, but if I had to pay for my public ones too, I'd just not host them on github.

DannoHung• 17 years ago

I'd be willing to pay Github for some other premium service besides private repos (they're not of much use to me, tbh).

Uhh... can't think of anything off the top of my head that I really need or want though... OH, wait, I know, how about Bespin integration?

omouse2• 17 years ago

Maybe those open source projects should charge a fee as well?

bestes• 17 years ago

I'm a paying GitHub customer and having big projects like Ruby on Rails and Perl hosted there makes me very comfortable. I figure if they can handle the big stuff, they can handle me! Plus, I use as much free software as everyone else and am happy to help out indirectly (by being a paying customer).

I expect prices will rise soon as well. I'm not sure anyone has mentioned this yet, even though it seems inevitable. Still, I think GitHub is fantastic and I'm planning to increase my usage.

oomkiller2• 17 years ago

The reliability will need to go WAAY up before they can even think about touching prices, I'm being about as patient as I can be for spending $$$ per month on it.

stephenjudkins3• 17 years ago

When evaluating hosting providers, the quote Engineyard gave us was tremendously expensive.

They offered a great deal of value-added services, but we found that it would be cheaper to go with the second-most expensive option (Amazon EC2) while also hiring a qualified full-time sysadmin.

Whether this was a wise choice or not given our experience with EC2 is up for debate. But even if we had gone with Engineyard and had no problems at all it would be hard to justify a couple developer's salaries for it.

lsb3• 17 years ago

Just out of curiosity, what were your options that were cheaper than EC2, and why'd you reject them?

Periodic• 17 years ago

Please, write it up and post it to HN! It deserves its own discussion.

stephenjudkins3• 17 years ago

There were several hosting services that were a good deal cheaper. It costs $576/mo to run an extra large EC2 instance fulltime, while purchasing a server with similar specs only costs $2k or so. The data center across town could give us an equal number of server power for much cheaper. We're in our own data center right now, in another city, and we simply have a contractor we pay to go in and occasionally replace broken equipment. It really is quite cheap to do it this way.

Various hosts offered greater degrees of managed services at various prices. None really offered the flexibility that EC2 offered, or offered that much over simply colocating our own servers. With less traffic or fewer servers I can see many of these hosts being a great deal.

There also were several places that gave us completely screwy quotes that I can imagine were only meant to trick CEOs who don't know any better. One example is a place that offered us the use of an HTTP load balancer for the "low, low price" of $1000 a month. A lot of places offered expensive managed services, but when talking to the people who would serve us it was clear they didn't know what they were talking about. We did not seriously consider these places.

To Engineyard's credit, they were only managed host who sounded like they had knowledge that would be really valuable to us. They also made clear that they would devote significant resources to getting our site working smoothly. We would have paid a premium for this service, but not the premium they asked.

EC2 offers an amazing amount of flexibility that we found highly valuable. A sysadmin and developer spent most of a day diagnosing an issue with a DB server that ended up being a flaky drive. In EC2, we would have simply killed the server, fired up a new instance, waited an hour for its replication to catch up, and returned it to the rotation.

To return to EC2 we would have to drastically reduce the load on our MySQL instances, since we found that EBS volumes had somewhat unpredictable and not that great performance. Reducing our reliance on MySQL is something we're doing anyways, since on its current course it would have been difficult and expensive to continue scaling up in any data center.

marclove4• 17 years ago

Its worth noting that you get more than just EC2 + a full-time sysadmin with EY (speaking as an EY customer myself), and I don't mean this to question your own decision...that's something we all have to make considering the budgets we have available.

With EY, you get EC2 + several heavily experienced sysadmins who work every day on the exact stack that you're working with...they know it in and out like the back of their hand + the EY stack which has been battle-tested for several years now in all sorts of unique situations + performance that you couldn't replicate yourself without all that experience.

Again, not attacking your choice at all, I'm just saying that its hard to appreciate all that you're getting until you actually use the service and once you are, its easy to justify the costs.

stephenjudkins3• 17 years ago

Yes, Engineyard offered by far the best impression of competence and having sysadmins who would actually work well with us. We've heard great things from everyone we've heard of who uses them.

I've also heard great things about the Porsche 911 turbo from everyone who drives one. We were expecting a very expensive quote, but we were all surprised by how high it actually was.

datums2• 17 years ago

Bottomline, it's a business decision, I'm sure if staying with EngineYard was an option they would have gone that route. Moving to EC2 is probably a lot cheaper than other pricey, like RackSpace solutions. It's part of growing up, you sometimes have new friends, but never forget those you grew up with. I wonder how many new EY customer use their github accounts ?

davidw4• 17 years ago

> We identified the bottlenecks and supported GitHub and the community by making patches to ssh to allow key lookup in MySQL rather than a text file. That remains, to this day, one of the finest examples of Engine Yard support and it makes me extremely proud just thinking of it.

This seems... odd to me. It doesn't feel like the right boundary between businesses. From a hosting provider, I expect good, steady service, reboots, a root console, and that they'll fix anything that's on their end (hardware, for instance).

Patching ssh is development work, and is something I would expect to pay for to meet specific goals, but not something that comes as part of my hosting package. I mean, what if you are just cruising along, and don't need any deep hacking over a couple of months. Is your money being wasted? With actual developers, I could redeploy them to do other things. Can I do that with EY?

They seem like really good, sharp guys, but I don't quite get the business, I guess.

tedunangst• 17 years ago

Maybe the developers building apps that run on EY aren't the best developers to be patching ssh? Is your money wasted if you hire a lawyer on retainer and don't ask enough questions?

davidw4• 17 years ago

EY publicizes their Ruby expertise, not their 'hack on security-critical crypto code' expertise, and while it's impressive, and to their credit that they were able to do it, it still leaves me thinking that a fixed monthly fee is the wrong way to pay for that kind of talent.

> if you hire a lawyer on retainer and don't ask enough questions?

At a certain point, yes.

tmornini_ey5• 17 years ago

From our perspective, it was a matter of helping our partner and the Ruby community succeed.

We're working really hard to make Ruby on Rails succeed. This is just one of the many ways that we've pitched in to help it do so.

If, during that downtime, when the "Twitter's problems are Ruby on Rails" FUD was running high, would anyone believe that a Github scale failure wasn't a Ruby on Rails problem? We weren't willing to test it, so we solved the problem, i.e. we put our money where our mouth is -- That Ruby on Rails scales just fine, the bottlenecks are generally elsewhere.

davidw4• 17 years ago

> From our perspective, it was a matter of helping our partner and the Ruby community succeed.

I suppose what seems confusing to me is that there are a lot of things you could do to help your customers succeed, but many of them are fairly expensive - such as cutting your rates, or doing high end development work.

With more basic hosting, say EC2, I know that Amazon isn't going to do beans for me, so I'm on my own. I know exactly what the price does and doesn't include, and what I have to provide myself. EY seems fuzzier... it's almost like having an extra developer on staff in terms of talent, but it is and it isn't: you can't tell that person to go off and do something else.

Say I move one of my sites to EY; is your "whatever it takes" attitude going to include fixing up my ugly design/graphics work?:-)

tmornini_ey5• 17 years ago

Hey all. Wanted to chime in the discussion.

My blog post is direct and to the point. There's honestly no mystery about this, I laid out all the details, possibly excepting one.

We designed our infrastructure offering 3 years ago. We knew that the vast majority of websites would produce nearly read-only file I/O. GFS's less than stellar write performance isn't a problem in that typical case.

Along comes Github, and they have an entirely different disk I/O profile to the rest of our customers. Github was built on a shared-something architecture because GFS made it quick and easy to get up and running, developing features, and attracting users. This is a good thing!

Github has been very vocal about their dislike of the GFS filesystem we use for shared filesystem access. GFS doesn't scale forever, and we've never suggested it does. It could scale far larger than it has at Github, but fizx hit the nail on the head: we weren't willing to do it for free, and Github was unwilling to pay our price.

We warned Github many, many moons ago that given their growth rate, in order to scale their application smoothly and inexpensively, a shared-nothing architecture was eventually going to be needed. We offered to help them with that architecture, as we saw Github running wonderfully atop a cloud service such as EC2.

Rather than do the re-architecture now, they've chosen instead to move to a vendor who can provide them a high performance, high availability, non-commodity, proprietary network file server infrastructure. I suspect it will work well for them, and we'll all enjoy a faster Github. :-)

From my perspective, I really want everyone to understand that there's no bad blood on the EY side, and hopefully none on the Github side. Business is business, decisions need to be made each and every day.

defunkt• 17 years ago

GitHub has been working on a re-architecture since our last major feature release, which was GitHub Issues in April.

And we can't wait to roll it out.

tmornini_ey5• 17 years ago

Great news, Chris, sorry to misrepresent.

So, the new Github architecture is shared nothing?

seiji2• 17 years ago

I much prefer being talked to this way instead of hearing about "non-Ruby repositories."

liquidcool• 17 years ago

Personally, I like hearing about both the technical and the commercial reasons driving a business decision. The fact that GitHub did not serve Ruby projects exclusively (or primarily) was an important factor to EY - why should they neglect to mention that?

dschobel• 17 years ago

Agreed. That bit sounded a bit pathetic when the issues were clearly technical and not ideological.

Are we really to believe that EY would have facilitated GitHub's growth had the repos remained overwhelmingly Ruby?

rit• 17 years ago

This is a much more... reasonable explanation than "GitHub offers the largest free storage quota among the big SCM hosters, and we came to the conclusion that we didn’t want to subsidize that quota for non-Ruby developers.".

Which had me honestly scratching my head. Then again, I prefer Mercurial and Python, so to each his own I suppose.

rapind• 17 years ago

I applaud the transparency of EY here. This is far more honesty than you get from most hosting companies.

I'm also pleased that github is upgrading since I've noticed gradually decreasing performance for about a year now which prompted me to setup my own private repo server (but I missed the github front-end). Kudos to github for doing something about it.

jasonjwwilliams2• 17 years ago

Would have been nice to see a joint statement (or at least concurrent statements) out of EY and Github. This way it looks like EY is just trying to beat them to the punch...either out of bad blood or to get in front of any potential bad PR.

calambrac• 17 years ago

Jesus Christ, there's a lot of drama-mongering in this thread. Grow the fuck up, it's a business decision that both sides had an interest in seeing happen.

callmeed• 17 years ago

Does this mean I'll no longer get a free GitHub account as an EngineYard customer?

EDIT: N/M, found the answer.

Does this mean I can have a free GitHub account because I'm also a Rackspace customer :)

yesimahuman3• 17 years ago

Engine Yard and Github's business relationship aside, is anyone else sick of everything these days being about the tool or language people are using? For some reason we hear more from the Ruby side than any other about how great their language is, but frankly the only thing that matters are the products made from it. I don't know, I'm just kind of sick of Ruby fan boys.

adw• 17 years ago

I'm a Python guy, and that community's both sinned against and sinning. You have prominent Ruby guys claiming there's no momentum - Youtube, Friendfeed, Dropbox, loads of newspapers - behind the language. But, then, I'm chippy enough to notice...

It's the Perl community who just seem to get it done. Amazon and BBC iPlayer have more traffic than most of us, I bet!

carbon82• 17 years ago

"we hear more from the Ruby side than any other about how great their language is"

Um, that crown unquestionably goes to a big chunk of the python community right now. Reddit alone provides a steady stream of both extreme fawning over python and vitriol targeted at ruby, and your comment and its upvotes demonstrate that it's present here, as well.

It's also quite a colossal double standard to complain about Ruby-specific hosts considering the existence of GAE, all the django-targeting hosting companies http://www.google.com/search?q=django+hosting and all the companies providing language-specific services for PHP, Java and every other mainstream language.

yesimahuman3• 17 years ago

I'm not complaining about the hosts, I'm complaining about the extreme focus we put on the language being used rather than the products being made.

jwr• 17 years ago

I think it's rather funny that this would never be news if not for the keywords. I mean, a customer changes a hosting provider for business reasons. Yawn. Who cares.

But, drop in "Ruby" and "Git" and see it as front page news on HN with a discussion going off on tangents.

llimllib• 17 years ago

Silly question: the GFS referred to here is Global File System (http://www.redhat.com/gfs/) not Google File System, right?

tmornini_ey5• 17 years ago

You are correct, sir.

Github and Engineyard part ways

Related Stories