imbriaco

We run rather a lot more than 2500 instances at peak and a variety of instance types dependent on game mode. Our waste is on the order of 5-7% depending on the time of day -- we can run less waste on the downward slope of the demand curve for instance.

We're also seeing some pretty compelling results from ARM based instances: https://www.wired.com/sponsored/story/changing-the-game/

It's very true in my experience, and we've looked pretty hard at all the options. Thanks for the offer of the slides, I have first hand experience with this at scale.

It certainly depends a lot on how sophisticated your autoscaling is and how closely you're able to follow demand to limit waste, how well you can manage per-host utilization, whether you are CPU or memory bound, and lots of other factors. But even at truly massive scale the cloud hosting option can be very competitive without nearly as much management overhead.

The vast majority of game servers are hosted on public cloud providers. They turn out to be a pretty perfect example of an ideal cloud workload:

- Very elastic demand curve

- Generally ephemeral with any necessary persistent state stored off-host

- Benefit from geographic distribution of cloud regions because they're generally quite latency sensitive

Totally agree, there's definitely an opportunity there if the surface area of what the system is doing gets smaller. The big difference today is that it's a lot easier to compose a debugging suite using additional tools on top of a traditional host-based runtime for now.

Definitely excited to see how the technology evolves over the next few years. It hasn't moved as fast as I'd have expected over the last 2-3 years but I'd love to see that accelerate.

No, I'm not moving any goal posts. I was reacting to the statement "Tooling/Perf/etc is not needed when running". I'm not sure where you'd get the idea that I'm anti-unikernel, I just wanted to not disregard that challenge since it does matter.

Tooling/Perf/etc is not needed when running (do you really want to debug in production) but tooling can be used in the process of development.

Whether or not you want to debug in production, reality often means that you will see things in a live environment that you will not see in other environments.

Unikernels are very interesting and have a number of compelling attributes, but let's not pretend that the current state of available tooling for troubleshooting, instrumentation, and general debugging isn't a challenge.

This is an asinine argument that seems to be driven by the belief that there is such a thing as efficient allocation of capital in the real world. Or perhaps just sour grapes that some of the unsophisticated investors they like to take advantage of are instead putting their money index funds.

Maybe both.

It actually is unreasonable for you to ask that. It's also a pretty bad take to bring up the 150,000 people die every day statistic as if the fact that a single person who many people had a connection to dying is not of significance.

If you truly don't care just move on. If you want to know more, it doesn't take a great deal of skill to use Twitter, GitHub, Google, and more to find out all that you want to know.

The Glowforge will only engrave from a bitmap. You need a vector to do cutting.

But MQDs.

This is extra delicious irony given the fact that the typical, and often only, requirement for these SDN layers is IP connectivity.

Yes. They've been hosting it outside of their production infrastructure for several years.

Hi, I'm Mark Imbriaco. That's my deck.

As I said in response to another comment in this thread, I don't often share my slides without the context of the recording of the talk that went with it. That recording isn't available yet, but these slides have more words than most of the ones I present usually do so I thought I'd share them.

In my talk, each of the main points was accompanied with an anecdote to give some background on the experiences that led me to believe them, as well as some expand on the content of the slide.

The talk was specifically built for an audience of attendees of the DevOps Enterprise Summit. It's a fantastic event with one of the most thoughtful and engaged group of attendees I've experienced, and I've been to a lot of conferences. These folks are, generally speaking, development or operations leaders at enterprise companies who are in the midst of a transition to a style of work that's a significant departure from traditional enterprise IT. Since much of what we call DevOps now is built on the way that many of us in the world of startups and Internet companies have been doing things for a long time, Gene Kim asked me to share some ideas that I think are universal.

It was one of the most fun talks I can remember since I was given the license to tell stories. ;)

Thanks, I appreciate that.

I generally don't like to post decks without the recording of the talk and I was kind of surprised this one got shared as widely as it seems to have been. I'm glad you enjoyed it.

I don't tend to recite my slides, so your prediction that the talk was more insightful was ... insightful.

Each of the slides was accompanied with an anecdote that attempted to both illustrate the point and to provide more background about why I think it is important.

If there was something specific you disagree with, I'd be happy to talk more about it.

Hi everybody, I'm the founder and CEO of Operable, developers of the Cog ChatOps platform. I'm happy to answer any questions folks might have about Cog or how we think about ops and ChatOps.

In a traditional RDBMS you would let the database enforce schema level authn if that was a concern, so wouldn't have to parse anything. You'd just connect to the database as the appropriate user and send the query.

That's irrelevant since we are specifically talking about the NC law.

The NC law was very specifically targeted at municipal broadband projects and was effectively written by Time Warner to prevent underserved customers from having another option. It's as simple as that and it's disgusting.

The judge conveniently disregards the difference between passive observation and active intrusion. Breaking into someone's computer is no different than if the officer had broken the blinds themself, which clearly would not meet the test described by Justice Breyer.

Google "Netflix downtime" for evidence that Netflix also has outages. Google has outages, sometimes very significant ones of Google Apps. Facebook has outages.

Complex systems fail. Period. All the time. Things like the Simian Army are fantastic tools that help you identify a host of problems and remediate them in advance, but they cannot test every combinatorial possibility in a complex distributed system.

At the end of the day, the best defense is to have skilled people who are practiced at responding to problems. GitHub has those in spades, which is why they could respond to a widespread failure of their physical layer in just over 2 hours.

The biggest win with the Simian Army isn't that it improves your redundancy. It's that it gives your people opportunities to _practice_ responses.

You're absolutely right.

We should collectively be using incidents like this as an opportunity to learn, much like the GitHub team does. Our entire industry is held back by the lack of knowledge sharing when it comes to problem response and the fact that so many companies are terrified of being transparent in the face of failure.

This is very well written retrospective that gives us a glimpse into the internal review that they conducted. Imagine how much we could collectively learn if everyone was fearless about sharing.

It turns out that it's even hard then. Complex systems, by their very nature, fail in unexpected and unpredictable ways. If that weren't bad enough, hindsight bias makes it way too easy for us to look back with perfect knowledge and opine "That was so obvious, how could they have missed such a rudimentary issue?"

If only things were that easy.

I don't know why I'm responding, but I can't help myself. I guess it's the good old XKCD "Someone is wrong on the Internet" effect and I should know better. But.

They are paying people who would otherwise work on Git in their spare time to work on it full time. That is a net win for everyone who uses git.

I'm going to give up at this point and just accept the fact that you're going to spin it such that any commercial entity that uses open source and doesn't open source their core product is a parasite. I think you're wrong, but you're certainly entitled to your opinion.

You're not entitled to your own facts, though.

At least the #2 and #7 (by number of commits) committers to Git are employed by GitHub. They're both very consistent contributors and have been for a very long time.

You specifically said that GitHub does not participate in the open source community and I think I've amply demonstrated that you're incorrect. If you'd like to continue to argue the point, I'd appreciate some facts.

That'd be a shame if it were true.

GitHub has 6 pages of public repositories (https://github.com/github) that you can peruse. To pre-empt your response, yes, they aren't all open source projects that GitHub has released. But a great many of them are. Some of their popular projects include things like Atom, Hubot, Boxen, Linguist, and many more.

They also have developers who contribute to a great many other projects as part of their daily work; Ruby springs to mind - Aman Gupta and Charlie Somerville have contributed quite a lot. And they pay at least two people to work effectively full time on git itself.

But don't let facts get in the way of your rant.

We're growing so fast that shutting down hardware is the last thing we would ever do. We have the opposite problem -- it's quite challenging to provision hardware fast enough to keep up with demand. :)

Rest in peace, Ezra.

Exactly this. It would have seemed pretty presumptuous to ask the Debian project to change the support model that they decided worked for them to support us.

We were also faced with a very large new infrastructure build and we knew we were at the end of the line with Squeeze but Wheezy had't quite yet shipped, which put us in an awkward position.

Debian is a great distribution, but at the time we felt like Ubuntu was the right way to go given the existing support policies.

I was part of the Ubuntu decision at GitHub, and it was based on the fact that Ubuntu gave you much longer support after the next release. Debian (at the time) only guaranteed even security updates on previous releases for one year after a new release.

That's incorrect. The way you apply bug fixes to 12.04 LTS is by installing the bug fixes, not by moving to 14.04. The whole point is that 12.04 LTS continues to get updates for 5 years.