jnewland

This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:"

1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.

2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.

Home Assistant is super dope. https://github.com/jnewland/ha-config is my config if anyone's interested!

While we're here, http://www.nytimes.com/2015/03/31/technology/china-appears-t... also happened in 2015. Not a legal request to remove content, per-say, but something...slightly different.

Those tools are certainly useful! In the recent production incident I mentioned, we we able to quickly use SLOWLOG to determine which calls were causing the latency spikes. Thanks deeply for providing all of these useful tools and docs.

there is a percentage of users that will not read the doc and just deploy it

I wonder what this percentage is? I'd wager that it is higher than you might have anticipated, especially given the other comments on this thread.

As an engineer on a team ultimately responsible for the availability of a production service, it's my responsibility to ensure that the percentage of engineers that know the latency side effects of any Redis calls they make is near 100%. In the presence of such variable latency, any means of making that variability more obvious to all users of Redis would be a positive step towards happy users and operators.

This line jumped out to me as well. After a recent production incident determined to have been heavily influenced by several thousand O(N) commands being called against a list several orders of magnitude larger than normal, I'm thinking about this a lot.

I'm confident many other users of Redis are using O(N) operations in production against small datasets without knowledge of how much latency will be introduced by those operations when that dataset grows. This is exactly the kind of situation that makes me immediately skeptical when I find Redis in a emergent system design.

I'm considering what initially felt like a draconian means of remediation: using rename-command to rename all O(N) commands to BIG_O_N_$COMMAND to ensure everyone using them knows the possible impact and to allow for easy detection during code review and/or Redis latency spikes.

The more I think about it this approach, the more I feel that this should be the default mode of operation for Redis in production. SREs around the world would collectively save decades of time if the every engineer writing Redis queries to knew this fact by heart:

Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes.

I threw out that prototype soon after the talk. At the time, there weren't a lot of other engineers at the company doing Erlang, so maintenance was considered to be a long-term problem. I'm glad we made that call.

Hi cdibona! I'm primary on-call at GitHub today, and I wanted to say thanks to everyone that worked on https://code.google.com/export-to-github/ and worked with us to load test it before this announcement. It's really cool to see so much work put into making it easy for folks to easily move their data.

Step 1: Stop saying "cyber"

If you have a subdomain, there's no need to use Cloudflare - if you CNAME this domain to pkulchenko.github.io you'll use GitHub's CDN automatically.

That's exactly right.

What domain? I'll get an issue filed to stop sending warning emails in cases like this. Thanks!

Hey folks, Jesse from GitHub Ops here.

First off, if you use a DNS provider that has support for a ALIAS records or something similar, pointing your apex domain to <username>.github.io will ensure your GitHub Pages site is served by our CDN without these redirects.

I wish we could provide better service for folks without a DNS provider that supports ALIAS domains in the face of the constant barrage of DDoS attacks we've seen against the IPs we've advertised for GitHub Pages over the years. We made the decision to keep DDoS mitigation enabled for apex domains after seeing GitHub Pages attacked and going down a handful of times in the same week. It's a bummer that this decision negatively impacts performance, but it does certainly improve the overall availability of the service.

FWIW, we considered pulling support for GitHub Pages on apex domains about a year ago because we knew it'd be slower than subdomains and would require DNS configuration that would be challenging and frustrating for a large number of our users. However, we ended up deciding not to go that route because of the number of existing users on apex domains.

We use https://github.com/soundcloud/large-hadron-migrator/

My wife is pretty convinced I brought this upon myself with my comment on that post. Lol, you never know.

1. IO and CPU load spiked so much that the system was basically unresponsive over SSH. We think it was due another Xen VM swapping out of control.

2. Was 10 seconds with a 10 second timeout (way to low to run `xm list` in a loaded situation). It's now 90 seconds with a 90 second timeout.

has anyone out there actually used or evaluated clustrix? i haven't been able to find anything on the internet about this thing that hasn't come straight from the company.

tl;dr version: http://files.jnewland.com/stonebraker-20101021-200946.jpg

For those on OSX, I just hacked together a homebrew forumla for membase beta 2 yesterday:

http://groups.google.com/group/membase/t/69a47f80a2b772b8

You can't gain any more cycles, of course, but you can gain throughput by adding processes to an already cpu-bound app by giving faster incoming requests a place to go instead of sitting on the global queue while waiting for other requests to process.

Excellent point, Jeremy. This article didn't talk at all about the CPU/RAM balance, which is a important to consider when scaling any application. In my experience, a large number of Rails apps are more RAM bound than CPU bound due to large gems/libraries/codebases.

Having additional Passenger processes available even when CPU bound will allow for greater throughput (if at a less-than-ideal speed) than having requests back up on the queue, however - especially if those requests spend a good amount of time waiting on DB, memcached, or other external API calls.

Nice, I'm really looking forward that to that. I'd love to hear about the transition too - big moves aren't easy, as I'm sure you guys know.

> From what I understand, we're at the data migration point, and we're waiting on EY for that.

s/working directly with/delaying/

http://support.github.com/discussions/site/721-network-feed-...

not currently, since the underlying configuration engine, Puppet, doesn't support rollbacks

I've been using git-wiki recently, with great success:

http://github.com/jnewland/git-wiki/tree/master