HN user

dmuino

1,143 karma

Daniel Muino

Software Engineer currently working for Netflix. Previously worked for Yahoo! and Inktomi on large scale infrastructure problems.

Posts42
Comments29
View on HN
www.thedailybeast.com 13y ago

DOJ drops charges against Aaron Swartz

dmuino
2pts0
www.rebelliouspixels.com 13y ago

Buffy vs. Edward Remix Unfairly Removed by Lionsgate

dmuino
177pts71
techblog.netflix.com 13y ago

Edda - Learn the Stories of Your Cloud Deployments

dmuino
4pts0
diegobasch.com 13y ago

Social Networks Implode Quickly

dmuino
45pts20
heather.cs.ucdavis.edu 13y ago

Programming on Parallel Machines; GPU, Multicore, Clusters and More - free ebook

dmuino
2pts0
www.infoq.com 13y ago

Rich Hickey: The Value of Values

dmuino
186pts42
www.businesswire.com 14y ago

ShopRunner appoints Scott Thompson as CEO

dmuino
1pts0
allthingsd.com 14y ago

Mayer to Get $40 Million in Overall Compensation for Yahoo’s Top Job

dmuino
2pts0
0x10co.de 14y ago

DCPU-16 Tetris

dmuino
1pts0
blakesmith.me 14y ago

Understanding clojure concurrency, part 2

dmuino
1pts0
www.clicktorelease.com 14y ago

Floating shiny knot

dmuino
1pts0
www.techdirt.com 14y ago

Al Gore Comes Out Against SOPA/PIPA

dmuino
169pts62
www.wagon-bertrand.fr 14y ago

South Park in CSS3

dmuino
11pts1
mbostock.github.com 14y ago

D3.js transition demo

dmuino
15pts1
github.com 14y ago

Elegant Weapons

dmuino
2pts0
mathoverflow.net 14y ago

Which math paper maximizes the ratio (importance)/(length)?

dmuino
2pts0
missingfaktor.blogspot.com 14y ago

10 Factor One-Liners to Impress Your Friends

dmuino
2pts0
docs.scala-lang.org 14y ago

Scala Style Guide

dmuino
2pts0
www.developerdrive.com 14y ago

New methods in jQuery 1.6

dmuino
3pts0
roycewhelan.com 14y ago

WebGL reflection: Emerald cube

dmuino
2pts0
a1k0n.net 14y ago

Html5 game of tron

dmuino
2pts0
www.bloomberg.com 15y ago

Apple Said to Consider Making Bid for Hulu

dmuino
9pts0
a1k0n.net 15y ago

Donut math - How donut.c works

dmuino
237pts21
lkml.org 15y ago

Re: Linux 3.0-rc3 "In other words: don't be stupid."

dmuino
38pts8
cassandra.apache.org 15y ago

Cassandra 0.8.0 has been released

dmuino
27pts4
newsandinsight.thomsonreuters.com 15y ago

How Yahoo won the Bedrock patent trial that Google lost

dmuino
1pts0
www.physicscentral.com 15y ago

The High Water Mark of American Science

dmuino
2pts0
www.acunu.com 15y ago

Big Dictionaries I: query/update tradeoffs

dmuino
1pts0
ocw.mit.edu 15y ago

Advanced Data Structures (MIT OCW)

dmuino
14pts0
herbsutter.com 15y ago

Herb Sutter's Effective Concurrency columns

dmuino
3pts0

Note that we didn't run stock freebsd. It was a custom freebsd4 with a custom gcc 2.95 toolchain. I don't remember all the mods, but migrating from 4 to 6 was a nice undertaking, and it was done in parts. People who worked there will remember the term "4 on 6". After the migration, the kernel was 6 (and you got the latest drivers) but we were still running those weird 4 user-land binaries.

Inktomi (YST) was acquired in 2003, later Overture (YSM) and both were linux shops. YST was the biggest property by far and we didn't have any kernel developers. We were doing just fine with stock debian, then stock RHEL. (I did a few very minor patches for 32-bit, like splitting user-space/kernel to 3.5/0.5G instead of 3/1G, and some caching improvements for /proc, but I wasn't a kernel developer). Linux just worked. Drivers were available and supported for the HW we needed. Oprofile, systemtap worked fine for the most part. We later hired one kernel developer to help Mail move to linux, but we always had more freebsd kernel developers.

For freebsd you needed to write the drivers and the tools. That took time and money. The community was also smaller and it was harder to hire people with experience. Acquisitions were running on linux too.

The decision was not easy. Y! already had great freebsd engineers. By announcing that linux would be the supported platform going forward it was a given that many of them wouldn't be happy and leave. Fortunately not all of them left. For example Peter W. still works there.

In any case I don't think there was really an alternative.

Even when Rick was at Y! the push to move to linux was there. Rick had written most of the base libraries for freebsd originally (mdbm, ylock, ylock_kern, yfor, etc.), and tools like the package manager yinst and ysar, but being the great engineer he is, he ported them to linux and got the same or better performance. (In the yinst case he delegated the maintenance to a very capable engineer)

Note that Yahoo! was employing a few very bright freebsd kernel engineers but when the decision was announced that linux was the future many of them decided to leave. That made the path forward even clearer. It wasn't just a matter of integrating new acquisitions easier, and the fact that it was easier to hire people with linux experience than with freebsd experience. That was around 2008 IIRC. I left in 2010, and Rick less than a year later.

Working at Netflix 12 years ago

Can you share the name of the recruiter? Just curious if that person is still working at Netflix.

I've been a happy Netflix employee for over 4 years and have never experienced a 'large layoff' event. All the layoffs I've witnessed were not a surprise for the people affected.

Senior engineers usually make more money than their direct managers, but I fail to see how that's an issue.

I'm guessing RayVR met with Patty who left Netflix 2 years ago. My interview with her was different but I guess it would depend on your background and what she was trying to determine. In my case it was mostly to see if I was going to be a good cultural fit since I was at the time working for Yahoo which had a completely different culture. To me that was an entirely reasonable thing to do. Actually the whole interview process was great, and it was one of the major reasons I chose Netflix over other companies.

I've listened to this audiobook and I was hooked very early in the story. Jon Ronson is a very good story teller and he narrates his book. He's quite funny and engaging. "A Journey Through the Madness Industry" is the subtitle, and "journey" is indeed a very adequate description of what you'll experience with this book. Jon wanders from place to place where he'll meet psychopaths and people who work in the "madness industry" and relates his experiences and thinking.

It's not a definitive treatise on psychopathy, just the adventures of the author as he dives into this fascinating world. Definitely worth getting it at this price.

I agree that the biggest issue is #1. Another problems I've run into are:

- Operations are not automatically aborted if the client closes the connection. Suppose a case where the client is sending many queries that are queued by the server (usually waiting for a lock to be released). Client times out, re-establishes a connection, retries, etc. Mongo will execute the operations even if the client is no longer waiting for the response. Operations are not automatically aborted.

- Sending slaveOk() queries need to be sent to the slaves explicitly by the driver, and I haven't seen it work reliably. Ideally you'd send them to mongos and let it send slaveOk() queries automatically to the slaves. All the drivers would be simpler and just work. But this is not the case. You can't do it unless sharding is enabled. Each driver needs to implement this functionality.

But I must say that overall my experience has been very positive. It's very developer friendly. You can go from not knowing anything about it to having a working prototype in an hour or so. The flexibility in querying the system is awesome.

You just need to understand the limitations of the system. Once you start pushing the limits it gets significantly more complicated.

I'm a Netflix employee. If you ask me I'll tell you in no uncertain terms that it's not a culture of fear. I've been here for one year and have only seen one person removed for performance reasons. Fear is never a driving factor in any of my decisions.

Employees ask tough questions during company meetings and our executives are incredibly open. I guess we've become complacent because their decisions had been bold and right before these past couple of months. I believe this is why there was not a stronger push back internally.

Los Gatos, CA - Netflix

I'm looking for a frontend developer for my team. Your main job would be to help develop a very interactive infrastructure monitoring system. We have lots and lots of metrics about our systems. We'd like to be able to present them in a useful format, make it easy to troubleshoot production issues, aggregate on different dimensions, slice and dice them, zoom in on interesting events, expand clusters into individual nodes, save your work into your custom dashboard, etc.

I'm mostly looking for a strong developer who knows javascript well (or a language that compiles to javascript) and is familiar with tools or frameworks that will make your job easier (think Backbone, Knockoutjs, Spine, JavascriptMVC, etc.).

Netflix is a great environment for engineers where the emphasis is on agility and there are no rules about what tools or technologies you use to get your job done. You know what's best and we trust you.

If this sounds interesting to you please email me at dmuino @ netflix.com

That's a very good point. Unfortunately is not as simple as having discrete states: 'healthy', 'degraded'. For example a tier might be deployed to an auto-scaling-group (ASG) running 100 instances. One of them is failing the health-check. Common sense dictates you can take action (kill the offending instance). What's the overall state of the ASG after this? Are you allowed to kill another instance that's having issues? How many after that? In general you want to put some rules into place. (Percentage of the ASG that's safe to kill, latency boundaries for the service, time to wait before taking the next action, etc.)

It wasn't one of the monkeys. Most monkeys only run when we have developers who could notice and fix problems. It also happens that our peak usage is while we're home and our quiet time is while we're at the office. That means, in general, the monkeys don't run on a Sunday night.

Unfortunately we're not running 100% on the cloud today. We're working on it, and we could use more help. The latest outage was caused by a component that still runs in our legacy infrastructure where we have no monkeys :)

ITerm2 1.0 Released 15 years ago

I use it with tmux. The main reason I use it is because of its full screen support and the unixy behavior for focus follows mouse, copy on select, middle button paste.

As I mentioned elsewhere Netflix does use Level3, Akamai, and Limelight. So the traffic is mainly coming from CDNs.

I'm not sure if by "content-aware routers" you mean something like Content Centric Networking:

http://www.ccnx.org/

If that's the case it'd be up to the ISPs to push that technology to their infrastructure. Van Jacobson gave a very informative presentation at Netflix about it a couple of months ago. It seems you can watch a similar presentation he gave at Google on youtube:

http://www.youtube.com/watch?v=8Z685OF-PS8

>I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people.

You can definitely have a good plan, and be able to execute on that plan, with fewer people. I was lucky enough to work in a team that was running the search backend for a big internet company. Tens of thousands of machines, constant code, config and data pushes, configuration management, os images, kernel, etc. The team was responsible for developing and running the tools needed to keep the site up even when a whole datacenter lost power and you had to shift load from one coast to the other. The size of that team was much lower than 10 people.

Obviously Reddit will have to invest in both infrastructure and human resources to get there, but it can be done.

You can do high availability deployments using AWS. AWS offers different availability zones within a region, and multiple regions if you're extra paranoid. Unfortunately the way reddit is using the available services is not ideal. IIRC they are using cassandra backed by EBS which is not a best practice. It's much better to run cassandra on local storage and have a backup system that's uploading incremental and full snapshots of your data to S3.

I agree with your overall point that having a business continuity plan is required. They should be able to answer questions like how do we minimize impact when we lose one cassandra node or two, ebs gets corrupted, a whole avail-zone disappears, etc. I'm just saying that AWS provides the building blocks for you to do that.

This is not the thing to test. DNS resolution time is important, but even if the google resolver returned results faster than your local ISP resolver it would still make your browsing in the general case slower. The problem is that the IPs for the resolvers have presence in very few locations around the world. I believe this number is just three at this time (Virginia, London, Taiwan). The big problem with this is the effect it has on CDNs and GSLBs. The DNS response you get for large sites, like www.bing.com for example, depends on the location of the machine asking the question. If you use your local ISP as your resolver then you'll get an answer that's close to you. If you use the google resolver you'll get an answer that's close to the google resolver and that will generate additional latency for all the rest of the browsing session. This obviously will affect all big sites that are NOT google. They're obviously aware of the problem since they have special cases for their own sites. If you have access to machines in different countries you can see the responses you get from 8.8.8.8 have good locality properties for google sites. So I'm guessing they're just forwarding the queries to their location-aware DNS servers. Obviously they won't do that for akamai, and others.

End result: you're farther away from the resolver, farther away from your web pages, farther away from your CDN. I don't see how you can make a case that this is an improvement in general.

If you're interested in cloud computing we have a few openings for developers. We're building a very large scale private cloud and we could use help all over the place. From the low level Xen foundation, linux kernel, and virtual machine setup and deployment, to the high level view of the system we'll present to the users (Yahoo developers).

Location: Sunnyvale, CA

Qualifications: You're a very good developer (in any language)

dmuino at yahoo.com

I leave my kindle at home most of the time. Sometimes, while at work, I wish I could take a look at a particular passage of a book. The iphone app is OK for waiting rooms, but is definitely not ideal for more than casual reading. Having a mac version would be very helpful for me.