Open-sourcing Riskquant, a library for quantifying risk

https://netflixtechblog.com/open-sourcing-riskquant-a-library-for-quantifying-risk-6720cc1e4968

by el_duderino • 6 years ago

213 55 6 years ago

netflixtechblog.com

fancyfredbot• 6 years ago

It doesnt seem to have an input for correlation between the loss scenarios? That would affect expected loss a lot.

rubyfan2• 6 years ago

This seems like applying insurance modeling to security. Is this a new way of looking at risk or a reinvention?

im_down_w_otp• 6 years ago

Looks like parts of FMEA w/ some specific measurements to inform occurance & severity.

Eridrus2• 6 years ago

This is part of an attempt to make information security risk modelling more quantitative that has been going on for a few years. There's very little in the way of data to really back most of this up, but actually putting numbers to things is significant progress IMO.

rubyfan2• 6 years ago

Agree it seems like a better way to inform decisions and manage risk. Is it backed by any sort of real understanding of litigation, settlement, statutory experience? I often hear reputational risk cited by security teams at public companies... it usually follows some sort of indecision or appeal to a higher level of management or beuqacratic tool.

TeMPOraL• 6 years ago

Neither; the idea has been around for many years now (source: worked professionally on exactly these calculations this library does, in the same space). It's just basic probability theory and Monte Carlo simulations; nothing anywhere near as complex as the quants/insurance folks do.

IME, the biggest problem in this space is: on the one side, you have all kinds of metrics related to hardware, software and operations in your company; on the other side, you have FAIR and related ideas that let you model scenarios in a sane way, and let you plug in those statistical methods. But, there's no good idea how to connect these two sides. There is a vast gulf between what you can reliably and accurately measure, and what you actually want to know. It's hard to cross it while maintaining any kind of accuracy or statistical validity. I've been doing work on this "middle layer", but never got the chance to test it out in the open, as the company I worked for suddenly imploded.

(If anyone is working in this space and is accepting remote workers, I'm available for a chat.)

leecb• 6 years ago

Requires registration to read.

Tab closed.

muglug• 6 years ago

If the authors are reading, I think you mistakenly marked this as a paywalled article. I can't believe this was done deliberately from someone on the Netflix side.

thenightcrawler• 6 years ago

Nassim Nicholas Taleb is enraged by this!

rq1• 6 years ago

Why would he be?

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

syndacks• 6 years ago

Why would he be?

Read Black Swan, by NNT. His thesis is that the most significant impacts on an individual and/or a collective are the function of outliers or "Black Swans". In a Malcolm Gladwell kind of way, he uses a bunch of anecdata to prove his thesis. It's a pretty good book I suppose, but it was also before the all the faux-philosophy Malcolm Gladwell type books became a thing.

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

What exactly do you mean by that? In all sincerity, I think you misused an idiom or at least didn't make a clear connection as to why you chose that one.

batterseapower• 6 years ago

If you like the idea of this library you'll probably like the book "How to Measure Anything" by Douglas Hubbard (https://www.goodreads.com/book/show/20933591-how-to-measure-...). It's all about how to get sensible confidence intervals for things that are often considered unmeasurable such as the value of IT security. The book mostly uses Excel to do this modelling, but it looks like riskquant would be an excellent alternative on that approach, that for the more technically minded practitioner.

jacques_chester3• 6 years ago

The relevant book for this is Measuring and Managing Information Risk: A FAIR Approach by Freund and Jones[0].

Both books are worth reading; Hubbard's influence on FAIR is noticeable and positive. FAIR has the advantage that it comes with a fairly built-out ontology for assembling data or estimates. The OP touches on the top level (Loss Event Magnitude and Loss Event Frequency), but the ontology goes quite deep and can be used at multiple levels of detail.

The calculations are not difficult, I've implemented them twice in proofs-of-concept, including one that produces pretty charts.

The difficult part, to be honest, is that developing good estimates is difficult and frequently uncomfortable and the gains are not easily internalised.

Additionally, serious tool support is lacking in the places where it would make a difference -- issue trackers, for example.

[0] https://www.amazon.com/Measuring-Managing-Information-Risk-A...

edit -- Another good book in this area is Waltzing with Bears by DeMarco & Lister. A short, funny, insightful read, as you'd expect from the authors of PeopleWare: https://www.amazon.com/Waltzing-Bears-Managing-Software-Proj...

lifeisstillgood• 6 years ago

I regularly put a risk / loss / impact assessment into my issue tracker tickets - it's not the tool support is not there, it's that

a) everyone else needs to do this across the board

b) it's still just a guess - normalising my guess and you're guess is hard

thanatropism3• 6 years ago

I have that book. Basically I’m the last in a giveaway chain and can’t honestly recommend it enough that someone should lug it home. Next time I move it’s going on the trash.

It’s really not very good, even for executives who shouldn’t care for technicalities. The best thing are the calibration exercises. But my advice is, skip this one.

qznc• 6 years ago

Can you elaborate?

I'm half-way through it. I know most of the general stuff already but my knowledge is from lots of sources I mostly forgot. This books seems to be a good collection for this topic. At least, I don't know any substitute.

mindcrime4• 6 years ago

It’s really not very good

Compared to what?

But my advice is, skip this one.

And read what instead?

Not trying to start an argument here, I'm genuinely curious, as I consider How To Measure Anything to be one of the best books I've ever read (and I read a lot of books), and I recommend it highly to, well, pretty much everybody. If you feel that there's a better resource out there that relates to these topics, I'd be curious to know about it.

caseyf7• 6 years ago

I wish there were detailed scenarios of how they were using it.

Dowwie2• 6 years ago

Focus on the stress tests and worst case scenarios, Netflix.

Jugurtha2• 6 years ago

I suppose this is the impetus. I think that this gives them a way to prioritize which issues to fix in a better than the arbitrary hunch way. We assign "weights" all the time to issues either based on expected "time to complete" or on the severity of not fixing this. However, someone's severe is another's meh. This could give a score in a systematic way and help alleviate the guesswork.

Dowwie2• 6 years ago

Risk is managed differently in places that have experienced actual loss. No need to try to educate me about weighted decision making.

sheerun• 6 years ago

Can someone explain to me why Netflix puts their posts behind a paywall?

nurettin• 6 years ago

It looks as if they just registered for medium.com and slapped their domain on it.

pps43• 6 years ago

Not clear why the losses should follow lognormal distribution rather than some less pleasant to work with distribution like Cauchy.

Also not clear why loss frequency is assumed to be independent from loss magnitude.

Instead of "percent chance that the loss will exceed X" (VaR), expected shortfall would be more useful.

jacques_chester3• 6 years ago

The distribution comes from PERT 3-point estimates.

LEF and LEM are built of lower factors that roll up, which to my mind at least seem independent.

I'm not sure what you meant by expected shortfall.

bernardv2• 6 years ago

The choice of distribution is subjective, especially if you little data to start with and are going on theoretical grounds. Agree on expected shortfall/TVaR.

juskrey5• 6 years ago

You can quantify in normal domains e.g. probability of seeing a human over 2.05m tall. You CAN'T quantify tail risk in unbounded (for practical purposes) domains, period. This is just ridiculous: "For this example, there’s about a 2% chance losses would exceed $60 million in a year." If you see you can lose everything, you are not quantifying, you are simply not going that way. If you don't know if you can protect yourself from losing everything, you are not going that way. If you can afford losing some limited amount you just write down the loss from the very start. That is that simple.

Netflix is just going straight to hell with such prediction approaches. And, mind that, this is not a prediction, this is a fragility statement.

bernardv2• 6 years ago

Oh yes you can. The insurance and reinsurance actuaries, and regulators, have done this for a very long time.

juskrey5• 6 years ago

Insurance and reinsurance just clip the tail contractually, they never take "infinity multiplied by some small percent" risk! They don't go underground when their client does.

maxymajzr• 6 years ago

I see an interesting library, risk quantification (relates to what I work with).

I click the link.

Sniff/Accountwall stops me. I hit ctrl+w before I even thought of doing it (yes for muscle memory!)

For people who went through the trouble to see the content, is the library worth it?

These days, anything behind paywall/sniffwall/must-be-authenticated-to-read-this-wall warrants a ctr + w from me.

Chris20486• 6 years ago

I feel "riskquant" is a little vague for a project name. It's like there are a million internal libraries called "riskLib" - "risk" is a very general concept!

veeralpatel9792• 6 years ago

Link to the GitHub repo: https://github.com/Netflix-Skunkworks/riskquant

veeralpatel9792• 6 years ago

I chatted with Markus de Shon, one of the creators of riskquant, a few days ago. I would also check out Ryan McGeehan's writings on this at scrty.io.

I'd be happy to discuss this topic with anyone; my email address is in my profile.

Open-sourcing Riskquant, a library for quantifying risk

Related Stories