Python at Netflix

https://medium.com/netflix-techblog/python-at-netflix-bba45dae649e

by luord • 7 years ago

440 234 7 years ago

1ba9115454• 7 years ago

Most of these use cases would be better served with a type safe language.

sametmax11• 7 years ago

I read a lot of data analysis, prototyping, configuration, monitoring and deployment, for which Python is very, very well suited. So I don't follow how you got there.

bouncing7• 7 years ago

I'm going to start a drinking game every time someone says that.

tclancy• 7 years ago

You'll want to get on a liver donor list now.

cuddlecake• 7 years ago

Will you start the game everytime someone says that, or will your game be to drink everytime someone says that? Honestly curious what your methodology is.

luord2• 7 years ago

It's been going on for sixty years (something the defenders of both paradigms often seem to ignore) and will never end, you'll die.

skywhopper• 7 years ago

This isn't a very useful comment in this form. What aspects of these specific use cases would be better served by a type-safe language? Why is type safety the most important attribute of language choice in this scenario?

throwaway555542• 7 years ago

Question: If they would be, don't you think Netflix would go that route?

"Better" is contextually dependent. There's no silver bullet to anything we do in tech.

jniedrauer• 7 years ago

don't you think Netflix would go that route?

Not necessarily. Python has a few huge advantages that aren't (directly) related to the quality or utility of the language. For example, hiring and training people to use Python is going to be easier and cheaper than hiring and training people to use C++. Additionally, as mentioned in the article, Python has a massive and healthy ecosystem of high quality packages.

And then there's the cost of porting an existing codebase to a new language...

inglor4• 7 years ago

You think you're being downvoted because people like dynamic typing very much.

You're being downvoted because:

A) You left a comment with a strong opinion without expressing it while dismissing other peoples' experience B) Python is in fact a type safe language for a while now https://docs.python.org/3/library/typing.html - you have pluggable "a la bracha" types.

enitihas• 7 years ago

I think you meant dynamic typing and not dynamic programming.

0815test• 7 years ago

Python is in fact a type safe language for a while now

Well it has a typing mechanism, which is better than nothing, but has the mechanism been shown to be safe? C has types too, but I'm not sure that it would be described as type safe.

philwelch3• 7 years ago

Python’s type annotation mechanism is somewhere between afterthought and joke. Nobody uses it and nothing is built with it in mind; it can be anywhere from needlessly complicated to utterly impossible for code that interacts with third-party libraries to successfully discover and import class names for type annotations. Making matters worse, you need to use even more extra libraries for the type annotations to actually do anything.

erk__• 7 years ago

Yeah it is a bit strange argument to use "type safe", the better wording would probably have been to use a statically typed language which is more what is getting more and more popular at the moment.

kbumsik• 7 years ago

Python is in fact a type safe language for a while now

I like the idea of type annotators but they do not make it a type safe language because types are not checked and they are more like comments.

Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.[1]

[1] https://www.python.org/dev/peps/pep-0484/#non-goals

mirceal• 7 years ago

you may be the exception, but most people i’ve met that make this claim have no idea what they are talking about.

how would the cases be better suited for a type safe language? what is type safety, in your opinion?

bredren• 7 years ago

Interesting to see a touch of Flask used for some internal APIs.

alexpotato• 7 years ago

People talk a lot about complicated stacks but Python Flask + some basic HTML/CSS/Vanilla JS can solve a LOT of problems.

This is especially true inside big orgs that have lots of silos and need "Rosetta Stones" that translate between the silos.

kgraves• 7 years ago

"Python is the industry standard for all of the major applications we use to create Animated and VFX content, so it goes without saying that we are using it very heavily..."

Interesting... very curious to see Netflix using tools in the VFX industry, Shotgun & Nuke to name a few, I wish they can expand more on this.

erlangNewb• 7 years ago

Netflix produces content too

mixmastamyk16• 7 years ago

A lot of it, which needs compositing, post-production, encoding, etc. Python is great for pipeline code to script the compiled libraries that do the number crunching.

Some of it is licensed/farmed out to places like DreamWorks however (new Voltron/She-ra), which is mostly a Java/Spring shop, to my knowledge.

coffeedoughnuts• 7 years ago

I used to work for a major vfx house who did much work on Netflix original shows

there will be a certain amount of sharing needed between vendors, and so Netflix will need to have some vfx expertise in-house to coordinate that (similar to how a film will have a vfx supervisor that works for the film production company, who liases with supervisors at each vendor company)

but it would be very sensible for them to have an in-house asset library (which may included work written in python, not just images or textures) so that if they want to switch vfx vendor between seasons of a show it's easy to do so

in general, python rules in the vfx world. there's some c++ too. Sadly, because it's been this way for so long, many houses are still stuck in a python 2 world with huge legacy codebases

sametmax11• 7 years ago

Do you know what the Netflix culture is regarding remote work ?

I've been looking for a job for the first time in 10 years of Python, and Netflix is one of the rare big companies that I still have respect for.

https://jobs.netflix.com/ has several offers I would be a good fit for, but I'm not ready to relocate to the US.

dudul2• 7 years ago

They don't do it.

sametmax11• 7 years ago

Thanks. Working at Netflix ?

drewg123• 7 years ago

We do have an office in Paris: https://jobs.netflix.com/locations/paris-france

Netflix is generally opposed to remote work, but some teams may make case by case exceptions (especially in Open Connect, the organization which manages the CDN). I'd encourage you to speak to a recruiter.

sametmax11• 7 years ago

Thank you. Will do.

2trill2spill• 7 years ago

How come the Open connect team is more receptive to remote work? Harder to find people to work on FreeBSD and low level C code?

jimmaswell3• 7 years ago

As someone who has worked on fully remote game dev volunteer projects for nearly 10 years, it's unbelievable how much the cultural inertia of physical presence has prevented companies from making the rational move of eliminating all possible office space in favor of remote work and skype meetings. So much money wasted on rent, security, and upkeep and so much time (and even lives) lost to commuting when there isn't a single compelling reason programmers need to cohabitate a physical office to get their job done.

mixmastamyk16• 7 years ago

Especially when their stuff runs on AWS.

sametmax11• 7 years ago

I get it personally. Remote requires better communication skill, and additional tooling and management to work around time differences.

Besides, I know many people that are not as productive at home.

theshrike79• 7 years ago

Communication between team members is a huge issue in remote work.

It works if everyone is on site. It also works if everyone is remote.

But when some are at the office and some aren't, communication and information sharing becomes a lot harder. Mostly it's a process and tool issue, but still humans will rather just turn around and ask the team than spend a minute writing their issue on Slack/whatever.

tcbawo• 7 years ago

Companies invest a lot in their employees. They don't want them to leave. In many managers' eyes, physical presence in the office makes it much easier to gauge employee satisfaction and whether they may be out interviewing. Also, it seems that in many organizations still, the employees that arrive to work earliest, are most punctual, most consistent, most organized, best dressed are viewed as more competent and are favored.

IanCal2• 7 years ago

So much money wasted on rent,

Although this now translates to a cost on me needing an extra room in the house for an office.

lame88• 7 years ago

I like remote work to an extent, but its not that simple, and whether programmers "need" something or what "can" happen is a really not the best point to argue, because for example you don't "need" anything more than C if you require portable code to do anything that pretty much any other general purpose language does. I've also done these volunteer projects. They work because timelines are pretty up in the air, there's often little or no funding to be concerned with, everyone is working on it because they want to, and they're only working when they are well-motivated to do so. There are many more constraints in a proper company and job. Remote communication doesn't even come close to the interaction of a roughly synchronized office environment. Of course, there are downsides to the traditional office setup too, but in eliminating those concerns you introduce more problems presented by remote setup. I know people who even have the option of remote work and choose not to, and I can empathize with that sentiment.

rifung2• 7 years ago

Even though I would absolutely prefer to work remotely, I have to disagree with you.

I find that there are significant benefits to working in the same physical location.

It's just way easier to go to someone's desk and ask them something than to have to set up a video chat. Human interaction is just naturally an in person thing. So much is lost if you can only video chat. Sure it may not affect the work directly but I think it really affects a team's dynamic. Lots of great ideas happen just from random office interactions, whether it be lunch or just talking over coffee. These kinds of things are really hard to reproduce in a remote environment.

That's not to say there aren't reasons working remotely is beneficial. I just don't think it's fair to just say the only reason companies opt for physical proximity is cultural.

gfs78• 7 years ago

Not every programmer does his/hers best working remote and communicating through slack/skype meetings. People have different personality traits. The remote devs in your volunteer projects were probably self-selecting.

Also, some dev positions require extensive communication and that is easier in person.

mixmastamyk16• 7 years ago

Their job site is a big /dev/null however. I spent several years dutifully submitting for numerous jobs with twenty years experience in internet and VFX in Hollywood and didn't even get an "other direction" form-letter for my trouble. (They don't date their posts so you have no idea which are active.)

Netflix is in biking distance, but instead I work for a company on the east coast, 4000km away, waking at the crack of dawn.

sametmax11• 7 years ago

Rejection is not a problem to me. The world is a big place, and it moves fast.

gonehome8• 7 years ago

The only way to get hired at competitive companies is via referral - if you live in the Bay Area you probably know someone or know someone who knows someone at Netflix.

Trying any other way is a lottery - and if you didn’t go to MIT/Stanford/CalTech/Harvard then your odds are pretty bad.

The other way is already working at a famous company and having another famous company send you a recruitment email via linked in, but that typically requires you to have done the referral way first.

jogjayr• 7 years ago

I was told it was not an option, when I discussed an engineering IC role with them.

winter_blue6• 7 years ago

but I'm not ready to relocate to the US

Even if you wanted to, it's nearly impossible to immigrate to the US as a skilled worker legally.

I have a friend who is an really talented programmer (and a really diligent worker) who studied Computer Science at some of the best universities in Europe: École Centrale Paris and the EPFL (École Polytechnique Fédérale de Lausanne) in Switzerland. One of his biggest desires for a while was to move to the US, and live and work in New York. He spent more than 2 years trying to the US, and then gave up. He had even hesitated buying too much furniture while in France, because he was anticipating being able to move to the US.

The only viable option to move to the US for most people in your (and his) situation would be the H-1B visa, which is extraordinarily difficult to get, and requires a miracle to obtain. You need to: (1) have a bachelor's degree that's related to the field you want to work in -- e.g. Computer Science degree for software development work, (2) find a company that's willing to offer you a job that pay at least what Americans make doing the same job, (3) the company must be willing to wait 7-8 months, (4) you need to win a lottery in which your odds of success are about 1 in 3.

(1) and (2) are not too difficult. (3) is quite difficult -- the majority companies do not want to wait for 7-8 months for you to join. And (4) is completely up to chance, and out of your control. (By the way, this lottery is conducted at the beginning of April every year, and you can only start working in October.

My friend fulfilled criteria (1), (2), and the difficult (3). But he did not get selected in the H-1B lottery two years in a row.

The US is essentially a closed country when it comes to skilled immigration. Most of the immigrants who get green cards here are family members, refugees/asylees, and diversity lottery winners. Getting even temporary permission to work in the U.S. as a educated/skilled person requires a miracle from God.

sametmax11• 7 years ago

I know at least 3 french persons that managed to make it work. They don't even work for a FAANGs, and only of of them is technically better than I am.

I'm an optimist :)

Thaxll3• 7 years ago

If you're European from a "good" country it's pretty easy to get an H1-B with a master / eng degree to work for a SV company.

dbancajas• 7 years ago

You forgot other loopholes: 1. Get F1-visa and do MS. 2. Marry a US citizen 3. use a J1 visa?

random42_• 7 years ago

(1) You don't need to have a degree to apply for an H1B as long as you can prove 3 years of experience to each year of college. So, if you are a developer with 12 years of experience that makes up for the lack of a 4-years degree. You can also combine years of college with experience to complete that requirement. Say, 2 years of college plus 6 years of experience.

(2) the salary must follow the prevailing wages in the area the employee is gonna work. That's not really an absurd thing specially considering that the market rates are usually higher than the prevailing wages.

(3) what I have seen about this point is that companies will usually find a way around that if they're really willing to hire you. Either work remotely as a contractor until the visa is stamped and you can move or something else. It's hard but not impossible.

(4) is really up to chance and I've had friends who were and were not selected.

I'd say it's hard but no miracles needed.

ThePawnBreak• 7 years ago

Even if you wanted to, it's nearly impossible to immigrate to the US as a skilled worker legally.

Nearly impossible? Come on, this is a gross exaggeration. Get a job at a FAANG or unicorn in Europe, work there for 1 - 2 years, transfer to the US on an L1 visa, then apply for H1B each year until you get it. I know several people who have done this. Hell, Microsoft opened an office in Vancouver specifically to move failed H1B's there until they can get an L1.

xtreak29• 7 years ago

I am wondering if PyPy is used somewhere for workloads that need performance.

mixmastamyk16• 7 years ago

Probably not, more likely to call out to their numerous Java projects and C/C++ video tools.

z3t4• 7 years ago

Wasn't Netflix a Node.JS shop two years ago when Node was popular and now when Python is the most popular they are a Python shop ? :P (sarcasm) It's good to use many languages as it favors a micro-service architecture.

tombert3• 7 years ago

Is Python the most popular language now? I'm not trying to argue, I'm trying to see how out-of-the-loop I am nowadays.

yifanl• 7 years ago

Depends on how you measure most popular. I think I saw a post here saying that the most new questions on SO were for python over js now, which might be a good enough proxy?

bouncing7• 7 years ago

If search metrics are a valid measure:

https://www.economist.com/graphic-detail/2018/07/26/python-i...

passthejoe• 7 years ago

I've been resisting Python for years, but it just keeps getting more popular (and support and more/better libraries come along with that), so I might just stop fighting using it for local scripts and small GUI projects.

kickopotomus• 7 years ago

It has gained a lot of popularity and is used across disciplines pretty heavily. Apart from the typical CS uses, it's used a lot by scientists and engineers because it's pretty easy to pick up and the available resources have snowballed such that someone has probably already solved your problem so all you have to do is import antigravity[0].

[0] https://xkcd.com/353/

wongarsu2• 7 years ago

Python has seen an increase because most ML is done in Python and ML is a popular topic right now.

"most popular" is hard to measure or even define.

aaronblohowiak5• 7 years ago

Python at Netflix 2013 edition: https://medium.com/netflix-techblog/python-at-netflix-86b602...

Though it is outside my area of expertise, Node usage continues to increase as well. Different use-cases.

bcrosby95• 7 years ago

At a company the size of netflix, I would imagine they're using all sorts of languages. I have a friend that works there and he primarily codes in c and/or c++.

batbomb• 7 years ago

I'd say a "shop" implies less than 150 employees. So there are many "shops" inside Netflix.

nerdponx• 7 years ago

It's good to use many languages as it favors a micro-service architecture.

I'd argue the converse: one advantage of a micro-service architecture is that it lets you use the best language for any particular task (where best could be "what your team is most comfortable with" or "has the most extensive library support" or something else).

bouncing7• 7 years ago

I had actually looked at Flask-RESTPlus for a relatively recent project before deciding to use FastAPI.

randomsearch• 7 years ago

Surprised they use Flask.

I'm working on my first larger, industrial-strength REST API in Python and I've found the Django Rest Framework to be more suited once you get to that level of complexity.

bouncing7• 7 years ago

Probably whoever made it just preferred Flask.

django-rest-framework is nice because it's maintained well and it's consistent with the rest of Django, but plenty of people just prefer the Flask way of doing things, even if it's a scrappier set of tools in some ways.

ajford• 7 years ago

Django works great if you're buying into the Django ecosystem. But if you're standing up a simple REST API, I've found Flask is much more flexible. Either approach will get you to where you're going, but I find Flask gets out of your way and lets you do your thing, while you have to find the Django Way of doing what you need to do.

This may be biased (I've been a Flask user since 0.7), and perhaps I'm stuck in my ways, but I've found I either need to find a supported and blessed Django plugin for the thing I need, or build something much more complex than I'd need in Flask.

rurp• 7 years ago

For typical apps I think that Django is far superior to Flask because every Flask project ends up recreating a lot of Django's core functionality; but in a more time consuming and less standardized way.

For a company as specialized as Netflix, they probably are doing so much customization it doesn't matter much which framework they started with.

Splendor• 7 years ago

You made the right choice.

glckr• 7 years ago

Interesting. I had it in my head that Netflix was a pure Java shop. Was I wrong in the first place, or has something changed?

2StepsOutOfLine• 7 years ago

From what I understand from their talk on how they (don't) do devops they adopt a polygot architecture and teams use different languages.

linuxdude314• 7 years ago

Different teams using different languages doesn't mean they don't do DevOps.

vbsteven• 7 years ago

I believe their is a fair bit of Kotlin used there too (based on some @netflix.com addresses in Kotlin libraries like Strikt)

wesleytodd2• 7 years ago

I mainly write js but our team maintains a scala app as well. It is much more of a polyglot company with a history of java (I have only been here 1.5y so this is from hearsay and my expirence, not a definitive source).

tanilama2• 7 years ago

Mainly a Java shop. They published Node and even some C stuff back then in their blog posts.

Fnoord• 7 years ago

Java or JavaScript? Some of the replies to this post indicate JS, others Java, while they're not at all related to each other...

abritinthebay• 7 years ago

We're everything!

JS is heavily used, as is Groovy, Java, Python, C and C ++, etc.

Depends on the project and the team. We're not about limiting people, but the choice should be justifiable (like... Brainfuck is probably not ok)

dorgo• 7 years ago

https://en.wikipedia.org/wiki/Brainfuck

Thaxll3• 7 years ago

If you read the article they don't talk about "online" services / API which are probably most of them in Java.

tracker1• 7 years ago

When I interviewed there a couple years ago (decided to stay in phx), it was a polyglot culture... the team I was interviewing with was largely node/js. I'm not surprised to see a lot of Python, Java or anything else that leverages existing tooling. For that matter, I wouldn't be surprised by custom C++, Go or Rust for some pieces.

brown9-2• 7 years ago

A sufficiently large company uses basically every language at some layer.

butterpeanut• 7 years ago

Id like to contribute to the netflix code base by offering what is obviously a missing but much needed piece of code:

def enableAutoPlay( flag ): annoyingAutoPlay = flag return

fokinsean• 7 years ago

I don't think that's valid syntax, so it likely won't make it in their repo

terryschiavo22• 7 years ago

Drats! We were so close!

hyeomans• 7 years ago

At least as an option hidden somewhere.

7813• 7 years ago

TV's should also have this feature. When you switch to a new channel, it should be displayed in a paused state and you should press a button on the remote to start playing.

Auto-play doesn't make sense anywhere, streaming apps, youtube, or TVs. Only crazy people expect video to auto-play in a video app.

icedchai• 7 years ago

That would make it super annoying for your average channel surfer.

pletnes2• 7 years ago

Afaik you can disable autoplay in the options?

bhauer• 7 years ago

At least with the web UI, you can mute the auto-playing video and subsequent auto-playing videos will remain muted.

driverdan2• 7 years ago

You can disable automatic playback of the next video but can't disable autoplay previews.

iblaine• 7 years ago

Netflix does have people from MySpace, which could explain the annoying auto play feature, but the reality is Netflix has a mature process of AB testing. If autoplay is permanently on then it's because AB testing wills it so. ¯\_(ツ)_/¯

gloflo• 7 years ago

Maybe they noticed people watching more content with autoplay on. Wait...

ptah• 7 years ago

now to get the apple tv app to use more than the bottom third of screen for letterbox sized navigation by making the massive "thumbnail"/trailer at the top a third of the screen, so i can actually find something good to watch...

gvd2• 7 years ago

I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle.

gshulegaard• 7 years ago

I fail to understand the use of python in a distributed environment while the language has such poor concurrency support

Because it's a distributed environment probably is exactly why. Python has (arguably) great concurrency support apart from Multi-threading.

https://www.youtube.com/watch?v=MCs5OvhV9S4

So if you need concurrency in the context of a single thread, then Python's GIL is a non-starter. But a distributed environment is not likely one of those.

Edit: I should amend concurrency in a single thread to: concurrency in a single thread that is compute gated...since coroutines can give you pseudo concurrency in a single thread provided you're workload has blocking steps like IO or TCP calls.

wongarsu2• 7 years ago

In fact it's precisely Python's deficiency in multithreading that lead to it having one of the best ecosystems for every other form of concurrency, like green threads and multiprocess applications.

pletnes2• 7 years ago

If you’re doing (data analysis|simulations|Image processing) you can offload computation to numpy, which releases the GIL. This allows nice multicore speedups with python and threading.

The same holds for various CPU intensive standard library functions implemented in C.

The GIL issue is real, but posts like this one confused me for years. Please, don’t exaggerate GIL issues.

weberc212• 7 years ago

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/async dichotomy.

zerkten• 7 years ago

Just because you don't see or understand doesn't mean their usage is null and void. Does distributed imply a need to "squeeze out every cycle" at the cost of productivity? If they find that they are CPU bound, or need more efficiency then it's likely they'll move to another platform. If not, why make the move, or lose productivity?

Personally, I've worked with some HPC apps in pharma. We found quite a mix of CPU and IO-bound challenges when we actually profiled and looked closely at what was slowing up the apps. Contrary to the original belief, rewriting and increasing improving the CPUs wouldn't have helped much.

larkost• 7 years ago

Some parts of the process are obviously not ones where it makes sense to squeeze out every CPU cycle. Sometimes programmer time (both in development and maintenance) is your scarce resource (either because of total numbers, or people with specific knowledge), and so it is far better to optimize for that.

In most cases (Python, Ruby, etc...) is it even possible to find your hot-loops and replace them with C/C++/Rust/etc... code. So you can really focus on those small areas where that would make an actual difference.

Additionally, in some places where you need concurrency you can split up the task into parts that have little to do with each other, and there Python is a great glue language to manage calling other executable that do the actual work.

gigantum• 7 years ago

Re: Type system -- at Gigantum we very aggressively enforce all classes and methods in the core libraries must be fully typed using mypy. The depth and expressiveness of mypy rivals that of any other strongly typed language.

tanilama2• 7 years ago

I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system).

Well, without knowing the exact issue people trying to solve, of course you won't understand the motivation behind.

Reading their post, it seems that they are not using Python for serving, mainly for long running daemon processes. Concurrency is probably not something people care about in such situation, nor squeeze CPU perf. In fact such workload probably wants to maintain a low-key finger print.

markbnj3• 7 years ago

I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle.

You're conflating several things that are orthogonal imo. A system can be distributed without concurrency. A concurrent system need not be distributed. Either kind of thing can be built with or without a specific kind of type system. And CPU efficiency has nothing to do specifically with any of the previous things.

To expand on that: distributed systems are quite often constructed from simple single-threaded processes, and where concurrency is needed it is probably more often achieved through multi-processing than multi-threading. A single-threaded event-dispatched request-response service probably describes a big chunk of all the stuff running in distributed environments today. In a lot of these cases the workloads are i/o bound, and instructions per cycle is not even close to the top of the list of concerns. There are a lot of reasons why python fits into that world very well.

badpun3• 7 years ago

A system can be distributed without concurrency. A concurrent system need not be distributed.

The only justification for such design I can think of is HA. I.e. you have very light workload that does not saturate a single machine, but nonetheless you create a distributed system with two machines for HA. Such light workload is probably not often the case at Netflix.

ApolloFortyNine• 7 years ago

To make a distributed application, you need some sort of orchestration. For example, a Redis job queue.

And at that point, you can just code your application as a single thread, and run one per core on the machine.

And the Python lack of types "issue" has always been a bit overblown in my opinion. If it's not immediately obvious what something is, then you need to either comment more, or come up with better variable names. And if you really, really feel the need for a type system, Python natively supports that now.

MaxBarraclough• 7 years ago

poor concurrency support

Poor intra-process parallelism, yes. Python is pretty capable when it comes to concurrency.

peterwwillis2• 7 years ago

Reason #8,201 not to model your org after FAANG: writing custom software just to operate your site is like building your own tools to till your farm. I suppose if you hire world-class toolmakers you'll have some very good tools, but that's not quite the point of the farm, is it?

gonehome8• 7 years ago

If you hire world class tool makers to make the best tools for your farm then your farm has a strategic advantage.

Really good tools can make a big difference.

You really do need world class though - the trade off of custom means your tools really do need to be a lot better than the standard.

prdonahue• 7 years ago

At Cloudflare we use their Python-based Lemur (https://github.com/Netflix/lemur) application to issue (on some days) 1M SSL certificates.

prolepunk2• 7 years ago

"You've read 11 stories this months, lets make things official".

Medium is trying to look like NY Times, minus any kind of effort to actually write the stories.

lma21• 7 years ago

Why can't I read their articles through Pocket?

Tycho2• 7 years ago

Does anyone have more info about this or know of similar projects?

We lean on the many of the statistical and mathematical libraries (numpy, scipy, ruptures, pandas) to help automate the analysis of 1000s of related signals when our alerting systems indicate problems. We’ve developed a time series correlation system...

jamespo• 7 years ago

There's https://prometheus.io/ which you're probably familiar with

aaronblohowiak5• 7 years ago

What would you like to know?

Tycho2• 7 years ago

The correlations part specifically, what does it do, what’s the modelling approach, and is there more info about it anywhere?

mistrial9• 7 years ago

push or pull ?

sandGorgon3• 7 years ago

The ability to drop into a bpython shell and improvise has saved the day more than once.

what do you really do here - connect to the flask instance and route requests manually ?

kabacha• 7 years ago

Also who uses anything but ptpython!

aaronblohowiak5• 7 years ago

Our python code does not handle user requests directly. Our team uses python to control the control plane of traffic — we flip dns records, control cross-region proxying, re-steer cdn-based reverse proxying and scaling of the hundreds of micro services that power the Netflix experience. We’ve improvised various custom traffic distribution patterns, operated outside of our normal traffic-shifting workflows, and written/run quick scripts to modify scaling fleetwide.

sandGorgon3• 7 years ago

hi - so im actually curious about the infrastructure that allows you to connect bpython to it. Is it a rq process that you connect to, etc ?

I would love to setup an architecture like this that lets me connect to stuff through a CLI. Also, I'm assuming you are on kubernetes (or something). How does this bpython business work through all those layers ?

whoevercares• 7 years ago

Metaflow looks like a superhero solution for ML/DL. Hope they open source it one day

Python at Netflix

Related Stories