Python at Scale: Strict Modules

https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834

by apadmarao • 7 years ago

388 249 7 years ago

ledauphin4• 7 years ago

I love the idea, but it feels like just an idea at this point. I'd rather read about them releasing their 'compile-time' analyzer and revealing their measurements for how much startup time it saves.

In our codebase, we have pretty strict developer-enforced rules about not doing I/O at the module level, usually through the use of simple "Lazy" wrappers for module-level objects. I'd be curious to know what other approaches people have taken with Python here.

rectangletangle3• 7 years ago

It is an interesting approach, though I feel like this could introduce some nasty unintended consequences given how dynamic and introspective Python can be (admittedly I haven't studied this particular implementation).

I always treated this a bit like single underscore private functions/methods, i.e., follow a convention that produces code that's easy to reason about, even if it's not strictly enforced by the language/compiler. So in practice this equates to separating out modules that mutate global state, and placing the majority of logic in "strict" modules that only declare a bunch of "pure" classes/routines. So the "non strict" code is really just a thin layer of wiring gluing everything together. For instance my Celery task files tend to be very thin.

ledauphin4• 7 years ago

well, we also heavily use static typing, so you end up with something like

my_db_conn: Lazy[DbConn] = Lazy(lambda: make_db_conn(...))

and MyPy will tell you if you're doing something silly when you try to use it.

EDIT: After typing up this response and submitting I realize you were talking about their strict approach rather than ours. whoops :)

avip• 7 years ago

It's very important to think about objects lifecycle management.

It's also important to... use pytest fixtures instead of arbitrarily patching around in tests.

brenden22• 7 years ago

It still blows my mind that people don't use strongly typed languages in the first place and spare themselves from all this future pain.

My guess (based on my experiences) is that companies wind up in this position from having inexperienced people building early versions of products instead of hiring experienced engineers (who are usually more expensive).

b3orn• 7 years ago

Python is strongly typed, just not static.

brenden22• 7 years ago

Python uses duck typing: https://en.wikipedia.org/wiki/Duck_typing

I would categorize it as a subset of dynamic typing, and that's what Wikipedia says too.

ainar-g2• 7 years ago

It's a constant struggle against the current. Dynamically-typed languages are often “good enough for the time being”. I have the same issue explaining to our C/C++/Obj-C team why they should use static (Clang-Tidy, Infer, PVS-Studo) and dynamic (ASan, MSan, UBSan) analysis tools. They just keep giving me basically the same response of “I am a good programmer, and my code is good, and shame on you for even daring to think that a mere machine could find bugs in my code!”. I don't know what kind of status anxiety causes it. It also makes me think about what kind of other I am missing because of the was I keep thinking that I do that thing well-enough myself.

ken3• 7 years ago

I'm confused. It should be easy to demonstrate the benefit, if there is one. Just show them the bugs!

For me, it's not "status anxiety". It's simply not worth the effort.

The last couple static analysis tools I ran on my programs, I spent a while getting the tool to not-crash (because even though the authors obviously had a static analysis tool themselves, they either didn't bother to run it on their own code, or it wasn't good enough to find actual issues). These tools flagged only a couple issues, and almost all of them were places where it couldn't really cause any problems, but the type system was not strong enough for me to prove why it couldn't go bad. So I spent a while sorting through false-positives.

I'm not going to spend hours with a tool to find only a couple (real) bugs, which no user has ever reported seeing, and which I've gotten no automated crash reports about. I have much better uses for my time.

zallarak• 7 years ago

This article is among the best argument for using a typed language I’ve yet seen.

kbd2• 7 years ago

This has nothing to do with types. It's more about static guarantees the language gives about module import behavior.

nothrabannosir• 7 years ago

In OP's defence:

So that's a third pain point for us. Mutable global state is not merely available in Python, it's underfoot everywhere you look: every module, every class, every list or dictionary or set attached to a module or class, every singleton object created at module level. It requires discipline and some Python expertise to avoid accidentally polluting global state at runtime of your program.
One reasonable take might be that we’re stretching Python beyond what it was intended for. It works great for smaller teams on smaller codebases that can maintain good discipline around how to use it, and we should switch to a less dynamic language.
But we’re past the point of codebase size where a rewrite is even feasible. And more importantly, despite these pain points, there’s a lot more that we like about Python, and overall our developers enjoy working in Python. So it’s up to us to figure out how we can make Python work at this scale, and continue to work as we grow.

Those are literal quotes from the article. That is quite damning. How did they get to this point? By starting when Python was appropriate, and taking it day by day.

iso-8859-1• 7 years ago

Depends what you mean by "type". A type in e.g. Haskell specifies whether there are side effects.

ken3• 7 years ago

Surely this is a typo and you mean “functional language”, as mutability and state is the main issue here, not dynamic typing.

allan_s• 7 years ago

This means that just by importing this module, we're mutating global state somewhere else.

Yes, this !

That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py`

Also as said below in the article, suddenly it's much harder to handle smoothly the "the database is momentary unavailable" (because someone has put the line starting the database connection in the global space of a module somewhere)

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function

tummybug• 7 years ago

I'm not sure about django but flasks Application object has a before_first_request method which takes a function designed to do this type of initialization operations.

rectangletangle3• 7 years ago

I'm a huge fan of Django, but I always felt that this was true. I wish there was more of a push to decouple parts of the framework. Keep the magic, but allow usage without it.

nerdponx2• 7 years ago

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function

Django _does_ have a "setup" function. You can't import and use Django database connections outside of a running application without it.

Flask also has a "run" method and does no i/o without it.

heavenlyblue3• 7 years ago

Every time I hear a comment alike parent's, it makes me think how many times a day I actually read a comment in the same fashion, but about something I actually know nothing about.

yen223• 7 years ago

Without calling setup, you cannot import anything that touches Django models, like constants defined in a file that transitively imports a Django model.

In practice, this means that any script that depends indirectly on Django code will incur a lengthy startup cost (from having to call setup()), and will fail to run if there's no database connection, even if the script itself doesn't need the db.

orf2• 7 years ago

Django doesn’t create database connections on import. That would be madness.

It does create an object that can (lazily) connect to the database, so it needs the required database drivers installed. It also needs the required information about _how_ to connect to the database, so it needs the settings loaded.

That's why you need to use `django.setup()` before, to tell it what settings to load. You should never be importing random Django models without this configured, simply because they cannot be used and will not work. We think an exception saying "don't do this, call django.setup()" is less confusing at import time is than "Databases not configured" at runtime. Not that it would even reach that, because you might be using a field from a third party application that needs to be initialized (i.e INSTALLED_APPS configured) or that relies on a configured settings (maybe an encrypted field that needs your SECRET_KEY available).

Stop making it hard, just write a management command. It's super easy.

ben509• 7 years ago

How do we know that the log_to_network or route functions are not safe to call at module level? We assume that anything imported from a non-strict module is unsafe, except for certain standard library functions that are known safe.

It's hard to know anything about the stdlib as it can be monkey patched, e.g. [1]

That said, you could solve this with diagnostics; calculate signatures of stdlib functions and classes to find any known safe ones that were patched. Run that check in your test suite to find problematic imports.

If the utils module is strict, then we’d rely on the analysis of that module to tell us in turn whether log_to_network is safe.

I like this. It seems far more usable than proposals like adding const decorators.[2]

[1] https://github.com/gevent/gevent/blob/master/src/gevent/monk...

[2] https://github.com/python/typing/issues/242

timothycrosley18• 7 years ago

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvement.

TylerE7• 7 years ago

nim

timothycrosley18• 7 years ago

I've been tracking nim, and would agree it's the most promising so far! I feel though that it's trying to be too flexible in many ways. Examples of this include allowing multiple different garbage collectors and encouraging heavy ast manipulation. I'm also afraid it is different enough to keep it from attracting a significant amount of developers from the Python community. Nonetheless, it's something I plan on using and contributing to, since it's the best option so far.

carapace11• 7 years ago

Cython? Nuitka?

timothycrosley18• 7 years ago

I use Cython a lot! But mostly to speed up existing Python code, and build C-extensions faster. I don't see it as a strict subset of Python or a new language to build a community around. Nuitka I just started experimenting with to build standalone Python executable, and I really like the direction and roadmap they are following. In the end though both of these technologies seem like ways to somewhat speedup existing Python code and not attempts to introduce a strict language subset that would allow the greatest amount of optimization, and finally fix long running issues, like the inability to have multiple versions of a package installed.

scrollaway10• 7 years ago

Python has some of my favourite syntax as well but I absolutely hate its annotations. TypeScript got typings right.

I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like.

And of course if something changes the syntax, better anonymous functions will be the absolute first thing I would look for...

timothycrosley18• 7 years ago

TypeScript got typings right.

I have not used TypeScript, but looking at it's documentation the syntax for type annotations look identical. Would you be willing to expand on why you think its approach is better / how it's different?

TylerE7• 7 years ago

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitterness about the whole situation).

breatheoften• 7 years ago

I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like.

I think this is an extremely good idea. Python is horrible but forced on a huge number of developers because of its ecosystem ... I think a bridging layer from typescript to python could be built in a way similar to swift’s Python Interop — and I don’t think it would require any special language support ...

I think could actually make a better/easier to use/more robust design than Swift by requiring all interactions with the python interpreter from node be async.

bratao• 7 years ago

There is https://github.com/python/mypy/tree/master/mypyc that I think is a great idea and approach

staticassertion2• 7 years ago

https://github.com/python/mypy/tree/master/mypyc

This may interest you. This is probably going to be the 'official' way to get what you're talking about.

timothycrosley18• 7 years ago

Thank you for sharing! This looks really promising, I'll try to think of ways I can contribute to the project.

ledauphin4• 7 years ago

yeah, I'm excited about this approach. but it's a long way from being a realistic approach for folks outside Dropbox, it seems.

weberc214• 7 years ago

Sounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.

timothycrosley18• 7 years ago

I hate the fact that you may be right, because I really don't like Go in many ways:

- I hate it's module system and package eco-system story. - I don't like its syntax. - I don't like its error handling. - I'd much prefer gradual typing. - I want to maintain the ability to use interactive interpreters. - I don't like the fact that instead of being community driven it is Google driven.

But, anecdotally, I see go being used as a second language to Python more than anything else and at an ever accelerating rate.

ledauphin4• 7 years ago

Go may "feel" like Python, but it's almost nothing like Python in actual practice. It's not dynamic (and doesn't even have generics), and its error handling is dramatically different.

nine_k3• 7 years ago

What Go adds in tooling and performance, it takes away in expressivity.

What takes 3 lines in Python, takes 10-30 on Go.

totalperspectiv• 7 years ago

I'm hoping someone with more experience will chime in, but what about rpython / pypy?

cpeterso• 7 years ago

I like Python, but I often wonder how many developers use Python because they actually use dynamic language features versus just liking the languages' clean syntax and library ecosystem. I'm surprised languages that offer both REPL (for development) and AOT native compilation (for production), like OCaml, are not more popular. Evidence that syntax matters, I guess. :)

mypy and mypyc are interesting but their compile-time checks and optimizations are still hampered by Python's dynamic language semantics.

Sophistifunk• 7 years ago

I think a great deal of this sort of thing could be done by just doing some eval in a dynamic state before you stop the vm and compile its stable state, rather than the actual source code.

peteradio2• 7 years ago

Names matter and OCaml is a crappy name.

clintonb• 7 years ago

Don’t underestimate inertia. I’ve worked with Python and Django for seven years. I know the libraries in the ecosystem. I know the framework. It’s far easier for me to start a project with Django than to learn another framework or language.

pjmlp12• 7 years ago

The dynamic language semantics of Lisp, Scheme, Smalltalk, JavaScript have not hampered the existence of good JIT/AOT compilers.

Smalltalk, for example you can completely change the structure of a class by sending a become: message.

What I think is missing is a bit of more PyPy love, and the Truffle and OpenJ9 Python support efforts.

Nimitz14• 7 years ago

Yeah I was hoping Nim would be it but I don't like the syntax they use.

jsmeaton• 7 years ago

For the sites I typically work on it’s very hard to give up the Django admin and all of the features it provides.

At the same time, I’d love a stronger type system to avoid a bunch of the pitfalls that the dynamism of python has.

So count me in.

paulie_a• 7 years ago

I completely agree. With python I need ten packages. With the shit show of JavaScript I need 100 conflicting packages. Why bother on a backend framework like js. it's a worthless language for backend development

chenzhekl• 7 years ago

How about Nim? It has Python-like syntax and is as fast as C. https://nim-lang.org/

timothycrosley18• 7 years ago

Answered this below:

I've been tracking nim, and would agree it's the most promising so far! I feel though that it's trying to be too flexible in many ways. Examples of this include allowing multiple different garbage collectors and encouraging heavy ast manipulation. I'm also afraid it is different enough to keep it from attracting a significant amount of developers from the Python community. Nonetheless, it's something I plan on using and contributing to, since it's the best option so far.

Though, now that another commenter pointed out mypyc: https://github.com/mypyc/mypyc I believe I'll invest my limited free-time in that project instead, as it will allow me to stay within the Python community and eco-system that I love so much.

Jefro118• 7 years ago

Just in case it's of interest to anyone reading this, I interviewed the designer of Nim, Andreas, about his design choices and what he learned from Python and the C family here: https://sourcesort.com/interview/andreas-rumpf-on-creating-a...

Gives some good insight into where Nim is going in the future too.

strokirk• 7 years ago

It's certainly interesting to use! However, it's type checker still have a lot of work to go, since you can easily segfault due to using a nil reference.

thelastbender122• 7 years ago

Very much this! For numerical computing, Numba + llvmlite attempts to do it.

I don't know however if this approach could be extended to other domains - say making a web framework. Given, python classes let you do so much tinkering, any attempts to port existing code will probably need a lot of rewriting?

moksly• 7 years ago

Aside from having a two decade history of using C#, types is the only thing preventing us from going full Python. Even so, the dynamic types in Python are more often a benefit than a disadvantage because Python is so great at handling them automatic.

We build our employee database, and from there our IDM, from a singel XML file in a really shitty format + three txt files in even worse formats (they are single line output files from an old mainframe system predating sap). We used to do it in a rather complicated Microsoft SSIS workflow with a lot of C# services. All in all it’s a 30 minute nightly runtime. I recently replaced it with around 500 lines of Python and a 1-5 minute Runtime (sometimes at the beginning of a school year we’ll see changes to around 1000 positions).

Python eats the XML like it wasn’t shit. It takes things like terrible date formats, we’re talking the output of a SAP free-text box shitty, and ports then seamlessly into a SQL date field. This alone was a nightmare in C# and Python just does it.

Still, after two decades of strict types it feels dangerous.

Rotareti2• 7 years ago

I can imagine the next big programming language will be one that is split into two language-variants: the "low-level-variant" and the "high-level-variant".

The high-level-variant is a dynamic language with optional typing, which is good for scripting, fast prototyping, fast time-to-market, etc.

The low-level-variant is similar to the high-level-variant (same syntax, same features mostly, same documentation), but it has no garbage collector, typing is mandatory and it runs fast like C/C++/Rust. Compiled packages that are written in the low-level-variant can be used from the high-level-variant without additional effort at all. The tooling to achieve this comes with the language.

A language like this would be insane, IMHO.

nikki93• 7 years ago

A key consideration here would probably be the expression and passing around of managed instances spawned in the high-level variant through low-level code. Would you explicitly retain and release them? -- etc. I think it should be an ergonomic solution for this language to provide an edge over just using C / C++ / etc. with Lua / Python / etc.

alex7o• 7 years ago

You can say that this is typescript and assemblyscript, they have the same syntax but one of them is compiled natively (wasm).

miki1232113• 7 years ago

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're wizarding. If you grow big enough, this approach slows you down greatly, and you need to switch to engineering. You sacrifice some speed for making the codebase more understandable to a larger group of people, you can no longer assume everyone knows all the code, you write unit tests, need types and dislike metaprogramming because of the confusion it creates. This is why languages like Python, Ruby, Lisp or Smalltalk are amazing for small startups, but Java is what enterprises use. They're different ends of the wizarding/engineering spectrum. I wish there was a language that let you move gradually from one end to the other, exactly when you need to.

[1] https://www.tedinski.com/2018/03/20/wizarding-vs-engineering...

ben_jones• 7 years ago

What you call Wizarding I call "ordinary Software Development". A software developer spends ~70% of their time writing features and the rest mixed between organization/planning/roadmapping etc. A software engineer spends ~30% of their time writing features and the rest of it managing technical debt and making long-term investments towards better features and processes.

Too many companies need devs but have engineers, or they need engineers but only have devs :/

heartbreak2• 7 years ago

I’ve never known anyone to distinguish between the two roles, and they seem to be used interchangeably in the industry.

(Except those people who claim software engineers aren’t real engineers)

TylerE7• 7 years ago

Who is starting large-scale new projects in Java in 2019?

victor1062• 7 years ago

One of my clients handles 90% of all the pbm routing in the US, which is millions of transactions per second. They started to completely modernize the application on java... primarily Spring Boot and Apache Geode. After some optimizations they are very happy with the performance and expressiveness of modern java.

pjmlp12• 7 years ago

Plenty of companies free of magpie developers.

jedberg4• 7 years ago

A big chuck of Netflix is written in Java, and they make new stuff in Java all the time.

hsaliak• 7 years ago

Google

coldtea11• 7 years ago

Countless companies, huge and small -- from Apple and Amazon, to Google and your friendly local startup, plus all the enterprise world that's not a .NET shop...

In what parallel universe is not Java immensely popular or not used for green projects?

lucidone• 7 years ago

I write JavaScript all day and wish I could write in Java.

lern_too_spel• 7 years ago

For large scale projects, Java and C++ remain the go-to languages. I've seen a little bit of Go start to show up but no others. Other languages are used for libraries (Rust, C), only at certain employers (OCaml, Erlang), or for small-scale projects (nearly everything else).

typon• 7 years ago

It's a good question. Why would you pick Java over Go or C++?

salex89• 7 years ago

Can't say about new ones, but after a couple of years of working on a huge Python project, I would accept rewriting it in Java without a whim. I have equal experience in Python and Java by now. Funny thing, I returned from my vacation a few months ago, and started writing my first code after it. Of course, I was more concerned with thinking what am I writing instead of how. Then I looked up and noticed I started typing Java instead of Python. And that's three years after the last time I've written some Java code.

kevan• 7 years ago

Plenty of teams within Amazon

pron• 7 years ago

Oh, just Apple, Amazon, Google, Netflix and likely every hospital, utility company, police force, military, or bank you depend on. And the people programming the robot swarms that pack your groceries (https://www.infoq.com/presentations/java-robot-swarms/).

bcherny2• 7 years ago

I wish there was a language that let you move gradually from one end to the other, exactly when you need to.

This is precisely what gradually typed languages — like TypeScript, Flow, and typed Pythons — solve!

I talked about this on Software Engineering Radio last week: https://www.se-radio.net/2019/10/episode-384-boris-cherny-on....

spraak• 7 years ago

Typescript was exactly what I was going to mention in reply

paulddraper2• 7 years ago

TypeScript is perfectly this. (And other gradually typed solutions; TS is simply the most popular one.)

You have the madness of thousand of developers flinging code at the universe due to the easiness of browsers, JS, and npm.

This results in great speed, but not great quality.

When your project/company now wants quality, you keep your code but transition to types. (In OSS space, Angular and Yarn projects have both done JS => TS migrations of some form.)

lostcolony4• 7 years ago

On that note, I'd include Erlang. It's not gradually typed, per se, but you can have a fully dynamic language (no type specs, no Dialyzer), a completely optimistic static analyzer for inferring types and warning where it's inconsistent (Dialyzer runs), and then you can add specs where needed to tighten up and improve what Dialyzer can catch, to basically be a fully static language.

miki1232113• 7 years ago

It's not just about static typing, though. Macros, metaprogramming, being able to reach as deep as you want to, ugly code full of side effects, global state etc. All of those might actualy benefit you when your project is small, and they make development way faster (see Rails). Later, however, they're a definite impediment.

zitterbewegung• 7 years ago

You can write fortran in any language. [1] https://blog.codinghorror.com/you-can-write-fortran-in-any-l... [2] https://queue.acm.org/detail.cfm?id=1039535

movedx• 7 years ago

You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're wizarding. If you grow big enough, this approach slows you down greatly, and you need to switch to engineering.

I've never heard of this before... I love it. Thanks for bringing this up.

heartbreak2• 7 years ago

This is similar to Martin Fowler’s Design-Stamina Hypothesis [0].

[0] https://martinfowler.com/bliki/DesignStaminaHypothesis.html

ken3• 7 years ago

When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter.

I’ve never worked on a program so small that readability didn’t matter. I consider it a crucial ingredient of expressiveness and development speed.

Though your perspective could explain a few of the more atrocious code bases I’ve seen.

jacquesm• 7 years ago

During exploratory programming I don't care at all about readability, just about find a path - any path - to something that works. As soon as I have that readability starts to matter and the first order of battle is then to refactor out all the dead ends and to make the whole thing look good. That's because the project now has long term perspective.

choiway• 7 years ago

I like the way you characterize this but what on your thoughts on why you can't be a wizard with a typed language? Seems to me that if you start with something like Go or Typescript you cover a decent middle ground of foregoing a lot of boilerplate while having code that you don't need to be a wizard to understand.

DonaldPShimoda• 7 years ago

When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing.

I'm not familiar with this use of the term "expressiveness".

My understanding is that expressiveness (as per "On the expressive power of programming languages", Felleisen 1991 [0]) has to do with capabilities that a language has that separate it from another language. C is more expressive than Python in that it gives you direct access to memory management, whereas Python is more expressive than C in that it provides inheritance/OO. (These are just examples.)

Type safety, performance, and readability are all wholly separate from expressiveness, I think. A language's type system and performance benchmarks have nothing to do with the expressive power of a language outright, and "readability" is entirely subjective to begin with.

So: would you mind elaborating on what you mean, exactly, by "expressiveness of [a] language" here?

---

In fact, most of what you (and the linked article) are talking about has to do with the dynamic/static spectrum, not this "wizarding/engineering" spectrum you've coined (though I do kind of like the idea of that for discussing development methodologies).

The article is all about how the dynamically-typed nature of Python allowed for rapid iteration at the beginning of the Instagram project, but has since hindered further progress as they've grown larger. But now they feel they can't just rewrite it all in a statically-typed language because of the engineering overhead involved.

On this note, I want to go to your last point:

I wish there was a language that let you move gradually from one end to the other, exactly when you need to.

With regard to the dynamic/static distinction, there are languages that allow you to move "gradually from one end to the other", and they are (aptly) called gradually-typed languages.

Gradual typing was invented by Jeremy Siek and his PhD student, Walid Taha, back in the mid-2000s at Indiana [1]. In this discipline, you can have a statically-typed codebase with local dynamically-typed regions. You get all of the static guarantees for everywhere that they can be made, and dynamic regions impose runtime checks to ensure consistency. (This connects closely to contracts, which are primarily worked on by Robby Findler at Northwestern, I think.)

Unfortunately (to me), it seems like a lot of these languages are implemented in terms of existing dynamically-typed languages. For example, Sam Tobin-Hochstadt (Indiana) created Typed Racket, which is (of course) built upon Racket but provides a gradual typing discipline. Wherever possible, static types are checked, and everywhere else utilizes contracts to guarantee runtime consistency.

Anyway, all this is to say: the technology exists, technically, but is in its infancy. There's no doubt it'll be some time before it sees widespread use throughout industry. Sam wrote up a brief overview for the SIGPLAN Perspectives blog recently, if you're interested [2].

[0] https://www.sciencedirect.com/science/article/pii/0167642391...

[1] https://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typin...

[2] https://blog.sigplan.org/2019/07/12/gradual-typing-theory-pr...

joshuamorton5• 7 years ago

I expect in this context, expressiveness means something like "the ability to describe the relevant stuff in the code with minimal noise", which might map to having good abstraction.

I find that pythons OOP + functional aspects, combined with a good understanding of the language hits a sweet spot here. One that simply can't be reached in c/cpp/go/java/haskell, and which is much easier to reach than in js/rust/other langs where I think it is possible.

miki1232113• 7 years ago

My definition of expressiveness is basically similar to what Paul Graham (@pg) says. He also calls this "powerfulness" in his famous "Beating the averages" essay[1]. In short, a more expressive language is a language that lets you express more with less. Rails, with its "has_many :books" is very expressive, Assembly is the other end.

The wizarding/engineering spectrum was coined by the article I've linked to[2]. I think the post is exactly about that, first Instagram was wizarding and they had a suitable language for wizarding, now they're engineering, but their language is still only good for wizarding.

As I've said in a sister comment, it's not just about static typing, but metaprogramming/macros/side effects everywhere etc. There's more to the expressiveness/powerfulness than just types. While gradual typing is certainly an improvement, I think we need more research in this direction.

[1] http://www.paulgraham.com/avg.html

andybak• 7 years ago

And the worst thing is using languages/libraries/frameworks that presume everyone needs engineering when you need to wizard.

There's two many people that have swallowed SOLID whole and can no longer see good engineering as a trade-off against other factors.

For example, being strict about having the smallest possible public API and making most methods private protects me from future breakage that might never be an issue (I might never upgrade) but forces me copy/paste vast globs of your code into my own if I need access to something you didn't anticipate. (and that's assuming I have access to your source. Worst case is that I have to reimplement things that already exist in the code I'm interfacing with)

Python got this right. Private methods are a weak or strong hint that you might want to think twice before calling them. But you're the boss at the end of the day.

pytester3• 7 years ago

And the worst thing is using languages/libraries/frameworks that presume everyone needs engineering when you need to wizard.

I think this is why it's easy to point a thousand things built in python which people use every day (like instagram), while in, say, haskell, there are barely a handful (pandoc, facebook spam filter, etc.).

carapace11• 7 years ago

Instagram Server is a several-million-line Python monolith

That's bananas.

Nothing Instagram does requires that much code.

Also, that much Python code means you're doing it wrong.

carapace11• 7 years ago

No, I'm seriously you guys.

Python is too expressive to require mega-LoC for that site.

You could implement an OS, relational DB, spreadsheet, and optimizing compiler all in less than that.

orf2• 7 years ago

You have no idea about their codebase, the implementation details of their features nor how they counted the lines (comments included?). So stating that it’s dumb is beyond ridiculous.

You are right in that it’s certainly a high LoC count for Python, but still...

scrollaway10• 7 years ago

As orf said you have no idea about their codebase. And you have no idea what's included in that statement -- given that they talk about startup time, they most likely are taking into account the whole framework, a plethora of admin and analytics tools, lots of debugging / debug-only infrastructure, migrations, lots of tooling whose sole purpose is making it easier to work in large teams, etc…

(And for the record, Linux is ~37 million lines of actual code, Postgres ~2 million, and gcc ~8 million)

There's nothing absurd about one of the most visited websites on earth to be a couple million LOC.

tahdig• 7 years ago

... many of whom are new to Python.

well, if you ask me to write language X, I would definitely make mistakes for the first couple of weeks/months/years, that is why you need code review, mentoring and education plans for your hires.

Here’s another thing we often find developers doing at import time: fetching configuration from a network configuration source.

  MY_CONFIG = get_config_from_network_service()

I am pretty sure this an anti-pattern, if this code passed the code review, you should make your review process more strict.

  def myview(request):
    SomeClass.id = request.GET.get("id")

> Likely you’ve already spotted the problem

Well, yes, why would you do this? why would this pass code review? why do we we have linters and other checks for dynamic languages

It works great for smaller teams on smaller codebases that can maintain good discipline around how to use it, and we should switch to a less dynamic language.

It seems we are here blaming python for shortcomings of a monolith also, instead of chunking out specific businesses modules to separate services/micro-services.

TO be honest the strict mode seems interesting, but I believe the problems they seem to be facing can be solved by a couple of changes to their pocess and code:

- everyone gets a mentor if they are not experienced in python or django

- code review atleast by two experienced python developers(does not count if you have coded for Java for 20 years)

- teams should try to move their logic outside the monolith(it sounds like they have a monolith)

- write CI tests to measure how much time it takes to import a file, if it takes more than T(line count * LINE_PROCESSING_THRESHOLD) you have to fix your code.

- prepare config and load it before running the actual server, no network call for getting config

All in all, python is suitable for big companies also, the thing is if don't care about the best practices, you would also have problems when you are a small startup, but in a big co it would make it impossible to move forward, trick is to independent of the company size follow best practices and have code review.

scrollaway10• 7 years ago

That's a long post to say "do more code review instead of investing into technical solutions to technical problems".

Clearly, Instagram's solution saves them time. That means faster code reviews which incidentally makes them more accurate. Your post doesn't really make sense.

zestyping• 7 years ago

I like this a lot.

jedberg4• 7 years ago

It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices.

Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again.

To be clear, I'm not saying microservices will solve all their problems or be less work. I'm just saying that with an equal level of effort, they would probably get more overall reliability by having multiple services, they'd be able to use multiple languages, whatever is suited to the task at hand, be able to deploy even more often with less risk, and be able to isolate these types of "change on import" behavior to a much smaller surface on any given deployment.

coldtea11• 7 years ago

Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again.

Yeah, now you'll have 10 interconnected services, 10x the complexity, and everything will have the ability to take down all of large parts of production, plus all the extra pain points of a distributed system...

jedberg4• 7 years ago

You won't have 10 times the complexity if you are taking a monolith and making each section services. You'll have to same dependency graph, it will just use the network to make calls between them instead of being local.

You'll have added complexity with the network calls, which is why I said it wouldn't be any less work, just different work.

civicsquid• 7 years ago

Beyond increasing complexity, I think this also assumes a dependency graph that _can_ be broken down into microservices by the author/the author's team. From my experience a lot of things at this scale have such complex dependencies that unteasing those dependencies is difficult if not impossible without asking several teams to do something differently. And who knows how long that will take?

jedberg4• 7 years ago

That's why you do it slowly. You take a small part of the monolith and make a service that does the same thing. Then you replace the code in the monolith with a call to the service, while keeping track of how often it is called in the monolith.

As you keep moving along, some things that depend on that first service will start calling the new service directly, and some will still call it in the monolith. But your tracking will tell you how often and who is doing that, so you can find out why.

In the meantime, nothing will break, because the monolith is still a pass through proxy to your service.

gtirloni• 7 years ago

I think your comment makes perfect sense.

However, at their scale and with their engineering resources, I can only imagine an attitude of "we can make this work" (the monolith) is easier to justify. The same goes for the micro-services approach (except here you have to justify changing what has been working so far?)

I'd love to read more about the history behind this approach at Instagram.

posedge• 7 years ago

Regardless of whether the monolith or microservices approach is the right way to go for their use case: I could very well imagine that it is too late for such a migration, and that it would hold them back for too long.

konschubert2• 7 years ago

I have a question about a detail in the article:

But if we moved the log_to_network call out into the outer log_calls function, [...] this would no longer compile as a strict module.

My current understanding is that the log_calls method would NOT get executed during module load time!?!

Why would having a side effect in this function violate the intention of __strict__ ?

scrollaway10• 7 years ago

My current understanding is that the log_calls method would NOT get executed during module load time!?!

That's incorrect. log_calls gets executed on import because it's a decorator, so equivalent to `hello_world = log_calls(hello_world)` at the top-level (which does also get executed).

log_to_network in the _wrapped() definition doesn't get executed until hello_world gets called; but outside of the definition of _wrapped does get executed.

konschubert2• 7 years ago

Right! I missed the fact that log_calls is used as a decorator further down.

marcoseliziario• 7 years ago

https://docs.python.org/3/library/importlib.html#importlib.u...

tln• 7 years ago

Avoiding module side effects and making classes and modules immutable seem like two separate concerns

bjoli2• 7 years ago

Not really. Mutation in general, and in modules in particular, inhibit a lot of reasoning about the code, and thus stops a whole lot of optimizations from being possible. Guile (a scheme dialect) recently got declarative modules for that reason, where a top level binding cannot change (i.e. you cannot set! a binding, but you can wrap in it a mutable container and change the contents of that container). This makes procedure calls and variable lookup a lot faster. Andy Wingo wrote about it here: https://wingolog.org/archives/2019/06/26/fibs-lies-and-bench... .

Those optimizations won't mean much for cpython, since Cpython doesn't try to run things fast, but for something like pypy this could be a big deal.

bjoli2• 7 years ago

To quote the article (from.memory): "adding static modules is probably the single most important optimization guile can do in the near future".

The quote is probably wrong, but it is right in spirit.

jbmsf• 7 years ago

I like the idea, but it feels a bit heavy handed outside of a very large team.

I think the first step here is to get away from the assumption that importing a module will have "interesting" side effects. This is not only a problem with Python...

I tend to create mini "dependency injection" frameworks that create a pattern for loading module code at some point well after import. This patterns tends to reduce to wrapping whatever code you have in the module in a function/closure instead of just running whenever.

Again, I like the idea of enforcing constraints with code, but I don't think it's a substitute for educating developers to avoid certain patterns and giving them infrastructure that makes the alternative easy.

accidentaldev• 7 years ago

who would have thought Instagram is a python monolith. ?

ianamartin• 7 years ago

Try Zope.Interface and Pyramid for a framework. You'll be really happy.

alexchamberlain• 7 years ago

I think this is an interesting idea, which appears to embed a stricter subset of Python within Python itself. Have the Instagram engineers tried floating this with the wider community via established channels like Python-Ideas or discuss.python.org?

time4tea• 7 years ago

Wow. Talk about solving the wrong problem!

Millions of lines of code in a monolith. 20s start up time. Meta monkey patching. One unit test per process... Yikes!

Software architecture, anyone?

Maybe Instagram should get a copy of Michael Feathers' book...

k_sze• 7 years ago

Another thing that I would like to see in some kind of strict mode is the ability to mark explicit exports like in JavaScript modules. I often want to import multiple things globally at the top of a module because they are shared by multiple class or function definitions that I am writing. However, such imports end up being exposed to and usable by the consumers of my module, even though the consumers should really have imported those things at their source instead of via my module.

There are currently maybe two ways to tackle this “problem”, without a strict mode:

1. Don’t import at the global module scope; but that’s a bit tedious.

2. Import with rename, like `import os as _os`, and then leave it to the principle of “we’re all consenting adults”. I.e. if anybody imports and used things that start with an underscore, it’s clearly their fault, not mine.

andreareina• 7 years ago

3. Import as normal, and leave it to the principle of "we're all consenting adults"; unless something is explicitly called out as being part of the public API I consider Law of Demeter[1] "violation" the same as accessing _var.

[1] https://en.wikipedia.org/wiki/Law_of_Demeter

rurban• 7 years ago

I like that idea, it's just not that easy. How to do define module versions and inheritance, when you are not allowed to do global assignments in the module. declarations only, and no IO or global side effect is fine, but declaring versions and inheritance need to be allowed in global scope.

I added these ideas here: https://github.com/perl11/cperl/issues/406

Python at Scale: Strict Modules

Related Stories