Reasons to avoid static type checking in Python

https://typing.readthedocs.io/en/latest/source/typing_anti_pitch.html

by samwho • 2 years ago

47 54 2 years ago

typing.readthedocs.io

luhg• 2 years ago

Feels like a bunch of BS excuses.

jupp0r• 2 years ago

Which is great as it documents that there aren't any real ones.

BenoitP• 2 years ago

Yeah the formulation is quite biased, but to some extent there is some truth in the statements:

Your codebase is old, large and has been working fine without static type checking for years. While Python’s type system is designed to allow gradual adoption of static type checking, the total cost of adding type annotations to a large extant codebase can be prohibitive.

That's a convoluted and quite dishonest way of warning that you're trapped if you don't enforce types from the start; but it does have the merit of being on the list.

typechksux• 2 years ago

Pleasing static type checkers requires a non-zero amount of busy work

Heck yeah! Why would I want to find problems ahead of time when the whole thing could blow up at runtime with cryptic errors. Or better yet, no errors because I’m such a ducking elite programmer that I never make mistakes. Read the code. The code don’t lie!

This is a lame, low effort blog post. Jesus Christ.

guax• 2 years ago

Your comment is incredibly hostile. They're expressing their opinion on the language they use and like. A language that's always been that way. I find it incredibly tiring that every language goes to the fase of people complaining it's not other language and fighting to change it instead of using another one.

nmca• 2 years ago

I eagerly await ruff-but-for-type-checking. Python type checkers are all excruciatingly slow at the moment.

diarrhea• 2 years ago

Yep. Speed is one factor, but also standardisation.

I’ve always been using mypy, but heard good things about pyright. It’s what VSCode uses by default as well, unless you jump through hoops. So I have pyright in the IDE and mypy everywhere else, yet they don’t agree with one another in various cases. Exhaustiveness checking and type inference in structural pattern matching, and async TaskGroups are some examples. One of my code bases has mixed “type: ignore” and “pyright: ignore” (forget the exact syntax) for that reason. Pretty bizarre and not something you’d see in any other language!

I wonder why mypy isn’t the blessed standard and pyright was able to catch up on (and in parts overtake) it. Mypy had a large head start and enjoys tons of development effort. Good job to Microsoft I suppose?

imron• 2 years ago

Use dmypy - the daemon version of mypy (installed when you install mypy) that runs 10x the speed for local edits/changes.

https://mypy.readthedocs.io/en/stable/mypy_daemon.html

(But I also await ruff type checking)

roenxi• 2 years ago

Scepticism is warranted for partial type checking. Something I've noticed in partially-typed situations is they are the worst of all worlds. The type checker soaks up a lot of time, but the silly bugs that type checking would have caught still get through. Having 1 library that doesn't support type hints is a major problem if I'm trying to use a type checker.

Therefore, I think "... the authors have no desire to ever make type hints mandatory, even by convention." is useless. I'm sure the authors are honest about their desires, but frankly either everything gets type checked or practically nothing does. It is unlikely that there is a social equilibrium where some things get checked and some do not. Not only that, but clever libraries that aren't written with type checking in mind are often a devil to hint retroactively.

galaxyLogic• 2 years ago

I think there is a big difference in what we type-check. The interfaces of public methods should be declared and type-checked. What happens inside a function not so much.

jandrewrogers2• 2 years ago

Strong agree that half-assed type checking is the worst of both worlds. Most of the effort of real type checking but only a fraction of the real-world benefit.

agubelu• 2 years ago

I agree with this. For example, it leads to weird situations where a linter with a strict type checking configuration marks some code as a type error, but it runs just fine.

Which, in turn, leads to you ignoring type errors entirely, since you don't know which one are spurious and which ones aren't.

mharig• 2 years ago

There is such an equilibrium: type check API functions and do not clutter the rest of the code with type hints/declarations.

treflop2• 2 years ago

Personal opinion here but I’d put TypeScript in the S-tier for “static typing overall enjoyable-ness,” Java in D-tier and it pains me to say Python in F-tier.

th0ma52• 2 years ago

Are those tiers good or bad? What is the order?

treflop2• 2 years ago

See https://en.m.wikipedia.org/wiki/Tier_list

bmacho• 2 years ago

SABCDEF descending

MrJohz2• 2 years ago

Yeah, I've recently removed type checking in a Python project because it was causing more false positives than true positives, and working around the type checker's complaints was just getting too tedious.

I'm a big fan of type checking in general, and Typescript is a great example of a type checker that allows you to statically define all the dynamic idioms that you'd normally use. But for various reasons, that just doesn't seem to have happened in the Python ecosystem. Instead, you can either write normal dynamic Python without types, or you can restrict yourself to a vaguely Java-like subset of Python with hard-to-use generics and confusing rules about forward references.

I wonder if it would have been better to have taken the Typescript approach and have Mypy and all its annotations developed as a separate language, rather than as part of the Python language, with the expectation that they could eventually be merged at some point. As it is, it feels like both sides have been very constrained: Mypy has struggled to innovate on expressive type-level syntax because it only operates on standard Python code, and Python has struggled with balancing the various goals that different groups have for youry annotations.

The result is a kind of mess where it's still painful to type basic things like generics (yes, that's changing, but not if you need to support older Python versions), but you also have weird syntax that requires type annotations, even if they're meaningless (like with dataclasses).

jowf8fa9sdf• 2 years ago

Have you tried pyright? I find it way better than mypy.

BigJono• 2 years ago

I don't know how you can put TS in the S-tier when the community is hell bent on making every codebase as unreadable and complicated as possible.

petesergeant2• 2 years ago

when the community is hell bent on making every codebase as unreadable and complicated as possible

I don’t understand what you’re getting at, but clearly you’re expressing some pain here: what are some examples?

jandrewrogers2• 2 years ago

I’ve written a lot of production Python and also a lot of thoroughly and rigorously statically typed code in C++. I use both regularly but there are clear tradeoffs in this regard. The main reason to avoid static type checking is that the programming language makes it difficult to do it effectively due to trading it away for other benefits. This does not change the intrinsic value of static type checking.

Python works great for cases where the scope of a particular function is tightly constrained around a tightly constrained set of inputs. In this scenario, it is relatively easy to test fitness for purpose, and that testing can replace static type checking as a practical matter. Many bugs sneak through in Python that C++ would find at compile-time, but that’s okay because the domain is so narrow that you can squeeze them out in testing with minimal overhead.

The minute you start trying to build a more generalized library of functionality in Python, the combinatorial bug factory starts to become a major drag. While you can add type checking, it is clumsy, slow, and less thorough in practice than you would expect with C++. Python is not designed to provide the degree of static type checking something like C++ can. Many types of code benefit from thorough static type checking, and for those use cases something like Python has a high cost.

This is fine. I understand the problems for which both languages are best suited. Right tool for the job and all that. Static type checking is generally beneficial in software but it isn’t always worth paying the tariff if the problem domain is narrow enough. It is quite possible to overfit static type checking to simple cases that don’t justify the effort.

jongjong6• 2 years ago

Yes. Great to read this type of nuanced comment. This nails it. Sometimes components are generic enough that type checking does not add any value. Type checking is useful when you want to frame your logic in terms of your business domain. But you don't want your business domain to leak into low level reusable components. Why constrain them to a specific domain when they could suit multiple domains without any effort.

Just think of a function which generates a random number for a lottery app. Do you want to call the function 'generateRandomNumber' or 'generateLotteryNumber'? There is no reason to constrain your function to the 'lottery' use case when it could also serve an infinite number of other use cases.

The function which is used as a dependency shouldn't concern itself with the specific business case towards which it will be applied.

As you move towards the leaves of the dependency hierarchy, the abstraction becomes more about technology and less about business domain.

a_t482• 2 years ago

I don’t quite follow here - you can still declare interfaces (Protocol) to declare what behavior your code needs.

baq• 2 years ago

I’d probably agree with you 5 years ago, but in those 5 years typescript has shown that you can take a completely dynamic language, add a very powerful and well thought out type system and the resulting language is so much better for non-trivial projects.

So if you ask me, who has been programming Python since before new-style classes, if types make sense for Python, the answer is a resounding ‘yes’: just steal as much typescript typing as possible.

LarsDu88• 2 years ago

And not only that, but you can take that statically typed code and build JIT compilers on top of them

BiteCode_dev• 2 years ago

I wish we'd steal more. TS type systel and syntax is much better that what we have.

And I hope one day iter[] and callable[] will be made available, cause importing typing sucks.

Chris_Newton• 2 years ago

At first glance, JavaScript/TypeScript and Python seem like similar cases, but in practice, I have never found them to be so.

The JS/TS community has enthusiastically adopted static typing, to the point where (a) the type system in TypeScript is reasonably clean and intuitive for everyday work, (b) almost every major library uses static types and specifies them in its documentation, (c) almost all major tools and development environments pick them up and work well with them, and probably as a result of those things, (d) almost everyone now uses static types extensively, at least for new development.

The Python community seems far more hesitant. I don’t know why. Maybe it’s because we’d just had such a long transition from Python 2 to Python 3 and introducing static typing feels like another big change? Maybe it’s because Python’s static type system simply hasn’t been as clean and tidy as TypeScript’s, particularly before the recent generics changes in 3.12? Whatever the reason, in practice none of (a)–(d) above reliably holds in the Python world. Personally I do still prefer to use static types in new Python code, but it’s frustrating and often needlessly time-consuming when popular libraries don’t.

jongjong6• 2 years ago

Personally, I don't use type checking because the benefits are extremely marginal given how I write software.

- I avoid complex function/method interfaces. My coding philosophy revolves around relying on simple primitives as arguments and return values and keep complex state fully encapsulated.

- I use a TDD e2e approach which gives me excellent code coverage and catches 'type mismatches' early as such issues cause functionality problems that are caught in my tests. Type checking is redundant.

- My software is made up of small modules. The lower level modules are generic enough that it doesn't make sense to restrict to specific types or assume a specific business domain.

At a psychological level, I find that type checking tends to encourage developers to design complex function/method interfaces which lead to worse, harder to maintain software and encourages leaky abstractions and passing around 'live instances'. It encourages developers to add unnecessary constraints which make low level modules less versatile and cover fewer use cases than they could otherwise have.

Interface names of low level modules often end up unnecessarily referencing the specific business domain which should be a higher level concern.

j-pb2• 2 years ago

Thats what I call "Uncle Bobs ossification disease" where you spend too much time on process and not enough on getting shit done. Because everything is broken up again and again it just piles and piles, making it harder to change things. Everything is intertwined through (often unwarranted) reuse and even the non-reused stuff is smeared over dozens and dozens of functions that are used only at a single call site. But wait there's more, for every of these functions theres half a dozen tests that ossify some implementation detail (because function boundaries are no longer meaningful).

You end up with thousands of lines of code that do very little except for being the physical incarnation of "the process" god.

That stuff is so 2000.

Type systems give you robustness and whole program consistency proofs without a single line of extra code. No manually writing tests to ensure type safety and you get help writing semantically meaningfull functions, instead of whole object graphs that need to be called in a complex dance just to avoid having 4 function arguments.

jongjong6• 2 years ago

If you saw how little code my projects require, you would realize that this is incorrect.

I've built a decentralized cryptocurrency exchange that can work with most blockchains in just 5000 lines of code. Has been running for 4 years with zero bugs.

I've written a cryptocurrency from scratch with only about 4000 lines of code and have not had any bug reported after 3 years of continuous operation. On the other hand, projects like Bitcoin and Ethereum are hundreds of thousands of lines.

I've also written a no-code platform in less than 4k lines of code which supports complex views with real time updates, pagination, indexing, access control, multi-tenancy and has been bug-free since it launched. I built the whole thing in about 3 months part time.

All of these built with plain JavaScript/Node.js. I can support all of these projects in parallel because I rarely need to update them.

code_biologist• 2 years ago

Very fragmented codebases like that have turned "inline function/method" into my favorite automated refactoring. I'm not an IDE person generally, but I open one up for that specific use. My favorite game to play is to see how many times I can inline with unchanged or net negative lines of code in a commit.

Getting rid of dumb tests is often where the real line savings is. No need to test your specialized internal function that's just wrapping filtering an array. Like, yes, array filter does in fact filter according the predicate you give it.

arialdomartini• 2 years ago

The wrong assumption in that page is that Static Typing implies the need for type hints or annotations.

In fact, type inference is a thing.

Most of the languages with Hindley Millner type system (such as oCaml, F#, Haskell, PureScript) do feel dynamic (e.g., developer does not need to hint the compiler) yet they are purely static typed.

What is the argument against having Python checking the types, without asking the developers to change their code?

tsss• 2 years ago

Haskell has to pay _a lot_ to achieve full type inference. A price that is way too high imo for a feature that nobody uses anyway. Among many other things it precludes subtyping, which I assume Python relies on heavily.

tome• 2 years ago

I suspect almost every Haskell uses full type inference almost constantly, if only to avoid having to write out top level types by hand. I know that applies to me. I've never considered that I paid a price for it. For me it was just something that came free with a language that is nice in other ways too. What price are you seeing? (For me, absence of subtyping is a feature.)

rich_sasha2• 2 years ago

There are cars and motorbikes, and many other things. They have different benefits and costs.

Python is a motorbike. It is small, convenient, nippy, fuel-efficient. It is also terribly unsafe, holds no luggage and no protection from the weather.

Statically-typed languages are cars. Stable, safe, predictable, require a lot of infrequent maintenance, weather-proof. Not convenient in heavy traffic, but hey, you put on the heater seats and the stereo and you don't mind queuing.

Now, I find the effort to add typing to Python like trying to make the motorbike safe. You double the wheels to have a full 4-wheel vehicle, add a roof and windshield... And you end up with something that has th upsides of neither a car nor a motorbike.

Where people find it convenient, sure. But the evangelism of "types everywhere" seems to not get it.

petesergeant2• 2 years ago

To steal someone else’s comment from above, I feel like TypeScript has disproved this, and that you lose almost nothing and gain a huge amount with a high-quality optional typing system

rich_sasha2• 2 years ago

Maybe. I don't use typescript, and rather this is my impression of state of things in Python.

Anything, including gradual typing, can be done well or not well.

crystaln• 2 years ago

None of these reasons are particularly compelling for any case I've encountered, other than your code base is old and / or poorly designed, and you chose to write it in a language without static type checking.

BerislavLopac• 2 years ago

I'm not really sure what is meant by "avoiding static typing" in Python -- it's just like saying "avoiding unit tests", which is done by not running them (or even writing them) in the first place.

Python is strongly typed, but it does not mandate static type analysis -- unlike most other typed languages, where it is mandatory at compilation time. But if you choose to use it, there are tools which will run it for you at any point in time (mypy being the reference implementation). You don't even have to write any special annotations for that: the checker will try to infer the type when possible, or use Any when it isn't.

The crucial thing to understand is that Python is still dynamic at runtime -- there are no checks that an argument is of the correct type defined for the corresponding parameter. So your program can fail the static analysis but correctly run in practice, unlike most other statically checked languages.

Another effect of the above, which is often overlooked, is that many developers tie themselves into knots to get the type annotations right on unit tests. This is completely missing the point, as unit tests and static typing are at completely cross purposes: while static analysis evaluates, as the name implies, the code in its static state, the unit tests only make sense when they are executed, i.e. at runtime.

So the practical rule of thumb, learned from practice, is:

1. Regularly run the static analysis on your application (and library) code, adding any necessary annotations, to weed out various problems that might arise from mismatched types.

2. Write thorough unit tests -- ideally with plenty of property tests with tools like hypothesis [0] -- and run them regularly.

[0] https://hypothesis.readthedocs.io

librasteve• 2 years ago

Python is a tool that is meant to serve you. Python is a big tent, multi-paradigm language that generally allows you to do things in the way that best suits your needs, as best determined by you.

^^ this excerpt from the post made me spit my wine out ... this is what https://raku.org is for!

Surely, according to the zen of python https://peps.python.org/pep-0020/, the pythonic way is:

There should be one-- and preferably only one --obvious way to do it.

At least raku is architected on consistent class types from the get go so its gradual typing is not just an annotation / type hint bolt on.

sgorawski• 2 years ago

Also, in Python (unlike TypeScript or PHP) you need to be sure that the code regularly passes type checking to have confidence in your type annotations.

I've dealt with my share of code with types that were almost correct but not really and it's been quite frustrating.

For interested I've written an article about it: https://medium.com/@sgorawski/python-types-have-an-expectati...

Reasons to avoid static type checking in Python

Related Stories