Tests aren’t enough: Case study after adding type hints to urllib3

https://sethmlarson.dev/blog/2021-10-18/tests-arent-enough-case-study-after-adding-types-to-urllib3

by quentinp • 5 years ago

205 198 5 years ago

I've seen a lot of push back on adding type checking to Python but we had a similar case at my company where we tried it out on a new project and the clarity and readability of the code was immediately beneficial to the entire team. Perhaps it's something well suited to larger codebases.

hobs• 5 years ago

I think it's well suited to anything really - the amount of casual problem solving and inference you can make from some simple types is pretty big in my experience, and Python's approach to allow you optionally buy into it is really nice.

tester756• 5 years ago

It's $current_year and there's still debate whether checking stuff at compilation time is better than at runtime?

novok3• 5 years ago

People who don't think types are a good thing need to work in a statically typed language for a year or two and then see what a difference it makes in reality. Unproductive Java bureaucracy != static typing.

I think the people debating it never tried it seriously.

kzrdude3• 5 years ago

That's not really the debate in Python :)

Almost every Python user now has to "deal" with type annotations. It's tempting to gradually add type annotations, it's nice documentation.

But it also rubs me the wrong way to have annotations that are never checked(!). In many codebases, you might just have "casual" style type annotations in Python, and nothing ever asserts that they hold. That's nagging on me, a bit.

handrous5• 5 years ago

I want type checking on pretty much anything that will ever exceed about two screenfuls of code. If I can't keep the whole thing in my head at once, I want the computer to do it for me. That's the point, right? Making computers do stuff for us so we don't have to?

david4224• 5 years ago

I kindof think of them as a giant set of unit tests. The compiler/linter etc. can check every variable and every function call to check to make sure you didn't mix up your types, which _will_ blow up at runtime if you got them wrong.

So rather than write them all by hand, just get your tools to do it.

matsemann• 5 years ago

Just wish Python's typing was better. But it's impossible to type hint the crazy "pythonic" code out there. Like the kwargs used to do a Django query.

dado3212• 5 years ago

Honestly will never go back to languages without type checking, it prevents so many bugs and is a huge help in understanding code you haven’t worked with previously.

SketchySeaBeast• 5 years ago

is a huge help in understanding code you haven’t worked with previously

This is huge for me. As someone who takes on already completed projects, it's a huge help with debugging and understand what's going on without requiring you to know the whole system forward and backwards. Sure, you still need to build a mental map of the general code flow, but you can look at a single function and clearly see the obvious inputs and outputs. Combine that with a a stack trace and you can debug that method as a single unit and then start to look at where it's called and what its downstream effects are. You don't need to start from the very beginning of the call and then follow it through, keeping mental track of what is available and in what form when and where.

seiferteric• 5 years ago

It is kind of ridiculous not to have types. I think in the old days handling types felt to heavy for scripting languages, but now with type inference and stuff I don't think it is any longer.

handrous5• 5 years ago

Honestly will never go back to languages without type checking, it prevents so many bugs and is a huge help in understanding code you haven’t worked with previously.

I see static types as one of the most powerful communication tools around, as far as code goes. I can't relate at all to people complaining that they waste time. They must work very differently from how I do, is all I can figure. It's that, or they don't realize how much time they're losing to communication-related tasks, or refactoring, or writing (and maintaining!) extra or more verbose tests, or having even one more bug per year make it to production, or whatever, that'd be saved by static types, so aren't correctly accounting for the time savings. One of the two.

adrianmonk• 5 years ago

Consider these 4 possible combinations for programming languages:

(1) Low-level, static types

(2) Low-level, dynamic types

(3) High-level, static types

(4) High-level, dynamic types

For whatever reason, historically #1 and #4 have been most popular. C, C++, Pascal, Ada, and Java are #1. Python, JavaScript, Perl, and BASIC are #4.

There haven't been a lot of #2 or #3 languages. Some #3 languages (TypeScript and Python with types) have come along, but relatively recently.

A person who experiences only #1 and #4 might notice that they can whip up programs faster in a #4 language than in a #1 language, then falsely attribute that difference to the static types. Whereas the real reason is working at a different level of abstraction.

alpaca128• 5 years ago

I can remember that in the beginning it felt cumbersome to me because I wasn't all too familiar with that language's type system and so I had all kinds of errors thrown at me.

But it's something you just have to get used to, and now that I understand it much better I feel more productive and have more confidence in my code. And the communication aspect is definitely a great help too. No handwritten documentation can be this consistent, completely independent of who touched the code (though to be fair, it's still difficult to get the naming right).

I can't imagine the people calling it a waste of time got over the hump in the beginning. To me it's obviously a timesaver. It does a tedious, difficult (for humans) task and does it quickly & with perfect accuracy. Beforehand worrying about all the type signatures and interfaces felt like 4D Sudoku across various modules, now I can concentrate on the interesting parts.

zz865• 5 years ago

I feel the same way but wonder if I'm right when the majority of jobs are JS and Python.

shrimp_emoji• 5 years ago

Both of those languages' communities have essentially admitted that not having type checking was a mistake and try to patch it with TypeScript and MyPy.

tunesmith2• 5 years ago

I've been a types advocate for years, but it wasn't until working with Typescript that I started experiencing some of the downsides...

To me, ideally, types are supposed to be a benefit not only in safety, but in understanding the intent of a piece of code more quickly. For an api or library interface, review the types to see what its intentions are.

But there's something about the typescript type system, with all the picks and keyof and typeof... sometimes it just feels like it's way too easy to go overboard, to the point that it occludes meaning. I understanding struggling with types if you're struggling with figuring out exactly what your boundary does and does not allow, but when you're struggling with types just because you're struggling with the kabillion different ways that some other typescript programmer chose to use the utility types... there are times when I feel like even Scala is easier.

Zababa4• 5 years ago

The problem is that typescript is here to type existing JS, and existing JS has been written without thinking about types. "fresh" TS might be better for that.

Shacklz• 5 years ago

with all the picks and keyof and typeof...

Depends of course a lot on the codebase but all typescript codebases that I've seen so far and considered "well-maintained" didn't really use keyof and typeof all that much. The only way I can imagine how one ends up with lots of those keywords is when you start with a dynamic language approach, and then tell the compiler afterwards what that type might be, instead of defining the type beforehand - might that be the issue?

IshKebab2• 5 years ago

Your criticism boils down to "it's possible to overcomplicate things". Sure if you completely remove static typing then you can't overcomplicate static typing. But is that really an argument against static typing?

m12k• 5 years ago

I've only been in the industry for ~15 years, but it still feels like every year, some ecosystem discovers the value of something that another ecosystem has taken for granted for decades - type-checking, immutability, unidirectional data-flow, AOT-compilation, closures, pure functions, you name it. I'm glad we seem to be converging on a set of best practices as an industry, but sometimes I wish we were spending less time rediscovering the wheel and more time building on top of and adding to the actual state of the art.

axiosgunnar• 5 years ago

I think the problem is to figure out what the best practices actually are.

What we are observing here is „the market fixing it“.

The process is messy and redundant, but effective.

taeric6• 5 years ago

I've been alive long enough to see that most things are useful, and all things are oversold.

More, the nice easy things to build with major restrictions pretty much gets thrown out the window for complicated things that have constraints that most efforts don't have. This isn't just a software thing. Building a little shed outside? Would be silly to use the same rigor that goes into a high rise. Which would be crazy to use the same materials engineering that goes into a little shed.

nllsh• 5 years ago

Not that I disagree, but I feel like you are overselling the simplicity of [building a shed](https://en.wiktionary.org/wiki/bikeshedding).

Zababa4• 5 years ago

The metaphor doesn't really works, as in software lots of high rise start as a little shed.

jeffbee• 5 years ago

Yes, and when that ecosystem discovers these obvious facts, the discovery is always described as a "journey" in the accompanying blog post. Having sense and good taste at the beginning of a project doesn't warrant a blog post but slowly stumbling over isolated aspects of good taste, now that's a journey.

d0mine• 5 years ago

Programming is sufficiently complex field that we can find examples when the opposite things are the best: it depends on context whether you need more or less types.

AlexCoventry• 5 years ago

I think the limiting factor in the case of python getting type hints was that it was never designed for type safety in mind, and that it took a while to establish consensus on a good type-hinting system.

I don't think it's a matter of reinventing the wheel, in this case, more a matter of bolting something like a wheel on a system which didn't start with wheels.

mdoms• 5 years ago

It is extremely funny to me watching Silicon Valley types slowly (very slowly) re-invent everything we knew about programming languages decades ago.

pjmlp3• 5 years ago

Well for the hipster culture what we were doing wasn't cool.

david4224• 5 years ago

I love static typing/type hints if for only 1 thing - code maintenance.

Even code I wrote six months ago.

Not having to dig through 6 functions deep to try to figure out whether "person" is a string, or an object, and if it's an object what attributes it has on it etc. is huge. And not to mention that some clever people decide - hey, if you pass a string I'll look up the person object - so you can pass an object or a string - which makes all sorts of convoluted code paths when someone else was looking at "person" and only saw one type so now their function doesn't work on both types etc.

I hate having to waste time figuring out the type of every variable and hold it in my head every single time I read a piece of code.

sseagull2• 5 years ago

This is absolutely how I feel. I've mentioned previously taking over a project, and just not knowing the type of anything took me months to overcome.

Also, type hints really help your IDE, even catching errors before you even run tests.

There's also a visual cue that you are doing something wrong: If a function returns 4 levels of Union[Tuple[List[int]], Optional[str]........ Then you are doing something too complex and the function should be broken up.

jfabre3• 5 years ago

I never understood this argument. In what kind of shop are you working that passing a string named person to a method expecting an object is tolerated. Or even passing different types that don't share a common interface.

This would never fly in a code review in any of the companies I've worked for.

kennywinker3• 5 years ago

I've seen essentially this code in so many organically grown codebases (when they grew up without types). It's usually close the the UI, because someone had to quickly add an alternate path to support some new user interaction

    function find_user(person) {
        if user is string {
            query_by_name(person)
        } else {
            query_by_name(person.name)
        }
    }

and yeah, we all know it's kinda messy, but also that logic has to live somewhere and we need this feature asap so it passes code review. I wrote a test for it, ship it.

elzbardico• 5 years ago

This was probably just a silly example for a quick explanation.

  But all it takes is a method that expects an integer Id to receive a string representation of said id because of some obscure path in code that notwithstanding your 100% line coverage the team is so proud of, was never exercised on tests because nobody can have 100% branch coverage

cyral• 5 years ago

If only there was a way to enforce these parameter types automatically

dec0dedab0de2• 5 years ago

I personally love it, and wish every library worked this way. My argument is why go out of my way to make it not work, when it would be easy to make it work. This is because I think of modules/packages as user facing programs that are easy to tie together, instead of simple building blocks.

What I really wish existed was a built in way to cast and validate, or normalize and validate. I never care if something is a string. I care that if I wrap it in str(), or use it in a fstring, the result matches a regex. Or if I run a handful of functions one of them returns what I need.

The only benefit I can see of type hints on their own is it makes it easy to change a callable's signature, but I think that's best avoided to begin with.

dboreham• 5 years ago

You'd think. But I've seen many many many examples of this pattern in production JS code.

tialaramex• 5 years ago

In C++ you're only ever one missing "explicit" from introducing such problems.

Suppose I call fire(bob). Programmers from other languages might reason that since fire is a function which takes a Person, bob must be a Person. Not in C++. In C++ the compiler is allowed to go, oh, bob is a string and I can see that there's a constructor for Person which takes a string as its only argument, therefore, I can just make a Person from this string bob and use that Person then throw it away.

To "fix" the inevitable cascade of misery caused by this "feature" C++ then introduces more syntax, an "explicit" keyword which means "Only use this when I actually ask you to" rather than as a sane person might, requiring an implicit keyword to flag any places you actually want this behaviour to just silently happen.

This way, hapless, lazy or short-sighted programmers cause the maximum amount of harm, very on-brand for C++. See also const.

madeofpalk2• 5 years ago

We have tests, and static types, because developers are people and people make mistakes.

You can't say "we simply don't allow bugs!" because it's a lie. Why rely on a another person manually checking for silly mistakes when the computer can do it for you?

wvenable3• 5 years ago

The main argument for dynamic typing is speed in prototyping but I find that's opposite for me. I'm much more comfortable rapid prototyping and ripping stuff apart when I have a strongly static typed environment telling me what I just broke.

Doing radical refactoring often involves just making those changes and then fixing all the IDE or compiler errors until it runs again.

ttymck• 5 years ago

I agree with your point on prototyping. I've never been more productive than when I have the (Scala) compiler acting as a second set of eyes, essentially looking over my shoulder, checking my business logic.

Enginerrrd• 5 years ago

Dynamic typing was great before I knew anything about programming. I'm talking like, at a middle school level. Fewer "Silly" errors.

After university, the opposite became true. No difficult to diagnose undefined behavior because of ambiguity in typing.

agumonkey• 5 years ago

I used to have loads of fun abusing Eclipse real time type checker, imagining software live with typed interfaces.

The IDE was my logical buddy, and every idea's possibility was rapidly shown with it. And I need to massage things a bit, I go faster because I know what's missing.

The only time I liked eclipse/java :)

munk-a• 5 years ago

When I'm prototyping I tend to go inside out in a layered fashion - some days I am really feeling the data layer - other days I like to work closer to the fringes. To this end type hinting serves as a quick and dirty code contract before all my pieces are in place. I can splat out a bunch of low level definitions that I know I'm going to need and then come back the next day to add in struts - remembering my choices easily as I go.

I know this isn't the approach of choice for most folks but hey - I'm working with ADHD so I've got to make some allowances for some neurodiversity.

tabbott2• 5 years ago

I would argue that the really big benefit of dynamic typing is that it enables a really nice interactive interpreter shell experience. I think it's also important from a prototyping standpoint that Python's static typing model does a lot of inference -- you don't have to add an explicit type annotation on every single variable.

orwin• 5 years ago

The biggest reason why i like type hints is because it force me to reflect on the datatype i want to use before implementing my code.

Last week, i could've done either a dataframe, a list of list, a list of tuple, a dict of tuples, a dict of lists (this was a bad idea that did not survive more than 2s in my head) or a list of dict. I started coding with a dataframe in mind (i guess i wanted to show off my numpy/pandas skills to my devops colleagues), but adding type hints to my prototypes shut down the idea pretty quick: lot of complexity for nothing.

tetha4• 5 years ago

Hm, growing somewhat experienced, I find myself adapating an old quote more and more: Sufficiently advanced static typing is indistinguishable from dynamic typing.

Now, I know, it's not true. It's entirely possible to build weird things in python that are provably impossible to typecheck statically. But modern language servers and their type inference capabilities in rust, terraform, or even straight up python are very impressive.

steve_adams_86• 5 years ago

when I have a strongly static typed environment telling me what I just broke.

Yes, I'm a total scatterbrain. Types let me remind myself later that I did in fact forget what I'm doing and what I did. It lets past-me protect future-me.

Barrin92• 5 years ago

the main argument for dynamic typing in particular in the context of object oriented programming is decoupling. It is always the receiving object's responsibility to handle whatever they get.

if you write dynamic OO languages with a static mentality in mind, i.e. you try to enforce some sort of global type expectation before the program runs, then obviously static languages are better, because you're trying to write static code.

Benefiting from dynamic languages means ditching that mindset altogether.

tshaddox• 5 years ago

I think the only reason dynamic typing can speed up prototyping is that it allows you to make certain type errors that you may never encounter at runtime while prototyping.

tiew9Vii• 5 years ago

It depends on the language but I find I'm also far more productive with strong types.

When I use a dynamic language I get no errors in dev, I need to run/invoke the program to see if it works. It may appear to work fine as I haven't executed a specific code path hence dynamic languages have extremely high test coverage. With dynamic languages I am delaying my feedback loop, I may get some visual output quicker but that doesn't mean my program is correct.

With a strongly typed language and utilizing types you use the compiler to guide you. The compiler says hey, this isn't correct, fix it, you go fix the error and recompile and repeat.

I've used Elm before and it's the only time I had a complex Javascript UI just compile and work first time. It's like a wow, did that just happen.

With Typescript it's not quite to the level of Elm but find my experience working with React etc far more productive. Typescript says hey, that's wrong, I expect ... you gave ..., you work through the errors and when it runs generally there's less silly mistakes than when I just use Javascript.

I'm learning Rust, the compiler error messages have greatly helped. When you compile it says hey, you tried to do ..., maybe you want ... instead. Not to sure what the suggestion is I try it and 9 times out of 10 it works, compiles, program runs.

With types you generally get better IDE auto complete support etc.

Now i'm using Python for my day job. My experience has been painful, discovering what arguments functions take, passing in wrong values, needing to run slow test suites, finding errors at runtime. Yes you can use type hints and I do but I find them far less reliable.

I guess I'm not a very good programmer so learnt to lean on a compiler to do the hard work for me, and if you have good type support you can lean on types more to get the compiler to help you more.

In Haskell I can write complex logic by writing out the types and ADT's. I've written whole programs with tests to verify the logic without writing a program. I find this incredible efficient for prototyping ideas, just write the types, the functions signitures etc etc. Once that is done you implement the functions, hit compile then boom, your shocked it just worked first time running.

mbrodersen2• 5 years ago

Agree. I usually think types first and quickly sketch the whole application without writing any code. So,when I start writing code it just works end-to-end.

handrous5• 5 years ago

I hate having to waste time figuring out the type of every variable and hold it in my head every single time I read a piece of code.

If a codebase doesn't have static types, it damn well better be set up to be highly grep-able. Including dependencies and frameworks.

This is why Rails pisses me off so much. No static types to help you out, and you can't grep (can barely google, even!) methods and properties that aren't defined anywhere until runtime. Is this from core? Is it from some 3rd party gem? Well fuck me, this file doesn't even tell me which gems it's relying on, so it could be literally anything in the entire goddamn dependency tree.

rightbyte• 5 years ago

... grepable ...

This is so important.

It is also the reason why I like global variables. They are accused of making a spaghetti mess but ... in my experience the opposite is true.

Fancy patterns are way worse to reverse engineer than simple flat long functions accessing globals. Easy to debug too!

blacktriangle2• 5 years ago

That's some Rails stupidity there, not a dynamic language problem. Autoloading symbols by name is straight up dumb.

As for greppable though...then you may as well be using a static language. The point of a dynamic language is to be dynamic, ie you can do those things at runtime.

layer84• 5 years ago

I hate having to waste time figuring out the type of every variable and hold it in my head every single time I read a piece of code.

For the same reason, I’m not a fan of type-inferring variable declarations.

dmart• 5 years ago

Yeah, when writing type inference is obviously nice, but it can be annoying to try to go back and read.

I think the best experience is having a language server annotate the inferred types (like how rust-analyzer does it.) But even then, it can become hard to read code on GitHub or somewhere where tools are not available. Granted that's becoming less and less of a problem, and even GitHub allows using some VS Code extensions now.

lamontcg2• 5 years ago

I'm okay with "var = new FooBarBazThingyWithALongName()" because I don't need to see the type name twice there.

In an IDE you can get the type annotation from the IDE over every inferred var type, but I don't like requiring an IDE to see that information and like it showing up in 'less' as well.

angelzen• 5 years ago

This is doubly true as experienced programmers argue that designing the data structures is the hardest part of coding. Code follows semi-automatically.

nitrogen2• 5 years ago

I'd add data flows as another level above data structures. It helps to think about how data flows into, through, and out of a system, then it's more clear how the data needs to be packaged, and from there, the code follows semi-automatically.

Tangentially related: I think it'd be cool if there was a development environment that combined a node-based dataflow editor with normal text editing, so pure plumbing could be implemented visually, but embedded within (and translated to) textual code.

karmakaze• 5 years ago

I learned the same thing on a project that was using Java 1 non-generics. Not exactly untyped to typed but an analogous experience. Everyone I asked said that it was too big to do. I started anyway by enabling the warnings for nongeneric use. I turned down the reporting limit to 1000 (I think) so as not to be discouraged. After months and months of incremental work alongside my main work, I got under the 1000 warnings. It got a bit trickier after that. In the end, there was exactly 1 bug, where an object.toString was being added to a dropdown box and we'd see it from time to time as Class@hexhash. What I learned then is that it isn't strictly about the bugs, it's the confident way you can navigate the codebase and understand and add in consistent ways. Now I add types to all my Ruby and it's seems normal again.

BeFlatXIII2• 5 years ago

some clever people decide - hey, if you pass a string I'll look up the person object - so you can pass an object or a string - which makes all sorts of convoluted code paths

Do you have hints on how to avoid being one of those 10x clever programmers while programming a prototype? I find that I am most likely to write functions like that when there's some variables that I don't want to pass 5 layers down the call stack and then, in your example, would accept either a string (in which case those variables use their default values) or the Person object, where the variables are pulled from the Person's attributes.

david4224• 5 years ago

I don't really, but I guess I could say that I have developed in statically typed languages and dynamically typed languages (professionally) for over a decade and I've always found that using the "power" of dynamic languages always ends up causing (me) more frustration in the long run- basically classes of bugs or time wasted that simply doesn't occur with statically typed languages. So for me, I tend to spend a little more time up front to try not to waste (my) time in the future.

I find that I am most likely to write functions like that when there's some variables that I don't want to pass 5 layers down the call stack

I agree for a prototype, there are some tradeoffs to be made. However, very often prototypes can end up becoming production. Temporary decisions often become permanent ones. Just something to keep in mind.

didibus• 5 years ago

I've been working in Clojure for the last few years, and what I learned is that the trick is to reverse the data dependencies, so that instead of your function asking: "What is a "person" and what attributes does it have if an object?". You have your function declaring: "I take a person as a map of keys :name and :age". And it is the caller who needs to ask itself: "What am I supposed to provide to this function?"

This is a very different mindset, but once you adopt this style, the lack of static types isn't as big an issue.

The reason you can do this in a dynamic language is that you can very easily adapt one structure to another, so its okay if not all your functions work directly on the same shared structures.

It also has the advantage that this style really favors making modular independent granular components that can be reused easily, because they aren't coupled to an application's shared domain structures, but to their own set of structures, creating a natural sub-domain.

There are other aspects to make this style work well, like keeping call-stacks shallow, and having a well defined domain model at the edge of your app with good querying capabilities for it.

Concretely it means say you need to add some feature X to the code, you might think, ok this existing function is one place where I could add the behavior, but for my new feature I need to have :age of "person", but I don't know if the "person" argument of this existing function would contain :age or not. Dammit, I wish I had static types to tell me.

Well, in this scenario, instead, what you do is that you don't add the behavior to that function. Instead, in my style you would have:

    A -> B
    A -> C

instead of:

    A -> B -> C

That means if after B is the right place for your logic, you don't do:

    A -> B -> B' -> C

And hope that the "person" passed to B had the :age key which is needed by B'.

Instead you would do:

    A -> B
    A -> B'
    A -> C

And when you implement B', you don't even care about "person", you can just say you need person-age, or that you need a Person object with key :age (which you don't care if it is the Person object shared in other places or not).

Finally, you modify A, where A was the function that creates the Person object in the first place, it has direct access to your actual database/payload and so finding whatever data you need is trivial in it.

blacktriangle2• 5 years ago

I would argue that any function that branches on argument type is straight up doing dynamic typing wrong. Well branching may not be the right word. Something resembling pattern maching is fine, but like you say having a function that takes a string for lookup OR the object is just a disaster, particularly when you start stacking function calls. Dynamic types should closer resemble things that all share an interface, not totally different representations of that data based on the shape of your code.

Javascript is by far the worst offender here with its ignoring extra arguments. Javascript functions that totally change effective type signatures based on number of args are the devil's work.

I'd argue that if the types that a function accepts are not easily defineable than you're doing dynamic typing wrong.

spicyramen• 5 years ago

I wrote Python code for 4 years, then moved to GoLang I really appreciate the typing languages as prevent so many bugs I just was used to handle

exdsq2• 5 years ago

I can’t understand programming without types - it’s just so weird…

kzrdude3• 5 years ago

Python is not without types, there are dynamic types.

exdsq2• 5 years ago

Sorry to be specific I find it so weird to program without defining what I want in and out of a function

chromatin• 5 years ago

Even though I had previously learned some rudimentary C, C++, and Java, I really came-of-age with Python. Now, having written (and maintained) nontrivial code bases in statically typed languages including D and Rust (and dabbling in others with contributions in C, OCaml, etc.), I am never going back — except perhaps in a few cases when a library like PyTorch or Pandas has no good substitute.

(edit: corrected "Linda's" to "Pandas" heh, mobile kbd)

papito• 5 years ago

Lack of type checking was a hot thing for a while. It made you "move faster". It was actually sold as an advantage. Until we realized that after moving faster you grind to a halt because now you have a massive codebase, with hundreds or thousands of files, and everything takes forever, and every change requires multiple rounds of testing.

I believe it really has to do with the size and complexity of modern projects. With a half-decent IDE you could sort of used non-type-checked Python in 2012, but times have changed, and now we are talking about statically checking Python and Ruby. And Javascript, of course, now has it in form of TypeScript.

tabbott2• 5 years ago

I agree with all the benefits of mypy cited in this article. For me, most important thing for the long-term health of a codebase is its readability/maintainability, and mypy static typing makes such a huge difference for that in large Python codebases. I'm really excited to see large libraries doing this migration.

I'll add for folks thinking about this transition that we took a pretty different strategy for converting Zulip to be type-checked: https://blog.zulip.com/2016/10/13/static-types-in-python-oh-...

The post is from 2016 and thus a bit stale in terms of the names of mypy options and the like, but the incremental approach we took involved only using mypy's native exclude tooling, and might be useful for some projects thinking about doing this transition.

One particular convention that I think many other projects may find useful is how we do `type: ignore` in comments in the Zulip codebase, which is to have a second comment on the line explaining why we needed a `type: ignore`, like so:

* # type: ignore[type-var] # https://github.com/python/typeshed/issues/4234

* # type: ignore[attr-defined] # private member missing from stubs

* # type: ignore[assignment] # Apparent mypy bug with Optional[int] setter.

* # type: ignore[misc] # This is an undocumented internal API

We've find this to be a lot more readable than using the commit message to record why we needed a `type: ignore`, and in particular it makes the work of removing these with time feel a lot more manageable to have the information organized this way.

(And we can have a linter enforce that `type: ignore` always comes with such a comment).

lrobinovitch• 5 years ago

I really like this documented type ignore strategy and will start incorporating it in our codebase. Thanks for sharing.

andybak2• 5 years ago

Having spent a decade with Python and more recently a few years with C# I still can't quite put my feelings into words but here's an attempt:

"The benefits of explicit typing are obvious and clear but they downsides are subtle and hard to communicate"

I still think typing in general is a net win but I'm not sure whether static typing is. You find yourself writing code that just wouldn't be neccesary in a dynamic language - and I don't just mean the direct code you write to declare and cast types. There are more subtle costs.

I need to spend time with a good type inference in a language with modern typing and dynamic features to sort out how I feel about this.

layer84• 5 years ago

Types are effectively assertions about the values they represent, and statically-typed code constitutes proofs that the assertions actually hold at runtime. The static typing forces you to be sufficiently rigorous in those proofs, which may require additional code as you mention. Without static typing, one has to rely on the "proofs" in one’s head to be correct (which humans aren’t really good at), instead of having the compiler double-check one’s reasoning.

andybak2• 5 years ago

I think this falls into the category of "The benefits of explicit typing are obvious and clear". It's the other side of the equation that I'm intrigued by and struggling most to formulate.

LadyCailin• 5 years ago

Oh look, they're finally discovering that strong typing is actually a benefit, and using a language without it is a huge step in the wrong direction.

KingMachiavelli• 5 years ago

I think mandatory type hints in method signatures and optional type hints at assignment are a good compromise.

But if I had to pick either a language without any type hint/inference or a verbosely strictly typed language - I would must rather use the strictly typed language.

richard_todd• 5 years ago

I think it's interesting that PEP 484 says ([1]): "the authors have no desire to ever make type hints mandatory, even by convention," while the opening of this article says "type hints have grown from a nice-to-have to an expectation for popular packages." Things don't always work out the way the PEP authors expect.

[1] https://www.python.org/dev/peps/pep-0484/

kkirsche• 5 years ago

It’s the most demoralizing aspect as even just typing the standard library online documentation and examples (such as Emil autoname example) would be extremely valuable.

kraf• 5 years ago

It would be really interesting to see some examples of the logic errors that were found that couldn't be found by tests. This seems to have been a very robust library. What kind of problems did you find? From what is mentioned in the article it really doesn't sound like the investment of hundreds of hours from multiple people has been actually worth it.

aitchnyu• 5 years ago

Tangential: did anybody find success with typed Model.objects methods with Django?

Tests aren’t enough: Case study after adding type hints to urllib3

Related Stories