A Failed Experiment with Python Type Annotations

https://mortoray.com/2019/06/11/a-failed-experiment-with-python-type-annotations/

by ingve • 7 years ago

127 80 7 years ago

Deimorz3• 7 years ago

The first issue is fixed by just using a string. Yes, you can't do Sequence[Node], but you can do Sequence["Node"].

The docs talk about this here ("class name forward references"): https://mypy.readthedocs.io/en/stable/kinds_of_types.html#cl...

mortoray• 7 years ago

Great!

Deimorz3• 7 years ago

The second issue is a bit of a judgment call too, I think. If the types were inferred you could end up with errors that you're not able to see because inferred types end up compatible. Personally, I think I'd rather write them manually so that I'm validating whether the method actually matches what it was intended to return.

I can't find a more authoritative source right now, but I also remember seeing the same thing that this SO answer says - that it was a deliberate choice not to do this so that people could incrementally add type annotations: https://stackoverflow.com/a/38775381

There are also some separate tools that can infer and generate the annotations for you. I haven't tried them personally, but the mypy docs recommends MonkeyType: https://mypy.readthedocs.io/en/stable/existing_code.html#aut...

rcfox6• 7 years ago

For Python 3.7, you can do:

    from __future__ import annotations

This will defer the parsing of annotations until after the file has been parsed, allowing you to use the actual symbol instead of a string.

anentropic• 7 years ago

TIL!

BurningFrog2• 7 years ago

OK, that works, but it does seem dumb, or even deliberately obtuse.

The language could just as well recognize Node as it recognizes 'Node', but instead it given as a burden for the human programmer to handle.

Is it maybe because mypy is not really part of the Python language, and can't really make changes to it?

shakna• 7 years ago

The language will recognise it in a future version by default, or a from future import can be used now.

mypy isn't behind (here), it's ahead.

_ZeD_• 7 years ago

This is not mypy, it's "python-the-language": the annotations are values, and, in this case, the method annotations are evaluated when encountered - which is before the end of the class. So you are trying to use a value still not defined. It's like doing something like

    def foo() -> Bar:
       ...

    class Bar:
       ...

obviously the python interpreter cannot find a "Bar" definition during the parsing of the foo function

Deimorz3• 7 years ago

PEP 563 is intended to resolve it: https://www.python.org/dev/peps/pep-0563/

As rcfox said, you can already enable this behavior, and it will be included in Python 4.0.

j-e-k2• 7 years ago

I would have to disagree with this post (first point is actually inaccurate and the second I could go either way on).

Type annotations/mypy, especially when coupled with dataclasses/pydantic has been very helpful in maintaining a rather substantial 3.7 codebase.

trimbo• 7 years ago

rather substantial 3.7 codebase

What size codebase is substantial?

andrelaszlo• 7 years ago

A rather sizable one.

j-e-k2• 7 years ago

Excluding tests/patches to third party libs, around 65K LOC

michaelanckaert• 7 years ago

I agree. I'm currently working on a project containing 45k lines of Python code and we have really benefited from having type annotations.

ggregoire3• 7 years ago

Why would you keep Python if you want static typing? I mean, there are several modern, performant (more than Python actually) statically typed languages. Rust, Go, Kotlin, Scala, Swift… All of them are mature and have a good library ecosystem. So, except if you need some math/AI/ML packages available only for Python, why bother? Choose the right tool for the job, no?

dragonwriter4• 7 years ago

Seems like all of this (adding static typing on top of Python, PHP, etc) started with the success of TypeScript

Mypy is a few years older than TypeScript, so, no.

ggregoire3• 7 years ago

Yep, already removed that part since I was not sure about the historic and it actually doesn't matter who started the trend.

rat9988• 7 years ago

Choose the right tool for the job, no?

The thing is, static typing is never the job by itself.

ggregoire3• 7 years ago

"job" doesn't refer to static typing but the project you are working on.

- script of 2,000 LOC to clean data => dynamic typing is fine => Python

- web service of 1,000,000 LOC => static typing seems a better choice => Rust, Go, Kotlin, Scala, Swift, C#, Java…

aflag• 7 years ago

Sometimes the right tool for the job is python + mypy.

darkr• 7 years ago

In general, for a new project of any size, or significance, or where performance/concurrency was a concern; hands down, yes, I would absolutely reach for a strong, statically typed, compiled language.

The fact remains though that Python is an excellent general purpose language, easy to learn, almost universally known, has libraries for everything (and that’s just the stdlib). It is well suited to rapid prototyping, scripting, and data wrangling. Adding incremental typing makes it even better.

pacala4• 7 years ago

Scala: Python + dataclasses [+ mypy] offers a programming experience close to pragmatic Scala. The tradeoff becomes:

* worse performance

* lack of immutable vectors, alleviated via style conventions

* rarely, awkward lambdas

vs.

* wide pool of people familiar with the language

* lack of JVM lockin: Scala native is not there yet, the library ecosystem is all JVM.

* [good] batteries included

* much better reflection

* access to modern ML

* gradual typing

* no actors

Rust: Manual memory management is verbose and distracting.

Go: Manual exception handling is verbose and distracting.

Kotlin: JVM lock-in.

Swift: Apple lock-in.

hermitdev• 7 years ago

No immutable vectors in Python? Are you not aware of tuples?

Lambdas in Python are quite limited, but offset by natural nested local functions.

Regarding performance, that's not always an issue, and depending on the use case can be addressed with async/await or using a multiprocessing pool. I have had issues with multiprocessing and some DB libs ib the past, but recently have had pretty good success.

Multithreading is still an issue because of the GIL (global interpreter lock) and really only useful if you're calling into a C/C++ lib that releases the lock while it does it's native things.

I've written a lot of Python in the last 15 years, and Ive written a ton of scripts that are faster than a well written identical C/C++/C# app, and was written in 1/10th of the time. It just depends on the use case.

joshuamorton5• 7 years ago

* lack of immutable vectors, alleviated via style conventions

Note that mypy/typed python fixes this: if you annotate a function as returning (or accepting) a Sequence/Collection/Iterable instead of a List, and attempt to mutate it, you'll get type errors. This is good practice anyway.

zapzupnz3• 7 years ago

Swift: Apple lock-in.

Swift runs on Linux and has done since it was made open source in 2015, a year after its initial announcement. Windows support is being worked on but, for the time being, it runs fine in WSL. No lock in.

swebs• 7 years ago

Scala: JVM lock-in
Kotlin: JVM lock-in

You know OpenJDK has been available for over a decade, right?

cwyers2• 7 years ago

Sometimes the job is "maintaining existing code," and nobody is going to pay for a ground up rewrite.

Sean1708• 7 years ago

Getting a whole team to try out static typing on a language that they already know is far easier than getting a whole team to learn an entirely new language.

dragonwriter4• 7 years ago

Or do you have any idea what type get_closure returns?

It doesn't.

Because of the syntax error.

joshuamorton5• 7 years ago

mypy has no trouble understanding this, but it’s unfortunately not valid Python code. You can’t refer to Node within the Node class.

No, the workaround is to stringify "Node".

    class Node(Object):
        def add_sub(self, sub: 'Node'):
            ...

        def get_subs(self) -> Sequence['Node']:  #Or maybe 'Sequence[Node]'

works just fine. As of python3.7, this stringification is done by automatically under a from future import, and will eventually become the default, so the original code will be valid.

This complexity helped drive the introduction of the auto keyword to C++. There are many situations where writing the type information isn’t workable. This is especially true when dealing with parametric container classes,

This is absolutely wrong! You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function.

This is an intentional choice: function declarations are your apis, and explicit and clear APIs are useful for human readers of your code. That's why you should annotate the public apis[0] even when you can have them be type inferred. Speaking of which, if you want type inference, check out pytype[1], it's like MyPy, but does do type inference on unannotated code. But you should still annotate your public apis. It serves as a sanity check that you aren't accidentally returning something you don't expect.

And of course, being familiar with the differences between iterables, iterators, sequences, containers, etc. is not a bad idea.

[0] https://google.github.io/styleguide/pyguide.html#3191-genera...

[1] https://github.com/google/pytype

dragonwriter4• 7 years ago

But you should still annotate your public apis.

The good thing about inference (especially with a REPL) is you can write it without the annotation, and then use the inferred type (in Haskell, I usually find that when I resist the temptation to explicitly annotate types, the actual types are more general than I would have specified.)

klodolph• 7 years ago

In Haskell, I usually find the quality of error messages to be much worse without top-level annotations.

joshuamorton5• 7 years ago

This is generally true with pytype as well, but questionable whether it's a good thing.

Inferred, loose parameter types are great, because while I think it's good to know the differences between iterator, generator, iterable, sequence, container, and so on, it's difficult, and inference will give you the loosest one. But with return values, the inferred type will be the tighest, in other words your function will return a Dict (or worse, a defaultdict) instead of a Mapping or MutableMapping. You almost always want a Mapping.

lozenge2• 7 years ago

As well as pytype, you can use a separate tool to discover the correct annotations, then verify them and add them to your source code. There are a couple - MonkeyType, pyannotate and pytypes.

redblacktree15• 7 years ago

This is absolutely wrong! You cannot annotate a function as returning `auto` in C++.

You can do that since C++14.

blub• 7 years ago

"[...]You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function.[...]"

Maybe I'm misunderstanding something, but one can write a function auto foo() in C++ and the return type is inferred. Lambda functions infer the return type by default.

joshuamorton5• 7 years ago

Yuck, apparently I missed this happening. Its still (I think) true in java, where you can't have a function return `var`.

ehsankia• 7 years ago

There's a great 5m lightning talk from Pycon 2019 explaining the difference between pytype and mypy[0]. It's great that there's different libraries approaching the problem from different angles. Speaking of which, I'm also curious to see how Facebook's Pyre[1] differs from the former two. I'm surprise there's isn't a comprehensive comparison of the 3 libraries yet.

[0] https://www.youtube.com/watch?v=yFcCuinRVnU&t=38m25s

[1] https://github.com/facebook/pyre-check

reissbaker• 7 years ago

Pyre is similar to mypy, in that it:

1. Is gradually typed,

2. Doesn't infer types, and

3. Is strict, in that it doesn't allow operations that change types.

It was originally developed as a replacement for mypy that was faster and scaled better to very large codebases.

neuland2• 7 years ago

Other people have mentioned that the self-referential case from this post is doable. But what is not doable (to my knowledge) is this kind of self-reference:

    class Node(typing.NamedTuple):  # or dataclass
        child: 'Node'

The error is example.py:4: error: Recursive types not fully supported yet, nested types replaced with "Any".

This error has been in there for several years at least. And it always bites me when I forget. It's particularly cryptic when the self-reference is many levels deep. I've even had the error message break in these cases where it prints out a line number from the wrong file. So, then you have to hunt to see where the self-reference is hiding.

rcfox6• 7 years ago

Just tried this myself, and it seems to only be an issue for typing.NamedTuple. Though, it still manages to do type checking correctly...

    from __future__ import annotations
    from typing import Optional, NamedTuple
    from dataclasses import dataclass

    class Foo:
        def __init__(self, foo: Foo = None) -> None:
            self.child: Optional[Foo] = foo

    Foo(Foo(Foo(None))) # Works
    Foo(Foo(Foo(1))) # error: Argument 1 to "Foo" has incompatible type "int"; expected "Optional[Foo]"

    @dataclass
    class Bar:
        child: Optional[Bar]

    Bar(Bar(Bar(None))) # Works
    Bar(Bar(Bar('bar'))) # error: Argument 1 to "Bar" has incompatible type "str"; expected "Optional[Bar]"

    class Hep(NamedTuple): # error: Recursive types not fully supported yet, nested types replaced with "Any"
        child: Optional[Hep]

    Hep(Hep(Hep(None))) # Works
    Hep(Hep(Hep({'a': 'b'}))) # error: Argument 1 to "Hep" has incompatible type "Dict[str, str]"; expected "Optional[Hep]"

lozenge2• 7 years ago

It looks like the constructor is being typechecked, but probably not attribute access. `val.child.junk` will pass because `val.child` is `Any` and you can do any-thing to an Any.

rcfox6• 7 years ago

I recently added type annotations to my Python 3.7 project and found it very helpful. Just the process of adding the annotations caught a handful of bugs!

Some things I found useful:

1. from __future__ import annotations

As I mentioned in another comment, this lets you write annotations naturally without worrying about when something gets defined.

2. typing.TYPE_CHECKING is only True while type checking.

This allows you to conditionally import files that would otherwise cause a circular dependency so that you can use the symbols for annotations.

3. def foo(bar: str = None) -> str

If an argument defaults to None, then mypy will recognize that its type is actually Optional[whatever]. So in the example above, bar is an Optional[str]. (I'm not sure if this is mypy-specific.) Optional[T] is equivalent to Union[T, None].

4. You can make typedefs easily.

MyType = Union[Sequence[str], Dict[str, int]]

Makes writing complicated annotations easier.

yegle• 7 years ago

If your code doesn't need to be py 2/3 compatible it's fine to annotate using str. But if you do, typing.Text and bytes is better.

rcfox6• 7 years ago

Good to know, but I hope to never need this.

quietbritishjim• 7 years ago

Am I missing something? Annotations aren't supported by Python 2 so if you're using them then you never need to worry about Python 2/3 compatibility.

davesque• 7 years ago

The first point can be rather easily circumvented by using strings as forward references and it only takes a bit of googling to figure that out. The second doesn't seem remotely like a deal breaker to me. Half the reason for having type annotations is to provide a description of what a function or method does at a glance. This is idiomatic even in hardcore functional languages like Haskell. With return type inference, you don't get that benefit.

cjwoodall• 7 years ago

It seems to me that type inference would be silly in this case. Type suggestion during mypy checks might be useful though.

Function x does not specify a type, but returns SomeHelpfulTypeString. We recommend you use that.

Put it behind a flag and call it a day. Maybe even have a way of requesting type suggestions for functions during development

totalperspectiv• 7 years ago

I really like using and writing type annotated Python, but every time I do, I wonder if I should just use a language where all the annotation time I put in gets me actual runtime improvements.

carlmr• 7 years ago

And accurate compile time checks.

I recommend checking F# if you want mostly inferred, strong static types and terse pythonesque Syntax (few braces, significant whitespace). Also much faster.

Considering you can use all of .NET from F# you can benefit from the work on C# as well.

Multicomp• 7 years ago

I'll also wave the F# flag here, but there's also Scala if moving to Javaland is better for you than Dotnetland.

whalesalad• 7 years ago

I cannot stand typing in Python. It's never given me any saving grace in a project, only headaches.

jimmaswell• 7 years ago

What's possibly so cumbersome about defining return types?

It does seem a little embarrassing that the annotations don't even support circular references.

adrianratnapala• 7 years ago

Circular references like that are called "recursive types" in formal type theory. They are one of those big issues that academics like to write papers about.

Even if you just want to write a practical interpreter, and choose to gloss over the issues, they will still come back in some disguised form and either by requiring some sort of implementation kludge, or just by creating weird edge cases.

heavenlyblue3• 7 years ago

And how is that an issue for type checking, would you elaborate?

heavenlyblue3• 7 years ago

Author of the subj didn’t even bother to Google that issue, which is solved by the way.

deepsun• 7 years ago

Try Kotlin. Syntax is similar to Python (but better), but it's actually just Java, with all the supporting ecosystem. Kotlin is just syntactic sugar.

amelius• 7 years ago

Try Kotlin. Syntax is similar to Python (but better)

Well, from [1], I'd say syntax is closer to C.

[1] https://kotlinlang.org/docs/reference/control-flow.html

sametmax• 7 years ago

While the first complaint is just a reflect of the lack of good tutorials on mypy, the second one is very valid: I too wish for automatic type detection for obvious cases.

What's more, while I found mypy useful myself in several instances, it's still cumbersome to use:

- you must know to use the magic command line arguments to avoid mypy complaining about other libs and imports all the time

- you need to use List[], Dict[], Set[], Iteratble[], Tuple[], etc. instead of list[], dict[], set[], iter[], tuple[], etc., which means an import in almost every file and a very unnatural workflow.

- you need to use Union and instead of |. This is ridiculous. I have to do:

    from typing import Union, List

    def mask(...) -> List[Union[bool, int]]

Instead of:

    def mask(...) -> list[bool|int]

And Guido explicitly rejected the proposal for those on github despite the fact most imports are due to those.

Now, mypy has improved a lot. It's way faster, shows many more things that before, way less false positive, and has support for duck typing and dunder methods (named "protocol").

But it's sad to thing TypeScript is actually easier to use. Having a JS tech easier to use than a Python tech is a good sign we can improve things.

nickm12• 7 years ago

This is a really poor article with a click-bait headline. I was expecting this to be a case study where Python type annotations just couldn't work due to some interesting specifics of the projects. Instead, it's some petty gripes about the syntax of annotations not being just to the author's liking.

Barrin92• 7 years ago

I feel baking type annotations on top of dynamic, late binding languages is fundamentally misunderstanding what they are about. Python might not go as far as say, smalltalk, but if you're picking a dynamic language just accept the runtime dynamism and don't program against it.

heavenlyblue3• 7 years ago

Or well, accept the dynamism and use type declarations as automated declarative teat coverage, at the very least.

perlgeek• 7 years ago

I think the premise of this blog post is misguided.

Yes, there are two things about introducing types that the author finds annoying. So don't do those two things, and use type annotations for all the rest.

The type annotation system and mypy are designed to give you benefits even if you don't fully annotate everything -- that's the starting point of nearly all existing python code bases.

jks• 7 years ago

What's the status with numpy types? I find two projects, https://github.com/machinalis/mypy-data/tree/master/numpy-my... and https://github.com/numpy/numpy-stubs but neither looks very complete and neither has been updated recently.

A Failed Experiment with Python Type Annotations

Related Stories