Statically-typed error handling in Python using Mypy

https://beepb00p.xyz/mypy-error-handling.html

by Agathos • 7 years ago

198 126 7 years ago

roseway42• 7 years ago

Having recently found myself switching between Python and Scala, I've come to value IntelliJ's type hinting and red squiggly prompts, not to mention compile-time errors.

While mypy and type hinting support in PyCharm and VSCode is great, it's not the seamless experience as with typed languages and the lack of (optional) runtime typing still allows all sorts of bad things to happen. A "safe" mode for Python, where typing and perhaps some other stuff is checked at run-time, might be an interesting advancement for those in the community building larger or mission-critical systems.

astrea• 7 years ago

My opinion is that python shouldn't be used for mission- critical systems anyway.

traverseda• 7 years ago

Aside from static types, why?

dfee2• 7 years ago

I’m on the other side; having gotten deeper into languages like c# and java; I think those technologies are vastly over used, and great languages like Python and Typescript are under used.

1e10• 7 years ago

Why? Seems that it is a well proven language. Just like php, asp.net, and other languages people like to hate on.

Or rather, how would you define mission critical?

jtdev3• 7 years ago

Care to elaborate? Which languages would you use for mission-critical systems? What specifically about Python is not conducive to such systems?

You realize that many mission-critical workloads run on Python with great success, right?

senko• 7 years ago

That depends on the mission, wouldn't you say?

staticassertion14• 7 years ago

People are gonna dogpile you, but if you'd said this about JS they wouldn't have blinked.

HALtheWise• 7 years ago

Would you rather build safety-critical systems in Python (type-unsafe but memory-safe) or C++(type-safe but memory-unsafe)?

While static typing does catch code errors, in my experience they are mostly errors that would be caught somewhere else in the testing process. Memory safety issues are rarer, but also have a penchant for showing up unexpectedly and for crashing the entire system in unrecoverable ways.

Obviously, the best would be to have both kinds of safety, but people program safety critical systems in C++ all the time, and deal with the consequences. At least you can catch a TypeError in Python and handle it.

Riverheart• 7 years ago

Appreciate how in depth this is, looking at all the possible ways of handling the issue. Makes the final result more interesting knowing the journey it took to get there.

karlicoss6• 7 years ago

Hi, I'm the author! Happy to answer your questions here!

dbrgn2• 7 years ago

Hi, thanks for featuring my result library!

Regarding your criticism that `result.ok()` returns `None` if the value is not an `Ok` type: That's what `result.unwrap()` and `result.expect(msg)` are for :) (There's no unwrap_err and expect_err so far, but PRs are welcome.) The lib is strongly inspired by Rust (see https://doc.rust-lang.org/std/result/enum.Result.html), that's why `result.ok()` returns something option-ish.

I like your `isinstance` approach btw! Will have to think about it a bit more. Reminds me a bit of Typescript as well. Does mypy have type guards (https://www.typescriptlang.org/docs/handbook/advanced-types....)? Maybe that could be used to create helper functions that check for success/failure and which help mypy to derive the correct type.

uryga• 7 years ago

Does mypy have type guards?

not yet, maybe in the future. might be possible to emulate some use cases with Literal Types. tracking issue:

https://github.com/python/mypy/issues/5206

karlicoss6• 7 years ago

Oh, not sure how I missed these methods, thanks! I'll update the post to reflect it.

Agathos• 7 years ago

Thanks for writing it! I'm a Python programmer in my day job. I have sometimes remarked that the reason I play around with Haskell in my free time is because I don't want to see another run time type error unless I'm being paid to do so. But I needed to do some spreadsheet-scraping and data-munging for a personal project and I figured it would be easier in Python. So I searched and found your article and it was just what I needed.

Now to sneak it in at work...

karlicoss6• 7 years ago

I'm a fan of cool type systems and Haskell in particular too.

Also doing lots of messing with data for personal stuff (e.g. https://beepb00p.xyz/mypkg.html#examples). But yep, at least for me, the reality is if I need some quick yet useful code, Python happens to get me there very fast, just because of the sheer amount of code people already have written in Python.

mplanchard• 7 years ago

I’ve written a similar python library called `safetywrap` that implements both Result and Option types from rust, with almost all of the interfaces implemented, including unwrap, map, flat map, err_map, etc. It fully supports types and works well with mypy.

Feel free to give it a shot! https://pypi.org/project/safetywrap/

dbrgn2• 7 years ago

At first glance, this looks really solid! Maybe I can deprecate my result library now :D

staticassertion14• 7 years ago

Here's my opinion, having written many thousands of lines of mypy code.

* Third party libraries are still not typed, which sucks

* Inference is weak. Sometimes an 'if' statement narrows accurately, sometimes it doesn't.

* Generics are extremely confusing, moreso than any other typed language I have used. Any Generics are even more confusing.

* Fundamentally, Python is structured such that the concept of interfaces are coupled tightly to implementation of classes, which is severely limiting. As an example, a module A can declare a class, a module B can declare an interface (Protocol), but module B can not implement that interface for A (without modifying A's code).

* Errors from mypy are still very very light. There are very few "Try XYZ to fix it", and it's just a single line of read text that amounts to an assertion.

* Still weak support for JSON (use Dict[str, Any])

* Slowly improving but still not amazing support for Self types

I could go on.

Mypy is awesome but it still feels very immature, and as it should since it's pre 1.0. It does not come close to matching the experience of other languages with static types.

karlicoss6• 7 years ago

* Third party libraries: I sometimes see them annotated, but what people don't suspect, is that you need to include 'py.typed' file with your package in order for it to be discoverable. Perhaps that's something setuptools could warn the developer about..

* Regarding errors: there are few quite recent flags: ' --pretty', '-show-error-context' and '--show-error-codes', they make it a bit more pleasant. You can see me using them in this section: https://beepb00p.xyz/mypy-error-handling.html#container

staticassertion14• 7 years ago

The 'py.typed' thing is extremely unintuitive, I package up python libs and I'm still not sure I've done it right.

Thank you for letting me know about those flags, I had not heard of them before!

Caligatio• 7 years ago

* Still weak support for JSON (use Dict[str, Any])

Python 3.8 added TypedDict which finally helps this particular shortcoming.

staticassertion14• 7 years ago

Wonderful, that's great to hear, and I can move to 3.8 quite easily myself.

It's nice to see strong progress in such important areas. My post is probably overly critical, but I write Python almost every day, and maintaining a highly-typed library can be demoralizing at times.

avremel2• 7 years ago

I was surprised to find out that MyPy will throw an error if you use a variable to access a key (https://github.com/python/mypy/issues/7178).

Optional keys are also a nuisance.

ledauphin• 7 years ago

in my experience, TypedDict has its own shortcomings, most of which boil down to the fact that it's not truly structural subtyping, because 'extra' keys are treated as errors.

foxes• 7 years ago

I think in the long run, python, ruby, js, etc are all bad ideas. You are solving high level problems with these languages, correctness should be the main priority.

Realising that type checking is a good thing, then bolting it on, I feel is a crappy solution, when you should just use something designed better from the start.

I completely disagree that there is a "productivity toll". If you dont know how to speak a language then obviously it seems difficult. I think you save time in the long run not dealing with all the type errors.

jtdev3• 7 years ago

> “ I think you save time in the long run not dealing with all the type errors.”

Have you written and deployed production code in the languages you mention above? Did you encounter a substantial number of type errors?

staticassertion14• 7 years ago

There is no good definition of a type error. It certainly goes beyond TypeError(Exception).

Is a '.close()' method being called twice a type error? It is in a language that can express states in the type system.

Is SQL injection a type error? It is when you use refined types in your sql library interface.

The vast, vast majority of errors I run into are errors that, with effort and the right type system, I could turn into type errors.

The point being that asking "would those be type errors?" is a really big question that is, when answered simply, "yes".

patrec• 7 years ago

In my case: Yes and yes. Wildly guessing, about 90% of the errors I find at runtime with dynamic languages would would be caught by a sane type system at compile time. If I'm comparing a completely mis-engineered statically typed language such as C++, the compilation overhead may be so bad that for code I'm writing right now feedback can still be faster with the dynamically typed language. But if we're talking interfacing a few bits of unfamiliar and not great dynamically typed code: well I've literally spent several weeks on a project that would have taken me half a day with even bad static typing.

h8hawk• 7 years ago

Type system's of typescript , mypy and julia are far better than type syatem's of java/c#/c++/go . Null safety, type inference, sanner generics, Union types, structural typing are come to mind. I think the reason is that these type systems should catch up expressiveness of their host language.

sa46• 7 years ago

Type system's of typescript [...] are far better than type systems [than java]

TypeScript's type system is far more powerful than Java's but I'm not sure that's a good thing. A lot of the concepts in TypeScript are necessary to paper over gratuitously dynamic APIs in JavaScript.

While Java certainly has warts in its type system (no generics over primitives, arrays), Java's type system straight-forward compared to TypeScript. TypeScript stuffs a staggering amount of functionality into the type system. To name a few of the more esoteric features: literal types, const assertions, asserts modifier for type predicates.

macca3212• 7 years ago

I maintain a library for doing exactly this in C# (https://github.com/mcintyre321/OneOf). I tried to replicate it in Python as I'm writing it at the moment, but couldn't quite manage it. Can you have a multi generic typed Union in python?

Recursing• 7 years ago

Do you mean something like this?

  T1 = TypeVar("T1")
  T2 = TypeVar("T2")
  GenericUnion = Union[T1, T2]

macca3212• 7 years ago

Yes I can't quite remember why I couldn't get it to work. I want to make something where this is valid, but couldn't manage it:

    def foo() -> OneOf[Bar, Baz]:
      if blah():
        return Bar()
      return Baz()

   

   def bar_to_blah(bar:Bar) -> Blah :
     ...

   def baz_to_blah(baz:Baz) -> Blah :
     ...


   one_of_bar_or_baz = foo()
   blah = one_of_bar_or_baz.match(bar_to_blah, baz_to_blah)

but the critical thing is to get a type error if you add another case to foo's return type, or change the generics to different types.

sandGorgon2• 7 years ago

Has anyone evaluated libraries like Pydantic ( https://pydantic-docs.helpmanual.io/) and Encode Typesystem (https://www.encode.io/typesystem/) versus MyPy as a real life thing ?

fermigier• 7 years ago

These are completely different objects: data validation libraries.

I think naming the latter "Encode Typesystem" is misleading (and your question is an indication that I am right).

sandGorgon2• 7 years ago

why is it misleading ? didnt understand. i just didnt want people mistaking "typesystem" as a word rather than a project name by Encode.

its a very common name.

ghostwriter• 7 years ago

There's also Typeit (https://typeit.readthedocs.io/en/latest/)

Statically-typed error handling in Python using Mypy

Related Stories