TypedDicts are better than you think

https://blog.changs.co.uk/typeddicts-are-better-than-you-think.html

by rbanffy • 2 years ago

134 147 2 years ago

blog.changs.co.uk

jjtheblunt3• 2 years ago

"thank you think" in a title is overtly condescending...you know your audience and their educational deficiencies?

"than i thought" would be compelling to read.

crazygringo• 2 years ago

Yup, using "you" in headlines is a pattern that needs to die.

I get that it's attention-grabbing, but it's because it's rude.

You don't know anything about me. You don't know what I think or what I already know or what I won't believe.

I know it's not a big deal in the grand scheme of things, but it's just one of those little aggravating things that makes life just a little bit worse each time you come across them.

robswc• 2 years ago

The one I hate the most is:

"X thing did Y, here's why Z"

Where X and Y can range from meaningless to national security but Z always remains an opinion (usually uninformed too).

Just search "here's why that's bad/good" on google news and there's so much "slop."

jjtheblunt3• 2 years ago

"you" in headlines is better said, a better criterion for skippable articles, agreed! good point

lopatin• 2 years ago

It's a qualifier. In a sense, they do know their audience fairly well because someone who clicks on that link is intrigued and feel they have more to learn about it. Anyone who is pretty knowledgeable about the subject will go "pff .. yeah right" and not even click on it.

That said, it does annoy me too.

Also what annoys me is that I constantly try to play devils advocate for things like this even if I don't always agree with the conclusion of the advocate.

tptacek2• 2 years ago

This is the kind of copyediting advice ChatGPT gives. I think everyone gets that the author doesn't know what you, in particular, think about TypedDicts. Read things more charitably; this is not a good use of time to discuss.

CyberDildonics2• 2 years ago

Or people could post what they think, since that's the point of an internet forum.

AmericanChopper• 2 years ago

I’m not a big fan of the “my opinion is fact” or “your opinion is wrong” headlines. They can be mildly funny in the right context, but it’s been done so much that they’re just a bit boring now. I’m especially bored of seeing this convention in conference presentation titles.

sqeaky• 2 years ago

100%! If people are dumb enough to think I am dumb, then they generally aren't saying anything worth listening to.

code_runner• 2 years ago

The irony of a typo in this meaningless and overly sensitive reply is hilarious

jjtheblunt3• 2 years ago

i agree (!), and it was my typo (amid Figaro the rescue kitten helping with the keyboard) and attempt to remark that such titles probably don't even get clicked.

benmmurphy• 2 years ago

  <Option<Option<subscription>>

miohtama• 2 years ago

Reasons to use dataclass(slots=True) instead of TypedDict

- Faster attribute access: your code is faster

- Slotted classes take less RAM, less L1 cache pressure, your code is faster

- Wrist friendly to with .foobar instead of ["foobar"]

- Runtime error if you misspell an attribute name

fovc2• 2 years ago

It’s Python. Does L1 matter at all? I assume anything you’re accessing is behind a few pointers and __dict__ accesses anyway.

For me it’s mostly about .attribute being more in line with the rest of the language. Kwargs aside, I find overuse of dicts to clunky in Python

gshulegaard2• 2 years ago

Nope, __slots__ exist explicitly as an alternative to __dict__:

https://wiki.python.org/moin/UsingSlots

Whether or not the performance matters...well that's somewhat subjective since Python has a fairly high performance floor which makes performance concerns a bit of a, "Why are you doing it in Python?" question rather than a, "How do I do this faster in Python?" most of the time. That said it _is_ more memory efficient and faster on attribute lookup.

https://medium.com/@stephenjayakar/a-quick-dive-into-pythons...

Anecdotally, I have used Slotted Objects to buy performance headroom before to delay/postpone a component rewrite.

zemo• 2 years ago

It’s Python. Does L1 matter at all?

depends on the type in question. If you are fetching and operating on a large number of records then it can matter. But otherwise the answer is more often that it does not really matter.

josephg6• 2 years ago

It’s Python. Does L1 matter at all?

In many ways it matters more because it’s Python.

I’ve met a lot of teams throughout my career who struggle daily with a badly performing Python codebase. You can write a no-frills web service in c#, go, rust or JavaScript. And, so long as you don’t do anything stupid, it’s usually plenty fast enough from day 1 to handle your users. But in my experience, the same isn’t true of Python. I’m sure Python web services can be made to run ok, but because it’s slow by default, I bet a lot more time is spent optimising Python programs around the world than optimising JavaScript.

Spivak3• 2 years ago

Better than both use Pydantic. You'll never want to use anything else ever again. It's truly transformative in how you write code. Full type hinting support as well as strong verification that your data actually conforms to the types you set. Full recursive parsing of types arbitrarily nested and can parse tagged and untagged unions.

serjester• 2 years ago

TypedDicts are enormously helpful in defining args a function takes. You can’t do that with either dataclasses / pydantic without passing instantiated objects as args - which is really cumbersome.

darkteflon• 2 years ago

Really like Pydantic - assuming you’re happy to step outside the standard library. It does have a few sharp edges, but out of the box it includes all the stuff you eventually end up needing anyway. attrs is the other big Python parsing / validation library. I know nothing about it other than it’s also popular and that it distinguishes itself philosophically from Pydantic in a number of ways.

Don’t forget to use ‘ConfigDict(frozen=True)’ absolutely everywhere!

neeleshs• 2 years ago

The author touches upon why typeddicts with total=False are more ergonomic than Pydantic for the examples they have.

flysand7• 2 years ago

Here we have a problem, for subscription does None mean don't change or remove subscription

I read this like 10 times and I still dont understand this mess of grammar. Am I having a fucking stroke rn?

roywiggins• 2 years ago

for "subscription", does `None` mean "don't change", or "remove subscription"?

Ericson2314• 2 years ago

Quoting, important when (meta-)programming and in natural language!

dan-robertson• 2 years ago

I’m going to send you some json to update your object – an HTTP PATCH request. There are three possible updates I might want to make to the subscription field:

- change it to a new value

- set it to None

- leave it as-is

In the json the first two options are specified by sending an object with a "subscription" field, either set to a string or to null. The third case is expressed by omitting the field.

The OP asks how all three cases could be represented in python, and points out that one could not use subscription=None to represent both case 2 and 3 above.

RHSeeger3• 2 years ago

If you need to be able to represent what looks like a tree of data

    set       - no value
              - new value
    do_no_set

(sorry about the horrible tree) then it seems like you should represent the data like that. Pass in action (set/ignore) and value (value for set, no value for ignore). Or even (set/unset/ignore), which allows you to have some sanity checking that, if the action is set, an actual value is provided (and vice versa for unset).

syockit• 2 years ago

Better punctuated:

Here we have a problem: for subscription, does None mean don't change or remove subscription?

gorgoiler• 2 years ago

Here we have a problem:

If the “subscription” part of the “patch” data-structure is set to “None” it is impossible to tell if this means:

1/ leave the “subscription” unchanged; or

2/ set the “subscription” to None.

—

You could solve this by requiring that a subscription is either a value or some other sentinel value meaning “not subscribed”:

  from typing import Union
  class T: pass
  NotSubscribed = T()
  Subscription = Union[int, T]
  def f(s: Subscription): pass
  f(44)
  f(NotSubscribed)
  f(None)  # wrong

Is there a better way of doing this? (One that isn’t just “use Haskell” :)

calibas2• 2 years ago

I feel like this needs to be mentioned more: If you don't have some kind of system to enforce types, then TypedDict does nothing at all.

You can store a float in an attribute annotated as a string, and default Python will not stop you, or display any kind of warning. The typing is purely for development, and does nothing once compiled.

If you want typing to actually be enforced, you need to use something like Pydantic.

MrJohz2• 2 years ago

Counter argument: this is actually a myth that gets repeated far too often, and misses the point of what static types do.

If your input data is the correct type (big if #1), and if your types are sounds and consistent with each other (big if #2), then you don't need something like Pydantic, because the types guarantee that there will never be a float stored somewhere that's annotated as a string. It cannot happen, because as soon as you try to store a float in a string attribute, your type checker (Pyright, Mypy, etc) will complain. And if you're consistently running that type checker over your code, e.g. in a CI job that runs over your codebase for every push, then you can never have checked-in code where the types are incorrect.

There are the two big caveats above, but these turn out not to be such big deals. Caveat #1 is that you can't rely on external data to match the types you've described internally. This is an ideal use case for Pydantic, like you say: you check external types once, at the boundaries of your program, and then internally you can be confident that the types will always be correct.

Caveat #2 is that you don't break the type checker's limits. This generally means avoiding things like `Any`, and ensuring that your typed code always calls other typed code, and never untyped, un-annotated code. The easiest way to do this is to start by typing the leaf files in your program (the ones that only get imported, and never import anything in turn). These can't import untyped code, so if they're fully typed and the type checker passes, then you can be confident that their types are correct. Now sure, a bad caller could try and call one of these functions with invalid input, but as they say, Python is for adults, and you're free to call a function with the wrong arguments, and you're free to deal with the results.

Now, you can add types to code that only imports typed code, and because you've checked the imported code you know it's correct, so if you also check the importer and it's also correct, then you can be confident at that level too. You can keep going until you've added types to the entire codebase, and now you can be confident that the entire codebase always passes the correct types, and therefore that no runtime enforcement is necessary.

There is a third caveat, which is that this relies on having a powerful enough type system to model all the things that you were originally doing in Python's dynamic type system. This is hard, because you can do some pretty wild things in Python, and my own experience with Python type checkers has been a bit disappointing - they handle more boilerplate-heavy code fairly well, but seem to struggle on more idiomatic code that uses Python's dynamic features. However, that seems to be improving all the time, and Typescript demonstrates that it's certainly possible to model a dynamic language with static types to a very reasonable accuracy.

vrighter5• 2 years ago

the lsp will complain about i~.

The language happily does it anyway.

calibas2• 2 years ago

What exactly is the "myth" that I'm promoting?

Type annotations do nothing in default Python, they require some other system to enforce them.

binary13210• 2 years ago

I feel like the dynamic typing crowd are slowly coming around to the fact that having a typesystem is just better.

While we’re on the topic, a better typesystem is better than a worse typesystem. Thanks for coming to my TED talk.

btilly2• 2 years ago

No. It's that when people who prefer type systems go to use a popular dynamic language, they try to drag in features that they like from other languages.

Very honestly, optional type systems tend to be the worst of all worlds. Because people who don't care, don't need to use it, you don't get safety. But people who enjoy ceremony can inflict verbosity on others. While missing the most important reason to do it.

paulddraper4• 2 years ago

Optional types are the best.*

I can prototype, script, move fast. Then I can solidify, stabilize, collaborate.

* To get top performance, of course you'll need compiler, types, static dispatch.

binary13210• 2 years ago

I’m not into the whole optional typing thing either. I would consider that far down the “worse typesystem” end of the spectrum I mentioned, only slightly better than “types are just arbitrary labels the language doesn’t even know about”, which is maybe even worse than lean dynamic typesystems, because you might make the mistake of believing in the shadows on the wall.

gklitz• 2 years ago

No. It's that when people who prefer type systems go to use a popular dynamic language, they try to drag in features that they like from other languages.

Very much. The “problem” people try to solve with things like dependency injection isn’t even a problem in python. 99.99% of the time you can just import you dependencies everywhere they are needed.

So many times now I’ve had to unteach bulky OOP patterns for people coming from strictly typed languages to python believe are “good practices” or are “needed for reliable code” and you go from 20 different interdependent classes down to 3 functions that just directly do what they are supposed to do. And you always end up with someone being disappointed that all of these complex patterns they learned to solve problems that don’t exist in python aren’t needed, rather than someone just happy that you can proceed directly to a value adding solution and skip the crud and design pattern spam.

And then they start arguing crazy stuff like the 200 extra lines unneeded code makes the solution “more readable” or “more reliable”, when in reality it’s just a desire to use the solutions they are used to even when the problems don’t exist.

HelloNurse• 2 years ago

Python type annotations are a good source of lightweight, easy to maintain metadata for many frameworks that provide valuable automatic handling of specific data types (for example defining deserialization and deserialization of nice class types with field declarations, like attrs/cattrs or Pydantic, or defining command line interfaces with function parameter declarations, like Appeal).

Expecting type annotations to be more generally useful is mostly projected, baseless declaration anxiety: tools and methods that are useful in other languages are expected to be relevant in Python, certainly comforting for the type-addicted programmer but not necessarily useful. Declaring types is perceived as normal, as a necessary burden and dynamic typing is perceived as missing important structural elements.

Consider the error pattern of accessing a value as if it had a different type: in C or C++ consequences are dire (possibly undetected and cascading memory corruption), likelihood is very high due to specific language features (pointers and references, raw arrays, casts, weak typing in general) and detailed type declarations are a useful mitigation because they turn run time catastrophes into actionable compile time reports, while in Python, thanks to robust duck typing without dangerous complications, consequences are mild (reasonable error messages or wrong results, before corrupting memory), likelihood is low (specific instances of unexpected and malformed data or gross API misunderstandings) and type declarations do nothing.

BiteCode_dev2• 2 years ago

Have an optional one is nice for python. Would suck to be forced to use it all the time.

echelon• 2 years ago

As an undergrad, Python was my favorite language. Now it's one of my least favorite because of dynamic typing and the poor dependency management.

Python dicts, when used as composite types or records, are a literal hellscape. Grepping through the code to find out where stringly-keyed fields get written takes way more time than thinking about types ever would. These should be structs.

Static typing has so many advantages:

- It lowers the software defect rate. All type errors are caught for free at compile time instead of runtime. This makes the software strong and rigid instead of brittle, and it removes an entire category of tests you would have to write and maintain.

- Static typing makes code maintainable for other people, including future you. It's self-documenting. You know precisely what things are in the immediate scope.

- Static typing makes bug-free automated refactoring with tools possible. There is no greater pleasure than mutating code via its AST.

Static typing is not hard, either. Most typed languages don't require type declarations except in structs and function declarations - that's really not a lot of effort.

vrighter5• 2 years ago

a type system is useless if you have no guarantees whether it will be used by any code you use. And if any code that declares types can violate them at will.

Optional is equivalent to no type system at all, in my opinion.

khafra• 2 years ago

I guess you could just write the program in Python, then have ChatGPT automatically translate it into a strongly-typed language.

globular-toast3• 2 years ago

Yes. I like to use type annotations on functions because it's documentation. If you use good names then it makes docstrings redundant. I'm thinking something like this:

    def bounding_box(points: list[Point]) -> Box:
        ...

A decent IDE will be able to quickly jump to definition of those Box and Point etc. if you need it. This is much better than having a docstring and having to look stuff up manually. The fact you can run a static type checker like mypy is a bonus!

I also really like being able to document the imperative code like `def do_thing() -> None`. Of course, it's completely up to the programmer to follow the rule of not doing side effects in routines that return something.

But having to do it everywhere? Ugh. I don't think people realise how powerful duck typing is for doing polymorphic code. I can write something like:

    def mean(things):
        return sum(things)/len(things)

And I don't care what concrete type you pass me as long as it supports `sum` and `len`. What am I going to do, define an ABC or typing.Protocol called `ListLike` or something? Hell no, I've got better things to do.

But of course learning when to define static types vs when not to comes down to experience. Python treats you like an adult. I feel like a lot of people who want static typing everywhere want it as training wheels for other devs they don't trust to make the right judgement calls.

zo12• 2 years ago

Python's optional type-hinting is leaps and bounds better than a static typing system (on its own). It's fluid, allows expressibility whilst balancing guard-rails, and is making crazy cool use of "execute-time" or "import-time" time concepts (as an alternative to design-time and run-time).

So no, we're not coming around. Static typing as dictated by a heavy-handed and super-strict compiler has it's place, but has gimped our industry for decades.

However, we do have to be very careful. Static, compile-time typing has kept the hipsters and junior-devs at-bay, and kept them from causing too-much havoc as we've seen in the JS world. So it's definitely an up-coming hazard for us to navigate around and make sure we don't fall prey to. Otherwise python will turn into another JS dumpster fire. Luckily, the JS developers are too-distracted and enthralled with node.js to jump ship.

scott_w13• 2 years ago

I don’t think that’s strictly true. Python developers are finding there are advantages to having a type system in larger programs such as readability and a reduction in certain classes of bugs.

That said, I’ve found Python’s approach lets me still build things through experimentation then fixing the types when I’m a bit more confident that I’ve built the right thing.

As a comparison, I found Go really clunky when I tried to learn it because it wouldn’t compile if the code wasn’t totally correct. It makes it hard to find the right solution through experimentation.

sinfulprogeny• 2 years ago

then fixing the types

man I wish my coworkers did this step

josephg6• 2 years ago

Python’s problem is that there’s multiple 3rd party type checkers and they all disagree in subtle ways about what the spec says. It’s a mess.

Python is probably an all-round better language than JavaScript. But typescript absolutely embarrasses Python’s “gradual” type system with just how good it is in comparison.

noelwelsh2• 2 years ago

Two things here:

1. You need to understand how to express concepts in a form the type system can understand. Each type system will have patterns it understands, and patterns it does not.

2. Go isn't a great example. It has a Java-circa-2000 type system. Modern type systems are much more expressive.

maxbond6• 2 years ago

It makes it hard to find the right solution through experimentation.

To me this suggests the tooling was insufficient. One of the things I value about type systems is being in a tight feedback loop with the compiler. Strong typing helps me run more experiments and to run them from within my editor. But without a good LSP and such, it would be miserable.

st3fan• 2 years ago

You can remove "larger programs".

Personally I also find python typing useful in "smaller" programs and I use them pretty much everywhere.

For example I use it in all my automation scripts which are usually not much longer than a few pages or a few modules. Adding type hints is a very minor task, really comes straight out of brain just like the code i type.

Type hints have given me wonderful completion and documentation which has helped me to write more correct and bug free code. Often little things pop up that would only have been caught at runtime. Now I see those things as soon as I type them.

Python + Type Hints == Awesome

paulddraper4• 2 years ago

because it wouldn’t compile if the code wasn’t totally correct

Go won't even compile if a variable is unused.

Makes it really hard to comment-out lines when debugging.

jgb1984• 2 years ago

I've been writing python for a living for almost 20 years. Big projects. Never felt the need for static typing. And I strongly dislike the verbosity and complexity it adds to the language. Python is beautiful because it's simple, easy to read, easy to understand. The typing tagged on to it looks like an ugly word salad to me.

I'll keep my dynamic typing, thanks!

IshKebab3• 2 years ago

Have you ever tried static typing though? I suspect you don't know what you're missing. Static types make Python way easier to understand.

zephyrfalcon• 2 years ago

There is definitely a push for it; not all of us think it's an improvement. When I see modern Python code, it looks nothing like the (relatively) clean, smallish and easy-to-understand language that it was 25 years ago. I get it, things change. The language now caters to a different crowd and attracts different people.

But I have often wondered, if someone wants static typing, why not just use a statically typed language?

Philpax• 2 years ago

I would and do, but a lot of existing code (especially ML code) is Python and I find myself having to interface with it. It's not an awful lot of fun, especially when there are no annotations to record what the code expects and what it outputs.

vundercind• 2 years ago

But I have often wondered, if someone wants static typing, why not just use a statically typed language?

A very high proportion of Python devs aren’t using it for the language, but for the libraries and ecosystem. Also lots of them weren’t the ones who picked it.

Meanwhile, not at least having the kind of autocomplete and documentation that type hints provide is kinda hellish on any project of more than 200 or so lines. The time savings from spotting runtime bugs before they happen is just a bonus.

Personally, I almost never need to add a type hint outside high-level definitions and function/method signatures, so they’re not really in the way even when I’m being pretty thorough with them.

lyu07282• 2 years ago

or perhaps a bad typesystem is worse than no typesystem at all? I think typing in python is obviously getting better, it just feels something went wrong when they constantly have to fix design mistakes other languages never made to begin with. Typescript is fun, its powerful and strict, typing in python is ugly and frustrating.

If this is the first time people are introduced to static typing in programming I can understand their frustrations and opposition to it. Its probably the worst type system in any modern, popular language out there, except perhaps when things like clojure(script) pretends to have types (shudder).

RHSeeger3• 2 years ago

Having a type system provides benefits. Having a type system also comes with costs. The trick is whether (you believe) your specific use case gets enough out of those benefits to pay the costs.

binary13210• 2 years ago

I can’t think of a time I have paid more for using a typesystem than I have been empowered by it. If you’re using types properly, it should be very empowering, and the cost not really perceptible. If anything, it reduces cost by moving things out of my brain and tests, into the typesystem.

lucasyvas• 2 years ago

Statically typed languages are in fact so much better nowadays that the industry should drop all dynamic / scripting languages for application development

That will sadly probably never happen given the momentum they’ve built so the only choice is to retroactively add static typing and begin enforcing it in individual projects.

When starting a new project, I would suggest nobody choose a dynamically typed language. Between Swift, Kotlin, C#, Go, Rust, etc… there’s no need for anything else. As long as front end web is around you might need TypeScript - and as long as ML is Python centric you might benefit from using a bit of it. But I wouldn’t make them the primary language.

Context: Recovering dynamically typed language addict of 12 years. They’re slow, error prone, and don’t scale to a large engineering team.

BiteCode_dev2• 2 years ago

TypedDict features are nice, but the syntax to declare it is aweful

psd12• 2 years ago

The article doesn't mention my train for loving TypedDict.

I worked on a python app built by a man with a stronger will to succeed than ability to code.

Data was passed as dicts. Many methods (in the god object, natch) destructured, unwrapped or mutated before passing the new dict to some other method.

TypedDict allowed me to annotate these shitty dicts. It became possible to reason about the code without spending two hours tracing code paths.

The real solution is to be better at code. But, given an app written by a dict fetishist, it's a pretty good solution.

sevensor• 2 years ago

I’ve been in the same boat. The amount of pain you can reduce by pushing type annotations through a codebase like that is spectacular. And although I love dataclasses, the non-totality that you get from typeddicts is key for modeling the kind of mess you describe.

psd12• 2 years ago

TypedDict is a drop-in solution. Dataclasses is a big scary refactor. I'd use dataclasses in a new app, but it was not really an option in that app.

mistahchris• 2 years ago

the link is 404ing, but i found the post in the author's github repo: https://github.com/Jamie-Chang/Jamie-Blog/blob/18c0aaf2266de...

TypedDicts are better than you think

Related Stories