Namedtuple in a Post-Dataclasses World

https://death.andgravity.com/namedtuples

113 88 5 years ago

death.andgravity.com

PaulHoule• 5 years ago

In practice the hashtable implementation in Python is so fast, particularly for cases like

  {"x": 51.2, "y": -74.1 }

that you don't gain a lot from namedtuples most of the time. I quit benching namedtuples for applications like that a long time ago.

johtso• 5 years ago

Immutability is nice though.

wyldfire2• 5 years ago

Also it's nice for bad references to trigger AttributeError which is almost always a design error, whereas KeyError is not always so.

I haven't tried it but type hints are probably smart enough to find errors like this statically with namedtuple, but probably not so with a dict.

toxik• 5 years ago

a.x is far easier to both read and write than a[”x”] (67% is operator syntax compared to 33% in a.x).

wcarss• 5 years ago

this and () => {} syntax have always been moderate to strong wins in my book for js when creative-coding

ziml772• 5 years ago

Not to mention that the first case gets assistance from autocomplete in an IDE or ipython session. That speeds up typing long, descriptive names so much.

masklinn5• 5 years ago

The main problem of dicts is that they're really heavy memory-wise, even with key-shared dicts and stuff.

Your dict is 232 bytes, the equivalent tuple is 56.

brandmeyer2• 5 years ago

How much does automatic interning of string literals help out here?

joshuamorton2• 5 years ago

I thought I'd add to this (because I was surprised) that the equivalent namedtuple is also 56 bytes (I expected it to be larger) and the equivalent dataclass is a mere 48 bytes. (although there's overhead for defining a namedtuple or a dataclass, on the order of a constant 1kb).

Although there seems to be some kind of trickery happening there, because if I make the class accept 3 floats instead of two, neither the class nor the instance get larger.

6gvONxR4sf7o• 5 years ago

In cases where you don't care about immutability, I'd think of it as a better version of TypedDict (though TypedDict still has its place). It makes my IDE more helpful, makes my code more self-documenting, and allows mypy to tell me when I'm being dumb.

donio• 5 years ago

That's only because Python is so slow in general and everything else is implemented in terms of hashtable lookups too.

heavenlyblue• 5 years ago

You could define __slots__ on the namedtuple that would not use a hashtable for lookups.

sfvisser• 5 years ago

Point = namedtuple('Point', 'x y')

I must come from a different world. What is going on here?

Are you dynamically creating a named tuple (a record?) by passing a space separated list of field names? Why?

cdcarter• 5 years ago

They are dynamically creating a named tuple class (or prototype). The namedtuple implementation in the python standard library indeed accepts a space separated list of field names. Once it's been defined (as above) then you can create instances (records) like "Point(2,3)"

selectnull• 5 years ago

I always wondered why is that a space separated string, when a list can work as well. The docs are not well written on that one. This works:

    Point = namedtuple('Point', ['x', 'y'])

xav0989• 5 years ago

The function was defined to either take a space separated list of names or a sequence of names. The docs seem pretty clear to me:

The field_names are a sequence of strings such as ['x', 'y']. Alternatively, field_names can be a single string with each fieldname separated by whitespace and/or commas, for example 'x y' or 'x, y'.

sgtnoodle2• 5 years ago

It's a class factory function, so right off the bat it's a bit weird. The original intent of using spaces was probably to minimize typing. Since they're attribute names they can't have spaces in them, so it's a safe delimiter.

You could imagine the function dynamically creates the class by manipulating the underlying dictionary (or whatever the "slot" alternative uses). At that level of python, attributes are strings anyway. Handling spaces is just a matter of calling .split().

In modern python, there's a whole metaclass system that would possibly let you do the equivalent without getting your hands dirty with internal data structures.

dec0dedab0de3• 5 years ago

I think it comes down to the idea that going out of your way to make the library work either way makes it easier for people to use, even if it makes the library itself a bit more complicated.

I wish more library devs would go out of their way to add such niceties.

A big one that I always do is if I'm expecting an iterator of objects I make it just work with one.

  from collections.abc import Iterable
  def my_function(arg):
      # slightly different if you're looking for a collection  of strings or bytes
      if not isinstance(arg, Iterable): 
          arg = [arg]

      for item in arg:
          do the thing

Or if you have a specific type of object you want it goes like this

  from collections.abc import Iterable
  def my_function(arg):
      if isinstance(arg, MyObjectIWant): 
          arg = [arg]

      for item in arg:
          do the thing

I like to think of my libraries as mini programs for users, and I hate when validation is too strict, when it could be so easy to fix. Like when a phone number validator insists on (XXX)XXX-XXXX or XXX.XXX.XXXX or XXXXXXXXXX when it could just ignore everything that isn't a number and make sure there is 10 of them.

goodside3• 5 years ago

I’d speculate it’s meant to mimic Perl’s `qw()` operator, which is like `str.split()` in Python. The module was originally written for contexts where you’re processing SQL result sets with fixed schemas, and before Python these tasks were traditionally handled in Perl. Python inherits a lot of these loose traditions. Similarly, some parts of the standard-lib (`sys`, `os`) follow shell- or C-like naming conventions that would seem bizarre to someone who’s never used a shell prompt.

dragonwriter2• 5 years ago

I always wondered why is that a space separated string, when a list can work as well.

Saves a bunch of typing. 5 chars:

  'x y'

vs 10:

  ['x', 'y']

raymondh2• 5 years ago

Using a list or tuple for the fields is generally best:

    Point = namedtuple('Point', ('x', 'y'))

Support for a space and/or comma separated strings was requested by users. It made life easier for them when syncing with other space/comma separated strings. For example, an SQL query, "SELECT name, rank, serial_number FROM Soldiers;" would have a corresponding named tuple where the field names could be cut-and-pasted from the SQL query.

    Soldier = namedtuple('Soldier', 'name, rank, serial_number')

wyldfire2• 5 years ago

Are you dynamically creating a named tuple (a record?) by passing a space separated list of field names? Why?

Very little in python is bound statically. This is akin to a type definition. The type will behave as an ordered tuple that can be indexed but also alias these attribute names to those ordinals.

    assert(a == Point(a, b).x)
    assert(b == Point(a, b).y)
    assert(a == Point(a, b)[0])

qbasic_forever• 5 years ago

It's effectively a 'macro' for metaprogramming.

masklinn5• 5 years ago

Are you dynamically creating a named tuple (a record?)

No, it's creating a namedtuple type, aka a subclass of `tuple`. So the fieldnames are literally the names given to the tuple's items: Point is a pair (a two-uple) whose 0th item can be accessed as `x` and 1st item `y`.

goodside3• 5 years ago

After your example, `Point` is a class. You are not creating a record, you’re creating the class that defines the schema of fields in any record. A class can be defined without using the `class foo:` syntax, and in this case the class is returned by the function `namedtuple` to produce an equivalent effect.

As for the unusual space-delimited syntax, the missing context here is that namedtuple is a very, very old part of Python that predates the conventions now considered good style. Using space delimiters for lists of strings is a common idiom in Perl scripting due to the `qw()` quote syntax. Note the archetypical context where namedtuple was imagined to apply (record-oriented processing of logs and SQL result sets) was commonly handled using Perl before Python became dominant.

Namedtuple is definitely the most prominent example of this syntax convention in Python, but other libraries use it too. `enum.Enum` supports a function-like interface directly modeled off namedtuple. It’s a mildly bad idea to keep using it IMHO because it complicates static analysis or refactoring. If someone does a wide search-and-replace of a field name literal it’s easy to miss this edge case.

jhgb2• 5 years ago

a very, very old part of Python that predates the conventions now considered good style. Using space delimiters for lists of strings is a common idiom in Perl scripting due to the `qw()` quote syntax.

Note that the Smalltalk-80 class creation method also expects a space-separated string of instance variable names, and that's an environment considerably older than Perl.

jhgb2• 5 years ago

Are you dynamically creating a named tuple (a record?) by passing a space separated list of field names? Why?

I believe that's called "procedural record interface" in Scheme and it does have its uses, for example if you need to create records for data the structure of which you don't know in advance.

nikeee• 5 years ago

There is also a NamedTuple (notice the different casing) from the typing library, which doesn't seem to be mentioned in the article:

https://docs.python.org/3/library/typing.html#typing.NamedTu...

    class Employee(NamedTuple):
        name: str
        id: int

This is equivalent to:

    Employee = collections.namedtuple('Employee', ['name', 'id'])

colejohnson66• 5 years ago

I don’t use Python much, but what’s the difference between a NamedTuple and a regular class?

akoumis• 5 years ago

NamedTuple is purely a data container. It does not have class functions or a constructor you can use for anything other than setting the data members.

ZuLuuuuuu• 5 years ago

NamedTuple has the features of a tuple, for example it is immutable. A regular class is mutable.

palotasb• 5 years ago

NamedTuple or namedtuple instances are tuple instances that have the same properties that regular tuples have. They are immutable (you cannot reassign their fields), you can index into them (a[0], a[1] instead of a.x and a.y), you can unpack them with *a. They can have methods like regular classes can, including additional @property methods. A NamedTuple class cannot inherit from another class, not even other named tuples.

dec0dedab0de3• 5 years ago

a named tuple works exactly like a tuple, except you can also use names to get the items in it.

so it is immutable and you can get it via slicing.

  >>> from collections import namedtuple
  >>> Hat = namedtuple('Hat', ['style', 'size', 'color'])
  >>> my_hat = Hat('safari', 'XL', 'Orange')
  >>> my_hat
  Hat(style='safari', size='XL', color='Orange')
  >>> my_hat[0]
  'safari'
  >>> my_hat.color
  'Orange'
  >>> my_hat[1:]
  ('XL', 'Orange')
  >>> style, size, color = my_hat
  >>> size
  'XL'
  >>>

If thats all your using your classes for, then a named tuple is probably a better solution, or a dataclass. Though I normally just use dicts in that situation. If I see someone create a class without any methods, or atleast planned methods, I don't let it through code review.

EDIT: Also, Raymond Hettinger created named tuples. I'm not normally one for call to authority, or hero worship, but I am a huge fan of his. I recommend that anyone interested in Python should watch as many of his talks as they can.

EDIT2: As masklinn pointed out, another really good use of named tuples is when you're already returning a tuple, and you realize it would be better if it had names. You could change it to a named tuple without breaking any of the existing code. Unless they're doing something dumb like halfassing type checking at runtime. (this use case is in the article, which i didn't read at first)

masklinn5• 5 years ago

Well in and of itself none, in the sense that anything a namedtuple can do you could do by hand (it really just defines a class). However namedtuple:

* extends tuples, so a namedtuple is literally a tuple (which is useful)

* sets up a bunch of properties for the "named fields", which are basically just names on the tuple elements

* sets up a few other utility methods e.g. nice formatting, `_make`, `_asdict`, `_replace`

Now the latter two are nice, and mostly replicated by dataclasses (or attrs). The first one is the raison d'être of namedtuples though: originally their purpose is to "upgrade" tuple return values into richer / clearer types e.g. urlparse originally returned a 6-utple which is not necessarily super wieldy / clear, you can probably infer that the 3rd element is the path but… after upgrading to namedtuple it's just `result.path which is usually much clearer.

And because namedtuples are still classes in and of themselves, you can inherit from them to create a class with a `__dict__` with relative ease.

agumonkey• 5 years ago

I think it was a pre-organized immutable data class in one line.

Spivak4• 5 years ago

I feel like none of the sibling answers actually answer your question which is "absolutely nothing." The function namedtuple is code generator that constructs a class definition and then eval()'s it.

The reason you reach for it is because it's tedious to write the same methods over and over to get things like a nice repr, methods covert between dicts, or pickling support.

The source from Python 3.6 is much more readable than 3.9 so I recommend reading that if you want to see how it works.

https://github.com/python/cpython/blob/3.6/Lib/collections/_...

Nagyman• 5 years ago

It is mentioned if you expand the "In case you've never used them, here's a comparison." element.

ziml772• 5 years ago

I like that method of defining a named tuple much more. I understand why it's required, but having to duplicate the name of the named tuple always bothered me. The type hints are excellent to have too.

Spivak4• 5 years ago

I don't know what you're talking about, I always confuse my coworkers by doing

    Pointe = namedtuple('Point'...

mikepurvis• 5 years ago

That's awesome and I was not previously aware of it. Literally a decade ago I was asking on SO about exactly this kind of use-case [1] and at the time the answers were pretty unsatisfying; it's great to hear that the story is better now.

[1] https://stackoverflow.com/questions/4071765/in-python-how-do...

sayhar• 5 years ago

A little unrelated, but this brings up a question I've had for a while.

Seems like one day, everyone around me was using dataclasses. I had not even heard of them. It felt like I had missed some memo or newsletter. It felt weird.

Here's my question: what should I have been reading / where should I have been "hanging out" online, so that I would have known that dataclasses were a thing? What are your go-tos for news about new language features, libraries that everyone is using, etc?

Hacker news is great, but it doesn't quite fill that need for me, it seems.

icegreentea2• 5 years ago

I follow the language specific sub-reddits, and I read release notes for major releases of languages (so for python that would be 3.X) even if I wasn't going to jump to the version set.

genericlemon242• 5 years ago

I read release notes for major releases of languages

_This_, so much.

If you are a heavy user of a language / library, it's immensely helpful to look at the release notes every once in a while. Even if you don't plan to upgrade now, it gives you an idea of where things are going (and may eventually tip the scales to a "fuck it, it's now worth upgrading" moment).

For Python specifically, PEPs are also a good way to keep track of what's happening (even if some of them don't get accepted): https://www.python.org/dev/peps/ ; there's also an RSS feed: view-source:https://www.python.org/dev/peps/peps.rss/

emddudley• 5 years ago

Python-centric forums, like r/Python:

https://www.reddit.com/r/Python/

I think the RealPython site is excellent for learning, even for mid to advanced users:

https://realpython.com/

They also have a great podcast:

https://realpython.com/podcasts/rpp/

Also just browsing the Python docs and release notes.

pydry• 5 years ago

r/Python is 90% newbies showing off toy projects. It's not great for news.

lbhdc• 5 years ago

I read the release notes. I often see posts for releases here on HN or on reddit, but often I will check in on the official repos or websites to see whats new.

I like to spend a few hours a week reading up on whats happening, or try something new to keep up. Checking out new language features is part of that processes to me.

bobbylarrybobby2• 5 years ago

For Python, you pretty much just need to be aware of when the new major version is released because the "what's new" pages are pretty good. Here's the one in which data classes were released: https://docs.python.org/3/whatsnew/3.7.html

chaos_emergent• 5 years ago

Concretely, I found out about dataclasses by using [pydantic](https://pydantic-docs.helpmanual.io/) and seeing their drop-in `@dataclass` annotation - it got me curious about the adjacent stdlib class. I was using pydantic because I started using FastAPI to build a REST interface, which has pydantic deeply integrated.

Generally, I find out about new features through PEP posts, and I reach those by seeing a keyword that I don't know in random code I read online

dknight10• 5 years ago

A couple good newsletters are Python Weekly and PyCoder's Weekly. They each put out a mix of news, articles/tutorials, and interesting projects.

https://www.pythonweekly.com/ https://pycoders.com/

genericlemon242• 5 years ago

Already duplicated in my reply to icegreentea2 below, but release notes for projects you use are a great place to get updates.

For Python specifically, PEPs are very helpful too: https://www.python.org/dev/peps/

dec0dedab0de3• 5 years ago

I found out about data classes on hn, before they were in the standard library. I also regularly search for python to see what stories I missed.

I also like to keep up to date with the PyCon videos, as well as some of the other python conferences. But, as others have said, the release notes are the main source for whats new, if a bit dry.

That said, I never actually use data classes. I normally just use dicts, and occasionally named tuples.

raymondh2• 5 years ago

FWIW, here is a PyCon video to get you up to speed on dataclasses:

https://www.youtube.com/watch?v=T-TwcmT6Rcw

nerdponx2• 5 years ago

In the past, Freenode #python, now Libera #python. Also the Python Discord server.

msravi• 5 years ago

The syntax is a lot more intuitive in Julia:

julia> point = (x=1, y=2)

(x = 1, y = 2)

julia> point.x

julia> point.y

julia> dump(point)

NamedTuple{(:x, :y), Tuple{Int64, Int64}}

  x: Int64 1

  y: Int64 2

bobbylarrybobby2• 5 years ago

And to create one as an anonymous type, you can use the `@NamedTuple` macro:

    julia> @NamedTuple{x::Int, y::String}
    NamedTuple{(:x, :y), Tuple{Int64, String}}

mdcfrancis• 5 years ago

they are also, type stable, strongly typed and have the same overhead as a struct. So an array of NamedTuples takes the minimal space and allocation.

rdtsc• 5 years ago

I guess one difference is that when you inspect it, it doesn't indicate that it is a `point`, just that it's named tuple of two variables, so it's not exactly equivalent.

the__alchemist2• 5 years ago

Dataclasses and Enums have, since their introduction, taken over as my foundation of Python data structures. They've obsoleted NamedTuple, namedtuple, and traditional classes in my code.

They're a clean way of defining how difference functions, methods etc should interact with each other.

synergy20• 5 years ago

what do you mean by 'traditional classes', dataclasses are more for data-only data structure with no methods?

the__alchemist2• 5 years ago

As in, classes that aren't tagged as `dataclass` or `Enum`. You can have methods with dataclasses.

bionhoward• 5 years ago

Along this line of reasoning I learned to love Pydantic which can make it a breeze to parse and coerce environment variables to the correct types: https://pydantic-docs.helpmanual.io/usage/settings/

You can make an env.py and override module __getattr__ and then import environment variables just like they're regular Python objects (even booleans, floats, collections, etc, despite .env files being string only

Huge force multiplier for ML cuz then you can do hyperparameter optimization just by passing different environment variables in an outer loop (even inside your infrastructure as code)

Edit: you can make these classes immutable too: https://pydantic-docs.helpmanual.io/usage/models/#faux-immut...

jkrubin• 5 years ago

Pydantic is an incredible lib. I use it for so many different things on top of misc parsing.

ElevenPhonons• 5 years ago

You can also create CLI tools that can load partial or full "presets" defined in JSON.

https://github.com/mpkocher/pydantic-cli

actually_a_dog• 5 years ago

Speaking of namedtuple, I would encourage anybody who uses Python and wants to learn a thing or two to read the source code for them. At least one of the things you learn should probably fall in the "what not to do" category. There's a lot going on in there to support all that magic you see from the outside, and it's a little scary in there.

RhysU• 5 years ago

This*1000.

I have been spending an embarrassing amount of time trying to merge a one-off version of dataclasses with dataclasses to pick up the codegen based on type hints. This stuff is nasty and subtle under the covers. I would rather be dealing with a real closure or with real compositional capabilities. Dataclasses are gross in the weeds.

For example, y = dataclass(x) mutates its argument. That is, y = x.

continuational• 5 years ago

Such a confusing title without "Python" in there, especially when it's specifically about some Python feature.

jammycrisp2• 5 years ago

Y'all may be interested in a fast dataclass-like library I maintain called msgspec (https://jcristharif.com/msgspec/) that provides many of the benefits of dataclasses (mutable, type declarations), but with speedy performance. The objects are mainly meant to be used for (de)serialization (currently only msgpack is supported, but JSON support is in the works), with native type validation (think a faster pydantic).

Mirroring the author's initialization benchmark:

    In [1]: import msgspec

    In [2]: from typing import NamedTuple

    In [3]: class Point(msgspec.Struct):
    ...:     x: int
    ...:     y: int
    ...: 

    In [4]: class PointNT(NamedTuple):
    ...:     x: int
    ...:     y: int
    ...: 

    In [5]: %timeit Point(1, 2)
    48.4 ns ± 0.195 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

    In [6]: %timeit PointNT(1, 2)
    185 ns ± 0.851 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

nerdponx2• 5 years ago

Is this faster than Attrs as well?

jammycrisp2• 5 years ago

Yes, but it's also less flexible. Tradeoffs.

Namedtuple in a Post-Dataclasses World

Related Stories