Numpy is absurd

https://gist.github.com/Moelf/59d6312c51c250ba251125e54bea7282

by moelf • 4 years ago

102 50 4 years ago

gist.github.com

I always gripe about Python not having built-in array type

  from array import array

it’s right there.

Numpy doesn't distinguish "vector of vector" from "matrix"

  numpy.matrix

though to be fair, it’s on its way out.

moelf4• 4 years ago

when built-in array type is not performant nor serves as an interface (abstract class), it's as if it doesn't exist. I guess the point is Numpy predates Python's array by a long shot.

Compare and contrast with languages with built-in N-dimensional array for scientific computing.

hprotagonist5• 4 years ago

like matlab, that lies?

jwilk• 4 years ago

Numpy predates Python's array by a long shot

I don't think so. The array module was included even in Python 1.0.

Or did you mean to use another verb?

cuteboy194• 4 years ago

Who needs this strange operation anyways? Deprecate the whole method and call it a day

moelf4• 4 years ago

that's what I'd go for, throw more errors when `__contains__` is called with ambiguity.

but, cough, mature and stable scientific computing foundation?

judofyr• 4 years ago

Here's a better resource for understanding how __conains__ works which actually looks at the source/documentation instead of trying to understand it intuitively: https://stackoverflow.com/questions/18320624/how-does-contai...

Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.

gnulinux2• 4 years ago

Well yeah you're right OP could have explained this better; but this argues their point. From the definition that `a in b` is `(a == b).any()` it's trivial to see things like `np.array([1, 0, 0]) in np.array([1, 2, 3])` will return `True` too, which is absurd, as the title says. I find it hard to believe when numpy team was designing the API they missed this.

kzrdude3• 4 years ago

It's trivial to see some things and other things use broadcasting and that's not quite trivial :)

cuteboy194• 4 years ago

Does == ignore dtype?

rich_sasha• 4 years ago

Yes. 1 == 1.0

Someone• 4 years ago

Even that took me a while to understand it and, I guess, is another example of Numpy’s absurdity.

My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans.

But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence.

Reading https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.

imiric• 4 years ago

For a language with "explicit is better than implicit" as part of its mantra[1], Python is chock full of surprising behavior like this where the programmer really has no idea what's happening under the hood.

I'm happy I moved on to Go, which actually adopts this as a core design tenet.

[1]: Even though PEP-20 is largely a meme.

qbasic_forever• 4 years ago

To be fair this person's gripes are with the behavior of Numpy arrays, not anything that's native to Python's core language. Numpy arrays are tricky beasts because they try to balance fast operations on huge amounts of data, with expressive Python operator syntax.

ModernMech4• 4 years ago

Although, another way of looking at what you just said is that Numpy arrays exist because core Python fails to strike a balance between being both fast and expressive.

ThePhysicist• 4 years ago

I've worked with numpy for years and I don't think I ever used the 'in' operator with arrays. What would be the use case for that? For me a larger gotcha was around object multiplication (i.e. element-wise vs. matrix multiplication) and how it is different based on whether you have a "matrix" or an "array" type (or a mix of both). That caused some pretty big headache in the beginning.

weare138• 4 years ago

But that just seems like a foot gun for the poor unfortunate soul that eventually does.

sillysaurusx• 4 years ago

Meh. Footguns are underrated. Let’s return to the Wild West of the 90s.

I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.

rowanG0772• 4 years ago

I don't think people who use python are concerned with footguns. Numpy is very much not at fault here.

__del__• 4 years ago

iirc matrix is deprecated

bo10242• 4 years ago

Interpreting “in” as “is an element of”, all of these are type errors. It only makes sense to ask if an X is in an Array<X>. In a language that lets you ask nonsense questions, expect nonsense answers.

kzrdude3• 4 years ago

Numpy has the highly useful operation of `a < b` where a and b are arrays and the answer is an array of booleans - i.e broadcasted elementwise compare.

They changed the language to change what question `<` asks.

kevin_thibedeau• 4 years ago

This gets ugly with !=. Instead of 'a != b' you usually want 'not (a == b).all()'.

bo10242• 4 years ago

Sure, and languages like C++ let you do this kind of thing very broadly, but it has a lot of pitfalls. One of my biggest gripes with python is that function interfaces and "type signatures" are hard to discover even when documentation is good.

moelf4• 4 years ago

Op here, I'd also like to say I blame this on Python's lack of broadcasting syntax, (dot in Matlab and Julia)

Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container

johnmyleswhite• 4 years ago

Julia's broadcasting syntax depends on having Julia-style compilation to be efficient. It's not clear to me Python could ever provide the same behavior.

dima552• 4 years ago

Core numpy is really weird, and has really sloppy logic around higher-dimensional objects. I've a module to work around lots of the warts. If somebody has smarter versions of intersect() or in() or whatever, contribute!

https://github.com/dkogan/numpysane

cuteboy194• 4 years ago

Apologies if this was explained in the talk, but why not upstream your changes to numpy?

dima552• 4 years ago

No real good reason, honestly. I wasn't feeling positively about numpy devs at the moment, and didn't want to negotiate getting the code in. The library is solid though, and should be merged. I need somebody to pester me about it, to actualy work on that today.

kakadzhun• 4 years ago

    import numpy as np
    print(np.__version__)
    a = np.array([1,2,3])
    b = np.array([1])

    print(a in b)
    print(b in a)
    print(np.isin(a, b))
    print(np.isin(b, a))
    print(np.isin(a.astype(float), b))
    print(np.isin(a, b.astype(float))))

    1.21.6
    True
    True
    [ True False False]
    [ True]
    [ True False False]
    [ True False False]

stosto88• 4 years ago

Finally! It had to be said. Im a huge Python fan and a huge Pandas fan and have written hundreds of scripts. But numpy is so "R" esque and technical and stiff and unintuitive for sure.

joshlk• 4 years ago

I always gripe about Python not having built-in array type

Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html

tpoacher• 4 years ago

Oh no! Undefined behaviour! What ever shall we do?

Numpy is absurd

Related Stories