ngoldbaum

I wonder why the github status page has an atlassian cookie request pop-up.

Either something about beanie babies or something riffing on "thank you". Couldn't ever make up my mind then basically forgot about it.

I gave away the “ty” project name on pypi to Astral a week or so ago. I wanted to use it for a joke a few years ago but this is a much better use for a two letter project name. They agreed to make a donation to the PSF to demonstrate their gratefulness.

You could do two passes over the string, first get the total length in bytes, then fill it in codepoint by codepoint.

You could also pessimistically over-allocate assuming four bytes per character and then resize afterwards.

With the API in the linked blog post it's up to the user to decide how they want to use the output [u8;4] array.

The main difference is the strings are stored in a single contiguous arena buffer (with some minor caveats if you mutate the array in-place). With object strings each string has its own heap allocation.

More details in NEP 55: https://numpy.org/neps/nep-0055-string_dtype.html

This post is based on the content of a 25 minute talk and it’s hard to explain everything fully…

Ah I could see how that’s confusing. I was trying to indicate that the size stored for the string in the example is 28, but it’s stored in a 64 bit uint.

They’re stored on the DType instance. This requires that there’s only one DType instance per “owned” array buffer, which I figured out how to do along with Sebastian Berg and others using the new DType system.

That is something I’d like to see but I don’t want to wade into the already very complicated discussion around arrow strings in pandas. If a Pandas developer wanted to take this on I think that would make things easier since there’s so much complexity around strings in Pandas.

That said there is a branch that gets most of the way there: https://github.com/pandas-dev/pandas/pull/58578. The remaining challenges are mostly around getting consensus around how to introduce this change.

If NumPy had StringDType in 2019 instead of 2024 I think Pandas might have had an easier time. Sadly the timing didn’t quite work out.

This was a case of convergent evolution, both projects ended up working simultaneously on similar ideas.

One issue with using Arrow directly in NumPy is PyArrow exposes an immutable 1D array, while NumPy exposes a mutable ND array.

See also https://numpy.org/neps/nep-0055-string_dtype.html#related-wo...

Well, there was no concept of sidecar storage. Now we have the hack we came up with for StringDType to store data on the DType instance and also make it so StringDType arrays don't share StringDType instances, unless the array is a view.

EDIT: looking back at the NEP, I'm not sure it does a great job explaining exactly how the per-array descriptor works. Ultimately it's powered by a hook in the DType API: https://github.com/numpy/numpy/pull/24988. There is only one spot in NumPy where array buffers are allocated, so we hooked there and made sure any arrays with newly allocated buffers get a new DType instance.

Thank you this really means a lot.

No, it wouldn’t make sense to. Use threading.Lock for that. PyMutex is available in the CPython C API.

https://github.com/numpy/numpy/issues/26510#issuecomment-229...

And now that I look at that again I realize I forgot to finish that up!

This style of mutex will also power PyMutex in Python 3.13. I have real-world benchmarks showing how much faster PyMutex is than the old PyThread_type_lock that was available before 3.13.

The complaint [1] is linked in the article. They're suing over Intel internally using and redistributing the anaconda software distribution.

[1] https://fingfx.thomsonreuters.com/gfx/legaldocs/znvnxomagvl/...

Clarifying a few days later: single-threaded performance in the normal ABI with the GIL does not have the same performance degradation. You only see the performance hit if you’re testing the experimental 3.13 free-threaded release.

It’s an experimental release in 3.13. Another example: objects that will have deffered reference counts in 3.14 are made immortal in 3.13 to avoid scaling issues from reference count thrashing. This wasn’t originally the plan but deferred reference counting didn’t land in time for 3.13. It will be several years before free-threading becomes the default, at that point there will no longer be any single-threaded performance drop. Of course that assumes everything shakes out as planned, we’ll see.

This post is a call to ask people to “kick the tires”, experiment, and report issues they run into, not announcing that all work is done.

Right now there is a significant single-threaded performance cost. Somewhere from 30-50%. Part of what my colleague Ken Jin and others are working on is getting back some of that lost performance by applying some optimizations. Expect single-threaded performance to improve for Python 3.14 next year.

We had a developer meeting to discuss what should go into 2.0 in April 2023: https://github.com/numpy/archive/tree/main/2.0_developer_mee...

Teeny nit, the sun produces light well into the x-rays (mostly from the corona though). You're probably talking about sunlight making it through the atmosphere.

The owner, tod sacerdoti, is using it to scrape top stories from lobste.rs and submit them to HN as a karma arbitrage pump. Search todsacerdoti on https://gerikson.com/hnlo/ to see what I’m talking about. Judging by the amount of HN karma he’s accumulated it seems to work very well.

If you like the art you should try and play Cosmic Osmo! It was my favorite game to play on the Performa PowerPC macs my elementary school had. It’s a very short and silly game made with hypercard. The creators later made Myst using the same technology.

It’s a screenshot from Cosmic Osmo and the Worlds Beyond the Mackerel.

It’ll be guarded by a fine-grained mutex, so it won’t seg fault. They’re using a performant mutex based on webkit’s wtf::lock.

Fun to see an open source project I contribute to (numpy) used to commit some light fraud.

I find py-spy really useful in python, it would be neat if you integrated with that somehow.

I’m working on adding missing data support for strings as part of adding a UTF-8 variable-width string type to NumPy. Not a general solution but should help with a lot of use-cases. https://numpy.org/neps/nep-0055-string_dtype.html

It’s a little easier to write idiomatic python bindings for a C/C++ library in Cython IMO, because you’re writing the bindings in a language that’s almost python.

I guess it’s mismatched expectations. IMO the primary use-case is making python wrappers for C/C++ code or converting prototype Python code into a fast C extension for production use by adding types or progressively converting code into a language that is a mix of Python and C.

Cython 3 also uses python type annotaions to infer C types now too, so if you add python types you might see surprising cython speedups.