Farewell Pandas, and thanks for all the fish

https://ibis-project.org/posts/farewell-pandas/

by nojito • 2 years ago

228 138 2 years ago

One thing I do like about pandas is it’s pretty extensible to columns of new types. Maybe I’m missing something, but does Polars allow this? Last time I checked there wasn’t a clear path forward.

datadrivenangel3• 2 years ago

Ibis allows this with custom datatypes.

porker• 2 years ago

I'm curious, when do you want to add a new column type? It's not a situation I've been in.

softwaredoug2• 2 years ago

GeoPandas is one popular library (https://geopandas.org/en/stable/)

I added a column type for full text search, searching tokenized text (https://github.com/softwaredoug/searcharray)

The pandas support for extensions is very robust with extensive testing to make sure you can do all the data aggregations functionality you expect in and pandas column.

deadlypointer• 2 years ago

OTOH the list and struct types allow lots of complex operations, e.g with `.eval`, which run blazingly fast.

maxdo• 2 years ago

curious to know what they use instead of Dask ?

datadrivenangel3• 2 years ago

DuckDB. It's faster on a single computer than Dask.

codyvoda9• 2 years ago

Ibis still supports over a dozen backends, including DuckDB and DataFusion and Polars (all great single-node OLAP query engine options that are often faster than trying something distributed) and PySpark, Snowflake, BigQuery, Trino, ClickHouse, Druid, and more

plenty of options for fast and scalable computation. in the fullness of time, it would be great to have a Dask backend but it'd need to be rewritten to use dask-expr, which isn't something the core Ibis team has the capacity for

lopatin2• 2 years ago

As a potential user of Ibis, how do I know they won't eventually drop support for other backends until it just becomes a frontend for DuckDB?

2-3-7-43-18073• 2 years ago

even vanilla is faster than dask on a single computer ... _OFFICIALLY_ ... cause dask is and only is for clusters

mistrial9• 2 years ago

can you expand or give more context on this ?

spywaregorilla• 2 years ago

What is this exactly?

The front page doesn't make it clear to make what ibis is besides being an alternative to pandas/polars.

prepend• 2 years ago

Check the project’s left nav, https://ibis-project.org/why

I didn’t know either. It’s a python dataframe library/api.

smu3l• 2 years ago

Never heard of ibis before but the front page give a pretty good overview of what it is to me at least. Looks like a dataframes api that can be executed on a variety of backend engines, both distributed and local.

Somewhat analogous to how the pandas api can be used in pyspark via pyspark pandas, but from the api -> implementation direction rather than (pandas) implementation -> api -> implementation maybe?

giovannibonetti• 2 years ago

As far as I can tell, it is an ORM for data engineers. They can write Python code that gets translated to either SQL or some other language understood by the DB engine that actually runs it.

cpcloud22• 2 years ago

Ibis is not an ORM. It has no opinion of or knowledge about your data model, or relationships between tables and their columns.

Check out SQLAlchemy's ORM (https://docs.sqlalchemy.org/en/20/orm/) for what that looks like.

joelschw• 2 years ago

Huge fan of Ibis, the value isn't that you can now use DuckDB... it's that your syntax will work when the next cool thing arrives too

infecto• 2 years ago

I will try it out but honestly the pain-point for me is the library api to pandas is not always intuitive/natural to how things should be in Python. The NaN/None bit is annoying but I find that to be a minor annoyance.

mananaysiempre3• 2 years ago

NULL indicates a missing value, and NaN is Not a Number.

That’s actually less true than it sounds. One of the primary functions of NaN is to be the result of 0/0, so there it means that there could be a value but we don’t know what it is because we didn’t take the limit properly. One of the primary functions of NULL is to say that a tuple satisfies a predicate except we don’t know what this one position is—it’s certainly is something out in the real world, we just don’t know what. These ideas are what motivates the comparison shenanigans both NaN and NULL are known for.

There’s certainly an argument to be made that the actual implementation of both of these ideas is half-baked in the respective standards, and that they are half-baked differently so we shouldn’t confuse them. But I don’t think it’s fair to say that they are just completely unrelated. If anything, it’s Python’s None that’s doesn’t belong.

cpcloud22• 2 years ago

100% this.

We'd love to have a docs contribution that lays this out in detail if you'd be up for it!

Disclaimer: (I lead the project and work on it full time).

aeonik• 2 years ago

Why don't we build a portable numeric system that just has these mathematical constants and definitions built in, in a portable and performance manner?

verandaguy• 2 years ago

We just don’t have the technology for that, else there’d be libraries floating around that let you do that.

miohtama2• 2 years ago

Related to this, below someone posted a link to the blog post of Wes McKinney where he discussed Pandas limitations and how PyArrow works around these

4. Doing missing data right All missing data in Arrow is represented as a packed bit array, separate from the rest of the data. This makes missing data handling simple and consistent across all data types. You can also do analytics on the null bits (AND-ing bitmaps, or counting set bits) using fast bit-wise built-in hardware operators and SIMD.
The null count in an array is also explicitly stored in its metadata, so if data does not have nulls, we can choose faster code paths that skip null checking. With pandas, we cannot assume that arrays do not have null sentinel values and so most analytics has extra null checking which hurts performance. If you have no nulls, you don’t even need to allocate the bit array.
Because missing data is not natively supported in NumPy, over time we have had to implement our own null-friendly versions of most key performance-critical algorithms. It would be better to have null-handling built into all algorithms and memory management from the ground up.

https://wesmckinney.com/blog/apache-arrow-pandas-internals/

jononor2• 2 years ago

Note that the post is from 2017, and pandas now has (optional) support for PyArrow backed dataframes. So there is movement away from the critiques that were presented there.

OptionOfT• 2 years ago

One of the primary functions of NULL is to say that a tuple satisfies a predicate except we don’t know what this one position is—it’s certainly is something out in the real world, we just don’t know what.

I'm not sure I understand this. Can you explain it with an example?

I'm thinking something like:

    let (x, y) = NULL;

But then I'm stuck.

kstrauser2• 2 years ago

In the broader DB theory context and not strictly Pandas, think of a `users` table with a `middle_name` column. There's a difference between `middle_name is null` (we don't know whether they have a middle name or what it is if they do) and `middle_name = ''` (we know that they don't have a middle name).

In that case, `select * from users where middle_name is null` gives us a set of users to prod for missing information next time we talk to them. `...where middle_name = ''` gives us a list of people without a middle name, should we care.

Edit:

A related aside is that Django's ORM's handling of NULL gives me freaking hives. From https://docs.djangoproject.com/en/5.1/ref/models/fields/#nul...:

If a string-based field has null=True, that means it has two possible values for “no data”: NULL, and the empty string. In most cases, it’s redundant to have two possible values for “no data;” the Django convention is to use the empty string, not NULL.

I've been around numerous Django devs when they found out that other DBs and ORMs do not remotely consider NULL and empty string to be the same thing.

mananaysiempre3• 2 years ago

What I mean is, the ivory-tower relational model is that each table (“relation”) represents a predicate (Boolean-valued function) accepting as many arguments as there are columns (“attributes”), and the rows (“tuples”) of the table are an exhaustive listing of those combinations of arguments for which the predicate yields true (“holds”). E.g. the relation Lived may contain a tuple (Edgar Codd, 1923-08-19, 2003-04-18) to represent the fact that Franklin was born on 19 August 1923 and died on 18 April 2003.

One well-established approach[1] to NULLs is that they represent the above “closed-world assumption” of exhaustiveness to encompass values we don’t know. For example, the same relation could also contain the tuple (Leslie Lamport, 1941-02-07, NULL) to represent that Lamport was born on 7 February 1941 and lives to the present day.

We could then try to have some sort of three-valued logic to propagate this notion of uncertainty: define NULL = x to be (neither true nor false but) NULL (“don’t know”), and then any Boolean operation involving NULL to also yield NULL; that’s more or less what SQL does. As far as I know, it’s possible to make this consistent, but it’ll always be weaker than necessary: a complete solution would instead assign a variable to each unknown value of this sort and recognize that those values are, at the very least, equal to themselves, but dealing with this is NP-complete, as it essentially amounts to implementing Prolog.

[1] http://www.esp.org/foundations/database-theory/holdings/codd...

ironmagma2• 2 years ago

Man, these frontend people are constantly churning through dependencies, a new framework every year. Don't reinvent the wheel, just be like backend folks and pick a dependency and stick with it. Oh wait...

vundercind2• 2 years ago

ETL and data analysis “notebook” world been warming up for years and now it’s red-hot due to every company down to Grandma’s Cookie Bakery LLC wanting to “do AI”. And most of the standard tools are surprisingly terrible so churn is increasing for reasons similar to why that happened in frontend (the underlying tools are ass and the existing interfaces bad and almost all of it’s being overcomplicated for bad reasons and a million more developers just jumped into the pool)

ironmagma2• 2 years ago

Yes, and so really what we are seeing is just the effects of a high quality bar. If you want high-quality software, you do more iterations. And iterations produce breaking changes.

dr_mee63• 2 years ago

Just as an FYI Ibis is being around for 10 years approximately...

frakt0x90• 2 years ago

Curious if you considered Polars. That's become the defacto standard in my group as we all dislike pandas.

cpcloud22• 2 years ago

We support Polars as a backend.

DuckDB was chosen as the default because Polars was too unstable to be the default backend two years ago.

amelius• 2 years ago

Good riddance.

Pandas turns 10x programmers into 1x programmers.

data-ottawa4• 2 years ago

I've been using pandas for over a decade and I'm very efficient with it.

The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it.

I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style.

It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing.

I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.

__mharrison__5• 2 years ago

Curious if you could expand on this?

Kalanos2• 2 years ago

Lol. As if different tools for filtering rows/columns determines a programmer's capability.

fifilura4• 2 years ago

What is the difference, other than parallel execution?

magnio2• 2 years ago

- Much faster and more memory efficient.

- Consistent Expression and SQL-like API.

- Lazy execution mode, where queries are compiled and optimized before running.

- Sane NaN and null handling.

- Much faster.

- Much more memory efficient.

codyvoda9• 2 years ago

Polars is a great choice but, like pandas, locks you into its execution engine(s). Polars is a supported backend for Ibis, though depending on your use case DuckDB or DataFusion may scale better. we did some benchmarking recently: https://ibis-project.org/posts/1tbc/

marcogorelli• 2 years ago

Also depending on your use case, Ibis may not support the part of the Polars API you actually need

(to be clear, I'm a fan of "DuckDB via Ibis", I'm much less a fan of statements to the effect of "use Polars via Ibis instead of Polars directly", as I think they miss a lot :wink:)

ang_cire2• 2 years ago

I wouldn't switch to Polars as a replacement to Pandas, given that Polars also seems likely to die soon. I'd also recommend avoiding Manatees.

cpcloud22• 2 years ago

On the flip side, Ibises (Ibisii ??) have been around for years and aren't going anywhere.

highfrequency2• 2 years ago

Why do you think polars is likely to die soon?

mrbluecoat• 2 years ago

Dask was really cool back in the day. Farewell, friend.

lmc• 2 years ago

Back in the day? What are people using now instead?

ZeroCool2u• 2 years ago

Ray seems to be a popular choice now if you need multi node. But Polars is really nice if you're okay with just vertically scaling on one machine.

cpcloud22• 2 years ago

I've been working with pencil and paper recently. Thinking about starting an Ibis backend for it.

dr_mee63• 2 years ago

Abacuses?

thsgtu• 2 years ago

About time. It always surprises me how long pandas has been able to hold on. Wes McKinney talked about pandas' limitations back in 2017 https://wesmckinney.com/blog/apache-arrow-pandas-internals/

__mharrison__5• 2 years ago

Not surprising. There are much better compute engines than pandas.

Folks then ask why not jump from pandas to [insert favorite tool]?

- Existing codebases. Lots of legacy pandas floating about.

- Third party integration. Everyone supports pandas. Lots of libraries work with tools like Polars, but everything works with pandas.

- YAGNI - For lots of small data tasks, pandas is perfectly fine.

cpcloud22• 2 years ago

All great reasons to stick with Pandas. If a thing works for you, then that's one less thing to manage.

Fully agree that if pandas is working for you, then you're better off sticking with it!

whimsicalism3• 2 years ago

If your task is too big for pandas, you should probably skip right over dask and polars for a better compute engine.

__mharrison__5• 2 years ago

Jump straight to cluster and skip Dask?

Not sure what "big" means here, but a combination of .pipe, pyarrow, and polars can speed up many slow Pandas operations.

Polars streaming is surprisingly good for larger than RAM. I get that clusters are cool, but I prefer to keep it on a single machine if possible.

Also, libraries like cudf can greatly speed up Pandas code on a single machine, while Snowpark can scale Pandas code to Snowflake scale.

0cf8612b2e1e• 2 years ago

Even medium tasks, the slow step is rarely computations, but the squishy human entering them.

When I work on a decent 50GB+ dataset, I have to do something fairly naive before I get frustrated at computation time.

Edit: pandas is now Boring Technology (not a criticism). It is a solid default choice. In contrast, we are still in a Cambrian explosion of NeoPandas wannabes. I have no idea who will win, but there is a lot of fragmentation which makes it difficult me for to jump head first into one of these alternatives. Most of them are faster or less RAM pressure which is really low on my list of problems.

halfcat4• 2 years ago

YAGNI - For lots of small data tasks, pandas is perfectly fine.

There’s a real possibility the growth of single node RAM capacity will perpetually outpace the growth of a business’s data. AWS has machines with 32 TB RAM now.

It’s a real question as to whether big data systems become more accessible before single node machines become massive enough to consume almost every data set.

(Also love your work, thank you)

__mharrison__5• 2 years ago

But... but we have to have big data and a cluster...

The hoops folks will jump through...

(Thanks!)

mrguyorama• 2 years ago

Pandas made me think I hated python.

e10v_me• 2 years ago

Funny, the same) When I was switching from R (data.table) to Python, it was painful. Not only because it was slow, but because of the API. At that time, I thought that maybe it's because of switching to something new. Several years later, switching from Pandas to Polars API was a real joy (Ibis is also good is that sense). So, I learned that it had been Pandas fault all along))

whimsicalism3• 2 years ago

Frankly, pandas/dask/polars - all are trying to recreate something that has existed for years (ie. sql, spark) and with a terrible API.

I truly thought I was terrible at python for a long time because of pandas - turns out it just has an absolutely terrible API surface

tech_ken• 2 years ago

I would not say pandas is a recreation of SQL/Spark, they have very different use-cases in my experience. SQL/Spark is like a bulk data management tool: I use it if I need to load massive data from some remote store, perform light preprocessing, join up a couple dimension tables, etc. Having normalized and joined my data, then pandas enters as a 'last-mile' processing engine, particularly when paired with ex. SKLearn or whatever other inference lib you're using. Pandas is awesome if you need to ex. apply string manipulation to a data table or daisy-chain some complicated computations together. Honestly in my opinion the API is really nice, I came over from R tidyverse and the 'chained methods' approach to pandas let's me carry over all my old patterns and paradigms. I find it far easier to use that approach than having to write like 20 dependent subqueries or staging tables

cmollis• 2 years ago

true spark has existed for years and is a great toolset.. i use it ever day. it's also a huge hassle spinning clusters up and down and configuration is complex.

I can execute some pretty hairy scans against a huge s3 parquet dataset in Duckdb that I would typically have to run in either spark or athena.. it's a little slower, but not ridiculously slower. And, it does all of that from my desktop.. no clusters, no mem or task configs.. just run the query. Being able to integrate all of the expensive historical scanning and knitting that back into an ML pipeline with desktop python is pretty nice.

dammaj• 2 years ago

Personnally, I tend to use Pandas because it is integrated everywhere, because of the ecosystem that uses it. Let's say I want to read data from json file (csv file, python dict, etc.) and I want to plot it using plotly. If Ibis is compatible with whatever Pands dataframe is compatible with, then for most of my usage I don't really care much about the "backend".

kremi3• 2 years ago

Pandas has been working fine for me. The most powerful feature that makes me stick to it is the multi-index (hierarchical indexes) [1]. Can be used for columns too. Not sure how the cool new kids like polars or ibis would fare in that category.

[1] https://pandas.pydata.org/docs/user_guide/advanced.html#adva...

cpcloud22• 2 years ago

Multi-indexes definitely have their place. In fact, I got involved in pandas development in 2013 as part of some work I was doing in graduate school, and I was a heavy user of multi-indexed columns. I loved them.

Over time, and after working on a variety of use cases, I personally have come to believe the baggage introduced by these data structures wasn't worth it. Take a look at the indexing code in pandas, and the staggering complexity of what's possible to put inside square brackets and how to decipher its meaning. The maintenance cost alone is quite high.

We don't plan to ever support multi-indexed rows or columns in Ibis. I don't think we'd fare well _at all_ there, intentionally so.

kremi3• 2 years ago

Take a look at the indexing code in pandas

As the end-user, not quite my concern.

and the staggering complexity of what's possible to put inside square brackets and how to decipher its meaning

I might not be aware of everything that's possible -- the usage I have of it doesn't give me an impression of staggering complexity. In fact I've found the functionality quite basic, and have been using pd.MultiIndex.from_* quite extensively for anything slightly more advanced than selecting a bunch of values at some level of the index.

highfrequency2• 2 years ago

Is your workload mostly single-threaded? If so, is that due to dataset size, or machine core count?

kremi3• 2 years ago

Sorry I don't know what to answer. I don't think what I do qualifies as "workload".

I have a process that generates lots of data. I put it in a huge multi-indexed dataframe that luckily fits in RAM. I then slice out the part I need and pass it on to some computation (at which point the data usually becomes a numpy array or a torch tensor). Core-count is not really a concern as there's not much going on other than slicing in memory.

The main gain I get of this approach is prototyping velocity and flexibility. Certainly sub-optimal in terms of performance.

jononor2• 2 years ago

I work a lot with IoT data, where basically everything is multi-variate time-series from multiple devices (at different physical locations and logical groupings). Pandas multi index is very nice for this, at least having time+space in the index.

UncleOxidant• 2 years ago

Related to this pandas?: https://pandas.pydata.org/

cpcloud22• 2 years ago

On the contrary, very related to that pandas!

Kalanos2• 2 years ago

I've only heard about Ibis maybe three times in the past two years and I pay pretty close attention to the space. If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge.

Sure, hip new frameworks are moving away from pandas/numpy, but I'll wait 5 years for the dust to settle here while the compatibility and edge cases sort themselves out. The pydata/numfocus ecosystem is extensive.

It's just tabular data. So what if I have to wait a few more milliseconds to get my result.

cpcloud22• 2 years ago

Sounds like a solid plan. No reason to incur switching costs if things are working!

I think you're misunderstanding what "removing pandas" means.

You can still compute on DataFrames, it'll just be with something other than Pandas itself, probably DuckDB, Polars, or DataFusion.

So, the bridge was there, is there, and isn't going anywhere.

halfcat4• 2 years ago

I think you're misunderstanding what "removing pandas" means.

Maybe it’s in the messaging, like the “Farewell Pandas” part?

codyvoda9• 2 years ago

If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge.

the bridge is that Ibis accepts pandas as input and has a `to_pandas()` method as output

Ibis also still depends on pandas (and thus numpy) internally. also, Ibis was created by the creator of pandas, and the lead developer (and other contributors) have also worked on pandas. the Ibis team understands pandas and the broader Python data ecosystem very well

It's just tabular data. So what if I have to wait a few more milliseconds to get my result.

usually as scale grows, milliseconds -> seconds -> minutes -> hours -> days. at some point along the way having a dataframe library that can scale up without rewriting your code might be useful. but if you're dealing with small tabular data and pandas meets your needs, it's a great library to use and stick with!

riezebos• 2 years ago

Nice to see, over the past months I've replaces pandas with ibis in all new projects and I am a huge fan!

- Syntax in general feels more fluid than pandas

- Chaining operations with deferred expressions makes code snippets very portable

- Duckdb backend is super fast

- Community is very active, friendly and responsive

I'm trying to promote it to all my peers but it's not a very well known project in my circles. (Unlike Polars which seems to be the subject of 10% of the talks at all Python conferences)

glial• 2 years ago

In my experience, the best thing about Pandas is how much it made me appreciate using dplyr and the tidyverse. If it wasn't for Pandas, I may not be the avid R user I am today.

antman2• 2 years ago

"TL; DR: we are deprecating the pandas and dask backends and will be removing them in version 10.0.

There is no feature gap between the pandas backend and our default DuckDB backend"

There is an important feature gap, pandas has all the users and no one knows you guys. Was trying to pitch this to my company but I guess its goodbye Ibis and thanks for all the fish.

codyvoda9• 2 years ago

were you using the pandas backend in Ibis and pitching that to your company? you'd have much better performance and efficiency using the default backend, even if you used pandas for data input and output (though the DuckDB data readers are generally more efficient too). I'd be happy to help you on the pitch!

and if you were using the pandas backend or know of many users doing so, please reach out! we love feedback from our users and would be sad to see y'all go, we're open to reconsidering any decisions based on feedback

antman2• 2 years ago

Yes I agree and have tested it, my point is that marketing wise it is a shame. I will even think that when looking into new tooling "pandae" compatibility as a search term and incentive would be far ahead the word "dataframe"

zhangyt26• 2 years ago

How does ibis manage ordering of the data frame for SQL based engine?

One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.

codyvoda9• 2 years ago

this is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data

so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up

rldjbpin• 2 years ago

you could completely respect the decision to deprecate some support, but the marketing speak comes across distasteful.

from a user's perspective, selecting a tool should be about what's "best" for you and not what is the "best" out there (from some arbitrary measure nonetheless).

pandas made sense and might still make sense for many, especially who are migrating from R and are already comfortable with dataframe paradigm. using the same paradigm and just saying "we run faster" does not solve the ergonomics or the reason why people adopted this way of processing data.

there are plenty players in the space going about this from the stack. would be nice to see more discussion on alternative generic data processing paradigms instead.

codyvoda9• 2 years ago

this is a blog for the Ibis project, directly from one of the main developers/maintainers of the project, about deprecating two (of 20+) backends the project supports and giving the justification

as a project, we're big on using the right tool for the job -- of course we often think that's Ibis. a lot of effort has been put into the ergonomics, taking learnings from pandas and dplyr and SQL and various other Python data projects the team has been instrumental in. if you weren't aware, the creator of pandas also created Ibis and the current lead developer (Phillip Cloud) was also very involved in pandas. we have plenty of other blogs and things on the documentation about the project's ergonomics and thoughts on data processing paradigms

this specific blog isn't intended as "marketing speak" or to somehow be distasteful, it's intended to inform the Ibis community that we're deprecating a backend and justifying why

cpcloud22• 2 years ago

Pandas support as input and output isn't going anywhere. People can continue to bring their pandas DataFrames to Ibis and run Ibis expressions against them.

pandas made sense and might still make sense for many, especially who are migrating from R and are already comfortable with dataframe paradigm.

Yes, and nothing is different about the paradigm after the changes outlined in the blog post, it's still a dataframe paradigm.

using the same paradigm and just saying "we run faster" does not solve the ergonomics or the reason why people adopted this way of processing data.

This seems to indicate you _do_ understand that the paradigm isn't changing but then does a complete 180 into complaining about Ibis not solving pandas ergonomics issues. Which is it?

log4shell• 2 years ago

Its great to have a single entrypoint for multiple backends. What I am trying to understand and couldn't find much information related to: How does the use of multiple engines in Ibis impact the consistency of results for the same input and query, particularly in relation to semantic differences among the engines?

Farewell Pandas, and thanks for all the fish

Related Stories