People typically start with Excel and graduate to Pandas somewhere in the GB range.
Most of the things I use Pandas for would easily fit in an Excel table but Pandas lets me iterate different analyses more quickly. A lot of Pandas users use it via Jupyter Notebooks, where the data is of a size that can fit in a clipboard for pasting into a cell.
I think the largest thing I’ve processed with Pandas was around 100 MiB (long run of an experiment that spat out a TSV as it ran).
The motivation for better performance simply isn’t there for any of these use cases. If you’re processing GiBs of data then you’re in a very different domain from, I would guess, most Pandas users.
So then the question I have for that domain is: where is the bottleneck? Is how Pandas manages data slow? Is NumPy slow? Is CPython adding overhead? If it’s the first, is this constrained by Pandas’ APIs or is it something that could be optimised without changing any consumers? If it’s the second, what happens if you switch in one of the GPU-offload NumPy variants? If it’s the last, what happens if you use a Python JIT instead? All of these can be explored without changing user code.
For people like me, who have been considering trying Polars for a while: is the Polars API better than Pandas? Is it more consistent? More concise? Less error-prone? Show me those things and you’ll convince me. Tell me that if I had 100x as much data as I have then Pandas would be slow and I’ll shrug.
Oh, pandas can certainly grow into a many gigs of data without any issue, it just doesn't usually start at that point.
As far as I understand, one of the larger issues limiting Pandas is that it is eager, whereas polars (and I guess DuckDB though it's kinda akward to apply this notion for it) are lazy - each call to pandas dataframe actually performs the operation, while with polars, each call just records the operation, and they are performed only when you want the resulting data. This means that in some cases, pandas will do unnecessary operations on more data than it actually needs for the end result because it doesn't get to know the full operation graph, while polars (and duckDB) does, and thus can optimize by for example not reading columns that aren't needed for the end result, doing everything in a streaming manner to conserve memory, etc.
Changing Pandas from eager to lazy would be a fairly significant work, and likely would have a lot of impact to the API.
I think 9/10 times I used Pandas for some serious work^TM I eventually hit memory and slowness issues, but I agree that Excel vs Pandas is much more about automation and iteration than data size.
I admit I never profiled it, but I think most of the time was spent loading data from disk to memory and copying data around in memory, and paging in and out once the machine ran out of memory.
At the point where you start downloading the dataset in order to compute, none of these single node alternatives are nice to work with anymore. Sending the query and getting back the results becomes more practical. Spark just happens to have a more mature story for that (and plenty of cloud offerings).
You’re not gaining much by switching from pandas to polars to scan a bucket with thousands small CSV files, no matter which side of the data size “cliff” you’re on.
I have high hopes for DuckDB and DuckLake bridging this gap, however. Even though I’d rather not review stringified SQL in Python ever again.
Even though I’d rather not review stringified SQL in Python ever again.
For a long time now, our department has put SQL code in .sql files and reading it in with app code. We chose this for security (to promote prepared statements over f-strings) but a side benefit was standard tooling applies to them, such as applying linters or reading the files directly with SQL clients. It also makes it much easier to search for SQL across projects in Github.
We haven't figured out a way to enforce it in CI, though.
You’re not gaining much by switching from pandas to polars to scan a bucket with thousands small CSV files, no matter which side of the data size “cliff” you’re on.
It seems to me like the post proved definitively that this is not true. Why make this claim in the face of this blog post?
I may have missed it l, but the CSV for the 1 billion row challenge is a single file, and the NYC data is monthly beginning in 2009 (far cry from thousands) and in a data format that’s already much friendlier for scanning (parquet).
david_chisnall | 19 hours ago
I think the problem with this is the framing:
Most of the things I use Pandas for would easily fit in an Excel table but Pandas lets me iterate different analyses more quickly. A lot of Pandas users use it via Jupyter Notebooks, where the data is of a size that can fit in a clipboard for pasting into a cell.
I think the largest thing I’ve processed with Pandas was around 100 MiB (long run of an experiment that spat out a TSV as it ran).
The motivation for better performance simply isn’t there for any of these use cases. If you’re processing GiBs of data then you’re in a very different domain from, I would guess, most Pandas users.
So then the question I have for that domain is: where is the bottleneck? Is how Pandas manages data slow? Is NumPy slow? Is CPython adding overhead? If it’s the first, is this constrained by Pandas’ APIs or is it something that could be optimised without changing any consumers? If it’s the second, what happens if you switch in one of the GPU-offload NumPy variants? If it’s the last, what happens if you use a Python JIT instead? All of these can be explored without changing user code.
For people like me, who have been considering trying Polars for a while: is the Polars API better than Pandas? Is it more consistent? More concise? Less error-prone? Show me those things and you’ll convince me. Tell me that if I had 100x as much data as I have then Pandas would be slow and I’ll shrug.
ignaloidas | 18 hours ago
Oh, pandas can certainly grow into a many gigs of data without any issue, it just doesn't usually start at that point.
As far as I understand, one of the larger issues limiting Pandas is that it is eager, whereas polars (and I guess DuckDB though it's kinda akward to apply this notion for it) are lazy - each call to pandas dataframe actually performs the operation, while with polars, each call just records the operation, and they are performed only when you want the resulting data. This means that in some cases, pandas will do unnecessary operations on more data than it actually needs for the end result because it doesn't get to know the full operation graph, while polars (and duckDB) does, and thus can optimize by for example not reading columns that aren't needed for the end result, doing everything in a streaming manner to conserve memory, etc.
Changing Pandas from eager to lazy would be a fairly significant work, and likely would have a lot of impact to the API.
[OP] ohrv | 16 hours ago
I think 9/10 times I used Pandas for some serious work^TM I eventually hit memory and slowness issues, but I agree that Excel vs Pandas is much more about automation and iteration than data size.
I admit I never profiled it, but I think most of the time was spent loading data from disk to memory and copying data around in memory, and paging in and out once the machine ran out of memory.
stig | 18 hours ago
As a friend & colleague remarked years ago:
gcarvalho | 19 hours ago
At the point where you start downloading the dataset in order to compute, none of these single node alternatives are nice to work with anymore. Sending the query and getting back the results becomes more practical. Spark just happens to have a more mature story for that (and plenty of cloud offerings).
You’re not gaining much by switching from pandas to polars to scan a bucket with thousands small CSV files, no matter which side of the data size “cliff” you’re on.
I have high hopes for DuckDB and DuckLake bridging this gap, however. Even though I’d rather not review stringified SQL in Python ever again.
ryan-duve | 18 hours ago
For a long time now, our department has put SQL code in
.sqlfiles and reading it in with app code. We chose this for security (to promote prepared statements over f-strings) but a side benefit was standard tooling applies to them, such as applying linters or reading the files directly with SQL clients. It also makes it much easier to search for SQL across projects in Github.We haven't figured out a way to enforce it in CI, though.
dmfay | 8 hours ago
Standardize on aiosql and fail CI if anything imports psycopg2 or the like?
jrwren | 13 hours ago
It seems to me like the post proved definitively that this is not true. Why make this claim in the face of this blog post?
gcarvalho | 13 hours ago
I may have missed it l, but the CSV for the 1 billion row challenge is a single file, and the NYC data is monthly beginning in 2009 (far cry from thousands) and in a data format that’s already much friendlier for scanning (parquet).