Published on: 31st October 2022
This post was created while writing my Data Analysis with Polars course. Check it out on Udemy
Polars has a built in tool to go on a dtype diet.
Call the shrink_dtype expression and it will convert the column to the dtype that requires the least amount of memory based on the data in the column.
import polars as pl
(
pl.DataFrame(
{
"a": [1, 2, 3],
"b": [1, 2, 2 << 32],
"c": [-1, 2, 1 << 30],
"d": [-112, 2, 112],
"e": [-112, 2, 129],
"f": ["a", "b", "c"],
"g": [0.1, 1.32, 0.12],
"h": [True, None, False],
}
)
.select(
pl.all().shrink_dtype()
)
)
┌─────┬────────────┬────────────┬──────┬──────┬─────┬──────┬───────┐
│ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ┆ h │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i8 ┆ i64 ┆ i32 ┆ i8 ┆ i16 ┆ str ┆ f32 ┆ bool │
╞═════╪════════════╪════════════╪══════╪══════╪═════╪══════╪═══════╡
│ 1 ┆ 1 ┆ -1 ┆ -112 ┆ -112 ┆ a ┆ 0.1 ┆ true │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
│ 2 ┆ 2 ┆ 2 ┆ 2 ┆ 2 ┆ b ┆ 1.32 ┆ null │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
│ 3 ┆ 8589934592 ┆ 1073741824 ┆ 112 ┆ 129 ┆ c ┆ 0.12 ┆ false │
└─────┴────────────┴────────────┴──────┴──────┴─────┴──────┴───────┘
Both floats and integers default to 64-bit precision. In the example below from the API docs Polars sees that column “a” could be 8-bit, column “b” must be 64-bit, but column “c” could be 32-bit.
Casting numeric columns from 64-bit to 32-bit is often the easiest win in data science. Memory usage halves and computation time might also be half that of 64-bit.
You do need to check that the loss of precision is ok. I had sensors accurate to 0.01 so a change of 10^-6 was 👍
In my udemy course I show that if you cast to 8- or 16-bits memory usage continues to fall proportionally…
…but computation time probably won’t be better than 32-bits!
Most modern CPUs don’t have native support for 8- or 16-bit so they have to emulate it.
String columns with lots of repeated entries can also usefully be cast to categoricals. But that’s a story for another day.
Want to know more about Polars for high performance data science and ML? Then you can:
or let me know if you would like a Polars workshop for your organisation.