Documentation: Python - Rust - Node.js - R | StackOverflow: Python - Rust - Node.js - R | User Guide | Discord
## Polars: Blazingly fast DataFrames in Rust, Python, Node.js, R and SQL Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using [Apache Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html) as the memory model. - Lazy | eager execution - Multi-threaded - SIMD - Query optimization - Powerful expression API - Hybrid Streaming (larger than RAM datasets) - Rust | Python | NodeJS | R | ... To learn more, read the [User Guide](https://pola-rs.github.io/polars/). ## Python ```python >>> import polars as pl >>> df = pl.DataFrame( ... { ... "A": [1, 2, 3, 4, 5], ... "fruits": ["banana", "banana", "apple", "apple", "banana"], ... "B": [5, 4, 3, 2, 1], ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"], ... } ... ) # embarrassingly parallel execution & very expressive query language >>> df.sort("fruits").select( ... "fruits", ... "cars", ... pl.lit("fruits").alias("literal_string_fruits"), ... pl.col("B").filter(pl.col("cars") == "beetle").sum(), ... pl.col("A").filter(pl.col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), ... pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"), ... pl.col("A").reverse().over("fruits").alias("rev_A_by_fruits"), ... pl.col("A").sort_by("B").over("fruits").alias("sort_A_by_B_by_fruits"), ... ) shape: (5, 8) ┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐ │ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │ │ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │ │ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │ │ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡ │ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │ │ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │ │ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │ │ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │ │ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │ └──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘ ``` ## SQL ```python >>> # create a sql context >>> context = pl.SQLContext() >>> # register a table >>> table = pl.scan_ipc("file.arrow") >>> context.register("my_table", table) >>> # the query we want to run >>> query = """ ... SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM my_table ... WHERE id1 = 'id016' ... LIMIT 10 ... """ >>> ## OPTION 1 >>> # run query to materialization >>> context.query(query) shape: (1, 2) ┌────────┬────────┐ │ sum_v1 ┆ min_v2 │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════╪════════╡ │ 298268 ┆ 1 │ └────────┴────────┘ >>> ## OPTION 2 >>> # Don't materialize the query, but return as LazyFrame >>> # and continue in python >>> lf = context.execute(query) >>> (lf.join(other_table) ... .group_by("foo") ... .agg( ... pl.col("sum_v1").count() ... ).collect()) ``` SQL commands can also be ran directly from your terminal using the Polars CLI: ```bash # run an inline sql query > polars -c "SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10" # run interactively > polars Polars CLI v0.3.0 Type .help for help. > SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10; ``` Refer to the [Polars CLI repository](https://github.com/pola-rs/polars-cli) for more information. ## Performance 🚀🚀 ### Blazingly fast Polars is very fast. In fact, it is one of the best performing solutions available. See the results in [DuckDB's db-benchmark](https://duckdblabs.github.io/db-benchmark/). In the [TPCH benchmarks](https://www.pola.rs/benchmarks.html) polars is orders of magnitudes faster than pandas, dask, modin and vaex on full queries (including IO). ### Lightweight Polars is also very lightweight. It comes with zero required dependencies, and this shows in the import times: - polars: 70ms - numpy: 104ms - pandas: 520ms ### Handles larger than RAM data If you have data that does not fit into memory, polars lazy is able to process your query (or parts of your query) in a streaming fashion, this drastically reduces memory requirements so you might be able to process your 250GB dataset on your laptop. Collect with `collect(streaming=True)` to run the query streaming. (This might be a little slower, but it is still very fast!) ## Setup ### Python Install the latest polars version with: ```sh pip install polars ``` We also have a conda package (`conda install -c conda-forge polars`), however pip is the preferred way to install Polars. Install Polars with all optional dependencies. ```sh pip install 'polars[all]' pip install 'polars[numpy,pandas,pyarrow]' # install a subset of all optional dependencies ``` You can also install the dependencies directly. | Tag | Description | | ---------- | ---------------------------------------------------------------------------- | | **all** | Install all optional dependencies (all of the following) | | pandas | Install with Pandas for converting data to and from Pandas Dataframes/Series | | numpy | Install with numpy for converting data to and from numpy arrays | | pyarrow | Reading data formats using PyArrow | | fsspec | Support for reading from remote file systems | | connectorx | Support for reading from SQL databases | | xlsx2csv | Support for reading from Excel files | | openpyxl | Support for reading from Excel files with native types | | deltalake | Support for reading from Delta Lake Tables | | pyiceberg | Support for reading from Apache Iceberg tables | | timezone | Timezone support, only needed if are on Python<3.9 or you are on Windows | Releases happen quite often (weekly / every few days) at the moment, so updating polars regularly to get the latest bugfixes / features might not be a bad idea. ### Rust You can take latest release from `crates.io`, or if you want to use the latest features / performance improvements point to the `main` branch of this repo. ```toml polars = { git = "https://github.com/pola-rs/polars", rev = "
](https://www.jetbrains.com)