postgres_to_polars

A Rust library for streaming PostgreSQL query results directly into Polars DataFrames using sqlx.

Derive IntoDataFrame on your struct, use sqlx's query_as! macro, and call .to_dataframe() on the stream. That's it.

Usage

Add to your Cargo.toml:

[dependencies]
postgres_to_polars = "1.0"
sqlx = { version = "0.8", features = ["runtime-tokio", "postgres", "chrono"] }
tokio = { version = "1", features = ["full"] }
polars = "0.53"

Basic example

use sqlx::PgPool;
use postgres_to_polars::{IntoDataFrame, StreamToDataFrame};

#[derive(sqlx::FromRow, IntoDataFrame)]
struct User {
    id: i32,
    name: Option<String>,
    email: Option<String>,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pool = PgPool::connect("postgres://user:pass@localhost/db").await?;

    // With estimated row count (avoids reallocations)
    let df = sqlx::query_as!(User, "SELECT id, name, email FROM users")
        .fetch(&pool)
        .to_dataframe(100_000)
        .await?;

    println!("{}", df);
    Ok(())
}

With default capacity

When you don't know the row count upfront:

let df = sqlx::query_as!(User, "SELECT id, name, email FROM users WHERE active = true")
    .fetch(&pool)
    .to_dataframe_default()
    .await?;

Supported types

All types support Option<T> for nullable columns.

How it works

1. #[derive(IntoDataFrame)] generates a columnar builder for your struct
2. .to_dataframe(capacity) streams rows from PostgreSQL and pushes each row into the builder (no intermediate Vec<Struct>)
3. The builder converts each column Vec into a Polars Series and assembles the DataFrame

This approach is memory-efficient: rows are consumed one at a time from the stream and stored directly in columnar format.

Performance

Benchmarks on 500K rows (PostgreSQL running locally in Docker):

License

Apache-2.0

lib.rs:

Proc-macro crate providing #[derive(IntoDataFrame)].

This is an implementation detail of the postgres_to_polars crate. Use postgres_to_polars::IntoDataFrame instead of depending on this crate directly.