xorq: Multi-engine ML pipelines made simple

xorq is a deferred computational framework that brings the replicability and performance of declarative pipelines to the Python ML ecosystem. It enables us to write pandas-style transformations that never run out of memory, automatically cache intermediate results, and seamlessly move between SQL engines and Python UDFs—all while maintaining replicability. xorq is built on top of Ibis and DataFusion.

Feature	Description
Declarative expressions	Express and execute complex data processing logic via declarative functions. Define transformations as Ibis expressions so that you are not tied to a specific execution engine.
Multi-engine	Create unified ML workflows that leverage the strengths of different data engines in a single pipeline. xorq orchestrates data movement between engines (e.g., Snowflake for initial extraction, DuckDB for transformations, and Python for ML model training).
Built-in caching	xorq automatically caches intermediate pipeline results, minimizing repeated work.
Serializable pipelines	All pipeline definitions, including UDFs, are serialized to YAML, enabling version control, reproducibility, and CI/CD integration. Ensures consistent results across environments and makes it easy to track changes over time.
Portable UDFs	Build pipelines as UDxFs- aggregates, windows, and transformations. The DataFusion-based xorq engine provides a portable runtime for UDF execution.
Arrow-native architecture	Built on the Apache Arrow columnar memory format and Arrow Flight transport layer, xorq achieves high-performance data transfer without cumbersome serialization overhead.

Getting Started

xorq functions as both an interactive library for building expressions and a command-line interface. This dual nature enables seamless transition from exploratory research to production-ready artifacts. The steps below will guide through using both the CLI and library components to get started.

[!CAUTION] This library does not currently have a stable release. Both the API and implementation are subject to change, and future updates may not be backward compatible.

Installation

xorq is available as xorq on PyPI:

pip install xorq

[!NOTE] We are changing the name from LETSQL to xorq.

Usage

# your_pipeline.py
import xorq as xo


pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

batting = pg.table("batting")
awards_players = xo.examples.awards_players.fetch(backend=db)

left = batting.filter(batting.yearID == 2015)

right = (awards_players.filter(awards_players.lgID == "NL")
                       .drop("yearID", "lgID")
                       .into_backend(pg, "filtered"))

expr = (left.join(right, ["playerID"], how="semi")
            .cache()
            .select(["yearID", "stint"]))

result = expr.execute()

xorq provides a CLI that enables you to build serialized artifacts from expressions, making your pipelines reproducible and deployable:

# Build an expression from a Python script
xorq build your_pipeline.py -e "expr" --target-dir builds

This will create a build artifact directory named by its expression hash:

builds
└── fce90c2d4bb8
   ├── abe2c934f4fe.sql
   ├── cec2eb9706bc.sql
   ├── deferred_reads.yaml
   ├── expr.yaml
   ├── metadata.json
   ├── profiles.yaml
   └── sql.yaml

The CLI converts Ibis expressions into serialized artifacts that capture the complete execution graph, ensuring consistent results across environments. More info can be found in the tutorial Building with xorq.

For more examples on how to use xorq, check the examples directory, note that in order to run some of the scripts in there, you need to install the library with examples extra:

pip install 'xorq[examples]'

Contributing

Contributions are welcome and highly appreciated. To get started, check out the contributing guidelines.

Acknowledgements

This project heavily relies on Ibis and DataFusion.

License

This repository is licensed under the Apache License