Welcome to Pathway Live Data Framework Developer Documentation!

The Pathway Live Data Framework is a Python data processing framework for analytics and AI pipelines over data streams. It's the ideal solution for real-time processing use cases like streaming ETL or RAG pipelines for unstructured data.

Getting Started

Install Pathway using pip:

pip install pathway

Starting examples

Try Our Templates

Pathway offers ready-to-go templates for RAG and ETL pipelines. Run Pathway on your data in minutes.

Pathway Live Data Framework Templates

Key Features:

Easy-to-use Python API: Pathway Live Data Framework is fully compatible with Python. Use your favorite Python tools and ML libraries.
Scalable Rust engine: your Python code is run by a powerful Rust engine with multithreading and multiprocessing. No JVM and no GIL!
Stateful operations: use stateful and temporal operations such as groupby and windows.
Incremental computations: using Differential Dataflow, Pathway Live Data Framework takes care of out-of-order data points for you, in real time.
Batch and streaming alike: use the same pipeline on static data and live data streams.
In-memory data processing: real-time updates, reduced latency, and higher throughput.
Easy to deploy with Docker or Kubernetes. The Pathway Live Data Framework comes with an orchestrator and is fully compatible with OpenTelemetry.
Exactly once consistency: obtain the same results in both batch and streaming.
Persistence and backfilling: save the state of the computation to quickly resume after a failure or a pipeline update.
LLM tooling: online ML, RAG pipelines, vector indexes... With Pathway Live Data Framework, your ML pipeline works on fresh data.
Connect to any data source: Pathway Live Data Framework comes with 350+ connectors, including SharePoint. Or implement your own.

Pathway Rust engine makes it fast, scalable, and safe.

What's next?

GitHub repository

The Pathway Live Data Framework sources are available on GitHub. Don't hesitate to clone the repo and contribute!

See the sources

License key

Some features of Pathway Live Data Framework such as monitoring or advanced connectors (e.g., SharePoint) require a free license key. To obtain a free license key, you need to register here.