Welcome to Pathway Developer Documentation!

Here, you will get to know the open Pathway framework for data transformation, start to develop code with Pathway, learn to perform rapid prototyping and production deployment, and discover solution templates for various use cases.

In the first place, Pathway was designed to be a life-saver (or at least a time-saver) for Python developers and ML/AI engineers faced with live data sources, where you need to react quickly to fresh data. Still, Pathway is a powerful tool that can be used for a lot of things. If you want to do streaming in Python, build an AI data pipeline, or if you are looking for your next Python data processing framework, keep reading.

Pathway is powered by a scalable Rust engine which is unified across batch and streaming and performs incremental computation. Tasks it can be used for include: Event Stream Processing, real-time Machine Learning, LLM-powered streaming pipelines, Retrieval Augmented Generation and document preparation for LLM, time-series analysis, and graph algorithms.

Pathway allows you to easily develop data transformation pipelines and Machine Learning applications that work with live data sources and changing data. You can start by using a local Python installation, and work in your preferred IDE or notebooks. In this way, you can make data experiments with static data samples and try out data source connections. Once satisfied with the test implementation, the same pipeline can be used in production with live data.

Pathway's engine provides consistent outputs for both batch and streaming data. You can combine real-time and historical data in the same code logic, working with multiple data sources including API's, Kafka-like event brokers, databases, and files.

• Effortless Batch to Streaming Transition.
• Realtime analytics.
• LLM-powered data pipelines.
• Event-driven pipelines with alerting.
• Advanced data transformation.