Pathway is a data processing framework which takes care of streaming data updates for you. It makes real time data processing as easy as it can be.
You can download the current Pathway release, which is now available on a free-to-use license:
pip install -U pathway
on a Python 3.10+ installation, and you are ready to roll!
⚠️ Pathway is available on MacOS and Linux. Pathway is currently not supported on Windows.
To use Pathway, you only need to import it:
import pathway as pw
Now, you can easily create your processing pipeline while letting Pathway handle the updates. Once your pipeline is created, you can launch the computation on streaming data by a one-line command:
This will launch the computation. Each update in the input data streams will automatically trigger the update of the whole pipeline. The computation will not stop, and will poll for new updates until the process is terminated.
To jumpstart a Pathway project, quickly check our cookiecutter template.
Learn about the main challenges when working with streaming data and how Pathway resolves them.
To get started with Pathway, follow our first realtime streaming app tutorial to build your end-to-end realtime app using Pathway.
Must-read for both first-timers and veterans alike, the basic operations guide describes the elements of Pathway you are most likely to use from day one.
Pathway is made for streaming data but also provides a static mode to facilitate debugging and testing. For these purposes, you can use the streaming and static modes of Pathway.
To use Pathway to its full potential, let's get a basic understanding of the data concepts behind Pathway, describing how Pathway represents and transforms data.
As you continue you will see some more advanced programming constructs which provide a lot of flexibility to Pathway, for example:
- How to integrate multiple live and static data sources?
- How to apply Machine Learning to data tables?
This, and a lot more, is covered in recipes in Pathway tutorials. You can try these for a start: