for developers

Getting Started

Pathway is a data processing framework which takes care of streaming data updates for you. It makes real time data processing as easy as it can be.

Installation

You can download the current Pathway release, which is now available on a free-to-use license:

            pip install -U pathway

on a Python 3.10+ installation, and you are ready to roll!

⚠️ Pathway is available on MacOS and Linux. Pathway is currently not supported on Windows.

You can also try these steps in an online notebook environment like Colab or Noteable.

To use Pathway, you only need to import it:

import pathway as pw

Now, you can easily create your processing pipeline while letting Pathway handle the updates. Once your pipeline is created, you can launch the computation on streaming data by a one-line command:

pw.run()

This will launch the computation. Each update in the input data streams will automatically trigger the update of the whole pipeline. The computation will not stop, and will poll for new updates until the process is terminated.

To jumpstart a Pathway project, quickly check our cookiecutter template.

Dealing with streaming data

Learn about the main challenges when working with streaming data and how Pathway resolves them.

Your first realtime app

To get started with Pathway, follow our first realtime streaming app tutorial to build your end-to-end realtime app using Pathway.

Basic operations

Must-read for both first-timers and veterans alike, the basic operations guide describes the elements of Pathway you are most likely to use from day one.

Streaming and static modes

Pathway is made for streaming data but also provides a static mode to facilitate debugging and testing. For these purposes, you can use the streaming and static modes of Pathway.

Data representation concepts

To use Pathway to its full potential, let's get a basic understanding of the data concepts behind Pathway, describing how Pathway represents and transforms data.

To go further

As you continue you will see some more advanced programming constructs which provide a lot of flexibility to Pathway, for example:

  • How to integrate multiple live and static data sources?
  • How to apply Machine Learning to data tables?

This, and a lot more, is covered in recipes in Pathway tutorials. You can try these for a start:

Introduction

Dealing with streaming data