Pathway Overview

Pathway is a data processing framework which takes care of streaming data updates for you. It makes realtime data processing as easy as it can be.

Installation

You can download the current Pathway release, which is now available in Open Beta on a free-to-use license:

                    
                        pip install --extra-index-url 
                        
                         pathway

on a Python 3.8+ installation, and we are ready to roll!

To use Pathway, we only need to import it:

import pathway as pw

Now, you can easily create your processing pipeline while letting Pathway handle the updates! Once your pipeline is created, you can launch the computation on streaming data by a one-line command:

pw.run()

This will launch the computation, each update in the input data streams automatically triggering the update of the whole pipeline! The computation will not stop, waiting for new updates, until the process is killed.

What is Pathway?

You wonder what might be the issues with streaming data and how Pathway resolves them? You can find all the answers and more in our introduction to Pathway.

Your first realtime app with Pathway

To get started with Pathway, follow our tutorial and build a simple your first realtime streaming app using Pathway. You will learn how easy it is to create a end-to-end realtime app using Pathway.

Key concepts

To use Pathway to its full potential, it is primordial to have a basic understanding on the underlying engine of Pathway. Our article on the key concepts of Pathway will provide you valuable insights on how Pathway works.

Streaming and static modes

Pathway is made for streaming data but also provides a static mode to facilitate debugging and testing. All you need to know about those two modes is presented in this article.

Survival guide

Must-read for both first-timers and veterans alike, the survival guide gathers the most commonly used basic elements of Pathway.

To go further

As we continue you will see some more advanced programming constructs which provide a lot of flexibility to Pathway:

  • Applying Machine Learning to data tables.
  • The ability to do iteration and recursion.

This, and a lot more, is covered in recipes in the Pathway cookbook - try these for a start:

Introduction

What is Pathway?