In the following, you can find instructions on how to start using Pathway.
You can download the current Pathway release, which is now available in Open Beta on a free-to-use license:
pip install [Sign in] to obtain package link
on a Python 3.10 installation, and we are ready to roll!
To use Pathway, we only need to import it:
import pathway as pw
The first thing you need to do is to access the data you want to manipulate. Pathway provides many connectors to access your data and manipulate them.
As an example, let's load a table using a csv connector:
table_dogs = pw.debug.table_from_markdown( """ | name | age 1 | Ace | 8 2 | Bella | 5 3 | Coco | 13 """)
Now the table is loaded. But if we try to print it, we obtain a very generic output:
That's perfectly normal: as explained in our introduction to programming in Pathway, Pathway is used to schedule the operations that will be later performed in realtime by the runtime engine. To process the actual data in our example, we need to use debug function called
| name | age ^2TMTFGY... | Ace | 8 ^YHZBTNY... | Bella | 5 ^SERVYWW... | Coco | 13
Now that we have a table, we are going to do some basic operations on it. You can find the full list of the supported operations in our API documentation.
The first thing we may want, is to filter on the age and keep only the dogs younger than 10 years old.
We can use the operator
filter on the column 'age'.
To access a column, we can either use the notation
table_name.column_name or use the more generic
table_dogs_young = table_dogs.filter( table_dogs.age <= 10) # table_dogs['age'] also workspw.debug.compute_and_print(table_dogs_young)
| name | age ^2TMTFGY... | Ace | 8 ^YHZBTNY... | Bella | 5
We can also apply a function to a given column.
Let's say that we want to change the value of a column.
Due to an error in rounding, all the age values are wrong and should be decreased by one.
We can modify the table using the
table_dogs_corrected = table_dogs.select( table_dogs.name, age=pw.apply((lambda x: x - 1), table_dogs["age"]))pw.debug.compute_and_print(table_dogs_corrected)
| name | age ^2TMTFGY... | Ace | 7 ^YHZBTNY... | Bella | 4 ^SERVYWW... | Coco | 12
What happens here is that we select from
table_dogs the column 'name' and a column 'age' which is obtained by the operator
f to each entry in
col (there may be several such columns).
To do more complicated operations, we may need a second table:
table_dogs_owners = pw.debug.table_from_markdown( """ | name | owner 1 | Ace | Alice 2 | Bella | Bob 3 | Coco | Alice """)pw.debug.compute_and_print(table_dogs_owners)
| name | owner ^2TMTFGY... | Ace | Alice ^YHZBTNY... | Bella | Bob ^SERVYWW... | Coco | Alice
We can build a table with both information using the operator
table_dogs_full = table_dogs_corrected.join( table_dogs_owners, table_dogs_corrected.name == table_dogs_owners.name).select(table_dogs_corrected.name, table_dogs_corrected.age, table_dogs_owners.owner)pw.debug.compute_and_print(table_dogs_full)
| name | age | owner ^VJ3K9DF... | Ace | 7 | Alice ^V1RPZW8... | Bella | 4 | Bob ^R0GE4WM... | Coco | 12 | Alice
As we continue you will see some more advanced programming constructs which provide a lot of flexibility to Pathway:
- Applying Machine Learning to data tables.
- The ability to do iteration and recursion.
We will also use Pathway connectors to external data sources (for data inputs) and sinks (for data outputs).
This, and a lot more, is covered in recipes in the Pathway cookbook - try these for a start: