Run In Colab  View in Github

A tour of Pathway's transformer classes

In this part of the guide, we will go through several examples of Pathway's transformer classes. This should give you a good overview of how to handle them and how useful they are.

We will not go into the details of the implementations, so you are strongly encouraged to read our introduction first.

In the following, we are going to see how to use transformer classes to perform simples operations on a single row, use transformers as a method, and use transformers to combine several tables at once.

Our guinea pig

We will experiment on the following table:

from typing import Anyimport pathway as pwguinea_pig = pw.debug.table_from_markdown(    """    | val  | aux 0  | 0    | 10 1  | 1    | 11 2  | 2    | 12 3  | 3    | 13 4  | 4    | 14 5  | 5    | 15 6  | 6    | 16 """)

Simple operations on a single row

First, we are going to perform simple operations on our table: adding a given number, obtaining the squared value, and performing the sum of two columns.

Adding 10 to each value:

@pw.transformerclass add_ten:    class table(pw.ClassArg):        val = pw.input_attribute()        @pw.output_attribute        def result(self) -> float:            return self.val + 10result = add_ten(guinea_pig).tablepw.debug.compute_and_print(result)
            | result
^8JFNKVV... | 10
^2TMTFGY... | 11
^YHZBTNY... | 12
^SERVYWW... | 13
^8GR6BSX... | 14
^76QPWK3... | 15
^C4S6S48... | 16

As you can see only the column val has been taken into account.

Obtaining the squared value of each value:

@pw.transformerclass squared_value:    class table(pw.ClassArg):        val = pw.input_attribute()        @pw.output_attribute        def result(self) -> float:            return self.val * self.valresult = squared_value(guinea_pig).tablepw.debug.compute_and_print(result)
            | result
^8JFNKVV... | 0
^2TMTFGY... | 1
^YHZBTNY... | 4
^SERVYWW... | 9
^8GR6BSX... | 16
^76QPWK3... | 25
^C4S6S48... | 36

Summing two columns

@pw.transformerclass summing_columns:    class table(pw.ClassArg):        val = pw.input_attribute()        aux = pw.input_attribute()        @pw.output_attribute        def result(self) -> float:            return self.val + self.auxresult = summing_columns(guinea_pig).tablepw.debug.compute_and_print(result)
            | result
^8JFNKVV... | 10
^2TMTFGY... | 12
^YHZBTNY... | 14
^SERVYWW... | 16
^8GR6BSX... | 18
^76QPWK3... | 20
^C4S6S48... | 22

Those three results can be obtained in a unique transformer:

@pw.transformerclass combined_transformer:    class table(pw.ClassArg):        val = pw.input_attribute()        aux = pw.input_attribute()        @pw.output_attribute        def result_add(self) -> float:            return self.val + 10        @pw.output_attribute        def result_squared(self) -> float:            return self.val * self.val        @pw.output_attribute        def result_sum(self) -> float:            return self.val + self.auxresult = combined_transformer(guinea_pig).tablepw.debug.compute_and_print(result)
            | result_add | result_squared | result_sum
^8JFNKVV... | 10         | 0              | 10
^2TMTFGY... | 11         | 1              | 12
^YHZBTNY... | 12         | 4              | 14
^SERVYWW... | 13         | 9              | 16
^8GR6BSX... | 14         | 16             | 18
^76QPWK3... | 15         | 25             | 20
^C4S6S48... | 16         | 36             | 22

Finally, we can use the new values inside the same transformer to perform more advanced operations:

@pw.transformerclass reusing_transformer:    class table(pw.ClassArg):        val = pw.input_attribute()        @pw.output_attribute        def result_add(self) -> float:            return self.val + 10        @pw.output_attribute        def result_double(self) -> float:            return self.result_add + self.result_addresult = reusing_transformer(guinea_pig).tablepw.debug.compute_and_print(result)
            | result_add | result_double
^8JFNKVV... | 10         | 20
^2TMTFGY... | 11         | 22
^YHZBTNY... | 12         | 24
^SERVYWW... | 13         | 26
^8GR6BSX... | 14         | 28
^76QPWK3... | 15         | 30
^C4S6S48... | 16         | 32

Transformers as a method

We are not bound to static computation as transformers provide way to obtain methods as new values. This is done using the method keyword:

@pw.transformerclass method_transformer:    class table(pw.ClassArg):        val: float = pw.input_attribute()        @pw.method        def method_result(self, arg) -> float:            return self.val + argmethod_table = method_transformer(guinea_pig).tableresult = method_table.select(res=method_table.method_result(10))pw.debug.compute_and_print(result)
            | res
^8JFNKVV... | 10
^2TMTFGY... | 11
^YHZBTNY... | 12
^SERVYWW... | 13
^8GR6BSX... | 14
^76QPWK3... | 15
^C4S6S48... | 16

Transformer Classes using two different tables

Now we want to do something more complicated which requires two different tables.

We have a table matchings which contains pairs of values a and b and a table profiles which contains the profile of each value of the pairs. We want to compute, for each pair, the sum of the profiles of the values of the pair.

First, we need the tables:

profiles = pw.debug.table_from_markdown(    """    | profile 0  | 1 1  | 10 2  | 100 3  | 1000 """)matchings = pw.debug.table_from_markdown(    """    | a  | b 0  | 0  | 2 1  | 1  | 3 """)matchings = matchings.select(    a=profiles.pointer_from(matchings.a), b=profiles.pointer_from(matchings.b))

Now, we do a transformer which takes the two tables as parameters. To access a given table inside the transformer, we use the notation self.transformer.my_table.

@pw.transformerclass using_two_tables:    class profiles_table(pw.ClassArg):        profile: float = pw.input_attribute()    class matchings_table(pw.ClassArg):        a: pw.Pointer = pw.input_attribute()        b: pw.Pointer = pw.input_attribute()        @pw.output_attribute        def sum_profiles(self) -> float:            pa = self.transformer.profiles_table[self.a].profile            pb = self.transformer.profiles_table[self.b].profile            return pa + pbresult = using_two_tables(    profiles_table=profiles, matchings_table=matchings).matchings_tablepw.debug.compute_and_print(result)
            | sum_profiles
^8JFNKVV... | 101
^2TMTFGY... | 1010

Other topics

We hope these examples make you feel comfortable using Pathway's transformer classes. You can take a look at our advanced example of transformer classes on a tree.

To continue your tour of Pathway, you can also check out our connectors, or see directly how to use Pathway to implement classical algorithm such as PageRank.

Olivier Ruas

Algorithm and Data Processing Magician