pathway package

Pathway main module

Typical use: import pathway as pw.


class pathway.ClassArg(ref: RowReference, ptr: Pointer)

Base class to inherit from when writing inner classes for class transformers.

Example:

>>> import pathway as pw>>> @pw.transformer... class simple_transformer:...     class table(pw.ClassArg):...         arg = pw.input_attribute()...         @pw.output_attribute...         def ret(self) -> int:...             return self.arg + 1>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7... ''')>>> t2 = simple_transformer(table=t1.select(arg=t1.age)).table>>> pw.debug.compute_and_print(t1 + t2, include_id=False)age | ret7   | 88   | 99   | 1010  | 11

pointer_from(*args, optional=False)

Pseudo-random hash of its argument. Produces pointer types. Applied value-wise.

  • Return type
    BasePointer

class pathway.ColumnReference(*, column, table, name)

Reference to the column.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice  dog... 2    9    Bob  dog... 3    8  Alice  cat... 4    7    Bob  dog''')>>> isinstance(t1.age, pw.ColumnReference)True>>> isinstance(t1["owner"], pw.ColumnReference)True

property name()

Name of the referred column.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice  dog... 2    9    Bob  dog... 3    8  Alice  cat... 4    7    Bob  dog''')>>> t1.age.name'age'

property table()

Table where the referred column belongs to.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice  dog... 2    9    Bob  dog... 3    8  Alice  cat... 4    7    Bob  dog''')>>> t1.age.table is t1True

class pathway.GroupedTable(table, grouping_columns, set_id=False)

Result of a groupby operation on a Table.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.groupby(t1.pet, t1.owner)>>> isinstance(t2, pw.GroupedTable)True

reduce(*args, **kwargs)

Reduces grouped table to a table.

  • Parameters
    • args (ColumnReference) – Column references.
    • kwargs (ColumnExpression) – Column expressions with their new assigned names.
  • Returns
    Created table.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.groupby(t1.pet, t1.owner).reduce(t1.owner, t1.pet, ageagg=pw.reducers.sum(t1.age))>>> pw.debug.compute_and_print(t2, include_id=False)owner | pet | ageaggAlice | cat | 8Alice | dog | 10Bob   | dog | 16

class pathway.JoinResult(left_table, right_table, on_left, on_right, substitution, allow_miss=True, id=None)

Result of a join between tables.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice    1... 2    9    Bob    1... 3    8  Alice    2... ''')>>> t2 = pw.debug.parse_to_table('''...     age  owner  pet size... 11   10  Alice    3    M... 12    9    Bob    1    L... 13    8    Tom    1   XL... ''')>>> t3 = t1.join(t2, t1.pet == t2.pet, t1.owner == t2.owner)   # noqa: E501>>> isinstance(t3, pw.JoinResult)True

select(*args, **kwargs)

Computes result of a join.

  • Parameters
    • args (ColumnReference) – Column references.
    • kwargs (Any) – Column expressions with their new assigned names.
  • Returns
    Created table.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age  | owner  | pet...  10  | Alice  | 1...   9  | Bob    | 1...   8  | Alice  | 2... ''')>>> t2 = pw.debug.parse_to_table('''... age  | owner  | pet | size...  10  | Alice  | 3   | M...  9   | Bob    | 1   | L...  8   | Tom    | 1   | XL... ''')>>> t3 = t1.join(t2, t1.pet == t2.pet, t1.owner == t2.owner).select(age=t1.age, owner_name=t2.owner, size=t2.size)   # noqa: E501>>> pw.debug.compute_and_print(t3, include_id=False)age | owner_name | size9   | Bob        | L

class pathway.Pointer()

Pointer to row type.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.select(col=t1.id)>>> t2.schema['col'] is pw.PointerTrue

class pathway.Schema()

Base class to inherit from when creating schemas. All schemas should be subclasses of this one.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice  dog... 2    9    Bob  dog... 3    8  Alice  cat... 4    7    Bob  dog''')>>> t1.schema.as_dict(){'age': <class 'int'>, 'owner': <class 'str'>, 'pet': <class 'str'>}>>> issubclass(t1.schema, pw.Schema)True>>> class NewSchema(pw.Schema):...   foo: int>>> SchemaSum = NewSchema | t1.schema>>> SchemaSum.as_dict(){'age': <class 'int'>, 'owner': <class 'str'>, 'pet': <class 'str'>, 'foo': <class 'int'>}

class pathway.Table(columns, universe, pk_columns={}, schema=None, id_column=None)

Collection of named columns over identical universes.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> isinstance(t1, pw.Table)True

static concat(*tables)

Concatenate contents of several tables.

This is similar to PySpark union. .. warning:: All tables must have the same schema.

  • Parameters
    tables (Table) – List of tables to concatenate. All tables must have the same schema.
  • Returns
    The concatenated table. It will have new, synthetic ids.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | pet... 1 | Dog... 7 | Cat... ''')>>> t2 = pw.debug.parse_to_table('''...   | pet... 1 | Manul... 8 | Octopus... ''')>>> t3 = pw.Table.concat(t1, t2)>>> pw.debug.compute_and_print(t3, include_id=False)petCatDogManulOctopus

copy()

Returns a copy of a table.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.copy()>>> pw.debug.compute_and_print(t2, include_id=False)age | owner | pet7   | Bob   | dog8   | Alice | cat9   | Bob   | dog10  | Alice | dog>>> t1 is t2False
  • Return type
    Table

difference(other)

Restrict self universe to keys not appearing in the other table.

  • Parameters
    other (Table) – table with ids to remove from self.
  • Returns
    table with restricted universe, with the same set of columns
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | age  | owner  | pet... 1 | 10   | Alice  | 1... 2 | 9    | Bob    | 1... 3 | 8    | Alice  | 2... ''')>>> t2 = pw.debug.parse_to_table('''...   | cost... 2 | 100... 3 | 200... 4 | 300... ''')>>> t3 = t1.difference(t2)>>> pw.debug.compute_and_print(t3, include_id=False)age | owner | pet10  | Alice | 1

static empty(**kwargs)

Create an empty table with schema specified by kwargs.

  • Parameters
    kwargs (NewType(DType, type)) – Dict whose keys are column names and values are column types.
  • Returns
    Created empty table.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.Table.empty(age=float, pet=float)>>> pw.debug.compute_and_print(t1, include_id=False)age | pet

filter(filter_expression)

Filter a table according to filter condition.

  • Parameters
    filter – ColumnExpression that specifies the filtering condition.
  • Returns
    Result has the same schema as self and its ids are subset of self.id.
  • Return type
    Table

Example:

>>> import pathway as pw>>> vertices = pw.debug.parse_to_table('''... label outdegree...     1         3...     7         0... ''')>>> filtered = vertices.filter(vertices.outdegree == 0)>>> pw.debug.compute_and_print(filtered, include_id=False)label | outdegree7     | 0

static from_columns(*args, **kwargs)

Build a table from columns.

All columns must have the same ids. Columns’ names must be pairwise distinct.

  • Parameters
    • args (ColumnReference) – List of columns.
    • kwargs (ColumnReference) – Columns with their new names.
  • Returns
    Created table.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.Table.empty(age=float, pet=float)>>> t2 = pw.Table.empty(foo=float, bar=float).unsafe_promise_same_universe_as(t1)>>> t3 = pw.Table.from_columns(t1.pet, qux=t2.foo)>>> pw.debug.compute_and_print(t3, include_id=False)pet | qux

groupby(*args, id=None)

Group table by groupby_columns.

NOTE: Usually followed by .reduce() that aggregates the result and returns a table.

  • Parameters
    • groupby_columns – columns to group by.
    • id (OptionalColumnReference) – if provided, is the column used to set id’s of the rows of the result
  • Returns
    Groupby object.
  • Return type
    GroupedTable

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.groupby(t1.pet, t1.owner).reduce(t1.owner, t1.pet, ageagg=pw.reducers.sum(t1.age))>>> pw.debug.compute_and_print(t2, include_id=False)owner | pet | ageaggAlice | cat | 8Alice | dog | 10Bob   | dog | 16

having(indexer)

Removes rows so that indexed.ixindexer is possible when some rows are missing.

  • Return type
    JoinResult

property id(: ColumnReferenc )

Get reference to pseudocolumn containing id’s of a table.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.select(ids = t1.id)>>> pw.debug.compute_and_print(t2.select(test=t2.id == t2.ids), include_id=False)testTrueTrueTrueTrue
  • Return type
    ColumnReference

intersect(*tables)

Restrict self universe to keys appearing in all of the tables.

  • Parameters
    tables (Table) – tables keys of which are used to restrict universe.
  • Returns
    table with restricted universe, with the same set of columns
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | age  | owner  | pet... 1 | 10   | Alice  | 1... 2 | 9    | Bob    | 1... 3 | 8    | Alice  | 2... ''')>>> t2 = pw.debug.parse_to_table('''...   | cost... 2 | 100... 3 | 200... 4 | 300... ''')>>> t3 = t1.intersect(t2)>>> pw.debug.compute_and_print(t3, include_id=False)age | owner | pet8   | Alice | 29   | Bob   | 1

property ix()

Return an object that indexed by column returns new table reindexed.

  • Returns
    an object that when indexed by some column returns a table with rows specified by that column.
  • Return type
    Indexer

Example:

>>> import pathway as pw>>> t_animals = pw.debug.parse_to_table('''...   | epithet    | genus... 1 | upupa      | epops... 2 | acherontia | atropos... 3 | bubo       | scandiacus... 4 | dynastes   | hercules... ''')>>> t_birds = pw.debug.parse_to_table('''...   | desc... 2 | hoopoe... 4 | owl... ''')>>> ret = t_birds.select(t_birds.desc, latin=t_animals.ix[t_birds.id].genus)>>> pw.debug.compute_and_print(ret, include_id=False)desc   | latinhoopoe | atroposowl    | hercules

join(other, *on, id=None)

Join self with other using the given join expression.

  • Parameters
    • other (Table) – the right side of the join.
    • on (ColumnBinaryOpExpression) – a list of column expressions. Each must have == as the top level operation and be of the form LHS: ColumnReference == RHS: ColumnReference.
    • id (OptionalColumnReference) – optional argument for id of result, can be only self.id or other.id
  • Returns
    an object on which .select() may be called to extract relevant
      columns from the result of the join.
  • Return type
    JoinResult

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age  | owner  | pet...  10  | Alice  | 1...   9  | Bob    | 1...   8  | Alice  | 2... ''')>>> t2 = pw.debug.parse_to_table('''... age  | owner  | pet | size...  10  | Alice  | 3   | M...  9   | Bob    | 1   | L...  8   | Tom    | 1   | XL... ''')>>> t3 = t1.join(t2, t1.pet == t2.pet, t1.owner == t2.owner).select(age=t1.age, owner_name=t2.owner, size=t2.size)  # noqa: E501>>> pw.debug.compute_and_print(t3, include_id = False)age | owner_name | size9   | Bob        | L

pointer_from(*args, optional=False)

Pseudo-random hash of its argument. Produces pointer types. Applied column-wise.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice  dog... 2    9    Bob  dog... 3    8  Alice  cat... 4    7    Bob  dog''')>>> g = t1.groupby(t1.owner).reduce(refcol = t1.pointer_from(t1.owner)) # g.id == g.refcol>>> pw.debug.compute_and_print(g.select(test = (g.id == g.refcol)), include_id=False)testTrueTrue

reduce(*args, **kwargs)

Reduce a table to a single row.

Equivalent to self.groupby().reduce(*args, **kwargs).

  • Parameters
    • args (ColumnReference) – reducer to reduce the table with
    • kwargs (ColumnExpression) – reducer to reduce the table with. Its key is the new name of a column.
  • Returns
    Reduced table.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t2 = t1.reduce(ageagg=pw.reducers.argmin(t1.age))>>> pw.debug.compute_and_print(t2, include_id=False) ageagg^...>>> t3 = t2.select(t1.ix[t2.ageagg].age, t1.ix[t2.ageagg].pet)>>> pw.debug.compute_and_print(t3, include_id=False)age | pet7   | dog

rename_columns(**kwargs)

Rename columns according to kwargs.

Columns not in keys(kwargs) are not changed. New name of a column must not be id.

  • Parameters
    kwargs (ColumnReference) – mapping from old column names to new names.
  • Returns
    self with columns renamed.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | 1... 9   | Bob   | 1... 8   | Alice | 2... ''')>>> t2 = t1.rename_columns(years_old=t1.age, animal=t1.pet)>>> pw.debug.compute_and_print(t2, include_id=False)owner | years_old | animalAlice | 8         | 2Alice | 10        | 1Bob   | 9         | 1

property schema(: Type[Schema )

Get schema of the table.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> t1.schema.as_dict(){'age': <class 'int'>, 'owner': <class 'str'>, 'pet': <class 'str'>}>>> t1.schema['age']<class 'int'>
  • Return type
    TypeSchema

select(*args, **kwargs)

Build a new table with columns specified by kwargs.

Output columns’ names are keys(kwargs). values(kwargs) can be raw values, boxed values, columns. Assigning to id reindexes the table.

  • Parameters
    • args (ColumnReference) – Column references.
    • kwargs (Any) – Column expressions with their new assigned names.
  • Returns
    Created table.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... pet... Dog... Cat... ''')>>> t2 = t1.select(animal=t1.pet, desc="fluffy")>>> pw.debug.compute_and_print(t2, include_id=False)animal | descCat    | fluffyDog    | fluffy

unsafe_promise_same_universe_as(other)

Returns a copy of self with exactly the same universe as others.

Semantics: Required precondition self.universe == other.universe Used in situation where pathway cannot deduce equality of universes, but those are equal as verified during runtime.

Note: unlike unsafe_promise_universe_is_subset_of, the assertion only works on a returned object.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | pet... 1 | Dog... 7 | Cat... ''')>>> t2 = pw.debug.parse_to_table('''...   | age... 1 | 10... 7 | 3... ''')>>> t2 = t2.unsafe_promise_same_universe_as(t1)>>> t3 = t1 + t2>>> pw.debug.compute_and_print(t3, include_id=False)pet | ageCat | 3Dog | 10
  • Return type
    Table

update_cells(other)

Updates cells of self, breaking ties in favor for the values in other.

Semantics:

* result.columns == self.columns


* result.id == self.id


* conflicts are resolved preferring other’s values

Requires:

* other.columns ⊆ self.columns


* other.id ⊆ self.id
  • Parameters
    other (Table) – the other table.
  • Returns
    self updated with cells form other.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | age | owner | pet... 1 | 10  | Alice | 1... 2 | 9   | Bob   | 1... 3 | 8   | Alice | 2... ''')>>> t2 = pw.debug.parse_to_table('''...     age | owner | pet... 1 | 10  | Alice | 30... ''')>>> t2 = t2.unsafe_promise_universe_is_subset_of(t1)>>> t3 = t1.update_cells(t2)>>> pw.debug.compute_and_print(t3, include_id=False)age | owner | pet8   | Alice | 29   | Bob   | 110  | Alice | 30

update_columns(*args, **kwargs)

Updates columns of self, according to

*

args and

**

kwargs. See table.select specification for evaluation of args and kwargs.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | age | owner | pet... 1 | 10  | Alice | 1... 2 | 9   | Bob   | 1... 3 | 8   | Alice | 2... ''')>>> t2 = pw.debug.parse_to_table('''...   | owner | pet | size... 1 | Tom   | 1   | 10... 2 | Bob   | 1   | 9... 3 | Tom   | 2   | 8... ''')>>> t2 = t2.unsafe_promise_same_universe_as(t1)>>> t3 = t1.update_columns(*t2)>>> pw.debug.compute_and_print(t3, include_id=False)age | owner | pet | size8   | Tom   | 2   | 89   | Bob   | 1   | 910  | Tom   | 1   | 10
  • Return type
    Table

update_rows(other)

Updates rows of self, breaking ties in favor for the rows in other.

Semantics:

* result.columns == self.columns == other.columns


* result.id == self.id ∪ other.id

Requires:

* other.columns == self.columns
  • Parameters
    other (Table) – the other table.
  • Returns
    self updated with rows form other.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | age | owner | pet... 1 | 10  | Alice | 1... 2 | 9   | Bob   | 1... 3 | 8   | Alice | 2... ''')>>> t2 = pw.debug.parse_to_table('''...    | age | owner | pet... 1  | 10  | Alice | 30... 12 | 12  | Tom   | 40... ''')>>> t3 = t1.update_rows(t2)>>> pw.debug.compute_and_print(t3, include_id=False)age | owner | pet8   | Alice | 29   | Bob   | 110  | Alice | 3012  | Tom   | 40

with_id(new_index)

Set new ids based on another column containing id-typed values.

To generate ids based on arbitrary valued columns, use with_id_from.

Values assigned must be row-wise unique.

  • Parameters
    new_id – column to be used as new index.
  • Return type
    Table
  • Returns
    Table with updated ids.

Example:

>>> import pytest; pytest.xfail("with_id is hard to test")>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...   | age | owner | pet... 1 | 10  | Alice | 1... 2 | 9   | Bob   | 1... 3 | 8   | Alice | 2... ''')>>> t2 = pw.debug.parse_to_table('''...   | new_id... 1 | 2... 2 | 3... 3 | 4... ''')>>> t3 = t1.unsafe_promise_universe_is_subset_of(t2).with_id(t2.new_id)>>> pw.debug.compute_and_print(t3)    age  owner  pet^2   10  Alice    1^3    9    Bob    1^4    8  Alice    2

with_id_from(*args)

Compute new ids based on values in columns. Ids computed from columns must be row-wise unique.

  • Parameters
    columns – columns to be used as primary keys.
  • Returns
    self updated with recomputed ids.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    | age | owner  | pet...  1 | 10  | Alice  | 1...  2 | 9   | Bob    | 1...  3 | 8   | Alice  | 2... ''')>>> t2 = t1 + t1.select(old_id=t1.id)>>> t3 = t2.with_id_from(t2.age)>>> pw.debug.compute_and_print(t3)      | age | owner | pet | old_id^... | 8   | Alice | 2   | ^...^... | 9   | Bob   | 1   | ^...^... | 10  | Alice | 1   | ^...>>> t4 = t3.select(t3.age, t3.owner, t3.pet, same_as_old=(t3.id == t3.old_id),...     same_as_new=(t3.id == t3.pointer_from(t3.age)))>>> pw.debug.compute_and_print(t4)      | age | owner | pet | same_as_old | same_as_new^... | 8   | Alice | 2   | False       | True^... | 9   | Bob   | 1   | False       | True^... | 10  | Alice | 1   | False       | True

without(*columns)

Selects all columns without named column references.

  • Parameters
    columns (ColumnReference) – columns to be dropped provided by table.column_name notation.
  • Returns
    self without specified columns.
  • Return type
    Table

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age  | owner  | pet...  10  | Alice  | 1...   9  | Bob    | 1...   8  | Alice  | 2... ''')>>> t2 = t1.without(t1.age, pw.this.pet)>>> pw.debug.compute_and_print(t2, include_id=False)ownerAliceAliceBob

class pathway.TableLike(universe)

Interface class for table-likes: Table, GroupedTable and JoinResult. All of those contain universe info, and thus support universe-related asserts.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | dog... 9   | Bob   | dog... 8   | Alice | cat... 7   | Bob   | dog... ''')>>> g1 = t1.groupby(t1.owner)>>> t2 = t1.filter(t1.age >= 9)>>> pw.debug.compute_and_print(t2, include_id=False)age | owner | pet9   | Bob   | dog10  | Alice | dog>>> g2 = t2.groupby(t2.owner)>>> g2 = g2.unsafe_promise_universe_is_subset_of(g1) # t2 is a subset of t1, so this is safe

unsafe_promise_universe_is_subset_of(*others)

Asserts to the pathway that an universe of self is a subset of universe of each of the others.

Semantics: Used in situation where pathway cannot deduce one universe being a subset of another.

Returns: self

Note: unlike unsafe_promise_same_universe_as, the assertion works in place.

Example:

import pathway as pw t1 = pw.debug.parse_to_table(‘’’ … | age | owner | pet … 1 | 10 | Alice | 1 … 2 | 9 | Bob | 1 … 3 | 8 | Alice | 2 … ‘’’) t2 = pw.debug.parse_to_table(‘’’ … | age | owner | pet … 1 | 10 | Alice | 30 … ‘’’) t2 = t2.unsafe_promise_universe_is_subset_of(t1) t3 = t1 << t2 pw.debug.compute_and_print(t3, include_id=False) age | owner | pet 8 | Alice | 2 9 | Bob | 1 10 | Alice | 30

  • Return type
    TypeVar(SelfTableLike, bound= TableLike)

pathway.apply(fun, *args, **kwargs)

Applies function to column expressions, column-wise. Output column type deduced from type-annotations of a function.

Example:

>>> import pathway as pw>>> def concat(left: str, right: str) -> str:...   return left+right>>> t1 = pw.debug.parse_to_table('''... age  owner  pet...  10  Alice  dog...   9    Bob  dog...   8  Alice  cat...   7    Bob  dog''')>>> t2 = t1.select(col = pw.apply(concat, t1.owner, t1.pet))>>> pw.debug.compute_and_print(t2, include_id=False)colAlicecatAlicedogBobdogBobdog

pathway.apply_async(fun, *args, **kwargs)

Applies asynchronous function to column expressions, column-wise (stub).


pathway.apply_with_type(fun, ret_type, *args, **kwargs)

Applies function to column expressions, column-wise. Output column type is provided explicitly.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    age  owner  pet... 1   10  Alice  dog... 2    9    Bob  dog... 3    8  Alice  cat... 4    7    Bob  dog''')>>> t2 = t1.select(col = pw.apply_with_type(lambda left, right: left+right, str, t1.owner, t1.pet))>>> pw.debug.compute_and_print(t2, include_id=False)colAlicecatAlicedogBobdogBobdog

pathway.attribute(func, **kwargs)

Decorator for creation of attributes.

Example:

>>> import pathway as pw>>> @pw.transformer... class simple_transformer:...     class table(pw.ClassArg):...         arg = pw.input_attribute()......         @pw.attribute...         def attr(self) -> float:...             return self.arg*2......         @pw.output_attribute...         def ret(self) -> float:...             return self.attr + 1...>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7''')>>> t2 = simple_transformer(table=t1.select(arg=t1.age)).table>>> pw.debug.compute_and_print(t1 + t2, include_id=False)age | ret7   | 158   | 179   | 1910  | 21

pathway.cast(target_type, col)

Used to change the type of a column to a particular type. Disclaimer: it only changes type in a schema, it does not affect values stored.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''...    val... 1   10... 2    9... 3    8... 4    7''')>>> t1.schema.as_dict(){'val': <class 'int'>}>>> t2 = t1.select(val = cast(int | float, t1.val))>>> t2.schema.as_dict(){'val': int | float}
  • Return type
    ApplyExpression

pathway.input_attribute(type=<class 'float'>)

Returns new input_attribute. To be used inside class transformers.

Example:

>>> import pathway as pw>>> @pw.transformer... class simple_transformer:...     class table(pw.ClassArg):...         arg = pw.input_attribute()......         @pw.output_attribute...         def ret(self) -> float:...             return self.arg + 1...>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7''')>>> t2 = simple_transformer(table=t1.select(arg=t1.age)).table>>> pw.debug.compute_and_print(t1 + t2, include_id=False)age | ret7   | 88   | 99   | 1010  | 11

pathway.input_method(type=<class 'float'>)

Decorator for defining input methods in class transformers.

Example:

>>> import pathway as pw>>> @pw.transformer... class first_transformer:...     class table(pw.ClassArg):...         a: float = pw.input_attribute()......         @pw.method...         def fun(self, arg) -> int:...             return self.a * arg...>>> @pw.transformer... class second_transformer:...     class table(pw.ClassArg):...         m = pw.input_method(int)......         @pw.output_attribute...         def val(self):...             return self.m(2)...>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7''')>>> t2 = first_transformer(table=t1.select(a=t1.age)).table>>> t2.schema.as_dict(){'fun': typing.Callable[..., int]}>>> t3 = second_transformer(table=t2.select(m=t2.fun)).table>>> pw.debug.compute_and_print(t1 + t3, include_id=False)age | val7   | 148   | 169   | 1810  | 20

class pathway.iterate_universe(table)


pathway.method(func, **kwargs)

Decorator for creation methods in class transformers.

Example:

>>> import pathway as pw>>> @pw.transformer... class simple_transformer:...     class table(pw.ClassArg):...         a: float = pw.input_attribute()......         @pw.output_attribute...         def b(self) -> float:...             return self.fun(self.a)......         @method...         def fun(self, arg) -> float:...             return self.a * arg...>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7''')>>> t2 = simple_transformer(table=t1.select(a=t1.age)).table>>> t2.schema.as_dict(){'b': <class 'float'>, 'fun': typing.Callable[..., float]}>>> pw.debug.compute_and_print(t1 + t2.select(t2.b), include_id=False)age | b7   | 498   | 649   | 8110  | 100>>> pw.debug.compute_and_print(t1 + t2.select(out = t2.fun(t2.b)), include_id=False)age | out7   | 3438   | 5129   | 72910  | 1000

pathway.output_attribute(func, **kwargs)

Decorator for creation of output_attributes.

Example:

>>> import pathway as pw>>> @pw.transformer... class simple_transformer:...     class table(pw.ClassArg):...         arg = pw.input_attribute()......         @pw.output_attribute...         def ret(self) -> float:...             return self.arg + 1...>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7''')>>> t2 = simple_transformer(table=t1.select(arg=t1.age)).table>>> pw.debug.compute_and_print(t1 + t2, include_id=False)age | ret7   | 88   | 99   | 1010  | 11

pathway.pandas_transformer(output_schema, output_universe=None)

Decorator that turns python function operating on

`

pandas.DataFrame`s into pathway transformer.

Input universes are converted into input DataFrame indexes. The resulting index is treated as the output universe, so it must maintain uniqueness and be of integer type.

  • Parameters
    • output_schema (TypeSchema) – Schema of a resulting table.
    • output_universe (Unionstr, int, None) – Index or name of an argument whose universe will be used
    • None. (in resulting table. Defaults to) –
  • Returns
    Transformer that can be applied on pathawy tables.

pathway.schema_from_types(_name=None, **kwargs)

Constructs schema from kwargs: field=type.

Example:

>>> import pathway as pw>>> s = pw.schema_from_types(foo=int, bar=str)>>> s.as_dict(){'foo': <class 'int'>, 'bar': <class 'str'>}>>> issubclass(s, pw.Schema)True
  • Return type
    TypeSchema

class pathway.that(*args, **kwargs)

Object for generating column references without holding actual table in hand. Needs to be evaluated in the proper context. For JoinResult, refers to the right input table. For all other situations, you need pw.this object.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age  | owner  | pet...  10  | Alice  | 1...   9  | Bob    | 1...   8  | Alice  | 2... ''')>>> t2 = pw.debug.parse_to_table('''... age  | owner  | pet | size...  10  | Alice  | 3   | M...  9   | Bob    | 1   | L...  8   | Tom    | 1   | XL... ''')>>> t3 = t1.join(t2, pw.this.pet == pw.that.pet, pw.this.owner == pw.that.owner).select(age=pw.this.age, owner_name=pw.that.owner, size=pw.that.size)  # noqa: E501>>> pw.debug.compute_and_print(t3, include_id=False)age | owner_name | size9   | Bob        | L

class pathway.this(*args, **kwargs)

Object for generating column references without holding actual table in hand. Needs to be evaluated in the proper context. For most of the Table methods, refers to self. For JoinResult, refers to the left input table.

Example:

>>> import pathway as pw>>> t1 = pw.debug.parse_to_table('''... age | owner | pet... 10  | Alice | 1... 9   | Bob   | 1... 8   | Alice | 2... ''')>>> t2 = t1.select(pw.this.owner, pw.this.age)>>> pw.debug.compute_and_print(t2, include_id=False)owner | ageAlice | 8Alice | 10Bob   | 9

pathway.transformer(cls)

Decorator that wraps the outer class when defining class transformers.

Example:

>>> import pathway as pw>>> @pw.transformer... class simple_transformer:...     class table(pw.ClassArg):...         arg = pw.input_attribute()......         @pw.output_attribute...         def ret(self) -> float:...             return self.arg + 1...>>> t1 = pw.debug.parse_to_table('''... age... 10... 9... 8... 7''')>>> t2 = simple_transformer(table=t1.select(arg=t1.age)).table>>> pw.debug.compute_and_print(t1 + t2, include_id=False)age | ret7   | 88   | 99   | 1010  | 11

Subpackages

Submodules

pathway.conftest module