for developers

Input / Output API

In Pathway, accessing the data is done using connectors. This page provides their API documentation. See connector articles for an overview of their architecture.

Available connectors

Read and write

Read only

Write only

class pw.io.CsvParserSettings(delimiter=',', quote='"', escape=None, enable_double_quote_escapes=True, enable_quoting=True, comment_character=None)

Class representing settings for the CSV parser.

class pw.io.OnChangeCallback(*args, **kwargs)

class pw.io.PersistenceConfig(*, metadata_storage, snapshot_storage)

This class aggregates the metadata and stream storage settings. This is the entry point for persistence configuration and should be used as `persistence_config` parameter in `pw.run(...)` command.

class pw.io.PersistentStorageBackend(engine_data_storage, fs_path)

This class works as a part of a high-level persistence config. User specifies the persistent storage parameters using one classmethod-marked methods.

In order to configure persistence in Pathway, you will need two settings of this kind: one for stream storage and one for snapshot storage.

Functions

pw.io.subscribe(table, on_change, on_end=<function <lambda>>)

Calls a callback function on_change on every change happening in table.
  • Parameters
    • table – the table to subscribe.
    • on_change (OnChangeCallback) – the callback function to be called on every change in the table. The function is required to accept three parameters: the row changed, the time of the change in microseconds and the flag stating if the change had been an addition of the row. These parameters of the callback are expected to have names row, time and is_addition respectively.
    • on_end (Callable[[], Any]) – the callback function to be called when the stream of changes ends. It will be called on each engine worker separately.
  • Returns
    None

Api Docs

Groupby API

Api Docs

Expressions API