Input / Output API
In Pathway, accessing the data is done using connectors. This page provides their API documentation. See connector articles for an overview of their architecture.
Available connectors
Read and write
- pathway.io.csv package
- pathway.io.fs package
- pathway.io.http package
- pathway.io.jsonlines package
- pathway.io.kafka package
- pathway.io.redpanda package
Read only
- pathway.io.debezium package
- pathway.io.plaintext package
- pathway.io.python package
- pathway.io.s3 package
- pathway.io.s3_csv package
- pathway.io.minio package
Write only
- pathway.io.elasticsearch package
- pathway.io.logstash package
- pathway.io.null package
- pathway.io.postgres package
class pw.io.CsvParserSettings(delimiter=',', quote='"', escape=None, enable_double_quote_escapes=True, enable_quoting=True, comment_character=None)
Class representing settings for the CSV parser.
class pw.io.OnChangeCallback(*args, **kwargs)
class pw.io.PersistenceConfig(*, metadata_storage, snapshot_storage)
This class aggregates the metadata and stream storage settings. This is the entry
point for persistence configuration and should be used as `persistence_config`
parameter in `pw.run(...)` command.
class pw.io.PersistentStorageBackend(engine_data_storage, fs_path)
This class works as a part of a high-level persistence config. User specifies
the persistent storage parameters using one classmethod-marked methods.
In order to configure persistence in Pathway, you will need two settings of this kind: one for stream storage and one for snapshot storage.
Functions
pw.io.subscribe(table, on_change, on_end=<function <lambda>>)
Calls a callback function on_change on every change happening in table.- Parameters
- table – the table to subscribe.
- on_change (
OnChangeCallback
) – the callback function to be called on every change in the table. The function is required to accept three parameters: the row changed, the time of the change in microseconds and the flag stating if the change had been an addition of the row. These parameters of the callback are expected to have names row, time and is_addition respectively. - on_end (
Callable
[[],Any
]) – the callback function to be called when the stream of changes ends. It will be called on each engine worker separately.
- Returns
None