for developers

pathway.io.python package

class pw.io.python.ConnectorSubject()

An abstract class allowing to create custom python connectors.

Custom python connector can be created by extending this class and implementing run() function responsible for filling the buffer with data. This function will be started by pathway engine in a separate thread.

In order to send a message one of the methods next_json(), next_str(), next_bytes() can be used.

close()

Sends a sentinel message.

Should be called to indicate that no new messages will be sent.

commit()

Sends a commit message.

next_bytes(message)

Sends a message.
  • Parameters
    message (bytes) – bytes encoded json string.

next_json(message)

Sends a message.
  • Parameters
    message (Dict) – Dict representing json.

next_str(message)

Sends a message.
  • Parameters
    message (str) – json string.

on_stop()

Called after the end of the `run()` function.

start()

Runs a separate thread with function feeding data into buffer.

Should not be called directly.

Functions

pw.io.python.read(subject, *, schema=None, format='json', autocommit_duration_ms=1500, debug_data=None, value_columns=None, primary_key=None, types=None, default_values=None, persistent_id=None)

Reads a table from a ConnectorSubject.
  • Parameters
    • subject (ConnectorSubject) – An instance of a ConnectorSubject.
    • schema (Optional[Type[Schema]]) – Schema of the resulting table.
    • format (str) – Format of the data produced by a subject, “json” or “raw”. In case of a “raw” format, table with single “data” column will be produced.
    • debug_data – Static data replacing original one when debug mode is active.
    • autocommit_duration_ms (int) – the maximum time between two commits. Every autocommit_duration_ms milliseconds, the updates received by the connector are committed and pushed into Pathway’s computation graph
    • value_columns (Optional[List[str]]) – Columns to extract for a table. [will be deprecated soon]
    • primary_key (Optional[List[str]]) – In case the table should have a primary key generated according to a subset of its columns, the set of columns should be specified in this field. Otherwise, the primary key will be generated randomly. [will be deprecated soon]
    • types (Optional[Dict[str, PathwayType]]) – Dictionary containing the mapping between the columns and the data types (pw.Type) of the values of those columns. This parameter is optional, and if not provided the default type is pw.Type.ANY. [will be deprecated soon]
    • default_values (Optional[Dict[str, Any]]) – dictionary containing default values for columns replacing blank entries. The default value of the column must be specified explicitly, otherwise there will be no default value. [will be deprecated soon]
    • persistent_id (Optional[str]) – (unstable) An identifier, under which the state of the table will be persisted or None, if there is no need to persist the state of this table. When a program restarts, it restores the state for all input tables according to what was saved for their persistent_id. This way it’s possible to configure the start of computations from the moment they were terminated last time.
  • Returns
    Table – The table read.