pathway.io.python package

class pathway.io.python.ConnectorSubject()

An abstract class allowing to create custom python connectors.

Custom python connector can be created by extending this class and implementing run() function responsible for filling the buffer with data. This function will be started by pathway engine in a separate thread.

In order to send a message one of the methods next_json(), next_str(), next_bytes() can be used.

close()

Sends a sentinel message.

Should be called to indicate that no new messages will be sent.

  • Return type
    None

commit()

Sends a commit message.

  • Return type
    None

next_bytes(message)

Sends a message.

  • Parameters
    message (bytes) – bytes encoded json string.
  • Return type
    None

next_json(message)

Sends a message.

  • Parameters
    message (Dict) – Dict representing json.
  • Return type
    None

next_str(message)

Sends a message.

  • Parameters
    message (str) – json string.
  • Return type
    None

on_stop()

Called after the end of the run() function.

  • Return type
    None

start()

Runs a separate thread with function feeding data into buffer.

Should not be called directly.

  • Return type
    None

pathway.io.python.read(subject, *, format='json', value_columns=None, primary_key=None, types=None, autocommit_duration_ms=10000, debug_data=None)

Reads a table from a ConnectorSubject.

  • Parameters
    • subject (ConnectorSubject) – An instance of a ConnectorSubject
    • format (str) – Format of the data produced by a subject, “json” or “raw”. In case of a “raw” format, table with single “data” column will be produced.
    • value_columns (OptionalListstr) – Columns to extract for a table.
    • primary_key (OptionalListstr) – In case the table should have a primary key generated according to a subset of its columns, the set of columns should be specified in this field. Otherwise, the primary key will be generated randomly.
    • autocommit_duration_ms (int) – the maximum time between two commits. Every autocommit_duration_ms milliseconds, the updates received by the connector are committed and pushed into Pathway’s computation graph.
    • debug_data – Static data replacing original one when debug mode is active.
    • types (OptionalDictstr, PathwayType) – Dictionary containing the mapping between the columns and the data types (pw.Type) of the values of those columns. This parameter is optional, and if not provided the default type is pw.Type.ANY.
  • Returns
    The table read.
  • Return type
    Table