pathway.persistence package
class pw.persistence.Backend(engine_data_storage, fs_path=None)
The settings of a backend, which is used to persist the computation state. There
are two kinds of data backends: metadata backend and snapshot backend. Both are
configurable via this class.
classmethod filesystem(path)
Configure the filesystem backend.- Parameters
path (str
|PathLike
[str
]) – the path to the root directory in the file system, which will be used to store the persisted data. - Returns
Class instance denoting the filesystem storage backend with root directory atpath
.
classmethod s3(root_path, bucket_settings)
Configure the S3 backend.- Parameters
- root_path (
str
) – path to the root in the S3 storage, which will be used to store persisted data; - bucket_settings (
AwsS3Settings
) – the settings for S3 bucket connection in the same format as they are used by S3 connectors.
- root_path (
- Returns
Class instance denoting the S3 storage backend with root directory asroot_path
and connection settings given bybucket_settings
.
class pw.persistence.Config(*, snapshot_interval_ms=0, metadata_storage, snapshot_storage, snapshot_access, persistence_mode, continue_after_replay)
Configure the data persistence. An instance of this class should be passed as a
parameter to pw.run in case persistence is enabled.
Please note that if you’d like to use the same backend for both metadata and
snapshot storages, you can use the convenience method simple_config
.
- Parameters
classmethod simple_config(backend, snapshot_interval_ms=0, snapshot_access=<pathway.engine.SnapshotAccess object>, persistence_mode=<pathway.engine.PersistenceMode object>, continue_after_replay=True)
Construct config from a single instance of the `Backend` class, using this backend to persist metadata and snapshot.- Parameters
- backend (
Backend
) – storage backend settings; - snapshot_interval_ms – the desired freshness of the persisted snapshot in milliseconds. The greater the value is, the more the amount of time that the snapshot may fall behind, and the less computational resources are required.
- backend (
- Returns
Persistence config.