Docker Deployment of Pathway Live Data Framework

Introduction

Deployment is a critical step that should never be overlooked.

The Pathway Live Data Framework is fully compatible with Python and leverages its flexibility to provide various deployment options.

The Pathway Live Data Framework is fully Python-compatible: you can use any existing Python deployment method.

Whether starting a new project or adding the Pathway Live Data Framework to an existing one, it allows you to choose the best approach that suits your needs.

In this article, we will explore how to deploy a Pathway Live Data Framework project using Docker.

The Pathway Live Data Framework is meant to be deployed in a containerized manner. Single-machine deployments can easily be achieved using Docker. The deployment can run concurrently on multiple cores using multiple processes or threads. We provide a pathway spawn command to launch multi-process and multi-threaded jobs. The choice between threads and multiple processes depends on the nature of the computation. While communication between threads is faster, Python-heavy workloads may require multi-process parallelism to bypass the GIL.

Prerequisites

Before we dive into the deployment process, ensure that you have Docker installed on your system. You can download and install Docker from the official website Docker Installation Guide.

Using the Pathway Live Data Framework image

To deploy your Pathway Live Data Framework application, you can use the Pathway Live Data Framework Docker image. It includes all the dependencies required to run the framework.

Using a Dockerfile

Using a Dockerfile is an easy way to configure your project using Docker. To use Pathway, you can set the image to use the Pathway Live Data Framework Docker image using the FROM command:

FROM pathwaycom/pathway:latest

# Set working directory
WORKDIR /app

# Copy requirements file and install dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Command to run the Pathway script
CMD [ "python", "./your-script.py" ]

Note that the Pathway Live Data Framework Docker image contains everything it needs to run a Pathway Live Data Framework application. If you don't use any other library than Pathway, then there is no need for a requirements.txt file.

You can then build and run the Docker image:

docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

Using multi-threading and multi-processing

The Pathway Live Data Framework allows you to use multi-threading and multi-processing using Pathway Live Data Framework CLI. To run a Pathway Live Data Framework application using 2 processes and 3 threads, replace this command:

CMD [ "python", "./your-script.py" ]

by this one:

CMD ["pathway", "spawn", "--processes", "2", "--threads", "3", "python", "./your-script.py"]

Running a single Python script

When dealing with single-file projects, creating a full-fledged Dockerfile might seem unnecessary. In such scenarios, you can execute a Python script directly using the Python Docker image. For example:

docker run -it --rm --name my-pathway-app -v "$PWD":/app pathwaycom/pathway:latest python my-pathway-app.py

Using a Python image

Alternatively, the Pathway Live Data Framework is fully Python-compatible: you can use any existing Python deployment method. If you prefer using a standard Python image and installing the framework via pip, you can use such a Dockerfile:

FROM --platform=linux/x86_64 python:3.10

# Set working directory
WORKDIR /app

# Copy requirements file and install dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Command to run the Pathway script
CMD [ "python", "./your-script.py" ]

⚠️ The Pathway Live Data Framework is not available on Windows and requires a Python 3.10+ installation. For compatibility reasons, you should use x86_64 Linux container and a Python 3.10+ image.

Otherwise, it's the same as with the Pathway image, except that you don't need to install Pathway. You can build and run your image with the same commands as before:

docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

Example: Log Monitoring with Docker

You can see how to deploy a Pathway Live Data Framework project with Docker by looking at our Realtime Server Log Monitoring.

Deploying on the cloud

If you want to scale your Pathway Live Data Framework application, you may be interested in deploying the framework in the cloud. You can learn more about this on the dedicated page.