Docker Deployment of Pathway

Introduction

Deployment is a critical step that should never be overlooked.

Pathway is fully compatible with Python and leverages its flexibility to provide various deployment options.

Pathway is fully Python-compatible: you can use any existing Python deployment method.

Whether starting a new project or adding Pathway to an existing one, it allows you to choose the best approach that suits your needs.

In this article, we will explore how to deploy a Pathway project using Docker.

Pathway is meant to be deployed in a containerized manner. Single-machine deployments can easily be achieved using Docker. The deployment can run concurrently on multiple cores using multiple processes or threads. We provide a pathway spawn command to launch multi-process and multi-threaded jobs. The choice between threads and multiple processes depends on the nature of the computation. While communication between threads is faster, Python-heavy workloads may require multi-process parallelism to bypass the GIL.

Prerequisites

Before we dive into the deployment process, ensure that you have Docker installed on your system. You can download and install Docker from the official website Docker Installation Guide.

Using the Pathway image

To deploy your Pathway application, you can use the Pathway Docker image. It includes all the dependencies required to run Pathway.

Using a Dockerfile

Using a Dockerfile is an easy way to configure your project using Docker. To use Pathway, you can set the image to use the Pathway Docker image using the FROM command:

FROM pathwaycom/pathway:latest

# Set working directory
WORKDIR /app

# Copy requirements file and install dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Command to run the Pathway script
CMD [ "python", "./your-script.py" ]

Note that the Pathway Docker image contains everything it needs to run a Pathway application. If you don't use any other library than Pathway, then there is no need for a requirements.txt file.

You can then build and run the Docker image:

docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

Using multi-threading and multi-processing

Pathway allows you to use multi-threading and multi-processing using Pathway CLI. To run a Pathway application using 2 processes and 3 threads, replace this command:

CMD [ "python", "./your-script.py" ]

by this one:

CMD ["pathway", "spawn", "--processes", "2", "--threads", "3", "python", "./your-script.py"]

Running a single Python script

When dealing with single-file projects, creating a full-fledged Dockerfile might seem unnecessary. In such scenarios, you can execute a Python script directly using the Python Docker image. For example:

docker run -it --rm --name my-pathway-app -v "$PWD":/app pathwaycom/pathway:latest python my-pathway-app.py

Using a Python image

Alternatively, Pathway is fully Python-compatible: you can use any existing Python deployment method. If you prefer using a standard Python image and installing Pathway via pip, you can use such a Dockerfile:

FROM --platform=linux/x86_64 python:3.10

# Set working directory
WORKDIR /app

# Copy requirements file and install dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Command to run the Pathway script
CMD [ "python", "./your-script.py" ]

⚠️ Pathway is not available on Windows and requires a Python 3.10+ installation. For compatibility reasons, you should use x86_64 Linux container and a Python 3.10+ image.

Otherwise, it's the same as with the Pathway image, except that you don't need to install Pathway. You can build and run your image with the same commands as before:

docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

Example: Log Monitoring with Docker

You can see how to deploy a Pathway project with Docker by looking at our Realtime Server Log Monitoring.

Deploying on the cloud

If you want to scale your Pathway application, you may be interested in deploying Pathway in the cloud. You can learn more about this on the dedicated page.