Deployment is a critical step that should never be overlooked.
Pathway is fully compatible with Python and leverages its flexibility to provide various deployment options.
Pathway is fully Python-compatible: you can use any existing Python deployment method.
Whether starting a new project or adding Pathway to an existing one, it allows you to choose the best approach that suits your needs.
In this article, we will explore how to deploy a Pathway project.
To begin, ensure you have Pathway installed in your project. You can do this using the following pip command:
pip install pathway
You can learn more about how to get started with Pathway in our dedicated article.
Once Pathway is part of your project, let's dive into the different deployment methods.
Pathway is meant to be deployed in a containerized manner.
Single-machine deployments can easily be achieved using Docker. The deployment can run concurrently on multiple cores using multiple processes or threads.
We provide a pathway spawn command to launch multi-process and multi-threaded jobs.
The choice between threads and multiple processes depends on the nature of the computation. While communication between threads is faster, Python-heavy workloads may require multi-process parallelism to bypass the GIL.
Docker provides a standardized packaging format that encapsulates your application and its dependencies. To use Pathway in Docker, you simply need a Python 3.10+ image. Create a Dockerfile in your project root with the following content:
# Use an official Python runtime as a parent image
# Set the working directory in the container
# Install Pathway and any needed package
RUN pip install pathway
# Copy the current directory contents into the container at /src
COPY . /src
# Run app.py when the container launches
CMD ["python", "main.py"]
Build your Docker image by executing the following command in the terminal:
docker build -t pathway-project .
Now, you can run your Pathway application in a Docker container:
docker run pathway-project
You can connect to the container to see your Pathway project running.
You can see how to deploy a Pathway project with Docker by looking at our Realtime Server Log Monitoring.
Most cloud platforms offer robust support for Docker containers and/or Python deployment, allowing you to deploy your Pathway project on these cloud environments without encountering compatibility issues. You can easily benefit from the scalability and flexibility of cloud deployment.
You can see how to build a real-time data pipeline with Azure Event Hubs, Pathway, and Azure OpenAI in our associated GitHub repository.
A multi-server (distributed) deployment can use Kubernetes and its cloud implementations. Pathway assumes a stateful set deployment with all pods present for a successful operation. Pathway enterprise offering covers distributed multi-server deployment for production use. Support with integration into existing helm charts and k8s tooling is provided.
You can learn more about Pathway Enterprise here.
Pathway can seamlessly be integrated into your CI/CD chain as it is inherently compatible with popular tools such as mypy or pytest. Your Pathway pipelines can be automatically tested, built, and deployed, like any other Python workflow.
You can run Pathway tests on offline data snippets locally in any CI/CD pipeline with Python. Tests can cover the handling of temporal (late, out of order) aspects of data by comparing results on multiple revisions. Pathway supports several session-replay mechanisms, such as the demo API. These allow predictably recreating streaming scenarios within standard CI/CD pipelines (Jenkins, GitHub Actions, etc.)
Deploying a Python project with the Pathway framework is a versatile process, allowing developers to choose deployment strategies that align with their project requirements. Whether you opt for containerization with Docker or leverage cloud services like Azure, Pathway's compatibility ensures a seamless deployment experience. Experiment with these examples and discover the deployment method that best suits your needs. Happy coding!