Run a Pathway Template

The Pathway Templates provide ready-to-use setups for creating real-time, AI-driven applications. With YAML-configured templates, it's easy to customize or create your own processing pipelines for use cases like RAG and ETL.

This quick start guide will help you set up and run a Pathway Template. Whether you're developing an ETL pipeline, a document indexing solution, a knowledge mining system, or a query-response interface, this guide will get you started quickly.

Prerequisites

To get started, you'll need:

• Git to clone the repository and manage updates.
• LLM API Key (e.g., OpenAI or Hugging Face) for embedding and querying models, if needed.

Running Options

1. Docker (recommended) will install all dependencies automatically.
2. Python 3.8+ with Pathway if you prefer a local setup.

Note: if you are using RAG pipelines locally, you will need to install Pathway LLM xpack with:

pip install pathway[all]

Optional: Install Streamlit for UI and pip for dependency management (if not using Docker).

Clone the Repository

First, you need to download the repository.

For ETL templates, clone Pathway repository:

git clone https://github.com/pathwaycom/pathway.git

For RAG templates, you need to clone the dedicated repository:

git clone https://github.com/pathwaycom/llm-app.git

Selecting Your Template

Pathway Templates provide several ready-to-go templates for common use cases. Whether you need a real-time ETL, document indexing, or context-based Q&A, you'll find templates for each.

Then you need to go the repository of the chosen template, let's take the demo-question-answering as an example.

cd llm-app/examples/pipelines/demo-question-answering

Configuring Pathway Templates

Most of the templates can be configured using a YAML file. You can learn how to configure them by reading the dedicated tutorial.

For non-YAML templates, the detailed configuration and usage steps can be found in the the README and articles included with each template.

Run a Template

You can run Pathway Templates either locally or using Docker.

Self-hosting

The exact information about how to run a given template is given in the dedicated article or GitHub repository. In general, the templates can be run in two different ways:

• Manually: by running the main Python file (usually called main.py). You'll need to install the dependencies manually.
• With Docker: by using docker compose up if a docker-compose.yml file is provided. The setup is automated, handling all required dependencies.

On the Cloud

Local and Docker deployment may be not enough. Most cloud platforms offer robust support for Docker containers and/or Python deployment, allowing you to deploy your Pathway project on these cloud environments without encountering compatibility issues.

You can learn more about how to deploy a Pathway Template in the cloud here.

Pathway for Enterprise

If you want to scale your Pathway application, you may be interested in our Pathway for Enterprise.

You can learn more about Pathway Enterprise here.