Building with Open AI

Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

docker --version

You should see the Docker version information if it's installed correctly. If not, you might want to revisit the prerequisites and the Docker Basics section.

Step 2: Clone the LLM App Templates Repository

Next, clone the llm-app repository from GitHub. This repository contains all the files you’ll need.

git clone https://github.com/pathwaycom/llm-app.git

If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:

git pull

This will update your local repository with the latest changes from the remote repository.

Step 3: Navigate to the relevant project directory

Change to the directory where the example is located.

cd examples/pipelines/demo-question-answering

Step 4: Changes in the app.py file

In the Pathway LLM App repo (which you’ve already starred ⭐), you’ll see the demo question-answering template that uses YAML files to help you quickly set up the pipeline.

While this makes RAG applications 10x easier to customize, in this bootcamp, we will take a different approach. You’ll skip the yaml shortcuts and dive deeper into the Python application code, to build a solid foundation to create powerful customizations as you progress in your Gen AI journey.

Now that you’ve cloned the repo, let’s walk through some changes you’ll make to the files in the directory. This hands-on step will help you better understand how everything works under the hood.

Changes in the app.py file

Delete app.yaml (not needed anymore). After cloning the repository and navigating to the correct directory, modify the app.py file with the following content to ensure proper functionality:

import logging
import os
import pathway as pw
from dotenv import load_dotenv
from pathway.udfs import DefaultCache
from pathway.udfs import ExponentialBackoffRetryStrategy
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.stdlib.indexing import UsearchKnnFactory, USearchMetricKind
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.document_store import DocumentStore

# License key for Pathway advanced features
pw.set_license_key("demo-license-key-with-telemetry")

# Logging for easier debugging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

# Load environment variables like API keys
load_dotenv()

def run():
    # Step 1: Read files from the "data" folder
    folder = pw.io.fs.read(
        path="./data",
        format="binary",
        with_metadata=True,
    )
    sources = [folder]

    # Step 2: Use Docling parser to process PDFs and other files
    parser = parsers.DoclingParser(async_mode="fully_async", chunk=False)

    # Step 3: Split text into chunks (max 800 tokens each)
    text_splitter = splitters.TokenCountSplitter(max_tokens=800)

    # Step 4: Generate embeddings with OpenAI and cache results
    embedder = embedders.OpenAIEmbedder(
        model="text-embedding-3-small",
        cache_strategy=DefaultCache(),
        retry_strategy=ExponentialBackoffRetryStrategy(),
    )

    # Step 5: Build a vector index using USearch (fast and scalable)
    index = UsearchKnnFactory(
        reserved_space=1000,
        embedder=embedder,
        metric=USearchMetricKind.COS,
    )

    # Step 6: Configure LLM for answering questions
    llm = llms.OpenAIChat(
        model="gpt-4o",
        cache_strategy=DefaultCache(),
        retry_strategy=ExponentialBackoffRetryStrategy(max_retries=2),
        temperature=0,
        capacity=8,
    )

    # Step 7: Define host and port for the REST server
    pathway_host: str = "0.0.0.0"
    pathway_port = int(os.environ.get("PATHWAY_PORT", 8000))

    # Step 8: Create a Document Store to link sources, parser, splitter, and index
    doc_store = DocumentStore(
        docs=sources,
        splitter=text_splitter,
        parser=parser,
        retriever_factory=index,
    )

    # Step 9: Wrap everything in a RAG question-answering app
    rag_app = BaseRAGQuestionAnswerer(llm=llm, indexer=doc_store)

    # Step 10: Build and run the REST API server
    rag_app.build_server(host=pathway_host, port=pathway_port)
    rag_app.run_server(with_cache=True, terminate_on_error=True)

if __name__ == "__main__":
    run()

Make sure to save the file after applying the changes.

Step 5: Update the .env File with your OpenAI API Key

Rename the .env.example already present in the project to .env and then replace the placeholder API key (sk-xxxxxx) with the OpenAI API key you retrieved from the website.

OPENAI_API_KEY=<your_openai_api_key>
PATHWAY_PORT="8000"
UI_PORT="8501"

Step 6: How to run the project

Locally

If you are on Windows, please refer to running with docker section below.

Please note that the local run requires the dependencies to be installed. It can be done with a simple pip command:

pip install -r requirements.txt

Then, run the app:

python app.py

With Docker

Let’s build and run the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

Build the Docker with:

docker compose build

And, run with:

docker compose up

This will start the pipeline and the ui for asking questions.

Note: You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings. You will see 0 entries (x minibatch(es)) have been... message. If there are no more updates, this means the app is ready for use! ::

Handling Port Conflicts: If port 8000 is already in use and you see an error related to it, you can specify a different port in the .env file.

Open up another terminal window and follow the next steps.

Step 7: Interacting with your deployed RAG pipeline

To seamlessly integrate into your workflow, Pathway AI Pipelines include a robust REST API. This REST API is the primary interface for interacting with the deployed AI Pipelines.

The API allows you to easily query the RAG and retrieve the generated results, making it simple to connect the AI Pipelines to other components of your application via simple HTTP requests.

You can check the various REST API endpoints here: Pathway AI Pipelines REST API

Summary of available endpoints:

This example spawns a lightweight webserver that accepts queries on five possible endpoints, divided into two categories: document indexing and RAG with LLM.

Document Indexing capabilities

• /v1/retrieve to perform similarity search;
• /v1/statistics to get the basic stats about the indexer's health;
• /v2/list_documents to retrieve the metadata of all files currently processed by the indexer.

LLM and RAG capabilities

• /v2/answer to ask questions about your documents, or directly talk with your LLM;
• /v2/summarize to summarize a list of texts;

Firstly, let's retrieve the list of files from which our LLMs will gather information. To check the available inputs and associated metadata, you can use either the curl command (for Linux/Mac users) or Invoke-WebRequest (for Windows PowerShell users) as described below:

For Linux/Mac Users (curl command):

Use the following curl command to query the app:

curl -X 'POST'   'http://localhost:8000/v2/list_documents'   -H 'accept: */*'   -H 'Content-Type: application/json'

This will return the list of files e.g. if you start with the data folder provided in the demo, the answer will be as follows:

[{"created_at": null, "modified_at": 1718810417, "owner": "root", "path":"data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf", "seen_at": 1718902304}]

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, you can use the following Invoke-WebRequest command to perform the same query:

Invoke-WebRequest -Uri 'http://localhost:8000/v2/list_documents' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" }

This will also return the list of files with the associated metadata, similar to the example above.

Key Differences:

• Use curl for Linux/Mac environments or for users who have installed curl on Windows.
• Use Invoke-WebRequest for users working within Windows PowerShell.

This ensures that no matter which environment you're using, you can retrieve the list of documents and associated metadata to confirm that the app is ready for queries.

Step 7: Query the Default Document

By default, the repo already includes a sample file: data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf Secondly, lets start asking questions to LLM:

For Linux/Mac Users (curl command):

You can use the following curl command to ask a simple question to the RAG service:

curl -X 'POST' \
  'http://0.0.0.0:8000/v2/answer' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What is the start date of the contract?"
}'

It should return the following answer:

December 21, 2015

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, use the Invoke-WebRequest command to ask the same question:

Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v2/answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What is the start date of the contract?"}'

This will return the same response with the answer:

December 21, 2015

The answer you get will come only from this contract document, since it’s the only one indexed so far.

Step 8: Add a New Document Dynamically

So far, you’ve only been querying the default document that ships with the demo (a contract PDF). That worked fine — but in the real world, your knowledge base won’t stay static. You’ll constantly be adding new reports, contracts, meeting notes, or PDFs.

Normally, with traditional systems, if you add a new file you’d have to stop your server, reprocess all documents, and restart the pipeline. That’s painful and doesn’t scale.

With Pathway, it’s different. You can simply drop a new file into the data/ folder, and the system will immediately pick it up — no restart, no manual refresh.

Let’s try it. Download the Alphabet Q2 2023 earnings release PDF into the data/ folder:

curl -L -o data/2023q2-alphabet-earnings-release.pdf \
https://github.com/pathwaycom/llm-app/raw/main/examples/pipelines/demo-document-indexing/...

That’s it. The moment this file lands in data/, Pathway will:

1. Detect the new file.
2. Parse it (using the DoclingParser).
3. Chunk it into sections.
4. Generate embeddings via OpenAI.
5. Insert them into the index for retrieval.

All of this happens in the background automatically.

Step 9: Query Again

Now repeat the same query as before:

curl -X 'POST' \
  'http://0.0.0.0:8000/v2/answer' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What are segment operating results?"
}'

The difference this time is that your answer might not only come from the original contract document, but also from the Alphabet earnings release you just added.

Without restarting anything, the pipeline has updated itself and expanded the knowledge it can use to answer questions.

Why this matters

This “always up-to-date” behavior is what we call LiveAI™. It allows you to stream data directly into your Large Language Model pipelines, instead of waiting for slow batch jobs. At the center of this is Pathway’s Document Store — think of it like a live knowledge hub. Every time new information appears (whether it’s a PDF, a SharePoint doc, or a Google Drive file), the Document Store automatically parses, chunks, embeds, and indexes it.

So unlike a static database that quickly goes stale, this store is always current. And because it combines vector embeddings for semantic similarity with BM25 full-text search for exact keyword matches, your queries are both relevant and precise.

This means you can point your AI applications at real-world, constantly changing data — and they’ll always have the latest context when answering.

But how do you tweak this for your use-case? Let's see that by understanding the contents of the repo which just used.