Building with Open AI

Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

docker --version

You should see the Docker version information if it's installed correctly. If not, you might want to revisit the prerequisites and the Docker Basics section.

Step 2: Clone the LLM App Templates Repository

Next, clone the llm-app repository from GitHub. This repository contains all the files you’ll need.

git clone https://github.com/pathwaycom/llm-app.git

If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:

git pull

This will update your local repository with the latest changes from the remote repository.

Step 3: Navigate to the relevant project directory

Change to the directory where the example is located.

cd examples/pipelines/demo-question-answering

Step 4: Changes in the app.py, app.yaml and docker-compose.yml files

In the Pathway LLM App repo (which you’ve already starred ⭐), you’ll see the demo question-answering template that uses YAML files to help you quickly set up the pipeline.

While this makes RAG applications 10x easier to customize, in this bootcamp, we will take a different approach. You’ll skip the yaml shortcuts and dive deeper into the Python application code, to build a solid foundation to create powerful customizations as you progress in your Gen AI journey.

Now that you’ve cloned the repo, let’s walk through some changes you’ll make to the files in the directory. This hands-on step will help you better understand how everything works under the hood.

Changes in the app.py file

After cloning the repository and navigating to the correct directory, modify the app.py file with the following content to ensure proper functionality:

import logging
import os
import sys
import click
import pathway as pw
import yaml
from dotenv import load_dotenv
from pathway.udfs import DiskCache
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.stdlib.indexing import BruteForceKnnFactory, HybridIndexFactory
from pathway.stdlib.indexing.bm25 import TantivyBM25Factory
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.document_store import DocumentStore

# Set your Pathway license key here to use advanced features.
pw.set_license_key("demo-license-key-with-telemetry")

# Set up basic logging to capture key events and errors.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

# Load environment variables (e.g., API keys) from the .env file.
load_dotenv()

# Command-line interface (CLI) function to run the app with a specified config file.
@click.command()
@click.option("--config_file", default="app.yaml", help="Config file to be used.")
def run(config_file: str = "app.yaml"):
    # Load the configuration from the YAML file.
    with open(config_file) as f:
        config = pw.load_yaml(f)

    sources = config["sources"]

    llm = llms.OpenAIChat(model="gpt-4o-mini", cache_strategy=DiskCache())

    # Initialize the OpenAI Embedder to handle embeddings with caching enabled.
    embedder = embedders.OpenAIEmbedder(
        model="text-embedding-ada-002",
        cache_strategy=DiskCache(),
    )

    parser = parsers.UnstructuredParser()

    index = HybridIndexFactory(
            [
                TantivyBM25Factory(),
                BruteForceKnnFactory(embedder=embedder),
            ]
        )

    text_splitter = splitters.TokenCountSplitter(max_tokens=400)

    # Host and port configuration for running the server.
    # Get host from app.yaml config and port from .env file
    pathway_host = config.get("host", "0.0.0.0")
    pathway_port = int(os.environ.get("PATHWAY_PORT", 8000))

    # Initialize the vector store for storing document embeddings in memory.
    # This vector store updates the index dynamically whenever the data source changes
    # and can scale to handle over a million documents.
    doc_store = DocumentStore(
            docs=sources,
            splitter=text_splitter,
            parser=parser,
            retriever_factory=index
        )

    # Create a RAG (Retrieve and Generate) question-answering application.
    rag_app = BaseRAGQuestionAnswerer(llm=llm, indexer=doc_store)

    # Build the server to handle requests at the specified host and port.
    rag_app.build_server(host=pathway_host, port=pathway_port)

    # Run the server with caching enabled, and handle errors without shutting down.
    rag_app.run_server(with_cache=True, terminate_on_error=False)

# Entry point to execute the app if the script is run directly.
if __name__ == "__main__":
    run()

Make sure to save the file after applying the changes.

Changes in the app.yaml file:

Update the content of the app.yaml file with the configuration below. It's easy to comprehend if you read the comments:

# Define the sources for document ingestion
sources:
  - !pw.io.fs.read
    path: data
    format: binary
    with_metadata: true

  # - !pw.xpacks.connectors.sharepoint.read 
  #   url: $SHAREPOINT_URL
  #   tenant: $SHAREPOINT_TENANT
  #   client_id: $SHAREPOINT_CLIENT_ID
  #   cert_path: sharepointcert.pem
  #   thumbprint: $SHAREPOINT_THUMBPRINT
  #   root_path: $SHAREPOINT_ROOT
  #   with_metadata: true
  #   refresh_interval: 30

  # - !pw.io.gdrive.read
  #   object_id: $DRIVE_ID
  #   service_user_credentials_file: gdrive_indexer.json
  #   file_name_pattern:
  #     - "*.pdf"
  #     - "*.pptx"
  #   object_size_limit: null
  #   with_metadata: true
  #   refresh_interval: 30

# Host and port configuration
host: "0.0.0.0"
port: $PATHWAY_PORT

Uncomment the commented blocks when using Google Drive or Microsoft SharePoint as your (live) data source.

Make sure to save the file after applying the changes.

Changes in the docker-compose.yml file

Update the content of the docker-compose.yml file with the configuration below.

services:
  app:
    build:
      context: .
    ports:
      - "${PATHWAY_PORT:-8000}:${PATHWAY_PORT:-8000}"
    networks:
      - network
    environment:
      PATHWAY_PORT: "${PATHWAY_PORT:-8000}"
      PATHWAY_LICENSE_KEY: $PATHWAY_LICENSE_KEY
    volumes:
      - ./data:/app/data
      - ./Cache:/app/Cache

  ui:
    build:
      context: ui
    networks:
      - network
    environment:
      PATHWAY_HOST: "app"
      PATHWAY_PORT: "${PATHWAY_PORT:-8000}"
      UI_PORT: "${UI_PORT:-8501}"
    ports:
      - "${UI_PORT:-8501}:${UI_PORT:-8501}"

networks:
  network:
    driver: bridge

Make sure to save the file after applying the changes.

Step 5: Update the .env File with your OpenAI API Key

Rename the .env.example already present in the project to .env and then replace the placeholder API key (sk-xxxxxx) with the OpenAI API key you retrieved from the website.

OPENAI_API_KEY=<your_openai_api_key>
PATHWAY_PORT="8000"
UI_PORT="8501"

Make sure to save the file after applying the changes.

Step 6: How to run the project

Locally

If you are on Windows, please refer to running with docker section below.

Please note that the local run requires the dependencies to be installed. It can be done with a simple pip command:

pip install -r requirements.txt

Then, run the app:

python app.py

With Docker

Let’s build and run the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

Build the Docker with:

docker compose build

And, run with:

docker compose up

This will start the pipeline and the ui for asking questions.

Note: You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings. You will see 0 entries (x minibatch(es)) have been... message. If there are no more updates, this means the app is ready for use! ::

Handling Port Conflicts: If port 8000 is already in use and you see an error related to it, you can specify a different port in the .env file.

Open up another terminal window and follow the next steps.

Step 7: Interacting with your deployed RAG pipeline

To seamlessly integrate into your workflow, Pathway AI Pipelines include a robust REST API. This REST API is the primary interface for interacting with the deployed AI Pipelines.

The API allows you to easily query the RAG and retrieve the generated results, making it simple to connect the AI Pipelines to other components of your application via simple HTTP requests.

You can check the various REST API endpoints here: Pathway AI Pipelines REST API

Summary of available endpoints:

This example spawns a lightweight webserver that accepts queries on five possible endpoints, divided into two categories: document indexing and RAG with LLM.

Document Indexing capabilities

• /v1/retrieve to perform similarity search;
• /v1/statistics to get the basic stats about the indexer's health;
• /v2/list_documents to retrieve the metadata of all files currently processed by the indexer.

LLM and RAG capabilities

• /v2/answer to ask questions about your documents, or directly talk with your LLM;
• /v2/summarize to summarize a list of texts;

Firstly, let's retrieve the list of files from which our LLMs will gather information. To check the available inputs and associated metadata, you can use either the curl command (for Linux/Mac users) or Invoke-WebRequest (for Windows PowerShell users) as described below:

For Linux/Mac Users (curl command):

Use the following curl command to query the app:

curl -X 'POST'   'http://localhost:8000/v2/list_documents'   -H 'accept: */*'   -H 'Content-Type: application/json'

This will return the list of files e.g. if you start with the data folder provided in the demo, the answer will be as follows:

[{"created_at": null, "modified_at": 1718810417, "owner": "root", "path":"data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf", "seen_at": 1718902304}]

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, you can use the following Invoke-WebRequest command to perform the same query:

Invoke-WebRequest -Uri 'http://localhost:8000/v2/list_documents' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" }

This will also return the list of files with the associated metadata, similar to the example above.

Key Differences:

• Use curl for Linux/Mac environments or for users who have installed curl on Windows.
• Use Invoke-WebRequest for users working within Windows PowerShell.

This ensures that no matter which environment you're using, you can retrieve the list of documents and associated metadata to confirm that the app is ready for queries.

Secondly, lets start asking questions to LLM:

For Linux/Mac Users (curl command):

You can use the following curl command to ask a simple question to the RAG service:

curl -X 'POST' \
  'http://0.0.0.0:8000/v2/answer' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What is the start date of the contract?"
}'

It should return the following answer:

December 21, 2015

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, use the Invoke-WebRequest command to ask the same question:

Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v2/answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What is the start date of the contract?"}'

This will return the same response with the answer:

December 21, 2015

But how do you tweak this for your use-case? Let's see that by understanding the contents of the repo which just used.