Realtime RAG with Gemini and other Open AI Alternatives

In this module, you’ll build a realtime RAG pipeline using Google Gemini (for generation) and Gemini Embeddings (for retrieval) via Pathway’s LLM xpack. You won’t need YAML: we’ll do it entirely in Python for clarity and flexibility.

If you already have the OpenAI-based pipeline, jump to Step 4 to swap in Gemini. A side-by-side “What changed vs OpenAI” comparison is at the end.

Key Features

• Live Document Store: in-memory index that updates in real time as documents appear/change (works with local dirs, Google Drive, SharePoint, etc.).
• Bring-your-own LLM: use Gemini via LiteLLM providers (and easily switch to Replicate, Cohere, Groq, etc.).
• Accurate answers: hybrid retrieval pipeline (embeddings + robust parsing) for precise responses.
• Powerful filters: ask about specific files/folders or the whole corpus.
• REST API out of the box:
  • /v2/answer
  • /v2/list_documents
  • /v1/retrieve
  • /v2/summarize
  • /v1/statistics

Prerequisites

• Docker Desktop (recommended).
• Optional: VS Code + Docker extension for a nice DX.
• A Gemini API key (set it as GEMINI_API_KEY in your .env).

Step-by-Step Process

Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

docker --version

You should see the Docker version information if it's installed correctly.

Step 2: Clone the LLM App Templates Repository

Next, clone the llm-app repository from GitHub. This repository contains all the files you’ll need.

git clone https://github.com/pathwaycom/llm-app.git

If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:

git pull

This will update your local repository with the latest changes from the remote repository.

Step 3: Navigate to the Project Directory

Change to the directory where the relevant example of your current project is located.

cd examples/pipelines/demo-question-answering

Step 4: Update your .env File with your Gemini and Google API Key

If you've already built a pipeline with Open AI, this is where things get slightly different. Configure your key in a .env file by providing it as follows:

GEMINI_API_KEY="AIza----"
GOOGLE_API_KEY="AIza----"
PATHWAY_PORT=8000

Replace AIza---- with your actual Gemini API key. Save the file as .env in the demo-question-answering folder.

Step 5: Update app.py Code

We’ll skip YAML and define the whole pipeline in Python. Delete app.yaml (not needed). Replace the contents of app.py with:

import logging
import os
import pathway as pw
from dotenv import load_dotenv
from pathway.udfs import DefaultCache
from pathway.udfs import ExponentialBackoffRetryStrategy
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.stdlib.indexing import UsearchKnnFactory, USearchMetricKind
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.document_store import DocumentStore

# Set your Pathway license key here to use advanced features.
pw.set_license_key("demo-license-key-with-telemetry")

# Set up basic logging to capture key events and errors.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

# Load environment variables (e.g., API keys) from the .env file.
load_dotenv()

def run():
    folder = pw.io.fs.read(
        path="./data",
        format="binary",
        with_metadata=True,
    )

    sources = [folder]

    parser = parsers.DoclingParser(async_mode = 'fully_async', chunk= False)

    text_splitter = splitters.TokenCountSplitter(max_tokens=800)

    embedder = embedders.GeminiEmbedder(model="models/text-embedding-004")

    index = UsearchKnnFactory(reserved_space=1000, embedder=embedder, metric=USearchMetricKind.COS)

    llm  = llms.LiteLLMChat(
        model="gemini/gemini-2.0-flash", cache_strategy=DefaultCache(), retry_strategy=ExponentialBackoffRetryStrategy(max_retries=2), temperature=0, capacity=8) 

    pathway_host: str = "0.0.0.0"
    pathway_port = int(os.environ.get("PATHWAY_PORT", 8000))

    # Initialize the vector store for storing document embeddings in memory.
    # This vector store updates the index dynamically whenever the data source changes
    # and can scale to handle over a million documents.
    doc_store = DocumentStore(
            docs=sources,
            splitter=text_splitter,
            parser=parser,
            retriever_factory=index
        )

    # Create a RAG (Retrieve and Generate) question-answering application.
    rag_app = BaseRAGQuestionAnswerer(llm=llm, indexer=doc_store)

    # Build the server to handle requests at the specified host and port.
    rag_app.build_server(host=pathway_host, port=pathway_port)

    # Run the server with caching enabled, and handle errors without shutting down.
    # rag_app.run_server(with_cache=True, terminate_on_error=False)
    rag_app.run_server(with_cache=True, terminate_on_error=True)

if __name__ == "__main__":
    run()

Key Changes:

• Embeddings: Switched from OpenAIEmbedder("text-embedding-3-small") to GeminiEmbedder("models/text-embedding-004"), and dropped the explicit cache/retry configs on the embedder.
• LLM backend: Replaced OpenAIChat(model="gpt-4o") with LiteLLMChat(model="gemini/gemini-2.0-flash"), keeping the same cache/retry/temperature/capacity settings.

Step 6: How to run the project

Locally

If you are on Windows, please refer to running with docker section below.

Please note that the local run requires the dependencies to be installed. It can be done with a simple pip command:

pip install -r requirements.txt

Then, run the app:

python app.py

With Docker

Let’s build and run the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

Build the Docker with:

docker compose build

And, run with:

docker compose up

This will start the pipeline and the ui for asking questions.

Note: You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings. You will see 0 entries (x minibatch(es)) have been... message. If there are no more updates, this means the app is ready for use! ::

Handling Port Conflicts: If port 8000 is already in use and you see an error related to it, you can specify a different port.

Open up another terminal window and follow the next steps.

Step 7: Interacting with your deployed RAG pipeline

To seamlessly integrate into your workflow, Pathway AI Pipelines include a robust REST API. This REST API is the primary interface for interacting with the deployed AI Pipelines.

The API allows you to easily query the RAG and retrieve the generated results, making it simple to connect the AI Pipelines to other components of your application via simple HTTP requests.

You can check the various REST API endpoints here: Pathway AI Pipelines REST API

Summary of available endpoints:

This example spawns a lightweight webserver that accepts queries on five possible endpoints, divided into two categories: document indexing and RAG with LLM.

Document Indexing capabilities

• /v1/retrieve to perform similarity search;
• /v1/statistics to get the basic stats about the indexer's health;
• /v2/list_documents to retrieve the metadata of all files currently processed by the indexer.

LLM and RAG capabilities

• /v2/answer to ask questions about your documents, or directly talk with your LLM;
• /v2/summarize to summarize a list of texts;

Firstly, let's retrieve the list of files from which our LLMs will gather information. To check the available inputs and associated metadata, you can use either the curl command (for Linux/Mac users) or Invoke-WebRequest (for Windows PowerShell users) as described below:

For Linux/Mac Users (curl command):

Use the following curl command to query the app:

curl -X 'POST'   'http://localhost:8000/v2/list_documents'   -H 'accept: */*'   -H 'Content-Type: application/json'

This will return the list of files e.g. if you start with the data folder provided in the demo, the answer will be as follows:

[{"created_at": null, "modified_at": 1718810417, "owner": "root", "path":"data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf", "seen_at": 1718902304}]

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, you can use the following Invoke-WebRequest command to perform the same query:

Invoke-WebRequest -Uri 'http://localhost:8000/v2/list_documents' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" }

This will also return the list of files with the associated metadata, similar to the example above.

Key Differences:

• Use curl for Linux/Mac environments or for users who have installed curl on Windows.
• Use Invoke-WebRequest for users working within Windows PowerShell.

This ensures that no matter which environment you're using, you can retrieve the list of documents and associated metadata to confirm that the app is ready for queries.

Secondly, lets start asking questions to LLM:

For Linux/Mac Users (curl command):

You can use the following curl command to ask a simple question to the RAG service:

curl -X 'POST' 'http://0.0.0.0:8000/v2/answer' -H 'accept: */*' -H 'Content-Type: application/json' -d '{
  "prompt": "What are the terms and conditions"
}'

It should return the following answer:

"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, use the Invoke-WebRequest command to ask the same question:

Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v2/answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What are the terms and conditions?}'

This will return the same response with the answer:

"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."

Step 8: Add a New Document (Live Indexing)

So far, you’ve only been querying the default document that ships with the demo (a contract PDF). That worked fine — but in the real world, your knowledge base won’t stay static. You’ll constantly be adding new reports, contracts, meeting notes, or PDFs.

Normally, with traditional systems, if you add a new file you’d have to stop your server, reprocess all documents, and restart the pipeline. That’s painful and doesn’t scale.

With Pathway, it’s different. You can simply drop a new file into the data/ folder, and the system will immediately pick it up — no restart, no manual refresh.

Let’s try it. Download the Alphabet Q2 2023 earnings release PDF into the data/ folder:

curl -L -o data/2023q2-alphabet-earnings-release.pdf \
https://github.com/pathwaycom/llm-app/raw/main/examples/pipelines/demo-document-indexing/...

That’s it. The moment this file lands in data/, Pathway will:

1. Detect the new file.
2. Parse it (using the DoclingParser).
3. Chunk it into sections.
4. Generate embeddings via OpenAI.
5. Insert them into the index for retrieval.

All of this happens in the background automatically.

Query Again

Now repeat the same query as before:

curl -X 'POST' \
  'http://0.0.0.0:8000/v2/answer' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What are segment operating results?"
}'

The difference this time is that your answer might not only come from the original contract document, but also from the Alphabet earnings release you just added.

Without restarting anything, the pipeline has updated itself and expanded the knowledge it can use to answer questions.

Why this matters

This “always up-to-date” behavior is what we call LiveAI™. It allows you to stream data directly into your Large Language Model pipelines, instead of waiting for slow batch jobs. At the center of this is Pathway’s Document Store — think of it like a live knowledge hub. Every time new information appears (whether it’s a PDF, a SharePoint doc, or a Google Drive file), the Document Store automatically parses, chunks, embeds, and indexes it.

So unlike a static database that quickly goes stale, this store is always current. And because it combines vector embeddings for semantic similarity with BM25 full-text search for exact keyword matches, your queries are both relevant and precise.

This means you can point your AI applications at real-world, constantly changing data — and they’ll always have the latest context when answering.

Conclusion

If you get stuck, you should explore the Pathway documentation here and try to find the issue yourself once. It will also help you understand the code better, and many of your queries can actually be figured out via LLMs as well.

If still needed, you are very welcomed to ask it in the Discord channel for this bootcamp or also post your query on LiteLLM's Discord. It is generally a great practice to post your queries in the most relevant open source communities. 😄