Realtime RAG with Gemini and other Open AI Alternatives

In this module, we’ll cover how to build a Realtime RAG project using alternate APIs by Gemini, Replicate, and 300+ LLMs apart from OpenAI.

Here you'll use Google Gemini Pro for chat completions/generation and Sentence Transformer for embeddings.

If you've already built a pipeline using OpenAI, you can jump to Step 4.

Introduction

We’ll walk you through setting up a Realtime RAG project using Gemini Pro and Pathway.

Key Features:

• Create an in-memory document store with real-time document indexing using Pathway that can easily work with documents in your Google Drive, Microsoft 365 SharePoint, Databases, Local directory, etc.
• Connect an LLM model of choice (https://docs.litellm.ai/docs/providers) to your knowledge base.
• Get quality, accurate, and precise responses to your questions.
• Ask questions about folders, files, or all your documents easily, with the help of filtering options.
• Get an executive outlook for a question on different files to easily access available knowledge in your documents.

Prerequisites

Before we begin, ensure you have the following requirements we shared earlier:

• Docker Desktop: This tool allows you to run applications in isolated containers. Download Docker Desktop. (Note: Your antivirus software might block the installation, so temporarily disable it if needed.)
• If you’re using VS Code, consider installing the Docker extension to manage containers directly from the editor.

Step-by-Step Process

Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

docker --version

You should see the Docker version information if it's installed correctly.

Step 2: Clone the LLM App Templates Repository

Next, clone the llm-app repository from GitHub. This repository contains all the files you’ll need.

git clone https://github.com/pathwaycom/llm-app.git

If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:

git pull

This will update your local repository with the latest changes from the remote repository.

Step 3: Navigate to the Project Directory

Change to the directory where the relevant example of your current project is located.

cd examples/pipelines/demo-question-answering

Step 4: Update your .env File with your Gemini API Key

If you've already built a pipeline with Open AI, this is where things get slightly different. Configure your key in a .env file by providing it as follows:

GEMINI_API_KEY=*******

Replace ******* with your actual Gemini API key. Save the file as .env in the demo-question-answering folder.

Step 5: Update requirements.txt File

Add the following dependencies to the requirements.txt file that enable us to use Pathway LiteLLM wrapper, Google's APIs, and Sentence Transformers (a.k.a. SBERT) for embeddings:

pathway[all]
python-dotenv==1.0.1
mpmath==1.3.0
litellm>=1.35
Google-generativeai
Sentence-transformers

Step 6: Update app.py Code

Replace the existing app.py code with the following:

import logging
import sys
import click
import pathway as pw
import yaml
from dotenv import load_dotenv
from pathway.udfs import DiskCache
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.stdlib.indexing import BruteForceKnnFactory
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.document_store import DocumentStore

# Set your Pathway license key here to use advanced features.
pw.set_license_key("demo-license-key-with-telemetry")

# Set up basic logging to capture key events and errors.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

# Load environment variables (e.g., API keys) from the .env file.
load_dotenv()

# Function to handle data sources
def data_sources(source_configs) -> list[pw.Table]:
    sources = []
    for source_config in source_configs:
        if source_config["kind"] == "local":
            source = pw.io.fs.read(
                **source_config["config"],
                format="binary",
                with_metadata=True,
            )
            sources.append(source)
    return sources

# Command-line interface (CLI) function to run the app with a specified config file.
@click.command()
@click.option("--config_file", default="config.yaml", help="Config file to be used.")
def run(config_file: str = "config.yaml"):
    # Load the configuration from the YAML file.
    with open(config_file) as config_f:
        configuration = yaml.safe_load(config_f)

    llm = llms.LiteLLMChat(model="gemini/gemini-pro", cache_strategy=DiskCache())


    parser = parsers.UnstructuredParser()

    text_splitter = splitters.TokenCountSplitter(max_tokens=400)

    embedding_model = "avsolatorio/GIST-small-Embedding-v0"

    embedder = embedders.SentenceTransformerEmbedder(
        embedding_model,
        call_kwargs={"show_progress_bar": False}
    )


    index = BruteForceKnnFactory(embedder=embedder)

    # Host and port configuration for running the server.
    host_config = configuration["host_config"]
    host, port = host_config["host"], host_config["port"]

    # Initialize the vector store for storing document embeddings in memory.
    # This vector store updates the index dynamically whenever the data source changes
    # and can scale to handle over a million documents.
    doc_store = DocumentStore(
            *data_sources(configuration["sources"]),
            splitter=text_splitter,
            parser=parser,
            retriever_factory=index,
        )

    # Create a RAG (Retrieve and Generate) question-answering application.
    rag_app = BaseRAGQuestionAnswerer(llm=llm, indexer=doc_store)

    # Build the server to handle requests at the specified host and port.
    rag_app.build_server(host=host, port=port)

    # Run the server with caching enabled, and handle errors without shutting down.
    rag_app.run_server(with_cache=True, terminate_on_error=False)

# Entry point to execute the app if the script is run directly.
if __name__ == "__main__":
    run()

Key Changes:

• Embedding Model Selection: Chose avsolatorio/GIST-small-Embedding-v0 for embedding chunked texts. This model is compact and performed well in tests. Other options include mixedbread-ai/mxbai-embed-large-v1 and avsolatorio/GIST-Embedding-v0 (For other possible choices, take a look at the MTEB Leaderboard managed by HuggingFace)
• LLM Initialization: Integrated the Gemini Pro model by updating the llm in app.py to use the LiteLLM Chat class and use Gemini Pro as the LLM.

Step 7: Update config.yaml File

Update the model specification in the config.yaml file to use Gemini Pro and keep rest of the items as is:

host_config:
  host: "0.0.0.0"  # Host for running the app.
  port: 8000    # Port for the app.

cache_options:
  with_cache: True             # Enable caching for better performance.
  cache_folder: "./Cache"      # Directory to store cached data.

sources:
  - local_files:               # Data source is local files.
    kind: local
    config:
      path: "data/"            # Path to the local data directory.

  # Optionally, you can configure Google Drive or SharePoint as additional data sources.
  # For Google Drive:
  # - google_drive_folder:
  #   kind: gdrive
  #   config:
  #     object_id: "1cULDv2OaViJBmOfG5WB0oWcgayNrGtVs"
  #     service_user_credentials_file: SERVICE_CREDENTIALS
  #     refresh_interval: 5

  # For SharePoint (commercial offering):
  # - sharepoint_folder:
  #   kind: sharepoint
  #   config:
  #     root_path: ROOT_PATH
  #     url: SHAREPOINT_URL
  #     tenant: SHAREPOINT_TENANT
  #     client_id: SHAREPOINT_CLIENT_ID
  #     cert_path: SHAREPOINT.pem
  #     thumbprint: SHAREPOINT_THUMBPRINT
  #     refresh_interval: 5

Step 8: How to run the project

Locally

If you are on Windows, please refer to running with docker section below.

Please note that the local run requires the dependencies to be installed. It can be done with a simple pip command:

pip install -r requirements.txt

Then, run the app:

python app.py

With Docker

Let’s build and run the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

Build the Docker with:

docker compose build

And, run with:

docker compose up

This will start the pipeline and the ui for asking questions.

Note: You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings. You will see 0 entries (x minibatch(es)) have been... message. If there are no more updates, this means the app is ready for use! ::

Handling Port Conflicts: If port 8000 is already in use and you see an error related to it, you can specify a different port.

Open up another terminal window and follow the next steps.

Step 9: Interacting with your deployed RAG pipeline

To seamlessly integrate into your workflow, Pathway AI Pipelines include a robust REST API. This REST API is the primary interface for interacting with the deployed AI Pipelines.

The API allows you to easily query the RAG and retrieve the generated results, making it simple to connect the AI Pipelines to other components of your application via simple HTTP requests.

You can check the various REST API endpoints here:

Pathway AI Pipelines REST API

Summary of available endpoints:

This example spawns a lightweight webserver that accepts queries on five possible endpoints, divided into two categories: document indexing and RAG with LLM.

Document Indexing capabilities

• /v1/retrieve to perform similarity search;
• /v1/statistics to get the basic stats about the indexer's health;
• /v1/pw_list_documents to retrieve the metadata of all files currently processed by the indexer.

LLM and RAG capabilities

• /v1/pw_ai_answer to ask questions about your documents, or directly talk with your LLM;
• /v1/pw_ai_summary to summarize a list of texts;

Firstly, let's retrieve the list of files from which our LLMs will gather information. To check the available inputs and associated metadata, you can use either the curl command (for Linux/Mac users) or Invoke-WebRequest (for Windows PowerShell users) as described below:

For Linux/Mac Users (curl command):

Use the following curl command to query the app:

curl -X 'POST'   'http://localhost:8000/v1/pw_list_documents'   -H 'accept: */*'   -H 'Content-Type: application/json'

This will return the list of files e.g. if you start with the data folder provided in the demo, the answer will be as follows:

[{"created_at": null, "modified_at": 1718810417, "owner": "root", "path":"data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf", "seen_at": 1718902304}]

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, you can use the following Invoke-WebRequest command to perform the same query:

Invoke-WebRequest -Uri 'http://localhost:8000/v1/pw_list_documents' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" }

This will also return the list of files with the associated metadata, similar to the example above.

Key Differences:

• Use curl for Linux/Mac environments or for users who have installed curl on Windows.
• Use Invoke-WebRequest for users working within Windows PowerShell.

This ensures that no matter which environment you're using, you can retrieve the list of documents and associated metadata to confirm that the app is ready for queries.

Secondly, lets start asking questions to LLM:

For Linux/Mac Users (curl command):

You can use the following curl command to ask a simple question to the RAG service:

curl -X 'POST' 'http://0.0.0.0:8000/v1/pw_ai_answer' -H 'accept: */*' -H 'Content-Type: application/json' -d '{
  "prompt": "What are the terms and conditions"
}'

It should return the following answer:

"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, use the Invoke-WebRequest command to ask the same question:

Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v1/pw_ai_answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What are the terms and conditions?}'

This will return the same response with the answer:

"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."

Conclusion

This will help you set up a powerful Realtime RAG pipeline with Gemini Pro.

If you get stuck, you should explore the Pathway documentation here and try to find the issue yourself once. It will also help you understand the code better, and many of your queries can actually be figured out via LLMs as well.

If still needed, you are very welcomed to ask it in the Discord channel for this bootcamp or also post your query on LiteLLM's Discord. It is generally a great practice to post your queries in the most relevant open source communities. 😄