Realtime RAG with Gemini and other Open AI Alternatives
In this module, we’ll cover how to build a Realtime RAG project using alternate APIs by Gemini, Replicate, and 300+ LLMs apart from OpenAI.
Here you'll use Google Gemini Pro for chat completions/generation and Sentence Transformer for embeddings.
If you've already built a pipeline using OpenAI, you can jump to Step 4.
Introduction
We’ll walk you through setting up a Realtime RAG project using Gemini Pro and Pathway.
Key Features:
- Create an in-memory document store with real-time document indexing using Pathway that can easily work with documents in your Google Drive, Microsoft 365 SharePoint, Databases, Local directory, etc.
- Connect an LLM model of choice (https://docs.litellm.ai/docs/providers) to your knowledge base.
- Get quality, accurate, and precise responses to your questions.
- Ask questions about folders, files, or all your documents easily, with the help of filtering options.
- Get an executive outlook for a question on different files to easily access available knowledge in your documents.
Prerequisites
Before we begin, ensure you have the following requirements we shared earlier:
- Docker Desktop: This tool allows you to run applications in isolated containers. Download Docker Desktop. (Note: Your antivirus software might block the installation, so temporarily disable it if needed.)
- If you’re using VS Code, consider installing the Docker extension to manage containers directly from the editor.
Step-by-Step Process
Step 1: Verify Docker Installation
First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:
docker --version
You should see the Docker version information if it's installed correctly.
Step 2: Clone the LLM App Templates Repository
Next, clone the llm-app
repository from GitHub. This repository contains all the files you’ll need.
git clone https://github.com/pathwaycom/llm-app.git
If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:
git pull
This will update your local repository with the latest changes from the remote repository.
Step 3: Navigate to the Project Directory
Change to the directory where the relevant example of your current project is located.
cd examples/pipelines/demo-question-answering
Step 4: Update your .env
File with your Gemini API Key
If you've already built a pipeline with Open AI, this is where things get slightly different. Configure your key in a .env
file by providing it as follows:
GEMINI_API_KEY=*******
Replace *******
with your actual Gemini API key. Save the file as .env
in the demo-question-answering
folder.
Step 5: Update requirements.txt
File
Add the following dependencies to the requirements.txt
file that enable us to use Pathway LiteLLM wrapper, Google's APIs, and Sentence Transformers (a.k.a. SBERT) for embeddings:
pathway[all]
python-dotenv==1.0.1
mpmath==1.3.0
litellm>=1.35
Google-generativeai
Sentence-transformers
Step 6: Update app.py
Code
Replace the existing app.py
code with the following:
import logging
import sys
import click
import pathway as pw
import yaml
from dotenv import load_dotenv
from pathway.udfs import DiskCache
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.stdlib.indexing import BruteForceKnnFactory
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.document_store import DocumentStore
# Set your Pathway license key here to use advanced features.
pw.set_license_key("demo-license-key-with-telemetry")
# Set up basic logging to capture key events and errors.
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(name)s %(levelname)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
# Load environment variables (e.g., API keys) from the .env file.
load_dotenv()
# Function to handle data sources
def data_sources(source_configs) -> list[pw.Table]:
sources = []
for source_config in source_configs:
if source_config["kind"] == "local":
source = pw.io.fs.read(
**source_config["config"],
format="binary",
with_metadata=True,
)
sources.append(source)
return sources
# Command-line interface (CLI) function to run the app with a specified config file.
@click.command()
@click.option("--config_file", default="config.yaml", help="Config file to be used.")
def run(config_file: str = "config.yaml"):
# Load the configuration from the YAML file.
with open(config_file) as config_f:
configuration = yaml.safe_load(config_f)
llm = llms.LiteLLMChat(model="gemini/gemini-pro", cache_strategy=DiskCache())
parser = parsers.UnstructuredParser()
text_splitter = splitters.TokenCountSplitter(max_tokens=400)
embedding_model = "avsolatorio/GIST-small-Embedding-v0"
embedder = embedders.SentenceTransformerEmbedder(
embedding_model,
call_kwargs={"show_progress_bar": False}
)
index = BruteForceKnnFactory(embedder=embedder)
# Host and port configuration for running the server.
host_config = configuration["host_config"]
host, port = host_config["host"], host_config["port"]
# Initialize the vector store for storing document embeddings in memory.
# This vector store updates the index dynamically whenever the data source changes
# and can scale to handle over a million documents.
doc_store = DocumentStore(
*data_sources(configuration["sources"]),
splitter=text_splitter,
parser=parser,
retriever_factory=index,
)
# Create a RAG (Retrieve and Generate) question-answering application.
rag_app = BaseRAGQuestionAnswerer(llm=llm, indexer=doc_store)
# Build the server to handle requests at the specified host and port.
rag_app.build_server(host=host, port=port)
# Run the server with caching enabled, and handle errors without shutting down.
rag_app.run_server(with_cache=True, terminate_on_error=False)
# Entry point to execute the app if the script is run directly.
if __name__ == "__main__":
run()
Key Changes:
- Embedding Model Selection: Chose
avsolatorio/GIST-small-Embedding-v0
for embedding chunked texts. This model is compact and performed well in tests. Other options includemixedbread-ai/mxbai-embed-large-v1
andavsolatorio/GIST-Embedding-v0
(For other possible choices, take a look at the MTEB Leaderboard managed by HuggingFace) - LLM Initialization: Integrated the Gemini Pro model by updating the
llm
in app.py to use the LiteLLM Chat class and use Gemini Pro as the LLM.
Step 7: Update config.yaml
File
Update the model specification in the config.yaml
file to use Gemini Pro and keep rest of the items as is:
host_config:
host: "0.0.0.0" # Host for running the app.
port: 8000 # Port for the app.
cache_options:
with_cache: True # Enable caching for better performance.
cache_folder: "./Cache" # Directory to store cached data.
sources:
- local_files: # Data source is local files.
kind: local
config:
path: "data/" # Path to the local data directory.
# Optionally, you can configure Google Drive or SharePoint as additional data sources.
# For Google Drive:
# - google_drive_folder:
# kind: gdrive
# config:
# object_id: "1cULDv2OaViJBmOfG5WB0oWcgayNrGtVs"
# service_user_credentials_file: SERVICE_CREDENTIALS
# refresh_interval: 5
# For SharePoint (commercial offering):
# - sharepoint_folder:
# kind: sharepoint
# config:
# root_path: ROOT_PATH
# url: SHAREPOINT_URL
# tenant: SHAREPOINT_TENANT
# client_id: SHAREPOINT_CLIENT_ID
# cert_path: SHAREPOINT.pem
# thumbprint: SHAREPOINT_THUMBPRINT
# refresh_interval: 5
Step 8: How to run the project
Locally
If you are on Windows, please refer to running with docker section below.
Please note that the local run requires the dependencies to be installed. It can be done with a simple pip command:
pip install -r requirements.txt
Then, run the app:
python app.py
With Docker
Let’s build and run the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).
Build the Docker with:
docker compose build
And, run with:
docker compose up
This will start the pipeline and the ui for asking questions.
Note: You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings. You will see 0 entries (x minibatch(es)) have been... message. If there are no more updates, this means the app is ready for use! ::
Handling Port Conflicts: If port 8000
is already in use and you see an error related to it, you can specify a different port.
Open up another terminal window and follow the next steps.
Step 9: Interacting with your deployed RAG pipeline
To seamlessly integrate into your workflow, Pathway AI Pipelines include a robust REST API. This REST API is the primary interface for interacting with the deployed AI Pipelines.
The API allows you to easily query the RAG and retrieve the generated results, making it simple to connect the AI Pipelines to other components of your application via simple HTTP requests.
You can check the various REST API endpoints here:
Summary of available endpoints:
This example spawns a lightweight webserver that accepts queries on five possible endpoints, divided into two categories: document indexing and RAG with LLM.
Document Indexing capabilities
/v1/retrieve
to perform similarity search;/v1/statistics
to get the basic stats about the indexer's health;/v1/pw_list_documents
to retrieve the metadata of all files currently processed by the indexer.
LLM and RAG capabilities
/v1/pw_ai_answer
to ask questions about your documents, or directly talk with your LLM;/v1/pw_ai_summary
to summarize a list of texts;
Firstly, let's retrieve the list of files from which our LLMs will gather information. To check the available inputs and associated metadata, you can use either the curl
command (for Linux/Mac users) or Invoke-WebRequest
(for Windows PowerShell users) as described below:
For Linux/Mac Users (curl
command):
Use the following curl
command to query the app:
curl -X 'POST' 'http://localhost:8000/v1/pw_list_documents' -H 'accept: */*' -H 'Content-Type: application/json'
This will return the list of files e.g. if you start with the data folder provided in the demo, the answer will be as follows:
[{"created_at": null, "modified_at": 1718810417, "owner": "root", "path":"data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf", "seen_at": 1718902304}]
For Windows Users (PowerShell Invoke-WebRequest
):
If you're using PowerShell on Windows, you can use the following Invoke-WebRequest
command to perform the same query:
Invoke-WebRequest -Uri 'http://localhost:8000/v1/pw_list_documents' `
-Method POST `
-Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" }
This will also return the list of files with the associated metadata, similar to the example above.
Key Differences:
- Use
curl
for Linux/Mac environments or for users who have installedcurl
on Windows. - Use
Invoke-WebRequest
for users working within Windows PowerShell.
This ensures that no matter which environment you're using, you can retrieve the list of documents and associated metadata to confirm that the app is ready for queries.
Secondly, lets start asking questions to LLM:
For Linux/Mac Users (curl
command):
You can use the following curl
command to ask a simple question to the RAG service:
curl -X 'POST' 'http://0.0.0.0:8000/v1/pw_ai_answer' -H 'accept: */*' -H 'Content-Type: application/json' -d '{
"prompt": "What are the terms and conditions"
}'
8000
to another value (e.g., 8080
), make sure to update the curl command accordingly. For example, replace 8000
with 8080
in the URL.It should return the following answer:
"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."
For Windows Users (PowerShell Invoke-WebRequest
):
If you're using PowerShell on Windows, use the Invoke-WebRequest
command to ask the same question:
Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v1/pw_ai_answer' `
-Method POST `
-Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
-Body '{"prompt": "What are the terms and conditions?}'
8080
), make sure to update the URL in the Invoke-WebRequest
command.This will return the same response with the answer:
"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."
Conclusion
This will help you set up a powerful Realtime RAG pipeline with Gemini Pro.
If you get stuck, you should explore the Pathway documentation here and try to find the issue yourself once. It will also help you understand the code better, and many of your queries can actually be figured out via LLMs as well.
If still needed, you are very welcomed to ask it in the Discord channel for this bootcamp or also post your query on LiteLLM's Discord. It is generally a great practice to post your queries in the most relevant open source communities. 😄