RAG with Gemini and other Open AI Alternatives

In this module, we’ll cover how to build a RAG project using alternate APIs by Gemini, Replicate, and 300+ LLMs apart from OpenAI.

Here you'll use Google Gemini Pro for chat completions/generation and Sentence Transformer for embeddings.

If you've already built a pipeline using OpenAI, you can jump to Step 4.

Introduction

We’ll walk you through setting up a RAG project using Gemini Pro and Pathway.

Key Features:

Create an in-memory persistent vector store with real-time document indexing using Pathway that can easily work with documents in your Google Drive, Microsoft 365 SharePoint, Databases, Local directory, etc.
Connect an LLM model of choice (https://docs.litellm.ai/docs/providers) to your knowledge base.
Get quality, accurate, and precise responses to your questions.
Ask questions about folders, files, or all your documents easily, with the help of filtering options.
Get an executive outlook for a question on different files to easily access available knowledge in your documents.

Prerequisites

Before we begin, ensure you have the following requirements we shared earlier:

Docker Desktop: This tool allows you to run applications in isolated containers. Download Docker Desktop. (Note: Your antivirus software might block the installation, so temporarily disable it if needed.)
If you’re using VS Code, consider installing the Docker extension to manage containers directly from the editor.

Step-by-Step Process

Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

docker --version

You should see the Docker version information if it's installed correctly.

Step 2: Clone the LLM App Templates Repository

Next, clone the llm-app repository from GitHub. This repository contains all the files you’ll need.

git clone https://github.com/pathwaycom/llm-app.git

If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:

git pull

This will update your local repository with the latest changes from the remote repository.

Step 3: Navigate to the Project Directory

Change to the directory where the relevant example of your current project is located.

cd examples/pipelines/demo-question-answering

Step 4: Update your `.env` File with your Gemini API Key

If you've already built a pipeline with Open AI, this is where things get slightly different. Configure your key in a .env file by providing it as follows:

GEMINI_API_KEY=*******

Replace ******* with your actual Gemini API key. Save the file as .env in the demo-question-answering folder.

Step 5: Update `requirements.txt` File

Add the following dependencies to the requirements.txt file that enable us to use Pathway LiteLLM wrapper, Google's APIs, and Sentence Transformers (a.k.a. SBERT) for embeddings:

pathway[all]>=0.11.0
python-dotenv==1.0.1
mpmath==1.3.0
litellm>=1.35
Google-generativeai
Sentence-transformers

Step 6: Update `app.py` Code

Replace the existing app.py code with the following:

app.py

import logging
import sys
import os
import click
import pathway as pw
import yaml
from dotenv import load_dotenv

from pathway.udfs import DiskCache, ExponentialBackoffRetryStrategy
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.xpacks.llm.vector_store import VectorStoreServer

# To use advanced features with Pathway Scale, get your free license key from
# https://pathway.com/features and paste it below.
# To use Pathway Community, comment out the line below.
pw.set_license_key("demo-license-key-with-telemetry")

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

load_dotenv()

def data_sources(source_configs) -> list[pw.Table]:
    sources = []
    for source_config in source_configs:
        if source_config["kind"] == "local":
            source = pw.io.fs.read(
                **source_config["config"],
                format="binary",
                with_metadata=True,
            )
            sources.append(source)
        elif source_config["kind"] == "gdrive":
            source = pw.io.gdrive.read(
                **source_config["config"],
                with_metadata=True,
            )
            sources.append(source)
        elif source_config["kind"] == "sharepoint":
            try:
                import pathway.xpacks.connectors.sharepoint as io_sp

                source = io_sp.read(**source_config["config"], with_metadata=True)
                sources.append(source)
            except ImportError:
                print(
                    "The Pathway Sharepoint connector is part of the commercial offering, "
                    "please contact us for a commercial license."
                )
                sys.exit(1)

    return sources

@click.command()
@click.option("--config_file", default="config.yaml", help="Config file to be used.")
def run(config_file: str = "config.yaml"):
    with open(config_file) as config_f:
        configuration = yaml.safe_load(config_f)

    LLM_MODEL = configuration["llm_config"]["model"]

    embedding_model = "avsolatorio/GIST-small-Embedding-v0"

    embedder = embedders.SentenceTransformerEmbedder(
        embedding_model,
        call_kwargs={"show_progress_bar": False}
    )

    chat = llms.LiteLLMChat(
        model=LLM_MODEL,
        retry_strategy=ExponentialBackoffRetryStrategy(max_retries=6),
        cache_strategy=DiskCache(),
    )

    host_config = configuration["host_config"]
    host, port = host_config["host"], host_config["port"]

    doc_store = VectorStoreServer(
        *data_sources(configuration["sources"]),
        embedder=embedder,
        splitter=splitters.TokenCountSplitter(max_tokens=400),
        parser=parsers.ParseUnstructured(),
    )

    rag_app = BaseRAGQuestionAnswerer(llm=chat, indexer=doc_store)

    rag_app.build_server(host=host, port=port)

    rag_app.run_server(with_cache=True, terminate_on_error=False)

if __name__ == "__main__":
    run()

Key Changes:

Embedding Model Selection: Chose avsolatorio/GIST-small-Embedding-v0 for embedding chunked texts. This model is compact and performed well in tests. Other options include mixedbread-ai/mxbai-embed-large-v1 and avsolatorio/GIST-Embedding-v0 (For other possible choices, take a look at the MTEB Leaderboard managed by HuggingFace)
LLM Initialization: Integrated the Gemini Pro model by updating the configuration and initialization script to use LLM_MODEL instead of GPT_MODEL.

Step 7: Update `config.yaml` File

Update the model specification in the config.yaml file to use Gemini Pro and keep rest of the items as is:

config.yaml

llm_config:
  model: "gemini/gemini-pro"
host_config:
  host: "0.0.0.0"
  port: 8000
cache_options:
  with_cache: True
  cache_folder: "./Cache"
sources:
  - local_files:
    kind: local
    config:
      # Please refer to
      # https://pathway.com/developers/api-docs/pathway-io/fs#pathway.io.fs.read
      # for options definition
      path: "data/"
  # - google_drive_folder:
  #   kind: gdrive
  #   config:
  #     # Please refer to
  #     # https://pathway.com/developers/api-docs/pathway-io/gdrive#pathway.io.gdrive.read
  #     # for options definition
  #     # Please follow https://pathway.com/developers/user-guide/connectors/gdrive-connector/#setting-up-google-drive
  #     # for instructions on getting credentials
  #     object_id: "1cULDv2OaViJBmOfG5WB0oWcgayNrGtVs" # folder used in the managed demo
  #     service_user_credentials_file: SERVICE_CREDENTIALS
  #     refresh_interval: 5
  # - sharepoint_folder:
  #   kind: sharepoint
  #   config:
  #     # The sharepoint is part of our commercial offering, please contact us to use it
  #     # Please contact here: `contact@pathway.com`
  #     root_path: ROOT_PATH
  #     url: SHAREPOINT_URL
  #     tenant: SHAREPOINT_TENANT
  #     client_id: SHAREPOINT_CLIENT_ID
  #     cert_path: SHAREPOINT.pem
  #     thumbprint: SHAREPOINT_THUMBPRINT
  #     refresh_interval: 5

Step 8: Changes in the `Dockerfile`:

Modify the Dockerfile with the following content:

FROM python:3.11
WORKDIR /app

RUN apt-get update
&& apt-get install -y python3-opencv
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*

COPY requirements.txt . RUN pip install -U --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["python", "app.py"]

Step 9: Build the Docker Image

Build the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

docker build -t raggem .

Step 10: Run the Docker Container

Run the Docker container, mounting the data folder and exposing port 8000.

For Windows:

docker run -v "${PWD}/data:/app/data" -p 8000:8000 raggem

For Linux/Mac:

docker run -v "$(pwd)/data:/app/data" -p 8000:8000 --env-file .env raggem

Handling Port Conflicts: If port 8000 is already in use, specify a different port. For example, to use port 8080:

For Windows:

docker run -v "${PWD}/data:/app/data" -p 8080:8000 raggem

For Linux/Mac:

docker run -v "$(pwd)/data:/app/data" -p 8080:8000 --env-file .env raggem

Step 11: Check the List of Files

To see the list of files in the data folder, use the curl command. While doing this ensure that you'll need to update the port here as well if you changed it in the previous step while managing port conflicts:

curl -X 'POST' 'http://localhost:8000/v1/pw_list_documents' -H 'accept: */*' -H 'Content-Type: application/json'

This will return the list of files that are ingested as an external data source for your RAG project.

Step 12: Last Step – Run the RAG Service

You can now run the RAG service. Start by asking a simple question. For example:

For Linux/Mac Users (`curl` command):

You can use the following curl command to ask a simple question to the RAG service:

curl -X 'POST' 'http://0.0.0.0:8000/v1/pw_ai_answer' -H 'accept: */*' -H 'Content-Type: application/json' -d '{
  "prompt": "What are the terms and conditions"
}'

Note: If you change the port from 8000 to another value (e.g., 8080), make sure to update the curl command accordingly. For example, replace 8000 with 8080 in the URL.

It should return the following answer:

"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."

For Windows Users (PowerShell `Invoke-WebRequest`):

If you're using PowerShell on Windows, use the Invoke-WebRequest command to ask the same question:

Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v1/pw_ai_answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What are the terms and conditions?}'

Note: Just like with curl, if you change the port to a different value (e.g., 8080), make sure to update the URL in the Invoke-WebRequest command.

This will return the same response with the answer:

"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."

Conclusion

This will help you set up a powerful RAG pipeline with Gemini Pro.

If you get stuck, you should explore the Pathway documentation here and try to find the issue yourself once. It will also help you understand the code better, and many of your queries can actually be figured out via LLMs as well.

If still needed, you are very welcomed to ask it in the Discord channel for this bootcamp or also post your query on LiteLLM's Discord. It is generally a great practice to post your queries in the most relevant open source communities. 😄