10k

1 – First RAG Pipeline

Welcome! In this guide, we’ll walk you through setting up a RAG project using the GPT-3.5 model (by Open AI) and Pathway. With this module, you will create a basic RAG pipeline that uses a set of external files stored in the data folder, extracts relevant information, and updates results as documents change or new ones arrive.

  • You'll use the in-memory vector store by Pathway that is persistent, scalable and production-ready. While using the same, you won't have to worry about using an external vector database (e.g. Pinecone, Weaviate, etc.).
  • For retrieval: By default, your app uses a local data source to read documents from the data folder. Retrieval is taken care by the Pathway framework hence you don't need to use additional librarires (e.g. FAISS, etc.) for retrievers.
  • Choice of LLM: In your first RAG pipeline you can go for GPT-3.5 as shown ahead. It's a powerful LLM and is one of the cost-effective options provided by the makers of ChatGPT. Alternatively should you wish to use multimodal LLMs such as GPT-4o, Claude-3.5 Sonnet, Gemini Pro, etc. – that's doable. But we'll look at them later so it's easy for you to follow a gradual process of hands-on learning.

Key Feature: Your application will stand out as it

  • Uses an in-memory vector store that is easily scalable in enterprise applications.
  • Automatically reacts to the latest changes in your external data store. For example, any change in your Google Drive or Data folder will be reflected in your RAG application right away.
  • Using this approach, you can make your AI application run in permanent connection and sync with your drive, in sync with your documents which include visually formatted elements: tables, charts, etc.

Prerequisites

Before we begin, ensure you have the following prerequisites:

  1. Docker Desktop: This tool allows you to run applications in isolated containers (quick introduction of containerization is below). It ensures consistency across different environments. Download Docker Desktop. (Note: Your antivirus software might block the installation, so temporarily disable it if needed.)
  2. OpenAI API Key: Sign up on the OpenAI website and generate an API key from the API Key Management page. (Remember, don’t share your OpenAI API key with anyone.)

Optional

  • VS Code Docker Extension: If you’re using VS Code, consider installing the Docker extension to manage containers directly from the editor.