Embedders

When storing a document in a vector store, you compute the embedding vector for the text and store the vector with a reference to the original document. You can then compute the embedding of a query and find the embedded documents closest to the query.

The following embedding wrappers are available through the Pathway xpack:

• OpenAIEmbedder - Embed text with any of OpenAI's embedding models
• LiteLLMEmbedder - Embed text with any model available through LiteLLM
• SentenceTransformersEmbedder - Embed text with any model available through SentenceTransformer (aka. SBERT) maintained by Hugging Face
• GeminiEmbedder - Embed text with any of Google's available embedding models

OpenAIEmbedder

The default model for OpenAIEmbedder is text-embedding-3-small.

embedder: !pw.xpacks.llm.embedders.OpenAIEmbedder
  model: "text-embedding-3-small"

LiteLLMEmbedder

The model for LiteLLMEmbedder has to be specified during initialization. No default is provided.

embedder: !pw.xpacks.llm.embedders.LiteLLMEmbedder
  model: "text-embedding-3-small"

SentenceTransformerEmbedder

This SentenceTransformerEmbedder embedder allows you to use the models from the Hugging Face Sentence Transformer models.

The model is specified during initialization. Here is a list of available models.

embedder: !pw.xpacks.llm.embedders.SentenceTransformerEmbedder
  model: "intfloat/e5-large-v2"

GeminiEmbedder

GeminiEmbedder is the embedder for Google's Gemini Embedding Services. Available models can be found here.

embedder: !pw.xpacks.llm.embedders.GeminiEmbedder
  model: "models/text-embedding-004"