Embedders

When storing a document in a vector store, you compute the embedding vector for the text and store the vector with a reference to the original document. You can then compute the embedding of a query and find the embedded documents closest to the query.

The following embedding wrappers are available through the Pathway xpack:

• OpenAIEmbedder - Embed text with any of OpenAI's embedding models
• LiteLLMEmbedder - Embed text with any model available through LiteLLM
• SentenceTransformersEmbedder - Embed text with any model available through SentenceTransformer (aka. SBERT) maintained by Hugging Face
• GeminiEmbedder - Embed text with any of Google's available embedding models

OpenAIEmbedder

The default model for OpenAIEmbedder is text-embedding-3-small.

LiteLLMEmbedder

The model for LiteLLMEmbedder has to be specified during initialization. No default is provided.

SentenceTransformerEmbedder

This SentenceTransformerEmbedder embedder allows you to use the models from the Hugging Face Sentence Transformer models.

The model is specified during initialization. Here is a list of available models.

GeminiEmbedder

GeminiEmbedder is the embedder for Google's Gemini Embedding Services. Available models can be found here.