pw.xpacks.llm.vector_store

Accompanying vector index server for the slide-search demo. Builds a document indexing pipeline and starts an HTTP REST server.

Modifies the VectorStoreServer’s pw_list_document endpoint to return set of metadata after the parsing and document post processing stages.

classmethod from_langchain_components(*docs, embedder, parser=None, splitter=None, **kwargs)

sourceInitializes VectorStoreServer by using LangChain components.

• Parameters
  • docs – pathway tables typically coming out of connectors which contain source documents
  • embedder (Embeddings) – Langchain component for embedding documents
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents
  • splitter (BaseDocumentTransformer | None) – Langchaing component for splitting documents into parts

classmethod from_llamaindex_components(*docs, transformations, parser=None, **kwargs)

sourceInitializes VectorStoreServer by using LlamaIndex TransformComponents.

• Parameters
  • docs – pathway tables typically coming out of connectors which contain source documents
  • transformations (list[TransformComponent]) – list of LlamaIndex components. The last component in this list is required to inherit from LlamaIndex BaseEmbedding
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents

run_server(host, port, threaded=False, with_cache=True, cache_backend=pw.persistence.Backend.filesystem('./Cache'), **kwargs)

sourceBuilds the document processing pipeline and runs it.

• Parameters
  • host – host to bind the HTTP listener
  • port – to bind the HTTP listener
  • threaded (bool) – if True, run in a thread. Else block computation
  • with_cache (bool) – if True, embedding requests for the same contents are cached
  • cache_backend (Backend | None) – the backend to use for caching if it is enabled. The default is the disk cache, hosted locally in the folder ./Cache. You can use Backend class of the [persistence API](/developers/api-docs/persistence-api/#pathway.persistence.Backend) to override it.
  • kwargs – optional parameters to be passed to run().
• Returns
  If threaded, return the Thread object. Else, does not return.

A client you can use to query VectorStoreServer.

Please provide either the url, or host and port.

• Parameters
  • host (str | None) – host on which VectorStoreServer listens
  • port (int | None) – port on which VectorStoreServer listens
  • url (str | None) – url at which VectorStoreServer listens
  • timeout (int | None) – timeout for the post requests in seconds

get_input_files(metadata_filter=None, filepath_globpattern=None)

sourceFetch information on documents in the the vector store.

• Parameters
  • metadata_filter (str | None) – optional string representing the metadata filtering query in the JMESPath format. The search will happen only for documents satisfying this filtering.
  • filepath_globpattern (str | None) – optional glob pattern specifying which documents will be searched for this query.

get_vectorstore_statistics()

sourceFetch basic statistics about the vector store.

query(query, k=3, metadata_filter=None, filepath_globpattern=None)

sourcePerform a query to the vector store and fetch results.

• Parameters
  • query (str) –
  • k (int) – number of documents to be returned
  • metadata_filter (str | None) – optional string representing the metadata filtering query in the JMESPath format. The search will happen only for documents satisfying this filtering.
  • filepath_globpattern (str | None) – optional glob pattern specifying which documents will be searched for this query.

Builds a document indexing pipeline and starts an HTTP REST server for nearest neighbors queries.

• Parameters
  • docs (Table) – pathway tables typically coming out of connectors which contain source documents.
  • embedder (Callable[[str], list[float] | Coroutine] | UDF) – callable that embeds a single document
  • parser (Callable[[bytes], list[tuple[str, dict]]] | UDF | None) – callable that parses file contents into a list of documents
  • splitter (Callable[[str], list[tuple[str, dict]]] | UDF | None) – callable that splits long documents
  • doc_post_processors (list[Callable[[str, dict], tuple[str, dict]] | UDF] | None) – optional list of callables that modify parsed files and metadata. any callable takes two arguments (text: str, metadata: dict) and returns them as a tuple.

classmethod from_langchain_components(*docs, embedder, parser=None, splitter=None, **kwargs)

sourceInitializes VectorStoreServer by using LangChain components.

• Parameters
  • docs – pathway tables typically coming out of connectors which contain source documents
  • embedder (Embeddings) – Langchain component for embedding documents
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents
  • splitter (BaseDocumentTransformer | None) – Langchaing component for splitting documents into parts

classmethod from_llamaindex_components(*docs, transformations, parser=None, **kwargs)

sourceInitializes VectorStoreServer by using LlamaIndex TransformComponents.

• Parameters
  • docs – pathway tables typically coming out of connectors which contain source documents
  • transformations (list[TransformComponent]) – list of LlamaIndex components. The last component in this list is required to inherit from LlamaIndex BaseEmbedding
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents

run_server(host, port, threaded=False, with_cache=True, cache_backend=pw.persistence.Backend.filesystem('./Cache'), **kwargs)

sourceBuilds the document processing pipeline and runs it.

• Parameters
  • host – host to bind the HTTP listener
  • port – to bind the HTTP listener
  • threaded (bool) – if True, run in a thread. Else block computation
  • with_cache (bool) – if True, embedding requests for the same contents are cached
  • cache_backend (Backend | None) – the backend to use for caching if it is enabled. The default is the disk cache, hosted locally in the folder ./Cache. You can use Backend class of the [persistence API](/developers/api-docs/persistence-api/#pathway.persistence.Backend) to override it.
  • kwargs – optional parameters to be passed to run().
• Returns
  If threaded, return the Thread object. Else, does not return.