pw.xpacks.llm.vector_store

Accompanying vector index server for the slide-search demo. Builds a document indexing pipeline and starts an HTTP REST server.

Modifies the VectorStoreServer’s pw_list_document endpoint to return set of metadata after the parsing and document post processing stages.

classmethod from_langchain_components(*docs, embedder, parser=None, splitter=None, **kwargs)

sourceInitializes VectorStoreServer by using LangChain components.

• Parameters
  • docs (-) – pathway tables typically coming out of connectors which contain source documents
  • embedder (-) – Langchain component for embedding documents
  • parser (-) – callable that parses file contents into a list of documents
  • splitter (-) – Langchaing component for splitting documents into parts

classmethod from_llamaindex_components(*docs, transformations, parser=None, **kwargs)

sourceInitializes VectorStoreServer by using LlamaIndex TransformComponents.

• Parameters
  • docs (-) – pathway tables typically coming out of connectors which contain source documents
  • transformations (-) – list of LlamaIndex components. The last component in this list is required to inherit from LlamaIndex BaseEmbedding
  • parser (-) – callable that parses file contents into a list of documents

run_server(host, port, threaded=False, with_cache=True, cache_backend=pw.persistence.Backend.filesystem('./Cache'), **kwargs)

sourceBuilds the document processing pipeline and runs it.

• Parameters
  • host (-) – host to bind the HTTP listener
  • port (-) – to bind the HTTP listener
  • threaded (-) – if True, run in a thread. Else block computation
  • with_cache (-) – if True, embedding requests for the same contents are cached
  • cache_backend (-) – the backend to use for caching if it is enabled. The default is the disk cache, hosted locally in the folder ./Cache. You can use Backend class of the [persistence API](/developers/api-docs/persistence-api/#pathway.persistence.Backend) to override it.
  • kwargs (-) – optional parameters to be passed to run().
• Returns
  If threaded, return the Thread object. Else, does not return.

A client you can use to query VectorStoreServer.

Please provide either the url, or host and port.

• Parameters
  • host (str | None) – host on which VectorStoreServer listens
  • port (int | None) – port on which VectorStoreServer listens
  • url (str | None) – url at which VectorStoreServer listens
  • timeout (int | None) – timeout for the post requests in seconds

get_input_files(metadata_filter=None, filepath_globpattern=None)

sourceFetch information on documents in the the vector store.

• Parameters
  • metadata_filter (str | None) – optional string representing the metadata filtering query in the JMESPath format. The search will happen only for documents satisfying this filtering.
  • filepath_globpattern (str | None) – optional glob pattern specifying which documents will be searched for this query.

get_vectorstore_statistics()

sourceFetch basic statistics about the vector store.

query(query, k=3, metadata_filter=None, filepath_globpattern=None)

sourcePerform a query to the vector store and fetch results.

• Parameters
  • query (str) –
  • k (int) – number of documents to be returned
  • metadata_filter (str | None) – optional string representing the metadata filtering query in the JMESPath format. The search will happen only for documents satisfying this filtering.
  • filepath_globpattern (str | None) – optional glob pattern specifying which documents will be searched for this query.

Builds a document indexing pipeline and starts an HTTP REST server for nearest neighbors queries.

• Parameters
  • docs (-) – pathway tables typically coming out of connectors which contain source documents.
  • embedder (-) – callable that embeds a single document
  • parser (-) – callable that parses file contents into a list of documents
  • splitter (-) – callable that splits long documents
  • doc_post_processors (-) – optional list of callables that modify parsed files and metadata. any callable takes two arguments (text: str, metadata: dict) and returns them as a tuple.

classmethod from_langchain_components(*docs, embedder, parser=None, splitter=None, **kwargs)

sourceInitializes VectorStoreServer by using LangChain components.

• Parameters
  • docs (-) – pathway tables typically coming out of connectors which contain source documents
  • embedder (-) – Langchain component for embedding documents
  • parser (-) – callable that parses file contents into a list of documents
  • splitter (-) – Langchaing component for splitting documents into parts

classmethod from_llamaindex_components(*docs, transformations, parser=None, **kwargs)

sourceInitializes VectorStoreServer by using LlamaIndex TransformComponents.

• Parameters
  • docs (-) – pathway tables typically coming out of connectors which contain source documents
  • transformations (-) – list of LlamaIndex components. The last component in this list is required to inherit from LlamaIndex BaseEmbedding
  • parser (-) – callable that parses file contents into a list of documents

run_server(host, port, threaded=False, with_cache=True, cache_backend=pw.persistence.Backend.filesystem('./Cache'), **kwargs)

sourceBuilds the document processing pipeline and runs it.

• Parameters
  • host (-) – host to bind the HTTP listener
  • port (-) – to bind the HTTP listener
  • threaded (-) – if True, run in a thread. Else block computation
  • with_cache (-) – if True, embedding requests for the same contents are cached
  • cache_backend (-) – the backend to use for caching if it is enabled. The default is the disk cache, hosted locally in the folder ./Cache. You can use Backend class of the [persistence API](/developers/api-docs/persistence-api/#pathway.persistence.Backend) to override it.
  • kwargs (-) – optional parameters to be passed to run().
• Returns
  If threaded, return the Thread object. Else, does not return.