pw.xpacks.llm.document_store

Builds a document indexing pipeline for processing documents and querying closest documents to a query according to a specified index.

• Parameters
  • docs (Union[Table, Iterable[Table]]) – pathway tables typically coming out of connectors which contain source documents. The table needs to contain a data column of type bytes - usually by setting format of the connector to be "raw"". Optionally, it can contain a _metadata column containing a dictionary with metadata which is then used for filters. Some connectors offer with_metadata argument for returning _metadata column.
  • retriever_factory (AbstractRetrieverFactory) – factory for building an index, which will be provided texts by the DocumentStore.
  • parser (Callable[[bytes], list[tuple[str, dict]]] | UDF | None) – callable that parses file contents into a list of documents.
  • splitter (Callable[[str], list[tuple[str, dict]]] | UDF | None) – callable that splits long documents.
  • doc_post_processors (list[Callable[[str, dict], tuple[str, dict]] | UDF] | None) – optional list of callables that modify parsed files and metadata. any callable takes two arguments (text: str, metadata: dict) and returns them as a tuple.

class InputsResultSchema

classmethod from_langchain_components(docs, retriever_factory, parser=None, splitter=None, **kwargs)

sourceInitializes DocumentStore by using LangChain components.

• Parameters
  • docs (Union[Table, Iterable[Table]]) – pathway tables typically coming out of connectors which contain source documents
  • retriever_factory (AbstractRetrieverFactory) – factory for building an index, which will be provided texts by the DocumentStore.
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents
  • splitter (BaseDocumentTransformer | None) – Langchain component for splitting documents into parts

classmethod from_llamaindex_components(docs, retriever_factory, transformations, parser=None, **kwargs)

sourceInitializes DocumentStore by using LlamaIndex TransformComponents.

• Parameters
  • docs (Union[Table, Iterable[Table]]) – pathway tables typically coming out of connectors which contain source documents
  • retriever_factory (AbstractRetrieverFactory) – factory for building an index, which will be provided texts by the DocumentStore.
  • transformations (list[TransformComponent]) – list of LlamaIndex components.
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents

inputs_query(input_queries)

sourceQuery DocumentStore for the list of input documents.

retrieve_query(retrieval_queries)

sourceQuery DocumentStore for the list of closest texts to a given query.

statistics_query(info_queries)

sourceQuery DocumentStore for statistics about indexed documents. It returns the number of indexed texts, time of last modification, and time of last indexing of input document.

Document store for the slide-search application. Builds a document indexing pipeline and starts an HTTP REST server.

Adds to the DocumentStore a new method parsed_documents a set of documents metadata after the parsing and document post processing stages.

class InputsResultSchema

classmethod from_langchain_components(docs, retriever_factory, parser=None, splitter=None, **kwargs)

sourceInitializes DocumentStore by using LangChain components.

• Parameters
  • docs (Union[Table, Iterable[Table]]) – pathway tables typically coming out of connectors which contain source documents
  • retriever_factory (AbstractRetrieverFactory) – factory for building an index, which will be provided texts by the DocumentStore.
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents
  • splitter (BaseDocumentTransformer | None) – Langchain component for splitting documents into parts

classmethod from_llamaindex_components(docs, retriever_factory, transformations, parser=None, **kwargs)

sourceInitializes DocumentStore by using LlamaIndex TransformComponents.

• Parameters
  • docs (Union[Table, Iterable[Table]]) – pathway tables typically coming out of connectors which contain source documents
  • retriever_factory (AbstractRetrieverFactory) – factory for building an index, which will be provided texts by the DocumentStore.
  • transformations (list[TransformComponent]) – list of LlamaIndex components.
  • parser (Callable[[bytes], list[tuple[str, dict]]] | None) – callable that parses file contents into a list of documents

inputs_query(input_queries)

sourceQuery DocumentStore for the list of input documents.

parsed_documents_query(parse_docs_queries)

sourceQuery the SlidesDocumentStore for the list of documents with the associated metadata after the parsing stage.

retrieve_query(retrieval_queries)

sourceQuery DocumentStore for the list of closest texts to a given query.

statistics_query(info_queries)

sourceQuery DocumentStore for statistics about indexed documents. It returns the number of indexed texts, time of last modification, and time of last indexing of input document.