
pathway.run()
index = KNNIndex(enriched_documents, d=embedding_dimension)
query_context = index.query(query, k=3).select(
pw.this.query, documents_list=pw.this.result
)
prompt = query_context.select(
prompt=build_prompt(pw.this.documents_list, pw.this.query)
)
model = OpenAIChatGPTModel(api_key=api_key)
responses = prompt.select(
query_id=pw.this.id,
result=model.apply(
pw.this.prompt,
locator=model_locator,
temperature=temperature,
max_tokens=max_tokens,
),
)
The fastest data processing engine supporting unified workflows for batch, streaming data, and LLM applications
Market Guide for Data Analytics and Intelligence Platforms in Supply Chain
Pathway is used by data scientists and engineers alike. You can program in Python or SQL.
Launch your data project quickly with static data sources. Then, connect live data (Kafka, S3/min.io, PostgreSQL,...) and run your code in Pathway's streaming mode.
You can call HTTP API's and integrate with your existing ML models or build your entire LLM stack using Pathway.
Pathway is powered by a distributed pure-Rust streaming engine. It achieves high throughput with astonishingly low latency.
Pathway works with event stream data and time series, data tables, graph data, and external blobs (video, text).
Pathway supports backfilling with a mix of streaming and static data sources. It allows for late data points and corrections to data.
Pathway is currently used for real-time anomaly detection, predictive analytics, IoT and logs data observability, recommender systems, and alerting.
You can also use Pathway simply to reduce cloud costs, when frequent batch recomputation is eating away at your data budgets.
The data processing layer of Pathway is free and source available.
using Pathway on top of a Kafka real-time data pipeline
to unlock Pathway-powered analytics for your ELK stack for free