The ultimate data processing framework for the AI era
import pathway as pw
log_table = pw.connector(input_log_stream)
ts_table = log_table.reduce(ts = pw.reducers.max(pw.this.ts))
def sliding_window(log_table, ts_table, length):
t_sliding_window = log_table.filter(pw.this.ts >= ts_table.ix_ref().ts - length)
return t_sliding_window
t_sliding_window = sliding_window(log_table, ts_table, 5*60)
t_alert = t_sliding_window.reduce(count=pw.reducers.count())
t_alert = t_alert.select(
alert=pw.this.count >= alert_threshold
)
Trusted by
The AI era is transforming data teams into software teams.
Today’s data applications blur the distinction between data scientists and data engineers. Leading data teams handle complex code logic: machine learning, time series processing, and graph algorithms. All this when working with real-time data sources and changing data.
If you are struggling to design, run, and monitor data applications, you are not alone.
Traditional data processing frameworks were not designed for the challenges we face today. They force developers to compromise on performance, scalability, or ease of use.
With Pathway you can focus on delivering value, not infrastructure.
Pathway is a Python-first framework that lets developers and ML engineers easily implement production-grade applications involving Machine Learning, LLM's, or advanced algorithms.
To provide a smooth and scalable experience with live data sources, Pathway takes care of data updates, consistency, and versioning. Developers can focus on the logic and not the plumbing.
Pathway is based on the fastest Rust runtime on the market. Its engine performs incremental computing as input data changes, leveraging the best features of Timely and Differential Dataflow, along with proprietary research.
Use it for time series analysis, alerting and anomaly detection, graph analysis, LLM applications, real-time ML, and any other data pipelines you like.
Check out our developer hub
What you get with Pathway
Dramatic ease of prototyping
Pathway provides a dramatic ease of prototyping with interactive code logic. You can easily work in notebooks, prototype your data processing pipeline, and connect or simulate live data sources.
All this with a standard Python environment. Forget the pains of setting up a local cluster just to test your application with a tuned model setting. Pathway is a pip-installable Python library.
Use the same production code and runtime engine for data science experiments, scaled production deployment in containers, and CI/CD unit tests.
Power and flexibility of code development
Pathway provides convenient library routines for processing event streams at scale and cross-referencing data from different sources.
Create live incremental data connections to databases, API’s, files, S3, logstash, Kafka, as data sources or sinks - or add your own custom connectors in Python.
Pathway ensures that all live data source updates are taken care of, so you don’t have to worry about source synchronization.
Your code can call external functions, API’s, and models such as LLM’s - do an “apply” or “async_apply”, the choice is yours.
From decision trees to nearest neighbors search, Pathway enables you to perform real-time learning and inference, with forgetting.
Pathway provides ways to quickly retrieve the most relevant data to your AI application’s search context - from vector indexes to knowledge graph retrieval.
Use Pathway as an in-memory service to answer queries, retrieve up-to-date features with sub-millisecond latency.
Automatically notify users and down-stream systems when answers to registered queries have changed based on fresh data.
Apps built with Pathway ran faster than those in traditional frameworks, in both streaming and batch.
See benchmarksProduction-grade code from Day 1
Your Pathway code is production-grade already as you create it in your IDE or notebook. You don’t have to worry about scalability issues when you put it in production.
And, as a Python developer, you will never again have to worry about gluttonous JVM’s running out of resources. Promise.
What users say
It's time to add Pathway to your workflow
using Pathway on top of a Kafka real-time data pipeline
to unlock Pathway-powered analytics for your ELK stack for free
Put Pathway at the heart of Operational Analytics
Pathway for Enterprise brings you horizontal scalability, Machine Learning toolboxes, support with SLA, and more.
See full features list
Vertical Solutions
AI Applications and data pipelines crafted for the needs of your industry. With turn-key solutions that can be owned and modified by your data team, Pathway reduces the time to build a production-grade system from long months to days.
Gartner's Representative VendorMarket Guide for Data Analytics and Intelligence Platforms in Supply Chain