All you need for your ETL and AI pipelines Pathway Live Data Framework is a scalable and robust data processing framework. Use it to build and power AI/ML applications with live data and real-time pipelines.
Supported up to 8 GB RAM - 4 cores
Free and open under BSL 1.1.
Self-hosted only.
Packaged for pip, poetry, docker.
What's included: 20+ App templates
basic RAG pipelines document pipelines log monitoring Kafka ETL social media sentiment analysis Core features High-performance input/output connectors (Kafka, S3, cloud file systems, databases) Predefined API connectors to 300+ data sources REST API endpoint for serving query/answer and realtime features with sub-millisecond latency Python programming API (Table API) SQL programming API Incremental stream and table operations: join, filter, group-by, reduce Advanced join types, temporal joins, windows, and ranges Custom stateful reducers Incremental "apply" and "map-reduce" Support for pointer-based data structures: trees, graphs, event sequences User Defined Functions: call external libraries Make API calls from Pathway Live Data Framework data flows Async data processing: call APIs, libraries, LLM services Data schema support (Python/mypy compatible typing) True streaming data processing engine in Rust Same engine for streaming and batch workloads Same code logic for streaming and for backfilling Parallelization with multi-processing Data persistence with S3 storage Fully interactive execution with streaming & live data sources Develop and run in Jupyter notebooks (also in streaming) Data table introspection & debugging in streaming Out-of-the-box code completion supported by IDEs (VS Code, PyCharm, ...) Dataflow and schema validation at compile time Schema autodetection from data samples Exception handling at runtime AI Toolkits Libraries for time-series operations, sampling, and filters LLM extension pack Unstructured data parsing toolkit (data source sync, parsing, extraction, indexing) Advanced data indexing built-in (vector with HNSW, BM24, hybrid) Support for custom indexing with knowledge graph and summarization techniques Support for fully local and API-based ML models Built-in support for temporal graph data Library for classification and clustering Input connectors Kafka PostgreSQL http JSON Lines Redpanda Logstash Slack File System Google PubSub Custom Python connector (APIs and other destinations) Support & Deployment Self hosted setup: Run Pathway Live Data Framework on your own machine or cloud Community support Supported up to 16 GB RAM - 4 cores (Free)
Free or paid, with license key.
Self-hosted only.
Packaged for pip, poetry, docker.
Everything in Community, plus: Extra App templates
high-accuracy RAG Sharepoint AI search Delta lake ETL monitored instances Advanced features Enterprise data source connectors for Sharepoint, Delta Table, Iceberg, BigQuery, Elastic Search, Quest DB (more coming soon) Monitoring and traces for Pathway Live Data Framework Instances OpenTelemetry compatible Grafana integration Business Support with tickets 1-business day reply time for paid tier customers Free 1st consultation call for pipeline design for your RAG, streaming and ETL use case #DeveloperAssist program Supported up to 24 TB RAM - 128 cores x 40 nodes
Pathway Live Data Framework Enterprise Licensing.
Self-hosted and managed.
Private package registry.
Deploy on cloud of choice
Everything in Scale, plus: Industry solutions
real-time GPS data analytics logistics automotive IoT analytics fraud detection RAG for sales and marketing search in slide decks Enterprise features Horizontal Scalability High availability with hot failover Pathway Live Data Framework Helm Chart models Kubernetes deployment guides Connectors for custom messaging formats (MQTT/IoT, ...) Pathway Live Data Framework Visual Explorer: Live Dashboards with Pathway Live Data Framework, including geospatial data viz Queryable historical data snapshots Real-time geospatial & trajectory mining library (GPS traces) Support for data schema & code schema versioning Pathway Live Data Framework managed services and hosting Solutioning for industry use cases Made-to-measure data pipelines Access to professional services Multi-zone SLA 24/7 phone support