Pathway Live Data Framework is now available to all developers!

The Pathway Live Data Framework - the stream processing framework which takes care of data updates for you - announces a $4.5M funding round, and opens to all developers. You can try it out in a cloud notebook directly from your browser, or run it on a local Linux machine.
Run it now

Why should you use the Pathway Live Data Framework?

The Pathway Live Data Framework is a programming framework which allows you to work with streaming data as if you were working with static data, in batch mode.

Have you ever tried to make sense of streaming data? If so, there is a high chance that you encountered at least one of these issues:

• There are these annoying data updates that need to be taken care of
• One needs to use the same logic to handle real-time and historical data
• Debugging is a nightmare, because how can you debug something against unknown data?
• Not to mention applying proper Machine Learning on top of streaming data to draw business insights from it. Business insights, which are necessary for key decision-making.

If these are problems you have been up against, you are in the right place. At Pathway, we design the Pathway Live Data Framework which quietly takes care of data updates for you.

It gives you:

• A native real-time approach. Every task is either real-time streaming or streaming with historical data (backfilling), no need for batch, no hacks required.
• Reactivity.
• Full power of Python (to make all your ML dreams come true) with an extra SQL syntax layer coming soon (to make sure all data engineers are happy with their Pathway Live Data Framework pipelines, too).

In the design of the framework's streaming engine, we opted for ease-of-use and scalability.

What does all this mean in practice, for a developer?

That you can write as if you were writing a batch data processing pipeline (well, it's actually a little more than that, as we support loops and iteration!) and have it run on streaming data.

For a start, check out this simple example of classification of handwritten digits. All of it is captured by the code below. We approach this task with a classifier from the framework's standard library, in this case, k-Nearest-Neighbors (read more on Wikipedia). The framework builds up the corresponding control flow graph, and updates it in streaming mode.

Using Pathway Live Data Framework means that all Machine Learning outcomes are updated as the models learn with new samples and improve over time. Classification decisions for tested elements will also be revisited whenever they change. Such an approach is called reactive processing of streaming data. If you would like to learn more about this topic, we explain it in detail in this fresh video talk.

You will also find many more examples in our Documentation - and we are also sharing with you the whole examples pack at https://github.com/pathwaycom/pathway-examples.

All of this is now open for you to play with it, test, and have fun. You can even run it in a cloud notebook from your browser, unless you prefer to pip install directly on to your own Linux machine.

If you have some feedback on Pathway Live Data Framework, or just some streaming use cases that are leaving you with sleepless nights, we would love to know. Join us on Discord or drop us a line!