Universes

The universe of a table is a central concept in Pathway. In a nutshell, the universe of a table is the collection of the ids of the said table. It is used to know if some operations can be performed: if you want to update the cells of one table t1 by the values contained in a table t2, you have to make sure that each id of t2 is also in t1.

Simple, right?

Simple, but there is a bit more to this than meets the eye! Pathway aims at working on ever-changing data tables: Pathway' engine ensures that each update in the data is taken into account in realtime on each table. In this context, the set of ids is not something static nor stable: universes are used to ensure operations are performed on compatible data tables, despite potential instability.

For example, two data tables from two different data sources may have the same set of ids at some point but may diverge with upcoming updates. By default, tables from different data sources will not have the same universe.

Pathway tries to infer whether the tables have the same universe but depending on how the tables are processed it may fail. You may want to force two tables to have the same universe (or one table have a universe which is a subset of the universe of the second): Pathway might have failed the inference or because you know that the two tables actually derive from the same data source: you can specify it manually with t1.promise_universe_is_equal_to(t2).