Columnar tables: the vectorized scan

A table is one Element of columns

Most of a Document is row-shaped: one Element per thing, each carrying a handful of named Properties. Some data is column-shaped instead. A price series, a ledger, a set of measurements: many rows, a few fields, read in bulk.

For that shape there is a convention. A table is one Element whose Properties are equal-length array Values, each Property a column. The amount column is a single float[] of shape {N}. The region column is a single int[] of shape {N}. Row i is the i-th element of every column.

def table "sales" {
    float[] amount = [10, 20, 30, 40, 50]
    int[]   region = [0, 1, 0, 1, 0]        # 0 = west, 1 = east
}

Document d;
d.append(Path("/sales"), "table")
 .set("amount", FloatArray{10, 20, 30, 40, 50})
 .set("region", IntArray{0, 1, 0, 1, 0});       // 0 = west, 1 = east

d = kg.Document()
d.append("/sales", "table") \
    .set_float_array("amount", [10, 20, 30, 40, 50]) \
    .set_int_array("region", [0, 1, 0, 1, 0])    # 0 = west, 1 = east

Why a column is fast

A column is one contiguous typed buffer. A float[] of a thousand rows is a thousand float32s, end to end, one dtype. So a filter or an aggregate runs as a tight loop over that buffer with a single dtype dispatch up front, not per-row work.

The row path is general and pays for it. A query over a thousand Elements does a map lookup, a Value copy, and a dtype check on every row. The column path does the dtype dispatch once, then walks raw numbers. Same answer, one decision instead of a thousand.

ColumnTable: filter, then aggregate

ColumnTable wraps a table Element and reads its columns. where returns the row indices that pass a numeric compare. sum, min, max, and avg reduce a column, optionally over a subset of rows. In C++ it wraps the Element; in Python it takes the Document and the table's Path.

ColumnTable t(*d.element(Path("/sales")));

t.rows("amount");                          // 5

auto big = t.where("amount", Cmp::Gt, 25.0);   // rows {2, 3, 4}: 30, 40, 50
t.sum("amount", &big);                     // 120
t.max("amount", &big);                     // 50

t.sum("amount");                           // 150  (whole column)
t.avg("amount");                           // 30
t.min("amount");                           // 10

t = kg.ColumnTable(d, "/sales")

t.rows("amount")                       # 5

big = t.where("amount", ">", 25.0)     # rows [2, 3, 4]: 30, 40, 50
t.sum("amount", big)                   # 120.0
t.max("amount", big)                   # 50.0

t.sum("amount")                        # 150.0  (whole column)
t.avg("amount")                        # 30.0
t.min("amount")                        # 10.0

where takes a column name, a comparison, and a value, and returns the matching row indices. The C++ Cmp enum (Eq, Ne, Lt, Le, Gt, Ge) is a string in Python ("eq", "ne", "lt", "le", "gt", "ge", or the symbols ==, !=, <, <=, >, >=). It dispatches once on the column's dtype and tight-loops the buffer, collecting the indices that match.

SUM(amount) WHERE region = west

Filter on one column, aggregate another. The row mask from where over region selects the rows; sum over amount reduces only those:

auto west = t.where("region", Cmp::Eq, 0.0);   // rows {0, 2, 4}
t.sum("amount", &west);                        // 90  (10 + 30 + 50)

west = t.where("region", "==", 0.0)    # rows [0, 2, 4]
t.sum("amount", west)                  # 90.0  (10 + 30 + 50)

That is SUM(amount) WHERE region = west as one vectorized pass. The region filter touches one buffer, the amount reduction touches one buffer, and each takes a single dtype dispatch.

A missing or non-numeric column is honest about it: rows is 0, where is empty, and the aggregates return NaN.

Schema makes the dispatch a single branch

The dtype dispatch is one branch the first time ColumnTable reads a column. When the column's dtype is declared in a schema, it is a known branch: the dtype is fixed by the type, not discovered from the data. A schema-backed table reads one shape, every time.

This is opt-in, for analytical and column-shaped data. The row and document model is unchanged: a Document is still a set of Elements indexed by Path, and a table is just one Element that happens to hold columns. For row-at-a-time selection by field, type, or Path, reach for the query engine; for repeated equality lookups on a row field, an index turns the scan into a direct hit.

ColumnTable over a table Element runs a filter and an aggregate as one typed pass per column.