kinogaki/Kinogaki Core

Querying: filter, aggregate, join

Reading a Document as a table

A Document is a set of Elements indexed by Path. Querying reads that set as rows: pick the Elements you want, shape the result, and pull a value out. doc.query builds a lazy query; you chain transforms onto it; a terminal runs it.

Take a small issue tracker.

def group "tracker" {
    def issue "login_500"  { str title = "login returns 500"; str status = "open";   int votes = 9; str owner = "dy" }
    def issue "slow_search" { str title = "search is slow";    str status = "open";   int votes = 4; str owner = "ana" }
    def issue "typo_footer" { str title = "footer typo";       str status = "closed"; int votes = 1; str owner = "dy" }
    def issue "dark_mode"   { str title = "dark mode";         str status = "open";   int votes = 7 }
}

A query starts at a glob (the Paths it draws from) and narrows from there. Nothing walks the Document until a terminal runs.

std::vector<const Element*> open =
    doc.query("/tracker/*")
       .where(field("status").eq("open"))
       .orderBy("votes")
       .reverse()                      // highest votes first
       .limit(3)
       .toList();

query() with no glob draws from every Element. The transforms where, orderBy, reverse, and limit each return a new query and evaluate nothing. The terminals run the pipeline as of the call: toList returns the matching Elements, paths their Paths, count the row count, any whether one matches, first the first match (null when empty), one the single match (it throws unless exactly one). orderBy sorts ascending and is stable; reverse flips the order. A query is read-only and borrows its rows from the Document, so do not use a result past the Document's lifetime or mutate the Document mid-iteration.

Predicates

A predicate is a test over one Element. field(name) references a Property by name; its comparison methods build predicates: eq, ne, lt, le, gt, ge, in (equals any of a list), contains and startsWith (string Properties), and exists. The operators ==, !=, <, <=, >, >= are sugar for the matching method.

auto hot = field("votes").gt(5) && field("status").eq("open");
std::size_t n = doc.query("/tracker/*").where(hot).count();

Predicates compose. In C++ use &&, ||, and !; in Python use &, |, and ~. Four more builders match on structure rather than a Property value: byType(type) matches the Element's type token, under(path) its descendants, nameIs(name) its leaf name, and hasMeta(key) the presence of a string-metadata key. meta(key).eq(value) compares a metadata value.

auto p = byType("issue") && under("/tracker") && field("owner").eq("dy");

Comparisons are query-layer, not raw Value equality. Numbers compare by widened value, so field("votes").eq(9) matches whether votes is stored as an int or a float. Strings compare as text. Cross-kind comparisons do not match.

Null-safe comparison

Prism states only what is present, so an absent Property is null: there is no separate null Value. This gives a three-valued rule. A positive comparison against a null field is unknown and does not match. ne matches present-and-unequal only, so a null field does not match ne. The negation !field(x).eq(v) does include null, because it is the negation of a non-match. Test presence deliberately with isNull (the Property is absent) and notNull (the Property is present). In the tracker, dark_mode has no owner.

doc.query("/tracker/*").where(field("owner").isNull()).count();    // 1  (dark_mode)
doc.query("/tracker/*").where(field("owner").ne("dy")).count();    // 1  (slow_search; dark_mode's null does NOT match)
doc.query("/tracker/*").where(!field("owner").eq("dy")).count();   // 2  (slow_search AND dark_mode)

isNull is is_null in Python, notNull is not_null. The full rule, and how constraints declare which Properties may be null, are on the schema page. Aggregations and joins skip null fields.

Aggregation

Numeric terminals fold a field across the matching rows. sum, avg, min, max, median, variance (population), and stddev each return an optional double: it is empty (None in Python) when no matching row has a numeric value at that field. A missing or non-numeric row is skipped, never an error. nunique counts the distinct values (COUNT(DISTINCT field)); count from above is the row count.

auto open = doc.query("/tracker/*").where(field("status").eq("open"));
double total   = *open.sum("votes");        // 20
double busiest = *open.max("votes");        // 9
double spread  = *open.stddev("votes");     // population stddev of {9, 4, 7}
std::size_t owners = open.nunique("owner"); // 2  (dy, ana; the null owner is skipped)

Group by

aggregateBy(groupField, valueField, op) buckets the matching rows by groupField's value, then aggregates valueField per bucket: the SQL SELECT group, AGG(value) ... GROUP BY group. The Agg enum picks the fold: Sum, Avg, Min, Max, or Count. Count counts rows and ignores valueField. A bucket whose rows have no numeric valueField contributes 0 to Sum and Count and is absent from Min, Max, and Avg. A row missing groupField lands in the "" bucket.

using Agg = Query::Agg;
// votes per owner
std::map<std::string, double> byOwner =
    doc.query("/tracker/*").aggregateBy("owner", "votes", Agg::Sum);
// → { "": 7, "ana": 4, "dy": 10 }      (dark_mode's missing owner is the "" bucket)

// issues per status
std::map<std::string, double> counts =
    doc.query("/tracker/*").aggregateBy("status", "votes", Agg::Count);
// → { "closed": 1, "open": 3 }

In C++ the Agg enum picks the fold (Sum/Avg/Min/Max/Count); in Python it is the string "sum"/"avg"/"min"/"max"/"count". A list of group fields buckets by the tuple of their values, the multi-key GROUP BY.

std::map<std::string, double> byStatusOwner =
    doc.query("/tracker/*").aggregateBy({"status", "owner"}, "votes", Agg::Sum);
// one bucket per (status, owner) pair

The terminal groupBy(field) returns the rows themselves bucketed by a string field, rather than a folded number, when you want the Elements in each group.

auto groups = doc.query("/tracker/*").groupBy("status");
for (auto& [status, issues] : groups) { /* issues is a vector<const Element*> */ }

Joins

joinOn is a value-equality hash join, the SQL left {type} JOIN right ON left.leftField = right.rightField. It pairs each left row with every right row whose rightField equals its leftField, by query-layer equality (numbers by widened value, strings by text), in O(L + R). A row missing or non-scalar at its join field does not match. The two queries may be over different Documents. In C++ it is a free function over two queries; in Python it is a method on the left query.

Pair each issue with its owner Element from a people table.

def group "people" {
    def person "dy"  { str name = "Dy";  str email = "dy@kinogaki.com" }
    def person "ana" { str name = "Ana"; str email = "ana@kinogaki.com" }
}
auto issues = doc.query("/tracker/*").where(byType("issue"));
auto people = doc.query("/people/*");
std::vector<JoinPair> pairs =
    joinOn(issues, "owner", people, "name", JoinType::Inner);
for (const JoinPair& row : pairs) {
    // row.left is an issue, row.right the matching person
}

The join kind picks which unmatched rows survive. Inner keeps matches only. Left adds unmatched left rows (the right side is null); Right adds unmatched right rows (the left side is null); Outer keeps both. Anti keeps only the unmatched left rows, which finds issues whose owner names no person.

auto orphans = joinOn(issues, "owner", people, "name", JoinType::Anti);   // dark_mode (null owner) and any unknown owner

joinByConnection follows a Prism connection as a foreign key, the model's native reference. It pairs each left row with the Element feeding its .slot input: the source of the connection into left.slot. It runs in O(L) through the Document's incoming index. A left row whose slot has no incoming connection does not match. Where an issue connects its assignee slot to a person's output, follow that wire:

auto issues = doc.query("/tracker/*").where(byType("issue"));
std::vector<JoinPair> assigned = joinByConnection(doc, issues, "assignee");
// row.left is an issue, row.right the person feeding its .assignee slot

Going faster

A query scans the Document each time a terminal runs. For repeated equality lookups on one field, a value index answers in O(matches) instead of the O(n) scan. For folding a column across many rows, a columnar read vectorizes the scan. Both are opt-in accelerators over the same query semantics.

The query layer is the read surface over the Document: build with doc.query, narrow with predicates, fold or join, end with a terminal. For null and constraint rules see schema; to speed a hot query see indexes and columnar.

‹ Composition & references Indexes ›
© 2025 Kinogaki. Made with ♥︎ by Dy Mokomi