Schema, validation & null

Null is absence

Prism states only what is present. An Element carries the Properties it has, and nothing stands in for the ones it does not. There is no separate null Value. An absent Property is null, and that is the whole of it.

def issue "bug-1" {
    str status = "open"
    int weight = 3
}
def issue "bug-2" {
    str status = "open"
    # no weight: weight is null on this issue
}

weight is null on bug-2 because the Property is absent. Test presence with the two predicates on a field:

doc.query("/tracker/*").where(field("weight").isNull());    // every issue with no weight
doc.query("/tracker/*").where(field("weight").notNull());   // every issue that has one

doc.query("/tracker/*").where(field("weight").is_null())    # every issue with no weight
doc.query("/tracker/*").where(field("weight").not_null())   # every issue that has one

notNull() (not_null() in Python) is exists(): the Property is present, with any Value. isNull() (is_null()) is its complement.

Three-valued comparison

A compare against a null field is unknown, and the query layer is honest about it. The rules follow from one idea: a positive test of an absent Property cannot be true.

A positive compare (eq, lt, gt, contains, startsWith) against a null field does not match. There is nothing to compare.
ne matches present-and-unequal only. A null field does not match ne. (Absence is not "unequal", it is unknown.)
!field(x).eq(v) does include null. It is the negation of "matched eq", and a null field never matched, so it falls into the negation.
Aggregations (sum, avg, min, max, count) and joins skip null fields. A row missing the field contributes nothing.

The pair to watch is ne against !eq. They differ exactly on null:

doc.query("/tracker/*").where(field("status").ne("closed"));      // present AND not "closed"
doc.query("/tracker/*").where(!field("status").eq("closed"));     // the above, PLUS rows with no status

doc.query("/tracker/*").where(field("status").ne("closed"))       # present AND not "closed"
doc.query("/tracker/*").where(~field("status").eq("closed"))      # the above, PLUS rows with no status

Pick deliberately. ne keeps rows that say something other than "closed". !eq keeps everything that is not "closed", which includes the rows that say nothing at all. Reach for isNull() / notNull() when presence is the question itself.

A schema is optional

Prism is schema-optional, and the per-element flexibility is a feature. The same container holds a 3-D object, a document heading, and a tracker issue, because the model stays neutral on shape. A Schema is the opt-in: a consumer declares the shape it expects, then validates a Document against it.

A Schema is a set of TypeRules, one per type token. Each TypeRule lists FieldRules, one per Property the type should carry. A FieldRule names a Property, its dtype (one of the Value dtypes: bool, the integer and float widths, str), and two flags:

required: the Property must be present (not null).
unique: no two elements of this type may share the value.

In C++ a Schema is a set of TypeRules; in Python it is built by chaining field(type, name, dtype, ...), where the dtype is a name like "i64" or "str" and the first argument is the type token.

#include "kinogaki/Schema.h"
using namespace kinogaki;

Schema schema{
    .types = {
        TypeRule{
            .type = "issue",
            .fields = {
                FieldRule{.name = "id",     .dtype = DType::I64, .required = true, .unique = true},
                FieldRule{.name = "status", .dtype = DType::Str, .required = true},
                FieldRule{.name = "weight", .dtype = DType::I64},
            },
        },
    },
};

schema = (kg.Schema()
          .field("issue", "id", "i64", required=True, unique=True)
          .field("issue", "status", "str", required=True)
          .field("issue", "weight", "i64"))

A Schema targets elements by type token. Every element whose type is issue is checked against this rule; an element of any other type is unconstrained and passes. A type with no rule is unconstrained. v1 checks dtype, presence, and uniqueness; it does not check shape.

validate returns Violations

validate(doc, schema) walks the document and returns a vector<Violation>. An empty result means the document is valid. Each Violation names the offending element by Path and says what is wrong.

In C++ validate is a free function; in Python it is a method on the Schema.

std::vector<Violation> bad = validate(doc, schema);
for (const Violation& v : bad)
    std::printf("%s: %s\n", v.element.str().c_str(), v.detail.c_str());

for v in schema.validate(doc):
    print(f"{v.element}: {v.detail}")

For a document where one issue omits the required status, one stores weight as a string, and two share an id, the call reports each:

/tracker/bug-2: missing required property 'status'
/tracker/bug-3: property 'weight' is str, expected i64
/tracker/bug-4: duplicate value for unique property 'id' (also at /tracker/bug-1)

Fix the writes, validate again, get an empty result.

Why declare a schema

A declared schema earns its keep two ways.

It catches malformed writes. A required field left out, a weight stored as text, a duplicate id: validate finds them before they reach the rest of the system, at the seam where a write enters.

It lets the fast paths assume one dtype per column. When a Schema guarantees that every issue stores weight as an i64, the columnar layout and the value index can pack that column as one dtype, with no per-row type check. The constraint a schema enforces is exactly the invariant those accelerators want.

‹ Columnar tables Versioning & history ›