Design

Architectural decisions

The reasoning behind each load-bearing decision in Core: the use case that drove it, a counter-example, the alternatives considered, and why they lost.

Why it's built this way

This is the long version of why. The model, format, and reference pages tell you what Kinogaki Core does. This page tells you the thinking behind each load-bearing decision: the use case that drove it, a counter-example where it would be the wrong call, the alternatives that were on the table, and why they lost.

Two principles run through it. The model is small and the format is honest. A foundation others build on can afford to do less, but it cannot afford to lie: a header that promises more than the implementation delivers is worse than one that says less. And the common case is clean while the rare case stays possible. The happy path reads like prose; the escape hatch sits off to the side, there when you need it.

Read top to bottom for the whole argument, or jump to the decision you are questioning.

Identity is the Path. There are no UUIDs

The decision. An Element is identified by its place in the tree, /world/props/bolt, and by nothing else. There is no hidden id. Rename or reparent an Element and its Path changes; every connection endpoint and asset key that referenced it is rewritten in the same operation.

Why. The Path is already meaningful, already unique, already hierarchical, and already readable. Reusing it as identity makes a saved Document complete: no id table to keep in sync, nothing to collide, nothing to leak across files. Two people editing the same scene produce a diff a human can read. def "bolt" moved; an opaque id did not change parent. The text is the truth, and a UUID is truth nobody can read.

Use case. A writer renames a chapter in Atlas. Every paragraph-to-beat link and every word-to-entity span follows automatically, because doc.rename(from, to) rewrites Paths and endpoints together. The novel.json diff shows the rename and nothing spurious.

Counter-example. Sometimes an object must keep one identity across renames, the way a database row stays "the same row" whatever you call it. There the Path is the wrong identity. Prism is for documents whose structure is the meaning, not for entity stores where the name is incidental.

Alternatives considered. Stable UUIDs, the choice in many scene graphs: identity survives renames, but every file carries an id table, diffs turn unreadable, and ids collide or leak when files merge. A name plus a generation counter: it solves nothing the Path does not, and adds bookkeeping. The Path won because it makes the saved file self-contained and the diff legible, and because rename-rewrites-everything is one testable operation instead of a distributed invariant.

One Value type: a dtype and a shape

The decision. Every value in the Document is a Value: an element dtype (bool, char, the signed and unsigned ints, float16/32/64, str) over a shape (empty is a scalar, {N} is a run, {M,N,…} is a multi-dimensional array) on one flat buffer. A radius, an albedo, a 4×4 transform, a mesh's million points, a string, an animation key: all one type. Vec3, a matrix, a Spectrum are not separate types. They are fixed float32 shapes with names.

Why. A renderer needs float[N,3] point arrays. A tracker needs an int weight. A document needs strings. A shader needs a 4×4 double matrix. A fixed enum of "the types we thought of" runs out the moment someone needs the type you did not. A dtype plus a shape is the model NumPy and the array languages use, because it spans scalars, vectors, and tensors uniformly. One serializer, one interpolator, one comparison, for everything from a bool to an 8×8 float64.

Use case. A mesh is point3f[] points, int32[] indices, float16[] per-vertex weights. All the same Value machinery, all diffable, all interpolating uniformly under animation.

Counter-example. A value that is genuinely a record, a struct of named fields of different types, is not one dtype over one shape. Prism models that as a child Element with Properties, not as a single Value. See the dictionaries decision.

Alternatives considered. A closed enum of role types (Float, Vec3, Matrix, Spectrum), the model Core actually started with. It read nicely but could not store a uint16 array or a float64 matrix without a new enum case and a new code path each time. It was a list of guesses. A tagged union over a dozen C++ types: every reader pays a visitor, and arrays of arbitrary rank still do not fit. The dtype-and-shape model covers both. The role types survive only as named shapes (below), sugar over the one container.

Named types are a label over a shape, and the catalog is fixed

The decision. A Property can declare a named type, vec4f, matrix4d, color3f, instead of a bare dtype and shape. The name resolves through one built-in catalog (there are no user-defined aliases) and is stored on the Property, so two types that share a representation stay distinct.

Why store the name. vec4f and quatf are both float32×4. point3f, normal3f, and color3f are all float32×3. A renderer tells a normal from a color; a tool round-trips the author's intent. Keep only the dtype and shape and the name collapses on save, and that intent is lost. So the name rides on the Property as a label. The value underneath is still the catalog's dtype and shape.

Why a fixed catalog. A shipped, closed vocabulary resolves the same way in every Document and every tool. There is no per-file alias table to read first, no scoping rules, no two files disagreeing on what vec3 means. The architecture stays clean because there is exactly one dictionary: a lot of types, all known, none invented per file.

Use case. matrix4d xform = … reads like the 3D type it is. A DCC importer keys off point3f against color3f to tell which float32×3 is geometry and which is shading.

Counter-example. A throwaway scalar nobody introspects does not need a name. float is fine. The named layer earns its keep where the meaning of a shape matters to a reader.

Alternatives considered. Pure shape aliasing (write vec4f, store only the shape): lossy, because quatf becomes vec4f on the next save. A semantic "role" field on the Value: it bloats every value with a field most do not use, and it changes value equality. User-declarable aliases (alias mat = float64[4,4]): flexible, but it adds scoping and cross-file disagreement for little gain. A name on the Property, drawn from one fixed catalog, keeps round-trips lossless and the model lean.

float16 sits last in the enum

The decision. When float16 joined the dtypes, its code went at the end of the list rather than between uint64 and float32 where it reads naturally.

Why. The binary format writes a dtype as its numeric code. Insert a value in the middle and every later dtype renumbers, so every .prism file written before would misread float32 as something else. Appending keeps every existing code stable. The cost is a one-line "this is the highest code" bound in the reader and a list that is not grouped by family.

The lesson it stands for. The name is the contract a human reads. float16 everywhere: text, API, docs. The numeric code is an internal detail. Cleanliness goes where it shows; stability goes where it does not. A prettier ordering was not worth invalidating files.

Counter-example. Before any files existed, renumbering for a tidy list would have been free and correct. The decision is a function of when. Once data exists, layout stability outranks source aesthetics.

Connections: the consumer names its source, and it is one-to-many only

The decision. A connection is written connect <target> = <source>. The consumer (target) is on the left and reads from the producer (source) on the right. Each target slot has at most one source: many-to-one is not allowed. One source drives many targets, and a * glob in the target reaches them all in one line, declarable anywhere.

Why the consumer side. This is where the meaning lives and where USD puts it. A material binding is a fact about the geometry ("my material is this"). An input reads from one output. Writing the connection on the consumer keeps the producer ignorant of who listens, and keeps value resolution unambiguous: when an evaluator asks what feeds an input, there is exactly one answer.

Why no many-to-one. Two sources into one input is a value conflict with no defined winner. Forbidding it makes the dataflow well-defined. When a domain reads as many-to-one, many geoms and one material, you invert it. The producer names its targets with a glob: connect </geo/*.material> = </mat/Plastic.out> is one source to many targets, still one source per target.

Why a glob, stored lazily. Binding a material to a hundred geoms by editing a hundred Elements is the bulk edit a format makes a one-liner. The glob is stored verbatim and matched against the Document on read, so geometry added later under /geo is bound automatically. The link is a living query, not a snapshot.

Use case. A shader graph: connect roughness = </noise.out> on the node that consumes it. A look: one connect </geo/*.material> = </mat/Plastic.out> in a MaterialAssignments Element binds the whole set.

Counter-example. Many-to-many wiring has no single owner. An object lit by several lights, each light lighting several objects, is modeled as one chosen direction plus explicit membership, not as a single connection. And a connection target is a real slot (/geo/foo.material), so slot names are identifiers. A slot literally named "in slot", with a space, is not expressible.

Alternatives considered. A source-side out -> [targets] with a new arrow operator: prototyped and rejected. The -> read foreign next to a format that is otherwise name = value, and authoring a binding on the producer is backwards from how USD and people think about it. A separate "relationship" concept alongside connections: two mechanisms for one idea, a directed link to one or more targets. The unified connect directive is the single mechanism. One keyword, target on the left, a bare slot when relative or a <path> (with globs) when absolute. A glob is simply how it reaches many.

Time lives on the Property, and the curve is held, linear, or bézier

The decision. Every Property is a default Value plus an optional set of time samples. Reading at a time interpolates the samples when animated, and returns the default when not. The sample interpolation is held, linear, or bézier (with handles).

Why. Animation is not a separate kind of data. It is the same Value, sampled over time. Putting time samples on the Property, rather than in a parallel animation system, makes a static value and an animated one the same type, read the same way, serialized the same way. Turning animation on is adding a sample, not migrating to a different object. Bézier handles live in Core because the editor needs a curve it can shape, not only held and linear.

Counter-example. A value that is a function, not a sampled curve, a procedural noise driving a parameter, is not time samples. It is a connection to a node that computes it. Time samples are for authored keyframes.

Alternatives considered. A separate animation track table keyed by Path: it doubles the addressing, and a renamed Property orphans its track. Linear-and-held only, as in USD's core: simpler, but a DCC needs bézier, and adding it later would change the keyframe wire format. Samples on the Property, with handles from the start, keep animation a property of the value and the format stable.

Composition is references and overlay layers, not variants, payloads, or inherits

The decision. Cross-file power comes from two mechanisms. A reference pulls another Document's subtree in at read time. overlay stacks sparse layers (a proposal, an opinion) on a base, matched by Path, later-wins, with delete, disconnect, and reorder tombstones. diff is the inverse of overlay. There are deliberately no variants, payloads, inherits, or specializes.

Why these two. References give multi-file assets that resolve fresh every read, never baked away. Overlays give non-destructive proposals: an agent suggests, a human commits, provenance survives. Together they cover "build from parts" and "edit without losing the original", which is the collaboration story Prism is for. diff and overlay being exact inverses power the server-driven-UI patch plane: re-render the view, diff against the last sent view, ship only the layer.

Why not the rest. Variants, payloads, and inherits are USD's heavyweight composition. Variant sets switch shading or LOD; payloads defer streaming of huge scenes; class inheritance broadcasts edits. They are powerful and they are a lot of machinery, with subtle opinion-resolution rules. For a format meant to be read in an afternoon, taking them on trades the point for parity. They are a deliberate non-goal, and the header says so plainly.

Use case. A proposal: overlay(base, proposal) previews an edit; committing writes the result as plain local opinions, references intact. A bulk look: a referenced library plus an overlay of assignments.

Counter-example. A 100,000-asset city scene that must stream on demand wants payloads. Prism would load it all. If that becomes a goal, payloads get reconsidered, as an explicit feature with its own design, not smuggled in.

Alternatives considered. Full USD-arc composition, rejected for weight, as above. No layering at all, where a Document is only itself: it loses the proposal-and-commit workflow that makes agents and multiplayer work. References plus overlay is the minimum that delivers multi-file and non-destructive editing without the arc zoo.

diff and overlay round-trip exactly, including reorders

The decision. overlay(base, diff(base, target)) reproduces target exactly, Element and value order included. A property change becomes an over opinion with removal tombstones, not a delete-and-recreate. A pure reorder of named siblings becomes a reorder directive. Connection order is inert: connections compare as a target-to-source map, not an ordered list.

Why. The patch plane ships only the diff and applies it on the other side. If the round-trip were not exact, the client's view would drift from the server's. "Exact" is the load-bearing invariant, so every way a Document can differ must be expressible as a layer: a changed value, a dropped key, a reordered child, a re-sourced wire. The reorder move op exists because appending at the end cannot say "this child moved here". Connection equality ignores order because re-sourcing a wire moves it in the list without changing meaning, and a target has one source anyway.

Counter-example. When you want only the squashed current state and not a history of layers, you do not need the layer vocabulary. apply collapses a patch into a clean standalone base. The exactness matters when the layer itself is the thing you transmit or keep.

Alternatives considered. Coarse diffs that replace a changed Element wholesale: simpler to compute, but they lose order and balloon the patch. Ignoring reorders, the original behavior: it left a real case where the round-trip silently dropped a sibling reordering. That is unacceptable for a patch-plane promise, so the move op was added rather than letting the contract lie.

The C ABI is the one foreign-language seam

The decision. Every non-C++ consumer, Python today and Rust or Swift later, goes through a flat C ABI: opaque handles, scalar and array setters and getters, a general (dtype, shape, bytes) escape hatch, and serialize-and-load. Python is a thin binding over it, not a separate C++ binding.

Why. A C ABI is the one interface every language calls without a C++ compiler, without name-mangling, without ABI fights. Widening it benefits every binding at once; a per-language C++ binding re-solves the same problem and drifts. The general set_array and get_array pair makes the ABI as expressive as the C++ model, any dtype at any shape, without a function per type. The typed setters are there for the common case, the general one for completeness.

The discipline it imposes. A borrowed string is valid only until the next mutation, so nothing transfers ownership across the boundary. A buffer the caller frees is explicit. A bool reports what happened, so a connect that made no wire returns false. The ABI says less, and it does not lie, and that honesty is what other languages depend on.

Counter-example. Inside C++, use the real API (Document, Value, handles) directly. The C ABI is the foreign seam, not a wrapper C++ code routes through.

Alternatives considered. A native C++ Python module (pybind11): ergonomic for Python, but it is a compiled artifact per Python version, it does not help Rust or Swift, and it couples the binding to C++ build details. The binding over the C ABI is version-agnostic, portable, and shares one contract with every future language.

Two serializations, both honest about versions

The decision. A Document writes as .prisma (sorted, diff-clean ASCII) or .prism (a binary form), and both read back exactly. A reader accepts a set of versions. The format grows by additive change: a new optional section under a bumped version, with the readable floor left in place so old files keep loading. The floor rises only on a layout change the current reader can no longer parse.

Why two. Text is for people, diffs, and review. Binary is for size and speed on large arrays. They stay in lock-step, so the same Document round-trips identically through either, and you never choose between "readable" and "real". The sorted, diff-clean text is deliberate: a value re-serialized produces the same bytes, so version control shows only true changes.

Why a version seam. A frozen foundation outlives its first format. Pin the reader to exactly the current version and the first future file breaks every old reader. A documented readable-version set with an additive-change rule lets the format grow without a flag day. The contract is written down, so the rule is enforceable instead of folklore.

Counter-example. A one-shot, never-persisted, same-build wire message does not need the version seam's care. It costs nothing there, and the same code serves the persisted case.

Alternatives considered. Text only: it loses binary's size and speed on geometry. Binary only: it loses readable diffs, the format's whole pitch. Exact-version pinning: simplest, but it makes the format un-evolvable without breaking files. Two lock-step encodings plus a readable-set seam keeps both audiences and a future.

Assets are a sidecar, and you convert into Prism on purpose

The decision. Opaque bytes, textures, fonts, baked meshes, live in an asset store: a byte-blob sidecar keyed by the owning Element's Path, packaged with the Document as one self-contained .prism. There is no asset value type pointing at an external file. To store an image, you convert it into Prism.

Why a sidecar. The numeric graph and the opaque bytes have different lifetimes and different tooling. Keeping blobs in a Path-keyed store, with the same rename and reparent fix-up the connections get, keeps the Document model pure numbers and the package one shippable file, the way usdz does.

Why explicit conversion. This is the RenderMan .tx philosophy. An asset enters the system by being converted into the system's form, on purpose, by tooling, not by a path string the format hopes a resolver will find. Dependencies stay explicit and a package stays self-contained. The cost is a conversion step the user's tooling owns.

Counter-example. A pipeline that must reference live external files by path, a shared texture server or assets too large to embed, wants an asset-typed reference and a resolver. Prism's bet is the opposite: bring it in, or do not. That bet is wrong for a reference-heavy DCC pipeline and right for a portable, self-contained document.

Alternatives considered. An asset value type (USD's @path@): convenient, and on the table, but it makes the Document depend on an external resolver and files that may move. And reference already covers "pull in another Prism Document". Keeping storage explicit chose self-containment over convenience.

Codecs share one document model, and round-trips are identity

The decision. Markdown, HTML, and JSON are codecs over a shared document model: heading, paragraph, list, table, inline runs. A Document read from one format and written back to the same format is faithful to the structure. Degradation to a weaker format is defined and documented.

Why. Foreign formats become first-class Prism Documents. A Markdown file is a typed tree you can query and diff, not an opaque blob. One model means many formats interoperate through a common center instead of one converter per pair. The within-format identity contract (text with *, _, |, or a leading # still round-trips) makes the codecs trustworthy: an edit-and-resave does not silently corrupt content.

Counter-example. A format with no structural analog in the model is a blob, not a codec target. Codecs are for content with a tree.

Alternatives considered. Direct format-to-format converters: quadratic and inconsistent. No canonical model, storing Markdown as a string: it loses the queryable tree that is the point. A shared document model with a stated lossiness table per format is the honest middle.

Structured data is the tree, not a Value

The decision. Nested, typed, record-like data is a child Element with its own Properties, not a recursive dictionary Value. A Value is a dtype over a shape; it does not nest.

Why. The Document already is a typed tree of named, typed slots, which is what a dictionary is. A recursive dictionary value would build a second tree inside the value model, with its own serialization, equality, and query story, duplicating the Element tree. Using the tree keeps one mechanism.

Counter-example. A free-form metadata bag, arbitrary nested config a tool attaches and a human never models as structure, is where a typed dictionary value would shine. Modeling it as Elements is heavier than it deserves. That case is deferred, on purpose, until it is needed.

Alternatives considered. A recursive dictionary Value: the clean general answer, but a large addition (a recursive value model touching every serializer, equality, and the bindings) for a need the Element tree already serves. Deferred to a later version rather than carried speculatively.

The query and analytics engine lives in Core, and every binding shares it

The decision. Reading a Document as data, filtering, ordering, aggregating, joining, indexing for equality and nearness and text and vectors, the vectorized columnar scan, runs in kinogaki-core (C++). A binding does not re-implement any of it. The Python doc.query(...) records the recipe and hands it to the native engine through the C ABI; the filtering and folding happen in C++, and Python reads back the result.

Why. A query engine is exactly the kind of thing that drifts when it is written twice. Two implementations of "numbers compare by widened value, a missing field is null, ordering is stable" are two chances to disagree, and the disagreement surfaces as a binding that returns different rows than the library. One engine, reached through the one seam, means the semantics are defined in one place and proven by one test suite. It is also where the speed is: an index lookup or a columnar fold is a tight C++ loop, and a Python reimplementation would be an order of magnitude slower over the same data, which defeats the point of having the accelerator at all.

Use case. A Python tool filters ten thousand issues by status, groups by owner, and sums the votes. Every comparison, the grouping, and the fold run in C++; Python sends the predicate and reads the map back. The same call in C++ runs the same code.

Counter-example. A user-supplied predicate is the one thing the binding cannot ship to C++: a Python lambda is Python code. So where(lambda ...) is the explicit escape hatch, and it runs in Python over the candidate set the native predicates already filtered. The heavy lifting stays in the engine; only the user's own function runs where it was written.

Alternatives considered. A parallel query layer in each binding, which is where the Python package started: it worked, but it was a second engine to keep in step, with no indexes and no vectorized scan, so it was both a drift risk and slow. Binding the C++ engine instead made the binding smaller, faster, and correct by construction. The cost is a wider C ABI, paid once, shared by every future language.

A rename-stable NodeId, in memory only and never in the file

The decision. Each Element carries a NodeId, an integer identity that survives a rename. It is assigned on load, excluded from value equality, and never serialized. The saved file still has no id table and no UUIDs (identity is the Path); the NodeId exists only for the lifetime of an in-memory Document and its copies.

Why. Two jobs need to recognize "the same Element" across an edit that moved it. Move detection (renames) tells a rename from a delete-plus-add, and a move-aware merge keeps an edit that landed on a renamed Element. The Path cannot answer "are these the same Element" once the Path itself changed, which is the whole point of a move. A runtime id can, as long as the two Documents share lineage (one is a copy-then-edited version of the other). Keeping it out of the file preserves the no-id-table invariant that makes a saved Document self-contained and its diff legible.

Use case. A writer renames a chapter, and a concurrent edit set a property on it. renames(base, edited) reports the move, and the merge applies the property edit to the chapter at its new Path instead of dropping it as an edit to a deleted Element.

Counter-example. Two Documents loaded independently number their ids from scratch, so their NodeId spaces are unrelated. Comparing them by NodeId is meaningless; compare them by diff. The NodeId is a within-lineage identity, not a global one.

Alternatives considered. Serializing the id (a real UUID in the file): it would survive a save, but it brings back the id table, the collisions on merge, and the unreadable diff the Path decision exists to avoid. A pure path-diff with no id: simplest, but it cannot recover a move, so a rename reads as a delete and an add and a concurrent edit to the moved Element is lost. An in-memory-only id is the narrow addition that buys move detection without touching the file.

A Document copies in O(1), and an edit un-shares

The decision. Copying a Document is copy-on-write: the copy shares the original's structure and duplicates nothing, and the first mutation on either side un-shares only what it touches. A snapshot is a pointer until something diverges.

Why. The version story rests on snapshots being cheap. diff wants a before and an after; an event log keeps a stack of past states; a merge holds a base and two edits. If a copy duplicated a fifty-thousand-Element Document every time, holding history would be the dominant cost and the whole versioning model would be too expensive to use. Copy-on-write makes a snapshot free until it is edited, so keeping the before-state around costs nothing in the common case where it is never touched again.

Use case. An editor keeps an undo stack of past Document states. Each is an O(1) snapshot; only an Element actually edited duplicates, and only the part of the structure on the path to it.

Counter-example. A Document you copy and then immediately rewrite wholesale gets no benefit, since the first broad edit un-shares it all. Copy-on-write wins when copies outnumber divergences, which is the versioning case it is built for.

Alternatives considered. A deep copy on every snapshot: simple and predictable, but it makes history proportional to size times depth, too costly to lean on. An explicit transaction or journal object: it would track changes, but it adds a second model beside the value-type Document, and the point of a Document is that it is a plain value you can copy. Copy-on-write keeps the value semantics and makes them cheap.

Indexes are snapshots, off the Document mutation path

The decision. A FieldIndex, SpatialIndex, TextIndex, or VectorIndex is built once over a Document, borrows its Elements, captures the field values at build time, and does not track later edits. Change the Document and you rebuild the index. The index is an opt-in accelerator that sits beside the Document, never on the path that mutates it.

Why. An index that auto-updated on every edit would put itself on the Document's write path: every set would have to find and patch every index that might cover the field, which makes a plain edit pay for accelerators it may not have, and couples the mutation code to every index type. A build-once snapshot keeps editing simple and fast, and it fits the model: a Document is a value you hold, and an index over an unchanging value stays correct as long as you hold it. The cost is a rebuild after an edit, which is the honest price of keeping the write path clean.

Use case. A read-heavy view builds a FieldIndex over status once and answers thousands of equality lookups in time proportional to the matches, not the Document size. When the view's Document is replaced by an edited one, it builds a fresh index.

Counter-example. A workload that interleaves one edit with one lookup, over and over, gets no value from an index: it would rebuild as often as it queries. There the plain query scan, which always sees the live Document, is the right tool. Indexes earn their keep when reads on an unchanging Document vastly outnumber edits.

Alternatives considered. A live, self-maintaining index hooked into every mutation: ergonomic, but it taxes every write and entangles the index types with the core edit path. A copy-on-write index that versions with the Document: more machinery than the accelerator is worth. A build-once snapshot, rebuilt after an edit, keeps indexes a thing you reach for, not a tax you always pay.

The small decisions, briefly

A few choices are not headline features, but they shape how the library feels.

• Anonymous Elements (/parent/[i], addressed by index): a converted document's paragraphs and runs do not have to invent names like "0" and "1". Ordered, nameless children take an index identity, so structural content stays clean while named content stays addressable.
• Append is O(1). Building a Document indexes only the new Element; a full re-index runs on remove and rename. Assembling a 50,000-Element Document, the native load path, stays linear.
• Strict getters coerce; bools are honest. p.requireFloat(name) accepts any numeric scalar (an int64 satisfies it) rather than demanding an exact dtype, because that is what callers mean. A mutation bool reports whether the mutation happened, so a no-op is distinct from a success.
• Types are tokens, and a schema is optional. An Element's type is a free string, and the substrate enforces no shape: a Document is valid without any schema. Core ships an opt-in Schema a consumer can declare and validate against (required fields, dtype, uniqueness), but nothing imposes it on the model. The validator is a layer you reach for, not a contract the substrate demands, so the model stays a substrate and not a framework.
• The strata do not leak. Core knows nothing of "novel" or "render" or any kind. It defines the substrate (Elements, Values, Properties, connections, time, composition, serialization) and the codec seam, and the kinds live above it. That separation is why the same Core carries a manuscript, a 3D scene, and an issue tracker.

If a decision here ages badly, that is the right thing to record too. This page makes the reasoning inspectable, so a future change argues with the argument instead of guessing at it.