moczarr.convention
The morton-hive layout convention, read side (pure functions, no I/O).
The layout is owned by the mortie spec (espg/mortie#62); zagg writes it
(zagg.hive); moczarr reads it. Summary::
{store_root}/
morton_hive.json <- static manifest; root-only exception
coverage.moc <- root ranges MOC; root-only exception
{sign+base}/{d1}/.../{d_n}/ <- digit component(s) per level
{full_id}.zarr/ <- vanilla zarr v3 leaf
{full_id}_{window}.zarr/ <- time-windowed leaf (morton-hive/2)
- Ids are morton decimal strings: sign + base digit (
1..6), then one digit1..4per order. A string prefix is a spatial ancestor. - Path components chunk the digit tail per the manifest's
path_grouping(spec §6.1; zagg D21):path_groupingdigits per component, the LAST component carrying the remainder when the order does not divide evenly (leading components stay full-width, so component boundaries are ancestor prefixes shared by deeper shards regardless of their order).1— the default, and every existing store retroactively — is a value of the one generic chunking, never a separate code path; readers chunk per the manifest, never by assumption. - Below the root a node holds only digit children and
*.zarrobjects (the node invariant); the manifest and rootcoverage.mocare the two root-only exceptions. - A leaf is complete iff its root zarr attrs carry the commit stamp
(
morton_hive_commit); an unstamped.zarr/prefix is debris. - Windowed leaf names split on the FIRST
_(morton ids and window labels never contain one); labels use the frozen charset[0-9A-Za-z-]{1,32}.
Everything here is arithmetic on ids and dict validation — the store layer
(phase 2) supplies the bytes. Golden vectors in tests/ pin this
implementation against zagg's writer so the two cannot drift silently.
area29_to_point(word)
Max-encoded POINT twin(s) of order-29 AREA word(s) — the p parse.
Inverse of :func:point_to_area29: area 28 + t28*5 + (t29 + 1) ->
point 48 + t28*4 + t29 (spec §1). Raises on any word that is not an
order-29 area encoding — points exist only at order 29 (§2/§4).
Source code in src/moczarr/convention.py
150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | |
check_node_invariant(rel_path, *, path_grouping=1)
Raise unless rel_path is a legal hive leaf path.
Below the root only digit components are allowed — {sign+base}
(optional -, one digit 1..6) at the first level, then digit
(1..4) components chunked per path_grouping (spec §6.1): every
component full-width except the last, which carries the remainder —
terminating in {full_id}.zarr (or the windowed
{full_id}_{window}.zarr) whose id equals the concatenated
components. This is the walker's contract: any other name under the root
(bar the manifest and the root coverage.moc) breaks child
classification.
Source code in src/moczarr/convention.py
552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 | |
column_name(window=None)
The §4.6 column basename for a (node, window): {stem}.pyramid.zarr.
The stem derives from the window alone (§4.6 naming, the §4.2
overview dialect) — never from the leaf's own basename stem — so the
unwindowed / schedule: none case takes the reserved all-time token
(window=None -> all.pyramid.zarr) and a windowed leaf's column
is {window}.pyramid.zarr. An explicit window="all" is refused at
the same seam every window= entry point shares (espg/moczarr#30).
Source code in src/moczarr/convention.py
482 483 484 485 486 487 488 489 490 491 492 493 494 495 | |
column_path(shard, window=None, *, path_grouping=1)
Store-relative path of a shard's §4.6 leaf column.
The column is a sibling of the leaf under the leaf's own node prefix
(§4.6), so this is :func:leaf_path's node directory plus
:func:column_name — the same shard/grouping validation, the window
seam included. Naming an object says nothing about existence: columns
are derived artifacts a reader never requires (§4.6), so the caller
checks the commit stamp (:func:moczarr.column.read_column_record).
Source code in src/moczarr/convention.py
498 499 500 501 502 503 504 505 506 507 508 509 510 | |
decimal_base(decimal)
The {sign+base} component of a decimal id.
Source code in src/moczarr/convention.py
228 229 230 | |
decimal_order(decimal)
HEALPix order of a decimal id (one digit per level past the base).
Source code in src/moczarr/convention.py
223 224 225 | |
decimal_rank(decimal)
Base-4 value of a decimal id's digit tail (digits 1..4 -> 0..3).
The bit/rank convention of the coverage encodings: ascending packed-word (Z-)order within one base cell at a fixed order.
Source code in src/moczarr/convention.py
233 234 235 236 237 238 239 240 241 242 | |
group_digits(digits, path_grouping)
Chunk a digit tail into hive path components (spec §6.1, zagg D21).
Left to right, path_grouping digits per component; the LAST component
carries the remainder when len(digits) % path_grouping != 0. Leading
components stay full-width so every component boundary is a spatial
ancestor prefix (§2) shared by deeper shards regardless of their order —
mixed-order shards share ancestor nodes, the property the digit tree
exists for.
Source code in src/moczarr/convention.py
379 380 381 382 383 384 385 386 387 388 389 | |
is_base_component(name)
Whether name is a {sign+base}-shaped hive root child.
Source code in src/moczarr/convention.py
373 374 375 376 | |
is_column_basename(name)
Whether name is a spec §4.6 leaf-column basename.
The one normative name seam: a basename ending in
:data:PYRAMID_COLUMN_SUFFIX is a per-(node, window) column and
MUST NOT be read as a leaf or an overview. Suffix-only by design — the
frozen label charset admits no ., so no legitimate leaf or overview
basename can end this way, and the seam needs no stem parse.
Source code in src/moczarr/convention.py
470 471 472 473 474 475 476 477 478 479 | |
is_point_word(word)
Whether packed word(s) encode an order-29 POINT (spec §1/§4).
Kind is carried by the word's 6-bit suffix — 48..=63 is the point
band; everything below (0..=47) is an AREA element, exact at its
encoded order — never by store or array metadata (§4). Suffix-mask
implementation, golden-tested against the spec §1 table. mortie's public
predicate (mortie.is_point) DID land in the new floor — espg/mortie#116
shipped it in 0.9.1 — and it is bit-identical across all 64 suffix values,
pinned in tests/test_convention.py. The local mask stays anyway: it is
moczarr's independent read of the §1 table, and delegating would leave the
golden test asserting mortie against itself rather than against the spec.
Scalar in -> bool; array in -> bool array.
Source code in src/moczarr/convention.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | |
leaf_name(full_id, window=None)
The leaf zarr basename: {full_id}_{window}.zarr, or bare.
Source code in src/moczarr/convention.py
445 446 447 448 449 450 | |
leaf_path(shard, window=None, *, path_grouping=1)
Store-relative hive path of a shard's leaf zarr.
path_grouping is the manifest's digit-chunking (spec §6.1; default
1 — every pre-D21 store). At 1 the path is computed by mortie's
hive_path (the convention owner) and re-checked against the node
invariant so drift on either side fails loudly; the grouped form chunks
the decimal here (:func:group_digits) until mortie grows the grouped
hive_path (espg/mortie#62). window selects the time-windowed
leaf at the same node.
Source code in src/moczarr/convention.py
513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 | |
manifest_path_grouping(manifest)
The manifest's path_grouping (D21: absent reads as 1).
Assumes a :func:parse_manifest-validated manifest; the single accessor
keeps the absent->1 normalization in one place.
Source code in src/moczarr/convention.py
392 393 394 395 396 397 398 | |
morton_decimal(word)
Decimal morton string of a packed word (pass-through for strings).
POINT words (§1 suffix 48..=63) render with the terminal p kind
marker — the §2 render/interchange form — via the order-29 area twin's
digits (same path; kind restored by the marker, so the round-trip is
lossless for BOTH kinds, pinned by goldens).
Source code in src/moczarr/convention.py
204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 | |
morton_word(label)
Packed uint64 morton word of a decimal id (pass-through for ints).
Accepts the §2 p kind-suffix on full order-29 POINT ids: the stem
parses as the area word and the §1 point suffix is restored moczarr-side
(:func:area29_to_point). Not a capability gap — the supported mortie
floor parses p natively and to the same word — but the branch is
moczarr's own read of the §2 grammar, so a misplaced marker gets an error
naming the spec rather than the parser's, and the §4 tie-break below
stays visibly ours. Both forms are pinned bit-for-bit against golden
literals in tests/test_convention.py. An UNMARKED order-29 string
parses as the AREA word, the normative §4 tie-break. Rides mortie's
public scalar decimal_to_word (espg/mortie#114; issue #38 — the
deprecated private _decimal_to_word carried no compatibility
promise). This seam parses ONE label by contract; a caller with a batch
reaches for the batched mortie.decimal_to_word instead of looping here.
Source code in src/moczarr/convention.py
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 | |
normalize_subtree(subtree)
Normalize a subtree= argument to (word, decimal, order).
Both currencies of the readers' subtree= keyword (issue #29, mirroring
zagg's readers._layout pair): a packed morton AREA word (int) or a
decimal morton string (str). An UNMARKED string always yields the area
word (the §4 tie-break); a p-marked order-29 id parses as the POINT
word and is refused below, so no separate kind flag is needed.
Raises ValueError
on a malformed member of either currency — an unparsable string, an int
outside the uint64 range (not a packed word; parse a decimal id by
passing it as a string), an invalid packed word, or an order-29 POINT
word (kind rides
the §1 suffix; a subtree names an AREA ancestor, and points have no
descendants).
Source code in src/moczarr/convention.py
254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 | |
parse_manifest(payload)
Validate a morton_hive.json payload; returns it as a dict.
Loud on malformed input (a manifest is the reader's bootstrap — there is
no degraded mode without it): unknown spec, missing/non-integer
orders, a /2 manifest without its temporal block, or a temporal block
without a schedule all raise ValueError.
Source code in src/moczarr/convention.py
591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 | |
point_to_area29(word)
Order-29 AREA twin(s) of point word(s); area words pass through.
Spec §1 suffix arithmetic on the same body: point 48 + t28*4 + t29
-> area 28 + t28*5 + (t29 + 1). The twin shares the point's full
path, so containment arithmetic (§4: membership of a point at a coarser
level is ordinary truncation) runs uniformly in area space — the
normalization :func:moczarr.coverage.aoi_mask applies. Scalar in ->
int; array in -> uint64 array.
Source code in src/moczarr/convention.py
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | |
rank_tail(rank, depth)
Inverse of :func:decimal_rank: the width-depth digit tail.
Source code in src/moczarr/convention.py
245 246 247 248 249 250 251 | |
split_leaf_name(name)
(full_id, window-or-None) from a leaf basename — split on the FIRST _.
Morton decimal ids never contain _ and window labels cannot
(charset), so the first underscore is the one separator. Raises on a
non-.zarr name or a malformed window label.
Source code in src/moczarr/convention.py
453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 | |
subtree_cell_span(subtree, anchor, anchor_index, cell_order, n_cells, field, *, stacklevel=2)
Cell-index span [start, stop) of subtree on a nested cells axis.
The zagg spec §1.5 subtree-span identity, resolved by pure arithmetic
(issue #29; logic-parallel with zagg readers/_layout.subtree_cell_span):
a cell's axis position is its HEALPix NESTED id, so the order-k
subtree below a word with nested id h occupies [h·4^d, (h+1)·4^d)
(d = cell_order - k) in fullsphere coordinates. A single-root
power-of-four axis (a hive leaf's shard subtree) subtracts its root's own
span start, with the root derived from anchor — any WRITTEN cell word
(the morton anchor slice the caller already read); no store bytes are
touched here.
That identity is CHECKED, not assumed: anchor sits at anchor_index
on the axis, so nested(anchor) - root_start must equal
anchor_index or the axis is not in canonical nested placement and
this arithmetic would hand back plausible-but-wrong cells silently — a
mismatch raises ValueError.
The span is returned intersected with the axis: a word at or above the
axis root clips to the whole axis (every stored cell is its descendant),
and a well-formed word DISJOINT from the axis warns — naming the word and
the axis root — and returns the empty span (0, 0). An empty read is
therefore ambiguous between "in-domain, nothing stored" and "outside the
domain" unless the warning is observed; DELIVERY is the caller's
warnings filter's call (the default action dedups a repeat of the
same warning in the same process), so the guarantee is "at most once per
call". stacklevel is the caller's depth from here — the readers pass
the depth that attributes the warning to THEIR caller rather than to a
moczarr frame. Raises ValueError for a malformed word, or one deeper
than the cells-axis order (grammatically impossible — nothing stored
could be its descendant).
Source code in src/moczarr/convention.py
285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 | |
validate_label(label)
Validate a window label against the frozen charset; returns it.
Source code in src/moczarr/convention.py
401 402 403 404 405 406 407 408 | |
validate_window(label, *, where=None)
Validate an explicit window= label; returns it.
:func:validate_label plus the one exclusion the charset cannot carry —
the reserved all-time token :data:ALL_TOKEN. all satisfies the
grammar because it is a basename (the unwindowed leaf's, and the
sweep's all-time overview fold's), so a reader entry point that only
validates the charset resolves it to objects that are not that window's:
open_hive names {shard}_all.zarr leaves that do not exist and
quietly empties, while an overview order would open the all-time folds
beside a 0-cell source node. Three behaviors for one reserved token is an
API trap (espg/moczarr#30), so every window= seam refuses it here,
with one message.
Name arithmetic that legitimately spells all — the overview sidecar
stem in :func:moczarr.stats.stats_sidecar_key, and the telemetry
readers that address an all.zarr object by name — keeps calling
:func:validate_label: reading one record cannot mislead about coverage
the way an all-time dataset node would.
Source code in src/moczarr/convention.py
411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 | |