moczarr.column

Leaf column artifacts (zagg spec §4.6, zagg-column/1) — issue #36.

A column is the leaf worker's own pyramid contribution, written at aggregation time while the shard's cell data is resident: one zarr per (leaf, window), a sibling of the leaf under the leaf's own node prefix ({window stem}.pyramid.zarr — the stem from the window ALONE), holding one single-chunk, unsharded group per resolution in the {order}/ {field} layout — every declared leaf resolution, every within-footprint rung of the fixed ladder, and the node-order member (the leaf's whole-footprint aggregate, its universal partial for every coarser cell). On the published ATL03 store that is groups at orders 13 down to 9.

The reader posture is §4.6's, twice over:

• Columns are derived artifacts a reader never requires — absence (or an unstamped prefix, debris) is never an error, and the manifest's pyramid block MAY lag what the fleet actually wrote, so which leaves carry columns is answered by the columns themselves (:func:moczarr.store.walk_columns, :func:read_column_record), not the manifest.
• A column MUST NOT be read as a leaf or an overview (the normative .pyramid.zarr name seam); classification is the role / zagg_column root attrs, checked here.

Both readers here are node-addressed: a SHARD id names the §4.6 leaf column, and an ANCESTOR node id names an issue-#384 stage column — the same artifact shape (zagg-column/1 attrs, D4 stamp, {window stem}.pyramid.zarr under the node's prefix) written at a staged sweep's dispatch nodes, whose groups are pure gathers (regime: "stage-gather") plus the mandatory relay member at shard_order. Addressing one is a DIRECT read of an object the caller already names (espg/moczarr#36b): stage-column existence at a given order stays orchestration, never contract ("a reader binds to the ladder artifacts of §4.4, not to stage columns" — §4.6), so nothing here enumerates or requires them — absence is the usual None — and no assembly surface consumes them (:func:moczarr.level.open_level binds to §4.4 artifacts). Discovery remains :func:moczarr.store.walk_columns, which classifies by suffix at every depth.

Reads go through the normal ragged/dense read path: :func:open_column returns the same read-only zarr store shape :func:moczarr.open.open_leaf does, so a dense field opens with zarr.open_array(store, path="13/count") and a ragged digest streams through :func:moczarr.ragged.read_ragged / :func:moczarr.ragged.read_cell at "13/h_tdigest_signal" — companion channels (locations=/times=) included, because a column group carries every sibling its field's §4.5 entry declares, exactly as a leaf does. Full multiscale assembly (a resolution-aware view over columns + overviews) is deliberately NOT here — that is espg/moczarr#36b, sequenced behind the #37 data-model ruling.

column_orders(record)

A column record's resolution-group cell orders, finest first.

The integer keys of the §4.6 groups provenance map — the group set actually on disk (every declared leaf resolution, the within-footprint ladder rungs, and the node-order member). The finest is the group whose populated-cell count the commit stamp records (cells_with_data_order).

Source code in src/moczarr/column.py

def column_orders(record: dict) -> tuple[int, ...]:
    """A column record's resolution-group cell orders, finest first.

    The integer keys of the §4.6 ``groups`` provenance map — the group set
    actually on disk (every declared leaf resolution, the within-footprint
    ladder rungs, and the node-order member). The finest is the group whose
    populated-cell count the commit stamp records
    (``cells_with_data_order``).
    """
    return tuple(sorted((int(k) for k in record.get("groups") or {}), reverse=True))

open_column(store_root, shard, *, window=None, product=None, manifest=None, anonymous=False, store=None, **store_kwargs)

Open ONE node's §4.6 column as a read-only zarr store.

The column twin of :func:moczarr.open.open_leaf — same parameters, same credential/anonymous handling, same validation (the window seam, the node-order check, D19 product= re-rooting), same deliberately bare return: a read-only zarr.storage.ObjectStore rooted at the column, for the per-leaf readers at "{order}/{field}" paths — zarr.open_array for dense fields, :func:moczarr.ragged.read_ragged / :func:moczarr.ragged.read_cell for ragged digests (companion channels included). shard is the column's node id: a shard id opens the leaf column, an ancestor node id the issue-#384 stage column at that dispatch node (a direct read — the module docstring's orchestration-not-contract caveat applies). The available group orders and fields come from :func:read_column_record, which also answers whether the column exists at all — opening names an object and never checks (§4.6 absence is never an error; the first read misses). store= shares a handle for the manifest GET only, exactly as open_leaf does.

Source code in src/moczarr/column.py

def open_column(
    store_root: str,
    shard: str | int,
    *,
    window: str | None = None,
    product: str | None = None,
    manifest: dict | None = None,
    anonymous: bool = False,
    store: Any = None,
    **store_kwargs: Any,
):
    """Open ONE node's §4.6 column as a read-only zarr store.

    The column twin of :func:`moczarr.open.open_leaf` — same parameters,
    same credential/``anonymous`` handling, same validation (the window
    seam, the node-order check, D19 ``product=`` re-rooting), same
    deliberately bare return: a read-only ``zarr.storage.ObjectStore``
    rooted at the column, for the per-leaf readers at ``"{order}/{field}"``
    paths — ``zarr.open_array`` for dense fields,
    :func:`moczarr.ragged.read_ragged` / :func:`moczarr.ragged.read_cell`
    for ragged digests (companion channels included). ``shard`` is the
    column's node id: a shard id opens the leaf column, an ancestor node id
    the issue-#384 stage column at that dispatch node (a direct read — the
    module docstring's orchestration-not-contract caveat applies). The
    available group orders and fields come from :func:`read_column_record`,
    which also answers whether the column exists at all — opening names an
    object and never checks (§4.6 absence is never an error; the first read
    misses). ``store=`` shares a handle for the manifest GET only, exactly
    as ``open_leaf`` does.
    """
    from zarr.storage import ObjectStore

    if anonymous:
        store_kwargs.setdefault("anonymous", True)
    store_root, rel = _column_target(
        store_root,
        shard,
        window=window,
        product=product,
        manifest=manifest,
        store=store,
        store_kwargs=store_kwargs,
    )
    column_root = f"{store_root.rstrip('/')}/{rel}"
    return ObjectStore(open_object_store(column_root, **store_kwargs), read_only=True)

read_column_record(store_root, shard, *, window=None, product=None, manifest=None, anonymous=False, store=None, **store_kwargs)

One node's validated zagg_column provenance block, or None.

One zarr.json GET (after the manifest's, when not threaded). None means no committed column: an absent object, or an unstamped prefix (D4 debris — a torn worker, or a declaration that never wrote one; §4.6 makes the two indistinguishable from here and neither an error, since readers never require a column). A present, stamped object that does not classify as a column raises: role must be "column", the :data:COLUMN_ATTR block must be present with spec :data:COLUMN_SPEC (strict-check, fail loudly — the conformance rule; this call names ONE object and cannot half-trust it), and its node/order/window must be the ones asked for — a column declaring another node's identity is the "interpretable but wrong" class (:func:moczarr.pyramid._object_entry's posture for an off-order overview), since its groups would be read as this node's.

shard is the column's node id at ITS OWN order: a shard id reads the §4.6 leaf column, an ancestor node id the issue-#384 stage column at that dispatch node — whose record rides verbatim too, its stage-side members included (regime: "stage-gather" groups, the generation skip-gate summary, source_children, run_id). Absence stays the ordinary None either way: stage-column existence at a given order is orchestration, never contract (§4.6).

The record is the §4.6 block verbatim — fields (the materialized fields with their composability classes: the zero-open way to see the published ATL03 split between count-only and four-field columns, englacial/zagg#547) and groups (the resolution groups on disk with per-group regime/merges_from_raw/n_cells; orders 9..13 on the published ATL03 store) — plus node/order/ source_cell_order/window/cells_with_data_order. Group orders as integers: :func:column_orders.

Source code in src/moczarr/column.py

def read_column_record(
    store_root: str,
    shard: str | int,
    *,
    window: str | None = None,
    product: str | None = None,
    manifest: dict | None = None,
    anonymous: bool = False,
    store: Any = None,
    **store_kwargs: Any,
) -> dict | None:
    """One node's validated ``zagg_column`` provenance block, or ``None``.

    One ``zarr.json`` GET (after the manifest's, when not threaded).
    ``None`` means *no committed column*: an absent object, or an unstamped
    prefix (D4 debris — a torn worker, or a declaration that never wrote
    one; §4.6 makes the two indistinguishable from here and neither an
    error, since readers never require a column). A present, **stamped**
    object that does not classify as a column raises: ``role`` must be
    ``"column"``, the :data:`COLUMN_ATTR` block must be present with
    ``spec`` :data:`COLUMN_SPEC` (strict-check, fail loudly — the
    conformance rule; this call names ONE object and cannot half-trust it),
    and its ``node``/``order``/``window`` must be the ones asked for — a
    column declaring another node's identity is the "interpretable but
    wrong" class (:func:`moczarr.pyramid._object_entry`'s posture for an
    off-order overview), since its groups would be read as this node's.

    ``shard`` is the column's node id at ITS OWN order: a shard id reads
    the §4.6 leaf column, an ancestor node id the issue-#384 **stage
    column** at that dispatch node — whose record rides verbatim too, its
    stage-side members included (``regime: "stage-gather"`` groups, the
    ``generation`` skip-gate summary, ``source_children``, ``run_id``).
    Absence stays the ordinary ``None`` either way: stage-column existence
    at a given order is orchestration, never contract (§4.6).

    The record is the §4.6 block verbatim — ``fields`` (the materialized
    fields with their composability classes: the zero-open way to see the
    published ATL03 split between count-only and four-field columns,
    englacial/zagg#547) and ``groups`` (the resolution groups on disk with
    per-group ``regime``/``merges_from_raw``/``n_cells``; orders 9..13 on
    the published ATL03 store) — plus ``node``/``order``/
    ``source_cell_order``/``window``/``cells_with_data_order``. Group
    orders as integers: :func:`column_orders`.
    """
    from moczarr.store import _resolve_store, _stamp_from_meta, read_leaf_metas

    if anonymous:
        store_kwargs.setdefault("anonymous", True)
    store_root, rel = _column_target(
        store_root,
        shard,
        window=window,
        product=product,
        manifest=manifest,
        store=store,
        store_kwargs=store_kwargs,
    )
    handle = _resolve_store(store_root, store, store_kwargs)
    meta = read_leaf_metas(store_root, [rel], store=handle)[0]
    if meta is None or _stamp_from_meta(meta) is None:
        return None  # absent, or unstamped debris (D4) — never an error
    attrs = meta.get("attributes") or {}
    role = attrs.get(ROLE_ATTR)
    if role != "column":
        raise ValueError(
            f"stamped object at {rel} declares role {role!r}, not 'column': the §4.6 "
            f"name seam promises a column at this basename, so a stamped non-column "
            f"here is non-conformant (zagg spec §4.6)"
        )
    block = attrs.get(COLUMN_ATTR)
    if not isinstance(block, dict):
        raise ValueError(
            f"column at {rel} lacks the {COLUMN_ATTR!r} provenance block (present "
            f"exactly when role is 'column' — zagg spec §4.6)"
        )
    if block.get("spec") != COLUMN_SPEC:
        raise ValueError(
            f"column at {rel} declares spec {block.get('spec')!r}; this reader "
            f"implements {COLUMN_SPEC!r} only (strict-check, fail loudly)"
        )
    # The block's own identity keys, checked against what was ASKED for.
    # §4.6 makes `node` the leaf's morton decimal, `order` its shard order,
    # and `window` the §4.2 key the basename round-trips with — all three
    # are already in hand, and a column that disagrees is the "interpretable
    # but wrong" class `_object_entry` raises on for overviews: this record
    # is the existence + field/group answer `open_column`'s
    # `"{order}/{field}"` reads are driven from, so a stale or misplaced
    # column (a copied prefix, a rename, a retrofit that moved a leaf) would
    # otherwise hand back ANOTHER node's cells under this shard's identity —
    # a wrong answer rather than a missing one.
    want_node = morton_decimal(shard)
    want_order = decimal_order(want_node)
    want_window = ALL_TOKEN if window is None else window
    if block.get("node") != want_node or block.get("order") != want_order:
        raise ValueError(
            f"column at {rel} declares node {block.get('node')!r}/order "
            f"{block.get('order')!r}, not this node's {want_node!r}/{want_order} — "
            f"its groups would be read under the wrong node (zagg spec §4.6)"
        )
    if block.get("window") != want_window:
        raise ValueError(
            f"column at {rel} declares window {block.get('window')!r}, not "
            f"{want_window!r}: §4.6 makes the basename and this key round-trip"
        )
    return block