mortie.arrow

Arrow interop: the morton_index pyarrow ExtensionType and the library-agnostic Arrow C Data Interface surface. MortonIndexType and MortonIndexExtArray are built lazily behind a module __getattr__ (pyarrow is optional), so they are documented narratively in Arrow interchange rather than here.

The morton_index Arrow skin: a pyarrow ExtensionType over the words.

A pyarrow :class:pyarrow.ExtensionType over uint64 storage carrying the morton_index tag (issue #35, phase 4; issue #58 flipped the storage to uint64).

This is the Arrow-interop sibling of the pandas ExtensionArray in :mod:mortie.morton_index. The packed 64-bit decimal-Morton words live in Rust (src_rust/src/decimal_morton.rs); this module only wraps them so the same words can travel through an Arrow array and survive a parquet round-trip with their morton_index identity attached as extension metadata. Storage is the raw uint64 words verbatim (over the kernel's bit layout), so the raw word order is the Z-order, the same convention as the pandas skin.

pyarrow is an optional dependency exactly like pandas: importing mortie succeeds with neither installed. The extension type is built lazily on first use and a clear ImportError is raised if it is touched without pyarrow.

morton_index_type()

Return the (registered) morton_index pyarrow extension type.

Returns:

Type	Description
ExtensionType	The singleton type instance, registered with pyarrow on first call.

Raises:

Type	Description
ImportError	If pyarrow is not installed.

Source code in mortie/arrow.py

def morton_index_type():
    """Return the (registered) ``morton_index`` pyarrow extension type.

    Returns
    -------
    pyarrow.ExtensionType
        The singleton type instance, registered with pyarrow on first call.

    Raises
    ------
    ImportError
        If pyarrow is not installed.
    """
    return _build_type()

polygons_to_morton_mocs(polygons, order=18, tolerance=None, max_cells=None, normalize=True)

Batch MOC coverage over an Arrow polygon column (issue #153).

The Arrow skin of :func:mortie.polygons_to_morton_mocs (plural MOCs: one MOC per input polygon, many→many — not the many→one ring union of the multipart scalar form): the ragged polygon batch goes in as an Arrow list array, its child arrays feed the numpy core directly, and the ragged result comes back as a ListArray whose values carry the registered morton_index extension type — parquet-ready, e.g. for a catalog's footprint_cells column.

Parameters:

Name	Type	Description	Default
polygons	Array or ChunkedArray or tuple	Either a list<struct<lat, lon>> array (fields in degrees), or a (lats, lons) pair of list<double> arrays with identical offsets. Each list entry is one ring — there is no multipart/hole spelling here, so a multi-ring footprint must be decomposed by the caller (and covered with :func:mortie.morton_coverage_moc's list-of-rings form if the union is what is wanted). Chunked inputs are combined; a sliced input is re-based (its offsets shifted to 0 and only its own vertex window passed on, so the untouched rest of the column is neither copied nor covered); nulls are rejected fail-fast with the polygon index named.	required
order	int	Finest HEALPix order (1-29), shared by every polygon. Default 18.	18
tolerance	(float, int)	The shared per-polygon stop criteria, exactly as on :func:mortie.polygons_to_morton_mocs (mutually exclusive; tolerance in degrees).	None
max_cells	(float, int)	The shared per-polygon stop criteria, exactly as on :func:mortie.polygons_to_morton_mocs (mutually exclusive; tolerance in degrees).	None
normalize	bool	Ring-orientation handling, as on :func:mortie.morton_coverage. Default True.	True

Returns:

Type	Description
ListArray	One entry per input polygon; entry i is that polygon's compact MOC as morton_index-typed words, byte-identical to the scalar :func:mortie.morton_coverage_moc on that ring. A LargeListArray is returned instead when the batch holds more than 2**31 - 1 cells.

Raises:

Type	Description
ImportError	If pyarrow is not installed.
ValueError	Fail-fast with the lowest-index offending polygon named, as on :func:mortie.polygons_to_morton_mocs; also for null polygons or an unrecognized layout.

Examples:

>>> import pyarrow as pa
>>> from mortie import arrow as marrow
>>> polys = pa.array(
...     [[{"lat": 40.0, "lon": -120.0}, {"lat": 50.0, "lon": -120.0},
...       {"lat": 45.0, "lon": -110.0}]])
>>> mocs = marrow.polygons_to_morton_mocs(polys, order=6)
>>> mocs.type
ListType(list<item: extension<mortie.morton_index<MortonIndexType>>>)

Source code in mortie/arrow.py

def polygons_to_morton_mocs(polygons, order=18, tolerance=None, max_cells=None,
                            normalize=True):
    """Batch MOC coverage over an Arrow polygon column (issue #153).

    The Arrow skin of :func:`mortie.polygons_to_morton_mocs` (plural *MOCs*:
    one MOC per input polygon, many→many — not the many→one ring union of the
    multipart scalar form): the ragged polygon batch goes in as an Arrow list
    array, its child arrays feed the numpy core directly, and the ragged
    result comes back as a ``ListArray`` whose values carry the registered
    ``morton_index`` extension type — parquet-ready, e.g. for a catalog's
    ``footprint_cells`` column.

    Parameters
    ----------
    polygons : pyarrow.Array or pyarrow.ChunkedArray or tuple
        Either a ``list<struct<lat, lon>>`` array (fields in degrees), or a
        ``(lats, lons)`` pair of ``list<double>`` arrays with identical
        offsets.  Each list entry is **one ring** — there is no multipart/hole
        spelling here, so a multi-ring footprint must be decomposed by the
        caller (and covered with :func:`mortie.morton_coverage_moc`'s
        list-of-rings form if the union is what is wanted).  Chunked inputs are combined; a **sliced** input is re-based
        (its offsets shifted to 0 and only its own vertex window passed on, so
        the untouched rest of the column is neither copied nor covered);
        nulls are rejected fail-fast with the polygon index named.
    order : int, optional
        Finest HEALPix order (1-29), shared by every polygon.  Default 18.
    tolerance, max_cells : float, int, optional
        The shared per-polygon stop criteria, exactly as on
        :func:`mortie.polygons_to_morton_mocs` (mutually exclusive;
        ``tolerance`` in degrees).
    normalize : bool, optional
        Ring-orientation handling, as on :func:`mortie.morton_coverage`.
        Default ``True``.

    Returns
    -------
    pyarrow.ListArray
        One entry per input polygon; entry ``i`` is that polygon's compact
        MOC as ``morton_index``-typed words, byte-identical to the scalar
        :func:`mortie.morton_coverage_moc` on that ring.  A
        ``LargeListArray`` is returned instead when the batch holds more than
        2**31 - 1 cells.

    Raises
    ------
    ImportError
        If pyarrow is not installed.
    ValueError
        Fail-fast with the lowest-index offending polygon named, as on
        :func:`mortie.polygons_to_morton_mocs`; also for null polygons or an
        unrecognized layout.

    Examples
    --------
    >>> import pyarrow as pa
    >>> from mortie import arrow as marrow
    >>> polys = pa.array(
    ...     [[{"lat": 40.0, "lon": -120.0}, {"lat": 50.0, "lon": -120.0},
    ...       {"lat": 45.0, "lon": -110.0}]])
    >>> mocs = marrow.polygons_to_morton_mocs(polys, order=6)
    >>> mocs.type
    ListType(list<item: extension<mortie.morton_index<MortonIndexType>>>)
    """
    pa = _require_pyarrow()
    from .batch import polygons_to_morton_mocs as _batch

    lats, lons, offsets = _ragged_from_arrow(pa, polygons)
    values, out_offsets = _batch(
        lats, lons, offsets, order=order, tolerance=tolerance,
        max_cells=max_cells, normalize=normalize,
    )
    ext_values = pa.ExtensionArray.from_storage(
        _build_type(), pa.array(values, type=pa.uint64())
    )
    if out_offsets[-1] <= np.iinfo(np.int32).max:
        return pa.ListArray.from_arrays(
            pa.array(out_offsets.astype(np.int32), type=pa.int32()), ext_values
        )
    return pa.LargeListArray.from_arrays(
        pa.array(out_offsets, type=pa.int64()), ext_values
    )

from_wkbs(column, order=18, tolerance=None, max_cells=None, normalize=True)

Batch MOC coverage over an Arrow WKB column (issue #163).

The Arrow skin of :func:mortie.from_wkbs: a geoparquet / STAC geometry column goes in as it comes off the file — binary or large_binary, chunked or not, sliced or not — and the same ragged (values, out_offsets) pair comes back, with every scalar parameter forwarded unchanged. Result i is byte-identical to the core called on blob i.

What this buys is correctness, not speed. The core already accepts byte buffers, so a caller can hand it memoryview slices off the column's value buffer today — but doing that by hand has to get the array offset, the chunk boundaries and the offset width all right, and gets different data with no error if it misses any of them (issue #163) — and, for a null, an empty blob reported as a truncated geometry rather than as a missing one. All four traps are handled once here, in :func:_wkb_blobs_from_arrow.

Memory is the core's posture unchanged, because this is the core: the blobs are zero-copy views into the column's own buffer, so no bytes object is built (the ~305 MB materialization to_pylist() costs on a 555,867-blob column, englacial/zagg#408), and the byte-capped chunk loop still bounds the peak at the result plus one chunk. What it does not remove is the per-chunk copy — releasing the GIL needs owned bytes — nor the one memoryview object per row.

Parameters:

Name	Type	Description	Default
column	Array or ChunkedArray	A binary or large_binary column, one WKB/EWKB geometry per entry. Chunked input is walked chunk by chunk (never combined, which would copy the column); a sliced input reads its own rows; nulls are rejected fail-fast with the index named. An extension column over either storage — a geoparquet column read with the geoarrow.wkb extension registered — is unwrapped to its storage, so the same file covers the same whether or not geoarrow is installed.	required
order	int	Finest HEALPix order (1-29), shared by every blob. Default 18.	18
tolerance	float	Shared per-blob stop radius in degrees, mutually exclusive with max_cells, exactly as on :func:mortie.from_wkbs.	None
max_cells	int	Shared per-blob cell budget, exactly as on :func:mortie.from_wkbs.	None
normalize	bool	Ring-orientation handling, as on :func:mortie.from_wkbs. Default True.	True

Returns:

Name	Type	Description
values	ndarray	Every blob's morton MOC words concatenated (uint64).
out_offsets	ndarray	int64 arrow list offsets into values, length len(column) + 1; out_offsets[0] is 0 and out_offsets[-1] is len(values).

Raises:

Type	Description
ImportError	If pyarrow is not installed.
ValueError	Fail-fast naming the offending blob in the logical column's frame (so an offender in the third chunk reports its column index, not its within-chunk one): a null entry, plus every failure class :func:mortie.from_wkbs raises. See the Notes for which one wins when a column carries both.
TypeError	For a column that is not an Arrow binary / large_binary, or an extension type over any other storage (named by its extension name).

Notes

Two ordered gates, as on :func:mortie.from_wkbs itself: nulls are screened by a vectorised pre-pass over the whole column, and only then are the blobs parsed and covered. Each gate reports its own lowest-index offender, so a null preempts a malformed blob at a lower index — a null at row 20 is raised ahead of a truncated blob at row 3. Within each class the lowest index wins. The pre-pass is an earlier gate, not a competing one: it is what turns a null from the core's misleading truncated WKB into the absence of a geometry, and it is per column rather than per blob because is_null() is one vectorised call.

See Also

mortie.from_wkbs : the core batch, and the contract in full.

Examples:

>>> import pyarrow as pa
>>> import shapely
>>> from mortie import arrow as marrow
>>> col = pa.array([shapely.to_wkb(geom)])
>>> values, off = marrow.from_wkbs(col, order=8)

Source code in mortie/arrow.py

def from_wkbs(column, order=18, tolerance=None, max_cells=None, normalize=True):
    """Batch MOC coverage over an Arrow WKB column (issue #163).

    The Arrow skin of :func:`mortie.from_wkbs`: a geoparquet / STAC geometry
    column goes in as it comes off the file — ``binary`` or ``large_binary``,
    chunked or not, sliced or not — and the same ragged
    ``(values, out_offsets)`` pair comes back, with every scalar parameter
    forwarded unchanged.  Result ``i`` is byte-identical to the core called on
    blob ``i``.

    What this buys is **correctness, not speed**.  The core already accepts
    byte buffers, so a caller can hand it ``memoryview`` slices off the
    column's value buffer today — but doing that by hand has to get the array
    offset, the chunk boundaries and the offset width all right, and gets
    *different data with no error* if it misses any of them (issue #163) —
    and, for a null, an empty blob reported as a truncated geometry rather
    than as a missing one.  All four traps are handled once here, in
    :func:`_wkb_blobs_from_arrow`.

    Memory is the core's posture unchanged, because this *is* the core: the
    blobs are zero-copy views into the column's own buffer, so no ``bytes``
    object is built (the ~305 MB materialization ``to_pylist()`` costs on a
    555,867-blob column, englacial/zagg#408), and the byte-capped chunk loop
    still bounds the peak at the result plus one chunk.  What it does **not**
    remove is the per-chunk copy — releasing the GIL needs owned bytes — nor
    the one ``memoryview`` object per row.

    Parameters
    ----------
    column : pyarrow.Array or pyarrow.ChunkedArray
        A ``binary`` or ``large_binary`` column, one WKB/EWKB geometry per
        entry.  Chunked input is walked chunk by chunk (never combined, which
        would copy the column); a **sliced** input reads its own rows; nulls
        are rejected fail-fast with the index named.  An **extension** column
        over either storage — a geoparquet column read with the
        ``geoarrow.wkb`` extension registered — is unwrapped to its storage,
        so the same file covers the same whether or not geoarrow is installed.
    order : int, optional
        Finest HEALPix order (1-29), shared by every blob.  Default 18.
    tolerance : float, optional
        Shared per-blob stop radius in **degrees**, mutually exclusive with
        ``max_cells``, exactly as on :func:`mortie.from_wkbs`.
    max_cells : int, optional
        Shared per-blob cell budget, exactly as on :func:`mortie.from_wkbs`.
    normalize : bool, optional
        Ring-orientation handling, as on :func:`mortie.from_wkbs`.  Default
        ``True``.

    Returns
    -------
    values : numpy.ndarray
        Every blob's morton MOC words concatenated (``uint64``).
    out_offsets : numpy.ndarray
        ``int64`` arrow list offsets into *values*, length ``len(column) + 1``;
        ``out_offsets[0]`` is 0 and ``out_offsets[-1]`` is ``len(values)``.

    Raises
    ------
    ImportError
        If pyarrow is not installed.
    ValueError
        Fail-fast naming the offending blob in the logical column's frame (so
        an offender in the third chunk reports its column index, not its
        within-chunk one): a null entry, plus every failure class
        :func:`mortie.from_wkbs` raises.  See the Notes for which one wins
        when a column carries both.
    TypeError
        For a column that is not an Arrow ``binary`` / ``large_binary``, or
        an extension type over any other storage (named by its extension
        name).

    Notes
    -----
    **Two ordered gates**, as on :func:`mortie.from_wkbs` itself: nulls are
    screened by a vectorised pre-pass over the whole column, and only then
    are the blobs parsed and covered.  Each gate reports its own lowest-index
    offender, so a **null preempts a malformed blob at a lower index** — a
    null at row 20 is raised ahead of a truncated blob at row 3.  Within each
    class the lowest index wins.  The pre-pass is an earlier gate, not a
    competing one: it is what turns a null from the core's misleading
    ``truncated WKB`` into the absence of a geometry, and it is per column
    rather than per blob because ``is_null()`` is one vectorised call.

    See Also
    --------
    mortie.from_wkbs : the core batch, and the contract in full.

    Examples
    --------
    >>> import pyarrow as pa
    >>> import shapely                                     # doctest: +SKIP
    >>> from mortie import arrow as marrow                 # doctest: +SKIP
    >>> col = pa.array([shapely.to_wkb(geom)])             # doctest: +SKIP
    >>> values, off = marrow.from_wkbs(col, order=8)       # doctest: +SKIP
    """
    pa = _require_pyarrow()
    from .batch import from_wkbs as _batch

    return _batch(
        _wkb_blobs_from_arrow(pa, column), order=order, tolerance=tolerance,
        max_cells=max_cells, normalize=normalize,
    )

from_morton_index(array)

Wrap a :class:~mortie.morton_index.MortonIndexArray as an Arrow array.

Builds a pyarrow ExtensionArray of the morton_index type over the same uint64 words. Missing elements -- a MortonIndexArray for which :meth:isna is True, i.e. the all-zero empty sentinel word -- emit Arrow nulls, so a null survives the round-trip back through :func:to_morton_index. (The missing mask is read off the uint64 words, so a sentinel word in a raw array is treated as a null too; an already-built Arrow array goes back through :func:to_morton_index, not here.)

Parameters:

Name	Type	Description	Default
array	MortonIndexArray or array_like	The words to wrap; may also be a raw uint64 array-like of words.	required

Returns:

Type	Description
ExtensionArray	A morton_index-typed Arrow array over the same words.

Raises:

Type	Description
ImportError	If pyarrow is not installed.

Source code in mortie/arrow.py

def from_morton_index(array):
    """Wrap a :class:`~mortie.morton_index.MortonIndexArray` as an Arrow array.

    Builds a pyarrow ``ExtensionArray`` of the ``morton_index`` type over the
    same ``uint64`` words. Missing elements -- a ``MortonIndexArray`` for which
    :meth:`isna` is True, i.e. the all-zero empty sentinel word -- emit Arrow
    nulls, so a null survives the round-trip back through
    :func:`to_morton_index`. (The missing mask is read off the ``uint64``
    words, so a sentinel word in a raw array is treated as a null too; an
    already-built Arrow array goes back through :func:`to_morton_index`, not
    here.)

    Parameters
    ----------
    array : MortonIndexArray or array_like
        The words to wrap; may also be a raw ``uint64`` array-like of words.

    Returns
    -------
    pyarrow.ExtensionArray
        A ``morton_index``-typed Arrow array over the same words.

    Raises
    ------
    ImportError
        If pyarrow is not installed.
    """
    pa = _require_pyarrow()
    ext_type = _build_type()
    data = np.asarray(getattr(array, "_data", array), dtype=np.uint64)
    # The empty sentinel (all-zero word, prefix 0) is the missing value on the
    # pandas side; mirror it as an Arrow null so isna() round-trips both ways.
    from .morton_index import MortonIndexArray

    mask = data == MortonIndexArray._SENTINEL
    storage = pa.array(data, type=pa.uint64(), mask=mask)
    return pa.ExtensionArray.from_storage(ext_type, storage)

to_morton_index(array)

Convert an Arrow morton_index array back to a MortonIndexArray.

Arrow nulls come back as the all-zero empty sentinel word, so the pandas :meth:isna reports them as missing.

Parameters:

Name	Type	Description	Default
array	ExtensionArray or Array	The extension array, or its plain uint64 storage.	required

Returns:

Type	Description
MortonIndexArray	The pandas-side :class:~mortie.morton_index.MortonIndexArray over the same words.

Raises:

Type	Description
ImportError	If pyarrow is not installed.

Source code in mortie/arrow.py

def to_morton_index(array):
    """Convert an Arrow ``morton_index`` array back to a ``MortonIndexArray``.

    Arrow nulls come back as the all-zero empty sentinel word, so the pandas
    :meth:`isna` reports them as missing.

    Parameters
    ----------
    array : pyarrow.ExtensionArray or pyarrow.Array
        The extension array, or its plain ``uint64`` storage.

    Returns
    -------
    MortonIndexArray
        The pandas-side :class:`~mortie.morton_index.MortonIndexArray` over
        the same words.

    Raises
    ------
    ImportError
        If pyarrow is not installed.
    """
    _require_pyarrow()
    from .morton_index import MortonIndexArray

    storage = getattr(array, "storage", array)
    # Fill nulls with the empty sentinel before materializing: a uint64 array
    # with a null buffer cannot go straight to numpy.
    if storage.null_count:
        storage = storage.fill_null(int(MortonIndexArray._SENTINEL))
    words = storage.to_numpy(zero_copy_only=False).astype(np.uint64, copy=False)
    return MortonIndexArray(words)

export_c_array(words)

Export packed uint64 words as an Arrow C Data Interface capsule pair.

Consumable by any Arrow lib without pandas or pyarrow.

Parameters:

Name	Type	Description	Default
words	array_like	Any uint64 array-like (e.g. a raw numpy array or a MortonIndexArray).	required

Returns:

Type	Description
tuple of PyCapsule	The (schema_capsule, array_capsule) pair, carrying the words as a morton_index extension column (ARROW:extension:name on the schema), with the all-zero empty sentinel mapped to an Arrow null via a real validity bitmap.

Source code in mortie/arrow.py

def export_c_array(words):
    """Export packed ``uint64`` words as an Arrow C Data Interface capsule pair.

    Consumable by any Arrow lib without pandas or pyarrow.

    Parameters
    ----------
    words : array_like
        Any ``uint64`` array-like (e.g. a raw numpy array or a
        ``MortonIndexArray``).

    Returns
    -------
    tuple of PyCapsule
        The ``(schema_capsule, array_capsule)`` pair, carrying the words as a
        ``morton_index`` extension column (``ARROW:extension:name`` on the
        schema), with the all-zero empty sentinel mapped to an Arrow null via
        a real validity bitmap.
    """
    from . import _rustie

    data = np.ascontiguousarray(
        np.asarray(getattr(words, "_data", words), dtype=np.uint64)
    )
    return _rustie.rust_mi_export_c_array(data)

export_c_schema()

Return the morton_index Arrow schema capsule.

The __arrow_c_schema__ half of the C Data Interface surface.

Returns:

Type	Description
PyCapsule	An ArrowSchema capsule carrying the morton_index extension type.

Source code in mortie/arrow.py

def export_c_schema():
    """Return the ``morton_index`` Arrow schema capsule.

    The ``__arrow_c_schema__`` half of the C Data Interface surface.

    Returns
    -------
    PyCapsule
        An ``ArrowSchema`` capsule carrying the ``morton_index`` extension
        type.
    """
    from . import _rustie

    return _rustie.rust_mi_export_c_schema()

import_c_array(source)

Import an Arrow C Data Interface array/stream as packed uint64 words.

Arrow nulls come back as the all-zero empty sentinel, so the null<->sentinel convention round-trips byte-for-byte. No pyarrow dependency on any path.

Parameters:

Name	Type	Description	Default
source	object or tuple	One of: an object exposing __arrow_c_array__ (a contiguous arro3-core / pyarrow / polars array), an object exposing __arrow_c_stream__ (a chunked column / multi-batch source -- every chunk is concatenated), or a (schema_capsule, array_capsule) tuple.	required

Returns:

Type	Description
ndarray	The packed words as a uint64 array.

Source code in mortie/arrow.py

def import_c_array(source):
    """Import an Arrow C Data Interface array/stream as packed ``uint64`` words.

    Arrow nulls come back as the all-zero empty sentinel, so the null<->sentinel
    convention round-trips byte-for-byte. No pyarrow dependency on any path.

    Parameters
    ----------
    source : object or tuple
        One of:

        * an object exposing ``__arrow_c_array__`` (a contiguous arro3-core /
          pyarrow / polars array),
        * an object exposing ``__arrow_c_stream__`` (a **chunked** column /
          multi-batch source -- every chunk is concatenated),
        * or a ``(schema_capsule, array_capsule)`` tuple.

    Returns
    -------
    numpy.ndarray
        The packed words as a ``uint64`` array.
    """
    from . import _rustie

    # A single contiguous array is preferred when both are present; only a
    # chunked source (no __arrow_c_array__) goes through the stream path.
    if hasattr(source, "__arrow_c_array__"):
        schema_capsule, array_capsule = source.__arrow_c_array__()
        return _rustie.rust_mi_import_c_array(schema_capsule, array_capsule)
    if hasattr(source, "__arrow_c_stream__"):
        return _rustie.rust_mi_import_c_stream(source.__arrow_c_stream__())
    schema_capsule, array_capsule = source
    return _rustie.rust_mi_import_c_array(schema_capsule, array_capsule)