Write, append, and open row-sharded HDF5 frames — write_sharded_frame

A sharded frame is a directory containing immutable FDS v1 HDF5 frame files and one small root manifest. Appending writes a new shard and then atomically replaces only the root manifest; previously committed HDF5 files are never opened for writing or rewritten.

Usage

write_sharded_frame_h5(
  x,
  path,
  shard_id = "shard-000001",
  chunk_dims = NULL,
  layout = c("balanced", "imagewise", "featurewise"),
  target_chunk_bytes = getOption("fmristore.target_chunk_bytes", 1024^2),
  memory_budget = getOption("fmridataset.block_budget", 512 * 1024^2)
)

append_frame_shard_h5(
  x,
  path,
  shard_id,
  chunk_dims = NULL,
  layout = c("balanced", "imagewise", "featurewise"),
  target_chunk_bytes = getOption("fmristore.target_chunk_bytes", 1024^2),
  memory_budget = getOption("fmridataset.block_budget", 512 * 1024^2)
)

frame_shard_manifest(path)

open_sharded_frame_h5(path)

Arguments

x: An `fmri_frame` containing one new observation shard.
path: Sharded frame directory.
shard_id: Stable unique shard identifier.
chunk_dims, layout, target_chunk_bytes, memory_budget: Passed to [write_frame_h5()].

Value

Writers invisibly return the normalized directory path. `open_sharded_frame_h5()` returns one lazy `fmri_frame`, and `frame_shard_manifest()` returns root metadata without opening a shard.

Details

All shards must have identical feature identity, assay declarations, observation-block definitions, and non-observation registries. Observation IDs must be globally unique. Numerical arrays remain lazy when reopened.