Hand off to multiprocessing

Pickle a typed map through a worker boundary.

Goal

Send a derived map to a worker pool (or a joblib cache) and have the receiving side still know what produced it — without shipping a separate manifest alongside the pickle.

Code

from pathlib import Path
import sys
import pickle

sys.path.insert(0, str(Path.cwd().parents[1] / "src"))

import neuroim as ni

repo = Path.cwd().parents[1] if Path.cwd().name == "cookbook" else Path.cwd()
bold = ni.read_image(str(repo / "golden_tests/fixtures/tiny_bold.nii.gz"))
mask = ni.read_image(str(repo / "golden_tests/fixtures/tiny_mask.nii.gz"), type="vol")

tsnr = bold.temporal_snr(mask=mask)

payload = pickle.dumps(tsnr, protocol=pickle.HIGHEST_PROTOCOL)
received = pickle.loads(payload)             # stands in for the worker side

(received.provenance is not None,
 received.provenance.method_name,
 received.provenance.input_space_hash == tsnr.provenance.input_space_hash,
 received.provenance.mask_hash == tsnr.provenance.mask_hash)
(True, 'temporal_snr', True, True)

The Receipt is part of the object, so it travels with the pickle. The worker does not need a side-channel describing what it received.

What’s guaranteed

  • DenseNeuroVol / result objects are picklable and the round-trip produces an equal Receipt (same hashes, same method_name).
  • This holds across a fresh process, not just in-process.

What survives the file

Pickle is not the file boundary, but the guarantee is the same shape: the provenance is intrinsic to the object. The on-disk equivalent is Save a derived map.

Trapdoors

  • Reducing to a bare ndarray before the handoff (tsnr.data instead of tsnr) ships numbers with no provenance — the receiving side cannot recover what produced them. Keep the typed carrier across the boundary.
  • A worker that re-wraps a bare array in a fresh DenseNeuroVol invents a new identity; it does not recover the original Receipt.

The important rule is to hand off the typed object, not the detached array. The typed object carries the Receipt; the array carries only numbers.

See also

Back to top