Skip to contents

Overview: Parade in 10 Minutes

parade-overview.Rmd

Introduction

Parade is a small framework for building typed, parallel dataflows in R. It wraps future/furrr for local parallelism, adds backends for SLURM and mirai, and provides a unified, ergonomic interface to submit, monitor, and collect results — with optional artifact sinks and typed schemas.

What you get

  • Flows and stages: Compose small, pure functions over a parameter grid.
  • Distribution: Run locally (future), via SLURM (batchtools), or mirai.
  • Deferred execution: submit() → deferred_await() → deferred_collect().
  • Artifacts and sinks: Save outputs reliably with typed schemas and manifests.
  • Ergonomics: Defaults, profiles, helpers for paths, naming, and monitoring.

Core concepts at a glance 

library(parade)

# Grid of inputs
grid <- data.frame(x = 1:4, group = rep(c("A", "B"), 2))

# Define a simple flow with one stage and an explicit return schema
fl <- flow(grid) |>
  stage("calc", function(x) x^2, schema = returns(result = dbl())) |>
  distribute(dist_local(by = "group"))

# Submit for deferred execution (local futures by default)
d <- submit(fl)

# Wait briefly and collect results
deferred_await(d, timeout = 30)
out <- deferred_collect(d)
out

When to use which surface

  • Use the flow DSL when you want typed, multi-stage pipelines with sinks.
  • Use the unified job API (slurm_call(), slurm_map()/slurm_pmap()) for quick function/script submission and jobset operations.

Distribution options

Artifacts and sinks (optional)

Define sinks in stages to write outputs to stable paths with format-specific read/write. Start with temporary sinks during development; switch to project paths (artifacts://) for production.

Next steps