Quickstart: Build and Run a Flow

This quickstart walks you through a minimal Parade pipeline you can run on your laptop. It uses local futures and small inputs to finish fast.

HPC note (optional)

On clusters, you typically want registry:// and artifacts:// to live on shared scratch (not SLURM_TMPDIR, which is usually node-local). Two common ways to get set up:

library(parade)

# One-command helper (recommended on clusters)
parade_init_hpc(persist = TRUE)

If you have a fixed allocation (e.g., 10 nodes × 196 cores), once you have a flow fl you can set up distribution with a single helper and keep the DAG code unchanged:

fl <- fl |>
  distribute(dist_slurm_allocation(
    nodes = 10,
    cores_per_node = 196,
    within = "multicore",
    # Optional: oversubscribe to reduce stragglers for variable task durations
    target_jobs = 20
  ))

Or a minimal non-persistent setup:

paths_init(profile = "hpc", create = TRUE)
parade_doctor()

# Copy/paste these into your SLURM template preamble if you want the same paths
cat(paste(paths_export(), collapse = "\n"))

1) Install and load

# install.packages("parade")  # CRAN or dev version if needed
library(parade)

2) Define a grid and a stage

grid <- data.frame(x = 1:6, group = rep(LETTERS[1:3], each = 2))

fl <- flow(grid) |>
  stage(
    id = "calc",
    fn = function(x) x^2,
    schema = returns(result = dbl())
  )

3) Choose distribution

# Local parallelism by logical group (A/B/C)
fl <- fl |> distribute(dist_local(by = "group"))

4) Submit, await, and collect

d <- submit(fl)
deferred_await(d, timeout = 30)  # finite timeout for safety
out <- deferred_collect(d)
out

You should see a tibble with the original grid plus the calc.result column.

5) Add a sink (optional)

# Write results to a temp sink; switch to artifacts:// in real projects
fl2 <- flow(grid) |>
  stage(
    "calc", function(x) x^2,
    schema = returns(result = dbl()),
    sink = sink_temp(prefix = "quickstart")
  ) |>
  distribute(dist_local(by = "group"))

d2 <- submit(fl2)
deferred_await(d2, timeout = 30)
deferred_collect(d2)

Deepen your understanding

• Overview: concepts, distribution backends, sinks, and ergonomics
  • website: Overview
  • R help: vignette("parade-overview")
• Unified API: submit functions/scripts and work with jobsets
  • website: Unified API
  • R help: vignette("parade-unified-api")
• Sinks & Artifacts: formats, manifests, and best practices
  • website: Sinks & Artifacts
  • R help: vignette("parade-sinks")

See also: Path management and project setup in Smart Path Management (vignette("parade-paths")).