Skip to contents

Create SLURM distribution specification

dist_slurm.Rd

Configure distributed execution on SLURM clusters using batchtools.

Usage

dist_slurm(
  by = NULL,
  within = c("multisession", "multicore", "callr", "sequential"),
  workers_within = NULL,
  template = slurm_template(),
  resources = list(),
  chunks_per_job = 1L,
  target_jobs = NULL
)

Arguments

by

Column names to group by for parallelization

within

Execution strategy within each SLURM job: "multisession", "multicore", "callr", or "sequential"

workers_within

Number of workers within each SLURM job

template

Path to SLURM batch template file

resources

Named list of SLURM resource specifications

chunks_per_job

Number of groups to process per SLURM job.

target_jobs

Optional integer; target number of SLURM jobs to create (overrides chunks_per_job at submit time).

Value

A parade_dist object for SLURM execution

Examples

# Create SLURM distribution specification
dist <- dist_slurm(by = "condition", resources = list(time = "1h", mem = "4GB"))

# Use multicore within each SLURM job for efficiency
dist <- dist_slurm(by = "condition", within = "multicore", workers_within = 8,
                   resources = list(cpus_per_task = 8))

# Use with a flow (configuration only, no execution)
grid <- data.frame(x = 1:4, group = rep(c("A", "B"), 2))
if (FALSE) { # \dontrun{
flow(grid) |>
  stage("process", function(x) x * 2) |>
  distribute(dist)
} # }