The TACC Launcher is a utility designed for simple, data-parallel, high-throughput computing (HTC) workflows on High-Performance Computing (HPC) clusters. It enables users to submit and run large numbers of serial or multithreaded tasks (a "parameter sweep") in parallel across compute nodes using a single Slurm job submission.
Important Notice
Do NOT use MATLAB with the TACC Launcher.
Running MATLAB instances via the launcher can trigger licensing bottlenecks and orphan background processes that consume compute node resources improperly.
Prerequisites
Before submitting jobs via TACC Launcher, ensure you have:
Getting Started
On FAU HPC clusters, TACC Launcher is pre-installed system-wide. You do not need to download or compile source code. Load Launcher into your environment with:
Bash
module load launcher/3.5
Loading the module automatically configures all necessary environment variables, including $LAUNCHER_DIR and execution wrappers.
1. Environment Variables
User-Defined Environment Variables
Set these variables in your Slurm submission script before calling $LAUNCHER_DIR/paramrun:
| Variable |
Description |
LAUNCHER_JOB_FILE |
(Required) Path to the file containing the list of commands/tasks to execute. |
LAUNCHER_WORKDIR |
(Required) Working directory where Launcher executes. All relative paths resolve here. |
LAUNCHER_SCHED |
Task scheduling method: dynamic (default), interleaved, or block. |
LAUNCHER_BIND |
Set to 1 to enable CPU core binding via hwloc. |
Launcher-Generated Environment Variables
Launcher automatically generates and exposes these variables to every running job task:
| Variable |
Description |
LAUNCHER_NPROCS |
Total number of simultaneous processes running in the job submission. |
LAUNCHER_NHOSTS |
Number of hosts/nodes assigned to the submission. |
LAUNCHER_PPN |
Number of processes per node. |
LAUNCHER_NJOBS |
Total number of task lines in your job file. |
LAUNCHER_TSK_ID |
Processing core ID running the job (indexed 0 to LAUNCHER_NPROCS - 1). |
LAUNCHER_JID |
Job line ID currently executing (indexed 1 to LAUNCHER_NJOBS). |
Example: Redirecting Task Output Using $LAUNCHER_JID
To route stdout to a unique log file for each task line, write the command in your job task file as:
Bash
./a.out > out.o$LAUNCHER_JID
If this task corresponds to line 1 of your task file, output will be written to out.o1.
2. Task Scheduling Behaviors (LAUNCHER_SCHED)
You can control how tasks are assigned across available cores by setting LAUNCHER_SCHED in your Slurm script:
-
dynamic (Default): Each task $k$ claims and executes the first available unexecuted line in the job file. Recommended for tasks with variable execution times.
-
interleaved: Each task $k$ executes every $(k + p)^{\text{th}}$ line, where $p$ is the total number of processes.
-
block: Each task $k$ executes a sequential contiguous chunk of lines:
$$\text{Lines } \left[ k \cdot \left(\frac{n}{p}\right) + 1 \quad \text{to} \quad (k + 1) \cdot \left(\frac{n}{p}\right) \right]$$
(where $n$ = total jobs, $p$ = total processes).
3. Hardware Optimizations
Core Binding (hwloc)
Launcher uses hwloc to detect CPU topology. To enable core binding for your tasks to improve cache locality, set:
Bash
export LAUNCHER_BIND=1
4. Job Scripts & Submission Examples
Running a job requires two files:
-
The Slurm Submission Script: Allocates nodes and configures Launcher parameters.
-
The Launcher Task File (LAUNCHER_JOB_FILE): Contains the actual list of commands to run in parallel line-by-line.
Example A: Serial Jobs
1. Create the Launcher Task File (jobfile.sh)
Bash
./Main 3
./Main 4
./Main 10
./Main 15
2. Create the Slurm Job Script (runserial.sh)
Bash
#!/bin/bash
#SBATCH -J launcher_job # Job name
#SBATCH -o launcher.o%j # Output and error log file (%j expands to JobID)
#SBATCH -N 4 # Total nodes requested
#SBATCH -n 16 # Total parallel launcher tasks across nodes
#SBATCH -p shortq7 # Partition name
#SBATCH -t 02:30:00 # Max runtime (hh:mm:ss)
# Load TACC Launcher module
module load launcher
# Define Launcher parameters
export LAUNCHER_WORKDIR=/home/your_username/TACCLauncher
export LAUNCHER_JOB_FILE=jobfile.sh
export LAUNCHER_SCHED=dynamic
# Execute launcher
$LAUNCHER_DIR/paramrun
Example B: Multithreaded Jobs (OpenMP)
To run multithreaded tasks, set thread counts using OMP_NUM_THREADS within your Slurm script.
Slurm Multithreaded Script (run_openmp.sh)
Bash
#!/bin/bash
#SBATCH -J launcher_omp # Job name
#SBATCH -o launcher_omp.o%j # Output and error log file (%j expands to JobID)
#SBATCH -N 2 # Total nodes requested
#SBATCH -n 4 # Total parallel launcher tasks
#SBATCH -p shortq7 # Partition name
#SBATCH -t 00:30:00 # Max runtime (hh:mm:ss)
# Load TACC Launcher module
module load launcher
# Configure threads and Launcher environment
export OMP_NUM_THREADS=6
export LAUNCHER_WORKDIR=/scratch/your_username/tests
export LAUNCHER_JOB_FILE=hello_openmp
# Execute launcher
$LAUNCHER_DIR/paramrun
5. Submitting & Monitoring Jobs
Submit Job
Bash
sbatch runserial.sh
Check Job Status
Bash
squeue -u $USER
Example Output:
Plaintext
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
450558 normal launcher nbc17000 R 0:53 4 compute-1-8-7,compute-7-3-[20-22]
6. Ensuring Accurate Tasks Per Node
To explicitly control task distribution across compute nodes, use the Slurm --tasks-per-node directive:
Bash
#SBATCH -N 2
#SBATCH --tasks-per-node=8
(Allocates 16 total tasks evenly across 2 compute nodes).
Inspect Compute Node Hardware
To view CPU count, core allocations, and hardware specs across cluster nodes, run:
Bash
sinfo --Node --long
7. Referencing Launcher in Publications
If you use TACC Launcher to generate published research results, please cite:
Wilson, M. L., et al. (2014). Launcher: A Flexible Framework for Large-Scale Parameter Sweeps.
BibTeX entry (Wilson:2014:LSF:2616498.2616533) is available in paper/paper.bib within the Launcher source repository.
References