Advanced workflows
The guides in this section cover workflows that need more coordination than a standard single-GPU training run. They assume that your image, command, data mounts, checkpoints, and first training result already work on the smallest approved compute resource.
If this is your first workload, start with the essential practices before using an advanced guide.
Choose the right guide
Section titled “Choose the right guide”| Goal | Start here |
|---|---|
| Submit and manage a normal training run from a terminal | Launch workloads with the Run:ai CLI |
| Adapt PyTorch code with a higher-level distributed launcher | Accelerate and FSDP |
| Understand single-node multi-GPU and multi-node Run:ai settings | Multi-GPU and multi-node training |
| Compare runs in TensorBoard or Weights & Biases | Experiment tracking |
| Measure performance without producing misleading results | Reproducible benchmarking |
| Call an SCC-managed model endpoint | LLM hosting service |
Translate an existing sbatch or srun workflow |
From Slurm to Run:ai |
Before you scale
Section titled “Before you scale”You should be able to answer all of the following:
- Does the same immutable image and command complete on one GPU?
- Are the dataset, outputs, and checkpoints on approved persistent storage?
- Can the application restart from a checkpoint without silently repeating or skipping data?
- Have you recorded the image tag, code revision, dataset version, random seed, and baseline result?
- Does your SCC AI Engineer know the intended GPU count, worker count, runtime, and expected benefit?
- Can you save diagnostic output promptly? Run:ai workload logs disappear shortly after a workload stops or fails.
What Run:ai manages—and what your code must manage
Section titled “What Run:ai manages—and what your code must manage”Run:ai schedules pods, applies project policy, attaches data sources, and allocates the approved compute resource. Your application still has to start the correct number of processes, assign one process to each GPU, coordinate ranks, aggregate metrics once, and write recoverable checkpoints.
For platform concepts, use NVIDIA’s current Run:ai workload documentation. For framework behavior, use the official PyTorch distributed overview or Hugging Face Accelerate documentation.