Skip to content
Spike Documentation

Run a training workload

Use a training workload for non-interactive work that should have a clear command, logs, and completion state. Use an interactive workspace only when you need a shell or notebook session, and use an inference workload for a serving endpoint.

Run:ai Workloads page with the New Workload menu open and synthetic cluster, project, and workload values
Open New Workload and choose Training for a batch run; Workspace and Inference use different lifecycle and connectivity patterns.

First training run

Section titled “First training run”
  1. In Run:ai, select the correct project and choose New workload → Training.
  2. Give the workload a descriptive, non-sensitive name.
  3. Select the environment or image tag you published in Harbor.
  4. Select the smallest approved compute-resource preset that can run the test.
  5. Attach the required data source or persistent volume. Use ephemeral storage only for disposable data.
  6. Enter a command that produces a visible result, submit it, then open the workload details.
  7. Follow the logs until the workload reaches Running and then Completed.
Run:ai New training form with a synthetic cluster and project-alpha selected
Select the project before configuring the training workload. The example uses synthetic names and neutral resource bars.
Run:ai New training form with standard architecture, no saved template, and training-demo as the synthetic name
Choose standard or distributed architecture, start from a template when available, and use a non-sensitive workload name.
Run:ai New training environment form showing a synthetic Harbor image, pull policy, and approved credentials label
Select the immutable image tag and pull policy; use the approved credential mechanism without exposing a token.
Run:ai New training form showing container security choices and synthetic compute-resource controls
Review the container identity and capabilities, then set only the compute resources required for the test.
Run:ai Compute Resources panel showing synthetic resource-small through resource-32xlarge presets
Choose an approved compute-resource preset; the example uses synthetic names and neutral capacity values.
Run:ai New training Data Sources panel with synthetic source-alpha through source-delta entries
Attach only the data sources required by the workload and keep disposable experiments separate from persistent project data.
Run:ai New training volume configuration with synthetic resource-example values and persistent or ephemeral choices
Configure the storage class, access mode, claim size, mount path, and persistence mode deliberately before submitting.
Run:ai New training ephemeral volume configuration with synthetic storage-ephemeral values
Use ephemeral storage for disposable data only; it is not a substitute for a persistent project volume.
Run:ai New training General settings with synthetic preemptibility, priority, retry, and cleanup values
Set retry, preemption, completion, and cleanup behavior explicitly so the workload lifecycle matches your experiment.
Run:ai Workload architecture panel showing synthetic distributed PyTorch worker choices
Use distributed architecture only when the launcher and worker configuration match the application design.
Run:ai distributed training environment with synthetic cluster, project, image, GPU, and worker values
Confirm the distributed image, project, worker count, and GPU request before starting a multi-worker run.

Shared memory for data loaders

Section titled “Shared memory for data loaders”

Linux containers normally have a small /dev/shm shared-memory filesystem. Multiprocessing data loaders, browser processes, and distributed communication libraries can exhaust it even when normal CPU and GPU memory are still available. Typical symptoms include a data-loader worker exiting unexpectedly, Bus error, or No space left on device for /dev/shm.

Check the current allocation inside a small running workload:

Terminal window
df -h /dev/shm

If the workload needs more shared memory, discuss the required size and reason with your SCC AI Engineer. They can approve a compute resource with Increase shared memory size enabled. NVIDIA documents this setting in the Run:ai compute-resource guide. Do not change the compute-resource preset yourself.

Some workloads instead need an engineer-approved ephemeral volume mounted at /dev/shm. In that case, use only the storage class, size, and persistence choice provided by the engineer. The volume is disposable and may use storage rather than RAM; it is not a location for checkpoints or the only copy of results.

After the change, rerun the smallest test, check df -h /dev/shm, and confirm that data-loader workers start and stop cleanly. Increasing shared memory does not fix an application that leaks workers or creates more processes than the approved CPU allocation can sustain.

GPU quota, priority, and retries

Section titled “GPU quota, priority, and retries”

Start with one GPU. Request a fractional GPU only where the selected compute-resource preset explicitly supports it. Multi-GPU and distributed training require an application launcher and architecture that match the selected resources; test the single-GPU path first.

The default scheduling state

Section titled “The default scheduling state”

Projects normally submit work in the over-quota state. Over-quota work may use otherwise idle GPUs, but it has no guaranteed start time and may be preempted. When several projects compete for over-quota capacity, scheduler fairness is proportional to each project’s configured GPU quota; it is not equal weighting. An ordinary request therefore receives neither reserved capacity nor a project-specific priority advantage.

Requesting a scale-up

Section titled “Requesting a scale-up”

When a project is ready for a larger run, request a temporary quota and priority change from the Spike administrators. Include:

  1. The project and workload owner.
  2. The GPU type, count, and compute-resource preset required.
  3. Why the scale-up is ready now (for example, the one-GPU run is validated).
  4. The requested start date and number of weeks for the scale-up.
  5. Expected utilization, checkpointing, and a contact for the work.

The administrators decide whether to grant the additional quota. If approved, they grant the project GPU priority for the agreed number of weeks. This is a temporary scheduling window for the scale-up, not a permanent change to the project’s allocation.

When the priority window ends

Section titled “When the priority window ends”

At the end of the agreed period, the administrators scale the project back to its normal over-quota state. Plan checkpoints, evaluation, and cleanup before the window expires; do not assume that the temporary priority will continue.

Priority does not remove workload limits or make an unsuitable request schedulable. The selected resource, available GPU type, workload state, preemptibility, and project policy still apply. A preemptible workload must persist checkpoints and tolerate being retried. Cancel stale test workloads and delete completed disposable workloads to keep project resources understandable.