Essential practices

These practices apply to every Spike workload, from a first one-GPU test to a distributed experiment or managed inference endpoint. Review them before moving the only copy of data or requesting additional GPUs.

Before submitting

Use an immutable Harbor image tag and a command you have tested on the smallest suitable compute-resource preset.
Confirm that inputs, outputs, logs, and checkpoints use the intended persistent data source or PVC.
Record the code revision, configuration, dataset version, image tag, random seed, and expected result.
Check that the project permits the data, external services, models, and software licences involved.
Know how the application will stop, checkpoint, resume, and clean up.

Make the run recoverable

Write checkpoints to persistent storage at useful intervals and test one restart before a costly run. A useful checkpoint includes the model, optimizer, scheduler, step or epoch, and any random or sampler state needed to continue without repeating or skipping data.

Save application logs to the project PVC as well as the Run:ai log view. Run:ai logs are operational diagnostics, not long-term storage, and can disappear after a workload stops or fails.

Observe before scaling

Start with one GPU and establish a baseline. Record GPU utilisation, CPU and input-pipeline behaviour, step time, memory use, and checkpoint cost. Profile a representative short run before assuming that more GPUs will help. Scaling an input bottleneck usually makes the bottleneck more expensive rather than faster.

Follow Monitor and profile a first workload for the beginner workflow. Use experiment tracking for run identity and metrics, and reproducible benchmarking for controlled comparisons after the baseline works.

Respect shared capacity

Over-quota work uses otherwise idle capacity, has no guaranteed start time, and may be preempted. Request the smallest resource that answers the current question, remove stale tests, and delete disposable completed workloads. Ask for a temporary quota and priority window only after the workload is reproducible and the requested period, utilisation, and recovery plan are clear.

Keep secrets and sensitive data out of the workload definition

Do not place passwords, API keys, private keys, access tokens, or storage credentials in an image, repository, notebook, command line, support ticket, or saved screenshot. Use the approved project secret mechanism. Do not log raw prompts, confidential samples, personal data, authorization headers, or full credentials.

The same boundary applies to inputs and derived artefacts. Review Data classification on Spike before moving data, and reassess it when a workload creates prompts, embeddings, checkpoints, weights, predictions, or logs that may expose restricted information.

Finish deliberately

Before stopping or deleting a workload:

save the final checkpoint, result, and relevant logs;
verify important files on persistent storage;
record the image tag, command, inputs, and result location;
remove temporary credentials and disposable resources; and
tell the project contact when a reserved priority window is no longer needed.