Workload lifecycle, logs, and cleanup

State	Meaning	Your next action
Pending	Waiting for a schedulable resource or prerequisite	Check quota, resource preset, image, and data source.
Initializing	Platform is preparing the workload	Wait for the transition or inspect the event details.
Running	Container command is active	Inspect and save relevant logs, then checkpoint important work.
Completed	Command ended successfully	Preserve results and delete disposable resources.
Failed	Startup or application command ended with an error	Open and save the logs immediately, then identify the first relevant error.
Stopped	User or policy stopped the workload	Capture any remaining logs, then decide whether to resume or remove it.
Deleted	Workload record has been removed	Confirm results were stored outside ephemeral storage.

Save application logs to persistent storage

For a workload whose output matters, write the application stream to a file on the project PVC while still showing it in the Run:ai log view:

set -o pipefail
mkdir -p /data/logs
python -u train.py 2>&1 | tee -a /data/logs/train.log

set -o pipefail preserves a failing application’s exit status instead of reporting only the result from tee. The -u option asks Python to flush output without long buffering delays.

This protects the log only when /data is the intended persistent mount. Confirm it with findmnt /data or df -h /data before the real run. Do not write credentials, access tokens, sensitive prompts, or confidential dataset values to the log. For distributed workloads, write a separate file per rank or let only rank 0 write the combined application log.

Planned maintenance

Spike normally has two planned maintenance periods each year: a broader summer window and a year-end window for required updates. SCC communicates the exact dates, scope, and expected downtime to active users in advance. Depending on the work involved, some or all Spike services may be unavailable.

Plan project deadlines, long-running jobs, checkpoints, and data transfers around the announced window. Your SCC AI engineer will discuss upcoming maintenance during planning and help identify a safe stopping point for work already in progress.

Safe recovery

Use status and logs before changing a configuration. When retrying, make one intentional change at a time and keep the previous image tag and command in the workload record. A retry is appropriate for transient scheduling or service failures; application errors need a corrected image, command, or data path.

When a workload is no longer needed, stop it and delete only disposable runtime resources. Preserve outputs, checkpoints, and reproducibility notes in approved persistent storage.