Storage and data
Choose storage before submitting a workload. The storage choice determines whether your data survives a restart, retry, or deletion.
| Storage choice | Use it for | Lifecycle |
|---|---|---|
| Existing data source | Project datasets and shared inputs | Managed outside one workload; verify permissions before use. |
| Persistent volume | Checkpoints, repeatable outputs, and long-lived work | Survives workload restarts according to project policy. |
| Ephemeral volume | Caches, temporary preprocessing, and disposable scratch data | Removed with the associated workload; never store the only copy of a result. |
| S3-compatible object storage | Large datasets, shared objects, and transfers between tools | Independent of a workload; access requires a project bucket and credentials. |
Platform storage architecture
Section titled “Platform storage architecture”High-performance DDN project storage
Section titled “High-performance DDN project storage”Spike 1 provides a 250 TiB high-performing project space (DDN) with an S3 connection option. Workloads access this storage via Persistent Volume Claims (PVCs) mounted into containers at /data or connect to S3 object storage endpoints for large dataset loading and model checkpointing.
TU/e-managed file storage and NetApp
Section titled “TU/e-managed file storage and NetApp”A TU/e-managed network share, including a NetApp-backed share, is an existing data source rather than Spike project storage. Do not assume that a share can be mounted directly into a workload. Before onboarding or a large transfer, give SCC the share owner, protocol, approximate size, data-classification decision, source path, and required access period. SCC will confirm whether the approved route is a direct managed connection or a staged transfer through S3 or an SSH-accessible workspace.
Test the chosen route with a small directory before moving the dataset. Confirm permissions, destination capacity, file counts, and sample checksums. Never copy a NetApp password, storage token, or private key into a container image, notebook, workload command, or support ticket.
VAST Data parallel enterprise storage
Section titled “VAST Data parallel enterprise storage”Spike A integrates an all-flash VAST Data parallel storage cluster engineered to deliver ultra-low latency and multi-gigabyte-per-second throughput to saturate NVIDIA Blackwell B300 GPU Tensor Cores.
- High-throughput parallel I/O: Built on Disaggregated Shared-Everything (DASE) architecture, VAST Data provides simultaneous parallel file access across distributed Blackwell compute nodes without metadata server contention.
- Protocol support: Supports both multi-protocol NFSv3/v4 mounts and S3-compatible object endpoints for fast dataset loading and checkpointing.
- Workload mounting: Configured within Run:AI as a dedicated project persistent
volume claim (PVC) mounted into workloads at
/dataor/vast. - Optimal data formats: For maximum training throughput, structure large datasets in chunked container formats (e.g. WebDataset, Sharded Parquet, or TFRecord) to leverage parallel NVMe read streams.
Before staging multi-terabyte datasets, coordinate with your Supercomputing Brabant (SCB) technical liaison to verify target volume quotas and establish the fastest network transfer path.
External cloud storage
Section titled “External cloud storage”External cloud storage is not automatically trusted or mounted on Spike. Record the provider, region, bucket or share owner, expected transfer volume, classification decision, and any egress implications during intake. Approval must cover both the source and the new Spike copy. Use a project-scoped credential through the approved secret mechanism and copy only into approved persistent storage. The file-transfer guide contains the supported S3, SSH, and rsync patterns.
Container filesystem versus PVC
Section titled “Container filesystem versus PVC”The writable container filesystem is part of the running container’s temporary runtime state. It is useful for package caches, logs, and short-lived scratch files, but it is not a durable data location. A new workload, image update, retry, or deletion can give you a fresh filesystem.
A persistent volume claim (PVC) is separate storage mounted into the workload.
When the SSH container workflow mounts a PVC at /data, files copied with SCP
to /data are written to that PVC rather than into the image layer. They can be
used by later workloads that mount the same claim, subject to project
permissions, quota, and the PVC’s lifecycle policy.
If /data is not backed by a PVC, it is only an ordinary directory in the
container and should be treated as temporary. Always verify the mount before a
large transfer and keep a second copy of irreplaceable data where project
policy requires it.
For the complete transfer choices and examples, see File transfer. Before cleanup, confirm important outputs exist in persistent storage and document the image tag, command, and input version used to produce them.
Project provisioning, volume creation, resizing, copying, and deletion are admin operations and are intentionally documented only in the private wiki.