Persistent Storage & I/O for AI/Scientific Workloads

Teaching: 15 min · Exercises: 15 min · Total: 30 min

Session 4 · 30 min

AI and scientific workloads live or die on I/O: where the dataset sits, how fast checkpoints write, and whether ten students can read the same files at once. This episode maps NRP's storage options to those needs and gets hands-on with each from a notebook terminal.

📘 Docs: Storage intro · Ceph · S3 · Policies

Storage options on NRP

Your JupyterHub home directory (/home/jovyan) is itself an RBD PVC — everything you save in the notebook survives server restarts, but it is sized in gigabytes; keep bulk data on CephFS or S3.

Choosing an access mode

• ReadWriteOnce (RWO) — one node mounts read-write. Block storage. You saw the consequence in Episode 2: the sidecar exercise had to delete the first pod before the second could mount.
• ReadWriteMany (RWX) — many pods on many nodes mount simultaneously. CephFS. This is what a classroom shared folder or a multi-worker training job wants.

Hands-on: an RWX CephFS volume shared by many pods

yamls/shared-pvc.yaml creates a CephFS-backed PVC. Apply it and note the RWX access mode:

kubectl apply -n nrp-training-k8s -f yamls/shared-pvc.yaml
kubectl get pvc -n nrp-training-k8s | grep shared

jupyterhub-shared-volume   Bound    pvc-…   1Gi   RWX   rook-cephfs   30s

Unlike the Episode 2 PVC, many pods can mount this claim at the same time — in the final episode this exact volume becomes the /home/shared folder every student sees in a course JupyterHub.

Hands-on: S3 object storage

NRP runs S3-compatible object storage on Ceph. It's the right tool when data must be reachable from outside the cluster, shared across sites, or published alongside a paper. Any S3 client works — aws CLI, boto3, s3fs, rclone.

yamls/pod-awscli.yaml starts a pod with the AWS CLI image, a 100 Gi emptyDir scratch volume at /scratch, and installs boto3 + torch on boot. It also pulls the shared tutorial S3 credentials in from a Secret (nrp-tutorial-s3) as environment variables — so inside the pod both aws and boto3 authenticate automatically, no aws configure step. Replace <username> and apply:

kubectl apply -n nrp-training-k8s -f yamls/pod-awscli.yaml
kubectl get pod tutorial-<username>-pod -n nrp-training-k8s -w

Wait for the install loop to log Done with installs, then exec in and talk to S3:

# carry your username into the pod (its shell has the S3 keys but not NRP_USER):
kubectl exec -it tutorial-<username>-pod -n nrp-training-k8s -- env NRP_USER=<username> bash

# inside the pod — the tutorial key is already in the environment
# ($AWS_ACCESS_KEY_ID / $AWS_SECRET_ACCESS_KEY / $AWS_ENDPOINT_URL / $S3_BUCKET):
aws --endpoint $AWS_ENDPOINT_URL s3 ls
aws --endpoint $AWS_ENDPOINT_URL s3 ls s3://$S3_BUCKET/

# stage the shared dataset onto the fast local scratch:
aws --endpoint $AWS_ENDPOINT_URL s3 cp s3://$S3_BUCKET/dataset.tar.gz /scratch/
tar xzf /scratch/dataset.tar.gz -C /scratch     # 30 sample images + labels.csv + shards

# publish your own results back — write under your username so you don't
# clobber the shared dataset (everyone shares one bucket in this tutorial):
echo "hello from <username>" > /scratch/result.txt
aws --endpoint $AWS_ENDPOINT_URL s3 cp /scratch/result.txt s3://$S3_BUCKET/<username>/result.txt

2026-07-20 18:02:11   10485760 dataset.tar.gz
download: s3://pearc26-tutorial/dataset.tar.gz to ../scratch/dataset.tar.gz
upload: ../scratch/result.txt to s3://pearc26-tutorial/alice/result.txt

On your own laptop (outside this pod) you'd first run aws configure and paste an access key / secret from nrp.ai/s3token, then use the same --endpoint commands. Request your own S3 credentials via the User Portal.

The same works from Python with boto3 — it reads the same credentials from the environment. Run this inside the awscli pod (kubectl exec … -- python3), where the nrp-tutorial-s3 Secret injects AWS_ENDPOINT_URL/S3_BUCKET; these are not set in your hub notebook, so pasting it into a hub cell raises KeyError.

import boto3, os
s3 = boto3.client("s3", endpoint_url=os.environ["AWS_ENDPOINT_URL"])
bucket = os.environ["S3_BUCKET"]
for obj in s3.list_objects_v2(Bucket=bucket).get("Contents", []):
    print(obj["Key"], obj["Size"])

alice/result.txt 18
dataset.tar.gz 10485760

I/O patterns for AI workloads

A pattern that serves nearly every training job on NRP:

1. Stage in — copy the dataset from S3 (or CephFS) to node-local scratch (emptyDir) at job start. Local NVMe is far faster for the random reads of a dataloader.
2. Checkpoint out — write checkpoints to an RWO block PVC (or push to S3) at epoch boundaries, not every step.
3. Publish — copy final artifacts to S3 where collaborators (or your future self) can fetch them without cluster access.

For classrooms, the equivalent pattern is: course materials on an RWX CephFS volume mounted read-only into every student server, student work on per-user RWO home volumes — exactly what we'll configure in the final episode.

Cleanup

kubectl delete pod tutorial-<username>-pod -n nrp-training-k8s --ignore-not-found

Keep jupyterhub-shared-volume — the custom JupyterHub episode mounts it. Verify with bash check.sh 4.