Basic Docker & Kubernetes Hands-On

Teaching: 10 min · Exercises: 60 min · Total: 70 min

Session 2 · 70 min

This episode is the core Kubernetes hands-on: scheduling pods and jobs, persistent storage, multi-container pods, ConfigMaps and Secrets, Deployments, exposing an HTTPS service, steering pods with taints/tolerations and node affinity, and launching a GPU pod.

Conventions. Hands-on examples use the nrp-training-k8s namespace. In any YAML or command, replace <username> with a short version of your name or username to avoid collisions with other participants. Manifests live in the workspace yamls/ folder.

📘 Docs: Kubernetes basics · GPU pods · Run jobs · Storage · Live resources

kubectl flags you'll reach for constantly

Hands-on: a simple pod

Open yamls/test-pod.yaml and replace <username> in metadata.name:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-<username>
  namespace: nrp-training-k8s
spec:
  containers:
  - name: mypod
    image: ubuntu:22.04
    command: ["sh", "-c", "echo 'Hello from NRP!' && sleep 3600"]
    resources:
      limits:  { memory: 100Mi, cpu: 100m }
      requests: { memory: 100Mi, cpu: 100m }

Notice requests and limits are identical — the Gatekeeper-safe default from Episode 1.

Launch and inspect:

kubectl apply -n nrp-training-k8s -f yamls/test-pod.yaml
kubectl wait --for=condition=Ready pod/test-pod-<username> -n nrp-training-k8s --timeout=60s
kubectl get pods -n nrp-training-k8s

sleep 5
kubectl logs test-pod-<username> -n nrp-training-k8s

The kubectl wait line pauses until the pod is actually running. We still sleep briefly before reading logs since Ready doesn't guarantee the container has flushed its first line of stdout.

pod/test-pod-<username> created

NAME                  READY   STATUS    RESTARTS   AGE
test-pod-<username>   1/1     Running   0          12s

Hello from NRP!

Run a command inside it, then open an interactive shell (Ctrl-D to exit):

kubectl exec test-pod-<username> -n nrp-training-k8s -- echo 'Command executed successfully'

kubectl exec -it test-pod-<username> -n nrp-training-k8s -- /bin/bash

kubectl describe pod test-pod-<username> -n nrp-training-k8s          # status + last events

kubectl get events -n nrp-training-k8s --sort-by=.metadata.creationTimestamp | tail -20

Clean up:

kubectl delete pod test-pod-<username> -n nrp-training-k8s

⏳ Bare pods on NRP are reaped after 6 hours. The cluster stamps every pod that isn't managed by a Deployment or Job with a 6-hour activeDeadlineSeconds. A pod you start this morning will be gone by mid-afternoon — that's expected, not a failure, and check.sh reports it as "already cleaned up". Anything that must outlive that (or survive a node reboot) belongs in a Deployment, and its data belongs on a PVC — which is exactly where we go next.

Hands-on: persistent storage with a PVC

Pods are ephemeral — anything written to the container filesystem disappears when the pod terminates. PersistentVolumeClaims ask Kubernetes for long-lived storage you can mount into pods. On NRP we typically use rook-ceph-block-east for general-purpose ReadWriteOnce block storage.

Open yamls/pvc.yaml — it contains a 1 GiB PVC and a writer pod that mounts it at /data. Replace <username> in both names, then:

kubectl apply -n nrp-training-k8s -f yamls/pvc.yaml
kubectl get pvc -n nrp-training-k8s

kubectl get pod pvc-pod-<username> -n nrp-training-k8s

NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
pvc-<username>   Bound    pvc-99a63070-eb3d-490a-82fd-4e5811e4a5df   1Gi        RWO            rook-ceph-block-east   45s

NAME                 READY   STATUS    RESTARTS   AGE
pvc-pod-<username>   1/1     Running   0          47s

Prove the data survives pod deletion — delete only the pod, re-apply, and read the file back:

kubectl wait --for=condition=Ready pod/pvc-pod-<username> -n nrp-training-k8s --timeout=90s
kubectl exec pvc-pod-<username> -n nrp-training-k8s -- cat /data/log.txt

kubectl delete pod pvc-pod-<username> -n nrp-training-k8s
kubectl apply -n nrp-training-k8s -f yamls/pvc.yaml
kubectl wait --for=condition=Ready pod/pvc-pod-<username> -n nrp-training-k8s --timeout=90s

kubectl exec pvc-pod-<username> -n nrp-training-k8s -- cat /data/log.txt   # previous line still there

Don't delete the PVC yet — the next section reuses it.

Hands-on: multi-container pod (sidecar pattern)

A pod can hold more than one container — they share the network namespace (same localhost) and any volumes mounted into both. This is the classic sidecar pattern: a main container plus a supporting one (log shipping, file syncing, format conversion).

yamls/multicontainer.yaml defines a pod whose writer container appends a tick line to a shared file every 5 seconds while a reader container tails the same file. It reuses pvc-<username> from the previous section — first delete the writer pod so the RWO volume detaches:

kubectl delete pod pvc-pod-<username> -n nrp-training-k8s --ignore-not-found
kubectl apply -n nrp-training-k8s -f yamls/multicontainer.yaml

kubectl get pod sidecar-<username> -n nrp-training-k8s

Read each container's log stream separately with -c:

sleep 5
kubectl logs sidecar-<username> -c writer -n nrp-training-k8s --tail=5

kubectl logs sidecar-<username> -c reader -n nrp-training-k8s --tail=5

# writer:
writer-tick 1 04:55:58
writer-tick 2 04:56:04

# reader (tailing the shared file from a different process):
reader started, tailing /shared/data.log
writer-tick 1 04:55:58
writer-tick 2 04:56:04

Every line the writer appends shows up in the reader's stream — both containers see the same volume. Clean up (this also releases the PVC):

kubectl delete -n nrp-training-k8s -f yamls/multicontainer.yaml
kubectl delete -n nrp-training-k8s -f yamls/pvc.yaml

Hands-on: ConfigMap, Secret, and env vars

Hard-coding paths, hostnames, or API tokens into images is a recipe for pain. Kubernetes gives you two purpose-built objects:

• ConfigMap — non-sensitive key/value config, stored as plain text.
• Secret — sensitive values (tokens, passwords, TLS keys), stored base64-encoded with separate RBAC.

yamls/configmap-secret.yaml ships a ConfigMap, a Secret, and a Pod that pulls ConfigMap keys in bulk via envFrom and the Secret via secretKeyRef. Replace <username> in all names and apply:

kubectl apply -n nrp-training-k8s -f yamls/configmap-secret.yaml
kubectl wait --for=condition=Ready pod/env-pod-<username> -n nrp-training-k8s --timeout=60s

sleep 5
kubectl logs env-pod-<username> -n nrp-training-k8s

GREETING=Hello from NRP
SERVER_PORT=8080
API_TOKEN starts with: tutorial…

Look inside each object:

kubectl get configmap app-config-<username> -n nrp-training-k8s -o yaml | grep -A2 '^data:'

kubectl get secret    app-secret-<username> -n nrp-training-k8s -o jsonpath='{.data.API_TOKEN}' | base64 -d ; echo

Base64 is storage format, not encryption — anyone who can get secret in your namespace can read it. Clean up:

kubectl delete -n nrp-training-k8s -f yamls/configmap-secret.yaml

Hands-on: Deployment

A Deployment keeps a set of identical pods running: it restarts them when they fail and rolls out new versions without downtime. Open yamls/deployment.yaml, replace <username>, apply:

kubectl apply -n nrp-training-k8s -f yamls/deployment.yaml

kubectl get deploy,rs,pod -n nrp-training-k8s -l app=hello-deploy-<username>

Try the basic operations:

# scale to 4 replicas
kubectl scale deployment hello-deploy-<username> -n nrp-training-k8s --replicas=4

# delete one pod and watch the Deployment immediately recreate it
VICTIM=$(kubectl get pod -n nrp-training-k8s -l app=hello-deploy-<username> -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod "$VICTIM" -n nrp-training-k8s
kubectl get pods -n nrp-training-k8s -l app=hello-deploy-<username>   # still 4

# rolling update to a different image
kubectl set image deployment/hello-deploy-<username> -n nrp-training-k8s hello=nginx:alpine
kubectl rollout status deployment/hello-deploy-<username> -n nrp-training-k8s

Working with running pods: cp, port-forward, patch

Pick one pod from the Deployment:

POD=$(kubectl get pod -n nrp-training-k8s -l app=hello-deploy-<username> -o jsonpath='{.items[0].metadata.name}')
echo "$POD"

Copy files in and out:

echo "training data v1" > /tmp/dataset.txt
kubectl cp /tmp/dataset.txt nrp-training-k8s/"$POD":/tmp/dataset.txt

kubectl exec "$POD" -n nrp-training-k8s -- cat /tmp/dataset.txt

Tunnel a pod port to your terminal (foreground; use a second terminal for curl):

kubectl port-forward "$POD" -n nrp-training-k8s 8080:80
# second terminal:
curl -s -o /dev/null -w "HTTP %{http_code}\n" http://localhost:8080

Patch a single field without re-applying YAML:

kubectl patch deployment hello-deploy-<username> -n nrp-training-k8s -p '{"spec":{"replicas":2}}'

Clean up:

kubectl delete -n nrp-training-k8s -f yamls/deployment.yaml

Hands-on: batch Job

A Job runs pods until a target number complete successfully. Open yamls/job.yaml, replace <username>, apply, and watch π get computed:

kubectl apply -n nrp-training-k8s -f yamls/job.yaml
kubectl get jobs -n nrp-training-k8s

kubectl wait --for=condition=complete job/pi-<username> -n nrp-training-k8s --timeout=180s

sleep 5
kubectl logs -n nrp-training-k8s -l job-name=pi-<username>

Computing 2000 digits of π takes 50–120s of CPU, so kubectl wait holds until the Job reports complete before we read the logs.

3.14159265358979323846264338327950288419716939937510582097494459230781640628620…

NAME             STATUS     COMPLETIONS   DURATION   AGE
pi-<username>    Complete   1/1           53s        57s

The Job auto-deletes 10 minutes after completion (ttlSecondsAfterFinished: 600).

Hands-on: exposing a service over HTTPS

To expose an HTTP application publicly you need three objects: a Deployment (runs the pods), a Service (stable in-cluster name), and an Ingress on the haproxy class that routes a public hostname to the Service. NRP runs HAProxy as the ingress controller, and Cert Manager issues a free Let's Encrypt TLS certificate automatically for any *.nrp-nautilus.io hostname.

Open yamls/ingress-demo.yaml and replace every <username> (the hostname hello-<username>.nrp-nautilus.io must be globally unique):

kubectl apply -n nrp-training-k8s -f yamls/ingress-demo.yaml

kubectl get deploy,svc,ingress -n nrp-training-k8s -l k8s-app=hello-web-<username>

Wait ~60 seconds for HAProxy and the certificate, then:

curl -sI https://hello-<username>.nrp-nautilus.io | head -5

HTTP/2 200
server: nginx/1.29.1
content-type: text/plain

Open the URL in your browser and reload a few times — the Server name line cycles between the two replicas. Clean up (this releases the public hostname):

kubectl delete -n nrp-training-k8s -f yamls/ingress-demo.yaml

Scheduling: labels, affinity, taints, and tolerations

NRP is a heterogeneous shared cluster — 500+ nodes, many GPU SKUs, and pools reserved for specific projects. Scheduling primitives are how you say "put my pod here, not there":

Labels + affinity are an attraction; taints + tolerations are a repulsion. You usually need both: a toleration to be allowed onto a reserved node, plus an affinity rule so the scheduler actually picks it.

The PEARC26 reserved GPU pool

For the tutorial, NRP has a pool of NVIDIA A10 GPU nodes reserved:

• Label nrp-training=true — marks the tutorial nodes.
• Taint nautilus.io/reservation=nrp:NoSchedule — keeps other workloads off them.

Explore the pool:

kubectl get nodes -l nrp-training=true -L nvidia.com/gpu.product

kubectl get nodes -l nrp-training=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

NAME                         STATUS   ROLES    AGE      VERSION    GPU.PRODUCT
hcc-nrp-shor-c5825.unl.edu   Ready    <none>   3y300d   v1.33.12   NVIDIA-A10
hcc-nrp-shor-c5905.unl.edu   Ready    <none>   3y300d   v1.33.8    NVIDIA-A10
…

hcc-nrp-shor-c5825.unl.edu	[{"effect":"NoSchedule","key":"nautilus.io/reservation","value":"nrp"}, …]

To land on the pool, a pod spec needs both blocks — this pattern appears in every GPU manifest in the workspace:

spec:
  tolerations:
  - key: nautilus.io/reservation
    operator: Equal
    value: nrp
    effect: NoSchedule
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: nrp-training
            operator: In
            values: ["true"]

preferred… is a soft hint — the scheduler picks a reserved node if one is free but won't strand your pod if all are busy. The required… variant blocks scheduling until a matching node frees up. Beyond this tutorial the same pattern targets specific GPU models (nvidia.com/gpu.product=NVIDIA-A100-PCIE-40GB), CUDA versions (nvidia.com/cuda.runtime.major=12), or regions (topology.kubernetes.io/region=us-west).

Hands-on: your first GPU pod

yamls/gpu-pod.yaml requests one GPU via resource limits:

    resources:
      limits:
        nvidia.com/gpu: 1
      requests:
        nvidia.com/gpu: 1

Resource keys by hardware type:

• NVIDIA GPUs (generic): nvidia.com/gpu: <count>
• Qualcomm Cloud AI 100: qualcomm.com/qaic: <count> — Nautilus has 8 Cloud AI 100 Ultra cards × 4 SoCs = 32 devices; each runs LLMs up to ~25B parameters
• Specific products: nvidia.com/a100, nvidia.com/rtxa6000, etc. — see GPU pods docs

Launch it, exec in, and run nvidia-smi:

kubectl apply -n nrp-training-k8s -f yamls/gpu-pod.yaml
kubectl get pods -n nrp-training-k8s

kubectl exec -it tutorial-<username>-gpu-pod -n nrp-training-k8s -- nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA A10                     On  |   00000000:06:00.0 Off |                    0 |
+-----------------------------------------+------------------------+----------------------+

Important — GPUs are scarce shared resources. Delete the pod as soon as you're done:

kubectl delete pod tutorial-<username>-gpu-pod -n nrp-training-k8s

Same operations from Python: the Kubernetes API

kubectl is just a client for the Kubernetes REST API — every get, apply, and delete you ran is an HTTP call to the API server. Anything you can do with kubectl, you can do from code, which is how automation, CI pipelines, dashboards, and operators drive a cluster. The official kubernetes Python client is the most common way to do it.

Install it once (a small, ~20-second pure-Python install):

pip install --quiet kubernetes

The script below does a full round-trip — list → create → wait → read logs → delete — against your namespace, entirely from Python. It reuses the same kubeconfig kubectl uses, so you're already authenticated; and it builds the Pod out of Python objects (V1Pod, V1Container, …) that map one-to-one to the YAML fields you saw earlier.

import os, time
from kubernetes import client, config

config.load_kube_config()          # the same config kubectl uses — already authenticated
v1   = client.CoreV1Api()
ns   = "nrp-training-k8s"
user = os.environ["NRP_USER"]      # set by the ⚙️ setup cell
name = f"pyapi-{user}"

# LIST — like `kubectl get pods -n nrp-training-k8s`
for p in v1.list_namespaced_pod(ns).items:
    print(p.metadata.name, "->", p.status.phase)

# CREATE — the same Pod you'd write in YAML, built as Python objects
pod = client.V1Pod(
    metadata=client.V1ObjectMeta(name=name, labels={"app": "pyapi-demo"}),
    spec=client.V1PodSpec(
        restart_policy="Never",
        tolerations=[client.V1Toleration(
            key="nautilus.io/reservation", operator="Equal",
            value="nrp", effect="NoSchedule")],
        containers=[client.V1Container(
            name="main", image="busybox:1.36",
            command=["sh", "-c", "echo hello from the kubernetes python client"],
            resources=client.V1ResourceRequirements(
                requests={"cpu": "200m", "memory": "128Mi"},
                limits={"cpu": "200m", "memory": "128Mi"}))]))
v1.create_namespaced_pod(ns, pod)
print("created", name)

# WAIT + READ LOGS — like `kubectl logs`
for _ in range(60):
    if v1.read_namespaced_pod(name, ns).status.phase in ("Succeeded", "Failed"):
        break
    time.sleep(2)
# _preload_content=False returns the raw HTTP response, giving clean log text
resp = v1.read_namespaced_pod_log(name, ns, _preload_content=False)
print("logs:", resp.read().decode().strip())

# DELETE — like `kubectl delete pod`
v1.delete_namespaced_pod(name, ns)
print("deleted", name)

Same three verbs (list / create / delete), same objects, just expressed in Python instead of YAML + kubectl. Every resource type has a matching API group — CoreV1Api for pods/services/configmaps, AppsV1Api for deployments, BatchV1Api for jobs — so the patterns from the whole morning carry straight over into code.

End of episode — cleanup

kubectl delete pod test-pod-<username>            -n nrp-training-k8s --ignore-not-found
kubectl delete -f yamls/multicontainer.yaml       -n nrp-training-k8s --ignore-not-found
kubectl delete -f yamls/pvc.yaml                  -n nrp-training-k8s --ignore-not-found
kubectl delete -f yamls/configmap-secret.yaml     -n nrp-training-k8s --ignore-not-found
kubectl delete -f yamls/deployment.yaml           -n nrp-training-k8s --ignore-not-found
kubectl delete -f yamls/job.yaml                  -n nrp-training-k8s --ignore-not-found
kubectl delete -f yamls/ingress-demo.yaml         -n nrp-training-k8s --ignore-not-found
kubectl delete pod tutorial-<username>-gpu-pod    -n nrp-training-k8s --ignore-not-found

Then verify: bash check.sh 2 in the workspace (or the last cell of the notebook).