Deploy a Custom JupyterHub & Build Images in NRP GitLab

Teaching: 15 min · Exercises: 30 min · Total: 45 min

Time: 00:15–01:00

Deploy your own JupyterHub with Helm — controlled access, custom images, per-profile resource limits, shared storage — then see how to build custom container images with NRP GitLab CI/CD. This is the recipe instructors and PIs use to stand up course and lab hubs on NRP.

mkdir -p ~/.kube
curl -o ~/.kube/config -fSL https://nrp.ai/config

kubectl config get-contexts     # should list the `nautilus` context
kubectl auth whoami             # triggers the CILogon browser login the first time
helm version --short

Conventions. Each participant works in their own pre-created namespace (nrp-training-000 … nrp-training-099) — JupyterHub can only be deployed once per namespace. Set your short username, render your personal manifests, and claim your namespace in one step. The claim is keyed by your hub login, so it is idempotent — you get the same slot back every time, and re-running this after a break is safe:

export NRP_USER=changeme   # ✏️ EDIT to your short name
cd ~/nairr-webinar/workspace
if [ "$NRP_USER" = changeme ]; then echo "⚠️  Edit NRP_USER above first, then re-run"; else
  mkdir -p my-yamls
  for f in yamls/*; do sed "s/<username>/$NRP_USER/g" "$f" > "my-yamls/$(basename "$f")"; done
  echo "✅ my-yamls/ rendered for $NRP_USER"
fi
# claim your own namespace for the session (idempotent — same slot every time you ask):
export NRP_NAMESPACE=$(curl -s "http://nrp-claim.nrp-training.svc.cluster.local/claim?user=${JUPYTERHUB_USER:-$NRP_USER}")
export NRP_RELEASE=jhub-$NRP_USER
echo "namespace=$NRP_NAMESPACE release=$NRP_RELEASE"

✅ my-yamls/ rendered for alice
namespace=nrp-training-042 release=jhub-alice

$NRP_NAMESPACE and $NRP_RELEASE are what the commands below (and check.sh 2) pick up — no hand-editing. Every manifest is rendered into my-yamls/ with <username> already filled in, so wherever this page says "replace <username>", it is already done in your copy.

Terminal steps don't share these variables — run the same export lines in any terminal you open. Re-running the render overwrites edits you made in my-yamls/.

📘 Docs: Deploy JupyterHub · Build images · NRP GitLab CI · Z2JH (upstream)

1. Helm in one paragraph

Helm is a package manager for Kubernetes — instead of authoring every Deployment, Service, and ConfigMap by hand, you install a chart (a reusable bundle of templates) and tune it through a values file. The Zero to JupyterHub chart packages the entire hub/proxy/spawner stack; your whole deployment is one YAML file of values.

In the training hub, helm is preinstalled — verify, then add the chart repository:

kubectl auth whoami && helm version --short

helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update
helm repo list

"jupyterhub" has been added to your repositories
Update Complete. ⎈Happy Helming!⎈

NAME         URL
jupyterhub   https://jupyterhub.github.io/helm-chart/

2. Examine the values file

Open yamls/jhub-values.yaml. Key sections:

hub:
  config:
    JupyterHub:
      authenticator_class: dummy      # tutorial only — swap for CILogon/OIDC in production
      admin_access: true
      admin_users: ["admin"]
    DummyAuthenticator:
      password: "training123"
  db:
    type: sqlite-pvc
    pvc:
      accessModes: [ReadWriteOnce]
      storage: 1Gi
      storageClassName: rook-ceph-block-east
proxy:
  secretToken: 'secret_token'         # replace before deploying!
singleuser:
  storage:
    type: dynamic
    capacity: 5Gi
    homeMountPath: /home/jovyan
    dynamic:
      storageClass: rook-ceph-block-east
      pvcNameTemplate: claim-{username}{servername}
      storageAccessModes: [ReadWriteOnce]
  image:
    name: quay.io/jupyter/scipy-notebook
    tag: 2024-04-22
  cpu: {limit: 2, guarantee: 2}
  memory: {limit: 8G, guarantee: 8G}
  defaultUrl: "/lab"
cull:                                  # required on NRP — close inactive sessions
  enabled: true
  timeout: 3600
  every: 600

Generate a real proxy token and put it in the file in place of secret_token:

openssl rand -hex 32

3. Deploy

First — what's already running in your namespace?

JupyterHub can only be deployed once per namespace: a second release fights the first one over the proxy-public service and the hub database. Your claimed slot should be empty, but check before you install — if you've run this tutorial before, or the slot was recycled, there may already be a hub sitting in it.

helm list -n $NRP_NAMESPACE
kubectl get pods -n $NRP_NAMESPACE

NAME	NAMESPACE	REVISION	STATUS	CHART	APP VERSION
No resources found in nrp-training-042 namespace.

An empty helm list and no pods means you're clear — skip ahead and deploy.

If a release is listed, look at the NAME column. Tear it down only if it's yours; in a shared namespace someone else's class may be running on it. Set OLD_RELEASE to that name — otherwise leave it alone and ask whoever owns the namespace.

# ⚠️  Optional — only if the command above listed a JupyterHub you want gone.
OLD_RELEASE=changeme   # ✏️ the NAME shown by `helm list` above

if [ "$OLD_RELEASE" = changeme ]; then
  echo "Nothing to do — set OLD_RELEASE only if you need to remove an existing hub."
else
  helm uninstall "$OLD_RELEASE" -n $NRP_NAMESPACE
  # wait for the pods to actually go away before redeploying
  kubectl wait --for=delete pod -l app=jupyterhub -n $NRP_NAMESPACE --timeout=120s 2>/dev/null || true
  helm list -n $NRP_NAMESPACE
  kubectl get pods -n $NRP_NAMESPACE
fi

helm uninstall leaves PVCs behind on purpose — the hub database and any user home directories survive, so a reinstall picks them back up. Section 8 shows how to delete those too if you really want a clean slate.

Install the chart

helm upgrade --cleanup-on-fail --install $NRP_RELEASE jupyterhub/jupyterhub \
  --namespace $NRP_NAMESPACE \
  --values my-yamls/jhub-values.yaml \
  --wait \
  --timeout=10m

Release "jhub-alice" does not exist. Installing it now.
NAME: jhub-alice
NAMESPACE: nrp-training-042
STATUS: deployed
REVISION: 1
NOTES:
       You have successfully installed the official JupyterHub Helm chart!

Inspect what the chart created — every one of these is an ordinary Kubernetes object:

kubectl get pods -n $NRP_NAMESPACE

kubectl get services -n $NRP_NAMESPACE

kubectl get pvc -n $NRP_NAMESPACE

You should see the hub pod (auth, sessions, spawning), the proxy pod (routing), a hub-db-dir PVC — and, once someone logs in, per-user pods and claim-<user> PVCs.

4. Expose it with an Ingress

my-yamls/jhub-values.yaml already ends with an ingress block, commented out, with your hostname rendered in — the setup step substituted <username> for you, so it is globally unique:

ingress:
  enabled: true
  ingressClassName: haproxy
  hosts: ["jhub-<username>.nrp-nautilus.io"]
  pathSuffix: ''
  tls:
    - hosts:
      - jhub-<username>.nrp-nautilus.io

The quickest way to enable it is a one-liner that strips the leading # from those lines:

sed -i '/^#ingress:/,$ s/^#//' my-yamls/jhub-values.yaml
tail -9 my-yamls/jhub-values.yaml

Upgrade the release and verify:

helm upgrade $NRP_RELEASE jupyterhub/jupyterhub \
  --namespace $NRP_NAMESPACE \
  --values my-yamls/jhub-values.yaml \
  --wait --timeout=10m

kubectl get ingress -n $NRP_NAMESPACE

After ~a minute for HAProxy + Let's Encrypt, open https://jhub-$NRP_USER.nrp-nautilus.io, log in as admin with the Dummy password, and spawn a server. You now have a working multi-user JupyterHub on national research infrastructure.

5. Make it yours

5.1 Multiple image profiles

Give users a menu of environments — add to singleuser:

singleuser:
  profileList:
  - display_name: Scipy
    kubespawner_override:
      image_spec: quay.io/jupyter/scipy-notebook:2024-04-22
    default: True
  - display_name: Tensorflow (CUDA)
    kubespawner_override:
      image_spec: quay.io/jupyter/tensorflow-notebook:cuda-2024-04-22
  - display_name: Pytorch (CUDA 12)
    kubespawner_override:
      image_spec: quay.io/jupyter/pytorch-notebook:cuda12-2024-04-22
  - display_name: Datascience (scipy, Julia, R)
    kubespawner_override:
      image_spec: quay.io/jupyter/datascience-notebook:2024-04-22

5.2 Per-profile resource limits

  - display_name: Small (2 CPU, 4GB RAM)
    kubespawner_override:
      image_spec: quay.io/jupyter/scipy-notebook:2024-04-22
      cpu_limit: 2
      cpu_guarantee: 2
      mem_limit: 4G
      mem_guarantee: 4G
  - display_name: Large (8 CPU, 16GB RAM)
    kubespawner_override:
      image_spec: quay.io/jupyter/scipy-notebook:2024-04-22
      cpu_limit: 8
      cpu_guarantee: 8
      mem_limit: 16G
      mem_guarantee: 16G

Add a profile or two to your values file, helm upgrade again, and reload the spawn page — the menu updates live. A GPU profile adds extra_resource_limits: {"nvidia.com/gpu": "1"}.

This is where a course gets shaped: give the intro unit a Small CPU profile and the deep-learning unit a GPU profile, and students pick the right one from a dropdown instead of you managing machines.

5.3 Shared storage for the whole class

Mount one RWX CephFS volume into every user server:

singleuser:
  storage:
    extraVolumes:
      - name: jupyterhub-shared
        persistentVolumeClaim:
          claimName: jupyterhub-shared-volume
    extraVolumeMounts:
      - name: jupyterhub-shared
        mountPath: /home/shared

Instructors drop datasets and notebooks into /home/shared once; every student sees them instantly. Mount it read-only for students in production.

5.4 Real authentication

For production, replace the Dummy authenticator with institutional login. yamls/cilogon-jupyterhub-config.yaml in the workspace shows a CILogon/OIDC configuration — campus credentials, an allowlist or admin-managed access, no passwords to distribute. For a class roster, the allowlist is your enrollment list.

Swapping it in needs a client_id and client_secret from CILogon, which you have to request from them and wait on — see the lead-time warning at the top of this page. That wait is the reason today's hub uses the Dummy authenticator.

6. Operating your hub

helm list -n $NRP_NAMESPACE

sleep 5
kubectl logs -n $NRP_NAMESPACE -l app=jupyterhub,component=hub --tail=50

kubectl get pods -n $NRP_NAMESPACE -l app=jupyterhub,component=singleuser-server

Troubleshooting is the standard Kubernetes trio: describe the failing pod, read namespace events, check hub/proxy logs.

7. Building custom images in NRP GitLab

The stock Jupyter images only go so far — real courses need their own package stacks. NRP GitLab (gitlab.nrp-nautilus.io) builds images for you in CI and hosts them in its container registry.

The workflow:

1. Create a project on NRP GitLab and add a Dockerfile — typically FROM quay.io/jupyter/scipy-notebook:… plus your pip/conda installs.
2. Add .gitlab-ci.yml — a single Kaniko job builds and pushes on every commit:

image: ghcr.io/osscontainertools/kaniko:debug

stages:
- build-and-push

build-and-push-job:
  stage: build-and-push
  variables:
    GODEBUG: "http2client=0"
  script:
  - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
  - /kaniko/executor --cache=true --push-retry=10 --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --destination $CI_REGISTRY_IMAGE:latest

3. Use the image anywhere on the cluster — in a pod spec, or as a hub profile:

  - display_name: My Course Image
    kubespawner_override:
      image_spec: gitlab-registry.nrp-nautilus.io/<group>/<project>:latest

Best practices: tag with commit SHAs (not just latest) so a course mid-semester never changes under your students; use --cache=true for fast rebuilds; keep credentials in CI variables, never in the Dockerfile.

8. Cleanup

If this was a trial run, uninstall your Helm release so the cluster is left clean:

helm uninstall $NRP_RELEASE -n $NRP_NAMESPACE

User PVCs are kept by default; delete them only if you're sure:

kubectl delete pvc -n $NRP_NAMESPACE -l app=jupyterhub,component=singleuser-storage

If this is a real course hub, leave it running — the cull settings in the values file close idle student sessions automatically.

Where to go next

• Get your own namespace if you were following along on the training hub — see Getting your own access.
• Docs: nrp.ai/documentation
• Live help: the NRP Matrix channel at nrp.ai/contact
• These materials: training.nrp-nautilus.io and GitHub