Persistent storage (CSI)¶

Stateful workloads need persistent volumes. CAPVCO ships the whitesky.cloud CSI driver as an add-on, so a PersistentVolumeClaim becomes a real whitesky.cloud disk, attached to the node that needs it. You get standard disks, low-latency physical disks, online expansion, and — with a key vault — LUKS2 at-rest encryption.

Turn it on¶

Add the CSI label to the Cluster (the base, prebaked and geo templates already do):

metadata:
  labels:
    capvco.infrastructure.cluster.x-k8s.io/csi: vco

CAPVCO then, per cluster:

deploys the driver (controller Deployments + a node DaemonSet) into the csi-driver namespace via a ClusterResourceSet, and
injects a per-cluster credential Secret (csi-driver-secret) so the driver can create/attach disks with your cluster credential — CAPVCO mints nothing.

Install the CSI add-on once — like the CNI add-on

Like the CNI, the CSI driver is delivered by a ClusterResourceSet you apply once to the management cluster; every cluster labelled csi: vco then gets the driver automatically (this is the "per cluster" deploy above). The driver is the existing whitesky.cloud CSI driver, not a rewrite. Two things about it are per-instance (white-label), so the bundle carries placeholders you substitute at apply time: its image (${VCO_CSI_IMAGE}, host = your portal) and its name (${VCO_CSI_DRIVER_NAME}). The driver names itself from the portal domain — reverse the domain and replace the host leaf with csi (e.g. try.whitesky.cloud → cloud.whitesky.csi) — and the shipped CSIDriver object and StorageClass provisioner must use that exact name, so PVCs bind (#21).

From a release (csi.yaml ships alongside cni.yaml):

host=<your-portal-host>                # e.g. try.whitesky.cloud
driver=$(echo "$host" | awk -F. '{o="";for(i=NF;i>1;i--)o=o $i".";print o"csi"}')   # -> cloud.whitesky.csi
curl -fsSL https://gitlab.whitesky.cloud/api/v4/projects/730/packages/generic/capvco/v0.6.1/csi.yaml \
  | VCO_CSI_IMAGE="$host/csi-driver:latest" VCO_CSI_DRIVER_NAME="$driver" \
    envsubst '$VCO_CSI_IMAGE $VCO_CSI_DRIVER_NAME' \
  | kubectl apply -f -

From a checkout: make csi-addon VCO_CSI_IMAGE=<your-portal-host>/csi-driver:latest | kubectl apply -f - (it derives the driver name from the image host automatically).

StorageClasses¶

Three classes ship, with stable, instance-independent names (vco-csi*). Their provisioner is the per-instance driver name, substituted at apply time (above), so they match your running driver on any instance:

StorageClass	Backing	Notes
`vco-csi`	Standard SDS disk	Default class. `WaitForFirstConsumer`, `allowVolumeExpansion: true`.
`vco-csi-physical`	Physical (local, low-latency) disk	`diskType: PHYSICAL`. For latency-sensitive workloads.
`vco-csi-encrypted-vault`	Standard disk + LUKS2 at-rest	Requires a key vault (below) and the encryption add-on.

If a PVC sits Pending with no error, check the driver name matched

Provisioning is silent when the StorageClass provisioner doesn't equal the driver's advertised name — the external-provisioner just ignores the PVC (#21). The apply step above substitutes the name derived from your portal domain; verify it matches what the driver actually registered:

# against the WORKLOAD cluster — should equal the derived name (e.g. cloud.whitesky.csi)
kubectl get csinode -o jsonpath='{range .items[*].spec.drivers[*]}{.name}{"\n"}{end}'
kubectl get storageclass vco-csi -o jsonpath='{.provisioner}{"\n"}'

If they differ, your instance's name doesn't follow the reverse-domain rule — set VCO_CSI_DRIVER_NAME=<the-name-from-csinode> explicitly when applying the add-on.

A basic PVC just uses the default class:

YAMLJSON

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 10Gi
  # storageClassName omitted → the default class

{
  "apiVersion": "v1",
  "kind": "PersistentVolumeClaim",
  "metadata": {
    "name": "data"
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "10Gi"
      }
    }
  }
}

WaitForFirstConsumer means the disk is created in the right location once a pod actually schedules — so the volume lands where the workload runs.

ReadWriteOnce is single-node — enforced, but plan for it

Volumes are ReadWriteOnce: the disk attaches to one node at a time. Pods on the same node share it fine. If a second pod lands on a different node, Kubernetes enforces single-node attach — the second pod is held with a clean Multi-Attach error for volume … and waits for the first to release, rather than double-mounting (no corruption path). This is real (a VolumeAttachment is tracked and the CSIDriver's attachRequired is honoured) and is validated on a multi-worker cluster by the feature-matrix e2e (#22).

So the failure mode is a pod that waits, not data loss — but design for it: for a disk per replica use a StatefulSet with volumeClaimTemplates; for a single-owner volume on a Deployment pin replicas: 1 (or strategy: Recreate) so a rolling update doesn't strand the new pod behind the old one's attachment.

Provisioned PVs carry no topology nodeAffinity, so when the scheduler places a pod whose volume already exists it gets no signal about where that volume lives. Placement happens before any CSI call, so the driver cannot compensate. Within one location that only costs time — the pod waits for the old node to detach. Across locations it cannot recover: a whitesky.cloud disk is location-scoped, so on a geo cluster (worker pools pinned to site-1/site-2/site-3, which may sit in different locations) a pod placed in a different site from its volume waits indefinitely. Until the driver reports accessible_topology at location granularity (the remaining driver-side item in #22), keep volume-bound workloads in one location with a nodeSelector on the driver's own node label:

nodeSelector:
  cloud.whitesky/location: <location>   # same location the volume was created in

Non-root pods and `fsGroup`¶

Volumes honour the pod's fsGroup, so a container running as a non-root user can write its own PVC — which is what every chart that sets a securityContext expects:

securityContext: {runAsUser: 26, runAsGroup: 26, fsGroup: 26, runAsNonRoot: true}

The mount ends up group-owned by that gid (drwxrwsr-x … 26) rather than root:root 0755.

Two settings in the bundle make that work, and they cover different volumes:

the provisioner runs with --default-fstype=ext4, so new PVs record a filesystem type on spec.csi.fsType. The driver has always formatted ext4, but it recorded that only in volumeAttributes, which kubelet ignores when deciding whether to apply fsGroup;
the CSIDriver sets fsGroupPolicy: File, which applies fsGroup regardless of fsType. This is the half that repairs volumes created before the fix — their spec.csi.fsType is already empty and nothing backfills it.

Why not a StorageClass parameter

csi.storage.k8s.io/fstype on the shipped classes would do the same job for new volumes, but StorageClass.parameters is immutable. The add-on is delivered by a ClusterResourceSet with strategy: Reconcile, so changing a shipped class would make every re-apply fail with updates to parameters are forbidden on every cluster that already has it. The provisioner flag has the same effect, covers StorageClasses you create yourself, and can be rolled out.

If you had already worked around this with your own class carrying the parameter, it keeps working — an explicit class parameter still wins over the flag.

Online expansion¶

The default and physical classes allow expansion. Grow a bound PVC by editing its request:

kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

The disk and filesystem grow online, no pod restart. The csi-resizer sidecar in the CSI add-on is what performs it — if you have replaced or trimmed that bundle, expansion silently stops working: the PVC accepts the larger request and then sits at ExternalExpanding with no error anywhere, because nothing is calling ControllerExpandVolume (#53).

No volume snapshots yet

A VolumeSnapshotClass / CSI snapshotter is not shipped in this release (it needs the snapshot CRDs + snapshot-controller, which are deferred). Don't rely on CSI snapshots for now; for cluster-level disaster recovery use etcd backup. Tracked in known limitations.

Encrypted volumes (LUKS2)¶

The …csi-encrypted-vault class encrypts data at rest with LUKS2, wrapping the volume key with a Vault/OpenBao transit engine. You choose the vault mode on the VCOCluster:

spec:
  storage:
    encryption:
      vault: external      # none | external | managed

Mode	What it does	Status
`none` (default)	No vault, no encrypted class — only standard/physical ship.	✅
`external`	You bring a Vault/OpenBao; CAPVCO wires the driver to it.	✅ Supported
`managed`	CAPVCO deploys and operates a Vault for you.	⏳ Deferred — see known limitations

For external, point at your vault:

spec:
  storage:
    encryption:
      vault: external
      external:
        address: https://openbao.example:8200
        transitMount: transit
        k8sAuthPath: kubernetes
        role: csi
        caRef:
          name: openbao-ca      # ConfigMap/Secret with the vault CA bundle

The encryption add-on (csi-encryption: openbao) then delivers the csi-key-backend-config and openbao-ca ConfigMaps the driver reads. The driver authenticates to the vault with its Kubernetes ServiceAccount token (via the vault's kubernetes-auth), so no static vault credential lives in the cluster.

Per-cluster vault auth is a manual step (for now)

Because auth is per-cluster, your vault needs a kubernetes-auth role (bound to the CSI ServiceAccounts, pointing at this cluster's TokenReview endpoint + CA) before encrypted volumes work. Operating the vault and wiring a cluster is covered in OpenBao operations. Automating this per-cluster is a follow-up.

Driver image must include cryptsetup

LUKS formatting needs cryptsetup in the CSI node plugin image. All of CAPVCO's provider-side wiring (transit login, encrypt/decrypt, the encrypted class) is validated end-to-end, but the reference …/csi-driver:latest image historically shipped without cryptsetup, in which case encrypted volumes fail to format. Confirm your driver image includes it. Tracked in known limitations.

How node ↔ disk identity works¶

The CSI node plugin resolves which whitesky.cloud VM it's running on from the host's SMBIOS UUID (/sys/class/dmi/id/product_uuid), matched against the VM's reference id — not from the Kubernetes providerID. So CSI works independently of the provider-id scheme, and there's nothing for you to configure here.

Startup cost grows with your account, not your cluster

To find that match the driver lists every cloudspace in the customer account and then every VM in each one, serially, before it starts serving. The driver pod therefore takes longer to become ready the more cloudspaces the account holds — including cloudspaces that have nothing to do with this cluster. The add-on gives it a startup probe with a five-minute budget, so a slow scan delays registration instead of killing the container. If csinode never gets a driver entry, read the csi-driver container's log before suspecting the image: the same scan aborts outright if any one cloudspace fails to list.

Security & isolation caveat¶

The injected csi-driver-secret holds your cluster credential. A tenant admin in the workload cluster can read that Secret and use the credential directly against the whitesky.cloud API — so the credential's grant scope is the workload cluster's blast radius. Scope the per-cluster credential to just what one cluster needs.

CSI disk-create needs a location-scoped grant

The driver creates disks through a location-scoped API path. A credential granted only at the cloudspace level can attach/snapshot but cannot create disks (a genuine 403). So a cluster running CSI needs a credential with a location grant — which spans every cloudspace in that location, slightly widening the blast radius beyond one cloudspace. This is the one isolation trade-off in the current design; spec.storage.credentialScope exists to select it (location is the default and the working option today). See known limitations.

Next: give etcd its own fast disk, or grow node disks automatically → Node disks.