Air-gapped clusters¶

Some environments can't reach the public internet at all — regulated networks, isolated sites, security-sensitive deployments. CAPVCO can run a cluster with no public-internet egress: every container image is served by a registry inside the cloudspace, and node binaries come from a pre-baked image.

How it works¶

Two sources of egress have to be eliminated, and CAPVCO handles both:

Node bootstrap — a stock image installs Kubernetes from pkgs.k8s.io on boot. A pre-baked image already has it, so no egress. This is a hard prerequisite.
Container images — kubeadm, the CNI, the CSI driver, etc. normally pull from registry.k8s.io, docker.io, and your portal's registry. CAPVCO stands up an in-cloudspace registry VM and points every node's containerd at it.

flowchart LR
  subgraph cs["Cloudspace (no egress)"]
    reg["Registry VM (TLS)"]
    cp["control-plane"]
    wk["workers"]
    cp -->|"containerd pulls"| reg
    wk -->|"containerd pulls"| reg
  end

The controller brings the registry up before the control plane, fetches a whitesky.cloud-issued TLS certificate for it, publishes its DNS name to its in-cloudspace IP, and wires every node's containerd to pull through it (with the cert trusted). CRS add-on manifests stay registry-agnostic — the containerd mirror redirects transparently, so nothing needs image-ref rewriting.

Turn it on¶

Use cluster-template-airgapped.yaml:

VCO_LOCATION=<loc> \
VCO_IMAGE_ID=<prebaked-node-image-id> \
VCO_REGISTRY_DOMAIN=ag-registry.<your-zone> \
VCO_REGISTRY_IMAGE_ID=<registry-image-id> \
VCO_REGISTRY_SEEDED=true \
VCO_REGISTRY_CERT_SECRET=ag1-operator \
clusterctl generate cluster ag1 \
  --kubernetes-version v1.36.4 --infrastructure vco --flavor airgapped > ag1.yaml
kubectl apply -f ag1.yaml

The `spec.registry` fields¶

spec.airGapped: true implies a registry; spec.registry configures it:

Field	Meaning
`domain`	Required. The registry's DNS name — used for its TLS cert and the containerd mirror config.
`seeded`	`true` = a pre-seeded, zero-egress registry image (below). `false` (default) = a `registry:2` pull-through cache (needs egress to seed on demand).
`imageId` / `locationImages`	The registry VM's image id (per location, like node images).
`certSource`	`CERTIFICATE_STORE` (operator-uploaded, air-gap friendly; default) or `LETS_ENCRYPT` (needs public reachability).
`certCredentialSecretRef`	Operator credential with access to the SSL-certificate store (per-cluster customer credentials get a 403 here).
`port`	Registry HTTPS port (default 5000).
`mirrorRegistries`	Upstreams redirected to the local registry. Defaults to the set kubeadm/CNI/CSI use; your portal's own registry is derived and mirrored automatically (never hardcoded).
`vcpus` / `memoryMiB` / `bootDiskGiB`	Registry VM sizing.

Seeded vs pull-through¶

Seeded (seeded: true) — true zero-egress. The registry VM image has the registry binary, a systemd unit, and a pre-seeded image store already baked in (kubeadm's image set, Calico, the CSI driver + sidecars, etc., each stored under its host-stripped repo path so containerd resolves it). The controller just drops the TLS cert and starts the baked service. Nothing is ever pulled from upstream.
Pull-through (seeded: false, default) — needs egress to warm. The registry runs as a registry:2 pull-through cache, fetching from upstream on first request and caching thereafter. Fine for a low-egress (not zero-egress) setup, or for validating the mirror path.

For a genuinely disconnected environment you want a seeded image. Build one with image-builder/build-registry-image.sh (the seeded counterpart to the node-image build): it stages every image an air-gapped cluster pulls — the kubeadm set, Calico, the CSI sidecars — into a local store with skopeo (each under its host-stripped repo path), then bakes that store, the registry binary, the capvco-registry.service unit, and the baked config.yml into the Ubuntu cloud image with packer's QEMU builder. It needs egress at build time only (to fetch the images); the resulting cluster needs none. Publish the qcow2 into your location(s) with capvco-imagepublish (image create is operator-privileged), then set seeded: true and point registry.imageId at the published id.

Validated end-to-end

The seeded path is exercised by the feature-matrix e2e (scenario_airgapped, seeded by default): a control plane comes up pulling Calico (docker.io) and kube-proxy/coredns (registry.k8s.io) entirely from the baked store, reaching Ready with zero upstream pulls.

The TLS certificate¶

The registry serves HTTPS with a certificate from whitesky.cloud's SSL-certificate API. The manager fetches the cert for domain and pushes it to the registry VM, and a periodic cron re-fetches and re-pushes so it never goes stale. Nodes trust the registry's CA via containerd.

Cert fetch needs an operator credential

The SSL-certificate store is operator-privileged — a per-cluster customer credential gets a 403 even listing certs. Provide an operator credential via registry.certCredentialSecretRef (same shape as the cluster credential Secret: jwt + customer_id). This mirrors the image-publish privilege model.

Verify¶

On a healthy air-gapped cluster, every image resolves locally. You can confirm the containerd mirror is in effect on a node:

cat /etc/containerd/certs.d/registry.k8s.io/hosts.toml
# → redirects to https://<registry-domain>:5000 with the CA trusted

A quick negative test: an image that isn't in a seeded registry (e.g. netshoot) will fail to pull — which is exactly the isolation you want.

Workers on the airgapped template

cluster-template-airgapped.yaml ships a control-plane-only cluster (no worker MachineDeployment, no MachineHealthChecks, no cert-rollout block) to keep the zero-egress example minimal. Add worker pools (and MHCs) as you would elsewhere — just make sure they use the pre-baked node image and the same CNI/CSI labels.

Removing the public endpoint too¶

Air-gapped removes workload egress; on its own it still runs on a normal cloudspace whose Kubernetes API is published on the external IP. To remove the public endpoint as well, combine this with a private cluster:

spec:
  controlPlaneExposure: private
  cloudspace:
    private: true          # no external network at all

spec.cloudspace.private is accepted only together with controlPlaneExposure: private. Read Reaching a private cluster first — the management cluster has to be somewhere that can see the cloudspace private network.

That's the feature set. For the exact fields behind any of it, see the CRD API reference; for every template knob, Templates & variables.