Kubernetes

The example in examples/k8s is a kustomize base for the synthetic acme team. It uses plain manifests and no operator or chart. CI builds it with a pinned kustomize and validates every resource against the Kubernetes schemas with a pinned kubeconform.

What you get

Resource	Name	Notes
Namespace	outstaff-acme	Labelled for the restricted Pod Security level, which the manifests meet.
Deployment, Service, PVC	outstaff-office	One replica with Recreate, so two offices never share a store. The PVC holds SQLite if you use no Postgres.
StatefulSet, PVC	outstaff-agent-<handle>, <handle>-home	One per person, from one template, each with its own 20 GiB home volume (a plain PVC, so the name is predictable).
NetworkPolicy	several	Default deny; DNS and HTTPS out; agents to the office; the office to Postgres.
ConfigMap	outstaff-team	Generated from your team.yaml. Mounted into the office only; each agent receives the team config from the office.
Secrets	outstaff-office, outstaff-agent-<handle>	Generated by kustomize from files you add under secrets/ (see below). Never in git.

Postgres is not included. Bring your own (a managed database, or your platform’s Postgres operator) and give the office its URL in a Secret. Without one, the office stores its state in SQLite on its volume, which is fine to try things out.

Layout

examples/k8s/
  kustomization.yaml        namespace, resources, images, team ConfigMap, secretGenerator
  secrets/                  your secret files (gitignored) and a README listing them
  namespace.yaml
  network-policy.yaml
  office/                   Deployment, Service, PVC
  agent/                    the template of one agent: StatefulSet and home PVC
  people/<handle>/          person label and patches for the names, the person, the Secret and the volume
examples/k8s-sandbox/       optional overlay: the agents with the subprocess sandbox, and its two node profiles

A person is a folder with a kustomization.yaml that reuses agent/ and patches the StatefulSet name, OUTSTAFF_PERSON, the name of the person’s Secret and volume, and the PVC name. To add or remove a person, copy or delete a folder, edit the list in the top kustomization.yaml and add or remove the person’s entry in its secretGenerator.

Deploy

1. Team file. Copy your team file next to the kustomization: cp .outstaff/team.yaml examples/k8s/team.yaml. For acme, copy examples/acme/.outstaff/team.yaml. It is gitignored there. Only the office mounts it. When it changes, kubectl apply -k rolls out the office (the ConfigMap’s name carries a hash of its content); the agents reconnect, receive the new team config and render their working files again once their running work is done, without a restart of their own.
2. Images. Set the release in the images: list of kustomization.yaml. Use a version tag, or a digest for fully reproducible rollouts.
3. People. Edit the list of people/ folders to match your team file.
4. Namespace. kubectl apply -f examples/k8s/namespace.yaml.
5. Secrets. Add the files under examples/k8s/secrets/ as described below. They become Secrets when you apply.
6. Apply. kubectl apply -k examples/k8s.
7. Check.

kubectl -n outstaff-acme get pods,pvc                  # office Ready 1/1, one agent pod per person Running, PVCs Bound
kubectl -n outstaff-acme logs deploy/outstaff-office | grep "agent connected"
kubectl -n outstaff-acme logs sts/outstaff-agent-sam   # why an agent does not start

On first start the office doesn’t pick up existing tickets or pull requests; assign or move a ticket to start work.

Secrets

kustomize generates the Secrets from files. Put one file per key in a gitignored tree, examples/k8s/secrets/office/<KEY> for the team and examples/k8s/secrets/<handle>/<KEY> for each person, and kubectl apply -k creates them. A file holds the bare value, so multi-line values such as PEM private keys survive intact. secrets/README.md lists every file per person.

The keys are the acme example’s (github, itsaplan, slack and claude-agent-sdk); yours depend on your integrations and on each person’s repository grants. ost setup office and ost setup <handle> list them, and the secretGenerator in kustomization.yaml is where you add or drop a file. Each key of a Secret becomes a file in /run/secrets/outstaff, which is where the containers look, so the key names are the file names.

Secret	Keys
outstaff-office	GH_OFFICE_APP_ID, GH_OFFICE_APP_INSTALLATION_ID, GH_OFFICE_APP_PRIVATE_KEY, GH_OFFICE_APP_SLUG, GH_OFFICE_APP_BOT_EMAIL, ITSAPLAN_API_KEY, SLACK_OFFICE_BOT_TOKEN, and OUTSTAFF_AGENT_TOKEN_<HANDLE> (upper case) for each person
outstaff-agent-<handle>	OUTSTAFF_AGENT_TOKEN (the office gets the same value as its token for this person), ANTHROPIC_API_KEY (or your other sign-in), SLACK_APP_TOKEN, SLACK_BOT_TOKEN, ITSAPLAN_API_KEY, and the GitHub keys the person’s repository grants need (below)
outstaff-office-store	url: the Postgres URL for the office, postgres://user:password@host:5432/db. Optional: without it the office uses SQLite on its volume. Not generated; create it with kubectl create secret generic outstaff-office-store --from-literal=url=....

The GitHub keys of a person follow their grants, the same way ost setup <handle> lists them. A person with any write grant gets their own App (GH_APP_ID, GH_APP_INSTALLATION_ID, GH_APP_PRIVATE_KEY, GH_APP_SLUG, GH_APP_BOT_EMAIL) and the office App keys (GH_OFFICE_APP_*, including GH_OFFICE_APP_BOT_EMAIL). A person with read grants only gets the office App keys. A person with no repository grants, such as the designer leo, gets no GitHub keys. The GH_OFFICE_APP_* files are stored once, under secrets/office/, and the generator copies them into each Secret that needs them. The tree has the same layout ost setup writes to .outstaff/secrets/ (one folder per person and one for the office), so you can copy or link those folders into secrets/.

Each agent pod mounts only its own Secret, so an agent never sees another person’s secrets or the office’s other keys. The generated Secrets carry a hash in their name, which the manifests follow, so changing a file rolls the pods that use it. Files are mounted with mode 0440 and group 10001 (fsGroup), so only the container user reads them. To keep the files in 1Password or SOPS instead of typing them in, see the Secrets guide; it ends with the same files.

Kubernetes Secrets are base64 in etcd. Turn on encryption at rest, limit who can get secrets in the namespace, or let an external store fill them (see the Secrets guide). The pods do not mount a service account token, so nothing inside the cluster is reachable through the Kubernetes API from an agent.

Migrating an existing deployment

A StatefulSet’s selector and serviceName are immutable, so kubectl apply cannot change them on StatefulSets you already run. If your agents run from other manifests (or from an older version of this example), move them over without losing their volumes or Secrets. The names below are acme’s; use your own.

1. Keep the old objects. Copy each old StatefulSet aside as outstaff-agent-sam-legacy, with replicas: 0 and a distinct selector label, for example outstaff.work/legacy: "true" in both selector.matchLabels and the pod template. Create the copies, so nothing selects both an old and a new pod.
2. Stop the originals and delete them without their pods or volumes. Scale each to zero first, then kubectl delete sts outstaff-agent-sam --cascade=orphan. The PVCs and Secrets stay as they are. Do not touch them.
3. Adopt the volumes. The new manifests expect a PVC named <handle>-home. A volume created by volumeClaimTemplates is called home-<statefulset name> (for example home-outstaff-agent-sam). A PVC’s name cannot change, so either name the new PVC after the old one (patch claimName and the PVC name in people/sam/kustomization.yaml) or create sam-home from a snapshot or a restored volume. Match the storage class and size of the old claim: a bound PVC is not resized by apply to a smaller value.
4. Provide the new Secrets. The example generates outstaff-agent-<handle>-<hash> from your files and never edits an existing Secret, so the old Secrets stay in place for the rollback.
5. Create the new objects. kubectl apply -k examples/k8s creates the new StatefulSets, PVCs and Secrets. Check that each agent connects (kubectl -n outstaff-acme logs deploy/outstaff-office | grep "agent connected").
6. Roll back if needed. Scale the new StatefulSet to zero and the -legacy copy back to one. Only one of the two may run for a person at a time, because both would use the same agent token and home volume.
7. Clean up when you are sure: delete the -legacy copies, then the old PVCs and Secrets you no longer use.

Hardening that is already in the manifests

• Every container is non-root (uid and gid 10001), with readOnlyRootFilesystem, all capabilities dropped, no privilege escalation and the RuntimeDefault seccomp profile. Writable places are the volumes: /home/outstaff (an agent), /var/lib/outstaff (the office) and a size-limited emptyDir for /tmp.
• terminationGracePeriodSeconds is 45 for the office and 60 for an agent. Both stop on SIGTERM: the office finishes its running handlers and an agent hands its running work item back as interrupted. The item goes first in the queue and the restarted agent resumes it with its session, so a rollout resumes running work.
• The office has a startup probe on /healthz with a generous window (three minutes), then a liveness probe on /healthz and a readiness probe on /readyz. The agent has no port, so no probes: it exits when it cannot run and the StatefulSet restarts it.

Subprocess sandbox (optional)

By default the agent’s shell commands can read the Claude credential from their environment, the same as Claude Code on a developer’s machine; the runtime adapter explains the trade-off and the mitigations. With sandbox: bubblewrap in the team file, Claude Code removes the credential from the environment of every command, hook and MCP server and runs each command in a bubblewrap sandbox with its own user and PID namespaces and a fresh /proc, so a command cannot read it from another process either. The runtime also hides the credential files under /run/secrets/outstaff inside that sandbox; the other secret files stay readable, because ost git-credential runs there.

integrations:
  runtime: { adapter: claude-agent-sdk, sandbox: bubblewrap }

The agent image has bubblewrap and socat; they do nothing until the option is on. A hardened pod blocks what bubblewrap needs, so the overlay in examples/k8s-sandbox changes four things on the agents, and each is needed. The office runs no commands and keeps the base settings.

Setting	Why
seccompProfile: { type: Localhost, localhostProfile: outstaff/bwrap.json }	The runtime’s default profile refuses clone and unshare with namespace flags, mount, umount2 and pivot_root. This profile is Docker’s default plus exactly those, without IPC, UTS or cgroup namespaces.
appArmorProfile: { type: Localhost, localhostProfile: outstaff-agent }	containerd’s default AppArmor profile denies every mount, and Ubuntu 24.04 denies user namespaces to any process without a profile that grants them. This profile is containerd’s default plus userns and the mounts bubblewrap makes.
hostUsers: false	The pod gets its own user namespace: uid 0 in the pod is an unprivileged uid on the node. Kubernetes allows the next setting only with it.
procMount: Unmasked (agent container)	bubblewrap mounts a fresh /proc for each command, which the kernel refuses while parts of the container’s /proc are masked.

Turn it on:

1. Check the nodes and install the two profiles (below).
2. Set sandbox: bubblewrap under integrations.runtime in team.yaml.
3. kubectl apply -k examples/k8s-sandbox instead of examples/k8s.

When the pod lacks one of these, the agent stops at start with one error that names the missing piece and its fix, for example /proc is masked, so bubblewrap cannot mount a fresh /proc .... Read it with kubectl -n outstaff-acme logs sts/outstaff-agent-sam. ost doctor <handle> run inside the agent container shows the same check.

What the nodes need

Requirement	Check on a node
Kubernetes 1.33 or later	kubectl version. On 1.30 to 1.32, turn on the UserNamespacesSupport and ProcMountType feature gates.
containerd 2.0 or later	kubectl get nodes -o wide (CONTAINER-RUNTIME column)
runc 1.2 or later	runc --version (k3s: /var/lib/rancher/k3s/data/current/bin/runc --version)
Linux 6.3 or later	uname -r
Unprivileged user namespaces	cat /proc/sys/user/max_user_namespaces is above 0
Volumes with idmapped mounts	The agent’s home volume and /var/lib/kubelet on ext4, xfs, btrfs or tmpfs. NFS does not work.
AppArmor	cat /sys/module/apparmor/parameters/enabled; Y means the AppArmor profile must be loaded

Managed clusters often do not let you add node profiles or user namespaces; there, leave the sandbox off.

Install the profiles on each node

The kubelet reads Localhost seccomp profiles from <kubelet root>/seccomp/, by default /var/lib/kubelet/seccomp/ (k3s uses the same path unless --kubelet-arg=root-dir changes it). AppArmor profiles are loaded into the kernel by name. On every node that may run an agent:

sudo install -D -m 0644 examples/k8s-sandbox/seccomp/outstaff-bwrap.json /var/lib/kubelet/seccomp/outstaff/bwrap.json
sudo install -m 0644 examples/k8s-sandbox/apparmor/outstaff-agent /etc/apparmor.d/outstaff-agent
sudo apparmor_parser -r -W /etc/apparmor.d/outstaff-agent    # only where AppArmor is enabled
sudo aa-status | grep outstaff-agent                          # loaded, in enforce mode

Files under /etc/apparmor.d/ load again at boot. Use your configuration management, or a DaemonSet with a hostPath mount, to keep the files on new nodes. A pod whose profile is missing on its node does not start (CreateContainerError).

Ubuntu 24.04 nodes enable AppArmor, run pods under containerd’s cri-containerd.apparmor.d profile and set kernel.apparmor_restrict_unprivileged_userns=1, so without the outstaff-agent profile even an unconfined process cannot create the user namespace bubblewrap needs. Load the profile there. Every rule it adds to containerd’s default was checked to be needed on Ubuntu 24.04 with that setting.

If your nodes do not enable AppArmor, drop the appArmorProfile patch from examples/k8s-sandbox/kustomization.yaml. If the AppArmor profile does not work on your nodes, read the kernel’s denials (sudo journalctl -k | grep 'apparmor="DENIED"', or dmesg) and open an issue with them. appArmorProfile: { type: Unconfined } is the last resort: it lifts every AppArmor restriction on the agent, not only the ones bubblewrap needs.

Pod Security

procMount: Unmasked is not allowed by the restricted Pod Security level, even with hostUsers: false, so the overlay makes the namespace enforce baseline and still warn and audit against restricted, which the rest of the manifests meet. A namespace without Pod Security labels enforces nothing and needs no change. An unmasked /proc shows read-only kernel information such as /proc/timer_list; writes under /proc/sys need capabilities on the node that a pod with its own user namespace does not have.

Network policy

network-policy.yaml denies all traffic in the namespace, then allows:

• DNS to kube-dns;
• TCP 443 to the internet but not to private ranges or the cloud metadata address (169.254.0.0/16): the code host, tracker, chat service, LLM provider and tool sources;
• agents to the office on 8080, and nothing else into the office;
• the office to Postgres on 5432. The example allows 10.0.0.0/8; put the address of your database, or delete the policy if you use SQLite.

Services and devices you grant to an agent (Network grants) are usually on private addresses, so each needs an egress rule of its own.

This needs a CNI that enforces NetworkPolicy. HTTPS to any public address is broader than the allow-list in the container contract (an agent can run shell commands, and sockets do not ask the permission rules). If your CNI supports it, replace the ipBlock rule with FQDN rules for the hosts you name: api.github.com, github.com, your tracker, slack.com and its Socket Mode hosts, and api.anthropic.com or your cloud provider’s model endpoint, and the package registries of the roles that build or test code (for example registry.npmjs.org, so pnpm install works). Agents also need the hosts of the tool sources on their first start (the tools install into the home volume and later starts reuse them), so allow those temporarily or build an image with the tools in it (Container images).

Sizing

These are the starting points from the container contract, set as requests, with a memory limit and no CPU limit so a build can use idle CPU. Watch the real use for a week, then adjust.

Pod	Requests	Memory limit	Storage
Office	0.5 CPU, 512 MiB	1 GiB	5 GiB volume for SQLite; none needed with Postgres
Agent	1 CPU, 2 GiB	4 GiB	20 GiB home volume (<handle>-home)
Agent that builds and tests	2 CPU, 4 GiB or more	8 GiB or more	Size the volume for the clones and worktrees of every repository the person works in

Give an agent that runs your tests the memory of your biggest test run, not of the agent. Raise it with a patch in that person’s people/<handle>/kustomization.yaml, so one person can differ from the rest. Size the home volume for the installed tools, the clones and the worktrees, and cap ephemeral-storage so one runaway job cannot fill a node. With memory.store: volume (the default) the home volume also holds the agent’s notes, so back it up; with memory.store: knowledge the notes are in the knowledge integration, and a single-replica storage class without backups is enough. The first start of an agent installs its tools and takes longer than later ones.

Check it works

CI runs this for every pull request that touches examples/:

cp examples/acme/.outstaff/team.yaml examples/k8s/team.yaml
kustomize build examples/k8s | kubeconform -strict -summary -kubernetes-version 1.33.0 -
kustomize build examples/k8s-sandbox | kubeconform -strict -summary -kubernetes-version 1.33.0 -