---
title: "Building container images in Kubernetes without privileged pods"
description: "Compare rootless BuildKit and maintained Kaniko with reproducible Kubernetes manifests, explicit permission tradeoffs, and OCI image validation."
url: https://9apes.com/blog/build-images-kubernetes-without-privileged-pods/
---

# Building container images in Kubernetes without privileged pods

You can build a container image in Kubernetes without setting `privileged: true` and without mounting the host's Docker socket. Rootless BuildKit and Kaniko offer two ways to do it. Their permission requirements are different, and neither tool name is a substitute for examining the pod it runs in.

This walkthrough builds the same small Go program with both tools. It produces an image archive, checks the image's content hashes and configuration, and runs the resulting executable on the build node. There are no registry pushes or registry credentials in the experiment.

The [manifests, fixture, verification scripts, and observed results](/downloads/ci-lab/builders-evidence.zip) are available as a download. The examples use the maintained `osscontainertools/kaniko` project. Its documentation identifies it as a replacement for the archived original repository. Pin the intended fork and image digest instead of copying an old `latest` reference. [Kaniko project](https://github.com/osscontainertools/kaniko)

## Tested environment and result

The complete recipe passed on September 15, 2026, between 23:13 and 23:17 UTC. Both tools ran on EKS with Kubernetes `v1.36.4-eks-4cc7921`, `c7i.xlarge` worker capacity, Amazon Linux 2023, and containerd 2.2.7. The recorded node kernel was `6.18.44-99.149.amzn2023.x86_64`.

| Check in the final run | BuildKit v0.33.0 rootless | Kaniko v1.28.4 |
|---|---|---|
| Builder exited successfully | Yes | Yes |
| Privileged mode | Disabled | Disabled |
| OCI descriptor and uncompressed layer hashes | Passed | Passed |
| Image architecture | Linux amd64 | Linux amd64 |
| Verified executable size | 1,933,474 bytes | 1,933,474 bytes |
| Executable mode and ownership in the layer | `0755`, UID/GID `0:0` | `0755`, UID/GID `0:0` |
| Remote executable matches verified bytes and SHA256 | Yes | Yes |
| Expected native program output | Passed | Passed |

The two executables had the same SHA256 in this run. The image manifests differed. Both archives, their full hashes, the actual builder image IDs, fixture checksums, logs, and cleanup confirmation are included in the download.

This is a functional check of the supplied configurations, not a speed comparison. The rootless configuration uses the native snapshotter, and a single completed recipe does not establish comparative performance. Preliminary runs exposed two output-handling issues described below; they did not require changing either builder's security context.

## Define what “without privileged” means

`privileged: false` is one property of a container's security context. It does not tell you whether the process runs as root, whether syscall filtering is enabled, what capabilities it has, or which networks it can reach.

Here are the choices made by these manifests:

| Property | Rootless BuildKit | Maintained Kaniko |
|---|---|---|
| Privileged container | Disabled | Disabled |
| Builder UID | 1000 | 0 inside the container |
| Export and native-smoke UID | 1000 | 0, with all capabilities dropped |
| Seccomp profile | Unconfined | RuntimeDefault |
| AppArmor profile | Unconfined | Runtime default behavior |
| Privilege escalation | Allowed for namespace setup helpers | Disabled |
| Dockerfile process sandbox | Disabled by the daemon flag | Executes within the executor's container environment |
| Docker socket or hostPath | None | None |
| Service-account token | Not mounted | Not mounted |
| Output | Local OCI archive | Local OCI layout, then archive |

These are compatibility configurations for a trusted fixture in a dedicated lab. They are not examples of compliance with Kubernetes Restricted Pod Security. No network-isolation test is implied by leaving credentials out of the manifest.

## Prepare the same build context

The workload has a Go module with one pinned dependency. Its Dockerfile uses a digest-pinned Go base, downloads the module dependencies, compiles a static executable, and copies that executable into a `scratch` image. The final image declares a non-root user.

Both tools receive exactly the same four files through a ConfigMap: `Dockerfile`, `go.mod`, `go.sum`, and `main.go`. This keeps source checkout, Git credentials, and registry authentication out of the builder comparison. It also makes the context small enough to inspect directly.

ConfigMap volume files are projected through symlinks. A non-root init container copies those four files with `cp -L` into an `emptyDir` workspace before either builder starts. The build then receives ordinary files with the recorded contents, avoiding differences in how a builder follows projected-volume links.

A ConfigMap is suitable for this tiny fixture. A real repository generally needs another context transport, such as a checked-out workspace or a carefully scoped artifact download. Treat source retrieval as part of the build's trust boundary and supply only the files that build requires.

The runner records the checksum of each input file. That connects the output to a particular context without assuming that a repository branch remained unchanged during the experiment.

## Run rootless BuildKit

The rootless image starts BuildKit under a non-root identity. Its Kubernetes example requires explicit runtime exceptions. This manifest follows that approach and chooses the native snapshotter, avoiding a dependency on a usable FUSE device or a particular rootless overlay configuration. [BuildKit Kubernetes example](https://github.com/moby/buildkit/blob/v0.33.0/examples/kubernetes/job.rootless.yaml)

The core container settings are:

```yaml
securityContext:
  privileged: false
  runAsUser: 1000
  runAsGroup: 1000
  allowPrivilegeEscalation: true
  seccompProfile:
    type: Unconfined
  appArmorProfile:
    type: Unconfined
env:
  - name: BUILDKITD_FLAGS
    value: --oci-worker-no-process-sandbox --oci-worker-snapshotter=native
```

The builder runs `buildctl-daemonless.sh` with the Dockerfile frontend and local context. Its output option is:

```text
--output type=oci,dest=/out/image.tar,compression=gzip
```

The output directory and BuildKit state live in separate `emptyDir` volumes. The state volume is mounted at the rootless image's expected location. The pod's group setting lets the non-root builder write its output, and the context is read-only.

The exception worth understanding is `--oci-worker-no-process-sandbox`. Dockerfile commands share a process environment with the builder daemon instead of receiving the usual separate process sandbox. Upstream documents the resulting ability to interfere with daemon-container processes and limitations in cleaning up leftover build processes. Use a short-lived builder for each trusted build and keep unrelated work out of that pod. [BuildKit rootless limitations](https://github.com/moby/buildkit/blob/v0.33.0/docs/rootless.md)

Rootless operation also depends on host support for user namespaces. The experiment does not change node sysctls, disable host security systems, or add a privileged helper to make a failing environment work. If the node cannot support the configuration, record that failure and choose a compatible execution environment.

## Run maintained Kaniko

Kaniko unpacks the base filesystem and executes Dockerfile instructions within its container environment. The permissions needed depend on the base image and build commands. This example runs the executor as container UID 0 with the runtime's default capabilities, while keeping privileged mode and privilege escalation disabled. Calling this particular manifest rootless would be incorrect. [Kaniko security guidance](https://github.com/osscontainertools/kaniko#security)

Its relevant settings are:

```yaml
securityContext:
  privileged: false
  runAsUser: 0
  allowPrivilegeEscalation: false
  seccompProfile:
    type: RuntimeDefault
args:
  - --dockerfile=/workspace/Dockerfile
  - --context=dir:///workspace
  - --destination=ci-lab:local
  - --no-push
  - --no-push-cache
  - --cache=false
  - --oci-layout-path=/out/layout
  - --image-format=oci
  - --compression=gzip
```

The destination is a local image name for this experiment. The no-push flags and disabled remote cache keep output local. Pulling the public base image and downloading the Go module still require network access. Local output is not an offline build.

The OCI layout lives on the same kind of temporary output volume used by BuildKit. A small sidecar keeps that volume available after the executor exits, so the script can retrieve the result without adding a shell to the builder image or modifying the executor. Kaniko created a root-owned layer blob that a UID1000 exporter could not read in this experiment. Its exporter therefore uses UID 0 with all capabilities dropped and privilege escalation disabled. BuildKit's exporter stays UID1000.

## Keep output available long enough to verify it

A Kubernetes pod can still be running after the builder succeeds because the export sidecar is sleeping. Waiting for the entire pod to finish would confuse output retention with build completion.

Instead, the runner checks the builder container's terminated state and exit code. It then copies the archive through the sidecar and verifies it locally. Inside the pod, it extracts the identified layer's executable from the retained archive and requires its byte count and SHA256 to match the local verification before running it. The pod has a ten-minute active deadline, so the retention container cannot wait indefinitely if the client disappears.

Both builders are limited to one CPU and 2GiB of memory. They run sequentially on the designated lab node pool. Those limits bound this functional experiment; they are not a recommended production capacity plan or a fair performance benchmark.

From the repository root, select the dedicated lab context and a new output directory:

```sh
python3 examples/ci-lab/builders/run.py \
  --context YOUR_DEDICATED_LAB_CONTEXT \
  --output /absolute/path/to/new-builder-results
```

The script creates its own namespace and refuses to adopt an existing one. It does not create a cluster, modify node settings, or provision registry services. The complete JSON manifests in the download can be read by `kubectl` directly.

## Check the result, not just the exit code

A successful builder exit is only the first assertion. The verifier checks:

1. The OCI index points to one image manifest, with matching descriptor sizes and SHA256 hashes.
2. The manifest's configuration and every layer match their declared hashes, including uncompressed layer digests.
3. The image declares Linux, the expected architecture, `/ci-lab` as its entrypoint, and the expected non-root image user. The executable's archived Unix mode must grant that configured user execute permission.
4. The final executable has the matching ELF architecture and produces the expected output on the build node.

```text
ci-lab baseline 762033b8-63d9-5dd3-9da8-66718f9c7f98
```

Both layer entries were owned by UID/GID `0:0` with mode `0755`, which grants the configured UID/GID `65532:65532` execute permission through the other-user bits. That is a static file-permission check.

The native smoke test runs the extracted executable in the export sidecar as UID1000 for BuildKit and UID0 for Kaniko. It does not run under the final image's configured UID65532, start the generated image through a registry, or prove that every runtime will accept it. The image metadata and layer checks cover different assertions from the executable test, so the evidence reports them separately.

Two harness failures were useful checks on the verification itself. In the first attempt, an `exec` stdin copy returned success but left a zero-byte remote file, which failed with an executable-format error. Direct extraction plus explicit byte and hash checks replaced that transfer. In the second attempt, the non-root export helper could not read Kaniko's layer blob. Its final-run permissions were recorded as root-owned mode `0600`, so the Kaniko helper uses the matching UID while retaining its other restrictions. Both builds had completed successfully before those failures.

Do not assume two valid builds must have identical archive hashes. Image metadata, layer representation, and timestamps can differ between builders. Reproducibility across tools is a separate claim that requires controls beyond “the same Dockerfile built successfully.”

## Choose the boundary before choosing the builder

Rootless BuildKit is a reasonable candidate when its BuildKit features fit the workflow and the runtime can support its namespace requirements. This manifest makes its relaxed process and syscall boundaries visible rather than hiding them behind the word rootless.

Kaniko is a candidate when its execution model and Dockerfile support fit the environment. In this example it needs container root, so a policy requiring every process to run as a non-root UID would exclude this configuration even though privileged mode is disabled.

For untrusted pull requests, the larger decision is where the builder runs, which credentials and networks are reachable, and whether another workload shares the kernel or persistent state. Neither successful build demonstrates a boundary against hostile source code.

After retrieving the required results, confirm that `ci-lab-builders` has been deleted. Its `emptyDir` volumes disappear with its pods. Worker nodes and the EKS control plane have separate lifecycles and billing, so the lab operator must finish that cleanup separately.

## Building images on 9apes

Getting the image built is one part of running CI on Kubernetes. Your team also maintains worker nodes, builder images, permissions, caching and capacity. Rootless builds still need someone to own that infrastructure.

With 9apes, we manage the runner infrastructure, so your team can focus on Dockerfiles, build steps and tests. You connect the GitHub App, select a runner and update your workflow's `runs-on` label. Check your builder's requirements when moving the job, especially its privileges, storage and registry access.

To try it with one workflow, follow the [9apes quickstart](/docs/quick-start/).