Change build limits
Raise or lower the CPU, memory and scratch disk one build may use on a cluster, and the quota that caps all builds together.
Every build runs with fixed limits, so one build cannot take a node from the workloads that share it. The defaults suit an ordinary application image. A build that is killed for memory, or evicted for disk, needs larger limits.
Before you begin
- You have shell access as root to a server node of the cluster.
- Know which limit the build hit. See Troubleshooting builds.
- The cluster's install manifest is from 0.42.0 or newer. On an older cluster, re-apply the manifest first. Until then builds use the defaults, and the settings below are ignored.
The limits
| Setting | Default | Limits |
|---|---|---|
buildkit-cpu | 4 | CPU of the BuildKit container, which runs the Dockerfile. |
buildkit-memory | 8Gi | Memory of the BuildKit container. A build over it is killed. |
state-size | 10Gi | BuildKit's layer storage: base images and every layer the build writes. |
source-size | 2Gi | The checked-out repository. |
cache-size | 4Gi | The build cache pulled from the registry. |
result-size | 6Gi | The finished image and the cache the build exports. |
The three helper containers that clone the repository, plan a Railpack build and push the image have fixed limits of 1 CPU and 1 GiB of memory each. They need no tuning.
A build can use the sum of the four sizes and a margin for BuildKit's own files, about 24 GiB at the defaults. A volume that fills ends the build as failed.
All builds on a cluster together are capped by a quota: 4 pods, 24 CPUs and 40 GiB of memory in limits, and 16 GiB of requested scratch disk. The control plane runs 2 builds at a time on a cluster. Raise the quota with a setting, and it must stay above what 2 builds ask for.
Steps
Set the value. Each key takes a Kubernetes quantity, for example 16Gi or 6. A value that is not a valid positive quantity is ignored and the default stays.
k3s kubectl -n nebula-build patch configmap nebula-build-resources --type merge -p '{"data":{"buildkit-memory":"16Gi"}}'If the new limits no longer fit the quota, raise the quota. QUOTA_LIMIT_MEMORY is the new total, for example 64Gi.
k3s kubectl -n nebula-build patch resourcequota nebula-build --type merge -p '{"spec":{"hard":{"limits.memory":"QUOTA_LIMIT_MEMORY"}}}'Applying the install manifest again resets the quota to its defaults. The values in nebula-build-resources are kept. After a re-apply, check the quota again.
Select Retry on the failed build. New builds read the settings when they start; a build that is running keeps its limits.
Verify
The new build's pod shows the new limit:
k3s kubectl -n nebula-build get pod -l nebula.dev/build -o jsonpath='{.items[0].spec.containers[0].resources.limits}'{"cpu":"4","ephemeral-storage":"24640Mi","memory":"16Gi"}Reset a limit
Delete its key, and the default applies again:
k3s kubectl -n nebula-build patch configmap nebula-build-resources --type json -p '[{"op":"remove","path":"/data/buildkit-memory"}]'Next steps
Run builds on a dedicated node
Reserve one node of a cluster for builds, so a repository's Dockerfile never runs on the nodes that host your services, the agent or its access broker.
nebula.toml service file
Every table and key of the nebula.toml that configures one service from its repository, with types, defaults, limits and the problems the parser reports.