---
title: "Autoscaling GitHub Actions runners with Karpenter"
description: "Connect ARC runner demand to EC2 capacity, compare zero-idle and warm runners, and measure startup and consolidation with reproducible burst tests."
url: https://9apes.com/blog/karpenter-github-actions-runners/
---

# Autoscaling GitHub Actions runners with Karpenter

Autoscaling a GitHub Actions runner pool involves two separate decisions: how
many runners should exist, and where Kubernetes can place their pods. Actions
Runner Controller handles the runner pool. Karpenter supplies nodes for pods
that cannot be scheduled. A fast response from one controller does not make the
other part instantaneous.

The useful question is how much startup time a team is willing to exchange for
idle capacity. This article uses a small EKS lab to make that tradeoff visible.
The [configuration, collector and results](/downloads/ci-lab/scaling-evidence.zip)
are included so that the experiment can be repeated with another workload.

## Separate runner scaling from node scaling

ARC's listener reacts to jobs assigned to its scale set. Runner pods request
CPU and memory. Kubernetes schedules those pods if suitable capacity exists;
otherwise Karpenter evaluates them against its NodePool constraints and asks
AWS for instances.

The path before a shell command can execute is therefore longer than a simple
queue:

1. GitHub creates and schedules the workflow jobs.
2. The ARC listener receives demand and runner resources are created.
3. Kubernetes places the pods, possibly after Karpenter creates EC2 capacity.
4. The nodes join the cluster and pull the runner image.
5. The runner starts, registers and begins the workflow step.

Measure those stages separately when their timestamps are available. A chart of
workflow creation to first command is useful, but it includes all of them. It
should not be labeled pure queue time.

## Keep the controllers on stable capacity

The lab uses one fixed `m7i.large` system node. ARC's controller and listener,
Karpenter and the cluster's system services run there. Runner pods select nodes
labeled `ci-lab-role: runner` and tolerate the matching worker taint.

This avoids a bootstrap dependency in which the only controller able to create
nodes is itself waiting for a node. It also makes the idle-cost baseline
explicit: scaling the worker NodePool to zero leaves the system node and EKS
control plane running.

A separate system node is not a security boundary against every Kubernetes
workload. Here it makes scheduling behavior easier to understand and keeps
ordinary runner CPU demand away from the controllers.

## Constrain the worker NodePool

The example uses Karpenter `1.14.1`, EKS Kubernetes minor `1.36`, AL2023 release
`1.36.3-20260911`, and on-demand `c7i.xlarge` workers. Restricting instance type
and purchase model removes variation that would otherwise complicate this
small comparison.

```yaml
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: [amd64]
        - key: karpenter.sh/capacity-type
          operator: In
          values: [on-demand]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: [c7i.xlarge]
  limits:
    cpu: '12'
    memory: 24Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 60s
```

This is an excerpt; the download includes the `nodeClassRef`, labels, taint,
AMI selection, subnet and security-group selectors, metadata configuration and
root-volume settings.

The limits allow a planned maximum of three four-vCPU workers. They are
[eventually consistent during provisioning](https://karpenter.sh/docs/concepts/nodepools/),
so they are not a hard upper bound on instantaneous resource creation or a
billing cutoff. A supervised session deadline and verified teardown remain
necessary.

Each runner requests and is limited to one CPU and 2 GiB of memory. Requests
matter to scheduling: a low request with a much higher actual workload can make
a node appear to have capacity that it cannot deliver reliably. Keep the
requests, limits and workload unchanged when comparing pool configurations.

## Compare zero-idle capacity with two warm runners

The ARC scale set always has `maxRunners: 8`. The experiment changes only the
minimum number of idle runners:

| Configuration | Preparation before each burst |
| --- | --- |
| Cold | Set `minRunners: 0`; wait until all runner worker Nodes, NodeClaims and runner pods are gone |
| Warm | Reset to the cold baseline, then set `minRunners: 2`; require exactly two Ready, connected idle runners |

Pod readiness alone does not establish a GitHub connection. The collector checks
for two online, non-busy runners in the repository runner API. If scale-set
runners are not exposed there, it requires each fresh runner's startup log to
contain `Listening for Jobs` and no indication that a job has started. The
sanitized observations record which criterion was used and when it was checked.

Every warm repetition starts from zero worker capacity before establishing that
reserve. Extra empty nodes and in-flight NodeClaims must be gone before the
timer starts. This prevents a later repetition from benefiting from spare
capacity left over by the preceding burst. Preparation is excluded from the
startup measurement. Initial worker counts are still recorded because two
protected idle runners can occupy one or two nodes.

The warm configuration reserves two runners, not enough capacity for every
possible burst. When eight jobs arrive, the maximum of eight runners takes
precedence over retaining two additional idle runners. Some work in that burst
can still require new nodes.

This is also different from keeping EC2 instances warm without registered
runners. An idle instance can avoid EC2 boot time while still requiring a pod,
image pull and runner registration. Choose the warm resource you actually mean
to purchase.

## Account for a reset problem in the tested ARC release

Our ARC `0.14.2` deployment hit a specific preparation failure when changing
`minRunners` from two to zero: the listener was removed, while two connected idle
EphemeralRunners remained and blocked its recreation. The unchanged 15-minute
preparation deadline expired before a workflow was dispatched.

The symptom matches [ARC issue 4432](https://github.com/actions/actions-runner-controller/issues/4432).
We checked the actual release ancestry: the tested tag does not include the
[in-place upgrade change in PR4516](https://github.com/actions/actions-runner-controller/pull/4516).
We kept the tested version unchanged.

The revised preparation helper resets only GitHub-confirmed idle resources
while the listener is absent and `minRunners` is zero. It checks that no repository
workflow is queued or running and verifies the exact controller UID chain from
each Pod through its EphemeralRunner and EphemeralRunnerSet to this scale set.
It refuses unknown, busy or differently owned resources and preserves normal
Kubernetes finalizer cleanup.

This operation happens before the startup timer. The zero-capacity and two-idle
readiness gates remain the same. Failed preparations and protocol revisions are
retained in the download, including the interruption used to harden the guard;
no timed job ran in that interrupted attempt.

## Use a workload that measures startup clearly

The included workflow contains no checkout or third-party action. Its first
shell command records an epoch timestamp; it then sleeps for 20 seconds and
records another timestamp. Bursts contain one, four or eight jobs, with three
repetitions of each burst for each capacity configuration.

The sleep is deliberate. This is synthetic occupancy used to examine startup
and placement, not a claim about application build speed. Real builds need their
own CPU, memory, disk and cache measurements.

The workflow bounds concurrency at eight and has a ten-minute per-job timeout.
The collector gives a workflow a separate overall deadline, preserves failures,
and stops the experiment when preparation or execution cannot complete. Builder
experiments run in a separate time slot so they cannot silently provide warm
nodes for a supposedly cold trial.

To run the protocol after completing the EKS lab setup:

```bash
python3 examples/ci-lab/scaling/collect.py \
  --mode suite \
  --output .omx/state/technical-authority/scaling
```

Use the private output directory for raw logs and identifiers. The accompanying
summary tool strips runner names, cloud instance identifiers and node addresses
before producing publishable evidence. Review the sanitized files before
sharing them.

## Measured startup results

Each cell summarizes three successful burst repetitions. The median shown is the
median of the three per-trial job medians; ranges span all jobs in those
repetitions. Values are displayed to one decimal place, with the clock limitation
described below.

| Submitted jobs | Cold median (s) | Two-idle median (s) | Cold job range (s) | Two-idle job range (s) |
| --- | ---: | ---: | --- | --- |
| 1 | 44.7 | 4.6 | 44.7 to 46.9 | 4.1 to 4.9 |
| 4 | 47.5 | 5.8 | 44.8 to 56.6 | 4.2 to 30.9 |
| 8 | 51.5 | 31.8 | 46.4 to 67.8 | 3.7 to 52.9 |

All 78 timed jobs succeeded. Every cold trial began with zero worker Nodes and
NodeClaims; every warm trial began with two connected idle runners. Every warm
trial began on one worker node. The download retains three preparation failures:
a Helm value-type error, an ARC reset timeout, and an interrupted recovery while
ownership checks were strengthened. None dispatched a job. Completed successful
keys were not redispatched during recovery.

The smallest bursts benefit from already registered runners. An eight-job burst
can still require additional capacity despite the two-runner reserve, which is
why the complete job range matters alongside a median. This is a synthetic
startup experiment, not a production build-throughput result.

## Define the timestamps before drawing a chart

The collector retains several different timestamps:

| Timestamp | What it establishes |
| --- | --- |
| Local dispatch start | When the operator began the GitHub workflow-dispatch request |
| GitHub workflow `created_at` | GitHub's recorded creation time for the workflow run |
| GitHub job `started_at` and `completed_at` | Provider-reported job lifecycle timestamps |
| First shell-command timestamp | When the synthetic workflow's first recorded command ran |
| Kubernetes node and pod creation/Ready timestamps | The observed infrastructure and scheduling lifecycle |
| Sampled disappearance of a node | A time interval in which the node stopped appearing in Kubernetes |

The principal comparison is GitHub workflow creation to first shell command.
Local dispatch to first command is retained separately because it also includes
the dispatch/API interval. These are cross-system UTC timestamps, and GitHub's
creation timestamp has whole-second resolution. The runner and GitHub wall
clocks were not independently calibrated, so their relative skew is unknown.
Decimal places in the table are display rounding, not evidence of subsecond
accuracy. This limitation matters especially for warm starts lasting only a few
seconds.

Kubernetes is sampled at roughly five-second intervals plus API-call overhead.
A node disappearing between observations does not identify the exact time AWS
stopped billing for its instance. To calculate operating cost, join those
observations to EC2 instance lifetime and traffic records instead of assuming
that a missing Kubernetes Node means zero infrastructure cost at that instant.

For each burst size, compare the three per-trial median startup times and retain
the range across all jobs. The eight-job trials contain more job observations
than the single-job trials; treating every job as an independent repetition
would overstate how much evidence the experiment provides.

## Observe consolidation after the job

The NodePool permits consolidation after its configured delay. Runner pods carry
`karpenter.sh/do-not-disrupt: 'true'` to reduce voluntary disruption while they
are active. A separate 180-second synthetic job lets the collector observe a
worker while the job is running and after the runner exits.

The right evidence is a timeline: worker appeared, runner became Ready, the
first command ran, the job finished, the runner disappeared, and the idle worker
was removed. A single successful long job is not evidence that a node is
protected against every interruption. Node expiry, forced disruption, instance
failure and other mechanisms have their own behavior. Consult Karpenter's
[current disruption documentation](https://karpenter.sh/docs/concepts/disruption/)
when choosing the policy for a production workload.

Do not shorten the consolidation interval solely because an empty node feels
wasteful. If closely spaced jobs keep arriving, repeated node creation and image
pulls can exchange a small amount of idle cost for recurring startup delays.
The appropriate interval depends on the arrival pattern you observe.

The separate held-job observation completed successfully: the shell interval was
180.0 seconds, and the worker remained registered in all
17 snapshots taken during that interval. The first observation with no
worker Nodes, NodeClaims or runner pods occurred about 119 seconds after the last shell
command. That is a sampled Kubernetes retirement observation, not an exact EC2
termination timestamp or a test of forced-disruption resistance. Observer and
runner clock skew was not independently calibrated.

## Put a price on the warm capacity

A warm pool buys time with idle resources. The relevant idle cost includes the
worker capacity kept alive for those runners, its storage and public addresses,
as well as the fixed system-node and control-plane baseline. Account for
consolidation and image-pull traffic when comparing the alternatives.

For a team making this decision, three questions are more useful than asking
whether autoscaling is generally faster:

- How often does work arrive after the worker pool has reached zero?
- How much does an extra minute before the first useful command affect developer wait time?
- How often do bursts exceed the number of runners kept warm?

Use the answer to pick an initial minimum, observe the workload, and revise it.
A two-runner warm pool is a test configuration, not a universal optimum.

## Finish by deleting the capacity

Restore `minRunners: 0` after the warm trials. Cancel outstanding lab jobs and
remove any builder pods before deleting the NodePool. Keep Karpenter and its
permissions running until worker deletion finishes; then remove the controllers,
EKS cluster and dedicated IAM stack using the included teardown procedure.

Verify cloud resources independently of Kubernetes objects. The experiment is
complete when its evidence is saved and its dedicated infrastructure has been
removed, not merely when the workflow dashboard turns green.