Auto-scaling

Capacity planning used to be a bet placed weeks in advance: measure your peak, add a safety margin, buy the hardware, and hope the forecast holds. The bet is wrong in both directions at once. Provision for the peak and the machines sit idle through every trough — you pay 24 hours a day for a load that arrives for two. Provision for the average and the first real spike falls over. Either way a human is in the loop, reading dashboards and racking servers, on a timescale far slower than demand actually moves.

Auto-scaling replaces the bet with a control loop. Instead of choosing a capacity, you choose a target — a utilization level, a queue depth, a request rate per instance — and hand a controller the authority to add and remove capacity to keep the system at that target as load rises and falls. Demand doubles at 9 a.m.; the controller notices and grows the fleet. Demand drains overnight; it shrinks the fleet back. Nobody is awake for either. You stop paying for capacity you might need and start paying for load you actually have.

The problem

Consider a stateless service behind a load balancer. Traffic is not flat — it has a daily rhythm, weekly peaks, and the occasional unforecast spike when a link goes viral or a batch job kicks off. You must decide how many instances to run, and every fixed answer is wrong:

• Provision for peak. Size the fleet for the busiest minute of the busiest day. It never falls over, and it burns money continuously — most instances idle most of the time, and you are paying for the 99th-percentile minute during the 50th-percentile hour. The unforecast spike above your peak still breaks you.
• Provision for average. Size for typical load and spend far less. Now every peak is a brownout: latency climbs, queues back up, requests time out, and recovery waits on a human noticing and reacting on a human timescale.
• Provision by hand, reactively. Keep someone watching, adding capacity when it gets busy. This does not scale as an operational practice — it needs a person awake, it reacts in minutes-to-hours, and it turns every traffic surge into a page.

All three couple capacity to a guess about the future instead of to observed present demand. What you actually want is for capacity to be a function of load, evaluated continuously, acted on automatically — the system provisioning itself in response to what is happening right now, not what someone predicted last quarter.

The subtlety that makes this genuinely hard: capacity does not appear instantly. Between "demand spiked" and "new instance serving traffic" sits a delay — metrics must be collected and averaged, a threshold must be breached for long enough to be believed, and then an instance must boot, pass health checks, register, and warm its caches. That lag is the whole game. Scale on a signal that is too noisy and you thrash; scale on one too smoothed and you react after the spike has already hurt you.

When to use it

Auto-scaling earns its keep when demand is variable and capacity is elastic and stateless:

• Load varies meaningfully over time. Daily peaks, weekly cycles, campaign spikes, unpredictable virality. If your traffic is genuinely flat, a fixed fleet is simpler and there is nothing to scale. The pattern pays off in proportion to how far peak sits above trough.
• The workload is horizontally scalable and stateless. Adding a copy adds capacity because any instance can serve any request. Twelve-factor services behind a load balancer, stateless workers pulling from a queue — these scale by count. State pinned to a specific instance does not.
• Capacity is rented, not owned. Elasticity only saves money if you can release capacity and stop paying for it. On cloud infrastructure you scale in and the meter stops; on hardware you own, the idle machine costs the same powered down, so the economic case is weaker.
• A clean demand signal exists. CPU utilization, request rate per instance, queue depth, p95 latency — something that reliably tracks "am I under-provisioned right now." The quality of that signal is the ceiling on how well any policy can perform.

When NOT to use it

The assumptions are also the disqualifiers:

• The bottleneck is not the tier you are scaling. Adding stateless web instances does nothing if the real constraint is a single primary database, a third-party rate limit, or a lock. You will scale the fleet, watch latency stay pinned, and pay more for the privilege. Find the actual bottleneck before you automate scaling the wrong tier. (This is where Circuit Breaker belongs — to protect the downstream that cannot scale from the fleet that just did.)
• Instances are slow or expensive to start. If boot-plus-warm-up takes many minutes — heavy images, huge caches to prime, long JIT warm-up — the controller reacts far too late to matter for spikes, and you are back to pre-provisioning headroom. Auto-scaling assumes capacity arrives on a timescale comparable to how fast demand moves.
• Load is flat, or perfectly predictable and constant. No variance, nothing to track; a right-sized fixed fleet is cheaper and has fewer moving parts. (Predictable variance — a known 9 a.m. surge — is a case for scheduled scaling, covered under performance below.)
• The workload is stateful and instance-pinned. Sessions in local memory, a leader that must be singular, sharded data bound to specific nodes — scaling by count corrupts or loses state. Externalize the state first, or this pattern actively harms you.
• Scaling actions are themselves costly or disruptive. If every scale-in drops in-flight work or every scale-out triggers an expensive rebalance, an eager controller manufactures the instability it was meant to absorb. The action must be cheap and safe to take, or the cure is worse than the load.

Architecture

Auto-scaling is a feedback loop wrapped around a fleet. A metrics source observes the running instances; a scaling policy compares the observed metric to a target and computes a desired capacity; an actuator reconciles the running count toward that desired count; and the changed fleet feeds new metrics back in. It is the same observe → decide → act loop a thermostat runs — the target is a set-point, the fleet is the room.

flowchart LR
    Users((Demand)) --> LB[Load balancer]
    LB --> Fleet[Instance fleet]
    Fleet -- CPU / RPS / queue depth --> Metrics[Metrics source]
    Metrics --> Policy{Scaling policy\ncompare to target}
    Policy -- desired count --> Actuator[Actuator]
    Actuator -- add / remove --> Fleet
    Fleet -. register / deregister .-> Registry[(Service registry)]
    Registry --> LB

The loop runs continuously, but two things keep it from oscillating. First, an evaluation period: the metric must stay past its threshold for a sustained window, not a single noisy sample, before the policy acts — this filters spikes-of-measurement from spikes-of-demand. Second, a cooldown: after a scaling action, the controller waits for the fleet to stabilize and for fresh metrics to reflect the change before acting again, so it does not stack a second scale-out on top of the first before the first has had any effect.

sequenceDiagram
    participant M as Metrics
    participant P as Policy
    participant A as Actuator
    participant F as Fleet
    loop every evaluation interval
        M->>P: avg CPU = 82% (target 65%)
        P->>P: desired = ceil(running × 82 / 65)
        P->>A: scale out to desired
        A->>F: launch instances
        F-->>A: healthy + registered
        Note over P: cooldown — ignore metrics until fleet settles
    end
    Note over M,F: demand drains → avg CPU = 30% → scale in (slowly)

The asymmetry is deliberate and load-bearing: scale out eagerly, scale in reluctantly. Being slow to add capacity risks an outage during a spike; being slow to remove it risks a little wasted spend. Those costs are not symmetric, so the policy should not treat them as such — aggressive thresholds and short cooldowns for growth, conservative thresholds and long cooldowns for shrink.

Code walkthrough

The heart of auto-scaling is the policy — a pure function from an observation to a desired capacity — kept rigorously separate from the actuator that makes the world match. Isolating the policy is what makes scaling behavior testable without booting a single machine. Start with the vocabulary.

interface Observation {
  runningInstances: number;
  metricValue: number;
}

interface ScalingBounds {
  min: number;
  max: number;
}

type ScalingDecision =
  | { action: "scale-out"; to: number }
  | { action: "scale-in"; to: number }
  | { action: "hold" };

Observation is what the metrics source reports; ScalingBounds are the floor and ceiling that keep the controller from scaling to zero or to bankruptcy. The decision is a discriminated union — three legal outcomes, nothing else representable.

The policy itself is target-tracking: pick the instance count that would bring the metric back to its target, using the standard proportional rule desired = ceil(running × current / target). It is a pure function, so it is exhaustively unit-testable.

interface TargetPolicy {
  targetMetric: number;
  scaleOutThreshold: number;
  scaleInThreshold: number;
}

function decide(
  obs: Observation,
  policy: TargetPolicy,
  bounds: ScalingBounds,
): ScalingDecision {
  const clamp = (n: number) => Math.max(bounds.min, Math.min(bounds.max, n));
  const desired = clamp(Math.ceil((obs.runningInstances * obs.metricValue) / policy.targetMetric));

  if (obs.metricValue >= policy.scaleOutThreshold && desired > obs.runningInstances) {
    return { action: "scale-out", to: desired };
  }
  if (obs.metricValue <= policy.scaleInThreshold && desired < obs.runningInstances) {
    return { action: "scale-in", to: desired };
  }
  return { action: "hold" };
}

Two thresholds straddle the target and create a dead band between them: as long as the metric sits between scaleInThreshold and scaleOutThreshold, the policy holds. Without that band the controller would hunt — scaling out at exactly the target, which drops the metric just below it, which scales back in, forever. The dead band is what turns a twitchy loop into a stable one.

Now the controller that runs the loop over time. It owns the one thing the policy must not: time-dependent state — the cooldown that keeps it from acting again before the last action has landed.

interface Clock {
  now(): number;
}

interface Fleet {
  runningInstances(): Promise<number>;
  setDesiredCapacity(count: number): Promise<void>;
}

class AutoScaler {
  private nextEligibleAt = 0;

  constructor(
    private readonly fleet: Fleet,
    private readonly policy: TargetPolicy,
    private readonly bounds: ScalingBounds,
    private readonly clock: Clock,
    private readonly cooldown: { scaleOutMs: number; scaleInMs: number },
  ) {}

  async evaluate(metricValue: number): Promise<ScalingDecision> {
    if (this.clock.now() < this.nextEligibleAt) return { action: "hold" };

    const running = await this.fleet.runningInstances();
    const decision = decide({ runningInstances: running, metricValue }, this.policy, this.bounds);

    if (decision.action === "hold") return decision;

    await this.fleet.setDesiredCapacity(decision.to);
    const wait = decision.action === "scale-out" ? this.cooldown.scaleOutMs : this.cooldown.scaleInMs;
    this.nextEligibleAt = this.clock.now() + wait;
    return decision;
  }
}

The cooldown is asymmetric by construction — scaleOutMs short so the controller can keep growing into a rising spike, scaleInMs long so it removes capacity slowly and never yanks away instances it will want back in a minute. The Clock is a port precisely so a test can advance time and assert the cooldown holds without any real waiting.

const scaler = new AutoScaler(
  fleet,
  { targetMetric: 65, scaleOutThreshold: 75, scaleInThreshold: 45 },
  { min: 2, max: 40 },
  systemClock,
  { scaleOutMs: 60_000, scaleInMs: 300_000 },
);

await scaler.evaluate(82); // → scale-out: 82 > 75, desired = ceil(running × 82/65)
await scaler.evaluate(88); // → hold: still inside the 60s scale-out cooldown
// …minutes pass, spike drains…
await scaler.evaluate(30); // → scale-in: 30 < 45, but only after the 5-min cooldown

The lesson is the separation: the policy decides what capacity is right from a single observation, and the controller decides when it is allowed to act. Keeping those apart is what makes scaling behavior something you can reason about and test, rather than an emergent property of a shell script and three CloudWatch alarms.

Choosing the signal

A scaling policy is only as good as the metric it tracks, and the default — CPU utilization — is often the wrong one.

• CPU / memory utilization. The classic signal, and fine when the work really is CPU-bound. It lies when your service is I/O-bound: an instance can be 100% saturated on downstream latency while its CPU sits at 20%, and a CPU policy will happily under-provision it into the ground.
• Requests per instance. Often better for web tiers — it tracks the thing you actually care about (load per unit of capacity) and is linear in demand, so the proportional rule behaves predictably.
• Queue depth / backlog. The right signal for worker fleets pulling from a Message Queue. Scaling on backlog-per-worker targets the latency your jobs experience directly — a growing queue is under-provisioning made visible, and it is the single cleanest auto-scaling signal there is.
• p95/p99 latency. Tempting because it is what users feel, but dangerous as a primary trigger: latency is a lagging, non-linear symptom, and by the time it moves you are already hurting. Use it as a guardrail, not the steering wheel.

The rule of thumb: scale on the signal closest to the resource that actually saturates first, and prefer one that is linear in demand so the proportional policy stays predictable.

Performance characteristics

The metrics above are indicative — orders of magnitude to build intuition, not measurements from a specific system.

Reaction time is the number that sizes everything else. The lag from spike to serving capacity is the sum of the metric window, the evaluation period, and instance boot-plus-warm-up — call it one to five minutes. You cannot policy your way below it; you can only hide it by carrying enough headroom (a lower utilization target) to absorb the demand that arrives during the lag. A faster-booting instance buys a higher target and a smaller idle margin — which is why warm pools and lightweight images matter more to cost than any threshold tuning.

Cost tracks the demand curve instead of the peak. The whole economic case is that the area under "capacity over time" shrinks from a rectangle sized to the peak down to something that hugs the load curve with a headroom margin on top. The savings are proportional to how spiky your traffic is: a peak-to-trough ratio of 5× makes auto-scaling transformative, a ratio of 1.2× makes it barely worth the operational complexity.

Thrashing is the failure mode, and it is a tuning failure. A controller that scales out and in repeatedly around the target wastes money on churn, destabilizes downstream systems, and can amplify the very load spikes it should absorb. The three defenses — a dead band between thresholds, an evaluation period that demands sustained breach, and asymmetric cooldowns — are not optional polish. They are what separates a stable control loop from an oscillator.

Predictable variance deserves prediction, not just reaction. Pure reactive scaling always pays the reaction-time tax on every spike. When the spike is scheduled — the 9 a.m. login surge, the Friday batch — scheduled scaling provisions ahead of it on a clock, and predictive scaling learns the recurring shape from history and pre-warms capacity before demand arrives. Both eliminate the lag for the load you can see coming; reactive scaling remains the backstop for the load you cannot.

Live demo

An interactive Auto-scaling playground is coming soon. You will be able to drive a synthetic demand curve — a gentle daily wave, a sudden viral spike, a slow drain — and watch a target-tracking controller chase it: instances launching as utilization crosses the scale-out threshold, the dead band holding steady through noise, cooldowns gating the next action, and the fleet shrinking reluctantly as load falls. A "misconfigured" mode will let you narrow the dead band and shorten the cooldown to feel the controller thrash, and a scheduled-scaling toggle will pre-warm capacity ahead of a known spike so you can see the reaction-time tax disappear. Check back shortly.

Related patterns

Service Discovery is the mechanism that makes an elastic fleet actually usable. Every instance the scaler launches must become reachable and every one it terminates must stop receiving traffic — which is exactly registration and deregistration in a service registry. Auto-scaling changes how many instances exist; service discovery keeps the load balancer's notion of which instances exist correct as that number moves. Without it, scaling out adds capacity nobody routes to and scaling in blackholes the requests still in flight to a terminated node.

Circuit Breaker is the pattern you reach for when you scale a tier and discover the bottleneck was downstream. A fleet that auto-scales in front of a database or third-party API that cannot scale will simply pile more concurrent load onto the fixed resource — turning a slow dependency into a collapsed one. The circuit breaker caps that pressure, failing fast and shedding load so the un-scalable downstream survives the scaled-up fleet. Auto-scaling and circuit breaking are the two halves of elasticity done responsibly: grow where you can, protect what you cannot.

Message Queue turns auto-scaling from reactive guesswork into a clean control problem. When workers pull from a queue, the backlog is the demand signal — queue-depth-per-worker directly measures whether you have enough capacity, and the queue itself buffers the reaction-time lag by holding work safely until new workers come online. A spike no longer risks dropped requests; it risks a temporarily longer queue that the scaler drains. It is the most robust substrate there is for auto-scaling a fleet, because the buffer and the signal are the same thing.