> ## Documentation Index
> Fetch the complete documentation index at: https://docs.simplismart.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Autoscaling
> How Simplismart automatically grows and shrinks your deployment's pods to match demand
Every deployment runs on a number of pods. Each pod is a running copy of your model, ready to serve requests. Too few pods, and requests queue up while response times climb. Too many, and you're paying for GPU capacity that sits idle. Autoscaling keeps this balanced automatically — Simplismart monitors your traffic and adds or removes pods accordingly.
**If you're just getting started:** you don't need to read this whole page to configure autoscaling. Choose a metric and a threshold on the deployment form and leave the rest at their defaults — see [Picking a starting point](#picking-a-starting-point) for sensible defaults by workload type. The rest of this page explains the reasoning behind that behavior, for when you need finer control.
This page walks through how autoscaling decides to add or remove pods. For the step-by-step form fields, see [Creating a Deployment](/model-suite/deployments/creating-a-deployment).
Pick the metric, threshold, and min/max range that drive scaling.
Guarantee capacity for traffic patterns you already know about.
Control how fast pods are added or removed once a threshold is crossed.
Drop idle deployments to 0 pods and stop paying for them.
## How it decides
At regular intervals, Simplismart checks one number — your chosen metric — against a target you've set, called the **threshold**. If the number is above the threshold, it adds pods. If it's below, it removes pods. At the threshold, it leaves things as they are.
Two settings bound this decision on either side:
* **Min pods** — the floor. Your deployment never drops below this, even at zero traffic, unless Scale to Zero is enabled (covered below).
* **Max pods** — the ceiling, capped by your account quota. Your deployment never grows past this, regardless of demand.
The metric determines the direction; min and max determine how far it's allowed to go.
```mermaid theme={null}
flowchart LR
A["Measure metric\n(Memory · Latency · Throughput · Concurrency)"] --> B{"Above or below\nthreshold?"}
B -- "Above" --> C["Candidate: scale up"]
B -- "Below" --> D["Candidate: scale down"]
C --> E["Autoscaling Policy\nhow fast, how much"]
D --> E
E --> F["Min / Max pods\nhard limits"]
F --> G["Pod count applied"]
```
### Choosing a metric
The metric is simply the number autoscaling watches. It falls into one of two categories:
* **Resource metrics** describe the pod itself — how hard the CPU or GPU is working, how much memory it's using.
* **Request-shape metrics** describe your traffic — how many requests are arriving, how quickly they're answered.
If you're unsure where to start, use **Concurrency** as your primary metric with **GPU Utilization** as a backup. This combination works well for most model deployments.
**Resource metrics**
| Metric | What it measures | Good fit for... |
| -------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------- |
| **CPU Utilization** | Average CPU usage across all pods | Workloads that lean on CPU, like pre/post-processing or tokenization |
| **GPU Utilization** | Average GPU compute usage across all pods | Inference that's GPU-bound, where compute (not memory) is the bottleneck |
| **Memory Usage** | Average system memory across all pods | Catching memory leaks or workloads whose memory use keeps growing |
| **GPU Memory Usage** | Average GPU memory (VRAM) usage across all pods | Getting an early warning before pods run out of VRAM (big batches, long-context LLMs) |
**Request-shape metrics**
| Metric | What it measures | Good fit for... |
| --------------- | --------------------------------------------- | ----------------------------------------------------------------------------- |
| **Concurrency** | How many requests are being handled right now | Bursty traffic made of many short requests |
| **Latency** | How long each response takes | Chat or voice, where you'd rather add pods early than let responses slow down |
| **Throughput** | Requests handled per second | Steadier traffic, where total volume matters more than any one request |
Some model runtimes expose a few extra specialized metrics too (like queue depth for async workers). Available metrics can vary by model and container type — see [Scaling Parameters](/model-suite/deployments/creating-a-deployment#scaling-parameters) for the exact list on your deployment.
Whichever metric you choose, the **threshold** is the value that determines the direction: above it, pods are added; below it, pods are removed.
## Try it: the autoscale console
Scaling up and scaling down are deliberately asymmetric. Simplismart adds pods quickly when demand rises, so requests don't queue. It removes pods more cautiously, so a brief lull doesn't remove a pod you need again moments later.
The console below runs a live simulation rather than a static illustration. Choose a traffic pattern, adjust the sliders, and watch the pod count respond in real time — no deployment required.
Two things worth exploring once it's loaded:
* The **Scaling ledger** logs every scale event as it happens. Select the **Flapping** pattern with a short scale-down period, and the ledger will flag (in rose) a scale-down that occurs sooner than expected. This is a real effect, explained in the next section.
* Under **Advanced settings**, enable **Schedule window**. It guarantees a minimum pod count during a time window you define, regardless of what the traffic metric shows — you'll see it hold that floor (shaded on the chart) and return control to standard scaling once the window ends.
Notice the shape of the two lines: traffic snaps up and pods follow within a step or two, but on the way down the pod line descends in a staircase, spread out well after traffic has already dropped. That staircase is controlled by the **Autoscaling Policy**, covered next.
## Controlling the speed: Autoscaling Policy
By default, Simplismart scales between your min and max pods with no additional rate-limiting — most deployments don't need to change this. **Autoscaling Policy** is an optional, advanced layer for controlling *how fast* pods are added or removed:
* **Stabilization window** — how long the autoscaler observes the metric before acting, so a brief spike or dip doesn't trigger a decision it would otherwise reverse moments later. Scale-up defaults to a short `60s` window (react quickly); scale-down defaults to a longer `180s` window (avoid overreacting to a blip).
* **Policies** — the size of each step: a fixed number of **Pods**, or a **Percent** of the current count, taken over a **Period** of time (1–1800s).
For most deployments, this is all you need to know — see [Autoscaling Policy](/model-suite/deployments/creating-a-deployment#autoscaling-policy) for the full field reference. The remainder of this section covers one setting that's easy to misread: the scale-down **Period**.
### A subtlety in the scale-down period
A policy such as *"remove 1 pod every 1200 seconds"* reads like a strict rate limit — one pod removed per 20-minute window, no exceptions. In practice, a second pod can be removed well before that window is up, if your traffic has been fluctuating.
**Why:** the autoscaler doesn't simply track time since the last scale-down. It looks at every scaling event — both additions and removals — within the trailing period, and a recent addition effectively offsets an earlier removal in that calculation. So if your metric has been crossing the threshold repeatedly, a scale-down can become eligible again well before the period would suggest.
You can observe this directly in the [console above](#try-it-the-autoscale-console): select **Flapping**, set a short scale-down period, and watch the ledger flag it in real time.
Here's what actually happens with a `1 pod / 1200s` (20 min) scale-down policy:
| Time | Event |
| ----- | ------------------------------- |
| 16:53 | Scale down |
| 16:54 | Scale up |
| 16:55 | Scale up |
| 17:05 | Check: can we scale down again? |
| 17:07 | Check: can we scale down again? |
**At 17:05** — currently at 4 pods. The trailing 1200s window holds 1 down and 2 ups.
`periodStartReplicas = 4 − 2 + 1 = 3`, so the floor is `3 − 1 = 2`. Scaling to 3 keeps us at 3 ≥ 2 → **allowed**. Pods go 4 → 3.
**At 17:07** — currently at 3 pods. The window now holds 2 downs and 2 ups.
`periodStartReplicas = 3 − 2 + 2 = 3`, so the floor is still `3 − 1 = 2`. Scaling to 2 keeps us at 2 ≥ 2 → **allowed**. Pods go 3 → 2 — just **2 minutes** after the previous scale-down, inside a policy meant to allow one pod removed every 20 minutes.
**How to mitigate it:**
* Give your threshold some headroom rather than tuning it to the exact edge, so ordinary noise doesn't flip the decision repeatedly.
* If your traffic is naturally variable, a smoother metric (such as Throughput) will trigger this less often than a spikier one (such as instantaneous Latency).
* Treat the period as a lookback window the autoscaler uses for its calculation, not as a timer that resets cleanly after each scale-down.
## Scale to Zero
Well suited to deployments that sit idle for extended periods — development and test environments, low-traffic endpoints. When there's no incoming traffic, **Scale to Zero** reduces your pod count to 0, so you stop paying for GPU capacity entirely. When a request arrives, a pod starts automatically, with a brief cold-start delay while it comes up. A **cooldown period** controls how long the deployment waits after traffic stops before scaling to zero, so a short pause doesn't trigger an unnecessary cold start.
Scale to Zero and Schedule-Based Autoscaling's "scale to 0 outside windows" option are mutually exclusive — use whichever fits how predictable your traffic is.
## Rapid Autoscaling
For sudden traffic bursts, **Rapid Autoscaling** reacts faster than the standard evaluation windows allow, reducing the latency spike that would otherwise occur while new pods are still starting up.
Rapid Autoscaling is available on Simplismart Cloud (Dedicated) deployments only. Contact [support@simplismart.tech](mailto:support@simplismart.tech) to enable it.
## Schedule-Based Autoscaling
If you already know your traffic pattern in advance — busy during office hours, quiet overnight, a recurring batch job — you don't need to rely on metrics for those periods. Define a time window with a guaranteed minimum pod count, and the deployment falls back to standard scaling (or zero) outside it. If a window and the regular metric-based scaling disagree, **whichever calls for more pods wins** — a window sets a floor, never a ceiling.
To see this in the [console above](#try-it-the-autoscale-console), open **Advanced settings** and enable **Schedule window**. The window appears shaded on the chart, and the pod count moves to the guaranteed minimum the instant the window opens, with none of the usual wait.
See [Schedule-Based Autoscaling](/model-suite/deployments/creating-a-deployment#schedule-based-autoscaling) for how to configure this, with both a guided picker and cron expressions.
## Picking a starting point
| Workload | Metric | Min pods | Scale to Zero | Notes |
| ---------------------------------- | --------------------------- | ---------------------- | ------------- | ------------------------------------------------------------- |
| Interactive chat / voice | Latency or Concurrency | ≥ 1 | Off | Avoid cold starts on the user-facing path |
| Bursty, cost-sensitive dev/test | Concurrency | 0 | On | Let it drop to zero between tests |
| Steady high-volume batch inference | Throughput or Memory | Sized to baseline load | Off | Default Autoscaling Policy is usually fine |
| Predictable business-hours traffic | Schedule-Based + any metric | Set per window | Depends | Use windows for the known pattern, metrics for the unexpected |
These are starting points, not rules — tune each setting against your own traffic once the deployment is live.
## Configuring autoscaling
* [Creating a Deployment](/model-suite/deployments/creating-a-deployment) walks through every field on the deployment form, including Scaling Parameters, Schedule-Based Autoscaling, and Autoscaling Policy.
* [Editing a Deployment](/model-suite/deployments/edit-deployment) covers adjusting any of these settings after the deployment is already live, applied as a rolling update.