What happens when the cloud doesn't have the machines your application needs? Let's look at how we can design applications and platforms to be more flexible when capacity gets constrained.

In Part 1, I wrote about capacity planning making a comeback. Cloud capacity isn't infinite, lead times matter again, and sometimes the answer from your CSP really is just: sorry, we don't have those machines right now.

Forecasting and securing capacity, which I covered in the previous part, are only one side of the problem. There's another question worth asking: What if our applications simply became better at using whatever capacity is available?

And I don't mean blindly throwing workloads onto random machines. Quite the opposite. We should understand what an application actually needs, test the alternatives, and deliberately design some flexibility into the system. Having a wider range of instance types our service can safely run on makes us much more resilient to OOC (Out of Capacity) problems.

Capacity is an application concern too

Developers build applications. A Platform team provides infrastructure. Correct? Not necessarily!

Infrastructure is ultimately provided by CSPs. Platform teams' responsibilities are usually more about infrastructure design, reliability, automation, and sometimes CI/CD. And as we established in Part 1: "An autoscaler allocates capacity. It doesn't create it."

So when we hit OOC for a particular instance type, we cannot magically make that capacity appear. And Platform engineers cannot magically make an application work well on infrastructure it was never designed or tested for either.

This is why I think capacity should become a normal application design consideration - just like performance, reliability or security.

The people developing a service usually understand it better than anyone else. They know which user journeys matter, its SLOs, whether latency or throughput matters more, and whether losing 20% of capacity means slightly slower processing or a complete disaster. They also know which parts of the workload can be delayed, degraded or simply dropped when things go wrong.

The Platform/Infrastructure team brings a different set of knowledge. They may know which machine families are realistically available in particular regions, which ones historically suffer from capacity shortages, what reservations can be obtained, how quotas work, what the CSP is telling them about upcoming supply, and what mechanisms the platform can provide for scheduling, storage, networking and resource isolation.

Neither side has enough information on its own. This is really important.

So perhaps the relationship should look more like this:

So again: cooperation. Design worked out together, followed by common work on rightsizing and performance testing.

Rightsizing: start with what the application actually needs

One of the first questions I'd ask is a basic one: What does this service actually require?

Imagine a service has been running on a 16-vCPU, 64 GB machine type for years. Does it actually require a 1:4 CPU-to-memory ratio? Or did somebody choose that instance type five years ago because it looked reasonable?

Does it care which CPU generation it runs on? Does it need x86 at all? Does it depend on local SSDs? How sensitive is it to storage latency or network throughput? What happens if CPU performance per core changes?

For some workloads, the answer really might be very specific. Distributed databases are an obvious example where CPU, memory, storage latency, local disks and network characteristics can all matter enormously. For many other services, however, the real requirements may be considerably less strict than the infrastructure configuration suggests.

That's where rightsizing and performance testing become true capacity tools:

Profiles, preferences and fallbacks

After rightsizing, we ideally start seeing groups of workloads with similar requirements. Some services might be compute-heavy and work best around a 1:2 CPU-to-memory ratio. Another group might need considerably more memory. And there will always be special cases - databases requiring local SSDs, workloads sensitive to network throughput, or applications depending on a particular CPU architecture or feature.

And then there should probably be a general-purpose profile for everything that doesn't really have any special requirements. Not every service needs its own carefully crafted infrastructure profile - and that's kind of the point. This can also be a good starting point for workloads that don't have well-understood requirements yet, especially early in their development.

This gives the Platform team an opportunity to define a relatively small set of infrastructure profiles instead of managing hundreds of application-specific configurations:

That's useful for standardization. But there's a trap here. Let's say performance testing shows that instance-type-A gives our general-purpose profile the best price/performance ratio. It would be tempting to standardize everything in that group on A. If hundreds or thousands of VMs eventually converge on the same instance family, though, we've also created a rather large dependency on one particular capacity pool. In other words:

The most efficient instance type is not necessarily the most resilient choice.

So instead of mapping a profile to exactly one VM type:

general-purpose
      ↓
instance-type-A

we can map it to a tested, ordered set of acceptable options:

general-purpose
      │
      ├── preferred   → instance-type-A
      ├── fallback 1  → instance-type-B
      ├── fallback 2  → instance-type-C
      └── emergency   → instance-type-D

Perhaps A gives us the best performance/cost ratio. B performs almost identically but costs slightly more. C has a different CPU-to-memory ratio but still keeps the application comfortably within its SLOs. D might be inefficient and expensive, but it's good enough to keep the service alive during a serious capacity shortage.

And different infrastructure profiles can have completely different lists:

general-purpose
  preferred:   A
  fallback:    B, C
  emergency:   D

compute-heavy
  preferred:   E
  fallback:    F, G

database-X
  preferred:   H
  fallback:    I
  requirements:
    - local SSD
    - high memory bandwidth

This doesn't mean support every instance type your CSP offers. That would create a completely different operational problem. It's about finding a reasonable balance between standardization and optionality: a small number of well-understood workload profiles, each backed by several tested infrastructure options wherever possible.

There's another subtle advantage here. If the alternatives use different CPU generations, machine families or underlying hardware, we're not just creating configuration flexibility - we're also potentially spreading our demand across different physical capacity pools.

So the application/platform contract can look like:

I prefer A, but we know exactly how this workload behaves on B, C and D.

That's a considerably better position to be in when your CSP tells you that A simply isn't available:

Turn those preferences into scheduling decisions

Once applications have a tested set of acceptable infrastructure choices, the platform can start doing something useful with that information. Conceptually, we want something like:

The exact implementation obviously depends on your environment. In Kubernetes, that might involve combinations of node pools, affinity and preference rules, topology constraints, scheduler configuration and autoscaler behaviour. Outside Kubernetes it might simply be logic in whatever provisioning or placement system you operate.

The important part is that the application's infrastructure preferences have become explicit data that the platform can act upon. And once you have that, you can go further.

Reconcile toward preferred capacity

Preferred capacity is a desired state, not a one-time placement decision. Let's say our preferred capacity disappears.The platform successfully places workloads onto fallback machines. Problem solved? Not quite.

Three days later the CSP has capacity again. Now you're running on machines that may be slower, more expensive, or simply less efficient than your preferred infrastructure. So ideally the system should eventually converge back toward the desired state:

That sounds easy until you actually try to automate it. How could this actually work?

In Kubernetes, one option is to make the preferred infrastructure visible through node labels and preferred node affinity. During an OOC event, workloads are allowed to land on fallback nodes. When preferred capacity becomes available again, the scheduler won't normally move already-running Pods just because a better node has appeared - scheduling is not continuous optimization.

This is where something like the Kubernetes Descheduler can help. For example, RemovePodsViolatingNodeAffinity can consider preferredDuringSchedulingIgnoredDuringExecution rules and evict workloads that are no longer running on their preferred nodes, allowing the normal scheduler to place them again. Conceptually:

Answering the question “is migration safe?” is the interesting problem here. For a stateless service, Kubernetes primitives may be enough: PodDisruptionBudgets, readiness checks, rollout limits and a carefully configured Descheduler policy can provide reasonable guardrails. For a stateful or distributed system, Kubernetes often doesn't know enough.

Imagine a database node currently rebuilding a shard, another replica recovering, or a cluster already close to its minimum redundancy. From Kubernetes' perspective the Pod might be perfectly healthy and evictable. From the application's perspective, moving it right now might be a terrible idea. The application needs some way to expose migration readiness - directly or indirectly - to the platform.

That doesn't necessarily mean Kubernetes needs to understand the application's internals. A small controller/operator could translate application state into a simple platform-level signal:

That signal could be represented in different ways depending on the platform: labels or annotations maintained by an operator, PodDisruptionBudgets, custom resources describing migration state, or an entirely separate controller coordinating the operation.

I wouldn't make the Descheduler responsible for understanding database replication, shard placement or application-specific recovery state. Its job can remain relatively dumb: identify placement that should be improved and perform evictions within the guardrails exposed to it. The application-specific controller decides whether those evictions are currently allowed.

This gives us a useful separation:

Application:
"Is it safe to move me?"

Platform:
"Is there somewhere better to put you?"

Scheduler:
"Where exactly should you run?"

And then the whole process becomes another reconciliation loop:

That last loop is the important part. We don't want "A is available again → move everything!". We want gradual convergence back to preferred capacity, constrained by application health, disruption budgets, available headroom and migration rate.

Now let's reiterate on the whole picture of this:

Design for OOC as a normal failure mode

This brings me to perhaps the most important design change. We already design distributed systems assuming that many components may fail - and that this is expected. I believe we should make the same assumption about Out of Capacity: OOC will happen, so design the system for it.

When a capacity request fails, the platform should already know which escape routes are available for that workload. Conceptually, the decision tree may look something like:

Of course not every application needs every one of those escape routes. Moving between regions can be enormously complicated as well as stateful workloads may have data locality constraints. Even some regulatory requirements might remove entire regions from consideration or part of services simply could not tolerate reduced capacity. This is all fine. The important thing is that these constraints should be known, instead of discovering it during an incident.

A service might explicitly say:

I'm deliberately not proposing this as an actual configuration format. The point is the information it contains. The platform shouldn't discover an application's capacity constraints during an OOC incident. It should already know which shapes, zones and regions are acceptable, how far the service may degrade, and what should happen when its minimum capacity can no longer be maintained.

Graceful degradation applies to capacity too

There's one more escape route in that decision tree. It's about agreeing on a service degradation model. Imagine a service with a critical path that needs to keep functioning, but also some reporting or background jobs that aren't that important and can wait. Both happen to run on the same infrastructure profile.

Now imagine a capacity constraint on the CSP side makes that particular profile very difficult to obtain. We already designed for this - we have preferred alternatives (remember A, B, C, D, etc.). But it may happen that none of those are available. Believe me - rarely, but it happens. All it takes is a severe enough outage on the CSP side and suddenly we cannot add any new capacity.

In that unfortunate situation, one of the remaining options is to deliberately degrade our service. In other words: continue scheduling what is really important, and stop scheduling what isn't.

How? This obviously depends on the infrastructure, schedulers and applications involved. One useful approach is to maintain a service dependency graph. If service A is critical but depends on B and C, keeping A alive while starving B would obviously not help much. The graph lets us reason about the minimum set of components required to keep an important user journey functioning.

Another useful mechanism is workload priority. Services or workloads can be assigned priority levels describing their relative importance. When capacity becomes constrained and workloads X and Y both ask for resources that simply aren't available, the platform can prioritize X, postpone Y, or perhaps even reclaim resources from lower-priority workloads.

Kubernetes actually has a mechanism that can help with this: PriorityClass and scheduler preemption. Pods can be assigned different priorities using PriorityClass. When a higher-priority Pod cannot be scheduled because there isn't enough capacity, kube-scheduler can look for lower-priority Pods whose removal would make enough room and preempt them. This doesn't solve the dependency problem or decide what's important to the business - but once we've made those decisions, it gives the platform a mechanism to enforce them.

For example, we could classify workloads roughly like this:

critical-user-path    → critical
supporting-services   → important
reporting / batch     → background

Then, during a severe capacity shortage, a new critical workload may be scheduled at the expense of lower-priority reporting or batch workloads. Kubernetes also supports non-preempting priority classes when we want something to be scheduled ahead of lower-priority work without evicting already-running Pods.

These two mechanisms complement each other: dependencies tell us what needs to survive together, while priority tells us what should survive first.

Deciding on priorities is ultimately a business decision. Engineering teams can provide the mechanisms, describe dependencies and technical constraints - but somebody needs to decide what the business actually considers critical:

Summing this up - a new equation

After this second part of this blog series, I think we can safely change our perspective on how we understand the capacity now:

Usable capacity is not only about how much infrastructure your cloud provider can provide at a given time. It's also about how much of that available infrastructure your applications are capable of using.

The CSP controls the first part of that equation. Through rightsizing, alternative instance types, scheduling, reconciliation and graceful degradation, we can influence the second:

Usable capacity = available cloud capacity × our ability to use it

What's next?

So far we've attacked the problem from two directions. In Part 1 we tried to answer:

How much capacity are we going to need, and when?

In this part:

How many different kinds of capacity can our applications safely use?

But there's still a rather important problem left. Imagine we've done everything right. We have forecasts and applications have been rightsized / benchmarked plus they can use several machine families understanding our fallback paths.

All those optimizations can get us too focused on ensuring capacity and reliability, while there's still one last part of the equation: cost-effectiveness. Remember - we're in a business, so in the end, we also need to optimize for money.

Now imagine that we look at our fleet and discover a huge amount of waste. Or even worse - we don't see it at all, but we certainly see the infrastructure bill. What then?

This opens another set of problems: bin-packing, scheduling optimization, reservations and utilization. And at sufficient scale, capacity management itself starts looking more like a fully-fledged control plane.

And somewhere along the way, we'll also get to my slightly provocative take on capacity automation: forget Terraform - just code it 😄

More on that in Part 3.

💡
Would you like to receive future blog posts via e-mail? Consider subscribing (free, no spam, no tracking, full privacy).