What If FFXII's Gambits Were a Genre?

2026-10-26T00:00:00+00:00

TL;DR: I made a little browser game where you program a fantasy party with FFXII-style rules, then try to win 16 fights out of 20. Play Root of All Evil.

2006's Final Fantasy XII was pretty sweet. Anyone play it? FFXII is one of the less popular entries in the series, but the combat was super interesting! It was in this transition stage between full turn-based and more real-time combat, and landed somewhere unusual.

You have your party, and you give them rules like "cast cure on ally with HP less than 30%" or "attack nearest enemy", so you build up the behaviour you want. Then as you go through the environment you unleash this clockwork destruction.

At least that's how it plays out when it works. If your rules are off, your party is doing completely the wrong things and getting stomped, and you have to jump around editing the rules on the fly. It actually feels a lot like being a sysadmin in an incident, trying to patch the system to muddle through just long enough that you can do a proper fix later. Not the easiest emotion to recreate, but they managed it!

I always thought that aspect of FFXII could be a genre, but it never really caught on and there aren't many other examples out there. So I spent the weekend whipping up a little game that plays with the idea.

You've got classic fantasy people (mage, cleric, knight, rogue) and you give them rules. This one is more puzzle game than the live action of FFXII: you program your party to beat a level, then proceed.

The first level at 4× speed. Nobody's steering; the party just follows its rules.

But while iterating on it, I stumbled into the part I'm most excited about. Remember in Avengers: Endgame where Dr Strange looks through all the possible futures to find the one where they win? Fights are random but seeded, so every one can be replayed exactly. That meant I could make you win across 20 universes, not just one: clear a level by winning 16 of 20.

As you adjust the rules, you see the fight play out. In a way you're playing the whole probability distribution, watching the graphs and numbers change, which is oddly compelling. It's fun to see how one seemingly inconsequential tweak shifts your odds from 9/20 to 17/20.

Healing at 20% HP instead of 10%: 8 wins out of 20 becomes 18.

On top of that, there are rewards for winning 16 times without anyone in your party falling, or for winning with the fewest rules. There's even an Endless mode with a fresh set of waves every day, the same for everyone, to see how far your party can get.

For a quick weekend project, it turned out better than I expected, which shows FFXII's 20-year-old combat system still has strong bones.

Give Root of All Evil a shot. See if you can find your 16 universes, and let me know your best win rate with the fewest rules.

"Just turn on autoscaling"

2026-10-09T00:00:00+00:00

Sometimes I hear people talk about autoscaling as an easy solution to performance or load issues. It's often a good one, and usually worth having, but it's not a switch you flip. There are a few problems that can catch you out, and even when you get past those, scaling out can just move the bottleneck somewhere else, so it helps to know where it's going to end up.

Here's the mental model I use to think about it, with a demo of each problem you can play with along the way. None of them are dead ends, so each one also comes with tips on how to deal with it.

TL;DR: This one's long, with a lot of interactive demos. If you'd rather skip the reading, open the full demo and have a play, it has every scenario from this post in a dropdown.

Thinking in funnels

Simplistically, every layer of your stack is a funnel.

Illustration: Pearson Scott Foresman, public domain

It can take in and hold a certain amount of water, and lets out water at a certain rate. Let's take an example of a server:

accepting TCP connections
handing them to worker threads
which have a database connection pool
to call a database

Picturing requests like water pouring in the top, here's how it looks:

Open this demo: Funnels only

Under normal conditions, all requests are served, water is flowing nicely. Right now CPU is the bottleneck, so if you click the "Send a burst" button, you'll see the CPU funnel fill up first and start to spill. You can also increase the incoming load to see how it handles a higher sustained volume.

In real life spilling is rarely a clean drop. Usually requests queue up, slow down, and eventually time out or error.

A note on CPU: it's a bit of a special case, as it doesn't really hold anything. A busy CPU just makes everything slower, which backs up into the worker threads, but it's easier to picture as its own funnel.

Adding autoscaling

It's worth being clear about how autoscaling differs from static capacity. With a static number of instances, the capacity is already there when load arrives, as long as you guessed the right number. Autoscaling, on the other hand, is a control loop: it measures something, decides whether that's too high or too low, adds or removes instances, and then waits for them to start. Every step of that takes time, so it's always reacting to load from a little while ago. Most of what follows comes down to that loop measuring the wrong thing, reacting too slowly, or believing a signal it shouldn't.

Scaling on the bottleneck

Back to the server from the first demo, where CPU was the bottleneck. You can add in autoscaling to solve that, right? Just autoscale based on CPU usage and job done? For this service, yes!

Open this demo: Autoscaling on the bottleneck

Send a burst or increase the load, and you'll see the instance scale out, relieving the pressure. Compare it with the first demo: at the same load, far more requests make it through. Push the load all the way up though, and it hits the maximum number of replicas and starts spilling again.

Scaling on the wrong bottleneck

That's a well-tuned service, with a known bottleneck and correctly configured autoscaling. But what if you're not sure where the bottleneck is? Maybe you haven't done sufficient testing, or a new feature or infrastructure change shifts the performance profile. Then there's a chance you're scaling on the wrong bottleneck:

Open this demo: Autoscaling on the wrong bottleneck

Here we can see that something changed and now worker threads are the bottleneck. As load increases, they run out of capacity first, and start spilling over. Which means CPU is never saturated and the autoscaling never kicks in.

Quick tip: CPU is a reasonable place to start, but load test to find your real bottleneck. If it's not CPU, scale on something closer to it: request rate, response time, in-flight requests, or queue depth and consumer lag for workers. On Kubernetes, KEDA makes scaling on these much easier.

Or come at it from the other direction and choose your bottleneck. I once worked with someone who spent about a month load testing an Akka HTTP service, tuning connection pools and the rest of its dials, until CPU was the bottleneck. Once it was, autoscaling on CPU just worked. Ok, maybe that's not such a quick tip.

Slow startups, small buffers

Ok, so imagine you've identified the bottleneck, and are autoscaling on it... what about startup time? Depending on how long your service takes to start up and start accepting load, the other layers will need sufficient buffering to survive the wait:

Open this demo: Autoscaling with slow startup and small buffers

As you increase the load here, you can see how the slow startup means the first instance starts dropping requests while the second instance is coming online.

The timings in these demos are sped up, and real ones vary a lot. Serverless platforms can add capacity in well under a second, but if your autoscaler has to start a new VM, we could be talking minutes. So how much buffer you need depends on your setup.

Quick tip: make startup faster (smaller images, pre-pulled images, less work at boot), keep some spare capacity warm, scale up ahead of predictable peaks so you're not waiting on startup when they hit, or give the layers in front enough buffer to hold out until new instances are ready. On Kubernetes, one trick for spare capacity is to run low-priority placeholder pods that real pods can preempt, so scaling out doesn't have to wait for a new node. There's more on this here.

Scaling down

You have your bottlenecks identified, sufficient buffers for your startup time, things are good right? Well, turns out scaling down can also be an issue:

Open this demo: Scaled down by an upstream outage, then dogpiled

Take this example where there's an incident upstream which causes your traffic to drop. Well, your autoscaler is watching the CPU, and its usage just dropped, so it scales down. That's probably what you want, e.g. at night if there's less traffic. However this is an artificial drop, so when the upstream incident clears you're now not ready for the load and your service gets dogpiled until it can scale back up. And often in these situations clients retry aggressively, creating even more traffic than usual. If your service keeps a local cache, the new instances also start with it empty, so more of that traffic goes straight through to the database. And if they load any state when they start up, they all do it at the same moment, from the same place, so even starting up can dogpile something.

On top of that, because CPU can't go past 100%, the autoscaler can't tell how overloaded it really is, so it climbs back in steps, waiting for each round of new instances to start before deciding again.^[1]

Quick tip: a longer scale-down window and a sensible minimum replica count give you headroom for traffic that comes back suddenly.

Backpressure

So far every funnel just pours into the next one, whether there's room or not. That's what's happening when the CPU funnel spills: the request was already accepted and handed to a worker thread, and then it times out halfway through. That's kind of the worst place to lose it. You've already done some of the work, and the client is probably just sitting there until it times out.

Backpressure is when a funnel only lets out as much water as the next one has room for. So when the CPU is full, the worker threads stop handing over work, so they fill up, then the TCP backlog fills up, and requests start getting turned away at the top instead.

Open this demo: Funnels with backpressure

Tick the "Backpressure" box and send a burst, and you'll see the funnels fill up from the bottom, and the spill moves up to the top. You're still dropping requests, but you're dropping them early and cheaply, e.g. by refusing the connection or sending a quick 503, instead of after doing half the work. That only really helps if clients back off when they're turned away, rather than retrying straight away and piling back in.

This isn't just about what happens inside one service: the same goes for every hop in the system, right up to the frontend. Each layer should expect the one after it to be slow or full, and fail fast instead of waiting on it. Michael Nygard calls this "cynical software" in Release It!, which is well worth a read. If you want to go deeper on backpressure itself, Backpressure explained is a good intro.

It's all funnels

These were all simplified examples, and in reality there are so many more funnels... load balancers, kernel queues, thread pools, file descriptors, disk IO, downstream services, even your cloud account's quota for new instances, and they all have their own limits.

And some of them are shared. Here's the same service again, but this time the database it's calling can't handle as much:

Open this demo: Scaling out into a shared database

As you increase the load, the service scales out just like before. But each new instance brings its own connection pool, all pouring into the same database. If each instance has a pool of 8 connections, 5 instances means 40 connections to the database, and scaling out to 20 means 160. Postgres only allows 100 by default. So autoscaling didn't remove the bottleneck, it just moved it further down, onto something that can't scale out.

None of this means you shouldn't autoscale. There's always another funnel, but each one fills up at a different point, and pushing that point further out can buy you enough time to deal with a problem before it turns into an incident. Fixing every possible bottleneck often isn't worth the cost anyway, in the same way that running multi-region isn't always worth it for availability.

For me, thinking in funnels makes all of this a lot easier to reason about. So next time someone says "just turn on autoscaling", they're probably right, it's usually worth doing. Just ask a few questions first:

Which funnel fills up first?
How long can it hold out?
Where does the water go when it spills?

Once you can answer those, you're on the path to autoscaling effectively.

"If you're short on wisdom in some things, let Nuremberg bring you the funnel." The Nuremberg Funnel, advertising stamp, 1910, public domain

The Kubernetes autoscaler works out how many instances it wants from current usage ÷ target usage. With a 70% target, CPU at 100% gives a ratio of at most 100/70 ≈ 1.4, so each round can only add about 40% more instances, however overloaded you really are. ↩

Alex Humphreys