Scaling Cloud IDE Provisioning by 5x: From 5K to 25K Workspaces in One Region

HackerRank evaluates candidates in a real VS Code workspace, in the browser. For a long time, one region could run about 5,000 of those at once.

Then the questions changed. Hiring moved toward multi-file project work, the kind of environment people now use with coding agents: several files open, a real repo, a language server, not a single editor pane. Demand for concurrent workspaces went up fast. 5,000 stopped being a comfortable ceiling and became a hard one.

Past that point the platform could not keep up. Some of that was infrastructure: quotas, IP space, NFS. Some of it was our own code: one-by-one VM creates, chatty APIs, a database that was not ready for the write rate. I owned the work end to end to take one region from 5,000 to 25,000 concurrent workspaces.

This was not a "spin up more VMs" problem.

The whole path was the ceiling

At this scale, provisioning is a coordination problem. A workspace request touches the cloud control plane, VM create, our database, git, NFS, and the service that keeps all of that in sync. If any one of those is slow or failing, the candidate sits on a spinner.

I did not get to pick a single villain. At 5,000, all of the above were already red. Fixing one service in isolation did not move the overall ceiling. The work was a loop: find where the system was actually breaking, fix that, load test again, watch the next thing catch fire.

GCP was the primary capacity. AWS was the fallback. That dual-cloud setup mattered later, when a bulk trick on one provider had no equivalent on the other.

First, the cloud had to stop saying no

Before I touched service code, I needed headroom. Otherwise every load test would die on provider throttling, and I would never see our own bugs.

Quota and capacity changes:

• Read requests per region/min: 15,000
• Write requests per region/min: 16,500
• Queries per region/min: 15,000
• Instance capacity per VPC: 50,000
• Private IP allocation: 65,000, by expanding the primary range to a /16

That removed the first hard blocker: the provider saying no before our code had a chance to fail.

Bulk APIs, or you lose on round trips

Single-resource operations were too expensive at this concurrency. Each create, each route insert, each status fetch was a round trip we could not afford when thousands of workspaces were coming up at once.

VM creation moved to bulk insert APIs, which cut control-plane overhead on each cloud.

After create, route entries went in through Redis pipelines, in batches, instead of one round trip per route.

Reconciliation was the awkward part. There is no bulk-read primitive that works the same way on GCP and AWS, so I used filtered list APIs in controlled batches to keep our state aligned with what actually existed.

On the non-bulk path, fetching instance details used to take 3 API calls. I cut that to 1. Isolated, it looks like a nothing-change. During a burst, with retries piled on, that path got about 3x faster, and it stopped the slow path from amplifying the incident.

Then the database became the thing we were waiting on

Once VMs were creating fast enough, the database showed up as the new ceiling. Connections ran out even when the cloud was keeping up. That was the first surprise: compute was no longer the scarce resource. The write path was.

I tightened connection pooling in the workspace service so we were not opening and closing connections on every state write.

Then I replaced one-by-one updates with bulk queries for workspace state and runtime fields. That dropped both connection count and write overhead.

A workspace is not provisioned until the repo is there

A VM with no project is still a failed workspace. Git repos for those IDEs lived on NFS: the clone the candidate actually opens. That path had to take the same 5x, and NFS was the second surprise hiding behind VM create.

What changed:

• Right-sized node pools and replicas for the git service
• One atomic push into git instead of multiple round trips
• git repack configuration to cut NFS IOPS
• Load-tested NFS mount options for metadata pressure
• Regional NFS in production, with a migration plan behind it

Fast VMs with a slow clone still look like a provisioning failure to the candidate. NFS metadata IOPS is how that failure shows up: not a dramatic outage, just thousands of small file operations stepping on each other while the workspace is supposed to be "ready."

I watched the load test

I ran JMeter against production-like conditions. The target was 25,000 workspaces in one region, at about 1,000 assignments per minute. This was not a pass/fail checkbox at the end. Each run told me what to fix next: hidden coupling, retry behavior, the NFS and database limits that only appear when the rest of the path is healthy.

The last run is this dashboard. I was watching it live.

What I was looking at:

• Running workspaces climbing through 25K
• Create requests holding around 1,000 per minute
• Average time to assign under a second (0.97s on this run)
• Two requests waiting, not a queue growing without bound
• 26,037 backends up
• 4 healthy contexts: the cloud-provider connections the workspace service can provision through. If that number drops, a provider is out of the pool and we have lost fallback.

What this left me with

Scale here is a systems problem. The VM was never the whole story.

Bulk primitives become mandatory past a certain rate. If you keep doing one-at-a-time against the cloud, the database, and git, you will hit 5,000 again under a new name.

The long tail matters more than the architecture diagram suggests. The 3-to-1 API change, the NFS mount options, the bulk SQL: none of those show up when you load test at 500 workspaces.

If you do not test like production, production will test you.