After Railway: My Agentic Infrastructure on AWS
On this page 21 sections
Cold open
In October, Railway had a security incident, and their incident report describes a very new way of attack. Someone broke out of a build and got as far as reading customers’ secrets, using fleets of agents to do the work, and model provider keys were the first thing they went looking for.
We had 250+ AI agents running on Railway, so we knew we couldn’t keep them there. The idea to migrate was proposed right away. The day after Railway got the attacker out I made the first commit of a service I called colony, and about two weeks later we deleted our Railway project.
I’ll tell it in four acts.
Act I: The setup
An agent that owns a disk
Every customer gets their own agent, and our agent design is a VM with a persistent volume that holds that customer’s data. If you’ve run infrastructure before, you know a stateful component is a nightmare, especially for scaling. I can’t run a second copy of an agent or put a load balancer in front of it, because each one is tied to its own disk, and we have 250 of them.
What the new home must do
The hard requirements were fast startup, a persistent volume because it holds customer info, and instances that scale.
We also have four work modes, and I’ll keep coming back to them:
- Fast initialization. A new customer should get a working agent environment in seconds.
- Persistent and reliable agents. They’re working most of the time, so they have to stay up and keep their disk.
- Complete isolation, because every agent runs code that a model wrote next to a customer’s private data.
- Fast deployments, since we roll out every new version to all of them.
Any one of these is easy. Fitting all four in one design is the hard part, because fast startup works against isolation and a disk per agent works against scaling.
What we would miss
The other challenge was the DX we had on Railway. You create a service, attach a volume and push, and that’s it. I couldn’t find that ease of use and deployment anywhere else, and they kept shipping new features on top of it.
It had some quirks. There wasn’t true parallelization between project deployments, so we were still bottlenecked whenever we rolled out to the whole fleet. But the experience was great, and that made it hard to go away from.
Act II: The wrong turn
First try: one machine per agent
I started with the simple approach, an EC2 machine with an EBS volume for each agent. Which is great, and it fits the criteria: the machine gives me isolation and the volume keeps the data.
The catch is that I needed to build the base for everything myself. Creating instances and volumes, the network, setting up each machine, replacing one when it dies, rolling out new versions, and then showing all of that in a simple UI because that’s what the team was used to.
That base became colony, the control plane. The name comes from a bee colony. Our backend tells colony what a customer needs, and colony is responsible for making it real.
One request to colony
Ask for an agent, not for a machine.
PUT /hosts/4012 { "memoryGiB": 2, "diskGiB": 32 } → 202 Accepted { "generation": 7 } - ✓create the disk
- ✓start the agent
- ✓check its health
- not in the request
machine typeregioncloud
I was rebuilding Kubernetes
I built a PoC this way and it worked. But while going, I noticed how much colony was doing on its own: picking machine sizes, deciding where each one runs, watching health, replacing dead machines, rolling versions across the fleet. I was rebuilding the same fundamentals of Kubernetes, only slower.
The reason I hadn’t started with Kubernetes was isolation. I wanted pure VM isolation for each agent, and a normal container shares the kernel with every other container on the machine.
Kata made it fit
That’s where Kata Containers came in. Kata runs each container inside its own small VM, while Kubernetes still treats it as a normal container. And it just clicked, because now I could have everything on the list:
- containers scale fast, so both a new agent and a new version take seconds
- each agent can still have its own EBS volume
- each agent still gets a VM of its own
- growing the fleet means adding machines to the cluster
One request, three hosting designs
The host changed three times. The request did not.
1 service + 1 disk per agent
The one thing to plan around is that a machine can only attach a limited number of volumes, so we keep 25 agents on each. That was manageable.
I found two things going down after the change. The first was lines of code in colony: with Kubernetes doing the scheduling and the restarts, one cleanup removed 10,219 lines. The second was the cost of hosting each agent, since 25 of them now share a machine. Scaling got easier too, and the backend didn’t have to change, because its request to colony never named a machine in the first place.
Act III: Rebuilding Railway on AWS
Then the question became how to deliver something close to the Railway experience on AWS and Kata. AWS gives you the pieces but not the product, so most of the two weeks went into building components. Here’s what I built for each work mode.
1. Start fast
A new machine in the cluster needs about four minutes before it can run an agent, which is far too slow for a new customer. So the cluster always keeps a few empty placeholders running that do nothing except hold a place.
Start time: about 20 seconds
An empty placeholder takes the four minute wait.
In my tests an agent that landed on a free place was ready in about 20 seconds. When there was no free place it took the full four minutes.
2. Stay up and keep its data
The worst mistake this system could make is giving an agent the wrong disk or wiping a good one, so the agent checks its own disk before it starts.
Agent 4012 checks its disk before it starts
Four disks, one of them wrong.
I tested it by giving one agent another customer’s disk on purpose. It refused to start and colony raised an alarm, with nothing written to the disk.
Colony also takes a backup of every disk each hour, gives more memory to an agent that runs out of it twice, grows a disk that fills up, and checks every agent once a minute.
3. Stay isolated
What one agent can reach
The wall around each agent.
The part I’d point at after the Railway incident is that an agent has no cloud keys at all. If someone breaks out of one, they don’t find credentials for anything else.
4. Release fast
Releases were where we got bottlenecked on Railway, so I wanted them fast here. They also had to be safe, since a release reaches 250 customers during their working day. Colony sends each release to a single agent first, called the canary, waits 30 minutes, and then rolls out to the rest in groups.
250 agents, one release
A bad release should stop at one agent.
Two details helped with speed. Every machine downloads the new version before the first agent switches, so agents don’t wait for a download. And when a fix is urgent, colony can skip the 30 minutes and update everyone at once, still with the health checks.
And a dashboard
Since the team was used to Railway’s UI, colony has a dashboard of its own. It lists every agent with its logs, shows the releases and the health of the system, and lets you restart an agent, give it more memory or disk, or move it to a new version early. It’s much smaller than Railway’s, but it has what we used day to day.
Act IV: Moving day
Test it for real, then destroy it
Before moving anyone I built each design in a real AWS account with fake customers, ran it, and tore it down again. Those runs turned up fourteen bugs. Most were small, like AWS rejecting a security group because its description had an apostrophe in it, but a few I don’t think I’d have found any other way.
The cluster once added two machines for a single agent. A new machine reports ready about three minutes before Kata finishes installing on it, so the agent still couldn’t start and the cluster asked for one more. New machines now carry a “not ready yet” mark until Kata is done.
A crashed agent also couldn’t come back, because Kata keeps the disk locked to the VM and a restart in place fails. Colony starts a fresh VM for it now.
A wrong version got past the canary, since nothing checked that the agent was running the version the release named. The agent checks that itself now. I found out it works when my own build script produced a bad version: all 20 test agents refused it and went back to the old one in under a minute.
And idle agents were holding about 2 GB of memory each that they weren’t using, for half an hour at a time, until I changed one setting.
Move one agent, 250 times
The move itself was a script that handled one agent at a time, and by default it only printed what it would do.
One move, run 250 times
The long copy happens while the old agent still works.
If something failed after the switch, the script took the customer back to Railway with the same steps in reverse. It also had a practice mode, which I used to time each move before doing it for real.
Epilogue
So I started with a machine per agent and ended up with less code and a lower hosting cost per agent, and all it took was to stop building what Kubernetes already does. Colony kept only the parts that are specific to us.
The agents have been running on AWS since then, and the backend still talks to colony the same way it did on day one.
Credits: how colony is built
Colony is written in TypeScript with Effect, and I’d pick it again. Most of this service is error handling, retries and time limits, and Effect has those built in. Each call to AWS gets a time limit, so a stuck call only fails the one agent it belongs to. I can also swap the cloud out in tests and run all of colony against a fake one.
For the AWS and Kubernetes calls I use distilled, which has clients written for Effect, so errors and retries look the same everywhere.
I also tried Alchemy in a small experiment and found colony can use its AWS code as a library without taking the rest of Alchemy. I’d like to move there, so that I maintain less cloud code by hand.
Two rules stayed the same through all the design changes. Before colony creates a disk it saves a unique id for the attempt and sends it to AWS, so if the answer gets lost and colony asks again, AWS returns the same disk and not a second one. And every disk carries a label with its customer, so if colony ever sees another customer’s label on a disk, it stops and leaves that disk alone.
Post-credits scene: Nitro Sandboxes
While I was writing this, AWS announced Nitro Sandboxes for EKS, ECS and EC2. With it, AWS puts each container in its own VM for you, and there’s nothing to install. That sounds really exciting, and if I was doing this project today maybe I would consider it.
But the key idea behind colony is to be provider agnostic. Kata works on any cloud and Nitro Sandboxes only on AWS, which matters to us after what happened with Railway.
If you’re running agents like this, or think a VM per agent is too much, find me on X or LinkedIn.