ahmedhesham.dev

After Railway: My Agentic Infrastructure on AWS

· 10 min read

Cold open

In October, Railway had a security incident, and their incident report describes a very new way of attack. Someone broke out of a build and got as far as reading customers’ secrets, using fleets of agents to do the work, and model provider keys were the first thing they went looking for.

We had 250+ AI agents running on Railway, so we knew we couldn’t keep them there. The idea to migrate was proposed right away. The day after Railway got the attacker out I made the first commit of a service I called colony, and about two weeks later we deleted our Railway project.

I’ll tell it in four acts.

Act I: The setup

An agent that owns a disk

Every customer gets their own agent, and our agent design is a VM with a persistent volume that holds that customer’s data. If you’ve run infrastructure before, you know a stateful component is a nightmare, especially for scaling. I can’t run a second copy of an agent or put a load balancer in front of it, because each one is tied to its own disk, and we have 250 of them.

What the new home must do

The hard requirements were fast startup, a persistent volume because it holds customer info, and instances that scale.

We also have four work modes, and I’ll keep coming back to them:

  1. Fast initialization. A new customer should get a working agent environment in seconds.
  2. Persistent and reliable agents. They’re working most of the time, so they have to stay up and keep their disk.
  3. Complete isolation, because every agent runs code that a model wrote next to a customer’s private data.
  4. Fast deployments, since we roll out every new version to all of them.

Any one of these is easy. Fitting all four in one design is the hard part, because fast startup works against isolation and a disk per agent works against scaling.

What we would miss

The other challenge was the DX we had on Railway. You create a service, attach a volume and push, and that’s it. I couldn’t find that ease of use and deployment anywhere else, and they kept shipping new features on top of it.

It had some quirks. There wasn’t true parallelization between project deployments, so we were still bottlenecked whenever we rolled out to the whole fleet. But the experience was great, and that made it hard to go away from.

Act II: The wrong turn

First try: one machine per agent

I started with the simple approach, an EC2 machine with an EBS volume for each agent. Which is great, and it fits the criteria: the machine gives me isolation and the volume keeps the data.

The catch is that I needed to build the base for everything myself. Creating instances and volumes, the network, setting up each machine, replacing one when it dies, rolling out new versions, and then showing all of that in a simple UI because that’s what the team was used to.

That base became colony, the control plane. The name comes from a bee colony. Our backend tells colony what a customer needs, and colony is responsible for making it real.

One request to colony

Ask for an agent, not for a machine.

Ask The backend asks for what the customer needs. Memory and disk, nothing else.

I was rebuilding Kubernetes

I built a PoC this way and it worked. But while going, I noticed how much colony was doing on its own: picking machine sizes, deciding where each one runs, watching health, replacing dead machines, rolling versions across the fleet. I was rebuilding the same fundamentals of Kubernetes, only slower.

The reason I hadn’t started with Kubernetes was isolation. I wanted pure VM isolation for each agent, and a normal container shares the kernel with every other container on the machine.

Kata made it fit

That’s where Kata Containers came in. Kata runs each container inside its own small VM, while Kubernetes still treats it as a normal container. And it just clicked, because now I could have everything on the list:

  • containers scale fast, so both a new agent and a new version take seconds
  • each agent can still have its own EBS volume
  • each agent still gets a VM of its own
  • growing the fleet means adding machines to the cluster

One request, three hosting designs

The host changed three times. The request did not.

Backend asks for an agent
colony not built yet
the backend knows its host
Railway deleted

1 service + 1 disk per agent

Before · Railway Where we started. Each agent is a Railway service with a disk, and the backend calls Railway itself.

The one thing to plan around is that a machine can only attach a limited number of volumes, so we keep 25 agents on each. That was manageable.

I found two things going down after the change. The first was lines of code in colony: with Kubernetes doing the scheduling and the restarts, one cleanup removed 10,219 lines. The second was the cost of hosting each agent, since 25 of them now share a machine. Scaling got easier too, and the backend didn’t have to change, because its request to colony never named a machine in the first place.

Act III: Rebuilding Railway on AWS

Then the question became how to deliver something close to the Railway experience on AWS and Kata. AWS gives you the pieces but not the product, so most of the two weeks went into building components. Here’s what I built for each work mode.

1. Start fast

A new machine in the cluster needs about four minutes before it can run an agent, which is far too slow for a new customer. So the cluster always keeps a few empty placeholders running that do nothing except hold a place.

Start time: about 20 seconds

An empty placeholder takes the four minute wait.

A warm machine Each machine keeps a few placeholders. A placeholder holds a place and does no work.

In my tests an agent that landed on a free place was ready in about 20 seconds. When there was no free place it took the full four minutes.

2. Stay up and keep its data

The worst mistake this system could make is giving an agent the wrong disk or wiping a good one, so the agent checks its own disk before it starts.

Agent 4012 checks its disk before it starts

Four disks, one of them wrong.

Right disk The label on the disk names this customer. The agent starts.

I tested it by giving one agent another customer’s disk on purpose. It refused to start and colony raised an alarm, with nothing written to the disk.

Colony also takes a backup of every disk each hour, gives more memory to an agent that runs out of it twice, grows a disk that fills up, and checks every agent once a minute.

3. Stay isolated

What one agent can reach

The wall around each agent.

Kata VM agent /data its own VM, no AWS keys router relay internet another agent cloud keys our network
Two ways in Only the router and the relay reach an agent. The router brings messages from chat apps. The relay brings calls from our own backend.

The part I’d point at after the Railway incident is that an agent has no cloud keys at all. If someone breaks out of one, they don’t find credentials for anything else.

4. Release fast

Releases were where we got bottlenecked on Railway, so I wanted them fast here. They also had to be safe, since a release reaches 250 customers during their working day. Colony sends each release to a single agent first, called the canary, waits 30 minutes, and then rolls out to the rest in groups.

250 agents, one release

A bad release should stop at one agent.

1 / 250 canary
Canary One agent takes the release. It checks that it runs the version the release names, then it must report that it is healthy.

Two details helped with speed. Every machine downloads the new version before the first agent switches, so agents don’t wait for a download. And when a fix is urgent, colony can skip the 30 minutes and update everyone at once, still with the health checks.

And a dashboard

Since the team was used to Railway’s UI, colony has a dashboard of its own. It lists every agent with its logs, shows the releases and the health of the system, and lets you restart an agent, give it more memory or disk, or move it to a new version early. It’s much smaller than Railway’s, but it has what we used day to day.

Act IV: Moving day

Test it for real, then destroy it

Before moving anyone I built each design in a real AWS account with fake customers, ran it, and tore it down again. Those runs turned up fourteen bugs. Most were small, like AWS rejecting a security group because its description had an apostrophe in it, but a few I don’t think I’d have found any other way.

The cluster once added two machines for a single agent. A new machine reports ready about three minutes before Kata finishes installing on it, so the agent still couldn’t start and the cluster asked for one more. New machines now carry a “not ready yet” mark until Kata is done.

A crashed agent also couldn’t come back, because Kata keeps the disk locked to the VM and a restart in place fails. Colony starts a fresh VM for it now.

A wrong version got past the canary, since nothing checked that the agent was running the version the release named. The agent checks that itself now. I found out it works when my own build script produced a bad version: all 20 test agents refused it and went back to the old one in under a minute.

And idle agents were holding about 2 GB of memory each that they weren’t using, for half an hour at a time, until I changed one setting.

Move one agent, 250 times

The move itself was a script that handled one agent at a time, and by default it only printed what it would do.

One move, run 250 times

The long copy happens while the old agent still works.

Hold Colony makes the new home but keeps it on hold. The disk is there. The agent does not run yet.

If something failed after the switch, the script took the customer back to Railway with the same steps in reverse. It also had a practice mode, which I used to time each move before doing it for real.

Epilogue

So I started with a machine per agent and ended up with less code and a lower hosting cost per agent, and all it took was to stop building what Kubernetes already does. Colony kept only the parts that are specific to us.

The agents have been running on AWS since then, and the backend still talks to colony the same way it did on day one.

Credits: how colony is built

Colony is written in TypeScript with Effect, and I’d pick it again. Most of this service is error handling, retries and time limits, and Effect has those built in. Each call to AWS gets a time limit, so a stuck call only fails the one agent it belongs to. I can also swap the cloud out in tests and run all of colony against a fake one.

For the AWS and Kubernetes calls I use distilled, which has clients written for Effect, so errors and retries look the same everywhere.

I also tried Alchemy in a small experiment and found colony can use its AWS code as a library without taking the rest of Alchemy. I’d like to move there, so that I maintain less cloud code by hand.

Two rules stayed the same through all the design changes. Before colony creates a disk it saves a unique id for the attempt and sends it to AWS, so if the answer gets lost and colony asks again, AWS returns the same disk and not a second one. And every disk carries a label with its customer, so if colony ever sees another customer’s label on a disk, it stops and leaves that disk alone.

Post-credits scene: Nitro Sandboxes

While I was writing this, AWS announced Nitro Sandboxes for EKS, ECS and EC2. With it, AWS puts each container in its own VM for you, and there’s nothing to install. That sounds really exciting, and if I was doing this project today maybe I would consider it.

But the key idea behind colony is to be provider agnostic. Kata works on any cloud and Nitro Sandboxes only on AWS, which matters to us after what happened with Railway.

If you’re running agents like this, or think a VM per agent is too much, find me on X or LinkedIn.

React: