Why we wrote our own scheduler instead of using a queue

The default architecture for a workflow engine looks like this: an API writes a job to a queue, workers poll the queue, each step round-trips through the broker. It works, and it is why a ten-step flow takes four seconds on tools you have used.

When we profiled that architecture, the steps themselves were rarely the cost. The cost was the space between steps: serialization, broker hops, polling intervals, and worker wake-ups. A flow spends most of its life waiting for its own infrastructure.

So GoRunner's engine schedules steps in-process. A run is a goroutine, a step transition is a function call, and the queue only appears at the edges where durability demands it. Steps that need external IO yield; everything else proceeds at memory speed.

The trade-offs are real: in-process scheduling means we own backpressure, fairness, and crash recovery ourselves rather than delegating to a broker. The post below walks through how we handle each, and where the design stops being clever and starts being careful.

This is the first post in a series on the engine's internals. The numbers we publish here will always come with a methodology and a way to reproduce them.