We write about the boring parts of running a small GPU cloud, because that is most of it. New posts go to the mailing list.
Model hosting is a pool of SGLang instances with a router in front. Why we picked it, how we pin it, and how we swap model weights without restarting an instance.
The scheduler we run for model hosting: what it does, what it does not do, and why a queue is fine.
Most of instance provisioning is not the GPU. It is the boring software around it. We shaved 30 minutes off and the support queue noticed.
Per-hour billing made idle time feel free. Per-second billing made us rebuild the metering, and it changed what customers do with us.