Where a rented GPU spends its first 40 minutes
Our test nodes are rented by the minute from a shared GPU cloud. Each run starts a fresh container that has no disk left over from the last run, so it fetches everything again. For the repaint-strength test we logged a timestamp at each step of the start, to see where the time goes.
What a start does
1. The cloud finds a machine and starts our container (a public ComfyUI image). 2. The container updates ComfyUI to a newer release from GitHub, because the image ships an older one. 3. It downloads the model files from Hugging Face: the Krea 2 Turbo model (12.24 GB), its text encoder (4.88 GB) and the VAE (0.24 GB). 4. ComfyUI starts and says it is ready.
What happened this time
| Step | First machine | Second machine |
|---|---|---|
| Machine found, container running | 54 s | 57 s |
| ComfyUI update from GitHub | 98 s | 59 s |
| Model 12.24 GB | 526 s (about 23 MB/s) | 1,171 s (about 10 MB/s) |
| Text encoder 4.88 GB | stopped after 25 s | 463 s (about 10 MB/s) |
| VAE 0.24 GB | — | 23 s |
| Ready | never | about 30 min after its start |
- On the first machine the container exited 25 seconds into the second download ("Failed to start server: terminated", then "Instance Exited"). We do not know why.
- Our start script treats that pair of log lines as a crash, stops the machine and asks for another. The second machine was ready 42 minutes after the first start.
- Downloads were about 93% of the second machine's start. The GitHub update was about 1 minute. The 18 pictures took about 5 minutes.
- Download speed was set by the machine, not by us: the same file came at 23 MB/s on one machine and 10 MB/s on the other.
What we are not doing
A common fix is to build our own container image with the models already inside, so a start is one download. In a separate project on the same cloud, an image of about 30 GB took 28 minutes just to download and once did not finish in 60, which was worse than fetching the models. So for now we keep the public image and the Hugging Face downloads, wait up to 30 minutes per machine, and only swap machines when the logs show a crash.
How much to trust this
- One run, two machines, one GPU type. Earlier runs did not log these timestamps.
- Download speeds are averages over each file, from the container's own log.
- The 30 GB image result comes from a different project and different models; we did not repeat it.