Training a character LoRA on a rented RTX 3090
*1,500 steps at about 2.8 seconds per step, a peak of 15,531 MiB of VRAM, and a total bill of roughly $0.71 — here is what it actually took to train a fixed-character LoRA for Qwen-Image 2.1 on a rented home-class GPU, failures included.*
Most LoRA training reports assume a GPU you own, or a cloud instance you control. We wanted to know whether the same job is realistic on the kind of card a home user might rent by the hour: a previous-generation 24 GB card, booked through a rental portal, with no say over which physical node you get. Our test subject was a LoRA that teaches one fixed character to Qwen-Image 2.1, an open image model. The short answer: the training itself is cheap and fast. Everything around it — moving files in, keeping the container alive, getting the results back out — is where the time went.
Setup
| Item | Value |
|---|---|
| GPU | RTX 3090, 24 GB (rented, high-priority tier, $0.170/hour) |
| Base model | Qwen-Image 2.1 (7B DiT, bf16; Qwen3VL-8B text encoder; bf16 VAE) |
| Trainer | ai-toolkit |
| LoRA settings | rank 32 / alpha 32, learning rate 1e-4, adamw8bit, bf16 |
| Training settings | batch size 1, resolution buckets 512 / 768 / 1024, gradient checkpointing, cached text embeddings, text encoder in qfloat8, low-VRAM mode |
| Dataset | 52 portrait and everyday-life images of one fixed character, each with a caption |
| Container | stock PyTorch runtime image, 4 CPUs, 32 GB RAM, 100 GB storage |
One early detour: our first-choice trainer, musubi-tuner, does not support Qwen-Image 2.1 yet — its documentation covers Qwen-Image and several Edit variants, but not this one. ai-toolkit lists 2.1 support explicitly, so we switched before spending anything.
Results
| Run | Steps | Seconds/step | Peak VRAM | Training time | Cost |
|---|---|---|---|---|---|
| Smoke test | 60 | ~2.8 (range 2.0–5.3 across resolution buckets) | not measured | 4 min 45 s including caching and saving | included in the day's experiment total |
| Full run v1 | 1,500 | 2.76, then 2.80 on the re-run | 15,531 MiB | ~89 min of training | ≈ $0.71 for both runs (see below) |
| Full run v2 | 1,500 | 2.69 | — | 1 h 07 min of stepping | ≈ $0.53, including about an hour of idle time after it finished |
Checkpoints were saved every 250 steps; each interval, sampling included, took about 14 minutes. Loss stayed in a healthy range throughout (0.2–0.55 on the full run). Two things stand out. First, the card was never the constraint: at a 15.5 GB peak there was comfortable headroom, and step time barely moved between runs or dataset versions. Second, our own mistakes cost more than the successful run — the first run's outputs were stranded on the node (see below), so we ran it twice. Both runs together, billed on metered running time, came to about $0.71.
What went wrong
Exit 127, over and over. Our first containers died within two seconds of starting, reproducibly, on more than one node. Exit 127 means "command not found", and the culprit was our own bootstrap: it fetched its setup script with curl, and the stock PyTorch runtime image contains neither curl nor wget. We confirmed the absence on a probe container and switched the fetch to python3, which the image does have. The same image bit us again later: training died instantly because libGL.so.1, required by opencv-python, was not installed either. One line in the package install list fixed that too.
A zip file that broke itself. The 171 MB dataset archive exceeded the upload size limit as a single file (HTTP 413), so we split it into eleven 16 MiB parts. Reassembling them with a plain glob pattern concatenated the parts in dictionary order — part 10 and part 11 landed before part 2 — corrupting the archive and trapping the container in a roughly 90-second restart loop. Zero-padding the part numbers fixed it. The upload limit is a fact of the platform; the join order was entirely our bug, and it is the kind of bug that only appears at exactly this file count.
The node lottery is bandwidth. Downloading roughly 31 GB of model files took about 8 minutes on a fast node (a measured 413 MB/s on the largest file) and 26½ minutes on a slow one, where the 17.5 GB text encoder crawled at about 20 MB/s. Same image, same commands, same price per hour — a threefold difference in dead time before the first step. On a rented node, startup cost is a distribution, not a number.
Stranded by IPv4. The first full run trained perfectly and then could not be collected: our small download server listened on IPv4 only, while the rental gateway required an IPv6 listener, so the instance never reported ready and the gateway returned 503. The trained weights sat intact on the node's disk, unreachable. For the re-run we bound the server to dual-stack IPv6 from the start, verified the ready flag and a 200 response *before* training, and retrieved everything — five checkpoints of 159,436,000 bytes each, samples, log and config. The lesson generalizes: on a rental platform, test the exit door before you start the expensive part.
The dataset lesson
Speed was never the problem, so the second experiment targeted the dataset instead. Version 2 grew the set from 52 to 75 images, adding face-focused material — including portraits generated by another AI image model, plus face crops — on the theory that more faces would mean a more faithful face. It trained fine (2.69 s/step, final loss 0.156) and we rejected it anyway: the outputs' face had drifted toward a different look from the reference character. The synthetic portraits had quietly taught the LoRA a slightly different person. Version 1's all-photo-real dataset remained the winner, and judged by eye against the reference images, its step-1000 checkpoint at strength 1.0 was selected as the best of the five. Our rule from here on: identity data must be photo-real and consistent, and synthetic additions are guilty until proven innocent — no matter how good they look individually.
What we would measure next
Three gaps remain. We never measured peak VRAM on the smoke run, only on the full run, so the low end of the memory curve is still unknown. We would also account cost per checkpoint rather than per run, since the useful checkpoint (step 1000) arrived about 28 minutes before the run ended — the last 500 steps were, for this dataset, optional spending. And the checkpoint comparison was at first judged on training samples, so we ran the side-by-side properly: a fixed, clothed validation set — the same two prompts, the same seed, strength 1.0 — through the step-500, step-1000 and step-1500 checkpoints.
Two honest caveats. Without reference images the renders lean semi-real rather than photo-real, whatever the prompt asks for — the LoRA carries the character, not the medium. And the grid cost almost nothing: about 14 seconds per image warm, and roughly $0.04 of rental time for the whole session including startup. The expensive part of this entire project was never the generation; it was the training logistics above.
Methods
*All runs used the settings in the table above, unchanged between versions; only the dataset differed between v1 and v2. Step times are server-side, measured on warm runs after model loading and caching. Costs are computed from metered container running time at the stated hourly rate, including startup, downloads and idle time until the container was stopped; the rental provider's own meter is the final authority. Likeness was judged by eye against reference images of the character, not by an automated metric.*