We audited our LoRA dataset against its own spec. Zero images passed.
*Our second dataset had about 50 percent more images than the first, and it did not beat it. Instead of training a fourth version on still more data, we stopped and audited what we had actually been training on.*
This is a follow-up to our earlier note on training a character LoRA for Qwen-Image 2.1 on a rented RTX 3090. That note ended with a dataset lesson: version 2 of the training set, grown from 52 to 75 images, trained fine and was rejected anyway because its face had drifted toward a different look. A later version 3 grew the set again, to 78 rows. In validation it did not clearly beat version 1. We had been treating that as a training question. It turned out to be a data question, and it had a measurable answer.
How we audited
Two parts of the audit were mechanical. Set membership — which images are actually in v1 and v3 — and duplication were established from the training archives themselves, by MD5 hash, not by filename or by eye. The rest was scoring: every unique image was read against the character's written specification on three axes — hair colour (a specific warm pearl-champagne greige), hairstyle (a specific folded-loop updo), and earrings (slim dangling earrings). One thing we deliberately did not score is likeness. Whether an output looks like the character is a human judgment, and we kept it out of the spreadsheet.
What the audit found
The headline is short: zero images pass all three axes. Not one of v1's 52 rows, and not one of v3's 68 unique images, has the specified hair colour, the specified hairstyle and the specified earrings at the same time.
| Axis | v1 (52 rows) | v3 (68 unique images) |
|---|---|---|
| Hair colour off-spec | 90.4% (47 of 52) | 91.2% |
| Hairstyle off-spec | 98.1% (51 of 52) | 97.1% |
| Earrings pass | 48.1% | 41.2% |
| Earrings confirmed off-spec | 15.4% | 16.2% |
| Earrings not assessable in frame | 36.5% | 42.6% |
The hairstyle numbers are the worst. Across v3's unique images, 47 percent wear braided styles and 31 percent wear the hair loose or undone; other buns and ponytails account for about 19 percent. The specified folded-loop updo — the character's defining hairstyle — appears in exactly two unique images. The earrings column needs a note: in over a third of the images the ears are out of frame, covered by hair, or too distant to score at all, which is its own finding about how the set was composed.
The hair-colour row explains a problem we already knew from the output side. About 91 percent of the unique images in both sets have silver-grey hair rather than the specified warm tone. The finished LoRA has a persistent silver-grey drift that we had been treating as a model quirk. It is not. The model learned the dataset faithfully; the dataset's hair colour is silver-grey. The same pattern held for the earrings: validation outputs kept producing oversized teardrop earrings, and that exact failure is present verbatim in the training data — including in the very image we had designated the set's face anchor. Worse, the few images with the best, on-spec hair colour systematically wear the wrong earrings (beaded drops). The axes are not independently fixable by picking better rows, because the rows that fix one axis break another.
Only six unique images have on-spec warm hair colour at all, and all six come from a single scene family. Zero full-body or clothed images pass hair and hairstyle together. A compliant next dataset cannot be mined from these sets: the passing categories are empty, and those images will have to be generated fresh.
There is also a lineage fact we had not written down before auditing. Version 1 was a mix of two source families: 28 images from our own gallery pipeline and 24 portraits — about 46 percent — produced by another image model. Our earlier note reported that synthetic portraits taught the LoRA a slightly different person. The audit put a number on how much of the set that risk applied to.
The expansion that wasn't
Version 3 was presented, including to ourselves, as v1 plus a batch of new gallery additions. By hash, the story is different. Of the 20 files counted as new additions, 10 were byte-identical renames of v1 files. Only 16 genuinely new unique images were added — and they carried the same off-spec profile as the rest, which is why the larger set behaved like the smaller one. Version 1 is a byte-identical subset of version 3.
The additions also quietly reintroduced a category the original v1 selection rules had explicitly excluded: frames containing a second person. Nobody decided to reverse that rule. It eroded, one unreviewed addition at a time.
A controlled comparison
The same week, we ran the comparison the audit called for: three checkpoints (v1 at step 1000, v3 at steps 1000 and 1500, all at strength 1.0) against four fixed cases, twelve images in total, for about $0.90 of rented GPU time. Two findings. With reference images attached, all three versions were near-identical — the references dominate, and a LoRA difference cannot be measured that way. In pure text-to-image the versions did separate: on a changed-condition case, a cafe scene unlike the training material, v1's face drifted the youngest and most doll-like, while the v3 pair held consistency slightly better. Within v3, step 1000 and step 1500 showed no visible difference. And the oversized-earring problem appeared in all twelve outputs, which is what the audit would have predicted.
What we would do differently
- Audit before adding data. A scored audit of 52 images costs an afternoon. Training on an unscored 78 cost more and answered nothing.
- More data with the same profile changes nothing. Version 3's failure was not insufficient data; it was the same data, plus duplicates.
- Deduplicate by hash, not by filename. Ten of our "new" images were renames. File names are labels; hashes are facts.
- A spec is only real if the dataset is scored against it. We had a written specification the whole time. Until this audit, nothing had ever checked the images against it.
- Synthetic portraits import another model's idea of the character. Nearly half of v1 came from a different image model, and it shows in what the LoRA believes the character looks like.
Methods
*Scoring was done by eye against the character's written spec, on the three axes above; likeness was not scored. Set membership and duplication were determined by MD5 hash of the training archives. The comparison run used the same LoRA recipe family as our earlier rented-3090 note (ai-toolkit, Qwen-Image 2.1) and a rented GPU; its cost is metered running time, as before.*