One rented GPU, three operators, no stop button
*One rented GPU, three operators, and no stop button. That is the whole setup. Sharing one GPU across operators exposed coordination risks; the register also records script, environment and capacity failures.*
We rent a single GPU in the cloud: a 24 GB card on a g2-standard-4 instance, in one region, at about $0.70 an hour while it runs and about $0.83 a day for the disk while it sits idle. The account's GPU quota is one, so exactly one job can run at a time. Three operators share it — two of our own work sessions and a third operator's jobs, which we only start and hand back — and coordination is left to a shared board. Jobs are non-interactive: a startup script is attached to the instance as metadata, the machine boots, runs the job by itself, uploads its results to object storage, and powers itself off. Jobs run without interactive control; operators can monitor logs and status.
That is a cheap, workable arrangement. It is also an arrangement where some coordination failures were silent, and where, in the cases that follow, the first sign of trouble was a missing result file. The register covers incidents from November 1 onward; on November 13 we formalised same-day filing and prevention-rule updates, each entry carrying its cause once known and a prevention rule attached. This note is what the register taught us.
The stop that stopped the wrong job
The incident we consider the worst happened on the second day. One operator meant to stop their own job and instead stopped another operator's portrait batch mid-run. Seventeen of thirty images had already been uploaded and were safe; the one being generated at the time was lost, and a recovery run finished the remaining thirteen.
The mechanism is worth spelling out. Starting an instance that is already running is a silent no-op: the machine simply keeps running the job it already had. The observed no-op combined with a failure to verify boot ownership — the operator believed they had started their own job, and the stop that followed landed on the running machine. The lesson we took is not "be more careful." It is: verify TERMINATED immediately before starting, and never stop a boot whose ownership you cannot establish; the runner therefore has no stop command.
Our answer was to remove the stop operation from the tooling entirely. All lane operations now go through a single runner script, and that script has no stop or delete command at all. The runner exposes no stop or delete command, so an operator cannot issue those operations through it. The same runner is the only way to claim the lane, swap a startup script, watch a job, or release the lane when a job finishes.
The board everyone misread
With no live coordination, the shared state lives in one file: a lane board that says who is using the GPU. On November 10 and 12, a stale board entry and a misread prose entry exposed coordination risks. Once, the board still showed an earlier attempt number while a later attempt was already running; nothing broke, but another operator could have made a mistaken decision from stale information. Days later, an operator read the phrase "back to free" inside a prose log entry, took it for the lane state, and overwrote the other session's startup script with their own. No instance was running and no generation was lost — the script was restored byte-for-byte once the mistake was noticed — but the failure mode was clear: state kept in human prose can be misread by a human, or by a script parsing prose.
The fix was to make exactly one line of that file the truth. The board now carries a single machine-readable state line, rewritten automatically by the runner on every claim and release, and operators are told to read that line and nothing else. The prose log underneath is history, not state. No recurrence of either misreading is recorded since the change. We distinguish written, implemented and verified rules; a rule is marked verified only after it is observed working on a real job.
The failures that looked like nothing
Several incidents produced no images; the initial upload failure produced no reports, and the typo sent reports to the wrong prefix. The most instructive took roughly a minute and a half. (The register labels it ninety-three seconds; its own timestamps put eighty-two seconds between the recorded start and the shutdown.) A validation batch booted, and roughly a minute and a half later the instance shut itself down. No result, no status file, no log reached the expected folder, even though the script had a shutdown handler meant to upload them. For about an hour and a half it looked like a silent death with no evidence at all.
The evidence existed; it was in the wrong place. The startup script's job name had a one-letter typo — an extra "o" in the middle of the name. The model fetch went to a storage prefix that did not exist, the job failed its own startup gate, and the status file and log were uploaded correctly — to the misspelled prefix. Every report had been filed under the wrong name. Once the typo was found, the batch was relaunched from a template script and all twelve images completed for about $0.90; the failed boot had cost about ninety-three seconds of runtime.
The typo prompted a new name-matching rule and reinforced the existing boot-marker and log-sync rule. First, the runner now cross-checks the job name inside the script against the claimed job name and refuses to install a script that disagrees — a mismatch is rejected before the machine ever boots. Second, every startup script must now post a boot marker within seconds of starting, sync its log every thirty seconds, report its stage in a status file, and upload a final log on shutdown, using a gsutil-to-REST fallback with retries. These checks reduce misrouting risk and provide periodic diagnostics when uploads succeed.
Two environment failures and a separate input-staging failure also produced no images; the latter reported a failed status. Once, a shared environment broke twice in a single night — first a vision-library operator missing because the installed torch and torchvision versions were incompatible, then an old video library failing an import — each costing a whole boot and producing zero images. The third attempt, with a preflight import gate added, completed six of six. Now the import checks and a disk-capacity check run before any generation starts, and a failure there ends the job immediately with a reason in its status file. And once, a reused script failed after 68 seconds because it unconditionally fetched reference images that the new job had never staged; the fix was to copy three files and rerun — four of four, about $0.21 — and the rule is now to compare a reused script's unconditional fetches against the new job's staged inputs before starting.
The failures you cannot prevent
Not everything in the register has a fix. The region's GPU capacity ran out repeatedly: one validation batch needed thirteen start attempts before one succeeded, and a diagnostic boot failed five more times the same evening. Failed starts cost nothing — these unsuccessful starts incurred no instance-running charge, and only the disk charges continued while the machine sat stopped — but they cost half a day of waiting, and there is no action on our side that creates capacity in a region.
What we could control was the retry. Automatic retry with the claim held — so a failed start does not lose your place in line — became standard. But the retry itself then failed silently: a background retry process died after eight stockouts, nobody was notified, and the lane board stayed "busy" for about ten hours while the instance sat idle and the other operator waited. The validation batch eventually ran the same evening — started on the second attempt, six of six images in twenty-three minutes, about $0.28 — but the ten-hour gap was pure waste. The rule that followed is deliberately modest: a claim that stays busy for hours without a successful start must be checked by a human — instance state, retry process, last log line — and the retry mechanism itself needs to survive restarts rather than depending on one background process. That last part is still prose, not code; the register says so honestly.
One more distinction earned its rule the quiet way. A monitoring session timed out after forty polls with only two of six images done, and it was tempting to call the job failed. The job was still running and finished six of six; the results were collected by hand. Since then, a monitoring timeout is explicitly not a job failure: success is judged by the status file and the results in storage, never by whether the watcher was still watching.
What the register is for
As of November 14, the register had thirteen entries and eleven prevention rules. As of November 14, two rules were prose-only: R10 (the input-manifest check on script reuse) and R11 (the retry-survival check) — and the register marks that plainly instead of pretending they are enforced. The enforced ones share a pattern: they remove the dangerous operation, make the state machine-readable, or move a check to before the expensive step instead of after it.
The register is, in our judgment, the cheapest part of the whole setup — a same-day entry is a small task next to failures that cost hours, lost images, and a day of waiting. Its value is not the list of mistakes. It is that we implement mechanically enforceable lessons in tooling, and explicitly label the manual or pending rules. The next operator to share a single rented GPU does not need our thirteen entries. They need the habit: write it down the day it happens, find the cause. Our goal is to make the next occurrence impossible — or, where it cannot be made impossible, make it loud.
Methods
*One rented GPU (24 GB, about $0.70/h running, about $0.83/day disk idle), one region, quota of one; monthly budget ¥10,000 within trial credit. Jobs run non-interactively from startup-script metadata, with inputs and results in object storage and automatic power-off on completion. Costs quoted are the approximate costs recorded in the incident entries for the named runs. The incident register and prevention rules are kept alongside the runner and updated on every incident; entries quoted here are the register's own record, with causes as determined at the time.*