Ansible Playbooks 07 / 09 / 10 · verusd · Clever

VRSCTEST auth-repair — the lifecycle

one scroll, the whole repair

Six states, one disk-full crash, one bootstrap restore. Scroll to advance the story; scroll back to rewind.

healthy/restored repair in flight broken/blocked
scroll ↓
frame 0 / 50

What happened on 2026-10-29

The vrsctest verusd daemon on bwd had been serving a stale tip — a corrupted identity index stopped it at block 1293672 with a blocktime ~40 hours old. Every login QR signed at that height embedded a block far outside the wallet's ±3600 s signature-time check, so Verus Mobile refused each scan before the approve screen.

Primary failure mode: the state is a corrupt index, not low disk space. df showed 22 GB free when we restarted the daemon — the space was gone from the diagnosis, but the ExtractIDMap: Corrupt Index verdict remained. A restart alone can never fix a restart-refusing state.

How it was fixed

A full -reindex was tried first (~52 blocks/sec, 6–13h ETA) and killed deliberately. The bootstrap flag path — VERUSD_BOOTSTRAP_FLAG=-bootstrap in docker-verusd/vrsctest/.env — downloaded the signed verustest-bootstrap.tar.gz (~6.3GB), extracted a fresh chainstate + identity index (tip lands ~4 days back), and then peer-synced to live in minutes.

Playbook sequence: 07-setup flips the flag, 09-hygiene confirms blocks==longestchain and flips it back to empty only if no fork is present, 10-shutdown clears it again. Today all three were exercised manually on the vrsctest path — they are still mainnet-hardcoded and deserve a chain-parameterized variant.

Lessons worth keeping

Never trust current df to diagnose a disk-full crash — read the daemon's own log line. Corrupt index ≠ disk-full: two different terminal states, two different exits. Bootstrap is faster than reindex when the chain state (not the blocks files) is the damaged part.