Verification matrix (W7)

Sprint 46 W7 calls for an end-to-end verification on each adapter: provision → cloud-init → KVM check → mvm smoke → mvmd smoke → teardown.

Status — 2026-05-06

Provider	Preflight	Live cycle	Seccomp probe	Notes
Hetzner	✅	n/a (no KVM tier)	n/a	Runs the no-KVM lane only
GCP	✅	✅ verified 2026-05-06	✅ pass	$0.10 / 25 min, n2-standard-4
AWS	✅	✅ verified 2026-05-06	✅ pass	$2.50 / 35 min, c5.metal
Azure	—	⏳ pending	?	Needs az CLI install
Vultr	—	⏳ pending	?	Needs vultr-cli + token
Latitude.sh	❌	⏳ pending	?	Needs LATITUDE_API_TOKEN
Equinix Metal	—	⏳ pending	?	Needs metal CLI + token
OVH	❌	⏳ pending	?	Needs three OVH_* env vars

What “verified” means here

• up.sh PROVIDER=<x> runs end-to-end without intervention.
• The seccomp functional probe (cargo test -p mvm-guest --test seccomp_apply) passes — that’s the load-bearing thing on Hetzner that gets skipped without /dev/kvm.
• smoke-mvmd.sh passes (modulo the upstream-known mvmd v0.3.0 tenant persistence gap, which both verified providers also hit).
• down.sh cleanly terminates the instance.

Bugs caught while verifying

The verification matrix is doing its job: each adapter has surfaced real bugs that wouldn’t have shown up in static lint.

GCP

1. SSH key injection (GCE doesn’t auto-inject to root). Fixed via --metadata=ssh-keys=root:<pubkey>.
2. 10GB default boot disk too small. Bumped to 50GB.
3. Stale ~/.ssh/known_hosts entries on IP reuse. Fixed in up.sh.
4. NAT idle timeouts during long cloud-init waits. Fixed via SSH keepalives.
5. mvmd and mvm-hostd live in different workspace packages. post-deploy.sh uses --package <crate> --bin <name> for each.
6. mvmd-coordinator validator rejects [autoscale] enabled = false. Template ships enabled = true with min=max=1 nodes.

AWS

1. Default-VPC default-SG blocks inbound SSH. Adapter now creates a per-region mvm-ephemeral-ssh SG with op-IP-only ingress.
2. Ubuntu AMI 8GB root volume runs out mid-build. Adapter now sets --block-device-mappings for 50GB gp3.
3. Ubuntu AMIs install /root/.ssh/authorized_keys with a forced “login as ubuntu” command. Adapter splices a clean entry into cloud-init’s existing write_files: list.

Cross-cutting

• Clippy fails on mvm-seccomp-apply.rs with 196× unnecessary_cast (i64 → i64). Fixed by tinylabscom/mvm#83.
• The cloud-init desired.json schema was using version instead of node_id/prune_unknown_*. Fixed.

Cost / time per cycle

A full verification cycle on each provider (provision → cloud-init → build → smoke → tests → teardown) takes 25-40 minutes wall-clock and $0.10-3.00 in cloud charges depending on the SKU. AWS bare metal is the priciest at ~$4.08/hr for c5.metal; GCP nested virt the cheapest at ~$0.20/hr for n2-standard-4.