# Hetzner GPU Builder

Hetzner Server Auction GPU provisioning and Nix remote builder orchestration.

## Docs

- `docs/hetzner-auction-hourly.md` — Auction billing model and pricing analysis
- `docs/gpu-training-runbook.md` — GPU training execution guide
- `docs/builder-operator-runbook.md` — Canonical builder lifecycle workflow

## Tools

- `tools/hetzner_auction_value.py` — Server value ranking and filtering
- `tools/hetzner_market.py` — Robot API for ordering, rescue, cancellation
- `tools/probe_hetzner_rescue.py` — Hardware/network fact collection in rescue mode
- `tools/prepare_rescue_stripe.sh` — N-disk RAID0 rootfs preparation
- `tools/kexec_builder.sh` — Kexec into canonical builder rootfs
- `tools/revalidate_builder_from_rescue.sh` — End-to-end revalidation workflow
- `tools/sync_dotenv_from_hetzner.py` — Sync Robot credentials to .env
- `tools/builder-progress-check.sh` — Monitor GPU/CPU/RAM/disk during builds

## Setup

```bash
# Copy Hetzner Robot credentials
cp hetzner .env  # or use sync_dotenv_from_hetzner.py

# Find GPU servers
python tools/hetzner_auction_value.py --only-gpu --top 10

# Order cheapest Ada GPU server
python tools/hetzner_market.py order --preset cheapest-ada --execute

# Wait for transaction
python tools/hetzner_market.py transaction --wait-for-server --timeout 600

# Full revalidation (rescue → probe → rootfs → kexec → register)
bash tools/revalidate_builder_from_rescue.sh
```

## Environment Variables

- `HETZNER_ROBOT_USER` — Robot Webservice username
- `HETZNER_ROBOT_PASSWORD` — Robot Webservice password
- `TOTALLY_SPIES_REPO_ROOT` — Repo root (auto-detected via git)
- `TOTALLY_SPIES_DEVENV_STATE_ROOT` — State directory (default: `.devenv/state`)

## License

Apache 2.0