# cultguard-chromium

Standalone flake repo for the patched Chromium build that powers the managed Cultguard browsers.

## What lives here

- the Chromium overlay and patch set
- the local/shared `sccache` plumbing
- a reusable raw NixOS builder image for Hetzner
- a packaged remote-build helper plus `devenv` tasks for the image/build lifecycle

## Flake outputs

- `packages.x86_64-linux.browser`
- `packages.x86_64-linux."chromium-cultguard"`
- `packages.x86_64-linux.image`
- `packages.x86_64-linux.remote`
- `packages.x86_64-linux.cache`
- `packages.x86_64-linux.proof`
- `packages.x86_64-linux.smoke-chrome-first`
- `packages.x86_64-linux.detect`
- `packages.x86_64-linux.detect-test-result`
- `overlays.default`
- `lib.mkCultguardChromeOverlay`

`packages.x86_64-linux.browser` is the stable downstream-consumer alias for the patched browser package. It resolves to the same derivation as `packages.x86_64-linux."chromium-cultguard"` and is the recommended surface for other flakes, including Home Manager / dotfiles repos.

## Downstream flake consumption

Downstream flakes should consume this repo as a normal flake input rather than pinning a raw browser store path:

```nix
{
  inputs.cultguard-chromium.url = "git+https://github.com/cultscale/cultguard-chromium.git";

  outputs = { self, nixpkgs, cultguard-chromium, ... }:
    let
      system = "x86_64-linux";
    in {
      packages.${system}.cultguard-browser =
        cultguard-chromium.packages.${system}.browser;
    };
}
```

That keeps downstream evaluation pure, reuses the browser derivation from this repo directly, and stays compatible with the repo-local `devenv` task flow because the existing `devenv.nix` contract remains unchanged.

## Secrets and defaults

Pure Chromium builds use a source-visible runtime env file: `nix/sccache-runtime.env`, committed in-repo on purpose so both local and remote builds stay Nix-pure.

`devenv` enables dotenv integration, so local `.env` and `.env.hcloud` files are loaded automatically into shells and tasks. Both are gitignored in this repo.

Setup is:

1. Export `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` locally, plus optional `AWS_SESSION_TOKEN`.
2. Run `nix run .#cache -- install`.
3. That writes `.devenv/sccache/config.toml` and populates `nix/sccache-runtime.env`.
4. Build locally with `nix build .#chromium-cultguard` or remotely with `nix run .#remote -- build`.

For remote builds, the VM does **not** need its own repo checkout. The local Nix client evaluates the flake, snapshots the needed source paths, and transfers the required source/store closure to the remote builder. That is why `nix/sccache-runtime.env` must already exist in the local flake source before a pure remote Chromium build.

Build caching uses an in-build `sccache` daemon. The S3 config is tracked in the repo, the runtime `AWS_*` env comes from `nix/sccache-runtime.env`, and both are embedded into the Chromium derivation. This keeps local and remote Chromium builds Nix-pure without any VM-level `sccache.service`.

Required local secrets for the full remote-build workflow:

- `HCLOUD_TOKEN` via env or `.env.hcloud`
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`

Optional local env:

- `AWS_SESSION_TOKEN`
- `CG_SSH_KEY`

## How remote Chromium builds work

1. You run `nix run .#remote -- build` locally.
2. `cg-remote` boots or reuses a Hetzner builder and prepends a remote-builder stanza to local `NIX_CONFIG`.
3. The local Nix client evaluates the requested flake/installable and validates `nix/sccache-runtime.env` for this Chromium build.
4. Nix transfers the needed source/store closure to the remote builder over SSH.
5. The remote nix-daemon executes the derivation under `nixbld*` users.
6. Chromium starts an in-build `sccache` daemon during `preConfigure` and uses the embedded runtime env plus S3 config.
7. Outputs are registered in the remote store and reported back through the normal remote-builder protocol.

In short: the **local machine** evaluates and supplies source; the **remote builder** executes the derivation.

The SSH contract is path-based and local-only:

- `CG_SSH_KEY` is an optional readable local private-key path
- if `CG_SSH_KEY` is unset, the helper falls back to the first existing key from `~/.ssh/id_ed25519`, `~/.ssh/id_ecdsa`, then `~/.ssh/id_rsa`
- cloud-init derives the matching public key and installs it into both `builder`'s and `root`'s `authorized_keys`
- no private-key material is ever sent to the VM

The user split is intentional:

- `cg-remote` connects as `builder` for helper/admin SSH and post-boot inspection
- the generated remote builder config uses `ssh://root@...` for Nix store/build transport only
- actual Nix derivation builds run through the daemon under `nixbld*` users

`sccache` is now build-contained for Chromium:

- the S3 config is a tracked repo asset and is baked into the derivation
- the runtime credential file is `nix/sccache-runtime.env`, which is intentionally part of the flake source
- the build starts its own `sccache` daemon inside the build directory
- the daemon connects directly to the Garage-backed S3 cache during the build
- Chromium builds export `SSL_CERT_FILE`, `NIX_SSL_CERT_FILE`, and `CURL_CA_BUNDLE` to `/etc/ssl/certs/ca-bundle.crt`
- local and remote Chromium builds stay Nix-pure because they no longer depend on evaluation-time process env

## Local cache setup

Materialize the local cache config and build-visible runtime env file:

```bash
nix run .#cache -- install
```

This writes:

- `.devenv/sccache/config.toml`
- `nix/sccache-runtime.env`

`nix/sccache-runtime.env` is intentionally tracked with real cache credentials so it becomes part of the flake source for pure builds. This is deliberate: the creds only gate compiler-cache access.

The helper commands below are optional operator tools; the Chromium build does **not** depend on an external sccache daemon anymore:

```bash
nix run .#cache -- start
nix run .#cache -- status
```

## Local build

Build normally; the runtime `AWS_*` env is read from `nix/sccache-runtime.env`, so the Chromium derivation stays pure:

```bash
nix shell nixpkgs#devenv -c devenv shell -- bash -lc '
  nice -n 20 ionice -c3 nix build .#chromium-cultguard \
    --max-jobs 1 \
    --cores 8 \
    -L
'
```

## Devenv tasks

List the task surface:

```bash
nix shell nixpkgs#devenv -c devenv tasks list
```

Supported tasks:

- `image:build` — build the current builder image locally
- `image:push` — ensure the current builder snapshot exists in Hetzner
- `image:purge` — delete obsolete repo-labeled builder snapshots
- `remote:test` — boot an ephemeral builder, run the remote preflight, then clean up
- `remote:build` — boot an ephemeral builder, run the preflight, run `nix build` with the local flake routed to the remote builder, then clean up
- `remote:clean` — delete the current builder and local helper state
- `worker:list` — inspect local helper state plus active Hetzner workers
- `worker:status` — inspect the currently selected builder plus live remote health over SSH
- `worker:ssh` — open an SSH session to the currently selected builder
- `detect:launch` — run the local smoke suite against a self-contained launched browser using an ephemeral profile/runtime and internal Xvfb
- `detect:test` — compatibility alias for `detect:launch`
- `canary:launch` — run the default local canary fixture sweep against a self-contained launched browser using an ephemeral profile/runtime and internal Xvfb
- `canary:public` — run the opt-in public-site comparison sweep against the same self-contained launched browser path
- `canary:test` — compatibility alias for `canary:launch`
- `gc:workers` — reap stale workers older than `CG_GC_TTL`
- `hold:builder` — boot a builder and keep it up for manual iteration
- `release:builder` — tear down a manually held builder
- `kill:worker` — interactively delete selected workers

Task chains:

- `devenv tasks run image` runs `image:build -> image:push -> image:purge`
- `devenv tasks run remote` runs `remote:test -> remote:build -> remote:clean`
- `devenv tasks run detect:test` is a compatibility alias for `detect:launch`
- `devenv tasks run canary:test` is a compatibility alias for `canary:launch`

The manual-control tasks intentionally live outside the `image:` and `remote:` groups so the grouped runs stay predictable. `kill:worker` remains the fully destructive manual escape hatch; `gc:workers` only reaps stale builders older than `CG_GC_TTL`.

## Local detection smoke

The repo now ships a dedicated local-first detection helper:

```bash
nix run .#detect -- smoke --launch --headful
```

That smoke path does **not** navigate to third-party detection sites and does **not** attach to your live desktop browsers. Instead it:

- launches a temporary Chromium instance from this flake with an isolated profile/runtime
- runs JS probes locally against a data URL
- inspects the resulting CDP events directly
- uses a deliberately unroutable `127.0.0.1:9` fetch only to observe the browser-generated request headers through CDP, so no third-party host sees the request

The stable assertions cover the repo's current stealth surface:

- `navigator.webdriver`
- `navigator.languages`
- `navigator.plugins`
- `navigator.vendor`
- `navigator.hardwareConcurrency`
- `window.outerWidth` / `outerHeight`
- `window.chrome`
- `iframe.contentWindow`
- local fetch request headers such as `Accept-Language`, `User-Agent`, and `Sec-CH-UA` when exposed
- the native-error CDP regression behind `patches/cdp-stealth.patch`

The helper also records softer observations that are useful but can legitimately vary between environments:

- WebGL vendor / renderer
- proprietary codec exposure
- `navigator.userAgentData`
- `chrome.runtime`, `chrome.csi`, and `chrome.loadTimes`
- sourceURL stack markers

Useful entry points:

```bash
# Probe a temporary browser launched from this flake.
nix run .#detect -- smoke --launch

# Probe a temporary desktop-like launch that avoids headless-only fingerprints.
nix run .#detect -- smoke --launch --headful

# Run the supported local coverage through devenv task wrappers.
devenv tasks run detect:launch
devenv tasks run detect:test
```

The `--launch` path is now the only supported smoke target. It starts the flake's `.#browser` binary with isolated profile, config, cache, and state directories plus remote debugging enabled, then tears it down automatically. That isolation avoids contention with the live desktop browser's crash-report lock under `~/.config/chromium`. By default the temporary browser runs headless; pass `--headful` when you want a desktop-like launch without the expected `HeadlessChrome` user-agent differences. The `--headful` mode is self-contained inside the detect runtime: it creates its own Xvfb display instead of depending on the caller's `DISPLAY`.

If the temporary browser still cannot expose DevTools or complete the local self-probe, the helper records a structured warning and artifacts instead of crashing. Add `--strict-warnings` when you want that condition to fail CI.

Every run writes JSON artifacts, page DOM, screenshots, and launch logs into either the directory passed via `--artifacts-dir` or a fresh temp directory under `TMPDIR`.

## Smoke vs canary

`smoke` is the assertive, local-first check. It runs Cultguard-owned JS probes against a local data URL and returns pass/warn/fail counts for our stealth surface, including the CDP native-error regression behind `patches/cdp-stealth.patch`.

`canary` is the observational comparison sweep. By default it opens curated local fixture pages shipped from this repo, captures screenshots, DOM, titles, text excerpts, and the same browser/header/CDP probes used by smoke, while keeping the whole run self-contained. When you need ecosystem comparison, pass `--public` to run the same launched browser against third-party fingerprinting pages. In both modes, canary stays report-only: it now emits a normalized score and weighted checks, but it does not fail the command based on those results.

## Local canary fixtures

The default canary path is now fully local. It serves curated in-repo fixture pages over a temporary loopback HTTP server, so the launched browser sees ordinary page URLs without any third-party traffic.

```bash
nix run .#detect -- canary --launch --headful

# Override the default fixture set with a one-off local page.
nix run .#detect -- canary --launch --headful --site 'data:text/html,<title>ok</title><h1>hello</h1>'

# Or use the matching devenv wrappers.
devenv tasks run canary:launch
devenv tasks run canary:test
```

The local fixtures are curated approximations of the public canaries, not wholesale mirrored copies of those sites. They are meant for routine inner-loop checks and offline/local validation.

The launched canary path reuses the same ephemeral profile/config/cache/state setup as the smoke launcher, so it does not touch your long-lived browser profile or attach to a live desktop instance.

Each canary page now includes structured evaluation data in `summary.json`:

- `evaluation.checks[]`: per-check `status`, `severity`, `weight`, `earnedWeight`, and `evidence`
- `evaluation.score`: the page score with earned/max weight and percentage
- target `summary`: rolled-up per-target counts and score
- top-level `aggregate`: the full run score and pass/warn/fail site counts

The CLI mirrors that summary with per-site scores plus a final aggregate line. This gives you a consistent collection/scoring model across both local fixtures and `--public` runs without turning canary into a hard gate yet.

The default local fixture set includes:

- `local-sannysoft`
- `local-areyouheadless`
- `local-intoli`
- `local-fpscanner`
- `local-amiunique`

## Public canary comparison

The public canaries remain **opt-in** because the destination sites can see and log each visit:

```bash
nix run .#detect -- canary --launch --headful --public

# Or use the matching devenv wrapper.
devenv tasks run canary:public
```

The built-in public comparison set is:

- `bot.sannysoft.com`
- Antoine Vastel `areyouheadless`
- Intoli `chrome-headless-test.html`
- `fpscanner.com/demo`
- AmiUnique

The canary command captures screenshots, DOM snapshots, a short text excerpt, and scored probe output for each page. The local fixture mode is the routine path; the public mode is there when you explicitly want outside comparison.

## Packaged helper

The low-level helper is:

```bash
nix run "git+file://$PWD#remote" -- <subcommand>
```

Supported subcommands:

- `help`
- `image`
- `push`
- `up`
- `status`
- `ssh [command...]`
- `down`
- `gc`
- `purge`
- `clean`
- `list`
- `kill`
- `build [nix-build-args...]`
- `test`

High-level behavior:

- `push` resolves the current `.#image` derivation, reuses a matching uploaded snapshot if available, or uploads a fresh one
- `up` boots a builder from that image, passes the derived SSH authorized key through cloud-init for both `builder` and `root`, waits for cloud-init to finish, uses configless SSH probes so host `ssh_config` cannot interfere, and writes explicit local helper state under `.devenv/` plus a `big-parallel`-capable remote builder config
- `status` turns the retained `.devenv/builder.json` state back into a concise operator view, including live `cloud-final` and basic host health over SSH when the builder is reachable
- `ssh` opens an interactive SSH session to the retained builder (or runs a single remote command) using the same packaged key/known-host settings as the helper itself
- `build` defaults to `git+file://<repo>#chromium-cultguard`, adds `--cores 0`, and validates that `nix/sccache-runtime.env` is already populated for pure Chromium builds
- `test` runs the remote-build proof; if the local proof output was GC'd or is otherwise not currently valid, the helper automatically retries that proof without `--rebuild`
- `clean`/`down` remove the VM plus local helper state

When you need to hold a builder open manually:

```bash
nix shell nixpkgs#devenv -c devenv tasks run hold:builder
export NIX_CONFIG="$(cat .devenv/builder.nix.conf)"$'\n'"${NIX_CONFIG:-}"
nix build "git+file://$PWD#chromium-cultguard" -L
nix shell nixpkgs#devenv -c devenv tasks run release:builder
```

The direct helper equivalents are still `nix run "git+file://$PWD#remote" -- up` and `... -- down`.

## Runtime env surface

Short repo-specific env vars:

- `CG_ROOT` — repo root; `devenv` sets this automatically
- `CG_REF` — flake ref; defaults to `git+file://<repo>`
- `CG_IMAGE` — explicit Hetzner image/snapshot override
- `CG_TYPE` — explicit Hetzner server type override
- `CG_LOC` — Hetzner location override, default `fsn1`
- `CG_DC` — exact Hetzner datacenter override
- `CG_NAME` — builder VM name prefix, default `cultguard-builder`
- `CG_OWNER` — worker namespace label, default `shared`
- `CG_GC_TTL` — stale-builder TTL for `gc`, default `86400`
- `CG_BUILD` — default installable for `build`, default `git+file://<repo>#chromium-cultguard`
- `CG_ARGS` — extra `nix build` args for `build`
- `CG_KEEP_VM` — keep the VM after `build`/`test`
- `CG_HOST` — explicit invoker-host label, default `explicit-none`
- `CG_RUN` — explicit run-kind label override
- `CG_TARGET` — explicit run-target label override
- `CG_IMPORT_TYPE` — Hetzner image-import worker type override
- `CG_IMPORT_LOC` — Hetzner image-import location override
- `CG_KILL` — scripted selection for `kill` (indexes, IDs, or `all`)
- `CG_SSH_KEY` — optional readable local private-key path; defaults to the first existing key from `~/.ssh/id_ed25519`, `~/.ssh/id_ecdsa`, then `~/.ssh/id_rsa`

Standard env vars still used directly:

- `HCLOUD_TOKEN`

Install-time env vars still used by `nix run .#cache -- install`:

- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- optional `AWS_SESSION_TOKEN`

`HCLOUD_TOKEN` is typically sourced from `.env.hcloud` when you use `devenv` task wrappers for remote commands.

## Runtime tips

- Keep the builder after an automated run for inspection:

  ```bash
  CG_KEEP_VM=1 nix run .#remote -- test
  ```

- Run a manual iteration loop against one held-open builder:

  ```bash
  nix shell nixpkgs#devenv -c devenv tasks run hold:builder
  export NIX_CONFIG="$(cat .devenv/builder.nix.conf)"$'\n'"${NIX_CONFIG:-}"
  nix build .#chromium-cultguard -L
  nix shell nixpkgs#devenv -c devenv tasks run release:builder
  ```

- Override the remote build target or flags without editing tracked files:

  ```bash
  CG_BUILD="git+file://$PWD#chromium-cultguard" \
  CG_ARGS="-L --keep-going" \
  nix run .#remote -- build
  ```

- The default remote build already requests all detected build cores. Only pass `--cores` in `CG_ARGS` when you want to cap it explicitly:

  ```bash
  CG_ARGS="--cores 24 -L" nix run .#remote -- build
  ```

- Monitor a live builder with the retained helper state:

  ```bash
  nix run .#remote -- status
  nix run .#remote -- ssh
  ```

  `devenv` wrappers are also available as `devenv tasks run worker:status` and `devenv tasks run worker:ssh`.

  Once attached, `nix build "$CG_BUILD" -L` is the closest reproduction of the real remote build path from the local side.

- The focused Chromium debug target is available as `git+file://$PWD#smoke-chrome-first`. Wrapper logging is enabled by default there; only set `SCCACHE_WRAPPER_STRACE="$TMPDIR/sccache-wrapper.strace"` when you explicitly want deep tracing.

- Use non-interactive cleanup when you already know what to delete:

  ```bash
  CG_KILL=all nix run .#remote -- kill
  CG_GC_TTL=3600 nix run .#remote -- gc
  ```

## Local helper state

The helper and the task wrappers may write explicit local state under `.devenv/`:

- `.devenv/builder.nix.conf`
- `.devenv/builder.json`
- `.devenv/known_hosts`

Those files are operational artifacts. They are primarily the handoff surface for manual hold-open workflows and ad hoc inspection.

## Validation

Useful local validation commands:

```bash
nix build .#cache --no-link
nix build .#remote --no-link
nix build .#image --no-link
nix build .#detect --no-link
nix flake check --no-build
nix shell nixpkgs#devenv -c devenv tasks list
```

The current cache path is build-contained:

- the builder image only needs SSH and outbound network access
- Chromium starts an in-build `sccache` daemon inside the derivation work directory
- the daemon uses the tracked S3 config plus `nix/sccache-runtime.env`
- the live cache backend is `s3, name: sccache, prefix: /compiler-cache/`
- both local and remote Chromium builds stay Nix-pure

A real Hetzner end-to-end preflight remains green:

- `nix run .#remote -- test` reused or uploaded the matching snapshot, booted a real Hetzner builder, wrote `.devenv/builder.nix.conf`, and built `.#proof` remotely
- the retained builder passed a direct post-run inspection with `cloud-final.service` in `active (exited)`
- the current live contract remains hybrid: `builder` is the human/helper SSH user, while `root` is constrained by `sshd` to `nix-store --serve --write` for transport only

A final pure non-root retry was also performed and is still blocked with the current transport stack:

- `ssh://builder@...` still fails with `unexpected end-of-file`
- `ssh-ng://builder@...` reaches the remote daemon as trusted `builder`, then disconnects with `Broken pipe`
- forcing `builder` directly to `nix-store --serve --write` or `nix daemon --stdio` via dedicated temporary keys did not change the failure
- because of that, the optimized live contract remains hybrid for now: `builder` for admin SSH, constrained `root` only for Nix transport

The live Hetzner proof also flushed out a few helper hardening fixes that are now baked in:

- cloud-init now relocks `root` and clears password-expiry metadata on first boot so provider-side forced password rotation cannot break key-only automation
- helper SSH calls now pass `-F /dev/null`, which prevents ambient host `ssh_config` from breaking the packaged helper
- the non-`cloud-init` fallback now waits only while `cloud-final.service` is `activating`; `active (exited)` is correctly treated as finished
- the `sshd` PAM account stack now replaces the default `pam_unix` account check with `pam_permit` so provider-side root password-expiry noise cannot corrupt the Nix transport stream