E2E sandbox hub cleanup
Use this when Hub lifecycle (Stripe test) or manual sandbox sign-ups leave e2e-* customer hubs running on platform-sandbox (https://platform-sandbox.lovely-home.co.uk) with Stripe test subscriptions.
Typical symptoms:
- Orphan hubs still answer at
{site_id}.lovely-hub.comafter a red lifecycle run
Since 17 Oct 2026 most of this is automatic:
- A lifecycle run that fails after signup cancels its own hub's Stripe subscription before reporting the failure; the cancel webhook starts teardown. Look for the
cleanup:annotation in the Playwright report (#938). It doesn't wait for teardown to finish — allow 10–15 minutes. - The cleanup's billing purge waits for its own registry drops to land on
main(up to 15 min,PURGE_REGISTRY_WAIT_MS), so one run clears the "billing only" cards (#938). e2e-*hubs are never listed on the production platform admin — they're sandbox-only (#940).
Related: ops-deprovision-stripe-test-hubs.md (production platform test hubs), platform-billing.md.
When to use
| Situation | Action |
|---|---|
Failed lifecycle left live e2e-* hubs on sandbox | Run E2E sandbox cleanup (this doc) |
You know exact site_id values | Workflow input site_ids (comma-separated) |
| After a Hub burst test (hub-burst-test.md) | site_ids = the comma list in that run's summary |
Unknown orphans from lovely-e2e@gmx.co.uk (or E2E_OWNER_EMAIL) test subs | Leave site_ids empty — discovery lists Stripe test customers by that email and reads site_id metadata on their subscriptions |
Production *.lovely-hub.com test-mode mistakes | ops-deprovision-stripe-test-hubs.md |
Sandbox only: the workflow refuses live Stripe keys and non-sandbox PLATFORM_API_ORIGIN.
Automatic sweep every 6 hours (#1086)
E2E stale hub sweep (.github/workflows/e2e-stale-hub-sweep.yml) runs at :41 past every sixth hour on the VPS, so it's free. It runs whether or not the regression ran.
- It lists
e2e-*sites inplatform/sites.yamlon main whose latestrecord … in site registrycommit is 3+ hours old. A lifecycle run records and drops its hubs within about 15 minutes, so a live test is never touched. - It dispatches E2E sandbox cleanup dispatch on the public repo with exactly those ids (
dry_run: false), usingPLATFORM_GITHUB_TOKEN. - It posts the hubs it's cleaning up to Slack #workflows. A failed sweep alerts through Workflow alerts.
Run it by hand from Actions → E2E stale hub sweep. Manual runs default to dry_run: true, which lists hubs only and lets you change stale_hours.
Typical leftovers: a registry drop lost during a queue backlog, or a late duplicate record that re-adds a hub after its drop (#1078).
Stripe test subscriptions: audit, cancel, nightly sweep (#1466)
Tearing a hub down never cancelled its Stripe test subscription, and Multi Home (account-plan) subscriptions had no cleanup outside a lifecycle run that finished its own tidy-up. One script, scripts/sweep-e2e-stripe-subscriptions.mjs, now does three jobs. It lists every live (trialing, active, past_due) subscription on the e2e owner's Stripe test customers and sorts each one as a hub subscription (site_id metadata is an e2e-* slug), an account plan (checkout_kind: account_plan or account_plan metadata, no site_id) or other, which it never touches.
| Where | What it does |
|---|---|
E2E sandbox cleanup → resolve-orphans → Stripe audit | Every run, dry or not, prints the subscriptions with age and what the nightly sweep would do with each. The job summary has the table. Nothing is cancelled. |
| E2E sandbox cleanup → each torn-down hub | After the inline teardown, scripts/mark-e2e-hub-torn-down.mjs sets the hub's billing row to status = canceled with deprovision_dispatched_at set, then the script cancels that hub's subscription (--site-ids {id} --apply). The cancel webhook finds a cancelled row whose teardown is already dispatched and does not enqueue a second one; the purge in the next job deletes the row as before. |
| E2E Stripe sweep (nightly, 05:17 UTC, VPS runner) | --apply with STALE_HOURS 3: cancels hub subscriptions 3+ hours old whose hub is not in platform/sites.yaml (a registered one is the stale hub sweep's job, and the cleanup cancels it then), and account-plan subscriptions 3+ hours old unless an e2e-* hub was recorded in the last 3 hours, because a running Multi Home test owns its plan and Stripe does not say which run a plan belongs to. Posts to Slack #workflows when it cancels something; a failed run alerts through Workflow alerts. |
Run the sweep by hand from Actions → E2E Stripe sweep. Manual runs default to dry_run: true and let you change stale_hours. Locally:
STRIPE_SECRET_KEY=sk_test_… E2E_OWNER_EMAIL=… node scripts/sweep-e2e-stripe-subscriptions.mjs # audit
STRIPE_SECRET_KEY=sk_test_… E2E_OWNER_EMAIL=… node scripts/sweep-e2e-stripe-subscriptions.mjs --apply # cancelStripe rate limit (#1491). The e2e owner gets a new Stripe test customer on every sign-up, so it has hundreds. The sweep and the lifecycle tests' Stripe helpers (listOwnerSubscriptions in e2e/lib/stripeApi.js) read the owner's customers with their subscriptions expanded (expand[]=data.subscriptions): one request per 100 customers, every page. Do not go back to listing subscriptions per customer and per status: that was hundreds of requests per call inside the tests' poll loops, and on 2026-11-07 it hit Stripe's test-mode limit (25 requests a second) and failed the nightly Free and Plus runs with "Request rate limit exceeded". Stripe embeds up to 10 subscriptions per customer (every status except canceled); a customer with more gets its own list call.
The script refuses live keys. A cancel webhook that Stripe retries after the purge recreates a cancelled billing row for the gone site; the platform does not tear anything down for it (the site is not in the manifest), and the next audit shows nothing live in Stripe.
GitHub Actions — E2E sandbox cleanup
- Actions → E2E sandbox cleanup → Run workflow.
site_ids— optional comma-separated list (e.g.e2e-iiatslo6,e2e-r7r7w6ed). Empty runsscripts/list-e2e-orphan-hubs.mjsagainst Stripe test + hub-status (orphans withreadyorregistered). Discovery misses hubs that are half torn down (no longerready/registeredbut with an active billing row — the "billing only" cards); name those explicitly — explicit ids skip the orphan filter.dry_run— default true (list only). Set false to run billing deprovision for each id.use_github_hosted— same as Platform site billing deprovision when self-hosted runners are unavailable.
Each non-dry site runs the same billing deprovision composite action as Platform site billing deprovision (customers stack, registry drop → terraform destroy), with skip_platform_hub_archive: true so disposable e2e-* hubs do not accumulate {site_id}/site-backup-* keys in platform R2. Matrix max-parallel: 3.
If older lifecycle runs already uploaded hub JSON under e2e-* prefixes in lovely-home-hub-archives, prune those keys locally (does not delete Playwright reports under e2e-reports/):
# Preview (default — no deletes)
CLOUDFLARE_ACCOUNT_ID=… CLOUDFLARE_API_TOKEN=… \
npm run prune:e2e-hub-site-archives-r2
# Delete all objects whose first path segment is an e2e sandbox site id
CLOUDFLARE_ACCOUNT_ID=… CLOUDFLARE_API_TOKEN=… \
npm run prune:e2e-hub-site-archives-r2 -- --applyOptional bucket override: PLATFORM_ARCHIVE_R2_BUCKET (default lovely-home-hub-archives). See also platform-site-archive.md.
Matrix jobs pass --skip-platform-admin so parallel teardowns do not race on a single Cloudflare Pages deploy. After all sites finish, redeploy-sandbox-platform-admin runs once: manifest rebuild, purge site_billing rows in lovely-home-platform-billing-sandbox for each cleaned id whose platform/sites.yaml entry is already gone on main (it first waits up to 15 min for the matrix's registry drops to land, re-fetching origin/main every 30 s; ids still on the registry after that are skipped with a warning so one stale orphan does not block deploy), then refresh-sandbox-platform.sh. That rebuilds the commit home-dashboard-platform-sandbox already serves with main's site list, so a cleanup never swaps the code under test for main (#1135). Production home-dashboard-platform is never touched by this workflow.
Registry drops land on main asynchronously via the hub-registry queue (platform: drop … from site registry commits). A green cleanup run can finish before your laptop’s checkout catches up — platform/sites.yaml on origin/main is the source of truth, not an stale local tree.
Required secrets
Same as Platform site billing deprovision: STRIPE_SECRET_KEY (sk_test_*), E2E_OWNER_EMAIL, Cloudflare / Terraform R2 state, PLATFORM_GITHUB_TOKEN, archive and hub provision secrets (see workflow file).
Public repo (faster)
Run the same cleanup on ubuntu-latest in marklovely/lovely-hub-provision (no VPS self-hosted queue). Template: public-repo-workflow-template-e2e-sandbox-cleanup-dispatch.yml → Actions → E2E sandbox cleanup dispatch.
After merging any change to public-repo-workflow-template-e2e-sandbox-cleanup-dispatch.yml on home-dashboard, re-publish the public workflow so lovely-hub-provision stays in sync (including SKIP_PLATFORM_HUB_ARCHIVE_FOR_E2E and skip_platform_hub_archive: true on matrix billing-deprovision — without these, teardown fails when the hub worker/Pages asset is already gone):
node scripts/sync-public-e2e-cleanup-workflow.mjs --repo marklovely/lovely-hub-provisionOr copy .github/workflows/e2e-sandbox-cleanup-dispatch.yml from the template into a PR on lovely-hub-provision.
| Secret / env | Purpose |
|---|---|
HOME_DASHBOARD_REPO | marklovely/home-dashboard |
HOME_DASHBOARD_DISPATCH_TOKEN | PAT Contents: Read on private repo (same as public-ci-dispatch) |
STRIPE_SECRET_KEY | Stripe test (sk_test_ / rk_test_) |
E2E_OWNER_EMAIL | Stripe discovery inbox (e.g. GMX lovely-e2e@gmx.co.uk) |
| Provision profile set | Reuse secrets already on the public repo for hub provision — CLOUDFLARE_*, TF_STATE_R2_*, PLATFORM_GITHUB_TOKEN, HUB_PROXY_SECRETS_JSON, archive/billing keys, etc. (hub-provision-public-actions.md) |
Job env PLATFORM_GITHUB_REPO | Set in the dispatch template to ${{ secrets.HOME_DASHBOARD_REPO }} on billing-deprovision and redeploy-sandbox-platform-admin (GitHub cannot override GITHUB_REPOSITORY). Required so sync-git-main-before-site-delete.mjs resets home-dashboard main, not the public repo (which has no platform/sites.yaml). Same pattern as hub-provision dispatch. |
Inputs match the private workflow: optional site_ids, dry_run (default true), plus optional checkout_ref (default main) to test a branch of home-dashboard. Matrix jobs always set prune_public_repo_secrets: true, skip_platform_admin: true, and skip_platform_hub_archive: true (job env SKIP_PLATFORM_HUB_ARCHIVE_FOR_E2E: '1'); redeploy-sandbox-platform-admin still checks out main after registry drops.
Private E2E sandbox cleanup remains available (self-hosted by default; enable use_github_hosted when the public dispatch workflow is not published yet).
Local discovery
export STRIPE_SECRET_KEY=sk_test_...
export E2E_OWNER_EMAIL=lovely-e2e@gmx.co.uk
export PLATFORM_API_ORIGIN=https://platform-sandbox.lovely-home.co.uk
# Stripe discovery + hub-status table
node scripts/list-e2e-orphan-hubs.mjs
# Explicit ids
node scripts/list-e2e-orphan-hubs.mjs e2e-iiatslo6,e2e-r7r7w6ed
# JSON
node scripts/list-e2e-orphan-hubs.mjs --json --orphans-onlyManual fallback (single site)
If the site is still in platform/sites.yaml:
- Actions → Platform site billing deprovision →
site_id,use_github_hostedif needed.
If the site is already off sites.yaml but infra remains:
- Actions → Platform site deprovision →
site_id,terraform_stack:customers.
Emergency local scripts: scripts/billing-deprovision-hub-site.mjs or scripts/deprovision-hub-site.mjs — see ops-deprovision-stripe-test-hubs.md.
Stale cards in sandbox platform admin
If hubs are gone from main but cards remain (often billing only):
- Confirm the site is off
origin/main:git fetch origin main && git show origin/main:platform/sites.yaml | rg '{site_id}'(no match = registry drop landed). - Purge sandbox billing D1 (Wrangler logged in):
git fetch origin main # optional; purge allows local drift when absent on origin/main
export PLATFORM_BILLING_D1_NAME=lovely-home-platform-billing-sandbox
node scripts/purge-platform-site-for-reuse.mjs e2e-exampleIf the site is still on origin/main, run Platform site billing deprovision (or E2E sandbox cleanup with dry_run: false) — do not purge D1 first.
After teardown, reuse slugs with scripts/purge-platform-site-for-reuse.mjs when billing D1 rows should be cleared. e2e-* slugs use the sandbox billing database automatically when PLATFORM_BILLING_D1_NAME is unset.