Skip to content

E2E sandbox hub cleanup ​

Use this when Hub lifecycle (Stripe test) or manual sandbox sign-ups leave e2e-* customer hubs running on platform-sandbox (https://platform-sandbox.lovely-home.co.uk) with Stripe test subscriptions.

Typical symptoms:

  • Orphan hubs still answer at {site_id}.lovely-hub.com after a red lifecycle run

Since 17 Oct 2026 most of this is automatic:

  • A lifecycle run that fails after signup cancels its own hub's Stripe subscription before reporting the failure; the cancel webhook starts teardown. Look for the cleanup: annotation in the Playwright report (#938). It doesn't wait for teardown to finish — allow 10–15 minutes.
  • The cleanup's billing purge waits for its own registry drops to land on main (up to 15 min, PURGE_REGISTRY_WAIT_MS), so one run clears the "billing only" cards (#938).
  • e2e-* hubs are never listed on the production platform admin — they're sandbox-only (#940).

Related: ops-deprovision-stripe-test-hubs.md (production platform test hubs), platform-billing.md.


When to use ​

SituationAction
Failed lifecycle left live e2e-* hubs on sandboxRun E2E sandbox cleanup (this doc)
You know exact site_id valuesWorkflow input site_ids (comma-separated)
After a Hub burst test (hub-burst-test.md)site_ids = the comma list in that run's summary
Unknown orphans from lovely-e2e@gmx.co.uk (or E2E_OWNER_EMAIL) test subsLeave site_ids empty — discovery lists Stripe test customers by that email and reads site_id metadata on their subscriptions
Production *.lovely-hub.com test-mode mistakesops-deprovision-stripe-test-hubs.md

Sandbox only: the workflow refuses live Stripe keys and non-sandbox PLATFORM_API_ORIGIN.


Automatic sweep every 6 hours (#1086) ​

E2E stale hub sweep (.github/workflows/e2e-stale-hub-sweep.yml) runs at :41 past every sixth hour on the VPS, so it's free. It runs whether or not the regression ran.

  1. It lists e2e-* sites in platform/sites.yaml on main whose latest record … in site registry commit is 3+ hours old. A lifecycle run records and drops its hubs within about 15 minutes, so a live test is never touched.
  2. It dispatches E2E sandbox cleanup dispatch on the public repo with exactly those ids (dry_run: false), using PLATFORM_GITHUB_TOKEN.
  3. It posts the hubs it's cleaning up to Slack #workflows. A failed sweep alerts through Workflow alerts.

Run it by hand from Actions → E2E stale hub sweep. Manual runs default to dry_run: true, which lists hubs only and lets you change stale_hours.

Typical leftovers: a registry drop lost during a queue backlog, or a late duplicate record that re-adds a hub after its drop (#1078).

Stripe test subscriptions: audit, cancel, nightly sweep (#1466) ​

Tearing a hub down never cancelled its Stripe test subscription, and Multi Home (account-plan) subscriptions had no cleanup outside a lifecycle run that finished its own tidy-up. One script, scripts/sweep-e2e-stripe-subscriptions.mjs, now does three jobs. It lists every live (trialing, active, past_due) subscription on the e2e owner's Stripe test customers and sorts each one as a hub subscription (site_id metadata is an e2e-* slug), an account plan (checkout_kind: account_plan or account_plan metadata, no site_id) or other, which it never touches.

WhereWhat it does
E2E sandbox cleanup → resolve-orphans → Stripe auditEvery run, dry or not, prints the subscriptions with age and what the nightly sweep would do with each. The job summary has the table. Nothing is cancelled.
E2E sandbox cleanup → each torn-down hubAfter the inline teardown, scripts/mark-e2e-hub-torn-down.mjs sets the hub's billing row to status = canceled with deprovision_dispatched_at set, then the script cancels that hub's subscription (--site-ids {id} --apply). The cancel webhook finds a cancelled row whose teardown is already dispatched and does not enqueue a second one; the purge in the next job deletes the row as before.
E2E Stripe sweep (nightly, 05:17 UTC, VPS runner)--apply with STALE_HOURS 3: cancels hub subscriptions 3+ hours old whose hub is not in platform/sites.yaml (a registered one is the stale hub sweep's job, and the cleanup cancels it then), and account-plan subscriptions 3+ hours old unless an e2e-* hub was recorded in the last 3 hours, because a running Multi Home test owns its plan and Stripe does not say which run a plan belongs to. Posts to Slack #workflows when it cancels something; a failed run alerts through Workflow alerts.

Run the sweep by hand from Actions → E2E Stripe sweep. Manual runs default to dry_run: true and let you change stale_hours. Locally:

bash
STRIPE_SECRET_KEY=sk_test_… E2E_OWNER_EMAIL=… node scripts/sweep-e2e-stripe-subscriptions.mjs          # audit
STRIPE_SECRET_KEY=sk_test_… E2E_OWNER_EMAIL=… node scripts/sweep-e2e-stripe-subscriptions.mjs --apply  # cancel

Stripe rate limit (#1491). The e2e owner gets a new Stripe test customer on every sign-up, so it has hundreds. The sweep and the lifecycle tests' Stripe helpers (listOwnerSubscriptions in e2e/lib/stripeApi.js) read the owner's customers with their subscriptions expanded (expand[]=data.subscriptions): one request per 100 customers, every page. Do not go back to listing subscriptions per customer and per status: that was hundreds of requests per call inside the tests' poll loops, and on 2026-11-07 it hit Stripe's test-mode limit (25 requests a second) and failed the nightly Free and Plus runs with "Request rate limit exceeded". Stripe embeds up to 10 subscriptions per customer (every status except canceled); a customer with more gets its own list call.

The script refuses live keys. A cancel webhook that Stripe retries after the purge recreates a cancelled billing row for the gone site; the platform does not tear anything down for it (the site is not in the manifest), and the next audit shows nothing live in Stripe.

GitHub Actions — E2E sandbox cleanup ​

  1. Actions → E2E sandbox cleanup → Run workflow.
  2. site_ids — optional comma-separated list (e.g. e2e-iiatslo6,e2e-r7r7w6ed). Empty runs scripts/list-e2e-orphan-hubs.mjs against Stripe test + hub-status (orphans with ready or registered). Discovery misses hubs that are half torn down (no longer ready/registered but with an active billing row — the "billing only" cards); name those explicitly — explicit ids skip the orphan filter.
  3. dry_run — default true (list only). Set false to run billing deprovision for each id.
  4. use_github_hosted — same as Platform site billing deprovision when self-hosted runners are unavailable.

Each non-dry site runs the same billing deprovision composite action as Platform site billing deprovision (customers stack, registry drop → terraform destroy), with skip_platform_hub_archive: true so disposable e2e-* hubs do not accumulate {site_id}/site-backup-* keys in platform R2. Matrix max-parallel: 3.

If older lifecycle runs already uploaded hub JSON under e2e-* prefixes in lovely-home-hub-archives, prune those keys locally (does not delete Playwright reports under e2e-reports/):

bash
# Preview (default — no deletes)
CLOUDFLARE_ACCOUNT_ID=… CLOUDFLARE_API_TOKEN=… \
  npm run prune:e2e-hub-site-archives-r2

# Delete all objects whose first path segment is an e2e sandbox site id
CLOUDFLARE_ACCOUNT_ID=… CLOUDFLARE_API_TOKEN=… \
  npm run prune:e2e-hub-site-archives-r2 -- --apply

Optional bucket override: PLATFORM_ARCHIVE_R2_BUCKET (default lovely-home-hub-archives). See also platform-site-archive.md.

Matrix jobs pass --skip-platform-admin so parallel teardowns do not race on a single Cloudflare Pages deploy. After all sites finish, redeploy-sandbox-platform-admin runs once: manifest rebuild, purge site_billing rows in lovely-home-platform-billing-sandbox for each cleaned id whose platform/sites.yaml entry is already gone on main (it first waits up to 15 min for the matrix's registry drops to land, re-fetching origin/main every 30 s; ids still on the registry after that are skipped with a warning so one stale orphan does not block deploy), then refresh-sandbox-platform.sh. That rebuilds the commit home-dashboard-platform-sandbox already serves with main's site list, so a cleanup never swaps the code under test for main (#1135). Production home-dashboard-platform is never touched by this workflow.

Registry drops land on main asynchronously via the hub-registry queue (platform: drop … from site registry commits). A green cleanup run can finish before your laptop’s checkout catches up — platform/sites.yaml on origin/main is the source of truth, not an stale local tree.

Required secrets ​

Same as Platform site billing deprovision: STRIPE_SECRET_KEY (sk_test_*), E2E_OWNER_EMAIL, Cloudflare / Terraform R2 state, PLATFORM_GITHUB_TOKEN, archive and hub provision secrets (see workflow file).

Public repo (faster) ​

Run the same cleanup on ubuntu-latest in marklovely/lovely-hub-provision (no VPS self-hosted queue). Template: public-repo-workflow-template-e2e-sandbox-cleanup-dispatch.yml → Actions → E2E sandbox cleanup dispatch.

After merging any change to public-repo-workflow-template-e2e-sandbox-cleanup-dispatch.yml on home-dashboard, re-publish the public workflow so lovely-hub-provision stays in sync (including SKIP_PLATFORM_HUB_ARCHIVE_FOR_E2E and skip_platform_hub_archive: true on matrix billing-deprovision — without these, teardown fails when the hub worker/Pages asset is already gone):

bash
node scripts/sync-public-e2e-cleanup-workflow.mjs --repo marklovely/lovely-hub-provision

Or copy .github/workflows/e2e-sandbox-cleanup-dispatch.yml from the template into a PR on lovely-hub-provision.

Secret / envPurpose
HOME_DASHBOARD_REPOmarklovely/home-dashboard
HOME_DASHBOARD_DISPATCH_TOKENPAT Contents: Read on private repo (same as public-ci-dispatch)
STRIPE_SECRET_KEYStripe test (sk_test_ / rk_test_)
E2E_OWNER_EMAILStripe discovery inbox (e.g. GMX lovely-e2e@gmx.co.uk)
Provision profile setReuse secrets already on the public repo for hub provision — CLOUDFLARE_*, TF_STATE_R2_*, PLATFORM_GITHUB_TOKEN, HUB_PROXY_SECRETS_JSON, archive/billing keys, etc. (hub-provision-public-actions.md)
Job env PLATFORM_GITHUB_REPOSet in the dispatch template to ${{ secrets.HOME_DASHBOARD_REPO }} on billing-deprovision and redeploy-sandbox-platform-admin (GitHub cannot override GITHUB_REPOSITORY). Required so sync-git-main-before-site-delete.mjs resets home-dashboard main, not the public repo (which has no platform/sites.yaml). Same pattern as hub-provision dispatch.

Inputs match the private workflow: optional site_ids, dry_run (default true), plus optional checkout_ref (default main) to test a branch of home-dashboard. Matrix jobs always set prune_public_repo_secrets: true, skip_platform_admin: true, and skip_platform_hub_archive: true (job env SKIP_PLATFORM_HUB_ARCHIVE_FOR_E2E: '1'); redeploy-sandbox-platform-admin still checks out main after registry drops.

Private E2E sandbox cleanup remains available (self-hosted by default; enable use_github_hosted when the public dispatch workflow is not published yet).


Local discovery ​

bash
export STRIPE_SECRET_KEY=sk_test_...
export E2E_OWNER_EMAIL=lovely-e2e@gmx.co.uk
export PLATFORM_API_ORIGIN=https://platform-sandbox.lovely-home.co.uk

# Stripe discovery + hub-status table
node scripts/list-e2e-orphan-hubs.mjs

# Explicit ids
node scripts/list-e2e-orphan-hubs.mjs e2e-iiatslo6,e2e-r7r7w6ed

# JSON
node scripts/list-e2e-orphan-hubs.mjs --json --orphans-only

Manual fallback (single site) ​

If the site is still in platform/sites.yaml:

  1. Actions → Platform site billing deprovision → site_id, use_github_hosted if needed.

If the site is already off sites.yaml but infra remains:

  1. Actions → Platform site deprovision → site_id, terraform_stack: customers.

Emergency local scripts: scripts/billing-deprovision-hub-site.mjs or scripts/deprovision-hub-site.mjs — see ops-deprovision-stripe-test-hubs.md.

Stale cards in sandbox platform admin ​

If hubs are gone from main but cards remain (often billing only):

  1. Confirm the site is off origin/main: git fetch origin main && git show origin/main:platform/sites.yaml | rg '{site_id}' (no match = registry drop landed).
  2. Purge sandbox billing D1 (Wrangler logged in):
bash
git fetch origin main   # optional; purge allows local drift when absent on origin/main
export PLATFORM_BILLING_D1_NAME=lovely-home-platform-billing-sandbox
node scripts/purge-platform-site-for-reuse.mjs e2e-example

If the site is still on origin/main, run Platform site billing deprovision (or E2E sandbox cleanup with dry_run: false) — do not purge D1 first.

After teardown, reuse slugs with scripts/purge-platform-site-for-reuse.mjs when billing D1 rows should be cleared. e2e-* slugs use the sandbox billing database automatically when PLATFORM_BILLING_D1_NAME is unset.