Platform site provisioning (v4)
Paid customer hubs are created from a Cloudflare Queue after Stripe confirms payment. Git (platform/sites.yaml) is recorded after the hub is live. Operator wizard creates still open a registry PR.
Flow
flowchart TD
A[Stripe checkout.session.completed] --> B[HUB_PROVISION_QUEUE]
B --> C[hub-jobs Worker]
C --> D[platform-site-provision.yml]
D --> E[ensure yaml locally]
E --> F[terraform apply + Worker + Pages]
F --> G[HUB_REGISTRY_QUEUE]
G --> H[hub-jobs waits for git write]
H --> I[platform-site-registry.yml overlays onto main]- Signup webhook enqueues
{ siteId, action: provision }(no GitHub create-PR). - hub-jobs Worker (Terraform:
lovely-home-hub-jobs) dispatchesplatform-site-provision.ymlon the private repo (or a public orchestrator whenGITHUB_PROVISION_REPOis set — see hub-provision-public-actions.md) with concurrency 4. - Provision writes the yaml/wrangler stubs locally, applies Terraform, deploys Worker/Pages, then enqueues a registry record.
- Registry queue (concurrency 1) overlays that site onto
origin/mainand pushes. Yaml-push auto-provision ignores those ledger commits.- hub-jobs waits up to ~13¾ minutes for each registry (and provision) run, within the 15-minute queue consumer limit. It used to give up after 5 minutes. When the VPS runner queue was long, the queue then retried and dispatched the same job again while the first run was still queued (#1078).
- Provision and teardown never send the job again while it is still in progress (#1100). If the run is still queued or running at the wait limit, the message is marked done, because the run will finish by itself. A run cancelled by the workflow's one-run-per-site concurrency also counts as handled. Only a run that actually failed is retried. On 2026-10-22 a provision that outlived the wait was sent again; the duplicate ran after the hub's teardown and brought
e2e-s5tlk6m4back. The registry queue still retries at the wait limit, so the next site keeps waiting its turn. A retried registry message waits on the run it already started (#1531): it looks for a run with the same action and site created after the message was sent, and dispatches again only if that run failed or was cancelled. On 2026-11-09, drop runs queued behind a busy VPS runner took up to 26 minutes, and each retry dispatched another run: one drop ran 5 times and held the queue for about 2 hours. The snapshot guard (#1079) still skips any duplicate that gets through. - Provision guard (#1106). Before building, the public Hub provision dispatch reads the home's
site_billingrow (scripts/check-provision-still-wanted.mjs). It skips, with a notice, when billing is cancelled or when a teardown was dispatched after the last provision claim, so a late or duplicate provision can't rebuild a torn-down home. A real re-subscription clears the teardown marker when it claims its provision, so that still builds. If billing can't be read, it provisions as before and leaves a warning. - Each run's title is
{action} <site id> · {workflow}(run-name). hub-jobs uses it to find its own run when several are dispatched within seconds, and it tells you in the Actions list which site a run is for. Copies of a workflow withoutrun-namefall back to the earliest run created after the dispatch. - Runner (#1080): registry runs for customer sites use GitHub-hosted
ubuntu-latest, so signups never queue behind other work on the VPS (about 1 private-repo Actions minute per record or drop).e2e-*regression hubs stay on the VPS, which is free and mostly runs at night. To override, set the repo variableHUB_REGISTRY_RUNNER(Settings → Secrets and variables → Actions → Variables) tohosted(everything on GitHub) orself-hosted(everything on the VPS, e.g. if minutes run low). Unset orautogives the split above. - One snapshot branch per site, newest state wins (#1120). Provision pushes
platform/hub-registry-{site}saying "this site is in the registry"; teardown force-pushes "it is not" over the same branch, even whenmainnever had the site. Each registry run (scripts/check-hub-registry-snapshot.mjs) applies whatever the branch holds at that moment, takes record or drop from the snapshot itself (not from the queue message), and deletes the branch only if it still points at the commit it applied. A message that finds the branch gone has nothing to do: an earlier run already applied the newest state. So duplicates and out-of-order messages are harmless. On 2026-10-23 a record delayed past its teardown pute2e-1ny6okafback onmain, and the second drop then looped. That can't happen with one branch per site. If a provision or teardown can't push its snapshot, that step fails in its own run. - Database region (#1148). A new hub's D1 database is created in Western Europe (
d1_location_hint, defaultweur, interraform/modules/hub_environment). Without a hint, D1 placed each database near the caller. That was GitHub's US runners, so hubs created before this have their database in North America and every query crosses the Atlantic. Terraform ignores the hint on existing databases (ignore_changes), since a D1 database can't be moved and replacing it would empty it. Moving an existing hub: Move hub database below.
- Cancel enqueues
{ action: teardown }→ archive + destroy → registry drop. - Operator wizard still dispatches
platform-site-manage.yml→ PR → merge → yaml-push provision.
One-time setup
1. Remote Terraform state (R2)
Why: Terraform state today lives on your laptop (terraform/terraform.tfstate). GitHub Actions cannot read that file, so automated provisioning needs state stored in Cloudflare R2 (S3-compatible object storage).
What you are doing: create an R2 bucket, create an R2 API token, point Terraform at the bucket, then copy your existing local state into R2 once. After that, both your laptop and GitHub Actions use the same remote state.
Step A — Create the R2 bucket
- Open Cloudflare dashboard → R2 object storage.
- Click Create bucket.
- Name it something like
lovely-home-terraform-state(globally unique within your account). - Leave other settings as default → Create bucket.
Write down the bucket name — you need it twice (local backend.hcl and GitHub secret TF_STATE_R2_BUCKET).
Step B — Create an R2 API token (for Terraform only)
This token is separate from your main CLOUDFLARE_API_TOKEN. It only accesses the state bucket.
- R2 → Manage R2 API tokens (or Overview → Manage API tokens).
- Create API token.
- Name: e.g.
terraform-state. - Permissions: Object Read & Write. Not Admin: Terraform only reads and writes state objects, and admin rights would let a leaked token delete the bucket.
- Scope: only the state bucket you just created. Never "all buckets": the state holds every hub's secrets (proxy secrets, service token secrets), and the platform archive bucket must not be reachable with this token.
- Create API token.
Cloudflare shows two values once:
| Cloudflare label | Used as |
|---|---|
| Access Key ID | AWS_ACCESS_KEY_ID locally, GitHub secret TF_STATE_R2_ACCESS_KEY_ID |
| Secret Access Key | AWS_SECRET_ACCESS_KEY locally, GitHub secret TF_STATE_R2_SECRET_ACCESS_KEY |
Copy both somewhere safe now — the secret is not shown again.
Step C — Find your account ID (for the endpoint URL)
- Dashboard → any zone → Overview → copy Account ID (32-character hex),
or Workers & Pages → overview sidebar.
Your R2 endpoint is always:
https://<ACCOUNT_ID>.r2.cloudflarestorage.comExample: account 2c810bbed7e633623b99ae7c51dd0aa2 →https://2c810bbed7e633623b99ae7c51dd0aa2.r2.cloudflarestorage.com
That URL goes in backend.hcl and GitHub secret TF_STATE_R2_ENDPOINT.
Step D — Create local backend.hcl (not committed)
From the repo root:
cp terraform/environments/backend.hcl.example terraform/environments/backend.hclEdit terraform/environments/backend.hcl — replace only the two placeholders:
bucket = "lovely-home-terraform-state" # Step A bucket name
endpoints = {
s3 = "https://2c810bbed7e633623b99ae7c51dd0aa2.r2.cloudflarestorage.com" # Step C
}Leave key, region, and the skip_* lines as in the example.
This file is gitignored — do not commit it.
Step E — Export R2 credentials in your terminal
Terraform’s S3 backend reads R2 tokens via AWS-named env vars (that is normal for R2):
export AWS_ACCESS_KEY_ID="paste Access Key ID from Step B"
export AWS_SECRET_ACCESS_KEY="paste Secret Access Key from Step B"Optional sanity check (requires AWS CLI configured the same way, or skip):
aws s3 ls s3://lovely-home-terraform-state \
--endpoint-url "https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com"Step F — Point Terraform at R2 and migrate state
You still need your usual Cloudflare token for the provider (resources), separate from the R2 token above:
export CLOUDFLARE_API_TOKEN="your existing terraform API token"
cd terraformCommand 1 — configure the backend and migrate state:
terraform init -backend-config=environments/backend.hclWhen prompted “Do you want to copy existing state to the new backend?” → type yes.
That single command both configures R2 and uploads your local terraform.tfstate. You do not need a second terraform init -migrate-state unless you answered no the first time.
Command 2 — verify nothing changed:
terraform plan -var-file=environments/hub.tfvarsExpect No changes (or only minor drift you already know about). If the plan wants to recreate production resources, stop and ask for help before applying.
Step G — Add matching GitHub secrets
In GitHub → repo → Settings → Secrets and variables → Actions:
| GitHub secret | Value |
|---|---|
TF_STATE_R2_BUCKET | Bucket name from Step A |
TF_STATE_R2_ENDPOINT | https://{ACCOUNT_ID}.r2.cloudflarestorage.com |
TF_STATE_R2_ACCESS_KEY_ID | Access Key ID from Step B |
TF_STATE_R2_SECRET_ACCESS_KEY | Secret Access Key from Step B |
CI uses these instead of backend.hcl (see platform-site-provision-reusable.yml).
The same four secrets are set on the public orchestrator repo (lovely-hub-provision), whose workflows also run Terraform.
Rotating the state token (#1171)
Rotate it once a year, when someone who had it leaves, or if it may have leaked (for example a log or artifact that printed the environment).
- Check no provision, teardown or Platform admin Terraform run is in progress, and none is queued.
- R2 → Manage R2 API tokens → Create API token with the same settings as Step B (Object Read & Write, state bucket only).
- Update
TF_STATE_R2_ACCESS_KEY_IDandTF_STATE_R2_SECRET_ACCESS_KEYon both repos (home-dashboardandlovely-hub-provision), and your localAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. - Run a read-only check:
terraform init -reconfigure -backend-config=environments/backend.hclthenterraform planfor one stack. Expect it to read state with no access error. - Delete the old token in R2. Until you do, both work.
A leaked state token exposes the secrets stored in state, so after a leak also rotate those secrets (hub proxy secrets, the platform health service token, PLATFORM_SITE_ARCHIVE_SECRET).
Hub job queues and the hub-jobs Worker live in module.platform_admin. After those Terraform files land on main, run Actions → Platform admin Terraform and type apply-platform-admin. That job inits platform.tfstate and applies -target=module.platform_admin[0] and -target=module.marketing_site[0] so production/demo/sandbox hubs are not touched. It keeps the pre-launch marketing Access gate on.
After migration — day-to-day use
Local applies use the same backend automatically once terraform init has been run:
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export CLOUDFLARE_API_TOKEN="..."
cd terraform
terraform plan -var-file=environments/hub.tfvars
terraform apply -var-file=environments/hub.tfvarsYou no longer maintain a separate “CI only” state file — laptop and GitHub Actions share R2.
Troubleshooting
| Problem | Fix |
|---|---|
No valid credential sources on init | Export AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY before terraform init |
Access Denied on init | R2 token needs Object Read & Write on the state bucket |
| Plan wants to recreate everything | State file not migrated — re-run terraform init -migrate-state and confirm yes |
backend.hcl not found | Run cp from Step D; file lives at terraform/environments/backend.hcl |
R2 Class A bill spiked on lovely-home-terraform-state | Terraform’s S3 backend lists env:/ on every init/plan by default. Use workspace_key_prefix = "workspaces" in backend.hcl (see backend.hcl.example) and re-run terraform init -reconfigure. In R2, delete stray env:/ objects if any were created by old workspace experiments. CI picks up the prefix via scripts/terraform-init-r2.sh. |
2. GitHub repository secrets (Cloudflare + app config)
| Secret | Purpose |
|---|---|
CLOUDFLARE_API_TOKEN | Terraform + Wrangler (D1, R2, Pages, Workers, Access, DNS, Workers Scripts, Queues Edit for hub job enqueue) |
CLOUDFLARE_ACCOUNT_ID | Account ID |
CLOUDFLARE_ZONE_ID | Zone ID for lovely-home.co.uk |
WORKERS_SUBDOMAIN | e.g. mark-lovely67 |
ACCESS_TEAM_DOMAIN | Zero Trust team slug, e.g. lovely-home |
TF_STATE_R2_ACCESS_KEY_ID | R2 API token access key |
TF_STATE_R2_SECRET_ACCESS_KEY | R2 API token secret |
TF_STATE_R2_BUCKET | R2 bucket name |
TF_STATE_R2_ENDPOINT | https://{account_id}.r2.cloudflarestorage.com |
Optional (strongly recommended — required for wizard PRs unless you enable Actions PR creation in repo settings):
| Secret | Purpose |
|---|---|
OWNER_EMAILS | Platform hubs only (*.lovely-home.co.uk). Never applied to customer hubs |
SUPPORT_OWNER_EMAILS | Owners of hubs on the customer zone (*.lovely-hub.com) that have no owner in the billing database, which are hubs you run yourself. Never added to a hub that has a billing owner |
CF_ACCESS_MANAGEMENT_TOKEN | Required. Cloudflare token with only Access: Apps and Policies: Edit on the account. Each hub Worker gets it to keep its House sitters, Family and Invited owners sign-in lists. Never the same value as CLOUDFLARE_API_TOKEN: provisioning stops if it is missing or equal (#1566). Set it in both repos (home-dashboard and lovely-hub-provision). |
PLATFORM_GITHUB_TOKEN | Same PAT as platform admin Pages env — must also be a GitHub Actions repo secret so site-manage can open PRs, provision can open the post-provision follow-up PR, and deprovision can prune HUB_PROXY_SECRETS_JSON (needs repo secrets write). For the Stripe Test/Live switch, also needs Actions variables write |
PLATFORM_CF_API_TOKEN | Same token as platform admin Pages env (optional — usage, preview toggles, marketing OTP guests). Needs Access: Apps and Policies Edit for the guest list |
STRIPE_SECRET_KEY | Test-mode key — same as stripe_secret_key in hub.tfvars — required for billing or CI terraform apply strips Stripe vars from platform Pages |
STRIPE_WEBHOOK_SECRET | Test webhook signing secret — same as stripe_webhook_secret |
STRIPE_PRICE_ID | Test monthly price — same as stripe_price_id |
STRIPE_PRICE_ID_YEARLY | Test yearly price — same as stripe_price_id_yearly |
STRIPE_SECRET_KEY_LIVE | Optional live key — same as stripe_secret_key_live. Store before public launch; mode is switched from platform admin |
STRIPE_WEBHOOK_SECRET_LIVE | Optional live webhook signing secret |
STRIPE_PRICE_ID_LIVE / STRIPE_PRICE_ID_YEARLY_LIVE | Optional live Price ids |
Repo variable (not a secret): STRIPE_MODE (test | live). Platform admin updates it on go-live. Provision workflows always pass both Stripe key sets into Terraform; they do not swap secrets based on this variable. | TURNSTILE_SITE_KEY / TURNSTILE_SECRET_KEY | Bot check on public signup. Both must be set, or CI terraform apply leaves the check off | | RESEND_API_KEY | Same as resend_api_key in hub.tfvars — customer lifecycle emails. Without it, billing still works but owners get no hub-URL mail | | CUSTOMER_EMAIL_FROM | Optional. Same as customer_email_from. Default is Lovely Home <support@lovely-home.co.uk> | | SLACK_WORKFLOWS_WEBHOOK_URL | Optional. Same as slack_workflows_webhook_url and the GitHub secret the workflow alerts use. The platform posts one #workflows message per sign-up whose hub build gives up (#1513). Unset = no Slack, the emails still go. | | PLATFORM_SIGNUP_NOTIFY_EMAIL | Optional fallback. Same as platform_signup_notify_email. Comma-separated inbox for one Resend email per public sign-up (Free or Plus checkout started). Requires RESEND_API_KEY. Preferred: set Platform admin → Sign-up limits → Per sign-up notify inbox (stored in D1 platform_settings.signup_notify_email). When that D1 value is non-empty it overrides this env var. Suggested: signups@lovely-home.co.uk | | PLATFORM_SITE_ARCHIVE_SECRET | Platform master. Each hub Worker gets only its own derived secret (#1166). CI uses it to pull a full-site JSON backup before destroy | | PLATFORM_ARCHIVE_R2_BUCKET | Platform archive bucket (lovely-home-hub-archives). Optional; the archive script defaults to that name | | PLATFORM_HEALTH_CF_ACCESS_CLIENT_ID | Platform health Access service token: the one hubs admit; Platform admin Terraform puts it on platform Pages (#1223) — required to fetch the hub through Access | | PLATFORM_HEALTH_CF_ACCESS_CLIENT_SECRET | Service token secret. Set by hand; never copied from Terraform state (#1223) | | HUB_PROXY_SECRETS_JSON | {"production":"...","test":"..."} — only needed if Terraform output is unavailable; CI normally reads secrets from remote state |
Provision and deprovision workflows set terraform_wrapper: false on hashicorp/setup-terraform. The wrapper wraps sensitive outputs as { sensitive: true, value } (or redacts them), which made generate-hub-tfvars think production had no hub_proxy_secret and abort a new customer hub. The generator also unwraps that JSON shape if a wrapper is present.
Production was imported with a locally pinned secret that is not in HUB_PROXY_SECRETS_JSON (that map currently has CI-created sites such as demo, sandbox, smith). A targeted apply (-target=module.hub_site["kitchen-home"]) does not touch production, so generate-hub-tfvars no longer requires production’s secret when provisioning a different site. It still throws if the site being applied is in state and the secret cannot be read. CI also merges terraform state pull (random_password + Pages HUB_PROXY_SECRET) into the readable map.
CI writes secrets to a separate sensitive var-file (hub.generated.secrets.tfvars.json, gitignored) so hub_proxy_secret values are not embedded in the main generated tfvars file or CI logs.
Provisioning runs one platform hub at a time (max-parallel: 1 + platform-terraform-state-platform) because those hubs share platform.tfstate. Customer hubs (*.lovely-hub.com) each have home-dashboard/customers/{siteId}.tfstate, so two household signups can Terraform-apply in parallel. R2 still has no DynamoDB lock, so two applies on the same state file must not overlap — CI uses platform-terraform-state-customers-{siteId} for that.
Hub names stay unique: signup checks the live manifest, active billing, and an atomic D1 reservation (signup_slug_reservations) before Stripe Checkout. A second person cannot take a name that is already live or held.
Registry PRs (platform-site-manage) lock per hub, not globally. GitHub concurrency is not a queue: one shared group keeps a single pending job and cancels the rest, which is how four overlapping signups only opened two PRs. Different households run in parallel; overlay onto origin/main plus a reconcile when another platform: PR merges (and a one-minute cron) replay leftover conflicting or behind PRs. Squash auto-merge will not land a branch that is only behind main, so overlay must run in that case too — not only on merge conflicts. Terraform apply retries Cloudflare 429s; a 403 account limit (for example D1 databases per account on the Workers Free plan) fails immediately instead of waiting through four retries. A Durable Object or Cloudflare Queue would be a real FIFO if we ever move the registry off git; while sites.yaml lives in this repo, GitHub Actions is the right writer and a bad waiter.
Split Terraform state (platform vs customers)
CI no longer uses the combined home-dashboard/hub.tfstate key. After that change landed, the one-time split (bash scripts/migrate-terraform-state-stacks.sh split-stacks, done September 2026) copied it into:
| R2 key | Contents |
|---|---|
home-dashboard/platform.tfstate | Platform admin, marketing Access, hubs on lovely-home.co.uk |
home-dashboard/customers.tfstate | Household hubs on lovely-hub.com (combined file — backup after the next split) |
home-dashboard/hub.tfstate | Left in place as a backup — do not apply against it |
Split customer Terraform state (per site)
The one-time per-site split (bash scripts/migrate-terraform-state-customer-sites.sh split-sites, done September 2026) copied each module.hub_site["id"] out of customers.tfstate into:
| R2 key | Contents |
|---|---|
home-dashboard/customers/{siteId}.tfstate | One household hub |
home-dashboard/customers.tfstate | Left in place as a backup — do not apply against it |
Provision/deprovision fail closed if a platform backend is still empty while yaml expects sites, or a per-site customers backend still looks like the combined estate.
Local backend.hcl must use platform.tfstate or customers/{siteId}.tfstate and you must pass -var=terraform_stack=platform (or customers) on every plan/apply. Customer applies also need -var=provision_site_id={siteId}.
3. Cloudflare API token permissions
The CI token needs everything in platform-terraform.md plus:
- Account → Workers Scripts → Edit (deploy Worker +
wrangler secret put) - Zone → Workers Routes → Edit (wrangler deploy enables
*.workers.devafter upload)
Without Workers Scripts Edit, set-worker-secrets-from-terraform.mjs fails in CI. Without Workers Routes Edit, wrangler deploy fails on /workers/subdomain even when the script upload succeeds.
Provision fails with 401 Unauthorized (Access, D1, R2 create): The GitHub secret CLOUDFLARE_API_TOKEN is invalid, expired, or does not include account CLOUDFLARE_ACCOUNT_ID. Re-create the token with the permissions above, update the repo secret, and run locally or in CI:
export CLOUDFLARE_API_TOKEN="..."
export CLOUDFLARE_ACCOUNT_ID="..."
export CLOUDFLARE_ZONE_ID="..."
bash scripts/verify-cloudflare-api-token.shDeploy fails with Authentication error [code: 10000] on /workers/subdomain: Add Workers Routes → Edit on zone lovely-home.co.uk (included in Cloudflare’s “Edit Cloudflare Workers” template). Re-run verify — it checks the subdomain endpoint before terraform apply.
Pages deploy fails with Unknown arguments: functions-directory: Wrangler 4.128 dropped that flag. deploy-cloudflare-pages-site.sh now stages pruned Functions as dist-hub-functions/functions and runs wrangler from that directory. Do not add the old flag back.
Platform admin says billing is active but infrastructure is not provisioned, while health checks are green: The card reads platform-manifest.json served by the platform Pages deployment. Git can already have the Terraform contract (attach_hub_api_binding: true + D1 id) while the live dashboard is still on an older Git-connected build. Rebuild/redeploy platform admin from current main — do not click Provision again. The amber “Terraform contract” row is that stale file; Worker/Pages/HUB_API rows come from live Cloudflare.
Post-worker apply fails with Invalid Service name () (8000022): CI runs scripts/attach-hub-api-pages-binding.mjs (direct Cloudflare API) and then deploys Pages, so the binding is active from the first deployment (#1500). Local: bash scripts/deploy-cloudflare-pages-site.sh {site_id} after Worker deploy.
Platform admin shows HUB_API binding no after provision: The binding is project-level; the active Pages deployment must be created after attach. deploy-cloudflare-pages-site.sh attaches, then deploys. Re-run that script or Platform site deploy for the site.
Sandbox (or another hub) loses HUB_API after a merge to main: Hub Pages projects keep a GitHub source for previews but production git deploys are off. Provision wrangler-deploys the hub; app-code rollouts use CD — Hub Pages (cd-hub-pages.yml). Registry-only merges must not rebuild every household (that queue starved platform Pages and made signup registered lag). Existing projects are patched by scripts/disable-hub-pages-git-production.mjs from Ensure hub HUB_API bindings. If a binding is still missing: node scripts/ensure-hub-api-bindings.mjs sandbox then bash scripts/deploy-cloudflare-pages-site.sh sandbox.
Terraform apply fails with Access 400 Bad Request / domain does not belong to zone on a new site: The Pages Access app includes *.pages.dev destinations, which Cloudflare rejects until the Pages project exists. CI now omits those destinations during the pre-worker apply and adds them on the post-worker apply. Merge the fix and re-run Platform site provision (workflow dispatch with the site id). Partial applies resume safely on retry.
Build tracking and feedback (#1505–#1511)
A Free sign-up is live about 3 minutes after the button (run 105: 02:02:50 → 02:05:45); the page, the steps and the welcome email say so. What happens when a build does not go to plan:
flowchart LR
Q[HUB_PROVISION_QUEUE] --> W[hub-jobs Worker]
W -->|run failed| F[POST /api/public/hub-build-run<br>attempt_failed n of 6]
W -->|run green| S[POST hub-build-run succeeded]
W -->|every 15 min| V[POST hub-build-run sweep]
F --> D[(site_billing + hub_build_attempts)]
S --> D
V --> D
P[/signup-success polls hub-status/] --> D
A[Platform admin Needs attention] --> D
B[/signup banner via signup/status/] --> D- Every provision attempt is recorded (#1505). The Worker waits for the GitHub run; when it fails it posts
attempt_failedwith the queue'smessage.attempts(1–6: the first try plusmax_retries = 5interraform/modules/platform_admin/queues.tf) and the error, then lets the queue retry as before. A green run postssucceeded.functions/api/platform/platformHubBuild.jswritesprovision_attempts,provision_last_error,provision_failed_at(set only by the last attempt) onsite_billingand one row per attempt inhub_build_attempts(migration0032_hub_build_status.sql). The report is best effort: a platform that does not answer never fails the job. The workflow's own "Record setup failure" step still writesprovision_last_errorfor runs the Worker did not wait for. - hub-status says
retrying, notfailed, while attempts remain.stateisretryingwhen an error is stored, no final failure is, the attempt count is below the maximum and the build was queued less than an hour ago. It isfailedonce the last attempt is reported, or an hour after a failure nothing followed. The payload also carriesattempt,maxAttempts,dispatchedAt(when the build was queued:provision_dispatched_at, orregistry_dispatched_atfor a new sign-up, which the registry path queues),failedAt,liveAt,typicalMinutes(3) andownerEmailMasked(m•••@gmail.com, live billing only) for the success page's steps and sign-in note. Underscore site ids still fail at once. - "Your hub is live" email (#1506). The first hub-status poll that sees the hub answer (or the sweep, if the owner closed the page) sets
provision_live_atand sends one email with the hub link, the setup checklist and how the sign-in code arrives.hub_live_email_sent_atis the claim, so two pollers send one mail. - Build gave up (#1508). The final attempt sends the owner "We could not build your hub today" (we have been alerted, finished by hand, email follows, nothing charged;
build_failed_email_sent_at) and the sign-up notify inbox (PLATFORM_SIGNUP_NOTIFY_EMAILor Platform admin → Sign-up limits) "Hub build stopped: {site}" with the last error. The success page shows the same words. The same moment posts one Slack message to #workflows, "Hub build stopped for {host} (Production|Sandbox) after n attempts", with the last error and what to do (SLACK_WORKFLOWS_WEBHOOK_URLon platform Pages, #1513). It rides on the samebuild_failed_email_sent_atclaim, so the sweep's re-queues never repeat it. Failed provision runs of the public orchestrator no longer post to Slack themselves; failed teardowns still do. - Sweep (#1509), every 15 minutes with the clean-email cron. Homes with live billing whose build gave up are queued again, at most once an hour (
provision_retry_at) and at most three times per build cycle (provision_retries; worst case 4 × 6 = 24 Actions runs), until they answer. After the third re-queue the home stays failed in Platform admin until someone presses Retry build. A home whose failed attempt nobody followed up for an hour is settled as given up, so its emails go. A home that was queued and now answers is marked live. The 2026-11-08 GitHub outage would have cost those sign-ups about two hours instead of a support email. - Platform admin "Needs attention" (#1510).
GET /api/platform/sitesreturnshubBuilds(homes that gave up, homes waiting more than 10 minutes, the last attempts). The strip at the top lists them with a Retry build button (the card's Provision action). - Slow-build banner on /signup (#1511).
GET /api/public/signup/statusreturnshubBuilds.slowwhen the last three attempts in the past day all failed, or a live home has waited more than 10 minutes since its build was queued without going live. The form shows an amber banner in place of the demo one ("Hub building is slower than usual right now. You can still sign up…") and clears it at the next successful build. Platform admin → Maintenance stays the manual switch. - Address check while typing (#1507).
/signupasksGET /api/public/signup/slug/{id}once the visitor pauses for 600 ms (one request per distinct name, within the 40-an-hour limit) and shows a tick, or a cross with two suggestions.
Apply migration 0032 (Platform operators workflow, production and sandbox) before the deploy that carries this.
Move hub database (#1148)
Hubs created before #1150 have their D1 database in North America, so every query from a Worker in Europe crosses the Atlantic (about 130 ms). Actions → Move hub database moves one hub's database to the module's region (Western Europe). Inputs: site_id, plus confirm with the same id typed again.
It's the normal provision with --move-d1:
- Export.
sync-wrangler-from-terraformfirst points the Worker config at the database Terraform state holds, so the export reads the live database even ifmain'swrangler.tomlis behind.scripts/hub-d1-move.mjs exportthen takes a SQL export of the old database and every table's row count, counting each table with its own subquery (D1 caps compoundSELECTs, so noUNION). It stops if the database has no tables. A copy goes to the platform archive bucket:lovely-home-hub-archives/{site}/d1-move/{time}.sql. - Terraform.
apply -replace=module.hub_site["{site}"].cloudflare_d1_database.manualsdeletes the old database and creates an empty one with the same name in the new region. Time Travel for the old database goes with it; the R2 copy is the backup. - Import and check.
sync-wrangler-from-terraformpoints the Worker config at the new id.hub-d1-move.mjs importloads the export, refusing a database that already has tables.verifyfails the run if any table's row count differs. - Back up. The final
terraform apply -refresh-onlyruns without-replace(Terraform refuses the two together). The hub Worker is redeployed (migrations are already in the export) and the registry records the new database id onmain. Sitter emails are not re-seeded fromsites.yaml; the hub keeps its own.
The registry step replaces the hub's [env.{site}] block in worker/wrangler.toml with the one the run wrote. It only matches [env.{site}] at the start of a line, so a comment that mentions the env is never taken for the block; before #1186 one was, the new id never reached main, and the next Deploy on merge failed migrating lovely's deleted database. worker/tests/wranglerConfig.test.js checks every hub env's D1 id against the platform manifest, so the two can't drift apart again unnoticed.
The hub is down for a minute or two, between the delete and the redeploy. Anything written to the hub after the export is lost, so run it when the hub is quiet. Don't merge hub changes while a move runs: Deploy on merge would redeploy the Worker with the old database id until the registry records the new one.
If a run fails after Terraform: the export is in R2. Re-running the move would export the new (empty) database and stop ("no tables"). Instead, download the R2 copy to .hub-d1-move/{site}.sql and run node scripts/hub-d1-move.mjs import {site}, then verify (needs .hub-d1-move/{site}.counts.json, which is in the run's log), then redeploy the Worker.
Who gets owner access (and why customer emails stay out of git)
platform/sites.yaml and platform-admin/public/platform-manifest.json are both public. Customer addresses must never appear in either, so owner access to a customer hub is assembled at provision time instead:
| Source | Applies to |
|---|---|
Billing database (site_billing.owner_email, read by scripts/fetch-site-owner-emails.mjs into SITE_OWNER_EMAILS_JSON) | The household that signed up |
SUPPORT_OWNER_EMAILS secret | Only a hub on the customer zone with no billing owner: one the operator runs for themselves. Never added to a customer's hub |
OWNER_EMAILS secret | Platform hubs (*.lovely-home.co.uk) only |
generate-hub-tfvars.mjs fails the run if a customer hub would end up with no owner addresses from any source, rather than silently applying a hub nobody can sign in to. fetch-site-owner-emails.mjs fails the run if the billing database can't be read: a failed lookup is not "no owner", and carrying on would leave the customer off their own hub.
The operator has no standing way into a customer's hub. To look at one, the customer invites the address under Settings → People → Owners and removes it afterwards. build-platform-manifest.mjs fails the same way if an email survives into the manifest.
What runs in CI
scripts/provision-hub-site.mjs:
generate-hub-tfvars.mjs— builds tfvars fromplatform/sites.yaml+ billing owner emails + secrets (never committed)terraform apply(attach_hub_api_binding=false for new site)sync-wrangler-from-terraform.mjsset-worker-secrets-from-terraform.mjs— HUB_PROXY_SECRET, Access AUD, vanilla dummy secrets, optionalGOOGLE_PLACES_API_KEY(setup wizard Find address: house number and street; Google does not list the houses in a postcode, #1493), optionalOS_PLACES_API_KEY(UK UPRN / bin auto-import)npm run d1:migrate:{site}+npm run deploy:{site}terraform apply -refresh-only(post-worker tfvars)deploy-cloudflare-pages-site.sh— attach HUB_API, then deploy Pages once, so the first live deployment already reaches the Worker. Before #1500 it deployed first and redeployed after the attach: for a minute or two the new hub served its page with every/api/*call failing, the platform hub-status said "ready", and signup (and the nightly regression) opened a hub showing "Hub setup unavailable". The attach checks the binding in the PATCH response; a GET straight after can still return the old projectenable-hub-pages-previews.mjsfor real hubs, ordisable-hub-pages-previews.mjsfor lifecyclee2e-*/pages_preview: falseinplatform/sites.yamlbuild-platform-manifest.mjs+deploy-platform-admin.sh
Local dry-run
After R2 backend is configured, export CLOUDFLARE_API_TOKEN (not stored in hub.tfvars). Other provision inputs are read from terraform/environments/hub.tfvars automatically when env vars are unset:
export CLOUDFLARE_API_TOKEN="..."
export AWS_ACCESS_KEY_ID="..." # R2
export AWS_SECRET_ACCESS_KEY="..."
cd terraform && terraform init -backend-config=environments/backend.hcl
cd ..
node scripts/provision-hub-site.mjs demoTo override tfvars values, export the usual env vars (WORKERS_SUBDOMAIN, CLOUDFLARE_ACCOUNT_ID, etc.) before running provision. See platform-terraform.md for the full list used by CI.
Workflows
| Workflow | Trigger |
|---|---|
platform-site-manage.yml | Wizard → PR |
platform-site-provision.yml | Push to main (sites.yaml) or manual / Provision button |
platform-site-deprovision.yml | Push to main (sites.yaml removal) or manual |
platform-site-deploy.yml | Worker-only redeploy |
platform-sync-google-places-key.yml | Manual — push GOOGLE_PLACES_API_KEY to all hub Workers (or one site) |
platform-sync-archive-secret.yml | Manual — push each hub Worker its own archive secret |
platform-terraform-plan.yml | Manual — plan only, never applies (see below) |
platform-terraform-apply.yml | Manual — applies a reviewed plan run's hub plans, Terraform only (see below) |
platform-site-billing-deprovision.yml | Stripe cancel/delete webhook or manual — archive, registry removal PR (auto-merge) → platform-site-deprovision.yml |
Plan before you apply (#1211)
Platform Terraform plan shows what Platform site provision and Platform admin Terraform would change, without applying. Run it before any apply that isn't a routine provision, for example after rotating a secret or changing a module.
- Inputs:
sites(all, or ids likelovely,demo) andinclude_platform_admin(on by default). - Each hub is planned with the same tfvars (
PROVISION_PHASE=pre-worker), owner emails, stack vars and-target=module.hub_site["{site}"]as its provision apply. Platform admin uses Platform admin Terraform's three targets. - Plans run with
-lock=falseand never apply, import or write state, so they can run beside a provision. - Each job's summary lists the planned creates, updates, replaces and destroys. The full plan is in the job log under "Full plan". A destroy or replace raises a warning.
- Platform admin caveat: the apply first imports the marketing, ops handbook and Grafana Access resources if they already exist in Cloudflare. The plan skips those imports, so a resource that exists in Cloudflare but not yet in state shows as a create. That is not a real create.
Apply a reviewed plan to the hubs (#1215)
For a Terraform-only change (an Access policy, an env var), Platform Terraform apply reaches every hub in minutes. Platform site provision is slower because it also deploys the Worker and commits to the registry for each hub.
- Run Platform Terraform plan and read each hub's summary.
- Run Platform Terraform apply with that run's id as
plan_run_idandapply-hubsinconfirm.
How it behaves:
- It checks out the commit the plan was made at and prepares each hub exactly as the plan did (shared action
.github/actions/terraform-hub-prepare). It plans again and applies only when the changes match the reviewed plan exactly. Otherwise it stops and asks for a new plan. - Hubs with no changes are skipped. Customer hubs have their own state, so they apply side by side (on GitHub-hosted runners by default).
- It refuses a run where any hub plan destroys or replaces a resource. Use Platform site provision for those.
- Platform admin is never applied here. Use Platform admin Terraform, which imports existing resources first.
- No Worker deploy, Pages deploy or registry commit happens, so Worker secrets and code stay as they are.
The plan run keeps only resource addresses and actions as an artifact (terraform-plan-changes-{target}, 7 days), never values.
v5 — automated deprovision
Fully automated hub teardown after the delete PR merges to main.
Flow
flowchart TD
A[Wizard: Delete via PR] --> B[Merge to main]
B --> C[platform-site-deprovision.yml]
C --> A1[archive-hub-site-backup.mjs]
A1 --> D[terraform destroy module.hub_site]
D --> E[DELETE Worker script via API]
E --> F[Refresh platform manifest + redeploy admin]- Wizard dispatches
platform-site-manage.yml(delete) → opens PR removing the site from registry and Wrangler stubs. - Merge PR to
main. - Auto-trigger: push to
mainchangingplatform/sites.yamlstartsplatform-site-deprovision.ymlfor each removed site (terraform: true, not protected). CI exports a full site backup from the hub Worker (worker_api_origin) to platform R2 before Terraform destroy — see platform-site-archive.md. Billing deprovision must complete that archive before it opens the registry-removal PR. Customer-facing close hub flows rely on that JSON for about 12 months so the same household can reinstate the name from Billing or My Homes (platform-billing.md). - Manual retry: workflow dispatch with
site_id(site must already be absent fromplatform/sites.yaml). Enable use_github_hosted to run Terraform teardown onubuntu-latest(same pattern as manual provision and billing deprovision). If Terraform already destroyed the hub, archive and destroy are skipped so a failed git follow-up can still drop the contract from the platform manifest.
Hub registry record/drop commits (platform: record|drop … in site registry) still touch platform/sites.yaml but do not start a second Terraform run: push detect is skipped and CI ignores registry-overlay-only pushes — see ci-runner-strategy.md.
What gets removed
| Resource | How |
|---|---|
| D1, R2×2, Pages, Access×2, DNS | terraform destroy -target=module.hub_site["{id}"] |
| Worker script + secrets | Cloudflare API DELETE /workers/scripts/{name} (after Terraform) |
| Platform admin site card | build-platform-manifest.mjs + deploy-platform-admin.sh |
Local hub.tfvars (if present) | prune-local-hub-tfvars.mjs during deprovision |
HUB_PROXY_SECRETS_JSON GitHub secret | prune-hub-proxy-secrets-github-secret.mjs in CI (requires PLATFORM_GITHUB_TOKEN with repo secrets write) |
Already handled by the delete PR: hub.tfvars.example is removed when the wizard opens the delete PR (step 1), before deprovision runs.
Per-site customer state: Per-hub Terraform backends may lack root sites / hub_proxy_secrets outputs until a full apply refreshes them. Deprovision scripts fall back to terraform state list, platform manifest, and HUB_PROXY_SECRETS_JSON when generating tfvars or reading site contracts. Archive uses the same manifest fallback for the live worker URL (split state does not block archive). If the worker returns 5xx, archive retries, tries the Pages hostname, then reuses a recent backup in platform R2. Before terraform destroy, empty-hub-site-r2-buckets.mjs deletes all objects from the hub guides/media buckets (Cloudflare returns 409 if a bucket is not empty; missing buckets are skipped so a partial destroy retry can continue), then prune-hub-pages-deployments.mjs deletes all Pages deployments (Cloudflare returns 400 / 8000076 if the project has too many deployments to delete).
Manual deprovision (local)
Same secrets as provision (see Manual provision). Site must not appear in platform/sites.yaml.
node scripts/deprovision-hub-site.mjs demoUse --skip-platform-admin to skip manifest rebuild and platform admin redeploy.
Recreating a site (e.g. demo)
Use this checklist when tearing down a site and provisioning it again with the same site id.
- Delete — Platform admin wizard → Delete → merge the delete PR to
main. - Wait for deprovision — In GitHub Actions, confirm Platform site deprovision is green for that site. Do not start a create for the same id while deprovision is still running (Terraform state race).
- Confirm manifest — After deprovision, the site card should disappear from platform admin. If create fails with “already exists in the platform manifest”, deprovision has not finished or platform admin was not redeployed.
- Create — Wizard → Create with owner emails (sitter emails optional) → merge create PR.
- Wait for provision — Platform site provision runs automatically; merge any follow-up PR that sets
attach_hub_api_binding: trueand syncs D1 ids (same as the first demo provision). - Smoke test — Open the hub hostname, complete setup wizard (including bin dates if you want home-screen reminders), and verify sitter Access login.
Listing URL import (Lovely Home+)
- Plan:
POST /api/setup/listing-importis gated to Lovely Home+ and Multi Home (listingImportonhub-plan-statusfeatures). Free hubs skip the fetch step in setup and enter details manually. - Worker cost: Each fetch runs from the household Worker (
subrequestto the public listing URL). Expect small per-import egress; degraded/partial imports still consume a fetch attempt. Monitor Workers request volume if imports spike during onboarding campaigns. - Providers: Airbnb, Booking.com, TrustedHousesitters, QuietStay, and generic HTTPS pages. Booking.com (and occasionally others) may block automated fetches (WAF) — the API returns a partial draft with a warning rather than failing setup.
Related
- platform-admin.md — operator UI
- platform-terraform.md — Terraform resource details