FAS Rollback Procedures

Suite-specific rollback steps for the Hello World Co-Op platform. This guide covers rollback for individual suites, shared packages, DNS, and emergency monolith fallback.

For the monolith emergency rollback procedure, see the FAS-8.1 rollback procedure document (in bmad-artifacts/implementation-artifacts/).

When to Rollback
Suite-Specific Rollback Steps
Package Rollback
DNS Rollback
Verification Checklist
Rollback Commands Reference
Related Documentation

When to Rollback

Criteria for Triggering a Rollback

Condition	Action
Error rate > 5% sustained for 10+ minutes	Rollback the affected suite
Uptime < 95% for the suite	Rollback the affected suite
Critical auth failure (SSO broken)	Rollback oracle-bridge or affected suites
Data loss or corruption	Rollback immediately + investigate
Production deployment broke a working feature	Rollback to last known good deployment

Do NOT Rollback For

Cosmetic/non-blocking UI issues (fix forward instead)
Intermittent errors below 5% threshold
Issues affecting only local development
Known issues with documented workarounds

Suite-Specific Rollback Steps

General Pattern (All Suites)

All suite rollbacks follow the same general process:

bash

# 1. Identify the last known good commit
git log --oneline -n 10

# 2. Revert the problematic commit
git revert <bad-commit-hash>
git push origin main

# 3. CI/CD will automatically redeploy
# Monitor: https://github.com/Hello-World-Co-Op/<suite-name>/actions

# 4. Verify the rollback
curl -I https://<suite-domain>

Estimated time: < 3 minutes (CI build + deploy).

think-tank-suite

Canister: wnfjk-biaaa-aaaao-a6dhq-caiDomain: staging-think-tank.helloworlddao.com

bash

cd ~/git/think-tank-suite
git log --oneline -n 5
git revert <commit-hash>
git push origin main

Special considerations:

Largest suite (4,642 tests) -- CI takes longest
If auth is broken, check oracle-bridge first (think-tank is the login hub)
If manualChunks causes JS crash, remove it and let Rollup auto-split (see FAS-8.1 fix)

Verification: Visit https://staging-think-tank.helloworlddao.com, verify login page renders.

governance-suite

Canister: wkep6-mqaaa-aaaao-a6dha-caiDomain: staging-governance.helloworlddao.com

bash

cd ~/git/governance-suite
git log --oneline -n 5
git revert <commit-hash>
git push origin main

Special considerations:

Uses direct nanostores (no AuthProviderBridge) -- auth issues are simpler
69 E2E tests may catch regressions before deployment

Verification: Visit https://staging-governance.helloworlddao.com, verify proposals page loads.

marketing-suite

Canister: d5fe6-hqaaa-aaaao-a6t5q-caiDomain: staging.helloworlddao.com (staging), www.helloworlddao.com (production)

bash

cd ~/git/marketing-suite
git log --oneline -n 5
git revert <commit-hash>
git push origin main

Special considerations:

Standalone suite -- rollback does not affect other suites
After rollback, verify SEO pre-rendering still works: check dist/index.html has content in <div id="root">
Verify dist/sitemap.xml is generated correctly
robots.txt must still be present

Verification:

bash

# Check pre-rendered HTML
curl -s https://staging.helloworlddao.com | grep -c '<div id="root">'
# Expected: 1 (with content inside)

# Check sitemap
curl -s https://staging.helloworlddao.com/sitemap.xml

otter-camp-suite

Canister: dzt3i-sqaaa-aaaao-a6uaa-caiDomain: staging-ottercamp.helloworlddao.com

bash

cd ~/git/otter-camp-suite
git log --oneline -n 5
git revert <commit-hash>
git push origin main

Special considerations:

Standalone suite -- rollback does not affect other suites
After rollback, verify Phaser.js chunk loads correctly (lazy-loaded ~340KB)
Game should gracefully degrade if WebSocket (oracle-bridge) is unavailable
Check that game assets in public/assets/ are intact

Verification: Visit https://staging-ottercamp.helloworlddao.com/otter-camp, verify game canvas renders.

dao-suite

Canister: d6s54-7iaaa-aaaao-a6uaq-caiDomain: staging-portal.helloworlddao.com

bash

cd ~/git/dao-suite
git log --oneline -n 5
git revert <commit-hash>
git push origin main

Special considerations:

After rollback, verify oracle-bridge compatibility (SSO may break if bridge API changed)
Check that authCookieClient still bridges cookies to localStorage
Test the full auth flow: login on think-tank -> navigate to dao-suite -> dashboard loads

Verification:

Visit https://staging-portal.helloworlddao.com
Should redirect to think-tank login
After login, should redirect back with authenticated view

dao-admin-suite

Canister: dxrwa-jaaaa-aaaao-a6uba-caiDomain: staging-admin.helloworlddao.com

bash

cd ~/git/dao-admin-suite
git log --oneline -n 5
git revert <commit-hash>
git push origin main

Special considerations:

Same oracle-bridge dependency as dao-suite
Verify AdminGuard still renders (RBAC placeholder)
Test admin-specific validateReturnUrl -- ensure it does not allow open redirects

Verification:

Visit https://staging-admin.helloworlddao.com
Should show auth gate / redirect to login
After login, admin dashboard should load

Package Rollback

If a shared package update breaks one or more suites:

1. Identify Affected Suites

bash

# Check which suites use the package
cd ~/git/think-tank-suite && npm list @hello-world-co-op/*
cd ~/git/governance-suite && npm list @hello-world-co-op/*
cd ~/git/marketing-suite && npm list @hello-world-co-op/*
cd ~/git/dao-suite && npm list @hello-world-co-op/*
cd ~/git/dao-admin-suite && npm list @hello-world-co-op/*

2. Downgrade Package Version

In each affected suite's package.json:

json

{
  "dependencies": {
    "@hello-world-co-op/ui": "0.1.0"
  }
}

Then:

bash

npm install
npm test                 # Verify tests pass with old version
npm run build            # Verify build succeeds
git add package.json package-lock.json
git commit -m "fix: downgrade @hello-world-co-op/ui to 0.1.0"
git push origin main     # Triggers redeploy

3. Redeploy All Affected Suites

Each suite with the downgraded package must be redeployed. Push to main triggers CI/CD automatically.

Current package versions (as of FAS-8.1):

@hello-world-co-op/api: 0.1.0
@hello-world-co-op/auth: 0.1.0
@hello-world-co-op/ui: 0.1.0

DNS Rollback

If a custom domain is misconfigured or pointing to the wrong canister:

1. Identify Current DNS State

bash

# Check where the domain currently points
dig +short staging-think-tank.helloworlddao.com
dig +short staging-governance.helloworlddao.com
dig +short staging.helloworlddao.com
dig +short staging-ottercamp.helloworlddao.com
dig +short staging-portal.helloworlddao.com
dig +short staging-admin.helloworlddao.com

2. Revert DNS CNAME

Update the DNS CNAME record to point to the previous canister ID:

Domain	Current Target	Rollback Target (if needed)
staging-think-tank.helloworlddao.com	`wnfjk-biaaa-aaaao-a6dhq-cai.icp0.io`	(same -- no rollback)
staging-governance.helloworlddao.com	`wkep6-mqaaa-aaaao-a6dha-cai.icp0.io`	(same -- no rollback)
staging.helloworlddao.com	`d5fe6-hqaaa-aaaao-a6t5q-cai.icp0.io`	(same -- no rollback)
staging-ottercamp.helloworlddao.com	`dzt3i-sqaaa-aaaao-a6uaa-cai.icp0.io`	(same -- no rollback)
staging-portal.helloworlddao.com	`d6s54-7iaaa-aaaao-a6uaq-cai.icp0.io`	(same -- no rollback)
staging-admin.helloworlddao.com	`dxrwa-jaaaa-aaaao-a6uba-cai.icp0.io`	(same -- no rollback)

3. DNS Propagation Time

Estimated: 5-60 minutes depending on TTL
Workaround during propagation: Use direct canister URLs (https://<canister-id>.icp0.io) for immediate access

4. Verify DNS Propagation

bash

# Verify each domain resolves correctly
dig +short staging-think-tank.helloworlddao.com
# Expected: icp1.io (or similar IC boundary node)

# Verify with curl
curl -I https://staging-think-tank.helloworlddao.com
# Expected: HTTP/2 200

Verification Checklist

After any rollback, verify these items:

Health Checks

[ ] Suite responds to HTTP requests (200 OK)
[ ] No JavaScript errors in browser console
[ ] Bundle loads correctly (no missing chunks)

Auth Flow (auth-integrated suites only)

[ ] Login redirects work (suite -> think-tank -> back)
[ ] Session cookie set on .helloworlddao.com
[ ] Session validation returns authenticated: true
[ ] Logout clears session across suites
[ ] No infinite redirect loops

Suite-Specific

[ ] Marketing: Pre-rendered HTML has content, sitemap.xml exists
[ ] Otter Camp: Phaser.js game canvas renders
[ ] DAO: Dashboard loads with user data
[ ] DAO Admin: Admin dashboard renders, KYC review accessible

Performance

[ ] Bundle size within NFR6 (< 500KB gzipped)
[ ] No console errors in production build
[ ] Page load time acceptable

Rollback Commands Reference

Git Commands

bash

# Revert a specific commit
git revert <commit-hash>

# Revert multiple commits
git revert <oldest-commit>..<newest-commit>

# Checkout a specific tag
git checkout <tag-name>

# Create a new branch from a tag
git checkout -b rollback/<tag-name> <tag-name>

dfx Commands

bash

# Check canister status
dfx canister status <canister-id> --network ic

# Reinstall canister with previous WASM
dfx canister install <canister-name> --mode reinstall --wasm <previous.wasm> --network ic

# Start a stopped canister
dfx canister start <canister-id> --network ic

# Stop a canister
dfx canister stop <canister-id> --network ic

npm Commands

bash

# Downgrade a package
npm install @hello-world-co-op/ui@0.1.0

# Check installed versions
npm list @hello-world-co-op/*

# Check available versions
npm view @hello-world-co-op/ui versions --registry=https://npm.pkg.github.com

DNS Verification

bash

# Check DNS resolution
dig +short <domain>

# Full DNS trace
dig +trace <domain>

# HTTP verification
curl -I https://<domain>

# Direct canister access (bypasses DNS)
curl -I https://<canister-id>.icp0.io

FAS-8.1 Emergency Rollback -- Monolith rollback procedure (see bmad-artifacts/implementation-artifacts/)
Architecture Overview -- Suite and package architecture
Troubleshooting Guide -- Common issues and fixes
Cross-Suite Auth Debugging -- SSO diagnosis
DNS Configuration Reference -- DNS records (see bmad-artifacts/implementation-artifacts/)
Canister ID Reference -- All canister IDs (see bmad-artifacts/implementation-artifacts/)

FAS Rollback Procedures ​

Table of Contents ​

When to Rollback ​

Criteria for Triggering a Rollback ​

Do NOT Rollback For ​

Suite-Specific Rollback Steps ​

General Pattern (All Suites) ​

think-tank-suite ​

governance-suite ​

marketing-suite ​

otter-camp-suite ​

dao-suite ​

dao-admin-suite ​

Package Rollback ​

1. Identify Affected Suites ​

2. Downgrade Package Version ​

3. Redeploy All Affected Suites ​

DNS Rollback ​

1. Identify Current DNS State ​

2. Revert DNS CNAME ​

3. DNS Propagation Time ​

4. Verify DNS Propagation ​

Verification Checklist ​

Health Checks ​

Auth Flow (auth-integrated suites only) ​

Suite-Specific ​

Performance ​

Rollback Commands Reference ​

Git Commands ​

dfx Commands ​

npm Commands ​

DNS Verification ​

Related Documentation ​

FAS Rollback Procedures

Table of Contents

When to Rollback

Criteria for Triggering a Rollback

Do NOT Rollback For

Suite-Specific Rollback Steps

General Pattern (All Suites)

think-tank-suite

governance-suite

marketing-suite

otter-camp-suite

dao-suite

dao-admin-suite

Package Rollback

1. Identify Affected Suites

2. Downgrade Package Version

3. Redeploy All Affected Suites

DNS Rollback

1. Identify Current DNS State

2. Revert DNS CNAME

3. DNS Propagation Time

4. Verify DNS Propagation

Verification Checklist

Health Checks

Auth Flow (auth-integrated suites only)

Suite-Specific

Performance

Rollback Commands Reference

Git Commands

dfx Commands

npm Commands

DNS Verification

Related Documentation