Infrastructure & Deployment

This document covers the production and staging infrastructure for Hello World DAO: the Hetzner VPS, Cloudflare DNS, Docker-based oracle-bridge deployment, and the FounderyOS Kubernetes cluster on AX42-U.

---

Hetzner VPS (oracle-bridge Staging)

The oracle-bridge off-chain service is deployed to a Hetzner server in Helsinki.

Property	Value
Provider	Hetzner Cloud
Location	Helsinki, Finland
IP	65.21.149.226
SSH	ssh -i ~/.ssh/oracle-bridge-deploy deploy@65.21.149.226

Docker Containers

Oracle-bridge runs as Docker containers managed by Docker Compose:

Container	Port	Purpose
oracle-bridge-staging	8787	Staging environment
oracle-bridge-production	8788	Production environment

# SSH to VPS
ssh -i ~/.ssh/oracle-bridge-deploy deploy@65.21.149.226

# View running containers
docker ps

# View staging logs (last 100 lines)
docker logs oracle-bridge-staging --tail 100 -f

# Restart staging container
docker compose -f ~/oracle-bridge/docker-compose.staging.yml restart

Docker Compose Directory

Compose files and .env files live in ~/oracle-bridge/ on the VPS. Never edit these manually in production — changes are managed by CI/CD.

Container Registry

Images are published to GitHub Container Registry:

ghcr.io/hello-world-co-op/oracle-bridge:staging   (latest staging build)
ghcr.io/hello-world-co-op/oracle-bridge:latest     (latest production build)
ghcr.io/hello-world-co-op/oracle-bridge:v0.x.x     (versioned release tags)

PEM and CA Certificates

Secrets stored on the VPS:

Path	Contents	Permissions
/etc/oracle-bridge/github-ci-identity.pem	IC identity for canister calls	640 root:docker
/etc/oracle-bridge/proton-bridge-ca.crt	Internal CA cert	640 root:docker

These are mounted read-only into containers via the compose file. To rotate: update the file on the VPS and restart the container.

---

CI/CD — Docker Build & Deploy

Oracle-bridge uses GitHub Actions for automated deployment. Manual dfx deploy or SSH deploys are not used.

Workflow Files

Workflow	File	Trigger
Staging deploy	docker-build.yml	Push to main
Production deploy	deploy-production.yml	GitHub Release event

Staging Deploy Flow

git push origin main
    │
    ▼
GHA: docker-build.yml
    ├── Build Docker image
    ├── Push to ghcr.io:staging
    └── SSH to VPS → docker compose pull + up -d

To check deploy status:

# View latest staging deploy run
gh run list --workflow=docker-build.yml --repo Hello-World-Co-Op/oracle-bridge --limit 5

# View logs for a specific run
gh run view <run-id> --log

Production Release Flow

gh release create v0.x.x --title "..." --notes "..."
    │
    ▼
GHA: deploy-production.yml
    ├── Build Docker image
    ├── Push to ghcr.io:latest + ghcr.io:v0.x.x
    └── SSH to VPS → docker compose pull + up -d (production container)

---

Cloudflare DNS

DNS for helloworlddao.com migrated from GoDaddy to Cloudflare in February 2026.

Property	Value
Nameservers	lewis.ns.cloudflare.com, audrey.ns.cloudflare.com
Zone ID	c568354061b31b7b1c96f9cd2f98c4a4
Credentials	~/.config/cloudflare/credentials.env

DNS Sync Script

DNS records are managed declaratively via ops-infra/scripts/cloudflare-dns-sync.sh. This script defines all 61+ records and syncs them to Cloudflare.

# Dry run — shows what would change
./ops-infra/scripts/cloudflare-dns-sync.sh --dry-run

# Apply changes
./ops-infra/scripts/cloudflare-dns-sync.sh

Important: IC boundary node records MUST be proxied: false (DNS-only, grey cloud). Cloudflare's auto-import sets proxied: true for all records — the sync script corrects this. Proxied IC records break canister routing.

DDNS Script

Sector7 (Graydon's network) uses a DDNS cron job to keep DNS current with the dynamic WAN IP:

# Script location
ops-infra/scripts/cloudflare-ddns-helloworlddao.sh

# Current Sector7 WAN IP (as of 2026-02-28)
71.73.242.88

Key DNS Records

Subdomain	Type	Target	Notes
www	CNAME	IC boundary node	Marketing suite canister
portal	CNAME	IC boundary node	DAO suite canister
admin	CNAME	IC boundary node	DAO admin suite canister
oracle	A	65.21.149.226	Oracle bridge (staging on 8787)
staging-oracle	A	65.21.149.226	Explicit staging endpoint
think-tank	CNAME	IC boundary node	Think Tank suite
ottercamp	CNAME	IC boundary node	Otter Camp suite
governance	CNAME	IC boundary node	Governance suite

Unmanaged Records

These records exist in Cloudflare but are NOT managed by the sync script:

Record	Reason
_domainconnect	GoDaddy artifact — safe to leave
_gh-hello-world-coop-dao-e	GitHub org verification — do not delete

IC Custom Domain API

To transfer a custom domain between IC canisters (e.g., promote staging canister to production):

# Trigger domain transfer
curl -X PATCH https://icp0.io/custom-domains/v1/helloworlddao.com \
  -H "Content-Type: application/json" \
  -d '{"canister_id": "<new-canister-id>"}'

You must also remove the domain from the old canister's .well-known/ic-domains file before or after the API call. See IC Custom Domain Runbook for the full procedure.

---

Sector7 Kubernetes Cluster (FounderyOS)

FounderyOS (the off-chain SaaS platform) runs on a Kubernetes cluster hosted on Graydon's Sector7 network.

Cluster Nodes

Node	IP	Role	RAM	GPU	Notes
Aurora	192.168.2.159	control-plane, gpu-compute	64 GB	GTX 1070	Primary Ollama, k8s control
Theo	192.168.2.160	worker, compute	16 GB	GTX 1060	Dedicated Ollama (Coby)
Library	192.168.2.231	worker, storage	32 GB	—	PostgreSQL PVCs
Knower	192.168.2.93	workstation	—	—	Tainted NoSchedule

kubectl Access

# kubectl binary
~/.local/bin/kubectl

# Kubeconfig
~/.kube/config

# Context and cluster
context: sector7-coby
cluster: sector7 → https://192.168.2.159:6443

# Check cluster status
kubectl cluster-info --context sector7-coby

# List pods in hello-world namespace
kubectl get pods -n hello-world

# View pod logs
kubectl logs -n hello-world <pod-name> --tail 100 -f

Namespaces

Namespace	Purpose	Quota
hello-world	FounderyOS API, workers, Ollama	ResourceQuota applied
the-flourish	Affiliate project	View-only for Coby

Coby's RBAC: edit on hello-world, view on the-flourish. No longer cluster-admin (post-S206).

FounderyOS Services

Service	Internal Address	External	Notes
API	founderyos-api-svc.hello-world	founderyos.dev	Node.js + Fastify
PostgreSQL	postgres-svc.hello-world:5432	—	Neon-compatible
Ollama (Theo)	ollama-theo-svc.hello-world:11434	192.168.2.159:31434 (NodePort)	Coby's dedicated
Ollama (Aurora)	—	192.168.2.159:31434	Graydon's primary
GlitchTip	glitchtip.founderyos.dev	—	Error tracking

Tailscale VPN

Cluster access (from outside Sector7 LAN) requires Tailscale:

# Tailnet
hello-world-co-op.org.github

# Coby's identity
RationalSolutions@hello-world-co-op.org.github

# Check Tailscale peers
tailscale status

# Ping Aurora control plane
ping 192.168.2.159

Database Migrations (FounderyOS)

FounderyOS uses Prisma for database schema management:

# Apply pending migrations (from founderyos-api repo)
npx prisma migrate deploy

# Generate Prisma client after schema changes
npx prisma generate

Migrations run automatically on deploy via the founderyos-api pod's startup script. For manual migration during incidents:

# Exec into API pod
kubectl exec -it -n hello-world <founderyos-api-pod> -- sh

# Run migration inside pod
npx prisma migrate deploy

Cycle Monitoring

The IC canister fleet (30+ canisters, ~30 TC total) is monitored via:

• Local script: ops-infra/scripts/check-cycles.sh
• GHA cron: ops-infra/.github/workflows/monitor-metrics.yml (runs every 6 hours)
• Minimum balance: 100B cycles per canister

Canisters with high burn rates (user-service ~378M cycles/day, membership) require monthly top-ups. Top-up command:

# Convert ICP to cycles and top up a canister
dfx ledger top-up <canister-id> --amount <icp> --network ic

---

Resource Stability Rules

These rules apply to all cluster workloads (learned from EPIC-033 OOM crash resolution):

1. k3s reserves 2 GiB memory on all nodes (systemReserved)
2. kubelet eviction triggers at < 500 Mi available
3. All pods must have resources.limits and resources.requests set
4. LimitRange and ResourceQuota are enforced per namespace
5. Vitest test workers: maxThreads: 4 to prevent OOM during CI

Violating these rules causes node OOM and pod eviction cascades. Review ops-infra/k8s/ for current LimitRange and ResourceQuota manifests.