Canister Cycle Monitoring

Last Updated: 2026-02-15 Owner: DevOps Team Related: Cycles Top-Up Runbook, Monitoring & Alerting

Overview

This document provides procedures for monitoring Internet Computer (IC) canister cycle balances, checking canister health, and topping up cycles before depletion. All production canisters run on the IC mainnet and consume cycles for computation and storage.

Canister Inventory

Backend Canisters

Canister	ID	Purpose	Budget Target
auth-service	`gexqh-jqaaa-aaaae-acsxq-cai`	Session management, login delegation	< 0.5 TC/month
user-service	`j4rvr-3aaaa-aaaao-qkvfq-cai`	User records, password verification	< 0.5 TC/month
blog	`dsqvo-niaaa-aaaao-a6ynq-cai`	Blog post storage, queries	< 1 TC/month (NFR31)

Frontend Asset Canisters (6 Suites)

Suite	ID	Domain	Budget Target
foundery-os-suite	`wnfjk-biaaa-aaaao-a6dhq-cai`	staging-foundery.helloworlddao.com foundery.helloworlddao.com	< 0.2 TC/month
governance-suite	`wkep6-mqaaa-aaaao-a6dha-cai`	staging-governance.helloworlddao.com governance.helloworlddao.com	< 0.2 TC/month
marketing-suite	`d5fe6-hqaaa-aaaao-a6t5q-cai`	www.helloworlddao.com	< 0.3 TC/month
otter-camp-suite	`dzt3i-sqaaa-aaaao-a6uaa-cai`	staging-ottercamp.helloworlddao.com ottercamp.helloworlddao.com	< 0.3 TC/month
dao-suite	`d6s54-7iaaa-aaaao-a6uaq-cai`	staging-portal.helloworlddao.com portal.helloworlddao.com	< 0.2 TC/month
dao-admin-suite	`dxrwa-jaaaa-aaaao-a6uba-cai`	staging-admin.helloworlddao.com admin.helloworlddao.com	< 0.2 TC/month

Legacy Canisters (Archived)

Canister	ID	Status
Frontend (old)	`vlmti-wqaaa-aaaad-acoiq-cai`	Archived (replaced by suite architecture)
Admin (old)	`ehbll-kiaaa-aaaac-qc2aa-cai`	Archived (replaced by dao-admin-suite)

Total Budget Target: < 3 TC/month across all active canisters

Current Wallet Balance: ~2.99 TC cycles, ~9.01 ICP remaining (as of 2026-02-14)

Checking Cycle Balances

Prerequisites

bash

# Set dfx identity for mainnet operations
dfx identity use github-ci

# Required environment variable for mainnet operations
export DFX_WARNING=-mainnet_plaintext_identity

Check Single Canister

bash

# Check canister status (includes cycles balance)
dfx canister status <canister-id> --network ic

# Example output:
# Canister status: Running
# Controllers: <principal-id>
# Memory allocation: 0
# Compute allocation: 0
# Freezing threshold: 2_592_000
# Memory Size: Nat(12345678)
# Balance: 1_234_567_890_123 Cycles
# Module hash: 0xabcdef...

Check All Canisters (Script)

Create a monitoring script to check all canisters:

bash

#!/bin/bash
# canister-check.sh - Check all canister balances

set -e

export DFX_WARNING=-mainnet_plaintext_identity

# Canister inventory
declare -A CANISTERS=(
  ["auth-service"]="gexqh-jqaaa-aaaae-acsxq-cai"
  ["user-service"]="j4rvr-3aaaa-aaaao-qkvfq-cai"
  ["blog"]="dsqvo-niaaa-aaaao-a6ynq-cai"
  ["foundery-os-suite"]="wnfjk-biaaa-aaaao-a6dhq-cai"
  ["governance-suite"]="wkep6-mqaaa-aaaao-a6dha-cai"
  ["marketing-suite"]="d5fe6-hqaaa-aaaao-a6t5q-cai"
  ["otter-camp-suite"]="dzt3i-sqaaa-aaaao-a6uaa-cai"
  ["dao-suite"]="d6s54-7iaaa-aaaao-a6uaq-cai"
  ["dao-admin-suite"]="dxrwa-jaaaa-aaaao-a6uba-cai"
)

# Thresholds
WARNING_TC=1
CRITICAL_TC=0.5
WARNING_CYCLES=$((WARNING_TC * 1000000000000))
CRITICAL_CYCLES=$((CRITICAL_TC * 1000000000000))

echo "=== Canister Cycle Balance Report ==="
echo "Date: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
echo ""

for name in "${!CANISTERS[@]}"; do
  canister_id="${CANISTERS[$name]}"

  echo "Checking: $name ($canister_id)..."

  status_output=$(dfx canister status "$canister_id" --network ic 2>&1)

  # Extract cycles balance
  cycles=$(echo "$status_output" | grep -i "balance:" | awk '{print $2}' | tr -d ',_')

  if [ -z "$cycles" ]; then
    echo "  ❌ ERROR: Could not retrieve balance"
    continue
  fi

  # Convert to TC for display
  cycles_tc=$(echo "scale=2; $cycles / 1000000000000" | bc)

  # Check thresholds
  if [ "$cycles" -lt "$CRITICAL_CYCLES" ]; then
    echo "  🔴 CRITICAL: $cycles_tc TC (< $CRITICAL_TC TC)"
  elif [ "$cycles" -lt "$WARNING_CYCLES" ]; then
    echo "  🟡 WARNING: $cycles_tc TC (< $WARNING_TC TC)"
  else
    echo "  ✅ OK: $cycles_tc TC"
  fi

  echo ""
done

echo "=== End Report ==="

Usage:

bash

chmod +x canister-check.sh
./canister-check.sh

Alert Thresholds

Balance Thresholds

Level	Threshold	Color	Response Time	Action
Healthy	> 1 TC	🟢 Green	None	Monitor
Warning	< 1 TC	🟡 Yellow	< 1 hour	Schedule top-up
Critical	< 0.5 TC	🔴 Red	< 15 minutes	Immediate top-up
Emergency	< 0.1 TC	🔴 Red	< 5 minutes	Emergency top-up

Freezing Threshold

IC canisters have a freezing threshold that prevents them from consuming all cycles. Default is ~2.5 TC. When a canister's balance approaches the freezing threshold:

Update calls are rejected with "out of cycles" errors
Query calls continue to work (read-only)
Canister state is preserved (not lost)

Key Point: A canister won't delete itself due to low cycles, but it will stop accepting updates.

Cycle Top-Up Procedures

Option 1: Direct Top-Up with ICP (Recommended)

Convert ICP to cycles and send to canister:

bash

# Set identity and environment
dfx identity use github-ci
export DFX_WARNING=-mainnet_plaintext_identity

# Top up canister with ICP (amount in ICP, e.g., 0.5 ICP)
dfx ledger top-up <canister-id> --amount <icp-amount> --network ic

# Example: Top up blog canister with 0.5 ICP
dfx ledger top-up dsqvo-niaaa-aaaao-a6ynq-cai --amount 0.5 --network ic

Conversion Rate: ~1 ICP ≈ 1.3 TC (varies with XDR price)

Option 2: Deposit Cycles from Wallet

If you already have cycles in a cycles wallet:

bash

# Deposit cycles to canister (amount in cycles)
dfx canister deposit-cycles <amount> <canister-id> --network ic

# Example: Deposit 500 billion cycles (0.5 TC)
dfx canister deposit-cycles 500000000000 dsqvo-niaaa-aaaao-a6ynq-cai --network ic

Option 3: Automated Top-Up (CI/CD)

The marketing-suite pre-rendering workflow includes automated cycle balance checks:

yaml

# .github/workflows/blog-pre-render.yml
- name: Check cycle balance
  run: |
    export DFX_WARNING=-mainnet_plaintext_identity
    CYCLES=$(dfx canister status marketing_suite_assets --network ic | grep "Balance:" | awk '{print $2}' | tr -d ',_')
    THRESHOLD=500000000000  # 0.5 TC
    CRITICAL_THRESHOLD=100000000000  # 0.1 TC

    if [ "$CYCLES" -lt "$THRESHOLD" ]; then
      echo "::warning::Canister cycle balance is low ($CYCLES cycles)"
      echo "Threshold: $THRESHOLD cycles (0.5 TC)"
      echo "Top up with: dfx ledger top-up <canister-id> --amount <ICP> --network ic"
    fi

    if [ "$CYCLES" -lt "$CRITICAL_THRESHOLD" ]; then
      echo "::error::Canister cycle balance critically low!"
      exit 1
    fi

This workflow:

Runs before each blog deployment
Warns if balance < 0.5 TC
Fails deployment if balance < 0.1 TC
Documents top-up procedure in output

Emergency Top-Up (Out of ICP)

If the ICP wallet is depleted:

Contact team lead for ICP transfer
Get dfx identity account ID:
bash
```
dfx ledger account-id --network ic
```
Transfer ICP from exchange or team wallet to account ID
Verify balance:
bash
```
dfx ledger balance --network ic
```
Top up canister using Option 1 above

Blog Canister Health Monitoring

The blog canister (BL-008.7.3) includes operational health monitoring accessible from the dao-admin-suite dashboard.

Canister Stats Query

The blog canister exposes a get_canister_stats() query method:

bash

# Query canister stats directly
dfx canister call dsqvo-niaaa-aaaao-a6ynq-cai get_canister_stats --network ic

# Response format:
# record {
#   total_posts: nat64;
#   published_count: nat64;
#   draft_count: nat64;
#   scheduled_count: nat64;
#   archived_count: nat64;
#   storage_usage_bytes: nat64;
# }

Admin Dashboard (Web UI)

Access canister health via dao-admin-suite:

Login to admin.helloworlddao.com (requires admin role)
Navigate to Blog → Dashboard → Operations tab
View the "Blog Canister Health" panel showing:
- Total posts
- Published count
- Drafts count
- Scheduled count
- Archived count
- Storage usage (MB) with color indicators

Storage Monitoring (NFR33)

Target: Stable memory < 100MB with 500+ posts

Color Indicators:

🟢 Green: < 50 MB (healthy)
🟡 Yellow: 50-75 MB (monitor)
🔴 Red: > 75 MB (investigate optimization)

Storage Calculation: The canister serializes entire state using serde_cbor to measure actual memory usage.

Broken Image Scanner

The admin dashboard includes a broken image scanner (AC#1 from BL-008.7.3):

Click "Scan for Broken Images" in Blog Health Panel
Wait for scan to complete (checks all image URLs in published posts)
Review results:
- 🔴 Broken images (404, 500, timeout)
- 🟡 CORS-blocked images (unable to verify)
- ✅ All images OK message

What gets scanned:

featured_image_url fields
og_image_url fields
All <img src="..."> tags in post HTML bodies

Scan method: HEAD requests with 10-second timeout (doesn't download full images)

Pipeline Success Rate (NFR19)

Track pre-rendering workflow success rate via GitHub Actions:

Navigate to marketing-suite Actions
Filter by workflow: "Pre-Render Blog Posts"
Review run history for failures or slowdowns

Wallet Management

Current Wallet Status

Identity: github-ciBalance (as of 2026-02-14):

Cycles: ~2.99 TC
ICP: ~9.01 ICP

Freezing Threshold: ~2 TC (wallet needs to maintain this balance)

Check Wallet Balance

bash

# Check ICP balance
dfx ledger balance --network ic --identity github-ci

# Check cycles wallet balance (if using cycles wallet)
dfx wallet balance --network ic --identity github-ci

ICP Funding Process

Determine funding need:
- Review monthly burn rate (~3 TC/month)
- Add buffer for growth (2-3x)
- Target: Maintain 6-12 months runway
Acquire ICP:
- Purchase from exchange (Coinbase, Binance, etc.)
- Transfer from team treasury
- Document transaction for accounting

Transfer to dfx identity:

bash

# Get account ID
dfx ledger account-id --network ic --identity github-ci

# Transfer ICP to this account ID from exchange/wallet
# Verify transfer
dfx ledger balance --network ic --identity github-ci

Update inventory:
- Document new balance in this file
- Update cycle runway projection
- Set next funding review date

Monitoring Checklist

Daily (Automated)

[ ] GitHub Actions monitoring workflow runs successfully
[ ] No critical cycle balance alerts triggered
[ ] All canisters report "Running" status

Weekly (Manual Review)

[ ] Run canister-check.sh to verify all balances
[ ] Review cycle burn rate trends
[ ] Check for unusual consumption spikes
[ ] Verify wallet has sufficient ICP reserve (> 1 ICP)

Monthly (Comprehensive Audit)

[ ] Review all canister budgets vs actuals
[ ] Update budget targets if needed
[ ] Calculate cycles runway (months remaining)
[ ] Plan ICP funding if runway < 6 months
[ ] Review and update this document

Quarterly (Strategic Review)

[ ] Optimize cycle consumption (upgrade patterns, query vs update calls)
[ ] Review canister architecture for efficiency
[ ] Test emergency top-up procedures
[ ] Update disaster recovery plans

Troubleshooting

Canister Shows "Out of Cycles" Error

Symptoms:

Update calls fail with "out of cycles" error
Query calls still work
dfx canister status shows balance > 0 but near freezing threshold

Diagnosis:

bash

# Check current balance
dfx canister status <canister-id> --network ic

# Look for:
# Balance: 1_234_567_890 Cycles  (low balance)
# Freezing threshold: 2_592_000 seconds

Resolution:

Immediate: Top up with 0.5-1 ICP (creates buffer)
Verify: Confirm balance increased above freezing threshold
Test: Try an update call to verify canister accepts requests
Document: Log incident and update budget projections

See: Cycles Top-Up Runbook for detailed procedure

High Cycle Consumption

Symptoms:

Canister burns through cycles faster than budgeted
Monthly consumption exceeds target by 2x+

Diagnosis:

Check call volume:
- Review IC dashboard for canister
- Identify spike in update calls
- Look for abnormal traffic patterns
Review recent changes:
- Recent code deployments
- New features with expensive operations
- Increased storage usage

Resolution:

Optimize code:
- Convert update calls to queries where possible
- Reduce inter-canister calls
- Optimize data structures for smaller stable memory
Adjust budget:
- Update target in this document
- Increase monitoring frequency
- Plan additional ICP funding
Scale monitoring:
- Add canister-specific alerts
- Track cycle consumption per feature

dfx Commands Fail with "Identity Not Found"

Symptoms:

Error: Identity github-ci does not exist

Resolution:

bash

# List available identities
dfx identity list

# If github-ci missing, restore from PEM file
# (PEM stored in GitHub Actions secret DFX_IDENTITY_PEM)

# Create identity directory
mkdir -p ~/.config/dfx/identity/github-ci

# Copy PEM file to identity directory
cp /path/to/identity.pem ~/.config/dfx/identity/github-ci/identity.pem

# Verify identity restored
dfx identity use github-ci
dfx identity get-principal

Wallet Balance Shows Zero Despite ICP Transfer

Symptoms:

ICP transferred to account ID
dfx ledger balance shows 0 ICP

Diagnosis:

Check correct network:

bash

# Ensure using --network ic (not local)
dfx ledger balance --network ic

Verify correct identity:

bash

# Confirm using correct identity
dfx identity whoami

# Check account ID matches transfer destination
dfx ledger account-id --network ic

Check transaction on IC:
- Visit IC Dashboard
- Search for account ID
- Verify transaction appears in history

Resolution:

If transfer pending: Wait 5-10 minutes for finalization
If wrong network: Re-transfer to correct account ID on IC mainnet
If wrong identity: Use dfx identity use <correct-identity>

References

Internal Documentation

Cycles Top-Up Runbook - Detailed top-up procedures
Monitoring & Alerting - Grafana dashboards and alert setup
Incident Response - General incident handling
BL-008.7.3 Story - Blog health monitoring implementation (see bmad-artifacts/implementation-artifacts/)

DFINITY Documentation

External Tools

ICP Dashboard - Official IC monitoring
IC Rocks - Community canister explorer
dfx CLI Reference

Change Log

Date	Change	Author
2026-02-15	Initial canister monitoring documentation created (TD-BL008-2)	Claude Opus 4.6

Appendix: Cycle Conversion Reference

ICP to Cycles Conversion

Cycles price is pegged to XDR (IMF Special Drawing Rights), not ICP directly:

1 TC (trillion cycles) ≈ 1 XDR
1 ICP = variable XDR price (changes with market)
Example: If 1 ICP = 1.3 XDR, then 1 ICP ≈ 1.3 TC

Current rate: Check DFINITY Pricing Page

Common Amounts

ICP	Approximate TC	Notes
0.1 ICP	~0.13 TC	Minimum top-up
0.5 ICP	~0.65 TC	Standard top-up
1.0 ICP	~1.3 TC	Large top-up
5.0 ICP	~6.5 TC	Quarterly funding
10.0 ICP	~13 TC	Annual funding

Cycle Budget Planning

Monthly consumption: ~3 TC/month (all canisters)

Annual projection: ~36 TC/year

ICP funding needed: ~30 ICP/year (at 1 ICP = 1.3 TC)

Recommended reserve: 6-12 months runway (18-36 TC = ~15-30 ICP)

Login Required

Canister Cycle Monitoring ​

Overview ​

Canister Inventory ​

Backend Canisters ​

Frontend Asset Canisters (6 Suites) ​

Legacy Canisters (Archived) ​

Checking Cycle Balances ​

Prerequisites ​

Check Single Canister ​

Check All Canisters (Script) ​

Alert Thresholds ​

Balance Thresholds ​

Freezing Threshold ​

Cycle Top-Up Procedures ​

Option 1: Direct Top-Up with ICP (Recommended) ​

Option 2: Deposit Cycles from Wallet ​

Option 3: Automated Top-Up (CI/CD) ​

Emergency Top-Up (Out of ICP) ​

Blog Canister Health Monitoring ​

Canister Stats Query ​

Admin Dashboard (Web UI) ​

Storage Monitoring (NFR33) ​

Broken Image Scanner ​

Pipeline Success Rate (NFR19) ​

Wallet Management ​

Current Wallet Status ​

Check Wallet Balance ​

ICP Funding Process ​

Monitoring Checklist ​

Daily (Automated) ​

Weekly (Manual Review) ​

Monthly (Comprehensive Audit) ​

Quarterly (Strategic Review) ​

Troubleshooting ​

Canister Shows "Out of Cycles" Error ​

High Cycle Consumption ​

dfx Commands Fail with "Identity Not Found" ​

Wallet Balance Shows Zero Despite ICP Transfer ​

References ​

Internal Documentation ​

DFINITY Documentation ​

External Tools ​

Change Log ​

Appendix: Cycle Conversion Reference ​

ICP to Cycles Conversion ​

Common Amounts ​

Cycle Budget Planning ​

Canister Cycle Monitoring

Overview

Canister Inventory

Backend Canisters

Frontend Asset Canisters (6 Suites)

Legacy Canisters (Archived)

Checking Cycle Balances

Prerequisites

Check Single Canister

Check All Canisters (Script)

Alert Thresholds

Balance Thresholds

Freezing Threshold

Cycle Top-Up Procedures

Option 1: Direct Top-Up with ICP (Recommended)

Option 2: Deposit Cycles from Wallet

Option 3: Automated Top-Up (CI/CD)

Emergency Top-Up (Out of ICP)

Blog Canister Health Monitoring

Canister Stats Query

Admin Dashboard (Web UI)

Storage Monitoring (NFR33)

Broken Image Scanner

Pipeline Success Rate (NFR19)

Wallet Management

Current Wallet Status

Check Wallet Balance

ICP Funding Process

Monitoring Checklist

Daily (Automated)

Weekly (Manual Review)

Monthly (Comprehensive Audit)

Quarterly (Strategic Review)

Troubleshooting

Canister Shows "Out of Cycles" Error

High Cycle Consumption

dfx Commands Fail with "Identity Not Found"

Wallet Balance Shows Zero Despite ICP Transfer

References

Internal Documentation

DFINITY Documentation

External Tools

Change Log

Appendix: Cycle Conversion Reference

ICP to Cycles Conversion

Common Amounts

Cycle Budget Planning