Operations And Recovery

Admin Settings > Operations is the master CMS recovery surface.

Use it to:

• run scheduled backups for the master CMS and every managed contributor subCMS
• test backup archives before relying on them
• send backup-created events to an offsite backup hook
• label staged releases by upgrade channel
• queue app rollbacks for the root worker
• export redacted support bundles
• run Rebuild all, Backup all, and Submit all sitemaps

Contributor subCMS users cannot access this page. Master support users can view the page and download existing support bundles, but they cannot run actions.

Scheduled Backups

Scheduled backups are controlled by:

• operations_backup_schedule_enabled
• operations_backup_interval_hours
• operations_backup_last_run_at

The OpenBSD operations worker checks the schedule every 15 minutes and runs backups only when the configured interval has elapsed.

Install the worker once per server:

doas perl /usr/local/www/desertcms/tools/openbsd-operations-worker.pl --install-cron

The cron entry runs the worker as _desertcms, not as root. Each backup still uses that site's own config, SQLite database, public root, originals directory, theme directory, and backup directory.

Backup All

Backup all sites creates backups for:

• the master CMS
• active contributor subCMS sites
• disabled contributor subCMS sites
• pending contributor sites when their config exists

Destroyed sites are skipped.

Each backup includes:

• SQLite database
• private originals
• generated public files
• editable themes
• manifest with SHA-256 checksums

Restore Testing

Restore testing extracts a selected master backup into a temporary private directory, verifies the manifest checksums, and runs SQLite PRAGMA integrity_check.

It does not replace the live database or public files. Use it before a major upgrade, before deleting old backups, and after changing backup storage.

Offsite Backup Hooks

Set an offsite hook URL and optional bearer token in Operations.

After each backup, DesertCMS posts a small JSON event:

{
  "event": "desertcms.backup.created",
  "site_id": "kaleb",
  "domain": "kaleb.example.com",
  "filename": "desertcms-20260630-010203.tar.gz",
  "bytes": 1234567,
  "sha256": "..."
}

The hook does not upload the archive itself. Use it to notify an external backup puller, object-storage sync job, or monitoring system.

Upgrade Channels

Operations stores a default channel:

• stable
• beta
• nightly

New upgrade jobs staged from Admin Settings > Upgrade DesertCMS include the current channel label. The label does not change validation or install behavior; it gives operators a clear release lane in the job list.

Rollback

The upgrade worker creates an application backup before every upgrade and rollback:

/var/backups/desertcms-app-YYYYMMDD-HHMMSS.tar.gz

Operations lists those app backups and can queue a rollback job. The admin CGI only writes the queued job. The root worker performs the rollback, validates the extracted app tree, backs up the current app again, migrates and rebuilds configured instances, then restarts desertcms_slowcgi and reloads httpd.

If a rollback is queued more than once for the same archive, DesertCMS reuses the existing queued, running, or completed job instead of creating duplicates.

Rebuild All

Rebuild all sites rebuilds:

• the master CMS
• every active contributor subCMS with a readable config path

It loads each site independently and writes generated output to that site's own public root. On OpenBSD, the sandbox unveils contributor public roots with write access so this action can complete from the master operations page.

Submit All Sitemaps

Submit all sitemaps walks the master and active contributor subCMS sites.

For each site:

• Google Search Console is submitted only when that site has a connected OAuth account.
• IndexNow is submitted only when enabled for that site.
• Unconfigured providers are skipped rather than treated as failures.

IndexNow treats subdomains as separate hosts, so each contributor subCMS keeps its own key and public key file.

Health Report

The Operations page summarizes:

• managed site count
• active contributor count
• fleet alerts
• open queue items
• latest master backup
• scheduled backup state
• current upgrade channel
• support bundle count

For deeper fleet detail, use Admin Settings > Master Control.

Support Bundles

Create support bundle exports a compressed archive under:

<backup_dir>/support-bundles/

The bundle includes:

• health report JSON
• redacted settings JSON
• recent upgrade and rollback jobs
• redacted master and contributor config files

It does not include:

• SQLite databases
• private originals
• backup archives
• session data
• raw secrets or tokens

Secret-like keys such as stripe_secret_key, stripe_webhook_secret, postmark_server_token, OAuth tokens, webhook tokens, and password fields are replaced with [redacted].

Recovery Order

For a failed release:

1. Open Admin Settings > Operations.
2. Create a support bundle.
3. Test the latest backup.
4. Queue rollback to the newest known-good app backup.
5. Wait for the root upgrade worker to mark the rollback job done.
6. Run Rebuild all sites if public output still looks stale.
7. Run Submit all sitemaps after public pages are correct.

For data loss in a single site, use that site's backup archive and the maintenance restore command for that site config. Test the archive first.