Email Alerts¶

Owlette sends email notifications when machines go offline, processes crash, or connections fail.

Alert Types¶

Alert	Trigger	Recipients
Machine Offline	Heartbeat missing for 3+ minutes	All site users with `healthAlerts` enabled
Process Crash	Configured process exits unexpectedly	Users subscribed to process alerts for that site
Process Start Failed	Process failed to launch	Users subscribed to process alerts
Connection Failure	Agent lost connection to Firestore	Site admins

Machine Offline Alerts¶

How It Works¶

A Railway cron job runs every 5 minutes, calling GET /api/cron/health-check:

Scans all sites/{siteId}/machines/{machineId} for stale heartbeats
A machine is considered offline if its last heartbeat is older than 3 minutes
Sends one grouped email per site listing all offline machines
Writes health.lastCronAlertAt to Firestore to prevent repeat alerts
Cooldown: 1 hour per machine — won't send another alert for the same machine within an hour

Email Content¶

The email includes:

Site name
List of offline machines with last-seen timestamps
Direct link to the dashboard

Process Crash Alerts¶

When the agent detects a process crash:

Agent sends POST /api/agent/alert with alert type process_crash
Server looks up users subscribed to process alerts for that site
Sends email with process name, machine name, and error details

Rate Limiting¶

Process alerts: 3 per hour per process per machine
Connection failures: 5 per hour per IP

Setting Up Alerts¶

Prerequisites¶

Resend API key — Set RESEND_API_KEY environment variable in Railway
Admin email — Set ADMIN_EMAIL_PROD (production) or ADMIN_EMAIL_DEV (development)
Cron job — Configure Railway cron schedule (see below)

Configuring the Cron Job¶

In Railway dashboard, open your web service
Go to Settings → "Cron Schedule"
Enter: */5 * * * * (every 5 minutes)

Add CRON_SECRET environment variable:

python -c "import secrets; print(secrets.token_hex(32))"

User Preferences¶

Users can control their alert preferences:

Health alerts: Opt in/out of machine offline notifications
Process alerts: Opt in/out of process crash notifications

Unsubscribe¶

Each alert email contains a one-click unsubscribe link that disables health alerts for that user.

Testing Emails¶

Test Email Page¶

Admins can send test emails from Admin Panel → Email Test (/admin/test-email):

Enter a recipient email
Optionally customize subject and message
Click "Send Test Email"
Verify delivery in your inbox

Simulate Events¶

Test the full alert pipeline from /api/admin/events/simulate:

curl -X POST https://your-dashboard.app/api/admin/events/simulate \
  -H "Content-Type: application/json" \
  -H "Cookie: session=..." \
  -d '{
    "siteId": "your-site",
    "event": "process_crash",
    "data": {
      "machineId": "DESKTOP-TEST",
      "machineName": "Test Machine",
      "processName": "TouchDesigner",
      "errorMessage": "Process exited with code 1"
    }
  }'

This triggers the same email alert flow as a real event, without requiring a real crash.

Environment Variables¶

Variable	Required	Description
`RESEND_API_KEY`	Yes	API key from Resend
`CRON_SECRET`	Yes	Shared secret for cron authentication
`ADMIN_EMAIL_PROD`	Yes	Fallback admin email (production)
`ADMIN_EMAIL_DEV`	Yes	Fallback admin email (development)

Troubleshooting¶

No Emails Received¶

Check RESEND_API_KEY is set in Railway environment variables
Verify the Resend API key is valid (test at resend.com)
Check spam/junk folder
Use the test email page to verify delivery
Check Railway logs for email-related errors

Duplicate Alerts¶

The system has built-in cooldowns (1 hour per machine for offline alerts, rate limits for process alerts). If you're getting duplicates, check that the cron job isn't running more frequently than expected.

Cron Not Running¶

Verify Railway cron schedule is set: */5 * * * *
Check that CRON_SECRET matches between the env var and the cron configuration
Review Railway deployment logs for cron execution