SigNoz Observability กับ Backup Recovery

Backup Recovery Strategy

Backup Recovery Strategy กำหนด RPO RTO สำหรับแต่ละระบบ ครอบคลุม Full Backup, Incremental, Offsite Storage ทดสอบ Recovery สม่ำเสมอ

เนื้อหาเกี่ยวข้อง — ทำความเข้าใจ ai วาดรูปผู้หญิง

SigNoz ช่วย Monitor Backup Jobs ติดตาม Duration, Success Rate, Storage Usage ตั้ง Alerts เมื่อ Backup ล้มเหลว สร้าง Dashboard แสดงสถานะทั้งหมด

เนื้อหาเกี่ยวข้อง — ดูเพิ่มเติมเรื่อง LLM Quantization GGUF Developer Experience DX

Backup Automation

# === Backup Automation Script ===
#!/bin/bash
# backup_automated.sh — Automated Backup with Monitoring

set -euo pipefail

# Configuration
BACKUP_DIR="/backup"
S3_BUCKET="s3://company-backups"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/backup/backup_.log"

# Databases to backup
DATABASES=("postgres_main" "postgres_analytics" "redis_cache" "clickhouse_events")

# OpenTelemetry — ส่ง Metrics ไป SigNoz
OTEL_ENDPOINT="http://signoz:4318/v1/metrics"

send_metric() {
    local name=$1
    local value=$2
    local labels=$3
    # ส่ง Metric ผ่าน OTLP HTTP
    curl -s -X POST "$OTEL_ENDPOINT" \
      -H "Content-Type: application/json" \
      -d "{\"resourceMetrics\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"backup-service\"}}]},\"scopeMetrics\":[{\"metrics\":[{\"name\":\"$name\",\"gauge\":{\"dataPoints\":[{\"asDouble\":$value}]}}]}]}]}" \
      > /dev/null 2>&1 || true
}

backup_database() {
    local db=$1
    local start_time=$(date +%s)
    local backup_file="/_.sql.gz"

    echo "[$(date)] Starting backup: $db" | tee -a "$LOG_FILE"

    case "$db" in
        postgres_*)
            pg_dump "" | gzip > "$backup_file" 2>> "$LOG_FILE"
            ;;
        redis_*)
            redis-cli BGSAVE
            cp /var/lib/redis/dump.rdb ".rdb" 2>> "$LOG_FILE"
            ;;
        clickhouse_*)
            clickhouse-client --query "SELECT * FROM " \
              --format Native | gzip > "$backup_file" 2>> "$LOG_FILE"
            ;;
    esac

    local end_time=$(date +%s)
    local duration=$((end_time - start_time))
    local size=$(stat -f%z "$backup_file" 2>/dev/null || stat -c%s "$backup_file" 2>/dev/null || echo 0)

    # Verify backup
    if gzip -t "$backup_file" 2>/dev/null; then
        echo "[$(date)] Backup OK: $db (s,  bytes)" | tee -a "$LOG_FILE"
        send_metric "backup.duration" "$duration" "db=$db"
        send_metric "backup.size_bytes" "$size" "db=$db"
        send_metric "backup.success" "1" "db=$db"

        # Upload to S3
        aws s3 cp "$backup_file" "//" --storage-class STANDARD_IA
        echo "[$(date)] Uploaded to S3: $db" | tee -a "$LOG_FILE"
    else
        echo "[$(date)] FAILED: $db" | tee -a "$LOG_FILE"
        send_metric "backup.success" "0" "db=$db"
    fi
}

# Main
echo "[$(date)] === Backup Started ===" | tee -a "$LOG_FILE"
total_start=$(date +%s)

for db in ""; do
    backup_database "$db"
done

# Cleanup old backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete
find "$BACKUP_DIR" -name "*.rdb" -mtime +$RETENTION_DAYS -delete

total_end=$(date +%s)
total_duration=$((total_end - total_start))

echo "[$(date)] === Backup Complete (s) ===" | tee -a "$LOG_FILE"
send_metric "backup.total_duration" "$total_duration" ""

# Crontab: 0 2 * * * /opt/scripts/backup_automated.sh

Disaster Recovery Plan

# === Disaster Recovery Plan ===

dr_plan = {
    "tier1_critical": {
        "systems": ["postgres_main", "api-gateway", "auth-service"],
        "rpo": "1 hour",
        "rto": "2 hours",
        "strategy": "Active-Active Cross-Region",
        "backup": "Continuous Replication + Hourly Snapshots",
        "recovery": [
            "1. Detect failure (SigNoz Alert)",
            "2. Verify scope of failure",
            "3. Failover DNS to DR region",
            "4. Verify services in DR region",
            "5. Notify stakeholders",
        ],
    },
    "tier2_important": {
        "systems": ["postgres_analytics", "clickhouse_events", "redis_cache"],
        "rpo": "4 hours",
        "rto": "4 hours",
        "strategy": "Warm Standby + S3 Backup",
        "backup": "4-hourly Full + Continuous WAL",
        "recovery": [
            "1. Detect failure (SigNoz Alert)",
            "2. Provision new instances from AMI",
            "3. Restore from latest backup",
            "4. Apply WAL logs to minimize data loss",
            "5. Update DNS and config",
            "6. Verify data integrity",
        ],
    },
    "tier3_non_critical": {
        "systems": ["dev-db", "staging-db", "log-archive"],
        "rpo": "24 hours",
        "rto": "8 hours",
        "strategy": "Daily Backup to S3",
        "backup": "Daily Full Backup",
        "recovery": [
            "1. Provision new instances",
            "2. Restore from S3 backup",
            "3. Verify services",
        ],
    },
}

# DR Drill Schedule
dr_drills = [
    {"quarter": "Q1", "type": "Tabletop Exercise", "scope": "All tiers"},
    {"quarter": "Q2", "type": "Tier 1 Failover Test", "scope": "Critical systems"},
    {"quarter": "Q3", "type": "Full DR Drill", "scope": "All tiers"},
    {"quarter": "Q4", "type": "Tier 2 Recovery Test", "scope": "Important systems"},
]

print("Disaster Recovery Plan:")
for tier, plan in dr_plan.items():
    print(f"\n  [{tier}]")
    print(f"    Systems: {', '.join(plan['systems'])}")
    print(f"    RPO: {plan['rpo']} | RTO: {plan['rto']}")
    print(f"    Strategy: {plan['strategy']}")
    print(f"    Steps: {len(plan['recovery'])}")

print(f"\n  DR Drill Schedule:")
for drill in dr_drills:
    print(f"    {drill['quarter']}: {drill['type']} ({drill['scope']})")

Best Practices

• 3-2-1 Rule: 3 copies, 2 media types, 1 offsite ใช้ Immutable Backups
• Monitor Backups: ใช้ SigNoz ติดตาม Backup Duration, Success Rate, Size
• Test Recovery: ทดสอบ Restore จาก Backup อย่างน้อยเดือนละครั้ง
• RPO/RTO: กำหนด RPO RTO ตาม Business Impact ของแต่ละระบบ
• DR Drills: ทำ DR Drill อย่างน้อยปีละ 2 ครั้ง ทั้ง Tabletop และ Full Test
• Alerts: ตั้ง Alerts เมื่อ Backup ล้มเหลว หรือ Backup Age เกิน RPO

Backup Recovery Strategy คืออะไร

แผนสำรองกู้คืนข้อมูล กำหนด RPO ข้อมูลหายได้มากสุด RTO กู้คืนเร็วแค่ไหน Full Incremental Backup Offsite Storage Automated Testing Disaster Recovery

แนะนำเพิ่มเติม — iCafeForex

เนื้อหาเกี่ยวข้อง — แนะนำให้อ่าน Shadcn UI AR VR Development