Airbyte ETL Zero Downtime Deployment

Airbyte Zero Downtime

Airbyte ETL ELT Data Integration Zero Downtime Deployment Connector CDC Incremental Sync Blue-Green Rolling Update Kubernetes Helm PostgreSQL BigQuery Snowflake

Airbyte Setup

# === Airbyte Installation ===

# Docker Compose — Quick Start
# git clone https://github.com/airbytehq/airbyte.git
# cd airbyte
# docker compose up -d
# # Access: http://localhost:8000

# Kubernetes — Helm Chart
# helm repo add airbyte https://airbytehq.github.io/helm-charts
# helm repo update
#
# helm install airbyte airbyte/airbyte \
#   --namespace airbyte --create-namespace \
#   --set webapp.service.type=ClusterIP \
#   --set global.database.type=external \
#   --set global.database.host=postgres.example.com \
#   --set global.database.port=5432 \
#   --set global.database.database=airbyte \
#   --set global.database.user=airbyte \
#   --set global.database.password=secret \
#   --set global.logs.storage.type=S3 \
#   --set global.logs.s3.bucket=airbyte-logs \
#   --set global.logs.s3.bucketRegion=ap-southeast-1

# values.yaml — Production Configuration
# global:
#   database:
#     type: external
#     host: postgres-rds.example.com
#   logs:
#     storage:
#       type: S3
#   jobs:
#     resources:
#       requests:
#         cpu: "1"
#         memory: "2Gi"
#       limits:
#         cpu: "2"
#         memory: "4Gi"
# webapp:
#   replicaCount: 2
# server:
#   replicaCount: 2
# worker:
#   replicaCount: 3

from dataclasses import dataclass

@dataclass
class Connection:
    name: str
    source: str
    destination: str
    sync_mode: str
    schedule: str
    last_sync: str
    status: str

connections = [
    Connection("Users DB", "PostgreSQL", "BigQuery", "Incremental CDC", "Every 1h", "14:00", "Active"),
    Connection("Orders DB", "MySQL", "Snowflake", "Incremental Append", "Every 30m", "14:15", "Active"),
    Connection("Stripe API", "Stripe", "BigQuery", "Incremental", "Every 6h", "12:00", "Active"),
    Connection("HubSpot CRM", "HubSpot", "PostgreSQL DWH", "Full Refresh", "Daily 02:00", "02:00", "Active"),
    Connection("S3 Logs", "S3", "BigQuery", "Incremental Append", "Every 1h", "14:00", "Active"),
    Connection("Salesforce", "Salesforce", "Snowflake", "Incremental", "Every 4h", "12:00", "Active"),
]

print("=== Airbyte Connections ===")
for c in connections:
    print(f"  [{c.status}] {c.name}")
    print(f"    {c.source} -> {c.destination} | Mode: {c.sync_mode}")
    print(f"    Schedule: {c.schedule} | Last: {c.last_sync}")

Zero Downtime Strategy

# === Zero Downtime Deployment ===

# Rolling Update — Kubernetes
# spec:
#   strategy:
#     type: RollingUpdate
#     rollingUpdate:
#       maxSurge: 1
#       maxUnavailable: 0
#   # New pod starts before old pod terminates

# Pre-deploy Checklist
# 1. Check running syncs — wait for completion
# 2. Pause scheduled syncs
# 3. Database migration compatibility check
# 4. Deploy new version (rolling update)
# 5. Verify health checks pass
# 6. Resume scheduled syncs
# 7. Monitor for errors

# Deployment Script
# #!/bin/bash
# # Check for running syncs
# running=$(curl -s http://airbyte:8001/api/v1/jobs/list \
#   -d '{"configTypes":["sync"],"statuses":["running"]}' | jq '.jobs | length')
#
# if [ "$running" -gt 0 ]; then
#   echo "Waiting for $running syncs to complete..."
#   sleep 300
# fi
#
# # Deploy new version
# helm upgrade airbyte airbyte/airbyte \
#   --namespace airbyte \
#   -f values.yaml \
#   --set global.image.tag=0.60.0 \
#   --wait --timeout 10m
#
# # Verify
# kubectl rollout status deployment/airbyte-webapp -n airbyte
# kubectl rollout status deployment/airbyte-server -n airbyte

@dataclass
class DeployStrategy:
    strategy: str
    downtime: str
    complexity: str
    rollback: str
    use_case: str

strategies = [
    DeployStrategy("Rolling Update", "0", "ต่ำ", "kubectl rollout undo", "Standard K8s"),
    DeployStrategy("Blue-Green", "0", "สูง", "Switch DNS/LB", "Critical systems"),
    DeployStrategy("Canary", "0", "สูง", "Scale down canary", "Gradual rollout"),
    DeployStrategy("Recreate", "Minutes", "ต่ำ", "Redeploy old version", "Dev/Test only"),
]

print("\n=== Deployment Strategies ===")
for s in strategies:
    print(f"  [{s.strategy}] Downtime: {s.downtime}")
    print(f"    Complexity: {s.complexity} | Rollback: {s.rollback}")
    print(f"    Use: {s.use_case}")

Monitoring และ Operations

# === Production Operations ===

# Health Check Endpoints
# /api/v1/health — Server health
# /api/v1/jobs/list — List sync jobs
# /api/v1/connections/list — List connections

# Monitoring Metrics
# airbyte_worker_job_running_count
# airbyte_worker_job_succeeded_count
# airbyte_worker_job_failed_count
# airbyte_sync_duration_seconds
# airbyte_records_emitted_total

# Prometheus + Grafana
# helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
# ServiceMonitor for Airbyte metrics

operational_metrics = {
    "Active Connections": "12",
    "Running Syncs": "3",
    "Syncs Today": "48",
    "Failed Syncs (24h)": "1",
    "Records Synced (24h)": "2.5M",
    "Avg Sync Duration": "8 minutes",
    "Worker Pods": "3/3 healthy",
    "DB Size": "15 GB",
    "Monthly Data Volume": "75 GB",
}

print("Operations Dashboard:")
for k, v in operational_metrics.items():
    print(f"  {k}: {v}")

# Troubleshooting
troubleshoot = [
    "Sync Failed: ดู Job Log ใน Web UI หรือ kubectl logs",
    "Slow Sync: เพิ่ม Worker Resources CPU/Memory",
    "OOM: เพิ่ม Memory Limit สำหรับ Job Container",
    "Connection Timeout: ตรวจ Network Policy Firewall",
    "Schema Change: Airbyte ตรวจจับอัตโนมัติ แต่อาจ Break Pipeline",
    "Disk Full: ใช้ External Log Storage S3 แทน Local",
    "Deploy Fail: kubectl rollout undo กลับ Version เก่า",
]

print(f"\n\nTroubleshooting:")
for i, t in enumerate(troubleshoot, 1):
    print(f"  {i}. {t}")

เคล็ดลับ

• External DB: ใช้ External PostgreSQL ไม่ใช้ Internal
• S3 Logs: เก็บ Log บน S3 ไม่เต็ม Disk
• CDC: ใช้ CDC Incremental Sync ประหยัด Resource
• Rolling: maxSurge 1 maxUnavailable 0 สำหรับ Zero Downtime
• Monitor: Alert เมื่อ Sync Failed ทันที

Airbyte คืออะไร

Open Source Data Integration ETL ELT 300+ Connector Database API SaaS CDC Incremental Sync Schedule Web UI Cloud Self-hosted

สรุป

Airbyte ETL Zero Downtime Deployment Kubernetes Helm Rolling Update CDC Incremental Sync Connector PostgreSQL BigQuery Snowflake S3 Monitoring Production Operations