ai
Airbyte ETL Log Management ELK — จัดการ Log ด้วย
Airbyte ELK Log Management
Airbyte ETL Log Management ELK Elasticsearch Logstash Kibana Filebeat Sync Connector Alert Dashboard Pipeline Production
| Log Type | Source | Volume | Retention |
|---|---|---|---|
| Sync Logs | Airbyte Worker Pods | สูง (ทุก Sync Run) | Hot 7d Warm 30d Cold 90d |
| Connector Logs | Source/Destination Containers | สูง (API Calls) | Hot 7d Warm 30d |
| Platform Logs | Scheduler Server DB | ปานกลาง | Hot 7d Warm 30d |
| Audit Logs | API Server | ต่ำ | Hot 30d Warm 90d Cold 365d |
Dashboard & Alert
# === Kibana Dashboard & Alert Rules ===
@dataclass
class AlertRule:
name: str
condition: str
severity: str
channel: str
alerts = [
AlertRule("Sync Failure Spike",
"status='failed' count > 3 in 1 hour",
"P1 Critical",
"Slack #data-alerts + PagerDuty"),
AlertRule("Error Log Spike",
"level='ERROR' count > 100 in 10 min",
"P1 Critical",
"Slack + PagerDuty"),
AlertRule("Sync Duration Anomaly",
"duration > 2x rolling_avg(7d)",
"P2 High",
"Slack #data-perf"),
AlertRule("Data Volume Drop",
"rows_synced < 50% of previous run",
"P2 High",
"Slack #data-quality"),
AlertRule("No Sync Activity (Flatline)",
"No sync logs for 2x schedule interval",
"P1 Critical",
"PagerDuty (Pipeline may be stuck)"),
AlertRule("Disk Usage High",
"Elasticsearch disk > 85%",
"P2 High",
"Slack #infra + Auto ILM Force Merge"),
]
print("=== Alert Rules ===")
for a in alerts:
print(f" [{a.name}] Severity: {a.severity}")
print(f" Condition: {a.condition}")
print(f" Channel: {a.channel}")เคล็ดลับ
- Filebeat: ใช้ Filebeat แทน Logstash สำหรับ Collection (เบากว่า)
- ILM: ตั้ง ILM ลบ Log เก่าอัตโนมัติ ประหยัด Storage
- JSON: ตั้ง Airbyte Output เป็น JSON ง่ายต่อ Parsing
- Index Pattern: แยก Index ต่อวัน ลบ/Archive ง่าย
- ElastAlert: ใช้ ElastAlert สำหรับ Complex Alert Rules
Airbyte Log Management คืออะไร
Sync Connector Platform Audit Log Container Pod ELK Elasticsearch Kibana Filebeat Dashboard Alert Full-text Search Centralized
