Skaffold Dev Batch Processing Pipeline — พัฒนา
Skaffold Batch Processing
Skaffold Dev Batch Processing Pipeline Kubernetes Job CronJob Argo Workflows Local Dev Loop CI/CD Production
| Tool | Use Case | Complexity | Scheduling | Best For |
|---|---|---|---|---|
| K8s Job | Single batch task | ต่ำ | Manual / CI trigger | Simple ETL, migration |
| K8s CronJob | Scheduled batch | ต่ำ | Cron expression | Daily reports, cleanup |
| Argo Workflows | Multi-step pipeline | กลาง | Cron + Event | Complex DAG pipelines |
| Spark on K8s | Big data processing | สูง | Airflow / Argo | TB-scale data |
| Tekton | CI/CD + batch | กลาง | Trigger-based | Build + process pipeline |
Skaffold Configuration
=== Skaffold + Batch Job Setup ===
อ่านเพิ่ม: LLM Inference vLLM Infrastructure as Code | SiamCafe Blog · อ่านเพิ่ม: LocalAI Self-hosted Event Driven Design | SiamCafe Blog · อ่านเพิ่ม: LLM Inference vLLM CQRS Event Sourcing | SiamCafe Blog
skaffold.yaml
apiVersion: skaffold/v4beta6
kind: Config
metadata:
name: batch-pipeline
build:
artifacts:
- image: batch-etl
docker:
dockerfile: Dockerfile.etl
- image: batch-report
docker:
dockerfile: Dockerfile.report
deploy:
kubectl:
manifests:
- k8s/etl-job.yaml
- k8s/report-cronjob.yaml
profiles:
- name: production
build:
tagPolicy:
sha256: {}
deploy:
kubectl:
manifests:
- k8s/production/*.yaml
k8s/etl-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: etl-daily
spec:
backoffLimit: 3
activeDeadlineSeconds: 3600
template:
spec:
restartPolicy: Never
containers:
- name: etl
image: batch-etl
command: ["python", "etl.py"]
resources:
requests: { cpu: "500m", memory: "1Gi" }
เนื้อหาเกี่ยวข้อง — ดูเพิ่มเติมเรื่อง Prometheus Federation Learning Path Roadmap
limits: { cpu: "2", memory: "4Gi" }
env:
- name: DB_URL
valueFrom:
secretKeyRef: { name: db-secret, key: url }
Commands
skaffold dev # Watch mode, auto rebuild+deploy on change
skaffold run # One-time build+deploy (CI/CD)
skaffold debug # Deploy with debug ports
แนะนำเพิ่มเติม — เรียนเทรดกับ iCafeForex
skaffold delete # Clean up deployed resources
skaffold render # Output rendered manifests
from dataclasses import dataclass
@dataclass
class SkaffoldCommand:
command: str
use_case: str
when: str
flags: str
commands = [
SkaffoldCommand("skaffold dev", "Watch mode, rebuild on change",
"Local development", "--port-forward --tail"),
SkaffoldCommand("skaffold run", "One-time build and deploy",
"CI/CD pipeline", "--tag=$GIT_SHA --profile=production"),
เนื้อหาเกี่ยวข้อง — แนะนำให้อ่าน Betteruptime Machine Learning Pipeline
SkaffoldCommand("skaffold debug", "Deploy with remote debug",
"Debugging batch job issues", "--port-forward"),
SkaffoldCommand("skaffold render", "Output K8s manifests",
"Review what will be deployed", "--output=rendered.yaml"),
SkaffoldCommand("skaffold delete", "Clean up resources",
"After testing, cleanup", ""),
SkaffoldCommand("skaffold build", "Build images only",
"CI build step", "--tag=$GIT_SHA --push"),
]
แนะนำเพิ่มเติม — ดูสัญญาณเทรดที่ XM Signal
Pipeline Orchestration
=== Argo Workflows for Complex Pipelines ===
argo-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: batch-pipeline
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: extract
template: etl-step
เนื้อหาเกี่ยวข้อง — แนะนำให้อ่าน LLM Inference vLLM Testing Strategy QA
arguments:
parameters: [{name: step, value: extract}]
- name: transform
template: etl-step
dependencies: [extract]
arguments:
parameters: [{name: step, value: transform}]
- name: load
template: etl-step
dependencies: [transform]
arguments:
parameters: [{name: step, value: load}]
- name: report
template: report-step
dependencies: [load]
- name: etl-step
container:
image: batch-etl
command: [python, etl.py, "{{inputs.parameters.step}}"]
- name: report-step
container:
image: batch-report
command: [python, generate_report.py]
@dataclass
class PipelineStage:
stage: str
image: str
resources: str
timeout: str
retry: int
เนื้อหาเกี่ยวข้อง — ดูเพิ่มเติมเรื่อง LlamaIndex RAG Machine Learning Pipeline
depends_on: str
pipeline = [
PipelineStage("Extract", "batch-etl",
"CPU: 1, Memory: 2Gi", "30 min", 3, "None"),
PipelineStage("Transform", "batch-etl",
"CPU: 2, Memory: 4Gi", "60 min", 2, "Extract"),
PipelineStage("Load", "batch-etl",
"CPU: 1, Memory: 2Gi", "30 min", 3, "Transform"),
PipelineStage("Report", "batch-report",
"CPU: 500m, Memory: 1Gi", "15 min", 2, "Load"),
PipelineStage("Notify", "batch-notify",
"CPU: 100m, Memory: 128Mi", "5 min", 1, "Report"),
]
เคล็ดลับ
- Dev Loop: ใช้ skaffold dev ลด Feedback Loop จาก 5 นาทีเหลือ 30 วินาที
- Spot: ใช้ Spot Node สำหรับ Batch Job ประหยัด 60-90%
- Idempotent: ออกแบบ Job ให้รันซ้ำได้ ป้องกัน Duplicate Processing
- Deadline: ตั้ง activeDeadlineSeconds ทุก Job ป้องกัน Runaway
- Cleanup: ตั้ง ttlSecondsAfterFinished ลบ Pod เก่าอัตโนมัติ
Skaffold คืออะไร
Google Kubernetes Dev Loop Build Deploy อัตโนมัติ Docker Helm Kustomize File Sync Watch Mode Debug CI/CD Pipeline
