data-engineering-medallion-pipeline

This skill teaches an AI agent to stand up and operate a complete, local end-to-end data engineering pipeline built on the Medallion Architecture (Bronze, Silver, Gold layers). It solves the problem of wiring together a modern open-source data stack from scratch, combining MinIO for S3-compatible object storage, Airbyte for ingestion, PostgreSQL as the warehouse, DBT for transformations, Apache Airflow for orchestration, and Prometheus/Grafana for monitoring. The whole stack is designed to run locally via Docker Compose, following an ELT pattern where raw immutable data lands in Bronze, gets cleaned and typed in Silver, and is aggregated into business-ready KPIs in Gold.

The skill is driven largely by a Makefile that exposes commands for infrastructure lifecycle (setup, start, stop, status, logs), DBT operations (run, test, docs, snapshots for SCD Type 2), data movement (uploading sample data to MinIO, triggering Airflow DAGs), and development chores (lint, format with black, validate). It documents concrete DBT model patterns for each layer, including extracting JSONB from Airbyte raw tables, deduplicating with window functions, casting types, and building aggregated metrics like product profit margins. Airflow DAGs orchestrate the bronze-ingestion, silver-transformation, and gold-aggregation stages.

It targets data engineers, analytics engineers, and platform learners who want a reproducible reference implementation of a data lakehouse without provisioning cloud infrastructure. Use cases include learning medallion architecture, prototyping ELT flows, implementing DBT data-quality tests, and demonstrating orchestrated pipelines. It requires Docker 20.10+, Docker Compose 2.0+, and at least 8GB of RAM, and clones an upstream companion repository to obtain the actual project code.