realtime-cinema-data-engineering-pipeline

Realtime Cinema Data Engineering Pipeline is a tutorial skill for building an end-to-end streaming data platform using Apache Kafka, PostgreSQL, Apache Airflow, and Streamlit, organized around the Medallion (Bronze/Silver/Gold) architecture. It solves the problem of learning how streaming ingestion, warehousing, orchestration, and live visualization fit together by providing a concrete, runnable reference implementation that can process large volumes of synthetic events.

The skill documents cloning a companion GitHub repository and starting the infrastructure with Docker Compose. It explains the Bronze layer (raw JSONB events), Silver layer (normalized 3NF customer/movie/showtime/transaction tables), and Gold layer (materialized analytics views), and provides Python code for a Kafka producer that generates synthetic cinema ticket-sale events with Faker, a Kafka consumer that writes raw events into the Bronze table via psycopg2, and an Airflow DAG that orchestrates the Bronze→Silver→Gold ELT transformations. Connection details for Kafka and PostgreSQL are read from environment variables with local development defaults.

Target users are aspiring and practicing data engineers who want hands-on experience with production-style streaming ELT patterns and the Medallion architecture. It is instructional in nature—value lies in the documented architecture and example code pointing to an upstream repo—and the operations involved (spinning up local containers, generating fake data, writing to a local warehouse) are self-contained and non-destructive to any external systems.