harvard-art-museums-etl-pipeline

This skill is a guided reference for building an end-to-end ETL pipeline and analytics dashboard on top of the Harvard Art Museums API using Python, SQL, and Streamlit. It solves the practical data-engineering problem of pulling nested cultural-heritage data from a public API, normalizing it into relational tables, and turning it into interactive analytics. It largely documents and points to an upstream reference implementation repository, so it functions partly as a catalogue entry that walks through that project's patterns rather than shipping the full application itself.

The skill covers the full pipeline: extracting artifact data from the Harvard Art Museums API with pagination and rate limiting, transforming nested JSON into normalized tables for metadata, media, and colors, loading into MySQL or TiDB Cloud with foreign-key relationships, running predefined SQL analytics queries, and visualizing results through interactive Plotly charts in a Streamlit dashboard. It includes installation steps, a documented API-key and database-credential setup via a .env file and environment variables, a database connection helper, and SQL schema definitions for the artifact tables. Credentials are read from environment variables rather than hardcoded.

Target users are data engineers, analysts, and learners who want a realistic example of API-to-warehouse ETL and dashboarding. Typical use cases include extracting museum collection data with pagination, designing a relational schema for artifact metadata, building analytics queries over cultural data, and standing up a Streamlit visualization dashboard.