harvard-art-museums-data-engineering-pipeline

This skill is a reference guide for building an end-to-end data engineering pipeline around the public Harvard Art Museums API. It demonstrates the full path from raw API data to an interactive analytics dashboard, solving the problem of how to structure a production-style ETL project: fetching paginated artifact data, normalizing nested JSON into relational tables, running SQL analytics, and visualizing results. It presents itself as a reference architecture accompanying an upstream sample application (the Harvard-Artifacts-Collection-Data-Engineering-Analytics-App repository).

The documented workflow covers API integration with pagination (querying the Harvard object endpoint with an API key), an ETL layer that transforms artifact records into separate metadata, media, and color tables using pandas, and a load step that writes to a MySQL or TiDB Cloud database with an auto-created schema. It layers on 20+ predefined analytical SQL queries and a Streamlit dashboard using Plotly charts for interactive exploration. Configuration is via a .env file holding the Harvard API key and standard database connection variables, and the app is run locally with Streamlit.

It targets data engineers, analysts, and learners who want a concrete, runnable example of a normalized-schema pipeline and dashboard. Because the primary document is essentially a tutorial and reference pointing to a separate cloneable repository, its depth in the skill itself is instructional rather than a self-contained tool; users get code snippets and setup steps rather than a packaged executable. Use cases include learning ETL patterns, prototyping museum/collection analytics, and adapting the schema design to other paginated JSON APIs.