harvard-art-museums-etl-analytics

harvard-art-museums-etl-analytics is a data-engineering skill that guides AI coding agents and developers in building end-to-end ETL and analytics applications on the Harvard Art Museums API with Python, SQL, and Streamlit. It solves the problem of transforming a paginated public museum API into structured, analyzable data with an interactive dashboard, and demonstrates real-world ETL, SQL analytics, and visualization patterns following an API to ETL to SQL to Analytics to Visualization architecture.

The skill documents API integration with pagination and rate limiting (noting the Harvard API's daily request budget), an ETL pipeline that extracts artifact metadata, media, and color data and transforms nested JSON into relational form, SQL storage in MySQL or TiDB Cloud with a proper relational schema, an analytics engine of 20-plus predefined SQL queries, and interactive Streamlit dashboards using Plotly. It provides concrete code for a schema-creation routine, a fetch function that only pulls artifacts with images, a bounded extractor that stops at a maximum record count, and transform functions producing separate dataframes. Configuration uses a .env file holding the Harvard API key and database host, port, user, password, and name, all read through environment variables rather than embedded in code.

Like its sibling entries, this is largely a catalogue-style walkthrough that documents and points to the same upstream reference implementation (the Harvard-Artifacts-Collection-Data-Engineering-Analytics-App repository) in the ara.so Data Skills collection. It suits data engineers, Python developers, and learners who want a reproducible example of cultural-heritage data engineering from API ingestion through to dashboard visualization.