harvard-art-museum-data-pipeline

harvard-art-museum-data-pipeline is a data-engineering skill for building an end-to-end pipeline that collects, transforms, stores, and analyzes artifact data from the Harvard Art Museums API using Python, MySQL, and Streamlit. It targets the problem of converting a paginated public art API into a normalized relational dataset that can be queried and visualized, and it presents production-oriented patterns for ETL, SQL analytics, and interactive dashboards.

The documented architecture runs from the Harvard Art Museums API through a Python ETL layer into MySQL or TiDB Cloud, then to SQL analytics and a Streamlit dashboard with Plotly charts. Key components include an API client with secure key management, a three-table relational schema (artifactmetadata, artifactmedia, artifactcolors), 20-plus analytical SQL queries, batch inserts for load performance, and interactive visualizations. The skill supplies concrete code for the API client class, a paginated extractor with rate limiting, transform functions that flatten nested records into metadata, media, and color rows, and the SQL DDL for the schema. Configuration is done through environment variables (the Harvard API key and database host, user, password, and name), read securely rather than hardcoded.

This is effectively a catalogue-style entry that documents and points to the same upstream reference implementation (the Harvard-Artifacts-Collection-Data-Engineering-Analytics-App repository) and reproduces its patterns; it is one of several closely related Harvard-museum ETL skills in the ara.so Data Skills collection. It suits data engineers, Python developers, and learners who want a worked, reproducible example of API-to-dashboard data engineering, particularly around cultural-heritage or museum-collection data. Prerequisites include Python 3.8 or later, a MySQL or TiDB Cloud account, and a Harvard Art Museums API key.