harvard-artifacts-etl-analytics

harvard-artifacts-etl-analytics is a data-engineering skill that helps developers build end-to-end ETL pipelines and analytics dashboards on top of the Harvard Art Museums API using Python, SQL, and Streamlit. It solves the problem of turning a paginated, rate-limited public art API into a normalized, queryable dataset with an interactive front end, and it demonstrates real-world patterns for API integration, data transformation, relational schema design, and visualization.

The skill describes an architecture of API to ETL to SQL to Analytics to Visualization. It shows how to extract artifact data with pagination and rate limiting, transform nested JSON into three normalized tables (artifact metadata, media, and colors), load into MySQL or TiDB Cloud, run 20-plus predefined analytical SQL queries, and visualize results with interactive Plotly charts inside Streamlit. It provides concrete code patterns for the API client, the paginated extractor, and the transform functions, along with the SQL DDL for the schema and instructions for configuration via a .env file holding the Harvard API key and database credentials. Credential handling follows standard practice -- values are read from environment variables via python-dotenv rather than hardcoded.

In practice this is largely a catalogue-style entry that documents and points to an upstream reference implementation (the Harvard-Artifacts-Collection-Data-Engineering-Analytics-App repository on GitHub) and reproduces its key code patterns. It targets data engineers and Python developers learning or assembling museum-data or cultural-heritage analytics pipelines, and anyone who wants a worked example of API-to-dashboard ETL. The skill itself is authored by the ara.so Data Skills collection.