harvard-artifacts-collection-data-engineering-analytics

harvard-artifacts-collection-data-engineering-analytics is a tutorial-style skill that walks through building an end-to-end data engineering and analytics application on top of the Harvard Art Museums API. It solves the problem of demonstrating a complete ETL-to-dashboard workflow by extracting artifact data from the API, transforming nested JSON into structured relational tables, loading it into a SQL database (MySQL or TiDB Cloud), and visualizing it with Streamlit and Plotly.

The skill provides concrete Python code: an extract step that paginates the API with rate limiting, a transform step that flattens artifacts into metadata, media, and color DataFrames, and a load step that creates a normalized schema (artifactmetadata, artifactmedia, artifactcolors with foreign keys) via mysql-connector-python. Configuration uses a .env file for the Harvard API key and database host, port, user, password, and name, following standard credential-setup practice with python-dotenv. It advertises 20+ analytical SQL queries and interactive visualizations, and points to an upstream companion repository for the full application.

It targets data engineering learners, analytics developers, and anyone wanting a reference implementation for API-driven ETL and dashboarding. Typical use cases include learning ETL pipeline design, practicing relational schema modeling from nested JSON, and standing up a Streamlit analytics dashboard. Because the skill largely mirrors and links to an external GitHub project, it functions partly as a guided catalogue entry into that upstream repo rather than a fully self-contained tool.