12 skills found
Provides guidance for migrating Apache Airflow DAGs in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers migration to Airflow 2.11.1 (MSAA Gen 2 and 3) and Airflow 3 (MSAA Gen 3), including environment inspection, GCS download/upload and scanning patterns for breaking changes. Use when migrating the DAG code to newer Airflow version. Don't use when checking DAG run failures unrelated to code migration.
Build end-to-end real-time data pipelines with Kafka, PostgreSQL, Airflow, and Streamlit using Medallion Architecture for streaming analytics.
End-to-end data engineering pipeline using MinIO, Airbyte, PostgreSQL, DBT, and Airflow with medallion architecture (Bronze/Silver/Gold layers)
Comprehensive study guide covering data engineering concepts, tools, and best practices for learning and reference
Incremental Load Setup - Auto-activating skill for Data Pipelines.Triggers on: incremental load setup, incremental load setupPart of the Data Pipelines skill category.
File Format Converter - Auto-activating skill for Data Pipelines.Triggers on: file format converter, file format converterPart of the Data Pipelines skill category.
Schema Validator - Auto-activating skill for Data Pipelines.Triggers on: schema validator, schema validatorPart of the Data Pipelines skill category.
Kafka Stream Processor - Auto-activating skill for Data Pipelines.Triggers on: kafka stream processor, kafka stream processorPart of the Data Pipelines skill category.
Pipeline Monitoring Setup - Auto-activating skill for Data Pipelines.Triggers on: pipeline monitoring setup, pipeline monitoring setupPart of the Data Pipelines skill category.
Data Lineage Tracker - Auto-activating skill for Data Pipelines.Triggers on: data lineage tracker, data lineage trackerPart of the Data Pipelines skill category.
Data Catalog Updater - Auto-activating skill for Data Pipelines.Triggers on: data catalog updater, data catalog updaterPart of the Data Pipelines skill category.
Sql Transform Helper - Auto-activating skill for Data Pipelines.Triggers on: sql transform helper, sql transform helperPart of the Data Pipelines skill category.
