data-scraper-agent
Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
Security Assessment
Detected risks:
About data-scraper-agent
The data-scraper-agent is an AI-powered tool designed to automate the collection of data from any public source. It allows users to scrape websites, APIs, RSS feeds, and other publicly accessible endpoints on a configurable schedule, enabling the monitoring of jobs, prices, news, GitHub repositories, sports scores, and other events or listings. The agent leverages Python-based scraping libraries and a free LLM (Gemini Flash) to enrich collected data, summarize, classify, or score items, and then store the results in platforms like Notion, Google Sheets, or Supabase. It operates fully on GitHub Actions, making it a cost-free solution for automated data collection and monitoring without requiring external infrastructure.
FAQ
How do I configure the data-scraper-agent for my own sources?
You can configure the agent using the provided 'config.yaml' file, specifying your target URLs or APIs, the fields you want to extract, how often to run the scraper, and the storage destination. Additional context or user-specific criteria can be added in 'profile/context.md'.
Which AI model does the agent use for enrichment?
The agent uses the free Gemini Flash LLM via REST API for enrichment tasks such as scoring, summarizing, and classifying items. It supports a model fallback chain to handle quota limits automatically.
Can the agent handle JavaScript-rendered websites?
Yes. For sites that require JavaScript execution, the agent uses the free Playwright library to render pages before scraping the content.
Where can I store the collected data?
Collected data can be stored in Notion, Google Sheets, Supabase, or a local JSON file. The choice depends on your preference and review workflow.
Does the agent require paid hosting?
No. The agent runs entirely on GitHub Actions for free, using a schedule to automate scraping, enrichment, and storage without additional infrastructure costs.
Install data-scraper-agent
Quick Setup:
- Copy the skill folder to
.claude/skills/ - Claude will automatically detect and use the skill
Repository
affaan-m/everything-claude-code