web-crawler

The web-crawler skill provides an integrated web scraping solution for extracting publicly available content from websites and social media platforms that are difficult to access through standard page fetching. It is designed to handle JavaScript-heavy pages, social media posts, transcripts, comments, profiles, and other publicly accessible data when conventional web retrieval methods are insufficient. The skill automatically routes requests to an appropriate backend, reducing the need for users to understand the underlying scraping implementation.

Its primary capabilities include retrieving public social media data across numerous supported platforms, extracting profiles, posts, videos, comments, transcripts, advertisements, trending content, and engagement-related information where supported by the underlying services. For ordinary web pages that cannot be fetched reliably, it provides a fallback page scraping mechanism focused on structured content extraction such as markdown, HTML, raw HTML, links, summaries, highlights, and constrained JSON-based extraction. The skill also recommends using standard web fetching for simple pages before invoking more specialized scraping backends.

This skill is intended for users who need structured extraction of public web or social media content, including developers, researchers, analysts, and automation workflows. Typical scenarios include downloading YouTube transcripts, collecting public social media information, extracting data from JavaScript-heavy pages, and accessing public content that blocks basic fetch requests while maintaining an automatic routing experience for the user.