Part of Run GTM Workflows With AI Agent Skills
Marketing Blueprint
Safety checkedExtract Competitor & Research Data From Web
Scrapes structured data from websites, single-page applications, and APIs. Includes pre-scrape API checks, embedded JSON detection, and evasion handling for protected sites.
What problem does this solve?
Extracts research data from competitor websites and target companies without manual copying by running a pre-scrape analysis to check for public APIs or embedded JSON first, then selecting the right tool (HTTPX for static HTML, Playwright for SPAs, beamsplitter for protected sites) and executing the extraction with rate limiting, retry logic, and validation. Instead of clicking through pages manually, you describe what you want ('scrape pricing from competitor.com into a table' or 'pull all job postings from our competitor'), and the skill handles the analysis, tool selection, and execution, returning clean structured data ready for review.
How does it work?
You request data extraction: 'Scrape competitor pricing.' The skill runs a pre-scrape analysis gate: (1) checks for a public API (Google API docs), (2) inspects network traffic for hidden JSON endpoints, (3) searches for embedded JSON in the raw HTML, (4) reads robots.txt for disallowed paths and crawl delays, (5) classifies the target (static HTML, JS-rendered SPA, behind auth, anti-bot protected). Based on classification, it selects the appropriate tool (BeautifulSoup for static, Playwright for SPAs, curl for APIs). It writes an extraction script with operational basics (rate limiting, user-agent headers, retry logic), tests on 5 items first, validates output (schema, nulls, encoding), and delivers the full dataset as JSON, CSV, or your requested format. Output: clean structured data ready to import or analyze.
Included Skills
No skills in this group
What's required to run this?
Uses Playwright for SPA rendering (bundled with Node 18+), BeautifulSoup via Python (requires Python 3.8+), HTTPX for HTTP (built in), and optional curl for API discovery. Reference files (static_scraping.md, dynamic_scraping.md, api_discovery.md, evasion_mobile.md, operational_basics.md) guide tool selection and implementation. No API keys required for basic scraping. Bright Data or ScrapingBee may be needed for Cloudflare Enterprise or aggressive anti-bot (not bundled, user-supplied).
Key Features
Pre-Scrape API Check
Searches for public APIs before writing scrapers. Uses API if available; it's always faster and more reliable than HTML parsing.
Embedded JSON Detection
Scans raw HTML source for embedded data (JSON-LD, Next.js hydration payloads, Redux state) before classifying as a spa. Extracts with regex when found.
Tool Selection Router
Classifies target (static vs SPA vs API vs protected) and picks the right tool: HTTPX plus BeautifulSoup, Playwright, curl, or evasion techniques.
Operational Basics
Every script includes rate limiting, user-agent headers, retry logic with backoff, and incremental saves to prevent data loss on timeout.
Block Escalation & Evasion
When 403/CAPTCHA/blocks occur, diagnoses the block type and applies fixes: add headers, switch to tls-client, rotate proxies, or escalate to paid scraping service.
About This Blueprint
- Industry
- Technology
More Blueprints to explore
⭐ Ask Nicely : Find your next reviewers
Pick promoters from your CRM, remove accounts with open escalations, overdue billing or a recent review, then create the G2 review campaign after you approve the list, send method and any incentive.
G2CRMSurvey or CS platform+4 moreCharlie Albuery
🤝 You Said We Did : Align your roadmap to real user feedback
Group recent G2 reviews into themes, check each theme against tickets in your work tracker, and return a ranked one-page brief that lists the themes with no ticket first.
G2Work trackerResearch repository+3 moreCharlie Albuery
👀 Wandering Eyes : Identify churn risk early
Match accounts renewing soon in your CRM to G2 buyer intent on competitors and your category, check usage, support and meetings, then rank the accounts and brief the CSM. Runs as a weekly sweep or on one named account.
G2CRMProduct usage analytics+4 moreCharlie Albuery