ShadowCrawl

ShadowCrawl (CortexScout) is a self-hostable Rust module that provides advanced web search, crawling, structured extraction, and browser automation capabilities, specifically optimized for AI agents and LLM workflows. It integrates with Model Context Protocol (MCP) for agent orchestration, supports robust anti-bot mechanisms, stateful browser automation, Human-in-the-Loop fallback, and is designed for efficient, repeatable deep web research and extraction tasks. Ideal for developers and automation teams building agentic or autonomous research pipelines or workflows.