WebClaw MCP Server

Source-backed facts for citing this resource, derived directly from the registry — also available as plain text for AI assistants.

Canonical URL

https://heyclau.de/entry/mcp/webclaw-mcp-server

Source URLs

https://raw.githubusercontent.com/0xMassi/webclaw/main/README.md, https://github.com/0xMassi/webclaw

Brand

WebClaw

Brand domain

webclaw.io

Brand asset source

brandfetch

Safety notes

WebClaw MCP Server can scrape one URL, crawl same-origin links, map URLs, batch multiple URLs, extract structured data, summarize pages, compare snapshots, and extract brand metadata., Scraping, crawling, search, research, and hosted fallback can trigger target-site rate limits, bot protections, legal restrictions, or terms-of-service limits., The server validates public HTTP URLs and rejects private or internal destinations, but operators should still avoid fetching internal, secret, customer-specific, or credential-bearing URLs., Cookie-bearing requests, proxies, and browser profiles can expose authenticated sessions or route traffic through external infrastructure., Setting `WEBCLAW_API_KEY` enables hosted API fallback, which can send target URLs and page context to the WebClaw service., Structured extraction and summarization may use configured LLM providers; review provider data handling before enabling those tools with sensitive pages.

Privacy notes

URLs, fetched page content, crawl results, extracted structured data, snapshots, brand metadata, cookies, proxy settings, API keys, and LLM prompts can be exposed to the MCP client., Crawled pages may include personal data, customer content, non-public links, access tokens embedded in URLs, or site-specific identifiers., Diff snapshots and exported JSON can preserve page content longer than expected; store and delete them according to the workflow's retention needs., Hosted fallback, proxy providers, search/research APIs, and LLM providers may observe target URLs, headers, cookies, prompts, and extracted content when enabled., Redact sensitive URLs and extracted data before sharing MCP transcripts, crawl results, logs, or generated RAG context.

Author

WebClaw

Submitted by

oktofeesh1

Claim status

unclaimed

Last verified

2026-06-06

• URLs, fetched page content, crawl results, extracted structured data, snapshots, brand metadata, cookies, proxy settings, API keys, and LLM prompts can be exposed to the MCP client.
• Crawled pages may include personal data, customer content, non-public links, access tokens embedded in URLs, or site-specific identifiers.
• Diff snapshots and exported JSON can preserve page content longer than expected; store and delete them according to the workflow's retention needs.
• Hosted fallback, proxy providers, search/research APIs, and LLM providers may observe target URLs, headers, cookies, prompts, and extracted content when enabled.
• Redact sensitive URLs and extracted data before sharing MCP transcripts, crawl results, logs, or generated RAG context.

• Node.js for the `create-webclaw` installer, or Rust/Cargo when building the MCP binary from source.
• Review of AGPL-3.0 obligations before redistributing or operating modified server versions.
• Review of target-site terms, robots expectations, rate limits, and scraping permissions.
• Optional `WEBCLAW_API_KEY` only when hosted API fallback is acceptable for bot-protected or JavaScript-heavy pages.
• Optional proxy, cookie, or LLM-provider configuration only for workflows that explicitly require it.

Show that WebClaw MCP Server is listed on HeyClaude. Paste this Markdown into your README — it renders the badge and links back to this page.

[![Listed on HeyClaude](https://heyclau.de/badge/mcp/webclaw-mcp-server.svg)](https://heyclau.de/entry/mcp/webclaw-mcp-server)

WebClaw MCP Server side by side with 3 alternatives on trust, install, platform support, and disclosed safety notes — all from reviewed registry metadata.

Field	WebClaw MCP Server Local-first MCP server for web scraping, crawling, URL mapping, batch extraction, structured extraction, summarization, diffing, brand extraction, and optional hosted API fallback for bot-protected pages. Open dossier	Omnisearch MCP Server MCP server that unifies web search, AI answer, GitHub search, and web extraction providers including Tavily, Brave, Kagi, Exa, Linkup, Firecrawl, and GitHub behind four consolidated tools. Open dossier	open-webSearch MCP Server No-key multi-engine MCP server, CLI, and local daemon for web search and public web content retrieval across engines such as Bing, DuckDuckGo, Brave, Exa, Baidu, CSDN, Juejin, Startpage, and Sogou. Open dossier	AgentQL MCP Server AgentQL MCP server for extracting structured JSON from public webpages using a URL and natural-language extraction prompt. Open dossier
Next steps	Open dossierCopy installCopy configAPI JSONOpen LLMMCP feedOpen sourceNewsletterClaim listing	Open dossierCopy installCopy configAPI JSONOpen LLMMCP feedOpen sourceNewsletterClaim listing	Open dossierCopy installCopy configAPI JSONOpen LLMMCP feedOpen sourceNewsletterClaim listing	Open dossierCopy installCopy configAPI JSONOpen LLMMCP feedOpen sourceNewsletterClaim listing
Trust	Review first	Review first	Review first	Review first
Review status	Not reviewed	Not reviewed	Not reviewed	Not reviewed
Package trust	Package not verified	Package not verified	Package not verified	Package not verified
Source provenance	Source-backed	Source-backed	Source-backed	Source-backed
Submitter	oktofeesh1	oktofeesh1	oktofeesh1	oktofeesh1
Install risk	Review first	Review first	Review first	Review first
Notes	Safety ✓ Privacy ✓	Safety ✓ Privacy ✓	Safety ✓ Privacy ✓	Safety ✓ Privacy ✓
Brand	WebClaw	—	—	AgentQL
Category	mcp	mcp	mcp	mcp
Source	Source-backed	Source-backed	Source-backed	Source-backed
Author	WebClaw	Scott Spence	Aas-ee	AgentQL
Added	2026-06-06	2026-06-06	2026-06-06	2026-06-06
Platforms	Claude CodeClaude Desktop	Claude CodeClaude Desktop	Claude CodeClaude Desktop	Claude CodeClaude Desktop
Harness	Claude CodeClaude Desktop	Claude CodeClaude Desktop	Claude CodeClaude Desktop	Claude CodeClaude Desktop
Source repo	1.8k repo stars	—	—	—
Safety notes	✓WebClaw MCP Server can scrape one URL, crawl same-origin links, map URLs, batch multiple URLs, extract structured data, summarize pages, compare snapshots, and extract brand metadata. Scraping, crawling, search, research, and hosted fallback can trigger target-site rate limits, bot protections, legal restrictions, or terms-of-service limits. The server validates public HTTP URLs and rejects private or internal destinations, but operators should still avoid fetching internal, secret, customer-specific, or credential-bearing URLs. Cookie-bearing requests, proxies, and browser profiles can expose authenticated sessions or route traffic through external infrastructure. Setting `WEBCLAW_API_KEY` enables hosted API fallback, which can send target URLs and page context to the WebClaw service. Structured extraction and summarization may use configured LLM providers; review provider data handling before enabling those tools with sensitive pages.	✓Omnisearch MCP Server can query multiple third-party search, AI answer, GitHub search, extraction, scraping, crawling, summarization, and content-processing providers. Firecrawl crawl, scrape, map, extract, and action modes can interact with websites and may trigger robots.txt, TOS, rate-limit, or authentication concerns. GitHub search depends on the configured token scope and can reveal private code, repositories, users, or organization metadata when a broad token is used. Large result mode can write response content to files; review where those files are stored before using it with private or regulated data. Require confirmation before crawling sites, extracting private URLs, using authenticated providers on sensitive targets, or running broad GitHub searches against private organizations.	✓open-webSearch can send search queries to multiple public engines and fetch public web pages, GitHub READMEs, CSDN articles, Juejin articles, and other supported targets. Search and fetch results can be incomplete, stale, rate-limited, blocked, region-dependent, or affected by search-engine ranking and scraping protections. Respect each target site's terms and robots expectations; do not use the server for abusive scraping, credentialed browsing, paywall bypass, or personal-data harvesting. The optional local daemon exposes HTTP endpoints for local tooling; keep it bound to trusted local interfaces and do not treat it as a public internet API. Proxy settings, Playwright WebSocket/CDP endpoints, and reused browser sessions can route traffic or cookies through external systems; configure them deliberately. Leave TLS verification enabled unless a specific target has a broken certificate chain and the risk has been accepted.	✓AgentQL MCP exposes one tool, `extract-web-data`, that sends a target URL and natural-language extraction prompt to the AgentQL API. The tool is intended for public webpages; do not use it to bypass access controls, scrape private pages, evade paywalls, or extract data where automated collection is prohibited. Web extraction can still trigger target-site rate limits, legal restrictions, robots guidance, or terms-of-service concerns. The source implementation uses AgentQL's query-data endpoint with fast mode, no screenshot capture, no scroll-to-bottom behavior, and no local browser cookies. Treat extracted output as untrusted web data that may include errors, stale content, ads, tracking text, or prompt-injection attempts.
Privacy notes	✓URLs, fetched page content, crawl results, extracted structured data, snapshots, brand metadata, cookies, proxy settings, API keys, and LLM prompts can be exposed to the MCP client. Crawled pages may include personal data, customer content, non-public links, access tokens embedded in URLs, or site-specific identifiers. Diff snapshots and exported JSON can preserve page content longer than expected; store and delete them according to the workflow's retention needs. Hosted fallback, proxy providers, search/research APIs, and LLM providers may observe target URLs, headers, cookies, prompts, and extracted content when enabled. Redact sensitive URLs and extracted data before sharing MCP transcripts, crawl results, logs, or generated RAG context.	✓Provider API keys, GitHub tokens, search queries, URLs, extracted page content, crawled site maps, scraped text, summaries, AI answers, GitHub search results, and large-result files can be exposed to the MCP client. Search and extraction providers may log queries, URLs, source content, account identifiers, usage metadata, and generated responses according to their own policies. Web extraction can capture personal data, paywalled content, internal URLs, or proprietary documents if supplied by the user or available to an authenticated provider. GitHub search output may include repository names, code snippets, file paths, commit metadata, and organization details. Keep provider credentials in local MCP configuration only, and avoid sharing transcripts that contain private queries, URLs, source text, or tokens.	✓Search queries, fetched URLs, result titles, snippets, article content, proxy URLs, browser endpoints, and fetched page text can be exposed to MCP clients, logs, and model context. Live search engines and fetched websites may observe queries, IP address, proxy exit, browser fingerprints, cookies, timing, and request headers. Playwright fallback or CDP reuse can expose browser state, existing cookies, logged-in sessions, or verification state to fetched pages. The project includes public URL validation and private-network target protections, but operators should still avoid fetching internal, secret, or customer-specific URLs. Redact sensitive search terms and downloaded page content before sharing transcripts or logs.	✓Target URLs, extraction prompts, API key-authenticated requests, and extracted structured data are sent to AgentQL's API. Extracted data can include personal data, copyrighted content, customer information, job postings, prices, social content, or other third-party material. AGENTQL_API_KEY should stay out of prompts, issues, logs, screenshots, and committed configuration files. Claude transcripts and downstream reports may retain extracted data, so avoid collecting information that is not approved for the model session.
Prerequisites	Node.js for the `create-webclaw` installer, or Rust/Cargo when building the MCP binary from source. Review of AGPL-3.0 obligations before redistributing or operating modified server versions. Review of target-site terms, robots expectations, rate limits, and scraping permissions. Optional `WEBCLAW_API_KEY` only when hosted API fallback is acceptable for bot-protected or JavaScript-heavy pages.	Node.js 22 or newer for the published npm package. At least one provider API key for Tavily, Kagi, Brave Search, GitHub, Exa, Linkup, or Firecrawl. Optional `FIRECRAWL_BASE_URL` when using a self-hosted Firecrawl instance. Optional `OMNISEARCH_LARGE_RESULT_MODE` choice for file or inline handling of large results.	Node.js 18 or newer for the published npm package. Review of search-engine terms, scraping limits, rate limits, and allowed use for the target websites. Optional proxy configuration only when live search or fetch traffic must route through an approved proxy. Optional Playwright or browser endpoint setup only when request-based search/fetch is insufficient.	Node.js and npx available to the MCP client runtime. AgentQL API key from the AgentQL developer portal. Approved list of public webpages or domains Claude may query. Review of target site terms, robots guidance, rate limits, and data-use rules before extraction.
Install	npx create-webclaw Copy install	npx mcp-omnisearch Copy install	npx -y open-websearch@latest Copy install	npx -y agentql-mcp Copy install
Config	Manual-only setup: npx create-webclaw Copy config	{ "mcpServers": { "omnisearch": { "command": "npx", "args": [ "mcp-omnisearch" ], "env": { "TAVILY_API_KEY": "your-tavily-key", "KAGI_API_KEY": "your-kagi-key", "BRAVE_API_KEY": "your-brave-key", "GITHUB_API_KEY": "your-github-token", "EXA_API_KEY": "your-exa-key", "LINKUP_API_KEY": "your-linkup-key", "FIRECRAWL_API_KEY": "your-firecrawl-key" }, "type": "stdio" } } } Copy config	{ "mcpServers": { "open-websearch": { "command": "npx", "args": [ "-y", "open-websearch@latest" ], "env": { "MODE": "stdio", "DEFAULT_SEARCH_ENGINE": "duckduckgo", "ALLOWED_SEARCH_ENGINES": "duckduckgo,startpage,bing", "NPM_CONFIG_IGNORE_SCRIPTS": "true" }, "type": "stdio" } } } Copy config	{ "mcpServers": { "agentql": { "command": "npx", "args": ["-y", "agentql-mcp"], "env": { "AGENTQL_API_KEY": "<your-agentql-api-key>" } } } } Copy config
Citations	Source repositorygithub.com 2026-07-21T07:17:01+00:00 Documentationraw.githubusercontent.com Submitted by oktofeesh12026-06-06 Source methodology →	Source repositorygithub.com 2026-07-21T07:17:01+00:00 Documentationraw.githubusercontent.com Submitted by oktofeesh12026-06-06 Source methodology →	Source repositorygithub.com 2026-07-21T07:17:01+00:00 Documentationraw.githubusercontent.com Submitted by oktofeesh12026-06-06 Source methodology →	Source repositorygithub.com 2026-07-21T07:17:01+00:00 Documentationgithub.com Submitted by oktofeesh12026-06-06 Source methodology →
Claim	Unclaimed	Unclaimed	Unclaimed	Unclaimed