A Claude skill that fetches web pages, strips boilerplate with Mozilla Readability, respects robots.txt, removes duplicate content, and turns the result into structured summaries.
Open the source and read safety notes before installing.
Source-backed facts for citing this resource, derived directly from the registry — also available as plain text for AI assistants.
Decision playbook
Signals are comparatively strong, but you should still validate source, privacy posture, and package provenance for your environment.
0
96
—
No baseline selected
No major trust-signal divergence detected in the current selection.
Confirm ownership and provenance before trusting install instructions.
Source link availableRequired
Open the canonical repository and verify ownership.
Source provenance statusRequired
Marked as first-party.
Metadata reviewed
Registry metadata indicates a reviewed listing.
Validate risk disclosures before installation or API wiring.
Safety notes presentRequired
Review the listed safety guidance before running commands.
Privacy notes presentRequired
Review data handling notes before connecting accounts or secrets.
Trust level risk gateRequired
Trust level does not block evaluation.
Check package metadata and artifact integrity signals.
Install payload available
Install or copy payload is available for review.
Package verification flag
Package marked verified.
Checksum metadata
SHA-256 hash is present.
Use compare context to validate trade-offs before adoption.
Compare tray has multiple entries
Add at least one more entry to compare trust differences.
Baseline comparison available
No baseline peer selected yet.
Diverging trust signals identified
No major trust-signal divergence found.
Setup at a glance
Copy-ready — paste the snippet to get started.
Adoption plan
Current risk score 0/100. Use staged verification before broader rollout.
Validate source and review signals before any execution.
Confirm source provenanceRequired
Source URL/provenance metadata is present.
Confirm metadata review state
Listing has review metadata.
Verify install payload
Install/config payload exists and can be inspected.
Confirm safety, privacy, and package integrity signals.
Review safety notesRequired
Safety notes are present.
Review privacy notesRequired
Privacy notes are present.
Verify package integrity metadata
Package verification/checksum metadata is available.
Adopt in controlled steps based on the selected plan.
Run in isolated sandbox firstRequired
Use a constrained sandbox and observe behavior across multiple tasks.
Roll out graduallyRequired
Roll out to a small cohort before wider usage.
Set monitoring and fallback
Define rollback path and monitor errors after adoption.
Evidence readiness
Required evidence gates are covered (6/6 signals complete).
Source repository/provenance is listed.
Required in this preset
Review metadata is present.
Required in this preset
Safety notes are present.
Required in this preset
Privacy notes are present.
Optional in this preset
Package integrity metadata is present.
Optional in this preset
Install payload is available.
Required in this preset
Required evidence gates are covered for this preset.
Decision timeline
6/6 steps complete with no blocking gaps for this preset.
triage
Source/provenance metadata is available.
triage
Review metadata is available.
verify
Safety notes are available.
verify
Privacy notes are available.
verify
Package integrity metadata is available.
rollout
Install payload is available.
No required blockers for this timeline preset.
Prerequisite readiness
6 prerequisites to line up before setup.
Safety & privacy surface
2 safety and 1 privacy notes across 3 risk areas. Review closely: credentials & tokens, network access.
| Platform | Support | Install path |
|---|---|---|
| claude-code | Native | .claude/skills/<skill-name>/SKILL.md |
| codex | Native | .agents/skills/<skill-name>/SKILL.md |
| windsurf | Native | .windsurf/skills/<skill-name>/SKILL.md |
| gemini | Native | .gemini/skills/<skill-name>/SKILL.md or .agents/skills/<skill-name>/SKILL.md |
| cursor | Adapter | .cursor/rules/<skill-name>.mdc |
| cli | Manual | AGENTS.md or tool-specific context file |
import { JSDOM } from 'jsdom';
import { Readability } from '@mozilla/readability';
import fetch from 'node-fetch';
const html = await (await fetch('https://example.com')).text();
const doc = new JSDOM(html, { url: 'https://example.com' });
const article = new Readability(doc.window.document).parse();
console.log(article.title);
console.log(article.textContent);Claude can crawl websites, extract content from web pages, clean HTML to readable text, respect robots.txt, and generate structured summaries or documentation from web content. Perfect for research, competitive analysis, and content aggregation.
SKILL.md.SKILL.md content as reusable workflow instructions..gemini/skills/<skill-name>/SKILL.md or .agents/skills/<skill-name>/SKILL.md where supported..cursor/rules/*.mdc adapter for project rules.Required:
What Claude handles:
Prompt: "Extract the main content from this webpage: https://example.com/article Convert to clean Markdown and save as article.md"
Claude will:
Prompt: "Crawl all pages linked from this site: Start URL: https://docs.example.com Only crawl pages under /docs/ Extract content from each page Save as individual Markdown files Max 50 pages"
Claude will:
Prompt: "Crawl these 10 blog posts and:
Claude will:
Prompt: "Download sitemap from https://example.com/sitemap.xml and:
Claude will:
"Analyze competitor website:
1. Crawl main site (max 20 pages)
2. Extract: services, pricing mentions, features
3. Identify key messaging themes
4. Create structured comparison report
Focus on product/service pages."
"Create local mirror of documentation:
1. Start at https://docs.example.com
2. Crawl all /docs/* pages
3. Download images referenced
4. Convert to Markdown
5. Preserve link structure
6. Generate offline-browseable site"
"Gather research from these 20 URLs:
1. Extract main content from each
2. Identify key findings and quotes
3. Extract citations and references
4. Group by topic/theme
5. Create annotated bibliography
Output as structured Markdown."
"Monitor this webpage for changes:
1. Fetch current version
2. Extract main content
3. Compare with version from last week
4. Highlight what changed
5. Generate change report"
Issue: Getting blocked or rate limited Solution: "Slow down to 1 request per 10 seconds" and "Add random delays between requests"
Issue: Content not extracting correctly Solution: "Show me the raw HTML first" then identify CSS selectors for main content
Issue: JavaScript content not loading Solution: "Use Playwright to render JavaScript" or "Wait 5 seconds for content to load"
Issue: Too many pages being crawled Solution: Set limits: "Max 50 pages" or "Only crawl 2 levels deep" or "Stick to /docs/* path"
Issue: Images/assets not downloading Solution: "Download all images referenced in articles" or provide specific asset types needed
Issue: Different page structures Solution: Provide multiple CSS selectors: "Try article.content, then div.post-body, then main"
Important: Always respect:
Use cases: Research, archival, accessibility, personal use Prohibited: Spam, unauthorized scraping, data theft, ToS violations
Show that Website Crawler + Summarizer Skill is listed on HeyClaude. Paste this Markdown into your README — it renders the badge and links back to this page.
[](https://heyclau.de/entry/skills/website-crawler-summarizer)Website Crawler + Summarizer Skill side by side with its closest alternative on trust, install, platform support, and disclosed safety notes — all from reviewed registry metadata.
2 trust signals differ across this comparison (Package trust, Source provenance).
| Field | A Claude skill that fetches web pages, strips boilerplate with Mozilla Readability, respects robots.txt, removes duplicate content, and turns the result into structured summaries. Open dossier | Write and maintain reliable end-to-end tests with Playwright — Microsoft's browser automation library that auto-waits for actionable elements and runs the same suite against Chromium, Firefox, and WebKit. Open dossier |
|---|---|---|
| Next steps | ||
| Trust | ||
| Review status | ReviewedMaintainer reviewed | ReviewedMaintainer reviewed |
| Package trustDiffers | Package verified2025-10-15 | Package verified2025-10-16 |
| Source provenanceDiffers | Source-backed | No submission link |
| Submitter | — | — |
| Install risk | Review first | Low risk |
| Notes | Safety ✓ Privacy ✓ | Safety ✓ Privacy ✓ |
| Brand | — | — |
| Category | skills | skills |
| Source | First-party | First-party |
| Author | JSONbored | JSONbored |
| Added | 2025-10-15 | 2025-10-16 |
| Platforms | ||
| Harness | ||
| Source repo | — | — |
| Safety notes | ✓Executes network requests to crawl third-party websites and installs scraping libraries (Playwright, BeautifulSoup) on first use; respect robots.txt, rate limits, and each site's terms of service, and review target URLs before running. Crawled pages are written to local files (Markdown, JSON, CSV); review output paths so the skill does not overwrite existing files. | ✓`npx playwright install --with-deps` downloads browser binaries (~500 MB for Chromium, Firefox, and WebKit) from Playwright's CDN and on Linux installs system-level packages. Review your network and environment policies before running in restricted infrastructure. Generated tests have full browser-level access to any URL they target. Review test scripts before running them against authenticated sessions or production environments — Playwright can submit forms, click buttons, and make network requests as a real user. |
| Privacy notes | ✓Fetches and stores content from external websites locally and may use site cookies or auth tokens to reach gated pages; do not crawl private or authenticated data without authorization, and keep any credentials out of saved output. | ✓Playwright runs entirely on your local machine. Traces, screenshots, and HTML test reports are written to local disk only and are not uploaded to Microsoft or any external service. Trace files and failure screenshots may capture sensitive content from the application under test (form data, rendered PII, session tokens visible in URLs). Store and rotate these files appropriately. |
| Prerequisites |
|
|
| Install | | |
| Config | — | — |
| Citations | ||
| Claim | Unclaimed | Unclaimed |
Loading live community signals…
A short, calm digest of reviewed Claude resources. Unsubscribe any time.