Why does CORS block some browser crawls?

Modern browser sandboxes enforce Cross-Origin Resource Sharing (CORS) rules. If a destination website does not return wildcard headers (Access-Control-Allow-Origin: *), the browser blocks direct fetching requests for security reasons.

How does robots.txt disallow rules work?

The robots.txt file defines crawl directives for different search engines. Directives like Disallow: /admin/ inform the crawler which sections are private, preventing requests to restricted folders.

Is my crawl data private?

Yes. All parsed details, links, images, and sitemaps are written to your local browser IndexedDB. No page data leaves your local device.

Free Website Crawler — Crawl Up to 10,000 Pages

Crawl and analyze up to 10,000 URLs from any website with detailed technical, SEO, links, content, HTTP, sitemaps, robots.txt, and security headers.

⚡ Interactive Calculation Engine Active

Please enable JavaScript in your browser to run live computations, 2D envelope diagrams, and export PDF reports.

How to Use Website Crawler & Technical Link Spider

Enter the starting target website URL (e.g., https://example.com).
Configure crawler settings: max URLs (up to 10,000), depth limit, robots.txt options, and subdomain settings.
Click Start to run the spider. Watch the queue updates, fetching latency speeds, and crawled pages logs in real-time.
Explore crawled pages list, technical issue groups, and the interactive SVG visual site architecture tree.
Export results as CSV tables, download crawl comparison logs, or clear history.

What is a Website Crawler & Link Spider?

A website crawler is an automated spider that parses HTML structures client-side to extract references, mapping out sitemap links, indexability categories, and redirects chains.

Respecting robots.txt Directives and Crawling Politeness

Directives declare which directories are crawlable. The crawler automatically loads /robots.txt to match rules and apply delay throttles, avoiding request flooding.

Troubleshooting CORS Network Block Skipped Pages

Client-side scripts are subject to browser CORS blocks. If external hosts restrict access, pages are flagged as skipped. You can test local domains or CORS-friendly test endpoints.

Client-Side IndexedDB Storage and Data Protection

All crawl runs metadata, image lists, and issues are stored locally inside IndexedDB. No data is stored on remote servers, providing strict privacy controls.

Frequently Asked Questions

What is a Website Crawler?: A website crawler is an automated bot that discovers links across pages sequentially to parse metadata, index status parameters, broken resources, headings outlines, and construct a visual layout representation of a site's depth hierarchy.
Why does CORS block some browser crawls?: Modern browser sandboxes enforce Cross-Origin Resource Sharing (CORS) rules. If a destination website does not return wildcard headers (Access-Control-Allow-Origin: *), the browser blocks direct fetching requests for security reasons.
How does robots.txt disallow rules work?: The robots.txt file defines crawl directives for different search engines. Directives like Disallow: /admin/ inform the crawler which sections are private, preventing requests to restricted folders.
Is my crawl data private?: Yes. All parsed details, links, images, and sitemaps are written to your local browser IndexedDB. No page data leaves your local device.