Site Crawler

Make Any Site Static

Crawl dynamic websites and serve them statically. Perfect for any CMS without native integration. Get static performance without rebuilding your site.

Start crawlingCrawler documentation
Site Crawler
example.com
Crawling
Active Workers 5 / 5
1
/products/overview
Capturing
2
/blog/latest-post
Rendering
3
/about
Done
4
/contact
Assets
5
/docs/getting-started
Done
Crawl Progress 156 / 247
156
Captured
91
Queued
0
Errors
2.3s
Avg time
Browser mode
Enabled
5 workers
1s delay
Depth: 10
1-20
Workers
JS
Rendering
Cron
Scheduling
Webhooks
Notifications
Crawler Configuration

Full Control Over Crawling

Configure every aspect of how your site is crawled. Set concurrent workers, request delays, crawl depth, and page limits. Use regex patterns to include or exclude specific URLs.

  • 1-20 concurrent workers
  • Configurable delay (0-10s)
  • Depth limiting
  • Include/exclude patterns

Crawler documentation

Crawler Settings
Active
Headless Browser
Render JavaScript content
Workers
5
Delay (ms)
1000
Max Depth
10
Max Pages
5,000
URL Patterns
include /docs/*, /blog/*
exclude /admin/*, *.pdf
JS rendering
Regex patterns
Full control
Automated Crawls
Scheduled
Schedule
0 2 * * *
Daily at 2:00 AM UTC
Recent activity
Crawl completed 2:14 AM
2,847 pages · 14 min
Crawl started 2:00 AM
Scheduled trigger
Webhook sent 2:14 AM
POST to your endpoint
Webhook endpoint
https://api.example.com/webhooks/crawl
Start
Complete
Error
Cron scheduling
Webhooks
Notifications
Webhooks & Scheduling

Automated Crawl Workflows

Set up cron-based schedules for automatic recrawls. Receive webhook notifications for crawl events including start, progress, completion, and errors.

  • Cron scheduling
  • Webhook notifications
  • Progress tracking
  • Error alerts

Schedule crawls

JavaScript Rendering

Capture Dynamic Content

Enable headless browser mode to render JavaScript before capture. Perfect for SPAs, client-rendered content, and interactive elements. Execute JavaScript during asset collection.

  • Chrome headless mode
  • JS asset collection
  • Network interception
  • Custom user agent

Crawler docs

Site Crawler
example.com
Crawling
Active Workers 5 / 5
1
/products/overview
Capturing
2
/blog/latest-post
Rendering
3
/about
Done
4
/contact
Assets
5
/docs/getting-started
Done
Crawl Progress 156 / 247
156
Captured
91
Queued
0
Errors
2.3s
Avg time
Browser mode
Enabled
5 workers
1s delay
Depth: 10

Crawler Use Cases

Static performance for any dynamic site.

WordPress

Serve WordPress sites statically. Keep the familiar admin, get static speed and security.

  • Headless browser
  • Asset capture
  • Multi-domain
Learn more

Drupal

Decoupled Drupal with static delivery. Perfect for editorial workflows with static performance.

  • Entity support
  • Webhooks
  • Scheduled sync
Learn more

Legacy Sites

Migrate legacy sites to static without rewriting. Capture and serve any site, regardless of technology.

  • Any technology
  • No code changes
  • Full capture
Learn more

Site Archival

Create static archives of sites for compliance, preservation, or historical reference.

  • Full capture
  • Asset preservation
  • Offline viewing
Learn more

AI Search Indexing

Enable AI Search integration to index crawled pages with semantic embeddings for search.

  • Semantic indexing
  • Auto summaries
  • Vector search
Learn more

Scheduled Sync

Keep static sites in sync with dynamic sources via cron-scheduled or webhook-triggered crawls.

  • Cron scheduling
  • Webhook triggers
  • Progress tracking
Learn more

Go Static Today

Crawl your first site in minutes. Start with a 30-day free trial.

Headless browser mode
Configurable workers
Cron scheduling
Webhook notifications
Start crawlingView pricing