Web Queries
Start Crawl
Crawl a website and scrape every discovered page
POST
Start a crawl over a website. The crawler discovers pages from the entry URL (and sitemaps) and scrapes each one with the same output options as Scrape URL.
Crawls are always asynchronous: the response returns a
crawl_id immediately — poll Crawl Status for progress and results, or register a webhook for delivery.
Request Body
string
required
The entry URL to start crawling from.
object
Crawler options: -
limit (number, default 100) — max pages to crawl -
include / exclude (string[]) — URL patterns to include or skip -
max_depth (number, default 3) — link depth from the entry URL -
include_entire_domain (boolean, default false) — allow crawling any path
on the entry domain - include_subdomains (boolean, default false) — allow
crawling subdomains of the entry domain - include_external_links (boolean,
default false) — allow crawling external links - sitemaps (boolean,
default true) — seed from sitemap.xmlobject
Output options per page — same as Scrape
URL.
object
Proxy options:
location (default US) and sticky_session.object
Request options — same as Scrape URL.
object
JavaScript rendering options — same as Scrape
URL.
object
Delivery webhook:
url, optional headers, events (default ["started", "completed", "failed"]).Response
string
Unique ID for this crawl job. Use it with Crawl
Status.
string
crawling on accept.