curl -X POST https://scrape.st/web/crawl \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"crawler": { "limit": 50, "max_depth": 2 },
"output": { "formats": ["markdown"] }
}'
{
"crawl_id": "crl_01HZX...",
"version": "1.0.0",
"status": "crawling"
}
Web Queries
Start Crawl
Crawl a website and scrape every discovered page
POST
/
web
/
crawl
curl -X POST https://scrape.st/web/crawl \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"crawler": { "limit": 50, "max_depth": 2 },
"output": { "formats": ["markdown"] }
}'
{
"crawl_id": "crl_01HZX...",
"version": "1.0.0",
"status": "crawling"
}
Start a crawl over a website. The crawler discovers pages from the entry URL (and sitemaps) and scrapes each one with the same output options as Scrape URL.
Crawls are always asynchronous: the response returns a
crawl_id immediately — poll Crawl Status for progress and results, or register a webhook for delivery.
Request Body
string
required
The entry URL to start crawling from.
object
Crawler options: -
limit (number, default 100) — max pages to crawl -
include / exclude (string[]) — URL patterns to include or skip -
max_depth (number, default 3) — link depth from the entry URL -
include_entire_domain (boolean, default false) — allow crawling any path
on the entry domain - include_subdomains (boolean, default false) — allow
crawling subdomains of the entry domain - include_external_links (boolean,
default false) — allow crawling external links - sitemaps (boolean,
default true) — seed from sitemap.xmlobject
Output options per page — same as Scrape
URL.
object
Proxy options:
location (default US) and sticky_session.object
Request options — same as Scrape URL.
object
JavaScript rendering options — same as Scrape
URL.
object
Delivery webhook:
url, optional headers, events (default ["started", "completed", "failed"]).Response
string
Unique ID for this crawl job. Use it with Crawl
Status.
string
crawling on accept.curl -X POST https://scrape.st/web/crawl \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"crawler": { "limit": 50, "max_depth": 2 },
"output": { "formats": ["markdown"] }
}'
{
"crawl_id": "crl_01HZX...",
"version": "1.0.0",
"status": "crawling"
}