Web - Scrapest

Monitor websites for changes and scrape any public URL for raw HTML content. Track pages for real-time change detection or perform one-off scrapes on demand.

What You Can Do

Capability	Endpoint	Description
Track Website	`POST /track/web`	Monitor a website URL for changes
Web Scraping	`POST /scrape`	Scrape any public URL for raw HTML
List Tracked	`GET /track/web`	List all tracked websites
Untrack	`DELETE /track/web`	Stop tracking a website

Track a Website

Start monitoring a website URL for content changes.

curl -X POST https://scrape.st/track/web \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"sid": "https://example.com"}'

const res = await fetch("https://scrape.st/track/web", {
  method: "POST",
  headers: {
    "x-api-key": "YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ sid: "https://example.com" }),
});
const data = await res.json();

import requests

res = requests.post(
    "https://scrape.st/track/web",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"sid": "https://example.com"},
)
print(res.json())

Response:

{
  "data": {
    "id": "src_abc",
    "source": "WEB",
    "externalId": "https://example.com"
  },
  "message": "Now tracking https://example.com on web"
}

Web Scraping

Scrape any public URL and get the raw HTML content back. This is a one-off request — no tracking required.

This endpoint is rate-limited to 10 requests per 60 seconds per API key.

curl -X POST https://scrape.st/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

const res = await fetch("https://scrape.st/scrape", {
  method: "POST",
  headers: {
    "x-api-key": "YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com" }),
});
const data = await res.json();

import requests

res = requests.post(
    "https://scrape.st/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "https://example.com"},
)
print(res.json())

Response:

{
  "raw": "<html><head><title>Example Domain</title>...</html>"
}

List Tracked Websites

Retrieve all websites you are currently tracking.

curl -X GET https://scrape.st/track/web \
  -H "x-api-key: YOUR_API_KEY"

Stop Tracking

Remove a website from your tracked sources. You can identify the source by its original URL (sid) or its internal ID (iid).

curl -X DELETE https://scrape.st/track/web \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"sid": "https://example.com"}'

curl -X DELETE https://scrape.st/track/web \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"iid": "src_abc"}'

const res = await fetch("https://scrape.st/track/web", {
  method: "DELETE",
  headers: {
    "x-api-key": "YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ sid: "https://example.com" }),
});
const data = await res.json();

import requests

res = requests.delete(
    "https://scrape.st/track/web",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"sid": "https://example.com"},
)
print(res.json())

Webhook Event Shape

When a tracked website publishes new content, your webhook receives a SourceEvent object. This is the exact shape sent via JSON.stringify to your webhook URL:

{
  "mid": "https://example.com/blog/new-post",
  "sid": "https://example.com",
  "source": "web",
  "timestamp": 1712072400000,
  "content": "Full text of the detected article…",
  "images": ["https://example.com/cover.jpg"],
  "payload": {
    "id": "https://example.com/blog/new-post",
    "sourceUrl": "https://example.com",
    "title": "New Post Title",
    "url": "https://example.com/blog/new-post",
    "content": "Full text of the detected article…",
    "publishedAt": "2024-04-02T15:30:00.000Z",
    "author": "Jane Doe",
    "imageUrl": "https://example.com/cover.jpg"
  }
}

Field	Type	Description
`mid`	string	Message ID — the article URL
`sid`	string	Source ID — the tracked website URL
`source`	string	`"web"`
`timestamp`	number	Unix timestamp (ms) when the event was received
`content`	string	Extracted article text
`images`	array	Cover image URL if detected, else empty
`payload`	object	Full `WebData` object (see fields above)

Delivery Methods

Web change events are delivered through all the same channels as other sources:

Webhooks — Push to your endpoint
WebSocket — Real-time bidirectional stream
SSE — Server-sent events with auto-reconnect

API Reference

Create Tracking — POST /track/web
List Tracked Sources — GET /track/web
Delete Tracking — DELETE /track/web

​What You Can Do

​Track a Website

​Web Scraping

​List Tracked Websites

​Stop Tracking

​Webhook Event Shape

​Delivery Methods

​API Reference

What You Can Do

Track a Website

Web Scraping

List Tracked Websites

Stop Tracking

Webhook Event Shape

Delivery Methods

API Reference