> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrape.st/llms.txt
> Use this file to discover all available pages before exploring further.
# Web
> Track websites and scrape web pages for real-time content monitoring
Monitor websites for changes and scrape any public URL for raw HTML content. Track pages for real-time change detection or perform one-off scrapes on demand.
## What You Can Do
| Capability | Endpoint | Description |
| ----------------- | ------------------- | ---------------------------------- |
| **Track Website** | `POST /track/web` | Monitor a website URL for changes |
| **Web Scraping** | `POST /scrape` | Scrape any public URL for raw HTML |
| **List Tracked** | `GET /track/web` | List all tracked websites |
| **Untrack** | `DELETE /track/web` | Stop tracking a website |
## Track a Website
Start monitoring a website URL for content changes.
```bash curl theme={null}
curl -X POST https://scrape.st/track/web \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"sid": "https://example.com"}'
```
```javascript JavaScript theme={null}
const res = await fetch("https://scrape.st/track/web", {
method: "POST",
headers: {
"x-api-key": "YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({ sid: "https://example.com" }),
});
const data = await res.json();
```
```python Python theme={null}
import requests
res = requests.post(
"https://scrape.st/track/web",
headers={"x-api-key": "YOUR_API_KEY"},
json={"sid": "https://example.com"},
)
print(res.json())
```
**Response:**
```json theme={null}
{
"data": {
"id": "src_abc",
"source": "WEB",
"externalId": "https://example.com"
},
"message": "Now tracking https://example.com on web"
}
```
## Web Scraping
Scrape any public URL and get the raw HTML content back. This is a one-off request — no tracking required.
This endpoint is rate-limited to **10 requests per 60 seconds** per API key.
```bash curl theme={null}
curl -X POST https://scrape.st/scrape \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
```javascript JavaScript theme={null}
const res = await fetch("https://scrape.st/scrape", {
method: "POST",
headers: {
"x-api-key": "YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({ url: "https://example.com" }),
});
const data = await res.json();
```
```python Python theme={null}
import requests
res = requests.post(
"https://scrape.st/scrape",
headers={"x-api-key": "YOUR_API_KEY"},
json={"url": "https://example.com"},
)
print(res.json())
```
**Response:**
```json theme={null}
{
"raw": "Example Domain..."
}
```
## List Tracked Websites
Retrieve all websites you are currently tracking.
```bash theme={null}
curl -X GET https://scrape.st/track/web \
-H "x-api-key: YOUR_API_KEY"
```
## Stop Tracking
Remove a website from your tracked sources. You can identify the source by its original URL (`sid`) or its internal ID (`iid`).
```bash curl (by URL) theme={null}
curl -X DELETE https://scrape.st/track/web \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"sid": "https://example.com"}'
```
```bash curl (by internal ID) theme={null}
curl -X DELETE https://scrape.st/track/web \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"iid": "src_abc"}'
```
```javascript JavaScript theme={null}
const res = await fetch("https://scrape.st/track/web", {
method: "DELETE",
headers: {
"x-api-key": "YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({ sid: "https://example.com" }),
});
const data = await res.json();
```
```python Python theme={null}
import requests
res = requests.delete(
"https://scrape.st/track/web",
headers={"x-api-key": "YOUR_API_KEY"},
json={"sid": "https://example.com"},
)
print(res.json())
```
## Webhook Event Shape
When a tracked website publishes new content, your webhook receives a `SourceEvent` object. This is the exact shape sent via `JSON.stringify` to your webhook URL:
```json theme={null}
{
"mid": "https://example.com/blog/new-post",
"sid": "https://example.com",
"source": "web",
"timestamp": 1712072400000,
"content": "Full text of the detected article…",
"images": ["https://example.com/cover.jpg"],
"payload": {
"id": "https://example.com/blog/new-post",
"sourceUrl": "https://example.com",
"title": "New Post Title",
"url": "https://example.com/blog/new-post",
"content": "Full text of the detected article…",
"publishedAt": "2024-04-02T15:30:00.000Z",
"author": "Jane Doe",
"imageUrl": "https://example.com/cover.jpg"
}
}
```
| Field | Type | Description |
| ----------- | ------ | ----------------------------------------------- |
| `mid` | string | Message ID — the article URL |
| `sid` | string | Source ID — the tracked website URL |
| `source` | string | `"web"` |
| `timestamp` | number | Unix timestamp (ms) when the event was received |
| `content` | string | Extracted article text |
| `images` | array | Cover image URL if detected, else empty |
| `payload` | object | Full `WebData` object (see fields above) |
## Delivery Methods
Web change events are delivered through all the same channels as other sources:
* **[Webhooks](/rest-api-reference/create-webhook)** — Push to your endpoint
* **[WebSocket](/streams/websocket-connections)** — Real-time bidirectional stream
* **[SSE](/streams/public-sse)** — Server-sent events with auto-reconnect
## API Reference
* [Create Tracking](/rest-api-reference/create-tracking) — `POST /track/web`
* [List Tracked Sources](/rest-api-reference/list-tracking) — `GET /track/web`
* [Delete Tracking](/rest-api-reference/delete-tracking) — `DELETE /track/web`