Create Webhook

Create a new webhook to receive real-time data from Scrapest when specified events occur.

Endpoint

HTTP Request

POST /webhooks
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

Request Body

{
  "url": "https://your-domain.com/webhook",
  "events": ["tweet.created", "user.updated"],
  "filters": {
    "keywords": ["javascript", "programming"],
    "users": ["twitter_user"],
    "language": "en"
  },
  "secret": "your_webhook_secret",
  "active": true,
  "description": "Webhook for programming tweets"
}

Request Parameters

Required Parameters

url

Type: String
Description: HTTPS endpoint URL to receive webhook events
Format: Valid HTTPS URL
Example: https://your-domain.com/webhook

events

Type: Array of strings
Description: List of event types to subscribe to
Options:
- tweet.created - New tweets created
- tweet.updated - Tweets updated (likes, retweets)
- user.updated - User profile updates
- webhook.delivery_failed - Delivery failure notifications
Example: ["tweet.created", "user.updated"]

Optional Parameters

filters

Type: Object
Description: Filters to apply to events before delivery
Properties:
- keywords: Array of keywords to match
- users: Array of usernames to track
- language: Language code (e.g., “en”, “es”, “fr”)
- min_followers: Minimum follower count for users
- verified: Boolean to filter verified users only

secret

Type: String
Description: Secret key for webhook signature validation
Length: 8-128 characters
Auto-generated: If not provided, will be auto-generated
Example: your_webhook_secret

active

Type: Boolean
Description: Whether webhook should be immediately active
Default: true
Example: true

description

Type: String
Description: Human-readable description of webhook purpose
Max Length: 255 characters
Example: Webhook for programming tweets

Response Format

Success Response (201 Created)

{
  "id": "webhook_1234567890",
  "url": "https://your-domain.com/webhook",
  "events": ["tweet.created", "user.updated"],
  "filters": {
    "keywords": ["javascript", "programming"],
    "users": ["twitter_user"],
    "language": "en"
  },
  "secret": "your_webhook_secret",
  "active": true,
  "description": "Webhook for programming tweets",
  "status": "pending_verification",
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:30:00Z",
  "delivery_count": 0,
  "failure_count": 0
}

Error Response (400 Bad Request)

{
  "error": "Invalid webhook URL",
  "code": 400,
  "details": {
    "field": "url",
    "message": "URL must be a valid HTTPS endpoint"
  },
  "timestamp": "2024-01-15T10:30:00Z",
  "requestId": "req_1234567890"
}

Implementation Examples

JavaScript (Node.js)

const axios = require("axios");
const crypto = require("crypto");

async function createWebhook(apiKey, webhookConfig) {
  try {
    const response = await axios.post("https://scrape.st/webhooks", webhookConfig, {
      headers: {
        "x-api-key": apiKey,
        "Content-Type": "application/json",
      },
    });

    const webhook = response.data;
    console.log("Webhook created successfully:", webhook.id);

    // Store webhook details for verification
    await storeWebhookDetails(webhook);

    return webhook;
  } catch (error) {
    console.error("Failed to create webhook:", error.response?.data || error.message);
    throw error;
  }
}

function storeWebhookDetails(webhook) {
  // Store webhook details securely
  console.log("Storing webhook details...");
  console.log("Webhook ID:", webhook.id);
  console.log("Secret:", webhook.secret);
  // Implement your secure storage logic here
}

// Usage
const webhookConfig = {
  url: "https://your-domain.com/webhook",
  events: ["tweet.created", "user.updated"],
  filters: {
    keywords: ["javascript", "programming"],
    language: "en",
    verified: true,
  },
  description: "Track programming tweets from verified users",
};

createWebhook("your_api_key_here", webhookConfig);

Python

import requests
import json
import hashlib
import hmac

def create_webhook(api_key, webhook_config):
    url = "https://scrape.st/webhooks"
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }

    try:
        response = requests.post(url, json=webhook_config, headers=headers)
        response.raise_for_status()
        webhook = response.json()

        print(f"Webhook created successfully: {webhook['id']}")

        # Store webhook details for verification
        store_webhook_details(webhook)

        return webhook
    except requests.exceptions.RequestException as error:
        print(f"Failed to create webhook: {error}")
        if error.response:
            print(f"Error details: {error.response.text}")
        raise

def store_webhook_details(webhook):
    print("Storing webhook details...")
    print(f"Webhook ID: {webhook['id']}")
    print(f"Secret: {webhook['secret']}")
    # Implement your secure storage logic here

# Usage
webhook_config = {
    "url": "https://your-domain.com/webhook",
    "events": ["tweet.created", "user.updated"],
    "filters": {
        "keywords": ["javascript", "programming"],
        "language": "en",
        "verified": True
    },
    "description": "Track programming tweets from verified users"
}

create_webhook("your_api_key_here", webhook_config)

cURL

curl -X POST https://scrape.st/webhooks \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-domain.com/webhook",
    "events": ["tweet.created", "user.updated"],
    "filters": {
      "keywords": ["javascript", "programming"],
      "language": "en",
      "verified": true
    },
    "description": "Track programming tweets from verified users"
  }'

Webhook Verification

After creation, webhooks require verification:

Verification Process

Scrapest sends a POST request to your webhook URL
Request contains verification challenge in the body
Your endpoint must respond with the challenge value
Webhook status changes to active

Verification Request Format

{
  "type": "webhook.verification",
  "challenge": "verification_challenge_1234567890",
  "webhook_id": "webhook_1234567890"
}

Verification Response

{
  "challenge": "verification_challenge_1234567890"
}

Verification Implementation

// Express.js example
app.post("/webhook", (req, res) => {
  const { type, challenge, webhook_id } = req.body;

  if (type === "webhook.verification") {
    // Respond with challenge to verify webhook
    return res.json({ challenge });
  }

  // Handle regular webhook events
  handleWebhookEvent(req.body);
  res.status(200).send("OK");
});

Signature Validation

HMAC Signature

Each webhook request includes an HMAC signature for validation:

X-Webhook-Signature: sha256=5d41402abc4b2a76b9719d911017c592

Validation Implementation

function validateWebhookSignature(payload, signature, secret) {
  const expectedSignature = crypto.createHmac("sha256", secret).update(JSON.stringify(payload)).digest("hex");

  const receivedSignature = signature.replace("sha256=", "");

  return crypto.timingSafeEqual(Buffer.from(expectedSignature), Buffer.from(receivedSignature));
}

// Usage in Express middleware
app.post("/webhook", (req, res) => {
  const signature = req.headers["x-webhook-signature"];
  const isValid = validateWebhookSignature(req.body, signature, webhookSecret);

  if (!isValid) {
    return res.status(401).send("Invalid signature");
  }

  // Process webhook event
  handleWebhookEvent(req.body);
  res.status(200).send("OK");
});

Error Handling

Common Error Codes

400 Bad Request

Invalid URL: Webhook URL is not a valid HTTPS endpoint
Invalid Events: Event types are not recognized
Invalid Filters: Filter parameters are malformed

401 Unauthorized

Invalid API Key: API key is missing or invalid
Insufficient Permissions: API key lacks webhook creation permissions

409 Conflict

Duplicate Webhook: Webhook with same URL already exists
Webhook Limit: Maximum number of webhooks reached

429 Too Many Requests

Rate Limit: Too many webhook creation requests
Retry After: Check Retry-After header for retry time

Error Handling Implementation

async function createWebhookWithRetry(apiKey, webhookConfig, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await createWebhook(apiKey, webhookConfig);
    } catch (error) {
      if (error.response?.status === 429) {
        const retryAfter = error.response.headers["retry-after"] || 60;
        console.log(`Rate limited, retrying in ${retryAfter} seconds...`);
        await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000));
        continue;
      }

      if (error.response?.status === 409) {
        console.error("Webhook already exists or limit reached");
        throw error;
      }

      if (attempt === maxRetries) {
        throw error;
      }

      const delay = Math.pow(2, attempt) * 1000;
      console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

Best Practices

Security

HTTPS Required: Always use HTTPS endpoints
Secret Management: Store webhook secrets securely
Signature Validation: Always validate webhook signatures
Input Validation: Validate all incoming webhook data

Performance

Fast Response: Respond to webhook requests quickly
Async Processing: Process webhook data asynchronously
Queue Management: Use message queues for high-volume webhooks
Load Balancing: Distribute webhook load across servers

Reliability

Idempotency: Handle duplicate webhook events
Error Handling: Implement robust error handling
Logging: Log all webhook deliveries for debugging
Monitoring: Monitor webhook delivery success rates

Next: List Webhooks

​Create Webhook

​Endpoint

​HTTP Request

​Request Body

​Request Parameters

​Required Parameters

​url

​events

​Optional Parameters

​filters

​secret

​active

​description

​Response Format

​Success Response (201 Created)

​Error Response (400 Bad Request)

​Implementation Examples

​JavaScript (Node.js)

​Python

​cURL

​Webhook Verification

​Verification Process

​Verification Request Format

​Verification Response

​Verification Implementation

​Signature Validation

​HMAC Signature

​Validation Implementation

​Error Handling

​Common Error Codes

​400 Bad Request

​401 Unauthorized

​409 Conflict

​429 Too Many Requests

​Error Handling Implementation

​Best Practices

​Security

​Performance

​Reliability

Create Webhook

Endpoint

HTTP Request

Request Body

Request Parameters

Required Parameters

url

events

Optional Parameters

filters

secret

active

description

Response Format

Success Response (201 Created)

Error Response (400 Bad Request)

Implementation Examples

JavaScript (Node.js)

Python

cURL

Webhook Verification

Verification Process

Verification Request Format

Verification Response

Verification Implementation

Signature Validation

HMAC Signature

Validation Implementation

Error Handling

Common Error Codes

400 Bad Request

401 Unauthorized

409 Conflict

429 Too Many Requests

Error Handling Implementation

Best Practices

Security

Performance

Reliability