Skip to main content

​
WebSocket Connections

WebSocket connections provide real-time, bidirectional communication with Scrapest for receiving Twitter/X data instantly.

​
Connection Setup

​
Endpoint

wss://api.scrape.st/ws

​
Authentication

Include your API key in the connection headers: 
const ws = new WebSocket("wss://api.scrape.st/ws", [], {
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
  },
});

​
Connection Example

const ws = new WebSocket("wss://api.scrape.st/ws");

ws.onopen = function () {
  console.log("Connected to Scrapest WebSocket");

  // Subscribe to data streams
  ws.send(
    JSON.stringify({
      action: "subscribe",
      filters: {
        keywords: ["javascript", "programming"],
        language: "en",
      },
    }),
  );
};

ws.onmessage = function (event) {
  const data = JSON.parse(event.data);
  console.log("Received:", data);
};

ws.onerror = function (error) {
  console.error("WebSocket Error:", error);
};

ws.onclose = function () {
  console.log("WebSocket connection closed");
};

​
Message Format

​
Subscription Request

{
  "action": "subscribe",
  "filters": {
    "keywords": ["string"],
    "users": ["string"],
    "language": "string",
    "min_retweets": "number"
  }
}

​
Data Response

{
  "type": "tweet",
  "data": {
    "id": "1234567890",
    "text": "Tweet content",
    "user": {
      "id": "user_id",
      "username": "username",
      "name": "Display Name"
    },
    "created_at": "2024-01-01T00:00:00Z",
    "metrics": {
      "retweets": 10,
      "likes": 25,
      "replies": 5
    }
  }
}

​
Managing Connections

​
Unsubscribe

ws.send(
  JSON.stringify({
    action: "unsubscribe",
    filters: {
      keywords: ["javascript"],
    },
  }),
);

​
Close Connection

ws.close();

​
Error Handling

Common WebSocket error codes:
  • 1000: Normal closure
  • 1001: Going away
  • 1006: Abnormal closure
  • 1008: Policy violation

​
Best Practices

  1. Reconnection Logic: Implement automatic reconnection with exponential backoff
  2. Rate Limiting: Respect connection limits and message rates
  3. Error Handling: Handle all WebSocket events properly
  4. Filter Optimization: Use specific filters to reduce data volume

​
Rate Limits

  • Maximum 5 concurrent WebSocket connections per API key
  • 100 messages per second rate limit
  • Automatic disconnection on sustained violations
Next: Public SSE