Getting Started with Scrapest

​

Prerequisites

• A Scrapest account (sign up at admin.scrape.st)
• Basic familiarity with APIs and webhooks
• A server or application that can receive HTTP requests

​

Step 1: Get Your API Key

1. Sign in to the Scrapest Dashboard
2. Navigate to API Keys in the sidebar
3. Click Generate New API Key
4. Give your key a descriptive name (e.g., “My First Integration”)
5. Copy the API key and store it securely

# Your API key will look like this:
sk_live_1234567890abcdef1234567890abcdef12345678

​

Step 2: Choose Your Integration Approach

​

Webhooks (Recommended for most applications)

• Best for: Real-time event-driven applications
• How it works: We send data to your endpoint when events occur
• Setup time: 5-10 minutes

​

Streaming (For real-time applications)

• Best for: High-frequency data needs, live dashboards
• How it works: Maintain persistent connection for continuous data flow
• Setup time: 10-15 minutes

​

REST API (For on-demand access)

• Best for: One-time queries, user lookups, specific data retrieval
• How it works: Make HTTP requests when you need data
• Setup time: 2-5 minutes

​

Step 3: Set Up Your First Integration

​

Option A: Webhook Integration

1. Create a webhook endpoint on your server:

// Express.js example
app.post("/webhook", (req, res) => {
  console.log("Received webhook:", req.body);
  res.status(200).send("OK");
});

2. Register the webhook using our API:

curl -X POST https://scrape.st/webhooks \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-domain.com/webhook",
    "name": "My first webhook"
  }'

3. Start tracking a user or channel:

curl -X POST https://scrape.st/tracking/twitter \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sid": "twitter_handle"
  }'

curl -X POST https://scrape.st/tracking/discord \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sid": "discord_channel_id"
  }'

​

Option B: REST API Integration

1. Make your first API call to get user information:

curl -X GET "https://scrape.st/x/user?username=twitter_handle" \
  -H "x-api-key: YOUR_API_KEY"

2. Get specific tweet information:

curl -X GET "https://scrape.st/x/tweet?tweetId=1234567890123456789" \
  -H "x-api-key: YOUR_API_KEY"

​

Option C: Streaming Integration

1. Set up WebSocket connection:

const ws = new WebSocket("wss://scrape.st/ws", [], {
  headers: {
    "x-api-key": "YOUR_API_KEY",
  },
});

ws.onmessage = function (event) {
  const data = JSON.parse(event.data);
  console.log("Received data:", data);
};

​

Step 4: Test Your Integration

​

Verify Webhook Delivery

1. Create a test post from a tracked account/channel
2. Check your server logs for the webhook payload
3. Verify the data format and content

​

Test API Calls

# Test health endpoint
curl -X GET "https://scrape.st/health" \
  -H "x-api-key: YOUR_API_KEY"

​

Test Streaming

// Monitor connection events
ws.onopen = () => console.log("Connected to Scrapest");
ws.onerror = (error) => console.error("WebSocket error:", error);
ws.onclose = () => console.log("Disconnected from Scrapest");

​

Set Up Monitoring

• Monitor webhook delivery success rates
• Track API usage and rate limits
• Set up alerts for failures or unusual activity

​

Scale Your Integration

• Implement retry logic for failed requests
• Use caching for frequently accessed data
• Consider batch processing for high-volume needs

​

Common Integration Patterns

​

Social Media Dashboard

// Real-time dashboard using webhooks
app.post("/webhook", (req, res) => {
  const { source, payload } = req.body;

  if (source === "twitter") {
    // Update dashboard with new tweet
    updateDashboard(payload);
  }

  res.status(200).send("OK");
});

​

Analytics Pipeline

// Process tweets for analytics
function processMessageForAnalytics(payload) {
  const analytics = {
    engagement: payload.like_count + payload.retweet_count || 0,
    sentiment: analyzeSentiment(payload.text),
    reach: payload.author?.followers_count || 0,
  };

  storeInDatabase(analytics);
}

​

Troubleshooting Common Issues

​

Webhook Not Receiving Data

• Check that your webhook URL is publicly accessible
• Verify your server responds with 200 OK status
• Ensure webhook is active in your dashboard

​

API Authentication Errors

• Verify your API key is correct and active
• Check that you’re using the correct endpoint URL
• Ensure API key has required permissions

​

Rate Limiting

• Implement exponential backoff for retries
• Monitor your usage in the dashboard
• Consider upgrading your plan for higher limits

​

Streaming Connection Issues

• Check your network connectivity
• Verify WebSocket headers are correct
• Implement automatic reconnection logic

​

Best Practices

​

Security

• Store API keys securely (environment variables, secret managers)
• Use HTTPS for all webhook endpoints
• Validate incoming webhook signatures
• Implement rate limiting on your endpoints

​

Performance

• Use appropriate data structures for high-volume processing
• Implement caching for frequently accessed data
• Use connection pooling for API requests
• Monitor and optimize your integration regularly

​

Reliability

• Implement comprehensive error handling
• Use retry logic with exponential backoff
• Set up monitoring and alerting
• Have backup plans for service disruptions

​

Next Steps

1. Explore Advanced Features: Learn about streaming options
2. Review API Documentation: Check the REST API Reference
3. Set Up Monitoring: Configure health monitoring
4. Join the Community: Get help and share your experiences

​

Need Help?

• Documentation: Complete guides and API references
• Support Team: support@scrape.st
• Community: Discord and GitHub discussions
• Status Page: Real-time system status