Skip to main content

​
Rate Limits

​
Understanding Rate Limits

Scrapest implements rate limiting to ensure fair usage and system stability:
  • API Requests: Limited per API key per time window
  • Webhook Delivery: Limited per webhook endpoint
  • Streaming Connections: Limited concurrent connections per API key
  • Data Processing: Limited based on your subscription tier

​
Handling Rate Limits

Always implement proper rate limit handling: 
// Good: Rate limit handling with exponential backoff
async function makeApiRequest(url, options = {}) {
  let retries = 0;
  const maxRetries = 5;

  while (retries < maxRetries) {
    try {
      const response = await axios.get(url, {
        ...options,
        headers: {
          "x-api-key": process.env.SCRAPEST_API_KEY,
          ...options.headers,
        },
      });
      return response.data;
    } catch (error) {
      if (error.response?.status === 429) {
        retries++;
        const delay = Math.min(1000 * Math.pow(2, retries), 30000);
        console.log(`Rate limited. Retrying in ${delay}ms (attempt ${retries}/${maxRetries})`);
        await new Promise((resolve) => setTimeout(resolve, delay));
      } else {
        throw error;
      }
    }
  }

  throw new Error("Max retries exceeded due to rate limiting");
}

​
Rate Limit Best Practices

  • Monitor Response Headers: Check X-RateLimit-Remaining and X-RateLimit-Reset
  • Implement Queuing: Use request queues for high-volume operations
  • Distribute Load: Spread requests across time windows
  • Handle Gracefully: Always catch 429 responses and retry appropriately

​
API Key Security

​
Secure API Key Management

Never expose API keys in your code or repositories: 
// Bad: Hardcoded API key
const apiKey = "sk_live_1234567890abcdef";

// Good: Environment variables
const apiKey = process.env.SCRAPEST_API_KEY;

// Even better: Secure configuration
class SecureConfig {
  constructor() {
    this.loadConfig();
  }

  loadConfig() {
    this.apiKey = process.env.SCRAPEST_API_KEY;

    if (!this.apiKey) {
      throw new Error("SCRAPEST_API_KEY environment variable is required");
    }
  }

  getApiKey() {
    return this.apiKey;
  }
}

​
Environment Configuration

Set up environment variables securely: 
# Development (.env)
SCRAPEST_API_KEY=your_development_api_key

# Production
export SCRAPEST_API_KEY=your_production_api_key

​
Webhook Security

Secure your webhook endpoints with token validation When registering webhooks, include a token for validation: 
// Register webhook with token
const webhookConfig = {
  url: "https://your-domain.com/webhook?token=your_secret_token",
  name: "Secure Webhook",
};

const response = await axios.post("https://scrape.st/webhooks", webhookConfig, {
  headers: {
    "x-api-key": process.env.SCRAPEST_API_KEY,
  },
});

​
Validating Incoming Webhooks

Always validate webhook requests before processing: 
function validateWebhook(req) {
  // Validate token from query parameter
  const token = req.query.token;
  const expectedToken = process.env.WEBHOOK_TOKEN;

  if (!token || token !== expectedToken) {
    return { valid: false, error: "Invalid token" };
  }

  return { valid: true };
}

// Express middleware example
function webhookMiddleware(req, res, next) {
  const validation = validateWebhook(req);

  if (!validation.valid) {
    return res.status(401).json({
      error: validation.error,
      message: "Webhook validation failed",
    });
  }

  // Process the webhook data
  const { source, payload } = req.body;

  console.log(`Received ${source} webhook:`, {
    id: payload.id,
    timestamp: payload.created_at,
    type: source,
  });

  next();
}

// Express route setup
app.post("/webhook", webhookMiddleware, (req, res) => {
  // Webhook is validated, process the data
  res.status(200).json({ status: "received" });
});

​
Security Best Practices

​
Essential Security Rules

  1. Never Commit API Keys: Add API keys to .gitignore
  2. Use Environment Variables: Store keys in environment, not code
  3. Implement Least Privilege: Use minimal required permissions
  4. Monitor Usage: Track API key usage and anomalies
  5. Rotate Regularly: Change API keys periodically
  6. Use HTTPS: Always use secure connections
  7. Validate Input: Sanitize all incoming data

​
Environment Setup

// Secure environment configuration
const config = {
  apiKey: process.env.SCRAPEST_API_KEY,
  webhookUrl: process.env.WEBHOOK_URL,
  environment: process.env.NODE_ENV || "development",
};

// Validate required configuration
if (!config.apiKey) {
  throw new Error("SCRAPEST_API_KEY is required");
}

​
Common Mistakes to Avoid

​
Rate Limit Mistakes

  • ❌ Ignoring 429 responses
  • ❌ No retry logic with backoff
  • ❌ Not monitoring rate limit headers

​
Security Mistakes

  • ❌ Hardcoding API keys in source code
  • ❌ Not validating webhook signatures
  • ❌ Sharing API keys publicly

​
Quick Checklist

Before deploying your Scrapest integration:
  • API keys stored in environment variables
  • Rate limit handling implemented
  • HTTPS used for all connections
  • Error handling with retry logic
Following these practices ensures secure, reliable, and efficient Scrapest API integration.