Health Endpoints

Technical documentation for health monitoring and metrics API endpoints.

Available Endpoints

Health Check

GET /health

General Metrics

GET /metrics

Source-Specific Metrics

GET /metrics/{source}

Health Check Endpoint

Request

GET /health

Response Format

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "version": "1.2.3",
  "uptime": 86400,
  "checks": {
    "database": "healthy",
    "redis": "healthy",
    "external_apis": "degraded"
  },
  "details": {
    "database": {
      "response_time": 8,
      "connections": 25,
      "max_connections": 100
    },
    "redis": {
      "response_time": 3,
      "memory_usage": "45%",
      "connected_clients": 12
    },
    "external_apis": {
      "twitter": "degraded",
      "reddit": "healthy",
      "response_times": {
        "twitter": 850,
        "reddit": 120
      }
    }
  }
}

Response Fields

Overall Status

status: Overall system health (healthy, degraded, unhealthy)
timestamp: ISO 8601 timestamp of health check
version: Current API version
uptime: System uptime in seconds

Component Checks

checks: Health status of individual components
details: Detailed information for each component

Component Details

database: Database connection and performance
redis: Cache status and memory usage
external_apis: External API availability and response times

Implementation Examples

Health Check (JavaScript)

const axios = require("axios");

async function checkSystemHealth() {
  try {
    const response = await axios.get("https://scrape.st/health");
    const health = response.data;

    console.log("System Health:", health.status);
    console.log("Uptime:", formatUptime(health.uptime));

    // Check individual components
    Object.entries(health.checks).forEach(([component, status]) => {
      const icon = status === "healthy" ? "✅" : status === "degraded" ? "⚠️" : "❌";
      console.log(`${icon} ${component}: ${status}`);
    });

    // Alert on unhealthy components
    const unhealthyComponents = Object.entries(health.checks)
      .filter(([, status]) => status !== "healthy")
      .map(([component]) => component);

    if (unhealthyComponents.length > 0) {
      console.warn("⚠️ Unhealthy components:", unhealthyComponents.join(", "));
    }

    return health;
  } catch (error) {
    console.error("Health check failed:", error.message);
    return { status: "error", message: error.message };
  }
}

function formatUptime(seconds) {
  const days = Math.floor(seconds / 86400);
  const hours = Math.floor((seconds % 86400) / 3600);
  const minutes = Math.floor((seconds % 3600) / 60);

  return `${days}d ${hours}h ${minutes}m`;
}

// Usage
checkSystemHealth();

Health Check (Python)

import requests
from datetime import datetime

def check_system_health():
    try:
        response = requests.get('https://scrape.st/health')
        health = response.json()

        print(f"System Health: {health['status']}")
        print(f"Uptime: {format_uptime(health['uptime'])}")

        # Check individual components
        for component, status in health['checks'].items():
            icon = "✅" if status == "healthy" else "⚠️" if status == "degraded" else "❌"
            print(f"{icon} {component}: {status}")

        # Alert on unhealthy components
        unhealthy_components = [
            component for component, status in health['checks'].items()
            if status != 'healthy'
        ]

        if unhealthy_components:
            print(f"⚠️ Unhealthy components: {', '.join(unhealthy_components)}")

        return health
    except Exception as error:
        print(f"Health check failed: {error}")
        return {"status": "error", "message": str(error)}

def format_uptime(seconds):
    days = seconds // 86400
    hours = (seconds % 86400) // 3600
    minutes = (seconds % 3600) // 60
    return f"{days}d {hours}h {minutes}m"

# Usage
check_system_health()

Metrics Endpoints

General Metrics

GET /metrics
Authorization: Bearer YOUR_API_KEY

Source-Specific Metrics

GET /metrics/{source}
Authorization: Bearer YOUR_API_KEY

Response Headers

Content-Type: application/json
X-Rate-Limit-Remaining: 95
X-Rate-Limit-Reset: 1642248000

Error Handling

HTTP Status Codes

Success Codes

200 OK: Request successful
207 Multi-Status: Health check with mixed component status

Error Codes

401 Unauthorized: Invalid or missing API key
404 Not Found: Invalid source or endpoint
429 Too Many Requests: Rate limit exceeded
503 Service Unavailable: Service temporarily unavailable

Error Response Format

{
  "error": "Error message",
  "code": 401,
  "timestamp": "2024-01-15T10:30:00Z",
  "requestId": "req_1234567890"
}

Error Handling Implementation

class HealthClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.baseUrl = "https://scrape.st";
  }

  async getHealth() {
    return this.makeRequest("/health");
  }

  async getMetrics(source = null) {
    const endpoint = source ? `/metrics/${source}` : "/metrics";
    return this.makeAuthenticatedRequest(endpoint);
  }

  async makeRequest(endpoint) {
    try {
      const response = await axios.get(`${this.baseUrl}${endpoint}`);
      return response.data;
    } catch (error) {
      throw this.handleError(error);
    }
  }

  async makeAuthenticatedRequest(endpoint) {
    try {
      const response = await axios.get(`${this.baseUrl}${endpoint}`, {
        headers: {
          Authorization: `Bearer ${this.apiKey}`,
          "Content-Type": "application/json",
        },
      });
      return response.data;
    } catch (error) {
      throw this.handleError(error);
    }
  }

  handleError(error) {
    if (error.response) {
      const { status, data } = error.response;

      switch (status) {
        case 401:
          return new Error("Invalid API key");
        case 404:
          return new Error(data.error || "Endpoint not found");
        case 429:
          const retryAfter = data.retryAfter || 60;
          return new Error(`Rate limit exceeded. Retry after ${retryAfter} seconds`);
        case 503:
          return new Error("Service temporarily unavailable");
        default:
          return new Error(data.error || "Unknown error");
      }
    }

    if (error.code === "ECONNABORTED") {
      return new Error("Request timeout");
    }

    return new Error("Network error");
  }
}

Rate Limiting

Health Endpoint Limits

Requests: 60 requests per minute
Burst: 10 requests per second
Authentication: Not required

Metrics Endpoint Limits

Requests: 100 requests per minute per API key
Burst: 20 requests per second per API key
Authentication: Required

Rate Limit Headers

X-Rate-Limit-Limit: 100
X-Rate-Limit-Remaining: 95
X-Rate-Limit-Reset: 1642248000
X-Rate-Limit-Retry-After: 30

Rate Limit Handling

class RateLimitedClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.rateLimits = {};
  }

  async makeRequest(endpoint) {
    // Check rate limits
    if (this.isRateLimited(endpoint)) {
      const waitTime = this.getWaitTime(endpoint);
      console.log(`Rate limited, waiting ${waitTime}ms...`);
      await new Promise((resolve) => setTimeout(resolve, waitTime));
    }

    try {
      const response = await axios.get(endpoint, {
        headers: { Authorization: `Bearer ${this.apiKey}` },
      });

      // Update rate limit info
      this.updateRateLimits(endpoint, response.headers);

      return response.data;
    } catch (error) {
      if (error.response?.status === 429) {
        const retryAfter = error.response.headers["x-rate-limit-retry-after"] || 60;
        this.setRateLimit(endpoint, retryAfter * 1000);
      }
      throw error;
    }
  }

  isRateLimited(endpoint) {
    const limit = this.rateLimits[endpoint];
    return limit && limit.until > Date.now();
  }

  getWaitTime(endpoint) {
    const limit = this.rateLimits[endpoint];
    return limit ? Math.max(0, limit.until - Date.now()) : 0;
  }

  setRateLimit(endpoint, waitTime) {
    this.rateLimits[endpoint] = {
      until: Date.now() + waitTime,
    };
  }

  updateRateLimits(endpoint, headers) {
    const remaining = headers["x-rate-limit-remaining"];
    const reset = headers["x-rate-limit-reset"];

    if (remaining !== undefined && reset !== undefined) {
      const resetTime = new Date(reset * 1000);
      const now = new Date();

      if (remaining === "0" && resetTime > now) {
        this.setRateLimit(endpoint, resetTime.getTime() - now.getTime());
      }
    }
  }
}

Monitoring and Alerting

Health Monitoring Setup

class HealthMonitor {
  constructor(apiKey, checkInterval = 60000) {
    this.apiKey = apiKey;
    this.checkInterval = checkInterval;
    this.alertHandlers = [];
    this.lastHealth = null;
  }

  start() {
    this.monitorInterval = setInterval(() => {
      this.performHealthCheck();
    }, this.checkInterval);

    console.log("Health monitoring started");
  }

  stop() {
    if (this.monitorInterval) {
      clearInterval(this.monitorInterval);
      console.log("Health monitoring stopped");
    }
  }

  async performHealthCheck() {
    try {
      const health = await this.getHealth();
      this.evaluateHealth(health);
      this.lastHealth = health;
    } catch (error) {
      console.error("Health check failed:", error);
      this.triggerAlert("health_check_failed", { error: error.message });
    }
  }

  evaluateHealth(health) {
    if (!this.lastHealth) {
      return; // First check, no comparison
    }

    // Check for status changes
    if (health.status !== this.lastHealth.status) {
      this.triggerAlert("status_changed", {
        from: this.lastHealth.status,
        to: health.status,
      });
    }

    // Check component changes
    Object.entries(health.checks).forEach(([component, status]) => {
      const lastStatus = this.lastHealth.checks[component];
      if (status !== lastStatus) {
        this.triggerAlert("component_changed", {
          component,
          from: lastStatus,
          to: status,
        });
      }
    });
  }

  triggerAlert(type, data) {
    const alert = {
      type,
      data,
      timestamp: new Date().toISOString(),
    };

    console.warn("🚨 Health Alert:", alert);

    this.alertHandlers.forEach((handler) => {
      try {
        handler(alert);
      } catch (error) {
        console.error("Alert handler failed:", error);
      }
    });
  }

  addAlertHandler(handler) {
    this.alertHandlers.push(handler);
  }
}

Best Practices

For Health Checks

Regular Monitoring: Check health every 1-5 minutes
Alert Thresholds: Set appropriate alert triggers
Graceful Degradation: Handle partial system failures
Documentation: Document all health states and responses

For Metrics

Authentication: Always include valid API key
Rate Limiting: Respect rate limits and implement backoff
Error Handling: Handle all error states gracefully
Caching: Cache metrics data appropriately

For Integration

Timeouts: Set appropriate request timeouts
Retries: Implement exponential backoff for failed requests
Logging: Log all health checks and metrics requests
Monitoring: Monitor your monitoring system

For more information about specific metrics, see Performance Metrics and Source-Specific Metrics.

​Health Endpoints

​Available Endpoints

​Health Check

​General Metrics

​Source-Specific Metrics

​Health Check Endpoint

​Request

​Response Format

​Response Fields

​Overall Status

​Component Checks

​Component Details

​Implementation Examples

​Health Check (JavaScript)

​Health Check (Python)

​Metrics Endpoints

​General Metrics

​Source-Specific Metrics

​Response Headers

​Error Handling

​HTTP Status Codes

​Success Codes

​Error Codes

​Error Response Format

​Error Handling Implementation

​Rate Limiting

​Health Endpoint Limits

​Metrics Endpoint Limits

​Rate Limit Headers

​Rate Limit Handling

​Monitoring and Alerting

​Health Monitoring Setup

​Best Practices

​For Health Checks

​For Metrics

​For Integration

Health Endpoints

Available Endpoints

Health Check

General Metrics

Source-Specific Metrics

Health Check Endpoint

Request

Response Format

Response Fields

Overall Status

Component Checks

Component Details

Implementation Examples

Health Check (JavaScript)

Health Check (Python)

Metrics Endpoints

General Metrics

Source-Specific Metrics

Response Headers

Error Handling

HTTP Status Codes

Success Codes

Error Codes

Error Response Format

Error Handling Implementation

Rate Limiting

Health Endpoint Limits

Metrics Endpoint Limits

Rate Limit Headers

Rate Limit Handling

Monitoring and Alerting

Health Monitoring Setup

Best Practices

For Health Checks

For Metrics

For Integration