Link Scraper

A lightweight API built with Express and Playwright to scrape and extract links from web pages.

Features

Singleton Browser Pattern: Fast and memory-efficient.
SSRF Protection: Built-in validation to prevent private and local network access.
API Key Security: Protected endpoints using header authentication.

Installation

npm install

Configuration

Set the following environment variables (e.g., in a .env file or in Coolify settings):

PORT: (Optional) Port the server will run on (Default: 3001).
API_KEY: The secret token to authenticate requests. If not set, requests are allowed in development mode but blocked in production.
NODE_ENV: Set to production in production mode to enforce strict security.

Running the Application

Development Mode

npm run dev

Production Build

npm run build
npm start

API Documentation

Health Check

GET /health - Checks if the API is running (does not require authentication).

POST /scrape - Extracts navigation links from the target URL.

Headers:

Content-Type: application/json
x-api-key: your_secret_api_key (Or Authorization: Bearer your_secret_api_key)

Request Body:

{
  "url": "https://example.com"
}

Response Body:

{
  "source": "https://example.com",
  "count": 12,
  "links": [
    { "label": "About Us", "url": "https://example.com/about" },
    { "label": "Contact", "url": "https://example.com/contact" }
  ]
}

1.5 KiB Raw Blame History