73 lines
1.5 KiB
Markdown
73 lines
1.5 KiB
Markdown
# Link Scraper
|
|
|
|
A lightweight API built with Express and Playwright to scrape and extract links from web pages.
|
|
|
|
## Features
|
|
|
|
- **Singleton Browser Pattern:** Fast and memory-efficient.
|
|
- **SSRF Protection:** Built-in validation to prevent private and local network access.
|
|
- **API Key Security:** Protected endpoints using header authentication.
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
npm install
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Set the following environment variables (e.g., in a `.env` file or in Coolify settings):
|
|
|
|
- `PORT`: (Optional) Port the server will run on (Default: `3001`).
|
|
- `API_KEY`: The secret token to authenticate requests. If not set, requests are allowed in development mode but blocked in production.
|
|
- `NODE_ENV`: Set to `production` in production mode to enforce strict security.
|
|
|
|
## Running the Application
|
|
|
|
### Development Mode
|
|
|
|
```bash
|
|
npm run dev
|
|
```
|
|
|
|
### Production Build
|
|
|
|
```bash
|
|
npm run build
|
|
npm start
|
|
```
|
|
|
|
## API Documentation
|
|
|
|
### Health Check
|
|
|
|
`GET /health` - Checks if the API is running (does not require authentication).
|
|
|
|
### Scrape Navigation Links
|
|
|
|
`POST /scrape` - Extracts navigation links from the target URL.
|
|
|
|
**Headers:**
|
|
- `Content-Type: application/json`
|
|
- `x-api-key: your_secret_api_key` (Or `Authorization: Bearer your_secret_api_key`)
|
|
|
|
**Request Body:**
|
|
```json
|
|
{
|
|
"url": "https://example.com"
|
|
}
|
|
```
|
|
|
|
**Response Body:**
|
|
```json
|
|
{
|
|
"source": "https://example.com",
|
|
"count": 12,
|
|
"links": [
|
|
{ "label": "About Us", "url": "https://example.com/about" },
|
|
{ "label": "Contact", "url": "https://example.com/contact" }
|
|
]
|
|
}
|
|
```
|
|
|