# Crawl4AI Stress Testing and Benchmarking

This directory contains tools for stress testing Crawl4AI's `arun_many` method and dispatcher system with high volumes of URLs to evaluate performance, concurrency handling, and potentially detect memory issues. It also includes a benchmarking system to track performance over time.

## Quick Start

```bash
# Run a default stress test (small config) and generate a report
# (Assumes run_all.sh is updated to call run_benchmark.py)
./run_all.sh
```
*Note: `run_all.sh` might need to be updated if it directly called the old script.*

## Overview

The stress testing system works by:

1.  Generating a local test site with heavy HTML pages (regenerated by default for each test).
2.  Starting a local HTTP server to serve these pages.
3.  Running Crawl4AI's `arun_many` method against this local site using the `MemoryAdaptiveDispatcher` with configurable concurrency (`max_sessions`).
4.  Monitoring performance metrics via the `CrawlerMonitor` and optionally logging memory usage.
5.  Optionally generating detailed benchmark reports with visualizations using `benchmark_report.py`.

## Available Tools

-   `test_stress_sdk.py` - Main stress testing script utilizing `arun_many` and dispatchers.
-   `benchmark_report.py` - Report generator for comparing test results (assumes compatibility with `test_stress_sdk.py` outputs).
-   `run_benchmark.py` - Python script with predefined test configurations that orchestrates tests using `test_stress_sdk.py`.
-   `run_all.sh` - Simple wrapper script (may need updating).

## Usage Guide

### Using Predefined Configurations (Recommended)

The `run_benchmark.py` script offers the easiest way to run standardized tests:

```bash
# Quick test (50 URLs, 4 max sessions)
python run_benchmark.py quick

# Medium test (500 URLs, 16 max sessions)
python run_benchmark.py medium

# Large test (1000 URLs, 32 max sessions)
python run_benchmark.py large

# Extreme test (2000 URLs, 64 max sessions)
python run_benchmark.py extreme

# Custom configuration
python run_benchmark.py custom --urls 300 --max-sessions 24 --chunk-size 50

# Run 'small' test in streaming mode
python run_benchmark.py small --stream

# Override max_sessions for the 'medium' config
python run_benchmark.py medium --max-sessions 20

# Skip benchmark report generation after the test
python run_benchmark.py small --no-report

# Clean up reports and site files before running
python run_benchmark.py medium --clean
```

#### `run_benchmark.py` Parameters

| Parameter            | Default         | Description                                                                 |
| -------------------- | --------------- | --------------------------------------------------------------------------- |
| `config`             | *required*      | Test configuration: `quick`, `small`, `medium`, `large`, `extreme`, `custom`|
| `--urls`             | config-specific | Number of URLs (required for `custom`)                                      |
| `--max-sessions`     | config-specific | Max concurrent sessions managed by dispatcher (required for `custom`)         |
| `--chunk-size`       | config-specific | URLs per batch for non-stream logging (required for `custom`)               |
| `--stream`           | False           | Enable streaming results (disables batch logging)                           |
| `--monitor-mode`     | DETAILED        | `DETAILED` or `AGGREGATED` display for the live monitor                     |
| `--use-rate-limiter` | False           | Enable basic rate limiter in the dispatcher                                 |
| `--port`             | 8000            | HTTP server port                                                            |
| `--no-report`        | False           | Skip generating comparison report via `benchmark_report.py`                 |
| `--clean`            | False           | Clean up reports and site files before running                              |
| `--keep-server-alive`| False           | Keep local HTTP server running after test                                   |
| `--use-existing-site`| False           | Use existing site on specified port (no local server start/site gen)        |
| `--skip-generation`  | False           | Use existing site files but start local server                              |
| `--keep-site`        | False           | Keep generated site files after test                                        |

#### Predefined Configurations

| Configuration | URLs   | Max Sessions | Chunk Size | Description                      |
| ------------- | ------ | ------------ | ---------- | -------------------------------- |
| `quick`       | 50     | 4            | 10         | Quick test for basic validation  |
| `small`       | 100    | 8            | 20         | Small test for routine checks    |
| `medium`      | 500    | 16           | 50         | Medium test for thorough checks  |
| `large`       | 1000   | 32           | 100        | Large test for stress testing    |
| `extreme`     | 2000   | 64           | 200        | Extreme test for limit testing   |

### Direct Usage of `test_stress_sdk.py`

For fine-grained control or debugging, you can run the stress test script directly:

```bash
# Test with 200 URLs and 32 max concurrent sessions
python test_stress_sdk.py --urls 200 --max-sessions 32 --chunk-size 40

# Clean up previous test data first
python test_stress_sdk.py --clean-reports --clean-site --urls 100 --max-sessions 16 --chunk-size 20

# Change the HTTP server port and use aggregated monitor
python test_stress_sdk.py --port 8088 --urls 100 --max-sessions 16 --monitor-mode AGGREGATED

# Enable streaming mode and use rate limiting
python test_stress_sdk.py --urls 50 --max-sessions 8 --stream --use-rate-limiter

# Change report output location
python test_stress_sdk.py --report-path custom_reports --urls 100 --max-sessions 16
```

#### `test_stress_sdk.py` Parameters

| Parameter            | Default    | Description                                                          |
| -------------------- | ---------- | -------------------------------------------------------------------- |
| `--urls`             | 100        | Number of URLs to test                                               |
| `--max-sessions`     | 16         | Maximum concurrent crawling sessions managed by the dispatcher       |
| `--chunk-size`       | 10         | Number of URLs per batch (relevant for non-stream logging)           |
| `--stream`           | False      | Enable streaming results (disables batch logging)                    |
| `--monitor-mode`     | DETAILED   | `DETAILED` or `AGGREGATED` display for the live `CrawlerMonitor`     |
| `--use-rate-limiter` | False      | Enable a basic `RateLimiter` within the dispatcher                   |
| `--site-path`        | "test_site"| Path to store/use the generated test site                            |
| `--port`             | 8000       | Port for the local HTTP server                                       |
| `--report-path`      | "reports"  | Path to save test result summary (JSON) and memory samples (CSV)   |
| `--skip-generation`  | False      | Use existing test site files but still start local server            |
| `--use-existing-site`| False      | Use existing site on specified port (no local server/site gen)     |
| `--keep-server-alive`| False      | Keep local HTTP server running after test completion                 |
| `--keep-site`        | False      | Keep the generated test site files after test completion             |
| `--clean-reports`    | False      | Clean up report directory before running                             |
| `--clean-site`       | False      | Clean up site directory before/after running (see script logic)    |

### Generating Reports Only

If you only want to generate a benchmark report from existing test results (assuming `benchmark_report.py` is compatible):

```bash
# Generate a report from existing test results in ./reports/
python benchmark_report.py

# Limit to the most recent 5 test results
python benchmark_report.py --limit 5

# Specify a custom source directory for test results
python benchmark_report.py --reports-dir alternate_results
```

#### `benchmark_report.py` Parameters (Assumed)

| Parameter       | Default              | Description                                                 |
| --------------- | -------------------- | ----------------------------------------------------------- |
| `--reports-dir` | "reports"            | Directory containing `test_stress_sdk.py` result files      |
| `--output-dir`  | "benchmark_reports"  | Directory to save generated HTML reports and charts         |
| `--limit`       | None (all results)   | Limit comparison to N most recent test results              |
| `--output-file` | Auto-generated       | Custom output filename for the HTML report                  |

## Understanding the Test Output

### Real-time Progress Display (`CrawlerMonitor`)

When running `test_stress_sdk.py`, the `CrawlerMonitor` provides a live view of the crawling process managed by the dispatcher.

-   **DETAILED Mode (Default):** Shows individual task status (Queued, Active, Completed, Failed), timings, memory usage per task (if `psutil` is available), overall queue statistics, and memory pressure status (if `psutil` available).
-   **AGGREGATED Mode:** Shows summary counts (Queued, Active, Completed, Failed), overall progress percentage, estimated time remaining, average URLs/sec, and memory pressure status.

### Batch Log Output (Non-Streaming Mode Only)

If running `test_stress_sdk.py` **without** the `--stream` flag, you will *also* see per-batch summary lines printed to the console *after* the monitor display, once each chunk of URLs finishes processing:

```
 Batch | Progress | Start Mem | End Mem   | URLs/sec | Success/Fail | Time (s) | Status
───────────────────────────────────────────────────────────────────────────────────────────
 1     |  10.0%   |  50.1 MB  |  55.3 MB  |    23.8    |    10/0      |     0.42   | Success
 2     |  20.0%   |  55.3 MB  |  60.1 MB  |    24.1    |    10/0      |     0.41   | Success
 ...
```

This display provides chunk-specific metrics:
-   **Batch**: The batch number being reported.
-   **Progress**: Overall percentage of total URLs processed *after* this batch.
-   **Start Mem / End Mem**: Memory usage before and after processing this batch (if tracked).
-   **URLs/sec**: Processing speed *for this specific batch*.
-   **Success/Fail**: Number of successful and failed URLs *in this batch*.
-   **Time (s)**: Wall-clock time taken to process *this batch*.
-   **Status**: Color-coded status for the batch outcome.

### Summary Output

After test completion, a final summary is displayed:

```
================================================================================
Test Completed
================================================================================
Test ID: 20250418_103015
Configuration: 100 URLs, 16 max sessions, Chunk: 10, Stream: False, Monitor: DETAILED
Results: 100 successful, 0 failed (100 processed, 100.0% success)
Performance: 5.85 seconds total, 17.09 URLs/second avg
Memory Usage: Start: 50.1 MB, End: 75.3 MB, Max: 78.1 MB, Growth: 25.2 MB
Results summary saved to reports/test_summary_20250418_103015.json
```

### HTML Report Structure (Generated by `benchmark_report.py`)

(This section remains the same, assuming `benchmark_report.py` generates these)
The benchmark report contains several sections:
1.  **Summary**: Overview of the latest test results and trends
2.  **Performance Comparison**: Charts showing throughput across tests
3.  **Memory Usage**: Detailed memory usage graphs for each test
4.  **Detailed Results**: Tabular data of all test metrics
5.  **Conclusion**: Automated analysis of performance and memory patterns

### Memory Metrics

(This section remains conceptually the same)
Memory growth is the key metric for detecting leaks...

### Performance Metrics

(This section remains conceptually the same, though "URLs per Worker" is less relevant - focus on overall URLs/sec)
Key performance indicators include:
-   **URLs per Second**: Higher is better (throughput)
-   **Success Rate**: Should be 100% in normal conditions
-   **Total Processing Time**: Lower is better
-   **Dispatcher Efficiency**: Observe queue lengths and wait times in the monitor (Detailed mode)

### Raw Data Files

Raw data is saved in the `--report-path` directory (default `./reports/`):

-   **JSON files** (`test_summary_*.json`): Contains the final summary for each test run.
-   **CSV files** (`memory_samples_*.csv`): Contains time-series memory samples taken during the test run.

Example of reading raw data:
```python
import json
import pandas as pd

# Load test summary
test_id = "20250418_103015" # Example ID
with open(f'reports/test_summary_{test_id}.json', 'r') as f:
    results = json.load(f)

# Load memory samples
memory_df = pd.read_csv(f'reports/memory_samples_{test_id}.csv')

# Analyze memory_df (e.g., calculate growth, plot)
if not memory_df['memory_info_mb'].isnull().all():
    growth = memory_df['memory_info_mb'].iloc[-1] - memory_df['memory_info_mb'].iloc[0]
    print(f"Total Memory Growth: {growth:.1f} MB")
else:
    print("No valid memory samples found.")

print(f"Avg URLs/sec: {results['urls_processed'] / results['total_time_seconds']:.2f}")
```

## Visualization Dependencies

(This section remains the same)
For full visualization capabilities in the HTML reports generated by `benchmark_report.py`, install additional dependencies...

## Directory Structure

```
benchmarking/          # Or your top-level directory name
├── benchmark_reports/ # Generated HTML reports (by benchmark_report.py)
├── reports/           # Raw test result data (from test_stress_sdk.py)
├── test_site/         # Generated test content (temporary)
├── benchmark_report.py# Report generator
├── run_benchmark.py   # Test runner with predefined configs
├── test_stress_sdk.py # Main stress test implementation using arun_many
└── run_all.sh         # Simple wrapper script (may need updates)
#└── requirements.txt   # Optional: Visualization dependencies for benchmark_report.py
```

## Cleanup

To clean up after testing:

```bash
# Remove the test site content (if not using --keep-site)
rm -rf test_site

# Remove all raw reports and generated benchmark reports
rm -rf reports benchmark_reports

# Or use the --clean flag with run_benchmark.py
python run_benchmark.py medium --clean
```

## Use in CI/CD

(This section remains conceptually the same, just update script names)
These tests can be integrated into CI/CD pipelines:
```bash
# Example CI script
python run_benchmark.py medium --no-report # Run test without interactive report gen
# Check exit code
if [ $? -ne 0 ]; then echo "Stress test failed!"; exit 1; fi
# Optionally, run report generator and check its output/metrics
# python benchmark_report.py
# check_report_metrics.py reports/test_summary_*.json || exit 1
exit 0
```

## Troubleshooting

-   **HTTP Server Port Conflict**: Use `--port` with `run_benchmark.py` or `test_stress_sdk.py`.
-   **Memory Tracking Issues**: The `SimpleMemoryTracker` uses platform commands (`ps`, `/proc`, `tasklist`). Ensure these are available and the script has permission. If it consistently fails, memory reporting will be limited.
-   **Visualization Missing**: Related to `benchmark_report.py` and its dependencies.
-   **Site Generation Issues**: Check permissions for creating `./test_site/`. Use `--skip-generation` if you want to manage the site manually.
-   **Testing Against External Site**: Ensure the external site is running and use `--use-existing-site --port <correct_port>`.