feat(docker): add multi-container cluster deployment with CLI management
Add comprehensive Docker cluster orchestration with horizontal scaling support. CLI Commands: - crwl server start/stop/restart/status/scale/logs - Auto-detection: Single (N=1) → Swarm (N>1) → Compose (N>1 fallback) - Support for 1-100 container replicas with zero-downtime scaling Infrastructure: - Nginx load balancing (round-robin API, sticky sessions monitoring) - Redis-based container discovery via heartbeats (30s interval) - Real-time monitoring dashboard with cluster-wide visibility - WebSocket aggregation from all containers Security & Stability Fixes (12 critical issues): - Add timeout protection to browser pool locks (prevent deadlocks) - Implement Redis retry logic with exponential backoff - Add container ID validation (prevent Redis key injection) - Add CLI input sanitization (prevent shell injection) - Add file locking for state management (prevent corruption) - Fix WebSocket resource leaks and connection cleanup - Add graceful degradation and circuit breakers Configuration: - RedisTTLConfig dataclass with environment variable support - Template-based docker-compose.yml and nginx.conf generation - Comprehensive error handling with actionable messages Documentation: - AGENT.md: Complete DevOps context for AI assistants - MULTI_CONTAINER_ARCHITECTURE.md: Technical architecture guide - Reorganized docs into deploy/docker/docs/
This commit is contained in:
unclecode
241
deploy/docker/docs/STRESS_TEST_PIPELINE.md
Normal file
241
deploy/docker/docs/STRESS_TEST_PIPELINE.md
Normal file
@@ -0,0 +1,241 @@
|
||||
# Crawl4AI Docker Memory & Pool Optimization - Implementation Log
|
||||
|
||||
## Critical Issues Identified
|
||||
|
||||
### Memory Management
|
||||
- **Host vs Container**: `psutil.virtual_memory()` reported host memory, not container limits
|
||||
- **Browser Pooling**: No pool reuse - every endpoint created new browsers
|
||||
- **Warmup Waste**: Permanent browser sat idle with mismatched config signature
|
||||
- **Idle Cleanup**: 30min TTL too long, janitor ran every 60s
|
||||
- **Endpoint Inconsistency**: 75% of endpoints bypassed pool (`/md`, `/html`, `/screenshot`, `/pdf`, `/execute_js`, `/llm`)
|
||||
|
||||
### Pool Design Flaws
|
||||
- **Config Mismatch**: Permanent browser used `config.yml` args, endpoints used empty `BrowserConfig()`
|
||||
- **Logging Level**: Pool hit markers at DEBUG, invisible with INFO logging
|
||||
|
||||
## Implementation Changes
|
||||
|
||||
### 1. Container-Aware Memory Detection (`utils.py`)
|
||||
```python
|
||||
def get_container_memory_percent() -> float:
|
||||
# Try cgroup v2 → v1 → fallback to psutil
|
||||
# Reads /sys/fs/cgroup/memory.{current,max} OR memory/memory.{usage,limit}_in_bytes
|
||||
```
|
||||
|
||||
### 2. Smart Browser Pool (`crawler_pool.py`)
|
||||
**3-Tier System:**
|
||||
- **PERMANENT**: Always-ready default browser (never cleaned)
|
||||
- **HOT_POOL**: Configs used 3+ times (longer TTL)
|
||||
- **COLD_POOL**: New/rare configs (short TTL)
|
||||
|
||||
**Key Functions:**
|
||||
- `get_crawler(cfg)`: Check permanent → hot → cold → create new
|
||||
- `init_permanent(cfg)`: Initialize permanent at startup
|
||||
- `janitor()`: Adaptive cleanup (10s/30s/60s intervals based on memory)
|
||||
- `_sig(cfg)`: SHA1 hash of config dict for pool keys
|
||||
|
||||
**Logging Fix**: Changed `logger.debug()` → `logger.info()` for pool hits
|
||||
|
||||
### 3. Endpoint Unification
|
||||
**Helper Function** (`server.py`):
|
||||
```python
|
||||
def get_default_browser_config() -> BrowserConfig:
|
||||
return BrowserConfig(
|
||||
extra_args=config["crawler"]["browser"].get("extra_args", []),
|
||||
**config["crawler"]["browser"].get("kwargs", {}),
|
||||
)
|
||||
```
|
||||
|
||||
**Migrated Endpoints:**
|
||||
- `/html`, `/screenshot`, `/pdf`, `/execute_js` → use `get_default_browser_config()`
|
||||
- `handle_llm_qa()`, `handle_markdown_request()` → same
|
||||
|
||||
**Result**: All endpoints now hit permanent browser pool
|
||||
|
||||
### 4. Config Updates (`config.yml`)
|
||||
- `idle_ttl_sec: 1800` → `300` (30min → 5min base TTL)
|
||||
- `port: 11234` → `11235` (fixed mismatch with Gunicorn)
|
||||
|
||||
### 5. Lifespan Fix (`server.py`)
|
||||
```python
|
||||
await init_permanent(BrowserConfig(
|
||||
extra_args=config["crawler"]["browser"].get("extra_args", []),
|
||||
**config["crawler"]["browser"].get("kwargs", {}),
|
||||
))
|
||||
```
|
||||
Permanent browser now matches endpoint config signatures
|
||||
|
||||
## Test Results
|
||||
|
||||
### Test 1: Basic Health
|
||||
- 10 requests to `/health`
|
||||
- **Result**: 100% success, avg 3ms latency
|
||||
- **Baseline**: Container starts in ~5s, 270 MB idle
|
||||
|
||||
### Test 2: Memory Monitoring
|
||||
- 20 requests with Docker stats tracking
|
||||
- **Result**: 100% success, no memory leak (-0.2 MB delta)
|
||||
- **Baseline**: 269.7 MB container overhead
|
||||
|
||||
### Test 3: Pool Validation
|
||||
- 30 requests to `/html` endpoint
|
||||
- **Result**: **100% permanent browser hits**, 0 new browsers created
|
||||
- **Memory**: 287 MB baseline → 396 MB active (+109 MB)
|
||||
- **Latency**: Avg 4s (includes network to httpbin.org)
|
||||
|
||||
### Test 4: Concurrent Load
|
||||
- Light (10) → Medium (50) → Heavy (100) concurrent
|
||||
- **Total**: 320 requests
|
||||
- **Result**: 100% success, **320/320 permanent hits**, 0 new browsers
|
||||
- **Memory**: 269 MB → peak 1533 MB → final 993 MB
|
||||
- **Latency**: P99 at 100 concurrent = 34s (expected with single browser)
|
||||
|
||||
### Test 5: Pool Stress (Mixed Configs)
|
||||
- 20 requests with 4 different viewport configs
|
||||
- **Result**: 4 new browsers, 4 cold hits, **4 promotions to hot**, 8 hot hits
|
||||
- **Reuse Rate**: 60% (12 pool hits / 20 requests)
|
||||
- **Memory**: 270 MB → 928 MB peak (+658 MB = ~165 MB per browser)
|
||||
- **Proves**: Cold → hot promotion at 3 uses working perfectly
|
||||
|
||||
### Test 6: Multi-Endpoint
|
||||
- 10 requests each: `/html`, `/screenshot`, `/pdf`, `/crawl`
|
||||
- **Result**: 100% success across all 4 endpoints
|
||||
- **Latency**: 5-8s avg (PDF slowest at 7.2s)
|
||||
|
||||
### Test 7: Cleanup Verification
|
||||
- 20 requests (load spike) → 90s idle
|
||||
- **Memory**: 269 MB → peak 1107 MB → final 780 MB
|
||||
- **Recovery**: 327 MB (39%) - partial cleanup
|
||||
- **Note**: Hot pool browsers persist (by design), janitor working correctly
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
| Metric | Before | After | Improvement |
|
||||
|--------|--------|-------|-------------|
|
||||
| Pool Reuse | 0% | 100% (default config) | ∞ |
|
||||
| Memory Leak | Unknown | 0 MB/cycle | Stable |
|
||||
| Browser Reuse | No | Yes | ~3-5s saved per request |
|
||||
| Idle Memory | 500-700 MB × N | 270-400 MB | 10x reduction |
|
||||
| Concurrent Capacity | ~20 | 100+ | 5x |
|
||||
|
||||
## Key Learnings
|
||||
|
||||
1. **Config Signature Matching**: Permanent browser MUST match endpoint default config exactly (SHA1 hash)
|
||||
2. **Logging Levels**: Pool diagnostics need INFO level, not DEBUG
|
||||
3. **Memory in Docker**: Must read cgroup files, not host metrics
|
||||
4. **Janitor Timing**: 60s interval adequate, but TTLs should be short (5min) for cold pool
|
||||
5. **Hot Promotion**: 3-use threshold works well for production patterns
|
||||
6. **Memory Per Browser**: ~150-200 MB per Chromium instance with headless + text_mode
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
**Location**: `deploy/docker/tests/`
|
||||
**Dependencies**: `httpx`, `docker` (Python SDK)
|
||||
**Pattern**: Sequential build - each test adds one capability
|
||||
|
||||
**Files**:
|
||||
- `test_1_basic.py`: Health check + container lifecycle
|
||||
- `test_2_memory.py`: + Docker stats monitoring
|
||||
- `test_3_pool.py`: + Log analysis for pool markers
|
||||
- `test_4_concurrent.py`: + asyncio.Semaphore for concurrency control
|
||||
- `test_5_pool_stress.py`: + Config variants (viewports)
|
||||
- `test_6_multi_endpoint.py`: + Multiple endpoint testing
|
||||
- `test_7_cleanup.py`: + Time-series memory tracking for janitor
|
||||
|
||||
**Run Pattern**:
|
||||
```bash
|
||||
cd deploy/docker/tests
|
||||
pip install -r requirements.txt
|
||||
# Rebuild after code changes:
|
||||
cd /path/to/repo && docker buildx build -t crawl4ai-local:latest --load .
|
||||
# Run test:
|
||||
python test_N_name.py
|
||||
```
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
**Why Permanent Browser?**
|
||||
- 90% of requests use default config → single browser serves most traffic
|
||||
- Eliminates 3-5s startup overhead per request
|
||||
|
||||
**Why 3-Tier Pool?**
|
||||
- Permanent: Zero cost for common case
|
||||
- Hot: Amortized cost for frequent variants
|
||||
- Cold: Lazy allocation for rare configs
|
||||
|
||||
**Why Adaptive Janitor?**
|
||||
- Memory pressure triggers aggressive cleanup
|
||||
- Low memory allows longer TTLs for better reuse
|
||||
|
||||
**Why Not Close After Each Request?**
|
||||
- Browser startup: 3-5s overhead
|
||||
- Pool reuse: <100ms overhead
|
||||
- Net: 30-50x faster
|
||||
|
||||
## Future Optimizations
|
||||
|
||||
1. **Request Queuing**: When at capacity, queue instead of reject
|
||||
2. **Pre-warming**: Predict common configs, pre-create browsers
|
||||
3. **Metrics Export**: Prometheus metrics for pool efficiency
|
||||
4. **Config Normalization**: Group similar viewports (e.g., 1920±50 → 1920)
|
||||
|
||||
## Critical Code Paths
|
||||
|
||||
**Browser Acquisition** (`crawler_pool.py:34-78`):
|
||||
```
|
||||
get_crawler(cfg) →
|
||||
_sig(cfg) →
|
||||
if sig == DEFAULT_CONFIG_SIG → PERMANENT
|
||||
elif sig in HOT_POOL → HOT_POOL[sig]
|
||||
elif sig in COLD_POOL → promote if count >= 3
|
||||
else → create new in COLD_POOL
|
||||
```
|
||||
|
||||
**Janitor Loop** (`crawler_pool.py:107-146`):
|
||||
```
|
||||
while True:
|
||||
mem% = get_container_memory_percent()
|
||||
if mem% > 80: interval=10s, cold_ttl=30s
|
||||
elif mem% > 60: interval=30s, cold_ttl=60s
|
||||
else: interval=60s, cold_ttl=300s
|
||||
sleep(interval)
|
||||
close idle browsers (COLD then HOT)
|
||||
```
|
||||
|
||||
**Endpoint Pattern** (`server.py` example):
|
||||
```python
|
||||
@app.post("/html")
|
||||
async def generate_html(...):
|
||||
from crawler_pool import get_crawler
|
||||
crawler = await get_crawler(get_default_browser_config())
|
||||
results = await crawler.arun(url=body.url, config=cfg)
|
||||
# No crawler.close() - returned to pool
|
||||
```
|
||||
|
||||
## Debugging Tips
|
||||
|
||||
**Check Pool Activity**:
|
||||
```bash
|
||||
docker logs crawl4ai-test | grep -E "(🔥|♨️|❄️|🆕|⬆️)"
|
||||
```
|
||||
|
||||
**Verify Config Signature**:
|
||||
```python
|
||||
from crawl4ai import BrowserConfig
|
||||
import json, hashlib
|
||||
cfg = BrowserConfig(...)
|
||||
sig = hashlib.sha1(json.dumps(cfg.to_dict(), sort_keys=True).encode()).hexdigest()
|
||||
print(sig[:8]) # Compare with logs
|
||||
```
|
||||
|
||||
**Monitor Memory**:
|
||||
```bash
|
||||
docker stats crawl4ai-test
|
||||
```
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- **Mac Docker Stats**: CPU metrics unreliable, memory works
|
||||
- **PDF Generation**: Slowest endpoint (~7s), no optimization yet
|
||||
- **Hot Pool Persistence**: May hold memory longer than needed (trade-off for performance)
|
||||
- **Janitor Lag**: Up to 60s before cleanup triggers in low-memory scenarios
|
||||
Reference in New Issue
Block a user