unclecode 91f7b9d129 feat(docker): add multi-container cluster deployment with CLI management ...

Add comprehensive Docker cluster orchestration with horizontal scaling support.

CLI Commands:
- crwl server start/stop/restart/status/scale/logs
- Auto-detection: Single (N=1) → Swarm (N>1) → Compose (N>1 fallback)
- Support for 1-100 container replicas with zero-downtime scaling

Infrastructure:
- Nginx load balancing (round-robin API, sticky sessions monitoring)
- Redis-based container discovery via heartbeats (30s interval)
- Real-time monitoring dashboard with cluster-wide visibility
- WebSocket aggregation from all containers

Security & Stability Fixes (12 critical issues):
- Add timeout protection to browser pool locks (prevent deadlocks)
- Implement Redis retry logic with exponential backoff
- Add container ID validation (prevent Redis key injection)
- Add CLI input sanitization (prevent shell injection)
- Add file locking for state management (prevent corruption)
- Fix WebSocket resource leaks and connection cleanup
- Add graceful degradation and circuit breakers

Configuration:
- RedisTTLConfig dataclass with environment variable support
- Template-based docker-compose.yml and nginx.conf generation
- Comprehensive error handling with actionable messages

Documentation:
- AGENT.md: Complete DevOps context for AI assistants
- MULTI_CONTAINER_ARCHITECTURE.md: Technical architecture guide
- Reorganized docs into deploy/docker/docs/

2025-10-19 13:31:14 +08:00

..

components

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

crawlers

Update all documentation to import extraction strategies directly from crawl4ai.

2025-06-10 18:08:27 +08:00

deep_crawling

Merge pull request #1389 from unclecode/fix/deep-crawl-scoring

2025-09-16 15:45:54 +08:00

html2text

2025 feb alpha 1 (#685)

2025-02-19 14:13:17 +08:00

js_snippet

fix: Add a nominal wait time for remove overlay elements since it's already controllable through delay_before_return_html

2025-04-14 12:39:05 +05:30

legacy

Squashed commit of the following:

2025-08-04 19:02:01 +08:00

processors/pdf

fix(pdf): add timeout to PDF downloads to prevent hanging (#1141)

2025-05-23 16:05:44 +08:00

script

Add C4A-Script support and documentation

2025-06-07 23:07:19 +08:00

templates

feat(docker): add multi-container cluster deployment with CLI management

2025-10-19 13:31:14 +08:00

__init__.py

feat: 🚀 Introduce revolutionary LLMTableExtraction with intelligent chunking for massive tables

2025-08-14 18:21:24 +08:00

__version__.py

chore(version): update version to 0.7.4

2025-08-17 19:22:23 +08:00

adaptive_crawler copy.py

feat(crawl4ai): Implement adaptive crawling feature

2025-07-04 15:16:53 +08:00

adaptive_crawler.py

fix: allow custom LLM providers for adaptive crawler embedding config. ref: #1291

2025-09-09 12:49:55 +08:00

async_configs.py

Merge pull request #1464 from unclecode/fix/proxy_deprecation

2025-09-16 15:48:45 +08:00

async_crawler_strategy.back.py

fix(crawler): Removed the incorrect reference in browser_config variable #1310

2025-08-18 10:59:14 +08:00

async_crawler_strategy.py

fix(http-crawler): bring back HTTP crawler strategy

2025-08-16 09:27:23 +08:00

async_database.py

WIP: logger migriate to rich

2025-04-11 00:44:43 +08:00

async_dispatcher.py

fix: always return a list, even if we catch an exception

2025-10-02 09:21:44 +02:00

async_logger.py

Merge branch '2025-MAY-2' into next-MAY

2025-07-08 11:46:13 +02:00

async_url_seeder.py

fix: Enable following redirects in sitemap fetching for seeder

2025-07-31 12:06:10 +08:00

async_webcrawler.py

feat: add preserve_https_for_internal_links flag to maintain HTTPS during crawling. Ref #1410

2025-08-28 17:38:40 +08:00

browser_adapter.py

feat(StealthAdapter): fix stealth features for Playwright integration. ref #1481

2025-09-18 15:39:06 +08:00

browser_manager.py

feat(StealthAdapter): fix stealth features for Playwright integration. ref #1481

2025-09-18 15:39:06 +08:00

browser_profiler.py

feat(browser-profiler): implement cross-platform keyboard listeners and improve quit handling

2025-08-08 11:18:34 +08:00

cache_context.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

chunking_strategy.py

refactor(core): reorganize project structure and remove legacy code

2025-01-30 19:35:06 +08:00

cli.py

feat(docker): add multi-container cluster deployment with CLI management

2025-10-19 13:31:14 +08:00

config.py

refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling

2025-04-15 22:27:18 +08:00

content_filter_strategy.py

Add use_stemming option to BM25ContentFilter (#1192)

2025-06-10 15:44:45 +08:00

content_scraping_strategy.py

feat: add preserve_https_for_internal_links flag to maintain HTTPS during crawling. Ref #1410

2025-08-28 17:38:40 +08:00

docker_client.py

fix(auth): add token authorization header in request preparation to ensure authenticated requests are made

2025-05-21 13:27:17 +08:00

extraction_strategy.py

Merge branch '2025-MAY-2' of https://github.com/unclecode/crawl4ai into 2025-MAY-2

2025-06-13 11:26:17 +02:00

hub.py

feat(logger): add abstract logger base class and file logger implementation

2025-02-23 21:23:41 +08:00

install.py

feat: add stealth mode and enhance undetected browser support

2025-07-17 16:59:10 +08:00

link_preview.py

refactor(link_extractor): remove link_extractor and rename to link_preview

2025-06-27 21:54:22 +08:00

markdown_generation_strategy.py

feat(markdown): add content source selection for markdown generation

2025-04-17 20:13:53 +08:00

migrations.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

model_loader.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

models.py

fix(docker): resolve filter serialization and JSON encoding errors in deep crawl strategy (ref #1419)

2025-08-25 14:04:08 +08:00

prompts.py

feat: add undetected browser support with adapter pattern

2025-07-14 17:29:50 +08:00

proxy_strategy.py

refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling

2025-04-15 22:27:18 +08:00

server_cli.py

feat(docker): add multi-container cluster deployment with CLI management

2025-10-19 13:31:14 +08:00

server_manager.py

feat(docker): add multi-container cluster deployment with CLI management

2025-10-19 13:31:14 +08:00

ssl_certificate.py

refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling

2025-04-15 22:27:18 +08:00

table_extraction.py

feat: 🚀 Introduce revolutionary LLMTableExtraction with intelligent chunking for massive tables

2025-08-14 18:21:24 +08:00

types.py

Squashed commit of the following:

2025-08-04 19:02:01 +08:00

user_agent_generator.py

feat(browser): add BrowserProfiler class for identity-based browsing

2025-03-02 20:32:29 +08:00

utils.py

fix: raise error on last attempt failure in perform_completion_with_backoff. ref #989

2025-09-02 16:49:01 +08:00