UncleCode 825c78a048 refactor(dispatcher): migrate to modular dispatcher system with enhanced monitoring ...

Reorganize dispatcher functionality into separate components:
- Create dedicated dispatcher classes (MemoryAdaptive, Semaphore)
- Add RateLimiter for smart request throttling
- Implement CrawlerMonitor for real-time progress tracking
- Move dispatcher config from CrawlerRunConfig to separate classes

BREAKING CHANGE: Dispatcher configuration moved from CrawlerRunConfig to dedicated dispatcher classes. Users need to update their configuration approach for multi-URL crawling.

2025-01-11 21:10:27 +08:00

4.5 KiB

Raw Blame History

View Raw