ca3e33122e
Reorganize documentation into core/advanced/extraction sections for better navigation. Update terminal theme styles and add rich library for better CLI output. Remove redundant tutorial files and consolidate content into core sections. Add personal story to index page for project context. BREAKING CHANGE: Documentation structure has been significantly reorganized
867 B
867 B
Crawl Dispatcher
We’re excited to announce a Crawl Dispatcher module that can handle thousands of crawling tasks simultaneously. By efficiently managing system resources (memory, CPU, network), this dispatcher ensures high-performance data extraction at scale. It also provides real-time monitoring of each crawler’s status, memory usage, and overall progress.
Stay tuned—this feature is coming soon in an upcoming release of Crawl4AI! For the latest news, keep an eye on our changelogs and follow @unclecode on X.
Below is a sample of how the dispatcher’s performance monitor might look in action:
We can’t wait to bring you this streamlined, scalable approach to multi-URL crawling—watch this space for updates!