UncleCode d024749633 refactor(deep-crawl): add max_pages limit and improve crawl control ...

Add max_pages parameter to all deep crawling strategies to limit total pages crawled.
Add score_threshold parameter to BFS/DFS strategies for quality control.
Remove legacy parameter handling in AsyncWebCrawler.
Improve error handling and logging in crawl strategies.

BREAKING CHANGE: Removed support for legacy parameters in AsyncWebCrawler.run_many()

2025-03-03 21:51:11 +08:00

18 KiB

Raw Blame History

View Raw