UncleCode 72d8e679ad feat(pipeline): add high-level Crawler utility class for simplified web crawling ...

Add new Crawler class that provides a simplified interface for both single and batch URL crawling operations. Key features include:
- Simple single URL crawling with configurable options
- Parallel batch crawling with concurrency control
- Shared browser hub support for resource efficiency
- Progress tracking and custom retry strategies
- Comprehensive error handling and retry logic

Remove demo and extended test files in favor of new focused test suite.

2025-04-07 22:50:44 +08:00

5.1 KiB

Raw Permalink Blame History

View Raw