UncleCode 8bb799068e feat(crawler): add HTTP crawler strategy for lightweight web scraping ...

Implements a new AsyncHTTPCrawlerStrategy class that provides a fast, memory-efficient alternative to browser-based crawling. Features include:
- Support for HTTP/HTTPS requests with configurable methods, headers, and timeouts
- File and raw content handling capabilities
- Streaming response processing for large files
- Customizable request/response hooks
- Comprehensive error handling

Also refactors browser management code into separate module for better organization.

2025-02-15 19:26:30 +08:00

2.3 KiB

Raw Blame History

View Raw