feat(crawler): add HTTP crawler strategy for lightweight web scraping

Browse Source

Implements a new AsyncHTTPCrawlerStrategy class that provides a fast, memory-efficient alternative to browser-based crawling. Features include:
- Support for HTTP/HTTPS requests with configurable methods, headers, and timeouts
- File and raw content handling capabilities
- Streaming response processing for large files
- Customizable request/response hooks
- Comprehensive error handling

Also refactors browser management code into separate module for better organization.

...

This commit is contained in:

UncleCode

2025-02-15 19:26:30 +08:00

parent 063df572b0

commit 8bb799068e

7 changed files with 1353 additions and 851 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1077

crawl4ai/async_crawler_strategy.py

File diff suppressed because it is too large Load Diff