UncleCode 768aa06ceb feat(crawler): Enhance stealth and flexibility, improve error handling ...

- Implement playwright_stealth for better bot detection avoidance
- Add user simulation and navigator override options
- Improve iframe processing and browser selection
- Enhance error reporting and debugging capabilities
- Optimize image processing and parallel crawling
- Add new example for user simulation feature
- Added support for including links in Markdown content, by definin g a new flag `include_links_on_markdown` in `crawl` method.

2024-10-17 21:37:48 +08:00

19 KiB

Raw Blame History

View Raw