ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(crawler): Enhance stealth and flexibility, improve error handling

Browse Source 
- Implement playwright_stealth for better bot detection avoidance
- Add user simulation and navigator override options
- Improve iframe processing and browser selection
- Enhance error reporting and debugging capabilities
- Optimize image processing and parallel crawling
- Add new example for user simulation feature
- Added support for including links in Markdown content, by definin g a new flag `include_links_on_markdown` in `crawl` method.
This commit is contained in:
UncleCode
2024-10-17 21:37:48 +08:00
parent 9ffa34b697
commit 768aa06ceb
8 changed files with 777 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@@ -202,5 +202,7 @@ todo.md
git_changes.py
git_changes.md
pypi_build.sh
git_issues.py
git_issues.md
.tests/