ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(crawler): optimize single URL handling and add performance comparison

Browse Source 
Add special handling for single URL requests in Docker API to use arun() instead of arun_many()
Add new example script demonstrating performance differences between sequential and parallel crawling
Update cache mode from aggressive to bypass in examples and tests
Remove unused dependencies (zstandard, msgpack)

BREAKING CHANGE: Changed default cache_mode from aggressive to bypass in examples
This commit is contained in:
UncleCode
2025-03-13 22:15:15 +08:00
parent dc36997a08
commit b750542e6d
6 changed files with 95 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
pyproject.toml
View File

@@ -43,8 +43,6 @@ dependencies = [
"faust-cchardet>=2.1.19",
"aiohttp>=3.11.11",
"humanize>=4.10.0",
"zstandard>=0.23.0",
"msgpack>=1.1.0"
]
classifiers = [
"Development Status :: 4 - Beta",