ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(browser): improve parallel crawling and browser management

Browse Source 
Remove PagePoolConfig in favor of direct page management in browser strategies.
Add get_pages() method for efficient parallel page creation.
Improve storage state handling and persistence.
Add comprehensive parallel crawling tests and performance analysis.

BREAKING CHANGE: Removed PagePoolConfig class and related functionality.
This commit is contained in:
UncleCode
2025-03-23 18:53:24 +08:00
parent 4ab0893ffb
commit 0094cac675
7 changed files with 2115 additions and 351 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
tests/browser/test_browser_manager.py
View File

@@ -171,9 +171,9 @@ async def run_tests():
"""Run all tests sequentially."""
results = []
# results.append(await test_basic_browser_manager())
# results.append(await test_custom_browser_config())
# results.append(await test_multiple_pages())
results.append(await test_basic_browser_manager())
results.append(await test_custom_browser_config())
results.append(await test_multiple_pages())
results.append(await test_session_management())
# Print summary