UncleCode 0094cac675 refactor(browser): improve parallel crawling and browser management ...

Remove PagePoolConfig in favor of direct page management in browser strategies.
Add get_pages() method for efficient parallel page creation.
Improve storage state handling and persistence.
Add comprehensive parallel crawling tests and performance analysis.

BREAKING CHANGE: Removed PagePoolConfig class and related functionality.

2025-03-23 18:53:24 +08:00

..

20241401

feat(schema): improve HTML preprocessing for schema generation

2025-03-12 22:40:46 +08:00

async

feat(browser): add standalone CDP browser launch and lxml extraction strategy

2025-03-07 20:55:56 +08:00

browser

refactor(browser): improve parallel crawling and browser management

2025-03-23 18:53:24 +08:00

cli

feat(cli): add command line interface with comprehensive features

2025-02-10 16:58:52 +08:00

docker

feat(crawler): optimize single URL handling and add performance comparison

2025-03-13 22:15:15 +08:00

hub

refactor(crawler): improve HTML handling and cleanup codebase

2025-02-07 21:56:27 +08:00

loggers

Release prep (#749)

2025-02-28 19:53:35 +08:00

memory

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

__init__.py

- Test all methods

2024-05-14 21:27:41 +08:00

docker_example.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_cli_docs.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_docker.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_llmtxt.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_main.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_scraping_strategy.py

Release prep (#749)

2025-02-28 19:53:35 +08:00

test_web_crawler.py

feat(browser): add standalone CDP browser launch and lxml extraction strategy

2025-03-07 20:55:56 +08:00