ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
16b231824295f561787d1473386473547b668510
crawl4ai/tests
History
UncleCode 16b2318242 feat(api): implement crawler pool manager for improved resource handling 
Adds a new CrawlerManager class to handle browser instance pooling and failover:
- Implements auto-scaling based on system resources
- Adds primary/backup crawler management
- Integrates memory monitoring and throttling
- Adds streaming support with memory tracking
- Updates API endpoints to use pooled crawlers

BREAKING CHANGE: API endpoints now require CrawlerManager initialization
2025-04-18 22:26:24 +08:00
..
async
feat(browser): add standalone CDP browser launch and lxml extraction strategy
2025-03-07 20:55:56 +08:00
browser
feat(browser): implement browser pooling and page pre-warming
2025-03-31 21:55:07 +08:00
cli
feat(cli): add command line interface with comprehensive features
2025-02-10 16:58:52 +08:00
docker
docs(tests): clarify server URL comments in deep crawl tests
2025-04-15 22:32:27 +08:00
general
feat(markdown): add content source selection for markdown generation
2025-04-17 20:13:53 +08:00
hub
refactor(crawler): improve HTML handling and cleanup codebase
2025-02-07 21:56:27 +08:00
loggers
Release prep (#749)
2025-02-28 19:53:35 +08:00
memory
feat(api): implement crawler pool manager for improved resource handling
2025-04-18 22:26:24 +08:00
__init__.py
- Test all methods
2024-05-14 21:27:41 +08:00
docker_example.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
test_cli_docs.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
test_docker.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
test_llmtxt.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
test_main.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
test_scraping_strategy.py
Release prep (#749)
2025-02-28 19:53:35 +08:00
test_web_crawler.py
feat(browser): add standalone CDP browser launch and lxml extraction strategy
2025-03-07 20:55:56 +08:00