UncleCode 16b2318242 feat(api): implement crawler pool manager for improved resource handling ...

Adds a new CrawlerManager class to handle browser instance pooling and failover:
- Implements auto-scaling based on system resources
- Adds primary/backup crawler management
- Integrates memory monitoring and throttling
- Adds streaming support with memory tracking
- Updates API endpoints to use pooled crawlers

BREAKING CHANGE: API endpoints now require CrawlerManager initialization

2025-04-18 22:26:24 +08:00

16 KiB

Raw Blame History

View Raw