2 Commits

Author	SHA1	Message	Date
UncleCode	a58c8000aa	refactor(server): migrate to pool-based crawler management ... Replace crawler_manager.py with simpler crawler_pool.py implementation: - Add global page semaphore for hard concurrency cap - Implement browser pool with idle cleanup - Add playground UI for testing and stress testing - Update API handlers to use pooled crawlers - Enhance logging levels and symbols BREAKING CHANGE: Removes CrawlerManager class in favor of simpler pool-based approach	2025-04-20 20:14:26 +08:00
UncleCode	16b2318242	feat(api): implement crawler pool manager for improved resource handling ... Adds a new CrawlerManager class to handle browser instance pooling and failover: - Implements auto-scaling based on system resources - Adds primary/backup crawler management - Integrates memory monitoring and throttling - Adds streaming support with memory tracking - Updates API endpoints to use pooled crawlers BREAKING CHANGE: API endpoints now require CrawlerManager initialization	2025-04-18 22:26:24 +08:00