UncleCode 0d357ab7d2 feat(scraper): Enhance URL filtering and scoring systems ...

Implement comprehensive URL filtering and scoring capabilities:

Filters:
- Add URLPatternFilter with glob/regex support
- Implement ContentTypeFilter with MIME type checking
- Add DomainFilter for domain control
- Create FilterChain with stats tracking

Scorers:
- Complete KeywordRelevanceScorer implementation
- Add PathDepthScorer for URL structure scoring
- Implement ContentTypeScorer for file type priorities
- Add FreshnessScorer for date-based scoring
- Add DomainAuthorityScorer for domain weighting
- Create CompositeScorer for combined strategies

Features:
- Add statistics tracking for both filters and scorers
- Implement logging support throughout
- Add resource cleanup methods
- Create comprehensive documentation
- Include performance optimizations

Tests and docs included.
Note: Review URL normalization overlap with recent crawler changes.

2024-11-08 19:02:28 +08:00

..

async_web_scraper.md

Refactored AsyncWebScraper to include comprehensive error handling and progress tracking capabilities. Introduced a ScrapingProgress data class to monitor processed and failed URLs. Enhanced scraping methods to log errors and track stats throughout the scraping process.

2024-11-06 21:09:47 +08:00

bfs_scraper_strategy.md

Enhanced BFS Strategy: Improved monitoring, resource management & configuration

2024-11-08 15:57:23 +08:00

filters_scrorers.md

feat(scraper): Enhance URL filtering and scoring systems

2024-11-08 18:45:12 +08:00

how_to_use.md

feat(scraper): Enhance URL filtering and scoring systems

2024-11-08 18:45:12 +08:00

scraper_quickstart.py

feat(scraper): Enhance URL filtering and scoring systems

2024-11-08 19:02:28 +08:00