UncleCode f78c46446b feat(deep-crawling): improve URL normalization and domain filtering ...

Enhance URL handling in deep crawling with:
- New URL normalization functions for consistent URL formats
- Improved domain filtering with subdomain support
- Added URLPatternFilter to public API
- Better URL deduplication in BFS strategy

These changes improve crawling accuracy and reduce duplicate visits.

2025-03-06 22:45:57 +08:00

..

__init__.py

Release prep (#749)

2025-02-28 19:53:35 +08:00

base_strategy.py

feat(profiles): add CLI command for crawling with browser profiles

2025-03-02 21:33:33 +08:00

bff_strategy.py

refactor(core): replace float('inf') with math.inf

2025-03-04 18:23:55 +08:00

bfs_strategy.py

feat(deep-crawling): improve URL normalization and domain filtering

2025-03-06 22:45:57 +08:00

crazy.py

refactor(config): enhance serialization and config handling

2025-02-19 17:23:25 +08:00

dfs_strategy.py

refactor(deep-crawl): add max_pages limit and improve crawl control

2025-03-03 21:51:11 +08:00

filters.py

feat(deep-crawling): improve URL normalization and domain filtering

2025-03-06 22:45:57 +08:00

scorers.py

2025 feb alpha 1 (#685)

2025-02-19 14:13:17 +08:00