ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(deep-crawl): reorganize deep crawling functionality into dedicated module

Browse Source 
Restructure deep crawling code into a dedicated module with improved organization:
- Move deep crawl logic from async_deep_crawl.py to deep_crawling/
- Create separate files for BFS strategy, filters, and scorers
- Improve code organization and maintainability
- Add optimized implementations for URL filtering and scoring
- Rename DeepCrawlHandler to DeepCrawlDecorator for clarity

BREAKING CHANGE: DeepCrawlStrategy and BreadthFirstSearchStrategy imports need to be updated to new package structure
This commit is contained in:
UncleCode
2025-02-04 23:28:17 +08:00
parent bc7559586f
commit c308a794e8
11 changed files with 2921 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
crawl4ai/async_configs.py
View File

@@ -13,7 +13,7 @@ from .chunking_strategy import ChunkingStrategy, RegexChunking
from .markdown_generation_strategy import MarkdownGenerationStrategy
from .content_filter_strategy import RelevantContentFilter # , BM25ContentFilter, LLMContentFilter, PruningContentFilter
from .content_scraping_strategy import ContentScrapingStrategy, WebScrapingStrategy
from .async_deep_crawl import DeepCrawlStrategy
from .deep_crawling import DeepCrawlStrategy
from typing import Union, List
from .cache_context import CacheMode