refactor(deep-crawling): reorganize deep crawling strategies and add new implementations

Split deep crawling code into separate strategy files for better organization and maintainability. Added new BFF (Best First) and DFS crawling strategies. Introduced base strategy class and common types.

BREAKING CHANGE: Deep crawling implementation has been split into multiple files. Import paths for deep crawling strategies have changed.
This commit is contained in:
UncleCode
2025-02-05 22:50:39 +08:00
parent c308a794e8
commit a9415aaaf6
10 changed files with 769 additions and 214 deletions

14
crawl4ai/types.py Normal file
View File

@@ -0,0 +1,14 @@
from typing import TYPE_CHECKING, Union
AsyncWebCrawler = Union['AsyncWebCrawlerType'] # Note the string literal
CrawlerRunConfig = Union['CrawlerRunConfigType']
CrawlResult = Union['CrawlResultType']
RunManyReturn = Union['RunManyReturnType']
if TYPE_CHECKING:
from . import (
AsyncWebCrawler as AsyncWebCrawlerType,
CrawlerRunConfig as CrawlerRunConfigType,
CrawlResult as CrawlResultType,
RunManyReturn as RunManyReturnType,
)