ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add advanced link head extraction with three-layer scoring system (#1)

Browse Source 
Squashed commit from feature/link-extractor branch implementing comprehensive link analysis:

- Extract HTML head content from discovered links with parallel processing
- Three-layer scoring: Intrinsic (URL quality), Contextual (BM25), and Total scores
- New LinkExtractionConfig class for type-safe configuration
- Pattern-based filtering for internal/external links
- Comprehensive documentation and examples
This commit is contained in:
UncleCode
2025-06-27 20:06:04 +08:00
parent e528086341
commit 5c9c305dbf
10 changed files with 2126 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
crawl4ai/models.py
View File

@@ -345,6 +345,12 @@ class Link(BaseModel):
text: Optional[str] = ""
title: Optional[str] = ""
base_domain: Optional[str] = ""
head_data: Optional[Dict[str, Any]] = None # Head metadata extracted from link target
head_extraction_status: Optional[str] = None # "success", "failed", "skipped"
head_extraction_error: Optional[str] = None # Error message if extraction failed
intrinsic_score: Optional[float] = None # Quality score based on URL structure, text, and context
contextual_score: Optional[float] = None # BM25 relevance score based on query and head content
total_score: Optional[float] = None # Combined score from intrinsic and contextual scores
class Media(BaseModel):