feat(scraping): add smart table extraction and analysis capabilities

Add comprehensive table detection and extraction functionality to the web scraping system:
- Implement intelligent table detection algorithm with scoring system
- Add table extraction with support for headers, rows, captions
- Update models to include tables in Media class
- Add table_score_threshold configuration option
- Add documentation and examples for table extraction
- Include crypto analysis example demonstrating table usage

This change enables users to extract structured data from HTML tables while intelligently filtering out layout tables.
This commit is contained in:
UncleCode
2025-03-09 21:31:33 +08:00
parent c6a605ccce
commit 9d69fce834
5 changed files with 586 additions and 9 deletions

View File

@@ -326,6 +326,7 @@ class Media(BaseModel):
audios: List[
MediaItem
] = [] # Using MediaItem model for now, can be extended with Audio model if needed
tables: List[Dict] = [] # Table data extracted from HTML tables
class Links(BaseModel):