chore: Add verbose option to ExtractionStrategy classes

This commit adds a new `verbose` option to the `ExtractionStrategy` classes. The `verbose` option allows for logging of extraction details, such as the number of extracted blocks and the URL being processed. This improves the debugging and monitoring capabilities of the code.
This commit is contained in:
unclecode
2024-05-17 18:06:10 +08:00
parent 32c87f0388
commit 36e46be23d
3 changed files with 14 additions and 7 deletions

View File

@@ -86,6 +86,7 @@ class WebCrawler:
**kwargs,
) -> CrawlResult:
extraction_strategy = extraction_strategy or NoExtractionStrategy()
extraction_strategy.verbose = verbose
# Check if extraction strategy is an instance of ExtractionStrategy if not raise an error
if not isinstance(extraction_strategy, ExtractionStrategy):
raise ValueError("Unsupported extraction strategy")