Enhance Markdown generation and external content control

Browse Source

- Integrate customized html2text library for flexible Markdown output
- Add options to exclude external links and images
- Improve content scraping efficiency and error handling
- Update AsyncPlaywrightCrawlerStrategy for faster closing
- Enhance CosineStrategy with generic embedding model loading

...

This commit is contained in:

UncleCode

2024-10-20 18:56:58 +08:00

parent e7cd8a1c2d

commit 6ec4cb33ca

14 changed files with 1981 additions and 21 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1015

crawl4ai/html2text/__init__.py Normal file

File diff suppressed because it is too large Load Diff