ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix Base64 image parsing in WebScrappingStrategy (issue 182)

Browse Source 
- Add support for extracting Base64 encoded images
- Improve image format detection to include Base64 images
- Enhance compatibility with locally saved HTML files using Base64 image encoding
This commit is contained in:
UncleCode
2024-10-20 19:25:25 +08:00
parent 1dd36f9035
commit 04d16e6d2b
2 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@@ -2,6 +2,9 @@
## [v0.3.72] - 2024-10-20 ## [v0.3.72] - 2024-10-20
### Fixed
- Added support for parsing Base64 encoded images in WebScrappingStrategy
### Added ### Added
- Forked and integrated a customized version of the html2text library for more control over Markdown generation - Forked and integrated a customized version of the html2text library for more control over Markdown generation
- New configuration options for controlling external content: - New configuration options for controlling external content:

6
crawl4ai/content_scrapping_strategy.py
View File

@@ -127,7 +127,11 @@ class WebScrappingStrategy(ContentScrappingStrategy):
image_width = img.get('width') image_width = img.get('width')
width_value, width_unit = parse_dimension(image_width) width_value, width_unit = parse_dimension(image_width)
image_size = 0 #int(fetch_image_file_size(img,base_url) or 0) image_size = 0 #int(fetch_image_file_size(img,base_url) or 0)
image_format = os.path.splitext(img.get('src',''))[1].lower() image_src = img.get('src','')
if "data:image/" in image_src:
image_format = image_src.split(',')[0].split(';')[0].split('/')[1]
else:
image_format = os.path.splitext(img.get('src',''))[1].lower()
# Remove . from format # Remove . from format
image_format = image_format.strip('.').split('?')[0] image_format = image_format.strip('.').split('?')[0]
score = 0 score = 0