ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add preserve_https_for_internal_links flag to maintain HTTPS during crawling. Ref #1410

Browse Source 
Added a new `preserve_https_for_internal_links` configuration flag that preserves the original HTTPS scheme for same-domain links even when the server redirects to HTTP.
This commit is contained in:
ntohidi
2025-08-28 17:38:40 +08:00
parent ef174a4c7a
commit f566c5a376
5 changed files with 224 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
crawl4ai/content_scraping_strategy.py
View File

@@ -258,7 +258,11 @@ class LXMLWebScrapingStrategy(ContentScrapingStrategy):
continue
try:
normalized_href = normalize_url(href, url)
normalized_href = normalize_url(
href, url,
preserve_https=kwargs.get('preserve_https_for_internal_links', False),
original_scheme=kwargs.get('original_scheme')
)
link_data = {
"href": normalized_href,
"text": link.text_content().strip(),