f6f7f1b551
* Fix: Use correct URL variable for raw HTML extraction (#1116) - Prevents full HTML content from being passed as URL to extraction strategies - Added unit tests to verify raw HTML and regular URL processing Fix: Wrong URL variable used for extraction of raw html * Fix #1181: Preserve whitespace in code blocks during HTML scraping The remove_empty_elements_fast() method was removing whitespace-only span elements inside <pre> and <code> tags, causing import statements like "import torch" to become "importtorch". Now skips elements inside code blocks where whitespace is significant. * Refactor Pydantic model configuration to use ConfigDict for arbitrary types * Fix EmbeddingStrategy: Uncomment response handling for the variations and clean up mock data. ref #1621 * Fix: permission issues with .cache/url_seeder and other runtime cache dirs. ref #1638 * fix: ensure BrowserConfig.to_dict serializes proxy_config * feat: make LLM backoff configurable end-to-end - extend LLMConfig with backoff delay/attempt/factor fields and thread them through LLMExtractionStrategy, LLMContentFilter, table extraction, and Docker API handlers - expose the backoff parameter knobs on perform_completion_with_backoff/aperform_completion_with_backoff and document them in the md_v2 guides * reproduced AttributeError from #1642 * pass timeout parameter to docker client request * added missing deep crawling objects to init * generalized query in ContentRelevanceFilter to be a str or list * import modules from enhanceable deserialization * parameterized tests * Fix: capture current page URL to reflect JavaScript navigation and add test for delayed redirects. ref #1268 * refactor: replace PyPDF2 with pypdf across the codebase. ref #1412 * Add browser_context_id and target_id parameters to BrowserConfig Enable Crawl4AI to connect to pre-created CDP browser contexts, which is essential for cloud browser services that pre-create isolated contexts. Changes: - Add browser_context_id and target_id parameters to BrowserConfig - Update from_kwargs() and to_dict() methods - Modify BrowserManager.start() to use existing context when provided - Add _get_page_by_target_id() helper method - Update get_page() to handle pre-existing targets - Add test for browser_context_id functionality This enables cloud services to: 1. Create isolated CDP contexts before Crawl4AI connects 2. Pass context/target IDs to BrowserConfig 3. Have Crawl4AI reuse existing contexts instead of creating new ones * Add cdp_cleanup_on_close flag to prevent memory leaks in cloud/server scenarios * Fix: add cdp_cleanup_on_close to from_kwargs * Fix: find context by target_id for concurrent CDP connections * Fix: use target_id to find correct page in get_page * Fix: use CDP to find context by browserContextId for concurrent sessions * Revert context matching attempts - Playwright cannot see CDP-created contexts * Add create_isolated_context flag for concurrent CDP crawls When True, forces creation of a new browser context instead of reusing the default context. Essential for concurrent crawls on the same browser to prevent navigation conflicts. * Add context caching to create_isolated_context branch Uses contexts_by_config cache (same as non-CDP mode) to reuse contexts for multiple URLs with same config. Still creates new page per crawl for navigation isolation. Benefits batch/deep crawls. * Add init_scripts support to BrowserConfig for pre-page-load JS injection This adds the ability to inject JavaScript that runs before any page loads, useful for stealth evasions (canvas/audio fingerprinting, userAgentData). - Add init_scripts parameter to BrowserConfig (list of JS strings) - Apply init_scripts in setup_context() via context.add_init_script() - Update from_kwargs() and to_dict() for serialization * Fix CDP connection handling: support WS URLs and proper cleanup Changes to browser_manager.py: 1. _verify_cdp_ready(): Support multiple URL formats - WebSocket URLs (ws://, wss://): Skip HTTP verification, Playwright handles directly - HTTP URLs with query params: Properly parse with urlparse to preserve query string - Fixes issue where naive f"{cdp_url}/json/version" broke WS URLs and query params 2. close(): Proper cleanup when cdp_cleanup_on_close=True - Close all sessions (pages) - Close all contexts - Call browser.close() to disconnect (doesn't terminate browser, just releases connection) - Wait 1 second for CDP connection to fully release - Stop Playwright instance to prevent memory leaks This enables: - Connecting to specific browsers via WS URL - Reusing the same browser with multiple sequential connections - No user wait needed between connections (internal 1s delay handles it) Added tests/browser/test_cdp_cleanup_reuse.py with comprehensive tests. * Update gitignore * Some debugging for caching * Add _generate_screenshot_from_html for raw: and file:// URLs Implements the missing method that was being called but never defined. Now raw: and file:// URLs can generate screenshots by: 1. Loading HTML into a browser page via page.set_content() 2. Taking screenshot using existing take_screenshot() method 3. Cleaning up the page afterward This enables cached HTML to be rendered with screenshots in crawl4ai-cloud. * Add PDF and MHTML support for raw: and file:// URLs - Replace _generate_screenshot_from_html with _generate_media_from_html - New method handles screenshot, PDF, and MHTML in one browser session - Update raw: and file:// URL handlers to use new method - Enables cached HTML to generate all media types * Add crash recovery for deep crawl strategies Add optional resume_state and on_state_change parameters to all deep crawl strategies (BFS, DFS, Best-First) for cloud deployment crash recovery. Features: - resume_state: Pass saved state to resume from checkpoint - on_state_change: Async callback fired after each URL for real-time state persistence to external storage (Redis, DB, etc.) - export_state(): Get last captured state manually - Zero overhead when features are disabled (None defaults) State includes visited URLs, pending queue/stack, depths, and pages_crawled count. All state is JSON-serializable. * Fix: HTTP strategy raw: URL parsing truncates at # character The AsyncHTTPCrawlerStrategy.crawl() method used urlparse() to extract content from raw: URLs. This caused HTML with CSS color codes like #eee to be truncated because # is treated as a URL fragment delimiter. Before: raw:body{background:#eee} -> parsed.path = 'body{background:' After: raw:body{background:#eee} -> raw_content = 'body{background:#eee' Fix: Strip the raw: or raw:// prefix directly instead of using urlparse, matching how the browser strategy handles it. * Add base_url parameter to CrawlerRunConfig for raw HTML processing When processing raw: HTML (e.g., from cache), the URL parameter is meaningless for markdown link resolution. This adds a base_url parameter that can be set explicitly to provide proper URL resolution context. Changes: - Add base_url parameter to CrawlerRunConfig.__init__ - Add base_url to CrawlerRunConfig.from_kwargs - Update aprocess_html to use base_url for markdown generation Usage: config = CrawlerRunConfig(base_url='https://example.com') result = await crawler.arun(url='raw:{html}', config=config) * Add prefetch mode for two-phase deep crawling - Add `prefetch` parameter to CrawlerRunConfig - Add `quick_extract_links()` function for fast link extraction - Add short-circuit in aprocess_html() for prefetch mode - Add 42 tests (unit, integration, regression) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Updates on proxy rotation and proxy configuration * Add proxy support to HTTP crawler strategy * Add browser pipeline support for raw:/file:// URLs - Add process_in_browser parameter to CrawlerRunConfig - Route raw:/file:// URLs through _crawl_web() when browser operations needed - Use page.set_content() instead of goto() for local content - Fix cookie handling for non-HTTP URLs in browser_manager - Auto-detect browser requirements: js_code, wait_for, screenshot, etc. - Maintain fast path for raw:/file:// without browser params Fixes #310 * Add smart TTL cache for sitemap URL seeder - Add cache_ttl_hours and validate_sitemap_lastmod params to SeedingConfig - New JSON cache format with metadata (version, created_at, lastmod, url_count) - Cache validation by TTL expiry and sitemap lastmod comparison - Auto-migration from old .jsonl to new .json format - Fixes bug where incomplete cache was used indefinitely * Update URL seeder docs with smart TTL cache parameters - Add cache_ttl_hours and validate_sitemap_lastmod to parameter table - Document smart TTL cache validation with examples - Add cache-related troubleshooting entries - Update key features summary * Add MEMORY.md to gitignore * Docs: Add multi-sample schema generation section Add documentation explaining how to pass multiple HTML samples to generate_schema() for stable selectors that work across pages with varying DOM structures. Includes: - Problem explanation (fragile nth-child selectors) - Solution with code example - Key points for multi-sample queries - Comparison table of fragile vs stable selectors * Fix critical RCE and LFI vulnerabilities in Docker API deployment Security fixes for vulnerabilities reported by ProjectDiscovery: 1. Remote Code Execution via Hooks (CVE pending) - Remove __import__ from allowed_builtins in hook_manager.py - Prevents arbitrary module imports (os, subprocess, etc.) - Hooks now disabled by default via CRAWL4AI_HOOKS_ENABLED env var 2. Local File Inclusion via file:// URLs (CVE pending) - Add URL scheme validation to /execute_js, /screenshot, /pdf, /html - Block file://, javascript:, data: and other dangerous schemes - Only allow http://, https://, and raw: (where appropriate) 3. Security hardening - Add CRAWL4AI_HOOKS_ENABLED=false as default (opt-in for hooks) - Add security warning comments in config.yml - Add validate_url_scheme() helper for consistent validation Testing: - Add unit tests (test_security_fixes.py) - 16 tests - Add integration tests (run_security_tests.py) for live server Affected endpoints: - POST /crawl (hooks disabled by default) - POST /crawl/stream (hooks disabled by default) - POST /execute_js (URL validation added) - POST /screenshot (URL validation added) - POST /pdf (URL validation added) - POST /html (URL validation added) Breaking changes: - Hooks require CRAWL4AI_HOOKS_ENABLED=true to function - file:// URLs no longer work on API endpoints (use library directly) * Enhance authentication flow by implementing JWT token retrieval and adding authorization headers to API requests * Add release notes for v0.7.9, detailing breaking changes, security fixes, new features, bug fixes, and documentation updates * Add release notes for v0.8.0, detailing breaking changes, security fixes, new features, bug fixes, and documentation updates Documentation for v0.8.0 release: - SECURITY.md: Security policy and vulnerability reporting guidelines - RELEASE_NOTES_v0.8.0.md: Comprehensive release notes - migration/v0.8.0-upgrade-guide.md: Step-by-step migration guide - security/GHSA-DRAFT-RCE-LFI.md: GitHub security advisory drafts - CHANGELOG.md: Updated with v0.8.0 changes Breaking changes documented: - Docker API hooks disabled by default (CRAWL4AI_HOOKS_ENABLED) - file:// URLs blocked on Docker API endpoints Security fixes credited to Neo by ProjectDiscovery * Add examples for deep crawl crash recovery and prefetch mode in documentation * Release v0.8.0: The v0.8.0 Update - Updated version to 0.8.0 - Added comprehensive demo and release notes - Updated all documentation * Update security researcher acknowledgment with a hyperlink for Neo by ProjectDiscovery * Add async agenerate_schema method for schema generation - Extract prompt building to shared _build_schema_prompt() method - Add agenerate_schema() async version using aperform_completion_with_backoff - Refactor generate_schema() to use shared prompt builder - Fixes Gemini/Vertex AI compatibility in async contexts (FastAPI) * Fix: Enable litellm.drop_params for O-series/GPT-5 model compatibility O-series (o1, o3) and GPT-5 models only support temperature=1. Setting litellm.drop_params=True auto-drops unsupported parameters instead of throwing UnsupportedParamsError. Fixes temperature=0.01 error for these models in LLM extraction. --------- Co-authored-by: rbushria <rbushri@gmail.com> Co-authored-by: AHMET YILMAZ <tawfik@kidocode.com> Co-authored-by: Soham Kukreti <kukretisoham@gmail.com> Co-authored-by: Chris Murphy <chris.murphy@klaviyo.com> Co-authored-by: unclecode <unclecode@kidocode.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
564 lines
20 KiB
Python
564 lines
20 KiB
Python
"""
|
|
BRUTAL edge case tests for raw:/file:// URL browser pipeline.
|
|
|
|
These tests try to break the system with tricky inputs, edge cases,
|
|
and compatibility checks to ensure we didn't break existing functionality.
|
|
"""
|
|
|
|
import pytest
|
|
import asyncio
|
|
import tempfile
|
|
import os
|
|
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: Hash characters in HTML (previously broke urlparse - Issue #283)
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_hash_in_css():
|
|
"""Test that # in CSS colors doesn't break HTML parsing (regression for #283)."""
|
|
html = """
|
|
<html>
|
|
<head>
|
|
<style>
|
|
body { background-color: #ff5733; color: #333333; }
|
|
.highlight { border: 1px solid #000; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="highlight" style="color: #ffffff;">Content with hash colors</div>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(js_code="document.body.innerHTML += '<div id=\"added\">Added</div>'")
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "#ff5733" in result.html or "ff5733" in result.html # Color should be preserved
|
|
assert "Added" in result.html # JS executed
|
|
assert "Content with hash colors" in result.html # Original content preserved
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_fragment_links():
|
|
"""Test HTML with # fragment links doesn't break."""
|
|
html = """
|
|
<html><body>
|
|
<a href="#section1">Go to section 1</a>
|
|
<a href="#section2">Go to section 2</a>
|
|
<div id="section1">Section 1</div>
|
|
<div id="section2">Section 2</div>
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(js_code="document.getElementById('section1').innerText = 'Modified Section 1'")
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "Modified Section 1" in result.html
|
|
assert "#section2" in result.html # Fragment link preserved
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: Special characters and unicode
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_unicode():
|
|
"""Test raw HTML with various unicode characters."""
|
|
html = """
|
|
<html><body>
|
|
<div id="unicode">日本語 中文 한국어 العربية 🎉 💻 🚀</div>
|
|
<div id="special">& < > " '</div>
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(js_code="document.getElementById('unicode').innerText += ' ✅ Modified'")
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "✅ Modified" in result.html or "Modified" in result.html
|
|
# Check unicode is preserved
|
|
assert "日本語" in result.html or "&#" in result.html # Either preserved or encoded
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_script_tags():
|
|
"""Test raw HTML with existing script tags doesn't interfere with js_code."""
|
|
html = """
|
|
<html><body>
|
|
<div id="counter">0</div>
|
|
<script>
|
|
// This script runs on page load
|
|
document.getElementById('counter').innerText = '10';
|
|
</script>
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
# Our js_code runs AFTER the page scripts
|
|
config = CrawlerRunConfig(
|
|
js_code="document.getElementById('counter').innerText = parseInt(document.getElementById('counter').innerText) + 5"
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
# The embedded script sets it to 10, then our js_code adds 5
|
|
assert ">15<" in result.html or "15" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: Empty and malformed HTML
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_empty():
|
|
"""Test empty raw HTML."""
|
|
html = ""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(js_code="document.body.innerHTML = '<div>Added to empty</div>'")
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "Added to empty" in result.html
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_minimal():
|
|
"""Test minimal HTML (just text, no tags)."""
|
|
html = "Just plain text, no HTML tags"
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(js_code="document.body.innerHTML += '<div id=\"injected\">Injected</div>'")
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
# Browser should wrap it in proper HTML
|
|
assert "Injected" in result.html
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_malformed():
|
|
"""Test malformed HTML with unclosed tags."""
|
|
html = "<html><body><div><span>Unclosed tags<div>More content"
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(js_code="document.body.innerHTML += '<div id=\"valid\">Valid Added</div>'")
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "Valid Added" in result.html
|
|
# Browser should have fixed the malformed HTML
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: Very large HTML
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_large():
|
|
"""Test large raw HTML (100KB+)."""
|
|
# Generate 100KB of HTML
|
|
items = "".join([f'<div class="item" id="item-{i}">Item {i} content here with some text</div>\n' for i in range(2000)])
|
|
html = f"<html><body>{items}</body></html>"
|
|
|
|
assert len(html) > 100000 # Verify it's actually large
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="document.getElementById('item-999').innerText = 'MODIFIED ITEM 999'"
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "MODIFIED ITEM 999" in result.html
|
|
assert "item-1999" in result.html # Last item should still exist
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: JavaScript errors and timeouts
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_js_error_doesnt_crash():
|
|
"""Test that JavaScript errors in js_code don't crash the crawl."""
|
|
html = "<html><body><div id='test'>Original</div></body></html>"
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code=[
|
|
"nonExistentFunction();", # This will throw an error
|
|
"document.getElementById('test').innerText = 'Still works'" # This should still run
|
|
]
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
# Crawl should succeed even with JS errors
|
|
assert result.success
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_wait_for_timeout():
|
|
"""Test wait_for with element that never appears times out gracefully."""
|
|
html = "<html><body><div id='test'>Original</div></body></html>"
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
wait_for="#never-exists",
|
|
wait_for_timeout=1000 # 1 second timeout
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
# Should timeout but still return the HTML we have
|
|
# The behavior might be success=False or success=True with partial content
|
|
# Either way, it shouldn't hang or crash
|
|
assert result is not None
|
|
|
|
|
|
# ============================================================================
|
|
# COMPATIBILITY: Normal HTTP URLs still work
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_http_urls_still_work():
|
|
"""Ensure we didn't break normal HTTP URL crawling."""
|
|
async with AsyncWebCrawler() as crawler:
|
|
result = await crawler.arun("https://example.com")
|
|
|
|
assert result.success
|
|
assert "Example Domain" in result.html
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_http_with_js_code_still_works():
|
|
"""Ensure HTTP URLs with js_code still work."""
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="document.body.innerHTML += '<div id=\"injected\">Injected via JS</div>'"
|
|
)
|
|
result = await crawler.arun("https://example.com", config=config)
|
|
|
|
assert result.success
|
|
assert "Injected via JS" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# COMPATIBILITY: File URLs
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_file_url_with_js_code():
|
|
"""Test file:// URLs with js_code execution."""
|
|
# Create a temp file
|
|
with tempfile.NamedTemporaryFile(mode='w', suffix='.html', delete=False) as f:
|
|
f.write("<html><body><div id='file-content'>File Content</div></body></html>")
|
|
temp_path = f.name
|
|
|
|
try:
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="document.getElementById('file-content').innerText = 'Modified File Content'"
|
|
)
|
|
result = await crawler.arun(f"file://{temp_path}", config=config)
|
|
|
|
assert result.success
|
|
assert "Modified File Content" in result.html
|
|
finally:
|
|
os.unlink(temp_path)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_file_url_fast_path():
|
|
"""Test file:// fast path (no browser params)."""
|
|
with tempfile.NamedTemporaryFile(mode='w', suffix='.html', delete=False) as f:
|
|
f.write("<html><body>Fast path file content</body></html>")
|
|
temp_path = f.name
|
|
|
|
try:
|
|
async with AsyncWebCrawler() as crawler:
|
|
result = await crawler.arun(f"file://{temp_path}")
|
|
|
|
assert result.success
|
|
assert "Fast path file content" in result.html
|
|
finally:
|
|
os.unlink(temp_path)
|
|
|
|
|
|
# ============================================================================
|
|
# COMPATIBILITY: Extraction strategies with raw HTML
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_css_extraction():
|
|
"""Test CSS extraction on raw HTML after js_code modifies it."""
|
|
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
|
|
|
html = """
|
|
<html><body>
|
|
<div class="products">
|
|
<div class="product"><span class="name">Original Product</span></div>
|
|
</div>
|
|
</body></html>
|
|
"""
|
|
|
|
schema = {
|
|
"name": "Products",
|
|
"baseSelector": ".product",
|
|
"fields": [
|
|
{"name": "name", "selector": ".name", "type": "text"}
|
|
]
|
|
}
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="""
|
|
document.querySelector('.products').innerHTML +=
|
|
'<div class="product"><span class="name">JS Added Product</span></div>';
|
|
""",
|
|
extraction_strategy=JsonCssExtractionStrategy(schema)
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
# Check that extraction found both products
|
|
import json
|
|
extracted = json.loads(result.extracted_content)
|
|
names = [p.get('name', '') for p in extracted]
|
|
assert any("JS Added Product" in name for name in names)
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: Concurrent raw: requests
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_concurrent_raw_requests():
|
|
"""Test multiple concurrent raw: requests don't interfere."""
|
|
htmls = [
|
|
f"<html><body><div id='test'>Request {i}</div></body></html>"
|
|
for i in range(5)
|
|
]
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
configs = [
|
|
CrawlerRunConfig(
|
|
js_code=f"document.getElementById('test').innerText += ' Modified {i}'"
|
|
)
|
|
for i in range(5)
|
|
]
|
|
|
|
# Run concurrently
|
|
tasks = [
|
|
crawler.arun(f"raw:{html}", config=config)
|
|
for html, config in zip(htmls, configs)
|
|
]
|
|
results = await asyncio.gather(*tasks)
|
|
|
|
for i, result in enumerate(results):
|
|
assert result.success
|
|
assert f"Request {i}" in result.html
|
|
assert f"Modified {i}" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: raw: with base_url for link resolution
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_base_url():
|
|
"""Test that base_url is used for link resolution in markdown."""
|
|
html = """
|
|
<html><body>
|
|
<a href="/page1">Page 1</a>
|
|
<a href="/page2">Page 2</a>
|
|
<img src="/images/logo.png" alt="Logo">
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
base_url="https://example.com",
|
|
process_in_browser=True # Force browser to test base_url handling
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
# Check markdown has absolute URLs
|
|
if result.markdown:
|
|
# Links should be absolute
|
|
md = result.markdown.raw_markdown if hasattr(result.markdown, 'raw_markdown') else str(result.markdown)
|
|
assert "example.com" in md or "/page1" in md
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: raw: with screenshot of complex page
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_screenshot_complex_page():
|
|
"""Test screenshot of complex raw HTML with CSS and JS modifications."""
|
|
html = """
|
|
<html>
|
|
<head>
|
|
<style>
|
|
body { font-family: Arial; background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); padding: 40px; }
|
|
.card { background: white; padding: 20px; border-radius: 10px; box-shadow: 0 4px 6px rgba(0,0,0,0.1); }
|
|
h1 { color: #333; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="card">
|
|
<h1 id="title">Original Title</h1>
|
|
<p>This is a test card with styling.</p>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="document.getElementById('title').innerText = 'Modified Title'",
|
|
screenshot=True
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert result.screenshot is not None
|
|
assert len(result.screenshot) > 1000 # Should be substantial
|
|
assert "Modified Title" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: JavaScript that tries to navigate away
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_js_navigation_blocked():
|
|
"""Test that JS trying to navigate doesn't break the crawl."""
|
|
html = """
|
|
<html><body>
|
|
<div id="content">Original Content</div>
|
|
<script>
|
|
// Try to navigate away (should be blocked or handled)
|
|
// window.location.href = 'https://example.com';
|
|
</script>
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
# Try to navigate via js_code
|
|
js_code=[
|
|
"document.getElementById('content').innerText = 'Before navigation attempt'",
|
|
# Actual navigation attempt commented - would cause issues
|
|
# "window.location.href = 'https://example.com'",
|
|
]
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "Before navigation attempt" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# EDGE CASE: Raw HTML with iframes
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_iframes():
|
|
"""Test raw HTML containing iframes."""
|
|
html = """
|
|
<html><body>
|
|
<div id="main">Main content</div>
|
|
<iframe id="frame1" srcdoc="<html><body><div id='iframe-content'>Iframe Content</div></body></html>"></iframe>
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="document.getElementById('main').innerText = 'Modified main'",
|
|
process_iframes=True
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "Modified main" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# TRICKY: Protocol inside raw content
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_raw_html_with_urls_inside():
|
|
"""Test raw: with http:// URLs inside the content."""
|
|
html = """
|
|
<html><body>
|
|
<a href="http://example.com">Example</a>
|
|
<a href="https://google.com">Google</a>
|
|
<img src="https://placekitten.com/200/300" alt="Cat">
|
|
<div id="test">Test content with URL: https://test.com</div>
|
|
</body></html>
|
|
"""
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
config = CrawlerRunConfig(
|
|
js_code="document.getElementById('test').innerText += ' - Modified'"
|
|
)
|
|
result = await crawler.arun(f"raw:{html}", config=config)
|
|
|
|
assert result.success
|
|
assert "Modified" in result.html
|
|
assert "http://example.com" in result.html or "example.com" in result.html
|
|
|
|
|
|
# ============================================================================
|
|
# TRICKY: Double raw: prefix
|
|
# ============================================================================
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_double_raw_prefix():
|
|
"""Test what happens with double raw: prefix (edge case)."""
|
|
html = "<html><body>Content</body></html>"
|
|
|
|
async with AsyncWebCrawler() as crawler:
|
|
# raw:raw:<html>... - the second raw: becomes part of content
|
|
result = await crawler.arun(f"raw:raw:{html}")
|
|
|
|
# Should either handle gracefully or return "raw:<html>..." as content
|
|
assert result is not None
|
|
|
|
|
|
if __name__ == "__main__":
|
|
import sys
|
|
|
|
async def run_tests():
|
|
# Run a few key tests manually
|
|
tests = [
|
|
("Hash in CSS", test_raw_html_with_hash_in_css),
|
|
("Unicode", test_raw_html_with_unicode),
|
|
("Large HTML", test_raw_html_large),
|
|
("HTTP still works", test_http_urls_still_work),
|
|
("Concurrent requests", test_concurrent_raw_requests),
|
|
("Complex screenshot", test_raw_html_screenshot_complex_page),
|
|
]
|
|
|
|
for name, test_fn in tests:
|
|
print(f"\n=== Running: {name} ===")
|
|
try:
|
|
await test_fn()
|
|
print(f"✅ {name} PASSED")
|
|
except Exception as e:
|
|
print(f"❌ {name} FAILED: {e}")
|
|
import traceback
|
|
traceback.print_exc()
|
|
|
|
asyncio.run(run_tests())
|