ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Release v0.8.0: Crash Recovery, Prefetch Mode & Security Fixes (#1712)

Browse Source 
* Fix: Use correct URL variable for raw HTML extraction (#1116)

- Prevents full HTML content from being passed as URL to extraction strategies
- Added unit tests to verify raw HTML and regular URL processing

Fix: Wrong URL variable used for extraction of raw html

* Fix #1181: Preserve whitespace in code blocks during HTML scraping

  The remove_empty_elements_fast() method was removing whitespace-only
  span elements inside <pre> and <code> tags, causing import statements
  like "import torch" to become "importtorch". Now skips elements inside
  code blocks where whitespace is significant.

* Refactor Pydantic model configuration to use ConfigDict for arbitrary types

* Fix EmbeddingStrategy: Uncomment response handling for the variations and clean up mock data. ref #1621

* Fix: permission issues with .cache/url_seeder and other runtime cache dirs. ref #1638

* fix: ensure BrowserConfig.to_dict serializes proxy_config

* feat: make LLM backoff configurable end-to-end

- extend LLMConfig with backoff delay/attempt/factor fields and thread them
  through LLMExtractionStrategy, LLMContentFilter, table extraction, and
  Docker API handlers
- expose the backoff parameter knobs on perform_completion_with_backoff/aperform_completion_with_backoff
  and document them in the md_v2 guides

* reproduced AttributeError from #1642

* pass timeout parameter to docker client request

* added missing deep crawling objects to init

* generalized query in ContentRelevanceFilter to be a str or list

* import modules from enhanceable deserialization

* parameterized tests

* Fix: capture current page URL to reflect JavaScript navigation and add test for delayed redirects. ref #1268

* refactor: replace PyPDF2 with pypdf across the codebase. ref #1412

* Add browser_context_id and target_id parameters to BrowserConfig

Enable Crawl4AI to connect to pre-created CDP browser contexts, which is
essential for cloud browser services that pre-create isolated contexts.

Changes:
- Add browser_context_id and target_id parameters to BrowserConfig
- Update from_kwargs() and to_dict() methods
- Modify BrowserManager.start() to use existing context when provided
- Add _get_page_by_target_id() helper method
- Update get_page() to handle pre-existing targets
- Add test for browser_context_id functionality

This enables cloud services to:
1. Create isolated CDP contexts before Crawl4AI connects
2. Pass context/target IDs to BrowserConfig
3. Have Crawl4AI reuse existing contexts instead of creating new ones

* Add cdp_cleanup_on_close flag to prevent memory leaks in cloud/server scenarios

* Fix: add cdp_cleanup_on_close to from_kwargs

* Fix: find context by target_id for concurrent CDP connections

* Fix: use target_id to find correct page in get_page

* Fix: use CDP to find context by browserContextId for concurrent sessions

* Revert context matching attempts - Playwright cannot see CDP-created contexts

* Add create_isolated_context flag for concurrent CDP crawls

When True, forces creation of a new browser context instead of reusing
the default context. Essential for concurrent crawls on the same browser
to prevent navigation conflicts.

* Add context caching to create_isolated_context branch

Uses contexts_by_config cache (same as non-CDP mode) to reuse contexts
for multiple URLs with same config. Still creates new page per crawl
for navigation isolation. Benefits batch/deep crawls.

* Add init_scripts support to BrowserConfig for pre-page-load JS injection

This adds the ability to inject JavaScript that runs before any page loads,
useful for stealth evasions (canvas/audio fingerprinting, userAgentData).

- Add init_scripts parameter to BrowserConfig (list of JS strings)
- Apply init_scripts in setup_context() via context.add_init_script()
- Update from_kwargs() and to_dict() for serialization

* Fix CDP connection handling: support WS URLs and proper cleanup

Changes to browser_manager.py:

1. _verify_cdp_ready(): Support multiple URL formats
   - WebSocket URLs (ws://, wss://): Skip HTTP verification, Playwright handles directly
   - HTTP URLs with query params: Properly parse with urlparse to preserve query string
   - Fixes issue where naive f"{cdp_url}/json/version" broke WS URLs and query params

2. close(): Proper cleanup when cdp_cleanup_on_close=True
   - Close all sessions (pages)
   - Close all contexts
   - Call browser.close() to disconnect (doesn't terminate browser, just releases connection)
   - Wait 1 second for CDP connection to fully release
   - Stop Playwright instance to prevent memory leaks

This enables:
- Connecting to specific browsers via WS URL
- Reusing the same browser with multiple sequential connections
- No user wait needed between connections (internal 1s delay handles it)

Added tests/browser/test_cdp_cleanup_reuse.py with comprehensive tests.

* Update gitignore

* Some debugging for caching

* Add _generate_screenshot_from_html for raw: and file:// URLs

Implements the missing method that was being called but never defined.
Now raw: and file:// URLs can generate screenshots by:
1. Loading HTML into a browser page via page.set_content()
2. Taking screenshot using existing take_screenshot() method
3. Cleaning up the page afterward

This enables cached HTML to be rendered with screenshots in crawl4ai-cloud.

* Add PDF and MHTML support for raw: and file:// URLs

- Replace _generate_screenshot_from_html with _generate_media_from_html
- New method handles screenshot, PDF, and MHTML in one browser session
- Update raw: and file:// URL handlers to use new method
- Enables cached HTML to generate all media types

* Add crash recovery for deep crawl strategies

Add optional resume_state and on_state_change parameters to all deep
crawl strategies (BFS, DFS, Best-First) for cloud deployment crash
recovery.

Features:
- resume_state: Pass saved state to resume from checkpoint
- on_state_change: Async callback fired after each URL for real-time
  state persistence to external storage (Redis, DB, etc.)
- export_state(): Get last captured state manually
- Zero overhead when features are disabled (None defaults)

State includes visited URLs, pending queue/stack, depths, and
pages_crawled count. All state is JSON-serializable.

* Fix: HTTP strategy raw: URL parsing truncates at # character

The AsyncHTTPCrawlerStrategy.crawl() method used urlparse() to extract
content from raw: URLs. This caused HTML with CSS color codes like #eee
to be truncated because # is treated as a URL fragment delimiter.

Before: raw:body{background:#eee} -> parsed.path = 'body{background:'
After:  raw:body{background:#eee} -> raw_content = 'body{background:#eee'

Fix: Strip the raw: or raw:// prefix directly instead of using urlparse,
matching how the browser strategy handles it.

* Add base_url parameter to CrawlerRunConfig for raw HTML processing

When processing raw: HTML (e.g., from cache), the URL parameter is meaningless
for markdown link resolution. This adds a base_url parameter that can be set
explicitly to provide proper URL resolution context.

Changes:
- Add base_url parameter to CrawlerRunConfig.__init__
- Add base_url to CrawlerRunConfig.from_kwargs
- Update aprocess_html to use base_url for markdown generation

Usage:
  config = CrawlerRunConfig(base_url='https://example.com')
  result = await crawler.arun(url='raw:{html}', config=config)

* Add prefetch mode for two-phase deep crawling

- Add `prefetch` parameter to CrawlerRunConfig
- Add `quick_extract_links()` function for fast link extraction
- Add short-circuit in aprocess_html() for prefetch mode
- Add 42 tests (unit, integration, regression)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Updates on proxy rotation and proxy configuration

* Add proxy support to HTTP crawler strategy

* Add browser pipeline support for raw:/file:// URLs

- Add process_in_browser parameter to CrawlerRunConfig
- Route raw:/file:// URLs through _crawl_web() when browser operations needed
- Use page.set_content() instead of goto() for local content
- Fix cookie handling for non-HTTP URLs in browser_manager
- Auto-detect browser requirements: js_code, wait_for, screenshot, etc.
- Maintain fast path for raw:/file:// without browser params

Fixes #310

* Add smart TTL cache for sitemap URL seeder

- Add cache_ttl_hours and validate_sitemap_lastmod params to SeedingConfig
- New JSON cache format with metadata (version, created_at, lastmod, url_count)
- Cache validation by TTL expiry and sitemap lastmod comparison
- Auto-migration from old .jsonl to new .json format
- Fixes bug where incomplete cache was used indefinitely

* Update URL seeder docs with smart TTL cache parameters

- Add cache_ttl_hours and validate_sitemap_lastmod to parameter table
- Document smart TTL cache validation with examples
- Add cache-related troubleshooting entries
- Update key features summary

* Add MEMORY.md to gitignore

* Docs: Add multi-sample schema generation section

Add documentation explaining how to pass multiple HTML samples
to generate_schema() for stable selectors that work across pages
with varying DOM structures.

Includes:
- Problem explanation (fragile nth-child selectors)
- Solution with code example
- Key points for multi-sample queries
- Comparison table of fragile vs stable selectors

* Fix critical RCE and LFI vulnerabilities in Docker API deployment

Security fixes for vulnerabilities reported by ProjectDiscovery:

1. Remote Code Execution via Hooks (CVE pending)
   - Remove __import__ from allowed_builtins in hook_manager.py
   - Prevents arbitrary module imports (os, subprocess, etc.)
   - Hooks now disabled by default via CRAWL4AI_HOOKS_ENABLED env var

2. Local File Inclusion via file:// URLs (CVE pending)
   - Add URL scheme validation to /execute_js, /screenshot, /pdf, /html
   - Block file://, javascript:, data: and other dangerous schemes
   - Only allow http://, https://, and raw: (where appropriate)

3. Security hardening
   - Add CRAWL4AI_HOOKS_ENABLED=false as default (opt-in for hooks)
   - Add security warning comments in config.yml
   - Add validate_url_scheme() helper for consistent validation

Testing:
   - Add unit tests (test_security_fixes.py) - 16 tests
   - Add integration tests (run_security_tests.py) for live server

Affected endpoints:
   - POST /crawl (hooks disabled by default)
   - POST /crawl/stream (hooks disabled by default)
   - POST /execute_js (URL validation added)
   - POST /screenshot (URL validation added)
   - POST /pdf (URL validation added)
   - POST /html (URL validation added)

Breaking changes:
   - Hooks require CRAWL4AI_HOOKS_ENABLED=true to function
   - file:// URLs no longer work on API endpoints (use library directly)

* Enhance authentication flow by implementing JWT token retrieval and adding authorization headers to API requests

* Add release notes for v0.7.9, detailing breaking changes, security fixes, new features, bug fixes, and documentation updates

* Add release notes for v0.8.0, detailing breaking changes, security fixes, new features, bug fixes, and documentation updates

Documentation for v0.8.0 release:

- SECURITY.md: Security policy and vulnerability reporting guidelines
- RELEASE_NOTES_v0.8.0.md: Comprehensive release notes
- migration/v0.8.0-upgrade-guide.md: Step-by-step migration guide
- security/GHSA-DRAFT-RCE-LFI.md: GitHub security advisory drafts
- CHANGELOG.md: Updated with v0.8.0 changes

Breaking changes documented:
- Docker API hooks disabled by default (CRAWL4AI_HOOKS_ENABLED)
- file:// URLs blocked on Docker API endpoints

Security fixes credited to Neo by ProjectDiscovery

* Add examples for deep crawl crash recovery and prefetch mode in documentation

* Release v0.8.0: The v0.8.0 Update

- Updated version to 0.8.0
- Added comprehensive demo and release notes
- Updated all documentation

* Update security researcher acknowledgment with a hyperlink for Neo by ProjectDiscovery

* Add async agenerate_schema method for schema generation

- Extract prompt building to shared _build_schema_prompt() method
- Add agenerate_schema() async version using aperform_completion_with_backoff
- Refactor generate_schema() to use shared prompt builder
- Fixes Gemini/Vertex AI compatibility in async contexts (FastAPI)

* Fix: Enable litellm.drop_params for O-series/GPT-5 model compatibility

O-series (o1, o3) and GPT-5 models only support temperature=1.
Setting litellm.drop_params=True auto-drops unsupported parameters
instead of throwing UnsupportedParamsError.

Fixes temperature=0.01 error for these models in LLM extraction.

---------

Co-authored-by: rbushria <rbushri@gmail.com>
Co-authored-by: AHMET YILMAZ <tawfik@kidocode.com>
Co-authored-by: Soham Kukreti <kukretisoham@gmail.com>
Co-authored-by: Chris Murphy <chris.murphy@klaviyo.com>
Co-authored-by: unclecode <unclecode@kidocode.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Nasrin
2026-01-17 14:19:15 +01:00
committed by GitHub
parent c85f56b085
commit f6f7f1b551
58 changed files with 11942 additions and 2411 deletions
Download Patch File Download Diff File Expand all files Collapse all files

301
docs/migration/v0.8.0-upgrade-guide.md Normal file
View File

@@ -0,0 +1,301 @@
# Migration Guide: Upgrading to Crawl4AI v0.8.0
This guide helps you upgrade from v0.7.x to v0.8.0, with special attention to breaking changes and security updates.
## Quick Summary
| Change | Impact | Action Required |
|--------|--------|-----------------|
| Hooks disabled by default | Docker API users with hooks | Set `CRAWL4AI_HOOKS_ENABLED=true` |
| file:// URLs blocked | Docker API users reading local files | Use Python library directly |
| Security fixes | All Docker API users | Update immediately |
---
## Step 1: Update the Package
### PyPI Installation
```bash
pip install --upgrade crawl4ai
```
### Docker Installation
```bash
docker pull unclecode/crawl4ai:latest
# or
docker pull unclecode/crawl4ai:0.8.0
```
### From Source
```bash
git pull origin main
pip install -e .
```
---
## Step 2: Check for Breaking Changes
### Are You Affected?
**You ARE affected if you:**
- Use the Docker API deployment
- Use the `hooks` parameter in `/crawl` requests
- Use `file://` URLs via API endpoints
**You are NOT affected if you:**
- Only use Crawl4AI as a Python library
- Don't use hooks in your API calls
- Don't use `file://` URLs via the API
---
## Step 3: Migrate Hooks Usage
### Before v0.8.0
Hooks worked by default:
```bash
# This worked without any configuration
curl -X POST http://localhost:11235/crawl \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"hooks": {
"code": {
"on_page_context_created": "async def hook(page, context, **kwargs):\n await context.add_cookies([...])\n return page"
}
}
}'
```
### After v0.8.0
You must explicitly enable hooks:
**Option A: Environment Variable (Recommended)**
```bash
# In your Docker run command or docker-compose.yml
export CRAWL4AI_HOOKS_ENABLED=true
```
```yaml
# docker-compose.yml
services:
crawl4ai:
image: unclecode/crawl4ai:0.8.0
environment:
- CRAWL4AI_HOOKS_ENABLED=true
```
**Option B: For Kubernetes**
```yaml
env:
- name: CRAWL4AI_HOOKS_ENABLED
value: "true"
```
### Security Warning
Only enable hooks if:
- You trust all users who can access the API
- The API is not exposed to the public internet
- You have other authentication/authorization in place
---
## Step 4: Migrate file:// URL Usage
### Before v0.8.0
```bash
# This worked via API
curl -X POST http://localhost:11235/execute_js \
-d '{"url": "file:///var/data/page.html", "scripts": ["document.title"]}'
```
### After v0.8.0
**Option A: Use the Python Library Directly**
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
async def process_local_file():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="file:///var/data/page.html",
config=CrawlerRunConfig(js_code=["document.title"])
)
return result
```
**Option B: Use raw: Protocol for HTML Content**
If you have the HTML content, you can still use the API:
```bash
# Read file content and send as raw:
HTML_CONTENT=$(cat /var/data/page.html)
curl -X POST http://localhost:11235/html \
-H "Content-Type: application/json" \
-d "{\"url\": \"raw:$HTML_CONTENT\"}"
```
**Option C: Create a Preprocessing Service**
```python
# preprocessing_service.py
from fastapi import FastAPI
from crawl4ai import AsyncWebCrawler
app = FastAPI()
@app.post("/process-local")
async def process_local(file_path: str):
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url=f"file://{file_path}")
return result.model_dump()
```
---
## Step 5: Review Security Configuration
### Recommended Production Settings
```yaml
# config.yml
security:
enabled: true
jwt_enabled: true
https_redirect: true # If behind HTTPS proxy
trusted_hosts:
- "your-domain.com"
- "api.your-domain.com"
```
### Environment Variables
```bash
# Required for JWT authentication
export SECRET_KEY="your-secure-random-key-minimum-32-characters"
# Only if you need hooks
export CRAWL4AI_HOOKS_ENABLED=true
```
### Generate a Secure Secret Key
```python
import secrets
print(secrets.token_urlsafe(32))
```
---
## Step 6: Test Your Integration
### Quick Validation Script
```python
import asyncio
import aiohttp
async def test_upgrade():
base_url = "http://localhost:11235"
# Test 1: Basic crawl should work
async with aiohttp.ClientSession() as session:
async with session.post(
f"{base_url}/crawl",
json={"urls": ["https://example.com"]}
) as resp:
assert resp.status == 200, "Basic crawl failed"
print("✓ Basic crawl works")
# Test 2: Hooks should be blocked (unless enabled)
async with aiohttp.ClientSession() as session:
async with session.post(
f"{base_url}/crawl",
json={
"urls": ["https://example.com"],
"hooks": {"code": {"on_page_context_created": "async def hook(page, context, **kwargs): return page"}}
}
) as resp:
if resp.status == 403:
print("✓ Hooks correctly blocked (default)")
elif resp.status == 200:
print("! Hooks enabled - ensure this is intentional")
# Test 3: file:// should be blocked
async with aiohttp.ClientSession() as session:
async with session.post(
f"{base_url}/execute_js",
json={"url": "file:///etc/passwd", "scripts": ["1"]}
) as resp:
assert resp.status == 400, "file:// should be blocked"
print("✓ file:// URLs correctly blocked")
asyncio.run(test_upgrade())
```
---
## Troubleshooting
### "Hooks are disabled" Error
**Symptom**: API returns 403 with "Hooks are disabled"
**Solution**: Set `CRAWL4AI_HOOKS_ENABLED=true` if you need hooks
### "URL must start with http://, https://" Error
**Symptom**: API returns 400 when using `file://` URLs
**Solution**: Use Python library directly or `raw:` protocol
### Authentication Errors After Enabling JWT
**Symptom**: API returns 401 Unauthorized
**Solution**:
1. Get a token: `POST /token` with your email
2. Include token in requests: `Authorization: Bearer <token>`
---
## Rollback Plan
If you need to rollback:
```bash
# PyPI
pip install crawl4ai==0.7.6
# Docker
docker pull unclecode/crawl4ai:0.7.6
```
**Warning**: Rolling back re-exposes the security vulnerabilities. Only do this temporarily while fixing integration issues.
---
## Getting Help
- **GitHub Issues**: [github.com/unclecode/crawl4ai/issues](https://github.com/unclecode/crawl4ai/issues)
- **Security Issues**: See [SECURITY.md](../../SECURITY.md)
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
---
## Changelog Reference
For complete list of changes, see:
- [Release Notes v0.8.0](../RELEASE_NOTES_v0.8.0.md)
- [CHANGELOG.md](../../CHANGELOG.md)