ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor proxy configuration documentation for clarity and consistency

Browse Source
This commit is contained in:
AHMET YILMAZ
2025-11-13 11:20:24 +08:00
parent 263ac890fd
commit fe353c4e27
1 changed files with 101 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files

231
docs/md_v2/advanced/proxy-security.md
View File

@@ -4,73 +4,34 @@ This guide covers proxy configuration and security features in Crawl4AI, includi
## Understanding Proxy Configuration ## Understanding Proxy Configuration
Crawl4AI supports proxy configuration at two levels: Crawl4AI recommends configuring proxies per request through `CrawlerRunConfig.proxy_config`. This gives you precise control, enables rotation strategies, and keeps examples simple enough to copy, paste, and run.
### BrowserConfig.proxy_config
Sets proxy at the **browser level** - affects all pages/tabs in that browser instance. Use this when:
- You want all crawls from this browser to use the same proxy
- You're using a single proxy for the entire session
- You need persistent proxy settings across multiple crawls
### CrawlerRunConfig.proxy_config
Sets proxy at the **request level** - can be different for each crawl operation. Use this when:
- You want per-request proxy control
- You're implementing proxy rotation
- Different URLs need different proxies
## Basic Proxy Setup ## Basic Proxy Setup
### Browser-Level Proxy (BrowserConfig) Configure proxies that apply to each crawl operation:
Configure proxies that apply to the entire browser session:
```python ```python
from crawl4ai import AsyncWebCrawler, BrowserConfig import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig
# Using dictionary configuration run_config = CrawlerRunConfig(proxy_config=ProxyConfig(server="http://proxy.example.com:8080"))
browser_config = BrowserConfig(proxy_config={ # run_config = CrawlerRunConfig(proxy_config={"server": "http://proxy.example.com:8080"})
"server": "http://proxy.example.com:8080" # run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080")
})
# Using ProxyConfig object
from crawl4ai import ProxyConfig
proxy = ProxyConfig(server="http://proxy.example.com:8080")
browser_config = BrowserConfig(proxy_config=proxy)
# Using string (auto-parsed) async def main():
browser_config = BrowserConfig(proxy_config="http://proxy.example.com:8080") browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com", config=run_config)
print(f"Success: {result.success} -> {result.url}")
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com") if __name__ == "__main__":
asyncio.run(main())
``` ```
### Request-Level Proxy (CrawlerRunConfig) !!! note "Why request-level?"
`CrawlerRunConfig.proxy_config` keeps each request self-contained, so swapping proxies or rotation strategies is just a matter of building a new run configuration.
Configure proxies that can be customized per crawl operation:
```python
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
# Using dictionary configuration
run_config = CrawlerRunConfig(proxy_config={
"server": "http://proxy.example.com:8080"
})
# Using ProxyConfig object
from crawl4ai import ProxyConfig
proxy = ProxyConfig(server="http://proxy.example.com:8080")
run_config = CrawlerRunConfig(proxy_config=proxy)
# Using string (auto-parsed)
run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080")
browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com", config=run_config)
```
!!! note "Priority Order"
When both `BrowserConfig.proxy_config` and `CrawlerRunConfig.proxy_config` are set, `CrawlerRunConfig.proxy_config` takes precedence for that specific crawl operation.
## Supported Proxy Formats ## Supported Proxy Formats
@@ -100,27 +61,33 @@ proxy5 = ProxyConfig.from_string("192.168.1.1:8080:user:pass")
For proxies requiring authentication: For proxies requiring authentication:
```python ```python
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig import asyncio
from crawl4ai import AsyncWebCrawler,BrowserConfig, CrawlerRunConfig, ProxyConfig
# Using dictionary run_config = CrawlerRunConfig(
run_config = CrawlerRunConfig(proxy_config={ proxy_config=ProxyConfig(
"server": "http://proxy.example.com:8080", server="http://proxy.example.com:8080",
"username": "your_username", username="your_username",
"password": "your_password" password="your_password",
}) )
# Using ProxyConfig object
from crawl4ai import ProxyConfig
proxy = ProxyConfig(
server="http://proxy.example.com:8080",
username="your_username",
password="your_password"
) )
run_config = CrawlerRunConfig(proxy_config=proxy) # Or dictionary style:
# run_config = CrawlerRunConfig(proxy_config={
# "server": "http://proxy.example.com:8080",
# "username": "your_username",
# "password": "your_password",
# })
browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler: async def main():
result = await crawler.arun(url="https://example.com", config=run_config) browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com", config=run_config)
print(f"Success: {result.success} -> {result.url}")
if __name__ == "__main__":
asyncio.run(main())
``` ```
## Environment Variable Configuration ## Environment Variable Configuration
@@ -149,9 +116,10 @@ Crawl4AI supports automatic proxy rotation to distribute requests across multipl
### Proxy Rotation (recommended) ### Proxy Rotation (recommended)
```python ```python
import asyncio
import re
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy from crawl4ai.proxy_strategy import RoundRobinProxyStrategy
import re
async def main(): async def main():
# Load proxies from environment # Load proxies from environment
@@ -195,7 +163,8 @@ async def main():
else: else:
print(f"❌ Request {i+1}: Failed - {result.error_message}") print(f"❌ Request {i+1}: Failed - {result.error_message}")
asyncio.run(main()) if __name__ == "__main__":
asyncio.run(main())
``` ```
## SSL Certificate Analysis ## SSL Certificate Analysis
@@ -204,45 +173,54 @@ Combine proxy usage with SSL certificate inspection for enhanced security analys
### Per-Request SSL Certificate Analysis ### Per-Request SSL Certificate Analysis
```python ```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
# Configure proxy with SSL certificate fetching per request
run_config = CrawlerRunConfig( run_config = CrawlerRunConfig(
proxy_config={ proxy_config={
"server": "http://proxy.example.com:8080", "server": "http://proxy.example.com:8080",
"username": "user", "username": "user",
"password": "pass" "password": "pass",
}, },
fetch_ssl_certificate=True # Enable SSL certificate analysis for this request fetch_ssl_certificate=True, # Enable SSL certificate analysis for this request
) )
browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com", config=run_config)
if result.success: async def main():
print(f"✅ Crawled via proxy: {result.url}") browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com", config=run_config)
# Analyze SSL certificate if result.success:
if result.ssl_certificate: print(f"✅ Crawled via proxy: {result.url}")
cert = result.ssl_certificate
print("🔒 SSL Certificate Info:")
print(f" Issuer: {cert.issuer}")
print(f" Subject: {cert.subject}")
print(f" Valid until: {cert.valid_until}")
print(f" Fingerprint: {cert.fingerprint}")
# Export certificate # Analyze SSL certificate
cert.to_json("certificate.json") if result.ssl_certificate:
print("💾 Certificate exported to certificate.json") cert = result.ssl_certificate
else: print("🔒 SSL Certificate Info:")
print("⚠️ No SSL certificate information available") print(f" Issuer: {cert.issuer}")
print(f" Subject: {cert.subject}")
print(f" Valid until: {cert.valid_until}")
print(f" Fingerprint: {cert.fingerprint}")
# Export certificate
cert.to_json("certificate.json")
print("💾 Certificate exported to certificate.json")
else:
print("⚠️ No SSL certificate information available")
if __name__ == "__main__":
asyncio.run(main())
``` ```
## Security Best Practices ## Security Best Practices
### 1. Proxy Rotation for Anonymity ### 1. Proxy Rotation for Anonymity
```python ```python
from crawl4ai import CrawlerRunConfig, ProxyConfig
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy
# Use multiple proxies to avoid IP blocking # Use multiple proxies to avoid IP blocking
proxies = ProxyConfig.from_env("PROXIES") proxies = ProxyConfig.from_env("PROXIES")
strategy = RoundRobinProxyStrategy(proxies) strategy = RoundRobinProxyStrategy(proxies)
@@ -250,12 +228,14 @@ strategy = RoundRobinProxyStrategy(proxies)
# Configure rotation per request (recommended) # Configure rotation per request (recommended)
run_config = CrawlerRunConfig(proxy_rotation_strategy=strategy) run_config = CrawlerRunConfig(proxy_rotation_strategy=strategy)
# If you want a single static proxy across all requests, set a fixed ProxyConfig at browser-level: # For a fixed proxy across all requests, just reuse the same run_config instance
# browser_config = BrowserConfig(proxy_config=proxies[0]) static_run_config = run_config
``` ```
### 2. SSL Certificate Verification ### 2. SSL Certificate Verification
```python ```python
from crawl4ai import CrawlerRunConfig
# Always verify SSL certificates when possible # Always verify SSL certificates when possible
# Per-request (affects specific requests) # Per-request (affects specific requests)
run_config = CrawlerRunConfig(fetch_ssl_certificate=True) run_config = CrawlerRunConfig(fetch_ssl_certificate=True)
@@ -270,30 +250,24 @@ export PROXIES="ip1:port1:user1:pass1,ip2:port2:user2:pass2"
### 4. SOCKS5 for Enhanced Security ### 4. SOCKS5 for Enhanced Security
```python ```python
# Prefer SOCKS5 proxies for better protocol support from crawl4ai import CrawlerRunConfig
# Browser-level
browser_config = BrowserConfig(proxy_config="socks5://proxy.example.com:1080")
# Or request-level # Prefer SOCKS5 proxies for better protocol support
run_config = CrawlerRunConfig(proxy_config="socks5://proxy.example.com:1080") run_config = CrawlerRunConfig(proxy_config="socks5://proxy.example.com:1080")
``` ```
## Migration from Deprecated `proxy` Parameter ## Migration from Deprecated `proxy` Parameter
!!! warning "Deprecation Notice" !!! warning "Deprecation Notice"
The `proxy` parameter in `BrowserConfig` is deprecated. Use `proxy_config` in either `BrowserConfig` or `CrawlerRunConfig` instead. The legacy `proxy` argument on `BrowserConfig` is deprecated. Configure proxies through `CrawlerRunConfig.proxy_config` so each request fully describes its network settings.
```python ```python
# Old (deprecated) # Old (deprecated) approach
browser_config = BrowserConfig(proxy="http://proxy.example.com:8080") # from crawl4ai import BrowserConfig
# browser_config = BrowserConfig(proxy="http://proxy.example.com:8080")
# You will see a warning similar to: # New (preferred) approach
# DeprecationWarning: BrowserConfig.proxy is deprecated and ignored. Use proxy_config instead. from crawl4ai import CrawlerRunConfig
# New (recommended) - Browser-level default
browser_config = BrowserConfig(proxy_config="http://proxy.example.com:8080")
# Or request-level override (takes precedence per request)
run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080") run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080")
``` ```
@@ -311,23 +285,20 @@ def safe_proxy_repr(proxy: ProxyConfig):
### Common Issues ### Common Issues
1. **Proxy Connection Failed** ???+ question "Proxy connection failed"
- Verify proxy server is accessible - Verify the proxy server is reachable from your network.
- Check authentication credentials - Double-check authentication credentials.
- Ensure correct protocol (http/https/socks5) - Ensure the protocol matches (`http`, `https`, or `socks5`).
2. **SSL Certificate Errors** ???+ question "SSL certificate errors"
- Some proxies may interfere with SSL inspection - Some proxies break SSL inspection; switch proxies if you see repeated failures.
- Try different proxy or disable SSL verification if necessary - Consider temporarily disabling certificate fetching to isolate the issue.
3. **Environment Variables Not Loading** ???+ question "Environment variables not loading"
- Ensure PROXIES variable is set correctly - Confirm `PROXIES` (or your custom env var) is set before running the script.
- Check comma separation and format: `ip:port:user:pass,ip:port:user:pass` - Check formatting: `ip:port:user:pass,ip:port:user:pass`.
4. **Proxy Rotation Not Working**
- Verify proxies are loaded: `len(proxies) > 0`
- Check proxy strategy is set on `CrawlerRunConfig` via `proxy_rotation_strategy`
- Ensure `proxy_config` is a valid `ProxyConfig` (when using a static proxy)
<!-- Removed duplicate Supported Proxy Formats section (already covered above) -->
???+ question "Proxy rotation not working"
- Ensure `ProxyConfig.from_env()` actually loaded entries (`len(proxies) > 0`).
- Attach `proxy_rotation_strategy` to `CrawlerRunConfig`.
- Validate the proxy definitions you pass into the strategy.