Implement a privacy-first, provider-agnostic telemetry system to help improve Crawl4AI stability
through anonymous crash reporting. The system is designed with user privacy as the top priority,
collecting only exception information without any PII, URLs, or crawled content.
Architecture & Design:
- Provider-agnostic architecture with base TelemetryProvider interface
- Sentry as the initial provider implementation with easy extensibility
- Separate handling for sync and async code paths
- Environment-aware behavior (CLI, Docker, Jupyter/Colab)
Key Features:
- Opt-in by default for CLI/library usage with interactive consent prompt
- Opt-out by default for Docker/API server (enabled unless CRAWL4AI_TELEMETRY=0)
- Jupyter/Colab support with widget-based consent (fallback to code snippets)
- Persistent consent storage in ~/.crawl4ai/config.json
- Optional email collection for critical issue follow-up
CLI Integration:
- `crwl telemetry enable [--email <email>] [--once]` - Enable telemetry
- `crwl telemetry disable` - Disable telemetry
- `crwl telemetry status` - Check current status
Python API:
- Decorators: @telemetry_decorator, @async_telemetry_decorator
- Context managers: telemetry_context(), async_telemetry_context()
- Manual capture: capture_exception(exc, context)
- Control: telemetry.enable(), telemetry.disable(), telemetry.status()
Privacy Safeguards:
- No URL collection
- No request/response data
- No authentication tokens or cookies
- No crawled content
- Automatic sanitization of sensitive fields
- Local consent storage only
Testing:
- Comprehensive test suite with 15 test cases
- Coverage for all environments and consent flows
- Mock providers for testing without external dependencies
Documentation:
- Detailed documentation in docs/md_v2/core/telemetry.md
- Added to mkdocs navigation under Core section
- Privacy commitment and FAQ included
- Examples for all usage patterns
Installation:
- Optional dependency: pip install crawl4ai[telemetry]
- Graceful degradation if sentry-sdk not installed
- Added to pyproject.toml optional dependencies
- Docker requirements updated
Integration Points:
- AsyncWebCrawler: Automatic exception capture in arun() and aprocess_html()
- Docker server: Automatic initialization with environment control
- Global exception handler for uncaught exceptions (CLI only)
This implementation provides valuable error insights to improve Crawl4AI while maintaining
complete transparency and user control over data collection.
Major updates to Docker deployment infrastructure:
- Switch default port to 11235 for all services
- Add MCP (Model Context Protocol) support with WebSocket/SSE endpoints
- Simplify docker-compose.yml with auto-platform detection
- Update documentation with new features and examples
- Consolidate configuration and improve resource management
BREAKING CHANGE: Default port changed from 8020 to 11235. Update your configurations and deployment scripts accordingly.
This commit introduces several significant enhancements to the Crawl4AI Docker deployment:
1. Add MCP Protocol Support:
- Implement WebSocket and SSE transport layers for MCP server communication
- Create mcp_bridge.py to expose existing API endpoints via MCP protocol
- Add comprehensive tests for both socket and SSE transport methods
2. Enhance Docker Server Capabilities:
- Add PDF generation endpoint with file saving functionality
- Add screenshot capture endpoint with configurable wait time
- Implement JavaScript execution endpoint for dynamic page interaction
- Add intelligent file path handling for saving generated assets
3. Improve Search and Context Functionality:
- Implement syntax-aware code function chunking using AST parsing
- Add BM25-based intelligent document search with relevance scoring
- Create separate code and documentation context endpoints
- Enhance response format with structured results and scores
4. Rename and Fix File Organization:
- Fix typo in test_docker_config_gen.py filename
- Update import statements and dependencies
- Add FileResponse for context endpoints
This enhancement significantly improves the machine-to-machine communication
capabilities of Crawl4AI, making it more suitable for integration with LLM agents
and other automated systems.
The CHANGELOG update has been applied successfully, highlighting the key features and improvements made in this release. The commit message provides a detailed explanation of all the
changes, which will be helpful for tracking the project's evolution.
Add supervisor configuration for managing Redis and Gunicorn processes
Replace direct process management with supervisord
Add secure and token-free API server variants
Implement JWT authentication for protected endpoints
Update datetime handling in async dispatcher
Add email domain verification
BREAKING CHANGE: Server startup now uses supervisord instead of direct process management
Complete overhaul of Docker deployment setup with improved architecture:
- Add Redis integration for task management
- Implement rate limiting and security middleware
- Add Prometheus metrics and health checks
- Improve error handling and logging
- Add support for streaming responses
- Implement proper configuration management
- Add platform-specific optimizations for ARM64/AMD64
BREAKING CHANGE: Docker deployment now requires Redis and new config.yml structure
Add comprehensive Docker deployment configuration with:
- New .dockerignore and .llm.env.example files
- Enhanced Dockerfile with multi-stage build and optimizations
- Detailed README with setup instructions and environment configurations
- Improved requirements.txt with Gunicorn
- Better error handling in async_configs.py
BREAKING CHANGE: Docker deployment now requires .llm.env file for API keys
Add Docker deployment setup with FastAPI server implementation for Crawl4AI:
- Create Dockerfile with Python 3.10 and Playwright dependencies
- Implement FastAPI server with streaming and non-streaming endpoints
- Add request/response models and JSON serialization
- Include test script for API verification
Also includes:
- Update .gitignore for Continue development files
- Add project rules in .continuerules
- Clean up async_dispatcher.py formatting
