feat(telemetry): Add opt-in telemetry system for error tracking and stability improvement
Implement a privacy-first, provider-agnostic telemetry system to help improve Crawl4AI stability through anonymous crash reporting. The system is designed with user privacy as the top priority, collecting only exception information without any PII, URLs, or crawled content. Architecture & Design: - Provider-agnostic architecture with base TelemetryProvider interface - Sentry as the initial provider implementation with easy extensibility - Separate handling for sync and async code paths - Environment-aware behavior (CLI, Docker, Jupyter/Colab) Key Features: - Opt-in by default for CLI/library usage with interactive consent prompt - Opt-out by default for Docker/API server (enabled unless CRAWL4AI_TELEMETRY=0) - Jupyter/Colab support with widget-based consent (fallback to code snippets) - Persistent consent storage in ~/.crawl4ai/config.json - Optional email collection for critical issue follow-up CLI Integration: - `crwl telemetry enable [--email <email>] [--once]` - Enable telemetry - `crwl telemetry disable` - Disable telemetry - `crwl telemetry status` - Check current status Python API: - Decorators: @telemetry_decorator, @async_telemetry_decorator - Context managers: telemetry_context(), async_telemetry_context() - Manual capture: capture_exception(exc, context) - Control: telemetry.enable(), telemetry.disable(), telemetry.status() Privacy Safeguards: - No URL collection - No request/response data - No authentication tokens or cookies - No crawled content - Automatic sanitization of sensitive fields - Local consent storage only Testing: - Comprehensive test suite with 15 test cases - Coverage for all environments and consent flows - Mock providers for testing without external dependencies Documentation: - Detailed documentation in docs/md_v2/core/telemetry.md - Added to mkdocs navigation under Core section - Privacy commitment and FAQ included - Examples for all usage patterns Installation: - Optional dependency: pip install crawl4ai[telemetry] - Graceful degradation if sentry-sdk not installed - Added to pyproject.toml optional dependencies - Docker requirements updated Integration Points: - AsyncWebCrawler: Automatic exception capture in arun() and aprocess_html() - Docker server: Automatic initialization with environment control - Global exception handler for uncaught exceptions (CLI only) This implementation provides valuable error insights to improve Crawl4AI while maintaining complete transparency and user control over data collection.
This commit is contained in:
@@ -15,3 +15,4 @@ PyJWT==2.10.1
|
||||
mcp>=1.6.0
|
||||
websockets>=15.0.1
|
||||
httpx[http2]>=0.27.2
|
||||
sentry-sdk>=2.0.0
|
||||
|
||||
@@ -74,6 +74,32 @@ setup_logging(config)
|
||||
|
||||
__version__ = "0.5.1-d1"
|
||||
|
||||
# ───────────────────── telemetry setup ────────────────────────
|
||||
# Docker/API server telemetry: enabled by default unless CRAWL4AI_TELEMETRY=0
|
||||
import os as _os
|
||||
if _os.environ.get('CRAWL4AI_TELEMETRY') != '0':
|
||||
# Set environment variable to indicate we're in API server mode
|
||||
_os.environ['CRAWL4AI_API_SERVER'] = 'true'
|
||||
|
||||
# Import and enable telemetry for Docker/API environment
|
||||
from crawl4ai.telemetry import enable as enable_telemetry
|
||||
from crawl4ai.telemetry import capture_exception
|
||||
|
||||
# Enable telemetry automatically in Docker mode
|
||||
enable_telemetry(always=True)
|
||||
|
||||
import logging
|
||||
telemetry_logger = logging.getLogger("telemetry")
|
||||
telemetry_logger.info("✅ Telemetry enabled for Docker/API server")
|
||||
else:
|
||||
# Define no-op for capture_exception if telemetry is disabled
|
||||
def capture_exception(exc, context=None):
|
||||
pass
|
||||
|
||||
import logging
|
||||
telemetry_logger = logging.getLogger("telemetry")
|
||||
telemetry_logger.info("❌ Telemetry disabled via CRAWL4AI_TELEMETRY=0")
|
||||
|
||||
# ── global page semaphore (hard cap) ─────────────────────────
|
||||
MAX_PAGES = config["crawler"]["pool"].get("max_pages", 30)
|
||||
GLOBAL_SEM = asyncio.Semaphore(MAX_PAGES)
|
||||
|
||||
Reference in New Issue
Block a user