From 40173eeb7374dd5d3ab84b355b28e88d43703ee0 Mon Sep 17 00:00:00 2001 From: Nasrin Date: Wed, 22 Oct 2025 22:34:19 +0800 Subject: [PATCH] Update Docker hooks and Webhook documents (#1557) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(docker-api): migrate to modern datetime library API Signed-off-by: Emmanuel Ferdman * Fix examples in README.md * feat(docker): add user-provided hooks support to Docker API Implements comprehensive hooks functionality allowing users to provide custom Python functions as strings that execute at specific points in the crawling pipeline. Key Features: - Support for all 8 crawl4ai hook points: • on_browser_created: Initialize browser settings • on_page_context_created: Configure page context • before_goto: Pre-navigation setup • after_goto: Post-navigation processing • on_user_agent_updated: User agent modification handling • on_execution_started: Crawl execution initialization • before_retrieve_html: Pre-extraction processing • before_return_html: Final HTML processing Implementation Details: - Created UserHookManager for validation, compilation, and safe execution - Added IsolatedHookWrapper for error isolation and timeout protection - AST-based validation ensures code structure correctness - Sandboxed execution with restricted builtins for security - Configurable timeout (1-120 seconds) prevents infinite loops - Comprehensive error handling ensures hooks don't crash main process - Execution tracking with detailed statistics and logging API Changes: - Added HookConfig schema with code and timeout fields - Extended CrawlRequest with optional hooks parameter - Added /hooks/info endpoint for hook discovery - Updated /crawl and /crawl/stream endpoints to support hooks Safety Features: - Malformed hooks return clear validation errors - Hook errors are isolated and reported without stopping crawl - Execution statistics track success/failure/timeout rates - All hook results are JSON-serializable Testing: - Comprehensive test suite covering all 8 hooks - Error handling and timeout scenarios validated - Authentication, performance, and content extraction examples - 100% success rate in production testing Documentation: - Added extensive hooks section to docker-deployment.md - Security warnings about user-provided code risks - Real-world examples using httpbin.org, GitHub, BBC - Best practices and troubleshooting guide ref #1377 * fix(deep-crawl): BestFirst priority inversion; remove pre-scoring truncation. ref #1253 Use negative scores in PQ to visit high-score URLs first and drop link cap prior to scoring; add test for ordering. * docs: Update URL seeding examples to use proper async context managers - Wrap all AsyncUrlSeeder usage with async context managers - Update URL seeding adventure example to use "sitemap+cc" source, focus on course posts, and add stream=True parameter to fix runtime error * fix(crawler): Removed the incorrect reference in browser_config variable #1310 * docs: update Docker instructions to use the latest release tag * fix(docker): Fix LLM API key handling for multi-provider support Previously, the system incorrectly used OPENAI_API_KEY for all LLM providers due to a hardcoded api_key_env fallback in config.yml. This caused authentication errors when using non-OpenAI providers like Gemini. Changes: - Remove api_key_env from config.yml to let litellm handle provider-specific env vars - Simplify get_llm_api_key() to return None, allowing litellm to auto-detect keys - Update validate_llm_provider() to trust litellm's built-in key detection - Update documentation to reflect the new automatic key handling The fix leverages litellm's existing capability to automatically find the correct environment variable for each provider (OPENAI_API_KEY, GEMINI_API_TOKEN, etc.) without manual configuration. ref #1291 * docs: update adaptive crawler docs and cache defaults; remove deprecated examples (#1330) - Replace BaseStrategy with CrawlStrategy in custom strategy examples (DomainSpecificStrategy, HybridStrategy) - Remove “Custom Link Scoring” and “Caching Strategy” sections no longer aligned with current library - Revise memory pruning example to use adaptive.get_relevant_content and index-based retention of top 500 docs - Correct Quickstart note: default cache mode is CacheMode.BYPASS; instruct enabling with CacheMode.ENABLED * fix(utils): Improve URL normalization by avoiding quote/unquote to preserve '+' signs. ref #1332 * feat: Add comprehensive website to API example with frontend This commit adds a complete, web scraping API example that demonstrates how to get structured data from any website and use it like an API using the crawl4ai library with a minimalist frontend interface. Core Functionality - AI-powered web scraping with plain English queries - Dual scraping approaches: Schema-based (faster) and LLM-based (flexible) - Intelligent schema caching for improved performance - Custom LLM model support with API key management - Automatic duplicate request prevention Modern Frontend Interface - Minimalist black-and-white design inspired by modern web apps - Responsive layout with smooth animations and transitions - Three main pages: Scrape Data, Models Management, API Request History - Real-time results display with JSON formatting - Copy-to-clipboard functionality for extracted data - Toast notifications for user feedback - Auto-scroll to results when scraping starts Model Management System - Web-based model configuration interface - Support for any LLM provider (OpenAI, Gemini, Anthropic, etc.) - Simplified configuration requiring only provider and API token - Add, list, and delete model configurations - Secure storage of API keys in local JSON files API Request History - Automatic saving of all API requests and responses - Display of request history with URL, query, and cURL commands - Duplicate prevention (same URL + query combinations) - Request deletion functionality - Clean, simplified display focusing on essential information Technical Implementation Backend (FastAPI) - RESTful API with comprehensive endpoints - Pydantic models for request/response validation - Async web scraping with crawl4ai library - Error handling with detailed error messages - File-based storage for models and request history Frontend (Vanilla JS/CSS/HTML) - No framework dependencies - pure HTML, CSS, JavaScript - Modern CSS Grid and Flexbox layouts - Custom dropdown styling with SVG arrows - Responsive design for mobile and desktop - Smooth scrolling and animations Core Library Integration - WebScraperAgent class for orchestration - ModelConfig class for LLM configuration management - Schema generation and caching system - LLM extraction strategy support - Browser configuration with headless mode * fix(dependencies): add cssselect to project dependencies Fixes bug reported in issue #1405 [Bug]: Excluded selector (excluded_selector) doesn't work This commit reintroduces the cssselect library which was removed by PR (https://github.com/unclecode/crawl4ai/pull/1368) and merged via (https://github.com/unclecode/crawl4ai/commit/437395e4902e2925ea55319353f64555b6da407b). Integration tested against 0.7.4 Docker container. Reintroducing cssselector package eliminated errors seen in logs and excluded_selector functionality was restored. Refs: #1405 * fix(docker): resolve filter serialization and JSON encoding errors in deep crawl strategy (ref #1419) - Fix URLPatternFilter serialization by preventing private __slots__ from being serialized as constructor params - Add public attributes to URLPatternFilter to store original constructor parameters for proper serialization - Handle property descriptors in CrawlResult.model_dump() to prevent JSON serialization errors - Ensure filter chains work correctly with Docker client and REST API The issue occurred because: 1. Private implementation details (_simple_suffixes, etc.) were being serialized and passed as constructor arguments during deserialization 2. Property descriptors were being included in the serialized output, causing "Object of type property is not JSON serializable" errors Changes: - async_configs.py: Comment out __slots__ serialization logic (lines 100-109) - filters.py: Add patterns, use_glob, reverse to URLPatternFilter __slots__ and store as public attributes - models.py: Convert property descriptors to strings in model_dump() instead of including them directly * fix(logger): ensure logger is a Logger instance in crawling strategies. ref #1437 * feat(docker): Add temperature and base_url parameters for LLM configuration. ref #1035 Implement hierarchical configuration for LLM parameters with support for: - Temperature control (0.0-2.0) to adjust response creativity - Custom base_url for proxy servers and alternative endpoints - 4-tier priority: request params > provider env > global env > defaults Add helper functions in utils.py, update API schemas and handlers, support environment variables (LLM_TEMPERATURE, OPENAI_TEMPERATURE, etc.), and provide comprehensive documentation with examples. * feat(docker): improve docker error handling - Return comprehensive error messages along with status codes for api internal errors. - Fix fit_html property serialization issue in both /crawl and /crawl/stream endpoints - Add sanitization to ensure fit_html is always JSON-serializable (string or None) - Add comprehensive error handling test suite. * #1375 : refactor(proxy) Deprecate 'proxy' parameter in BrowserConfig and enhance proxy string parsing - Updated ProxyConfig.from_string to support multiple proxy formats, including URLs with credentials. - Deprecated the 'proxy' parameter in BrowserConfig, replacing it with 'proxy_config' for better flexibility. - Added warnings for deprecated usage and clarified behavior when both parameters are provided. - Updated documentation and tests to reflect changes in proxy configuration handling. * Remove deprecated test for 'proxy' parameter in BrowserConfig and update .gitignore to include test_scripts directory. * feat: add preserve_https_for_internal_links flag to maintain HTTPS during crawling. Ref #1410 Added a new `preserve_https_for_internal_links` configuration flag that preserves the original HTTPS scheme for same-domain links even when the server redirects to HTTP. * feat: update documentation for preserve_https_for_internal_links. ref #1410 * fix: drop Python 3.9 support and require Python >=3.10. The library no longer supports Python 3.9 and so it was important to drop all references to python 3.9. Following changes have been made: - pyproject.toml: set requires-python to ">=3.10"; remove 3.9 classifier - setup.py: set python_requires to ">=3.10"; remove 3.9 classifier - docs: update Python version mentions - deploy/docker/c4ai-doc-context.md: options -> 3.10, 3.11, 3.12, 3.13 * issue #1329 refactor(crawler): move unwanted properties to CrawlerRunConfig class * fix(auth): fixed Docker JWT authentication. ref #1442 * remove: delete unused yoyo snapshot subproject * fix: raise error on last attempt failure in perform_completion_with_backoff. ref #989 * Commit without API * fix: update option labels in request builder for clarity * fix: allow custom LLM providers for adaptive crawler embedding config. ref: #1291 - Change embedding_llm_config from Dict to Union[LLMConfig, Dict] for type safety - Add backward-compatible conversion property _embedding_llm_config_dict - Replace all hardcoded OpenAI embedding configs with configurable options - Fix LLMConfig object attribute access in query expansion logic - Add comprehensive example demonstrating multiple provider configurations - Update documentation with both LLMConfig object and dictionary usage patterns Users can now specify any LLM provider for query expansion in embedding strategy: - New: embedding_llm_config=LLMConfig(provider='anthropic/claude-3', api_token='key') - Old: embedding_llm_config={'provider': 'openai/gpt-4', 'api_token': 'key'} (still works) * refactor(BrowserConfig): change deprecation warning for 'proxy' parameter to UserWarning * feat(StealthAdapter): fix stealth features for Playwright integration. ref #1481 * #1505 fix(api): update config handling to only set base config if not provided by user * fix(docker-deployment): replace console.log with print for metadata extraction * Release v0.7.5: The Update - Updated version to 0.7.5 - Added comprehensive demo and release notes - Updated documentation * refactor(release): remove memory management section for cleaner documentation. ref #1443 * feat(docs): add brand book and page copy functionality - Add comprehensive brand book with color system, typography, components - Add page copy dropdown with markdown copy/view functionality - Update mkdocs.yml with new assets and branding navigation - Use terminal-style ASCII icons and condensed menu design * Update gitignore add local scripts folder * fix: remove this import as it causes python to treat "json" as a variable in the except block * fix: always return a list, even if we catch an exception * feat(marketplace): Add Crawl4AI marketplace with secure configuration - Implement marketplace frontend and admin dashboard - Add FastAPI backend with environment-based configuration - Use .env file for secrets management - Include data generation scripts - Add proper CORS configuration - Remove hardcoded password from admin login - Update gitignore for security * fix(marketplace): Update URLs to use /marketplace path and relative API endpoints - Change API_BASE to relative '/api' for production - Move marketplace to /marketplace instead of /marketplace/frontend - Update MkDocs navigation - Fix logo path in marketplace index * fix(docs): hide copy menu on non-markdown pages * feat(marketplace): add sponsor logo uploads Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * feat(docs): add chatgpt quick link to page actions * fix(marketplace): align admin api with backend endpoints * fix(marketplace): isolate api under marketplace prefix * fix(marketplace): resolve app detail page routing and styling issues - Fixed JavaScript errors from missing HTML elements (install-code, usage-code, integration-code) - Added missing CSS classes for tabs, overview layout, sidebar, and integration content - Fixed tab navigation to display horizontally in single line - Added proper padding to tab content sections (removed from container, added to content) - Fixed tab selector from .nav-tab to .tab-btn to match HTML structure - Added sidebar styling with stats grid and metadata display - Improved responsive design with mobile-friendly tab scrolling - Fixed code block positioning for copy buttons - Removed margin from first headings to prevent extra spacing - Added null checks for DOM elements in JavaScript to prevent errors These changes resolve the routing issue where clicking on apps caused page redirects, and fix the broken layout where CSS was not properly applied to the app detail page. * fix(marketplace): prevent hero image overflow and secondary card stretching - Fixed hero image to 200px height with min/max constraints - Added object-fit: cover to hero-image img elements - Changed secondary-featured align-items from stretch to flex-start - Fixed secondary-card height to 118px (no flex: 1 stretching) - Updated responsive grid layouts for wider screens - Added flex: 1 to hero-content for better content distribution These changes ensure a rigid, predictable layout that prevents: 1. Large images from pushing text content down 2. Single secondary cards from stretching to fill entire height * feat: Add hooks utility for function-based hooks with Docker client integration. ref #1377 Add hooks_to_string() utility function that converts Python function objects to string representations for the Docker API, enabling developers to write hooks as regular Python functions instead of strings. Core Changes: - New hooks_to_string() utility in crawl4ai/utils.py using inspect.getsource() - Docker client now accepts both function objects and strings for hooks - Automatic detection and conversion in Crawl4aiDockerClient._prepare_request() - New hooks and hooks_timeout parameters in client.crawl() method Documentation: - Docker client examples with function-based hooks (docs/examples/docker_client_hooks_example.py) - Updated main Docker deployment guide with comprehensive hooks section - Added unit tests for hooks utility (tests/docker/test_hooks_utility.py) * feat: Add hooks utility for function-based hooks with Docker client integration. ref #1377 Add hooks_to_string() utility function that converts Python function objects to string representations for the Docker API, enabling developers to write hooks as regular Python functions instead of strings. Core Changes: - New hooks_to_string() utility in crawl4ai/utils.py using inspect.getsource() - Docker client now accepts both function objects and strings for hooks - Automatic detection and conversion in Crawl4aiDockerClient._prepare_request() - New hooks and hooks_timeout parameters in client.crawl() method Documentation: - Docker client examples with function-based hooks (docs/examples/docker_client_hooks_example.py) - Updated main Docker deployment guide with comprehensive hooks section - Added unit tests for hooks utility (tests/docker/test_hooks_utility.py) * fix(docs): clarify Docker Hooks System with function-based API in README * docs: Add demonstration files for v0.7.5 release, showcasing the new Docker Hooks System and all other features. * docs: Update 0.7.5 video walkthrough * docs: add complete SDK reference documentation Add comprehensive single-page SDK reference combining: - Installation & Setup - Quick Start - Core API (AsyncWebCrawler, arun, arun_many, CrawlResult) - Configuration (BrowserConfig, CrawlerConfig, Parameters) - Crawling Patterns - Content Processing (Markdown, Fit Markdown, Selection, Interaction, Link & Media) - Extraction Strategies (LLM and No-LLM) - Advanced Features (Session Management, Hooks & Auth) Generated using scripts/generate_sdk_docs.py in ultra-dense mode optimized for AI assistant consumption. Stats: 23K words, 185 code blocks, 220KB * feat: add AI assistant skill package for Crawl4AI - Create comprehensive skill package for AI coding assistants - Include complete SDK reference (23K words, v0.7.4) - Add three extraction scripts (basic, batch, pipeline) - Implement version tracking in skill and scripts - Add prominent download section on homepage - Place skill in docs/assets for web distribution The skill enables AI assistants like Claude, Cursor, and Windsurf to effectively use Crawl4AI with optimized workflows for markdown generation and data extraction. * fix: remove non-existent wiki link and clarify skill usage instructions * fix: update Crawl4AI skill with corrected parameters and examples - Fixed CrawlerConfig → CrawlerRunConfig throughout - Fixed parameter names (timeout → page_timeout, store_html removed) - Fixed schema format (selector → baseSelector) - Corrected proxy configuration (in BrowserConfig, not CrawlerRunConfig) - Fixed fit_markdown usage with content filters - Added comprehensive references to docs/examples/ directory - Created safe packaging script to avoid root directory pollution - All scripts tested and verified working * fix: thoroughly verify and fix all Crawl4AI skill examples - Cross-checked every section against actual docs - Fixed BM25ContentFilter parameters (user_query, bm25_threshold) - Removed incorrect wait_for selector from basic example - Added comprehensive test suite (4 test files) - All examples now tested and verified working - Tests validate: basic crawling, markdown generation, data extraction, advanced patterns - Package size: 76.6 KB (includes tests for future validation) * feat(ci): split release pipeline and add Docker caching - Split release.yml into PyPI/GitHub release and Docker workflows - Add GitHub Actions cache for Docker builds (10-15x faster rebuilds) - Implement dual-trigger for docker-release.yml (auto + manual) - Add comprehensive workflow documentation in .github/workflows/docs/ - Backup original workflow as release.yml.backup * feat: add webhook notifications for crawl job completion Implements webhook support for the crawl job API to eliminate polling requirements. Changes: - Added WebhookConfig and WebhookPayload schemas to schemas.py - Created webhook.py with WebhookDeliveryService class - Integrated webhook notifications in api.py handle_crawl_job - Updated job.py CrawlJobPayload to accept webhook_config - Added webhook configuration section to config.yml - Included comprehensive usage examples in WEBHOOK_EXAMPLES.md Features: - Webhook notifications on job completion (success/failure) - Configurable data inclusion in webhook payload - Custom webhook headers support - Global default webhook URL configuration - Exponential backoff retry logic (5 attempts: 1s, 2s, 4s, 8s, 16s) - 30-second timeout per webhook call Usage: POST /crawl/job with optional webhook_config: - webhook_url: URL to receive notifications - webhook_data_in_payload: include full results (default: false) - webhook_headers: custom headers for authentication Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude * docs: add webhook documentation to Docker README Added comprehensive webhook section to README.md including: - Overview of asynchronous job queue with webhooks - Benefits and use cases - Quick start examples - Webhook authentication - Global webhook configuration - Job status polling alternative Updated table of contents and summary to include webhook feature. Maintains consistent tone and style with rest of README. Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude * docs: add webhook example for Docker deployment Added docker_webhook_example.py demonstrating: - Submitting crawl jobs with webhook configuration - Flask-based webhook receiver implementation - Three usage patterns: 1. Webhook notification only (fetch data separately) 2. Webhook with full data in payload 3. Traditional polling approach for comparison Includes comprehensive comments explaining: - Webhook payload structure - Authentication headers setup - Error handling - Production deployment tips Example is fully functional and ready to run with Flask installed. Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude * test: add webhook implementation validation tests Added comprehensive test suite to validate webhook implementation: - Module import verification - WebhookDeliveryService initialization - Pydantic model validation (WebhookConfig) - Payload construction logic - Exponential backoff calculation - API integration checks All tests pass (6/6), confirming implementation is correct. Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude * test: add comprehensive webhook feature test script Added end-to-end test script that automates webhook feature testing: Script Features (test_webhook_feature.sh): - Automatic branch switching and dependency installation - Redis and server startup/shutdown management - Webhook receiver implementation - Integration test for webhook notifications - Comprehensive cleanup and error handling - Returns to original branch after completion Test Flow: 1. Fetch and checkout webhook feature branch 2. Activate venv and install dependencies 3. Start Redis and Crawl4AI server 4. Submit crawl job with webhook config 5. Verify webhook delivery and payload 6. Clean up all processes and return to original branch Documentation: - WEBHOOK_TEST_README.md with usage instructions - Troubleshooting guide - Exit codes and safety features Usage: ./tests/test_webhook_feature.sh Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude * fix: properly serialize Pydantic HttpUrl in webhook config Use model_dump(mode='json') instead of deprecated dict() method to ensure Pydantic special types (HttpUrl, UUID, etc.) are properly serialized to JSON-compatible native Python types. This fixes webhook delivery failures caused by HttpUrl objects remaining as Pydantic types in the webhook_config dict, which caused JSON serialization errors and httpx request failures. Also update mcp requirement to >=1.18.0 for compatibility. * feat: add webhook support for /llm/job endpoint Add comprehensive webhook notification support for the /llm/job endpoint, following the same pattern as the existing /crawl/job implementation. Changes: - Add webhook_config field to LlmJobPayload model (job.py) - Implement webhook notifications in process_llm_extraction() with 4 notification points: success, provider validation failure, extraction failure, and general exceptions (api.py) - Store webhook_config in Redis task data for job tracking - Initialize WebhookDeliveryService with exponential backoff retry logic Documentation: - Add Example 6 to WEBHOOK_EXAMPLES.md showing LLM extraction with webhooks - Update Flask webhook handler to support both crawl and llm_extraction tasks - Add TypeScript client examples for LLM jobs - Add comprehensive examples to docker_webhook_example.py with schema support - Clarify data structure differences between webhook and API responses Testing: - Add test_llm_webhook_feature.py with 7 validation tests (all passing) - Verify pattern consistency with /crawl/job implementation - Add implementation guide (WEBHOOK_LLM_JOB_IMPLEMENTATION.md) * fix: remove duplicate comma in webhook_config parameter * fix: update Crawl4AI Docker container port from 11234 to 11235 * docs: enhance README and docker-deployment documentation with Job Queue and Webhook API details * docs: update docker_hooks_examples.py with comprehensive examples and improved structure --------- Signed-off-by: Emmanuel Ferdman Co-authored-by: Emmanuel Ferdman Co-authored-by: Nezar Ali Co-authored-by: Soham Kukreti Co-authored-by: James T. Wood Co-authored-by: AHMET YILMAZ Co-authored-by: nafeqq-1306 Co-authored-by: unclecode Co-authored-by: Martin Sjöborg Co-authored-by: Martin Sjöborg Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Claude --- deploy/docker/README.md | 48 ++ docs/examples/docker_hooks_examples.py | 922 ++++++++++++++----------- docs/md_v2/core/docker-deployment.md | 466 +++++++++++++ 3 files changed, 1032 insertions(+), 404 deletions(-) diff --git a/deploy/docker/README.md b/deploy/docker/README.md index 2c920ef1..cee8af7f 100644 --- a/deploy/docker/README.md +++ b/deploy/docker/README.md @@ -785,6 +785,54 @@ curl http://localhost:11235/crawl/job/crawl_xyz The response includes `status` field: `"processing"`, `"completed"`, or `"failed"`. +#### LLM Extraction Jobs with Webhooks + +The same webhook system works for LLM extraction jobs via `/llm/job`: + +```bash +# Submit LLM extraction job with webhook +curl -X POST http://localhost:11235/llm/job \ + -H "Content-Type: application/json" \ + -d '{ + "url": "https://example.com/article", + "q": "Extract the article title, author, and main points", + "provider": "openai/gpt-4o-mini", + "webhook_config": { + "webhook_url": "https://myapp.com/webhooks/llm-complete", + "webhook_data_in_payload": true, + "webhook_headers": { + "X-Webhook-Secret": "your-secret-token" + } + } + }' + +# Response: {"task_id": "llm_1234567890"} +``` + +**Your webhook receives:** +```json +{ + "task_id": "llm_1234567890", + "task_type": "llm_extraction", + "status": "completed", + "timestamp": "2025-10-22T12:30:00.000000+00:00", + "urls": ["https://example.com/article"], + "data": { + "extracted_content": { + "title": "Understanding Web Scraping", + "author": "John Doe", + "main_points": ["Point 1", "Point 2", "Point 3"] + } + } +} +``` + +**Key Differences for LLM Jobs:** +- Task type is `"llm_extraction"` instead of `"crawl"` +- Extracted data is in `data.extracted_content` +- Single URL only (not an array) +- Supports schema-based extraction with `schema` parameter + > 💡 **Pro tip**: See [WEBHOOK_EXAMPLES.md](./WEBHOOK_EXAMPLES.md) for detailed examples including TypeScript client code, Flask webhook handlers, and failure handling. --- diff --git a/docs/examples/docker_hooks_examples.py b/docs/examples/docker_hooks_examples.py index a9c94d03..b64caf02 100644 --- a/docs/examples/docker_hooks_examples.py +++ b/docs/examples/docker_hooks_examples.py @@ -1,235 +1,451 @@ #!/usr/bin/env python3 """ -Comprehensive test demonstrating all hook types from hooks_example.py -adapted for the Docker API with real URLs +🚀 Crawl4AI Docker Hooks System - Complete Examples +==================================================== + +This file demonstrates the Docker Hooks System with three different approaches: + +1. String-based hooks for REST API +2. hooks_to_string() utility to convert functions +3. Docker Client with automatic conversion (most convenient) + +Requirements: +- Docker container running: docker run -p 11235:11235 unclecode/crawl4ai:latest +- crawl4ai installed: pip install crawl4ai """ +import asyncio import requests import json import time from typing import Dict, Any -# API_BASE_URL = "http://localhost:11234" -API_BASE_URL = "http://localhost:11235" +# Import Crawl4AI components +from crawl4ai import hooks_to_string +from crawl4ai.docker_client import Crawl4aiDockerClient + +# Configuration +DOCKER_URL = "http://localhost:11235" +TEST_URLS = [ + "https://www.kidocode.com", + "https://quotes.toscrape.com", + "https://httpbin.org/html", +] -def test_all_hooks_demo(): - """Demonstrate all 8 hook types with practical examples""" - print("=" * 70) - print("Testing: All Hooks Comprehensive Demo") - print("=" * 70) - - hooks_code = { - "on_browser_created": """ -async def hook(browser, **kwargs): - # Hook called after browser is created - print("[HOOK] on_browser_created - Browser is ready!") - # Browser-level configurations would go here - return browser -""", - - "on_page_context_created": """ -async def hook(page, context, **kwargs): - # Hook called after a new page and context are created - print("[HOOK] on_page_context_created - New page created!") - - # Set viewport size for consistent rendering - await page.set_viewport_size({"width": 1920, "height": 1080}) - - # Add cookies for the session (using httpbin.org domain) - await context.add_cookies([ - { - "name": "test_session", - "value": "abc123xyz", - "domain": ".httpbin.org", - "path": "/", - "httpOnly": True, - "secure": True - } - ]) - - # Block ads and tracking scripts to speed up crawling - await context.route("**/*.{png,jpg,jpeg,gif,webp,svg}", lambda route: route.abort()) +def print_section(title: str, description: str = ""): + """Print a formatted section header""" + print("\n" + "=" * 70) + print(f" {title}") + if description: + print(f" {description}") + print("=" * 70 + "\n") + + +def check_docker_service() -> bool: + """Check if Docker service is running""" + try: + response = requests.get(f"{DOCKER_URL}/health", timeout=3) + return response.status_code == 200 + except: + return False + + +# ============================================================================ +# REUSABLE HOOK LIBRARY +# ============================================================================ + +async def performance_optimization_hook(page, context, **kwargs): + """ + Performance Hook: Block unnecessary resources to speed up crawling + """ + print(" [Hook] 🚀 Optimizing performance - blocking images and ads...") + + # Block images + await context.route( + "**/*.{png,jpg,jpeg,gif,webp,svg,ico}", + lambda route: route.abort() + ) + + # Block ads and analytics await context.route("**/analytics/*", lambda route: route.abort()) await context.route("**/ads/*", lambda route: route.abort()) - - print("[HOOK] Viewport set, cookies added, and ads blocked") + await context.route("**/google-analytics.com/*", lambda route: route.abort()) + + print(" [Hook] ✓ Performance optimization applied") return page -""", - - "on_user_agent_updated": """ -async def hook(page, context, user_agent, **kwargs): - # Hook called when user agent is updated - print(f"[HOOK] on_user_agent_updated - User agent: {user_agent[:50]}...") + + +async def viewport_setup_hook(page, context, **kwargs): + """ + Viewport Hook: Set consistent viewport size for rendering + """ + print(" [Hook] 🖥️ Setting viewport to 1920x1080...") + await page.set_viewport_size({"width": 1920, "height": 1080}) + print(" [Hook] ✓ Viewport configured") return page -""", - - "before_goto": """ -async def hook(page, context, url, **kwargs): - # Hook called before navigating to each URL - print(f"[HOOK] before_goto - About to visit: {url}") - - # Add custom headers for the request + + +async def authentication_headers_hook(page, context, url, **kwargs): + """ + Headers Hook: Add custom authentication and tracking headers + """ + print(f" [Hook] 🔐 Adding custom headers for {url[:50]}...") + await page.set_extra_http_headers({ - "X-Custom-Header": "crawl4ai-test", - "Accept-Language": "en-US,en;q=0.9", - "DNT": "1" + 'X-Crawl4AI': 'docker-hooks', + 'X-Custom-Hook': 'function-based', + 'Accept-Language': 'en-US,en;q=0.9', }) - + + print(" [Hook] ✓ Custom headers added") return page -""", - - "after_goto": """ -async def hook(page, context, url, response, **kwargs): - # Hook called after navigating to each URL - print(f"[HOOK] after_goto - Successfully loaded: {url}") - - # Wait a moment for dynamic content to load + + +async def lazy_loading_handler_hook(page, context, **kwargs): + """ + Content Hook: Handle lazy-loaded content by scrolling + """ + print(" [Hook] 📜 Scrolling to load lazy content...") + + # Scroll to bottom + await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") await page.wait_for_timeout(1000) - - # Check if specific elements exist (with error handling) - try: - # For httpbin.org, wait for body element - await page.wait_for_selector("body", timeout=2000) - print("[HOOK] Body element found and loaded") - except: - print("[HOOK] Timeout waiting for body, continuing anyway") - - return page -""", - - "on_execution_started": """ -async def hook(page, context, **kwargs): - # Hook called after custom JavaScript execution - print("[HOOK] on_execution_started - Custom JS executed!") - - # You could inject additional JavaScript here if needed - await page.evaluate("console.log('[INJECTED] Hook JS running');") - - return page -""", - - "before_retrieve_html": """ -async def hook(page, context, **kwargs): - # Hook called before retrieving the HTML content - print("[HOOK] before_retrieve_html - Preparing to get HTML") - - # Scroll to bottom to trigger lazy loading - await page.evaluate("window.scrollTo(0, document.body.scrollHeight);") + + # Scroll to middle + await page.evaluate("window.scrollTo(0, document.body.scrollHeight / 2)") await page.wait_for_timeout(500) - + # Scroll back to top - await page.evaluate("window.scrollTo(0, 0);") + await page.evaluate("window.scrollTo(0, 0)") await page.wait_for_timeout(500) - - # One more scroll to middle for good measure - await page.evaluate("window.scrollTo(0, document.body.scrollHeight / 2);") - - print("[HOOK] Scrolling completed for lazy-loaded content") + + print(" [Hook] ✓ Lazy content loaded") return page -""", - - "before_return_html": """ -async def hook(page, context, html, **kwargs): - # Hook called before returning the HTML content - print(f"[HOOK] before_return_html - HTML length: {len(html)} characters") - - # Log some page metrics - metrics = await page.evaluate('''() => { - return { + + +async def page_analytics_hook(page, context, **kwargs): + """ + Analytics Hook: Log page metrics before extraction + """ + print(" [Hook] 📊 Collecting page analytics...") + + metrics = await page.evaluate(''' + () => ({ + title: document.title, images: document.images.length, links: document.links.length, - scripts: document.scripts.length - } - }''') - - print(f"[HOOK] Page metrics - Images: {metrics['images']}, Links: {metrics['links']}, Scripts: {metrics['scripts']}") - + scripts: document.scripts.length, + headings: document.querySelectorAll('h1, h2, h3').length, + paragraphs: document.querySelectorAll('p').length + }) + ''') + + print(f" [Hook] 📈 Page: {metrics['title'][:50]}...") + print(f" Links: {metrics['links']}, Images: {metrics['images']}, " + f"Headings: {metrics['headings']}, Paragraphs: {metrics['paragraphs']}") + + return page + + +# ============================================================================ +# APPROACH 1: String-Based Hooks (REST API) +# ============================================================================ + +def example_1_string_based_hooks(): + """ + Demonstrate string-based hooks with REST API + Use this when working with REST API directly or non-Python clients + """ + print_section( + "APPROACH 1: String-Based Hooks (REST API)", + "Define hooks as strings for REST API requests" + ) + + # Define hooks as strings + hooks_config = { + "on_page_context_created": """ +async def hook(page, context, **kwargs): + print(" [String Hook] Setting up page context...") + # Block images for performance + await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort()) + await page.set_viewport_size({"width": 1920, "height": 1080}) + return page +""", + + "before_goto": """ +async def hook(page, context, url, **kwargs): + print(f" [String Hook] Navigating to {url[:50]}...") + await page.set_extra_http_headers({ + 'X-Crawl4AI': 'string-based-hooks', + }) + return page +""", + + "before_retrieve_html": """ +async def hook(page, context, **kwargs): + print(" [String Hook] Scrolling page...") + await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") + await page.wait_for_timeout(1000) return page """ } - - # Create request payload + + # Prepare request payload payload = { - "urls": ["https://httpbin.org/html"], + "urls": [TEST_URLS[2]], # httpbin.org "hooks": { - "code": hooks_code, + "code": hooks_config, "timeout": 30 }, "crawler_config": { - "js_code": "window.scrollTo(0, document.body.scrollHeight);", - "wait_for": "body", "cache_mode": "bypass" } } - - print("\nSending request with all 8 hooks...") - start_time = time.time() - - response = requests.post(f"{API_BASE_URL}/crawl", json=payload) - - elapsed_time = time.time() - start_time - print(f"Request completed in {elapsed_time:.2f} seconds") - - if response.status_code == 200: - data = response.json() - print("\n✅ Request successful!") - - # Check hooks execution - if 'hooks' in data: - hooks_info = data['hooks'] - print("\n📊 Hooks Execution Summary:") - print(f" Status: {hooks_info['status']['status']}") - print(f" Attached hooks: {len(hooks_info['status']['attached_hooks'])}") - - for hook_name in hooks_info['status']['attached_hooks']: - print(f" ✓ {hook_name}") - - if 'summary' in hooks_info: - summary = hooks_info['summary'] - print(f"\n📈 Execution Statistics:") - print(f" Total executions: {summary['total_executions']}") - print(f" Successful: {summary['successful']}") - print(f" Failed: {summary['failed']}") - print(f" Timed out: {summary['timed_out']}") - print(f" Success rate: {summary['success_rate']:.1f}%") - - if hooks_info.get('execution_log'): - print(f"\n📝 Execution Log:") - for log_entry in hooks_info['execution_log']: - status_icon = "✅" if log_entry['status'] == 'success' else "❌" - exec_time = log_entry.get('execution_time', 0) - print(f" {status_icon} {log_entry['hook_point']}: {exec_time:.3f}s") - - # Check crawl results - if 'results' in data and len(data['results']) > 0: - print(f"\n📄 Crawl Results:") - for result in data['results']: - print(f" URL: {result['url']}") - print(f" Success: {result.get('success', False)}") - if result.get('html'): - print(f" HTML length: {len(result['html'])} characters") - - else: - print(f"❌ Error: {response.status_code}") - try: - error_data = response.json() - print(f"Error details: {json.dumps(error_data, indent=2)}") - except: - print(f"Error text: {response.text[:500]}") + + print(f"🎯 Target URL: {TEST_URLS[2]}") + print(f"🔧 Configured {len(hooks_config)} string-based hooks") + print(f"📡 Sending request to Docker API...\n") + + try: + start_time = time.time() + response = requests.post(f"{DOCKER_URL}/crawl", json=payload, timeout=60) + execution_time = time.time() - start_time + + if response.status_code == 200: + result = response.json() + + print(f"\n✅ Request successful! (took {execution_time:.2f}s)") + + # Display results + if result.get('results') and result['results'][0].get('success'): + crawl_result = result['results'][0] + html_length = len(crawl_result.get('html', '')) + markdown_length = len(crawl_result.get('markdown', '')) + + print(f"\n📊 Results:") + print(f" • HTML length: {html_length:,} characters") + print(f" • Markdown length: {markdown_length:,} characters") + print(f" • URL: {crawl_result.get('url')}") + + # Check hooks execution + if 'hooks' in result: + hooks_info = result['hooks'] + print(f"\n🎣 Hooks Execution:") + print(f" • Status: {hooks_info['status']['status']}") + print(f" • Attached hooks: {len(hooks_info['status']['attached_hooks'])}") + + if 'summary' in hooks_info: + summary = hooks_info['summary'] + print(f" • Total executions: {summary['total_executions']}") + print(f" • Successful: {summary['successful']}") + print(f" • Success rate: {summary['success_rate']:.1f}%") + else: + print(f"⚠️ Crawl completed but no results") + + else: + print(f"❌ Request failed with status {response.status_code}") + print(f" Error: {response.text[:200]}") + + except requests.exceptions.Timeout: + print("⏰ Request timed out after 60 seconds") + except Exception as e: + print(f"❌ Error: {str(e)}") + + print("\n" + "─" * 70) + print("✓ String-based hooks example complete\n") -def test_authentication_flow(): - """Test a complete authentication flow with multiple hooks""" - print("\n" + "=" * 70) - print("Testing: Authentication Flow with Multiple Hooks") - print("=" * 70) - +# ============================================================================ +# APPROACH 2: Function-Based Hooks with hooks_to_string() Utility +# ============================================================================ + +def example_2_hooks_to_string_utility(): + """ + Demonstrate the hooks_to_string() utility for converting functions + Use this when you want to write hooks as functions but use REST API + """ + print_section( + "APPROACH 2: hooks_to_string() Utility", + "Convert Python functions to strings for REST API" + ) + + print("📦 Creating hook functions...") + print(" • performance_optimization_hook") + print(" • authentication_headers_hook") + print(" • lazy_loading_handler_hook") + + # Convert function objects to strings using the utility + print("\n🔄 Converting functions to strings with hooks_to_string()...") + + hooks_dict = { + "on_page_context_created": performance_optimization_hook, + "before_goto": authentication_headers_hook, + "before_retrieve_html": lazy_loading_handler_hook, + } + + hooks_as_strings = hooks_to_string(hooks_dict) + + print(f"✅ Successfully converted {len(hooks_as_strings)} functions to strings") + + # Show a preview + print("\n📝 Sample converted hook (first 200 characters):") + print("─" * 70) + sample_hook = list(hooks_as_strings.values())[0] + print(sample_hook[:200] + "...") + print("─" * 70) + + # Use the converted hooks with REST API + print("\n📡 Using converted hooks with REST API...") + + payload = { + "urls": [TEST_URLS[2]], + "hooks": { + "code": hooks_as_strings, + "timeout": 30 + } + } + + try: + start_time = time.time() + response = requests.post(f"{DOCKER_URL}/crawl", json=payload, timeout=60) + execution_time = time.time() - start_time + + if response.status_code == 200: + result = response.json() + print(f"\n✅ Request successful! (took {execution_time:.2f}s)") + + if result.get('results') and result['results'][0].get('success'): + crawl_result = result['results'][0] + print(f" • HTML length: {len(crawl_result.get('html', '')):,} characters") + print(f" • Hooks executed successfully!") + else: + print(f"❌ Request failed: {response.status_code}") + + except Exception as e: + print(f"❌ Error: {str(e)}") + + print("\n💡 Benefits of hooks_to_string():") + print(" ✓ Write hooks as regular Python functions") + print(" ✓ Full IDE support (autocomplete, syntax highlighting)") + print(" ✓ Type checking and linting") + print(" ✓ Easy to test and debug") + print(" ✓ Reusable across projects") + print(" ✓ Works with any REST API client") + + print("\n" + "─" * 70) + print("✓ hooks_to_string() utility example complete\n") + + +# ============================================================================ +# APPROACH 3: Docker Client with Automatic Conversion (RECOMMENDED) +# ============================================================================ + +async def example_3_docker_client_auto_conversion(): + """ + Demonstrate Docker Client with automatic hook conversion (RECOMMENDED) + Use this for the best developer experience with Python + """ + print_section( + "APPROACH 3: Docker Client with Auto-Conversion (RECOMMENDED)", + "Pass function objects directly - conversion happens automatically!" + ) + + print("🐳 Initializing Crawl4AI Docker Client...") + client = Crawl4aiDockerClient(base_url=DOCKER_URL) + + print("✅ Client ready!\n") + + # Use our reusable hook library - just pass the function objects! + print("📚 Using reusable hook library:") + print(" • performance_optimization_hook") + print(" • authentication_headers_hook") + print(" • lazy_loading_handler_hook") + print(" • page_analytics_hook") + + print("\n🎯 Target URL: " + TEST_URLS[0]) + print("🚀 Starting crawl with automatic hook conversion...\n") + + try: + start_time = time.time() + + # Pass function objects directly - NO manual conversion needed! ✨ + results = await client.crawl( + urls=[TEST_URLS[0]], + hooks={ + "on_page_context_created": performance_optimization_hook, + "before_goto": authentication_headers_hook, + "before_retrieve_html": lazy_loading_handler_hook, + "before_return_html": page_analytics_hook, + }, + hooks_timeout=30 + ) + + execution_time = time.time() - start_time + + print(f"\n✅ Crawl completed! (took {execution_time:.2f}s)\n") + + # Display results + if results and results.success: + result = results + print(f"📊 Results:") + print(f" • URL: {result.url}") + print(f" • Success: {result.success}") + print(f" • HTML length: {len(result.html):,} characters") + print(f" • Markdown length: {len(result.markdown):,} characters") + + # Show metadata + if result.metadata: + print(f"\n📋 Metadata:") + print(f" • Title: {result.metadata.get('title', 'N/A')[:50]}...") + + # Show links + if result.links: + internal_count = len(result.links.get('internal', [])) + external_count = len(result.links.get('external', [])) + print(f"\n🔗 Links Found:") + print(f" • Internal: {internal_count}") + print(f" • External: {external_count}") + else: + print(f"⚠️ Crawl completed but no successful results") + if results: + print(f" Error: {results.error_message}") + + except Exception as e: + print(f"❌ Error: {str(e)}") + import traceback + traceback.print_exc() + + print("\n🌟 Why Docker Client is RECOMMENDED:") + print(" ✓ Automatic function-to-string conversion") + print(" ✓ No manual hooks_to_string() calls needed") + print(" ✓ Cleaner, more Pythonic code") + print(" ✓ Full type hints and IDE support") + print(" ✓ Built-in error handling") + print(" ✓ Async/await support") + + print("\n" + "─" * 70) + print("✓ Docker Client auto-conversion example complete\n") + + +# ============================================================================ +# APPROACH 4: Authentication Example +# ============================================================================ + +def example_4_authentication_flow(): + """ + Demonstrate authentication flow with multiple hooks + """ + print_section( + "EXAMPLE 4: Authentication Flow", + "Using hooks for authentication with cookies and headers" + ) + hooks_code = { "on_page_context_created": """ async def hook(page, context, **kwargs): print("[HOOK] Setting up authentication context") - + # Add authentication cookies await context.add_cookies([ { @@ -241,50 +457,42 @@ async def hook(page, context, **kwargs): "secure": True } ]) - - # Set localStorage items (for SPA authentication) - await page.evaluate(''' - localStorage.setItem('user_id', '12345'); - localStorage.setItem('auth_time', new Date().toISOString()); - ''') - + return page """, - + "before_goto": """ async def hook(page, context, url, **kwargs): print(f"[HOOK] Adding auth headers for {url}") - + # Add Authorization header import base64 credentials = base64.b64encode(b"user:passwd").decode('ascii') - + await page.set_extra_http_headers({ 'Authorization': f'Basic {credentials}', 'X-API-Key': 'test-api-key-123' }) - + return page """ } - + payload = { - "urls": [ - "https://httpbin.org/basic-auth/user/passwd" - ], + "urls": ["https://httpbin.org/basic-auth/user/passwd"], "hooks": { "code": hooks_code, "timeout": 15 } } - + print("\nTesting authentication with httpbin endpoints...") - response = requests.post(f"{API_BASE_URL}/crawl", json=payload) - + response = requests.post(f"{DOCKER_URL}/crawl", json=payload) + if response.status_code == 200: data = response.json() print("✅ Authentication test completed") - + if 'results' in data: for i, result in enumerate(data['results']): print(f"\n URL {i+1}: {result['url']}") @@ -300,214 +508,120 @@ async def hook(page, context, url, **kwargs): else: print(f"❌ Error: {response.status_code}") + print("\n" + "─" * 70) + print("✓ Authentication example complete\n") -def test_performance_optimization_hooks(): - """Test hooks for performance optimization""" + +# ============================================================================ +# MAIN EXECUTION +# ============================================================================ + +async def main(): + """ + Run all example demonstrations + """ print("\n" + "=" * 70) - print("Testing: Performance Optimization Hooks") + print(" 🚀 Crawl4AI - Docker Hooks System Examples") print("=" * 70) - - hooks_code = { - "on_page_context_created": """ -async def hook(page, context, **kwargs): - print("[HOOK] Optimizing page for performance") - - # Block resource-heavy content - await context.route("**/*.{png,jpg,jpeg,gif,webp,svg,ico}", lambda route: route.abort()) - await context.route("**/*.{woff,woff2,ttf,otf}", lambda route: route.abort()) - await context.route("**/*.{mp4,webm,ogg,mp3,wav}", lambda route: route.abort()) - await context.route("**/googletagmanager.com/*", lambda route: route.abort()) - await context.route("**/google-analytics.com/*", lambda route: route.abort()) - await context.route("**/doubleclick.net/*", lambda route: route.abort()) - await context.route("**/facebook.com/*", lambda route: route.abort()) - - # Disable animations and transitions - await page.add_style_tag(content=''' - *, *::before, *::after { - animation-duration: 0s !important; - animation-delay: 0s !important; - transition-duration: 0s !important; - transition-delay: 0s !important; - } - ''') - - print("[HOOK] Performance optimizations applied") - return page -""", - - "before_retrieve_html": """ -async def hook(page, context, **kwargs): - print("[HOOK] Removing unnecessary elements before extraction") - - # Remove ads, popups, and other unnecessary elements - await page.evaluate('''() => { - // Remove common ad containers - const adSelectors = [ - '.ad', '.ads', '.advertisement', '[id*="ad-"]', '[class*="ad-"]', - '.popup', '.modal', '.overlay', '.cookie-banner', '.newsletter-signup' - ]; - - adSelectors.forEach(selector => { - document.querySelectorAll(selector).forEach(el => el.remove()); - }); - - // Remove script tags to clean up HTML - document.querySelectorAll('script').forEach(el => el.remove()); - - // Remove style tags we don't need - document.querySelectorAll('style').forEach(el => el.remove()); - }''') - - return page -""" - } - - payload = { - "urls": ["https://httpbin.org/html"], - "hooks": { - "code": hooks_code, - "timeout": 10 - } - } - - print("\nTesting performance optimization hooks...") - start_time = time.time() - - response = requests.post(f"{API_BASE_URL}/crawl", json=payload) - - elapsed_time = time.time() - start_time - print(f"Request completed in {elapsed_time:.2f} seconds") - - if response.status_code == 200: - data = response.json() - print("✅ Performance optimization test completed") - - if 'results' in data and len(data['results']) > 0: - result = data['results'][0] - if result.get('html'): - print(f" HTML size: {len(result['html'])} characters") - print(" Resources blocked, ads removed, animations disabled") - else: - print(f"❌ Error: {response.status_code}") + # Check Docker service + print("\n🔍 Checking Docker service status...") + if not check_docker_service(): + print("❌ Docker service is not running!") + print("\n📋 To start the Docker service:") + print(" docker run -p 11235:11235 unclecode/crawl4ai:latest") + print("\nPlease start the service and run this example again.") + return -def test_content_extraction_hooks(): - """Test hooks for intelligent content extraction""" - print("\n" + "=" * 70) - print("Testing: Content Extraction Hooks") - print("=" * 70) - - hooks_code = { - "after_goto": """ -async def hook(page, context, url, response, **kwargs): - print(f"[HOOK] Waiting for dynamic content on {url}") - - # Wait for any lazy-loaded content - await page.wait_for_timeout(2000) - - # Trigger any "Load More" buttons - try: - load_more = await page.query_selector('[class*="load-more"], [class*="show-more"], button:has-text("Load More")') - if load_more: - await load_more.click() - await page.wait_for_timeout(1000) - print("[HOOK] Clicked 'Load More' button") - except: - pass - - return page -""", - - "before_retrieve_html": """ -async def hook(page, context, **kwargs): - print("[HOOK] Extracting structured data") - - # Extract metadata - metadata = await page.evaluate('''() => { - const getMeta = (name) => { - const element = document.querySelector(`meta[name="${name}"], meta[property="${name}"]`); - return element ? element.getAttribute('content') : null; - }; - - return { - title: document.title, - description: getMeta('description') || getMeta('og:description'), - author: getMeta('author'), - keywords: getMeta('keywords'), - ogTitle: getMeta('og:title'), - ogImage: getMeta('og:image'), - canonical: document.querySelector('link[rel="canonical"]')?.href, - jsonLd: Array.from(document.querySelectorAll('script[type="application/ld+json"]')) - .map(el => el.textContent).filter(Boolean) - }; - }''') - - print(f"[HOOK] Extracted metadata: {json.dumps(metadata, indent=2)}") - - # Infinite scroll handling - for i in range(3): - await page.evaluate("window.scrollTo(0, document.body.scrollHeight);") - await page.wait_for_timeout(1000) - print(f"[HOOK] Scroll iteration {i+1}/3") - - return page -""" - } - - payload = { - "urls": ["https://httpbin.org/html", "https://httpbin.org/json"], - "hooks": { - "code": hooks_code, - "timeout": 20 - } - } - - print("\nTesting content extraction hooks...") - response = requests.post(f"{API_BASE_URL}/crawl", json=payload) - - if response.status_code == 200: - data = response.json() - print("✅ Content extraction test completed") - - if 'hooks' in data and 'summary' in data['hooks']: - summary = data['hooks']['summary'] - print(f" Hooks executed: {summary['successful']}/{summary['total_executions']}") - - if 'results' in data: - for result in data['results']: - print(f"\n URL: {result['url']}") - print(f" Success: {result.get('success', False)}") - else: - print(f"❌ Error: {response.status_code}") + print("✅ Docker service is running!\n") - -def main(): - """Run comprehensive hook tests""" - print("🔧 Crawl4AI Docker API - Comprehensive Hooks Testing") - print("Based on docs/examples/hooks_example.py") - print("=" * 70) - - tests = [ - ("All Hooks Demo", test_all_hooks_demo), - ("Authentication Flow", test_authentication_flow), - ("Performance Optimization", test_performance_optimization_hooks), - ("Content Extraction", test_content_extraction_hooks), + # Run all examples + examples = [ + ("String-Based Hooks (REST API)", example_1_string_based_hooks, False), + ("hooks_to_string() Utility", example_2_hooks_to_string_utility, False), + ("Docker Client Auto-Conversion (Recommended)", example_3_docker_client_auto_conversion, True), + ("Authentication Flow", example_4_authentication_flow, False), ] - - for i, (name, test_func) in enumerate(tests, 1): - print(f"\n📌 Test {i}/{len(tests)}: {name}") + + for i, (name, example_func, is_async) in enumerate(examples, 1): + print(f"\n{'🔷' * 35}") + print(f"Example {i}/{len(examples)}: {name}") + print(f"{'🔷' * 35}\n") + try: - test_func() - print(f"✅ {name} completed") + if is_async: + await example_func() + else: + example_func() + + print(f"✅ Example {i} completed successfully!") + + # Pause between examples (except the last one) + if i < len(examples): + print("\n⏸️ Press Enter to continue to next example...") + input() + + except KeyboardInterrupt: + print(f"\n⏹️ Examples interrupted by user") + break except Exception as e: - print(f"❌ {name} failed: {e}") + print(f"\n❌ Example {i} failed: {str(e)}") import traceback traceback.print_exc() - + print("\nContinuing to next example...\n") + continue + + # Final summary print("\n" + "=" * 70) - print("🎉 All comprehensive hook tests completed!") + print(" 🎉 All Examples Complete!") print("=" * 70) + print("\n📊 Summary - Three Approaches to Docker Hooks:") + + print("\n✨ 1. String-Based Hooks:") + print(" • Write hooks as strings directly in JSON") + print(" • Best for: REST API, non-Python clients, simple use cases") + print(" • Cons: No IDE support, harder to debug") + + print("\n✨ 2. hooks_to_string() Utility:") + print(" • Write hooks as Python functions, convert to strings") + print(" • Best for: Python with REST API, reusable hook libraries") + print(" • Pros: IDE support, type checking, easy debugging") + + print("\n✨ 3. Docker Client (RECOMMENDED):") + print(" • Pass function objects directly, automatic conversion") + print(" • Best for: Python applications, best developer experience") + print(" • Pros: All benefits of #2 + cleaner code, no manual conversion") + + print("\n💡 Recommendation:") + print(" Use Docker Client (#3) for Python applications") + print(" Use hooks_to_string() (#2) when you need REST API flexibility") + print(" Use string-based (#1) for non-Python clients or simple scripts") + + print("\n🎯 8 Hook Points Available:") + print(" • on_browser_created, on_page_context_created") + print(" • on_user_agent_updated, before_goto, after_goto") + print(" • on_execution_started, before_retrieve_html, before_return_html") + + print("\n📚 Resources:") + print(" • Docs: https://docs.crawl4ai.com/core/docker-deployment") + print(" • GitHub: https://github.com/unclecode/crawl4ai") + print(" • Discord: https://discord.gg/jP8KfhDhyN") + + print("\n" + "=" * 70) + print(" Happy Crawling! 🕷️") + print("=" * 70 + "\n") + if __name__ == "__main__": - main() \ No newline at end of file + print("\n🎬 Starting Crawl4AI Docker Hooks Examples...") + print("Press Ctrl+C anytime to exit\n") + + try: + asyncio.run(main()) + except KeyboardInterrupt: + print("\n\n👋 Examples stopped by user. Thanks for exploring Crawl4AI!") + except Exception as e: + print(f"\n\n❌ Error: {str(e)}") + import traceback + traceback.print_exc() diff --git a/docs/md_v2/core/docker-deployment.md b/docs/md_v2/core/docker-deployment.md index ec2c69a5..6d5a9eae 100644 --- a/docs/md_v2/core/docker-deployment.md +++ b/docs/md_v2/core/docker-deployment.md @@ -27,6 +27,14 @@ - [Hook Response Information](#hook-response-information) - [Error Handling](#error-handling) - [Hooks Utility: Function-Based Approach (Python)](#hooks-utility-function-based-approach-python) +- [Job Queue & Webhook API](#job-queue-webhook-api) + - [Why Use the Job Queue API?](#why-use-the-job-queue-api) + - [Available Endpoints](#available-endpoints) + - [Webhook Configuration](#webhook-configuration) + - [Usage Examples](#usage-examples) + - [Webhook Best Practices](#webhook-best-practices) + - [Use Cases](#use-cases) + - [Troubleshooting](#troubleshooting) - [Dockerfile Parameters](#dockerfile-parameters) - [Using the API](#using-the-api) - [Playground Interface](#playground-interface) @@ -1110,6 +1118,464 @@ if __name__ == "__main__": --- +## Job Queue & Webhook API + +The Docker deployment includes a powerful asynchronous job queue system with webhook support for both crawling and LLM extraction tasks. Instead of waiting for long-running operations to complete, submit jobs and receive real-time notifications via webhooks when they finish. + +### Why Use the Job Queue API? + +**Traditional Synchronous API (`/crawl`):** +- Client waits for entire crawl to complete +- Timeout issues with long-running crawls +- Resource blocking during execution +- Constant polling required for status updates + +**Asynchronous Job Queue API (`/crawl/job`, `/llm/job`):** +- ✅ Submit job and continue immediately +- ✅ No timeout concerns for long operations +- ✅ Real-time webhook notifications on completion +- ✅ Better resource utilization +- ✅ Perfect for batch processing +- ✅ Ideal for microservice architectures + +### Available Endpoints + +#### 1. Crawl Job Endpoint + +``` +POST /crawl/job +``` + +Submit an asynchronous crawl job with optional webhook notification. + +**Request Body:** +```json +{ + "urls": ["https://example.com"], + "cache_mode": "bypass", + "extraction_strategy": { + "type": "JsonCssExtractionStrategy", + "schema": { + "title": "h1", + "content": ".article-body" + } + }, + "webhook_config": { + "webhook_url": "https://your-app.com/webhook/crawl-complete", + "webhook_data_in_payload": true, + "webhook_headers": { + "X-Webhook-Secret": "your-secret-token", + "X-Custom-Header": "value" + } + } +} +``` + +**Response:** +```json +{ + "task_id": "crawl_1698765432", + "message": "Crawl job submitted" +} +``` + +#### 2. LLM Extraction Job Endpoint + +``` +POST /llm/job +``` + +Submit an asynchronous LLM extraction job with optional webhook notification. + +**Request Body:** +```json +{ + "url": "https://example.com/article", + "q": "Extract the article title, author, publication date, and main points", + "provider": "openai/gpt-4o-mini", + "schema": "{\"title\": \"string\", \"author\": \"string\", \"date\": \"string\", \"points\": [\"string\"]}", + "cache": false, + "webhook_config": { + "webhook_url": "https://your-app.com/webhook/llm-complete", + "webhook_data_in_payload": true, + "webhook_headers": { + "X-Webhook-Secret": "your-secret-token" + } + } +} +``` + +**Response:** +```json +{ + "task_id": "llm_1698765432", + "message": "LLM job submitted" +} +``` + +#### 3. Job Status Endpoint + +``` +GET /job/{task_id} +``` + +Check the status and retrieve results of a submitted job. + +**Response (In Progress):** +```json +{ + "task_id": "crawl_1698765432", + "status": "processing", + "message": "Job is being processed" +} +``` + +**Response (Completed):** +```json +{ + "task_id": "crawl_1698765432", + "status": "completed", + "result": { + "markdown": "# Page Title\n\nContent...", + "extracted_content": {...}, + "links": {...} + } +} +``` + +### Webhook Configuration + +Webhooks provide real-time notifications when your jobs complete, eliminating the need for constant polling. + +#### Webhook Config Parameters + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `webhook_url` | string | Yes | Your HTTP(S) endpoint to receive notifications | +| `webhook_data_in_payload` | boolean | No | Include full result data in webhook payload (default: false) | +| `webhook_headers` | object | No | Custom headers for authentication/identification | + +#### Webhook Payload Format + +**Success Notification (Crawl Job):** +```json +{ + "task_id": "crawl_1698765432", + "task_type": "crawl", + "status": "completed", + "timestamp": "2025-10-22T12:30:00.000000+00:00", + "urls": ["https://example.com"], + "data": { + "markdown": "# Page content...", + "extracted_content": {...}, + "links": {...} + } +} +``` + +**Success Notification (LLM Job):** +```json +{ + "task_id": "llm_1698765432", + "task_type": "llm_extraction", + "status": "completed", + "timestamp": "2025-10-22T12:30:00.000000+00:00", + "urls": ["https://example.com/article"], + "data": { + "extracted_content": { + "title": "Understanding Web Scraping", + "author": "John Doe", + "date": "2025-10-22", + "points": ["Point 1", "Point 2"] + } + } +} +``` + +**Failure Notification:** +```json +{ + "task_id": "crawl_1698765432", + "task_type": "crawl", + "status": "failed", + "timestamp": "2025-10-22T12:30:00.000000+00:00", + "urls": ["https://example.com"], + "error": "Connection timeout after 30 seconds" +} +``` + +#### Webhook Delivery & Retry + +- **Delivery Method:** HTTP POST to your `webhook_url` +- **Content-Type:** `application/json` +- **Retry Policy:** Exponential backoff with 5 attempts + - Attempt 1: Immediate + - Attempt 2: 1 second delay + - Attempt 3: 2 seconds delay + - Attempt 4: 4 seconds delay + - Attempt 5: 8 seconds delay +- **Success Status Codes:** 200-299 +- **Custom Headers:** Your `webhook_headers` are included in every request + +### Usage Examples + +#### Example 1: Python with Webhook Handler (Flask) + +```python +from flask import Flask, request, jsonify +import requests + +app = Flask(__name__) + +# Webhook handler +@app.route('/webhook/crawl-complete', methods=['POST']) +def handle_crawl_webhook(): + payload = request.json + + if payload['status'] == 'completed': + print(f"✅ Job {payload['task_id']} completed!") + print(f"Task type: {payload['task_type']}") + + # Access the crawl results + if 'data' in payload: + markdown = payload['data'].get('markdown', '') + extracted = payload['data'].get('extracted_content', {}) + print(f"Extracted {len(markdown)} characters") + print(f"Structured data: {extracted}") + else: + print(f"❌ Job {payload['task_id']} failed: {payload.get('error')}") + + return jsonify({"status": "received"}), 200 + +# Submit a crawl job with webhook +def submit_crawl_job(): + response = requests.post( + "http://localhost:11235/crawl/job", + json={ + "urls": ["https://example.com"], + "extraction_strategy": { + "type": "JsonCssExtractionStrategy", + "schema": { + "name": "Example Schema", + "baseSelector": "body", + "fields": [ + {"name": "title", "selector": "h1", "type": "text"}, + {"name": "description", "selector": "meta[name='description']", "type": "attribute", "attribute": "content"} + ] + } + }, + "webhook_config": { + "webhook_url": "https://your-app.com/webhook/crawl-complete", + "webhook_data_in_payload": True, + "webhook_headers": { + "X-Webhook-Secret": "your-secret-token" + } + } + } + ) + + task_id = response.json()['task_id'] + print(f"Job submitted: {task_id}") + return task_id + +if __name__ == '__main__': + app.run(port=5000) +``` + +#### Example 2: LLM Extraction with Webhooks + +```python +import requests + +def submit_llm_job_with_webhook(): + response = requests.post( + "http://localhost:11235/llm/job", + json={ + "url": "https://example.com/article", + "q": "Extract the article title, author, and main points", + "provider": "openai/gpt-4o-mini", + "webhook_config": { + "webhook_url": "https://your-app.com/webhook/llm-complete", + "webhook_data_in_payload": True, + "webhook_headers": { + "X-Webhook-Secret": "your-secret-token" + } + } + } + ) + + task_id = response.json()['task_id'] + print(f"LLM job submitted: {task_id}") + return task_id + +# Webhook handler for LLM jobs +@app.route('/webhook/llm-complete', methods=['POST']) +def handle_llm_webhook(): + payload = request.json + + if payload['status'] == 'completed': + extracted = payload['data']['extracted_content'] + print(f"✅ LLM extraction completed!") + print(f"Results: {extracted}") + else: + print(f"❌ LLM extraction failed: {payload.get('error')}") + + return jsonify({"status": "received"}), 200 +``` + +#### Example 3: Without Webhooks (Polling) + +If you don't use webhooks, you can poll for results: + +```python +import requests +import time + +# Submit job +response = requests.post( + "http://localhost:11235/crawl/job", + json={"urls": ["https://example.com"]} +) +task_id = response.json()['task_id'] + +# Poll for results +while True: + result = requests.get(f"http://localhost:11235/job/{task_id}") + data = result.json() + + if data['status'] == 'completed': + print("Job completed!") + print(data['result']) + break + elif data['status'] == 'failed': + print(f"Job failed: {data.get('error')}") + break + + print("Still processing...") + time.sleep(2) +``` + +#### Example 4: Global Webhook Configuration + +Set a default webhook URL in your `config.yml` to avoid repeating it in every request: + +```yaml +# config.yml +api: + crawler: + # ... other settings ... + webhook: + default_url: "https://your-app.com/webhook/default" + default_headers: + X-Webhook-Secret: "your-secret-token" +``` + +Then submit jobs without webhook config: + +```python +# Uses the global webhook configuration +response = requests.post( + "http://localhost:11235/crawl/job", + json={"urls": ["https://example.com"]} +) +``` + +### Webhook Best Practices + +1. **Authentication:** Always use custom headers for webhook authentication + ```json + "webhook_headers": { + "X-Webhook-Secret": "your-secret-token" + } + ``` + +2. **Idempotency:** Design your webhook handler to be idempotent (safe to receive duplicate notifications) + +3. **Fast Response:** Return HTTP 200 quickly; process data asynchronously if needed + ```python + @app.route('/webhook', methods=['POST']) + def webhook(): + payload = request.json + # Queue for background processing + queue.enqueue(process_webhook, payload) + return jsonify({"status": "received"}), 200 + ``` + +4. **Error Handling:** Handle both success and failure notifications + ```python + if payload['status'] == 'completed': + # Process success + elif payload['status'] == 'failed': + # Log error, retry, or alert + ``` + +5. **Validation:** Verify webhook authenticity using custom headers + ```python + secret = request.headers.get('X-Webhook-Secret') + if secret != os.environ['EXPECTED_SECRET']: + return jsonify({"error": "Unauthorized"}), 401 + ``` + +6. **Logging:** Log webhook deliveries for debugging + ```python + logger.info(f"Webhook received: {payload['task_id']} - {payload['status']}") + ``` + +### Use Cases + +**1. Batch Processing** +Submit hundreds of URLs and get notified as each completes: +```python +urls = ["https://site1.com", "https://site2.com", ...] +for url in urls: + submit_crawl_job(url, webhook_url="https://app.com/webhook") +``` + +**2. Microservice Integration** +Integrate with event-driven architectures: +```python +# Service A submits job +task_id = submit_crawl_job(url) + +# Service B receives webhook and triggers next step +@app.route('/webhook') +def webhook(): + process_result(request.json) + trigger_next_service() + return "OK", 200 +``` + +**3. Long-Running Extractions** +Handle complex LLM extractions without timeouts: +```python +submit_llm_job( + url="https://long-article.com", + q="Comprehensive summary with key points and analysis", + webhook_url="https://app.com/webhook/llm" +) +``` + +### Troubleshooting + +**Webhook not receiving notifications?** +- Check your webhook URL is publicly accessible +- Verify firewall/security group settings +- Use webhook testing tools like webhook.site for debugging +- Check server logs for delivery attempts +- Ensure your handler returns 200-299 status code + +**Job stuck in processing?** +- Check Redis connection: `docker logs | grep redis` +- Verify worker processes: `docker exec ps aux | grep worker` +- Check server logs: `docker logs ` + +**Need to cancel a job?** +Jobs are processed asynchronously. If you need to cancel: +- Delete the task from Redis (requires Redis CLI access) +- Or implement a cancellation endpoint in your webhook handler + +--- + ## Dockerfile Parameters You can customize the image build process using build arguments (`--build-arg`). These are typically used via `docker buildx build` or within the `docker-compose.yml` file.