Release v0.7.3: Merge release branch

- Merge release/v0.7.3 into main
- Version: 0.7.3
- Ready for tag and publication

.github/FUNDING.yml

@@ -0,0 +1,7 @@
+# These are supported funding model platforms
+
+# GitHub Sponsors
+github: unclecode
+
+# Custom links for enterprise inquiries (uncomment when ready)
+# custom: ["https://crawl4ai.com/enterprise"]

README-first.md

@@ -0,0 +1,809 @@
+# 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.
+
+<div align="center">
+
+<a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
+
+[![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
+[![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
+
+[![PyPI version](https://badge.fury.io/py/crawl4ai.svg)](https://badge.fury.io/py/crawl4ai)
+[![Python Version](https://img.shields.io/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/)
+[![Downloads](https://static.pepy.tech/badge/crawl4ai/month)](https://pepy.tech/project/crawl4ai)
+[![GitHub Sponsors](https://img.shields.io/github/sponsors/unclecode?style=flat&logo=GitHub-Sponsors&label=Sponsors&color=pink)](https://github.com/sponsors/unclecode)
+
+<p align="center">
+    <a href="https://x.com/crawl4ai">
+      <img src="https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white" alt="Follow on X" />
+    </a>
+    <a href="https://www.linkedin.com/company/crawl4ai">
+      <img src="https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" alt="Follow on LinkedIn" />
+    </a>
+    <a href="https://discord.gg/jP8KfhDhyN">
+      <img src="https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join our Discord" />
+    </a>
+  </p>
+</div>
+
+Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.  
+
+[✨ Check out latest update v0.7.0](#-recent-updates)
+
+🎉 **Version 0.7.0 is now available!** The Adaptive Intelligence Update introduces groundbreaking features: Adaptive Crawling that learns website patterns, Virtual Scroll support for infinite pages, intelligent Link Preview with 3-layer scoring, Async URL Seeder for massive discovery, and significant performance improvements. [Read the release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.0.md)
+
+<details>
+<summary>🤓 <strong>My Personal Story</strong></summary>
+
+My journey with computers started in childhood when my dad, a computer scientist, introduced me to an Amstrad computer. Those early days sparked a fascination with technology, leading me to pursue computer science and specialize in NLP during my postgraduate studies. It was during this time that I first delved into web crawling, building tools to help researchers organize papers and extract information from publications a challenging yet rewarding experience that honed my skills in data extraction.
+
+Fast forward to 2023, I was working on a tool for a project and needed a crawler to convert a webpage into markdown. While exploring solutions, I found one that claimed to be open-source but required creating an account and generating an API token. Worse, it turned out to be a SaaS model charging $16, and its quality didn’t meet my standards. Frustrated, I realized this was a deeper problem. That frustration turned into turbo anger mode, and I decided to build my own solution. In just a few days, I created Crawl4AI. To my surprise, it went viral, earning thousands of GitHub stars and resonating with a global community.
+
+I made Crawl4AI open-source for two reasons. First, it’s my way of giving back to the open-source community that has supported me throughout my career. Second, I believe data should be accessible to everyone, not locked behind paywalls or monopolized by a few. Open access to data lays the foundation for the democratization of AI, a vision where individuals can train their own models and take ownership of their information. This library is the first step in a larger journey to create the best open-source data extraction and generation tool the world has ever seen, built collaboratively by a passionate community.
+
+Thank you to everyone who has supported this project, used it, and shared feedback. Your encouragement motivates me to dream even bigger. Join us, file issues, submit PRs, or spread the word. Together, we can build a tool that truly empowers people to access their own data and reshape the future of AI.
+</details>
+
+## 🧐 Why Crawl4AI?
+
+1. **Built for LLMs**: Creates smart, concise Markdown optimized for RAG and fine-tuning applications.  
+2. **Lightning Fast**: Delivers results faster with real-time, cost-efficient performance.  
+3. **Flexible Browser Control**: Offers session management, proxies, and custom hooks for seamless data access.  
+4. **Heuristic Intelligence**: Uses advanced algorithms for efficient extraction, reducing reliance on costly models.  
+5. **Open Source & Deployable**: Fully open-source with no API keys—ready for Docker and cloud integration.  
+6. **Thriving Community**: Actively maintained by a vibrant community and the #1 trending GitHub repository.
+
+## 🚀 Quick Start 
+
+1. Install Crawl4AI:
+```bash
+# Install the package
+pip install -U crawl4ai
+
+# For pre release versions
+pip install crawl4ai --pre
+
+# Run post-installation setup
+crawl4ai-setup
+
+# Verify your installation
+crawl4ai-doctor
+```
+
+If you encounter any browser-related issues, you can install them manually:
+```bash
+python -m playwright install --with-deps chromium
+```
+
+2. Run a simple web crawl with Python:
+```python
+import asyncio
+from crawl4ai import *
+
+async def main():
+    async with AsyncWebCrawler() as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+        )
+        print(result.markdown)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+3. Or use the new command-line interface:
+```bash
+# Basic crawl with markdown output
+crwl https://www.nbcnews.com/business -o markdown
+
+# Deep crawl with BFS strategy, max 10 pages
+crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
+
+# Use LLM extraction with a specific question
+crwl https://www.example.com/products -q "Extract all product prices"
+```
+
+## ✨ Features 
+
+<details>
+<summary>📝 <strong>Markdown Generation</strong></summary>
+
+- 🧹 **Clean Markdown**: Generates clean, structured Markdown with accurate formatting.
+- 🎯 **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
+- 🔗 **Citations and References**: Converts page links into a numbered reference list with clean citations.
+- 🛠️ **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs.
+- 📚 **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content. 
+</details>
+
+<details>
+<summary>📊 <strong>Structured Data Extraction</strong></summary>
+
+- 🤖 **LLM-Driven Extraction**: Supports all LLMs (open-source and proprietary) for structured data extraction.
+- 🧱 **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
+- 🌌 **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction.
+- 🔎 **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors.
+- 🔧 **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns.
+
+</details>
+
+<details>
+<summary>🌐 <strong>Browser Integration</strong></summary>
+
+- 🖥️ **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection.
+- 🔄 **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
+- 👤 **Browser Profiler**: Create and manage persistent profiles with saved authentication states, cookies, and settings.
+- 🔒 **Session Management**: Preserve browser states and reuse them for multi-step crawling.
+- 🧩 **Proxy Support**: Seamlessly connect to proxies with authentication for secure access.
+- ⚙️ **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups.
+- 🌍 **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit.
+- 📐 **Dynamic Viewport Adjustment**: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.
+
+</details>
+
+<details>
+<summary>🔎 <strong>Crawling & Scraping</strong></summary>
+
+- 🖼️ **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`.
+- 🚀 **Dynamic Crawling**: Execute JS and wait for async or sync for dynamic content extraction.
+- 📸 **Screenshots**: Capture page screenshots during crawling for debugging or analysis.
+- 📂 **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file://`).
+- 🔗 **Comprehensive Link Extraction**: Extracts internal, external links, and embedded iframe content.
+- 🛠️ **Customizable Hooks**: Define hooks at every step to customize crawling behavior.
+- 💾 **Caching**: Cache data for improved speed and to avoid redundant fetches.
+- 📄 **Metadata Extraction**: Retrieve structured metadata from web pages.
+- 📡 **IFrame Content Extraction**: Seamless extraction from embedded iframe content.
+- 🕵️ **Lazy Load Handling**: Waits for images to fully load, ensuring no content is missed due to lazy loading.
+- 🔄 **Full-Page Scanning**: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.
+
+</details>
+
+<details>
+<summary>🚀 <strong>Deployment</strong></summary>
+
+- 🐳 **Dockerized Setup**: Optimized Docker image with FastAPI server for easy deployment.
+- 🔑 **Secure Authentication**: Built-in JWT token authentication for API security.
+- 🔄 **API Gateway**: One-click deployment with secure token authentication for API-based workflows.
+- 🌐 **Scalable Architecture**: Designed for mass-scale production and optimized server performance.
+- ☁️ **Cloud Deployment**: Ready-to-deploy configurations for major cloud platforms.
+
+</details>
+
+<details>
+<summary>🎯 <strong>Additional Features</strong></summary>
+
+- 🕶️ **Stealth Mode**: Avoid bot detection by mimicking real users.
+- 🏷️ **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata.
+- 🔗 **Link Analysis**: Extract and analyze all links for detailed data exploration.
+- 🛡️ **Error Handling**: Robust error management for seamless execution.
+- 🔐 **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests.
+- 📖 **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage.
+- 🙌 **Community Recognition**: Acknowledges contributors and pull requests for transparency.
+
+</details>
+
+## Try it Now!
+
+✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)
+
+✨ Visit our [Documentation Website](https://docs.crawl4ai.com/)
+
+## Installation 🛠️
+
+Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
+
+<details>
+<summary>🐍 <strong>Using pip</strong></summary>
+
+Choose the installation option that best fits your needs:
+
+### Basic Installation
+
+For basic web crawling and scraping tasks:
+
+```bash
+pip install crawl4ai
+crawl4ai-setup # Setup the browser
+```
+
+By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.
+
+👉 **Note**: When you install Crawl4AI, the `crawl4ai-setup` should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
+
+1. Through the command line:
+
+   ```bash
+   playwright install
+   ```
+
+2. If the above doesn't work, try this more specific command:
+
+   ```bash
+   python -m playwright install chromium
+   ```
+
+This second method has proven to be more reliable in some cases.
+
+---
+
+### Installation with Synchronous Version
+
+The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:
+
+```bash
+pip install crawl4ai[sync]
+```
+
+---
+
+### Development Installation
+
+For contributors who plan to modify the source code:
+
+```bash
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+pip install -e .                    # Basic installation in editable mode
+```
+
+Install optional features:
+
+```bash
+pip install -e ".[torch]"           # With PyTorch features
+pip install -e ".[transformer]"     # With Transformer features
+pip install -e ".[cosine]"          # With cosine similarity features
+pip install -e ".[sync]"            # With synchronous crawling (Selenium)
+pip install -e ".[all]"             # Install all optional features
+```
+
+</details>
+
+<details>
+<summary>🐳 <strong>Docker Deployment</strong></summary>
+
+> 🚀 **Now Available!** Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.
+
+### New Docker Features
+
+The new Docker implementation includes:
+- **Browser pooling** with page pre-warming for faster response times
+- **Interactive playground** to test and generate request code
+- **MCP integration** for direct connection to AI tools like Claude Code
+- **Comprehensive API endpoints** including HTML extraction, screenshots, PDF generation, and JavaScript execution
+- **Multi-architecture support** with automatic detection (AMD64/ARM64)
+- **Optimized resources** with improved memory management
+
+### Getting Started
+
+```bash
+# Pull and run the latest release candidate
+docker pull unclecode/crawl4ai:0.7.0
+docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:0.7.0
+
+# Visit the playground at http://localhost:11235/playground
+```
+
+For complete documentation, see our [Docker Deployment Guide](https://docs.crawl4ai.com/core/docker-deployment/).
+
+</details>
+
+---
+
+### Quick Test
+
+Run a quick test (works for both Docker options):
+
+```python
+import requests
+
+# Submit a crawl job
+response = requests.post(
+    "http://localhost:11235/crawl",
+    json={"urls": ["https://example.com"], "priority": 10}
+)
+if response.status_code == 200:
+    print("Crawl job submitted successfully.")
+    
+if "results" in response.json():
+    results = response.json()["results"]
+    print("Crawl job completed. Results:")
+    for result in results:
+        print(result)
+else:
+    task_id = response.json()["task_id"]
+    print(f"Crawl job submitted. Task ID:: {task_id}")
+    result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+For more examples, see our [Docker Examples](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_example.py). For advanced configuration, environment variables, and usage examples, see our [Docker Deployment Guide](https://docs.crawl4ai.com/basic/docker-deployment/).
+
+</details>
+
+
+## 🔬 Advanced Usage Examples 🔬
+
+You can check the project structure in the directory [docs/examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.
+
+<details>
+<summary>📝 <strong>Heuristic Markdown Generation with Clean and Fit Markdown</strong></summary>
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
+from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
+from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
+
+async def main():
+    browser_config = BrowserConfig(
+        headless=True,  
+        verbose=True,
+    )
+    run_config = CrawlerRunConfig(
+        cache_mode=CacheMode.ENABLED,
+        markdown_generator=DefaultMarkdownGenerator(
+            content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
+        ),
+        # markdown_generator=DefaultMarkdownGenerator(
+        #     content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)
+        # ),
+    )
+    
+    async with AsyncWebCrawler(config=browser_config) as crawler:
+        result = await crawler.arun(
+            url="https://docs.micronaut.io/4.7.6/guide/",
+            config=run_config
+        )
+        print(len(result.markdown.raw_markdown))
+        print(len(result.markdown.fit_markdown))
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+</details>
+
+<details>
+<summary>🖥️ <strong>Executing JavaScript & Extract Structured Data without LLMs</strong></summary>
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
+from crawl4ai import JsonCssExtractionStrategy
+import json
+
+async def main():
+    schema = {
+    "name": "KidoCode Courses",
+    "baseSelector": "section.charge-methodology .w-tab-content > div",
+    "fields": [
+        {
+            "name": "section_title",
+            "selector": "h3.heading-50",
+            "type": "text",
+        },
+        {
+            "name": "section_description",
+            "selector": ".charge-content",
+            "type": "text",
+        },
+        {
+            "name": "course_name",
+            "selector": ".text-block-93",
+            "type": "text",
+        },
+        {
+            "name": "course_description",
+            "selector": ".course-content-text",
+            "type": "text",
+        },
+        {
+            "name": "course_icon",
+            "selector": ".image-92",
+            "type": "attribute",
+            "attribute": "src"
+        }
+    }
+}
+
+    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+    browser_config = BrowserConfig(
+        headless=False,
+        verbose=True
+    )
+    run_config = CrawlerRunConfig(
+        extraction_strategy=extraction_strategy,
+        js_code=["""(async () => {const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();"""],
+        cache_mode=CacheMode.BYPASS
+    )
+        
+    async with AsyncWebCrawler(config=browser_config) as crawler:
+        
+        result = await crawler.arun(
+            url="https://www.kidocode.com/degrees/technology",
+            config=run_config
+        )
+
+        companies = json.loads(result.extracted_content)
+        print(f"Successfully extracted {len(companies)} companies")
+        print(json.dumps(companies[0], indent=2))
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+</details>
+
+<details>
+<summary>📚 <strong>Extracting Structured Data with LLMs</strong></summary>
+
+```python
+import os
+import asyncio
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
+from crawl4ai import LLMExtractionStrategy
+from pydantic import BaseModel, Field
+
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
+
+async def main():
+    browser_config = BrowserConfig(verbose=True)
+    run_config = CrawlerRunConfig(
+        word_count_threshold=1,
+        extraction_strategy=LLMExtractionStrategy(
+            # Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
+            # provider="ollama/qwen2", api_token="no-token", 
+            llm_config = LLMConfig(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY')), 
+            schema=OpenAIModelFee.schema(),
+            extraction_type="schema",
+            instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
+            Do not miss any models in the entire content. One extracted model JSON format should look like this: 
+            {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
+        ),            
+        cache_mode=CacheMode.BYPASS,
+    )
+    
+    async with AsyncWebCrawler(config=browser_config) as crawler:
+        result = await crawler.arun(
+            url='https://openai.com/api/pricing/',
+            config=run_config
+        )
+        print(result.extracted_content)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+</details>
+
+<details>
+<summary>🤖 <strong>Using Your own Browser with Custom User Profile</strong></summary>
+
+```python
+import os, sys
+from pathlib import Path
+import asyncio, time
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
+
+async def test_news_crawl():
+    # Create a persistent user data directory
+    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
+    os.makedirs(user_data_dir, exist_ok=True)
+
+    browser_config = BrowserConfig(
+        verbose=True,
+        headless=True,
+        user_data_dir=user_data_dir,
+        use_persistent_context=True,
+    )
+    run_config = CrawlerRunConfig(
+        cache_mode=CacheMode.BYPASS
+    )
+    
+    async with AsyncWebCrawler(config=browser_config) as crawler:
+        url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
+        
+        result = await crawler.arun(
+            url,
+            config=run_config,
+            magic=True,
+        )
+        
+        print(f"Successfully crawled {url}")
+        print(f"Content length: {len(result.markdown)}")
+```
+
+</details>
+
+## ✨ Recent Updates
+
+### Version 0.7.0 Release Highlights - The Adaptive Intelligence Update
+
+- **🧠 Adaptive Crawling**: Your crawler now learns and adapts to website patterns automatically:
+  ```python
+  config = AdaptiveConfig(
+      confidence_threshold=0.7, # Min confidence to stop crawling
+      max_depth=5, # Maximum crawl depth
+      max_pages=20, # Maximum number of pages to crawl
+      strategy="statistical"
+  )
+  
+  async with AsyncWebCrawler() as crawler:
+      adaptive_crawler = AdaptiveCrawler(crawler, config)
+      state = await adaptive_crawler.digest(
+          start_url="https://news.example.com",
+          query="latest news content"
+      )
+  # Crawler learns patterns and improves extraction over time
+  ```
+
+- **🌊 Virtual Scroll Support**: Complete content extraction from infinite scroll pages:
+  ```python
+  scroll_config = VirtualScrollConfig(
+      container_selector="[data-testid='feed']",
+      scroll_count=20,
+      scroll_by="container_height",
+      wait_after_scroll=1.0
+  )
+  
+  result = await crawler.arun(url, config=CrawlerRunConfig(
+      virtual_scroll_config=scroll_config
+  ))
+  ```
+
+- **🔗 Intelligent Link Analysis**: 3-layer scoring system for smart link prioritization:
+  ```python
+  link_config = LinkPreviewConfig(
+      query="machine learning tutorials",
+      score_threshold=0.3,
+      concurrent_requests=10
+  )
+  
+  result = await crawler.arun(url, config=CrawlerRunConfig(
+      link_preview_config=link_config,
+      score_links=True
+  ))
+  # Links ranked by relevance and quality
+  ```
+
+- **🎣 Async URL Seeder**: Discover thousands of URLs in seconds:
+  ```python
+  seeder = AsyncUrlSeeder(SeedingConfig(
+      source="sitemap+cc",
+      pattern="*/blog/*",
+      query="python tutorials",
+      score_threshold=0.4
+  ))
+  
+  urls = await seeder.discover("https://example.com")
+  ```
+
+- **⚡ Performance Boost**: Up to 3x faster with optimized resource handling and memory efficiency
+
+Read the full details in our [0.7.0 Release Notes](https://docs.crawl4ai.com/blog/release-v0.7.0) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md).
+
+## Version Numbering in Crawl4AI
+
+Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
+
+### Version Numbers Explained
+
+Our version numbers follow this pattern: `MAJOR.MINOR.PATCH` (e.g., 0.4.3)
+
+#### Pre-release Versions
+We use different suffixes to indicate development stages:
+
+- `dev` (0.4.3dev1): Development versions, unstable
+- `a` (0.4.3a1): Alpha releases, experimental features
+- `b` (0.4.3b1): Beta releases, feature complete but needs testing
+- `rc` (0.4.3): Release candidates, potential final version
+
+#### Installation
+- Regular installation (stable version):
+  ```bash
+  pip install -U crawl4ai
+  ```
+
+- Install pre-release versions:
+  ```bash
+  pip install crawl4ai --pre
+  ```
+
+- Install specific version:
+  ```bash
+  pip install crawl4ai==0.4.3b1
+  ```
+
+#### Why Pre-releases?
+We use pre-releases to:
+- Test new features in real-world scenarios
+- Gather feedback before final releases
+- Ensure stability for production users
+- Allow early adopters to try new features
+
+For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the `--pre` flag.
+
+## 📖 Documentation & Roadmap 
+
+> 🚨 **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
+
+For current documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://docs.crawl4ai.com/).
+
+To check our development plans and upcoming features, visit our [Roadmap](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md).
+
+<details>
+<summary>📈 <strong>Development TODOs</strong></summary>
+
+- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
+- [ ] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
+- [ ] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
+- [ ] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
+- [ ] 4. Automated Schema Generator: Convert natural language to extraction schemas
+- [ ] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
+- [ ] 6. Web Embedding Index: Semantic search infrastructure for crawled content
+- [ ] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
+- [ ] 8. Performance Monitor: Real-time insights into crawler operations
+- [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers
+- [ ] 10. Sponsorship Program: Structured support system with tiered benefits
+- [ ] 11. Educational Content: "How to Crawl" video series and interactive tutorials
+
+</details>
+
+## 🤝 Contributing 
+
+We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md) for more information.
+
+I'll help modify the license section with badges. For the halftone effect, here's a version with it:
+
+Here's the updated license section:
+
+## 📄 License & Attribution
+
+This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.
+
+### Attribution Requirements
+When using Crawl4AI, you must include one of the following attribution methods:
+
+#### 1. Badge Attribution (Recommended)
+Add one of these badges to your README, documentation, or website:
+
+| Theme | Badge |
+|-------|-------|
+| **Disco Theme (Animated)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/></a> |
+| **Night Theme (Dark with Neon)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/></a> |
+| **Dark Theme (Classic)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/></a> |
+| **Light Theme (Classic)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/></a> |
+ 
+
+HTML code for adding the badges:
+```html
+<!-- Disco Theme (Animated) -->
+<a href="https://github.com/unclecode/crawl4ai">
+  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/>
+</a>
+
+<!-- Night Theme (Dark with Neon) -->
+<a href="https://github.com/unclecode/crawl4ai">
+  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/>
+</a>
+
+<!-- Dark Theme (Classic) -->
+<a href="https://github.com/unclecode/crawl4ai">
+  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/>
+</a>
+
+<!-- Light Theme (Classic) -->
+<a href="https://github.com/unclecode/crawl4ai">
+  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/>
+</a>
+
+<!-- Simple Shield Badge -->
+<a href="https://github.com/unclecode/crawl4ai">
+  <img src="https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square" alt="Powered by Crawl4AI"/>
+</a>
+```
+
+#### 2. Text Attribution
+Add this line to your documentation:
+```
+This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
+```
+
+## 📚 Citation
+
+If you use Crawl4AI in your research or project, please cite:
+
+```bibtex
+@software{crawl4ai2024,
+  author = {UncleCode},
+  title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},
+  year = {2024},
+  publisher = {GitHub},
+  journal = {GitHub Repository},
+  howpublished = {\url{https://github.com/unclecode/crawl4ai}},
+  commit = {Please use the commit hash you're working with}
+}
+```
+
+Text citation format:
+```
+UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software]. 
+GitHub. https://github.com/unclecode/crawl4ai
+```
+
+## 📧 Contact 
+
+For questions, suggestions, or feedback, feel free to reach out:
+
+- GitHub: [unclecode](https://github.com/unclecode)
+- Twitter: [@unclecode](https://twitter.com/unclecode)
+- Website: [crawl4ai.com](https://crawl4ai.com)
+
+Happy Crawling! 🕸️🚀
+
+## 💖 Support Crawl4AI
+
+> 🎉 **Sponsorship Program Just Launched!** Be among the first 50 **Founding Sponsors** and get permanent recognition in our Hall of Fame!
+
+Crawl4AI is the #1 trending open-source web crawler with 51K+ stars. Your support ensures we stay independent, innovative, and free forever.
+
+<div align="center">
+
+[![Become a Sponsor](https://img.shields.io/badge/Become%20a%20Sponsor-pink?style=for-the-badge&logo=github-sponsors&logoColor=white)](https://github.com/sponsors/unclecode)
+[![Current Sponsors](https://img.shields.io/github/sponsors/unclecode?style=for-the-badge&logo=github&label=Current%20Sponsors&color=green)](https://github.com/sponsors/unclecode)
+
+</div>
+
+### 🤝 Sponsorship Tiers
+
+- **🌱 Believer ($5/mo)**: Join the movement for data democratization
+- **🚀 Builder ($50/mo)**: Get priority support and early feature access  
+- **💼 Growing Team ($500/mo)**: Bi-weekly syncs and optimization help
+- **🏢 Data Infrastructure Partner ($2000/mo)**: Full partnership with dedicated support
+
+**Why sponsor?** Every tier includes real benefits. No more rate-limited APIs. Own your data pipeline. Build data sovereignty together.
+
+[View All Tiers & Benefits →](https://github.com/sponsors/unclecode)
+
+### 🏆 Our Sponsors
+
+#### 👑 Founding Sponsors (First 50)
+*Be part of history - [Become a Founding Sponsor](https://github.com/sponsors/unclecode)*
+
+<!-- Founding sponsors will be permanently recognized here -->
+
+#### Current Sponsors
+Thank you to all our sponsors who make this project possible!
+
+<!-- Sponsors will be automatically added here -->
+
+## 🗾 Mission
+
+Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.  
+
+We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
+
+<details>
+<summary>🔑 <strong>Key Opportunities</strong></summary>
+ 
+- **Data Capitalization**: Transform digital footprints into measurable, valuable assets.  
+- **Authentic AI Data**: Provide AI systems with real human insights.  
+- **Shared Economy**: Create a fair data marketplace that benefits data creators.  
+
+</details>
+
+<details>
+<summary>🚀 <strong>Development Pathway</strong></summary>
+
+1. **Open-Source Tools**: Community-driven platforms for transparent data extraction.  
+2. **Digital Asset Structuring**: Tools to organize and value digital knowledge.  
+3. **Ethical Data Marketplace**: A secure, fair platform for exchanging structured data.  
+
+For more details, see our [full mission statement](./MISSION.md).
+</details>
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)

README.md

@@ -10,6 +10,7 @@
 [![PyPI version](https://badge.fury.io/py/crawl4ai.svg)](https://badge.fury.io/py/crawl4ai)
 [![Python Version](https://img.shields.io/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/)
 [![Downloads](https://static.pepy.tech/badge/crawl4ai/month)](https://pepy.tech/project/crawl4ai)
+[![GitHub Sponsors](https://img.shields.io/github/sponsors/unclecode?style=flat&logo=GitHub-Sponsors&label=Sponsors&color=pink)](https://github.com/sponsors/unclecode)
 
 <p align="center">
     <a href="https://x.com/crawl4ai">
@@ -24,32 +25,33 @@
   </p>
 </div>
 
-Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.  
+Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
 
 [✨ Check out latest update v0.7.0](#-recent-updates)
 
-🎉 **Version 0.7.0 is now available!** The Adaptive Intelligence Update introduces groundbreaking features: Adaptive Crawling that learns website patterns, Virtual Scroll support for infinite pages, intelligent Link Preview with 3-layer scoring, Async URL Seeder for massive discovery, and significant performance improvements. [Read the release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.0.md)
+✨ New in v0.7.0, Adaptive Crawling, Virtual Scroll, Link Preview scoring, Async URL Seeder, big performance gains. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.0.md)
 
 <details>
-<summary>🤓 <strong>My Personal Story</strong></summary>
+  <summary>🤓 <strong>My Personal Story</strong></summary>
 
-My journey with computers started in childhood when my dad, a computer scientist, introduced me to an Amstrad computer. Those early days sparked a fascination with technology, leading me to pursue computer science and specialize in NLP during my postgraduate studies. It was during this time that I first delved into web crawling, building tools to help researchers organize papers and extract information from publications a challenging yet rewarding experience that honed my skills in data extraction.
+I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.
 
-Fast forward to 2023, I was working on a tool for a project and needed a crawler to convert a webpage into markdown. While exploring solutions, I found one that claimed to be open-source but required creating an account and generating an API token. Worse, it turned out to be a SaaS model charging $16, and its quality didn’t meet my standards. Frustrated, I realized this was a deeper problem. That frustration turned into turbo anger mode, and I decided to build my own solution. In just a few days, I created Crawl4AI. To my surprise, it went viral, earning thousands of GitHub stars and resonating with a global community.
+In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.
 
-I made Crawl4AI open-source for two reasons. First, it’s my way of giving back to the open-source community that has supported me throughout my career. Second, I believe data should be accessible to everyone, not locked behind paywalls or monopolized by a few. Open access to data lays the foundation for the democratization of AI, a vision where individuals can train their own models and take ownership of their information. This library is the first step in a larger journey to create the best open-source data extraction and generation tool the world has ever seen, built collaboratively by a passionate community.
-
-Thank you to everyone who has supported this project, used it, and shared feedback. Your encouragement motivates me to dream even bigger. Join us, file issues, submit PRs, or spread the word. Together, we can build a tool that truly empowers people to access their own data and reshape the future of AI.
+I made it open source for **availability**, anyone can use it without a gate. Now I’m building the platform for **affordability**, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
 </details>
 
-## 🧐 Why Crawl4AI?
 
-1. **Built for LLMs**: Creates smart, concise Markdown optimized for RAG and fine-tuning applications.  
-2. **Lightning Fast**: Delivers results 6x faster with real-time, cost-efficient performance.  
-3. **Flexible Browser Control**: Offers session management, proxies, and custom hooks for seamless data access.  
-4. **Heuristic Intelligence**: Uses advanced algorithms for efficient extraction, reducing reliance on costly models.  
-5. **Open Source & Deployable**: Fully open-source with no API keys—ready for Docker and cloud integration.  
-6. **Thriving Community**: Actively maintained by a vibrant community and the #1 trending GitHub repository.
+<details>
+  <summary>Why developers pick Crawl4AI</summary>
+
+- **LLM ready output**, smart Markdown with headings, tables, code, citation hints
+- **Fast in practice**, async browser pool, caching, minimal hops
+- **Full control**, sessions, proxies, cookies, user scripts, hooks
+- **Adaptive intelligence**, learns site patterns, explores only what matters
+- **Deploy anywhere**, zero keys, CLI and Docker, cloud friendly
+</details>
+
 
 ## 🚀 Quick Start 
 
@@ -101,6 +103,33 @@ crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
 crwl https://www.example.com/products -q "Extract all product prices"
 ```
 
+## 💖 Support Crawl4AI
+
+> 🎉 **Sponsorship Program Now Open!** After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for **startups** and **enterprises**. Be among the first 50 **Founding Sponsors** for permanent recognition in our Hall of Fame.
+
+Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.
+
+<div align="">
+  
+[![Become a Sponsor](https://img.shields.io/badge/Become%20a%20Sponsor-pink?style=for-the-badge&logo=github-sponsors&logoColor=white)](https://github.com/sponsors/unclecode)  
+[![Current Sponsors](https://img.shields.io/github/sponsors/unclecode?style=for-the-badge&logo=github&label=Current%20Sponsors&color=green)](https://github.com/sponsors/unclecode)
+
+</div>
+
+### 🤝 Sponsorship Tiers
+
+- **🌱 Believer ($5/mo)** — Join the movement for data democratization  
+- **🚀 Builder ($50/mo)** — Priority support & early access to features  
+- **💼 Growing Team ($500/mo)** — Bi-weekly syncs & optimization help  
+- **🏢 Data Infrastructure Partner ($2000/mo)** — Full partnership with dedicated support  
+  *Custom arrangements available - see [SPONSORS.md](SPONSORS.md) for details & contact*
+
+**Why sponsor?**  
+No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.
+
+[See All Tiers & Benefits →](https://github.com/sponsors/unclecode)
+
+
 ## ✨ Features 
 
 <details>
@@ -280,12 +309,6 @@ docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:0.
 # Visit the playground at http://localhost:11235/playground
 ```
 
-For complete documentation, see our [Docker Deployment Guide](https://docs.crawl4ai.com/core/docker-deployment/).
-
-</details>
-
----
-
 ### Quick Test
 
 Run a quick test (works for both Docker options):
@@ -316,10 +339,11 @@ For more examples, see our [Docker Examples](https://github.com/unclecode/crawl4
 
 </details>
 
+---
 
 ## 🔬 Advanced Usage Examples 🔬
 
-You can check the project structure in the directory [https://github.com/unclecode/crawl4ai/docs/examples](docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.
+You can check the project structure in the directory [docs/examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.
 
 <details>
 <summary>📝 <strong>Heuristic Markdown Generation with Clean and Fit Markdown</strong></summary>
@@ -478,7 +502,7 @@ if __name__ == "__main__":
 </details>
 
 <details>
-<summary>🤖 <strong>Using You own Browser with Custom User Profile</strong></summary>
+<summary>🤖 <strong>Using Your own Browser with Custom User Profile</strong></summary>
 
 ```python
 import os, sys
@@ -583,97 +607,12 @@ async def test_news_crawl():
 
 Read the full details in our [0.7.0 Release Notes](https://docs.crawl4ai.com/blog/release-v0.7.0) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md).
 
-### Previous Version: 0.6.0 Release Highlights
-
-- **🌎 World-aware Crawling**: Set geolocation, language, and timezone for authentic locale-specific content:
-  ```python
-    crun_cfg = CrawlerRunConfig(
-        url="https://browserleaks.com/geo",          # test page that shows your location
-        locale="en-US",                              # Accept-Language & UI locale
-        timezone_id="America/Los_Angeles",           # JS Date()/Intl timezone
-        geolocation=GeolocationConfig(                 # override GPS coords
-            latitude=34.0522,
-            longitude=-118.2437,
-            accuracy=10.0,
-        )
-    )
-  ```
-
-- **📊 Table-to-DataFrame Extraction**: Extract HTML tables directly to CSV or pandas DataFrames:
-  ```python
-    crawler = AsyncWebCrawler(config=browser_config)
-    await crawler.start()
-
-    try:
-        # Set up scraping parameters
-        crawl_config = CrawlerRunConfig(
-            table_score_threshold=8,  # Strict table detection
-        )
-
-        # Execute market data extraction
-        results: List[CrawlResult] = await crawler.arun(
-            url="https://coinmarketcap.com/?page=1", config=crawl_config
-        )
-
-        # Process results
-        raw_df = pd.DataFrame()
-        for result in results:
-            if result.success and result.tables:
-                raw_df = pd.DataFrame(
-                    result.tables[0]["rows"],
-                    columns=result.tables[0]["headers"],
-                )
-                break
-        print(raw_df.head())
-
-    finally:
-        await crawler.close()
-  ```
-
-- **🚀 Browser Pooling**: Pages launch hot with pre-warmed browser instances for lower latency and memory usage
-
-- **🕸️ Network and Console Capture**: Full traffic logs and MHTML snapshots for debugging:
-  ```python
-  crawler_config = CrawlerRunConfig(
-      capture_network=True,
-      capture_console=True,
-      mhtml=True
-  )
-  ```
-
-- **🔌 MCP Integration**: Connect to AI tools like Claude Code through the Model Context Protocol
-  ```bash
-  # Add Crawl4AI to Claude Code
-  claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse
-  ```
-
-- **🖥️ Interactive Playground**: Test configurations and generate API requests with the built-in web interface at `http://localhost:11235//playground`
-
-- **🐳 Revamped Docker Deployment**: Streamlined multi-architecture Docker image with improved resource efficiency
-
-- **📱 Multi-stage Build System**: Optimized Dockerfile with platform-specific performance enhancements
-
-
-### Previous Version: 0.5.0 Major Release Highlights
-
--   **🚀 Deep Crawling System**: Explore websites beyond initial URLs with BFS, DFS, and BestFirst strategies
--   **⚡ Memory-Adaptive Dispatcher**: Dynamically adjusts concurrency based on system memory
--   **🔄 Multiple Crawling Strategies**: Browser-based and lightweight HTTP-only crawlers
--   **💻 Command-Line Interface**: New `crwl` CLI provides convenient terminal access
--   **👤 Browser Profiler**: Create and manage persistent browser profiles
--   **🧠 Crawl4AI Coding Assistant**: AI-powered coding assistant
--   **🏎️ LXML Scraping Mode**: Fast HTML parsing using the `lxml` library
--   **🌐 Proxy Rotation**: Built-in support for proxy switching
--   **🤖 LLM Content Filter**: Intelligent markdown generation using LLMs
--   **📄 PDF Processing**: Extract text, images, and metadata from PDF files
-
-Read the full details in our [0.5.0 Release Notes](https://docs.crawl4ai.com/blog/releases/0.5.0.html).
-
 ## Version Numbering in Crawl4AI
 
 Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
 
-### Version Numbers Explained
+<details>
+<summary>📈 <strong>Version Numbers Explained</strong></summary>
 
 Our version numbers follow this pattern: `MAJOR.MINOR.PATCH` (e.g., 0.4.3)
 
@@ -710,6 +649,8 @@ We use pre-releases to:
 
 For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the `--pre` flag.
 
+</details>
+
 ## 📖 Documentation & Roadmap 
 
 > 🚨 **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
@@ -722,16 +663,16 @@ To check our development plans and upcoming features, visit our [Roadmap](https:
 <summary>📈 <strong>Development TODOs</strong></summary>
 
 - [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
-- [ ] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
-- [ ] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
-- [ ] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
-- [ ] 4. Automated Schema Generator: Convert natural language to extraction schemas
-- [ ] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
-- [ ] 6. Web Embedding Index: Semantic search infrastructure for crawled content
-- [ ] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
-- [ ] 8. Performance Monitor: Real-time insights into crawler operations
+- [x] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
+- [x] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
+- [x] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
+- [x] 4. Automated Schema Generator: Convert natural language to extraction schemas
+- [x] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
+- [x] 6. Web Embedding Index: Semantic search infrastructure for crawled content
+- [x] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
+- [x] 8. Performance Monitor: Real-time insights into crawler operations
 - [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers
-- [ ] 10. Sponsorship Program: Structured support system with tiered benefits
+- [x] 10. Sponsorship Program: Structured support system with tiered benefits
 - [ ] 11. Educational Content: "How to Crawl" video series and interactive tutorials
 
 </details>
@@ -746,12 +687,13 @@ Here's the updated license section:
 
 ## 📄 License & Attribution
 
-This project is licensed under the Apache License 2.0 with a required attribution clause. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.
+This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.
 
 ### Attribution Requirements
 When using Crawl4AI, you must include one of the following attribution methods:
 
-#### 1. Badge Attribution (Recommended)
+<details>
+<summary>📈 <strong>1. Badge Attribution (Recommended)</strong></summary>
 Add one of these badges to your README, documentation, or website:
 
 | Theme | Badge |
@@ -790,11 +732,15 @@ HTML code for adding the badges:
 </a>
 ```
 
-#### 2. Text Attribution
+</details>
+
+<details>
+<summary>📖 <strong>2. Text Attribution</strong></summary>
 Add this line to your documentation:
 ```
 This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
 ```
+</details>
 
 ## 📚 Citation
 

SPONSORS.md

@@ -0,0 +1,65 @@
+# 💖 Sponsors & Supporters
+
+Thank you to everyone supporting Crawl4AI! Your sponsorship helps keep this project open-source and actively maintained.
+
+## 👑 Founding Sponsors
+*The first 50 sponsors who believed in our vision - permanently recognized*
+
+<!-- Founding sponsors will be listed here with special recognition -->
+🎉 **Become a Founding Sponsor!** Only [X/50] spots remaining! [Join now →](https://github.com/sponsors/unclecode)
+
+---
+
+## 🏢 Data Infrastructure Partners ($2000/month)
+*These organizations are building their data sovereignty with Crawl4AI at the core*
+
+<!-- Data Infrastructure Partners will be listed here -->
+*Be the first Data Infrastructure Partner! [Join us →](https://github.com/sponsors/unclecode)*
+
+---
+
+## 💼 Growing Teams ($500/month)
+*Teams scaling their data extraction with Crawl4AI*
+
+<!-- Growing Teams will be listed here -->
+*Your team could be here! [Become a sponsor →](https://github.com/sponsors/unclecode)*
+
+---
+
+## 🚀 Builders ($50/month)
+*Developers and entrepreneurs building with Crawl4AI*
+
+<!-- Builders will be listed here -->
+*Join the builders! [Start sponsoring →](https://github.com/sponsors/unclecode)*
+
+---
+
+## 🌱 Believers ($5/month)
+*The community supporting data democratization*
+
+<!-- Believers will be listed here -->
+*Thank you to all our community believers!*
+
+---
+
+## 🤝 Want to Sponsor?
+
+Crawl4AI is the #1 trending open-source web crawler. We're building the future of data extraction - where organizations own their data pipelines instead of relying on rate-limited APIs.
+
+### Available Sponsorship Tiers:
+- **🌱 Believer** ($5/mo) - Support the movement
+- **🚀 Builder** ($50/mo) - Priority support & early access
+- **💼 Growing Team** ($500/mo) - Bi-weekly syncs & optimization
+- **🏢 Data Infrastructure Partner** ($2000/mo) - Full partnership & dedicated support
+
+[View all tiers and benefits →](https://github.com/sponsors/unclecode)
+
+### Enterprise & Custom Partnerships
+
+Building data extraction at scale? Need dedicated support or infrastructure? Let's talk about a custom partnership.
+
+📧 Contact: [hello@crawl4ai.com](mailto:hello@crawl4ai.com) | 📅 [Schedule a call](https://calendar.app.google/rEpvi2UBgUQjWHfJ9)
+
+---
+
+*This list is updated regularly. Sponsors at $50+ tiers can submit their logos via [hello@crawl4ai.com](mailto:hello@crawl4ai.com)*

crawl4ai/__version__.py

@@ -1,7 +1,7 @@
 # crawl4ai/__version__.py
 
 # This is the version that will be used for stable releases
-__version__ = "0.7.2"
+__version__ = "0.7.3"
 
 # For nightly builds, this gets set during build process
 __nightly_version__ = None

deploy/docker/api.py

@@ -419,15 +419,13 @@ async def handle_crawl_request(
     urls: List[str],
     browser_config: dict,
     crawler_config: dict,
-    config: dict,
-    hooks_config: Optional[dict] = None
+    config: dict
 ) -> dict:
-    """Handle non-streaming crawl requests with optional hooks."""
+    """Handle non-streaming crawl requests."""
     start_mem_mb = _get_memory_mb() # <--- Get memory before
     start_time = time.time()
     mem_delta_mb = None
     peak_mem_mb = start_mem_mb
-    hook_manager = None
     
     try:
         urls = [('https://' + url) if not url.startswith(('http://', 'https://')) else url for url in urls]
@@ -447,19 +445,6 @@ async def handle_crawl_request(
         # crawler: AsyncWebCrawler = AsyncWebCrawler(config=browser_config)
         # await crawler.start()
         
-        # Attach hooks if provided
-        hooks_status = {}
-        if hooks_config:
-            from hook_manager import attach_user_hooks_to_crawler, UserHookManager
-            hook_manager = UserHookManager(timeout=hooks_config.get('timeout', 30))
-            hooks_status, hook_manager = await attach_user_hooks_to_crawler(
-                crawler,
-                hooks_config.get('code', {}),
-                timeout=hooks_config.get('timeout', 30),
-                hook_manager=hook_manager
-            )
-            logger.info(f"Hooks attachment status: {hooks_status['status']}")
-        
         base_config = config["crawler"]["base_config"]
         # Iterate on key-value pairs in global_config then use haseattr to set them 
         for key, value in base_config.items():
@@ -473,10 +458,6 @@ async def handle_crawl_request(
                                 config=crawler_config, 
                                 dispatcher=dispatcher)
         results = await partial_func()
-        
-        # Ensure results is always a list
-        if not isinstance(results, list):
-            results = [results]
 
         # await crawler.close()
         
@@ -491,68 +472,19 @@ async def handle_crawl_request(
         # Process results to handle PDF bytes
         processed_results = []
         for result in results:
-            try:
-                # Check if result has model_dump method (is a proper CrawlResult)
-                if hasattr(result, 'model_dump'):
-                    result_dict = result.model_dump()
-                elif isinstance(result, dict):
-                    result_dict = result
-                else:
-                    # Handle unexpected result type
-                    logger.warning(f"Unexpected result type: {type(result)}")
-                    result_dict = {
-                        "url": str(result) if hasattr(result, '__str__') else "unknown",
-                        "success": False,
-                        "error_message": f"Unexpected result type: {type(result).__name__}"
-                    }
-                
-                # If PDF exists, encode it to base64
-                if result_dict.get('pdf') is not None and isinstance(result_dict.get('pdf'), bytes):
-                    result_dict['pdf'] = b64encode(result_dict['pdf']).decode('utf-8')
-                    
-                processed_results.append(result_dict)
-            except Exception as e:
-                logger.error(f"Error processing result: {e}")
-                processed_results.append({
-                    "url": "unknown",
-                    "success": False,
-                    "error_message": str(e)
-                })
+            result_dict = result.model_dump()
+            # If PDF exists, encode it to base64
+            if result_dict.get('pdf') is not None:
+                result_dict['pdf'] = b64encode(result_dict['pdf']).decode('utf-8')
+            processed_results.append(result_dict)
             
-        response = {
+        return {
             "success": True,
             "results": processed_results,
             "server_processing_time_s": end_time - start_time,
             "server_memory_delta_mb": mem_delta_mb,
             "server_peak_memory_mb": peak_mem_mb
         }
-        
-        # Add hooks information if hooks were used
-        if hooks_config and hook_manager:
-            from hook_manager import UserHookManager
-            if isinstance(hook_manager, UserHookManager):
-                try:
-                    # Ensure all hook data is JSON serializable
-                    import json
-                    hook_data = {
-                        "status": hooks_status,
-                        "execution_log": hook_manager.execution_log,
-                        "errors": hook_manager.errors,
-                        "summary": hook_manager.get_summary()
-                    }
-                    # Test that it's serializable
-                    json.dumps(hook_data)
-                    response["hooks"] = hook_data
-                except (TypeError, ValueError) as e:
-                    logger.error(f"Hook data not JSON serializable: {e}")
-                    response["hooks"] = {
-                        "status": {"status": "error", "message": "Hook data serialization failed"},
-                        "execution_log": [],
-                        "errors": [{"error": str(e)}],
-                        "summary": {}
-                    }
-        
-        return response
 
     except Exception as e:
         logger.error(f"Crawl error: {str(e)}", exc_info=True)
@@ -581,11 +513,9 @@ async def handle_stream_crawl_request(
     urls: List[str],
     browser_config: dict,
     crawler_config: dict,
-    config: dict,
-    hooks_config: Optional[dict] = None
-) -> Tuple[AsyncWebCrawler, AsyncGenerator, Optional[Dict]]:
-    """Handle streaming crawl requests with optional hooks."""
-    hooks_info = None
+    config: dict
+) -> Tuple[AsyncWebCrawler, AsyncGenerator]:
+    """Handle streaming crawl requests."""
     try:
         browser_config = BrowserConfig.load(browser_config)
         # browser_config.verbose = True # Set to False or remove for production stress testing
@@ -606,20 +536,6 @@ async def handle_stream_crawl_request(
 
         # crawler = AsyncWebCrawler(config=browser_config)
         # await crawler.start()
-        
-        # Attach hooks if provided
-        if hooks_config:
-            from hook_manager import attach_user_hooks_to_crawler, UserHookManager
-            hook_manager = UserHookManager(timeout=hooks_config.get('timeout', 30))
-            hooks_status, hook_manager = await attach_user_hooks_to_crawler(
-                crawler,
-                hooks_config.get('code', {}),
-                timeout=hooks_config.get('timeout', 30),
-                hook_manager=hook_manager
-            )
-            logger.info(f"Hooks attachment status for streaming: {hooks_status['status']}")
-            # Include hook manager in hooks_info for proper tracking
-            hooks_info = {'status': hooks_status, 'manager': hook_manager}
 
         results_gen = await crawler.arun_many(
             urls=urls,
@@ -627,7 +543,7 @@ async def handle_stream_crawl_request(
             dispatcher=dispatcher
         )
 
-        return crawler, results_gen, hooks_info
+        return crawler, results_gen
 
     except Exception as e:
         # Make sure to close crawler if started during an error here

deploy/docker/hook_manager.py

@@ -1,512 +0,0 @@
-"""
-Hook Manager for User-Provided Hook Functions
-Handles validation, compilation, and safe execution of user-provided hook code
-"""
-
-import ast
-import asyncio
-import traceback
-from typing import Dict, Callable, Optional, Tuple, List, Any
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class UserHookManager:
-    """Manages user-provided hook functions with error isolation"""
-    
-    # Expected signatures for each hook point
-    HOOK_SIGNATURES = {
-        "on_browser_created": ["browser"],
-        "on_page_context_created": ["page", "context"],
-        "before_goto": ["page", "context", "url"],
-        "after_goto": ["page", "context", "url", "response"],
-        "on_user_agent_updated": ["page", "context", "user_agent"],
-        "on_execution_started": ["page", "context"],
-        "before_retrieve_html": ["page", "context"],
-        "before_return_html": ["page", "context", "html"]
-    }
-    
-    # Default timeout for hook execution (in seconds)
-    DEFAULT_TIMEOUT = 30
-    
-    def __init__(self, timeout: int = DEFAULT_TIMEOUT):
-        self.timeout = timeout
-        self.errors: List[Dict[str, Any]] = []
-        self.compiled_hooks: Dict[str, Callable] = {}
-        self.execution_log: List[Dict[str, Any]] = []
-    
-    def validate_hook_structure(self, hook_code: str, hook_point: str) -> Tuple[bool, str]:
-        """
-        Validate the structure of user-provided hook code
-        
-        Args:
-            hook_code: The Python code string containing the hook function
-            hook_point: The hook point name (e.g., 'on_page_context_created')
-            
-        Returns:
-            Tuple of (is_valid, error_message)
-        """
-        try:
-            # Parse the code
-            tree = ast.parse(hook_code)
-            
-            # Check if it's empty
-            if not tree.body:
-                return False, "Hook code is empty"
-            
-            # Find the function definition
-            func_def = None
-            for node in tree.body:
-                if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
-                    func_def = node
-                    break
-            
-            if not func_def:
-                return False, "Hook must contain a function definition (def or async def)"
-            
-            # Check if it's async (all hooks should be async)
-            if not isinstance(func_def, ast.AsyncFunctionDef):
-                return False, f"Hook function must be async (use 'async def' instead of 'def')"
-            
-            # Get function name for better error messages
-            func_name = func_def.name
-            
-            # Validate parameters
-            expected_params = self.HOOK_SIGNATURES.get(hook_point, [])
-            if not expected_params:
-                return False, f"Unknown hook point: {hook_point}"
-            
-            func_params = [arg.arg for arg in func_def.args.args]
-            
-            # Check if it has **kwargs for flexibility
-            has_kwargs = func_def.args.kwarg is not None
-            
-            # Must have at least the expected parameters
-            missing_params = []
-            for expected in expected_params:
-                if expected not in func_params:
-                    missing_params.append(expected)
-            
-            if missing_params and not has_kwargs:
-                return False, f"Hook function '{func_name}' must accept parameters: {', '.join(expected_params)} (missing: {', '.join(missing_params)})"
-            
-            # Check if it returns something (should return page or browser)
-            has_return = any(isinstance(node, ast.Return) for node in ast.walk(func_def))
-            if not has_return:
-                # Warning, not error - we'll handle this
-                logger.warning(f"Hook function '{func_name}' should return the {expected_params[0]} object")
-            
-            return True, "Valid"
-            
-        except SyntaxError as e:
-            return False, f"Syntax error at line {e.lineno}: {str(e)}"
-        except Exception as e:
-            return False, f"Failed to parse hook code: {str(e)}"
-    
-    def compile_hook(self, hook_code: str, hook_point: str) -> Optional[Callable]:
-        """
-        Compile user-provided hook code into a callable function
-        
-        Args:
-            hook_code: The Python code string
-            hook_point: The hook point name
-            
-        Returns:
-            Compiled function or None if compilation failed
-        """
-        try:
-            # Create a safe namespace for the hook
-            # Use a more complete builtins that includes __import__
-            import builtins
-            safe_builtins = {}
-            
-            # Add safe built-in functions
-            allowed_builtins = [
-                'print', 'len', 'str', 'int', 'float', 'bool',
-                'list', 'dict', 'set', 'tuple', 'range', 'enumerate',
-                'zip', 'map', 'filter', 'any', 'all', 'sum', 'min', 'max',
-                'sorted', 'reversed', 'abs', 'round', 'isinstance', 'type',
-                'getattr', 'hasattr', 'setattr', 'callable', 'iter', 'next',
-                '__import__', '__build_class__'  # Required for exec
-            ]
-            
-            for name in allowed_builtins:
-                if hasattr(builtins, name):
-                    safe_builtins[name] = getattr(builtins, name)
-            
-            namespace = {
-                '__name__': f'user_hook_{hook_point}',
-                '__builtins__': safe_builtins
-            }
-            
-            # Add commonly needed imports
-            exec("import asyncio", namespace)
-            exec("import json", namespace)
-            exec("import re", namespace)
-            exec("from typing import Dict, List, Optional", namespace)
-            
-            # Execute the code to define the function
-            exec(hook_code, namespace)
-            
-            # Find the async function in the namespace
-            for name, obj in namespace.items():
-                if callable(obj) and not name.startswith('_') and asyncio.iscoroutinefunction(obj):
-                    return obj
-            
-            # If no async function found, look for any function
-            for name, obj in namespace.items():
-                if callable(obj) and not name.startswith('_'):
-                    logger.warning(f"Found non-async function '{name}' - wrapping it")
-                    # Wrap sync function in async
-                    async def async_wrapper(*args, **kwargs):
-                        return obj(*args, **kwargs)
-                    return async_wrapper
-            
-            raise ValueError("No callable function found in hook code")
-            
-        except Exception as e:
-            error = {
-                'hook_point': hook_point,
-                'error': f"Failed to compile hook: {str(e)}",
-                'type': 'compilation_error',
-                'traceback': traceback.format_exc()
-            }
-            self.errors.append(error)
-            logger.error(f"Hook compilation failed for {hook_point}: {str(e)}")
-            return None
-    
-    async def execute_hook_safely(
-        self, 
-        hook_func: Callable, 
-        hook_point: str,
-        *args, 
-        **kwargs
-    ) -> Tuple[Any, Optional[Dict]]:
-        """
-        Execute a user hook with error isolation and timeout
-        
-        Args:
-            hook_func: The compiled hook function
-            hook_point: The hook point name
-            *args, **kwargs: Arguments to pass to the hook
-            
-        Returns:
-            Tuple of (result, error_dict)
-        """
-        start_time = asyncio.get_event_loop().time()
-        
-        try:
-            # Add timeout to prevent infinite loops
-            result = await asyncio.wait_for(
-                hook_func(*args, **kwargs),
-                timeout=self.timeout
-            )
-            
-            # Log successful execution
-            execution_time = asyncio.get_event_loop().time() - start_time
-            self.execution_log.append({
-                'hook_point': hook_point,
-                'status': 'success',
-                'execution_time': execution_time,
-                'timestamp': start_time
-            })
-            
-            return result, None
-            
-        except asyncio.TimeoutError:
-            error = {
-                'hook_point': hook_point,
-                'error': f'Hook execution timed out ({self.timeout}s limit)',
-                'type': 'timeout',
-                'execution_time': self.timeout
-            }
-            self.errors.append(error)
-            self.execution_log.append({
-                'hook_point': hook_point,
-                'status': 'timeout',
-                'error': error['error'],
-                'execution_time': self.timeout,
-                'timestamp': start_time
-            })
-            # Return the first argument (usually page/browser) to continue
-            return args[0] if args else None, error
-            
-        except Exception as e:
-            execution_time = asyncio.get_event_loop().time() - start_time
-            error = {
-                'hook_point': hook_point,
-                'error': str(e),
-                'type': type(e).__name__,
-                'traceback': traceback.format_exc(),
-                'execution_time': execution_time
-            }
-            self.errors.append(error)
-            self.execution_log.append({
-                'hook_point': hook_point,
-                'status': 'failed',
-                'error': str(e),
-                'error_type': type(e).__name__,
-                'execution_time': execution_time,
-                'timestamp': start_time
-            })
-            # Return the first argument (usually page/browser) to continue
-            return args[0] if args else None, error
-    
-    def get_summary(self) -> Dict[str, Any]:
-        """Get a summary of hook execution"""
-        total_hooks = len(self.execution_log)
-        successful = sum(1 for log in self.execution_log if log['status'] == 'success')
-        failed = sum(1 for log in self.execution_log if log['status'] == 'failed')
-        timed_out = sum(1 for log in self.execution_log if log['status'] == 'timeout')
-        
-        return {
-            'total_executions': total_hooks,
-            'successful': successful,
-            'failed': failed,
-            'timed_out': timed_out,
-            'success_rate': (successful / total_hooks * 100) if total_hooks > 0 else 0,
-            'total_errors': len(self.errors)
-        }
-
-
-class IsolatedHookWrapper:
-    """Wraps user hooks with error isolation and reporting"""
-    
-    def __init__(self, hook_manager: UserHookManager):
-        self.hook_manager = hook_manager
-    
-    def create_hook_wrapper(self, user_hook: Callable, hook_point: str) -> Callable:
-        """
-        Create a wrapper that isolates hook errors from main process
-        
-        Args:
-            user_hook: The compiled user hook function
-            hook_point: The hook point name
-            
-        Returns:
-            Wrapped async function that handles errors gracefully
-        """
-        
-        async def wrapped_hook(*args, **kwargs):
-            """Wrapped hook with error isolation"""
-            # Get the main return object (page/browser)
-            # This ensures we always have something to return
-            return_obj = None
-            if args:
-                return_obj = args[0]
-            elif 'page' in kwargs:
-                return_obj = kwargs['page']
-            elif 'browser' in kwargs:
-                return_obj = kwargs['browser']
-            
-            try:
-                # Execute user hook with safety
-                result, error = await self.hook_manager.execute_hook_safely(
-                    user_hook, 
-                    hook_point,
-                    *args, 
-                    **kwargs
-                )
-                
-                if error:
-                    # Hook failed but we continue with original object
-                    logger.warning(f"User hook failed at {hook_point}: {error['error']}")
-                    return return_obj
-                
-                # Hook succeeded - return its result or the original object
-                if result is None:
-                    logger.debug(f"Hook at {hook_point} returned None, using original object")
-                    return return_obj
-                
-                return result
-                
-            except Exception as e:
-                # This should rarely happen due to execute_hook_safely
-                logger.error(f"Unexpected error in hook wrapper for {hook_point}: {e}")
-                return return_obj
-        
-        # Set function name for debugging
-        wrapped_hook.__name__ = f"wrapped_{hook_point}"
-        return wrapped_hook
-
-
-async def process_user_hooks(
-    hooks_input: Dict[str, str],
-    timeout: int = 30
-) -> Tuple[Dict[str, Callable], List[Dict], UserHookManager]:
-    """
-    Process and compile user-provided hook functions
-    
-    Args:
-        hooks_input: Dictionary mapping hook points to code strings
-        timeout: Timeout for each hook execution
-        
-    Returns:
-        Tuple of (compiled_hooks, validation_errors, hook_manager)
-    """
-    
-    hook_manager = UserHookManager(timeout=timeout)
-    wrapper = IsolatedHookWrapper(hook_manager)
-    compiled_hooks = {}
-    validation_errors = []
-    
-    for hook_point, hook_code in hooks_input.items():
-        # Skip empty hooks
-        if not hook_code or not hook_code.strip():
-            continue
-        
-        # Validate hook point
-        if hook_point not in UserHookManager.HOOK_SIGNATURES:
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': f'Unknown hook point. Valid points: {", ".join(UserHookManager.HOOK_SIGNATURES.keys())}',
-                'code_preview': hook_code[:100] + '...' if len(hook_code) > 100 else hook_code
-            })
-            continue
-        
-        # Validate structure
-        is_valid, message = hook_manager.validate_hook_structure(hook_code, hook_point)
-        if not is_valid:
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': message,
-                'code_preview': hook_code[:100] + '...' if len(hook_code) > 100 else hook_code
-            })
-            continue
-        
-        # Compile the hook
-        hook_func = hook_manager.compile_hook(hook_code, hook_point)
-        if hook_func:
-            # Wrap with error isolation
-            wrapped_hook = wrapper.create_hook_wrapper(hook_func, hook_point)
-            compiled_hooks[hook_point] = wrapped_hook
-            logger.info(f"Successfully compiled hook for {hook_point}")
-        else:
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': 'Failed to compile hook function - check syntax and structure',
-                'code_preview': hook_code[:100] + '...' if len(hook_code) > 100 else hook_code
-            })
-    
-    return compiled_hooks, validation_errors, hook_manager
-
-
-async def process_user_hooks_with_manager(
-    hooks_input: Dict[str, str],
-    hook_manager: UserHookManager
-) -> Tuple[Dict[str, Callable], List[Dict]]:
-    """
-    Process and compile user-provided hook functions with existing manager
-    
-    Args:
-        hooks_input: Dictionary mapping hook points to code strings
-        hook_manager: Existing UserHookManager instance
-        
-    Returns:
-        Tuple of (compiled_hooks, validation_errors)
-    """
-    
-    wrapper = IsolatedHookWrapper(hook_manager)
-    compiled_hooks = {}
-    validation_errors = []
-    
-    for hook_point, hook_code in hooks_input.items():
-        # Skip empty hooks
-        if not hook_code or not hook_code.strip():
-            continue
-        
-        # Validate hook point
-        if hook_point not in UserHookManager.HOOK_SIGNATURES:
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': f'Unknown hook point. Valid points: {", ".join(UserHookManager.HOOK_SIGNATURES.keys())}',
-                'code_preview': hook_code[:100] + '...' if len(hook_code) > 100 else hook_code
-            })
-            continue
-        
-        # Validate structure
-        is_valid, message = hook_manager.validate_hook_structure(hook_code, hook_point)
-        if not is_valid:
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': message,
-                'code_preview': hook_code[:100] + '...' if len(hook_code) > 100 else hook_code
-            })
-            continue
-        
-        # Compile the hook
-        hook_func = hook_manager.compile_hook(hook_code, hook_point)
-        if hook_func:
-            # Wrap with error isolation
-            wrapped_hook = wrapper.create_hook_wrapper(hook_func, hook_point)
-            compiled_hooks[hook_point] = wrapped_hook
-            logger.info(f"Successfully compiled hook for {hook_point}")
-        else:
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': 'Failed to compile hook function - check syntax and structure',
-                'code_preview': hook_code[:100] + '...' if len(hook_code) > 100 else hook_code
-            })
-    
-    return compiled_hooks, validation_errors
-
-
-async def attach_user_hooks_to_crawler(
-    crawler,  # AsyncWebCrawler instance
-    user_hooks: Dict[str, str],
-    timeout: int = 30,
-    hook_manager: Optional[UserHookManager] = None
-) -> Tuple[Dict[str, Any], UserHookManager]:
-    """
-    Attach user-provided hooks to crawler with full error reporting
-    
-    Args:
-        crawler: AsyncWebCrawler instance
-        user_hooks: Dictionary mapping hook points to code strings
-        timeout: Timeout for each hook execution
-        hook_manager: Optional existing UserHookManager instance
-        
-    Returns:
-        Tuple of (status_dict, hook_manager)
-    """
-    
-    # Use provided hook_manager or create a new one
-    if hook_manager is None:
-        hook_manager = UserHookManager(timeout=timeout)
-    
-    # Process hooks with the hook_manager
-    compiled_hooks, validation_errors = await process_user_hooks_with_manager(
-        user_hooks, hook_manager
-    )
-    
-    # Log validation errors
-    if validation_errors:
-        logger.warning(f"Hook validation errors: {validation_errors}")
-    
-    # Attach successfully compiled hooks
-    attached_hooks = []
-    for hook_point, wrapped_hook in compiled_hooks.items():
-        try:
-            crawler.crawler_strategy.set_hook(hook_point, wrapped_hook)
-            attached_hooks.append(hook_point)
-            logger.info(f"Attached hook to {hook_point}")
-        except Exception as e:
-            logger.error(f"Failed to attach hook to {hook_point}: {e}")
-            validation_errors.append({
-                'hook_point': hook_point,
-                'error': f'Failed to attach hook: {str(e)}'
-            })
-    
-    status = 'success' if not validation_errors else ('partial' if attached_hooks else 'failed')
-    
-    status_dict = {
-        'status': status,
-        'attached_hooks': attached_hooks,
-        'validation_errors': validation_errors,
-        'total_hooks_provided': len(user_hooks),
-        'successfully_attached': len(attached_hooks),
-        'failed_validation': len(validation_errors)
-    }
-    
-    return status_dict, hook_manager

deploy/docker/schemas.py

@@ -9,50 +9,6 @@ class CrawlRequest(BaseModel):
     browser_config: Optional[Dict] = Field(default_factory=dict)
     crawler_config: Optional[Dict] = Field(default_factory=dict)
 
-
-class HookConfig(BaseModel):
-    """Configuration for user-provided hooks"""
-    code: Dict[str, str] = Field(
-        default_factory=dict,
-        description="Map of hook points to Python code strings"
-    )
-    timeout: int = Field(
-        default=30,
-        ge=1,
-        le=120,
-        description="Timeout in seconds for each hook execution"
-    )
-    
-    class Config:
-        schema_extra = {
-            "example": {
-                "code": {
-                    "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Block images to speed up crawling
-    await context.route("**/*.{png,jpg,jpeg,gif}", lambda route: route.abort())
-    return page
-""",
-                    "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # Scroll to load lazy content
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-    await page.wait_for_timeout(2000)
-    return page
-"""
-                },
-                "timeout": 30
-            }
-        }
-
-
-class CrawlRequestWithHooks(CrawlRequest):
-    """Extended crawl request with hooks support"""
-    hooks: Optional[HookConfig] = Field(
-        default=None,
-        description="Optional user-provided hook functions"
-    )
-
 class MarkdownRequest(BaseModel):
     """Request body for the /md endpoint."""
     url: str                    = Field(...,  description="Absolute http/https URL to fetch")

deploy/docker/server.py

@@ -23,7 +23,7 @@ from api import (
     stream_results
 )
 from schemas import (
-    CrawlRequestWithHooks,
+    CrawlRequest,
     MarkdownRequest,
     RawCode,
     HTMLRequest,
@@ -414,72 +414,6 @@ async def get_schema():
             "crawler": CrawlerRunConfig().dump()}
 
 
-@app.get("/hooks/info")
-async def get_hooks_info():
-    """Get information about available hook points and their signatures"""
-    from hook_manager import UserHookManager
-    
-    hook_info = {}
-    for hook_point, params in UserHookManager.HOOK_SIGNATURES.items():
-        hook_info[hook_point] = {
-            "parameters": params,
-            "description": get_hook_description(hook_point),
-            "example": get_hook_example(hook_point)
-        }
-    
-    return JSONResponse({
-        "available_hooks": hook_info,
-        "timeout_limits": {
-            "min": 1,
-            "max": 120,
-            "default": 30
-        }
-    })
-
-
-def get_hook_description(hook_point: str) -> str:
-    """Get description for each hook point"""
-    descriptions = {
-        "on_browser_created": "Called after browser instance is created",
-        "on_page_context_created": "Called after page and context are created - ideal for authentication",
-        "before_goto": "Called before navigating to the target URL",
-        "after_goto": "Called after navigation is complete",
-        "on_user_agent_updated": "Called when user agent is updated",
-        "on_execution_started": "Called when custom JavaScript execution begins",
-        "before_retrieve_html": "Called before retrieving the final HTML - ideal for scrolling",
-        "before_return_html": "Called just before returning the HTML content"
-    }
-    return descriptions.get(hook_point, "")
-
-
-def get_hook_example(hook_point: str) -> str:
-    """Get example code for each hook point"""
-    examples = {
-        "on_page_context_created": """async def hook(page, context, **kwargs):
-    # Add authentication cookie
-    await context.add_cookies([{
-        'name': 'session',
-        'value': 'my-session-id',
-        'domain': '.example.com'
-    }])
-    return page""",
-        
-        "before_retrieve_html": """async def hook(page, context, **kwargs):
-    # Scroll to load lazy content
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-    await page.wait_for_timeout(2000)
-    return page""",
-        
-        "before_goto": """async def hook(page, context, url, **kwargs):
-    # Set custom headers
-    await page.set_extra_http_headers({
-        'X-Custom-Header': 'value'
-    })
-    return page"""
-    }
-    return examples.get(hook_point, "# Implement your hook logic here\nreturn page")
-
-
 @app.get(config["observability"]["health_check"]["endpoint"])
 async def health():
     return {"status": "ok", "timestamp": time.time(), "version": __version__}
@@ -495,30 +429,19 @@ async def metrics():
 @mcp_tool("crawl")
 async def crawl(
     request: Request,
-    crawl_request: CrawlRequestWithHooks,
+    crawl_request: CrawlRequest,
     _td: Dict = Depends(token_dep),
 ):
     """
     Crawl a list of URLs and return the results as JSON.
-    Supports optional user-provided hook functions for customization.
     """
     if not crawl_request.urls:
         raise HTTPException(400, "At least one URL required")
-    
-    # Prepare hooks config if provided
-    hooks_config = None
-    if crawl_request.hooks:
-        hooks_config = {
-            'code': crawl_request.hooks.code,
-            'timeout': crawl_request.hooks.timeout
-        }
-    
     res = await handle_crawl_request(
         urls=crawl_request.urls,
         browser_config=crawl_request.browser_config,
         crawler_config=crawl_request.crawler_config,
         config=config,
-        hooks_config=hooks_config
     )
     return JSONResponse(res)
 
@@ -527,42 +450,25 @@ async def crawl(
 @limiter.limit(config["rate_limiting"]["default_limit"])
 async def crawl_stream(
     request: Request,
-    crawl_request: CrawlRequestWithHooks,
+    crawl_request: CrawlRequest,
     _td: Dict = Depends(token_dep),
 ):
     if not crawl_request.urls:
         raise HTTPException(400, "At least one URL required")
-    
-    # Prepare hooks config if provided
-    hooks_config = None
-    if crawl_request.hooks:
-        hooks_config = {
-            'code': crawl_request.hooks.code,
-            'timeout': crawl_request.hooks.timeout
-        }
-    
-    crawler, gen, hooks_info = await handle_stream_crawl_request(
+    crawler, gen = await handle_stream_crawl_request(
         urls=crawl_request.urls,
         browser_config=crawl_request.browser_config,
         crawler_config=crawl_request.crawler_config,
         config=config,
-        hooks_config=hooks_config
     )
-    
-    # Add hooks info to response headers if available
-    headers = {
-        "Cache-Control": "no-cache",
-        "Connection": "keep-alive",
-        "X-Stream-Status": "active",
-    }
-    if hooks_info:
-        import json
-        headers["X-Hooks-Status"] = json.dumps(hooks_info['status']['status'])
-    
     return StreamingResponse(
         stream_results(crawler, gen),
         media_type="application/x-ndjson",
-        headers=headers,
+        headers={
+            "Cache-Control": "no-cache",
+            "Connection": "keep-alive",
+            "X-Stream-Status": "active",
+        },
     )
 
 

docs/blog/release-v0.7.3.md

@@ -0,0 +1,170 @@
+# 🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update
+
+*August 6, 2025 • 5 min read*
+
+---
+
+Today I'm releasing Crawl4AI v0.7.3—the Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.
+
+## 🎯 What's New at a Glance
+
+- **Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
+- **Flexible Docker LLM Providers**: Configure LLM providers via environment variables
+- **Bug Fixes**: Resolved several critical issues for better stability
+- **Documentation Updates**: Clearer examples and improved API documentation
+
+## 🎨 Multi-URL Configurations: One Size Doesn't Fit All
+
+**The Problem:** You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handling—caching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.
+
+**My Solution:** I implemented URL-specific configurations that let you define different strategies for different URL patterns in a single crawl batch. First match wins, with optional fallback support.
+
+### Technical Implementation
+
+```python
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
+
+# Define specialized configs for different content types
+configs = [
+    # Documentation sites - aggressive caching, include links
+    CrawlerRunConfig(
+        url_matcher=["*docs*", "*documentation*"],
+        cache_mode="write",
+        markdown_generator_options={"include_links": True}
+    ),
+    
+    # News/blog sites - fresh content, scroll for lazy loading
+    CrawlerRunConfig(
+        url_matcher=lambda url: 'blog' in url or 'news' in url,
+        cache_mode="bypass",
+        js_code="window.scrollTo(0, document.body.scrollHeight/2);"
+    ),
+    
+    # API endpoints - structured extraction
+    CrawlerRunConfig(
+        url_matcher=["*.json", "*api*"],
+        extraction_strategy=LLMExtractionStrategy(
+            provider="openai/gpt-4o-mini",
+            extraction_type="structured"
+        )
+    ),
+    
+    # Default fallback for everything else
+    CrawlerRunConfig()  # No url_matcher = matches everything
+]
+
+# Crawl multiple URLs with appropriate configs
+async with AsyncWebCrawler() as crawler:
+    results = await crawler.arun_many(
+        urls=[
+            "https://docs.python.org/3/",      # → Uses documentation config
+            "https://blog.python.org/",        # → Uses blog config  
+            "https://api.github.com/users",    # → Uses API config
+            "https://example.com/"             # → Uses default config
+        ],
+        config=configs
+    )
+```
+
+**Matching Capabilities:**
+- **String Patterns**: Wildcards like `"*.pdf"`, `"*/blog/*"`
+- **Function Matchers**: Lambda functions for complex logic
+- **Mixed Matchers**: Combine strings and functions with AND/OR logic
+- **Fallback Support**: Default config when nothing matches
+
+**Expected Real-World Impact:**
+- **Mixed Content Sites**: Handle blogs, docs, and downloads in one crawl
+- **Multi-Domain Crawling**: Different strategies per domain without separate runs
+- **Reduced Complexity**: No more if/else forests in your extraction code
+- **Better Performance**: Each URL gets exactly the processing it needs
+
+## 🐳 Docker: Flexible LLM Provider Configuration
+
+**The Problem:** Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.
+
+**My Solution:** Configure LLM providers via environment variables. Switch providers without touching code or rebuilding images.
+
+### Deployment Flexibility
+
+```bash
+# Option 1: Direct environment variables
+docker run -d \
+  -e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
+  -e GROQ_API_KEY="your-key" \
+  -p 11235:11235 \
+  unclecode/crawl4ai:latest
+
+# Option 2: Using .llm.env file (recommended for production)
+# Create .llm.env file:
+# LLM_PROVIDER=openai/gpt-4o-mini
+# OPENAI_API_KEY=your-openai-key
+# GROQ_API_KEY=your-groq-key
+
+docker run -d \
+  --env-file .llm.env \
+  -p 11235:11235 \
+  unclecode/crawl4ai:latest
+```
+
+Override per request when needed:
+```python
+# Use default provider from .llm.env
+response = requests.post("http://localhost:11235/crawl", json={
+    "url": "https://example.com",
+    "extraction_strategy": {"type": "llm"}
+})
+
+# Override to use different provider for this specific request
+response = requests.post("http://localhost:11235/crawl", json={
+    "url": "https://complex-page.com",
+    "extraction_strategy": {
+        "type": "llm",
+        "provider": "openai/gpt-4"  # Override default
+    }
+})
+```
+
+**Expected Real-World Impact:**
+- **Cost Optimization**: Use cheaper models for simple tasks, premium for complex
+- **A/B Testing**: Compare provider performance without deployment changes
+- **Fallback Strategies**: Switch providers on-the-fly during outages
+- **Development Flexibility**: Test locally with one provider, deploy with another
+- **Secure Configuration**: Keep API keys in `.llm.env` file, not in commands
+
+## 🔧 Bug Fixes & Improvements
+
+This release includes several important bug fixes that improve stability and reliability:
+
+- **URL Matcher Fallback**: Fixed edge cases in URL pattern matching logic
+- **Memory Management**: Resolved memory leaks in long-running crawl sessions
+- **Sitemap Processing**: Fixed redirect handling in sitemap fetching
+- **Table Extraction**: Improved table detection and extraction accuracy
+- **Error Handling**: Better error messages and recovery from network failures
+
+## 📚 Documentation Enhancements
+
+Based on community feedback, we've updated:
+- Clearer examples for multi-URL configuration
+- Improved CrawlResult documentation with all available fields
+- Fixed typos and inconsistencies across documentation
+- Added real-world URLs in examples for better understanding
+- New comprehensive demo showcasing all v0.7.3 features
+
+## 🙏 Acknowledgments
+
+Thanks to our contributors and the entire community for feedback and bug reports.
+
+## 📚 Resources
+
+- [Full Documentation](https://docs.crawl4ai.com)
+- [GitHub Repository](https://github.com/unclecode/crawl4ai)
+- [Discord Community](https://discord.gg/crawl4ai)
+- [Feature Demo](https://github.com/unclecode/crawl4ai/blob/main/docs/releases_review/demo_v0.7.3.py)
+
+---
+
+*Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deployment—they're game changers!*
+
+**Happy Crawling! 🕷️**
+
+*- The Crawl4AI Team*

docs/examples/docker_hooks_examples.py

@@ -1,513 +0,0 @@
-#!/usr/bin/env python3
-"""
-Comprehensive test demonstrating all hook types from hooks_example.py
-adapted for the Docker API with real URLs
-"""
-
-import requests
-import json
-import time
-from typing import Dict, Any
-
-# API_BASE_URL = "http://localhost:11234"
-API_BASE_URL = "http://localhost:11235"
-
-
-def test_all_hooks_demo():
-    """Demonstrate all 8 hook types with practical examples"""
-    print("=" * 70)
-    print("Testing: All Hooks Comprehensive Demo")
-    print("=" * 70)
-    
-    hooks_code = {
-        "on_browser_created": """
-async def hook(browser, **kwargs):
-    # Hook called after browser is created
-    print("[HOOK] on_browser_created - Browser is ready!")
-    # Browser-level configurations would go here
-    return browser
-""",
-        
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Hook called after a new page and context are created
-    print("[HOOK] on_page_context_created - New page created!")
-    
-    # Set viewport size for consistent rendering
-    await page.set_viewport_size({"width": 1920, "height": 1080})
-    
-    # Add cookies for the session (using httpbin.org domain)
-    await context.add_cookies([
-        {
-            "name": "test_session",
-            "value": "abc123xyz",
-            "domain": ".httpbin.org",
-            "path": "/",
-            "httpOnly": True,
-            "secure": True
-        }
-    ])
-    
-    # Block ads and tracking scripts to speed up crawling
-    await context.route("**/*.{png,jpg,jpeg,gif,webp,svg}", lambda route: route.abort())
-    await context.route("**/analytics/*", lambda route: route.abort())
-    await context.route("**/ads/*", lambda route: route.abort())
-    
-    print("[HOOK] Viewport set, cookies added, and ads blocked")
-    return page
-""",
-        
-        "on_user_agent_updated": """
-async def hook(page, context, user_agent, **kwargs):
-    # Hook called when user agent is updated
-    print(f"[HOOK] on_user_agent_updated - User agent: {user_agent[:50]}...")
-    return page
-""",
-        
-        "before_goto": """
-async def hook(page, context, url, **kwargs):
-    # Hook called before navigating to each URL
-    print(f"[HOOK] before_goto - About to visit: {url}")
-    
-    # Add custom headers for the request
-    await page.set_extra_http_headers({
-        "X-Custom-Header": "crawl4ai-test",
-        "Accept-Language": "en-US,en;q=0.9",
-        "DNT": "1"
-    })
-    
-    return page
-""",
-        
-        "after_goto": """
-async def hook(page, context, url, response, **kwargs):
-    # Hook called after navigating to each URL
-    print(f"[HOOK] after_goto - Successfully loaded: {url}")
-    
-    # Wait a moment for dynamic content to load
-    await page.wait_for_timeout(1000)
-    
-    # Check if specific elements exist (with error handling)
-    try:
-        # For httpbin.org, wait for body element
-        await page.wait_for_selector("body", timeout=2000)
-        print("[HOOK] Body element found and loaded")
-    except:
-        print("[HOOK] Timeout waiting for body, continuing anyway")
-    
-    return page
-""",
-        
-        "on_execution_started": """
-async def hook(page, context, **kwargs):
-    # Hook called after custom JavaScript execution
-    print("[HOOK] on_execution_started - Custom JS executed!")
-    
-    # You could inject additional JavaScript here if needed
-    await page.evaluate("console.log('[INJECTED] Hook JS running');")
-    
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # Hook called before retrieving the HTML content
-    print("[HOOK] before_retrieve_html - Preparing to get HTML")
-    
-    # Scroll to bottom to trigger lazy loading
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
-    await page.wait_for_timeout(500)
-    
-    # Scroll back to top
-    await page.evaluate("window.scrollTo(0, 0);")
-    await page.wait_for_timeout(500)
-    
-    # One more scroll to middle for good measure
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight / 2);")
-    
-    print("[HOOK] Scrolling completed for lazy-loaded content")
-    return page
-""",
-        
-        "before_return_html": """
-async def hook(page, context, html, **kwargs):
-    # Hook called before returning the HTML content
-    print(f"[HOOK] before_return_html - HTML length: {len(html)} characters")
-    
-    # Log some page metrics
-    metrics = await page.evaluate('''() => {
-        return {
-            images: document.images.length,
-            links: document.links.length,
-            scripts: document.scripts.length
-        }
-    }''')
-    
-    print(f"[HOOK] Page metrics - Images: {metrics['images']}, Links: {metrics['links']}, Scripts: {metrics['scripts']}")
-    
-    return page
-"""
-    }
-    
-    # Create request payload
-    payload = {
-        "urls": ["https://httpbin.org/html"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 30
-        },
-        "crawler_config": {
-            "js_code": "window.scrollTo(0, document.body.scrollHeight);",
-            "wait_for": "body",
-            "cache_mode": "bypass"
-        }
-    }
-    
-    print("\nSending request with all 8 hooks...")
-    start_time = time.time()
-    
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    elapsed_time = time.time() - start_time
-    print(f"Request completed in {elapsed_time:.2f} seconds")
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("\n✅ Request successful!")
-        
-        # Check hooks execution
-        if 'hooks' in data:
-            hooks_info = data['hooks']
-            print("\n📊 Hooks Execution Summary:")
-            print(f"  Status: {hooks_info['status']['status']}")
-            print(f"  Attached hooks: {len(hooks_info['status']['attached_hooks'])}")
-            
-            for hook_name in hooks_info['status']['attached_hooks']:
-                print(f"    ✓ {hook_name}")
-            
-            if 'summary' in hooks_info:
-                summary = hooks_info['summary']
-                print(f"\n📈 Execution Statistics:")
-                print(f"  Total executions: {summary['total_executions']}")
-                print(f"  Successful: {summary['successful']}")
-                print(f"  Failed: {summary['failed']}")
-                print(f"  Timed out: {summary['timed_out']}")
-                print(f"  Success rate: {summary['success_rate']:.1f}%")
-            
-            if hooks_info.get('execution_log'):
-                print(f"\n📝 Execution Log:")
-                for log_entry in hooks_info['execution_log']:
-                    status_icon = "✅" if log_entry['status'] == 'success' else "❌"
-                    exec_time = log_entry.get('execution_time', 0)
-                    print(f"  {status_icon} {log_entry['hook_point']}: {exec_time:.3f}s")
-        
-        # Check crawl results
-        if 'results' in data and len(data['results']) > 0:
-            print(f"\n📄 Crawl Results:")
-            for result in data['results']:
-                print(f"  URL: {result['url']}")
-                print(f"  Success: {result.get('success', False)}")
-                if result.get('html'):
-                    print(f"  HTML length: {len(result['html'])} characters")
-    
-    else:
-        print(f"❌ Error: {response.status_code}")
-        try:
-            error_data = response.json()
-            print(f"Error details: {json.dumps(error_data, indent=2)}")
-        except:
-            print(f"Error text: {response.text[:500]}")
-
-
-def test_authentication_flow():
-    """Test a complete authentication flow with multiple hooks"""
-    print("\n" + "=" * 70)
-    print("Testing: Authentication Flow with Multiple Hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Setting up authentication context")
-    
-    # Add authentication cookies
-    await context.add_cookies([
-        {
-            "name": "auth_token",
-            "value": "fake_jwt_token_here",
-            "domain": ".httpbin.org",
-            "path": "/",
-            "httpOnly": True,
-            "secure": True
-        }
-    ])
-    
-    # Set localStorage items (for SPA authentication)
-    await page.evaluate('''
-        localStorage.setItem('user_id', '12345');
-        localStorage.setItem('auth_time', new Date().toISOString());
-    ''')
-    
-    return page
-""",
-        
-        "before_goto": """
-async def hook(page, context, url, **kwargs):
-    print(f"[HOOK] Adding auth headers for {url}")
-    
-    # Add Authorization header
-    import base64
-    credentials = base64.b64encode(b"user:passwd").decode('ascii')
-    
-    await page.set_extra_http_headers({
-        'Authorization': f'Basic {credentials}',
-        'X-API-Key': 'test-api-key-123'
-    })
-    
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": [
-            "https://httpbin.org/basic-auth/user/passwd"
-        ],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 15
-        }
-    }
-    
-    print("\nTesting authentication with httpbin endpoints...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("✅ Authentication test completed")
-        
-        if 'results' in data:
-            for i, result in enumerate(data['results']):
-                print(f"\n  URL {i+1}: {result['url']}")
-                if result.get('success'):
-                    # Check for authentication success indicators
-                    html_content = result.get('html', '')
-                    if '"authenticated"' in html_content and 'true' in html_content:
-                        print("    ✅ Authentication successful! Basic auth worked.")
-                    else:
-                        print("    ⚠️ Page loaded but auth status unclear")
-                else:
-                    print(f"    ❌ Failed: {result.get('error_message', 'Unknown error')}")
-    else:
-        print(f"❌ Error: {response.status_code}")
-
-
-def test_performance_optimization_hooks():
-    """Test hooks for performance optimization"""
-    print("\n" + "=" * 70)
-    print("Testing: Performance Optimization Hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Optimizing page for performance")
-    
-    # Block resource-heavy content
-    await context.route("**/*.{png,jpg,jpeg,gif,webp,svg,ico}", lambda route: route.abort())
-    await context.route("**/*.{woff,woff2,ttf,otf}", lambda route: route.abort())
-    await context.route("**/*.{mp4,webm,ogg,mp3,wav}", lambda route: route.abort())
-    await context.route("**/googletagmanager.com/*", lambda route: route.abort())
-    await context.route("**/google-analytics.com/*", lambda route: route.abort())
-    await context.route("**/doubleclick.net/*", lambda route: route.abort())
-    await context.route("**/facebook.com/*", lambda route: route.abort())
-    
-    # Disable animations and transitions
-    await page.add_style_tag(content='''
-        *, *::before, *::after {
-            animation-duration: 0s !important;
-            animation-delay: 0s !important;
-            transition-duration: 0s !important;
-            transition-delay: 0s !important;
-        }
-    ''')
-    
-    print("[HOOK] Performance optimizations applied")
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Removing unnecessary elements before extraction")
-    
-    # Remove ads, popups, and other unnecessary elements
-    await page.evaluate('''() => {
-        // Remove common ad containers
-        const adSelectors = [
-            '.ad', '.ads', '.advertisement', '[id*="ad-"]', '[class*="ad-"]',
-            '.popup', '.modal', '.overlay', '.cookie-banner', '.newsletter-signup'
-        ];
-        
-        adSelectors.forEach(selector => {
-            document.querySelectorAll(selector).forEach(el => el.remove());
-        });
-        
-        // Remove script tags to clean up HTML
-        document.querySelectorAll('script').forEach(el => el.remove());
-        
-        // Remove style tags we don't need
-        document.querySelectorAll('style').forEach(el => el.remove());
-    }''')
-    
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/html"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 10
-        }
-    }
-    
-    print("\nTesting performance optimization hooks...")
-    start_time = time.time()
-    
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    elapsed_time = time.time() - start_time
-    print(f"Request completed in {elapsed_time:.2f} seconds")
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("✅ Performance optimization test completed")
-        
-        if 'results' in data and len(data['results']) > 0:
-            result = data['results'][0]
-            if result.get('html'):
-                print(f"  HTML size: {len(result['html'])} characters")
-                print("  Resources blocked, ads removed, animations disabled")
-    else:
-        print(f"❌ Error: {response.status_code}")
-
-
-def test_content_extraction_hooks():
-    """Test hooks for intelligent content extraction"""
-    print("\n" + "=" * 70)
-    print("Testing: Content Extraction Hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "after_goto": """
-async def hook(page, context, url, response, **kwargs):
-    print(f"[HOOK] Waiting for dynamic content on {url}")
-    
-    # Wait for any lazy-loaded content
-    await page.wait_for_timeout(2000)
-    
-    # Trigger any "Load More" buttons
-    try:
-        load_more = await page.query_selector('[class*="load-more"], [class*="show-more"], button:has-text("Load More")')
-        if load_more:
-            await load_more.click()
-            await page.wait_for_timeout(1000)
-            print("[HOOK] Clicked 'Load More' button")
-    except:
-        pass
-    
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Extracting structured data")
-    
-    # Extract metadata
-    metadata = await page.evaluate('''() => {
-        const getMeta = (name) => {
-            const element = document.querySelector(`meta[name="${name}"], meta[property="${name}"]`);
-            return element ? element.getAttribute('content') : null;
-        };
-        
-        return {
-            title: document.title,
-            description: getMeta('description') || getMeta('og:description'),
-            author: getMeta('author'),
-            keywords: getMeta('keywords'),
-            ogTitle: getMeta('og:title'),
-            ogImage: getMeta('og:image'),
-            canonical: document.querySelector('link[rel="canonical"]')?.href,
-            jsonLd: Array.from(document.querySelectorAll('script[type="application/ld+json"]'))
-                .map(el => el.textContent).filter(Boolean)
-        };
-    }''')
-    
-    print(f"[HOOK] Extracted metadata: {json.dumps(metadata, indent=2)}")
-    
-    # Infinite scroll handling
-    for i in range(3):
-        await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
-        await page.wait_for_timeout(1000)
-        print(f"[HOOK] Scroll iteration {i+1}/3")
-    
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/html", "https://httpbin.org/json"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 20
-        }
-    }
-    
-    print("\nTesting content extraction hooks...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("✅ Content extraction test completed")
-        
-        if 'hooks' in data and 'summary' in data['hooks']:
-            summary = data['hooks']['summary']
-            print(f"  Hooks executed: {summary['successful']}/{summary['total_executions']}")
-        
-        if 'results' in data:
-            for result in data['results']:
-                print(f"\n  URL: {result['url']}")
-                print(f"  Success: {result.get('success', False)}")
-    else:
-        print(f"❌ Error: {response.status_code}")
-
-
-def main():
-    """Run comprehensive hook tests"""
-    print("🔧 Crawl4AI Docker API - Comprehensive Hooks Testing")
-    print("Based on docs/examples/hooks_example.py")
-    print("=" * 70)
-    
-    tests = [
-        ("All Hooks Demo", test_all_hooks_demo),
-        ("Authentication Flow", test_authentication_flow),
-        ("Performance Optimization", test_performance_optimization_hooks),
-        ("Content Extraction", test_content_extraction_hooks),
-    ]
-    
-    for i, (name, test_func) in enumerate(tests, 1):
-        print(f"\n📌 Test {i}/{len(tests)}: {name}")
-        try:
-            test_func()
-            print(f"✅ {name} completed")
-        except Exception as e:
-            print(f"❌ {name} failed: {e}")
-            import traceback
-            traceback.print_exc()
-    
-    print("\n" + "=" * 70)
-    print("🎉 All comprehensive hook tests completed!")
-    print("=" * 70)
-
-
-if __name__ == "__main__":
-    main()

docs/md_v2/blog/index.md

@@ -20,24 +20,30 @@ Ever wondered why your AI coding assistant struggles with your library despite c
 
 ## Latest Release
 
-### [Crawl4AI v0.7.0 – The Adaptive Intelligence Update](releases/0.7.0.md)
-*January 28, 2025*
+### [Crawl4AI v0.7.3 – The Multi-Config Intelligence Update](releases/0.7.3.md)
+*August 6, 2025*
 
-Crawl4AI v0.7.0 introduces groundbreaking intelligence features that transform how crawlers understand and adapt to websites. This release brings Adaptive Crawling that learns website patterns, Virtual Scroll support for infinite pages, intelligent Link Preview with 3-layer scoring, and the powerful Async URL Seeder for massive URL discovery.
+Crawl4AI v0.7.3 brings smarter URL-specific configurations, flexible Docker deployments, and critical stability improvements. Configure different crawling strategies for different URL patterns in a single batch—perfect for mixed content sites with docs, blogs, and APIs.
 
 Key highlights:
-- **Adaptive Crawling**: Crawlers that learn and adapt to website structures automatically
-- **Virtual Scroll Support**: Complete content extraction from modern infinite scroll pages  
-- **Link Preview**: 3-layer scoring system for intelligent link prioritization
-- **Async URL Seeder**: Discover thousands of URLs in seconds with smart filtering
-- **Performance Boost**: Up to 3x faster with optimized resource handling
+- **Multi-URL Configurations**: Different strategies for different URL patterns in one crawl
+- **Flexible Docker LLM Providers**: Configure providers via environment variables  
+- **Bug Fixes**: Critical stability improvements for production deployments
+- **Documentation Updates**: Clearer examples and improved API documentation
 
-[Read full release notes →](releases/0.7.0.md)
+[Read full release notes →](releases/0.7.3.md)
 
 ---
 
 ## Previous Releases
 
+### [Crawl4AI v0.7.0 – The Adaptive Intelligence Update](releases/0.7.0.md)
+*January 28, 2025*
+
+Introduced groundbreaking intelligence features including Adaptive Crawling, Virtual Scroll support, intelligent Link Preview, and the Async URL Seeder for massive URL discovery.
+
+[Read release notes →](releases/0.7.0.md)
+
 ### [Crawl4AI v0.6.0 – World-Aware Crawling, Pre-Warmed Browsers, and the MCP API](releases/0.6.0.md)
 *December 23, 2024*
 

docs/md_v2/blog/releases/0.7.3.md

@@ -0,0 +1,170 @@
+# 🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update
+
+*August 6, 2025 • 5 min read*
+
+---
+
+Today I'm releasing Crawl4AI v0.7.3—the Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.
+
+## 🎯 What's New at a Glance
+
+- **Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
+- **Flexible Docker LLM Providers**: Configure LLM providers via environment variables
+- **Bug Fixes**: Resolved several critical issues for better stability
+- **Documentation Updates**: Clearer examples and improved API documentation
+
+## 🎨 Multi-URL Configurations: One Size Doesn't Fit All
+
+**The Problem:** You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handling—caching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.
+
+**My Solution:** I implemented URL-specific configurations that let you define different strategies for different URL patterns in a single crawl batch. First match wins, with optional fallback support.
+
+### Technical Implementation
+
+```python
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
+
+# Define specialized configs for different content types
+configs = [
+    # Documentation sites - aggressive caching, include links
+    CrawlerRunConfig(
+        url_matcher=["*docs*", "*documentation*"],
+        cache_mode="write",
+        markdown_generator_options={"include_links": True}
+    ),
+    
+    # News/blog sites - fresh content, scroll for lazy loading
+    CrawlerRunConfig(
+        url_matcher=lambda url: 'blog' in url or 'news' in url,
+        cache_mode="bypass",
+        js_code="window.scrollTo(0, document.body.scrollHeight/2);"
+    ),
+    
+    # API endpoints - structured extraction
+    CrawlerRunConfig(
+        url_matcher=["*.json", "*api*"],
+        extraction_strategy=LLMExtractionStrategy(
+            provider="openai/gpt-4o-mini",
+            extraction_type="structured"
+        )
+    ),
+    
+    # Default fallback for everything else
+    CrawlerRunConfig()  # No url_matcher = matches everything
+]
+
+# Crawl multiple URLs with appropriate configs
+async with AsyncWebCrawler() as crawler:
+    results = await crawler.arun_many(
+        urls=[
+            "https://docs.python.org/3/",      # → Uses documentation config
+            "https://blog.python.org/",        # → Uses blog config  
+            "https://api.github.com/users",    # → Uses API config
+            "https://example.com/"             # → Uses default config
+        ],
+        config=configs
+    )
+```
+
+**Matching Capabilities:**
+- **String Patterns**: Wildcards like `"*.pdf"`, `"*/blog/*"`
+- **Function Matchers**: Lambda functions for complex logic
+- **Mixed Matchers**: Combine strings and functions with AND/OR logic
+- **Fallback Support**: Default config when nothing matches
+
+**Expected Real-World Impact:**
+- **Mixed Content Sites**: Handle blogs, docs, and downloads in one crawl
+- **Multi-Domain Crawling**: Different strategies per domain without separate runs
+- **Reduced Complexity**: No more if/else forests in your extraction code
+- **Better Performance**: Each URL gets exactly the processing it needs
+
+## 🐳 Docker: Flexible LLM Provider Configuration
+
+**The Problem:** Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.
+
+**My Solution:** Configure LLM providers via environment variables. Switch providers without touching code or rebuilding images.
+
+### Deployment Flexibility
+
+```bash
+# Option 1: Direct environment variables
+docker run -d \
+  -e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
+  -e GROQ_API_KEY="your-key" \
+  -p 11235:11235 \
+  unclecode/crawl4ai:latest
+
+# Option 2: Using .llm.env file (recommended for production)
+# Create .llm.env file:
+# LLM_PROVIDER=openai/gpt-4o-mini
+# OPENAI_API_KEY=your-openai-key
+# GROQ_API_KEY=your-groq-key
+
+docker run -d \
+  --env-file .llm.env \
+  -p 11235:11235 \
+  unclecode/crawl4ai:latest
+```
+
+Override per request when needed:
+```python
+# Use default provider from .llm.env
+response = requests.post("http://localhost:11235/crawl", json={
+    "url": "https://example.com",
+    "extraction_strategy": {"type": "llm"}
+})
+
+# Override to use different provider for this specific request
+response = requests.post("http://localhost:11235/crawl", json={
+    "url": "https://complex-page.com",
+    "extraction_strategy": {
+        "type": "llm",
+        "provider": "openai/gpt-4"  # Override default
+    }
+})
+```
+
+**Expected Real-World Impact:**
+- **Cost Optimization**: Use cheaper models for simple tasks, premium for complex
+- **A/B Testing**: Compare provider performance without deployment changes
+- **Fallback Strategies**: Switch providers on-the-fly during outages
+- **Development Flexibility**: Test locally with one provider, deploy with another
+- **Secure Configuration**: Keep API keys in `.llm.env` file, not in commands
+
+## 🔧 Bug Fixes & Improvements
+
+This release includes several important bug fixes that improve stability and reliability:
+
+- **URL Matcher Fallback**: Fixed edge cases in URL pattern matching logic
+- **Memory Management**: Resolved memory leaks in long-running crawl sessions
+- **Sitemap Processing**: Fixed redirect handling in sitemap fetching
+- **Table Extraction**: Improved table detection and extraction accuracy
+- **Error Handling**: Better error messages and recovery from network failures
+
+## 📚 Documentation Enhancements
+
+Based on community feedback, we've updated:
+- Clearer examples for multi-URL configuration
+- Improved CrawlResult documentation with all available fields
+- Fixed typos and inconsistencies across documentation
+- Added real-world URLs in examples for better understanding
+- New comprehensive demo showcasing all v0.7.3 features
+
+## 🙏 Acknowledgments
+
+Thanks to our contributors and the entire community for feedback and bug reports.
+
+## 📚 Resources
+
+- [Full Documentation](https://docs.crawl4ai.com)
+- [GitHub Repository](https://github.com/unclecode/crawl4ai)
+- [Discord Community](https://discord.gg/crawl4ai)
+- [Feature Demo](https://github.com/unclecode/crawl4ai/blob/main/docs/releases_review/demo_v0.7.3.py)
+
+---
+
+*Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deployment—they're game changers!*
+
+**Happy Crawling! 🕷️**
+
+*- The Crawl4AI Team*

docs/md_v2/core/docker-deployment.md

@@ -58,15 +58,15 @@ Pull and run images directly from Docker Hub without building locally.
 
 #### 1. Pull the Image
 
-Our latest release candidate is `0.7.0-r1`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.
+Our latest release is `0.7.3`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.
 
-> ⚠️ **Important Note**: The `latest` tag currently points to the stable `0.6.0` version. After testing and validation, `0.7.0` (without -r1) will be released and `latest` will be updated. For now, please use `0.7.0-r1` to test the new features.
+> 💡 **Note**: The `latest` tag points to the stable `0.7.3` version.
 
 ```bash
-# Pull the release candidate (for testing new features)
-docker pull unclecode/crawl4ai:0.7.0-r1
+# Pull the latest version
+docker pull unclecode/crawl4ai:0.7.3
 
-# Or pull the current stable version (0.6.0)
+# Or pull using the latest tag
 docker pull unclecode/crawl4ai:latest
 ```
 
@@ -126,7 +126,7 @@ docker stop crawl4ai && docker rm crawl4ai
 #### Docker Hub Versioning Explained
 
 *   **Image Name:** `unclecode/crawl4ai`
-*   **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.7.0-r1`)
+*   **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.7.3`)
     *   `LIBRARY_VERSION`: The semantic version of the core `crawl4ai` Python library
     *   `SUFFIX`: Optional tag for release candidates (``) and revisions (`r1`)
 *   **`latest` Tag:** Points to the most recent stable version
@@ -405,409 +405,6 @@ Executes JavaScript snippets on the specified URL and returns the full crawl res
 
 ---
 
-## User-Provided Hooks API
-
-The Docker API supports user-provided hook functions, allowing you to customize the crawling behavior by injecting your own Python code at specific points in the crawling pipeline. This powerful feature enables authentication, performance optimization, and custom content extraction without modifying the server code.
-
-> ⚠️ **IMPORTANT SECURITY WARNING**: 
-> - **Never use hooks with untrusted code or on untrusted websites**
-> - **Be extremely careful when crawling sites that might be phishing or malicious**
-> - **Hook code has access to page context and can interact with the website**
-> - **Always validate and sanitize any data extracted through hooks**
-> - **Never expose credentials or sensitive data in hook code**
-> - **Consider running the Docker container in an isolated network when testing**
-
-### Hook Information Endpoint
-
-```
-GET /hooks/info
-```
-
-Returns information about available hook points and their signatures:
-
-```bash
-curl http://localhost:11235/hooks/info
-```
-
-### Available Hook Points
-
-The API supports 8 hook points that match the local SDK:
-
-| Hook Point | Parameters | Description | Best Use Cases |
-|------------|------------|-------------|----------------|
-| `on_browser_created` | `browser` | After browser instance creation | Light setup tasks |
-| `on_page_context_created` | `page, context` | After page/context creation | **Authentication, cookies, route blocking** |
-| `before_goto` | `page, context, url` | Before navigating to URL | Custom headers, logging |
-| `after_goto` | `page, context, url, response` | After navigation completes | Verification, waiting for elements |
-| `on_user_agent_updated` | `page, context, user_agent` | When user agent changes | UA-specific logic |
-| `on_execution_started` | `page, context` | When JS execution begins | JS-related setup |
-| `before_retrieve_html` | `page, context` | Before getting final HTML | **Scrolling, lazy loading** |
-| `before_return_html` | `page, context, html` | Before returning HTML | Final modifications, metrics |
-
-### Using Hooks in Requests
-
-Add hooks to any crawl request by including the `hooks` parameter:
-
-```json
-{
-  "urls": ["https://httpbin.org/html"],
-  "hooks": {
-    "code": {
-      "hook_point_name": "async def hook(...): ...",
-      "another_hook": "async def hook(...): ..."
-    },
-    "timeout": 30  // Optional, default 30 seconds (max 120)
-  }
-}
-```
-
-### Hook Examples with Real URLs
-
-#### 1. Authentication with Cookies (GitHub)
-
-```python
-import requests
-
-# Example: Setting GitHub session cookie (use your actual session)
-hooks_code = {
-    "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Add authentication cookies for GitHub
-    # WARNING: Never hardcode real credentials!
-    await context.add_cookies([
-        {
-            'name': 'user_session',
-            'value': 'your_github_session_token',  # Replace with actual token
-            'domain': '.github.com',
-            'path': '/',
-            'httpOnly': True,
-            'secure': True,
-            'sameSite': 'Lax'
-        }
-    ])
-    return page
-"""
-}
-
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://github.com/settings/profile"],  # Protected page
-    "hooks": {"code": hooks_code, "timeout": 30}
-})
-```
-
-#### 2. Basic Authentication (httpbin.org for testing)
-
-```python
-# Safe testing with httpbin.org (a service designed for HTTP testing)
-hooks_code = {
-    "before_goto": """
-async def hook(page, context, url, **kwargs):
-    import base64
-    # httpbin.org/basic-auth expects username="user" and password="passwd"
-    credentials = base64.b64encode(b"user:passwd").decode('ascii')
-    
-    await page.set_extra_http_headers({
-        'Authorization': f'Basic {credentials}'
-    })
-    return page
-"""
-}
-
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://httpbin.org/basic-auth/user/passwd"],
-    "hooks": {"code": hooks_code, "timeout": 15}
-})
-```
-
-#### 3. Performance Optimization (News Sites)
-
-```python
-# Example: Optimizing crawling of news sites like CNN or BBC
-hooks_code = {
-    "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Block images, fonts, and media to speed up crawling
-    await context.route("**/*.{png,jpg,jpeg,gif,webp,svg,ico}", lambda route: route.abort())
-    await context.route("**/*.{woff,woff2,ttf,otf,eot}", lambda route: route.abort())
-    await context.route("**/*.{mp4,webm,ogg,mp3,wav,flac}", lambda route: route.abort())
-    
-    # Block common tracking and ad domains
-    await context.route("**/googletagmanager.com/*", lambda route: route.abort())
-    await context.route("**/google-analytics.com/*", lambda route: route.abort())
-    await context.route("**/doubleclick.net/*", lambda route: route.abort())
-    await context.route("**/facebook.com/tr/*", lambda route: route.abort())
-    await context.route("**/amazon-adsystem.com/*", lambda route: route.abort())
-    
-    # Disable CSS animations for faster rendering
-    await page.add_style_tag(content='''
-        *, *::before, *::after {
-            animation-duration: 0s !important;
-            transition-duration: 0s !important;
-        }
-    ''')
-    
-    return page
-"""
-}
-
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://www.bbc.com/news"],  # Heavy news site
-    "hooks": {"code": hooks_code, "timeout": 30}
-})
-```
-
-#### 4. Handling Infinite Scroll (Twitter/X)
-
-```python
-# Example: Scrolling on Twitter/X (requires authentication)
-hooks_code = {
-    "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # Scroll to load more tweets
-    previous_height = 0
-    for i in range(5):  # Limit scrolls to avoid infinite loop
-        current_height = await page.evaluate("document.body.scrollHeight")
-        if current_height == previous_height:
-            break  # No more content to load
-            
-        await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-        await page.wait_for_timeout(2000)  # Wait for content to load
-        previous_height = current_height
-    
-    return page
-"""
-}
-
-# Note: Twitter requires authentication for most content
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://twitter.com/nasa"],  # Public profile
-    "hooks": {"code": hooks_code, "timeout": 30}
-})
-```
-
-#### 5. E-commerce Login (Example Pattern)
-
-```python
-# SECURITY WARNING: This is a pattern example. 
-# Never use real credentials in code!
-# Always use environment variables or secure vaults.
-
-hooks_code = {
-    "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Example pattern for e-commerce sites
-    # DO NOT use real credentials here!
-    
-    # Navigate to login page first
-    await page.goto("https://example-shop.com/login")
-    
-    # Wait for login form to load
-    await page.wait_for_selector("#email", timeout=5000)
-    
-    # Fill login form (use environment variables in production!)
-    await page.fill("#email", "test@example.com")  # Never use real email
-    await page.fill("#password", "test_password")   # Never use real password
-    
-    # Handle "Remember Me" checkbox if present
-    try:
-        await page.uncheck("#remember_me")  # Don't remember on shared systems
-    except:
-        pass
-    
-    # Submit form
-    await page.click("button[type='submit']")
-    
-    # Wait for redirect after login
-    await page.wait_for_url("**/account/**", timeout=10000)
-    
-    return page
-"""
-}
-```
-
-#### 6. Extracting Structured Data (Wikipedia)
-
-```python
-# Safe example using Wikipedia
-hooks_code = {
-    "after_goto": """
-async def hook(page, context, url, response, **kwargs):
-    # Wait for Wikipedia content to load
-    await page.wait_for_selector("#content", timeout=5000)
-    return page
-""",
-    
-    "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # Extract structured data from Wikipedia infobox
-    metadata = await page.evaluate('''() => {
-        const infobox = document.querySelector('.infobox');
-        if (!infobox) return null;
-        
-        const data = {};
-        const rows = infobox.querySelectorAll('tr');
-        
-        rows.forEach(row => {
-            const header = row.querySelector('th');
-            const value = row.querySelector('td');
-            if (header && value) {
-                data[header.innerText.trim()] = value.innerText.trim();
-            }
-        });
-        
-        return data;
-    }''')
-    
-    if metadata:
-        console.log("Extracted metadata:", metadata)
-    
-    return page
-"""
-}
-
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": ["https://en.wikipedia.org/wiki/Python_(programming_language)"],
-    "hooks": {"code": hooks_code, "timeout": 20}
-})
-```
-
-### Security Best Practices
-
-> 🔒 **Critical Security Guidelines**:
-
-1. **Never Trust User Input**: If accepting hook code from users, always validate and sandbox it
-2. **Avoid Phishing Sites**: Never use hooks on suspicious or unverified websites
-3. **Protect Credentials**: 
-   - Never hardcode passwords, tokens, or API keys in hook code
-   - Use environment variables or secure secret management
-   - Rotate credentials regularly
-4. **Network Isolation**: Run the Docker container in an isolated network when testing
-5. **Audit Hook Code**: Always review hook code before execution
-6. **Limit Permissions**: Use the least privileged access needed
-7. **Monitor Execution**: Check hook execution logs for suspicious behavior
-8. **Timeout Protection**: Always set reasonable timeouts (default 30s)
-
-### Hook Response Information
-
-When hooks are used, the response includes detailed execution information:
-
-```json
-{
-  "success": true,
-  "results": [...],
-  "hooks": {
-    "status": {
-      "status": "success",  // or "partial" or "failed"
-      "attached_hooks": ["on_page_context_created", "before_retrieve_html"],
-      "validation_errors": [],
-      "successfully_attached": 2,
-      "failed_validation": 0
-    },
-    "execution_log": [
-      {
-        "hook_point": "on_page_context_created",
-        "status": "success",
-        "execution_time": 0.523,
-        "timestamp": 1234567890.123
-      }
-    ],
-    "errors": [],  // Any runtime errors
-    "summary": {
-      "total_executions": 2,
-      "successful": 2,
-      "failed": 0,
-      "timed_out": 0,
-      "success_rate": 100.0
-    }
-  }
-}
-```
-
-### Error Handling
-
-The hooks system is designed to be resilient:
-
-1. **Validation Errors**: Caught before execution (syntax errors, wrong parameters)
-2. **Runtime Errors**: Handled gracefully - crawl continues with original page object
-3. **Timeout Protection**: Hooks automatically terminated after timeout (configurable 1-120s)
-
-### Complete Example: Safe Multi-Hook Crawling
-
-```python
-import requests
-import json
-import os
-
-# Safe example using httpbin.org for testing
-hooks_code = {
-    "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Set viewport and test cookies
-    await page.set_viewport_size({"width": 1920, "height": 1080})
-    await context.add_cookies([
-        {"name": "test_cookie", "value": "test_value", "domain": ".httpbin.org", "path": "/"}
-    ])
-    
-    # Block unnecessary resources for httpbin
-    await context.route("**/*.{png,jpg,jpeg}", lambda route: route.abort())
-    return page
-""",
-    
-    "before_goto": """
-async def hook(page, context, url, **kwargs):
-    # Add custom headers for testing
-    await page.set_extra_http_headers({
-        "X-Test-Header": "crawl4ai-test",
-        "Accept-Language": "en-US,en;q=0.9"
-    })
-    print(f"[HOOK] Navigating to: {url}")
-    return page
-""",
-    
-    "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # Simple scroll for any lazy-loaded content
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-    await page.wait_for_timeout(1000)
-    return page
-"""
-}
-
-# Make the request to safe testing endpoints
-response = requests.post("http://localhost:11235/crawl", json={
-    "urls": [
-        "https://httpbin.org/html",
-        "https://httpbin.org/json"
-    ],
-    "hooks": {
-        "code": hooks_code,
-        "timeout": 30
-    },
-    "crawler_config": {
-        "cache_mode": "bypass"
-    }
-})
-
-# Check results
-if response.status_code == 200:
-    data = response.json()
-    
-    # Check hook execution
-    if data['hooks']['status']['status'] == 'success':
-        print(f"✅ All {len(data['hooks']['status']['attached_hooks'])} hooks executed successfully")
-        print(f"Execution stats: {data['hooks']['summary']}")
-    
-    # Process crawl results
-    for result in data['results']:
-        print(f"Crawled: {result['url']} - Success: {result['success']}")
-else:
-    print(f"Error: {response.status_code}")
-```
-
-> 💡 **Remember**: Always test your hooks on safe, known websites first before using them on production sites. Never crawl sites that you don't have permission to access or that might be malicious.
-
----
-
 ## Dockerfile Parameters
 
 You can customize the image build process using build arguments (`--build-arg`). These are typically used via `docker buildx build` or within the `docker-compose.yml` file.

tests/docker/test_hooks_client.py

@@ -1,372 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test client for demonstrating user-provided hooks in Crawl4AI Docker API
-"""
-
-import requests
-import json
-from typing import Dict, Any
-
-
-API_BASE_URL = "http://localhost:11234"  # Adjust if needed
-
-
-def test_hooks_info():
-    """Get information about available hooks"""
-    print("=" * 70)
-    print("Testing: GET /hooks/info")
-    print("=" * 70)
-    
-    response = requests.get(f"{API_BASE_URL}/hooks/info")
-    if response.status_code == 200:
-        data = response.json()
-        print("Available Hook Points:")
-        for hook, info in data['available_hooks'].items():
-            print(f"\n{hook}:")
-            print(f"  Parameters: {', '.join(info['parameters'])}")
-            print(f"  Description: {info['description']}")
-    else:
-        print(f"Error: {response.status_code}")
-        print(response.text)
-
-
-def test_basic_crawl_with_hooks():
-    """Test basic crawling with user-provided hooks"""
-    print("\n" + "=" * 70)
-    print("Testing: POST /crawl with hooks")
-    print("=" * 70)
-    
-    # Define hooks as Python code strings
-    hooks_code = {
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    print("Hook: Setting up page context")
-    # Block images to speed up crawling
-    await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
-    print("Hook: Images blocked")
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    print("Hook: Before retrieving HTML")
-    # Scroll to bottom to load lazy content
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-    await page.wait_for_timeout(1000)
-    print("Hook: Scrolled to bottom")
-    return page
-""",
-        
-        "before_goto": """
-async def hook(page, context, url, **kwargs):
-    print(f"Hook: About to navigate to {url}")
-    # Add custom headers
-    await page.set_extra_http_headers({
-        'X-Test-Header': 'crawl4ai-hooks-test'
-    })
-    return page
-"""
-    }
-    
-    # Create request payload
-    payload = {
-        "urls": ["https://httpbin.org/html"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 30
-        }
-    }
-    
-    print("Sending request with hooks...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("\n✅ Crawl successful!")
-        
-        # Check hooks status
-        if 'hooks' in data:
-            hooks_info = data['hooks']
-            print("\nHooks Execution Summary:")
-            print(f"  Status: {hooks_info['status']['status']}")
-            print(f"  Attached hooks: {', '.join(hooks_info['status']['attached_hooks'])}")
-            
-            if hooks_info['status']['validation_errors']:
-                print("\n⚠️ Validation Errors:")
-                for error in hooks_info['status']['validation_errors']:
-                    print(f"  - {error['hook_point']}: {error['error']}")
-            
-            if 'summary' in hooks_info:
-                summary = hooks_info['summary']
-                print(f"\nExecution Statistics:")
-                print(f"  Total executions: {summary['total_executions']}")
-                print(f"  Successful: {summary['successful']}")
-                print(f"  Failed: {summary['failed']}")
-                print(f"  Timed out: {summary['timed_out']}")
-                print(f"  Success rate: {summary['success_rate']:.1f}%")
-            
-            if hooks_info['execution_log']:
-                print("\nExecution Log:")
-                for log_entry in hooks_info['execution_log']:
-                    status_icon = "✅" if log_entry['status'] == 'success' else "❌"
-                    print(f"  {status_icon} {log_entry['hook_point']}: {log_entry['status']} ({log_entry.get('execution_time', 0):.2f}s)")
-            
-            if hooks_info['errors']:
-                print("\n❌ Hook Errors:")
-                for error in hooks_info['errors']:
-                    print(f"  - {error['hook_point']}: {error['error']}")
-        
-        # Show crawl results
-        if 'results' in data:
-            print(f"\nCrawled {len(data['results'])} URL(s)")
-            for result in data['results']:
-                print(f"  - {result['url']}: {'✅' if result['success'] else '❌'}")
-    
-    else:
-        print(f"❌ Error: {response.status_code}")
-        print(response.text)
-
-
-def test_invalid_hook():
-    """Test with an invalid hook to see error handling"""
-    print("\n" + "=" * 70)
-    print("Testing: Invalid hook handling")
-    print("=" * 70)
-    
-    # Intentionally broken hook
-    hooks_code = {
-        "on_page_context_created": """
-def hook(page, context):  # Missing async!
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # This will cause an error
-    await page.non_existent_method()
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/html"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 5
-        }
-    }
-    
-    print("Sending request with invalid hooks...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        
-        if 'hooks' in data:
-            hooks_info = data['hooks']
-            print(f"\nHooks Status: {hooks_info['status']['status']}")
-            
-            if hooks_info['status']['validation_errors']:
-                print("\n✅ Validation caught errors (as expected):")
-                for error in hooks_info['status']['validation_errors']:
-                    print(f"  - {error['hook_point']}: {error['error']}")
-            
-            if hooks_info['errors']:
-                print("\n✅ Runtime errors handled gracefully:")
-                for error in hooks_info['errors']:
-                    print(f"  - {error['hook_point']}: {error['error']}")
-            
-            # The crawl should still succeed despite hook errors
-            if data.get('success'):
-                print("\n✅ Crawl succeeded despite hook errors (error isolation working!)")
-    
-    else:
-        print(f"Error: {response.status_code}")
-        print(response.text)
-
-
-def test_authentication_hook():
-    """Test authentication using hooks"""
-    print("\n" + "=" * 70)
-    print("Testing: Authentication with hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "before_goto": """
-async def hook(page, context, url, **kwargs):
-    # For httpbin.org basic auth test, set Authorization header
-    import base64
-    
-    # httpbin.org/basic-auth/user/passwd expects username="user" and password="passwd"
-    credentials = base64.b64encode(b"user:passwd").decode('ascii')
-    
-    await page.set_extra_http_headers({
-        'Authorization': f'Basic {credentials}'
-    })
-    
-    print(f"Hook: Set Authorization header for {url}")
-    return page
-""",
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Example: Add cookies for session tracking
-    await context.add_cookies([
-        {
-            'name': 'session_id',
-            'value': 'test_session_123',
-            'domain': '.httpbin.org',
-            'path': '/',
-            'httpOnly': True,
-            'secure': True
-        }
-    ])
-    
-    print("Hook: Added session cookie")
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/basic-auth/user/passwd"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 30
-        }
-    }
-    
-    print("Sending request with authentication hook...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        if data.get('success'):
-            print("✅ Crawl with authentication hook successful")
-            
-            # Check if hooks executed
-            if 'hooks' in data:
-                hooks_info = data['hooks']
-                if hooks_info.get('summary', {}).get('successful', 0) > 0:
-                    print(f"✅ Authentication hooks executed: {hooks_info['summary']['successful']} successful")
-                
-                # Check for any hook errors
-                if hooks_info.get('errors'):
-                    print("⚠️ Hook errors:")
-                    for error in hooks_info['errors']:
-                        print(f"  - {error}")
-            
-            # Check if authentication worked by looking at the result
-            if 'results' in data and len(data['results']) > 0:
-                result = data['results'][0]
-                if result.get('success'):
-                    print("✅ Page crawled successfully (authentication worked!)")
-                    # httpbin.org/basic-auth returns JSON with authenticated=true when successful
-                    if 'authenticated' in str(result.get('html', '')):
-                        print("✅ Authentication confirmed in response content")
-                else:
-                    print(f"❌ Crawl failed: {result.get('error_message', 'Unknown error')}")
-        else:
-            print("❌ Request failed")
-            print(f"Response: {json.dumps(data, indent=2)}")
-    else:
-        print(f"❌ Error: {response.status_code}")
-        try:
-            error_data = response.json()
-            print(f"Error details: {json.dumps(error_data, indent=2)}")
-        except:
-            print(f"Error text: {response.text[:500]}")
-
-
-def test_streaming_with_hooks():
-    """Test streaming endpoint with hooks"""
-    print("\n" + "=" * 70)
-    print("Testing: POST /crawl/stream with hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    await page.evaluate("document.querySelectorAll('img').forEach(img => img.remove())")
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/html", "https://httpbin.org/json"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 10
-        }
-    }
-    
-    print("Sending streaming request with hooks...")
-    
-    with requests.post(f"{API_BASE_URL}/crawl/stream", json=payload, stream=True) as response:
-        if response.status_code == 200:
-            # Check headers for hooks status
-            hooks_status = response.headers.get('X-Hooks-Status')
-            if hooks_status:
-                print(f"Hooks Status (from header): {hooks_status}")
-            
-            print("\nStreaming results:")
-            for line in response.iter_lines():
-                if line:
-                    try:
-                        result = json.loads(line)
-                        if 'url' in result:
-                            print(f"  Received: {result['url']}")
-                        elif 'status' in result:
-                            print(f"  Stream status: {result['status']}")
-                    except json.JSONDecodeError:
-                        print(f"  Raw: {line.decode()}")
-        else:
-            print(f"Error: {response.status_code}")
-
-
-def test_basic_without_hooks():
-    """Test basic crawl without hooks"""
-    print("\n" + "=" * 70)
-    print("Testing: POST /crawl with no hooks")
-    print("=" * 70)
-
-    payload = {
-        "urls": ["https://httpbin.org/html", "https://httpbin.org/json"]
-    }
-
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    if response.status_code == 200:
-        data = response.json()
-        print(f"Response: {json.dumps(data, indent=2)}")
-    else:
-        print(f"Error: {response.status_code}")
-
-
-def main():
-    """Run all tests"""
-    print("🔧 Crawl4AI Docker API - Hooks Testing")
-    print("=" * 70)
-    
-    # Test 1: Get hooks information
-    # test_hooks_info()
-    
-    # Test 2: Basic crawl with hooks
-    # test_basic_crawl_with_hooks()
-    
-    # Test 3: Invalid hooks (error handling)
-    test_invalid_hook()
-    
-    # # Test 4: Authentication hook
-    # test_authentication_hook()
-    
-    # # Test 5: Streaming with hooks
-    # test_streaming_with_hooks()
-
-    # # Test 6: Basic crawl without hooks
-    # test_basic_without_hooks()
-
-    print("\n" + "=" * 70)
-    print("✅ All tests completed!")
-    print("=" * 70)
-
-
-if __name__ == "__main__":
-    main()

tests/docker/test_hooks_comprehensive.py

@@ -1,512 +0,0 @@
-#!/usr/bin/env python3
-"""
-Comprehensive test demonstrating all hook types from hooks_example.py
-adapted for the Docker API with real URLs
-"""
-
-import requests
-import json
-import time
-from typing import Dict, Any
-
-API_BASE_URL = "http://localhost:11234"
-
-
-def test_all_hooks_demo():
-    """Demonstrate all 8 hook types with practical examples"""
-    print("=" * 70)
-    print("Testing: All Hooks Comprehensive Demo")
-    print("=" * 70)
-    
-    hooks_code = {
-        "on_browser_created": """
-async def hook(browser, **kwargs):
-    # Hook called after browser is created
-    print("[HOOK] on_browser_created - Browser is ready!")
-    # Browser-level configurations would go here
-    return browser
-""",
-        
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    # Hook called after a new page and context are created
-    print("[HOOK] on_page_context_created - New page created!")
-    
-    # Set viewport size for consistent rendering
-    await page.set_viewport_size({"width": 1920, "height": 1080})
-    
-    # Add cookies for the session (using httpbin.org domain)
-    await context.add_cookies([
-        {
-            "name": "test_session",
-            "value": "abc123xyz",
-            "domain": ".httpbin.org",
-            "path": "/",
-            "httpOnly": True,
-            "secure": True
-        }
-    ])
-    
-    # Block ads and tracking scripts to speed up crawling
-    await context.route("**/*.{png,jpg,jpeg,gif,webp,svg}", lambda route: route.abort())
-    await context.route("**/analytics/*", lambda route: route.abort())
-    await context.route("**/ads/*", lambda route: route.abort())
-    
-    print("[HOOK] Viewport set, cookies added, and ads blocked")
-    return page
-""",
-        
-        "on_user_agent_updated": """
-async def hook(page, context, user_agent, **kwargs):
-    # Hook called when user agent is updated
-    print(f"[HOOK] on_user_agent_updated - User agent: {user_agent[:50]}...")
-    return page
-""",
-        
-        "before_goto": """
-async def hook(page, context, url, **kwargs):
-    # Hook called before navigating to each URL
-    print(f"[HOOK] before_goto - About to visit: {url}")
-    
-    # Add custom headers for the request
-    await page.set_extra_http_headers({
-        "X-Custom-Header": "crawl4ai-test",
-        "Accept-Language": "en-US,en;q=0.9",
-        "DNT": "1"
-    })
-    
-    return page
-""",
-        
-        "after_goto": """
-async def hook(page, context, url, response, **kwargs):
-    # Hook called after navigating to each URL
-    print(f"[HOOK] after_goto - Successfully loaded: {url}")
-    
-    # Wait a moment for dynamic content to load
-    await page.wait_for_timeout(1000)
-    
-    # Check if specific elements exist (with error handling)
-    try:
-        # For httpbin.org, wait for body element
-        await page.wait_for_selector("body", timeout=2000)
-        print("[HOOK] Body element found and loaded")
-    except:
-        print("[HOOK] Timeout waiting for body, continuing anyway")
-    
-    return page
-""",
-        
-        "on_execution_started": """
-async def hook(page, context, **kwargs):
-    # Hook called after custom JavaScript execution
-    print("[HOOK] on_execution_started - Custom JS executed!")
-    
-    # You could inject additional JavaScript here if needed
-    await page.evaluate("console.log('[INJECTED] Hook JS running');")
-    
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    # Hook called before retrieving the HTML content
-    print("[HOOK] before_retrieve_html - Preparing to get HTML")
-    
-    # Scroll to bottom to trigger lazy loading
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
-    await page.wait_for_timeout(500)
-    
-    # Scroll back to top
-    await page.evaluate("window.scrollTo(0, 0);")
-    await page.wait_for_timeout(500)
-    
-    # One more scroll to middle for good measure
-    await page.evaluate("window.scrollTo(0, document.body.scrollHeight / 2);")
-    
-    print("[HOOK] Scrolling completed for lazy-loaded content")
-    return page
-""",
-        
-        "before_return_html": """
-async def hook(page, context, html, **kwargs):
-    # Hook called before returning the HTML content
-    print(f"[HOOK] before_return_html - HTML length: {len(html)} characters")
-    
-    # Log some page metrics
-    metrics = await page.evaluate('''() => {
-        return {
-            images: document.images.length,
-            links: document.links.length,
-            scripts: document.scripts.length
-        }
-    }''')
-    
-    print(f"[HOOK] Page metrics - Images: {metrics['images']}, Links: {metrics['links']}, Scripts: {metrics['scripts']}")
-    
-    return page
-"""
-    }
-    
-    # Create request payload
-    payload = {
-        "urls": ["https://httpbin.org/html"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 30
-        },
-        "crawler_config": {
-            "js_code": "window.scrollTo(0, document.body.scrollHeight);",
-            "wait_for": "body",
-            "cache_mode": "bypass"
-        }
-    }
-    
-    print("\nSending request with all 8 hooks...")
-    start_time = time.time()
-    
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    elapsed_time = time.time() - start_time
-    print(f"Request completed in {elapsed_time:.2f} seconds")
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("\n✅ Request successful!")
-        
-        # Check hooks execution
-        if 'hooks' in data:
-            hooks_info = data['hooks']
-            print("\n📊 Hooks Execution Summary:")
-            print(f"  Status: {hooks_info['status']['status']}")
-            print(f"  Attached hooks: {len(hooks_info['status']['attached_hooks'])}")
-            
-            for hook_name in hooks_info['status']['attached_hooks']:
-                print(f"    ✓ {hook_name}")
-            
-            if 'summary' in hooks_info:
-                summary = hooks_info['summary']
-                print(f"\n📈 Execution Statistics:")
-                print(f"  Total executions: {summary['total_executions']}")
-                print(f"  Successful: {summary['successful']}")
-                print(f"  Failed: {summary['failed']}")
-                print(f"  Timed out: {summary['timed_out']}")
-                print(f"  Success rate: {summary['success_rate']:.1f}%")
-            
-            if hooks_info.get('execution_log'):
-                print(f"\n📝 Execution Log:")
-                for log_entry in hooks_info['execution_log']:
-                    status_icon = "✅" if log_entry['status'] == 'success' else "❌"
-                    exec_time = log_entry.get('execution_time', 0)
-                    print(f"  {status_icon} {log_entry['hook_point']}: {exec_time:.3f}s")
-        
-        # Check crawl results
-        if 'results' in data and len(data['results']) > 0:
-            print(f"\n📄 Crawl Results:")
-            for result in data['results']:
-                print(f"  URL: {result['url']}")
-                print(f"  Success: {result.get('success', False)}")
-                if result.get('html'):
-                    print(f"  HTML length: {len(result['html'])} characters")
-    
-    else:
-        print(f"❌ Error: {response.status_code}")
-        try:
-            error_data = response.json()
-            print(f"Error details: {json.dumps(error_data, indent=2)}")
-        except:
-            print(f"Error text: {response.text[:500]}")
-
-
-def test_authentication_flow():
-    """Test a complete authentication flow with multiple hooks"""
-    print("\n" + "=" * 70)
-    print("Testing: Authentication Flow with Multiple Hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Setting up authentication context")
-    
-    # Add authentication cookies
-    await context.add_cookies([
-        {
-            "name": "auth_token",
-            "value": "fake_jwt_token_here",
-            "domain": ".httpbin.org",
-            "path": "/",
-            "httpOnly": True,
-            "secure": True
-        }
-    ])
-    
-    # Set localStorage items (for SPA authentication)
-    await page.evaluate('''
-        localStorage.setItem('user_id', '12345');
-        localStorage.setItem('auth_time', new Date().toISOString());
-    ''')
-    
-    return page
-""",
-        
-        "before_goto": """
-async def hook(page, context, url, **kwargs):
-    print(f"[HOOK] Adding auth headers for {url}")
-    
-    # Add Authorization header
-    import base64
-    credentials = base64.b64encode(b"user:passwd").decode('ascii')
-    
-    await page.set_extra_http_headers({
-        'Authorization': f'Basic {credentials}',
-        'X-API-Key': 'test-api-key-123'
-    })
-    
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": [
-            "https://httpbin.org/basic-auth/user/passwd"
-        ],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 15
-        }
-    }
-    
-    print("\nTesting authentication with httpbin endpoints...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("✅ Authentication test completed")
-        
-        if 'results' in data:
-            for i, result in enumerate(data['results']):
-                print(f"\n  URL {i+1}: {result['url']}")
-                if result.get('success'):
-                    # Check for authentication success indicators
-                    html_content = result.get('html', '')
-                    if '"authenticated"' in html_content and 'true' in html_content:
-                        print("    ✅ Authentication successful! Basic auth worked.")
-                    else:
-                        print("    ⚠️ Page loaded but auth status unclear")
-                else:
-                    print(f"    ❌ Failed: {result.get('error_message', 'Unknown error')}")
-    else:
-        print(f"❌ Error: {response.status_code}")
-
-
-def test_performance_optimization_hooks():
-    """Test hooks for performance optimization"""
-    print("\n" + "=" * 70)
-    print("Testing: Performance Optimization Hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "on_page_context_created": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Optimizing page for performance")
-    
-    # Block resource-heavy content
-    await context.route("**/*.{png,jpg,jpeg,gif,webp,svg,ico}", lambda route: route.abort())
-    await context.route("**/*.{woff,woff2,ttf,otf}", lambda route: route.abort())
-    await context.route("**/*.{mp4,webm,ogg,mp3,wav}", lambda route: route.abort())
-    await context.route("**/googletagmanager.com/*", lambda route: route.abort())
-    await context.route("**/google-analytics.com/*", lambda route: route.abort())
-    await context.route("**/doubleclick.net/*", lambda route: route.abort())
-    await context.route("**/facebook.com/*", lambda route: route.abort())
-    
-    # Disable animations and transitions
-    await page.add_style_tag(content='''
-        *, *::before, *::after {
-            animation-duration: 0s !important;
-            animation-delay: 0s !important;
-            transition-duration: 0s !important;
-            transition-delay: 0s !important;
-        }
-    ''')
-    
-    print("[HOOK] Performance optimizations applied")
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Removing unnecessary elements before extraction")
-    
-    # Remove ads, popups, and other unnecessary elements
-    await page.evaluate('''() => {
-        // Remove common ad containers
-        const adSelectors = [
-            '.ad', '.ads', '.advertisement', '[id*="ad-"]', '[class*="ad-"]',
-            '.popup', '.modal', '.overlay', '.cookie-banner', '.newsletter-signup'
-        ];
-        
-        adSelectors.forEach(selector => {
-            document.querySelectorAll(selector).forEach(el => el.remove());
-        });
-        
-        // Remove script tags to clean up HTML
-        document.querySelectorAll('script').forEach(el => el.remove());
-        
-        // Remove style tags we don't need
-        document.querySelectorAll('style').forEach(el => el.remove());
-    }''')
-    
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/html"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 10
-        }
-    }
-    
-    print("\nTesting performance optimization hooks...")
-    start_time = time.time()
-    
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    elapsed_time = time.time() - start_time
-    print(f"Request completed in {elapsed_time:.2f} seconds")
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("✅ Performance optimization test completed")
-        
-        if 'results' in data and len(data['results']) > 0:
-            result = data['results'][0]
-            if result.get('html'):
-                print(f"  HTML size: {len(result['html'])} characters")
-                print("  Resources blocked, ads removed, animations disabled")
-    else:
-        print(f"❌ Error: {response.status_code}")
-
-
-def test_content_extraction_hooks():
-    """Test hooks for intelligent content extraction"""
-    print("\n" + "=" * 70)
-    print("Testing: Content Extraction Hooks")
-    print("=" * 70)
-    
-    hooks_code = {
-        "after_goto": """
-async def hook(page, context, url, response, **kwargs):
-    print(f"[HOOK] Waiting for dynamic content on {url}")
-    
-    # Wait for any lazy-loaded content
-    await page.wait_for_timeout(2000)
-    
-    # Trigger any "Load More" buttons
-    try:
-        load_more = await page.query_selector('[class*="load-more"], [class*="show-more"], button:has-text("Load More")')
-        if load_more:
-            await load_more.click()
-            await page.wait_for_timeout(1000)
-            print("[HOOK] Clicked 'Load More' button")
-    except:
-        pass
-    
-    return page
-""",
-        
-        "before_retrieve_html": """
-async def hook(page, context, **kwargs):
-    print("[HOOK] Extracting structured data")
-    
-    # Extract metadata
-    metadata = await page.evaluate('''() => {
-        const getMeta = (name) => {
-            const element = document.querySelector(`meta[name="${name}"], meta[property="${name}"]`);
-            return element ? element.getAttribute('content') : null;
-        };
-        
-        return {
-            title: document.title,
-            description: getMeta('description') || getMeta('og:description'),
-            author: getMeta('author'),
-            keywords: getMeta('keywords'),
-            ogTitle: getMeta('og:title'),
-            ogImage: getMeta('og:image'),
-            canonical: document.querySelector('link[rel="canonical"]')?.href,
-            jsonLd: Array.from(document.querySelectorAll('script[type="application/ld+json"]'))
-                .map(el => el.textContent).filter(Boolean)
-        };
-    }''')
-    
-    print(f"[HOOK] Extracted metadata: {json.dumps(metadata, indent=2)}")
-    
-    # Infinite scroll handling
-    for i in range(3):
-        await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
-        await page.wait_for_timeout(1000)
-        print(f"[HOOK] Scroll iteration {i+1}/3")
-    
-    return page
-"""
-    }
-    
-    payload = {
-        "urls": ["https://httpbin.org/html", "https://httpbin.org/json"],
-        "hooks": {
-            "code": hooks_code,
-            "timeout": 20
-        }
-    }
-    
-    print("\nTesting content extraction hooks...")
-    response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
-    
-    if response.status_code == 200:
-        data = response.json()
-        print("✅ Content extraction test completed")
-        
-        if 'hooks' in data and 'summary' in data['hooks']:
-            summary = data['hooks']['summary']
-            print(f"  Hooks executed: {summary['successful']}/{summary['total_executions']}")
-        
-        if 'results' in data:
-            for result in data['results']:
-                print(f"\n  URL: {result['url']}")
-                print(f"  Success: {result.get('success', False)}")
-    else:
-        print(f"❌ Error: {response.status_code}")
-
-
-def main():
-    """Run comprehensive hook tests"""
-    print("🔧 Crawl4AI Docker API - Comprehensive Hooks Testing")
-    print("Based on docs/examples/hooks_example.py")
-    print("=" * 70)
-    
-    tests = [
-        ("All Hooks Demo", test_all_hooks_demo),
-        ("Authentication Flow", test_authentication_flow),
-        ("Performance Optimization", test_performance_optimization_hooks),
-        ("Content Extraction", test_content_extraction_hooks),
-    ]
-    
-    for i, (name, test_func) in enumerate(tests, 1):
-        print(f"\n📌 Test {i}/{len(tests)}: {name}")
-        try:
-            test_func()
-            print(f"✅ {name} completed")
-        except Exception as e:
-            print(f"❌ {name} failed: {e}")
-            import traceback
-            traceback.print_exc()
-    
-    print("\n" + "=" * 70)
-    print("🎉 All comprehensive hook tests completed!")
-    print("=" * 70)
-
-
-if __name__ == "__main__":
-    main()