chore(version): bump version to 0.3.73

Major changes:
- Add browser takeover feature using CDP for authentic browsing
- Implement Docker support with full API server documentation
- Enhance Mockdown with tag preservation system
- Improve parallel crawling performance

This release focuses on authenticity and scalability, introducing the ability
to use users' own browsers while providing containerized deployment options.
Breaking changes include modified browser handling and API response structure.

See CHANGELOG.md for detailed migration guide.

.gitignore

@@ -207,4 +207,5 @@ git_issues.md
 
 .tests/
 .issues/
-.docs/
+.docs/
+.issues/

CHANGELOG.md

@@ -1,8 +1,104 @@
 # Changelog
 
+# CHANGELOG
+
+## [v0.3.73] - 2024-11-05
+
+### Major Features
+- **New Doctor Feature**
+  - Added comprehensive system diagnostics tool
+  - Available through package hub and CLI
+  - Provides automated troubleshooting and system health checks
+  - Includes detailed reporting of configuration issues
+
+- **Dockerized API Server**
+  - Released complete Docker implementation for API server
+  - Added comprehensive documentation for Docker deployment
+  - Implemented container communication protocols
+  - Added environment configuration guides
+
+- **Managed Browser Integration**
+  - Added support for user-controlled browser instances
+  - Implemented `ManagedBrowser` class for better browser lifecycle management
+  - Added ability to connect to existing Chrome DevTools Protocol (CDP) endpoints
+  - Introduced user data directory support for persistent browser profiles
+
+- **Enhanced HTML Processing**
+  - Added HTML tag preservation feature during markdown conversion
+  - Introduced configurable tag preservation system
+  - Improved pre-tag and code block handling
+  - Added support for nested preserved tags with attribute retention
+
+### Improvements
+- **Browser Handling**
+  - Added flag to ignore body visibility for problematic pages
+  - Improved browser process cleanup and management
+  - Enhanced temporary directory handling for browser profiles
+  - Added configurable browser launch arguments
+
+- **Database Management**
+  - Implemented connection pooling for better performance
+  - Added retry logic for database operations
+  - Improved error handling and logging
+  - Enhanced cleanup procedures for database connections
+
+- **Resource Management**
+  - Added memory and CPU monitoring
+  - Implemented dynamic task slot allocation based on system resources
+  - Added configurable cleanup intervals
+
+### Technical Improvements
+- **Code Structure**
+  - Moved version management to dedicated _version.py file
+  - Improved error handling throughout the codebase
+  - Enhanced logging system with better error reporting
+  - Reorganized core components for better maintainability
+
+### Bug Fixes
+- Fixed issues with browser process termination
+- Improved handling of connection timeouts
+- Enhanced error recovery in database operations
+- Fixed memory leaks in long-running processes
+
+### Dependencies
+- Updated Playwright to v1.47
+- Updated core dependencies with more flexible version constraints
+- Added new development dependencies for testing
+
+### Breaking Changes
+- Changed default browser handling behavior
+- Modified database connection management approach
+- Updated API response structure for better consistency
+
+## Migration Guide
+When upgrading to v0.3.73, be aware of the following changes:
+
+1. Docker Deployment:
+   - Review Docker documentation for new deployment options
+   - Update environment configurations as needed
+   - Check container communication settings
+
+2. If using custom browser management:
+   - Update browser initialization code to use new ManagedBrowser class
+   - Review browser cleanup procedures
+
+3. For database operations:
+   - Check custom database queries for compatibility with new connection pooling
+   - Update error handling to work with new retry logic
+
+4. Using the Doctor:
+   - Run doctor command for system diagnostics: `crawl4ai doctor`
+   - Review generated reports for potential issues
+   - Follow recommended fixes for any identified problems
+
+
+## [2024-11-04 - 13:21:42] Comprehensive Update of Crawl4AI Features and Dependencies
+This commit introduces several key enhancements, including improved error handling and robust database operations in `async_database.py`, which now features a connection pool and retry logic for better reliability. Updates to the README.md provide clearer instructions and a better user experience with links to documentation sections. The `.gitignore` file has been refined to include additional directories, while the async web crawler now utilizes a managed browser for more efficient crawling. Furthermore, multiple dependency updates and introduction of the `CustomHTML2Text` class enhance text extraction capabilities.
+
 ## [v0.3.73] - 2024-10-24
 
 ### Added
+- preserve_tags: Added support for preserving specific HTML tags during markdown conversion.
 - Smart overlay removal system in AsyncPlaywrightCrawlerStrategy:
   - Automatic removal of popups, modals, and cookie notices
   - Detection and removal of fixed/sticky position elements

Dockerfile

@@ -0,0 +1,121 @@
+# syntax=docker/dockerfile:1.4
+
+# Build arguments
+ARG PYTHON_VERSION=3.10
+
+# Base stage with system dependencies
+FROM python:${PYTHON_VERSION}-slim as base
+
+# Declare ARG variables again within the build stage
+ARG INSTALL_TYPE=all
+ARG ENABLE_GPU=false
+
+# Platform-specific labels
+LABEL maintainer="unclecode"
+LABEL description="Crawl4AI - Advanced Web Crawler with AI capabilities"
+LABEL version="1.0"
+
+# Environment setup
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1 \
+    PIP_DEFAULT_TIMEOUT=100 \
+    DEBIAN_FRONTEND=noninteractive
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    curl \
+    wget \
+    gnupg \
+    git \
+    cmake \
+    pkg-config \
+    python3-dev \
+    libjpeg-dev \
+    libpng-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Playwright system dependencies for Linux
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libglib2.0-0 \
+    libnss3 \
+    libnspr4 \
+    libatk1.0-0 \
+    libatk-bridge2.0-0 \
+    libcups2 \
+    libdrm2 \
+    libdbus-1-3 \
+    libxcb1 \
+    libxkbcommon0 \
+    libx11-6 \
+    libxcomposite1 \
+    libxdamage1 \
+    libxext6 \
+    libxfixes3 \
+    libxrandr2 \
+    libgbm1 \
+    libpango-1.0-0 \
+    libcairo2 \
+    libasound2 \
+    libatspi2.0-0 \
+    && rm -rf /var/lib/apt/lists/*
+
+# GPU support if enabled
+RUN if [ "$ENABLE_GPU" = "true" ] ; then \
+    apt-get update && apt-get install -y --no-install-recommends \
+    nvidia-cuda-toolkit \
+    && rm -rf /var/lib/apt/lists/* ; \
+    fi
+
+# Create and set working directory
+WORKDIR /app
+
+# Copy the entire project
+COPY . .
+
+# Install base requirements
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Install required library for FastAPI
+RUN pip install fastapi uvicorn psutil
+
+# Install ML dependencies first for better layer caching
+RUN if [ "$INSTALL_TYPE" = "all" ] ; then \
+        pip install --no-cache-dir \
+            torch \
+            torchvision \
+            torchaudio \
+            scikit-learn \
+            nltk \
+            transformers \
+            tokenizers && \
+        python -m nltk.downloader punkt stopwords ; \
+    fi
+
+# Install the package
+RUN if [ "$INSTALL_TYPE" = "all" ] ; then \
+        pip install -e ".[all]" && \
+        python -m crawl4ai.model_loader ; \
+    elif [ "$INSTALL_TYPE" = "torch" ] ; then \
+        pip install -e ".[torch]" ; \
+    elif [ "$INSTALL_TYPE" = "transformer" ] ; then \
+        pip install -e ".[transformer]" && \
+        python -m crawl4ai.model_loader ; \
+    else \
+        pip install -e "." ; \
+    fi
+
+# Install Playwright and browsers
+RUN playwright install
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+# Expose port
+EXPOSE 8000
+
+# Start the FastAPI server
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "11235"]

MISSION.md

@@ -0,0 +1,46 @@
+# Mission
+
+![Mission Diagram](./docs/assets/pitch-dark.svg)
+
+### 1. The Data Capitalization Opportunity
+
+We live in an unprecedented era of digital wealth creation. Every day, individuals and enterprises generate massive amounts of valuable digital footprints across various platforms, social media channels, messenger apps, and cloud services. While people can interact with their data within these platforms, there's an immense untapped opportunity to transform this data into true capital assets. Just as physical property became a foundational element of wealth creation, personal and enterprise data has the potential to become a new form of capital on balance sheets.
+
+For individuals, this represents an opportunity to transform their digital activities into valuable assets. For enterprises, their internal communications, team discussions, and collaborative documents contain rich insights that could be structured and valued as intellectual capital. This wealth of information represents an unprecedented opportunity for value creation in the digital age.
+
+### 2. The Potential of Authentic Data
+
+While synthetic data has played a crucial role in AI development, there's an enormous untapped potential in the authentic data generated by individuals and organizations. Every message, document, and interaction contains unique insights and patterns that could enhance AI development. The challenge isn't a lack of data - it's that most authentic human-generated data remains inaccessible for productive use.
+
+By enabling willing participation in data sharing, we can unlock this vast reservoir of authentic human knowledge. This represents an opportunity to enhance AI development with diverse, real-world data that reflects the full spectrum of human experience and knowledge.
+
+## Our Pathway to Data Democracy
+
+### 1. Open-Source Foundation
+
+Our first step is creating an open-source data extraction engine that empowers developers and innovators to build tools for data structuring and organization. This foundation ensures transparency, security, and community-driven development. By making these tools openly available, we enable the technical infrastructure needed for true data ownership and capitalization.
+
+### 2. Data Capitalization Platform
+
+Building on this open-source foundation, we're developing a platform that helps individuals and enterprises transform their digital footprints into structured, valuable assets. This platform will provide the tools and frameworks needed to organize, understand, and value personal and organizational data as true capital assets.
+
+### 3. Creating a Data Marketplace
+
+The final piece is establishing a marketplace where individuals and organizations can willingly share their data assets. This creates opportunities for:
+- Individuals to earn equity, revenue, or other forms of value from their data
+- Enterprises to access diverse, high-quality data for AI development
+- Researchers to work with authentic human-generated data
+- Startups to build innovative solutions using real-world data
+
+## Economic Vision: A Shared Data Economy
+
+We envision a future where data becomes a fundamental asset class in a thriving shared economy. This transformation will democratize AI development by enabling willing participation in data sharing, ensuring that the benefits of AI advancement flow back to data creators. Just as property rights revolutionized economic systems, establishing data as a capital asset will create new opportunities for wealth creation and economic participation.
+
+This shared data economy will:
+- Enable individuals to capitalize on their digital footprints
+- Create new revenue streams for data creators
+- Provide AI developers with access to diverse, authentic data
+- Foster innovation through broader access to real-world data
+- Ensure more equitable distribution of AI's economic benefits
+
+Our vision is to facilitate this transformation from the ground up - starting with open-source tools, progressing to data capitalization platforms, and ultimately creating a thriving marketplace where data becomes a true asset class in a shared economy. This approach ensures that the future of AI is built on a foundation of authentic human knowledge, with benefits flowing back to the individuals and organizations who create and share their valuable data.

README.md

@@ -1,6 +1,9 @@
-# Crawl4AI (Async Version) 🕷️🤖
+# 🔥🕷️ Crawl4AI: LLM Friendly Web Crawler & Scrapper
+
+<a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 
 [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
+![PyPI - Downloads](https://img.shields.io/pypi/dm/Crawl4AI)
 [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
 [![GitHub Issues](https://img.shields.io/github/issues/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/issues)
 [![GitHub Pull Requests](https://img.shields.io/github/issues-pr/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/pulls)
@@ -8,6 +11,14 @@
 
 Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
 
+## 🌟 Meet the Crawl4AI Assistant: Your Copilot for Crawling
+
+Use the [Crawl4AI GPT Assistant](https://tinyurl.com/crawl4ai-gpt) as your AI-powered copilot! With this assistant, you can:
+
+- 🧑‍💻 Generate code for complex crawling and extraction tasks
+- 💡 Get tailored support and examples
+- 📘 Learn Crawl4AI faster with step-by-step guidance
+
 ## New in 0.3.72 ✨
 
 - 📄 Fit markdown generation for extracting main article content.
@@ -19,7 +30,7 @@ Crawl4AI simplifies asynchronous web crawling and data extraction, making it acc
 
 ## Try it Now!
 
-✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1REChY6fXQf-EaVYLv0eHEWvzlYxGm0pd?usp=sharing)
+✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)
 
 ✨ Visit our [Documentation Website](https://crawl4ai.com/mkdocs/)
 
@@ -72,11 +83,13 @@ By default, this will install the asynchronous version of Crawl4AI, using Playwr
 👉 Note: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
 
 1. Through the command line:
+
    ```bash
    playwright install
    ```
 
 2. If the above doesn't work, try this more specific command:
+
    ```bash
    python -m playwright install chromium
    ```
@@ -103,9 +116,53 @@ pip install -e .
 
 ### Using Docker 🐳
 
-We're in the process of creating Docker images and pushing them to Docker Hub. This will provide an easy way to run Crawl4AI in a containerized environment. Stay tuned for updates!
+Crawl4AI is available as Docker images for easy deployment. You can either pull directly from Docker Hub (recommended) or build from the repository.
+
+#### Option 1: Docker Hub (Recommended)
+
+```bash
+# Pull and run from Docker Hub (choose one):
+docker pull unclecode/crawl4ai:basic    # Basic crawling features
+docker pull unclecode/crawl4ai:all      # Full installation (ML, LLM support)
+docker pull unclecode/crawl4ai:gpu      # GPU-enabled version
+
+# Run the container
+docker run -p 11235:11235 unclecode/crawl4ai:basic  # Replace 'basic' with your chosen version
+```
+
+#### Option 2: Build from Repository
+
+```bash
+# Clone the repository
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+
+# Build the image
+docker build -t crawl4ai:local \
+  --build-arg INSTALL_TYPE=basic \  # Options: basic, all
+  .
+
+# Run your local build
+docker run -p 11235:11235 crawl4ai:local
+```
+
+Quick test (works for both options):
+```python
+import requests
+
+# Submit a crawl job
+response = requests.post(
+    "http://localhost:11235/crawl",
+    json={"urls": "https://example.com", "priority": 10}
+)
+task_id = response.json()["task_id"]
+
+# Get results
+result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+For advanced configuration, environment variables, and usage examples, see our [Docker Deployment Guide](https://crawl4ai.com/mkdocs/basic/docker-deployment/).
 
-For more detailed installation instructions and options, please refer to our [Installation Guide](https://crawl4ai.com/mkdocs/installation).
 
 ## Quick Start 🚀
 
@@ -235,7 +292,7 @@ if __name__ == "__main__":
     asyncio.run(extract_news_teasers())
 ```
 
-For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/advanced_jsoncss_extraction.md) section in the documentation.
+For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/extraction/css-advanced/) section in the documentation.
 
 ### Extracting Structured Data with OpenAI
 
@@ -338,7 +395,8 @@ if __name__ == "__main__":
 
 This example demonstrates Crawl4AI's ability to handle complex scenarios where content is loaded asynchronously. It crawls multiple pages of GitHub commits, executing JavaScript to load new content and using custom hooks to ensure data is loaded before proceeding.
 
-For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/session_based_crawling.md) section in the documentation.
+For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/tutorial/episode_12_Session-Based_Crawling_for_Dynamic_Websites/) section in the documentation.
+</details>
 
 
 ## Speed Comparison 🚀
@@ -347,7 +405,7 @@ Crawl4AI is designed with speed as a primary focus. Our goal is to provide the f
 
 We've conducted a speed comparison between Crawl4AI and Firecrawl, a paid service. The results demonstrate Crawl4AI's superior performance:
 
-```
+```bash
 Firecrawl:
 Time taken: 7.02 seconds
 Content length: 42074 characters
@@ -365,6 +423,7 @@ Images found: 89
 ```
 
 As you can see, Crawl4AI outperforms Firecrawl significantly:
+
 - Simple crawl: Crawl4AI is over 4 times faster than Firecrawl.
 - With JavaScript execution: Even when executing JavaScript to load more content (doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.
 
@@ -392,6 +451,34 @@ For questions, suggestions, or feedback, feel free to reach out:
 
 Happy Crawling! 🕸️🚀
 
+
+# Mission
+
+Our mission is to unlock the untapped potential of personal and enterprise data in the digital age. In today's world, individuals and organizations generate vast amounts of valuable digital footprints, yet this data remains largely uncapitalized as a true asset. 
+
+Our open-source solution empowers developers and innovators to build tools for data extraction and structuring, laying the foundation for a new era of data ownership. By transforming personal and enterprise data into structured, tradeable assets, we're creating opportunities for individuals to capitalize on their digital footprints and for organizations to unlock the value of their collective knowledge.
+
+This democratization of data represents the first step toward a shared data economy, where willing participation in data sharing drives AI advancement while ensuring the benefits flow back to data creators. Through this approach, we're building a future where AI development is powered by authentic human knowledge rather than synthetic alternatives.
+
+![Mission Diagram](./docs/assets/pitch-dark.svg)
+
+For a detailed exploration of our vision, opportunities, and pathway forward, please see our [full mission statement](./MISSION.md).
+
+## Key Opportunities
+
+- **Data Capitalization**: Transform digital footprints into valuable assets that can appear on personal and enterprise balance sheets
+- **Authentic Data**: Unlock the vast reservoir of real human insights and knowledge for AI advancement
+- **Shared Economy**: Create new value streams where data creators directly benefit from their contributions
+
+## Development Pathway
+
+1. **Open-Source Foundation**: Building transparent, community-driven data extraction tools
+2. **Data Capitalization Platform**: Creating tools to structure and value digital assets
+3. **Shared Data Marketplace**: Establishing an economic platform for ethical data exchange
+
+For a detailed exploration of our vision, challenges, and solutions, please see our [full mission statement](./MISSION.md).
+
+
 ## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)

crawl4ai/__init__.py

@@ -2,8 +2,8 @@
 
 from .async_webcrawler import AsyncWebCrawler
 from .models import CrawlResult
-
-__version__ = "0.3.72"
+from ._version import __version__
+# __version__ = "0.3.73"
 
 __all__ = [
     "AsyncWebCrawler",

crawl4ai/_version.py

@@ -0,0 +1,2 @@
+# crawl4ai/_version.py
+__version__ = "0.3.73"

crawl4ai/async_crawler_strategy copy.py

@@ -1,558 +0,0 @@
-import asyncio
-import base64
-import time
-from abc import ABC, abstractmethod
-from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
-from playwright.async_api import async_playwright, Page, Browser, Error
-from io import BytesIO
-from PIL import Image, ImageDraw, ImageFont
-from pathlib import Path
-from playwright.async_api import ProxySettings
-from pydantic import BaseModel
-import hashlib
-import json
-import uuid
-from playwright_stealth import stealth_async
-
-class AsyncCrawlResponse(BaseModel):
-    html: str
-    response_headers: Dict[str, str]
-    status_code: int
-    screenshot: Optional[str] = None
-    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
-
-    class Config:
-        arbitrary_types_allowed = True
-
-class AsyncCrawlerStrategy(ABC):
-    @abstractmethod
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        pass
-    
-    @abstractmethod
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        pass
-    
-    @abstractmethod
-    async def take_screenshot(self, url: str) -> str:
-        pass
-    
-    @abstractmethod
-    def update_user_agent(self, user_agent: str):
-        pass
-    
-    @abstractmethod
-    def set_hook(self, hook_type: str, hook: Callable):
-        pass
-
-class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
-    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
-        self.use_cached_html = use_cached_html
-        self.user_agent = kwargs.get(
-            "user_agent",
-            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
-            "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
-        )
-        self.proxy = kwargs.get("proxy")
-        self.headless = kwargs.get("headless", True)
-        self.browser_type = kwargs.get("browser_type", "chromium")
-        self.headers = kwargs.get("headers", {})
-        self.sessions = {}
-        self.session_ttl = 1800 
-        self.js_code = js_code
-        self.verbose = kwargs.get("verbose", False)
-        self.playwright = None
-        self.browser = None
-        self.hooks = {
-            'on_browser_created': None,
-            'on_user_agent_updated': None,
-            'on_execution_started': None,
-            'before_goto': None,
-            'after_goto': None,
-            'before_return_html': None,
-            'before_retrieve_html': None
-        }
-
-    async def __aenter__(self):
-        await self.start()
-        return self
-
-    async def __aexit__(self, exc_type, exc_val, exc_tb):
-        await self.close()
-
-    async def start(self):
-        if self.playwright is None:
-            self.playwright = await async_playwright().start()
-        if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-                
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
-            else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
-
-            await self.execute_hook('on_browser_created', self.browser)
-
-    async def close(self):
-        if self.browser:
-            await self.browser.close()
-            self.browser = None
-        if self.playwright:
-            await self.playwright.stop()
-            self.playwright = None
-
-    def __del__(self):
-        if self.browser or self.playwright:
-            asyncio.get_event_loop().run_until_complete(self.close())
-
-    def set_hook(self, hook_type: str, hook: Callable):
-        if hook_type in self.hooks:
-            self.hooks[hook_type] = hook
-        else:
-            raise ValueError(f"Invalid hook type: {hook_type}")
-
-    async def execute_hook(self, hook_type: str, *args):
-        hook = self.hooks.get(hook_type)
-        if hook:
-            if asyncio.iscoroutinefunction(hook):
-                return await hook(*args)
-            else:
-                return hook(*args)
-        return args[0] if args else None
-
-    def update_user_agent(self, user_agent: str):
-        self.user_agent = user_agent
-
-    def set_custom_headers(self, headers: Dict[str, str]):
-        self.headers = headers
-
-    async def kill_session(self, session_id: str):
-        if session_id in self.sessions:
-            context, page, _ = self.sessions[session_id]
-            await page.close()
-            await context.close()
-            del self.sessions[session_id]
-
-    def _cleanup_expired_sessions(self):
-        current_time = time.time()
-        expired_sessions = [
-            sid for sid, (_, _, last_used) in self.sessions.items() 
-            if current_time - last_used > self.session_ttl
-        ]
-        for sid in expired_sessions:
-            asyncio.create_task(self.kill_session(sid))
-            
-    async def smart_wait(self, page: Page, wait_for: str, timeout: float = 30000):
-        wait_for = wait_for.strip()
-        
-        if wait_for.startswith('js:'):
-            # Explicitly specified JavaScript
-            js_code = wait_for[3:].strip()
-            return await self.csp_compliant_wait(page, js_code, timeout)
-        elif wait_for.startswith('css:'):
-            # Explicitly specified CSS selector
-            css_selector = wait_for[4:].strip()
-            try:
-                await page.wait_for_selector(css_selector, timeout=timeout)
-            except Error as e:
-                if 'Timeout' in str(e):
-                    raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{css_selector}'")
-                else:
-                    raise ValueError(f"Invalid CSS selector: '{css_selector}'")
-        else:
-            # Auto-detect based on content
-            if wait_for.startswith('()') or wait_for.startswith('function'):
-                # It's likely a JavaScript function
-                return await self.csp_compliant_wait(page, wait_for, timeout)
-            else:
-                # Assume it's a CSS selector first
-                try:
-                    await page.wait_for_selector(wait_for, timeout=timeout)
-                except Error as e:
-                    if 'Timeout' in str(e):
-                        raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{wait_for}'")
-                    else:
-                        # If it's not a timeout error, it might be an invalid selector
-                        # Let's try to evaluate it as a JavaScript function as a fallback
-                        try:
-                            return await self.csp_compliant_wait(page, f"() => {{{wait_for}}}", timeout)
-                        except Error:
-                            raise ValueError(f"Invalid wait_for parameter: '{wait_for}'. "
-                                             "It should be either a valid CSS selector, a JavaScript function, "
-                                             "or explicitly prefixed with 'js:' or 'css:'.")
-    
-    async def csp_compliant_wait(self, page: Page, user_wait_function: str, timeout: float = 30000):
-        wrapper_js = f"""
-        async () => {{
-            const userFunction = {user_wait_function};
-            const startTime = Date.now();
-            while (true) {{
-                if (await userFunction()) {{
-                    return true;
-                }}
-                if (Date.now() - startTime > {timeout}) {{
-                    throw new Error('Timeout waiting for condition');
-                }}
-                await new Promise(resolve => setTimeout(resolve, 100));
-            }}
-        }}
-        """
-        
-        try:
-            await page.evaluate(wrapper_js)
-        except TimeoutError:
-            raise TimeoutError(f"Timeout after {timeout}ms waiting for condition")
-        except Exception as e:
-            raise RuntimeError(f"Error in wait condition: {str(e)}")
-
-    async def process_iframes(self, page):
-        # Find all iframes
-        iframes = await page.query_selector_all('iframe')
-        
-        for i, iframe in enumerate(iframes):
-            try:
-                # Add a unique identifier to the iframe
-                await iframe.evaluate(f'(element) => element.id = "iframe-{i}"')
-                
-                # Get the frame associated with this iframe
-                frame = await iframe.content_frame()
-                
-                if frame:
-                    # Wait for the frame to load
-                    await frame.wait_for_load_state('load', timeout=30000)  # 30 seconds timeout
-                    
-                    # Extract the content of the iframe's body
-                    iframe_content = await frame.evaluate('() => document.body.innerHTML')
-                    
-                    # Generate a unique class name for this iframe
-                    class_name = f'extracted-iframe-content-{i}'
-                    
-                    # Replace the iframe with a div containing the extracted content
-                    _iframe = iframe_content.replace('`', '\\`')
-                    await page.evaluate(f"""
-                        () => {{
-                            const iframe = document.getElementById('iframe-{i}');
-                            const div = document.createElement('div');
-                            div.innerHTML = `{_iframe}`;
-                            div.className = '{class_name}';
-                            iframe.replaceWith(div);
-                        }}
-                    """)
-                else:
-                    print(f"Warning: Could not access content frame for iframe {i}")
-            except Exception as e:
-                print(f"Error processing iframe {i}: {str(e)}")
-
-        # Return the page object
-        return page  
-    
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        response_headers = {}
-        status_code = None
-        
-        self._cleanup_expired_sessions()
-        session_id = kwargs.get("session_id")
-        if session_id:
-            context, page, _ = self.sessions.get(session_id, (None, None, None))
-            if not context:
-                context = await self.browser.new_context(
-                    user_agent=self.user_agent,
-                    viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None
-                )
-                await context.set_extra_http_headers(self.headers)
-                page = await context.new_page()
-                self.sessions[session_id] = (context, page, time.time())
-        else:
-            context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
-            )
-            await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
-            page = await context.new_page()
-
-        try:
-            if self.verbose:
-                print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                if os.path.exists(cache_file_path):
-                    html = ""
-                    with open(cache_file_path, "r") as f:
-                        html = f.read()
-                    # retrieve response headers and status code from cache
-                    with open(cache_file_path + ".meta", "r") as f:
-                        meta = json.load(f)
-                        response_headers = meta.get("response_headers", {})
-                        status_code = meta.get("status_code")
-                    response = AsyncCrawlResponse(
-                        html=html, response_headers=response_headers, status_code=status_code
-                    )
-                    return response
-
-            if not kwargs.get("js_only", False):
-                await self.execute_hook('before_goto', page)
-                
-                response = await page.goto("about:blank")
-                await stealth_async(page)
-                response = await page.goto(
-                    url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000)
-                )
-                
-                # await stealth_async(page)
-                # response = await page.goto("about:blank")
-                # await stealth_async(page)
-                # await page.evaluate(f"window.location.href = '{url}'")
-                
-                await self.execute_hook('after_goto', page)
-                
-                # Get status code and headers
-                status_code = response.status
-                response_headers = response.headers
-            else:
-                status_code = 200
-                response_headers = {}
-
-            await page.wait_for_selector('body')
-            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-
-            js_code = kwargs.get("js_code", kwargs.get("js", self.js_code))
-            if js_code:
-                if isinstance(js_code, str):
-                    await page.evaluate(js_code)
-                elif isinstance(js_code, list):
-                    for js in js_code:
-                        await page.evaluate(js)
-                
-                await page.wait_for_load_state('networkidle')
-                # Check for on execution event
-                await self.execute_hook('on_execution_started', page)
-                
-            if kwargs.get("simulate_user", False):
-                # Simulate user interactions
-                await page.mouse.move(100, 100)
-                await page.mouse.down()
-                await page.mouse.up()
-                await page.keyboard.press('ArrowDown')
-
-            # Handle the wait_for parameter
-            wait_for = kwargs.get("wait_for")
-            if wait_for:
-                try:
-                    await self.smart_wait(page, wait_for, timeout=kwargs.get("page_timeout", 60000))
-                except Exception as e:
-                    raise RuntimeError(f"Wait condition failed: {str(e)}")
-
-
-            
-            # Update image dimensions
-            update_image_dimensions_js = """
-            () => {
-                return new Promise((resolve) => {
-                    const filterImage = (img) => {
-                        // Filter out images that are too small
-                        if (img.width < 100 && img.height < 100) return false;
-                        
-                        // Filter out images that are not visible
-                        const rect = img.getBoundingClientRect();
-                        if (rect.width === 0 || rect.height === 0) return false;
-                        
-                        // Filter out images with certain class names (e.g., icons, thumbnails)
-                        if (img.classList.contains('icon') || img.classList.contains('thumbnail')) return false;
-                        
-                        // Filter out images with certain patterns in their src (e.g., placeholder images)
-                        if (img.src.includes('placeholder') || img.src.includes('icon')) return false;
-                        
-                        return true;
-                    };
-
-                    const images = Array.from(document.querySelectorAll('img')).filter(filterImage);
-                    let imagesLeft = images.length;
-                    
-                    if (imagesLeft === 0) {
-                        resolve();
-                        return;
-                    }
-
-                    const checkImage = (img) => {
-                        if (img.complete && img.naturalWidth !== 0) {
-                            img.setAttribute('width', img.naturalWidth);
-                            img.setAttribute('height', img.naturalHeight);
-                            imagesLeft--;
-                            if (imagesLeft === 0) resolve();
-                        }
-                    };
-
-                    images.forEach(img => {
-                        checkImage(img);
-                        if (!img.complete) {
-                            img.onload = () => {
-                                checkImage(img);
-                            };
-                            img.onerror = () => {
-                                imagesLeft--;
-                                if (imagesLeft === 0) resolve();
-                            };
-                        }
-                    });
-
-                    // Fallback timeout of 5 seconds
-                    setTimeout(() => resolve(), 5000);
-                });
-            }
-            """
-            await page.evaluate(update_image_dimensions_js)
-
-            # Wait a bit for any onload events to complete
-            await page.wait_for_timeout(100)
-
-            # Process iframes
-            if kwargs.get("process_iframes", False):
-                page = await self.process_iframes(page)
-            
-            await self.execute_hook('before_retrieve_html', page)
-            # Check if delay_before_return_html is set then wait for that time
-            delay_before_return_html = kwargs.get("delay_before_return_html")
-            if delay_before_return_html:
-                await asyncio.sleep(delay_before_return_html)
-                
-            html = await page.content()
-            await self.execute_hook('before_return_html', page, html)
-            
-            # Check if kwargs has screenshot=True then take screenshot
-            screenshot_data = None
-            if kwargs.get("screenshot"):
-                screenshot_data = await self.take_screenshot(url)            
-
-            if self.verbose:
-                print(f"[LOG] ✅ Crawled {url} successfully!")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                with open(cache_file_path, "w", encoding="utf-8") as f:
-                    f.write(html)
-                # store response headers and status code in cache
-                with open(cache_file_path + ".meta", "w", encoding="utf-8") as f:
-                    json.dump({
-                        "response_headers": response_headers,
-                        "status_code": status_code
-                    }, f)
-
-            async def get_delayed_content(delay: float = 5.0) -> str:
-                if self.verbose:
-                    print(f"[LOG] Waiting for {delay} seconds before retrieving content for {url}")
-                await asyncio.sleep(delay)
-                return await page.content()
-                
-            response = AsyncCrawlResponse(
-                html=html, 
-                response_headers=response_headers, 
-                status_code=status_code,
-                screenshot=screenshot_data,
-                get_delayed_content=get_delayed_content
-            )
-            return response
-        except Error as e:
-            raise Error(f"Failed to crawl {url}: {str(e)}")
-        finally:
-            if not session_id:
-                await page.close()
-                await context.close()
-
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        semaphore_count = kwargs.get('semaphore_count', 5)  # Adjust as needed
-        semaphore = asyncio.Semaphore(semaphore_count)
-
-        async def crawl_with_semaphore(url):
-            async with semaphore:
-                return await self.crawl(url, **kwargs)
-
-        tasks = [crawl_with_semaphore(url) for url in urls]
-        results = await asyncio.gather(*tasks, return_exceptions=True)
-        return [result if not isinstance(result, Exception) else str(result) for result in results]
-
-    async def take_screenshot(self, url: str, wait_time=1000) -> str:
-        async with await self.browser.new_context(user_agent=self.user_agent) as context:
-            page = await context.new_page()
-            try:
-                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
-                # Wait for a specified time (default is 1 second)
-                await page.wait_for_timeout(wait_time)
-                screenshot = await page.screenshot(full_page=True)
-                return base64.b64encode(screenshot).decode('utf-8')
-            except Exception as e:
-                error_message = f"Failed to take screenshot: {str(e)}"
-                print(error_message)
-
-                # Generate an error image
-                img = Image.new('RGB', (800, 600), color='black')
-                draw = ImageDraw.Draw(img)
-                font = ImageFont.load_default()
-                draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
-                
-                buffered = BytesIO()
-                img.save(buffered, format="JPEG")
-                return base64.b64encode(buffered.getvalue()).decode('utf-8')
-            finally:
-                await page.close()
-

crawl4ai/async_crawler_strategy.py

@@ -3,7 +3,8 @@ import base64
 import time
 from abc import ABC, abstractmethod
 from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
+import os, sys, shutil
+import tempfile, subprocess
 from playwright.async_api import async_playwright, Page, Browser, Error
 from io import BytesIO
 from PIL import Image, ImageDraw, ImageFont
@@ -13,6 +14,7 @@ from pydantic import BaseModel
 import hashlib
 import json
 import uuid
+
 from playwright_stealth import StealthConfig, stealth_async
 
 stealth_config = StealthConfig(
@@ -31,6 +33,106 @@ stealth_config = StealthConfig(
 )
 
 
+class ManagedBrowser:
+    def __init__(self, browser_type: str = "chromium", user_data_dir: Optional[str] = None, headless: bool = False):
+        self.browser_type = browser_type
+        self.user_data_dir = user_data_dir
+        self.headless = headless
+        self.browser_process = None
+        self.temp_dir = None
+        self.debugging_port = 9222
+
+    async def start(self) -> str:
+        """
+        Starts the browser process and returns the CDP endpoint URL.
+        If user_data_dir is not provided, creates a temporary directory.
+        """
+        
+        # Create temp dir if needed
+        if not self.user_data_dir:
+            self.temp_dir = tempfile.mkdtemp(prefix="browser-profile-")
+            self.user_data_dir = self.temp_dir
+
+        # Get browser path and args based on OS and browser type
+        browser_path = self._get_browser_path()
+        args = self._get_browser_args()
+
+        # Start browser process
+        try:
+            self.browser_process = subprocess.Popen(
+                args,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE
+            )
+            await asyncio.sleep(2)  # Give browser time to start
+            return f"http://localhost:{self.debugging_port}"
+        except Exception as e:
+            await self.cleanup()
+            raise Exception(f"Failed to start browser: {e}")
+
+    def _get_browser_path(self) -> str:
+        """Returns the browser executable path based on OS and browser type"""
+        if sys.platform == "darwin":  # macOS
+            paths = {
+                "chromium": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
+                "firefox": "/Applications/Firefox.app/Contents/MacOS/firefox",
+                "webkit": "/Applications/Safari.app/Contents/MacOS/Safari"
+            }
+        elif sys.platform == "win32":  # Windows
+            paths = {
+                "chromium": "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
+                "firefox": "C:\\Program Files\\Mozilla Firefox\\firefox.exe",
+                "webkit": None  # WebKit not supported on Windows
+            }
+        else:  # Linux
+            paths = {
+                "chromium": "google-chrome",
+                "firefox": "firefox",
+                "webkit": None  # WebKit not supported on Linux
+            }
+        
+        return paths.get(self.browser_type)
+
+    def _get_browser_args(self) -> List[str]:
+        """Returns browser-specific command line arguments"""
+        base_args = [self._get_browser_path()]
+        
+        if self.browser_type == "chromium":
+            args = [
+                f"--remote-debugging-port={self.debugging_port}",
+                f"--user-data-dir={self.user_data_dir}",
+            ]
+            if self.headless:
+                args.append("--headless=new")
+        elif self.browser_type == "firefox":
+            args = [
+                "--remote-debugging-port", str(self.debugging_port),
+                "--profile", self.user_data_dir,
+            ]
+            if self.headless:
+                args.append("--headless")
+        else:
+            raise NotImplementedError(f"Browser type {self.browser_type} not supported")
+            
+        return base_args + args
+
+    async def cleanup(self):
+        """Cleanup browser process and temporary directory"""
+        if self.browser_process:
+            try:
+                self.browser_process.terminate()
+                await asyncio.sleep(1)
+                if self.browser_process.poll() is None:
+                    self.browser_process.kill()
+            except Exception as e:
+                print(f"Error terminating browser: {e}")
+
+        if self.temp_dir and os.path.exists(self.temp_dir):
+            try:
+                shutil.rmtree(self.temp_dir)
+            except Exception as e:
+                print(f"Error removing temporary directory: {e}")
+
 class AsyncCrawlResponse(BaseModel):
     html: str
     response_headers: Dict[str, str]
@@ -82,6 +184,9 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         self.playwright = None
         self.browser = None
         self.sleep_on_close = kwargs.get("sleep_on_close", False)
+        self.use_managed_browser = kwargs.get("use_managed_browser", False)
+        self.user_data_dir = kwargs.get("user_data_dir", None)
+        self.managed_browser = None
         self.hooks = {
             'on_browser_created': None,
             'on_user_agent_updated': None,
@@ -103,36 +208,46 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         if self.playwright is None:
             self.playwright = await async_playwright().start()
         if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-            elif self.proxy_config:
-                proxy_settings = ProxySettings(server=self.proxy_config.get("server"), username=self.proxy_config.get("username"), password=self.proxy_config.get("password"))
-                browser_args["proxy"] = proxy_settings
-                
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
+            if self.use_managed_browser:
+                # Use managed browser approach
+                self.managed_browser = ManagedBrowser(
+                    browser_type=self.browser_type,
+                    user_data_dir=self.user_data_dir,
+                    headless=self.headless
+                )
+                cdp_url = await self.managed_browser.start()
+                self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)
             else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
+                browser_args = {
+                    "headless": self.headless,
+                    "args": [
+                        "--disable-gpu",
+                        "--no-sandbox",
+                        "--disable-dev-shm-usage",
+                        "--disable-blink-features=AutomationControlled",
+                        "--disable-infobars",
+                        "--window-position=0,0",
+                        "--ignore-certificate-errors",
+                        "--ignore-certificate-errors-spki-list",
+                        # "--headless=new",  # Use the new headless mode
+                    ]
+                }
+                
+                # Add proxy settings if a proxy is specified
+                if self.proxy:
+                    proxy_settings = ProxySettings(server=self.proxy)
+                    browser_args["proxy"] = proxy_settings
+                elif self.proxy_config:
+                    proxy_settings = ProxySettings(server=self.proxy_config.get("server"), username=self.proxy_config.get("username"), password=self.proxy_config.get("password"))
+                    browser_args["proxy"] = proxy_settings
+                    
+                # Select the appropriate browser based on the browser_type
+                if self.browser_type == "firefox":
+                    self.browser = await self.playwright.firefox.launch(**browser_args)
+                elif self.browser_type == "webkit":
+                    self.browser = await self.playwright.webkit.launch(**browser_args)
+                else:
+                    self.browser = await self.playwright.chromium.launch(**browser_args)
 
             await self.execute_hook('on_browser_created', self.browser)
 
@@ -142,6 +257,9 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         if self.browser:
             await self.browser.close()
             self.browser = None
+        if self.managed_browser:
+            await self.managed_browser.cleanup()
+            self.managed_browser = None
         if self.playwright:
             await self.playwright.stop()
             self.playwright = None
@@ -399,7 +517,48 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 status_code = 200
                 response_headers = {}
 
-            await page.wait_for_selector('body')
+            # Replace the current wait_for_selector line with this more robust check:
+            try:
+                # First wait for body to exist, regardless of visibility
+                await page.wait_for_selector('body', state='attached', timeout=30000)
+                
+                # Then wait for it to become visible by checking CSS
+                await page.wait_for_function("""
+                    () => {
+                        const body = document.body;
+                        const style = window.getComputedStyle(body);
+                        return style.display !== 'none' && 
+                            style.visibility !== 'hidden' && 
+                            style.opacity !== '0';
+                    }
+                """, timeout=30000)
+                
+            except Error as e:
+                # If waiting fails, let's try to diagnose the issue
+                visibility_info = await page.evaluate("""
+                    () => {
+                        const body = document.body;
+                        const style = window.getComputedStyle(body);
+                        return {
+                            display: style.display,
+                            visibility: style.visibility,
+                            opacity: style.opacity,
+                            hasContent: body.innerHTML.length,
+                            classList: Array.from(body.classList)
+                        }
+                    }
+                """)
+                
+                if self.verbose:
+                    print(f"Body visibility debug info: {visibility_info}")
+                
+                # Even if body is hidden, we might still want to proceed
+                if kwargs.get('ignore_body_visibility', True):
+                    if self.verbose:
+                        print("Proceeding despite hidden body...")
+                    pass
+                else:
+                    raise Error(f"Body element is hidden: {visibility_info}")
             
             await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
 

crawl4ai/async_database.py

@@ -2,18 +2,82 @@ import os
 from pathlib import Path
 import aiosqlite
 import asyncio
-from typing import Optional, Tuple
+from typing import Optional, Tuple, Dict
+from contextlib import asynccontextmanager
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
 
 DB_PATH = os.path.join(Path.home(), ".crawl4ai")
 os.makedirs(DB_PATH, exist_ok=True)
 DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")
 
 class AsyncDatabaseManager:
-    def __init__(self):
+    def __init__(self, pool_size: int = 10, max_retries: int = 3):
         self.db_path = DB_PATH
+        self.pool_size = pool_size
+        self.max_retries = max_retries
+        self.connection_pool: Dict[int, aiosqlite.Connection] = {}
+        self.pool_lock = asyncio.Lock()
+        self.connection_semaphore = asyncio.Semaphore(pool_size)
+        
+    async def initialize(self):
+        """Initialize the database and connection pool"""
+        await self.ainit_db()
+        
+    async def cleanup(self):
+        """Cleanup connections when shutting down"""
+        async with self.pool_lock:
+            for conn in self.connection_pool.values():
+                await conn.close()
+            self.connection_pool.clear()
+
+    @asynccontextmanager
+    async def get_connection(self):
+        """Connection pool manager"""
+        async with self.connection_semaphore:
+            task_id = id(asyncio.current_task())
+            try:
+                async with self.pool_lock:
+                    if task_id not in self.connection_pool:
+                        conn = await aiosqlite.connect(
+                            self.db_path,
+                            timeout=30.0
+                        )
+                        await conn.execute('PRAGMA journal_mode = WAL')
+                        await conn.execute('PRAGMA busy_timeout = 5000')
+                        self.connection_pool[task_id] = conn
+                    
+                yield self.connection_pool[task_id]
+                
+            except Exception as e:
+                logger.error(f"Connection error: {e}")
+                raise
+            finally:
+                async with self.pool_lock:
+                    if task_id in self.connection_pool:
+                        await self.connection_pool[task_id].close()
+                        del self.connection_pool[task_id]
+
+    async def execute_with_retry(self, operation, *args):
+        """Execute database operations with retry logic"""
+        for attempt in range(self.max_retries):
+            try:
+                async with self.get_connection() as db:
+                    result = await operation(db, *args)
+                    await db.commit()
+                    return result
+            except Exception as e:
+                if attempt == self.max_retries - 1:
+                    logger.error(f"Operation failed after {self.max_retries} attempts: {e}")
+                    raise
+                await asyncio.sleep(1 * (attempt + 1))  # Exponential backoff
 
     async def ainit_db(self):
-        async with aiosqlite.connect(self.db_path) as db:
+        """Initialize database schema"""
+        async def _init(db):
             await db.execute('''
                 CREATE TABLE IF NOT EXISTS crawled_data (
                     url TEXT PRIMARY KEY,
@@ -28,87 +92,101 @@ class AsyncDatabaseManager:
                     screenshot TEXT DEFAULT ""
                 )
             ''')
-            await db.commit()
+        
+        await self.execute_with_retry(_init)
         await self.update_db_schema()
 
     async def update_db_schema(self):
-        async with aiosqlite.connect(self.db_path) as db:
-            # Check if the 'media' column exists
+        """Update database schema if needed"""
+        async def _check_columns(db):
             cursor = await db.execute("PRAGMA table_info(crawled_data)")
             columns = await cursor.fetchall()
-            column_names = [column[1] for column in columns]
-            
-            if 'media' not in column_names:
-                await self.aalter_db_add_column('media')
-            
-            # Check for other missing columns and add them if necessary
-            for column in ['links', 'metadata', 'screenshot']:
-                if column not in column_names:
-                    await self.aalter_db_add_column(column)
+            return [column[1] for column in columns]
+
+        column_names = await self.execute_with_retry(_check_columns)
+        
+        for column in ['media', 'links', 'metadata', 'screenshot']:
+            if column not in column_names:
+                await self.aalter_db_add_column(column)
 
     async def aalter_db_add_column(self, new_column: str):
-        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
-                await db.commit()
-            print(f"Added column '{new_column}' to the database.")
-        except Exception as e:
-            print(f"Error altering database to add {new_column} column: {e}")
+        """Add new column to the database"""
+        async def _alter(db):
+            await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
+            logger.info(f"Added column '{new_column}' to the database.")
+
+        await self.execute_with_retry(_alter)
 
     async def aget_cached_url(self, url: str) -> Optional[Tuple[str, str, str, str, str, str, str, bool, str]]:
+        """Retrieve cached URL data"""
+        async def _get(db):
+            async with db.execute(
+                'SELECT url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot FROM crawled_data WHERE url = ?',
+                (url,)
+            ) as cursor:
+                return await cursor.fetchone()
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                async with db.execute('SELECT url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot FROM crawled_data WHERE url = ?', (url,)) as cursor:
-                    return await cursor.fetchone()
+            return await self.execute_with_retry(_get)
         except Exception as e:
-            print(f"Error retrieving cached URL: {e}")
+            logger.error(f"Error retrieving cached URL: {e}")
             return None
 
     async def acache_url(self, url: str, html: str, cleaned_html: str, markdown: str, extracted_content: str, success: bool, media: str = "{}", links: str = "{}", metadata: str = "{}", screenshot: str = ""):
+        """Cache URL data with retry logic"""
+        async def _cache(db):
+            await db.execute('''
+                INSERT INTO crawled_data (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                ON CONFLICT(url) DO UPDATE SET
+                    html = excluded.html,
+                    cleaned_html = excluded.cleaned_html,
+                    markdown = excluded.markdown,
+                    extracted_content = excluded.extracted_content,
+                    success = excluded.success,
+                    media = excluded.media,      
+                    links = excluded.links,    
+                    metadata = excluded.metadata,      
+                    screenshot = excluded.screenshot
+            ''', (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot))
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('''
-                    INSERT INTO crawled_data (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot)
-                    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-                    ON CONFLICT(url) DO UPDATE SET
-                        html = excluded.html,
-                        cleaned_html = excluded.cleaned_html,
-                        markdown = excluded.markdown,
-                        extracted_content = excluded.extracted_content,
-                        success = excluded.success,
-                        media = excluded.media,      
-                        links = excluded.links,    
-                        metadata = excluded.metadata,      
-                        screenshot = excluded.screenshot
-                ''', (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot))
-                await db.commit()
+            await self.execute_with_retry(_cache)
         except Exception as e:
-            print(f"Error caching URL: {e}")
+            logger.error(f"Error caching URL: {e}")
 
     async def aget_total_count(self) -> int:
+        """Get total number of cached URLs"""
+        async def _count(db):
+            async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
+                result = await cursor.fetchone()
+                return result[0] if result else 0
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
-                    result = await cursor.fetchone()
-                    return result[0] if result else 0
+            return await self.execute_with_retry(_count)
         except Exception as e:
-            print(f"Error getting total count: {e}")
+            logger.error(f"Error getting total count: {e}")
             return 0
 
     async def aclear_db(self):
+        """Clear all data from the database"""
+        async def _clear(db):
+            await db.execute('DELETE FROM crawled_data')
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('DELETE FROM crawled_data')
-                await db.commit()
+            await self.execute_with_retry(_clear)
         except Exception as e:
-            print(f"Error clearing database: {e}")
+            logger.error(f"Error clearing database: {e}")
 
     async def aflush_db(self):
-        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('DROP TABLE IF EXISTS crawled_data')
-                await db.commit()
-        except Exception as e:
-            print(f"Error flushing database: {e}")
+        """Drop the entire table"""
+        async def _flush(db):
+            await db.execute('DROP TABLE IF EXISTS crawled_data')
 
+        try:
+            await self.execute_with_retry(_flush)
+        except Exception as e:
+            logger.error(f"Error flushing database: {e}")
+
+# Create a singleton instance
 async_db_manager = AsyncDatabaseManager()

crawl4ai/async_webcrawler.py

@@ -16,7 +16,7 @@ from .utils import (
     InvalidCSSSelectorError,
     format_html
 )
-
+from ._version import __version__ as crawl4ai_version
 
 class AsyncWebCrawler:
     def __init__(
@@ -46,9 +46,12 @@ class AsyncWebCrawler:
         await self.crawler_strategy.__aexit__(exc_type, exc_val, exc_tb)
 
     async def awarmup(self):
+        # Print a message for crawl4ai and its version
+        print(f"[LOG] 🚀 Crawl4AI {crawl4ai_version}")
         if self.verbose:
             print("[LOG] 🌤️  Warming up the AsyncWebCrawler")
-        await async_db_manager.ainit_db()
+        # await async_db_manager.ainit_db()
+        await async_db_manager.initialize()
         await self.arun(
             url="https://google.com/",
             word_count_threshold=5,
@@ -125,6 +128,7 @@ class AsyncWebCrawler:
                 verbose,
                 bool(cached),
                 async_response=async_response,
+                bypass_cache=bypass_cache,
                 **kwargs,
             )
             crawl_result.status_code = async_response.status_code if async_response else 200
@@ -168,7 +172,6 @@ class AsyncWebCrawler:
         ]
         return await asyncio.gather(*tasks)
 
-
     async def aprocess_html(
         self,
         url: str,
@@ -243,7 +246,7 @@ class AsyncWebCrawler:
 
         screenshot = None if not screenshot else screenshot
 
-        if not is_cached:
+        if not is_cached or kwargs.get("bypass_cache", False) or self.always_by_pass_cache:
             await async_db_manager.acache_url(
                 url,
                 html,
@@ -274,10 +277,13 @@ class AsyncWebCrawler:
         )
 
     async def aclear_cache(self):
-        await async_db_manager.aclear_db()
+        # await async_db_manager.aclear_db()
+        await async_db_manager.cleanup()
 
     async def aflush_cache(self):
         await async_db_manager.aflush_db()
 
     async def aget_cache_size(self):
         return await async_db_manager.aget_total_count()
+
+

crawl4ai/content_scrapping_strategy.py

@@ -14,12 +14,97 @@ from .utils import (
     sanitize_html,
     extract_metadata,
     InvalidCSSSelectorError,
-    CustomHTML2Text,
+    # CustomHTML2Text,
     normalize_url,
     is_external_url
     
 )
 
+from .html2text import HTML2Text
+class CustomHTML2Text(HTML2Text):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.inside_pre = False
+        self.inside_code = False
+        self.preserve_tags = set()  # Set of tags to preserve
+        self.current_preserved_tag = None
+        self.preserved_content = []
+        self.preserve_depth = 0
+        
+        # Configuration options
+        self.skip_internal_links = False
+        self.single_line_break = False
+        self.mark_code = False
+        self.include_sup_sub = False
+        self.body_width = 0
+        self.ignore_mailto_links = True
+        self.ignore_links = False
+        self.escape_backslash = False
+        self.escape_dot = False
+        self.escape_plus = False
+        self.escape_dash = False
+        self.escape_snob = False
+
+    def update_params(self, **kwargs):
+        """Update parameters and set preserved tags."""
+        for key, value in kwargs.items():
+            if key == 'preserve_tags':
+                self.preserve_tags = set(value)
+            else:
+                setattr(self, key, value)
+
+    def handle_tag(self, tag, attrs, start):
+        # Handle preserved tags
+        if tag in self.preserve_tags:
+            if start:
+                if self.preserve_depth == 0:
+                    self.current_preserved_tag = tag
+                    self.preserved_content = []
+                    # Format opening tag with attributes
+                    attr_str = ''.join(f' {k}="{v}"' for k, v in attrs.items() if v is not None)
+                    self.preserved_content.append(f'<{tag}{attr_str}>')
+                self.preserve_depth += 1
+                return
+            else:
+                self.preserve_depth -= 1
+                if self.preserve_depth == 0:
+                    self.preserved_content.append(f'</{tag}>')
+                    # Output the preserved HTML block with proper spacing
+                    preserved_html = ''.join(self.preserved_content)
+                    self.o('\n' + preserved_html + '\n')
+                    self.current_preserved_tag = None
+                return
+
+        # If we're inside a preserved tag, collect all content
+        if self.preserve_depth > 0:
+            if start:
+                # Format nested tags with attributes
+                attr_str = ''.join(f' {k}="{v}"' for k, v in attrs.items() if v is not None)
+                self.preserved_content.append(f'<{tag}{attr_str}>')
+            else:
+                self.preserved_content.append(f'</{tag}>')
+            return
+
+        # Handle pre tags
+        if tag == 'pre':
+            if start:
+                self.o('```\n')
+                self.inside_pre = True
+            else:
+                self.o('\n```')
+                self.inside_pre = False
+        elif tag in ["h1", "h2", "h3", "h4", "h5", "h6"]:
+            pass
+        else:
+            super().handle_tag(tag, attrs, start)
+
+    def handle_data(self, data, entity_char=False):
+        """Override handle_data to capture content within preserved tags."""
+        if self.preserve_depth > 0:
+            self.preserved_content.append(data)
+            return
+        super().handle_data(data, entity_char)
+
 class ContentScrappingStrategy(ABC):
     @abstractmethod
     def scrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:

crawl4ai/models/onnx/config.json

@@ -1,25 +0,0 @@
-{
-  "_name_or_path": "sentence-transformers/all-MiniLM-L6-v2",
-  "architectures": [
-    "BertModel"
-  ],
-  "attention_probs_dropout_prob": 0.1,
-  "classifier_dropout": null,
-  "gradient_checkpointing": false,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 384,
-  "initializer_range": 0.02,
-  "intermediate_size": 1536,
-  "layer_norm_eps": 1e-12,
-  "max_position_embeddings": 512,
-  "model_type": "bert",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 6,
-  "pad_token_id": 0,
-  "position_embedding_type": "absolute",
-  "transformers_version": "4.27.4",
-  "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
-}

crawl4ai/models/onnx/special_tokens_map.json

@@ -1,7 +0,0 @@
-{
-  "cls_token": "[CLS]",
-  "mask_token": "[MASK]",
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "unk_token": "[UNK]"
-}

crawl4ai/models/onnx/tokenizer_config.json

@@ -1,15 +0,0 @@
-{
-  "cls_token": "[CLS]",
-  "do_basic_tokenize": true,
-  "do_lower_case": true,
-  "mask_token": "[MASK]",
-  "model_max_length": 512,
-  "never_split": null,
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "special_tokens_map_file": "/Users/hammad/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125/special_tokens_map.json",
-  "strip_accents": null,
-  "tokenize_chinese_chars": true,
-  "tokenizer_class": "BertTokenizer",
-  "unk_token": "[UNK]"
-}

crawl4ai/utils.py

@@ -178,7 +178,7 @@ def escape_json_string(s):
     
     return s
 
-class CustomHTML2Text(HTML2Text):
+class CustomHTML2Text_v0(HTML2Text):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.inside_pre = False
@@ -981,6 +981,19 @@ def format_html(html_string):
     return soup.prettify()
 
 def normalize_url(href, base_url):
+    """Normalize URLs to ensure consistent format"""
+    from urllib.parse import urljoin, urlparse
+
+    # Parse base URL to get components
+    parsed_base = urlparse(base_url)
+    if not parsed_base.scheme or not parsed_base.netloc:
+        raise ValueError(f"Invalid base URL format: {base_url}")
+
+    # Use urljoin to handle all cases
+    normalized = urljoin(base_url, href.strip())
+    return normalized
+
+def normalize_url_tmp(href, base_url):
     """Normalize URLs to ensure consistent format"""
     # Extract protocol and domain from base URL
     try:

docs/assets/pitch-dark.svg

@@ -0,0 +1,64 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 500">
+    <!-- Background -->
+    <rect width="800" height="500" fill="#1a1a1a"/>
+    
+    <!-- Opportunities Section -->
+    <g transform="translate(50,50)">
+        <!-- Opportunity 1 Box -->
+        <rect x="0" y="0" width="300" height="150" rx="10" fill="#1a2d3d" stroke="#64b5f6" stroke-width="2"/>
+        <text x="150" y="30" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#64b5f6">Data Capitalization Opportunity</text>
+        <text x="150" y="60" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
+            <tspan x="150" dy="0">Transform digital footprints into assets</tspan>
+            <tspan x="150" dy="20">Personal data as capital</tspan>
+            <tspan x="150" dy="20">Enterprise knowledge valuation</tspan>
+            <tspan x="150" dy="20">New form of wealth creation</tspan>
+        </text>
+
+        <!-- Opportunity 2 Box -->
+        <rect x="0" y="200" width="300" height="150" rx="10" fill="#1a2d1a" stroke="#81c784" stroke-width="2"/>
+        <text x="150" y="230" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#81c784">Authentic Data Potential</text>
+        <text x="150" y="260" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
+            <tspan x="150" dy="0">Vast reservoir of real insights</tspan>
+            <tspan x="150" dy="20">Enhanced AI development</tspan>
+            <tspan x="150" dy="20">Diverse human knowledge</tspan>
+            <tspan x="150" dy="20">Willing participation model</tspan>
+        </text>
+    </g>
+
+    <!-- Development Pathway -->
+    <g transform="translate(450,50)">
+        <!-- Step 1 Box -->
+        <rect x="0" y="0" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">1. Open-Source Foundation</text>
+        <text x="150" y="65" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Data extraction engine &amp; community development</text>
+
+        <!-- Step 2 Box -->
+        <rect x="0" y="125" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="160" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">2. Data Capitalization Platform</text>
+        <text x="150" y="190" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Tools to structure &amp; value digital assets</text>
+
+        <!-- Step 3 Box -->
+        <rect x="0" y="250" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="285" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">3. Shared Data Marketplace</text>
+        <text x="150" y="315" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Economic platform for data exchange</text>
+    </g>
+
+    <!-- Connecting Arrows -->
+    <g transform="translate(400,125)">
+        <path d="M-20,0 L40,0" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
+        <path d="M-20,200 L40,200" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
+    </g>
+
+    <!-- Arrow Marker -->
+    <defs>
+        <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#666"/>
+        </marker>
+    </defs>
+
+    <!-- Vision Box at Bottom -->
+    <g transform="translate(200,420)">
+        <rect x="0" y="0" width="400" height="60" rx="10" fill="#2d2613" stroke="#ffd54f" stroke-width="2"/>
+        <text x="200" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ffd54f">Economic Vision: Shared Data Economy</text>
+    </g>
+</svg>

docs/examples/quickstart_async.py

@@ -383,10 +383,11 @@ async def crawl_with_user_simultion():
     async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
         url = "YOUR-URL-HERE"
         result = await crawler.arun(
-            url=url,
+            url=url,            
             bypass_cache=True,
-            simulate_user = True,# Causes a series of random mouse movements and clicks to simulate user interaction
-            override_navigator = True # Overrides the navigator object to make it look like a real user
+            magic = True, # Automatically detects and removes overlays, popups, and other elements that block content
+            # simulate_user = True,# Causes a series of random mouse movements and clicks to simulate user interaction
+            # override_navigator = True # Overrides the navigator object to make it look like a real user
         )
         
         print(result.markdown)    

docs/examples/quickstart_v0.ipynb

@@ -0,0 +1,735 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6yLvrXn7yZQI"
+      },
+      "source": [
+        "# Crawl4AI: Advanced Web Crawling and Data Extraction\n",
+        "\n",
+        "Welcome to this interactive notebook showcasing Crawl4AI, an advanced asynchronous web crawling and data extraction library.\n",
+        "\n",
+        "- GitHub Repository: [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
+        "- Twitter: [@unclecode](https://twitter.com/unclecode)\n",
+        "- Website: [https://crawl4ai.com](https://crawl4ai.com)\n",
+        "\n",
+        "Let's explore the powerful features of Crawl4AI!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KIn_9nxFyZQK"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "First, let's install Crawl4AI from GitHub:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mSnaxLf3zMog"
+      },
+      "outputs": [],
+      "source": [
+        "!sudo apt-get update && sudo apt-get install -y libwoff1 libopus0 libwebp6 libwebpdemux2 libenchant1c2a libgudev-1.0-0 libsecret-1-0 libhyphen0 libgdk-pixbuf2.0-0 libegl1 libnotify4 libxslt1.1 libevent-2.1-7 libgles2 libvpx6 libxcomposite1 libatk1.0-0 libatk-bridge2.0-0 libepoxy0 libgtk-3-0 libharfbuzz-icu0"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xlXqaRtayZQK"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install crawl4ai\n",
+        "!pip install nest-asyncio\n",
+        "!playwright install"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qKCE7TI7yZQL"
+      },
+      "source": [
+        "Now, let's import the necessary libraries:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "I67tr7aAyZQL"
+      },
+      "outputs": [],
+      "source": [
+        "import asyncio\n",
+        "import nest_asyncio\n",
+        "from crawl4ai import AsyncWebCrawler\n",
+        "from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy\n",
+        "import json\n",
+        "import time\n",
+        "from pydantic import BaseModel, Field\n",
+        "\n",
+        "nest_asyncio.apply()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "h7yR_Rt_yZQM"
+      },
+      "source": [
+        "## Basic Usage\n",
+        "\n",
+        "Let's start with a simple crawl example:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "yBh6hf4WyZQM",
+        "outputId": "0f83af5c-abba-4175-ed95-70b7512e6bcc"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.05 seconds\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.05 seconds.\n",
+            "18102\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def simple_crawl():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(url=\"https://www.nbcnews.com/business\")\n",
+        "        print(len(result.markdown))\n",
+        "await simple_crawl()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9rtkgHI28uI4"
+      },
+      "source": [
+        "💡 By default, **Crawl4AI** caches the result of every URL, so the next time you call it, you’ll get an instant result. But if you want to bypass the cache, just set `bypass_cache=True`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MzZ0zlJ9yZQM"
+      },
+      "source": [
+        "## Advanced Features\n",
+        "\n",
+        "### Executing JavaScript and Using CSS Selectors"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "gHStF86xyZQM",
+        "outputId": "34d0fb6d-4dec-4677-f76e-85a1f082829b"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
+            "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 6.06 seconds\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.10 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.11 seconds.\n",
+            "41135\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def js_and_css():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        js_code = [\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"]\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            js_code=js_code,\n",
+        "            # css_selector=\"YOUR_CSS_SELECTOR_HERE\",\n",
+        "            bypass_cache=True\n",
+        "        )\n",
+        "        print(len(result.markdown))\n",
+        "\n",
+        "await js_and_css()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cqE_W4coyZQM"
+      },
+      "source": [
+        "### Using a Proxy\n",
+        "\n",
+        "Note: You'll need to replace the proxy URL with a working proxy for this example to run successfully."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QjAyiAGqyZQM"
+      },
+      "outputs": [],
+      "source": [
+        "async def use_proxy():\n",
+        "    async with AsyncWebCrawler(verbose=True, proxy=\"http://your-proxy-url:port\") as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            bypass_cache=True\n",
+        "        )\n",
+        "        print(result.markdown[:500])  # Print first 500 characters\n",
+        "\n",
+        "# Uncomment the following line to run the proxy example\n",
+        "# await use_proxy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XTZ88lbayZQN"
+      },
+      "source": [
+        "### Extracting Structured Data with OpenAI\n",
+        "\n",
+        "Note: You'll need to set your OpenAI API key as an environment variable for this example to work."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 14,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "fIOlDayYyZQN",
+        "outputId": "cb8359cc-dee0-4762-9698-5dfdcee055b8"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://openai.com/api/pricing/ using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://openai.com/api/pricing/ successfully!\n",
+            "[LOG] 🚀 Crawling done for https://openai.com/api/pricing/, success: True, time taken: 3.77 seconds\n",
+            "[LOG] 🚀 Content extracted for https://openai.com/api/pricing/, success: True, time taken: 0.21 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://openai.com/api/pricing/, Strategy: AsyncWebCrawler\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 0\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 1\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 2\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 3\n",
+            "[LOG] Extracted 4 blocks from URL: https://openai.com/api/pricing/ block index: 3\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 4\n",
+            "[LOG] Extracted 5 blocks from URL: https://openai.com/api/pricing/ block index: 0\n",
+            "[LOG] Extracted 1 blocks from URL: https://openai.com/api/pricing/ block index: 4\n",
+            "[LOG] Extracted 8 blocks from URL: https://openai.com/api/pricing/ block index: 1\n",
+            "[LOG] Extracted 12 blocks from URL: https://openai.com/api/pricing/ block index: 2\n",
+            "[LOG] 🚀 Extraction done for https://openai.com/api/pricing/, time taken: 8.55 seconds.\n",
+            "5029\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "from google.colab import userdata\n",
+        "os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')\n",
+        "\n",
+        "class OpenAIModelFee(BaseModel):\n",
+        "    model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n",
+        "    input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n",
+        "    output_fee: str = Field(..., description=\"Fee for output token for the OpenAI model.\")\n",
+        "\n",
+        "async def extract_openai_fees():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url='https://openai.com/api/pricing/',\n",
+        "            word_count_threshold=1,\n",
+        "            extraction_strategy=LLMExtractionStrategy(\n",
+        "                provider=\"openai/gpt-4o\", api_token=os.getenv('OPENAI_API_KEY'),\n",
+        "                schema=OpenAIModelFee.schema(),\n",
+        "                extraction_type=\"schema\",\n",
+        "                instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens.\n",
+        "                Do not miss any models in the entire content. One extracted model JSON format should look like this:\n",
+        "                {\"model_name\": \"GPT-4\", \"input_fee\": \"US$10.00 / 1M tokens\", \"output_fee\": \"US$30.00 / 1M tokens\"}.\"\"\"\n",
+        "            ),\n",
+        "            bypass_cache=True,\n",
+        "        )\n",
+        "        print(len(result.extracted_content))\n",
+        "\n",
+        "# Uncomment the following line to run the OpenAI extraction example\n",
+        "await extract_openai_fees()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BypA5YxEyZQN"
+      },
+      "source": [
+        "### Advanced Multi-Page Crawling with JavaScript Execution"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tfkcVQ0b7mw-"
+      },
+      "source": [
+        "## Advanced Multi-Page Crawling with JavaScript Execution\n",
+        "\n",
+        "This example demonstrates Crawl4AI's ability to handle complex crawling scenarios, specifically extracting commits from multiple pages of a GitHub repository. The challenge here is that clicking the \"Next\" button doesn't load a new page, but instead uses asynchronous JavaScript to update the content. This is a common hurdle in modern web crawling.\n",
+        "\n",
+        "To overcome this, we use Crawl4AI's custom JavaScript execution to simulate clicking the \"Next\" button, and implement a custom hook to detect when new data has loaded. Our strategy involves comparing the first commit's text before and after \"clicking\" Next, waiting until it changes to confirm new data has rendered. This showcases Crawl4AI's flexibility in handling dynamic content and its ability to implement custom logic for even the most challenging crawling tasks."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "qUBKGpn3yZQN",
+        "outputId": "3e555b6a-ed33-42f4-cce9-499a923fbe17"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 5.16 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.28 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.28 seconds.\n",
+            "Page 1: Found 35 commits\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.78 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.90 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.90 seconds.\n",
+            "Page 2: Found 35 commits\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 2.00 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.74 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.75 seconds.\n",
+            "Page 3: Found 35 commits\n",
+            "Successfully crawled 105 commits across 3 pages\n"
+          ]
+        }
+      ],
+      "source": [
+        "import re\n",
+        "from bs4 import BeautifulSoup\n",
+        "\n",
+        "async def crawl_typescript_commits():\n",
+        "    first_commit = \"\"\n",
+        "    async def on_execution_started(page):\n",
+        "        nonlocal first_commit\n",
+        "        try:\n",
+        "            while True:\n",
+        "                await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4')\n",
+        "                commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4')\n",
+        "                commit = await commit.evaluate('(element) => element.textContent')\n",
+        "                commit = re.sub(r'\\s+', '', commit)\n",
+        "                if commit and commit != first_commit:\n",
+        "                    first_commit = commit\n",
+        "                    break\n",
+        "                await asyncio.sleep(0.5)\n",
+        "        except Exception as e:\n",
+        "            print(f\"Warning: New content didn't appear after JavaScript execution: {e}\")\n",
+        "\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started)\n",
+        "\n",
+        "        url = \"https://github.com/microsoft/TypeScript/commits/main\"\n",
+        "        session_id = \"typescript_commits_session\"\n",
+        "        all_commits = []\n",
+        "\n",
+        "        js_next_page = \"\"\"\n",
+        "        const button = document.querySelector('a[data-testid=\"pagination-next-button\"]');\n",
+        "        if (button) button.click();\n",
+        "        \"\"\"\n",
+        "\n",
+        "        for page in range(3):  # Crawl 3 pages\n",
+        "            result = await crawler.arun(\n",
+        "                url=url,\n",
+        "                session_id=session_id,\n",
+        "                css_selector=\"li.Box-sc-g0xbh4-0\",\n",
+        "                js=js_next_page if page > 0 else None,\n",
+        "                bypass_cache=True,\n",
+        "                js_only=page > 0\n",
+        "            )\n",
+        "\n",
+        "            assert result.success, f\"Failed to crawl page {page + 1}\"\n",
+        "\n",
+        "            soup = BeautifulSoup(result.cleaned_html, 'html.parser')\n",
+        "            commits = soup.select(\"li\")\n",
+        "            all_commits.extend(commits)\n",
+        "\n",
+        "            print(f\"Page {page + 1}: Found {len(commits)} commits\")\n",
+        "\n",
+        "        await crawler.crawler_strategy.kill_session(session_id)\n",
+        "        print(f\"Successfully crawled {len(all_commits)} commits across 3 pages\")\n",
+        "\n",
+        "await crawl_typescript_commits()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EJRnYsp6yZQN"
+      },
+      "source": [
+        "### Using JsonCssExtractionStrategy for Fast Structured Output"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1ZMqIzB_8SYp"
+      },
+      "source": [
+        "The JsonCssExtractionStrategy is a powerful feature of Crawl4AI that allows for precise, structured data extraction from web pages. Here's how it works:\n",
+        "\n",
+        "1. You define a schema that describes the pattern of data you're interested in extracting.\n",
+        "2. The schema includes a base selector that identifies repeating elements on the page.\n",
+        "3. Within the schema, you define fields, each with its own selector and type.\n",
+        "4. These field selectors are applied within the context of each base selector element.\n",
+        "5. The strategy supports nested structures, lists within lists, and various data types.\n",
+        "6. You can even include computed fields for more complex data manipulation.\n",
+        "\n",
+        "This approach allows for highly flexible and precise data extraction, transforming semi-structured web content into clean, structured JSON data. It's particularly useful for extracting consistent data patterns from pages like product listings, news articles, or search results.\n",
+        "\n",
+        "For more details and advanced usage, check out the full documentation on the Crawl4AI website."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "trCMR2T9yZQN",
+        "outputId": "718d36f4-cccf-40f4-8d8c-c3ba73524d16"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
+            "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 7.00 seconds\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.32 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.48 seconds.\n",
+            "Successfully extracted 11 news teasers\n",
+            "{\n",
+            "  \"category\": \"Business News\",\n",
+            "  \"headline\": \"NBC ripped up its Olympics playbook for 2024 \\u2014 so far, the new strategy paid off\",\n",
+            "  \"summary\": \"The Olympics have long been key to NBCUniversal. Paris marked the 18th Olympic Games broadcast by NBC in the U.S.\",\n",
+            "  \"time\": \"13h ago\",\n",
+            "  \"image\": {\n",
+            "    \"src\": \"https://media-cldnry.s-nbcnews.com/image/upload/t_focal-200x100,f_auto,q_auto:best/rockcms/2024-09/240903-nbc-olympics-ch-1344-c7a486.jpg\",\n",
+            "    \"alt\": \"Mike Tirico.\"\n",
+            "  },\n",
+            "  \"link\": \"https://www.nbcnews.com/business\"\n",
+            "}\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def extract_news_teasers():\n",
+        "    schema = {\n",
+        "        \"name\": \"News Teaser Extractor\",\n",
+        "        \"baseSelector\": \".wide-tease-item__wrapper\",\n",
+        "        \"fields\": [\n",
+        "            {\n",
+        "                \"name\": \"category\",\n",
+        "                \"selector\": \".unibrow span[data-testid='unibrow-text']\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"headline\",\n",
+        "                \"selector\": \".wide-tease-item__headline\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"summary\",\n",
+        "                \"selector\": \".wide-tease-item__description\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"time\",\n",
+        "                \"selector\": \"[data-testid='wide-tease-date']\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"image\",\n",
+        "                \"type\": \"nested\",\n",
+        "                \"selector\": \"picture.teasePicture img\",\n",
+        "                \"fields\": [\n",
+        "                    {\"name\": \"src\", \"type\": \"attribute\", \"attribute\": \"src\"},\n",
+        "                    {\"name\": \"alt\", \"type\": \"attribute\", \"attribute\": \"alt\"},\n",
+        "                ],\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"link\",\n",
+        "                \"selector\": \"a[href]\",\n",
+        "                \"type\": \"attribute\",\n",
+        "                \"attribute\": \"href\",\n",
+        "            },\n",
+        "        ],\n",
+        "    }\n",
+        "\n",
+        "    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n",
+        "\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            extraction_strategy=extraction_strategy,\n",
+        "            bypass_cache=True,\n",
+        "        )\n",
+        "\n",
+        "        assert result.success, \"Failed to crawl the page\"\n",
+        "\n",
+        "        news_teasers = json.loads(result.extracted_content)\n",
+        "        print(f\"Successfully extracted {len(news_teasers)} news teasers\")\n",
+        "        print(json.dumps(news_teasers[0], indent=2))\n",
+        "\n",
+        "await extract_news_teasers()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FnyVhJaByZQN"
+      },
+      "source": [
+        "## Speed Comparison\n",
+        "\n",
+        "Let's compare the speed of Crawl4AI with Firecrawl, a paid service. Note that we can't run Firecrawl in this Colab environment, so we'll simulate its performance based on previously recorded data."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "agDD186f3wig"
+      },
+      "source": [
+        "💡 **Note on Speed Comparison:**\n",
+        "\n",
+        "The speed test conducted here is running on Google Colab, where the internet speed and performance can vary and may not reflect optimal conditions. When we call Firecrawl's API, we're seeing its best performance, while Crawl4AI's performance is limited by Colab's network speed.\n",
+        "\n",
+        "For a more accurate comparison, it's recommended to run these tests on your own servers or computers with a stable and fast internet connection. Despite these limitations, Crawl4AI still demonstrates faster performance in this environment.\n",
+        "\n",
+        "If you run these tests locally, you may observe an even more significant speed advantage for Crawl4AI compared to other services."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "F7KwHv8G1LbY"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install firecrawl"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "91813zILyZQN",
+        "outputId": "663223db-ab89-4976-b233-05ceca62b19b"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Firecrawl (simulated):\n",
+            "Time taken: 4.38 seconds\n",
+            "Content length: 41967 characters\n",
+            "Images found: 49\n",
+            "\n",
+            "Crawl4AI (simple crawl):\n",
+            "Time taken: 4.22 seconds\n",
+            "Content length: 18221 characters\n",
+            "Images found: 49\n",
+            "\n",
+            "Crawl4AI (with JavaScript execution):\n",
+            "Time taken: 9.13 seconds\n",
+            "Content length: 34243 characters\n",
+            "Images found: 89\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "from google.colab import userdata\n",
+        "os.environ['FIRECRAWL_API_KEY'] = userdata.get('FIRECRAWL_API_KEY')\n",
+        "import time\n",
+        "from firecrawl import FirecrawlApp\n",
+        "\n",
+        "async def speed_comparison():\n",
+        "    # Simulated Firecrawl performance\n",
+        "    app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])\n",
+        "    start = time.time()\n",
+        "    scrape_status = app.scrape_url(\n",
+        "    'https://www.nbcnews.com/business',\n",
+        "    params={'formats': ['markdown', 'html']}\n",
+        "    )\n",
+        "    end = time.time()\n",
+        "    print(\"Firecrawl (simulated):\")\n",
+        "    print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "    print(f\"Content length: {len(scrape_status['markdown'])} characters\")\n",
+        "    print(f\"Images found: {scrape_status['markdown'].count('cldnry.s-nbcnews.com')}\")\n",
+        "    print()\n",
+        "\n",
+        "    async with AsyncWebCrawler() as crawler:\n",
+        "        # Crawl4AI simple crawl\n",
+        "        start = time.time()\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            word_count_threshold=0,\n",
+        "            bypass_cache=True,\n",
+        "            verbose=False\n",
+        "        )\n",
+        "        end = time.time()\n",
+        "        print(\"Crawl4AI (simple crawl):\")\n",
+        "        print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "        print(f\"Content length: {len(result.markdown)} characters\")\n",
+        "        print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
+        "        print()\n",
+        "\n",
+        "        # Crawl4AI with JavaScript execution\n",
+        "        start = time.time()\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            js_code=[\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"],\n",
+        "            word_count_threshold=0,\n",
+        "            bypass_cache=True,\n",
+        "            verbose=False\n",
+        "        )\n",
+        "        end = time.time()\n",
+        "        print(\"Crawl4AI (with JavaScript execution):\")\n",
+        "        print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "        print(f\"Content length: {len(result.markdown)} characters\")\n",
+        "        print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
+        "\n",
+        "await speed_comparison()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "OBFFYVJIyZQN"
+      },
+      "source": [
+        "If you run on a local machine with a proper internet speed:\n",
+        "- Simple crawl: Crawl4AI is typically over 3-4 times faster than Firecrawl.\n",
+        "- With JavaScript execution: Even when executing JavaScript to load more content (potentially doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.\n",
+        "\n",
+        "Please note that actual performance may vary depending on network conditions and the specific content being crawled."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "A6_1RK1_yZQO"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "In this notebook, we've explored the powerful features of Crawl4AI, including:\n",
+        "\n",
+        "1. Basic crawling\n",
+        "2. JavaScript execution and CSS selector usage\n",
+        "3. Proxy support\n",
+        "4. Structured data extraction with OpenAI\n",
+        "5. Advanced multi-page crawling with JavaScript execution\n",
+        "6. Fast structured output using JsonCssExtractionStrategy\n",
+        "7. Speed comparison with other services\n",
+        "\n",
+        "Crawl4AI offers a fast, flexible, and powerful solution for web crawling and data extraction tasks. Its asynchronous architecture and advanced features make it suitable for a wide range of applications, from simple web scraping to complex, multi-page data extraction scenarios.\n",
+        "\n",
+        "For more information and advanced usage, please visit the [Crawl4AI documentation](https://crawl4ai.com/mkdocs/).\n",
+        "\n",
+        "Happy crawling!"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "venv",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.13"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/md_v2/api/parameters.md

@@ -0,0 +1,35 @@
+# Parameter Reference Table
+
+| File Name | Parameter Name | Code Usage | Strategy/Class | Description |
+|-----------|---------------|------------|----------------|-------------|
+| async_crawler_strategy.py | user_agent | `kwargs.get("user_agent")` | AsyncPlaywrightCrawlerStrategy | User agent string for browser identification |
+| async_crawler_strategy.py | proxy | `kwargs.get("proxy")` | AsyncPlaywrightCrawlerStrategy | Proxy server configuration for network requests |
+| async_crawler_strategy.py | proxy_config | `kwargs.get("proxy_config")` | AsyncPlaywrightCrawlerStrategy | Detailed proxy configuration including auth |
+| async_crawler_strategy.py | headless | `kwargs.get("headless", True)` | AsyncPlaywrightCrawlerStrategy | Whether to run browser in headless mode |
+| async_crawler_strategy.py | browser_type | `kwargs.get("browser_type", "chromium")` | AsyncPlaywrightCrawlerStrategy | Type of browser to use (chromium/firefox/webkit) |
+| async_crawler_strategy.py | headers | `kwargs.get("headers", {})` | AsyncPlaywrightCrawlerStrategy | Custom HTTP headers for requests |
+| async_crawler_strategy.py | verbose | `kwargs.get("verbose", False)` | AsyncPlaywrightCrawlerStrategy | Enable detailed logging output |
+| async_crawler_strategy.py | sleep_on_close | `kwargs.get("sleep_on_close", False)` | AsyncPlaywrightCrawlerStrategy | Add delay before closing browser |
+| async_crawler_strategy.py | use_managed_browser | `kwargs.get("use_managed_browser", False)` | AsyncPlaywrightCrawlerStrategy | Use managed browser instance |
+| async_crawler_strategy.py | user_data_dir | `kwargs.get("user_data_dir", None)` | AsyncPlaywrightCrawlerStrategy | Custom directory for browser profile data |
+| async_crawler_strategy.py | session_id | `kwargs.get("session_id")` | AsyncPlaywrightCrawlerStrategy | Unique identifier for browser session |
+| async_crawler_strategy.py | override_navigator | `kwargs.get("override_navigator", False)` | AsyncPlaywrightCrawlerStrategy | Override browser navigator properties |
+| async_crawler_strategy.py | simulate_user | `kwargs.get("simulate_user", False)` | AsyncPlaywrightCrawlerStrategy | Simulate human-like behavior |
+| async_crawler_strategy.py | magic | `kwargs.get("magic", False)` | AsyncPlaywrightCrawlerStrategy | Enable advanced anti-detection features |
+| async_crawler_strategy.py | log_console | `kwargs.get("log_console", False)` | AsyncPlaywrightCrawlerStrategy | Log browser console messages |
+| async_crawler_strategy.py | js_only | `kwargs.get("js_only", False)` | AsyncPlaywrightCrawlerStrategy | Only execute JavaScript without page load |
+| async_crawler_strategy.py | page_timeout | `kwargs.get("page_timeout", 60000)` | AsyncPlaywrightCrawlerStrategy | Timeout for page load in milliseconds |
+| async_crawler_strategy.py | ignore_body_visibility | `kwargs.get("ignore_body_visibility", True)` | AsyncPlaywrightCrawlerStrategy | Process page even if body is hidden |
+| async_crawler_strategy.py | js_code | `kwargs.get("js_code", kwargs.get("js", self.js_code))` | AsyncPlaywrightCrawlerStrategy | Custom JavaScript code to execute |
+| async_crawler_strategy.py | wait_for | `kwargs.get("wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait for specific element/condition |
+| async_crawler_strategy.py | process_iframes | `kwargs.get("process_iframes", False)` | AsyncPlaywrightCrawlerStrategy | Extract content from iframes |
+| async_crawler_strategy.py | delay_before_return_html | `kwargs.get("delay_before_return_html")` | AsyncPlaywrightCrawlerStrategy | Additional delay before returning HTML |
+| async_crawler_strategy.py | remove_overlay_elements | `kwargs.get("remove_overlay_elements", False)` | AsyncPlaywrightCrawlerStrategy | Remove pop-ups and overlay elements |
+| async_crawler_strategy.py | screenshot | `kwargs.get("screenshot")` | AsyncPlaywrightCrawlerStrategy | Take page screenshot |
+| async_crawler_strategy.py | screenshot_wait_for | `kwargs.get("screenshot_wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait before taking screenshot |
+| async_crawler_strategy.py | semaphore_count | `kwargs.get("semaphore_count", 5)` | AsyncPlaywrightCrawlerStrategy | Concurrent request limit |
+| async_webcrawler.py | verbose | `kwargs.get("verbose", False)` | AsyncWebCrawler | Enable detailed logging |
+| async_webcrawler.py | warmup | `kwargs.get("warmup", True)` | AsyncWebCrawler | Initialize crawler with warmup request |
+| async_webcrawler.py | session_id | `kwargs.get("session_id", None)` | AsyncWebCrawler | Session identifier for browser reuse |
+| async_webcrawler.py | only_text | `kwargs.get("only_text", False)` | AsyncWebCrawler | Extract only text content |
+| async_webcrawler.py | bypass_cache | `kwargs.get("bypass_cache", False)` | AsyncWebCrawler | Skip cache and force fresh crawl |

docs/md_v2/assets/styles.css

@@ -150,4 +150,11 @@ strong,
 .tab-content pre {
     margin: 0;
     max-height: 300px; overflow: auto; border:none;
+}
+
+ol li::before {
+    content: counters(item, ".") ". ";
+    counter-increment: item;
+    /* float: left; */
+    /* padding-right: 5px; */
 }

docs/md_v2/basic/docker-deploymeny.md

@@ -0,0 +1,459 @@
+# Docker Deployment
+
+Crawl4AI provides official Docker images for easy deployment and scalability. This guide covers installation, configuration, and usage of Crawl4AI in Docker environments.
+
+## Quick Start 🚀
+
+Pull and run the basic version:
+
+```bash
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+Test the deployment:
+```python
+import requests
+
+# Test health endpoint
+health = requests.get("http://localhost:11235/health")
+print("Health check:", health.json())
+
+# Test basic crawl
+response = requests.post(
+    "http://localhost:11235/crawl",
+    json={
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+)
+task_id = response.json()["task_id"]
+print("Task ID:", task_id)
+```
+
+## Available Images 🏷️
+
+- `unclecode/crawl4ai:basic` - Basic web crawling capabilities
+- `unclecode/crawl4ai:all` - Full installation with all features
+- `unclecode/crawl4ai:gpu` - GPU-enabled version for ML features
+
+## Configuration Options 🔧
+
+### Environment Variables
+
+```bash
+docker run -p 11235:11235 \
+    -e MAX_CONCURRENT_TASKS=5 \
+    -e OPENAI_API_KEY=your_key \
+    unclecode/crawl4ai:all
+```
+
+### Volume Mounting
+
+Mount a directory for persistent data:
+```bash
+docker run -p 11235:11235 \
+    -v $(pwd)/data:/app/data \
+    unclecode/crawl4ai:all
+```
+
+### Resource Limits
+
+Control container resources:
+```bash
+docker run -p 11235:11235 \
+    --memory=4g \
+    --cpus=2 \
+    unclecode/crawl4ai:all
+```
+
+## Usage Examples 📝
+
+### Basic Crawling
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "priority": 10
+}
+
+response = requests.post("http://localhost:11235/crawl", json=request)
+task_id = response.json()["task_id"]
+
+# Get results
+result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+### Structured Data Extraction
+
+```python
+schema = {
+    "name": "Crypto Prices",
+    "baseSelector": ".cds-tableRow-t45thuk",
+    "fields": [
+        {
+            "name": "crypto",
+            "selector": "td:nth-child(1) h2",
+            "type": "text",
+        },
+        {
+            "name": "price",
+            "selector": "td:nth-child(2)",
+            "type": "text",
+        }
+    ],
+}
+
+request = {
+    "urls": "https://www.coinbase.com/explore",
+    "extraction_config": {
+        "type": "json_css",
+        "params": {"schema": schema}
+    }
+}
+```
+
+### Dynamic Content Handling
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "js_code": [
+        "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+    ],
+    "wait_for": "article.tease-card:nth-child(10)"
+}
+```
+
+### AI-Powered Extraction (Full Version)
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "extraction_config": {
+        "type": "cosine",
+        "params": {
+            "semantic_filter": "business finance economy",
+            "word_count_threshold": 10,
+            "max_dist": 0.2,
+            "top_k": 3
+        }
+    }
+}
+```
+
+## Platform-Specific Instructions 💻
+
+### macOS
+```bash
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+### Ubuntu
+```bash
+# Basic version
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+
+# With GPU support
+docker pull unclecode/crawl4ai:gpu
+docker run --gpus all -p 11235:11235 unclecode/crawl4ai:gpu
+```
+
+### Windows (PowerShell)
+```powershell
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+## Testing 🧪
+
+Save this as `test_docker.py`:
+
+```python
+import requests
+import json
+import time
+import sys
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:11235"):
+        self.base_url = base_url
+        
+    def submit_and_wait(self, request_data: dict, timeout: int = 300) -> dict:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data)
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} timeout")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}")
+            status = result.json()
+            
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+
+def test_deployment():
+    tester = Crawl4AiTester()
+    
+    # Test basic crawl
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Basic crawl successful!")
+    print(f"Content length: {len(result['result']['markdown'])}")
+
+if __name__ == "__main__":
+    test_deployment()
+```
+
+## Advanced Configuration ⚙️
+
+### Crawler Parameters
+
+The `crawler_params` field allows you to configure the browser instance and crawling behavior. Here are key parameters you can use:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        # Browser Configuration
+        "headless": True,                    # Run in headless mode
+        "browser_type": "chromium",          # chromium/firefox/webkit
+        "user_agent": "custom-agent",        # Custom user agent
+        "proxy": "http://proxy:8080",        # Proxy configuration
+        
+        # Performance & Behavior
+        "page_timeout": 30000,               # Page load timeout (ms)
+        "verbose": True,                     # Enable detailed logging
+        "semaphore_count": 5,               # Concurrent request limit
+        
+        # Anti-Detection Features
+        "simulate_user": True,               # Simulate human behavior
+        "magic": True,                       # Advanced anti-detection
+        "override_navigator": True,          # Override navigator properties
+        
+        # Session Management
+        "user_data_dir": "./browser-data",   # Browser profile location
+        "use_managed_browser": True,         # Use persistent browser
+    }
+}
+```
+
+### Extra Parameters
+
+The `extra` field allows passing additional parameters directly to the crawler's `arun` function:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "extra": {
+        "word_count_threshold": 10,          # Min words per block
+        "only_text": True,                   # Extract only text
+        "bypass_cache": True,                # Force fresh crawl
+        "process_iframes": True,             # Include iframe content
+    }
+}
+```
+
+### Complete Examples
+
+1. **Advanced News Crawling**
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "crawler_params": {
+        "headless": True,
+        "page_timeout": 30000,
+        "remove_overlay_elements": True      # Remove popups
+    },
+    "extra": {
+        "word_count_threshold": 50,          # Longer content blocks
+        "bypass_cache": True                 # Fresh content
+    },
+    "css_selector": ".article-body"
+}
+```
+
+2. **Anti-Detection Configuration**
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        "simulate_user": True,
+        "magic": True,
+        "override_navigator": True,
+        "user_agent": "Mozilla/5.0 ...",
+        "headers": {
+            "Accept-Language": "en-US,en;q=0.9"
+        }
+    }
+}
+```
+
+3. **LLM Extraction with Custom Parameters**
+```python
+request = {
+    "urls": "https://openai.com/pricing",
+    "extraction_config": {
+        "type": "llm",
+        "params": {
+            "provider": "openai/gpt-4",
+            "schema": pricing_schema
+        }
+    },
+    "crawler_params": {
+        "verbose": True,
+        "page_timeout": 60000
+    },
+    "extra": {
+        "word_count_threshold": 1,
+        "only_text": True
+    }
+}
+```
+
+4. **Session-Based Dynamic Content**
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        "session_id": "dynamic_session",
+        "headless": False,
+        "page_timeout": 60000
+    },
+    "js_code": ["window.scrollTo(0, document.body.scrollHeight);"],
+    "wait_for": "js:() => document.querySelectorAll('.item').length > 10",
+    "extra": {
+        "delay_before_return_html": 2.0
+    }
+}
+```
+
+5. **Screenshot with Custom Timing**
+```python
+request = {
+    "urls": "https://example.com",
+    "screenshot": True,
+    "crawler_params": {
+        "headless": True,
+        "screenshot_wait_for": ".main-content"
+    },
+    "extra": {
+        "delay_before_return_html": 3.0
+    }
+}
+```
+
+### Parameter Reference Table
+
+| Category | Parameter | Type | Description |
+|----------|-----------|------|-------------|
+| Browser | headless | bool | Run browser in headless mode |
+| Browser | browser_type | str | Browser engine selection |
+| Browser | user_agent | str | Custom user agent string |
+| Network | proxy | str | Proxy server URL |
+| Network | headers | dict | Custom HTTP headers |
+| Timing | page_timeout | int | Page load timeout (ms) |
+| Timing | delay_before_return_html | float | Wait before capture |
+| Anti-Detection | simulate_user | bool | Human behavior simulation |
+| Anti-Detection | magic | bool | Advanced protection |
+| Session | session_id | str | Browser session ID |
+| Session | user_data_dir | str | Profile directory |
+| Content | word_count_threshold | int | Minimum words per block |
+| Content | only_text | bool | Text-only extraction |
+| Content | process_iframes | bool | Include iframe content |
+| Debug | verbose | bool | Detailed logging |
+| Debug | log_console | bool | Browser console logs |
+
+## Troubleshooting 🔍
+
+### Common Issues
+
+1. **Connection Refused**
+   ```
+   Error: Connection refused at localhost:11235
+   ```
+   Solution: Ensure the container is running and ports are properly mapped.
+
+2. **Resource Limits**
+   ```
+   Error: No available slots
+   ```
+   Solution: Increase MAX_CONCURRENT_TASKS or container resources.
+
+3. **GPU Access**
+   ```
+   Error: GPU not found
+   ```
+   Solution: Ensure proper NVIDIA drivers and use `--gpus all` flag.
+
+### Debug Mode
+
+Access container for debugging:
+```bash
+docker run -it --entrypoint /bin/bash unclecode/crawl4ai:all
+```
+
+View container logs:
+```bash
+docker logs [container_id]
+```
+
+## Best Practices 🌟
+
+1. **Resource Management**
+   - Set appropriate memory and CPU limits
+   - Monitor resource usage via health endpoint
+   - Use basic version for simple crawling tasks
+
+2. **Scaling**
+   - Use multiple containers for high load
+   - Implement proper load balancing
+   - Monitor performance metrics
+
+3. **Security**
+   - Use environment variables for sensitive data
+   - Implement proper network isolation
+   - Regular security updates
+
+## API Reference 📚
+
+### Health Check
+```http
+GET /health
+```
+
+### Submit Crawl Task
+```http
+POST /crawl
+Content-Type: application/json
+
+{
+    "urls": "string or array",
+    "extraction_config": {
+        "type": "basic|llm|cosine|json_css",
+        "params": {}
+    },
+    "priority": 1-10,
+    "ttl": 3600
+}
+```
+
+### Get Task Status
+```http
+GET /task/{task_id}
+```
+
+For more details, visit the [official documentation](https://crawl4ai.com/mkdocs/).

docs/md_v2/extraction/extraction_strategies.md

@@ -1,185 +0,0 @@
-## Extraction Strategies 🧠
-
-Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into three of the most important strategies: `CosineStrategy`, `LLMExtractionStrategy`, and the new `JsonCssExtractionStrategy`.
-
-### LLMExtractionStrategy
-
-`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
-
-#### When to Use
-- Suitable for complex extraction tasks requiring nuanced understanding.
-- Ideal for scenarios where detailed instructions can guide the extraction process.
-- Perfect for extracting specific types of information or content with precise guidelines.
-
-#### Parameters
-- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
-- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
-- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
-
-#### Example Without Instructions
-```python
-import asyncio
-import os
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define extraction strategy without instructions
-        strategy = LLMExtractionStrategy(
-            provider='openai',
-            api_token=os.getenv('OPENAI_API_KEY')
-        )
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        print(result.extracted_content)
-
-asyncio.run(main())
-```
-
-#### Example With Instructions
-```python
-import asyncio
-import os
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define extraction strategy with instructions
-        strategy = LLMExtractionStrategy(
-            provider='openai',
-            api_token=os.getenv('OPENAI_API_KEY'),
-            instruction="Extract only financial news and summarize key points."
-        )
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        print(result.extracted_content)
-
-asyncio.run(main())
-```
-
-### JsonCssExtractionStrategy
-
-`JsonCssExtractionStrategy` is a powerful tool for extracting structured data from HTML using CSS selectors. It allows you to define a schema that maps CSS selectors to specific fields, enabling precise and efficient data extraction.
-
-#### When to Use
-- Ideal for extracting structured data from websites with consistent HTML structures.
-- Perfect for scenarios where you need to extract specific elements or attributes from a webpage.
-- Suitable for creating datasets from web pages with tabular or list-based information.
-
-#### Parameters
-- `schema` (Dict[str, Any]): A dictionary defining the extraction schema, including base selector and field definitions.
-
-#### Example
-```python
-import asyncio
-import json
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define the extraction schema
-        schema = {
-            "name": "News Articles",
-            "baseSelector": "article.tease-card",
-            "fields": [
-                {
-                    "name": "title",
-                    "selector": "h2",
-                    "type": "text",
-                },
-                {
-                    "name": "summary",
-                    "selector": "div.tease-card__info",
-                    "type": "text",
-                },
-                {
-                    "name": "link",
-                    "selector": "a",
-                    "type": "attribute",
-                    "attribute": "href"
-                }
-            ],
-        }
-
-        # Create the extraction strategy
-        strategy = JsonCssExtractionStrategy(schema, verbose=True)
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        
-        # Parse and print the extracted content
-        extracted_data = json.loads(result.extracted_content)
-        print(json.dumps(extracted_data, indent=2))
-
-asyncio.run(main())
-```
-
-#### Use Cases for JsonCssExtractionStrategy
-- Extracting product information from e-commerce websites.
-- Gathering news articles and their metadata from news portals.
-- Collecting user reviews and ratings from review websites.
-- Extracting job listings from job boards.
-
-By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy`, nuanced, instruction-based extraction with `LLMExtractionStrategy`, or precise structured data extraction with `JsonCssExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨
-
-For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](./css-advanced.md).
-
-
-### CosineStrategy
-
-`CosineStrategy` uses hierarchical clustering based on cosine similarity to group text chunks into meaningful clusters. This method converts each chunk into its embedding and then clusters them to form semantical chunks.
-
-#### When to Use
-- Ideal for fast, accurate semantic segmentation of text.
-- Perfect for scenarios where LLMs might be overkill or too slow.
-- Suitable for narrowing down content based on specific queries or keywords.
-
-#### Parameters
-- `semantic_filter` (str, optional): Keywords for filtering relevant documents before clustering. Documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.
-- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.
-- `max_dist` (float, optional): Maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.
-- `linkage_method` (str, optional): Linkage method for hierarchical clustering. Default is `'ward'`.
-- `top_k` (int, optional): Number of top categories to extract. Default is `3`.
-- `model_name` (str, optional): Model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.
-
-#### Example
-```python
-import asyncio
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import CosineStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define extraction strategy
-        strategy = CosineStrategy(
-            semantic_filter="finance economy stock market",
-            word_count_threshold=10,
-            max_dist=0.2,
-            linkage_method='ward',
-            top_k=3,
-            model_name='BAAI/bge-small-en-v1.5'
-        )
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        print(result.extracted_content)
-
-asyncio.run(main())
-```

docs/md_v2/index.md

@@ -72,7 +72,7 @@ Our documentation is organized into several sections:
 ### Advanced Features
 - [Magic Mode](advanced/magic-mode.md)
 - [Session Management](advanced/session-management.md)
-- [Hooks & Authentication](advanced/hooks.md)
+- [Hooks & Authentication](advanced/hooks-auth.md)
 - [Proxy & Security](advanced/proxy-security.md)
 - [Content Processing](advanced/content-processing.md)
 

docs/md_v2/tutorial/episode_01_Introduction_to_Crawl4AI_and_Basic_Installation.md

@@ -0,0 +1,51 @@
+# Crawl4AI
+
+## Episode 1: Introduction to Crawl4AI and Basic Installation
+
+### Quick Intro
+Walk through installation from PyPI, setup, and verification. Show how to install with options like `torch` or `transformer` for advanced capabilities.
+
+Here's a condensed outline of the **Installation and Setup** video content:
+
+---
+
+1) **Introduction to Crawl4AI**: Briefly explain that Crawl4AI is a powerful tool for web scraping, data extraction, and content processing, with customizable options for various needs.
+
+2) **Installation Overview**:   
+   
+   - **Basic Install**: Run `pip install crawl4ai` and `playwright install` (to set up browser dependencies).
+ 
+   - **Optional Advanced Installs**:
+     - `pip install crawl4ai[torch]` - Adds PyTorch for clustering.
+     - `pip install crawl4ai[transformer]` - Adds support for LLM-based extraction.
+     - `pip install crawl4ai[all]` - Installs all features for complete functionality.
+
+3) **Verifying the Installation**:
+   
+   - Walk through a simple test script to confirm the setup:
+      ```python
+      import asyncio
+      from crawl4ai import AsyncWebCrawler
+      
+      async def main():
+          async with AsyncWebCrawler(verbose=True) as crawler:
+              result = await crawler.arun(url="https://www.example.com")
+              print(result.markdown[:500])  # Show first 500 characters
+
+      asyncio.run(main())
+      ```
+   - Explain that this script initializes the crawler and runs it on a test URL, displaying part of the extracted content to verify functionality.
+
+4) **Important Tips**:
+   
+   - **Run** `playwright install` **after installation** to set up dependencies.
+   - **For full performance** on text-related tasks, run `crawl4ai-download-models` after installing with `[torch]`, `[transformer]`, or `[all]` options.
+   - If you encounter issues, refer to the documentation or GitHub issues.
+
+5) **Wrap Up**:
+   
+   - Introduce the next topic in the series, which will cover Crawl4AI's browser configuration options (like choosing between `chromium`, `firefox`, and `webkit`).
+
+---
+
+This structure provides a concise, effective guide to get viewers up and running with Crawl4AI in minutes.

docs/md_v2/tutorial/episode_02_Overview_of_Advanced_Features.md

@@ -0,0 +1,78 @@
+# Crawl4AI
+
+## Episode 2: Overview of Advanced Features
+
+### Quick Intro
+A general overview of advanced features like hooks, CSS selectors, and JSON CSS extraction.
+
+Here's a condensed outline for an **Overview of Advanced Features** video covering Crawl4AI's powerful customization and extraction options:
+
+---
+
+### **Overview of Advanced Features**
+
+1) **Introduction to Advanced Features**:
+ 
+   - Briefly introduce Crawl4AI’s advanced tools, which let users go beyond basic crawling to customize and fine-tune their scraping workflows.
+
+2) **Taking Screenshots**:
+ 
+   - Explain the screenshot capability for capturing page state and verifying content.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", screenshot=True)
+      ```
+   - Mention that screenshots are saved as a base64 string in `result`, allowing easy decoding and saving.
+
+3) **Media and Link Extraction**:
+ 
+   - Demonstrate how to pull all media (images, videos) and links (internal and external) from a page for deeper analysis or content gathering.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com")
+      print("Media:", result.media)
+      print("Links:", result.links)
+      ```
+
+4) **Custom User Agent**:
+ 
+   - Show how to set a custom user agent to disguise the crawler or simulate specific devices/browsers.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
+      ```
+
+5) **Custom Hooks for Enhanced Control**:
+ 
+   - Briefly cover how to use hooks, which allow custom actions like setting headers or handling login during the crawl.
+   - **Example**: Setting a custom header with `before_get_url` hook.
+      ```python
+      async def before_get_url(page):
+          await page.set_extra_http_headers({"X-Test-Header": "test"})
+      ```
+
+6) **CSS Selectors for Targeted Extraction**:
+ 
+   - Explain the use of CSS selectors to extract specific elements, ideal for structured data like articles or product details.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", css_selector="h2")
+      print("H2 Tags:", result.extracted_content)
+      ```
+
+7) **Crawling Inside Iframes**:
+ 
+   - Mention how enabling `process_iframes=True` allows extracting content within iframes, useful for sites with embedded content or ads.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", process_iframes=True)
+      ```
+
+8) **Wrap-Up**:
+ 
+   - Summarize these advanced features and how they allow users to customize every part of their web scraping experience.
+   - Tease upcoming videos where each feature will be explored in detail.
+
+---
+
+This covers each advanced feature with a brief example, providing a useful overview to prepare viewers for the more in-depth videos.

docs/md_v2/tutorial/episode_03_Browser_Configurations_&_Headless_Crawling.md

@@ -0,0 +1,65 @@
+# Crawl4AI
+
+## Episode 3: Browser Configurations & Headless Crawling
+
+### Quick Intro
+Explain browser options (`chromium`, `firefox`, `webkit`) and settings for headless mode, caching, and verbose logging.
+
+Here’s a streamlined outline for the **Browser Configurations & Headless Crawling** video:
+
+---
+
+### **Browser Configurations & Headless Crawling**
+
+1) **Overview of Browser Options**:
+
+   - Crawl4AI supports three browser engines:
+     - **Chromium** (default) - Highly compatible.
+     - **Firefox** - Great for specialized use cases.
+     - **Webkit** - Lightweight, ideal for basic needs.
+   - **Example**:
+      ```python
+      # Using Chromium (default)
+      crawler = AsyncWebCrawler(browser_type="chromium")
+      
+      # Using Firefox
+      crawler = AsyncWebCrawler(browser_type="firefox")
+      
+      # Using WebKit
+      crawler = AsyncWebCrawler(browser_type="webkit")
+      ```
+
+2) **Headless Mode**:
+
+   - Headless mode runs the browser without a visible GUI, making it faster and less resource-intensive.
+   - To enable or disable:
+      ```python
+      # Headless mode (default is True)
+      crawler = AsyncWebCrawler(headless=True)
+      
+      # Disable headless mode for debugging
+      crawler = AsyncWebCrawler(headless=False)
+      ```
+
+3) **Verbose Logging**:
+   - Use `verbose=True` to get detailed logs for each action, useful for debugging:
+      ```python
+      crawler = AsyncWebCrawler(verbose=True)
+      ```
+
+4) **Running a Basic Crawl with Configuration**:
+   - Example of a simple crawl with custom browser settings:
+      ```python
+      async with AsyncWebCrawler(browser_type="firefox", headless=True, verbose=True) as crawler:
+          result = await crawler.arun(url="https://www.example.com")
+          print(result.markdown[:500])  # Show first 500 characters
+      ```
+   - This example uses Firefox in headless mode with logging enabled, demonstrating the flexibility of Crawl4AI’s setup.
+
+5) **Recap & Next Steps**:
+   - Recap the power of selecting different browsers and running headless mode for speed and efficiency.
+   - Tease the next video: **Proxy & Security Settings** for navigating blocked or restricted content and protecting IP identity.
+
+---
+
+This breakdown covers browser configuration essentials in Crawl4AI, providing users with practical steps to optimize their scraping setup.

docs/md_v2/tutorial/episode_04_Advanced_Proxy_and_Security_Settings.md

@@ -0,0 +1,90 @@
+# Crawl4AI
+
+## Episode 4: Advanced Proxy and Security Settings
+
+### Quick Intro
+Showcase proxy configurations (HTTP, SOCKS5, authenticated proxies). Demo: Use rotating proxies and set custom headers to avoid IP blocking and enhance security.
+
+Here’s a focused outline for the **Proxy and Security Settings** video:
+
+---
+
+### **Proxy & Security Settings**
+
+1) **Why Use Proxies in Web Crawling**:
+
+   - Proxies are essential for bypassing IP-based restrictions, improving anonymity, and managing rate limits.
+   - Crawl4AI supports simple proxies, authenticated proxies, and proxy rotation for robust web scraping.
+
+2) **Basic Proxy Setup**:
+
+   - **Using a Simple Proxy**:
+     ```python
+     # HTTP proxy
+     crawler = AsyncWebCrawler(proxy="http://proxy.example.com:8080")
+     
+     # SOCKS proxy
+     crawler = AsyncWebCrawler(proxy="socks5://proxy.example.com:1080")
+     ```
+
+3) **Authenticated Proxies**:
+
+   - Use `proxy_config` for proxies requiring a username and password:
+     ```python
+     proxy_config = {
+         "server": "http://proxy.example.com:8080",
+         "username": "user",
+         "password": "pass"
+     }
+     crawler = AsyncWebCrawler(proxy_config=proxy_config)
+     ```
+
+4) **Rotating Proxies**:
+
+   - Rotating proxies helps avoid IP bans by switching IP addresses for each request:
+     ```python
+     async def get_next_proxy():
+         # Define proxy rotation logic here
+         return {"server": "http://next.proxy.com:8080"}
+     
+     async with AsyncWebCrawler() as crawler:
+         for url in urls:
+             proxy = await get_next_proxy()
+             crawler.update_proxy(proxy)
+             result = await crawler.arun(url=url)
+     ```
+   - This setup periodically switches the proxy for enhanced security and access.
+
+5) **Custom Headers for Additional Security**:
+
+   - Set custom headers to mask the crawler’s identity and avoid detection:
+     ```python
+     headers = {
+         "X-Forwarded-For": "203.0.113.195",
+         "Accept-Language": "en-US,en;q=0.9",
+         "Cache-Control": "no-cache",
+         "Pragma": "no-cache"
+     }
+     crawler = AsyncWebCrawler(headers=headers)
+     ```
+
+6) **Combining Proxies with Magic Mode for Anti-Bot Protection**:
+
+   - For sites with aggressive bot detection, combine `proxy` settings with `magic=True`:
+     ```python
+     async with AsyncWebCrawler(proxy="http://proxy.example.com:8080", headers={"Accept-Language": "en-US"}) as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             magic=True  # Enables anti-detection features
+         )
+     ```
+   - **Magic Mode** automatically enables user simulation, random timing, and browser property masking.
+
+7) **Wrap Up & Next Steps**:
+
+   - Summarize the importance of proxies and anti-detection in accessing restricted content and avoiding bans.
+   - Tease the next video: **JavaScript Execution and Handling Dynamic Content** for working with interactive and dynamically loaded pages.
+
+---
+
+This outline provides a practical guide to setting up proxies and security configurations, empowering users to navigate restricted sites while staying undetected.

docs/md_v2/tutorial/episode_05_JavaScript_Execution_and_Dynamic_Content_Handling.md

@@ -0,0 +1,97 @@
+# Crawl4AI
+
+## Episode 5: JavaScript Execution and Dynamic Content Handling
+
+### Quick Intro
+Explain JavaScript code injection with examples (e.g., simulating scrolling, clicking ‘load more’). Demo: Extract content from a page that uses dynamic loading with lazy-loaded images.
+
+Here’s a focused outline for the **JavaScript Execution and Dynamic Content Handling** video:
+
+---
+
+### **JavaScript Execution & Dynamic Content Handling**
+
+1) **Why JavaScript Execution Matters**:
+
+   - Many modern websites load content dynamically via JavaScript, requiring special handling to access all elements.
+   - Crawl4AI can execute JavaScript on pages, enabling it to interact with elements like “load more” buttons, infinite scrolls, and content that appears only after certain actions.
+
+2) **Basic JavaScript Execution**:
+
+   - Use `js_code` to execute JavaScript commands on a page:
+     ```python
+     # Scroll to bottom of the page
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code="window.scrollTo(0, document.body.scrollHeight);"
+     )
+     ```
+   - This command scrolls to the bottom, triggering any lazy-loaded or dynamically added content.
+
+3) **Multiple Commands & Simulating Clicks**:
+
+   - Combine multiple JavaScript commands to interact with elements like “load more” buttons:
+     ```python
+     js_commands = [
+         "window.scrollTo(0, document.body.scrollHeight);",
+         "document.querySelector('.load-more').click();"
+     ]
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code=js_commands
+     )
+     ```
+   - This script scrolls down and then clicks the “load more” button, useful for loading additional content blocks.
+
+4) **Waiting for Dynamic Content**:
+
+   - Use `wait_for` to ensure the page loads specific elements before proceeding:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code="window.scrollTo(0, document.body.scrollHeight);",
+         wait_for="css:.dynamic-content"  # Wait for elements with class `.dynamic-content`
+     )
+     ```
+   - This example waits until elements with `.dynamic-content` are loaded, helping to capture content that appears after JavaScript actions.
+
+5) **Handling Complex Dynamic Content (e.g., Infinite Scroll)**:
+
+   - Combine JavaScript execution with conditional waiting to handle infinite scrolls or paginated content:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code=[
+             "window.scrollTo(0, document.body.scrollHeight);",
+             "const loadMore = document.querySelector('.load-more'); if (loadMore) loadMore.click();"
+         ],
+         wait_for="js:() => document.querySelectorAll('.item').length > 10"  # Wait until 10 items are loaded
+     )
+     ```
+   - This example scrolls and clicks "load more" repeatedly, waiting each time for a specified number of items to load.
+
+6) **Complete Example: Dynamic Content Handling with Extraction**:
+
+   - Full example demonstrating a dynamic load and content extraction in one process:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             js_code=[
+                 "window.scrollTo(0, document.body.scrollHeight);",
+                 "document.querySelector('.load-more').click();"
+             ],
+             wait_for="css:.main-content",
+             css_selector=".main-content"
+         )
+         print(result.markdown[:500])  # Output the main content extracted
+     ```
+
+7) **Wrap Up & Next Steps**:
+
+   - Recap how JavaScript execution allows access to dynamic content, enabling powerful interactions.
+   - Tease the next video: **Content Cleaning and Fit Markdown** to show how Crawl4AI can extract only the most relevant content from complex pages.
+
+---
+
+This outline explains how to handle dynamic content and JavaScript-based interactions effectively, enabling users to scrape and interact with complex, modern websites.

docs/md_v2/tutorial/episode_06_Magic_Mode_and_Anti-Bot_Protection.md

@@ -0,0 +1,86 @@
+# Crawl4AI
+
+## Episode 6: Magic Mode and Anti-Bot Protection
+
+### Quick Intro
+Highlight `Magic Mode` and anti-bot features like user simulation, navigator overrides, and timing randomization. Demo: Access a site with anti-bot protection and show how `Magic Mode` seamlessly handles it.
+
+Here’s a concise outline for the **Magic Mode and Anti-Bot Protection** video:
+
+---
+
+### **Magic Mode & Anti-Bot Protection**
+
+1) **Why Anti-Bot Protection is Important**:
+
+   - Many websites use bot detection mechanisms to block automated scraping. Crawl4AI’s anti-detection features help avoid IP bans, CAPTCHAs, and access restrictions.
+   - **Magic Mode** is a one-step solution to enable a range of anti-bot features without complex configuration.
+
+2) **Enabling Magic Mode**:
+
+   - Simply set `magic=True` to activate Crawl4AI’s full anti-bot suite:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         magic=True  # Enables all anti-detection features
+     )
+     ```
+   - This enables a blend of stealth techniques, including masking automation signals, randomizing timings, and simulating real user behavior.
+
+3) **What Magic Mode Does Behind the Scenes**:
+
+   - **User Simulation**: Mimics human actions like mouse movements and scrolling.
+   - **Navigator Overrides**: Hides signals that indicate an automated browser.
+   - **Timing Randomization**: Adds random delays to simulate natural interaction patterns.
+   - **Cookie Handling**: Accepts and manages cookies dynamically to avoid triggers from cookie pop-ups.
+
+4) **Manual Anti-Bot Options (If Not Using Magic Mode)**:
+
+   - For granular control, you can configure individual settings without Magic Mode:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         simulate_user=True,        # Enables human-like behavior
+         override_navigator=True    # Hides automation fingerprints
+     )
+     ```
+   - **Use Cases**: This approach allows more specific adjustments when certain anti-bot features are needed but others are not.
+
+5) **Combining Proxies with Magic Mode**:
+
+   - To avoid rate limits or IP blocks, combine Magic Mode with a proxy:
+     ```python
+     async with AsyncWebCrawler(
+         proxy="http://proxy.example.com:8080",
+         headers={"Accept-Language": "en-US"}
+     ) as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             magic=True  # Full anti-detection
+         )
+     ```
+   - This setup maximizes stealth by pairing anti-bot detection with IP obfuscation.
+
+6) **Example of Anti-Bot Protection in Action**:
+
+   - Full example with Magic Mode and proxies to scrape a protected page:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com/protected-content",
+             magic=True,
+             proxy="http://proxy.example.com:8080",
+             wait_for="css:.content-loaded"  # Wait for the main content to load
+         )
+         print(result.markdown[:500])  # Display first 500 characters of the content
+     ```
+   - This example ensures seamless access to protected content by combining anti-detection and waiting for full content load.
+
+7) **Wrap Up & Next Steps**:
+
+   - Recap the power of Magic Mode and anti-bot features for handling restricted websites.
+   - Tease the next video: **Content Cleaning and Fit Markdown** to show how to extract clean and focused content from a page.
+
+---
+
+This outline shows users how to easily avoid bot detection and access restricted content, demonstrating both the power and simplicity of Magic Mode in Crawl4AI.

docs/md_v2/tutorial/episode_07_Content_Cleaning_and_Fit_Markdown.md

@@ -0,0 +1,89 @@
+# Crawl4AI
+
+## Episode 7: Content Cleaning and Fit Markdown
+
+### Quick Intro
+Explain content cleaning options, including `fit_markdown` to keep only the most relevant content. Demo: Extract and compare regular vs. fit markdown from a news site or blog.
+
+Here’s a streamlined outline for the **Content Cleaning and Fit Markdown** video:
+
+---
+
+### **Content Cleaning & Fit Markdown**
+
+1) **Overview of Content Cleaning in Crawl4AI**:
+
+   - Explain that web pages often include extra elements like ads, navigation bars, footers, and popups.
+   - Crawl4AI’s content cleaning features help extract only the main content, reducing noise and enhancing readability.
+
+2) **Basic Content Cleaning Options**:
+
+   - **Removing Unwanted Elements**: Exclude specific HTML tags, like forms or navigation bars:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         word_count_threshold=10,       # Filter out blocks with fewer than 10 words
+         excluded_tags=['form', 'nav'], # Exclude specific tags
+         remove_overlay_elements=True   # Remove popups and modals
+     )
+     ```
+   - This example extracts content while excluding forms, navigation, and modal overlays, ensuring clean results.
+
+3) **Fit Markdown for Main Content Extraction**:
+
+   - **What is Fit Markdown**: Uses advanced analysis to identify the most relevant content (ideal for articles, blogs, and documentation).
+   - **How it Works**: Analyzes content density, removes boilerplate elements, and maintains formatting for a clear output.
+   - **Example**:
+     ```python
+     result = await crawler.arun(url="https://example.com")
+     main_content = result.fit_markdown  # Extracted main content
+     print(main_content[:500])  # Display first 500 characters
+     ```
+   - Fit Markdown is especially helpful for long-form content like news articles or blog posts.
+
+4) **Comparing Fit Markdown with Regular Markdown**:
+
+   - **Fit Markdown** returns the primary content without extraneous elements.
+   - **Regular Markdown** includes all extracted text in markdown format.
+   - Example to show the difference:
+     ```python
+     all_content = result.markdown      # Full markdown
+     main_content = result.fit_markdown # Only the main content
+     
+     print(f"All Content Length: {len(all_content)}")
+     print(f"Main Content Length: {len(main_content)}")
+     ```
+   - This comparison shows the effectiveness of Fit Markdown in focusing on essential content.
+
+5) **Media and Metadata Handling with Content Cleaning**:
+
+   - **Media Extraction**: Crawl4AI captures images and videos with metadata like alt text, descriptions, and relevance scores:
+     ```python
+     for image in result.media["images"]:
+         print(f"Source: {image['src']}, Alt Text: {image['alt']}, Relevance Score: {image['score']}")
+     ```
+   - **Use Case**: Useful for saving only relevant images or videos from an article or content-heavy page.
+
+6) **Example of Clean Content Extraction in Action**:
+
+   - Full example extracting cleaned content and Fit Markdown:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             word_count_threshold=10,
+             excluded_tags=['nav', 'footer'],
+             remove_overlay_elements=True
+         )
+         print(result.fit_markdown[:500])  # Show main content
+     ```
+   - This example demonstrates content cleaning with settings for filtering noise and focusing on the core text.
+
+7) **Wrap Up & Next Steps**:
+
+   - Summarize the power of Crawl4AI’s content cleaning features and Fit Markdown for capturing clean, relevant content.
+   - Tease the next video: **Link Analysis and Smart Filtering** to focus on analyzing and filtering links within crawled pages.
+
+---
+
+This outline covers Crawl4AI’s content cleaning features and the unique benefits of Fit Markdown, showing users how to retrieve focused, high-quality content from web pages.

docs/md_v2/tutorial/episode_08_Media_Handling:_Images,_Videos,_and_Audio.md

@@ -0,0 +1,116 @@
+# Crawl4AI
+
+## Episode 8: Media Handling: Images, Videos, and Audio
+
+### Quick Intro
+Showcase Crawl4AI’s media extraction capabilities, including lazy-loaded media and metadata. Demo: Crawl a multimedia page, extract images, and show metadata (alt text, context, relevance score).
+
+Here’s a clear and focused outline for the **Media Handling: Images, Videos, and Audio** video:
+
+---
+
+### **Media Handling: Images, Videos, and Audio**
+
+1) **Overview of Media Extraction in Crawl4AI**:
+
+   - Crawl4AI can detect and extract different types of media (images, videos, and audio) along with useful metadata.
+   - This functionality is essential for gathering visual content from multimedia-heavy pages like e-commerce sites, news articles, and social media feeds.
+
+2) **Image Extraction and Metadata**:
+
+   - Crawl4AI captures images with detailed metadata, including:
+     - **Source URL**: The direct URL to the image.
+     - **Alt Text**: Image description if available.
+     - **Relevance Score**: A score (0–10) indicating how relevant the image is to the main content.
+     - **Context**: Text surrounding the image on the page.
+   - **Example**:
+     ```python
+     result = await crawler.arun(url="https://example.com")
+     
+     for image in result.media["images"]:
+         print(f"Source: {image['src']}")
+         print(f"Alt Text: {image['alt']}")
+         print(f"Relevance Score: {image['score']}")
+         print(f"Context: {image['context']}")
+     ```
+   - This example shows how to access each image’s metadata, making it easy to filter for the most relevant visuals.
+
+3) **Handling Lazy-Loaded Images**:
+
+   - Crawl4AI automatically supports lazy-loaded images, which are commonly used to optimize webpage loading.
+   - **Example with Wait for Lazy-Loaded Content**:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         wait_for="css:img[data-src]",  # Wait for lazy-loaded images
+         delay_before_return_html=2.0   # Allow extra time for images to load
+     )
+     ```
+   - This setup waits for lazy-loaded images to appear, ensuring they are fully captured.
+
+4) **Video Extraction and Metadata**:
+
+   - Crawl4AI captures video elements, including:
+     - **Source URL**: The video’s direct URL.
+     - **Type**: Format of the video (e.g., MP4).
+     - **Thumbnail**: A poster or thumbnail image if available.
+     - **Duration**: Video length, if metadata is provided.
+   - **Example**:
+     ```python
+     for video in result.media["videos"]:
+         print(f"Video Source: {video['src']}")
+         print(f"Type: {video['type']}")
+         print(f"Thumbnail: {video.get('poster')}")
+         print(f"Duration: {video.get('duration')}")
+     ```
+   - This allows users to gather video content and relevant details for further processing or analysis.
+
+5) **Audio Extraction and Metadata**:
+
+   - Audio elements can also be extracted, with metadata like:
+     - **Source URL**: The audio file’s direct URL.
+     - **Type**: Format of the audio file (e.g., MP3).
+     - **Duration**: Length of the audio, if available.
+   - **Example**:
+     ```python
+     for audio in result.media["audios"]:
+         print(f"Audio Source: {audio['src']}")
+         print(f"Type: {audio['type']}")
+         print(f"Duration: {audio.get('duration')}")
+     ```
+   - Useful for sites with podcasts, sound bites, or other audio content.
+
+6) **Filtering Media by Relevance**:
+
+   - Use metadata like relevance score to filter only the most useful media content:
+     ```python
+     relevant_images = [img for img in result.media["images"] if img['score'] > 5]
+     ```
+   - This is especially helpful for content-heavy pages where you only want media directly related to the main content.
+
+7) **Example: Full Media Extraction with Content Filtering**:
+
+   - Full example extracting images, videos, and audio along with filtering by relevance:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             word_count_threshold=10,  # Filter content blocks for relevance
+             exclude_external_images=True  # Only keep internal images
+         )
+         
+         # Display media summaries
+         print(f"Relevant Images: {len(relevant_images)}")
+         print(f"Videos: {len(result.media['videos'])}")
+         print(f"Audio Clips: {len(result.media['audios'])}")
+     ```
+   - This example shows how to capture and filter various media types, focusing on what’s most relevant.
+
+8) **Wrap Up & Next Steps**:
+
+   - Recap the comprehensive media extraction capabilities, emphasizing how metadata helps users focus on relevant content.
+   - Tease the next video: **Link Analysis and Smart Filtering** to explore how Crawl4AI handles internal, external, and social media links for more focused data gathering.
+
+---
+
+This outline provides users with a complete guide to handling images, videos, and audio in Crawl4AI, using metadata to enhance relevance and precision in multimedia extraction.

docs/md_v2/tutorial/episode_09_Link_Analysis_and_Smart_Filtering.md

@@ -0,0 +1,95 @@
+# Crawl4AI
+
+## Episode 9: Link Analysis and Smart Filtering
+
+### Quick Intro
+Walk through internal and external link classification, social media link filtering, and custom domain exclusion. Demo: Analyze links on a website, focusing on internal navigation vs. external or ad links.
+
+Here’s a focused outline for the **Link Analysis and Smart Filtering** video:
+
+---
+
+### **Link Analysis & Smart Filtering**
+
+1) **Importance of Link Analysis in Web Crawling**:
+
+   - Explain that web pages often contain numerous links, including internal links, external links, social media links, and ads.
+   - Crawl4AI’s link analysis and filtering options help extract only relevant links, enabling more targeted and efficient crawls.
+
+2) **Automatic Link Classification**:
+
+   - Crawl4AI categorizes links automatically into internal, external, and social media links.
+   - **Example**:
+     ```python
+     result = await crawler.arun(url="https://example.com")
+
+     # Access internal and external links
+     internal_links = result.links["internal"]
+     external_links = result.links["external"]
+
+     # Print first few links for each type
+     print("Internal Links:", internal_links[:3])
+     print("External Links:", external_links[:3])
+     ```
+
+3) **Filtering Out Unwanted Links**:
+
+   - **Exclude External Links**: Remove all links pointing to external sites.
+   - **Exclude Social Media Links**: Filter out social media domains like Facebook or Twitter.
+   - **Example**:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         exclude_external_links=True,         # Remove external links
+         exclude_social_media_links=True      # Remove social media links
+     )
+     ```
+
+4) **Custom Domain Filtering**:
+
+   - **Exclude Specific Domains**: Filter links from particular domains, e.g., ad sites.
+   - **Custom Social Media Domains**: Add additional social media domains if needed.
+   - **Example**:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         exclude_domains=["ads.com", "trackers.com"],
+         exclude_social_media_domains=["facebook.com", "linkedin.com"]
+     )
+     ```
+
+5) **Accessing Link Context and Metadata**:
+
+   - Crawl4AI provides additional metadata for each link, including its text, type (e.g., navigation or content), and surrounding context.
+   - **Example**:
+     ```python
+     for link in result.links["internal"]:
+         print(f"Link: {link['href']}, Text: {link['text']}, Context: {link['context']}")
+     ```
+   - **Use Case**: Helps users understand the relevance of links based on where they are placed on the page (e.g., navigation vs. article content).
+
+6) **Example of Comprehensive Link Filtering and Analysis**:
+
+   - Full example combining link filtering, metadata access, and contextual information:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             exclude_external_links=True,
+             exclude_social_media_links=True,
+             exclude_domains=["ads.com"],
+             css_selector=".main-content"  # Focus only on main content area
+         )
+         for link in result.links["internal"]:
+             print(f"Internal Link: {link['href']}, Text: {link['text']}, Context: {link['context']}")
+     ```
+   - This example filters unnecessary links, keeping only internal and relevant links from the main content area.
+
+7) **Wrap Up & Next Steps**:
+
+   - Summarize the benefits of link filtering for efficient crawling and relevant content extraction.
+   - Tease the next video: **Custom Headers, Identity Management, and User Simulation** to explain how to configure identity settings and simulate user behavior for stealthier crawls.
+
+---
+
+This outline provides a practical overview of Crawl4AI’s link analysis and filtering features, helping users target only essential links while eliminating distractions.

docs/md_v2/tutorial/episode_10_Custom_Headers,_Identity,_and_User_Simulation.md

@@ -0,0 +1,93 @@
+# Crawl4AI
+
+## Episode 10: Custom Headers, Identity, and User Simulation
+
+### Quick Intro
+Teach how to use custom headers, user-agent strings, and simulate real user interactions. Demo: Set custom user-agent and headers to access a site that blocks typical crawlers.
+
+Here’s a concise outline for the **Custom Headers, Identity Management, and User Simulation** video:
+
+---
+
+### **Custom Headers, Identity Management, & User Simulation**
+
+1) **Why Customize Headers and Identity in Crawling**:
+
+   - Websites often track request headers and browser properties to detect bots. Customizing headers and managing identity help make requests appear more human, improving access to restricted sites.
+
+2) **Setting Custom Headers**:
+
+   - Customize HTTP headers to mimic genuine browser requests or meet site-specific requirements:
+     ```python
+     headers = {
+         "Accept-Language": "en-US,en;q=0.9",
+         "X-Requested-With": "XMLHttpRequest",
+         "Cache-Control": "no-cache"
+     }
+     crawler = AsyncWebCrawler(headers=headers)
+     ```
+   - **Use Case**: Customize the `Accept-Language` header to simulate local user settings, or `Cache-Control` to bypass cache for fresh content.
+
+3) **Setting a Custom User Agent**:
+
+   - Some websites block requests from common crawler user agents. Setting a custom user agent string helps bypass these restrictions:
+     ```python
+     crawler = AsyncWebCrawler(
+         user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
+     )
+     ```
+   - **Tip**: Use user-agent strings from popular browsers (e.g., Chrome, Firefox) to improve access and reduce detection risks.
+
+4) **User Simulation for Human-like Behavior**:
+
+   - Enable `simulate_user=True` to mimic natural user interactions, such as random timing and simulated mouse movements:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         simulate_user=True  # Simulates human-like behavior
+     )
+     ```
+   - **Behavioral Effects**: Adds subtle variations in interactions, making the crawler harder to detect on bot-protected sites.
+
+5) **Navigator Overrides and Magic Mode for Full Identity Masking**:
+
+   - Use `override_navigator=True` to mask automation indicators like `navigator.webdriver`, which websites check to detect bots:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         override_navigator=True  # Masks bot-related signals
+     )
+     ```
+   - **Combining with Magic Mode**: For a complete anti-bot setup, combine these identity options with `magic=True` for maximum protection:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             magic=True,  # Enables all anti-bot detection features
+             user_agent="Custom-Agent",  # Custom agent with Magic Mode
+         )
+     ```
+   - This setup includes all anti-detection techniques like navigator masking, random timing, and user simulation.
+
+6) **Example: Comprehensive Setup for Identity Management**:
+
+   - A full example combining custom headers, user-agent, and user simulation for a realistic browsing profile:
+     ```python
+     async with AsyncWebCrawler(
+         headers={"Accept-Language": "en-US", "Cache-Control": "no-cache"},
+         user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/91.0",
+         simulate_user=True
+     ) as crawler:
+         result = await crawler.arun(url="https://example.com/secure-page")
+         print(result.markdown[:500])  # Display extracted content
+     ```
+   - This example enables detailed customization for evading detection and accessing protected pages smoothly.
+
+7) **Wrap Up & Next Steps**:
+
+   - Recap the value of headers, user-agent customization, and simulation in bypassing bot detection.
+   - Tease the next video: **Extraction Strategies: JSON CSS, LLM, and Cosine** to dive into structured data extraction methods for high-quality content retrieval.
+
+---
+
+This outline equips users with tools for managing crawler identity and human-like behavior, essential for accessing bot-protected or restricted websites.

docs/md_v2/tutorial/episode_11_1_Extraction_Strategies:_JSON_CSS.md

@@ -0,0 +1,186 @@
+Here’s a detailed outline for the **JSON-CSS Extraction Strategy** video, covering all key aspects and supported structures in Crawl4AI:
+
+---
+
+### **10.1 JSON-CSS Extraction Strategy**
+
+#### **1. Introduction to JSON-CSS Extraction**
+   - JSON-CSS Extraction is used for pulling structured data from pages with repeated patterns, like product listings, article feeds, or directories.
+   - This strategy allows defining a schema with CSS selectors and data fields, making it easy to capture nested, list-based, or singular elements.
+
+#### **2. Basic Schema Structure**
+   - **Schema Fields**: The schema has two main components:
+     - `baseSelector`: A CSS selector to locate the main elements you want to extract (e.g., each article or product block).
+     - `fields`: Defines the data fields for each element, supporting various data types and structures.
+
+#### **3. Simple Field Extraction**
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Sample Product</h2>
+         <span class="price">$19.99</span>
+         <p class="description">This is a sample product.</p>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "price", "selector": ".price", "type": "text"},
+             {"name": "description", "selector": ".description", "type": "text"}
+         ]
+     }
+     ```
+   - **Explanation**: Each field captures text content from specified CSS selectors within each `.product` element.
+
+#### **4. Supported Field Types: Text, Attribute, HTML, Regex**
+   - **Field Type Options**:
+     - `text`: Extracts visible text.
+     - `attribute`: Captures an HTML attribute (e.g., `src`, `href`).
+     - `html`: Extracts the raw HTML of an element.
+     - `regex`: Allows regex patterns to extract part of the text.
+
+   - **Example HTML** (including an image):
+     ```html
+     <div class="product">
+         <h2 class="title">Sample Product</h2>
+         <img class="product-image" src="image.jpg" alt="Product Image">
+         <span class="price">$19.99</span>
+         <p class="description">Limited time offer.</p>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "image_url", "selector": ".product-image", "type": "attribute", "attribute": "src"},
+             {"name": "price", "selector": ".price", "type": "regex", "pattern": r"\$(\d+\.\d+)"},
+             {"name": "description_html", "selector": ".description", "type": "html"}
+         ]
+     }
+     ```
+   - **Explanation**:
+     - `attribute`: Extracts the `src` attribute from `.product-image`.
+     - `regex`: Extracts the numeric part from `$19.99`.
+     - `html`: Retrieves the full HTML of the description element.
+
+#### **5. Nested Field Extraction**
+   - **Use Case**: Useful when content contains sub-elements, such as an article with author details within it.
+   - **Example HTML**:
+     ```html
+     <div class="article">
+         <h1 class="title">Sample Article</h1>
+         <div class="author">
+             <span class="name">John Doe</span>
+             <span class="bio">Writer and editor</span>
+         </div>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".article",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "author", "type": "nested", "selector": ".author", "fields": [
+                 {"name": "name", "selector": ".name", "type": "text"},
+                 {"name": "bio", "selector": ".bio", "type": "text"}
+             ]}
+         ]
+     }
+     ```
+   - **Explanation**:
+     - `nested`: Extracts `name` and `bio` within `.author`, grouping the author details in a single `author` object.
+
+#### **6. List and Nested List Extraction**
+   - **List**: Extracts multiple elements matching the selector as a list.
+   - **Nested List**: Allows lists within lists, useful for items with sub-lists (e.g., specifications for each product).
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Product with Features</h2>
+         <ul class="features">
+             <li class="feature">Feature 1</li>
+             <li class="feature">Feature 2</li>
+             <li class="feature">Feature 3</li>
+         </ul>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "features", "type": "list", "selector": ".features .feature", "fields": [
+                 {"name": "feature", "type": "text"}
+             ]}
+         ]
+     }
+     ```
+   - **Explanation**:
+     - `list`: Captures each `.feature` item within `.features`, outputting an array of features under the `features` field.
+
+#### **7. Transformations for Field Values**
+   - Transformations allow you to modify extracted values (e.g., converting to lowercase).
+   - Supported transformations: `lowercase`, `uppercase`, `strip`.
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Special Product</h2>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text", "transform": "uppercase"}
+         ]
+     }
+     ```
+   - **Explanation**: The `transform` property changes the `title` to uppercase, useful for standardized outputs.
+
+#### **8. Full JSON-CSS Extraction Example**
+   - Combining all elements in a single schema example for a comprehensive crawl:
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Featured Product</h2>
+         <img class="product-image" src="product.jpg">
+         <span class="price">$99.99</span>
+         <p class="description">Best product of the year.</p>
+         <ul class="features">
+             <li class="feature">Durable</li>
+             <li class="feature">Eco-friendly</li>
+         </ul>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text", "transform": "uppercase"},
+             {"name": "image_url", "selector": ".product-image", "type": "attribute", "attribute": "src"},
+             {"name": "price", "selector": ".price", "type": "regex", "pattern": r"\$(\d+\.\d+)"},
+             {"name": "description", "selector": ".description", "type": "html"},
+             {"name": "features", "type": "list", "selector": ".features .feature", "fields": [
+                 {"name": "feature", "type": "text"}
+             ]}
+         ]
+     }
+     ```
+   - **Explanation**: This schema captures and transforms each aspect of the product, illustrating the JSON-CSS strategy’s versatility for structured extraction.
+
+#### **9. Wrap Up & Next Steps**
+   - Summarize JSON-CSS Extraction’s flexibility for structured, pattern-based extraction.
+   - Tease the next video: **10.2 LLM Extraction Strategy**, focusing on using language models to extract data based on intelligent content analysis.
+
+---
+
+This outline covers each JSON-CSS Extraction option in Crawl4AI, with practical examples and schema configurations, making it a thorough guide for users.

docs/md_v2/tutorial/episode_11_2_Extraction_Strategies:_LLM.md

@@ -0,0 +1,153 @@
+# Crawl4AI
+
+## Episode 11: Extraction Strategies: JSON CSS, LLM, and Cosine
+
+### Quick Intro
+Introduce JSON CSS Extraction Strategy for structured data, LLM Extraction Strategy for intelligent parsing, and Cosine Strategy for clustering similar content. Demo: Use JSON CSS to scrape product details from an e-commerce site.
+
+Here’s a comprehensive outline for the **LLM Extraction Strategy** video, covering key details and example applications.
+
+---
+
+### **10.2 LLM Extraction Strategy**
+
+#### **1. Introduction to LLM Extraction Strategy**
+   - The LLM Extraction Strategy leverages language models to interpret and extract structured data from complex web content.
+   - Unlike traditional CSS selectors, this strategy uses natural language instructions and schemas to guide the extraction, ideal for unstructured or diverse content.
+   - Supports **OpenAI**, **Azure OpenAI**, **HuggingFace**, and **Ollama** models, enabling flexibility with both proprietary and open-source providers.
+
+#### **2. Key Components of LLM Extraction Strategy**
+   - **Provider**: Specifies the LLM provider (e.g., OpenAI, HuggingFace, Azure).
+   - **API Token**: Required for most providers, except Ollama (local LLM model).
+   - **Instruction**: Custom extraction instructions sent to the model, providing flexibility in how the data is structured and extracted.
+   - **Schema**: Optional, defines structured fields to organize extracted data into JSON format.
+   - **Extraction Type**: Supports `"block"` for simpler text blocks or `"schema"` when a structured output format is required.
+   - **Chunking Parameters**: Breaks down large documents, with options to adjust chunk size and overlap rate for more accurate extraction across lengthy texts.
+
+#### **3. Basic Extraction Example: OpenAI Model Pricing**
+   - **Goal**: Extract model names and their input and output fees from the OpenAI pricing page.
+   - **Schema Definition**:
+     - **Model Name**: Text for model identification.
+     - **Input Fee**: Token cost for input processing.
+     - **Output Fee**: Token cost for output generation.
+
+   - **Schema**:
+     ```python
+     class OpenAIModelFee(BaseModel):
+         model_name: str = Field(..., description="Name of the OpenAI model.")
+         input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+         output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
+     ```
+
+   - **Example Code**:
+     ```python
+     async def extract_openai_pricing():
+         async with AsyncWebCrawler() as crawler:
+             result = await crawler.arun(
+                 url="https://openai.com/api/pricing/",
+                 extraction_strategy=LLMExtractionStrategy(
+                     provider="openai/gpt-4o",
+                     api_token=os.getenv("OPENAI_API_KEY"),
+                     schema=OpenAIModelFee.schema(),
+                     extraction_type="schema",
+                     instruction="Extract model names and fees for input and output tokens from the page."
+                 ),
+                 bypass_cache=True
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - The extraction strategy combines a schema and detailed instruction to guide the LLM in capturing structured data.
+     - Each model’s name, input fee, and output fee are extracted in a JSON format.
+
+#### **4. Knowledge Graph Extraction Example**
+   - **Goal**: Extract entities and their relationships from a document for use in a knowledge graph.
+   - **Schema Definition**:
+     - **Entities**: Individual items with descriptions (e.g., people, organizations).
+     - **Relationships**: Connections between entities, including descriptions and relationship types.
+
+   - **Schema**:
+     ```python
+     class Entity(BaseModel):
+         name: str
+         description: str
+
+     class Relationship(BaseModel):
+         entity1: Entity
+         entity2: Entity
+         description: str
+         relation_type: str
+
+     class KnowledgeGraph(BaseModel):
+         entities: List[Entity]
+         relationships: List[Relationship]
+     ```
+
+   - **Example Code**:
+     ```python
+     async def extract_knowledge_graph():
+         extraction_strategy = LLMExtractionStrategy(
+             provider="azure/gpt-4o-mini",
+             api_token=os.getenv("AZURE_API_KEY"),
+             schema=KnowledgeGraph.schema(),
+             extraction_type="schema",
+             instruction="Extract entities and relationships from the content to build a knowledge graph."
+         )
+         async with AsyncWebCrawler() as crawler:
+             result = await crawler.arun(
+                 url="https://example.com/some-article",
+                 extraction_strategy=extraction_strategy,
+                 bypass_cache=True
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - In this setup, the LLM extracts entities and their relationships based on the schema and instruction.
+     - The schema organizes results into a JSON-based knowledge graph format.
+
+#### **5. Key Settings in LLM Extraction**
+   - **Chunking Options**:
+     - For long pages, set `chunk_token_threshold` to specify maximum token count per section.
+     - Adjust `overlap_rate` to control the overlap between chunks, useful for contextual consistency.
+   - **Example**:
+     ```python
+     extraction_strategy = LLMExtractionStrategy(
+         provider="openai/gpt-4",
+         api_token=os.getenv("OPENAI_API_KEY"),
+         chunk_token_threshold=3000,
+         overlap_rate=0.2,  # 20% overlap between chunks
+         instruction="Extract key insights and relationships."
+     )
+     ```
+   - This setup ensures that longer texts are divided into manageable chunks with slight overlap, enhancing the quality of extraction.
+
+#### **6. Flexible Provider Options for LLM Extraction**
+   - **Using Proprietary Models**: OpenAI, Azure, and HuggingFace provide robust language models, often suited for complex or detailed extractions.
+   - **Using Open-Source Models**: Ollama and other open-source models can be deployed locally, suitable for offline or cost-effective extraction.
+   - **Example Call**:
+     ```python
+     await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
+     await extract_structured_data_using_llm("openai/gpt-4o", os.getenv("OPENAI_API_KEY"))
+     await extract_structured_data_using_llm("ollama/llama3.2")   
+     ```
+
+#### **7. Complete Example of LLM Extraction Setup**
+   - Code to run both the OpenAI pricing and Knowledge Graph extractions, using various providers:
+     ```python
+     async def main():
+         await extract_openai_pricing()
+         await extract_knowledge_graph()
+     
+     if __name__ == "__main__":
+         asyncio.run(main())
+     ```
+
+#### **8. Wrap Up & Next Steps**
+   - Recap the power of LLM extraction for handling unstructured or complex data extraction tasks.
+   - Tease the next video: **10.3 Cosine Similarity Strategy** for clustering similar content based on semantic similarity.
+
+---
+
+This outline explains LLM Extraction in Crawl4AI, with examples showing how to extract structured data using custom schemas and instructions. It demonstrates flexibility with multiple providers, ensuring practical application for different use cases.

docs/md_v2/tutorial/episode_11_3_Extraction_Strategies:_Cosine.md

@@ -0,0 +1,136 @@
+# Crawl4AI
+
+## Episode 11: Extraction Strategies: JSON CSS, LLM, and Cosine
+
+### Quick Intro
+Introduce JSON CSS Extraction Strategy for structured data, LLM Extraction Strategy for intelligent parsing, and Cosine Strategy for clustering similar content. Demo: Use JSON CSS to scrape product details from an e-commerce site.
+
+Here’s a structured outline for the **Cosine Similarity Strategy** video, covering key concepts, configuration, and a practical example.
+
+---
+
+### **10.3 Cosine Similarity Strategy**
+
+#### **1. Introduction to Cosine Similarity Strategy**
+   - The Cosine Similarity Strategy clusters content by semantic similarity, offering an efficient alternative to LLM-based extraction, especially when speed is a priority.
+   - Ideal for grouping similar sections of text, this strategy is well-suited for pages with content sections that may need to be classified or tagged, like news articles, product descriptions, or reviews.
+
+#### **2. Key Configuration Options**
+   - **semantic_filter**: A keyword-based filter to focus on relevant content.
+   - **word_count_threshold**: Minimum number of words per cluster, filtering out shorter, less meaningful clusters.
+   - **max_dist**: Maximum allowable distance between elements in clusters, impacting cluster tightness.
+   - **linkage_method**: Method for hierarchical clustering, such as `'ward'` (for well-separated clusters).
+   - **top_k**: Specifies the number of top categories for each cluster.
+   - **model_name**: Defines the model for embeddings, such as `sentence-transformers/all-MiniLM-L6-v2`.
+   - **sim_threshold**: Minimum similarity threshold for filtering, allowing control over cluster relevance.
+
+#### **3. How Cosine Similarity Clustering Works**
+   - **Step 1**: Embeddings are generated for each text section, transforming them into vectors that capture semantic meaning.
+   - **Step 2**: Hierarchical clustering groups similar sections based on cosine similarity, forming clusters with related content.
+   - **Step 3**: Clusters are filtered based on word count, removing those below the `word_count_threshold`.
+   - **Step 4**: Each cluster is then categorized with tags, if enabled, providing context to each grouped content section.
+
+#### **4. Example Use Case: Clustering Blog Article Sections**
+   - **Goal**: Group related sections of a blog or news page to identify distinct topics or discussion areas.
+   - **Example HTML Sections**:
+     ```text
+     "The economy is showing signs of recovery, with markets up this quarter.",
+     "In the sports world, several major teams are preparing for the upcoming season.",
+     "New advancements in AI technology are reshaping the tech landscape.",
+     "Market analysts are optimistic about continued growth in tech stocks."
+     ```
+
+   - **Code Setup**:
+     ```python
+     async def extract_blog_sections():
+         extraction_strategy = CosineStrategy(
+             word_count_threshold=15,
+             max_dist=0.3,
+             sim_threshold=0.2,
+             model_name="sentence-transformers/all-MiniLM-L6-v2",
+             top_k=2
+         )
+         async with AsyncWebCrawler() as crawler:
+             url = "https://example.com/blog-page"
+             result = await crawler.arun(
+                 url=url,
+                 extraction_strategy=extraction_strategy,
+                 bypass_cache=True
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - **word_count_threshold**: Ensures only clusters with meaningful content are included.
+     - **sim_threshold**: Filters out clusters with low similarity, focusing on closely related sections.
+     - **top_k**: Selects top tags, useful for identifying main topics.
+
+#### **5. Applying Semantic Filtering with Cosine Similarity**
+   - **Semantic Filter**: Filters sections based on relevance to a specific keyword, such as “technology” for tech articles.
+   - **Example Code**:
+     ```python
+     extraction_strategy = CosineStrategy(
+         semantic_filter="technology",
+         word_count_threshold=10,
+         max_dist=0.25,
+         model_name="sentence-transformers/all-MiniLM-L6-v2"
+     )
+     ```
+   - **Explanation**:
+     - **semantic_filter**: Only sections with high similarity to the “technology” keyword will be included in the clustering, making it easy to focus on specific topics within a mixed-content page.
+
+#### **6. Clustering Product Reviews by Similarity**
+   - **Goal**: Organize product reviews by themes, such as “price,” “quality,” or “durability.”
+   - **Example Reviews**:
+     ```text
+     "The quality of this product is outstanding and well worth the price.",
+     "I found the product to be durable but a bit overpriced.",
+     "Great value for the money and long-lasting.",
+     "The build quality is good, but I expected a lower price point."
+     ```
+
+   - **Code Setup**:
+     ```python
+     async def extract_product_reviews():
+         extraction_strategy = CosineStrategy(
+             word_count_threshold=20,
+             max_dist=0.35,
+             sim_threshold=0.25,
+             model_name="sentence-transformers/all-MiniLM-L6-v2"
+         )
+         async with AsyncWebCrawler() as crawler:
+             url = "https://example.com/product-reviews"
+             result = await crawler.arun(
+                 url=url,
+                 extraction_strategy=extraction_strategy,
+                 bypass_cache=True
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - This configuration clusters similar reviews, grouping feedback by common themes, helping businesses understand customer sentiments around particular product aspects.
+
+#### **7. Performance Advantages of Cosine Strategy**
+   - **Speed**: The Cosine Similarity Strategy is faster than LLM-based extraction, as it doesn’t rely on API calls to external LLMs.
+   - **Local Processing**: The strategy runs locally with pre-trained sentence embeddings, ideal for high-throughput scenarios where cost and latency are concerns.
+   - **Comparison**: With a well-optimized local model, this method can perform clustering on large datasets quickly, making it suitable for tasks requiring rapid, repeated analysis.
+
+#### **8. Full Code Example for Clustering News Articles**
+   - **Code**:
+     ```python
+     async def main():
+         await extract_blog_sections()
+         await extract_product_reviews()
+     
+     if __name__ == "__main__":
+         asyncio.run(main())
+     ```
+
+#### **9. Wrap Up & Next Steps**
+   - Recap the efficiency and effectiveness of Cosine Similarity for clustering related content quickly.
+   - Close with a reminder of Crawl4AI’s flexibility across extraction strategies, and prompt users to experiment with different settings to optimize clustering for their specific content.
+
+---
+
+This outline covers Cosine Similarity Strategy’s speed and effectiveness, providing examples that showcase its potential for clustering various content types efficiently.

docs/md_v2/tutorial/episode_12_Session-Based_Crawling_for_Dynamic_Websites.md

@@ -0,0 +1,140 @@
+# Crawl4AI
+
+## Episode 12: Session-Based Crawling for Dynamic Websites
+
+### Quick Intro
+Show session management for handling websites with multiple pages or actions (like “load more” buttons). Demo: Crawl a paginated content page, persisting session data across multiple requests.
+
+Here’s a detailed outline for the **Session-Based Crawling for Dynamic Websites** video, explaining why sessions are necessary, how to use them, and providing practical examples and a visual diagram to illustrate the concept.
+
+---
+
+### **11. Session-Based Crawling for Dynamic Websites**
+
+#### **1. Introduction to Session-Based Crawling**
+   - **What is Session-Based Crawling**: Session-based crawling maintains a continuous browsing session across multiple page states, allowing the crawler to interact with a page and retrieve content that loads dynamically or based on user interactions.
+   - **Why It’s Needed**:
+     - In static pages, all content is available directly from a single URL.
+     - In dynamic websites, content often loads progressively or based on user actions (e.g., clicking “load more,” submitting forms, scrolling).
+     - Session-based crawling helps simulate user actions, capturing content that is otherwise hidden until specific actions are taken.
+
+#### **2. Conceptual Diagram for Session-Based Crawling**
+
+   ```mermaid
+   graph TD
+       Start[Start Session] --> S1[Initial State (S1)]
+       S1 -->|Crawl| Content1[Extract Content S1]
+       S1 -->|Action: Click Load More| S2[State S2]
+       S2 -->|Crawl| Content2[Extract Content S2]
+       S2 -->|Action: Scroll Down| S3[State S3]
+       S3 -->|Crawl| Content3[Extract Content S3]
+       S3 -->|Action: Submit Form| S4[Final State]
+       S4 -->|Crawl| Content4[Extract Content S4]
+       Content4 --> End[End Session]
+   ```
+
+   - **Explanation of Diagram**:
+     - **Start**: Initializes the session and opens the starting URL.
+     - **State Transitions**: Each action (e.g., clicking “load more,” scrolling) transitions to a new state, where additional content becomes available.
+     - **Session Persistence**: Keeps the same browsing session active, preserving the state and allowing for a sequence of actions to unfold.
+     - **End**: After reaching the final state, the session ends, and all accumulated content has been extracted.
+
+#### **3. Key Components of Session-Based Crawling in Crawl4AI**
+   - **Session ID**: A unique identifier to maintain the state across requests, allowing the crawler to “remember” previous actions.
+   - **JavaScript Execution**: Executes JavaScript commands (e.g., clicks, scrolls) to simulate interactions.
+   - **Wait Conditions**: Ensures the crawler waits for content to load in each state before moving on.
+   - **Sequential State Transitions**: By defining actions and wait conditions between states, the crawler can navigate through the page as a user would.
+
+#### **4. Basic Session Example: Multi-Step Content Loading**
+   - **Goal**: Crawl an article feed that requires several “load more” clicks to display additional content.
+   - **Code**:
+     ```python
+     async def crawl_article_feed():
+         async with AsyncWebCrawler() as crawler:
+             session_id = "feed_session"
+             
+             for page in range(3):
+                 result = await crawler.arun(
+                     url="https://example.com/articles",
+                     session_id=session_id,
+                     js_code="document.querySelector('.load-more-button').click();" if page > 0 else None,
+                     wait_for="css:.article",
+                     css_selector=".article"  # Target article elements
+                 )
+                 print(f"Page {page + 1}: Extracted {len(result.extracted_content)} articles")
+     ```
+   - **Explanation**:
+     - **session_id**: Ensures all requests share the same browsing state.
+     - **js_code**: Clicks the “load more” button after the initial page load, expanding content on each iteration.
+     - **wait_for**: Ensures articles have loaded after each click before extraction.
+
+#### **5. Advanced Example: E-Commerce Product Search with Filter Selection**
+   - **Goal**: Interact with filters on an e-commerce page to extract products based on selected criteria.
+   - **Example Steps**:
+     1. **State 1**: Load the main product page.
+     2. **State 2**: Apply a filter (e.g., “On Sale”) by selecting a checkbox.
+     3. **State 3**: Scroll to load additional products and capture updated results.
+
+   - **Code**:
+     ```python
+     async def extract_filtered_products():
+         async with AsyncWebCrawler() as crawler:
+             session_id = "product_session"
+             
+             # Step 1: Open product page
+             result = await crawler.arun(
+                 url="https://example.com/products",
+                 session_id=session_id,
+                 wait_for="css:.product-item"
+             )
+             
+             # Step 2: Apply filter (e.g., "On Sale")
+             result = await crawler.arun(
+                 url="https://example.com/products",
+                 session_id=session_id,
+                 js_code="document.querySelector('#sale-filter-checkbox').click();",
+                 wait_for="css:.product-item"
+             )
+
+             # Step 3: Scroll to load additional products
+             for _ in range(2):  # Scroll down twice
+                 result = await crawler.arun(
+                     url="https://example.com/products",
+                     session_id=session_id,
+                     js_code="window.scrollTo(0, document.body.scrollHeight);",
+                     wait_for="css:.product-item"
+                 )
+                 print(f"Loaded {len(result.extracted_content)} products after scroll")
+     ```
+   - **Explanation**:
+     - **State Persistence**: Each action (filter selection and scroll) builds on the previous session state.
+     - **Multiple Interactions**: Combines clicking a filter with scrolling, demonstrating how the session preserves these actions.
+
+#### **6. Key Benefits of Session-Based Crawling**
+   - **Accessing Hidden Content**: Retrieves data that loads only after user actions.
+   - **Simulating User Behavior**: Handles interactive elements such as “load more” buttons, dropdowns, and filters.
+   - **Maintaining Continuity Across States**: Enables a sequential process, moving logically from one state to the next, capturing all desired content without reloading the initial state each time.
+
+#### **7. Additional Configuration Tips**
+   - **Manage Session End**: Always conclude the session after the final state to release resources.
+   - **Optimize with Wait Conditions**: Use `wait_for` to ensure complete loading before each extraction.
+   - **Handling Errors in Session-Based Crawling**: Include error handling for interactions that may fail, ensuring robustness across state transitions.
+
+#### **8. Complete Code Example: Multi-Step Session Workflow**
+   - **Example**:
+     ```python
+     async def main():
+         await crawl_article_feed()
+         await extract_filtered_products()
+     
+     if __name__ == "__main__":
+         asyncio.run(main())
+     ```
+
+#### **9. Wrap Up & Next Steps**
+   - Recap the usefulness of session-based crawling for dynamic content extraction.
+   - Tease the next video: **Hooks and Custom Workflow with AsyncWebCrawler** to cover advanced customization options for further control over the crawling process.
+
+---
+
+This outline covers session-based crawling from both a conceptual and practical perspective, helping users understand its importance, configure it effectively, and use it to handle complex dynamic content.

docs/md_v2/tutorial/episode_13_Chunking_Strategies_for_Large_Text_Processing.md

@@ -0,0 +1,138 @@
+# Crawl4AI
+
+## Episode 13: Chunking Strategies for Large Text Processing
+
+### Quick Intro
+Explain Regex, NLP, and Fixed-Length chunking, and when to use each. Demo: Chunk a large article or document for processing by topics or sentences.
+
+Here’s a structured outline for the **Chunking Strategies for Large Text Processing** video, emphasizing how chunking works within extraction and why it’s crucial for effective data aggregation.
+
+Here’s a structured outline for the **Chunking Strategies for Large Text Processing** video, explaining each strategy, when to use it, and providing examples to illustrate.
+
+---
+
+### **12. Chunking Strategies for Large Text Processing**
+
+#### **1. Introduction to Chunking in Crawl4AI**
+   - **What is Chunking**: Chunking is the process of dividing large text into manageable sections or “chunks,” enabling efficient processing in extraction tasks.
+   - **Why It’s Needed**:
+     - When processing large text, feeding it directly into an extraction function (like `F(x)`) can overwhelm memory or token limits.
+     - Chunking breaks down `x` (the text) into smaller pieces, which are processed sequentially or in parallel by the extraction function, with the final result being an aggregation of all chunks’ processed output.
+
+#### **2. Key Chunking Strategies and Use Cases**
+   - Crawl4AI offers various chunking strategies to suit different text structures, chunk sizes, and processing requirements.
+   - **Choosing a Strategy**: Select based on the type of text (e.g., articles, transcripts) and extraction needs (e.g., simple splitting or context-sensitive processing).
+
+#### **3. Strategy 1: Regex-Based Chunking**
+   - **Description**: Uses regular expressions to split text based on specified patterns (e.g., paragraphs or section breaks).
+   - **Use Case**: Ideal for dividing text by paragraphs or larger logical blocks where sections are clearly separated by line breaks or punctuation.
+   - **Example**:
+     - **Pattern**: `r'\n\n'` for double line breaks.
+     ```python
+     chunker = RegexChunking(patterns=[r'\n\n'])
+     text_chunks = chunker.chunk(long_text)
+     print(text_chunks)  # Output: List of paragraphs
+     ```
+   - **Pros**: Flexible for pattern-based chunking.
+   - **Cons**: Limited to text with consistent formatting.
+
+#### **4. Strategy 2: NLP Sentence-Based Chunking**
+   - **Description**: Uses NLP to split text by sentences, ensuring grammatically complete segments.
+   - **Use Case**: Useful for extracting individual statements, such as in news articles, quotes, or legal text.
+   - **Example**:
+     ```python
+     chunker = NlpSentenceChunking()
+     sentence_chunks = chunker.chunk(long_text)
+     print(sentence_chunks)  # Output: List of sentences
+     ```
+   - **Pros**: Maintains sentence structure, ideal for tasks needing semantic completeness.
+   - **Cons**: May create very small chunks, which could limit contextual extraction.
+
+#### **5. Strategy 3: Topic-Based Segmentation Using TextTiling**
+   - **Description**: Segments text into topics using TextTiling, identifying topic shifts and key segments.
+   - **Use Case**: Ideal for long articles, reports, or essays where each section covers a different topic.
+   - **Example**:
+     ```python
+     chunker = TopicSegmentationChunking(num_keywords=3)
+     topic_chunks = chunker.chunk_with_topics(long_text)
+     print(topic_chunks)  # Output: List of topic segments with keywords
+     ```
+   - **Pros**: Groups related content, preserving topical coherence.
+   - **Cons**: Depends on identifiable topic shifts, which may not be present in all texts.
+
+#### **6. Strategy 4: Fixed-Length Word Chunking**
+   - **Description**: Splits text into chunks based on a fixed number of words.
+   - **Use Case**: Ideal for text where exact segment size is required, such as processing word-limited documents for LLMs.
+   - **Example**:
+     ```python
+     chunker = FixedLengthWordChunking(chunk_size=100)
+     word_chunks = chunker.chunk(long_text)
+     print(word_chunks)  # Output: List of 100-word chunks
+     ```
+   - **Pros**: Ensures uniform chunk sizes, suitable for token-based extraction limits.
+   - **Cons**: May split sentences, affecting semantic coherence.
+
+#### **7. Strategy 5: Sliding Window Chunking**
+   - **Description**: Uses a fixed window size with a step, creating overlapping chunks to maintain context.
+   - **Use Case**: Useful for maintaining context across sections, as with documents where context is needed for neighboring sections.
+   - **Example**:
+     ```python
+     chunker = SlidingWindowChunking(window_size=100, step=50)
+     window_chunks = chunker.chunk(long_text)
+     print(window_chunks)  # Output: List of overlapping word chunks
+     ```
+   - **Pros**: Retains context across adjacent chunks, ideal for complex semantic extraction.
+   - **Cons**: Overlap increases data size, potentially impacting processing time.
+
+#### **8. Strategy 6: Overlapping Window Chunking**
+   - **Description**: Similar to sliding windows but with a defined overlap, allowing chunks to share content at the edges.
+   - **Use Case**: Suitable for handling long texts with essential overlapping information, like research articles or medical records.
+   - **Example**:
+     ```python
+     chunker = OverlappingWindowChunking(window_size=1000, overlap=100)
+     overlap_chunks = chunker.chunk(long_text)
+     print(overlap_chunks)  # Output: List of overlapping chunks with defined overlap
+     ```
+   - **Pros**: Allows controlled overlap for consistent content coverage across chunks.
+   - **Cons**: Redundant data in overlapping areas may increase computation.
+
+#### **9. Practical Example: Using Chunking with an Extraction Strategy**
+   - **Goal**: Combine chunking with an extraction strategy to process large text effectively.
+   - **Example Code**:
+     ```python
+     from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+     async def extract_large_text():
+         # Initialize chunker and extraction strategy
+         chunker = FixedLengthWordChunking(chunk_size=200)
+         extraction_strategy = LLMExtractionStrategy(provider="openai/gpt-4", api_token="your_api_token")
+         
+         # Split text into chunks
+         text_chunks = chunker.chunk(large_text)
+         
+         async with AsyncWebCrawler() as crawler:
+             for chunk in text_chunks:
+                 result = await crawler.arun(
+                     url="https://example.com",
+                     extraction_strategy=extraction_strategy,
+                     content=chunk
+                 )
+                 print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - `chunker.chunk()`: Divides the `large_text` into smaller segments based on the chosen strategy.
+     - `extraction_strategy`: Processes each chunk separately, and results are then aggregated to form the final output.
+
+#### **10. Choosing the Right Chunking Strategy**
+   - **Text Structure**: If text has clear sections (e.g., paragraphs, topics), use Regex or Topic Segmentation.
+   - **Extraction Needs**: If context is crucial, consider Sliding or Overlapping Window Chunking.
+   - **Processing Constraints**: For word-limited extractions (e.g., LLMs with token limits), Fixed-Length Word Chunking is often most effective.
+
+#### **11. Wrap Up & Next Steps**
+   - Recap the benefits of each chunking strategy and when to use them in extraction workflows.
+   - Tease the next video: **Hooks and Custom Workflow with AsyncWebCrawler**, focusing on customizing crawler behavior with hooks for a fine-tuned extraction process.
+
+---
+
+This outline provides a complete understanding of chunking strategies, explaining each method’s strengths and best-use scenarios to help users process large texts effectively in Crawl4AI.

docs/md_v2/tutorial/episode_14_Hooks_and_Custom_Workflow_with_AsyncWebCrawler.md

@@ -0,0 +1,185 @@
+# Crawl4AI
+
+## Episode 14: Hooks and Custom Workflow with AsyncWebCrawler
+
+### Quick Intro
+Cover hooks (`on_browser_created`, `before_goto`, `after_goto`) to add custom workflows. Demo: Use hooks to add custom cookies or headers, log HTML, or trigger specific events on page load.
+
+Here’s a detailed outline for the **Hooks and Custom Workflow with AsyncWebCrawler** video, covering each hook’s purpose, usage, and example implementations.
+
+---
+
+### **13. Hooks and Custom Workflow with AsyncWebCrawler**
+
+#### **1. Introduction to Hooks in Crawl4AI**
+   - **What are Hooks**: Hooks are customizable entry points in the crawling process that allow users to inject custom actions or logic at specific stages.
+   - **Why Use Hooks**:
+     - They enable fine-grained control over the crawling workflow.
+     - Useful for performing additional tasks (e.g., logging, modifying headers) dynamically during the crawl.
+     - Hooks provide the flexibility to adapt the crawler to complex site structures or unique project needs.
+
+#### **2. Overview of Available Hooks**
+   - Crawl4AI offers seven key hooks to modify and control different stages in the crawling lifecycle:
+     - `on_browser_created`
+     - `on_user_agent_updated`
+     - `on_execution_started`
+     - `before_goto`
+     - `after_goto`
+     - `before_return_html`
+     - `before_retrieve_html`
+
+#### **3. Hook-by-Hook Explanation and Examples**
+
+---
+
+##### **Hook 1: `on_browser_created`**
+   - **Purpose**: Triggered right after the browser instance is created.
+   - **Use Case**:
+     - Initializing browser-specific settings or performing setup actions.
+     - Configuring browser extensions or scripts before any page is opened.
+   - **Example**:
+     ```python
+     async def log_browser_creation(browser):
+         print("Browser instance created:", browser)
+     
+     crawler.crawler_strategy.set_hook('on_browser_created', log_browser_creation)
+     ```
+   - **Explanation**: This hook logs the browser creation event, useful for tracking when a new browser instance starts.
+
+---
+
+##### **Hook 2: `on_user_agent_updated`**
+   - **Purpose**: Called whenever the user agent string is updated.
+   - **Use Case**:
+     - Modifying the user agent based on page requirements, e.g., changing to a mobile user agent for mobile-only pages.
+   - **Example**:
+     ```python
+     def update_user_agent(user_agent):
+         print(f"User Agent Updated: {user_agent}")
+     
+     crawler.crawler_strategy.set_hook('on_user_agent_updated', update_user_agent)
+     crawler.update_user_agent("Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X)")
+     ```
+   - **Explanation**: This hook provides a callback every time the user agent changes, helpful for debugging or dynamically altering user agent settings based on conditions.
+
+---
+
+##### **Hook 3: `on_execution_started`**
+   - **Purpose**: Called right before the crawler begins any interaction (e.g., JavaScript execution, clicks).
+   - **Use Case**:
+     - Performing setup actions, such as inserting cookies or initiating custom scripts.
+   - **Example**:
+     ```python
+     async def log_execution_start(page):
+         print("Execution started on page:", page.url)
+     
+     crawler.crawler_strategy.set_hook('on_execution_started', log_execution_start)
+     ```
+   - **Explanation**: Logs the start of any major interaction on the page, ideal for cases where you want to monitor each interaction.
+
+---
+
+##### **Hook 4: `before_goto`**
+   - **Purpose**: Triggered before navigating to a new URL with `page.goto()`.
+   - **Use Case**:
+     - Modifying request headers or setting up conditions right before the page loads.
+     - Adding headers or dynamically adjusting options for specific URLs.
+   - **Example**:
+     ```python
+     async def modify_headers_before_goto(page):
+         await page.set_extra_http_headers({"X-Custom-Header": "CustomValue"})
+         print("Custom headers set before navigation")
+     
+     crawler.crawler_strategy.set_hook('before_goto', modify_headers_before_goto)
+     ```
+   - **Explanation**: This hook allows injecting headers or altering settings based on the page’s needs, particularly useful for pages with custom requirements.
+
+---
+
+##### **Hook 5: `after_goto`**
+   - **Purpose**: Executed immediately after a page has loaded (after `page.goto()`).
+   - **Use Case**:
+     - Checking the loaded page state, modifying the DOM, or performing post-navigation actions (e.g., scrolling).
+   - **Example**:
+     ```python
+     async def post_navigation_scroll(page):
+         await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
+         print("Scrolled to the bottom after navigation")
+     
+     crawler.crawler_strategy.set_hook('after_goto', post_navigation_scroll)
+     ```
+   - **Explanation**: This hook scrolls to the bottom of the page after loading, which can help load dynamically added content like infinite scroll elements.
+
+---
+
+##### **Hook 6: `before_return_html`**
+   - **Purpose**: Called right before HTML content is retrieved and returned.
+   - **Use Case**:
+     - Removing overlays or cleaning up the page for a cleaner HTML extraction.
+   - **Example**:
+     ```python
+     async def remove_advertisements(page, html):
+         await page.evaluate("document.querySelectorAll('.ad-banner').forEach(el => el.remove());")
+         print("Advertisements removed before returning HTML")
+     
+     crawler.crawler_strategy.set_hook('before_return_html', remove_advertisements)
+     ```
+   - **Explanation**: The hook removes ad banners from the HTML before it’s retrieved, ensuring a cleaner data extraction.
+
+---
+
+##### **Hook 7: `before_retrieve_html`**
+   - **Purpose**: Runs right before Crawl4AI initiates HTML retrieval.
+   - **Use Case**:
+     - Finalizing any page adjustments (e.g., setting timers, waiting for specific elements).
+   - **Example**:
+     ```python
+     async def wait_for_content_before_retrieve(page):
+         await page.wait_for_selector('.main-content')
+         print("Main content loaded, ready to retrieve HTML")
+     
+     crawler.crawler_strategy.set_hook('before_retrieve_html', wait_for_content_before_retrieve)
+     ```
+   - **Explanation**: This hook waits for the main content to load before retrieving the HTML, ensuring that all essential content is captured.
+
+#### **4. Setting Hooks in Crawl4AI**
+   - **How to Set Hooks**:
+     - Use `set_hook` to define a custom function for each hook.
+     - Each hook function can be asynchronous (useful for actions like waiting or retrieving async data).
+   - **Example Setup**:
+     ```python
+     crawler.crawler_strategy.set_hook('on_browser_created', log_browser_creation)
+     crawler.crawler_strategy.set_hook('before_goto', modify_headers_before_goto)
+     crawler.crawler_strategy.set_hook('after_goto', post_navigation_scroll)
+     ```
+
+#### **5. Complete Example: Using Hooks for a Customized Crawl Workflow**
+   - **Goal**: Log each key step, set custom headers before navigation, and clean up the page before retrieving HTML.
+   - **Example Code**:
+     ```python
+     async def custom_crawl():
+         async with AsyncWebCrawler() as crawler:
+             # Set hooks for custom workflow
+             crawler.crawler_strategy.set_hook('on_browser_created', log_browser_creation)
+             crawler.crawler_strategy.set_hook('before_goto', modify_headers_before_goto)
+             crawler.crawler_strategy.set_hook('after_goto', post_navigation_scroll)
+             crawler.crawler_strategy.set_hook('before_return_html', remove_advertisements)
+             
+             # Perform the crawl
+             url = "https://example.com"
+             result = await crawler.arun(url=url)
+             print(result.html)  # Display or process HTML
+     ```
+
+#### **6. Benefits of Using Hooks in Custom Crawling Workflows**
+   - **Enhanced Control**: Hooks offer precise control over each stage, allowing adjustments based on content and structure.
+   - **Efficient Modifications**: Avoid reloading or restarting the session; hooks can alter actions dynamically.
+   - **Context-Sensitive Actions**: Hooks enable custom logic tailored to specific pages or sections, maximizing extraction quality.
+
+#### **7. Wrap Up & Next Steps**
+   - Recap how hooks empower customized workflows in Crawl4AI, enabling flexibility at every stage.
+   - Tease the next video: **Automating Post-Processing with Crawl4AI**, covering automated steps after data extraction.
+
+---
+
+This outline provides a thorough understanding of hooks, their practical applications, and examples for customizing the crawling workflow in Crawl4AI.

docs/notebooks/Crawl4AI_v0.3.72_Release_Announcement.ipynb

@@ -0,0 +1,235 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 🚀 Crawl4AI v0.3.72 Release Announcement\n",
+    "\n",
+    "Welcome to the new release of **Crawl4AI v0.3.72**! This notebook highlights the latest features and demonstrates how they work in real-time. Follow along to see each feature in action!\n",
+    "\n",
+    "### What’s New?\n",
+    "- ✨ `Fit Markdown`: Extracts only the main content from articles and blogs\n",
+    "- 🛡️ **Magic Mode**: Comprehensive anti-bot detection bypass\n",
+    "- 🌐 **Multi-browser support**: Switch between Chromium, Firefox, WebKit\n",
+    "- 🔍 **Knowledge Graph Extraction**: Generate structured graphs of entities & relationships from any URL\n",
+    "- 🤖 **Crawl4AI GPT Assistant**: Chat directly with our AI assistant for help, code generation, and faster learning (available [here](https://tinyurl.com/your-gpt-assistant-link))\n",
+    "\n",
+    "---\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 📥 Setup\n",
+    "To start, we'll install `Crawl4AI` along with Playwright and `nest_asyncio` to ensure compatibility with Colab’s asynchronous environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install Crawl4AI and dependencies\n",
+    "!pip install crawl4ai\n",
+    "!playwright install\n",
+    "!pip install nest_asyncio"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import nest_asyncio and apply it to allow asyncio in Colab\n",
+    "import nest_asyncio\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "print('Setup complete!')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## ✨ Feature 1: `Fit Markdown`\n",
+    "Extracts only the main content from articles and blog pages, removing sidebars, ads, and other distractions.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "from crawl4ai import AsyncWebCrawler\n",
+    "\n",
+    "async def fit_markdown_demo():\n",
+    "    async with AsyncWebCrawler() as crawler:\n",
+    "        result = await crawler.arun(url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\")\n",
+    "        print(result.fit_markdown)  # Shows main content in Markdown format\n",
+    "\n",
+    "# Run the demo\n",
+    "await fit_markdown_demo()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 🛡️ Feature 2: Magic Mode\n",
+    "Magic Mode bypasses anti-bot detection to make crawling more reliable on protected websites.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def magic_mode_demo():\n",
+    "    async with AsyncWebCrawler() as crawler:  # Enables anti-bot detection bypass\n",
+    "        result = await crawler.arun(\n",
+    "            url=\"https://www.reuters.com/markets/us/global-markets-view-usa-pix-2024-08-29/\",\n",
+    "            magic=True  # Enables magic mode\n",
+    "        )\n",
+    "        print(result.markdown)  # Shows the full content in Markdown format\n",
+    "\n",
+    "# Run the demo\n",
+    "await magic_mode_demo()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 🌐 Feature 3: Multi-Browser Support\n",
+    "Crawl4AI now supports Chromium, Firefox, and WebKit. Here’s how to specify Firefox for a crawl.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def multi_browser_demo():\n",
+    "    async with AsyncWebCrawler(browser_type=\"firefox\") as crawler:  # Using Firefox instead of default Chromium\n",
+    "        result = await crawler.arun(url=\"https://crawl4i.com\")\n",
+    "        print(result.markdown)  # Shows content extracted using Firefox\n",
+    "\n",
+    "# Run the demo\n",
+    "await multi_browser_demo()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## ✨ Put them all together\n",
+    "\n",
+    "Let's combine all the features to extract the main content from a blog post, bypass anti-bot detection, and generate a knowledge graph from the extracted content."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from crawl4ai.extraction_strategy import LLMExtractionStrategy\n",
+    "from pydantic import BaseModel\n",
+    "import json, os\n",
+    "from typing import List\n",
+    "\n",
+    "# Define classes for the knowledge graph structure\n",
+    "class Landmark(BaseModel):\n",
+    "    name: str\n",
+    "    description: str\n",
+    "    activities: list[str]  # E.g., visiting, sightseeing, relaxing\n",
+    "\n",
+    "class City(BaseModel):\n",
+    "    name: str\n",
+    "    description: str\n",
+    "    landmarks: list[Landmark]\n",
+    "    cultural_highlights: list[str]  # E.g., food, music, traditional crafts\n",
+    "\n",
+    "class TravelKnowledgeGraph(BaseModel):\n",
+    "    cities: list[City]  # Central Mexican cities to visit\n",
+    "\n",
+    "async def combined_demo():\n",
+    "    # Define the knowledge graph extraction strategy\n",
+    "    strategy = LLMExtractionStrategy(\n",
+    "        # provider=\"ollama/nemotron\",\n",
+    "        provider='openai/gpt-4o-mini', # Or any other provider, including Ollama and open source models\n",
+    "        pi_token=os.getenv('OPENAI_API_KEY'), # In case of Ollama just pass \"no-token\"\n",
+    "        schema=TravelKnowledgeGraph.schema(),\n",
+    "        instruction=(\n",
+    "            \"Extract cities, landmarks, and cultural highlights for places to visit in Central Mexico. \"\n",
+    "            \"For each city, list main landmarks with descriptions and activities, as well as cultural highlights.\"\n",
+    "        )\n",
+    "    )\n",
+    "\n",
+    "    # Set up the AsyncWebCrawler with multi-browser support, Magic Mode, and Fit Markdown\n",
+    "    async with AsyncWebCrawler(browser_type=\"firefox\") as crawler:\n",
+    "        result = await crawler.arun(\n",
+    "            url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\",\n",
+    "            extraction_strategy=strategy,\n",
+    "            bypass_cache=True,\n",
+    "            magic=True\n",
+    "        )\n",
+    "        \n",
+    "        # Display main article content in Fit Markdown format\n",
+    "        print(\"Extracted Main Content:\\n\", result.fit_markdown)\n",
+    "        \n",
+    "        # Display extracted knowledge graph of cities, landmarks, and cultural highlights\n",
+    "        if result.extracted_content:\n",
+    "            travel_graph = json.loads(result.extracted_content)\n",
+    "            print(\"\\nExtracted Knowledge Graph:\\n\", json.dumps(travel_graph, indent=2))\n",
+    "\n",
+    "# Run the combined demo\n",
+    "await combined_demo()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 🤖 Crawl4AI GPT Assistant\n",
+    "Chat with the Crawl4AI GPT Assistant for code generation, support, and learning Crawl4AI faster. Try it out [here](https://tinyurl.com/crawl4ai-gpt)!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

main.py

@@ -1,254 +1,348 @@
-import os
-import importlib
 import asyncio
-from functools import lru_cache
-import logging
-logging.basicConfig(level=logging.DEBUG)
-
-from fastapi import FastAPI, HTTPException, Request
-from fastapi.responses import HTMLResponse, JSONResponse
-from fastapi.staticfiles import StaticFiles
-from fastapi.middleware.cors import CORSMiddleware  
-from fastapi.templating import Jinja2Templates
-from fastapi.exceptions import RequestValidationError
-from starlette.middleware.base import BaseHTTPMiddleware
-from starlette.responses import FileResponse
-from fastapi.responses import RedirectResponse
-
-from pydantic import BaseModel, HttpUrl
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import List, Optional
-
-from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.database import get_total_count, clear_db
-
+from fastapi import FastAPI, HTTPException, BackgroundTasks, Request
+from fastapi.responses import JSONResponse
+from pydantic import BaseModel, HttpUrl, Field
+from typing import Optional, List, Dict, Any, Union
+import psutil
 import time
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
-
-# load .env file
-from dotenv import load_dotenv
-load_dotenv()
-
-# Configuration
-__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
-MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
-current_requests = 0
-lock = asyncio.Lock()
-
-app = FastAPI()
-
-# Initialize rate limiter
-def rate_limit_key_func(request: Request):
-    access_token = request.headers.get("access-token")
-    if access_token == os.environ.get('ACCESS_TOKEN'):
-        return None
-    return get_remote_address(request)
-
-limiter = Limiter(key_func=rate_limit_key_func)
-app.state.limiter = limiter
-
-# Dictionary to store last request times for each client
-last_request_times = {}
-last_rate_limit = {}
-
-
-def get_rate_limit():
-    limit = os.environ.get('ACCESS_PER_MIN', "5")
-    return f"{limit}/minute"
-
-# Custom rate limit exceeded handler
-async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
-    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
-        last_rate_limit[request.client.host] = time.time()
-    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
-    reset_at = time.time() + retry_after
-    return JSONResponse(
-        status_code=429,
-        content={
-            "detail": "Rate limit exceeded",
-            "limit": str(exc.limit.limit),
-            "retry_after": retry_after,
-            'reset_at': reset_at,
-            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
-        }
-    )
-    
-app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
-
-
-# Middleware for token-based bypass and per-request limit
-class RateLimitMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
-        access_token = request.headers.get("access-token")
-        if access_token == os.environ.get('ACCESS_TOKEN'):
-            return await call_next(request)
-        
-        path = request.url.path
-        if path in ["/crawl", "/old"]:
-            client_ip = request.client.host
-            current_time = time.time()
-            
-            # Check time since last request
-            if client_ip in last_request_times:
-                time_since_last_request = current_time - last_request_times[client_ip]
-                if time_since_last_request < SPAN:
-                    return JSONResponse(
-                        status_code=429,
-                        content={
-                            "detail": "Too many requests",
-                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
-                            "retry_after": max(0, SPAN - time_since_last_request),
-                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
-                        }
-                    )
-            
-            last_request_times[client_ip] = current_time
-
-        return await call_next(request)
-
-app.add_middleware(RateLimitMiddleware)
-
-# CORS configuration
-origins = ["*"]  # Allow all origins
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=origins,  # List of origins that are allowed to make requests
-    allow_credentials=True,
-    allow_methods=["*"],  # Allows all methods
-    allow_headers=["*"],  # Allows all headers
+import uuid
+from collections import defaultdict
+from urllib.parse import urlparse
+import math
+import logging
+from enum import Enum
+from dataclasses import dataclass
+import json
+from crawl4ai import AsyncWebCrawler, CrawlResult
+from crawl4ai.extraction_strategy import (
+    LLMExtractionStrategy,
+    CosineStrategy,
+    JsonCssExtractionStrategy,
 )
 
-# Mount the pages directory as a static directory
-app.mount("/pages", StaticFiles(directory=__location__ + "/pages"), name="pages")
-app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
-site_templates = Jinja2Templates(directory=__location__ + "/site")
-templates = Jinja2Templates(directory=__location__ + "/pages")
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
 
-@lru_cache()
-def get_crawler():
-    # Initialize and return a WebCrawler instance
-    crawler = WebCrawler(verbose = True)
-    crawler.warmup()
-    return crawler
+class TaskStatus(str, Enum):
+    PENDING = "pending"
+    PROCESSING = "processing"
+    COMPLETED = "completed"
+    FAILED = "failed"
+
+class CrawlerType(str, Enum):
+    BASIC = "basic"
+    LLM = "llm"
+    COSINE = "cosine"
+    JSON_CSS = "json_css"
+
+class ExtractionConfig(BaseModel):
+    type: CrawlerType
+    params: Dict[str, Any] = {}
 
 class CrawlRequest(BaseModel):
-    urls: List[str]
-    include_raw_html: Optional[bool] = False
-    bypass_cache: bool = False
-    extract_blocks: bool = True
-    word_count_threshold: Optional[int] = 5
-    extraction_strategy: Optional[str] = "NoExtractionStrategy"
-    extraction_strategy_args: Optional[dict] = {}
-    chunking_strategy: Optional[str] = "RegexChunking"
-    chunking_strategy_args: Optional[dict] = {}
+    urls: Union[HttpUrl, List[HttpUrl]]
+    extraction_config: Optional[ExtractionConfig] = None
+    crawler_params: Dict[str, Any] = {}
+    priority: int = Field(default=5, ge=1, le=10)
+    ttl: Optional[int] = 3600
+    js_code: Optional[List[str]] = None
+    wait_for: Optional[str] = None
     css_selector: Optional[str] = None
-    screenshot: Optional[bool] = False
-    user_agent: Optional[str] = None
-    verbose: Optional[bool] = True
+    screenshot: bool = False
+    magic: bool = False
 
-@app.get("/")
-def read_root():
-    return RedirectResponse(url="/mkdocs")
+@dataclass
+class TaskInfo:
+    id: str
+    status: TaskStatus
+    result: Optional[Union[CrawlResult, List[CrawlResult]]] = None
+    error: Optional[str] = None
+    created_at: float = time.time()
+    ttl: int = 3600
 
-@app.get("/old", response_class=HTMLResponse)
-@limiter.limit(get_rate_limit())
-async def read_index(request: Request):
-    partials_dir = os.path.join(__location__, "pages", "partial")
-    partials = {}
+class ResourceMonitor:
+    def __init__(self, max_concurrent_tasks: int = 10):
+        self.max_concurrent_tasks = max_concurrent_tasks
+        self.memory_threshold = 0.85
+        self.cpu_threshold = 0.90
+        self._last_check = 0
+        self._check_interval = 1  # seconds
+        self._last_available_slots = max_concurrent_tasks
 
-    for filename in os.listdir(partials_dir):
-        if filename.endswith(".html"):
-            with open(os.path.join(partials_dir, filename), "r", encoding="utf8") as file:
-                partials[filename[:-5]] = file.read()
+    async def get_available_slots(self) -> int:
+        current_time = time.time()
+        if current_time - self._last_check < self._check_interval:
+            return self._last_available_slots
 
-    return templates.TemplateResponse("index.html", {"request": request, **partials})
+        mem_usage = psutil.virtual_memory().percent / 100
+        cpu_usage = psutil.cpu_percent() / 100
 
-@app.get("/total-count")
-async def get_total_url_count():
-    count = get_total_count()
-    return JSONResponse(content={"count": count})
+        memory_factor = max(0, (self.memory_threshold - mem_usage) / self.memory_threshold)
+        cpu_factor = max(0, (self.cpu_threshold - cpu_usage) / self.cpu_threshold)
 
-@app.get("/clear-db")
-async def clear_database():
-    # clear_db()
-    return JSONResponse(content={"message": "Database cleared."})
+        self._last_available_slots = math.floor(
+            self.max_concurrent_tasks * min(memory_factor, cpu_factor)
+        )
+        self._last_check = current_time
 
-def import_strategy(module_name: str, class_name: str, *args, **kwargs):
-    try:
-        module = importlib.import_module(module_name)
-        strategy_class = getattr(module, class_name)
-        return strategy_class(*args, **kwargs)
-    except ImportError:
-        print("ImportError: Module not found.")
-        raise HTTPException(status_code=400, detail=f"Module {module_name} not found.")
-    except AttributeError:
-        print("AttributeError: Class not found.")
-        raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")
+        return self._last_available_slots
+
+class TaskManager:
+    def __init__(self, cleanup_interval: int = 300):
+        self.tasks: Dict[str, TaskInfo] = {}
+        self.high_priority = asyncio.PriorityQueue()
+        self.low_priority = asyncio.PriorityQueue()
+        self.cleanup_interval = cleanup_interval
+        self.cleanup_task = None
+
+    async def start(self):
+        self.cleanup_task = asyncio.create_task(self._cleanup_loop())
+
+    async def stop(self):
+        if self.cleanup_task:
+            self.cleanup_task.cancel()
+            try:
+                await self.cleanup_task
+            except asyncio.CancelledError:
+                pass
+
+    async def add_task(self, task_id: str, priority: int, ttl: int) -> None:
+        task_info = TaskInfo(id=task_id, status=TaskStatus.PENDING, ttl=ttl)
+        self.tasks[task_id] = task_info
+        queue = self.high_priority if priority > 5 else self.low_priority
+        await queue.put((-priority, task_id))  # Negative for proper priority ordering
+
+    async def get_next_task(self) -> Optional[str]:
+        try:
+            # Try high priority first
+            _, task_id = await asyncio.wait_for(self.high_priority.get(), timeout=0.1)
+            return task_id
+        except asyncio.TimeoutError:
+            try:
+                # Then try low priority
+                _, task_id = await asyncio.wait_for(self.low_priority.get(), timeout=0.1)
+                return task_id
+            except asyncio.TimeoutError:
+                return None
+
+    def update_task(self, task_id: str, status: TaskStatus, result: Any = None, error: str = None):
+        if task_id in self.tasks:
+            task_info = self.tasks[task_id]
+            task_info.status = status
+            task_info.result = result
+            task_info.error = error
+
+    def get_task(self, task_id: str) -> Optional[TaskInfo]:
+        return self.tasks.get(task_id)
+
+    async def _cleanup_loop(self):
+        while True:
+            try:
+                await asyncio.sleep(self.cleanup_interval)
+                current_time = time.time()
+                expired_tasks = [
+                    task_id
+                    for task_id, task in self.tasks.items()
+                    if current_time - task.created_at > task.ttl
+                    and task.status in [TaskStatus.COMPLETED, TaskStatus.FAILED]
+                ]
+                for task_id in expired_tasks:
+                    del self.tasks[task_id]
+            except Exception as e:
+                logger.error(f"Error in cleanup loop: {e}")
+
+class CrawlerPool:
+    def __init__(self, max_size: int = 10):
+        self.max_size = max_size
+        self.active_crawlers: Dict[AsyncWebCrawler, float] = {}
+        self._lock = asyncio.Lock()
+
+    async def acquire(self, **kwargs) -> AsyncWebCrawler:
+        async with self._lock:
+            # Clean up inactive crawlers
+            current_time = time.time()
+            inactive = [
+                crawler
+                for crawler, last_used in self.active_crawlers.items()
+                if current_time - last_used > 600  # 10 minutes timeout
+            ]
+            for crawler in inactive:
+                await crawler.__aexit__(None, None, None)
+                del self.active_crawlers[crawler]
+
+            # Create new crawler if needed
+            if len(self.active_crawlers) < self.max_size:
+                crawler = AsyncWebCrawler(**kwargs)
+                await crawler.__aenter__()
+                self.active_crawlers[crawler] = current_time
+                return crawler
+
+            # Reuse least recently used crawler
+            crawler = min(self.active_crawlers.items(), key=lambda x: x[1])[0]
+            self.active_crawlers[crawler] = current_time
+            return crawler
+
+    async def release(self, crawler: AsyncWebCrawler):
+        async with self._lock:
+            if crawler in self.active_crawlers:
+                self.active_crawlers[crawler] = time.time()
+
+    async def cleanup(self):
+        async with self._lock:
+            for crawler in list(self.active_crawlers.keys()):
+                await crawler.__aexit__(None, None, None)
+            self.active_crawlers.clear()
+
+class CrawlerService:
+    def __init__(self, max_concurrent_tasks: int = 10):
+        self.resource_monitor = ResourceMonitor(max_concurrent_tasks)
+        self.task_manager = TaskManager()
+        self.crawler_pool = CrawlerPool(max_concurrent_tasks)
+        self._processing_task = None
+
+    async def start(self):
+        await self.task_manager.start()
+        self._processing_task = asyncio.create_task(self._process_queue())
+
+    async def stop(self):
+        if self._processing_task:
+            self._processing_task.cancel()
+            try:
+                await self._processing_task
+            except asyncio.CancelledError:
+                pass
+        await self.task_manager.stop()
+        await self.crawler_pool.cleanup()
+
+    def _create_extraction_strategy(self, config: ExtractionConfig):
+        if not config:
+            return None
+
+        if config.type == CrawlerType.LLM:
+            return LLMExtractionStrategy(**config.params)
+        elif config.type == CrawlerType.COSINE:
+            return CosineStrategy(**config.params)
+        elif config.type == CrawlerType.JSON_CSS:
+            return JsonCssExtractionStrategy(**config.params)
+        return None
+
+    async def submit_task(self, request: CrawlRequest) -> str:
+        task_id = str(uuid.uuid4())
+        await self.task_manager.add_task(task_id, request.priority, request.ttl or 3600)
+        
+        # Store request data with task
+        self.task_manager.tasks[task_id].request = request
+        
+        return task_id
+
+    async def _process_queue(self):
+        while True:
+            try:
+                available_slots = await self.resource_monitor.get_available_slots()
+                if available_slots <= 0:
+                    await asyncio.sleep(1)
+                    continue
+
+                task_id = await self.task_manager.get_next_task()
+                if not task_id:
+                    await asyncio.sleep(1)
+                    continue
+
+                task_info = self.task_manager.get_task(task_id)
+                if not task_info:
+                    continue
+
+                request = task_info.request
+                self.task_manager.update_task(task_id, TaskStatus.PROCESSING)
+
+                try:
+                    crawler = await self.crawler_pool.acquire(**request.crawler_params)
+                    
+                    extraction_strategy = self._create_extraction_strategy(request.extraction_config)
+                    
+                    if isinstance(request.urls, list):
+                        results = await crawler.arun_many(
+                            urls=[str(url) for url in request.urls],
+                            extraction_strategy=extraction_strategy,
+                            js_code=request.js_code,
+                            wait_for=request.wait_for,
+                            css_selector=request.css_selector,
+                            screenshot=request.screenshot,
+                            magic=request.magic,
+                            **request.extra,
+                        )
+                    else:
+                        results = await crawler.arun(
+                            url=str(request.urls),
+                            extraction_strategy=extraction_strategy,
+                            js_code=request.js_code,
+                            wait_for=request.wait_for,
+                            css_selector=request.css_selector,
+                            screenshot=request.screenshot,
+                            magic=request.magic,
+                            **request.extra,
+                        )
+
+                    await self.crawler_pool.release(crawler)
+                    self.task_manager.update_task(task_id, TaskStatus.COMPLETED, results)
+
+                except Exception as e:
+                    logger.error(f"Error processing task {task_id}: {str(e)}")
+                    self.task_manager.update_task(task_id, TaskStatus.FAILED, error=str(e))
+
+            except Exception as e:
+                logger.error(f"Error in queue processing: {str(e)}")
+                await asyncio.sleep(1)
+
+app = FastAPI(title="Crawl4AI API")
+crawler_service = CrawlerService()
+
+@app.on_event("startup")
+async def startup_event():
+    await crawler_service.start()
+
+@app.on_event("shutdown")
+async def shutdown_event():
+    await crawler_service.stop()
 
 @app.post("/crawl")
-@limiter.limit(get_rate_limit())
-async def crawl_urls(crawl_request: CrawlRequest, request: Request):
-    logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
-    global current_requests
-    async with lock:
-        if current_requests >= MAX_CONCURRENT_REQUESTS:
-            raise HTTPException(status_code=429, detail="Too many requests - please try again later.")
-        current_requests += 1
+async def crawl(request: CrawlRequest) -> Dict[str, str]:
+    task_id = await crawler_service.submit_task(request)
+    return {"task_id": task_id}
 
-    try:
-        logging.debug("[LOG] Loading extraction and chunking strategies...")
-        crawl_request.extraction_strategy_args['verbose'] = True
-        crawl_request.chunking_strategy_args['verbose'] = True
-        
-        extraction_strategy = import_strategy("crawl4ai.extraction_strategy", crawl_request.extraction_strategy, **crawl_request.extraction_strategy_args)
-        chunking_strategy = import_strategy("crawl4ai.chunking_strategy", crawl_request.chunking_strategy, **crawl_request.chunking_strategy_args)
+@app.get("/task/{task_id}")
+async def get_task_status(task_id: str):
+    task_info = crawler_service.task_manager.get_task(task_id)
+    if not task_info:
+        raise HTTPException(status_code=404, detail="Task not found")
 
-        # Use ThreadPoolExecutor to run the synchronous WebCrawler in async manner
-        logging.debug("[LOG] Running the WebCrawler...")
-        with ThreadPoolExecutor() as executor:
-            loop = asyncio.get_event_loop()
-            futures = [
-                loop.run_in_executor(
-                    executor, 
-                    get_crawler().run,
-                    str(url),
-                    crawl_request.word_count_threshold,
-                    extraction_strategy,
-                    chunking_strategy,
-                    crawl_request.bypass_cache,
-                    crawl_request.css_selector,
-                    crawl_request.screenshot,
-                    crawl_request.user_agent,
-                    crawl_request.verbose
-                )
-                for url in crawl_request.urls
-            ]
-            results = await asyncio.gather(*futures)
+    response = {
+        "status": task_info.status,
+        "created_at": task_info.created_at,
+    }
 
-        # if include_raw_html is False, remove the raw HTML content from the results
-        if not crawl_request.include_raw_html:
-            for result in results:
-                result.html = None
+    if task_info.status == TaskStatus.COMPLETED:
+        # Convert CrawlResult to dict for JSON response
+        if isinstance(task_info.result, list):
+            response["results"] = [result.dict() for result in task_info.result]
+        else:
+            response["result"] = task_info.result.dict()
+    elif task_info.status == TaskStatus.FAILED:
+        response["error"] = task_info.error
 
-        return {"results": [result.model_dump() for result in results]}
-    finally:
-        async with lock:
-            current_requests -= 1
-            
-@app.get("/strategies/extraction", response_class=JSONResponse)
-async def get_extraction_strategies():
-    with open(f"{__location__}/docs/extraction_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
-
-@app.get("/strategies/chunking", response_class=JSONResponse)
-async def get_chunking_strategies():
-    with open(f"{__location__}/docs/chunking_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
+    return response
 
+@app.get("/health")
+async def health_check():
+    available_slots = await crawler_service.resource_monitor.get_available_slots()
+    memory = psutil.virtual_memory()
+    return {
+        "status": "healthy",
+        "available_slots": available_slots,
+        "memory_usage": memory.percent,
+        "cpu_usage": psutil.cpu_percent(),
+    }
 
 if __name__ == "__main__":
     import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8888)
+    uvicorn.run(app, host="0.0.0.0", port=11235)

main_v0.py

@@ -0,0 +1,254 @@
+import os
+import importlib
+import asyncio
+from functools import lru_cache
+import logging
+logging.basicConfig(level=logging.DEBUG)
+
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.responses import HTMLResponse, JSONResponse
+from fastapi.staticfiles import StaticFiles
+from fastapi.middleware.cors import CORSMiddleware  
+from fastapi.templating import Jinja2Templates
+from fastapi.exceptions import RequestValidationError
+from starlette.middleware.base import BaseHTTPMiddleware
+from starlette.responses import FileResponse
+from fastapi.responses import RedirectResponse
+
+from pydantic import BaseModel, HttpUrl
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from typing import List, Optional
+
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.database import get_total_count, clear_db
+
+import time
+from slowapi import Limiter, _rate_limit_exceeded_handler
+from slowapi.util import get_remote_address
+from slowapi.errors import RateLimitExceeded
+
+# load .env file
+from dotenv import load_dotenv
+load_dotenv()
+
+# Configuration
+__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
+MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
+current_requests = 0
+lock = asyncio.Lock()
+
+app = FastAPI()
+
+# Initialize rate limiter
+def rate_limit_key_func(request: Request):
+    access_token = request.headers.get("access-token")
+    if access_token == os.environ.get('ACCESS_TOKEN'):
+        return None
+    return get_remote_address(request)
+
+limiter = Limiter(key_func=rate_limit_key_func)
+app.state.limiter = limiter
+
+# Dictionary to store last request times for each client
+last_request_times = {}
+last_rate_limit = {}
+
+
+def get_rate_limit():
+    limit = os.environ.get('ACCESS_PER_MIN', "5")
+    return f"{limit}/minute"
+
+# Custom rate limit exceeded handler
+async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
+    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
+        last_rate_limit[request.client.host] = time.time()
+    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
+    reset_at = time.time() + retry_after
+    return JSONResponse(
+        status_code=429,
+        content={
+            "detail": "Rate limit exceeded",
+            "limit": str(exc.limit.limit),
+            "retry_after": retry_after,
+            'reset_at': reset_at,
+            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
+        }
+    )
+    
+app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
+
+
+# Middleware for token-based bypass and per-request limit
+class RateLimitMiddleware(BaseHTTPMiddleware):
+    async def dispatch(self, request: Request, call_next):
+        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
+        access_token = request.headers.get("access-token")
+        if access_token == os.environ.get('ACCESS_TOKEN'):
+            return await call_next(request)
+        
+        path = request.url.path
+        if path in ["/crawl", "/old"]:
+            client_ip = request.client.host
+            current_time = time.time()
+            
+            # Check time since last request
+            if client_ip in last_request_times:
+                time_since_last_request = current_time - last_request_times[client_ip]
+                if time_since_last_request < SPAN:
+                    return JSONResponse(
+                        status_code=429,
+                        content={
+                            "detail": "Too many requests",
+                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
+                            "retry_after": max(0, SPAN - time_since_last_request),
+                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
+                        }
+                    )
+            
+            last_request_times[client_ip] = current_time
+
+        return await call_next(request)
+
+app.add_middleware(RateLimitMiddleware)
+
+# CORS configuration
+origins = ["*"]  # Allow all origins
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=origins,  # List of origins that are allowed to make requests
+    allow_credentials=True,
+    allow_methods=["*"],  # Allows all methods
+    allow_headers=["*"],  # Allows all headers
+)
+
+# Mount the pages directory as a static directory
+app.mount("/pages", StaticFiles(directory=__location__ + "/pages"), name="pages")
+app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
+site_templates = Jinja2Templates(directory=__location__ + "/site")
+templates = Jinja2Templates(directory=__location__ + "/pages")
+
+@lru_cache()
+def get_crawler():
+    # Initialize and return a WebCrawler instance
+    crawler = WebCrawler(verbose = True)
+    crawler.warmup()
+    return crawler
+
+class CrawlRequest(BaseModel):
+    urls: List[str]
+    include_raw_html: Optional[bool] = False
+    bypass_cache: bool = False
+    extract_blocks: bool = True
+    word_count_threshold: Optional[int] = 5
+    extraction_strategy: Optional[str] = "NoExtractionStrategy"
+    extraction_strategy_args: Optional[dict] = {}
+    chunking_strategy: Optional[str] = "RegexChunking"
+    chunking_strategy_args: Optional[dict] = {}
+    css_selector: Optional[str] = None
+    screenshot: Optional[bool] = False
+    user_agent: Optional[str] = None
+    verbose: Optional[bool] = True
+
+@app.get("/")
+def read_root():
+    return RedirectResponse(url="/mkdocs")
+
+@app.get("/old", response_class=HTMLResponse)
+@limiter.limit(get_rate_limit())
+async def read_index(request: Request):
+    partials_dir = os.path.join(__location__, "pages", "partial")
+    partials = {}
+
+    for filename in os.listdir(partials_dir):
+        if filename.endswith(".html"):
+            with open(os.path.join(partials_dir, filename), "r", encoding="utf8") as file:
+                partials[filename[:-5]] = file.read()
+
+    return templates.TemplateResponse("index.html", {"request": request, **partials})
+
+@app.get("/total-count")
+async def get_total_url_count():
+    count = get_total_count()
+    return JSONResponse(content={"count": count})
+
+@app.get("/clear-db")
+async def clear_database():
+    # clear_db()
+    return JSONResponse(content={"message": "Database cleared."})
+
+def import_strategy(module_name: str, class_name: str, *args, **kwargs):
+    try:
+        module = importlib.import_module(module_name)
+        strategy_class = getattr(module, class_name)
+        return strategy_class(*args, **kwargs)
+    except ImportError:
+        print("ImportError: Module not found.")
+        raise HTTPException(status_code=400, detail=f"Module {module_name} not found.")
+    except AttributeError:
+        print("AttributeError: Class not found.")
+        raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")
+
+@app.post("/crawl")
+@limiter.limit(get_rate_limit())
+async def crawl_urls(crawl_request: CrawlRequest, request: Request):
+    logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
+    global current_requests
+    async with lock:
+        if current_requests >= MAX_CONCURRENT_REQUESTS:
+            raise HTTPException(status_code=429, detail="Too many requests - please try again later.")
+        current_requests += 1
+
+    try:
+        logging.debug("[LOG] Loading extraction and chunking strategies...")
+        crawl_request.extraction_strategy_args['verbose'] = True
+        crawl_request.chunking_strategy_args['verbose'] = True
+        
+        extraction_strategy = import_strategy("crawl4ai.extraction_strategy", crawl_request.extraction_strategy, **crawl_request.extraction_strategy_args)
+        chunking_strategy = import_strategy("crawl4ai.chunking_strategy", crawl_request.chunking_strategy, **crawl_request.chunking_strategy_args)
+
+        # Use ThreadPoolExecutor to run the synchronous WebCrawler in async manner
+        logging.debug("[LOG] Running the WebCrawler...")
+        with ThreadPoolExecutor() as executor:
+            loop = asyncio.get_event_loop()
+            futures = [
+                loop.run_in_executor(
+                    executor, 
+                    get_crawler().run,
+                    str(url),
+                    crawl_request.word_count_threshold,
+                    extraction_strategy,
+                    chunking_strategy,
+                    crawl_request.bypass_cache,
+                    crawl_request.css_selector,
+                    crawl_request.screenshot,
+                    crawl_request.user_agent,
+                    crawl_request.verbose
+                )
+                for url in crawl_request.urls
+            ]
+            results = await asyncio.gather(*futures)
+
+        # if include_raw_html is False, remove the raw HTML content from the results
+        if not crawl_request.include_raw_html:
+            for result in results:
+                result.html = None
+
+        return {"results": [result.model_dump() for result in results]}
+    finally:
+        async with lock:
+            current_requests -= 1
+            
+@app.get("/strategies/extraction", response_class=JSONResponse)
+async def get_extraction_strategies():
+    with open(f"{__location__}/docs/extraction_strategies.json", "r") as file:
+        return JSONResponse(content=file.read())
+
+@app.get("/strategies/chunking", response_class=JSONResponse)
+async def get_chunking_strategies():
+    with open(f"{__location__}/docs/chunking_strategies.json", "r") as file:
+        return JSONResponse(content=file.read())
+
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=8888)

mkdocs.yml

@@ -8,6 +8,7 @@ docs_dir: docs/md_v2
 nav:
   - Home: 'index.md'
   - 'Installation': 'basic/installation.md'
+  - 'Docker Deplotment': 'basic/docker-deploymeny.md'
   - 'Quick Start': 'basic/quickstart.md'
   
   - Basic:
@@ -34,10 +35,29 @@ nav:
     - 'Chunking': 'extraction/chunking.md'
 
   - API Reference:
+    - 'Parameters Table': 'api/parameters.md'
     - 'AsyncWebCrawler': 'api/async-webcrawler.md'
     - 'AsyncWebCrawler.arun()': 'api/arun.md'
     - 'CrawlResult': 'api/crawl-result.md'
     - 'Strategies': 'api/strategies.md'
+    
+  - Tutorial:
+    - '1. Getting Started': 'tutorial/episode_01_Introduction_to_Crawl4AI_and_Basic_Installation.md'
+    - '2. Advanced Features': 'tutorial/episode_02_Overview_of_Advanced_Features.md'
+    - '3. Browser Setup': 'tutorial/episode_03_Browser_Configurations_&_Headless_Crawling.md'
+    - '4. Proxy Settings': 'tutorial/episode_04_Advanced_Proxy_and_Security_Settings.md'
+    - '5. Dynamic Content': 'tutorial/episode_05_JavaScript_Execution_and_Dynamic_Content_Handling.md'
+    - '6. Magic Mode': 'tutorial/episode_06_Magic_Mode_and_Anti-Bot_Protection.md'
+    - '7. Content Cleaning': 'tutorial/episode_07_Content_Cleaning_and_Fit_Markdown.md'
+    - '8. Media Handling': 'tutorial/episode_08_Media_Handling:_Images,_Videos,_and_Audio.md'
+    - '9. Link Analysis': 'tutorial/episode_09_Link_Analysis_and_Smart_Filtering.md'
+    - '10. User Simulation': 'tutorial/episode_10_Custom_Headers,_Identity,_and_User_Simulation.md'
+    - '11.1. JSON CSS': 'tutorial/episode_11_1_Extraction_Strategies:_JSON_CSS.md'
+    - '11.2. LLM Strategy': 'tutorial/episode_11_2_Extraction_Strategies:_LLM.md'
+    - '11.3. Cosine Strategy': 'tutorial/episode_11_3_Extraction_Strategies:_Cosine.md'
+    - '12. Session Crawling': 'tutorial/episode_12_Session-Based_Crawling_for_Dynamic_Websites.md'
+    - '13. Text Chunking': 'tutorial/episode_13_Chunking_Strategies_for_Large_Text_Processing.md'
+    - '14. Custom Workflows': 'tutorial/episode_14_Hooks_and_Custom_Workflow_with_AsyncWebCrawler.md'
 
 
 theme:

requirements-dev.txt

@@ -1,2 +1,5 @@
 -r requirements.txt
-pytest
+pytest
+pytest-asyncio
+selenium
+setuptools

requirements.txt

@@ -1,11 +1,11 @@
-aiosqlite==0.20.0
-html2text==2024.2.26
-lxml==5.3.0
-litellm==1.48.0
-numpy>=1.26.0,<2.1.1
-pillow==10.4.0
-playwright==1.47.0
-python-dotenv==1.0.1
-requests>=2.26.0,<2.32.3
-beautifulsoup4==4.12.3
-playwright_stealth==1.0.6
+aiosqlite~=0.20
+html2text~=2024.2
+lxml~=5.3
+litellm~=1.48
+numpy>=1.26.0,<3
+pillow~=10.4
+playwright>=1.47,<1.48
+python-dotenv~=1.0
+requests~=2.26
+beautifulsoup4~=4.12
+playwright_stealth~=1.0

setup.py

@@ -23,7 +23,7 @@ with open(os.path.join(__location__, "requirements.txt")) as f:
     requirements = f.read().splitlines()
     
 # Read version from __init__.py
-with open("crawl4ai/__init__.py") as f:
+with open("crawl4ai/_version.py") as f:
     for line in f:
         if line.startswith("__version__"):
             version = line.split("=")[1].strip().strip('"')
@@ -31,9 +31,11 @@ with open("crawl4ai/__init__.py") as f:
 
 # Define the requirements for different environments
 default_requirements = requirements
-torch_requirements = ["torch", "nltk", "spacy", "scikit-learn"]
-transformer_requirements = ["transformers", "tokenizers", "onnxruntime"]
-cosine_similarity_requirements = ["torch", "transformers", "nltk", "spacy"]
+# torch_requirements = ["torch", "nltk", "spacy", "scikit-learn"]
+# transformer_requirements = ["transformers", "tokenizers", "onnxruntime"]
+torch_requirements = ["torch", "nltk",  "scikit-learn"]
+transformer_requirements = ["transformers", "tokenizers"]
+cosine_similarity_requirements = ["torch", "transformers", "nltk" ]
 sync_requirements = ["selenium"]
 
 def install_playwright():

tests/test_docker.py

@@ -0,0 +1,299 @@
+import requests
+import json
+import time
+import sys
+import base64
+import os
+from typing import Dict, Any
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:8000"):
+        self.base_url = base_url
+        
+    def submit_and_wait(self, request_data: Dict[str, Any], timeout: int = 300) -> Dict[str, Any]:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data)
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}")
+            status = result.json()
+            
+            if status["status"] == "failed":
+                print("Task failed:", status.get("error"))
+                raise Exception(f"Task failed: {status.get('error')}")
+                
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+
+def test_docker_deployment(version="basic"):
+    tester = Crawl4AiTester()
+    print(f"Testing Crawl4AI Docker {version} version")
+    
+    # Health check with timeout and retry
+    max_retries = 5
+    for i in range(max_retries):
+        try:
+            health = requests.get(f"{tester.base_url}/health", timeout=10)
+            print("Health check:", health.json())
+            break
+        except requests.exceptions.RequestException as e:
+            if i == max_retries - 1:
+                print(f"Failed to connect after {max_retries} attempts")
+                sys.exit(1)
+            print(f"Waiting for service to start (attempt {i+1}/{max_retries})...")
+            time.sleep(5)
+    
+    # Test cases based on version
+    test_basic_crawl(tester)
+    if version in ["full", "transformer"]:
+        test_cosine_extraction(tester)
+
+    # test_js_execution(tester)
+    # test_css_selector(tester)
+    # test_structured_extraction(tester)
+    # test_llm_extraction(tester)
+    # test_llm_with_ollama(tester)
+    # test_screenshot(tester)
+    
+
+def test_basic_crawl(tester: Crawl4AiTester):
+    print("\n=== Testing Basic Crawl ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+    assert len(result["result"]["markdown"]) > 0
+
+def test_js_execution(tester: Crawl4AiTester):
+    print("\n=== Testing JS Execution ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "js_code": [
+            "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+        ],
+        "wait_for": "article.tease-card:nth-child(10)",
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"JS execution result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_css_selector(tester: Crawl4AiTester):
+    print("\n=== Testing CSS Selector ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 7,
+        "css_selector": ".wide-tease-item__description",
+        "crawler_params": {
+            "headless": True
+        },
+        "extra": {"word_count_threshold": 10}
+        
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"CSS selector result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_structured_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Structured Extraction ===")
+    schema = {
+        "name": "Coinbase Crypto Prices",
+        "baseSelector": ".cds-tableRow-t45thuk",
+        "fields": [
+            {
+                "name": "crypto",
+                "selector": "td:nth-child(1) h2",
+                "type": "text",
+            },
+            {
+                "name": "symbol",
+                "selector": "td:nth-child(1) p",
+                "type": "text",
+            },
+            {
+                "name": "price",
+                "selector": "td:nth-child(2)",
+                "type": "text",
+            }
+        ],
+    }
+    
+    request = {
+        "urls": "https://www.coinbase.com/explore",
+        "priority": 9,
+        "extraction_config": {
+            "type": "json_css",
+            "params": {
+                "schema": schema
+            }
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    extracted = json.loads(result["result"]["extracted_content"])
+    print(f"Extracted {len(extracted)} items")
+    print("Sample item:", json.dumps(extracted[0], indent=2))
+    assert result["result"]["success"]
+    assert len(extracted) > 0
+
+def test_llm_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing LLM Extraction ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "model_name": {
+                "type": "string",
+                "description": "Name of the OpenAI model."
+            },
+            "input_fee": {
+                "type": "string",
+                "description": "Fee for input token for the OpenAI model."
+            },
+            "output_fee": {
+                "type": "string",
+                "description": "Fee for output token for the OpenAI model."
+            }
+        },
+        "required": ["model_name", "input_fee", "output_fee"]
+    }
+    
+    request = {
+        "urls": "https://openai.com/api/pricing",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "openai/gpt-4o-mini",
+                "api_token": os.getenv("OPENAI_API_KEY"),
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": """From the crawled content, extract all mentioned model names along with their fees for input and output tokens."""
+            }
+        },
+        "crawler_params": {"word_count_threshold": 1}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} model pricing entries")
+        print("Sample entry:", json.dumps(extracted[0], indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"LLM extraction test failed (might be due to missing API key): {str(e)}")
+
+def test_llm_with_ollama(tester: Crawl4AiTester):
+    print("\n=== Testing LLM with Ollama ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "article_title": {
+                "type": "string",
+                "description": "The main title of the news article"
+            },
+            "summary": {
+                "type": "string",
+                "description": "A brief summary of the article content"
+            },
+            "main_topics": {
+                "type": "array",
+                "items": {"type": "string"},
+                "description": "Main topics or themes discussed in the article"
+            }
+        }
+    }
+    
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "ollama/llama2",
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": "Extract the main article information including title, summary, and main topics."
+            }
+        },
+        "extra": {"word_count_threshold": 1},
+        "crawler_params": {"verbose": True}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print("Extracted content:", json.dumps(extracted, indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Ollama extraction test failed: {str(e)}")
+
+def test_cosine_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Cosine Extraction ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "cosine",
+            "params": {
+                "semantic_filter": "business finance economy",
+                "word_count_threshold": 10,
+                "max_dist": 0.2,
+                "top_k": 3
+            }
+        }
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} text clusters")
+        print("First cluster tags:", extracted[0]["tags"])
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Cosine extraction test failed: {str(e)}")
+
+def test_screenshot(tester: Crawl4AiTester):
+    print("\n=== Testing Screenshot ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 5,
+        "screenshot": True,
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Screenshot captured:", bool(result["result"]["screenshot"]))
+    
+    if result["result"]["screenshot"]:
+        # Save screenshot
+        screenshot_data = base64.b64decode(result["result"]["screenshot"])
+        with open("test_screenshot.jpg", "wb") as f:
+            f.write(screenshot_data)
+        print("Screenshot saved as test_screenshot.jpg")
+    
+    assert result["result"]["success"]
+
+if __name__ == "__main__":
+    version = sys.argv[1] if len(sys.argv) > 1 else "basic"
+    # version = "full"
+    test_docker_deployment(version)

tests/test_main.py

@@ -0,0 +1,281 @@
+import asyncio
+import aiohttp
+import json
+import time
+import os
+from typing import Optional, Dict, Any
+from pydantic import BaseModel, HttpUrl
+
+class NBCNewsAPITest:
+    def __init__(self, base_url: str = "http://localhost:8000"):
+        self.base_url = base_url
+        self.session = None
+
+    async def __aenter__(self):
+        self.session = aiohttp.ClientSession()
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        if self.session:
+            await self.session.close()
+
+    async def submit_crawl(self, request_data: Dict[str, Any]) -> str:
+        async with self.session.post(f"{self.base_url}/crawl", json=request_data) as response:
+            result = await response.json()
+            return result["task_id"]
+
+    async def get_task_status(self, task_id: str) -> Dict[str, Any]:
+        async with self.session.get(f"{self.base_url}/task/{task_id}") as response:
+            return await response.json()
+
+    async def wait_for_task(self, task_id: str, timeout: int = 300, poll_interval: int = 2) -> Dict[str, Any]:
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
+
+            status = await self.get_task_status(task_id)
+            if status["status"] in ["completed", "failed"]:
+                return status
+
+            await asyncio.sleep(poll_interval)
+
+    async def check_health(self) -> Dict[str, Any]:
+        async with self.session.get(f"{self.base_url}/health") as response:
+            return await response.json()
+
+async def test_basic_crawl():
+    print("\n=== Testing Basic Crawl ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 10
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+        assert result["status"] == "completed"
+        assert "result" in result
+        assert result["result"]["success"]
+
+async def test_js_execution():
+    print("\n=== Testing JS Execution ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 8,
+            "js_code": [
+                "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+            ],
+            "wait_for": "article.tease-card:nth-child(10)",
+            "crawler_params": {
+                "headless": True
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"JS execution result length: {len(result['result']['markdown'])}")
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+
+async def test_css_selector():
+    print("\n=== Testing CSS Selector ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 7,
+            "css_selector": ".wide-tease-item__description"
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"CSS selector result length: {len(result['result']['markdown'])}")
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+
+async def test_structured_extraction():
+    print("\n=== Testing Structured Extraction ===")
+    async with NBCNewsAPITest() as api:
+        schema = {
+            "name": "NBC News Articles",
+            "baseSelector": "article.tease-card",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h2",
+                    "type": "text"
+                },
+                {
+                    "name": "description",
+                    "selector": ".tease-card__description",
+                    "type": "text"
+                },
+                {
+                    "name": "link",
+                    "selector": "a",
+                    "type": "attribute",
+                    "attribute": "href"
+                }
+            ]
+        }
+        
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 9,
+            "extraction_config": {
+                "type": "json_css",
+                "params": {
+                    "schema": schema
+                }
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} articles")
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+        assert len(extracted) > 0
+
+async def test_batch_crawl():
+    print("\n=== Testing Batch Crawl ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": [
+                "https://www.nbcnews.com/business",
+                "https://www.nbcnews.com/business/consumer",
+                "https://www.nbcnews.com/business/economy"
+            ],
+            "priority": 6,
+            "crawler_params": {
+                "headless": True
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"Batch crawl completed, got {len(result['results'])} results")
+        assert result["status"] == "completed"
+        assert "results" in result
+        assert len(result["results"]) == 3
+
+async def test_llm_extraction():
+    print("\n=== Testing LLM Extraction with Ollama ===")
+    async with NBCNewsAPITest() as api:
+        schema = {
+            "type": "object",
+            "properties": {
+                "article_title": {
+                    "type": "string",
+                    "description": "The main title of the news article"
+                },
+                "summary": {
+                    "type": "string",
+                    "description": "A brief summary of the article content"
+                },
+                "main_topics": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Main topics or themes discussed in the article"
+                }
+            },
+            "required": ["article_title", "summary", "main_topics"]
+        }
+
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 8,
+            "extraction_config": {
+                "type": "llm",
+                "params": {
+                    "provider": "openai/gpt-4o-mini",
+                    "api_key": os.getenv("OLLAMA_API_KEY"),
+                    "schema": schema,
+                    "extraction_type": "schema",
+                    "instruction": """Extract the main article information including title, a brief summary, and main topics discussed. 
+                    Focus on the primary business news article on the page."""
+                }
+            },
+            "crawler_params": {
+                "headless": True,
+                "word_count_threshold": 1
+            }
+        }
+        
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        
+        if result["status"] == "completed":
+            extracted = json.loads(result["result"]["extracted_content"])
+            print(f"Extracted article analysis:")
+            print(json.dumps(extracted, indent=2))
+        
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+
+async def test_screenshot():
+    print("\n=== Testing Screenshot ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 5,
+            "screenshot": True,
+            "crawler_params": {
+                "headless": True
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print("Screenshot captured:", bool(result["result"]["screenshot"]))
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+        assert result["result"]["screenshot"] is not None
+
+async def test_priority_handling():
+    print("\n=== Testing Priority Handling ===")
+    async with NBCNewsAPITest() as api:
+        # Submit low priority task first
+        low_priority = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 1,
+            "crawler_params": {"headless": True}
+        }
+        low_task_id = await api.submit_crawl(low_priority)
+
+        # Submit high priority task
+        high_priority = {
+            "urls": "https://www.nbcnews.com/business/consumer",
+            "priority": 10,
+            "crawler_params": {"headless": True}
+        }
+        high_task_id = await api.submit_crawl(high_priority)
+
+        # Get both results
+        high_result = await api.wait_for_task(high_task_id)
+        low_result = await api.wait_for_task(low_task_id)
+
+        print("Both tasks completed")
+        assert high_result["status"] == "completed"
+        assert low_result["status"] == "completed"
+
+async def main():
+    try:
+        # Start with health check
+        async with NBCNewsAPITest() as api:
+            health = await api.check_health()
+            print("Server health:", health)
+
+        # Run all tests
+        # await test_basic_crawl()
+        # await test_js_execution()
+        # await test_css_selector()
+        # await test_structured_extraction()
+        await test_llm_extraction()
+        # await test_batch_crawl()
+        # await test_screenshot()
+        # await test_priority_handling()
+
+    except Exception as e:
+        print(f"Test failed: {str(e)}")
+        raise
+
+if __name__ == "__main__":
+    asyncio.run(main())