Merge branch 'main' of https://github.com/unclecode/crawl4ai

2024-11-05 12:58:30 +00:00
parent e8aaa57cb2 1e7db0d293
commit 1222e456fb
32 changed files with 2280 additions and 62170 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -207,4 +207,5 @@ git_issues.md

 .tests/
 .issues/
-.docs/
+.docs/
+.issues/
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,8 +1,104 @@
 # Changelog

+# CHANGELOG
+
+## [v0.3.73] - 2024-11-05
+
+### Major Features
+- **New Doctor Feature**
+  - Added comprehensive system diagnostics tool
+  - Available through package hub and CLI
+  - Provides automated troubleshooting and system health checks
+  - Includes detailed reporting of configuration issues
+
+- **Dockerized API Server**
+  - Released complete Docker implementation for API server
+  - Added comprehensive documentation for Docker deployment
+  - Implemented container communication protocols
+  - Added environment configuration guides
+
+- **Managed Browser Integration**
+  - Added support for user-controlled browser instances
+  - Implemented `ManagedBrowser` class for better browser lifecycle management
+  - Added ability to connect to existing Chrome DevTools Protocol (CDP) endpoints
+  - Introduced user data directory support for persistent browser profiles
+
+- **Enhanced HTML Processing**
+  - Added HTML tag preservation feature during markdown conversion
+  - Introduced configurable tag preservation system
+  - Improved pre-tag and code block handling
+  - Added support for nested preserved tags with attribute retention
+
+### Improvements
+- **Browser Handling**
+  - Added flag to ignore body visibility for problematic pages
+  - Improved browser process cleanup and management
+  - Enhanced temporary directory handling for browser profiles
+  - Added configurable browser launch arguments
+
+- **Database Management**
+  - Implemented connection pooling for better performance
+  - Added retry logic for database operations
+  - Improved error handling and logging
+  - Enhanced cleanup procedures for database connections
+
+- **Resource Management**
+  - Added memory and CPU monitoring
+  - Implemented dynamic task slot allocation based on system resources
+  - Added configurable cleanup intervals
+
+### Technical Improvements
+- **Code Structure**
+  - Moved version management to dedicated _version.py file
+  - Improved error handling throughout the codebase
+  - Enhanced logging system with better error reporting
+  - Reorganized core components for better maintainability
+
+### Bug Fixes
+- Fixed issues with browser process termination
+- Improved handling of connection timeouts
+- Enhanced error recovery in database operations
+- Fixed memory leaks in long-running processes
+
+### Dependencies
+- Updated Playwright to v1.47
+- Updated core dependencies with more flexible version constraints
+- Added new development dependencies for testing
+
+### Breaking Changes
+- Changed default browser handling behavior
+- Modified database connection management approach
+- Updated API response structure for better consistency
+
+## Migration Guide
+When upgrading to v0.3.73, be aware of the following changes:
+
+1. Docker Deployment:
+   - Review Docker documentation for new deployment options
+   - Update environment configurations as needed
+   - Check container communication settings
+
+2. If using custom browser management:
+   - Update browser initialization code to use new ManagedBrowser class
+   - Review browser cleanup procedures
+
+3. For database operations:
+   - Check custom database queries for compatibility with new connection pooling
+   - Update error handling to work with new retry logic
+
+4. Using the Doctor:
+   - Run doctor command for system diagnostics: `crawl4ai doctor`
+   - Review generated reports for potential issues
+   - Follow recommended fixes for any identified problems
+
+
+## [2024-11-04 - 13:21:42] Comprehensive Update of Crawl4AI Features and Dependencies
+This commit introduces several key enhancements, including improved error handling and robust database operations in `async_database.py`, which now features a connection pool and retry logic for better reliability. Updates to the README.md provide clearer instructions and a better user experience with links to documentation sections. The `.gitignore` file has been refined to include additional directories, while the async web crawler now utilizes a managed browser for more efficient crawling. Furthermore, multiple dependency updates and introduction of the `CustomHTML2Text` class enhance text extraction capabilities.
+
 ## [v0.3.73] - 2024-10-24

 ### Added
+- preserve_tags: Added support for preserving specific HTML tags during markdown conversion.
 - Smart overlay removal system in AsyncPlaywrightCrawlerStrategy:
  - Automatic removal of popups, modals, and cookie notices
  - Detection and removal of fixed/sticky position elements
--- a/121
+++ b/121
@@ -0,0 +1,121 @@
+# syntax=docker/dockerfile:1.4
+
+# Build arguments
+ARG PYTHON_VERSION=3.10
+
+# Base stage with system dependencies
+FROM python:${PYTHON_VERSION}-slim as base
+
+# Declare ARG variables again within the build stage
+ARG INSTALL_TYPE=all
+ARG ENABLE_GPU=false
+
+# Platform-specific labels
+LABEL maintainer="unclecode"
+LABEL description="Crawl4AI - Advanced Web Crawler with AI capabilities"
+LABEL version="1.0"
+
+# Environment setup
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1 \
+    PIP_DEFAULT_TIMEOUT=100 \
+    DEBIAN_FRONTEND=noninteractive
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    curl \
+    wget \
+    gnupg \
+    git \
+    cmake \
+    pkg-config \
+    python3-dev \
+    libjpeg-dev \
+    libpng-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Playwright system dependencies for Linux
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libglib2.0-0 \
+    libnss3 \
+    libnspr4 \
+    libatk1.0-0 \
+    libatk-bridge2.0-0 \
+    libcups2 \
+    libdrm2 \
+    libdbus-1-3 \
+    libxcb1 \
+    libxkbcommon0 \
+    libx11-6 \
+    libxcomposite1 \
+    libxdamage1 \
+    libxext6 \
+    libxfixes3 \
+    libxrandr2 \
+    libgbm1 \
+    libpango-1.0-0 \
+    libcairo2 \
+    libasound2 \
+    libatspi2.0-0 \
+    && rm -rf /var/lib/apt/lists/*
+
+# GPU support if enabled
+RUN if [ "$ENABLE_GPU" = "true" ] ; then \
+    apt-get update && apt-get install -y --no-install-recommends \
+    nvidia-cuda-toolkit \
+    && rm -rf /var/lib/apt/lists/* ; \
+    fi
+
+# Create and set working directory
+WORKDIR /app
+
+# Copy the entire project
+COPY . .
+
+# Install base requirements
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Install required library for FastAPI
+RUN pip install fastapi uvicorn psutil
+
+# Install ML dependencies first for better layer caching
+RUN if [ "$INSTALL_TYPE" = "all" ] ; then \
+        pip install --no-cache-dir \
+            torch \
+            torchvision \
+            torchaudio \
+            scikit-learn \
+            nltk \
+            transformers \
+            tokenizers && \
+        python -m nltk.downloader punkt stopwords ; \
+    fi
+
+# Install the package
+RUN if [ "$INSTALL_TYPE" = "all" ] ; then \
+        pip install -e ".[all]" && \
+        python -m crawl4ai.model_loader ; \
+    elif [ "$INSTALL_TYPE" = "torch" ] ; then \
+        pip install -e ".[torch]" ; \
+    elif [ "$INSTALL_TYPE" = "transformer" ] ; then \
+        pip install -e ".[transformer]" && \
+        python -m crawl4ai.model_loader ; \
+    else \
+        pip install -e "." ; \
+    fi
+
+# Install Playwright and browsers
+RUN playwright install
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+# Expose port
+EXPOSE 8000
+
+# Start the FastAPI server
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "11235"]
--- a/MISSION.md
+++ b/MISSION.md
@@ -0,0 +1,46 @@
+# Mission
+
+![Mission Diagram](./docs/assets/pitch-dark.svg)
+
+### 1. The Data Capitalization Opportunity
+
+We live in an unprecedented era of digital wealth creation. Every day, individuals and enterprises generate massive amounts of valuable digital footprints across various platforms, social media channels, messenger apps, and cloud services. While people can interact with their data within these platforms, there's an immense untapped opportunity to transform this data into true capital assets. Just as physical property became a foundational element of wealth creation, personal and enterprise data has the potential to become a new form of capital on balance sheets.
+
+For individuals, this represents an opportunity to transform their digital activities into valuable assets. For enterprises, their internal communications, team discussions, and collaborative documents contain rich insights that could be structured and valued as intellectual capital. This wealth of information represents an unprecedented opportunity for value creation in the digital age.
+
+### 2. The Potential of Authentic Data
+
+While synthetic data has played a crucial role in AI development, there's an enormous untapped potential in the authentic data generated by individuals and organizations. Every message, document, and interaction contains unique insights and patterns that could enhance AI development. The challenge isn't a lack of data - it's that most authentic human-generated data remains inaccessible for productive use.
+
+By enabling willing participation in data sharing, we can unlock this vast reservoir of authentic human knowledge. This represents an opportunity to enhance AI development with diverse, real-world data that reflects the full spectrum of human experience and knowledge.
+
+## Our Pathway to Data Democracy
+
+### 1. Open-Source Foundation
+
+Our first step is creating an open-source data extraction engine that empowers developers and innovators to build tools for data structuring and organization. This foundation ensures transparency, security, and community-driven development. By making these tools openly available, we enable the technical infrastructure needed for true data ownership and capitalization.
+
+### 2. Data Capitalization Platform
+
+Building on this open-source foundation, we're developing a platform that helps individuals and enterprises transform their digital footprints into structured, valuable assets. This platform will provide the tools and frameworks needed to organize, understand, and value personal and organizational data as true capital assets.
+
+### 3. Creating a Data Marketplace
+
+The final piece is establishing a marketplace where individuals and organizations can willingly share their data assets. This creates opportunities for:
+- Individuals to earn equity, revenue, or other forms of value from their data
+- Enterprises to access diverse, high-quality data for AI development
+- Researchers to work with authentic human-generated data
+- Startups to build innovative solutions using real-world data
+
+## Economic Vision: A Shared Data Economy
+
+We envision a future where data becomes a fundamental asset class in a thriving shared economy. This transformation will democratize AI development by enabling willing participation in data sharing, ensuring that the benefits of AI advancement flow back to data creators. Just as property rights revolutionized economic systems, establishing data as a capital asset will create new opportunities for wealth creation and economic participation.
+
+This shared data economy will:
+- Enable individuals to capitalize on their digital footprints
+- Create new revenue streams for data creators
+- Provide AI developers with access to diverse, authentic data
+- Foster innovation through broader access to real-world data
+- Ensure more equitable distribution of AI's economic benefits
+
+Our vision is to facilitate this transformation from the ground up - starting with open-source tools, progressing to data capitalization platforms, and ultimately creating a thriving marketplace where data becomes a true asset class in a shared economy. This approach ensures that the future of AI is built on a foundation of authentic human knowledge, with benefits flowing back to the individuals and organizations who create and share their valuable data.
--- a/README.md
+++ b/README.md
@@ -12,19 +12,20 @@
 Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐

 ## 🌟 Meet the Crawl4AI Assistant: Your Copilot for Crawling
+
 Use the [Crawl4AI GPT Assistant](https://tinyurl.com/crawl4ai-gpt) as your AI-powered copilot! With this assistant, you can:
+
 - 🧑‍💻 Generate code for complex crawling and extraction tasks
 - 💡 Get tailored support and examples
 - 📘 Learn Crawl4AI faster with step-by-step guidance

-## New in 0.3.72 ✨
+## New in 0.3.73 ✨

- 📄 Fit markdown generation for extracting main article content.
- 🪄 Magic mode for comprehensive anti-bot detection bypass.
- 🌐 Enhanced multi-browser support with seamless switching (Chromium, Firefox, WebKit)
- 📚 New chunking strategies(Sliding window, Overlapping window, Flexible size control)
- 💾 Improved caching system for better performance
- ⚡ Optimized batch processing with automatic rate limiting
+- 🐳 Docker Ready: Full API server with seamless deployment & scaling
+- 🎯 Smart Browser: Managed browser integration with CDP support
+- 📝 Mockdown+: Enhanced tag preservation & content extraction
+- ⚡️ Parallel Power: Supercharged multi-URL crawling performance
+- 🌟 And many more exciting updates...

 ## Try it Now!

@@ -81,11 +82,13 @@ By default, this will install the asynchronous version of Crawl4AI, using Playwr
 👉 Note: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

 1. Through the command line:
+
   ```bash
   playwright install
   ```

 2. If the above doesn't work, try this more specific command:
+
   ```bash
   python -m playwright install chromium
   ```
@@ -112,9 +115,53 @@ pip install -e .

 ### Using Docker 🐳

-We're in the process of creating Docker images and pushing them to Docker Hub. This will provide an easy way to run Crawl4AI in a containerized environment. Stay tuned for updates!
+Crawl4AI is available as Docker images for easy deployment. You can either pull directly from Docker Hub (recommended) or build from the repository.
+
+#### Option 1: Docker Hub (Recommended)
+
+```bash
+# Pull and run from Docker Hub (choose one):
+docker pull unclecode/crawl4ai:basic    # Basic crawling features
+docker pull unclecode/crawl4ai:all      # Full installation (ML, LLM support)
+docker pull unclecode/crawl4ai:gpu      # GPU-enabled version
+
+# Run the container
+docker run -p 11235:11235 unclecode/crawl4ai:basic  # Replace 'basic' with your chosen version
+```
+
+#### Option 2: Build from Repository
+
+```bash
+# Clone the repository
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+
+# Build the image
+docker build -t crawl4ai:local \
+  --build-arg INSTALL_TYPE=basic \  # Options: basic, all
+  .
+
+# Run your local build
+docker run -p 11235:11235 crawl4ai:local
+```
+
+Quick test (works for both options):
+```python
+import requests
+
+# Submit a crawl job
+response = requests.post(
+    "http://localhost:11235/crawl",
+    json={"urls": "https://example.com", "priority": 10}
+)
+task_id = response.json()["task_id"]
+
+# Get results
+result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+For advanced configuration, environment variables, and usage examples, see our [Docker Deployment Guide](https://crawl4ai.com/mkdocs/basic/docker-deployment/).

-For more detailed installation instructions and options, please refer to our [Installation Guide](https://crawl4ai.com/mkdocs/installation).

 ## Quick Start 🚀

@@ -244,7 +291,7 @@ if __name__ == "__main__":
    asyncio.run(extract_news_teasers())
 ```

-For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/advanced_jsoncss_extraction.md) section in the documentation.
+For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/extraction/css-advanced/) section in the documentation.

 ### Extracting Structured Data with OpenAI

@@ -347,7 +394,8 @@ if __name__ == "__main__":

 This example demonstrates Crawl4AI's ability to handle complex scenarios where content is loaded asynchronously. It crawls multiple pages of GitHub commits, executing JavaScript to load new content and using custom hooks to ensure data is loaded before proceeding.

-For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/session_based_crawling.md) section in the documentation.
+For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/tutorial/episode_12_Session-Based_Crawling_for_Dynamic_Websites/) section in the documentation.
+</details>


 ## Speed Comparison 🚀
@@ -356,7 +404,7 @@ Crawl4AI is designed with speed as a primary focus. Our goal is to provide the f

 We've conducted a speed comparison between Crawl4AI and Firecrawl, a paid service. The results demonstrate Crawl4AI's superior performance:

-```
+```bash
 Firecrawl:
 Time taken: 7.02 seconds
 Content length: 42074 characters
@@ -374,6 +422,7 @@ Images found: 89
 ```

 As you can see, Crawl4AI outperforms Firecrawl significantly:
+
 - Simple crawl: Crawl4AI is over 4 times faster than Firecrawl.
 - With JavaScript execution: Even when executing JavaScript to load more content (doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.

@@ -401,6 +450,34 @@ For questions, suggestions, or feedback, feel free to reach out:

 Happy Crawling! 🕸️🚀

+
+# Mission
+
+Our mission is to unlock the untapped potential of personal and enterprise data in the digital age. In today's world, individuals and organizations generate vast amounts of valuable digital footprints, yet this data remains largely uncapitalized as a true asset. 
+
+Our open-source solution empowers developers and innovators to build tools for data extraction and structuring, laying the foundation for a new era of data ownership. By transforming personal and enterprise data into structured, tradeable assets, we're creating opportunities for individuals to capitalize on their digital footprints and for organizations to unlock the value of their collective knowledge.
+
+This democratization of data represents the first step toward a shared data economy, where willing participation in data sharing drives AI advancement while ensuring the benefits flow back to data creators. Through this approach, we're building a future where AI development is powered by authentic human knowledge rather than synthetic alternatives.
+
+![Mission Diagram](./docs/assets/pitch-dark.svg)
+
+For a detailed exploration of our vision, opportunities, and pathway forward, please see our [full mission statement](./MISSION.md).
+
+## Key Opportunities
+
+- **Data Capitalization**: Transform digital footprints into valuable assets that can appear on personal and enterprise balance sheets
+- **Authentic Data**: Unlock the vast reservoir of real human insights and knowledge for AI advancement
+- **Shared Economy**: Create new value streams where data creators directly benefit from their contributions
+
+## Development Pathway
+
+1. **Open-Source Foundation**: Building transparent, community-driven data extraction tools
+2. **Data Capitalization Platform**: Creating tools to structure and value digital assets
+3. **Shared Data Marketplace**: Establishing an economic platform for ethical data exchange
+
+For a detailed exploration of our vision, challenges, and solutions, please see our [full mission statement](./MISSION.md).
+
+
 ## Star History

 [![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)
--- a/crawl4ai/init.py
+++ b/crawl4ai/init.py
@@ -2,8 +2,8 @@

 from .async_webcrawler import AsyncWebCrawler
 from .models import CrawlResult
-
-__version__ = "0.3.72"
+from ._version import __version__
+# __version__ = "0.3.73"

 __all__ = [
    "AsyncWebCrawler",
--- a/crawl4ai/_version.py
+++ b/crawl4ai/_version.py
@@ -0,0 +1,2 @@
+# crawl4ai/_version.py
+__version__ = "0.3.73"
--- a/crawl4ai/async_crawler_strategy
+++ b/crawl4ai/async_crawler_strategy
@@ -1,558 +0,0 @@
-import asyncio
-import base64
-import time
-from abc import ABC, abstractmethod
-from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
-from playwright.async_api import async_playwright, Page, Browser, Error
-from io import BytesIO
-from PIL import Image, ImageDraw, ImageFont
-from pathlib import Path
-from playwright.async_api import ProxySettings
-from pydantic import BaseModel
-import hashlib
-import json
-import uuid
-from playwright_stealth import stealth_async
-
-class AsyncCrawlResponse(BaseModel):
-    html: str
-    response_headers: Dict[str, str]
-    status_code: int
-    screenshot: Optional[str] = None
-    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
-
-    class Config:
-        arbitrary_types_allowed = True
-
-class AsyncCrawlerStrategy(ABC):
-    @abstractmethod
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        pass
-    
-    @abstractmethod
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        pass
-    
-    @abstractmethod
-    async def take_screenshot(self, url: str) -> str:
-        pass
-    
-    @abstractmethod
-    def update_user_agent(self, user_agent: str):
-        pass
-    
-    @abstractmethod
-    def set_hook(self, hook_type: str, hook: Callable):
-        pass
-
-class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
-    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
-        self.use_cached_html = use_cached_html
-        self.user_agent = kwargs.get(
-            "user_agent",
-            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
-            "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
-        )
-        self.proxy = kwargs.get("proxy")
-        self.headless = kwargs.get("headless", True)
-        self.browser_type = kwargs.get("browser_type", "chromium")
-        self.headers = kwargs.get("headers", {})
-        self.sessions = {}
-        self.session_ttl = 1800 
-        self.js_code = js_code
-        self.verbose = kwargs.get("verbose", False)
-        self.playwright = None
-        self.browser = None
-        self.hooks = {
-            'on_browser_created': None,
-            'on_user_agent_updated': None,
-            'on_execution_started': None,
-            'before_goto': None,
-            'after_goto': None,
-            'before_return_html': None,
-            'before_retrieve_html': None
-        }
-
-    async def __aenter__(self):
-        await self.start()
-        return self
-
-    async def __aexit__(self, exc_type, exc_val, exc_tb):
-        await self.close()
-
-    async def start(self):
-        if self.playwright is None:
-            self.playwright = await async_playwright().start()
-        if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-                
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
-            else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
-
-            await self.execute_hook('on_browser_created', self.browser)
-
-    async def close(self):
-        if self.browser:
-            await self.browser.close()
-            self.browser = None
-        if self.playwright:
-            await self.playwright.stop()
-            self.playwright = None
-
-    def __del__(self):
-        if self.browser or self.playwright:
-            asyncio.get_event_loop().run_until_complete(self.close())
-
-    def set_hook(self, hook_type: str, hook: Callable):
-        if hook_type in self.hooks:
-            self.hooks[hook_type] = hook
-        else:
-            raise ValueError(f"Invalid hook type: {hook_type}")
-
-    async def execute_hook(self, hook_type: str, *args):
-        hook = self.hooks.get(hook_type)
-        if hook:
-            if asyncio.iscoroutinefunction(hook):
-                return await hook(*args)
-            else:
-                return hook(*args)
-        return args[0] if args else None
-
-    def update_user_agent(self, user_agent: str):
-        self.user_agent = user_agent
-
-    def set_custom_headers(self, headers: Dict[str, str]):
-        self.headers = headers
-
-    async def kill_session(self, session_id: str):
-        if session_id in self.sessions:
-            context, page, _ = self.sessions[session_id]
-            await page.close()
-            await context.close()
-            del self.sessions[session_id]
-
-    def _cleanup_expired_sessions(self):
-        current_time = time.time()
-        expired_sessions = [
-            sid for sid, (_, _, last_used) in self.sessions.items() 
-            if current_time - last_used > self.session_ttl
-        ]
-        for sid in expired_sessions:
-            asyncio.create_task(self.kill_session(sid))
-            
-    async def smart_wait(self, page: Page, wait_for: str, timeout: float = 30000):
-        wait_for = wait_for.strip()
-        
-        if wait_for.startswith('js:'):
-            # Explicitly specified JavaScript
-            js_code = wait_for[3:].strip()
-            return await self.csp_compliant_wait(page, js_code, timeout)
-        elif wait_for.startswith('css:'):
-            # Explicitly specified CSS selector
-            css_selector = wait_for[4:].strip()
-            try:
-                await page.wait_for_selector(css_selector, timeout=timeout)
-            except Error as e:
-                if 'Timeout' in str(e):
-                    raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{css_selector}'")
-                else:
-                    raise ValueError(f"Invalid CSS selector: '{css_selector}'")
-        else:
-            # Auto-detect based on content
-            if wait_for.startswith('()') or wait_for.startswith('function'):
-                # It's likely a JavaScript function
-                return await self.csp_compliant_wait(page, wait_for, timeout)
-            else:
-                # Assume it's a CSS selector first
-                try:
-                    await page.wait_for_selector(wait_for, timeout=timeout)
-                except Error as e:
-                    if 'Timeout' in str(e):
-                        raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{wait_for}'")
-                    else:
-                        # If it's not a timeout error, it might be an invalid selector
-                        # Let's try to evaluate it as a JavaScript function as a fallback
-                        try:
-                            return await self.csp_compliant_wait(page, f"() => {{{wait_for}}}", timeout)
-                        except Error:
-                            raise ValueError(f"Invalid wait_for parameter: '{wait_for}'. "
-                                             "It should be either a valid CSS selector, a JavaScript function, "
-                                             "or explicitly prefixed with 'js:' or 'css:'.")
-    
-    async def csp_compliant_wait(self, page: Page, user_wait_function: str, timeout: float = 30000):
-        wrapper_js = f"""
-        async () => {{
-            const userFunction = {user_wait_function};
-            const startTime = Date.now();
-            while (true) {{
-                if (await userFunction()) {{
-                    return true;
-                }}
-                if (Date.now() - startTime > {timeout}) {{
-                    throw new Error('Timeout waiting for condition');
-                }}
-                await new Promise(resolve => setTimeout(resolve, 100));
-            }}
-        }}
-        """
-        
-        try:
-            await page.evaluate(wrapper_js)
-        except TimeoutError:
-            raise TimeoutError(f"Timeout after {timeout}ms waiting for condition")
-        except Exception as e:
-            raise RuntimeError(f"Error in wait condition: {str(e)}")
-
-    async def process_iframes(self, page):
-        # Find all iframes
-        iframes = await page.query_selector_all('iframe')
-        
-        for i, iframe in enumerate(iframes):
-            try:
-                # Add a unique identifier to the iframe
-                await iframe.evaluate(f'(element) => element.id = "iframe-{i}"')
-                
-                # Get the frame associated with this iframe
-                frame = await iframe.content_frame()
-                
-                if frame:
-                    # Wait for the frame to load
-                    await frame.wait_for_load_state('load', timeout=30000)  # 30 seconds timeout
-                    
-                    # Extract the content of the iframe's body
-                    iframe_content = await frame.evaluate('() => document.body.innerHTML')
-                    
-                    # Generate a unique class name for this iframe
-                    class_name = f'extracted-iframe-content-{i}'
-                    
-                    # Replace the iframe with a div containing the extracted content
-                    _iframe = iframe_content.replace('`', '\\`')
-                    await page.evaluate(f"""
-                        () => {{
-                            const iframe = document.getElementById('iframe-{i}');
-                            const div = document.createElement('div');
-                            div.innerHTML = `{_iframe}`;
-                            div.className = '{class_name}';
-                            iframe.replaceWith(div);
-                        }}
-                    """)
-                else:
-                    print(f"Warning: Could not access content frame for iframe {i}")
-            except Exception as e:
-                print(f"Error processing iframe {i}: {str(e)}")
-
-        # Return the page object
-        return page  
-    
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        response_headers = {}
-        status_code = None
-        
-        self._cleanup_expired_sessions()
-        session_id = kwargs.get("session_id")
-        if session_id:
-            context, page, _ = self.sessions.get(session_id, (None, None, None))
-            if not context:
-                context = await self.browser.new_context(
-                    user_agent=self.user_agent,
-                    viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None
-                )
-                await context.set_extra_http_headers(self.headers)
-                page = await context.new_page()
-                self.sessions[session_id] = (context, page, time.time())
-        else:
-            context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
-            )
-            await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
-            page = await context.new_page()
-
-        try:
-            if self.verbose:
-                print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                if os.path.exists(cache_file_path):
-                    html = ""
-                    with open(cache_file_path, "r") as f:
-                        html = f.read()
-                    # retrieve response headers and status code from cache
-                    with open(cache_file_path + ".meta", "r") as f:
-                        meta = json.load(f)
-                        response_headers = meta.get("response_headers", {})
-                        status_code = meta.get("status_code")
-                    response = AsyncCrawlResponse(
-                        html=html, response_headers=response_headers, status_code=status_code
-                    )
-                    return response
-
-            if not kwargs.get("js_only", False):
-                await self.execute_hook('before_goto', page)
-                
-                response = await page.goto("about:blank")
-                await stealth_async(page)
-                response = await page.goto(
-                    url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000)
-                )
-                
-                # await stealth_async(page)
-                # response = await page.goto("about:blank")
-                # await stealth_async(page)
-                # await page.evaluate(f"window.location.href = '{url}'")
-                
-                await self.execute_hook('after_goto', page)
-                
-                # Get status code and headers
-                status_code = response.status
-                response_headers = response.headers
-            else:
-                status_code = 200
-                response_headers = {}
-
-            await page.wait_for_selector('body')
-            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-
-            js_code = kwargs.get("js_code", kwargs.get("js", self.js_code))
-            if js_code:
-                if isinstance(js_code, str):
-                    await page.evaluate(js_code)
-                elif isinstance(js_code, list):
-                    for js in js_code:
-                        await page.evaluate(js)
-                
-                await page.wait_for_load_state('networkidle')
-                # Check for on execution event
-                await self.execute_hook('on_execution_started', page)
-                
-            if kwargs.get("simulate_user", False):
-                # Simulate user interactions
-                await page.mouse.move(100, 100)
-                await page.mouse.down()
-                await page.mouse.up()
-                await page.keyboard.press('ArrowDown')
-
-            # Handle the wait_for parameter
-            wait_for = kwargs.get("wait_for")
-            if wait_for:
-                try:
-                    await self.smart_wait(page, wait_for, timeout=kwargs.get("page_timeout", 60000))
-                except Exception as e:
-                    raise RuntimeError(f"Wait condition failed: {str(e)}")
-
-
-            
-            # Update image dimensions
-            update_image_dimensions_js = """
-            () => {
-                return new Promise((resolve) => {
-                    const filterImage = (img) => {
-                        // Filter out images that are too small
-                        if (img.width < 100 && img.height < 100) return false;
-                        
-                        // Filter out images that are not visible
-                        const rect = img.getBoundingClientRect();
-                        if (rect.width === 0 || rect.height === 0) return false;
-                        
-                        // Filter out images with certain class names (e.g., icons, thumbnails)
-                        if (img.classList.contains('icon') || img.classList.contains('thumbnail')) return false;
-                        
-                        // Filter out images with certain patterns in their src (e.g., placeholder images)
-                        if (img.src.includes('placeholder') || img.src.includes('icon')) return false;
-                        
-                        return true;
-                    };
-
-                    const images = Array.from(document.querySelectorAll('img')).filter(filterImage);
-                    let imagesLeft = images.length;
-                    
-                    if (imagesLeft === 0) {
-                        resolve();
-                        return;
-                    }
-
-                    const checkImage = (img) => {
-                        if (img.complete && img.naturalWidth !== 0) {
-                            img.setAttribute('width', img.naturalWidth);
-                            img.setAttribute('height', img.naturalHeight);
-                            imagesLeft--;
-                            if (imagesLeft === 0) resolve();
-                        }
-                    };
-
-                    images.forEach(img => {
-                        checkImage(img);
-                        if (!img.complete) {
-                            img.onload = () => {
-                                checkImage(img);
-                            };
-                            img.onerror = () => {
-                                imagesLeft--;
-                                if (imagesLeft === 0) resolve();
-                            };
-                        }
-                    });
-
-                    // Fallback timeout of 5 seconds
-                    setTimeout(() => resolve(), 5000);
-                });
-            }
-            """
-            await page.evaluate(update_image_dimensions_js)
-
-            # Wait a bit for any onload events to complete
-            await page.wait_for_timeout(100)
-
-            # Process iframes
-            if kwargs.get("process_iframes", False):
-                page = await self.process_iframes(page)
-            
-            await self.execute_hook('before_retrieve_html', page)
-            # Check if delay_before_return_html is set then wait for that time
-            delay_before_return_html = kwargs.get("delay_before_return_html")
-            if delay_before_return_html:
-                await asyncio.sleep(delay_before_return_html)
-                
-            html = await page.content()
-            await self.execute_hook('before_return_html', page, html)
-            
-            # Check if kwargs has screenshot=True then take screenshot
-            screenshot_data = None
-            if kwargs.get("screenshot"):
-                screenshot_data = await self.take_screenshot(url)            
-
-            if self.verbose:
-                print(f"[LOG] ✅ Crawled {url} successfully!")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                with open(cache_file_path, "w", encoding="utf-8") as f:
-                    f.write(html)
-                # store response headers and status code in cache
-                with open(cache_file_path + ".meta", "w", encoding="utf-8") as f:
-                    json.dump({
-                        "response_headers": response_headers,
-                        "status_code": status_code
-                    }, f)
-
-            async def get_delayed_content(delay: float = 5.0) -> str:
-                if self.verbose:
-                    print(f"[LOG] Waiting for {delay} seconds before retrieving content for {url}")
-                await asyncio.sleep(delay)
-                return await page.content()
-                
-            response = AsyncCrawlResponse(
-                html=html, 
-                response_headers=response_headers, 
-                status_code=status_code,
-                screenshot=screenshot_data,
-                get_delayed_content=get_delayed_content
-            )
-            return response
-        except Error as e:
-            raise Error(f"Failed to crawl {url}: {str(e)}")
-        finally:
-            if not session_id:
-                await page.close()
-                await context.close()
-
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        semaphore_count = kwargs.get('semaphore_count', 5)  # Adjust as needed
-        semaphore = asyncio.Semaphore(semaphore_count)
-
-        async def crawl_with_semaphore(url):
-            async with semaphore:
-                return await self.crawl(url, **kwargs)
-
-        tasks = [crawl_with_semaphore(url) for url in urls]
-        results = await asyncio.gather(*tasks, return_exceptions=True)
-        return [result if not isinstance(result, Exception) else str(result) for result in results]
-
-    async def take_screenshot(self, url: str, wait_time=1000) -> str:
-        async with await self.browser.new_context(user_agent=self.user_agent) as context:
-            page = await context.new_page()
-            try:
-                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
-                # Wait for a specified time (default is 1 second)
-                await page.wait_for_timeout(wait_time)
-                screenshot = await page.screenshot(full_page=True)
-                return base64.b64encode(screenshot).decode('utf-8')
-            except Exception as e:
-                error_message = f"Failed to take screenshot: {str(e)}"
-                print(error_message)
-
-                # Generate an error image
-                img = Image.new('RGB', (800, 600), color='black')
-                draw = ImageDraw.Draw(img)
-                font = ImageFont.load_default()
-                draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
-                
-                buffered = BytesIO()
-                img.save(buffered, format="JPEG")
-                return base64.b64encode(buffered.getvalue()).decode('utf-8')
-            finally:
-                await page.close()
-
--- a/crawl4ai/async_crawler_strategy.py
+++ b/crawl4ai/async_crawler_strategy.py
@@ -3,7 +3,8 @@ import base64
 import time
 from abc import ABC, abstractmethod
 from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
+import os, sys, shutil
+import tempfile, subprocess
 from playwright.async_api import async_playwright, Page, Browser, Error
 from io import BytesIO
 from PIL import Image, ImageDraw, ImageFont
@@ -13,6 +14,7 @@ from pydantic import BaseModel
 import hashlib
 import json
 import uuid
+
 from playwright_stealth import StealthConfig, stealth_async

 stealth_config = StealthConfig(
@@ -31,6 +33,106 @@ stealth_config = StealthConfig(
 )


+class ManagedBrowser:
+    def __init__(self, browser_type: str = "chromium", user_data_dir: Optional[str] = None, headless: bool = False):
+        self.browser_type = browser_type
+        self.user_data_dir = user_data_dir
+        self.headless = headless
+        self.browser_process = None
+        self.temp_dir = None
+        self.debugging_port = 9222
+
+    async def start(self) -> str:
+        """
+        Starts the browser process and returns the CDP endpoint URL.
+        If user_data_dir is not provided, creates a temporary directory.
+        """
+        
+        # Create temp dir if needed
+        if not self.user_data_dir:
+            self.temp_dir = tempfile.mkdtemp(prefix="browser-profile-")
+            self.user_data_dir = self.temp_dir
+
+        # Get browser path and args based on OS and browser type
+        browser_path = self._get_browser_path()
+        args = self._get_browser_args()
+
+        # Start browser process
+        try:
+            self.browser_process = subprocess.Popen(
+                args,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE
+            )
+            await asyncio.sleep(2)  # Give browser time to start
+            return f"http://localhost:{self.debugging_port}"
+        except Exception as e:
+            await self.cleanup()
+            raise Exception(f"Failed to start browser: {e}")
+
+    def _get_browser_path(self) -> str:
+        """Returns the browser executable path based on OS and browser type"""
+        if sys.platform == "darwin":  # macOS
+            paths = {
+                "chromium": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
+                "firefox": "/Applications/Firefox.app/Contents/MacOS/firefox",
+                "webkit": "/Applications/Safari.app/Contents/MacOS/Safari"
+            }
+        elif sys.platform == "win32":  # Windows
+            paths = {
+                "chromium": "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
+                "firefox": "C:\\Program Files\\Mozilla Firefox\\firefox.exe",
+                "webkit": None  # WebKit not supported on Windows
+            }
+        else:  # Linux
+            paths = {
+                "chromium": "google-chrome",
+                "firefox": "firefox",
+                "webkit": None  # WebKit not supported on Linux
+            }
+        
+        return paths.get(self.browser_type)
+
+    def _get_browser_args(self) -> List[str]:
+        """Returns browser-specific command line arguments"""
+        base_args = [self._get_browser_path()]
+        
+        if self.browser_type == "chromium":
+            args = [
+                f"--remote-debugging-port={self.debugging_port}",
+                f"--user-data-dir={self.user_data_dir}",
+            ]
+            if self.headless:
+                args.append("--headless=new")
+        elif self.browser_type == "firefox":
+            args = [
+                "--remote-debugging-port", str(self.debugging_port),
+                "--profile", self.user_data_dir,
+            ]
+            if self.headless:
+                args.append("--headless")
+        else:
+            raise NotImplementedError(f"Browser type {self.browser_type} not supported")
+            
+        return base_args + args
+
+    async def cleanup(self):
+        """Cleanup browser process and temporary directory"""
+        if self.browser_process:
+            try:
+                self.browser_process.terminate()
+                await asyncio.sleep(1)
+                if self.browser_process.poll() is None:
+                    self.browser_process.kill()
+            except Exception as e:
+                print(f"Error terminating browser: {e}")
+
+        if self.temp_dir and os.path.exists(self.temp_dir):
+            try:
+                shutil.rmtree(self.temp_dir)
+            except Exception as e:
+                print(f"Error removing temporary directory: {e}")
+
 class AsyncCrawlResponse(BaseModel):
    html: str
    response_headers: Dict[str, str]
@@ -82,6 +184,9 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
        self.playwright = None
        self.browser = None
        self.sleep_on_close = kwargs.get("sleep_on_close", False)
+        self.use_managed_browser = kwargs.get("use_managed_browser", False)
+        self.user_data_dir = kwargs.get("user_data_dir", None)
+        self.managed_browser = None
        self.hooks = {
            'on_browser_created': None,
            'on_user_agent_updated': None,
@@ -103,36 +208,46 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
        if self.playwright is None:
            self.playwright = await async_playwright().start()
        if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-            elif self.proxy_config:
-                proxy_settings = ProxySettings(server=self.proxy_config.get("server"), username=self.proxy_config.get("username"), password=self.proxy_config.get("password"))
-                browser_args["proxy"] = proxy_settings
-                
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
+            if self.use_managed_browser:
+                # Use managed browser approach
+                self.managed_browser = ManagedBrowser(
+                    browser_type=self.browser_type,
+                    user_data_dir=self.user_data_dir,
+                    headless=self.headless
+                )
+                cdp_url = await self.managed_browser.start()
+                self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)
            else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
+                browser_args = {
+                    "headless": self.headless,
+                    "args": [
+                        "--disable-gpu",
+                        "--no-sandbox",
+                        "--disable-dev-shm-usage",
+                        "--disable-blink-features=AutomationControlled",
+                        "--disable-infobars",
+                        "--window-position=0,0",
+                        "--ignore-certificate-errors",
+                        "--ignore-certificate-errors-spki-list",
+                        # "--headless=new",  # Use the new headless mode
+                    ]
+                }
+                
+                # Add proxy settings if a proxy is specified
+                if self.proxy:
+                    proxy_settings = ProxySettings(server=self.proxy)
+                    browser_args["proxy"] = proxy_settings
+                elif self.proxy_config:
+                    proxy_settings = ProxySettings(server=self.proxy_config.get("server"), username=self.proxy_config.get("username"), password=self.proxy_config.get("password"))
+                    browser_args["proxy"] = proxy_settings
+                    
+                # Select the appropriate browser based on the browser_type
+                if self.browser_type == "firefox":
+                    self.browser = await self.playwright.firefox.launch(**browser_args)
+                elif self.browser_type == "webkit":
+                    self.browser = await self.playwright.webkit.launch(**browser_args)
+                else:
+                    self.browser = await self.playwright.chromium.launch(**browser_args)

            await self.execute_hook('on_browser_created', self.browser)

@@ -142,6 +257,9 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
        if self.browser:
            await self.browser.close()
            self.browser = None
+        if self.managed_browser:
+            await self.managed_browser.cleanup()
+            self.managed_browser = None
        if self.playwright:
            await self.playwright.stop()
            self.playwright = None
@@ -399,7 +517,48 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                status_code = 200
                response_headers = {}

-            await page.wait_for_selector('body')
+            # Replace the current wait_for_selector line with this more robust check:
+            try:
+                # First wait for body to exist, regardless of visibility
+                await page.wait_for_selector('body', state='attached', timeout=30000)
+                
+                # Then wait for it to become visible by checking CSS
+                await page.wait_for_function("""
+                    () => {
+                        const body = document.body;
+                        const style = window.getComputedStyle(body);
+                        return style.display !== 'none' && 
+                            style.visibility !== 'hidden' && 
+                            style.opacity !== '0';
+                    }
+                """, timeout=30000)
+                
+            except Error as e:
+                # If waiting fails, let's try to diagnose the issue
+                visibility_info = await page.evaluate("""
+                    () => {
+                        const body = document.body;
+                        const style = window.getComputedStyle(body);
+                        return {
+                            display: style.display,
+                            visibility: style.visibility,
+                            opacity: style.opacity,
+                            hasContent: body.innerHTML.length,
+                            classList: Array.from(body.classList)
+                        }
+                    }
+                """)
+                
+                if self.verbose:
+                    print(f"Body visibility debug info: {visibility_info}")
+                
+                # Even if body is hidden, we might still want to proceed
+                if kwargs.get('ignore_body_visibility', True):
+                    if self.verbose:
+                        print("Proceeding despite hidden body...")
+                    pass
+                else:
+                    raise Error(f"Body element is hidden: {visibility_info}")
            
            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")

--- a/crawl4ai/async_database.py
+++ b/crawl4ai/async_database.py
@@ -2,18 +2,82 @@ import os
 from pathlib import Path
 import aiosqlite
 import asyncio
-from typing import Optional, Tuple
+from typing import Optional, Tuple, Dict
+from contextlib import asynccontextmanager
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)

 DB_PATH = os.path.join(Path.home(), ".crawl4ai")
 os.makedirs(DB_PATH, exist_ok=True)
 DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")

 class AsyncDatabaseManager:
-    def __init__(self):
+    def __init__(self, pool_size: int = 10, max_retries: int = 3):
        self.db_path = DB_PATH
+        self.pool_size = pool_size
+        self.max_retries = max_retries
+        self.connection_pool: Dict[int, aiosqlite.Connection] = {}
+        self.pool_lock = asyncio.Lock()
+        self.connection_semaphore = asyncio.Semaphore(pool_size)
+        
+    async def initialize(self):
+        """Initialize the database and connection pool"""
+        await self.ainit_db()
+        
+    async def cleanup(self):
+        """Cleanup connections when shutting down"""
+        async with self.pool_lock:
+            for conn in self.connection_pool.values():
+                await conn.close()
+            self.connection_pool.clear()
+
+    @asynccontextmanager
+    async def get_connection(self):
+        """Connection pool manager"""
+        async with self.connection_semaphore:
+            task_id = id(asyncio.current_task())
+            try:
+                async with self.pool_lock:
+                    if task_id not in self.connection_pool:
+                        conn = await aiosqlite.connect(
+                            self.db_path,
+                            timeout=30.0
+                        )
+                        await conn.execute('PRAGMA journal_mode = WAL')
+                        await conn.execute('PRAGMA busy_timeout = 5000')
+                        self.connection_pool[task_id] = conn
+                    
+                yield self.connection_pool[task_id]
+                
+            except Exception as e:
+                logger.error(f"Connection error: {e}")
+                raise
+            finally:
+                async with self.pool_lock:
+                    if task_id in self.connection_pool:
+                        await self.connection_pool[task_id].close()
+                        del self.connection_pool[task_id]
+
+    async def execute_with_retry(self, operation, *args):
+        """Execute database operations with retry logic"""
+        for attempt in range(self.max_retries):
+            try:
+                async with self.get_connection() as db:
+                    result = await operation(db, *args)
+                    await db.commit()
+                    return result
+            except Exception as e:
+                if attempt == self.max_retries - 1:
+                    logger.error(f"Operation failed after {self.max_retries} attempts: {e}")
+                    raise
+                await asyncio.sleep(1 * (attempt + 1))  # Exponential backoff

    async def ainit_db(self):
-        async with aiosqlite.connect(self.db_path) as db:
+        """Initialize database schema"""
+        async def _init(db):
            await db.execute('''
                CREATE TABLE IF NOT EXISTS crawled_data (
                    url TEXT PRIMARY KEY,
@@ -28,87 +92,101 @@ class AsyncDatabaseManager:
                    screenshot TEXT DEFAULT ""
                )
            ''')
-            await db.commit()
+        
+        await self.execute_with_retry(_init)
        await self.update_db_schema()

    async def update_db_schema(self):
-        async with aiosqlite.connect(self.db_path) as db:
-            # Check if the 'media' column exists
+        """Update database schema if needed"""
+        async def _check_columns(db):
            cursor = await db.execute("PRAGMA table_info(crawled_data)")
            columns = await cursor.fetchall()
-            column_names = [column[1] for column in columns]
-            
-            if 'media' not in column_names:
-                await self.aalter_db_add_column('media')
-            
-            # Check for other missing columns and add them if necessary
-            for column in ['links', 'metadata', 'screenshot']:
-                if column not in column_names:
-                    await self.aalter_db_add_column(column)
+            return [column[1] for column in columns]
+
+        column_names = await self.execute_with_retry(_check_columns)
+        
+        for column in ['media', 'links', 'metadata', 'screenshot']:
+            if column not in column_names:
+                await self.aalter_db_add_column(column)

    async def aalter_db_add_column(self, new_column: str):
-        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
-                await db.commit()
-            print(f"Added column '{new_column}' to the database.")
-        except Exception as e:
-            print(f"Error altering database to add {new_column} column: {e}")
+        """Add new column to the database"""
+        async def _alter(db):
+            await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
+            logger.info(f"Added column '{new_column}' to the database.")
+
+        await self.execute_with_retry(_alter)

    async def aget_cached_url(self, url: str) -> Optional[Tuple[str, str, str, str, str, str, str, bool, str]]:
+        """Retrieve cached URL data"""
+        async def _get(db):
+            async with db.execute(
+                'SELECT url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot FROM crawled_data WHERE url = ?',
+                (url,)
+            ) as cursor:
+                return await cursor.fetchone()
+
        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                async with db.execute('SELECT url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot FROM crawled_data WHERE url = ?', (url,)) as cursor:
-                    return await cursor.fetchone()
+            return await self.execute_with_retry(_get)
        except Exception as e:
-            print(f"Error retrieving cached URL: {e}")
+            logger.error(f"Error retrieving cached URL: {e}")
            return None

    async def acache_url(self, url: str, html: str, cleaned_html: str, markdown: str, extracted_content: str, success: bool, media: str = "{}", links: str = "{}", metadata: str = "{}", screenshot: str = ""):
+        """Cache URL data with retry logic"""
+        async def _cache(db):
+            await db.execute('''
+                INSERT INTO crawled_data (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                ON CONFLICT(url) DO UPDATE SET
+                    html = excluded.html,
+                    cleaned_html = excluded.cleaned_html,
+                    markdown = excluded.markdown,
+                    extracted_content = excluded.extracted_content,
+                    success = excluded.success,
+                    media = excluded.media,      
+                    links = excluded.links,    
+                    metadata = excluded.metadata,      
+                    screenshot = excluded.screenshot
+            ''', (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot))
+
        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('''
-                    INSERT INTO crawled_data (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot)
-                    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-                    ON CONFLICT(url) DO UPDATE SET
-                        html = excluded.html,
-                        cleaned_html = excluded.cleaned_html,
-                        markdown = excluded.markdown,
-                        extracted_content = excluded.extracted_content,
-                        success = excluded.success,
-                        media = excluded.media,      
-                        links = excluded.links,    
-                        metadata = excluded.metadata,      
-                        screenshot = excluded.screenshot
-                ''', (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot))
-                await db.commit()
+            await self.execute_with_retry(_cache)
        except Exception as e:
-            print(f"Error caching URL: {e}")
+            logger.error(f"Error caching URL: {e}")

    async def aget_total_count(self) -> int:
+        """Get total number of cached URLs"""
+        async def _count(db):
+            async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
+                result = await cursor.fetchone()
+                return result[0] if result else 0
+
        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
-                    result = await cursor.fetchone()
-                    return result[0] if result else 0
+            return await self.execute_with_retry(_count)
        except Exception as e:
-            print(f"Error getting total count: {e}")
+            logger.error(f"Error getting total count: {e}")
            return 0

    async def aclear_db(self):
+        """Clear all data from the database"""
+        async def _clear(db):
+            await db.execute('DELETE FROM crawled_data')
+
        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('DELETE FROM crawled_data')
-                await db.commit()
+            await self.execute_with_retry(_clear)
        except Exception as e:
-            print(f"Error clearing database: {e}")
+            logger.error(f"Error clearing database: {e}")

    async def aflush_db(self):
-        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('DROP TABLE IF EXISTS crawled_data')
-                await db.commit()
-        except Exception as e:
-            print(f"Error flushing database: {e}")
+        """Drop the entire table"""
+        async def _flush(db):
+            await db.execute('DROP TABLE IF EXISTS crawled_data')

+        try:
+            await self.execute_with_retry(_flush)
+        except Exception as e:
+            logger.error(f"Error flushing database: {e}")
+
+# Create a singleton instance
 async_db_manager = AsyncDatabaseManager()
--- a/crawl4ai/async_webcrawler.py
+++ b/crawl4ai/async_webcrawler.py
@@ -16,7 +16,7 @@ from .utils import (
    InvalidCSSSelectorError,
    format_html
 )
-
+from ._version import __version__ as crawl4ai_version

 class AsyncWebCrawler:
    def __init__(
@@ -46,9 +46,12 @@ class AsyncWebCrawler:
        await self.crawler_strategy.__aexit__(exc_type, exc_val, exc_tb)

    async def awarmup(self):
+        # Print a message for crawl4ai and its version
+        print(f"[LOG] 🚀 Crawl4AI {crawl4ai_version}")
        if self.verbose:
            print("[LOG] 🌤️  Warming up the AsyncWebCrawler")
-        await async_db_manager.ainit_db()
+        # await async_db_manager.ainit_db()
+        await async_db_manager.initialize()
        await self.arun(
            url="https://google.com/",
            word_count_threshold=5,
@@ -125,6 +128,7 @@ class AsyncWebCrawler:
                verbose,
                bool(cached),
                async_response=async_response,
+                bypass_cache=bypass_cache,
                **kwargs,
            )
            crawl_result.status_code = async_response.status_code if async_response else 200
@@ -168,7 +172,6 @@ class AsyncWebCrawler:
        ]
        return await asyncio.gather(*tasks)

-
    async def aprocess_html(
        self,
        url: str,
@@ -243,7 +246,7 @@ class AsyncWebCrawler:

        screenshot = None if not screenshot else screenshot

-        if not is_cached:
+        if not is_cached or kwargs.get("bypass_cache", False) or self.always_by_pass_cache:
            await async_db_manager.acache_url(
                url,
                html,
@@ -274,10 +277,13 @@ class AsyncWebCrawler:
        )

    async def aclear_cache(self):
-        await async_db_manager.aclear_db()
+        # await async_db_manager.aclear_db()
+        await async_db_manager.cleanup()

    async def aflush_cache(self):
        await async_db_manager.aflush_db()

    async def aget_cache_size(self):
        return await async_db_manager.aget_total_count()
+
+
--- a/crawl4ai/content_scrapping_strategy.py
+++ b/crawl4ai/content_scrapping_strategy.py
@@ -14,12 +14,97 @@ from .utils import (
    sanitize_html,
    extract_metadata,
    InvalidCSSSelectorError,
-    CustomHTML2Text,
+    # CustomHTML2Text,
    normalize_url,
    is_external_url
    
 )

+from .html2text import HTML2Text
+class CustomHTML2Text(HTML2Text):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.inside_pre = False
+        self.inside_code = False
+        self.preserve_tags = set()  # Set of tags to preserve
+        self.current_preserved_tag = None
+        self.preserved_content = []
+        self.preserve_depth = 0
+        
+        # Configuration options
+        self.skip_internal_links = False
+        self.single_line_break = False
+        self.mark_code = False
+        self.include_sup_sub = False
+        self.body_width = 0
+        self.ignore_mailto_links = True
+        self.ignore_links = False
+        self.escape_backslash = False
+        self.escape_dot = False
+        self.escape_plus = False
+        self.escape_dash = False
+        self.escape_snob = False
+
+    def update_params(self, **kwargs):
+        """Update parameters and set preserved tags."""
+        for key, value in kwargs.items():
+            if key == 'preserve_tags':
+                self.preserve_tags = set(value)
+            else:
+                setattr(self, key, value)
+
+    def handle_tag(self, tag, attrs, start):
+        # Handle preserved tags
+        if tag in self.preserve_tags:
+            if start:
+                if self.preserve_depth == 0:
+                    self.current_preserved_tag = tag
+                    self.preserved_content = []
+                    # Format opening tag with attributes
+                    attr_str = ''.join(f' {k}="{v}"' for k, v in attrs.items() if v is not None)
+                    self.preserved_content.append(f'<{tag}{attr_str}>')
+                self.preserve_depth += 1
+                return
+            else:
+                self.preserve_depth -= 1
+                if self.preserve_depth == 0:
+                    self.preserved_content.append(f'</{tag}>')
+                    # Output the preserved HTML block with proper spacing
+                    preserved_html = ''.join(self.preserved_content)
+                    self.o('\n' + preserved_html + '\n')
+                    self.current_preserved_tag = None
+                return
+
+        # If we're inside a preserved tag, collect all content
+        if self.preserve_depth > 0:
+            if start:
+                # Format nested tags with attributes
+                attr_str = ''.join(f' {k}="{v}"' for k, v in attrs.items() if v is not None)
+                self.preserved_content.append(f'<{tag}{attr_str}>')
+            else:
+                self.preserved_content.append(f'</{tag}>')
+            return
+
+        # Handle pre tags
+        if tag == 'pre':
+            if start:
+                self.o('```\n')
+                self.inside_pre = True
+            else:
+                self.o('\n```')
+                self.inside_pre = False
+        elif tag in ["h1", "h2", "h3", "h4", "h5", "h6"]:
+            pass
+        else:
+            super().handle_tag(tag, attrs, start)
+
+    def handle_data(self, data, entity_char=False):
+        """Override handle_data to capture content within preserved tags."""
+        if self.preserve_depth > 0:
+            self.preserved_content.append(data)
+            return
+        super().handle_data(data, entity_char)
+
 class ContentScrappingStrategy(ABC):
    @abstractmethod
    def scrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
--- a/crawl4ai/models/onnx/config.json
+++ b/crawl4ai/models/onnx/config.json
@@ -1,25 +0,0 @@
-{
-  "_name_or_path": "sentence-transformers/all-MiniLM-L6-v2",
-  "architectures": [
-    "BertModel"
-  ],
-  "attention_probs_dropout_prob": 0.1,
-  "classifier_dropout": null,
-  "gradient_checkpointing": false,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 384,
-  "initializer_range": 0.02,
-  "intermediate_size": 1536,
-  "layer_norm_eps": 1e-12,
-  "max_position_embeddings": 512,
-  "model_type": "bert",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 6,
-  "pad_token_id": 0,
-  "position_embedding_type": "absolute",
-  "transformers_version": "4.27.4",
-  "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
-}
--- a/crawl4ai/models/onnx/model.onnx
+++ b/crawl4ai/models/onnx/model.onnx
--- a/crawl4ai/models/onnx/special_tokens_map.json
+++ b/crawl4ai/models/onnx/special_tokens_map.json
@@ -1,7 +0,0 @@
-{
-  "cls_token": "[CLS]",
-  "mask_token": "[MASK]",
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "unk_token": "[UNK]"
-}
--- a/crawl4ai/models/onnx/tokenizer.json
+++ b/crawl4ai/models/onnx/tokenizer.json
--- a/crawl4ai/models/onnx/tokenizer_config.json
+++ b/crawl4ai/models/onnx/tokenizer_config.json
@@ -1,15 +0,0 @@
-{
-  "cls_token": "[CLS]",
-  "do_basic_tokenize": true,
-  "do_lower_case": true,
-  "mask_token": "[MASK]",
-  "model_max_length": 512,
-  "never_split": null,
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "special_tokens_map_file": "/Users/hammad/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125/special_tokens_map.json",
-  "strip_accents": null,
-  "tokenize_chinese_chars": true,
-  "tokenizer_class": "BertTokenizer",
-  "unk_token": "[UNK]"
-}
--- a/crawl4ai/models/onnx/vocab.txt
+++ b/crawl4ai/models/onnx/vocab.txt
--- a/crawl4ai/utils.py
+++ b/crawl4ai/utils.py
@@ -178,7 +178,7 @@ def escape_json_string(s):
    
    return s

-class CustomHTML2Text(HTML2Text):
+class CustomHTML2Text_v0(HTML2Text):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.inside_pre = False
@@ -981,6 +981,19 @@ def format_html(html_string):
    return soup.prettify()

 def normalize_url(href, base_url):
+    """Normalize URLs to ensure consistent format"""
+    from urllib.parse import urljoin, urlparse
+
+    # Parse base URL to get components
+    parsed_base = urlparse(base_url)
+    if not parsed_base.scheme or not parsed_base.netloc:
+        raise ValueError(f"Invalid base URL format: {base_url}")
+
+    # Use urljoin to handle all cases
+    normalized = urljoin(base_url, href.strip())
+    return normalized
+
+def normalize_url_tmp(href, base_url):
    """Normalize URLs to ensure consistent format"""
    # Extract protocol and domain from base URL
    try:
--- a/docs/assets/pitch-dark.png
+++ b/docs/assets/pitch-dark.png
--- a/docs/assets/pitch-dark.svg
+++ b/docs/assets/pitch-dark.svg
@@ -0,0 +1,64 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 500">
+    <!-- Background -->
+    <rect width="800" height="500" fill="#1a1a1a"/>
+    
+    <!-- Opportunities Section -->
+    <g transform="translate(50,50)">
+        <!-- Opportunity 1 Box -->
+        <rect x="0" y="0" width="300" height="150" rx="10" fill="#1a2d3d" stroke="#64b5f6" stroke-width="2"/>
+        <text x="150" y="30" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#64b5f6">Data Capitalization Opportunity</text>
+        <text x="150" y="60" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
+            <tspan x="150" dy="0">Transform digital footprints into assets</tspan>
+            <tspan x="150" dy="20">Personal data as capital</tspan>
+            <tspan x="150" dy="20">Enterprise knowledge valuation</tspan>
+            <tspan x="150" dy="20">New form of wealth creation</tspan>
+        </text>
+
+        <!-- Opportunity 2 Box -->
+        <rect x="0" y="200" width="300" height="150" rx="10" fill="#1a2d1a" stroke="#81c784" stroke-width="2"/>
+        <text x="150" y="230" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#81c784">Authentic Data Potential</text>
+        <text x="150" y="260" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
+            <tspan x="150" dy="0">Vast reservoir of real insights</tspan>
+            <tspan x="150" dy="20">Enhanced AI development</tspan>
+            <tspan x="150" dy="20">Diverse human knowledge</tspan>
+            <tspan x="150" dy="20">Willing participation model</tspan>
+        </text>
+    </g>
+
+    <!-- Development Pathway -->
+    <g transform="translate(450,50)">
+        <!-- Step 1 Box -->
+        <rect x="0" y="0" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">1. Open-Source Foundation</text>
+        <text x="150" y="65" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Data extraction engine &amp; community development</text>
+
+        <!-- Step 2 Box -->
+        <rect x="0" y="125" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="160" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">2. Data Capitalization Platform</text>
+        <text x="150" y="190" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Tools to structure &amp; value digital assets</text>
+
+        <!-- Step 3 Box -->
+        <rect x="0" y="250" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="285" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">3. Shared Data Marketplace</text>
+        <text x="150" y="315" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Economic platform for data exchange</text>
+    </g>
+
+    <!-- Connecting Arrows -->
+    <g transform="translate(400,125)">
+        <path d="M-20,0 L40,0" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
+        <path d="M-20,200 L40,200" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
+    </g>
+
+    <!-- Arrow Marker -->
+    <defs>
+        <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#666"/>
+        </marker>
+    </defs>
+
+    <!-- Vision Box at Bottom -->
+    <g transform="translate(200,420)">
+        <rect x="0" y="0" width="400" height="60" rx="10" fill="#2d2613" stroke="#ffd54f" stroke-width="2"/>
+        <text x="200" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ffd54f">Economic Vision: Shared Data Economy</text>
+    </g>
+</svg>
--- a/docs/md_v2/advanced/hooks.md
+++ b/docs/md_v2/advanced/hooks.md
--- a/docs/md_v2/api/parameters.md
+++ b/docs/md_v2/api/parameters.md
@@ -0,0 +1,35 @@
+# Parameter Reference Table
+
+| File Name | Parameter Name | Code Usage | Strategy/Class | Description |
+|-----------|---------------|------------|----------------|-------------|
+| async_crawler_strategy.py | user_agent | `kwargs.get("user_agent")` | AsyncPlaywrightCrawlerStrategy | User agent string for browser identification |
+| async_crawler_strategy.py | proxy | `kwargs.get("proxy")` | AsyncPlaywrightCrawlerStrategy | Proxy server configuration for network requests |
+| async_crawler_strategy.py | proxy_config | `kwargs.get("proxy_config")` | AsyncPlaywrightCrawlerStrategy | Detailed proxy configuration including auth |
+| async_crawler_strategy.py | headless | `kwargs.get("headless", True)` | AsyncPlaywrightCrawlerStrategy | Whether to run browser in headless mode |
+| async_crawler_strategy.py | browser_type | `kwargs.get("browser_type", "chromium")` | AsyncPlaywrightCrawlerStrategy | Type of browser to use (chromium/firefox/webkit) |
+| async_crawler_strategy.py | headers | `kwargs.get("headers", {})` | AsyncPlaywrightCrawlerStrategy | Custom HTTP headers for requests |
+| async_crawler_strategy.py | verbose | `kwargs.get("verbose", False)` | AsyncPlaywrightCrawlerStrategy | Enable detailed logging output |
+| async_crawler_strategy.py | sleep_on_close | `kwargs.get("sleep_on_close", False)` | AsyncPlaywrightCrawlerStrategy | Add delay before closing browser |
+| async_crawler_strategy.py | use_managed_browser | `kwargs.get("use_managed_browser", False)` | AsyncPlaywrightCrawlerStrategy | Use managed browser instance |
+| async_crawler_strategy.py | user_data_dir | `kwargs.get("user_data_dir", None)` | AsyncPlaywrightCrawlerStrategy | Custom directory for browser profile data |
+| async_crawler_strategy.py | session_id | `kwargs.get("session_id")` | AsyncPlaywrightCrawlerStrategy | Unique identifier for browser session |
+| async_crawler_strategy.py | override_navigator | `kwargs.get("override_navigator", False)` | AsyncPlaywrightCrawlerStrategy | Override browser navigator properties |
+| async_crawler_strategy.py | simulate_user | `kwargs.get("simulate_user", False)` | AsyncPlaywrightCrawlerStrategy | Simulate human-like behavior |
+| async_crawler_strategy.py | magic | `kwargs.get("magic", False)` | AsyncPlaywrightCrawlerStrategy | Enable advanced anti-detection features |
+| async_crawler_strategy.py | log_console | `kwargs.get("log_console", False)` | AsyncPlaywrightCrawlerStrategy | Log browser console messages |
+| async_crawler_strategy.py | js_only | `kwargs.get("js_only", False)` | AsyncPlaywrightCrawlerStrategy | Only execute JavaScript without page load |
+| async_crawler_strategy.py | page_timeout | `kwargs.get("page_timeout", 60000)` | AsyncPlaywrightCrawlerStrategy | Timeout for page load in milliseconds |
+| async_crawler_strategy.py | ignore_body_visibility | `kwargs.get("ignore_body_visibility", True)` | AsyncPlaywrightCrawlerStrategy | Process page even if body is hidden |
+| async_crawler_strategy.py | js_code | `kwargs.get("js_code", kwargs.get("js", self.js_code))` | AsyncPlaywrightCrawlerStrategy | Custom JavaScript code to execute |
+| async_crawler_strategy.py | wait_for | `kwargs.get("wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait for specific element/condition |
+| async_crawler_strategy.py | process_iframes | `kwargs.get("process_iframes", False)` | AsyncPlaywrightCrawlerStrategy | Extract content from iframes |
+| async_crawler_strategy.py | delay_before_return_html | `kwargs.get("delay_before_return_html")` | AsyncPlaywrightCrawlerStrategy | Additional delay before returning HTML |
+| async_crawler_strategy.py | remove_overlay_elements | `kwargs.get("remove_overlay_elements", False)` | AsyncPlaywrightCrawlerStrategy | Remove pop-ups and overlay elements |
+| async_crawler_strategy.py | screenshot | `kwargs.get("screenshot")` | AsyncPlaywrightCrawlerStrategy | Take page screenshot |
+| async_crawler_strategy.py | screenshot_wait_for | `kwargs.get("screenshot_wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait before taking screenshot |
+| async_crawler_strategy.py | semaphore_count | `kwargs.get("semaphore_count", 5)` | AsyncPlaywrightCrawlerStrategy | Concurrent request limit |
+| async_webcrawler.py | verbose | `kwargs.get("verbose", False)` | AsyncWebCrawler | Enable detailed logging |
+| async_webcrawler.py | warmup | `kwargs.get("warmup", True)` | AsyncWebCrawler | Initialize crawler with warmup request |
+| async_webcrawler.py | session_id | `kwargs.get("session_id", None)` | AsyncWebCrawler | Session identifier for browser reuse |
+| async_webcrawler.py | only_text | `kwargs.get("only_text", False)` | AsyncWebCrawler | Extract only text content |
+| async_webcrawler.py | bypass_cache | `kwargs.get("bypass_cache", False)` | AsyncWebCrawler | Skip cache and force fresh crawl |
--- a/docs/md_v2/basic/docker-deploymeny.md
+++ b/docs/md_v2/basic/docker-deploymeny.md
@@ -0,0 +1,459 @@
+# Docker Deployment
+
+Crawl4AI provides official Docker images for easy deployment and scalability. This guide covers installation, configuration, and usage of Crawl4AI in Docker environments.
+
+## Quick Start 🚀
+
+Pull and run the basic version:
+
+```bash
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+Test the deployment:
+```python
+import requests
+
+# Test health endpoint
+health = requests.get("http://localhost:11235/health")
+print("Health check:", health.json())
+
+# Test basic crawl
+response = requests.post(
+    "http://localhost:11235/crawl",
+    json={
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+)
+task_id = response.json()["task_id"]
+print("Task ID:", task_id)
+```
+
+## Available Images 🏷️
+
+- `unclecode/crawl4ai:basic` - Basic web crawling capabilities
+- `unclecode/crawl4ai:all` - Full installation with all features
+- `unclecode/crawl4ai:gpu` - GPU-enabled version for ML features
+
+## Configuration Options 🔧
+
+### Environment Variables
+
+```bash
+docker run -p 11235:11235 \
+    -e MAX_CONCURRENT_TASKS=5 \
+    -e OPENAI_API_KEY=your_key \
+    unclecode/crawl4ai:all
+```
+
+### Volume Mounting
+
+Mount a directory for persistent data:
+```bash
+docker run -p 11235:11235 \
+    -v $(pwd)/data:/app/data \
+    unclecode/crawl4ai:all
+```
+
+### Resource Limits
+
+Control container resources:
+```bash
+docker run -p 11235:11235 \
+    --memory=4g \
+    --cpus=2 \
+    unclecode/crawl4ai:all
+```
+
+## Usage Examples 📝
+
+### Basic Crawling
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "priority": 10
+}
+
+response = requests.post("http://localhost:11235/crawl", json=request)
+task_id = response.json()["task_id"]
+
+# Get results
+result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+### Structured Data Extraction
+
+```python
+schema = {
+    "name": "Crypto Prices",
+    "baseSelector": ".cds-tableRow-t45thuk",
+    "fields": [
+        {
+            "name": "crypto",
+            "selector": "td:nth-child(1) h2",
+            "type": "text",
+        },
+        {
+            "name": "price",
+            "selector": "td:nth-child(2)",
+            "type": "text",
+        }
+    ],
+}
+
+request = {
+    "urls": "https://www.coinbase.com/explore",
+    "extraction_config": {
+        "type": "json_css",
+        "params": {"schema": schema}
+    }
+}
+```
+
+### Dynamic Content Handling
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "js_code": [
+        "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+    ],
+    "wait_for": "article.tease-card:nth-child(10)"
+}
+```
+
+### AI-Powered Extraction (Full Version)
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "extraction_config": {
+        "type": "cosine",
+        "params": {
+            "semantic_filter": "business finance economy",
+            "word_count_threshold": 10,
+            "max_dist": 0.2,
+            "top_k": 3
+        }
+    }
+}
+```
+
+## Platform-Specific Instructions 💻
+
+### macOS
+```bash
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+### Ubuntu
+```bash
+# Basic version
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+
+# With GPU support
+docker pull unclecode/crawl4ai:gpu
+docker run --gpus all -p 11235:11235 unclecode/crawl4ai:gpu
+```
+
+### Windows (PowerShell)
+```powershell
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+## Testing 🧪
+
+Save this as `test_docker.py`:
+
+```python
+import requests
+import json
+import time
+import sys
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:11235"):
+        self.base_url = base_url
+        
+    def submit_and_wait(self, request_data: dict, timeout: int = 300) -> dict:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data)
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} timeout")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}")
+            status = result.json()
+            
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+
+def test_deployment():
+    tester = Crawl4AiTester()
+    
+    # Test basic crawl
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Basic crawl successful!")
+    print(f"Content length: {len(result['result']['markdown'])}")
+
+if __name__ == "__main__":
+    test_deployment()
+```
+
+## Advanced Configuration ⚙️
+
+### Crawler Parameters
+
+The `crawler_params` field allows you to configure the browser instance and crawling behavior. Here are key parameters you can use:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        # Browser Configuration
+        "headless": True,                    # Run in headless mode
+        "browser_type": "chromium",          # chromium/firefox/webkit
+        "user_agent": "custom-agent",        # Custom user agent
+        "proxy": "http://proxy:8080",        # Proxy configuration
+        
+        # Performance & Behavior
+        "page_timeout": 30000,               # Page load timeout (ms)
+        "verbose": True,                     # Enable detailed logging
+        "semaphore_count": 5,               # Concurrent request limit
+        
+        # Anti-Detection Features
+        "simulate_user": True,               # Simulate human behavior
+        "magic": True,                       # Advanced anti-detection
+        "override_navigator": True,          # Override navigator properties
+        
+        # Session Management
+        "user_data_dir": "./browser-data",   # Browser profile location
+        "use_managed_browser": True,         # Use persistent browser
+    }
+}
+```
+
+### Extra Parameters
+
+The `extra` field allows passing additional parameters directly to the crawler's `arun` function:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "extra": {
+        "word_count_threshold": 10,          # Min words per block
+        "only_text": True,                   # Extract only text
+        "bypass_cache": True,                # Force fresh crawl
+        "process_iframes": True,             # Include iframe content
+    }
+}
+```
+
+### Complete Examples
+
+1. **Advanced News Crawling**
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "crawler_params": {
+        "headless": True,
+        "page_timeout": 30000,
+        "remove_overlay_elements": True      # Remove popups
+    },
+    "extra": {
+        "word_count_threshold": 50,          # Longer content blocks
+        "bypass_cache": True                 # Fresh content
+    },
+    "css_selector": ".article-body"
+}
+```
+
+2. **Anti-Detection Configuration**
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        "simulate_user": True,
+        "magic": True,
+        "override_navigator": True,
+        "user_agent": "Mozilla/5.0 ...",
+        "headers": {
+            "Accept-Language": "en-US,en;q=0.9"
+        }
+    }
+}
+```
+
+3. **LLM Extraction with Custom Parameters**
+```python
+request = {
+    "urls": "https://openai.com/pricing",
+    "extraction_config": {
+        "type": "llm",
+        "params": {
+            "provider": "openai/gpt-4",
+            "schema": pricing_schema
+        }
+    },
+    "crawler_params": {
+        "verbose": True,
+        "page_timeout": 60000
+    },
+    "extra": {
+        "word_count_threshold": 1,
+        "only_text": True
+    }
+}
+```
+
+4. **Session-Based Dynamic Content**
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        "session_id": "dynamic_session",
+        "headless": False,
+        "page_timeout": 60000
+    },
+    "js_code": ["window.scrollTo(0, document.body.scrollHeight);"],
+    "wait_for": "js:() => document.querySelectorAll('.item').length > 10",
+    "extra": {
+        "delay_before_return_html": 2.0
+    }
+}
+```
+
+5. **Screenshot with Custom Timing**
+```python
+request = {
+    "urls": "https://example.com",
+    "screenshot": True,
+    "crawler_params": {
+        "headless": True,
+        "screenshot_wait_for": ".main-content"
+    },
+    "extra": {
+        "delay_before_return_html": 3.0
+    }
+}
+```
+
+### Parameter Reference Table
+
+| Category | Parameter | Type | Description |
+|----------|-----------|------|-------------|
+| Browser | headless | bool | Run browser in headless mode |
+| Browser | browser_type | str | Browser engine selection |
+| Browser | user_agent | str | Custom user agent string |
+| Network | proxy | str | Proxy server URL |
+| Network | headers | dict | Custom HTTP headers |
+| Timing | page_timeout | int | Page load timeout (ms) |
+| Timing | delay_before_return_html | float | Wait before capture |
+| Anti-Detection | simulate_user | bool | Human behavior simulation |
+| Anti-Detection | magic | bool | Advanced protection |
+| Session | session_id | str | Browser session ID |
+| Session | user_data_dir | str | Profile directory |
+| Content | word_count_threshold | int | Minimum words per block |
+| Content | only_text | bool | Text-only extraction |
+| Content | process_iframes | bool | Include iframe content |
+| Debug | verbose | bool | Detailed logging |
+| Debug | log_console | bool | Browser console logs |
+
+## Troubleshooting 🔍
+
+### Common Issues
+
+1. **Connection Refused**
+   ```
+   Error: Connection refused at localhost:11235
+   ```
+   Solution: Ensure the container is running and ports are properly mapped.
+
+2. **Resource Limits**
+   ```
+   Error: No available slots
+   ```
+   Solution: Increase MAX_CONCURRENT_TASKS or container resources.
+
+3. **GPU Access**
+   ```
+   Error: GPU not found
+   ```
+   Solution: Ensure proper NVIDIA drivers and use `--gpus all` flag.
+
+### Debug Mode
+
+Access container for debugging:
+```bash
+docker run -it --entrypoint /bin/bash unclecode/crawl4ai:all
+```
+
+View container logs:
+```bash
+docker logs [container_id]
+```
+
+## Best Practices 🌟
+
+1. **Resource Management**
+   - Set appropriate memory and CPU limits
+   - Monitor resource usage via health endpoint
+   - Use basic version for simple crawling tasks
+
+2. **Scaling**
+   - Use multiple containers for high load
+   - Implement proper load balancing
+   - Monitor performance metrics
+
+3. **Security**
+   - Use environment variables for sensitive data
+   - Implement proper network isolation
+   - Regular security updates
+
+## API Reference 📚
+
+### Health Check
+```http
+GET /health
+```
+
+### Submit Crawl Task
+```http
+POST /crawl
+Content-Type: application/json
+
+{
+    "urls": "string or array",
+    "extraction_config": {
+        "type": "basic|llm|cosine|json_css",
+        "params": {}
+    },
+    "priority": 1-10,
+    "ttl": 3600
+}
+```
+
+### Get Task Status
+```http
+GET /task/{task_id}
+```
+
+For more details, visit the [official documentation](https://crawl4ai.com/mkdocs/).
--- a/docs/md_v2/index.md
+++ b/docs/md_v2/index.md
@@ -72,7 +72,7 @@ Our documentation is organized into several sections:
 ### Advanced Features
 - [Magic Mode](advanced/magic-mode.md)
 - [Session Management](advanced/session-management.md)
- [Hooks & Authentication](advanced/hooks.md)
+- [Hooks & Authentication](advanced/hooks-auth.md)
 - [Proxy & Security](advanced/proxy-security.md)
 - [Content Processing](advanced/content-processing.md)

--- a/main.py
+++ b/main.py
@@ -1,254 +1,348 @@
-import os
-import importlib
 import asyncio
-from functools import lru_cache
-import logging
-logging.basicConfig(level=logging.DEBUG)
-
-from fastapi import FastAPI, HTTPException, Request
-from fastapi.responses import HTMLResponse, JSONResponse
-from fastapi.staticfiles import StaticFiles
-from fastapi.middleware.cors import CORSMiddleware  
-from fastapi.templating import Jinja2Templates
-from fastapi.exceptions import RequestValidationError
-from starlette.middleware.base import BaseHTTPMiddleware
-from starlette.responses import FileResponse
-from fastapi.responses import RedirectResponse
-
-from pydantic import BaseModel, HttpUrl
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import List, Optional
-
-from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.database import get_total_count, clear_db
-
+from fastapi import FastAPI, HTTPException, BackgroundTasks, Request
+from fastapi.responses import JSONResponse
+from pydantic import BaseModel, HttpUrl, Field
+from typing import Optional, List, Dict, Any, Union
+import psutil
 import time
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
-
-# load .env file
-from dotenv import load_dotenv
-load_dotenv()
-
-# Configuration
-__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
-MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
-current_requests = 0
-lock = asyncio.Lock()
-
-app = FastAPI()
-
-# Initialize rate limiter
-def rate_limit_key_func(request: Request):
-    access_token = request.headers.get("access-token")
-    if access_token == os.environ.get('ACCESS_TOKEN'):
-        return None
-    return get_remote_address(request)
-
-limiter = Limiter(key_func=rate_limit_key_func)
-app.state.limiter = limiter
-
-# Dictionary to store last request times for each client
-last_request_times = {}
-last_rate_limit = {}
-
-
-def get_rate_limit():
-    limit = os.environ.get('ACCESS_PER_MIN', "5")
-    return f"{limit}/minute"
-
-# Custom rate limit exceeded handler
-async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
-    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
-        last_rate_limit[request.client.host] = time.time()
-    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
-    reset_at = time.time() + retry_after
-    return JSONResponse(
-        status_code=429,
-        content={
-            "detail": "Rate limit exceeded",
-            "limit": str(exc.limit.limit),
-            "retry_after": retry_after,
-            'reset_at': reset_at,
-            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
-        }
-    )
-    
-app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
-
-
-# Middleware for token-based bypass and per-request limit
-class RateLimitMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
-        access_token = request.headers.get("access-token")
-        if access_token == os.environ.get('ACCESS_TOKEN'):
-            return await call_next(request)
-        
-        path = request.url.path
-        if path in ["/crawl", "/old"]:
-            client_ip = request.client.host
-            current_time = time.time()
-            
-            # Check time since last request
-            if client_ip in last_request_times:
-                time_since_last_request = current_time - last_request_times[client_ip]
-                if time_since_last_request < SPAN:
-                    return JSONResponse(
-                        status_code=429,
-                        content={
-                            "detail": "Too many requests",
-                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
-                            "retry_after": max(0, SPAN - time_since_last_request),
-                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
-                        }
-                    )
-            
-            last_request_times[client_ip] = current_time
-
-        return await call_next(request)
-
-app.add_middleware(RateLimitMiddleware)
-
-# CORS configuration
-origins = ["*"]  # Allow all origins
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=origins,  # List of origins that are allowed to make requests
-    allow_credentials=True,
-    allow_methods=["*"],  # Allows all methods
-    allow_headers=["*"],  # Allows all headers
+import uuid
+from collections import defaultdict
+from urllib.parse import urlparse
+import math
+import logging
+from enum import Enum
+from dataclasses import dataclass
+import json
+from crawl4ai import AsyncWebCrawler, CrawlResult
+from crawl4ai.extraction_strategy import (
+    LLMExtractionStrategy,
+    CosineStrategy,
+    JsonCssExtractionStrategy,
 )

-# Mount the pages directory as a static directory
-app.mount("/pages", StaticFiles(directory=__location__ + "/pages"), name="pages")
-app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
-site_templates = Jinja2Templates(directory=__location__ + "/site")
-templates = Jinja2Templates(directory=__location__ + "/pages")
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)

-@lru_cache()
-def get_crawler():
-    # Initialize and return a WebCrawler instance
-    crawler = WebCrawler(verbose = True)
-    crawler.warmup()
-    return crawler
+class TaskStatus(str, Enum):
+    PENDING = "pending"
+    PROCESSING = "processing"
+    COMPLETED = "completed"
+    FAILED = "failed"
+
+class CrawlerType(str, Enum):
+    BASIC = "basic"
+    LLM = "llm"
+    COSINE = "cosine"
+    JSON_CSS = "json_css"
+
+class ExtractionConfig(BaseModel):
+    type: CrawlerType
+    params: Dict[str, Any] = {}

 class CrawlRequest(BaseModel):
-    urls: List[str]
-    include_raw_html: Optional[bool] = False
-    bypass_cache: bool = False
-    extract_blocks: bool = True
-    word_count_threshold: Optional[int] = 5
-    extraction_strategy: Optional[str] = "NoExtractionStrategy"
-    extraction_strategy_args: Optional[dict] = {}
-    chunking_strategy: Optional[str] = "RegexChunking"
-    chunking_strategy_args: Optional[dict] = {}
+    urls: Union[HttpUrl, List[HttpUrl]]
+    extraction_config: Optional[ExtractionConfig] = None
+    crawler_params: Dict[str, Any] = {}
+    priority: int = Field(default=5, ge=1, le=10)
+    ttl: Optional[int] = 3600
+    js_code: Optional[List[str]] = None
+    wait_for: Optional[str] = None
    css_selector: Optional[str] = None
-    screenshot: Optional[bool] = False
-    user_agent: Optional[str] = None
-    verbose: Optional[bool] = True
+    screenshot: bool = False
+    magic: bool = False

-@app.get("/")
-def read_root():
-    return RedirectResponse(url="/mkdocs")
+@dataclass
+class TaskInfo:
+    id: str
+    status: TaskStatus
+    result: Optional[Union[CrawlResult, List[CrawlResult]]] = None
+    error: Optional[str] = None
+    created_at: float = time.time()
+    ttl: int = 3600

-@app.get("/old", response_class=HTMLResponse)
-@limiter.limit(get_rate_limit())
-async def read_index(request: Request):
-    partials_dir = os.path.join(__location__, "pages", "partial")
-    partials = {}
+class ResourceMonitor:
+    def __init__(self, max_concurrent_tasks: int = 10):
+        self.max_concurrent_tasks = max_concurrent_tasks
+        self.memory_threshold = 0.85
+        self.cpu_threshold = 0.90
+        self._last_check = 0
+        self._check_interval = 1  # seconds
+        self._last_available_slots = max_concurrent_tasks

-    for filename in os.listdir(partials_dir):
-        if filename.endswith(".html"):
-            with open(os.path.join(partials_dir, filename), "r", encoding="utf8") as file:
-                partials[filename[:-5]] = file.read()
+    async def get_available_slots(self) -> int:
+        current_time = time.time()
+        if current_time - self._last_check < self._check_interval:
+            return self._last_available_slots

-    return templates.TemplateResponse("index.html", {"request": request, **partials})
+        mem_usage = psutil.virtual_memory().percent / 100
+        cpu_usage = psutil.cpu_percent() / 100

-@app.get("/total-count")
-async def get_total_url_count():
-    count = get_total_count()
-    return JSONResponse(content={"count": count})
+        memory_factor = max(0, (self.memory_threshold - mem_usage) / self.memory_threshold)
+        cpu_factor = max(0, (self.cpu_threshold - cpu_usage) / self.cpu_threshold)

-@app.get("/clear-db")
-async def clear_database():
-    # clear_db()
-    return JSONResponse(content={"message": "Database cleared."})
+        self._last_available_slots = math.floor(
+            self.max_concurrent_tasks * min(memory_factor, cpu_factor)
+        )
+        self._last_check = current_time

-def import_strategy(module_name: str, class_name: str, *args, **kwargs):
-    try:
-        module = importlib.import_module(module_name)
-        strategy_class = getattr(module, class_name)
-        return strategy_class(*args, **kwargs)
-    except ImportError:
-        print("ImportError: Module not found.")
-        raise HTTPException(status_code=400, detail=f"Module {module_name} not found.")
-    except AttributeError:
-        print("AttributeError: Class not found.")
-        raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")
+        return self._last_available_slots
+
+class TaskManager:
+    def __init__(self, cleanup_interval: int = 300):
+        self.tasks: Dict[str, TaskInfo] = {}
+        self.high_priority = asyncio.PriorityQueue()
+        self.low_priority = asyncio.PriorityQueue()
+        self.cleanup_interval = cleanup_interval
+        self.cleanup_task = None
+
+    async def start(self):
+        self.cleanup_task = asyncio.create_task(self._cleanup_loop())
+
+    async def stop(self):
+        if self.cleanup_task:
+            self.cleanup_task.cancel()
+            try:
+                await self.cleanup_task
+            except asyncio.CancelledError:
+                pass
+
+    async def add_task(self, task_id: str, priority: int, ttl: int) -> None:
+        task_info = TaskInfo(id=task_id, status=TaskStatus.PENDING, ttl=ttl)
+        self.tasks[task_id] = task_info
+        queue = self.high_priority if priority > 5 else self.low_priority
+        await queue.put((-priority, task_id))  # Negative for proper priority ordering
+
+    async def get_next_task(self) -> Optional[str]:
+        try:
+            # Try high priority first
+            _, task_id = await asyncio.wait_for(self.high_priority.get(), timeout=0.1)
+            return task_id
+        except asyncio.TimeoutError:
+            try:
+                # Then try low priority
+                _, task_id = await asyncio.wait_for(self.low_priority.get(), timeout=0.1)
+                return task_id
+            except asyncio.TimeoutError:
+                return None
+
+    def update_task(self, task_id: str, status: TaskStatus, result: Any = None, error: str = None):
+        if task_id in self.tasks:
+            task_info = self.tasks[task_id]
+            task_info.status = status
+            task_info.result = result
+            task_info.error = error
+
+    def get_task(self, task_id: str) -> Optional[TaskInfo]:
+        return self.tasks.get(task_id)
+
+    async def _cleanup_loop(self):
+        while True:
+            try:
+                await asyncio.sleep(self.cleanup_interval)
+                current_time = time.time()
+                expired_tasks = [
+                    task_id
+                    for task_id, task in self.tasks.items()
+                    if current_time - task.created_at > task.ttl
+                    and task.status in [TaskStatus.COMPLETED, TaskStatus.FAILED]
+                ]
+                for task_id in expired_tasks:
+                    del self.tasks[task_id]
+            except Exception as e:
+                logger.error(f"Error in cleanup loop: {e}")
+
+class CrawlerPool:
+    def __init__(self, max_size: int = 10):
+        self.max_size = max_size
+        self.active_crawlers: Dict[AsyncWebCrawler, float] = {}
+        self._lock = asyncio.Lock()
+
+    async def acquire(self, **kwargs) -> AsyncWebCrawler:
+        async with self._lock:
+            # Clean up inactive crawlers
+            current_time = time.time()
+            inactive = [
+                crawler
+                for crawler, last_used in self.active_crawlers.items()
+                if current_time - last_used > 600  # 10 minutes timeout
+            ]
+            for crawler in inactive:
+                await crawler.__aexit__(None, None, None)
+                del self.active_crawlers[crawler]
+
+            # Create new crawler if needed
+            if len(self.active_crawlers) < self.max_size:
+                crawler = AsyncWebCrawler(**kwargs)
+                await crawler.__aenter__()
+                self.active_crawlers[crawler] = current_time
+                return crawler
+
+            # Reuse least recently used crawler
+            crawler = min(self.active_crawlers.items(), key=lambda x: x[1])[0]
+            self.active_crawlers[crawler] = current_time
+            return crawler
+
+    async def release(self, crawler: AsyncWebCrawler):
+        async with self._lock:
+            if crawler in self.active_crawlers:
+                self.active_crawlers[crawler] = time.time()
+
+    async def cleanup(self):
+        async with self._lock:
+            for crawler in list(self.active_crawlers.keys()):
+                await crawler.__aexit__(None, None, None)
+            self.active_crawlers.clear()
+
+class CrawlerService:
+    def __init__(self, max_concurrent_tasks: int = 10):
+        self.resource_monitor = ResourceMonitor(max_concurrent_tasks)
+        self.task_manager = TaskManager()
+        self.crawler_pool = CrawlerPool(max_concurrent_tasks)
+        self._processing_task = None
+
+    async def start(self):
+        await self.task_manager.start()
+        self._processing_task = asyncio.create_task(self._process_queue())
+
+    async def stop(self):
+        if self._processing_task:
+            self._processing_task.cancel()
+            try:
+                await self._processing_task
+            except asyncio.CancelledError:
+                pass
+        await self.task_manager.stop()
+        await self.crawler_pool.cleanup()
+
+    def _create_extraction_strategy(self, config: ExtractionConfig):
+        if not config:
+            return None
+
+        if config.type == CrawlerType.LLM:
+            return LLMExtractionStrategy(**config.params)
+        elif config.type == CrawlerType.COSINE:
+            return CosineStrategy(**config.params)
+        elif config.type == CrawlerType.JSON_CSS:
+            return JsonCssExtractionStrategy(**config.params)
+        return None
+
+    async def submit_task(self, request: CrawlRequest) -> str:
+        task_id = str(uuid.uuid4())
+        await self.task_manager.add_task(task_id, request.priority, request.ttl or 3600)
+        
+        # Store request data with task
+        self.task_manager.tasks[task_id].request = request
+        
+        return task_id
+
+    async def _process_queue(self):
+        while True:
+            try:
+                available_slots = await self.resource_monitor.get_available_slots()
+                if available_slots <= 0:
+                    await asyncio.sleep(1)
+                    continue
+
+                task_id = await self.task_manager.get_next_task()
+                if not task_id:
+                    await asyncio.sleep(1)
+                    continue
+
+                task_info = self.task_manager.get_task(task_id)
+                if not task_info:
+                    continue
+
+                request = task_info.request
+                self.task_manager.update_task(task_id, TaskStatus.PROCESSING)
+
+                try:
+                    crawler = await self.crawler_pool.acquire(**request.crawler_params)
+                    
+                    extraction_strategy = self._create_extraction_strategy(request.extraction_config)
+                    
+                    if isinstance(request.urls, list):
+                        results = await crawler.arun_many(
+                            urls=[str(url) for url in request.urls],
+                            extraction_strategy=extraction_strategy,
+                            js_code=request.js_code,
+                            wait_for=request.wait_for,
+                            css_selector=request.css_selector,
+                            screenshot=request.screenshot,
+                            magic=request.magic,
+                            **request.extra,
+                        )
+                    else:
+                        results = await crawler.arun(
+                            url=str(request.urls),
+                            extraction_strategy=extraction_strategy,
+                            js_code=request.js_code,
+                            wait_for=request.wait_for,
+                            css_selector=request.css_selector,
+                            screenshot=request.screenshot,
+                            magic=request.magic,
+                            **request.extra,
+                        )
+
+                    await self.crawler_pool.release(crawler)
+                    self.task_manager.update_task(task_id, TaskStatus.COMPLETED, results)
+
+                except Exception as e:
+                    logger.error(f"Error processing task {task_id}: {str(e)}")
+                    self.task_manager.update_task(task_id, TaskStatus.FAILED, error=str(e))
+
+            except Exception as e:
+                logger.error(f"Error in queue processing: {str(e)}")
+                await asyncio.sleep(1)
+
+app = FastAPI(title="Crawl4AI API")
+crawler_service = CrawlerService()
+
+@app.on_event("startup")
+async def startup_event():
+    await crawler_service.start()
+
+@app.on_event("shutdown")
+async def shutdown_event():
+    await crawler_service.stop()

@app.post("/crawl")
-@limiter.limit(get_rate_limit())
-async def crawl_urls(crawl_request: CrawlRequest, request: Request):
-    logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
-    global current_requests
-    async with lock:
-        if current_requests >= MAX_CONCURRENT_REQUESTS:
-            raise HTTPException(status_code=429, detail="Too many requests - please try again later.")
-        current_requests += 1
+async def crawl(request: CrawlRequest) -> Dict[str, str]:
+    task_id = await crawler_service.submit_task(request)
+    return {"task_id": task_id}

-    try:
-        logging.debug("[LOG] Loading extraction and chunking strategies...")
-        crawl_request.extraction_strategy_args['verbose'] = True
-        crawl_request.chunking_strategy_args['verbose'] = True
-        
-        extraction_strategy = import_strategy("crawl4ai.extraction_strategy", crawl_request.extraction_strategy, **crawl_request.extraction_strategy_args)
-        chunking_strategy = import_strategy("crawl4ai.chunking_strategy", crawl_request.chunking_strategy, **crawl_request.chunking_strategy_args)
+@app.get("/task/{task_id}")
+async def get_task_status(task_id: str):
+    task_info = crawler_service.task_manager.get_task(task_id)
+    if not task_info:
+        raise HTTPException(status_code=404, detail="Task not found")

-        # Use ThreadPoolExecutor to run the synchronous WebCrawler in async manner
-        logging.debug("[LOG] Running the WebCrawler...")
-        with ThreadPoolExecutor() as executor:
-            loop = asyncio.get_event_loop()
-            futures = [
-                loop.run_in_executor(
-                    executor, 
-                    get_crawler().run,
-                    str(url),
-                    crawl_request.word_count_threshold,
-                    extraction_strategy,
-                    chunking_strategy,
-                    crawl_request.bypass_cache,
-                    crawl_request.css_selector,
-                    crawl_request.screenshot,
-                    crawl_request.user_agent,
-                    crawl_request.verbose
-                )
-                for url in crawl_request.urls
-            ]
-            results = await asyncio.gather(*futures)
+    response = {
+        "status": task_info.status,
+        "created_at": task_info.created_at,
+    }

-        # if include_raw_html is False, remove the raw HTML content from the results
-        if not crawl_request.include_raw_html:
-            for result in results:
-                result.html = None
+    if task_info.status == TaskStatus.COMPLETED:
+        # Convert CrawlResult to dict for JSON response
+        if isinstance(task_info.result, list):
+            response["results"] = [result.dict() for result in task_info.result]
+        else:
+            response["result"] = task_info.result.dict()
+    elif task_info.status == TaskStatus.FAILED:
+        response["error"] = task_info.error

-        return {"results": [result.model_dump() for result in results]}
-    finally:
-        async with lock:
-            current_requests -= 1
-            
-@app.get("/strategies/extraction", response_class=JSONResponse)
-async def get_extraction_strategies():
-    with open(f"{__location__}/docs/extraction_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
-
-@app.get("/strategies/chunking", response_class=JSONResponse)
-async def get_chunking_strategies():
-    with open(f"{__location__}/docs/chunking_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
+    return response

+@app.get("/health")
+async def health_check():
+    available_slots = await crawler_service.resource_monitor.get_available_slots()
+    memory = psutil.virtual_memory()
+    return {
+        "status": "healthy",
+        "available_slots": available_slots,
+        "memory_usage": memory.percent,
+        "cpu_usage": psutil.cpu_percent(),
+    }

 if __name__ == "__main__":
    import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8888)
+    uvicorn.run(app, host="0.0.0.0", port=11235)
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -8,6 +8,7 @@ docs_dir: docs/md_v2
 nav:
  - Home: 'index.md'
  - 'Installation': 'basic/installation.md'
+  - 'Docker Deplotment': 'basic/docker-deploymeny.md'
  - 'Quick Start': 'basic/quickstart.md'
  
  - Basic:
@@ -34,6 +35,7 @@ nav:
    - 'Chunking': 'extraction/chunking.md'

  - API Reference:
+    - 'Parameters Table': 'api/parameters.md'
    - 'AsyncWebCrawler': 'api/async-webcrawler.md'
    - 'AsyncWebCrawler.arun()': 'api/arun.md'
    - 'CrawlResult': 'api/crawl-result.md'
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,2 +1,5 @@
 -r requirements.txt
-pytest
+pytest
+pytest-asyncio
+selenium
+setuptools
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,11 +1,11 @@
-aiosqlite==0.20.0
-html2text==2024.2.26
-lxml==5.3.0
-litellm==1.48.0
-numpy>=1.26.0,<2.1.1
-pillow==10.4.0
-playwright==1.47.0
-python-dotenv==1.0.1
-requests>=2.26.0,<2.32.3
-beautifulsoup4==4.12.3
-playwright_stealth==1.0.6
+aiosqlite~=0.20
+html2text~=2024.2
+lxml~=5.3
+litellm~=1.48
+numpy>=1.26.0,<3
+pillow~=10.4
+playwright>=1.47,<1.48
+python-dotenv~=1.0
+requests~=2.26
+beautifulsoup4~=4.12
+playwright_stealth~=1.0
--- a/setup.py
+++ b/setup.py
@@ -23,7 +23,7 @@ with open(os.path.join(__location__, "requirements.txt")) as f:
    requirements = f.read().splitlines()
    
 # Read version from __init__.py
-with open("crawl4ai/__init__.py") as f:
+with open("crawl4ai/_version.py") as f:
    for line in f:
        if line.startswith("__version__"):
            version = line.split("=")[1].strip().strip('"')
@@ -31,9 +31,11 @@ with open("crawl4ai/__init__.py") as f:

 # Define the requirements for different environments
 default_requirements = requirements
-torch_requirements = ["torch", "nltk", "spacy", "scikit-learn"]
-transformer_requirements = ["transformers", "tokenizers", "onnxruntime"]
-cosine_similarity_requirements = ["torch", "transformers", "nltk", "spacy"]
+# torch_requirements = ["torch", "nltk", "spacy", "scikit-learn"]
+# transformer_requirements = ["transformers", "tokenizers", "onnxruntime"]
+torch_requirements = ["torch", "nltk",  "scikit-learn"]
+transformer_requirements = ["transformers", "tokenizers"]
+cosine_similarity_requirements = ["torch", "transformers", "nltk" ]
 sync_requirements = ["selenium"]

 def install_playwright():
--- a/tests/test_docker.py
+++ b/tests/test_docker.py
@@ -0,0 +1,299 @@
+import requests
+import json
+import time
+import sys
+import base64
+import os
+from typing import Dict, Any
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:8000"):
+        self.base_url = base_url
+        
+    def submit_and_wait(self, request_data: Dict[str, Any], timeout: int = 300) -> Dict[str, Any]:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data)
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}")
+            status = result.json()
+            
+            if status["status"] == "failed":
+                print("Task failed:", status.get("error"))
+                raise Exception(f"Task failed: {status.get('error')}")
+                
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+
+def test_docker_deployment(version="basic"):
+    tester = Crawl4AiTester()
+    print(f"Testing Crawl4AI Docker {version} version")
+    
+    # Health check with timeout and retry
+    max_retries = 5
+    for i in range(max_retries):
+        try:
+            health = requests.get(f"{tester.base_url}/health", timeout=10)
+            print("Health check:", health.json())
+            break
+        except requests.exceptions.RequestException as e:
+            if i == max_retries - 1:
+                print(f"Failed to connect after {max_retries} attempts")
+                sys.exit(1)
+            print(f"Waiting for service to start (attempt {i+1}/{max_retries})...")
+            time.sleep(5)
+    
+    # Test cases based on version
+    test_basic_crawl(tester)
+    if version in ["full", "transformer"]:
+        test_cosine_extraction(tester)
+
+    # test_js_execution(tester)
+    # test_css_selector(tester)
+    # test_structured_extraction(tester)
+    # test_llm_extraction(tester)
+    # test_llm_with_ollama(tester)
+    # test_screenshot(tester)
+    
+
+def test_basic_crawl(tester: Crawl4AiTester):
+    print("\n=== Testing Basic Crawl ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+    assert len(result["result"]["markdown"]) > 0
+
+def test_js_execution(tester: Crawl4AiTester):
+    print("\n=== Testing JS Execution ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "js_code": [
+            "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+        ],
+        "wait_for": "article.tease-card:nth-child(10)",
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"JS execution result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_css_selector(tester: Crawl4AiTester):
+    print("\n=== Testing CSS Selector ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 7,
+        "css_selector": ".wide-tease-item__description",
+        "crawler_params": {
+            "headless": True
+        },
+        "extra": {"word_count_threshold": 10}
+        
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"CSS selector result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_structured_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Structured Extraction ===")
+    schema = {
+        "name": "Coinbase Crypto Prices",
+        "baseSelector": ".cds-tableRow-t45thuk",
+        "fields": [
+            {
+                "name": "crypto",
+                "selector": "td:nth-child(1) h2",
+                "type": "text",
+            },
+            {
+                "name": "symbol",
+                "selector": "td:nth-child(1) p",
+                "type": "text",
+            },
+            {
+                "name": "price",
+                "selector": "td:nth-child(2)",
+                "type": "text",
+            }
+        ],
+    }
+    
+    request = {
+        "urls": "https://www.coinbase.com/explore",
+        "priority": 9,
+        "extraction_config": {
+            "type": "json_css",
+            "params": {
+                "schema": schema
+            }
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    extracted = json.loads(result["result"]["extracted_content"])
+    print(f"Extracted {len(extracted)} items")
+    print("Sample item:", json.dumps(extracted[0], indent=2))
+    assert result["result"]["success"]
+    assert len(extracted) > 0
+
+def test_llm_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing LLM Extraction ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "model_name": {
+                "type": "string",
+                "description": "Name of the OpenAI model."
+            },
+            "input_fee": {
+                "type": "string",
+                "description": "Fee for input token for the OpenAI model."
+            },
+            "output_fee": {
+                "type": "string",
+                "description": "Fee for output token for the OpenAI model."
+            }
+        },
+        "required": ["model_name", "input_fee", "output_fee"]
+    }
+    
+    request = {
+        "urls": "https://openai.com/api/pricing",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "openai/gpt-4o-mini",
+                "api_token": os.getenv("OPENAI_API_KEY"),
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": """From the crawled content, extract all mentioned model names along with their fees for input and output tokens."""
+            }
+        },
+        "crawler_params": {"word_count_threshold": 1}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} model pricing entries")
+        print("Sample entry:", json.dumps(extracted[0], indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"LLM extraction test failed (might be due to missing API key): {str(e)}")
+
+def test_llm_with_ollama(tester: Crawl4AiTester):
+    print("\n=== Testing LLM with Ollama ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "article_title": {
+                "type": "string",
+                "description": "The main title of the news article"
+            },
+            "summary": {
+                "type": "string",
+                "description": "A brief summary of the article content"
+            },
+            "main_topics": {
+                "type": "array",
+                "items": {"type": "string"},
+                "description": "Main topics or themes discussed in the article"
+            }
+        }
+    }
+    
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "ollama/llama2",
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": "Extract the main article information including title, summary, and main topics."
+            }
+        },
+        "extra": {"word_count_threshold": 1},
+        "crawler_params": {"verbose": True}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print("Extracted content:", json.dumps(extracted, indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Ollama extraction test failed: {str(e)}")
+
+def test_cosine_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Cosine Extraction ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "cosine",
+            "params": {
+                "semantic_filter": "business finance economy",
+                "word_count_threshold": 10,
+                "max_dist": 0.2,
+                "top_k": 3
+            }
+        }
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} text clusters")
+        print("First cluster tags:", extracted[0]["tags"])
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Cosine extraction test failed: {str(e)}")
+
+def test_screenshot(tester: Crawl4AiTester):
+    print("\n=== Testing Screenshot ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 5,
+        "screenshot": True,
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Screenshot captured:", bool(result["result"]["screenshot"]))
+    
+    if result["result"]["screenshot"]:
+        # Save screenshot
+        screenshot_data = base64.b64decode(result["result"]["screenshot"])
+        with open("test_screenshot.jpg", "wb") as f:
+            f.write(screenshot_data)
+        print("Screenshot saved as test_screenshot.jpg")
+    
+    assert result["result"]["success"]
+
+if __name__ == "__main__":
+    version = sys.argv[1] if len(sys.argv) > 1 else "basic"
+    # version = "full"
+    test_docker_deployment(version)
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -0,0 +1,281 @@
+import asyncio
+import aiohttp
+import json
+import time
+import os
+from typing import Optional, Dict, Any
+from pydantic import BaseModel, HttpUrl
+
+class NBCNewsAPITest:
+    def __init__(self, base_url: str = "http://localhost:8000"):
+        self.base_url = base_url
+        self.session = None
+
+    async def __aenter__(self):
+        self.session = aiohttp.ClientSession()
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        if self.session:
+            await self.session.close()
+
+    async def submit_crawl(self, request_data: Dict[str, Any]) -> str:
+        async with self.session.post(f"{self.base_url}/crawl", json=request_data) as response:
+            result = await response.json()
+            return result["task_id"]
+
+    async def get_task_status(self, task_id: str) -> Dict[str, Any]:
+        async with self.session.get(f"{self.base_url}/task/{task_id}") as response:
+            return await response.json()
+
+    async def wait_for_task(self, task_id: str, timeout: int = 300, poll_interval: int = 2) -> Dict[str, Any]:
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
+
+            status = await self.get_task_status(task_id)
+            if status["status"] in ["completed", "failed"]:
+                return status
+
+            await asyncio.sleep(poll_interval)
+
+    async def check_health(self) -> Dict[str, Any]:
+        async with self.session.get(f"{self.base_url}/health") as response:
+            return await response.json()
+
+async def test_basic_crawl():
+    print("\n=== Testing Basic Crawl ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 10
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+        assert result["status"] == "completed"
+        assert "result" in result
+        assert result["result"]["success"]
+
+async def test_js_execution():
+    print("\n=== Testing JS Execution ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 8,
+            "js_code": [
+                "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+            ],
+            "wait_for": "article.tease-card:nth-child(10)",
+            "crawler_params": {
+                "headless": True
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"JS execution result length: {len(result['result']['markdown'])}")
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+
+async def test_css_selector():
+    print("\n=== Testing CSS Selector ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 7,
+            "css_selector": ".wide-tease-item__description"
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"CSS selector result length: {len(result['result']['markdown'])}")
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+
+async def test_structured_extraction():
+    print("\n=== Testing Structured Extraction ===")
+    async with NBCNewsAPITest() as api:
+        schema = {
+            "name": "NBC News Articles",
+            "baseSelector": "article.tease-card",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h2",
+                    "type": "text"
+                },
+                {
+                    "name": "description",
+                    "selector": ".tease-card__description",
+                    "type": "text"
+                },
+                {
+                    "name": "link",
+                    "selector": "a",
+                    "type": "attribute",
+                    "attribute": "href"
+                }
+            ]
+        }
+        
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 9,
+            "extraction_config": {
+                "type": "json_css",
+                "params": {
+                    "schema": schema
+                }
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} articles")
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+        assert len(extracted) > 0
+
+async def test_batch_crawl():
+    print("\n=== Testing Batch Crawl ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": [
+                "https://www.nbcnews.com/business",
+                "https://www.nbcnews.com/business/consumer",
+                "https://www.nbcnews.com/business/economy"
+            ],
+            "priority": 6,
+            "crawler_params": {
+                "headless": True
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print(f"Batch crawl completed, got {len(result['results'])} results")
+        assert result["status"] == "completed"
+        assert "results" in result
+        assert len(result["results"]) == 3
+
+async def test_llm_extraction():
+    print("\n=== Testing LLM Extraction with Ollama ===")
+    async with NBCNewsAPITest() as api:
+        schema = {
+            "type": "object",
+            "properties": {
+                "article_title": {
+                    "type": "string",
+                    "description": "The main title of the news article"
+                },
+                "summary": {
+                    "type": "string",
+                    "description": "A brief summary of the article content"
+                },
+                "main_topics": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Main topics or themes discussed in the article"
+                }
+            },
+            "required": ["article_title", "summary", "main_topics"]
+        }
+
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 8,
+            "extraction_config": {
+                "type": "llm",
+                "params": {
+                    "provider": "openai/gpt-4o-mini",
+                    "api_key": os.getenv("OLLAMA_API_KEY"),
+                    "schema": schema,
+                    "extraction_type": "schema",
+                    "instruction": """Extract the main article information including title, a brief summary, and main topics discussed. 
+                    Focus on the primary business news article on the page."""
+                }
+            },
+            "crawler_params": {
+                "headless": True,
+                "word_count_threshold": 1
+            }
+        }
+        
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        
+        if result["status"] == "completed":
+            extracted = json.loads(result["result"]["extracted_content"])
+            print(f"Extracted article analysis:")
+            print(json.dumps(extracted, indent=2))
+        
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+
+async def test_screenshot():
+    print("\n=== Testing Screenshot ===")
+    async with NBCNewsAPITest() as api:
+        request = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 5,
+            "screenshot": True,
+            "crawler_params": {
+                "headless": True
+            }
+        }
+        task_id = await api.submit_crawl(request)
+        result = await api.wait_for_task(task_id)
+        print("Screenshot captured:", bool(result["result"]["screenshot"]))
+        assert result["status"] == "completed"
+        assert result["result"]["success"]
+        assert result["result"]["screenshot"] is not None
+
+async def test_priority_handling():
+    print("\n=== Testing Priority Handling ===")
+    async with NBCNewsAPITest() as api:
+        # Submit low priority task first
+        low_priority = {
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 1,
+            "crawler_params": {"headless": True}
+        }
+        low_task_id = await api.submit_crawl(low_priority)
+
+        # Submit high priority task
+        high_priority = {
+            "urls": "https://www.nbcnews.com/business/consumer",
+            "priority": 10,
+            "crawler_params": {"headless": True}
+        }
+        high_task_id = await api.submit_crawl(high_priority)
+
+        # Get both results
+        high_result = await api.wait_for_task(high_task_id)
+        low_result = await api.wait_for_task(low_task_id)
+
+        print("Both tasks completed")
+        assert high_result["status"] == "completed"
+        assert low_result["status"] == "completed"
+
+async def main():
+    try:
+        # Start with health check
+        async with NBCNewsAPITest() as api:
+            health = await api.check_health()
+            print("Server health:", health)
+
+        # Run all tests
+        # await test_basic_crawl()
+        # await test_js_execution()
+        # await test_css_selector()
+        # await test_structured_extraction()
+        await test_llm_extraction()
+        # await test_batch_crawl()
+        # await test_screenshot()
+        # await test_priority_handling()
+
+    except Exception as e:
+        print(f"Test failed: {str(e)}")
+        raise
+
+if __name__ == "__main__":
+    asyncio.run(main())