chore: update version to 0.3.744 and add publish.sh to .gitignore

- Updated version to 0.3.743
  - Improved ManagedBrowser configuration with dynamic host/port
  - Implemented fast HTML formatting in web crawler
  - Enhanced markdown generation with a new generator class
  - Improved sanitization and utility functions
  - Added contributor details and pull request acknowledgments
  - Updated documentation for clearer usage scenarios
  - Adjusted tests to reflect class name changes

.do/app.yaml

@@ -0,0 +1,19 @@
+alerts:
+- rule: DEPLOYMENT_FAILED
+- rule: DOMAIN_FAILED
+name: crawl4ai
+region: nyc
+services:
+- dockerfile_path: Dockerfile
+  github:
+    branch: 0.3.74
+    deploy_on_push: true
+    repo: unclecode/crawl4ai 
+  health_check:
+    http_path: /health
+  http_port: 11235
+  instance_count: 1
+  instance_size_slug: professional-xs
+  name: web
+  routes:
+  - path: /

.do/deploy.template.yaml

@@ -0,0 +1,22 @@
+spec:
+  name: crawl4ai
+  services:
+    - name: crawl4ai
+      git:
+        branch: 0.3.74
+        repo_clone_url: https://github.com/unclecode/crawl4ai.git
+      dockerfile_path: Dockerfile
+      http_port: 11235
+      instance_count: 1
+      instance_size_slug: professional-xs
+      health_check:
+        http_path: /health
+      envs:
+        - key: INSTALL_TYPE
+          value: "basic"
+        - key: PYTHON_VERSION  
+          value: "3.10"
+        - key: ENABLE_GPU
+          value: "false"
+      routes:
+        - path: /

.gitignore

@@ -199,6 +199,7 @@ test_env/
 **/.DS_Store
 
 todo.md
+todo_executor.md
 git_changes.py
 git_changes.md
 pypi_build.sh
@@ -207,4 +208,10 @@ git_issues.md
 
 .tests/
 .issues/
-.docs/
+.docs/
+.issues/
+.gitboss/
+todo_executor.md
+protect-all-except-feature.sh
+manage-collab.sh
+publish.sh

CHANGELOG.md

@@ -1,8 +1,401 @@
 # Changelog
 
+## [0.3.743] November 27, 2024
+
+Enhance features and documentation
+- Updated version to 0.3.743
+- Improved ManagedBrowser configuration with dynamic host/port
+- Implemented fast HTML formatting in web crawler
+- Enhanced markdown generation with a new generator class
+- Improved sanitization and utility functions
+- Added contributor details and pull request acknowledgments
+- Updated documentation for clearer usage scenarios
+- Adjusted tests to reflect class name changes
+
+### CONTRIBUTORS.md
+Added new contributors and pull request details.
+Updated community contributions and acknowledged pull requests.
+
+### crawl4ai/__version__.py
+Version update.
+Bumped version to 0.3.743.
+
+### crawl4ai/async_crawler_strategy.py
+Improved ManagedBrowser configuration.
+Enhanced browser initialization with configurable host and debugging port; improved hook execution.
+
+### crawl4ai/async_webcrawler.py
+Optimized HTML processing.
+Implemented 'fast_format_html' for optimized HTML formatting; applied it when 'prettiify' is enabled.
+
+### crawl4ai/content_scraping_strategy.py
+Enhanced markdown generation strategy.
+Updated to use DefaultMarkdownGenerator and improved markdown generation with filters option.
+
+### crawl4ai/markdown_generation_strategy.py
+Refactored markdown generation class.
+Renamed DefaultMarkdownGenerationStrategy to DefaultMarkdownGenerator; added content filter handling.
+
+### crawl4ai/utils.py
+Enhanced utility functions.
+Improved input sanitization and enhanced HTML formatting method.
+
+### docs/md_v2/advanced/hooks-auth.md
+Improved documentation for hooks.
+Updated code examples to include cookies in crawler strategy initialization.
+
+### tests/async/test_markdown_genertor.py
+Refactored tests to match class renaming.
+Updated tests to use renamed DefaultMarkdownGenerator class.
+
+## [0.3.74] November 17, 2024
+
+This changelog details the updates and changes introduced in Crawl4AI version 0.3.74. It's designed to inform developers about new features, modifications to existing components, removals, and other important information.
+
+### 1. File Download Processing
+
+- Users can now specify download folders using the `downloads_path` parameter in the `AsyncWebCrawler` constructor or the `arun` method. If not specified, downloads are saved to a "downloads" folder within the `.crawl4ai` directory.
+- File download tracking is integrated into the `CrawlResult` object.  Successfully downloaded files are listed in the `downloaded_files` attribute, providing their paths.
+- Added `accept_downloads` parameter to the crawler strategies (defaults to `False`). If set to True you can add JS code and `wait_for` parameter for file download.
+
+**Example:**
+
+```python
+import asyncio
+import os
+from pathlib import Path
+from crawl4ai import AsyncWebCrawler
+
+async def download_example():
+    downloads_path = os.path.join(Path.home(), ".crawl4ai", "downloads")
+    os.makedirs(downloads_path, exist_ok=True)
+
+    async with AsyncWebCrawler(
+        accept_downloads=True, 
+        downloads_path=downloads_path, 
+        verbose=True
+    ) as crawler:
+        result = await crawler.arun(
+            url="https://www.python.org/downloads/",
+            js_code="""
+                const downloadLink = document.querySelector('a[href$=".exe"]');
+                if (downloadLink) { downloadLink.click(); }
+            """,
+            wait_for=5 # To ensure download has started
+        )
+
+        if result.downloaded_files:
+            print("Downloaded files:")
+            for file in result.downloaded_files:
+                print(f"- {file}")
+
+asyncio.run(download_example())
+
+```
+
+### 2. Refined Content Filtering
+
+- Introduced the `RelevanceContentFilter` strategy (and its implementation `BM25ContentFilter`) for extracting relevant content from web pages, replacing Fit Markdown and other content cleaning strategy. This new strategy leverages the BM25 algorithm to identify chunks of text relevant to the page's title, description, keywords, or a user-provided query.
+- The `fit_markdown` flag in the content scraper is used to filter content based on title, meta description, and keywords.
+
+**Example:**
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.content_filter_strategy import BM25ContentFilter
+
+async def filter_content(url, query):
+    async with AsyncWebCrawler() as crawler:
+        content_filter = BM25ContentFilter(user_query=query)
+        result = await crawler.arun(url=url, extraction_strategy=content_filter, fit_markdown=True)
+        print(result.extracted_content)  # Or result.fit_markdown for the markdown version
+        print(result.fit_html) # Or result.fit_html to show HTML with only the filtered content
+
+asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple", "fruit nutrition health"))
+```
+
+### 3. Raw HTML and Local File Support
+
+- Added support for crawling local files and raw HTML content directly.
+- Use the `file://` prefix for local file paths.
+- Use the `raw:` prefix for raw HTML strings.
+
+**Example:**
+
+```python
+async def crawl_local_or_raw(crawler, content, content_type):
+    prefix = "file://" if content_type == "local" else "raw:"
+    url = f"{prefix}{content}"
+    result = await crawler.arun(url=url)
+    if result.success:
+        print(f"Markdown Content from {content_type.title()} Source:")
+        print(result.markdown)
+
+# Example usage with local file and raw HTML
+async def main():
+    async with AsyncWebCrawler() as crawler:
+        # Local File
+        await crawl_local_or_raw(
+            crawler, os.path.abspath('tests/async/sample_wikipedia.html'), "local"
+        )
+        # Raw HTML
+        await crawl_raw_html(crawler, "<h1>Raw Test</h1><p>This is raw HTML.</p>")
+        
+
+asyncio.run(main())
+```
+
+### 4. Browser Management
+
+- New asynchronous crawler strategy implemented using Playwright.
+- `ManagedBrowser` class introduced for improved browser session handling, offering features like persistent browser sessions between requests (using  `session_id`  parameter) and browser process monitoring.
+- Updated to tf-playwright-stealth for enhanced stealth capabilities.
+- Added `use_managed_browser`, `use_persistent_context`, and `chrome_channel` parameters to AsyncPlaywrightCrawlerStrategy.
+
+
+**Example:**
+```python
+async def browser_management_demo():
+    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "user-data-dir")
+    os.makedirs(user_data_dir, exist_ok=True)  # Ensure directory exists
+    async with AsyncWebCrawler(
+        use_managed_browser=True,
+        user_data_dir=user_data_dir,
+        use_persistent_context=True,
+        verbose=True
+    ) as crawler:
+        result1 = await crawler.arun(
+            url="https://example.com", session_id="my_session"
+        )
+        result2 = await crawler.arun(
+            url="https://example.com/anotherpage", session_id="my_session"
+        )
+
+asyncio.run(browser_management_demo())
+```
+
+
+### 5. API Server & Cache Improvements
+
+- Added CORS support to API server.
+- Implemented static file serving.
+- Enhanced root redirect functionality.
+- Cache database updated to store response headers and downloaded files information. It utilizes a file system approach to manage large content efficiently.
+- New, more efficient caching database built using xxhash and file system approach.
+- Introduced `CacheMode` enum (`ENABLED`, `DISABLED`, `READ_ONLY`, `WRITE_ONLY`, `BYPASS`) and `always_bypass_cache` parameter in AsyncWebCrawler for fine-grained cache control. This replaces `bypass_cache`, `no_cache_read`, `no_cache_write`, and `always_by_pass_cache`.
+
+
+### 🗑️ Removals
+
+- Removed deprecated: `crawl4ai/content_cleaning_strategy.py`.
+- Removed internal class ContentCleaningStrategy
+- Removed legacy cache control flags:  `bypass_cache`,  `disable_cache`,  `no_cache_read`,  `no_cache_write`, and `always_by_pass_cache`.  These have been superseded by  `cache_mode`.
+
+
+### ⚙️ Other Changes
+
+- Moved version file to `crawl4ai/__version__.py`.
+- Added `crawl4ai/cache_context.py`.
+- Added `crawl4ai/version_manager.py`.
+- Added `crawl4ai/migrations.py`.
+- Added `crawl4ai-migrate` entry point.
+- Added config `NEED_MIGRATION` and `SHOW_DEPRECATION_WARNINGS`.
+- API server now requires an API token for authentication, configurable with the `CRAWL4AI_API_TOKEN` environment variable.  This enhances API security.
+- Added synchronous crawl endpoint `/crawl_sync` for immediate result retrieval, and direct crawl endpoint `/crawl_direct` bypassing the task queue.
+
+
+### ⚠️ Deprecation Notices
+
+- The synchronous version of `WebCrawler` is being phased out.  While still available via `crawl4ai[sync]`, it will eventually be removed. Transition to `AsyncWebCrawler` is strongly recommended. Boolean cache control flags in `arun` are also deprecated, migrate to using the `cache_mode` parameter.  See examples in the "New Features" section above for correct usage.
+
+
+### 🐛 Bug Fixes
+
+- Resolved issue with browser context closing unexpectedly in Docker. This significantly improves stability, particularly within containerized environments. 
+- Fixed memory leaks associated with incorrect asynchronous cleanup by removing the `__del__` method and ensuring the browser context is closed explicitly using context managers.
+- Improved error handling in `WebScrapingStrategy`. More detailed error messages and suggestions for debugging will minimize frustration when running into unexpected issues.
+- Fixed issue with incorrect text parsing in specific HTML structures.
+
+
+### Example of migrating to the new CacheMode:
+
+**Old way:**
+
+```python
+crawler = AsyncWebCrawler(always_by_pass_cache=True)
+result = await crawler.arun(url="https://example.com", bypass_cache=True)
+```
+
+**New way:**
+
+```python
+from crawl4ai import CacheMode
+
+crawler = AsyncWebCrawler(always_bypass_cache=True)
+result = await crawler.arun(url="https://example.com", cache_mode=CacheMode.BYPASS)
+```
+
+
+## [0.3.74] - November 13, 2024
+
+1. **File Download Processing** (Nov 14, 2024)
+   - Added capability for users to specify download folders
+   - Implemented file download tracking in crowd result object
+   - Created new file: `tests/async/test_async_doanloader.py`
+
+2. **Content Filtering Improvements** (Nov 14, 2024)
+   - Introduced Relevance Content Filter as an improvement over Fit Markdown
+   - Implemented BM25 algorithm for content relevance matching
+   - Added new file: `crawl4ai/content_filter_strategy.py`
+   - Removed deprecated: `crawl4ai/content_cleaning_strategy.py`
+
+3. **Local File and Raw HTML Support** (Nov 13, 2024)
+   - Added support for processing local files
+   - Implemented raw HTML input handling in AsyncWebCrawler
+   - Enhanced `crawl4ai/async_webcrawler.py` with significant performance improvements
+
+4. **Browser Management Enhancements** (Nov 12, 2024)
+   - Implemented new async crawler strategy using Playwright
+   - Introduced ManagedBrowser for better browser session handling
+   - Added support for persistent browser sessions
+   - Updated from playwright_stealth to tf-playwright-stealth
+
+5. **API Server Component**
+   - Added CORS support
+   - Implemented static file serving
+   - Enhanced root redirect functionality
+
+
+
+## [0.3.731] - November 13, 2024
+
+### Added
+- Support for raw HTML and local file crawling via URL prefixes ('raw:', 'file://')
+- Browser process monitoring for managed browser instances
+- Screenshot capability for raw HTML and local file content
+- Response headers storage in cache database
+- New `fit_markdown` flag for optional markdown generation
+
+### Changed
+- Switched HTML parser from 'html.parser' to 'lxml' for ~4x performance improvement 
+- Optimized BeautifulSoup text conversion and element selection
+- Pre-compiled regular expressions for better performance
+- Improved metadata extraction efficiency
+- Response headers now stored alongside HTML in cache
+
+### Removed
+- `__del__` method from AsyncPlaywrightCrawlerStrategy to prevent async cleanup issues
+
+### Fixed 
+- Issue #256: Added support for crawling raw HTML content
+- Issue #253: Implemented file:// protocol handling
+- Missing response headers in cached results
+- Memory leaks from improper async cleanup
+
+## [v0.3.731] - 2024-11-13 Changelog for Issue 256 Fix
+- Fixed: Browser context unexpectedly closing in Docker environment during crawl operations.
+- Removed: __del__ method from AsyncPlaywrightCrawlerStrategy to prevent unreliable asynchronous cleanup, ensuring - browser context is closed explicitly within context managers.
+- Added: Monitoring for ManagedBrowser subprocess to detect and log unexpected terminations.
+- Updated: Dockerfile configurations to expose debugging port (9222) and allocate additional shared memory for improved browser stability.
+- Improved: Error handling and resource cleanup processes for browser lifecycle management within the Docker environment.
+
+## [v0.3.73] - 2024-11-05
+
+### Major Features
+- **New Doctor Feature**
+  - Added comprehensive system diagnostics tool
+  - Available through package hub and CLI
+  - Provides automated troubleshooting and system health checks
+  - Includes detailed reporting of configuration issues
+
+- **Dockerized API Server**
+  - Released complete Docker implementation for API server
+  - Added comprehensive documentation for Docker deployment
+  - Implemented container communication protocols
+  - Added environment configuration guides
+
+- **Managed Browser Integration**
+  - Added support for user-controlled browser instances
+  - Implemented `ManagedBrowser` class for better browser lifecycle management
+  - Added ability to connect to existing Chrome DevTools Protocol (CDP) endpoints
+  - Introduced user data directory support for persistent browser profiles
+
+- **Enhanced HTML Processing**
+  - Added HTML tag preservation feature during markdown conversion
+  - Introduced configurable tag preservation system
+  - Improved pre-tag and code block handling
+  - Added support for nested preserved tags with attribute retention
+
+### Improvements
+- **Browser Handling**
+  - Added flag to ignore body visibility for problematic pages
+  - Improved browser process cleanup and management
+  - Enhanced temporary directory handling for browser profiles
+  - Added configurable browser launch arguments
+
+- **Database Management**
+  - Implemented connection pooling for better performance
+  - Added retry logic for database operations
+  - Improved error handling and logging
+  - Enhanced cleanup procedures for database connections
+
+- **Resource Management**
+  - Added memory and CPU monitoring
+  - Implemented dynamic task slot allocation based on system resources
+  - Added configurable cleanup intervals
+
+### Technical Improvements
+- **Code Structure**
+  - Moved version management to dedicated _version.py file
+  - Improved error handling throughout the codebase
+  - Enhanced logging system with better error reporting
+  - Reorganized core components for better maintainability
+
+### Bug Fixes
+- Fixed issues with browser process termination
+- Improved handling of connection timeouts
+- Enhanced error recovery in database operations
+- Fixed memory leaks in long-running processes
+
+### Dependencies
+- Updated Playwright to v1.47
+- Updated core dependencies with more flexible version constraints
+- Added new development dependencies for testing
+
+### Breaking Changes
+- Changed default browser handling behavior
+- Modified database connection management approach
+- Updated API response structure for better consistency
+
+### Migration Guide
+When upgrading to v0.3.73, be aware of the following changes:
+
+1. Docker Deployment:
+   - Review Docker documentation for new deployment options
+   - Update environment configurations as needed
+   - Check container communication settings
+
+2. If using custom browser management:
+   - Update browser initialization code to use new ManagedBrowser class
+   - Review browser cleanup procedures
+
+3. For database operations:
+   - Check custom database queries for compatibility with new connection pooling
+   - Update error handling to work with new retry logic
+
+4. Using the Doctor:
+   - Run doctor command for system diagnostics: `crawl4ai doctor`
+   - Review generated reports for potential issues
+   - Follow recommended fixes for any identified problems
+
+
+## [v0.3.73] - 2024-11-04
+This commit introduces several key enhancements, including improved error handling and robust database operations in `async_database.py`, which now features a connection pool and retry logic for better reliability. Updates to the README.md provide clearer instructions and a better user experience with links to documentation sections. The `.gitignore` file has been refined to include additional directories, while the async web crawler now utilizes a managed browser for more efficient crawling. Furthermore, multiple dependency updates and introduction of the `CustomHTML2Text` class enhance text extraction capabilities.
+
 ## [v0.3.73] - 2024-10-24
 
 ### Added
+- preserve_tags: Added support for preserving specific HTML tags during markdown conversion.
 - Smart overlay removal system in AsyncPlaywrightCrawlerStrategy:
   - Automatic removal of popups, modals, and cookie notices
   - Detection and removal of fixed/sticky position elements
@@ -84,7 +477,7 @@
 ## [v0.3.72] - 2024-10-20
 
 ### Fixed
-- Added support for parsing Base64 encoded images in WebScrappingStrategy
+- Added support for parsing Base64 encoded images in WebScrapingStrategy
 
 ### Added
 - Forked and integrated a customized version of the html2text library for more control over Markdown generation
@@ -107,7 +500,7 @@
 ### Developer Notes
 - The customized html2text library is now located within the crawl4ai package
 - New configuration options are available in the `config.py` file for external content handling
-- The `WebScrappingStrategy` class has been updated to accommodate new external content exclusion options
+- The `WebScrapingStrategy` class has been updated to accommodate new external content exclusion options
 
 ## [v0.3.71] - 2024-10-19
 
@@ -184,7 +577,7 @@ These updates aim to provide more flexibility in text processing, improve perfor
 
 ### Improvements
 1. **Better Error Handling**:
-   - Enhanced error reporting in WebScrappingStrategy with detailed error messages and suggestions.
+   - Enhanced error reporting in WebScrapingStrategy with detailed error messages and suggestions.
    - Added console message and error logging for better debugging.
 
 2. **Image Processing Enhancements**:
@@ -242,43 +635,43 @@ These updates aim to provide more flexibility in text processing, improve perfor
 - Allows retrieval of content after a specified delay, useful for dynamically loaded content.
 - **How to use**: Access `result.get_delayed_content(delay_in_seconds)` after crawling.
 
-## Improvements and Optimizations
+### Improvements and Optimizations
 
-### 1. AsyncWebCrawler Enhancements
+#### 1. AsyncWebCrawler Enhancements
 - **Flexible Initialization**: Now accepts arbitrary keyword arguments, passed directly to the crawler strategy.
 - Allows for more customized setups.
 
-### 2. Image Processing Optimization
-- Enhanced image handling in WebScrappingStrategy.
+#### 2. Image Processing Optimization
+- Enhanced image handling in WebScrapingStrategy.
 - Added filtering for small, invisible, or irrelevant images.
 - Improved image scoring system for better content relevance.
 - Implemented JavaScript-based image dimension updating for more accurate representation.
 
-### 3. Database Schema Auto-updates
+#### 3. Database Schema Auto-updates
 - Automatic database schema updates ensure compatibility with the latest version.
 
-### 4. Enhanced Error Handling and Logging
+#### 4. Enhanced Error Handling and Logging
 - Improved error messages and logging for easier debugging.
 
-### 5. Content Extraction Refinements
+#### 5. Content Extraction Refinements
 - Refined HTML sanitization process.
 - Improved handling of base64 encoded images.
 - Enhanced Markdown conversion process.
 - Optimized content extraction algorithms.
 
-### 6. Utility Function Enhancements
+#### 6. Utility Function Enhancements
 - `perform_completion_with_backoff` function now supports additional arguments for more customized API calls to LLM providers.
 
-## Bug Fixes
+### Bug Fixes
 - Fixed an issue where image tags were being prematurely removed during content extraction.
 
-## Examples and Documentation
+### Examples and Documentation
 - Updated `quickstart_async.py` with examples of:
   - Using custom headers in LLM extraction.
   - Different LLM provider usage (OpenAI, Hugging Face, Ollama).
   - Custom browser type usage.
 
-## Developer Notes
+### Developer Notes
 - Refactored code for better maintainability, flexibility, and performance.
 - Enhanced type hinting throughout the codebase for improved development experience.
 - Expanded error handling for more robust operation.

CONTRIBUTORS.md

@@ -10,11 +10,19 @@ We would like to thank the following people for their contributions to Crawl4AI:
 
 ## Community Contributors
 
+- [aadityakanjolia4](https://github.com/aadityakanjolia4) - Fix for `CustomHTML2Text` is not defined.
 - [FractalMind](https://github.com/FractalMind) - Created the first official Docker Hub image and fixed Dockerfile errors
 - [ketonkss4](https://github.com/ketonkss4) - Identified Selenium's new capabilities, helping reduce dependencies
 - [jonymusky](https://github.com/jonymusky) - Javascript execution documentation, and wait_for
 - [datehoer](https://github.com/datehoer) - Add browser prxy support
 
+## Pull Requests
+
+- [nelzomal](https://github.com/nelzomal) - Enhance development installation instructions [#286](https://github.com/unclecode/crawl4ai/pull/286)
+- [HamzaFarhan](https://github.com/HamzaFarhan) - Handled the cases where markdown_with_citations, references_markdown, and filtered_html might not be defined [#293](https://github.com/unclecode/crawl4ai/pull/293)
+- [NanmiCoder](https://github.com/NanmiCoder) - fix: crawler strategy exception handling and fixes [#271](https://github.com/unclecode/crawl4ai/pull/271)
+
+
 ## Other Contributors
 
 - [Gokhan](https://github.com/gkhngyk) 

Dockerfile

@@ -0,0 +1,129 @@
+# syntax=docker/dockerfile:1.4
+
+# Build arguments
+ARG PYTHON_VERSION=3.10
+
+# Base stage with system dependencies
+FROM python:${PYTHON_VERSION}-slim as base
+
+# Declare ARG variables again within the build stage
+ARG INSTALL_TYPE=all
+ARG ENABLE_GPU=false
+
+# Platform-specific labels
+LABEL maintainer="unclecode"
+LABEL description="🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & scraper"
+LABEL version="1.0"
+
+# Environment setup
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1 \
+    PIP_DEFAULT_TIMEOUT=100 \
+    DEBIAN_FRONTEND=noninteractive
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    curl \
+    wget \
+    gnupg \
+    git \
+    cmake \
+    pkg-config \
+    python3-dev \
+    libjpeg-dev \
+    libpng-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Playwright system dependencies for Linux
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libglib2.0-0 \
+    libnss3 \
+    libnspr4 \
+    libatk1.0-0 \
+    libatk-bridge2.0-0 \
+    libcups2 \
+    libdrm2 \
+    libdbus-1-3 \
+    libxcb1 \
+    libxkbcommon0 \
+    libx11-6 \
+    libxcomposite1 \
+    libxdamage1 \
+    libxext6 \
+    libxfixes3 \
+    libxrandr2 \
+    libgbm1 \
+    libpango-1.0-0 \
+    libcairo2 \
+    libasound2 \
+    libatspi2.0-0 \
+    && rm -rf /var/lib/apt/lists/*
+
+# GPU support if enabled and architecture is supported
+RUN if [ "$ENABLE_GPU" = "true" ] && [ "$(dpkg --print-architecture)" != "arm64" ] ; then \
+        apt-get update && apt-get install -y --no-install-recommends \
+        nvidia-cuda-toolkit \
+        && rm -rf /var/lib/apt/lists/* ; \
+    else \
+        echo "Skipping NVIDIA CUDA Toolkit installation (unsupported architecture or GPU disabled)"; \
+    fi
+
+# Create and set working directory
+WORKDIR /app
+
+# Copy the entire project
+COPY . .
+
+# Install base requirements
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Install required library for FastAPI
+RUN pip install fastapi uvicorn psutil
+
+# Install ML dependencies first for better layer caching
+RUN if [ "$INSTALL_TYPE" = "all" ] ; then \
+        pip install --no-cache-dir \
+            torch \
+            torchvision \
+            torchaudio \
+            scikit-learn \
+            nltk \
+            transformers \
+            tokenizers && \
+        python -m nltk.downloader punkt stopwords ; \
+    fi
+
+# Install the package
+RUN if [ "$INSTALL_TYPE" = "all" ] ; then \
+        pip install ".[all]" && \
+        python -m crawl4ai.model_loader ; \
+    elif [ "$INSTALL_TYPE" = "torch" ] ; then \
+        pip install ".[torch]" ; \
+    elif [ "$INSTALL_TYPE" = "transformer" ] ; then \
+        pip install ".[transformer]" && \
+        python -m crawl4ai.model_loader ; \
+    else \
+        pip install "." ; \
+    fi
+
+    # Install MkDocs and required plugins
+RUN pip install --no-cache-dir \
+    mkdocs \
+    mkdocs-material \
+    mkdocs-terminal \
+    pymdown-extensions
+
+# Build MkDocs documentation
+RUN mkdocs build
+
+# Install Playwright and browsers
+RUN playwright install
+
+# Expose port
+EXPOSE 8000 11235 9222 8080
+
+# Start the FastAPI server
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "11235"]

MISSION.md

@@ -0,0 +1,46 @@
+# Mission
+
+![Mission Diagram](./docs/assets/pitch-dark.svg)
+
+### 1. The Data Capitalization Opportunity
+
+We live in an unprecedented era of digital wealth creation. Every day, individuals and enterprises generate massive amounts of valuable digital footprints across various platforms, social media channels, messenger apps, and cloud services. While people can interact with their data within these platforms, there's an immense untapped opportunity to transform this data into true capital assets. Just as physical property became a foundational element of wealth creation, personal and enterprise data has the potential to become a new form of capital on balance sheets.
+
+For individuals, this represents an opportunity to transform their digital activities into valuable assets. For enterprises, their internal communications, team discussions, and collaborative documents contain rich insights that could be structured and valued as intellectual capital. This wealth of information represents an unprecedented opportunity for value creation in the digital age.
+
+### 2. The Potential of Authentic Data
+
+While synthetic data has played a crucial role in AI development, there's an enormous untapped potential in the authentic data generated by individuals and organizations. Every message, document, and interaction contains unique insights and patterns that could enhance AI development. The challenge isn't a lack of data - it's that most authentic human-generated data remains inaccessible for productive use.
+
+By enabling willing participation in data sharing, we can unlock this vast reservoir of authentic human knowledge. This represents an opportunity to enhance AI development with diverse, real-world data that reflects the full spectrum of human experience and knowledge.
+
+## Our Pathway to Data Democracy
+
+### 1. Open-Source Foundation
+
+Our first step is creating an open-source data extraction engine that empowers developers and innovators to build tools for data structuring and organization. This foundation ensures transparency, security, and community-driven development. By making these tools openly available, we enable the technical infrastructure needed for true data ownership and capitalization.
+
+### 2. Data Capitalization Platform
+
+Building on this open-source foundation, we're developing a platform that helps individuals and enterprises transform their digital footprints into structured, valuable assets. This platform will provide the tools and frameworks needed to organize, understand, and value personal and organizational data as true capital assets.
+
+### 3. Creating a Data Marketplace
+
+The final piece is establishing a marketplace where individuals and organizations can willingly share their data assets. This creates opportunities for:
+- Individuals to earn equity, revenue, or other forms of value from their data
+- Enterprises to access diverse, high-quality data for AI development
+- Researchers to work with authentic human-generated data
+- Startups to build innovative solutions using real-world data
+
+## Economic Vision: A Shared Data Economy
+
+We envision a future where data becomes a fundamental asset class in a thriving shared economy. This transformation will democratize AI development by enabling willing participation in data sharing, ensuring that the benefits of AI advancement flow back to data creators. Just as property rights revolutionized economic systems, establishing data as a capital asset will create new opportunities for wealth creation and economic participation.
+
+This shared data economy will:
+- Enable individuals to capitalize on their digital footprints
+- Create new revenue streams for data creators
+- Provide AI developers with access to diverse, authentic data
+- Foster innovation through broader access to real-world data
+- Ensure more equitable distribution of AI's economic benefits
+
+Our vision is to facilitate this transformation from the ground up - starting with open-source tools, progressing to data capitalization platforms, and ultimately creating a thriving marketplace where data becomes a true asset class in a shared economy. This approach ensures that the future of AI is built on a foundation of authentic human knowledge, with benefits flowing back to the individuals and organizations who create and share their valuable data.

README.md

@@ -1,25 +1,127 @@
-# Crawl4AI (Async Version) 🕷️🤖
+# 🔥🕷️ Crawl4AI: Crawl Smarter, Faster, Freely. For AI.
+
+<a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 
 [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
+![PyPI - Downloads](https://img.shields.io/pypi/dm/Crawl4AI)
 [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
 [![GitHub Issues](https://img.shields.io/github/issues/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/issues)
 [![GitHub Pull Requests](https://img.shields.io/github/issues-pr/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/pulls)
 [![License](https://img.shields.io/github/license/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE)
 
-Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
+Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.  
+
+[✨ Check out what's new in the latest update!](#recent-updates)  
+
+## 🧐 Why Crawl4AI?
+
+1. **Built for LLMs**: Creates smart, concise Markdown optimized for RAG and fine-tuning applications.  
+2. **Lightning Fast**: Delivers results 6x faster with real-time, cost-efficient performance.  
+3. **Flexible Browser Control**: Offers session management, proxies, and custom hooks for seamless data access.  
+4. **Heuristic Intelligence**: Uses advanced algorithms for efficient extraction, reducing reliance on costly models.  
+5. **Open Source & Deployable**: Fully open-source with no API keys—ready for Docker and cloud integration.  
+6. **Thriving Community**: Actively maintained by a vibrant community and the #1 trending GitHub repository.
+
+## 🚀 Quick Start 
+
+1. Install Crawl4AI:
+```bash
+pip install crawl4ai
+```
+
+2. Run a simple web crawl:
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CacheMode
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.nbcnews.com/business")
+        # Soone will be change to result.markdown
+        print(result.markdown_v2.raw_markdown) 
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## ✨ Features 
+
+<details>
+<summary>📝 <strong>Markdown Generation</strong></summary>
+
+- 🧹 **Clean Markdown**: Generates clean, structured Markdown with accurate formatting.
+- 🎯 **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
+- 🔗 **Citations and References**: Converts page links into a numbered reference list with clean citations.
+- 🛠️ **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs.
+- 📚 **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content. 
+</details>
+
+<details>
+<summary>📊 <strong>Structured Data Extraction</strong></summary>
+
+- 🤖 **LLM-Driven Extraction**: Supports all LLMs (open-source and proprietary) for structured data extraction.
+- 🧱 **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
+- 🌌 **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction.
+- 🔎 **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors.
+- 🔧 **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns.
+
+</details>
+
+<details>
+<summary>🌐 <strong>Browser Integration</strong></summary>
+
+- 🖥️ **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection.
+- 🔄 **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
+- 🔒 **Session Management**: Preserve browser states and reuse them for multi-step crawling.
+- 🧩 **Proxy Support**: Seamlessly connect to proxies with authentication for secure access.
+- ⚙️ **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups.
+- 🌍 **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit.
+
+</details>
+
+<details>
+<summary>🔎 <strong>Crawling & Scraping</strong></summary>
+
+- 🖼️ **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`.
+- 🚀 **Dynamic Crawling**: Execute JS and wait for async or sync for dynamic content extraction.
+- 📸 **Screenshots**: Capture page screenshots during crawling for debugging or analysis.
+- 📂 **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file://`).
+- 🔗 **Comprehensive Link Extraction**: Extracts internal, external links, and embedded iframe content.
+- 🛠️ **Customizable Hooks**: Define hooks at every step to customize crawling behavior.
+- 💾 **Caching**: Cache data for improved speed and to avoid redundant fetches.
+- 📄 **Metadata Extraction**: Retrieve structured metadata from web pages.
+- 📡 **IFrame Content Extraction**: Seamless extraction from embedded iframe content.
+
+</details>
+
+<details>
+<summary>🚀 <strong>Deployment</strong></summary>
+
+- 🐳 **Dockerized Setup**: Optimized Docker image with API server for easy deployment.
+- 🔄 **API Gateway**: One-click deployment with secure token authentication for API-based workflows.
+- 🌐 **Scalable Architecture**: Designed for mass-scale production and optimized server performance.
+- ⚙️ **DigitalOcean Deployment**: Ready-to-deploy configurations for DigitalOcean and similar platforms.
+
+</details>
+
+<details>
+<summary>🎯 <strong>Additional Features</strong></summary>
+
+- 🕶️ **Stealth Mode**: Avoid bot detection by mimicking real users.
+- 🏷️ **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata.
+- 🔗 **Link Analysis**: Extract and analyze all links for detailed data exploration.
+- 🛡️ **Error Handling**: Robust error management for seamless execution.
+- 🔐 **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests.
+- 📖 **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage.
+- 🙌 **Community Recognition**: Acknowledges contributors and pull requests for transparency.
+
+</details>
 
-## New in 0.3.72 ✨
 
-- 📄 Fit markdown generation for extracting main article content.
-- 🪄 Magic mode for comprehensive anti-bot detection bypass.
-- 🌐 Enhanced multi-browser support with seamless switching (Chromium, Firefox, WebKit)
-- 📚 New chunking strategies(Sliding window, Overlapping window, Flexible size control)
-- 💾 Improved caching system for better performance
-- ⚡ Optimized batch processing with automatic rate limiting
 
 ## Try it Now!
 
-✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1REChY6fXQf-EaVYLv0eHEWvzlYxGm0pd?usp=sharing)
+✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)
 
 ✨ Visit our [Documentation Website](https://crawl4ai.com/mkdocs/)
 
@@ -55,11 +157,12 @@ Crawl4AI simplifies asynchronous web crawling and data extraction, making it acc
 
 Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
 
-### Using pip 🐍
+<details>
+<summary>🐍 <strong>Using pip</strong></summary>
 
 Choose the installation option that best fits your needs:
 
-#### Basic Installation
+### Basic Installation
 
 For basic web crawling and scraping tasks:
 
@@ -69,180 +172,272 @@ pip install crawl4ai
 
 By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.
 
-👉 Note: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
+👉 **Note**: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
 
 1. Through the command line:
+
    ```bash
    playwright install
    ```
 
 2. If the above doesn't work, try this more specific command:
+
    ```bash
    python -m playwright install chromium
    ```
 
 This second method has proven to be more reliable in some cases.
 
-#### Installation with Synchronous Version
+---
 
-If you need the synchronous version using Selenium:
+### Installation with Synchronous Version
+
+The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:
 
 ```bash
 pip install crawl4ai[sync]
 ```
 
-#### Development Installation
+---
+
+### Development Installation
 
 For contributors who plan to modify the source code:
 
 ```bash
 git clone https://github.com/unclecode/crawl4ai.git
 cd crawl4ai
-pip install -e .
+pip install -e .                    # Basic installation in editable mode
 ```
 
-### Using Docker 🐳
+Install optional features:
 
-We're in the process of creating Docker images and pushing them to Docker Hub. This will provide an easy way to run Crawl4AI in a containerized environment. Stay tuned for updates!
+```bash
+pip install -e ".[torch]"           # With PyTorch features
+pip install -e ".[transformer]"     # With Transformer features
+pip install -e ".[cosine]"          # With cosine similarity features
+pip install -e ".[sync]"            # With synchronous crawling (Selenium)
+pip install -e ".[all]"             # Install all optional features
+```
 
-For more detailed installation instructions and options, please refer to our [Installation Guide](https://crawl4ai.com/mkdocs/installation).
+</details>
 
-## Quick Start 🚀
+<details>
+<summary>🚀 <strong>One-Click Deployment</strong></summary>
+
+Deploy your own instance of Crawl4AI with one click:
+
+[![DigitalOcean Referral Badge](https://web-platforms.sfo2.cdn.digitaloceanspaces.com/WWW/Badge%203.svg)](https://www.digitalocean.com/?repo=https://github.com/unclecode/crawl4ai/tree/0.3.74&refcode=a0780f1bdb3d&utm_campaign=Referral_Invite&utm_medium=Referral_Program&utm_source=badge)
+
+> 💡 **Recommended specs**: 4GB RAM minimum. Select "professional-xs" or higher when deploying for stable operation.
+
+The deploy will:
+- Set up a Docker container with Crawl4AI
+- Configure Playwright and all dependencies
+- Start the FastAPI server on port `11235`
+- Set up health checks and auto-deployment
+
+</details>
+
+<details>
+<summary>🐳 <strong>Using Docker</strong></summary>
+
+Crawl4AI is available as Docker images for easy deployment. You can either pull directly from Docker Hub (recommended) or build from the repository.
+
+---
+
+### Option 1: Docker Hub (Recommended)
+
+```bash
+# Pull and run from Docker Hub (choose one):
+docker pull unclecode/crawl4ai:basic    # Basic crawling features
+docker pull unclecode/crawl4ai:all      # Full installation (ML, LLM support)
+docker pull unclecode/crawl4ai:gpu      # GPU-enabled version
+
+# Run the container
+docker run -p 11235:11235 unclecode/crawl4ai:basic  # Replace 'basic' with your chosen version
+
+# In case you want to set platform to arm64
+docker run --platform linux/arm64 -p 11235:11235 unclecode/crawl4ai:basic
+
+# In case to allocate more shared memory for the container
+docker run --shm-size=2gb -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+---
+
+### Option 2: Build from Repository
+
+```bash
+# Clone the repository
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+
+# Build the image
+docker build -t crawl4ai:local \
+  --build-arg INSTALL_TYPE=basic \  # Options: basic, all
+  .
+
+# In case you want to set platform to arm64
+docker build -t crawl4ai:local \
+  --build-arg INSTALL_TYPE=basic \  # Options: basic, all
+  --platform linux/arm64 \
+  .
+
+# Run your local build
+docker run -p 11235:11235 crawl4ai:local
+```
+
+---
+
+### Quick Test
+
+Run a quick test (works for both Docker options):
+
+```python
+import requests
+
+# Submit a crawl job
+response = requests.post(
+    "http://localhost:11235/crawl",
+    json={"urls": "https://example.com", "priority": 10}
+)
+task_id = response.json()["task_id"]
+
+# Get results
+result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+For advanced configuration, environment variables, and usage examples, see our [Docker Deployment Guide](https://crawl4ai.com/mkdocs/basic/docker-deployment/).
+
+</details>
+
+
+## 🔬 Advanced Usage Examples 🔬
+
+You can check the project structure in the directory [https://github.com/unclecode/crawl4ai/docs/examples](docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.
+
+<details>
+<summary>📝 <strong>Heuristic Markdown Generation with Clean and Fit Markdown</strong></summary>
 
 ```python
 import asyncio
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
+from crawl4ai.content_filter_strategy import BM25ContentFilter
+from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
 
 async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        result = await crawler.arun(url="https://www.nbcnews.com/business")
-        print(result.markdown)
-
-if __name__ == "__main__":
-    asyncio.run(main())
-```
-
-## Advanced Usage 🔬
-
-### Executing JavaScript and Using CSS Selectors
-
-```python
-import asyncio
-from crawl4ai import AsyncWebCrawler
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        js_code = ["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"]
+    async with AsyncWebCrawler(
+        headless=True,  
+        verbose=True,
+    ) as crawler:
         result = await crawler.arun(
-            url="https://www.nbcnews.com/business",
-            js_code=js_code,
-            css_selector=".wide-tease-item__description",
-            bypass_cache=True
+            url="https://docs.micronaut.io/4.7.6/guide/",
+            cache_mode=CacheMode.ENABLED,
+            markdown_generator=DefaultMarkdownGenerator(
+                content_filter=BM25ContentFilter(user_query=None, bm25_threshold=1.0)
+            ),
         )
-        print(result.extracted_content)
+        print(len(result.markdown))
+        print(len(result.fit_markdown))
+        print(len(result.markdown_v2.fit_markdown))
 
 if __name__ == "__main__":
     asyncio.run(main())
 ```
 
-### Using a Proxy
+</details>
+
+<details>
+<summary>🖥️ <strong>Executing JavaScript & Extract Structured Data without LLMs</strong></summary>
 
 ```python
 import asyncio
-from crawl4ai import AsyncWebCrawler
-
-async def main():
-    async with AsyncWebCrawler(verbose=True, proxy="http://127.0.0.1:7890") as crawler:
-        result = await crawler.arun(
-            url="https://www.nbcnews.com/business",
-            bypass_cache=True
-        )
-        print(result.markdown)
-
-if __name__ == "__main__":
-    asyncio.run(main())
-```
-
-### Extracting Structured Data without LLM
-
-The `JsonCssExtractionStrategy` allows for precise extraction of structured data from web pages using CSS selectors.
-
-```python
-import asyncio
-import json
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
 from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+import json
 
-async def extract_news_teasers():
+async def main():
     schema = {
-        "name": "News Teaser Extractor",
-        "baseSelector": ".wide-tease-item__wrapper",
-        "fields": [
-            {
-                "name": "category",
-                "selector": ".unibrow span[data-testid='unibrow-text']",
-                "type": "text",
-            },
-            {
-                "name": "headline",
-                "selector": ".wide-tease-item__headline",
-                "type": "text",
-            },
-            {
-                "name": "summary",
-                "selector": ".wide-tease-item__description",
-                "type": "text",
-            },
-            {
-                "name": "time",
-                "selector": "[data-testid='wide-tease-date']",
-                "type": "text",
-            },
-            {
-                "name": "image",
-                "type": "nested",
-                "selector": "picture.teasePicture img",
-                "fields": [
-                    {"name": "src", "type": "attribute", "attribute": "src"},
-                    {"name": "alt", "type": "attribute", "attribute": "alt"},
-                ],
-            },
-            {
-                "name": "link",
-                "selector": "a[href]",
-                "type": "attribute",
-                "attribute": "href",
-            },
-        ],
-    }
+    "name": "KidoCode Courses",
+    "baseSelector": "section.charge-methodology .w-tab-content > div",
+    "fields": [
+        {
+            "name": "section_title",
+            "selector": "h3.heading-50",
+            "type": "text",
+        },
+        {
+            "name": "section_description",
+            "selector": ".charge-content",
+            "type": "text",
+        },
+        {
+            "name": "course_name",
+            "selector": ".text-block-93",
+            "type": "text",
+        },
+        {
+            "name": "course_description",
+            "selector": ".course-content-text",
+            "type": "text",
+        },
+        {
+            "name": "course_icon",
+            "selector": ".image-92",
+            "type": "attribute",
+            "attribute": "src"
+        }
+    ]
+}
 
     extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
 
-    async with AsyncWebCrawler(verbose=True) as crawler:
+    async with AsyncWebCrawler(
+        headless=False,
+        verbose=True
+    ) as crawler:
+        
+        # Create the JavaScript that handles clicking multiple times
+        js_click_tabs = """
+        (async () => {
+            const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");
+            
+            for(let tab of tabs) {
+                // scroll to the tab
+                tab.scrollIntoView();
+                tab.click();
+                // Wait for content to load and animations to complete
+                await new Promise(r => setTimeout(r, 500));
+            }
+        })();
+        """     
+
         result = await crawler.arun(
-            url="https://www.nbcnews.com/business",
-            extraction_strategy=extraction_strategy,
-            bypass_cache=True,
+            url="https://www.kidocode.com/degrees/technology",
+            extraction_strategy=JsonCssExtractionStrategy(schema, verbose=True),
+            js_code=[js_click_tabs],
+            cache_mode=CacheMode.BYPASS
         )
 
-        assert result.success, "Failed to crawl the page"
+        companies = json.loads(result.extracted_content)
+        print(f"Successfully extracted {len(companies)} companies")
+        print(json.dumps(companies[0], indent=2))
 
-        news_teasers = json.loads(result.extracted_content)
-        print(f"Successfully extracted {len(news_teasers)} news teasers")
-        print(json.dumps(news_teasers[0], indent=2))
 
 if __name__ == "__main__":
-    asyncio.run(extract_news_teasers())
+    asyncio.run(main())
 ```
 
-For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/advanced_jsoncss_extraction.md) section in the documentation.
+</details>
 
-### Extracting Structured Data with OpenAI
+<details>
+<summary>📚 <strong>Extracting Structured Data with LLMs</strong></summary>
 
 ```python
 import os
 import asyncio
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
 from crawl4ai.extraction_strategy import LLMExtractionStrategy
 from pydantic import BaseModel, Field
 
@@ -257,6 +452,8 @@ async def main():
             url='https://openai.com/api/pricing/',
             word_count_threshold=1,
             extraction_strategy=LLMExtractionStrategy(
+                # Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
+                # provider="ollama/qwen2", api_token="no-token", 
                 provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY'), 
                 schema=OpenAIModelFee.schema(),
                 extraction_type="schema",
@@ -264,7 +461,7 @@ async def main():
                 Do not miss any models in the entire content. One extracted model JSON format should look like this: 
                 {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
             ),            
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
         )
         print(result.extracted_content)
 
@@ -272,117 +469,98 @@ if __name__ == "__main__":
     asyncio.run(main())
 ```
 
-### Session Management and Dynamic Content Crawling
+</details>
 
-Crawl4AI excels at handling complex scenarios, such as crawling multiple pages with dynamic content loaded via JavaScript. Here's an example of crawling GitHub commits across multiple pages:
+<details>
+<summary>🤖 <strong>Using You own Browswer with Custome User Profile</strong></summary>
 
 ```python
-import asyncio
-import re
-from bs4 import BeautifulSoup
+import os, sys
+from pathlib import Path
+import asyncio, time
 from crawl4ai import AsyncWebCrawler
 
-async def crawl_typescript_commits():
-    first_commit = ""
-    async def on_execution_started(page):
-        nonlocal first_commit 
-        try:
-            while True:
-                await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4')
-                commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4')
-                commit = await commit.evaluate('(element) => element.textContent')
-                commit = re.sub(r'\s+', '', commit)
-                if commit and commit != first_commit:
-                    first_commit = commit
-                    break
-                await asyncio.sleep(0.5)
-        except Exception as e:
-            print(f"Warning: New content didn't appear after JavaScript execution: {e}")
+async def test_news_crawl():
+    # Create a persistent user data directory
+    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
+    os.makedirs(user_data_dir, exist_ok=True)
 
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started)
-
-        url = "https://github.com/microsoft/TypeScript/commits/main"
-        session_id = "typescript_commits_session"
-        all_commits = []
-
-        js_next_page = """
-        const button = document.querySelector('a[data-testid="pagination-next-button"]');
-        if (button) button.click();
-        """
-
-        for page in range(3):  # Crawl 3 pages
-            result = await crawler.arun(
-                url=url,
-                session_id=session_id,
-                css_selector="li.Box-sc-g0xbh4-0",
-                js=js_next_page if page > 0 else None,
-                bypass_cache=True,
-                js_only=page > 0
-            )
-
-            assert result.success, f"Failed to crawl page {page + 1}"
-
-            soup = BeautifulSoup(result.cleaned_html, 'html.parser')
-            commits = soup.select("li")
-            all_commits.extend(commits)
-
-            print(f"Page {page + 1}: Found {len(commits)} commits")
-
-        await crawler.crawler_strategy.kill_session(session_id)
-        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
-
-if __name__ == "__main__":
-    asyncio.run(crawl_typescript_commits())
+    async with AsyncWebCrawler(
+        verbose=True,
+        headless=True,
+        user_data_dir=user_data_dir,
+        use_persistent_context=True,
+        headers={
+            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
+            "Accept-Language": "en-US,en;q=0.5",
+            "Accept-Encoding": "gzip, deflate, br",
+            "DNT": "1",
+            "Connection": "keep-alive",
+            "Upgrade-Insecure-Requests": "1",
+            "Sec-Fetch-Dest": "document",
+            "Sec-Fetch-Mode": "navigate",
+            "Sec-Fetch-Site": "none",
+            "Sec-Fetch-User": "?1",
+            "Cache-Control": "max-age=0",
+        }
+    ) as crawler:
+        url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
+        
+        result = await crawler.arun(
+            url,
+            cache_mode=CacheMode.BYPASS,
+            magic=True,
+        )
+        
+        print(f"Successfully crawled {url}")
+        print(f"Content length: {len(result.markdown)}")
 ```
 
-This example demonstrates Crawl4AI's ability to handle complex scenarios where content is loaded asynchronously. It crawls multiple pages of GitHub commits, executing JavaScript to load new content and using custom hooks to ensure data is loaded before proceeding.
-
-For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/session_based_crawling.md) section in the documentation.
+</details>
 
 
-## Speed Comparison 🚀
+## ✨ Recent Updates   
 
-Crawl4AI is designed with speed as a primary focus. Our goal is to provide the fastest possible response with high-quality data extraction, minimizing abstractions between the data and the user.
+- 🚀 **Improved ManagedBrowser Configuration**: Dynamic host and port support for more flexible browser management.  
+- 📝 **Enhanced Markdown Generation**: New generator class for better formatting and customization.  
+- ⚡ **Fast HTML Formatting**: Significantly optimized HTML formatting in the web crawler.  
+- 🛠️ **Utility & Sanitization Upgrades**: Improved sanitization and expanded utility functions for streamlined workflows.  
+- 👥 **Acknowledgments**: Added contributor details and pull request acknowledgments for better transparency.  
 
-We've conducted a speed comparison between Crawl4AI and Firecrawl, a paid service. The results demonstrate Crawl4AI's superior performance:
 
-```
-Firecrawl:
-Time taken: 7.02 seconds
-Content length: 42074 characters
-Images found: 49
-
-Crawl4AI (simple crawl):
-Time taken: 1.60 seconds
-Content length: 18238 characters
-Images found: 49
-
-Crawl4AI (with JavaScript execution):
-Time taken: 4.64 seconds
-Content length: 40869 characters
-Images found: 89
-```
-
-As you can see, Crawl4AI outperforms Firecrawl significantly:
-- Simple crawl: Crawl4AI is over 4 times faster than Firecrawl.
-- With JavaScript execution: Even when executing JavaScript to load more content (doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.
-
-You can find the full comparison code in our repository at `docs/examples/crawl4ai_vs_firecrawl.py`.
-
-## Documentation 📚
+## 📖 Documentation & Roadmap 
 
 For detailed documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://crawl4ai.com/mkdocs/).
 
-## Contributing 🤝
+Moreover to check our development plans and upcoming features, check out our [Roadmap](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md).
+
+<details>
+<summary>📈 <strong>Development TODOs</strong></summary>
+
+- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
+- [ ] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
+- [ ] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
+- [ ] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
+- [ ] 4. Automated Schema Generator: Convert natural language to extraction schemas
+- [ ] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
+- [ ] 6. Web Embedding Index: Semantic search infrastructure for crawled content
+- [ ] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
+- [ ] 8. Performance Monitor: Real-time insights into crawler operations
+- [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers
+- [ ] 10. Sponsorship Program: Structured support system with tiered benefits
+- [ ] 11. Educational Content: "How to Crawl" video series and interactive tutorials
+
+</details>
+
+## 🤝 Contributing 
 
 We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md) for more information.
 
-## License 📄
+## 📄 License 
 
 Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
 
-## Contact 📧
+## 📧 Contact 
 
 For questions, suggestions, or feedback, feel free to reach out:
 
@@ -392,6 +570,34 @@ For questions, suggestions, or feedback, feel free to reach out:
 
 Happy Crawling! 🕸️🚀
 
+## 🗾 Mission
+
+Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.  
+
+We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
+
+<details>
+<summary>🔑 <strong>Key Opportunities</strong></summary>
+ 
+- **Data Capitalization**: Transform digital footprints into measurable, valuable assets.  
+- **Authentic AI Data**: Provide AI systems with real human insights.  
+- **Shared Economy**: Create a fair data marketplace that benefits data creators.  
+
+</details>
+
+<details>
+<summary>🚀 <strong>Development Pathway</strong></summary>
+
+1. **Open-Source Tools**: Community-driven platforms for transparent data extraction.  
+2. **Digital Asset Structuring**: Tools to organize and value digital knowledge.  
+3. **Ethical Data Marketplace**: A secure, fair platform for exchanging structured data.  
+
+For more details, see our [full mission statement](./MISSION.md).
+</details>
+
+
+
+
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)
+[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)

ROADMAP.md

@@ -0,0 +1,503 @@
+# Crawl4AI Strategic Roadmap
+
+```mermaid
+%%{init: {'themeVariables': { 'fontSize': '14px'}}}%%
+graph TD
+    subgraph A1[Advanced Crawling Systems 🔧]
+        A["`
+        • Graph Crawler ✓
+        • Question-Based Crawler
+        • Knowledge-Optimal Crawler
+        • Agentic Crawler
+        `"]
+    end
+
+    subgraph A2[Specialized Features 🛠️]
+        B["`
+        • Automated Schema Generator
+        • Domain-Specific Scrapers
+        • 
+        • 
+        `"]
+    end
+
+    subgraph A3[Development Tools 🔨]
+        C["`
+        • Interactive Playground
+        • Performance Monitor
+        • Cloud Integration
+        • 
+        `"]
+    end
+
+    subgraph A4[Community & Growth 🌱]
+        D["`
+        • Sponsorship Program
+        • Educational Content
+        • 
+        • 
+        `"]
+    end
+
+    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px
+    classDef section fill:#f0f0f0,stroke:#333,stroke-width:4px,rx:10
+    class A1,A2,A3,A4 section
+
+    %% Layout hints
+    A1 --> A2[" "]
+    A3 --> A4[" "]
+    linkStyle 0,1 stroke:none
+```
+
+Crawl4AI is evolving to provide more intelligent, efficient, and versatile web crawling capabilities. This roadmap outlines the key developments and features planned for the project, organized into strategic sections that build upon our current foundation.
+
+## 1. Advanced Crawling Systems 🔧
+
+This section introduces three powerful crawling systems that extend Crawl4AI's capabilities from basic web crawling to intelligent, purpose-driven data extraction.
+
+### 1.1 Question-Based Crawler
+The Question-Based Crawler enhances our core engine by enabling automatic discovery and extraction of relevant web content based on natural language questions.
+
+Key Features:
+- SerpiAPI integration for intelligent web search
+- Relevancy scoring for search results
+- Automatic URL discovery and prioritization
+- Cross-source validation
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.discovery import QuestionBasedDiscovery
+
+async with AsyncWebCrawler() as crawler:
+    discovery = QuestionBasedDiscovery(crawler)
+    results = await discovery.arun(
+        question="What are the system requirements for major cloud providers' GPU instances?",
+        max_urls=5,
+        relevance_threshold=0.7
+    )
+    
+    for result in results:
+        print(f"Source: {result.url} (Relevance: {result.relevance_score})")
+        print(f"Content: {result.markdown}\n")
+```
+
+### 1.2 Knowledge-Optimal Crawler
+An intelligent crawling system that solves the optimization problem of minimizing data extraction while maximizing knowledge acquisition for specific objectives.
+
+Key Features:
+- Smart content prioritization
+- Minimal data extraction for maximum knowledge
+- Probabilistic relevance assessment
+- Objective-driven crawling paths
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.optimization import KnowledgeOptimizer
+
+async with AsyncWebCrawler() as crawler:
+    optimizer = KnowledgeOptimizer(
+        objective="Understand GPU instance pricing and limitations across cloud providers",
+        required_knowledge=[
+            "pricing structure",
+            "GPU specifications",
+            "usage limits",
+            "availability zones"
+        ],
+        confidence_threshold=0.85
+    )
+    
+    result = await crawler.arun(
+        urls=[
+            "https://aws.amazon.com/ec2/pricing/",
+            "https://cloud.google.com/gpu",
+            "https://azure.microsoft.com/pricing/"
+        ],
+        optimizer=optimizer,
+        optimization_mode="minimal_extraction"
+    )
+    
+    print(f"Knowledge Coverage: {result.knowledge_coverage}")
+    print(f"Data Efficiency: {result.efficiency_ratio}")
+    print(f"Extracted Content: {result.optimal_content}")
+```
+
+### 1.3 Agentic Crawler
+An autonomous system capable of understanding complex goals and automatically planning and executing multi-step crawling operations.
+
+Key Features:
+- Autonomous goal interpretation
+- Dynamic step planning
+- Interactive navigation capabilities
+- Visual recognition and interaction
+- Automatic error recovery
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.agents import CrawlerAgent
+
+async with AsyncWebCrawler() as crawler:
+    agent = CrawlerAgent(crawler)
+    
+    # Automatic planning and execution
+    result = await agent.arun(
+        goal="Find research papers about quantum computing published in 2023 with more than 50 citations",
+        auto_retry=True
+    )
+    print("Generated Plan:", result.executed_steps)
+    print("Extracted Data:", result.data)
+    
+    # Using custom steps with automatic execution
+    result = await agent.arun(
+        goal="Extract conference deadlines from ML conferences",
+        custom_plan=[
+            "Navigate to conference page",
+            "Find important dates section",
+            "Extract submission deadlines",
+            "Verify dates are for 2024"
+        ]
+    )
+    
+    # Monitoring execution
+    print("Step Completion:", result.step_status)
+    print("Execution Time:", result.execution_time)
+    print("Success Rate:", result.success_rate)
+```
+
+# Section 2: Specialized Features 🛠️
+
+This section introduces specialized tools and features that enhance Crawl4AI's capabilities for specific use cases and data extraction needs.
+
+### 2.1 Automated Schema Generator
+A system that automatically generates JsonCssExtractionStrategy schemas from natural language descriptions, making structured data extraction accessible to all users.
+
+Key Features:
+- Natural language schema generation
+- Automatic pattern detection
+- Predefined schema templates
+- Chrome extension for visual schema building
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.schema import SchemaGenerator
+
+# Generate schema from natural language description
+generator = SchemaGenerator()
+schema = await generator.generate(
+    url="https://news-website.com",
+    description="For each news article on the page, I need the headline, publication date, and main image"
+)
+
+# Use generated schema with crawler
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://news-website.com",
+        extraction_strategy=schema
+    )
+
+# Example of generated schema:
+"""
+{
+    "name": "News Article Extractor",
+    "baseSelector": "article.news-item",
+    "fields": [
+        {
+            "name": "headline",
+            "selector": "h2.article-title",
+            "type": "text"
+        },
+        {
+            "name": "date",
+            "selector": "span.publish-date",
+            "type": "text"
+        },
+        {
+            "name": "image",
+            "selector": "img.article-image",
+            "type": "attribute",
+            "attribute": "src"
+        }
+    ]
+}
+"""
+```
+
+### 2.2 Domain Specific Scrapers
+Specialized extraction strategies optimized for common website types and platforms, providing consistent and reliable data extraction without additional configuration.
+
+Key Features:
+- Pre-configured extractors for popular platforms
+- Academic site specialization (arXiv, NCBI)
+- E-commerce standardization
+- Documentation site handling
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extractors import AcademicExtractor, EcommerceExtractor
+
+async with AsyncWebCrawler() as crawler:
+    # Academic paper extraction
+    papers = await crawler.arun(
+        url="https://arxiv.org/list/cs.AI/recent",
+        extractor="academic",  # Built-in extractor type
+        site_type="arxiv",     # Specific site optimization
+        extract_fields=[
+            "title", 
+            "authors", 
+            "abstract", 
+            "citations"
+        ]
+    )
+    
+    # E-commerce product data
+    products = await crawler.arun(
+        url="https://store.example.com/products",
+        extractor="ecommerce",
+        extract_fields=[
+            "name",
+            "price",
+            "availability",
+            "reviews"
+        ]
+    )
+```
+
+### 2.3 Web Embedding Index
+Creates and maintains a semantic search infrastructure for crawled content, enabling efficient retrieval and querying of web content through vector embeddings.
+
+Key Features:
+- Automatic embedding generation
+- Intelligent content chunking
+- Efficient vector storage and indexing
+- Semantic search capabilities
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.indexing import WebIndex
+
+# Initialize and build index
+index = WebIndex(model="efficient-mini")
+
+async with AsyncWebCrawler() as crawler:
+    # Crawl and index content
+    await index.build(
+        urls=["https://docs.example.com"],
+        crawler=crawler,
+        options={
+            "chunk_method": "semantic",
+            "update_policy": "incremental",
+            "embedding_batch_size": 100
+        }
+    )
+
+    # Search through indexed content
+    results = await index.search(
+        query="How to implement OAuth authentication?",
+        filters={
+            "content_type": "technical",
+            "recency": "6months"
+        },
+        top_k=5
+    )
+
+    # Get similar content
+    similar = await index.find_similar(
+        url="https://docs.example.com/auth/oauth",
+        threshold=0.85
+    )
+```
+
+Each of these specialized features builds upon Crawl4AI's core functionality while providing targeted solutions for specific use cases. They can be used independently or combined for more complex data extraction and processing needs.
+
+# Section 3: Development Tools 🔧
+
+This section covers tools designed to enhance the development experience, monitoring, and deployment of Crawl4AI applications.
+
+### 3.1 Crawl4AI Playground 🎮
+
+The Crawl4AI Playground is an interactive web-based development environment that simplifies web scraping experimentation, development, and deployment. With its intuitive interface and AI-powered assistance, users can quickly prototype, test, and deploy web scraping solutions.
+
+#### Key Features 🌟
+
+##### Visual Strategy Builder
+- Interactive point-and-click interface for building extraction strategies
+- Real-time preview of selected elements
+- Side-by-side comparison of different extraction approaches
+- Visual validation of CSS selectors and XPath queries
+
+##### AI Assistant Integration
+- Strategy recommendations based on target website analysis
+- Parameter optimization suggestions
+- Best practices guidance for specific use cases
+- Automated error detection and resolution
+- Performance optimization tips
+
+##### Real-Time Testing & Validation
+- Live preview of extraction results
+- Side-by-side comparison of multiple strategies
+- Performance metrics visualization
+- Automatic validation of extracted data
+- Error detection and debugging tools
+
+##### Project Management
+- Save and organize multiple scraping projects
+- Version control for configurations
+- Export/import project settings
+- Share configurations with team members
+- Project templates for common use cases
+
+##### Deployment Pipeline
+- One-click deployment to various environments
+- Docker container generation
+- Cloud deployment templates (AWS, GCP, Azure)
+- Scaling configuration management
+- Monitoring setup automation
+
+
+### 3.2 Performance Monitoring System
+A comprehensive monitoring solution providing real-time insights into crawler operations, resource usage, and system health through both CLI and GUI interfaces.
+
+Key Features:
+- Real-time resource tracking
+- Active crawl monitoring
+- Performance statistics
+- Customizable alerting system
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.monitor import CrawlMonitor
+
+# Initialize monitoring
+monitor = CrawlMonitor()
+
+# Start monitoring with CLI interface
+await monitor.start(
+    mode="cli",  # or "gui"
+    refresh_rate="1s",
+    metrics={
+        "resources": ["cpu", "memory", "network"],
+        "crawls": ["active", "queued", "completed"],
+        "performance": ["success_rate", "response_times"]
+    }
+)
+
+# Example CLI output:
+"""
+Crawl4AI Monitor (Live) - Press Q to exit
+────────────────────────────────────────
+System Usage:
+ ├─ CPU: ███████░░░ 70%
+ └─ Memory: ████░░░░░ 2.1GB/8GB
+
+Active Crawls:
+ID    URL                   Status    Progress
+001   docs.example.com     🟢 Active   75%
+002   api.service.com      🟡 Queue    -
+
+Metrics (Last 5min):
+ ├─ Success Rate: 98%
+ ├─ Avg Response: 0.6s
+ └─ Pages/sec: 8.5
+"""
+```
+
+### 3.3 Cloud Integration
+Streamlined deployment tools for setting up Crawl4AI in various cloud environments, with support for scaling and monitoring.
+
+Key Features:
+- One-click deployment solutions
+- Auto-scaling configuration
+- Load balancing setup
+- Cloud-specific optimizations
+- Monitoring integration
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.deploy import CloudDeployer
+
+# Initialize deployer
+deployer = CloudDeployer()
+
+# Deploy crawler service
+deployment = await deployer.deploy(
+    service_name="crawler-cluster",
+    platform="aws",  # or "gcp", "azure"
+    config={
+        "instance_type": "compute-optimized",
+        "auto_scaling": {
+            "min_instances": 2,
+            "max_instances": 10,
+            "scale_based_on": "cpu_usage"
+        },
+        "region": "us-east-1",
+        "monitoring": True
+    }
+)
+
+# Get deployment status and endpoints
+print(f"Service Status: {deployment.status}")
+print(f"API Endpoint: {deployment.endpoint}")
+print(f"Monitor URL: {deployment.monitor_url}")
+```
+
+These development tools work together to provide a comprehensive environment for developing, testing, monitoring, and deploying Crawl4AI applications. The Playground helps users experiment and generate optimal configurations, the Performance Monitor ensures smooth operation, and the Cloud Integration tools simplify deployment and scaling.
+
+# Section 4: Community & Growth 🌱
+
+This section outlines initiatives designed to build and support the Crawl4AI community, provide educational resources, and ensure sustainable project growth.
+
+### 4.1 Sponsorship Program
+A structured program to support ongoing development and maintenance of Crawl4AI while providing valuable benefits to sponsors.
+
+Key Features:
+- Multiple sponsorship tiers
+- Sponsor recognition system
+- Priority support for sponsors
+- Early access to new features
+- Custom feature development opportunities
+
+Program Structure (not yet finalized):
+```
+Sponsorship Tiers:
+
+🥉 Bronze Supporter
+- GitHub Sponsor badge
+- Priority issue response
+- Community Discord role
+
+🥈 Silver Supporter
+- All Bronze benefits
+- Technical support channel
+- Vote on roadmap priorities
+- Early access to beta features
+
+🥇 Gold Supporter
+- All Silver benefits
+- Custom feature requests
+- Direct developer access
+- Private support sessions
+
+💎 Diamond Partner
+- All Gold benefits
+- Custom development
+- On-demand consulting
+- Integration support
+```
+
+### 4.2 "How to Crawl" Video Series
+A comprehensive educational resource teaching users how to effectively use Crawl4AI for various web scraping and data extraction scenarios.
+
+Key Features:
+- Step-by-step tutorials
+- Real-world use cases
+- Best practices
+- Integration guides
+- Advanced feature deep-dives
+
+These community initiatives are designed to:
+- Provide comprehensive learning resources
+- Foster a supportive user community
+- Ensure sustainable project development
+- Share knowledge and best practices
+- Create opportunities for collaboration
+
+The combination of structured support through sponsorship, educational content through video series, and interactive learning through the playground creates a robust ecosystem for both new and experienced users of Crawl4AI.

crawl4ai/__init__.py

@@ -1,13 +1,15 @@
 # __init__.py
 
-from .async_webcrawler import AsyncWebCrawler
-from .models import CrawlResult
+from .async_webcrawler import AsyncWebCrawler, CacheMode
 
-__version__ = "0.3.72"
+from .models import CrawlResult
+from .__version__ import __version__
+# __version__ = "0.3.73"
 
 __all__ = [
     "AsyncWebCrawler",
     "CrawlResult",
+    "CacheMode",
 ]
 
 def is_sync_version_installed():
@@ -26,5 +28,5 @@ if is_sync_version_installed():
         print("Warning: Failed to import WebCrawler even though selenium is installed. This might be due to other missing dependencies.")
 else:
     WebCrawler = None
-    import warnings
-    print("Warning: Synchronous WebCrawler is not available. Install crawl4ai[sync] for synchronous support. However, please note that the synchronous version will be deprecated soon.")
+    # import warnings
+    # print("Warning: Synchronous WebCrawler is not available. Install crawl4ai[sync] for synchronous support. However, please note that the synchronous version will be deprecated soon.")

crawl4ai/__version__.py

@@ -0,0 +1,2 @@
+# crawl4ai/_version.py
+__version__ = "0.3.744"

crawl4ai/async_crawler_strategy copy.py

@@ -1,558 +0,0 @@
-import asyncio
-import base64
-import time
-from abc import ABC, abstractmethod
-from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
-from playwright.async_api import async_playwright, Page, Browser, Error
-from io import BytesIO
-from PIL import Image, ImageDraw, ImageFont
-from pathlib import Path
-from playwright.async_api import ProxySettings
-from pydantic import BaseModel
-import hashlib
-import json
-import uuid
-from playwright_stealth import stealth_async
-
-class AsyncCrawlResponse(BaseModel):
-    html: str
-    response_headers: Dict[str, str]
-    status_code: int
-    screenshot: Optional[str] = None
-    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
-
-    class Config:
-        arbitrary_types_allowed = True
-
-class AsyncCrawlerStrategy(ABC):
-    @abstractmethod
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        pass
-    
-    @abstractmethod
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        pass
-    
-    @abstractmethod
-    async def take_screenshot(self, url: str) -> str:
-        pass
-    
-    @abstractmethod
-    def update_user_agent(self, user_agent: str):
-        pass
-    
-    @abstractmethod
-    def set_hook(self, hook_type: str, hook: Callable):
-        pass
-
-class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
-    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
-        self.use_cached_html = use_cached_html
-        self.user_agent = kwargs.get(
-            "user_agent",
-            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
-            "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
-        )
-        self.proxy = kwargs.get("proxy")
-        self.headless = kwargs.get("headless", True)
-        self.browser_type = kwargs.get("browser_type", "chromium")
-        self.headers = kwargs.get("headers", {})
-        self.sessions = {}
-        self.session_ttl = 1800 
-        self.js_code = js_code
-        self.verbose = kwargs.get("verbose", False)
-        self.playwright = None
-        self.browser = None
-        self.hooks = {
-            'on_browser_created': None,
-            'on_user_agent_updated': None,
-            'on_execution_started': None,
-            'before_goto': None,
-            'after_goto': None,
-            'before_return_html': None,
-            'before_retrieve_html': None
-        }
-
-    async def __aenter__(self):
-        await self.start()
-        return self
-
-    async def __aexit__(self, exc_type, exc_val, exc_tb):
-        await self.close()
-
-    async def start(self):
-        if self.playwright is None:
-            self.playwright = await async_playwright().start()
-        if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-                
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
-            else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
-
-            await self.execute_hook('on_browser_created', self.browser)
-
-    async def close(self):
-        if self.browser:
-            await self.browser.close()
-            self.browser = None
-        if self.playwright:
-            await self.playwright.stop()
-            self.playwright = None
-
-    def __del__(self):
-        if self.browser or self.playwright:
-            asyncio.get_event_loop().run_until_complete(self.close())
-
-    def set_hook(self, hook_type: str, hook: Callable):
-        if hook_type in self.hooks:
-            self.hooks[hook_type] = hook
-        else:
-            raise ValueError(f"Invalid hook type: {hook_type}")
-
-    async def execute_hook(self, hook_type: str, *args):
-        hook = self.hooks.get(hook_type)
-        if hook:
-            if asyncio.iscoroutinefunction(hook):
-                return await hook(*args)
-            else:
-                return hook(*args)
-        return args[0] if args else None
-
-    def update_user_agent(self, user_agent: str):
-        self.user_agent = user_agent
-
-    def set_custom_headers(self, headers: Dict[str, str]):
-        self.headers = headers
-
-    async def kill_session(self, session_id: str):
-        if session_id in self.sessions:
-            context, page, _ = self.sessions[session_id]
-            await page.close()
-            await context.close()
-            del self.sessions[session_id]
-
-    def _cleanup_expired_sessions(self):
-        current_time = time.time()
-        expired_sessions = [
-            sid for sid, (_, _, last_used) in self.sessions.items() 
-            if current_time - last_used > self.session_ttl
-        ]
-        for sid in expired_sessions:
-            asyncio.create_task(self.kill_session(sid))
-            
-    async def smart_wait(self, page: Page, wait_for: str, timeout: float = 30000):
-        wait_for = wait_for.strip()
-        
-        if wait_for.startswith('js:'):
-            # Explicitly specified JavaScript
-            js_code = wait_for[3:].strip()
-            return await self.csp_compliant_wait(page, js_code, timeout)
-        elif wait_for.startswith('css:'):
-            # Explicitly specified CSS selector
-            css_selector = wait_for[4:].strip()
-            try:
-                await page.wait_for_selector(css_selector, timeout=timeout)
-            except Error as e:
-                if 'Timeout' in str(e):
-                    raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{css_selector}'")
-                else:
-                    raise ValueError(f"Invalid CSS selector: '{css_selector}'")
-        else:
-            # Auto-detect based on content
-            if wait_for.startswith('()') or wait_for.startswith('function'):
-                # It's likely a JavaScript function
-                return await self.csp_compliant_wait(page, wait_for, timeout)
-            else:
-                # Assume it's a CSS selector first
-                try:
-                    await page.wait_for_selector(wait_for, timeout=timeout)
-                except Error as e:
-                    if 'Timeout' in str(e):
-                        raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{wait_for}'")
-                    else:
-                        # If it's not a timeout error, it might be an invalid selector
-                        # Let's try to evaluate it as a JavaScript function as a fallback
-                        try:
-                            return await self.csp_compliant_wait(page, f"() => {{{wait_for}}}", timeout)
-                        except Error:
-                            raise ValueError(f"Invalid wait_for parameter: '{wait_for}'. "
-                                             "It should be either a valid CSS selector, a JavaScript function, "
-                                             "or explicitly prefixed with 'js:' or 'css:'.")
-    
-    async def csp_compliant_wait(self, page: Page, user_wait_function: str, timeout: float = 30000):
-        wrapper_js = f"""
-        async () => {{
-            const userFunction = {user_wait_function};
-            const startTime = Date.now();
-            while (true) {{
-                if (await userFunction()) {{
-                    return true;
-                }}
-                if (Date.now() - startTime > {timeout}) {{
-                    throw new Error('Timeout waiting for condition');
-                }}
-                await new Promise(resolve => setTimeout(resolve, 100));
-            }}
-        }}
-        """
-        
-        try:
-            await page.evaluate(wrapper_js)
-        except TimeoutError:
-            raise TimeoutError(f"Timeout after {timeout}ms waiting for condition")
-        except Exception as e:
-            raise RuntimeError(f"Error in wait condition: {str(e)}")
-
-    async def process_iframes(self, page):
-        # Find all iframes
-        iframes = await page.query_selector_all('iframe')
-        
-        for i, iframe in enumerate(iframes):
-            try:
-                # Add a unique identifier to the iframe
-                await iframe.evaluate(f'(element) => element.id = "iframe-{i}"')
-                
-                # Get the frame associated with this iframe
-                frame = await iframe.content_frame()
-                
-                if frame:
-                    # Wait for the frame to load
-                    await frame.wait_for_load_state('load', timeout=30000)  # 30 seconds timeout
-                    
-                    # Extract the content of the iframe's body
-                    iframe_content = await frame.evaluate('() => document.body.innerHTML')
-                    
-                    # Generate a unique class name for this iframe
-                    class_name = f'extracted-iframe-content-{i}'
-                    
-                    # Replace the iframe with a div containing the extracted content
-                    _iframe = iframe_content.replace('`', '\\`')
-                    await page.evaluate(f"""
-                        () => {{
-                            const iframe = document.getElementById('iframe-{i}');
-                            const div = document.createElement('div');
-                            div.innerHTML = `{_iframe}`;
-                            div.className = '{class_name}';
-                            iframe.replaceWith(div);
-                        }}
-                    """)
-                else:
-                    print(f"Warning: Could not access content frame for iframe {i}")
-            except Exception as e:
-                print(f"Error processing iframe {i}: {str(e)}")
-
-        # Return the page object
-        return page  
-    
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        response_headers = {}
-        status_code = None
-        
-        self._cleanup_expired_sessions()
-        session_id = kwargs.get("session_id")
-        if session_id:
-            context, page, _ = self.sessions.get(session_id, (None, None, None))
-            if not context:
-                context = await self.browser.new_context(
-                    user_agent=self.user_agent,
-                    viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None
-                )
-                await context.set_extra_http_headers(self.headers)
-                page = await context.new_page()
-                self.sessions[session_id] = (context, page, time.time())
-        else:
-            context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
-            )
-            await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
-            page = await context.new_page()
-
-        try:
-            if self.verbose:
-                print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                if os.path.exists(cache_file_path):
-                    html = ""
-                    with open(cache_file_path, "r") as f:
-                        html = f.read()
-                    # retrieve response headers and status code from cache
-                    with open(cache_file_path + ".meta", "r") as f:
-                        meta = json.load(f)
-                        response_headers = meta.get("response_headers", {})
-                        status_code = meta.get("status_code")
-                    response = AsyncCrawlResponse(
-                        html=html, response_headers=response_headers, status_code=status_code
-                    )
-                    return response
-
-            if not kwargs.get("js_only", False):
-                await self.execute_hook('before_goto', page)
-                
-                response = await page.goto("about:blank")
-                await stealth_async(page)
-                response = await page.goto(
-                    url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000)
-                )
-                
-                # await stealth_async(page)
-                # response = await page.goto("about:blank")
-                # await stealth_async(page)
-                # await page.evaluate(f"window.location.href = '{url}'")
-                
-                await self.execute_hook('after_goto', page)
-                
-                # Get status code and headers
-                status_code = response.status
-                response_headers = response.headers
-            else:
-                status_code = 200
-                response_headers = {}
-
-            await page.wait_for_selector('body')
-            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-
-            js_code = kwargs.get("js_code", kwargs.get("js", self.js_code))
-            if js_code:
-                if isinstance(js_code, str):
-                    await page.evaluate(js_code)
-                elif isinstance(js_code, list):
-                    for js in js_code:
-                        await page.evaluate(js)
-                
-                await page.wait_for_load_state('networkidle')
-                # Check for on execution event
-                await self.execute_hook('on_execution_started', page)
-                
-            if kwargs.get("simulate_user", False):
-                # Simulate user interactions
-                await page.mouse.move(100, 100)
-                await page.mouse.down()
-                await page.mouse.up()
-                await page.keyboard.press('ArrowDown')
-
-            # Handle the wait_for parameter
-            wait_for = kwargs.get("wait_for")
-            if wait_for:
-                try:
-                    await self.smart_wait(page, wait_for, timeout=kwargs.get("page_timeout", 60000))
-                except Exception as e:
-                    raise RuntimeError(f"Wait condition failed: {str(e)}")
-
-
-            
-            # Update image dimensions
-            update_image_dimensions_js = """
-            () => {
-                return new Promise((resolve) => {
-                    const filterImage = (img) => {
-                        // Filter out images that are too small
-                        if (img.width < 100 && img.height < 100) return false;
-                        
-                        // Filter out images that are not visible
-                        const rect = img.getBoundingClientRect();
-                        if (rect.width === 0 || rect.height === 0) return false;
-                        
-                        // Filter out images with certain class names (e.g., icons, thumbnails)
-                        if (img.classList.contains('icon') || img.classList.contains('thumbnail')) return false;
-                        
-                        // Filter out images with certain patterns in their src (e.g., placeholder images)
-                        if (img.src.includes('placeholder') || img.src.includes('icon')) return false;
-                        
-                        return true;
-                    };
-
-                    const images = Array.from(document.querySelectorAll('img')).filter(filterImage);
-                    let imagesLeft = images.length;
-                    
-                    if (imagesLeft === 0) {
-                        resolve();
-                        return;
-                    }
-
-                    const checkImage = (img) => {
-                        if (img.complete && img.naturalWidth !== 0) {
-                            img.setAttribute('width', img.naturalWidth);
-                            img.setAttribute('height', img.naturalHeight);
-                            imagesLeft--;
-                            if (imagesLeft === 0) resolve();
-                        }
-                    };
-
-                    images.forEach(img => {
-                        checkImage(img);
-                        if (!img.complete) {
-                            img.onload = () => {
-                                checkImage(img);
-                            };
-                            img.onerror = () => {
-                                imagesLeft--;
-                                if (imagesLeft === 0) resolve();
-                            };
-                        }
-                    });
-
-                    // Fallback timeout of 5 seconds
-                    setTimeout(() => resolve(), 5000);
-                });
-            }
-            """
-            await page.evaluate(update_image_dimensions_js)
-
-            # Wait a bit for any onload events to complete
-            await page.wait_for_timeout(100)
-
-            # Process iframes
-            if kwargs.get("process_iframes", False):
-                page = await self.process_iframes(page)
-            
-            await self.execute_hook('before_retrieve_html', page)
-            # Check if delay_before_return_html is set then wait for that time
-            delay_before_return_html = kwargs.get("delay_before_return_html")
-            if delay_before_return_html:
-                await asyncio.sleep(delay_before_return_html)
-                
-            html = await page.content()
-            await self.execute_hook('before_return_html', page, html)
-            
-            # Check if kwargs has screenshot=True then take screenshot
-            screenshot_data = None
-            if kwargs.get("screenshot"):
-                screenshot_data = await self.take_screenshot(url)            
-
-            if self.verbose:
-                print(f"[LOG] ✅ Crawled {url} successfully!")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                with open(cache_file_path, "w", encoding="utf-8") as f:
-                    f.write(html)
-                # store response headers and status code in cache
-                with open(cache_file_path + ".meta", "w", encoding="utf-8") as f:
-                    json.dump({
-                        "response_headers": response_headers,
-                        "status_code": status_code
-                    }, f)
-
-            async def get_delayed_content(delay: float = 5.0) -> str:
-                if self.verbose:
-                    print(f"[LOG] Waiting for {delay} seconds before retrieving content for {url}")
-                await asyncio.sleep(delay)
-                return await page.content()
-                
-            response = AsyncCrawlResponse(
-                html=html, 
-                response_headers=response_headers, 
-                status_code=status_code,
-                screenshot=screenshot_data,
-                get_delayed_content=get_delayed_content
-            )
-            return response
-        except Error as e:
-            raise Error(f"Failed to crawl {url}: {str(e)}")
-        finally:
-            if not session_id:
-                await page.close()
-                await context.close()
-
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        semaphore_count = kwargs.get('semaphore_count', 5)  # Adjust as needed
-        semaphore = asyncio.Semaphore(semaphore_count)
-
-        async def crawl_with_semaphore(url):
-            async with semaphore:
-                return await self.crawl(url, **kwargs)
-
-        tasks = [crawl_with_semaphore(url) for url in urls]
-        results = await asyncio.gather(*tasks, return_exceptions=True)
-        return [result if not isinstance(result, Exception) else str(result) for result in results]
-
-    async def take_screenshot(self, url: str, wait_time=1000) -> str:
-        async with await self.browser.new_context(user_agent=self.user_agent) as context:
-            page = await context.new_page()
-            try:
-                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
-                # Wait for a specified time (default is 1 second)
-                await page.wait_for_timeout(wait_time)
-                screenshot = await page.screenshot(full_page=True)
-                return base64.b64encode(screenshot).decode('utf-8')
-            except Exception as e:
-                error_message = f"Failed to take screenshot: {str(e)}"
-                print(error_message)
-
-                # Generate an error image
-                img = Image.new('RGB', (800, 600), color='black')
-                draw = ImageDraw.Draw(img)
-                font = ImageFont.load_default()
-                draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
-                
-                buffered = BytesIO()
-                img.save(buffered, format="JPEG")
-                return base64.b64encode(buffered.getvalue()).decode('utf-8')
-            finally:
-                await page.close()
-

crawl4ai/async_crawler_strategy.py

@@ -3,7 +3,8 @@ import base64
 import time
 from abc import ABC, abstractmethod
 from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
+import os, sys, shutil
+import tempfile, subprocess
 from playwright.async_api import async_playwright, Page, Browser, Error
 from io import BytesIO
 from PIL import Image, ImageDraw, ImageFont
@@ -13,6 +14,8 @@ from pydantic import BaseModel
 import hashlib
 import json
 import uuid
+from .models import AsyncCrawlResponse
+from .utils import create_box_message
 from playwright_stealth import StealthConfig, stealth_async
 
 stealth_config = StealthConfig(
@@ -31,15 +34,166 @@ stealth_config = StealthConfig(
 )
 
 
-class AsyncCrawlResponse(BaseModel):
-    html: str
-    response_headers: Dict[str, str]
-    status_code: int
-    screenshot: Optional[str] = None
-    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
+class ManagedBrowser:
+    def __init__(self, browser_type: str = "chromium", user_data_dir: Optional[str] = None, headless: bool = False, logger = None, host: str = "localhost", debugging_port: int = 9222):
+        self.browser_type = browser_type
+        self.user_data_dir = user_data_dir
+        self.headless = headless
+        self.browser_process = None
+        self.temp_dir = None
+        self.debugging_port = debugging_port
+        self.host = host
+        self.logger = logger
+        self.shutting_down = False
+
+    async def start(self) -> str:
+        """
+        Starts the browser process and returns the CDP endpoint URL.
+        If user_data_dir is not provided, creates a temporary directory.
+        """
+        
+        # Create temp dir if needed
+        if not self.user_data_dir:
+            self.temp_dir = tempfile.mkdtemp(prefix="browser-profile-")
+            self.user_data_dir = self.temp_dir
+
+        # Get browser path and args based on OS and browser type
+        browser_path = self._get_browser_path()
+        args = self._get_browser_args()
+
+        # Start browser process
+        try:
+            self.browser_process = subprocess.Popen(
+                args,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE
+            )
+            # Monitor browser process output for errors
+            asyncio.create_task(self._monitor_browser_process())
+            await asyncio.sleep(2)  # Give browser time to start
+            return f"http://{self.host}:{self.debugging_port}"
+        except Exception as e:
+            await self.cleanup()
+            raise Exception(f"Failed to start browser: {e}")
+
+    async def _monitor_browser_process(self):
+        """Monitor the browser process for unexpected termination."""
+        if self.browser_process:
+            try:
+                stdout, stderr = await asyncio.gather(
+                    asyncio.to_thread(self.browser_process.stdout.read),
+                    asyncio.to_thread(self.browser_process.stderr.read)
+                )
+                
+                # Check shutting_down flag BEFORE logging anything
+                if self.browser_process.poll() is not None:
+                    if not self.shutting_down:
+                        self.logger.error(
+                            message="Browser process terminated unexpectedly | Code: {code} | STDOUT: {stdout} | STDERR: {stderr}",
+                            tag="ERROR",
+                            params={
+                                "code": self.browser_process.returncode,
+                                "stdout": stdout.decode(),
+                                "stderr": stderr.decode()
+                            }
+                        )                
+                        await self.cleanup()
+                    else:
+                        self.logger.info(
+                            message="Browser process terminated normally | Code: {code}",
+                            tag="INFO",
+                            params={"code": self.browser_process.returncode}
+                        )
+            except Exception as e:
+                if not self.shutting_down:
+                    self.logger.error(
+                        message="Error monitoring browser process: {error}",
+                        tag="ERROR",
+                        params={"error": str(e)}
+                    )
+
+    def _get_browser_path(self) -> str:
+        """Returns the browser executable path based on OS and browser type"""
+        if sys.platform == "darwin":  # macOS
+            paths = {
+                "chromium": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
+                "firefox": "/Applications/Firefox.app/Contents/MacOS/firefox",
+                "webkit": "/Applications/Safari.app/Contents/MacOS/Safari"
+            }
+        elif sys.platform == "win32":  # Windows
+            paths = {
+                "chromium": "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
+                "firefox": "C:\\Program Files\\Mozilla Firefox\\firefox.exe",
+                "webkit": None  # WebKit not supported on Windows
+            }
+        else:  # Linux
+            paths = {
+                "chromium": "google-chrome",
+                "firefox": "firefox",
+                "webkit": None  # WebKit not supported on Linux
+            }
+        
+        return paths.get(self.browser_type)
+
+    def _get_browser_args(self) -> List[str]:
+        """Returns browser-specific command line arguments"""
+        base_args = [self._get_browser_path()]
+        
+        if self.browser_type == "chromium":
+            args = [
+                f"--remote-debugging-port={self.debugging_port}",
+                f"--user-data-dir={self.user_data_dir}",
+            ]
+            if self.headless:
+                args.append("--headless=new")
+        elif self.browser_type == "firefox":
+            args = [
+                "--remote-debugging-port", str(self.debugging_port),
+                "--profile", self.user_data_dir,
+            ]
+            if self.headless:
+                args.append("--headless")
+        else:
+            raise NotImplementedError(f"Browser type {self.browser_type} not supported")
+            
+        return base_args + args
+
+    async def cleanup(self):
+        """Cleanup browser process and temporary directory"""
+        # Set shutting_down flag BEFORE any termination actions
+        self.shutting_down = True
+        
+        if self.browser_process:
+            try:
+                self.browser_process.terminate()
+                # Wait for process to end gracefully
+                for _ in range(10):  # 10 attempts, 100ms each
+                    if self.browser_process.poll() is not None:
+                        break
+                    await asyncio.sleep(0.1)
+                
+                # Force kill if still running
+                if self.browser_process.poll() is None:
+                    self.browser_process.kill()
+                    await asyncio.sleep(0.1)  # Brief wait for kill to take effect
+                    
+            except Exception as e:
+                self.logger.error(
+                    message="Error terminating browser: {error}",
+                    tag="ERROR",
+                    params={"error": str(e)}
+                )
+
+        if self.temp_dir and os.path.exists(self.temp_dir):
+            try:
+                shutil.rmtree(self.temp_dir)
+            except Exception as e:
+                self.logger.error(
+                    message="Error removing temporary directory: {error}",
+                    tag="ERROR",
+                    params={"error": str(e)}
+                )
 
-    class Config:
-        arbitrary_types_allowed = True
 
 class AsyncCrawlerStrategy(ABC):
     @abstractmethod
@@ -63,7 +217,8 @@ class AsyncCrawlerStrategy(ABC):
         pass
 
 class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
-    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
+    def __init__(self, use_cached_html=False, js_code=None, logger = None, **kwargs):
+        self.logger = logger
         self.use_cached_html = use_cached_html
         self.user_agent = kwargs.get(
             "user_agent",
@@ -75,6 +230,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         self.headless = kwargs.get("headless", True)
         self.browser_type = kwargs.get("browser_type", "chromium")
         self.headers = kwargs.get("headers", {})
+        self.cookies = kwargs.get("cookies", [])
         self.sessions = {}
         self.session_ttl = 1800 
         self.js_code = js_code
@@ -82,6 +238,12 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         self.playwright = None
         self.browser = None
         self.sleep_on_close = kwargs.get("sleep_on_close", False)
+        self.use_managed_browser = kwargs.get("use_managed_browser", False)
+        self.user_data_dir = kwargs.get("user_data_dir", None)
+        self.use_persistent_context = kwargs.get("use_persistent_context", False)
+        self.chrome_channel = kwargs.get("chrome_channel", "chrome")
+        self.managed_browser = None
+        self.default_context = None
         self.hooks = {
             'on_browser_created': None,
             'on_user_agent_updated': None,
@@ -91,6 +253,14 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             'before_return_html': None,
             'before_retrieve_html': None
         }
+        self.extra_args = kwargs.get("extra_args", [])
+        self.accept_downloads = kwargs.get("accept_downloads", False)
+        self.downloads_path = kwargs.get("downloads_path")
+        self._downloaded_files = []  # Track downloaded files for current crawl
+        if self.accept_downloads and not self.downloads_path:
+            self.downloads_path = os.path.join(os.getcwd(), "downloads")
+            os.makedirs(self.downloads_path, exist_ok=True)        
+        
 
     async def __aenter__(self):
         await self.start()
@@ -103,52 +273,143 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         if self.playwright is None:
             self.playwright = await async_playwright().start()
         if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-            elif self.proxy_config:
-                proxy_settings = ProxySettings(server=self.proxy_config.get("server"), username=self.proxy_config.get("username"), password=self.proxy_config.get("password"))
-                browser_args["proxy"] = proxy_settings
+            if self.use_managed_browser:
+                # Use managed browser approach
+                self.managed_browser = ManagedBrowser(
+                    browser_type=self.browser_type,
+                    user_data_dir=self.user_data_dir,
+                    headless=self.headless,
+                    logger=self.logger
+                )
+                cdp_url = await self.managed_browser.start()
+                self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)
                 
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
+                # Get the default context that maintains the user profile
+                contexts = self.browser.contexts
+                if contexts:
+                    self.default_context = contexts[0]
+                else:
+                    # If no default context exists, create one
+                    self.default_context = await self.browser.new_context(
+                        viewport={"width": 1920, "height": 1080}
+                    )
+                
+                # Set up the default context
+                if self.default_context:
+                    await self.default_context.set_extra_http_headers(self.headers)
+                    if self.cookies:
+                        await self.default_context.add_cookies(self.cookies)                    
+                    if self.accept_downloads:
+                        await self.default_context.set_default_timeout(60000)
+                        await self.default_context.set_default_navigation_timeout(60000)
+                        self.default_context._impl_obj._options["accept_downloads"] = True
+                        self.default_context._impl_obj._options["downloads_path"] = self.downloads_path
+                        
+                    if self.user_agent:
+                        await self.default_context.set_extra_http_headers({
+                            "User-Agent": self.user_agent
+                        })
             else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
+                # Base browser arguments
+                browser_args = {
+                    "headless": self.headless,
+                    "args": [
+                        "--no-sandbox",
+                        "--disable-dev-shm-usage",
+                        "--no-first-run",
+                        "--no-default-browser-check",
+                        "--disable-infobars",
+                        "--window-position=0,0",
+                        "--ignore-certificate-errors",
+                        "--ignore-certificate-errors-spki-list"
+                    ]
+                }
+                
+                # Add channel if specified (try Chrome first)
+                if self.chrome_channel:
+                    browser_args["channel"] = self.chrome_channel
+                
+                # Add extra args if provided
+                if self.extra_args:
+                    browser_args["args"].extend(self.extra_args)
+                    
+                # Add downloads path if downloads are enabled
+                if self.accept_downloads:
+                    browser_args["downloads_path"] = self.downloads_path
+                
+                # Add proxy settings if a proxy is specified
+                if self.proxy:
+                    proxy_settings = ProxySettings(server=self.proxy)
+                    browser_args["proxy"] = proxy_settings
+                elif self.proxy_config:
+                    proxy_settings = ProxySettings(
+                        server=self.proxy_config.get("server"),
+                        username=self.proxy_config.get("username"),
+                        password=self.proxy_config.get("password")
+                    )
+                    browser_args["proxy"] = proxy_settings
+                    
+                try:
+                    # Select the appropriate browser based on the browser_type
+                    if self.browser_type == "firefox":
+                        self.browser = await self.playwright.firefox.launch(**browser_args)
+                    elif self.browser_type == "webkit":
+                        self.browser = await self.playwright.webkit.launch(**browser_args)
+                    else:
+                        if self.use_persistent_context and self.user_data_dir:
+                            self.browser = await self.playwright.chromium.launch_persistent_context(
+                                user_data_dir=self.user_data_dir,
+                                accept_downloads=self.accept_downloads,
+                                downloads_path=self.downloads_path if self.accept_downloads else None,                                
+                                **browser_args
+                            )
+                            self.default_context = self.browser
+                        else:
+                            self.browser = await self.playwright.chromium.launch(**browser_args)
+                                
+                except Exception as e:
+                    # Fallback to chromium if Chrome channel fails
+                    if "chrome" in str(e) and browser_args.get("channel") == "chrome":
+                        browser_args["channel"] = "chromium"
+                        if self.use_persistent_context and self.user_data_dir:
+                            self.browser = await self.playwright.chromium.launch_persistent_context(
+                                user_data_dir=self.user_data_dir,
+                                **browser_args
+                            )
+                            self.default_context = self.browser
+                        else:
+                            self.browser = await self.playwright.chromium.launch(**browser_args)
+                    else:
+                        raise
 
             await self.execute_hook('on_browser_created', self.browser)
 
     async def close(self):
         if self.sleep_on_close:
             await asyncio.sleep(0.5)
+            
+        # Close all active sessions
+        session_ids = list(self.sessions.keys())
+        for session_id in session_ids:
+            await self.kill_session(session_id)
+            
         if self.browser:
             await self.browser.close()
             self.browser = None
+            
+        if self.managed_browser:
+            await asyncio.sleep(0.5)
+            await self.managed_browser.cleanup()
+            self.managed_browser = None
+            
         if self.playwright:
             await self.playwright.stop()
             self.playwright = None
 
-    def __del__(self):
-        if self.browser or self.playwright:
-            asyncio.get_event_loop().run_until_complete(self.close())
+    # Issue #256: Remove __del__ method to avoid potential issues with async cleanup
+    # def __del__(self):
+    #     if self.browser or self.playwright:
+    #         asyncio.get_event_loop().run_until_complete(self.close())
 
     def set_hook(self, hook_type: str, hook: Callable):
         if hook_type in self.hooks:
@@ -156,13 +417,13 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         else:
             raise ValueError(f"Invalid hook type: {hook_type}")
 
-    async def execute_hook(self, hook_type: str, *args):
+    async def execute_hook(self, hook_type: str, *args, **kwargs):
         hook = self.hooks.get(hook_type)
         if hook:
             if asyncio.iscoroutinefunction(hook):
-                return await hook(*args)
+                return await hook(*args, **kwargs)
             else:
-                return hook(*args)
+                return hook(*args, **kwargs)
         return args[0] if args else None
 
     def update_user_agent(self, user_agent: str):
@@ -175,7 +436,8 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         if session_id in self.sessions:
             context, page, _ = self.sessions[session_id]
             await page.close()
-            await context.close()
+            if not self.use_managed_browser:
+                await context.close()
             del self.sessions[session_id]
 
     def _cleanup_expired_sessions(self):
@@ -284,74 +546,189 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                         }}
                     """)
                 else:
-                    print(f"Warning: Could not access content frame for iframe {i}")
+                    # print(f"Warning: Could not access content frame for iframe {i}")
+                    self.logger.warning(
+                        message="Could not access content frame for iframe {index}",
+                        tag="SCRAPE",
+                        params={"index": i}
+                    )                    
             except Exception as e:
-                print(f"Error processing iframe {i}: {str(e)}")
+                self.logger.error(
+                    message="Error processing iframe {index}: {error}",
+                    tag="ERROR",
+                    params={"index": i, "error": str(e)}
+                )                
+                # print(f"Error processing iframe {i}: {str(e)}")
 
         # Return the page object
         return page  
     
     async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
+        """
+        Crawls a given URL or processes raw HTML/local file content based on the URL prefix.
+
+        Args:
+            url (str): The URL to crawl. Supported prefixes:
+                - 'http://' or 'https://': Web URL to crawl.
+                - 'file://': Local file path to process.
+                - 'raw:': Raw HTML content to process.
+            **kwargs: Additional parameters:
+                - 'screenshot' (bool): Whether to take a screenshot.
+                - ... [other existing parameters]
+
+        Returns:
+            AsyncCrawlResponse: The response containing HTML, headers, status code, and optional screenshot.
+        """
+        response_headers = {}
+        status_code = 200  # Default to 200 for local/raw HTML
+        screenshot_requested = kwargs.get('screenshot', False)
+        screenshot_data = None
+
+        if url.startswith(('http://', 'https://')):
+            # Proceed with standard web crawling
+            return await self._crawl_web(url, **kwargs)
+
+        elif url.startswith('file://'):
+            # Process local file
+            local_file_path = url[7:]  # Remove 'file://' prefix
+            if not os.path.exists(local_file_path):
+                raise FileNotFoundError(f"Local file not found: {local_file_path}")
+            with open(local_file_path, 'r', encoding='utf-8') as f:
+                html = f.read()
+            if screenshot_requested:
+                screenshot_data = await self._generate_screenshot_from_html(html)
+            return AsyncCrawlResponse(
+                html=html,
+                response_headers=response_headers,
+                status_code=status_code,
+                screenshot=screenshot_data,
+                get_delayed_content=None
+            )
+
+        elif url.startswith('raw:'):
+            # Process raw HTML content
+            raw_html = url[4:]  # Remove 'raw:' prefix
+            html = raw_html
+            if screenshot_requested:
+                screenshot_data = await self._generate_screenshot_from_html(html)
+            return AsyncCrawlResponse(
+                html=html,
+                response_headers=response_headers,
+                status_code=status_code,
+                screenshot=screenshot_data,
+                get_delayed_content=None
+            )
+        else:
+            raise ValueError("URL must start with 'http://', 'https://', 'file://', or 'raw:'")
+
+
+    async def _crawl_web(self, url: str, **kwargs) -> AsyncCrawlResponse:
+        """
+        Existing web crawling logic remains unchanged.
+
+        Args:
+            url (str): The web URL to crawl.
+            **kwargs: Additional parameters.
+
+        Returns:
+            AsyncCrawlResponse: The response containing HTML, headers, status code, and optional screenshot.
+        """
         response_headers = {}
         status_code = None
         
+        # Reset downloaded files list for new crawl
+        self._downloaded_files = []
+        
         self._cleanup_expired_sessions()
         session_id = kwargs.get("session_id")
-        if session_id:
-            context, page, _ = self.sessions.get(session_id, (None, None, None))
-            if not context:
-                context = await self.browser.new_context(
-                    user_agent=self.user_agent,
-                    viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None,
-                    accept_downloads=True,
-                    java_script_enabled=True
-                )
-                await context.add_cookies([{"name": "cookiesEnabled", "value": "true", "url": url}])
-                await context.set_extra_http_headers(self.headers)
-                page = await context.new_page()
-                self.sessions[session_id] = (context, page, time.time())
+        
+        # Handle page creation differently for managed browser
+        context = None
+        if self.use_managed_browser:
+            if session_id:
+                # Reuse existing session if available
+                context, page, _ = self.sessions.get(session_id, (None, None, None))
+                if not page:
+                    # Create new page in default context if session doesn't exist
+                    page = await self.default_context.new_page()
+                    self.sessions[session_id] = (self.default_context, page, time.time())
+            else:
+                # Create new page in default context for non-session requests
+                page = await self.default_context.new_page()
         else:
-            context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
-            )
-            await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False) or kwargs.get("simulate_user", False) or kwargs.get("magic", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
-            page = await context.new_page()
-            # await stealth_async(page) #, stealth_config)
+            if session_id:
+                context, page, _ = self.sessions.get(session_id, (None, None, None))
+                if not context:
+                    if self.use_persistent_context and self.browser_type in ["chrome", "chromium"]:
+                        # In persistent context, browser is the context
+                        context = self.browser
+                        page = await context.new_page()
+                    else:
+                        # Normal context creation for non-persistent or non-Chrome browsers
+                        context = await self.browser.new_context(
+                            user_agent=self.user_agent,
+                            viewport={"width": 1200, "height": 800},
+                            proxy={"server": self.proxy} if self.proxy else None,
+                            java_script_enabled=True,
+                            accept_downloads=self.accept_downloads,
+                            # downloads_path=self.downloads_path if self.accept_downloads else None
+                        )
+                        await context.add_cookies([{"name": "cookiesEnabled", "value": "true", "url": url}])
+                        if self.cookies:
+                            await context.add_cookies(self.cookies)
+                        await context.set_extra_http_headers(self.headers)
+                        page = await context.new_page()
+                    self.sessions[session_id] = (context, page, time.time())
+            else:
+                if self.use_persistent_context and self.browser_type in ["chrome", "chromium"]:
+                    # In persistent context, browser is the context
+                    context = self.browser
+                else:
+                    # Normal context creation
+                    context = await self.browser.new_context(
+                        user_agent=self.user_agent,
+                        viewport={"width": 1920, "height": 1080},
+                        proxy={"server": self.proxy} if self.proxy else None,
+                        accept_downloads=self.accept_downloads,
+                    )
+                    if self.cookies:
+                            await context.add_cookies(self.cookies)
+                    await context.set_extra_http_headers(self.headers)
+                
+                if kwargs.get("override_navigator", False) or kwargs.get("simulate_user", False) or kwargs.get("magic", False):
+                    # Inject scripts to override navigator properties
+                    await context.add_init_script("""
+                        // Pass the Permissions Test.
+                        const originalQuery = window.navigator.permissions.query;
+                        window.navigator.permissions.query = (parameters) => (
+                            parameters.name === 'notifications' ?
+                                Promise.resolve({ state: Notification.permission }) :
+                                originalQuery(parameters)
+                        );
+                        Object.defineProperty(navigator, 'webdriver', {
+                            get: () => undefined
+                        });
+                        window.navigator.chrome = {
+                            runtime: {},
+                            // Add other properties if necessary
+                        };
+                        Object.defineProperty(navigator, 'plugins', {
+                            get: () => [1, 2, 3, 4, 5],
+                        });
+                        Object.defineProperty(navigator, 'languages', {
+                            get: () => ['en-US', 'en'],
+                        });
+                        Object.defineProperty(document, 'hidden', {
+                            get: () => false
+                        });
+                        Object.defineProperty(document, 'visibilityState', {
+                            get: () => 'visible'
+                        });
+                    """)
+                
+                page = await context.new_page()
+                if kwargs.get("magic", False):
+                    await stealth_async(page, stealth_config)
 
         # Add console message and error logging
         if kwargs.get("log_console", False):
@@ -359,12 +736,16 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             page.on("pageerror", lambda exc: print(f"Page Error: {exc}"))
         
         try:
-            if self.verbose:
-                print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
+            # Set up download handling if enabled
+            if self.accept_downloads:
+                page.on("download", lambda download: asyncio.create_task(self._handle_download(download)))
+
+            # if self.verbose:
+            #     print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
 
             if self.use_cached_html:
                 cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
+                    os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
                 )
                 if os.path.exists(cache_file_path):
                     html = ""
@@ -381,16 +762,23 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                     return response
 
             if not kwargs.get("js_only", False):
-                await self.execute_hook('before_goto', page)
+                await self.execute_hook('before_goto', page, context = context)
                 
-                response = await page.goto(
-                    url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000)
-                )
+
+                try:
+                    response = await page.goto(
+                        url,
+                        # wait_until=kwargs.get("wait_until", ["domcontentloaded", "networkidle"]),
+                        wait_until=kwargs.get("wait_until", "domcontentloaded"),
+                        timeout=kwargs.get("page_timeout", 60000),
+                    )
+                except Error as e:
+                    raise RuntimeError(f"Failed on navigating ACS-GOTO :\n{str(e)}")
                 
                 # response = await page.goto("about:blank")
                 # await page.evaluate(f"window.location.href = '{url}'")
                 
-                await self.execute_hook('after_goto', page)
+                await self.execute_hook('after_goto', page, context = context)
                 
                 # Get status code and headers
                 status_code = response.status
@@ -399,7 +787,48 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 status_code = 200
                 response_headers = {}
 
-            await page.wait_for_selector('body')
+            # Replace the current wait_for_selector line with this more robust check:
+            try:
+                # First wait for body to exist, regardless of visibility
+                await page.wait_for_selector('body', state='attached', timeout=30000)
+                
+                # Then wait for it to become visible by checking CSS
+                await page.wait_for_function("""
+                    () => {
+                        const body = document.body;
+                        const style = window.getComputedStyle(body);
+                        return style.display !== 'none' && 
+                            style.visibility !== 'hidden' && 
+                            style.opacity !== '0';
+                    }
+                """, timeout=30000)
+                
+            except Error as e:
+                # If waiting fails, let's try to diagnose the issue
+                visibility_info = await page.evaluate("""
+                    () => {
+                        const body = document.body;
+                        const style = window.getComputedStyle(body);
+                        return {
+                            display: style.display,
+                            visibility: style.visibility,
+                            opacity: style.opacity,
+                            hasContent: body.innerHTML.length,
+                            classList: Array.from(body.classList)
+                        }
+                    }
+                """)
+                
+                if self.verbose:
+                    print(f"Body visibility debug info: {visibility_info}")
+                
+                # Even if body is hidden, we might still want to proceed
+                if kwargs.get('ignore_body_visibility', True):
+                    if self.verbose:
+                        print("Proceeding despite hidden body...")
+                    pass
+                else:
+                    raise Error(f"Body element is hidden: {visibility_info}")
             
             await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
 
@@ -411,9 +840,10 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                     for js in js_code:
                         await page.evaluate(js)
                 
-                await page.wait_for_load_state('networkidle')
+                # await page.wait_for_timeout(100)
+                
                 # Check for on execution event
-                await self.execute_hook('on_execution_started', page)
+                await self.execute_hook('on_execution_started', page, context = context)
                 
             if kwargs.get("simulate_user", False) or kwargs.get("magic", False):
                 # Simulate user interactions
@@ -429,6 +859,9 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                     await self.smart_wait(page, wait_for, timeout=kwargs.get("page_timeout", 60000))
                 except Exception as e:
                     raise RuntimeError(f"Wait condition failed: {str(e)}")
+            
+            # if not wait_for and js_code:
+            #     await page.wait_for_load_state('networkidle', timeout=5000)
 
             # Update image dimensions
             update_image_dimensions_js = """
@@ -496,7 +929,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             if kwargs.get("process_iframes", False):
                 page = await self.process_iframes(page)
             
-            await self.execute_hook('before_retrieve_html', page)
+            await self.execute_hook('before_retrieve_html', page, context = context)
             # Check if delay_before_return_html is set then wait for that time
             delay_before_return_html = kwargs.get("delay_before_return_html")
             if delay_before_return_html:
@@ -507,7 +940,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 await self.remove_overlay_elements(page)
             
             html = await page.content()
-            await self.execute_hook('before_return_html', page, html)
+            await self.execute_hook('before_return_html', page, html, context = context)
             
             # Check if kwargs has screenshot=True then take screenshot
             screenshot_data = None
@@ -518,12 +951,12 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                     await asyncio.sleep(screenshot_wait_for)
                 screenshot_data = await self.take_screenshot(page)          
 
-            if self.verbose:
-                print(f"[LOG] ✅ Crawled {url} successfully!")
-
+            # if self.verbose:
+            #     print(f"[LOG] ✅ Crawled {url} successfully!")
+           
             if self.use_cached_html:
                 cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
+                    os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
                 )
                 with open(cache_file_path, "w", encoding="utf-8") as f:
                     f.write(html)
@@ -545,16 +978,49 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 response_headers=response_headers, 
                 status_code=status_code,
                 screenshot=screenshot_data,
-                get_delayed_content=get_delayed_content
+                get_delayed_content=get_delayed_content,
+                downloaded_files=self._downloaded_files if self._downloaded_files else None
             )
             return response
         except Error as e:
-            raise Error(f"[ERROR] 🚫 crawl(): Failed to crawl {url}: {str(e)}")
+            raise Error(f"async_crawler_strategy.py:_crawleb(): {str(e)}")
         # finally:
         #     if not session_id:
         #         await page.close()
         #         await context.close()
 
+    async def _handle_download(self, download):
+        """Handle file downloads."""
+        try:
+            suggested_filename = download.suggested_filename
+            download_path = os.path.join(self.downloads_path, suggested_filename)
+            
+            self.logger.info(
+                message="Downloading {filename} to {path}",
+                tag="FETCH",
+                params={"filename": suggested_filename, "path": download_path}
+            )
+                
+            start_time = time.perf_counter()
+            await download.save_as(download_path)
+            end_time = time.perf_counter()
+            self._downloaded_files.append(download_path)
+
+            self.logger.success(
+                message="Downloaded {filename} successfully",
+                tag="COMPLETE",
+                params={"filename": suggested_filename, "path": download_path, "duration": f"{end_time - start_time:.2f}s"}
+            )            
+        except Exception as e:
+            self.logger.error(
+                message="Failed to handle download: {error}",
+                tag="ERROR",
+                params={"error": str(e)}
+            )
+            
+            # if self.verbose:
+            #     print(f"[ERROR] Failed to handle download: {str(e)}")
+    
     async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
         semaphore_count = kwargs.get('semaphore_count', 5)  # Adjust as needed
         semaphore = asyncio.Semaphore(semaphore_count)
@@ -696,17 +1162,36 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             await page.evaluate(remove_overlays_js)
             await page.wait_for_timeout(500)  # Wait for any animations to complete
         except Exception as e:
-            if self.verbose:
-                print(f"Warning: Failed to remove overlay elements: {str(e)}")
+            self.logger.warning(
+                message="Failed to remove overlay elements: {error}",
+                tag="SCRAPE",
+                params={"error": str(e)}
+            )            
+            # if self.verbose:
+            #     print(f"Warning: Failed to remove overlay elements: {str(e)}")
 
     async def take_screenshot(self, page: Page) -> str:
+        """
+        Takes a screenshot of the current page.
+        
+        Args:
+            page (Page): The Playwright page instance
+            
+        Returns:
+            str: Base64-encoded screenshot image
+        """
         try:
             # The page is already loaded, just take the screenshot
             screenshot = await page.screenshot(full_page=True)
             return base64.b64encode(screenshot).decode('utf-8')
         except Exception as e:
             error_message = f"Failed to take screenshot: {str(e)}"
-            print(error_message)
+            self.logger.error(
+                message="Screenshot failed: {error}",
+                tag="ERROR",
+                params={"error": error_message}
+            )
+            
 
             # Generate an error image
             img = Image.new('RGB', (800, 600), color='black')
@@ -719,4 +1204,41 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             return base64.b64encode(buffered.getvalue()).decode('utf-8')
         finally:
             await page.close()
+            
+    async def _generate_screenshot_from_html(self, html: str) -> Optional[str]:
+        """
+        Generates a screenshot from raw HTML content.
+
+        Args:
+            html (str): The HTML content to render and capture.
+
+        Returns:
+            Optional[str]: Base64-encoded screenshot image or an error image if failed.
+        """
+        try:
+            if not self.browser:
+                await self.start()
+            page = await self.browser.new_page()
+            await page.set_content(html, wait_until='networkidle')
+            screenshot = await page.screenshot(full_page=True)
+            await page.close()
+            return base64.b64encode(screenshot).decode('utf-8')
+        except Exception as e:
+            error_message = f"Failed to take screenshot: {str(e)}"
+            # print(error_message)
+            self.logger.error(
+                message="Screenshot failed: {error}",
+                tag="ERROR",
+                params={"error": error_message}
+            )            
+
+            # Generate an error image
+            img = Image.new('RGB', (800, 600), color='black')
+            draw = ImageDraw.Draw(img)
+            font = ImageFont.load_default()
+            draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
+
+            buffered = BytesIO()
+            img.save(buffered, format="JPEG")
+            return base64.b64encode(buffered.getvalue()).decode('utf-8')
 

crawl4ai/async_database.py

@@ -2,18 +2,165 @@ import os
 from pathlib import Path
 import aiosqlite
 import asyncio
-from typing import Optional, Tuple
+from typing import Optional, Tuple, Dict
+from contextlib import asynccontextmanager
+import logging
+import json  # Added for serialization/deserialization
+from .utils import ensure_content_dirs, generate_content_hash
+from .models import CrawlResult
+import xxhash
+import aiofiles
+from .config import NEED_MIGRATION
+from .version_manager import VersionManager
+from .async_logger import AsyncLogger
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
 
-DB_PATH = os.path.join(Path.home(), ".crawl4ai")
+base_directory = DB_PATH = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
 os.makedirs(DB_PATH, exist_ok=True)
-DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")
+DB_PATH = os.path.join(base_directory, "crawl4ai.db")
 
 class AsyncDatabaseManager:
-    def __init__(self):
+    def __init__(self, pool_size: int = 10, max_retries: int = 3):
         self.db_path = DB_PATH
+        self.content_paths = ensure_content_dirs(os.path.dirname(DB_PATH))
+        self.pool_size = pool_size
+        self.max_retries = max_retries
+        self.connection_pool: Dict[int, aiosqlite.Connection] = {}
+        self.pool_lock = asyncio.Lock()
+        self.init_lock = asyncio.Lock()
+        self.connection_semaphore = asyncio.Semaphore(pool_size)
+        self._initialized = False  
+        self.version_manager = VersionManager()
+        self.logger = AsyncLogger(
+            log_file=os.path.join(base_directory, ".crawl4ai", "crawler_db.log"),
+            verbose=False,
+            tag_width=10
+        )
+        
+        
+    async def initialize(self):
+        """Initialize the database and connection pool"""
+        try:
+            self.logger.info("Initializing database", tag="INIT")
+            # Ensure the database file exists
+            os.makedirs(os.path.dirname(self.db_path), exist_ok=True)
+            
+            # Check if version update is needed
+            needs_update = self.version_manager.needs_update()
+            
+            # Always ensure base table exists
+            await self.ainit_db()
+            
+            # Verify the table exists
+            async with aiosqlite.connect(self.db_path, timeout=30.0) as db:
+                async with db.execute(
+                    "SELECT name FROM sqlite_master WHERE type='table' AND name='crawled_data'"
+                ) as cursor:
+                    result = await cursor.fetchone()
+                    if not result:
+                        raise Exception("crawled_data table was not created")
+            
+            # If version changed or fresh install, run updates
+            if needs_update:
+                self.logger.info("New version detected, running updates", tag="INIT")
+                await self.update_db_schema()
+                from .migrations import run_migration  # Import here to avoid circular imports
+                await run_migration()
+                self.version_manager.update_version()  # Update stored version after successful migration
+                self.logger.success("Version update completed successfully", tag="COMPLETE")
+            else:
+                self.logger.success("Database initialization completed successfully", tag="COMPLETE")
+
+                
+        except Exception as e:
+            self.logger.error(
+                message="Database initialization error: {error}",
+                tag="ERROR",
+                params={"error": str(e)}
+            )
+            self.logger.info(
+                message="Database will be initialized on first use",
+                tag="INIT"
+            )
+                        
+            raise
+
+            
+    async def cleanup(self):
+        """Cleanup connections when shutting down"""
+        async with self.pool_lock:
+            for conn in self.connection_pool.values():
+                await conn.close()
+            self.connection_pool.clear()
+
+    @asynccontextmanager
+    async def get_connection(self):
+        """Connection pool manager"""
+        if not self._initialized:
+            # Use an asyncio.Lock to ensure only one initialization occurs
+            async with self.init_lock:
+                if not self._initialized:
+                    await self.initialize()
+                    self._initialized = True
+
+        await self.connection_semaphore.acquire()
+        task_id = id(asyncio.current_task())
+        try:
+            async with self.pool_lock:
+                if task_id not in self.connection_pool:
+                    conn = await aiosqlite.connect(
+                        self.db_path,
+                        timeout=30.0
+                    )
+                    await conn.execute('PRAGMA journal_mode = WAL')
+                    await conn.execute('PRAGMA busy_timeout = 5000')
+                    self.connection_pool[task_id] = conn
+
+            yield self.connection_pool[task_id]
+
+        except Exception as e:
+            self.logger.error(
+                message="Connection error: {error}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"error": str(e)}
+            )
+            raise
+        finally:
+            async with self.pool_lock:
+                if task_id in self.connection_pool:
+                    await self.connection_pool[task_id].close()
+                    del self.connection_pool[task_id]
+            self.connection_semaphore.release()
+
+
+    async def execute_with_retry(self, operation, *args):
+        """Execute database operations with retry logic"""
+        for attempt in range(self.max_retries):
+            try:
+                async with self.get_connection() as db:
+                    result = await operation(db, *args)
+                    await db.commit()
+                    return result
+            except Exception as e:
+                if attempt == self.max_retries - 1:
+                    self.logger.error(
+                        message="Operation failed after {retries} attempts: {error}",
+                        tag="ERROR",
+                        force_verbose=True,
+                        params={
+                            "retries": self.max_retries,
+                            "error": str(e)
+                        }
+                    )                    
+                    raise
+                await asyncio.sleep(1 * (attempt + 1))  # Exponential backoff
 
     async def ainit_db(self):
-        async with aiosqlite.connect(self.db_path) as db:
+        """Initialize database schema"""
+        async with aiosqlite.connect(self.db_path, timeout=30.0) as db:
             await db.execute('''
                 CREATE TABLE IF NOT EXISTS crawled_data (
                     url TEXT PRIMARY KEY,
@@ -25,90 +172,250 @@ class AsyncDatabaseManager:
                     media TEXT DEFAULT "{}",
                     links TEXT DEFAULT "{}",
                     metadata TEXT DEFAULT "{}",
-                    screenshot TEXT DEFAULT ""
+                    screenshot TEXT DEFAULT "",
+                    response_headers TEXT DEFAULT "{}",
+                    downloaded_files TEXT DEFAULT "{}"  -- New column added
                 )
             ''')
             await db.commit()
-        await self.update_db_schema()
+
+        
 
     async def update_db_schema(self):
-        async with aiosqlite.connect(self.db_path) as db:
-            # Check if the 'media' column exists
+        """Update database schema if needed"""
+        async with aiosqlite.connect(self.db_path, timeout=30.0) as db:
             cursor = await db.execute("PRAGMA table_info(crawled_data)")
             columns = await cursor.fetchall()
             column_names = [column[1] for column in columns]
             
-            if 'media' not in column_names:
-                await self.aalter_db_add_column('media')
+            # List of new columns to add
+            new_columns = ['media', 'links', 'metadata', 'screenshot', 'response_headers', 'downloaded_files']
             
-            # Check for other missing columns and add them if necessary
-            for column in ['links', 'metadata', 'screenshot']:
+            for column in new_columns:
                 if column not in column_names:
-                    await self.aalter_db_add_column(column)
+                    await self.aalter_db_add_column(column, db)
+            await db.commit()
 
-    async def aalter_db_add_column(self, new_column: str):
-        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
-                await db.commit()
-            print(f"Added column '{new_column}' to the database.")
-        except Exception as e:
-            print(f"Error altering database to add {new_column} column: {e}")
+    async def aalter_db_add_column(self, new_column: str, db):
+        """Add new column to the database"""
+        if new_column == 'response_headers':
+            await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT "{{}}"')
+        else:
+            await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
+        self.logger.info(
+            message="Added column '{column}' to the database",
+            tag="INIT",
+            params={"column": new_column}
+        )        
+
+
+    async def aget_cached_url(self, url: str) -> Optional[CrawlResult]:
+        """Retrieve cached URL data as CrawlResult"""
+        async def _get(db):
+            async with db.execute(
+                'SELECT * FROM crawled_data WHERE url = ?', (url,)
+            ) as cursor:
+                row = await cursor.fetchone()
+                if not row:
+                    return None
+                    
+                # Get column names
+                columns = [description[0] for description in cursor.description]
+                # Create dict from row data
+                row_dict = dict(zip(columns, row))
+                
+                # Load content from files using stored hashes
+                content_fields = {
+                    'html': row_dict['html'],
+                    'cleaned_html': row_dict['cleaned_html'],
+                    'markdown': row_dict['markdown'],
+                    'extracted_content': row_dict['extracted_content'],
+                    'screenshot': row_dict['screenshot']
+                }
+                
+                for field, hash_value in content_fields.items():
+                    if hash_value:
+                        content = await self._load_content(
+                            hash_value, 
+                            field.split('_')[0]  # Get content type from field name
+                        )
+                        row_dict[field] = content or ""
+                    else:
+                        row_dict[field] = ""
+
+                # Parse JSON fields
+                json_fields = ['media', 'links', 'metadata', 'response_headers']
+                for field in json_fields:
+                    try:
+                        row_dict[field] = json.loads(row_dict[field]) if row_dict[field] else {}
+                    except json.JSONDecodeError:
+                        row_dict[field] = {}
+
+                # Parse downloaded_files
+                try:
+                    row_dict['downloaded_files'] = json.loads(row_dict['downloaded_files']) if row_dict['downloaded_files'] else []
+                except json.JSONDecodeError:
+                    row_dict['downloaded_files'] = []
+
+                # Remove any fields not in CrawlResult model
+                valid_fields = CrawlResult.__annotations__.keys()
+                filtered_dict = {k: v for k, v in row_dict.items() if k in valid_fields}
+                
+                return CrawlResult(**filtered_dict)
 
-    async def aget_cached_url(self, url: str) -> Optional[Tuple[str, str, str, str, str, str, str, bool, str]]:
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                async with db.execute('SELECT url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot FROM crawled_data WHERE url = ?', (url,)) as cursor:
-                    return await cursor.fetchone()
+            return await self.execute_with_retry(_get)
         except Exception as e:
-            print(f"Error retrieving cached URL: {e}")
+            self.logger.error(
+                message="Error retrieving cached URL: {error}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"error": str(e)}
+            )
             return None
 
-    async def acache_url(self, url: str, html: str, cleaned_html: str, markdown: str, extracted_content: str, success: bool, media: str = "{}", links: str = "{}", metadata: str = "{}", screenshot: str = ""):
+    async def acache_url(self, result: CrawlResult):
+        """Cache CrawlResult data"""
+        # Store content files and get hashes
+        content_map = {
+            'html': (result.html, 'html'),
+            'cleaned_html': (result.cleaned_html or "", 'cleaned'),
+            'markdown': (result.markdown or "", 'markdown'),
+            'extracted_content': (result.extracted_content or "", 'extracted'),
+            'screenshot': (result.screenshot or "", 'screenshots')
+        }
+        
+        content_hashes = {}
+        for field, (content, content_type) in content_map.items():
+            content_hashes[field] = await self._store_content(content, content_type)
+
+        async def _cache(db):
+            await db.execute('''
+                INSERT INTO crawled_data (
+                    url, html, cleaned_html, markdown,
+                    extracted_content, success, media, links, metadata,
+                    screenshot, response_headers, downloaded_files
+                )
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                ON CONFLICT(url) DO UPDATE SET
+                    html = excluded.html,
+                    cleaned_html = excluded.cleaned_html,
+                    markdown = excluded.markdown,
+                    extracted_content = excluded.extracted_content,
+                    success = excluded.success,
+                    media = excluded.media,
+                    links = excluded.links,
+                    metadata = excluded.metadata,
+                    screenshot = excluded.screenshot,
+                    response_headers = excluded.response_headers,
+                    downloaded_files = excluded.downloaded_files
+            ''', (
+                result.url,
+                content_hashes['html'],
+                content_hashes['cleaned_html'],
+                content_hashes['markdown'],
+                content_hashes['extracted_content'],
+                result.success,
+                json.dumps(result.media),
+                json.dumps(result.links),
+                json.dumps(result.metadata or {}),
+                content_hashes['screenshot'],
+                json.dumps(result.response_headers or {}),
+                json.dumps(result.downloaded_files or [])
+            ))
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('''
-                    INSERT INTO crawled_data (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot)
-                    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-                    ON CONFLICT(url) DO UPDATE SET
-                        html = excluded.html,
-                        cleaned_html = excluded.cleaned_html,
-                        markdown = excluded.markdown,
-                        extracted_content = excluded.extracted_content,
-                        success = excluded.success,
-                        media = excluded.media,      
-                        links = excluded.links,    
-                        metadata = excluded.metadata,      
-                        screenshot = excluded.screenshot
-                ''', (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot))
-                await db.commit()
+            await self.execute_with_retry(_cache)
         except Exception as e:
-            print(f"Error caching URL: {e}")
+            self.logger.error(
+                message="Error caching URL: {error}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"error": str(e)}
+            )
+            
 
     async def aget_total_count(self) -> int:
+        """Get total number of cached URLs"""
+        async def _count(db):
+            async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
+                result = await cursor.fetchone()
+                return result[0] if result else 0
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
-                    result = await cursor.fetchone()
-                    return result[0] if result else 0
+            return await self.execute_with_retry(_count)
         except Exception as e:
-            print(f"Error getting total count: {e}")
+            self.logger.error(
+                message="Error getting total count: {error}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"error": str(e)}
+            )
             return 0
 
     async def aclear_db(self):
+        """Clear all data from the database"""
+        async def _clear(db):
+            await db.execute('DELETE FROM crawled_data')
+
         try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('DELETE FROM crawled_data')
-                await db.commit()
+            await self.execute_with_retry(_clear)
         except Exception as e:
-            print(f"Error clearing database: {e}")
+            self.logger.error(
+                message="Error clearing database: {error}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"error": str(e)}
+            )
 
     async def aflush_db(self):
-        try:
-            async with aiosqlite.connect(self.db_path) as db:
-                await db.execute('DROP TABLE IF EXISTS crawled_data')
-                await db.commit()
-        except Exception as e:
-            print(f"Error flushing database: {e}")
+        """Drop the entire table"""
+        async def _flush(db):
+            await db.execute('DROP TABLE IF EXISTS crawled_data')
 
-async_db_manager = AsyncDatabaseManager()
+        try:
+            await self.execute_with_retry(_flush)
+        except Exception as e:
+            self.logger.error(
+                message="Error flushing database: {error}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"error": str(e)}
+            )
+            
+                
+    async def _store_content(self, content: str, content_type: str) -> str:
+        """Store content in filesystem and return hash"""
+        if not content:
+            return ""
+            
+        content_hash = generate_content_hash(content)
+        file_path = os.path.join(self.content_paths[content_type], content_hash)
+        
+        # Only write if file doesn't exist
+        if not os.path.exists(file_path):
+            async with aiofiles.open(file_path, 'w', encoding='utf-8') as f:
+                await f.write(content)
+                
+        return content_hash
+
+    async def _load_content(self, content_hash: str, content_type: str) -> Optional[str]:
+        """Load content from filesystem by hash"""
+        if not content_hash:
+            return None
+            
+        file_path = os.path.join(self.content_paths[content_type], content_hash)
+        try:
+            async with aiofiles.open(file_path, 'r', encoding='utf-8') as f:
+                return await f.read()
+        except:
+            self.logger.error(
+                message="Failed to load content: {file_path}",
+                tag="ERROR",
+                force_verbose=True,
+                params={"file_path": file_path}
+            )
+            return None
+
+# Create a singleton instance
+async_db_manager = AsyncDatabaseManager()

crawl4ai/async_logger.py

@@ -0,0 +1,231 @@
+from enum import Enum
+from typing import Optional, Dict, Any, Union
+from colorama import Fore, Back, Style, init
+import time
+import os
+from datetime import datetime
+
+class LogLevel(Enum):
+    DEBUG = 1
+    INFO = 2
+    SUCCESS = 3
+    WARNING = 4
+    ERROR = 5
+
+class AsyncLogger:
+    """
+    Asynchronous logger with support for colored console output and file logging.
+    Supports templated messages with colored components.
+    """
+    
+    DEFAULT_ICONS = {
+        'INIT': '→',
+        'READY': '✓',
+        'FETCH': '↓',
+        'SCRAPE': '◆',
+        'EXTRACT': '■',
+        'COMPLETE': '●',
+        'ERROR': '×',
+        'DEBUG': '⋯',
+        'INFO': 'ℹ',
+        'WARNING': '⚠',
+    }
+
+    DEFAULT_COLORS = {
+        LogLevel.DEBUG: Fore.LIGHTBLACK_EX,
+        LogLevel.INFO: Fore.CYAN,
+        LogLevel.SUCCESS: Fore.GREEN,
+        LogLevel.WARNING: Fore.YELLOW,
+        LogLevel.ERROR: Fore.RED,
+    }
+
+    def __init__(
+        self,
+        log_file: Optional[str] = None,
+        log_level: LogLevel = LogLevel.INFO,
+        tag_width: int = 10,
+        icons: Optional[Dict[str, str]] = None,
+        colors: Optional[Dict[LogLevel, str]] = None,
+        verbose: bool = True
+    ):
+        """
+        Initialize the logger.
+        
+        Args:
+            log_file: Optional file path for logging
+            log_level: Minimum log level to display
+            tag_width: Width for tag formatting
+            icons: Custom icons for different tags
+            colors: Custom colors for different log levels
+            verbose: Whether to output to console
+        """
+        init()  # Initialize colorama
+        self.log_file = log_file
+        self.log_level = log_level
+        self.tag_width = tag_width
+        self.icons = icons or self.DEFAULT_ICONS
+        self.colors = colors or self.DEFAULT_COLORS
+        self.verbose = verbose
+        
+        # Create log file directory if needed
+        if log_file:
+            os.makedirs(os.path.dirname(os.path.abspath(log_file)), exist_ok=True)
+
+    def _format_tag(self, tag: str) -> str:
+        """Format a tag with consistent width."""
+        return f"[{tag}]".ljust(self.tag_width, ".")
+
+    def _get_icon(self, tag: str) -> str:
+        """Get the icon for a tag, defaulting to info icon if not found."""
+        return self.icons.get(tag, self.icons['INFO'])
+
+    def _write_to_file(self, message: str):
+        """Write a message to the log file if configured."""
+        if self.log_file:
+            timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
+            with open(self.log_file, 'a', encoding='utf-8') as f:
+                # Strip ANSI color codes for file output
+                clean_message = message.replace(Fore.RESET, '').replace(Style.RESET_ALL, '')
+                for color in vars(Fore).values():
+                    if isinstance(color, str):
+                        clean_message = clean_message.replace(color, '')
+                f.write(f"[{timestamp}] {clean_message}\n")
+
+    def _log(
+        self,
+        level: LogLevel,
+        message: str,
+        tag: str,
+        params: Optional[Dict[str, Any]] = None,
+        colors: Optional[Dict[str, str]] = None,
+        base_color: Optional[str] = None,
+        **kwargs
+    ):
+        """
+        Core logging method that handles message formatting and output.
+        
+        Args:
+            level: Log level for this message
+            message: Message template string
+            tag: Tag for the message
+            params: Parameters to format into the message
+            colors: Color overrides for specific parameters
+            base_color: Base color for the entire message
+        """
+        if level.value < self.log_level.value:
+            return
+
+        # Format the message with parameters if provided
+        if params:
+            try:
+                # First format the message with raw parameters
+                formatted_message = message.format(**params)
+                
+                # Then apply colors if specified
+                if colors:
+                    for key, color in colors.items():
+                        # Find the formatted value in the message and wrap it with color
+                        if key in params:
+                            value_str = str(params[key])
+                            formatted_message = formatted_message.replace(
+                                value_str, 
+                                f"{color}{value_str}{Style.RESET_ALL}"
+                            )
+                            
+            except KeyError as e:
+                formatted_message = f"LOGGING ERROR: Missing parameter {e} in message template"
+                level = LogLevel.ERROR
+        else:
+            formatted_message = message
+
+        # Construct the full log line
+        color = base_color or self.colors[level]
+        log_line = f"{color}{self._format_tag(tag)} {self._get_icon(tag)} {formatted_message}{Style.RESET_ALL}"
+
+        # Output to console if verbose
+        if self.verbose or kwargs.get("force_verbose", False):
+            print(log_line)
+
+        # Write to file if configured
+        self._write_to_file(log_line)
+
+    def debug(self, message: str, tag: str = "DEBUG", **kwargs):
+        """Log a debug message."""
+        self._log(LogLevel.DEBUG, message, tag, **kwargs)
+
+    def info(self, message: str, tag: str = "INFO", **kwargs):
+        """Log an info message."""
+        self._log(LogLevel.INFO, message, tag, **kwargs)
+
+    def success(self, message: str, tag: str = "SUCCESS", **kwargs):
+        """Log a success message."""
+        self._log(LogLevel.SUCCESS, message, tag, **kwargs)
+
+    def warning(self, message: str, tag: str = "WARNING", **kwargs):
+        """Log a warning message."""
+        self._log(LogLevel.WARNING, message, tag, **kwargs)
+
+    def error(self, message: str, tag: str = "ERROR", **kwargs):
+        """Log an error message."""
+        self._log(LogLevel.ERROR, message, tag, **kwargs)
+
+    def url_status(
+        self,
+        url: str,
+        success: bool,
+        timing: float,
+        tag: str = "FETCH",
+        url_length: int = 50
+    ):
+        """
+        Convenience method for logging URL fetch status.
+        
+        Args:
+            url: The URL being processed
+            success: Whether the operation was successful
+            timing: Time taken for the operation
+            tag: Tag for the message
+            url_length: Maximum length for URL in log
+        """
+        self._log(
+            level=LogLevel.SUCCESS if success else LogLevel.ERROR,
+            message="{url:.{url_length}}... | Status: {status} | Time: {timing:.2f}s",
+            tag=tag,
+            params={
+                "url": url,
+                "url_length": url_length,
+                "status": success,
+                "timing": timing
+            },
+            colors={
+                "status": Fore.GREEN if success else Fore.RED,
+                "timing": Fore.YELLOW
+            }
+        )
+
+    def error_status(
+        self,
+        url: str,
+        error: str,
+        tag: str = "ERROR",
+        url_length: int = 50
+    ):
+        """
+        Convenience method for logging error status.
+        
+        Args:
+            url: The URL being processed
+            error: Error message
+            tag: Tag for the message
+            url_length: Maximum length for URL in log
+        """
+        self._log(
+            level=LogLevel.ERROR,
+            message="{url:.{url_length}}... | Error: {error}",
+            tag=tag,
+            params={
+                "url": url,
+                "url_length": url_length,
+                "error": error
+            }
+        )

crawl4ai/async_webcrawler.py

@@ -1,36 +1,109 @@
 import os
 import time
+import warnings
+from enum import Enum
+from colorama import init, Fore, Back, Style
 from pathlib import Path
-from typing import Optional
+from typing import Optional, List, Union
 import json
 import asyncio
-from .models import CrawlResult
+from .models import CrawlResult, MarkdownGenerationResult
 from .async_database import async_db_manager
 from .chunking_strategy import *
+from .content_filter_strategy import *
 from .extraction_strategy import *
 from .async_crawler_strategy import AsyncCrawlerStrategy, AsyncPlaywrightCrawlerStrategy, AsyncCrawlResponse
-from .content_scrapping_strategy import WebScrappingStrategy
-from .config import MIN_WORD_THRESHOLD, IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD
+from .cache_context import CacheMode, CacheContext, _legacy_to_cache_mode
+from .content_scraping_strategy import WebScrapingStrategy
+from .async_logger import AsyncLogger
+
+from .config import (
+    MIN_WORD_THRESHOLD, 
+    IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD,
+    URL_LOG_SHORTEN_LENGTH
+)
 from .utils import (
     sanitize_input_encode,
     InvalidCSSSelectorError,
-    format_html
+    format_html,
+    fast_format_html,
+    create_box_message
 )
 
+from urllib.parse import urlparse
+import random
+from .__version__ import __version__ as crawl4ai_version
+
 
 class AsyncWebCrawler:
+    """
+    Asynchronous web crawler with flexible caching capabilities.
+    
+    Migration Guide (from version X.X.X):
+    Old way (deprecated):
+        crawler = AsyncWebCrawler(always_by_pass_cache=True)
+        result = await crawler.arun(
+            url="https://example.com",
+            bypass_cache=True,
+            no_cache_read=True,
+            no_cache_write=False
+        )
+    
+    New way (recommended):
+        crawler = AsyncWebCrawler(always_bypass_cache=True)
+        result = await crawler.arun(
+            url="https://example.com",
+            cache_mode=CacheMode.WRITE_ONLY
+        )
+    
+    To disable deprecation warnings:
+        Pass warning=False to suppress the warning.
+    """
+    _domain_last_hit = {}
+
     def __init__(
         self,
         crawler_strategy: Optional[AsyncCrawlerStrategy] = None,
-        always_by_pass_cache: bool = False,
-        base_directory: str = str(Path.home()),
+        always_bypass_cache: bool = False,
+        always_by_pass_cache: Optional[bool] = None,  # Deprecated parameter
+        base_directory: str = str(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())),
         **kwargs,
     ):
+        """
+        Initialize the AsyncWebCrawler.
+
+        Args:
+            crawler_strategy: Strategy for crawling web pages
+            always_bypass_cache: Whether to always bypass cache (new parameter)
+            always_by_pass_cache: Deprecated, use always_bypass_cache instead
+            base_directory: Base directory for storing cache
+        """  
+        self.verbose = kwargs.get("verbose", False)
+        self.logger = AsyncLogger(
+            log_file=os.path.join(base_directory, ".crawl4ai", "crawler.log"),
+            verbose=self.verbose,
+            tag_width=10
+        )
+        
         self.crawler_strategy = crawler_strategy or AsyncPlaywrightCrawlerStrategy(
+            logger = self.logger,
             **kwargs
         )
-        self.always_by_pass_cache = always_by_pass_cache
-        # self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        
+        # Handle deprecated parameter
+        if always_by_pass_cache is not None:
+            if kwargs.get("warning", True):
+                warnings.warn(
+                    "'always_by_pass_cache' is deprecated and will be removed in version X.X.X. "
+                    "Use 'always_bypass_cache' instead. "
+                    "Pass warning=False to suppress this warning.",
+                    DeprecationWarning,
+                    stacklevel=2
+                )
+            self.always_bypass_cache = always_by_pass_cache
+        else:
+            self.always_bypass_cache = always_bypass_cache
+
         self.crawl4ai_folder = os.path.join(base_directory, ".crawl4ai")
         os.makedirs(self.crawl4ai_folder, exist_ok=True)
         os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
@@ -46,18 +119,14 @@ class AsyncWebCrawler:
         await self.crawler_strategy.__aexit__(exc_type, exc_val, exc_tb)
 
     async def awarmup(self):
-        if self.verbose:
-            print("[LOG] 🌤️  Warming up the AsyncWebCrawler")
-        await async_db_manager.ainit_db()
-        await self.arun(
-            url="https://google.com/",
-            word_count_threshold=5,
-            bypass_cache=False,
-            verbose=False,
-        )
+        """Initialize the crawler with warm-up sequence."""
+        self.logger.info(f"Crawl4AI {crawl4ai_version}", tag="INIT")
+        # if self.verbose:
+        #     print(f"{Fore.CYAN}{self.tag_format('INIT')} {self.log_icons['INIT']} Crawl4AI {crawl4ai_version}{Style.RESET_ALL}")
+        #     print(f"{Fore.CYAN}{self.tag_format('INIT')} {self.log_icons['INIT']} Warming up AsyncWebCrawler{Style.RESET_ALL}")
         self.ready = True
-        if self.verbose:
-            print("[LOG] 🌞 AsyncWebCrawler is ready to crawl")
+        # if self.verbose:
+        #     print(f"{Fore.GREEN}{self.tag_format('READY')} {self.log_icons['READY']} AsyncWebCrawler initialized{Style.RESET_ALL}")
 
     async def arun(
         self,
@@ -65,14 +134,82 @@ class AsyncWebCrawler:
         word_count_threshold=MIN_WORD_THRESHOLD,
         extraction_strategy: ExtractionStrategy = None,
         chunking_strategy: ChunkingStrategy = RegexChunking(),
+        content_filter: RelevantContentFilter = None,
+        cache_mode: Optional[CacheMode] = None,
+        # Deprecated parameters
         bypass_cache: bool = False,
+        disable_cache: bool = False,
+        no_cache_read: bool = False,
+        no_cache_write: bool = False,
+        # Other parameters
         css_selector: str = None,
         screenshot: bool = False,
         user_agent: str = None,
         verbose=True,
         **kwargs,
     ) -> CrawlResult:
+        """
+        Runs the crawler for a single source: URL (web, local file, or raw HTML).
+
+        Migration from legacy cache parameters:
+            Old way (deprecated):
+                await crawler.arun(url, bypass_cache=True, no_cache_read=True)
+            
+            New way:
+                await crawler.arun(url, cache_mode=CacheMode.BYPASS)
+
+        Args:
+            url: The URL to crawl (http://, https://, file://, or raw:)
+            cache_mode: Cache behavior control (recommended)
+            word_count_threshold: Minimum word count threshold
+            extraction_strategy: Strategy for content extraction
+            chunking_strategy: Strategy for content chunking
+            css_selector: CSS selector for content extraction
+            screenshot: Whether to capture screenshot
+            user_agent: Custom user agent
+            verbose: Enable verbose logging
+            
+            Deprecated Args:
+                bypass_cache: Use cache_mode=CacheMode.BYPASS instead
+                disable_cache: Use cache_mode=CacheMode.DISABLED instead
+                no_cache_read: Use cache_mode=CacheMode.WRITE_ONLY instead
+                no_cache_write: Use cache_mode=CacheMode.READ_ONLY instead
+
+        Returns:
+            CrawlResult: The result of crawling and processing
+        """
         try:
+            # Handle deprecated parameters
+            if any([bypass_cache, disable_cache, no_cache_read, no_cache_write]):
+                if kwargs.get("warning", True):
+                    warnings.warn(
+                        "Cache control boolean flags are deprecated and will be removed in version X.X.X. "
+                        "Use 'cache_mode' parameter instead. Examples:\n"
+                        "- For bypass_cache=True, use cache_mode=CacheMode.BYPASS\n"
+                        "- For disable_cache=True, use cache_mode=CacheMode.DISABLED\n"
+                        "- For no_cache_read=True, use cache_mode=CacheMode.WRITE_ONLY\n"
+                        "- For no_cache_write=True, use cache_mode=CacheMode.READ_ONLY\n"
+                        "Pass warning=False to suppress this warning.",
+                        DeprecationWarning,
+                        stacklevel=2
+                    )
+                
+                # Convert legacy parameters if cache_mode not provided
+                if cache_mode is None:
+                    cache_mode = _legacy_to_cache_mode(
+                        disable_cache=disable_cache,
+                        bypass_cache=bypass_cache,
+                        no_cache_read=no_cache_read,
+                        no_cache_write=no_cache_write
+                    )
+            
+            # Default to ENABLED if no cache mode specified
+            if cache_mode is None:
+                cache_mode = CacheMode.ENABLED
+
+            # Create cache context
+            cache_context = CacheContext(url, cache_mode, self.always_bypass_cache)
+
             extraction_strategy = extraction_strategy or NoExtractionStrategy()
             extraction_strategy.verbose = verbose
             if not isinstance(extraction_strategy, ExtractionStrategy):
@@ -83,60 +220,126 @@ class AsyncWebCrawler:
             word_count_threshold = max(word_count_threshold, MIN_WORD_THRESHOLD)
 
             async_response: AsyncCrawlResponse = None
-            cached = None
+            cached_result = None
             screenshot_data = None
             extracted_content = None
-            if not bypass_cache and not self.always_by_pass_cache:
-                cached = await async_db_manager.aget_cached_url(url)
-
-            if kwargs.get("warmup", True) and not self.ready:
-                return None
-
-            if cached:
-                html = sanitize_input_encode(cached[1])
-                extracted_content = sanitize_input_encode(cached[4])
+            
+            start_time = time.perf_counter()
+            
+            # Try to get cached result if appropriate
+            if cache_context.should_read():
+                cached_result = await async_db_manager.aget_cached_url(url)
+                        
+            if cached_result:
+                html = sanitize_input_encode(cached_result.html)
+                extracted_content = sanitize_input_encode(cached_result.extracted_content or "")
                 if screenshot:
-                    screenshot_data = cached[9]
+                    screenshot_data = cached_result.screenshot
                     if not screenshot_data:
-                        cached = None
+                        cached_result = None
+                # if verbose:
+                #     print(f"{Fore.BLUE}{self.tag_format('FETCH')} {self.log_icons['FETCH']} Cache hit for {cache_context.display_url} | Status: {Fore.GREEN if bool(html) else Fore.RED}{bool(html)}{Style.RESET_ALL} | Time: {time.perf_counter() - start_time:.2f}s")
+                self.logger.url_status(
+                        url=cache_context.display_url,
+                        success=bool(html),
+                        timing=time.perf_counter() - start_time,
+                        tag="FETCH"
+                    )                    
 
-            if not cached or not html:
-                t1 = time.time()
+
+            # Fetch fresh content if needed
+            if not cached_result or not html:
+                t1 = time.perf_counter()
+                
                 if user_agent:
                     self.crawler_strategy.update_user_agent(user_agent)
-                async_response: AsyncCrawlResponse = await self.crawler_strategy.crawl(url, screenshot=screenshot, **kwargs)
+                async_response: AsyncCrawlResponse = await self.crawler_strategy.crawl(
+                    url, 
+                    screenshot=screenshot, 
+                    **kwargs
+                )
                 html = sanitize_input_encode(async_response.html)
                 screenshot_data = async_response.screenshot
-                t2 = time.time()
-                if verbose:
-                    print(
-                        f"[LOG] 🚀 Crawling done for {url}, success: {bool(html)}, time taken: {t2 - t1:.2f} seconds"
-                    )
+                t2 = time.perf_counter()
+                self.logger.url_status(
+                    url=cache_context.display_url,
+                    success=bool(html),
+                    timing=t2 - t1,
+                    tag="FETCH"
+                )
+                # if verbose:
+                #     print(f"{Fore.BLUE}{self.tag_format('FETCH')} {self.log_icons['FETCH']} Live fetch for {cache_context.display_url}... | Status: {Fore.GREEN if bool(html) else Fore.RED}{bool(html)}{Style.RESET_ALL} | Time: {t2 - t1:.2f}s")
 
+            # Process the HTML content
             crawl_result = await self.aprocess_html(
-                url,
-                html,
-                extracted_content,
-                word_count_threshold,
-                extraction_strategy,
-                chunking_strategy,
-                css_selector,
-                screenshot_data,
-                verbose,
-                bool(cached),
+                url=url,
+                html=html,
+                extracted_content=extracted_content,
+                word_count_threshold=word_count_threshold,
+                extraction_strategy=extraction_strategy,
+                chunking_strategy=chunking_strategy,
+                content_filter=content_filter,
+                css_selector=css_selector,
+                screenshot=screenshot_data,
+                verbose=verbose,
+                is_cached=bool(cached_result),
                 async_response=async_response,
+                is_web_url=cache_context.is_web_url,
+                is_local_file=cache_context.is_local_file,
+                is_raw_html=cache_context.is_raw_html,
                 **kwargs,
             )
-            crawl_result.status_code = async_response.status_code if async_response else 200
-            crawl_result.response_headers = async_response.response_headers if async_response else {}
+            
+            # Set response data
+            if async_response:
+                crawl_result.status_code = async_response.status_code
+                crawl_result.response_headers = async_response.response_headers
+                crawl_result.downloaded_files = async_response.downloaded_files
+            else:
+                crawl_result.status_code = 200
+                crawl_result.response_headers = cached_result.response_headers if cached_result else {}
+
             crawl_result.success = bool(html)
             crawl_result.session_id = kwargs.get("session_id", None)
+
+            # if verbose:
+            #     print(f"{Fore.GREEN}{self.tag_format('COMPLETE')} {self.log_icons['COMPLETE']} {cache_context.display_url[:URL_LOG_SHORTEN_LENGTH]}... | Status: {Fore.GREEN if crawl_result.success else Fore.RED}{crawl_result.success} | {Fore.YELLOW}Total: {time.perf_counter() - start_time:.2f}s{Style.RESET_ALL}")
+            self.logger.success(
+                    message="{url:.50}... | Status: {status} | Total: {timing}",
+                    tag="COMPLETE",
+                    params={
+                        "url": cache_context.display_url,
+                        "status": crawl_result.success,
+                        "timing": f"{time.perf_counter() - start_time:.2f}s"
+                    },
+                    colors={
+                        "status": Fore.GREEN if crawl_result.success else Fore.RED,
+                        "timing": Fore.YELLOW
+                    }
+                )
+
+            # Update cache if appropriate
+            if cache_context.should_write() and not bool(cached_result):
+                await async_db_manager.acache_url(crawl_result)
+
             return crawl_result
+        
         except Exception as e:
             if not hasattr(e, "msg"):
                 e.msg = str(e)
-            print(f"[ERROR] 🚫 arun(): Failed to crawl {url}, error: {e.msg}")
-            return CrawlResult(url=url, html="", markdown = f"[ERROR] 🚫 arun(): Failed to crawl {url}, error: {e.msg}", success=False, error_message=e.msg)
+            # print(f"{Fore.RED}{self.tag_format('ERROR')} {self.log_icons['ERROR']} Failed to crawl {cache_context.display_url[:URL_LOG_SHORTEN_LENGTH]}... | {e.msg}{Style.RESET_ALL}")
+            
+            self.logger.error_status(
+                url=cache_context.display_url,
+                error=create_box_message(e.msg, type = "error"),
+                tag="ERROR"
+            )            
+            return CrawlResult(
+                url=url, 
+                html="", 
+                success=False, 
+                error_message=e.msg
+            )
 
     async def arun_many(
         self,
@@ -144,6 +347,9 @@ class AsyncWebCrawler:
         word_count_threshold=MIN_WORD_THRESHOLD,
         extraction_strategy: ExtractionStrategy = None,
         chunking_strategy: ChunkingStrategy = RegexChunking(),
+        content_filter: RelevantContentFilter = None,
+        cache_mode: Optional[CacheMode] = None,
+        # Deprecated parameters
         bypass_cache: bool = False,
         css_selector: str = None,
         screenshot: bool = False,
@@ -151,22 +357,101 @@ class AsyncWebCrawler:
         verbose=True,
         **kwargs,
     ) -> List[CrawlResult]:
-        tasks = [
-            self.arun(
-                url,
-                word_count_threshold,
-                extraction_strategy,
-                chunking_strategy,
-                bypass_cache,
-                css_selector,
-                screenshot,
-                user_agent,
-                verbose,
-                **kwargs
-            )
-            for url in urls
-        ]
-        return await asyncio.gather(*tasks)
+        """
+        Runs the crawler for multiple URLs concurrently.
+
+        Migration from legacy parameters:
+            Old way (deprecated):
+                results = await crawler.arun_many(urls, bypass_cache=True)
+            
+            New way:
+                results = await crawler.arun_many(urls, cache_mode=CacheMode.BYPASS)
+
+        Args:
+            urls: List of URLs to crawl
+            cache_mode: Cache behavior control (recommended)
+            [other parameters same as arun()]
+
+        Returns:
+            List[CrawlResult]: Results for each URL
+        """
+        if bypass_cache:
+            if kwargs.get("warning", True):
+                warnings.warn(
+                    "'bypass_cache' is deprecated and will be removed in version X.X.X. "
+                    "Use 'cache_mode=CacheMode.BYPASS' instead. "
+                    "Pass warning=False to suppress this warning.",
+                    DeprecationWarning,
+                    stacklevel=2
+                )
+            if cache_mode is None:
+                cache_mode = CacheMode.BYPASS
+
+        semaphore_count = kwargs.get('semaphore_count', 10)
+        semaphore = asyncio.Semaphore(semaphore_count)
+
+        async def crawl_with_semaphore(url):
+            domain = urlparse(url).netloc
+            current_time = time.time()
+            
+            # print(f"{Fore.LIGHTBLACK_EX}{self.tag_format('PARALLEL')} Started task for {url[:50]}...{Style.RESET_ALL}")
+            self.logger.debug(
+                message="Started task for {url:.50}...",
+                tag="PARALLEL",
+                params={"url": url}
+            )            
+            
+            # Get delay settings from kwargs or use defaults
+            mean_delay = kwargs.get('mean_delay', 0.1)  # 0.5 seconds default mean delay
+            max_range = kwargs.get('max_range', 0.3)    # 1 seconds default max additional delay
+            
+            # Check if we need to wait
+            if domain in self._domain_last_hit:
+                time_since_last = current_time - self._domain_last_hit[domain]
+                if time_since_last < mean_delay:
+                    delay = mean_delay + random.uniform(0, max_range)
+                    await asyncio.sleep(delay)
+            
+            # Update last hit time
+            self._domain_last_hit[domain] = current_time    
+                    
+            async with semaphore:
+                return await self.arun(
+                    url,
+                    word_count_threshold=word_count_threshold,
+                    extraction_strategy=extraction_strategy,
+                    chunking_strategy=chunking_strategy,
+                    content_filter=content_filter,
+                    cache_mode=cache_mode,
+                    css_selector=css_selector,
+                    screenshot=screenshot,
+                    user_agent=user_agent,
+                    verbose=verbose,
+                    **kwargs,
+                )
+
+        # Print start message
+        # print(f"{Fore.CYAN}{self.tag_format('INIT')} {self.log_icons['INIT']} Starting concurrent crawling for {len(urls)} URLs...{Style.RESET_ALL}")
+        self.logger.info(
+            message="Starting concurrent crawling for {count} URLs...",
+            tag="INIT",
+            params={"count": len(urls)}
+        )        
+        start_time = time.perf_counter()
+        tasks = [crawl_with_semaphore(url) for url in urls]
+        results = await asyncio.gather(*tasks, return_exceptions=True)
+        end_time = time.perf_counter()
+        # print(f"{Fore.YELLOW}{self.tag_format('COMPLETE')} {self.log_icons['COMPLETE']} Concurrent crawling completed for {len(urls)} URLs | Total time: {end_time - start_time:.2f}s{Style.RESET_ALL}")
+        self.logger.success(
+            message="Concurrent crawling completed for {count} URLs | " + Fore.YELLOW + " Total time: {timing}" + Style.RESET_ALL,
+            tag="COMPLETE",
+            params={
+                "count": len(urls),
+                "timing": f"{end_time - start_time:.2f}s"
+            },
+            colors={"timing": Fore.YELLOW}
+        )        
+        return [result if not isinstance(result, Exception) else str(result) for result in results]
 
 
     async def aprocess_html(
@@ -177,33 +462,30 @@ class AsyncWebCrawler:
         word_count_threshold: int,
         extraction_strategy: ExtractionStrategy,
         chunking_strategy: ChunkingStrategy,
+        content_filter: RelevantContentFilter,
         css_selector: str,
         screenshot: str,
         verbose: bool,
-        is_cached: bool,
         **kwargs,
     ) -> CrawlResult:
-        t = time.time()
         # Extract content from HTML
         try:
-            t1 = time.time()
-            scrapping_strategy = WebScrappingStrategy()
+            _url = url if not kwargs.get("is_raw_html", False) else "Raw HTML"
+            t1 = time.perf_counter()
+            scrapping_strategy = WebScrapingStrategy()
             # result = await scrapping_strategy.ascrap(
             result = scrapping_strategy.scrap(
                 url,
                 html,
                 word_count_threshold=word_count_threshold,
                 css_selector=css_selector,
-                only_text=kwargs.get("only_text", False),
-                image_description_min_word_threshold=kwargs.get(
+                only_text=kwargs.pop("only_text", False),
+                image_description_min_word_threshold=kwargs.pop(
                     "image_description_min_word_threshold", IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD
                 ),
+                content_filter = content_filter,
                 **kwargs,
             )
-            if verbose:
-                print(
-                    f"[LOG] 🚀 Content extracted for {url}, success: True, time taken: {time.time() - t1:.2f} seconds"
-                )
 
             if result is None:
                 raise ValueError(f"Process HTML, Failed to extract content from the website: {url}")
@@ -212,6 +494,8 @@ class AsyncWebCrawler:
         except Exception as e:
             raise ValueError(f"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}")
 
+        markdown_v2: MarkdownGenerationResult = result.get("markdown_v2", None)
+        
         cleaned_html = sanitize_input_encode(result.get("cleaned_html", ""))
         markdown = sanitize_input_encode(result.get("markdown", ""))
         fit_markdown = sanitize_input_encode(result.get("fit_markdown", ""))
@@ -219,13 +503,21 @@ class AsyncWebCrawler:
         media = result.get("media", [])
         links = result.get("links", [])
         metadata = result.get("metadata", {})
+        
+        # if verbose:
+        #     print(f"{Fore.MAGENTA}{self.tag_format('SCRAPE')} {self.log_icons['SCRAPE']} Processed {_url[:URL_LOG_SHORTEN_LENGTH]}...{Style.RESET_ALL} | Time: {int((time.perf_counter() - t1) * 1000)}ms")
+        self.logger.info(
+            message="Processed {url:.50}... | Time: {timing}ms",
+            tag="SCRAPE",
+            params={
+                "url": _url,
+                "timing": int((time.perf_counter() - t1) * 1000)
+            }
+        )
 
-        if extracted_content is None and extraction_strategy and chunking_strategy:
-            if verbose:
-                print(
-                    f"[LOG] 🔥 Extracting semantic blocks for {url}, Strategy: {self.__class__.__name__}"
-                )
 
+        if extracted_content is None and extraction_strategy and chunking_strategy and not isinstance(extraction_strategy, NoExtractionStrategy):
+            t1 = time.perf_counter()
             # Check if extraction strategy is type of JsonCssExtractionStrategy
             if isinstance(extraction_strategy, JsonCssExtractionStrategy) or isinstance(extraction_strategy, JsonCssExtractionStrategy):
                 extraction_strategy.verbose = verbose
@@ -235,32 +527,28 @@ class AsyncWebCrawler:
                 sections = chunking_strategy.chunk(markdown)
                 extracted_content = extraction_strategy.run(url, sections)
                 extracted_content = json.dumps(extracted_content, indent=4, default=str, ensure_ascii=False)
-
-        if verbose:
-            print(
-                f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t:.2f} seconds."
+            # if verbose:
+                # print(f"{Fore.YELLOW}{self.tag_format('EXTRACT')} {self.log_icons['EXTRACT']} Completed for {_url[:URL_LOG_SHORTEN_LENGTH]}...{Style.RESET_ALL} | Time: {time.perf_counter() - t1:.2f}s{Style.RESET_ALL}")
+            self.logger.info(
+                message="Completed for {url:.50}... | Time: {timing}s",
+                tag="EXTRACT",
+                params={
+                    "url": _url,
+                    "timing": time.perf_counter() - t1
+                }
             )
 
         screenshot = None if not screenshot else screenshot
-
-        if not is_cached:
-            await async_db_manager.acache_url(
-                url,
-                html,
-                cleaned_html,
-                markdown,
-                extracted_content,
-                True,
-                json.dumps(media),
-                json.dumps(links),
-                json.dumps(metadata),
-                screenshot=screenshot,
-            )
-
+        
+        
+        if kwargs.get("prettiify", False):
+            cleaned_html = fast_format_html(cleaned_html)
+        
         return CrawlResult(
             url=url,
             html=html,
-            cleaned_html=format_html(cleaned_html),
+            cleaned_html=cleaned_html,
+            markdown_v2=markdown_v2,
             markdown=markdown,
             fit_markdown=fit_markdown,
             fit_html= fit_html,
@@ -274,10 +562,15 @@ class AsyncWebCrawler:
         )
 
     async def aclear_cache(self):
-        await async_db_manager.aclear_db()
+        """Clear the cache database."""
+        await async_db_manager.cleanup()
 
     async def aflush_cache(self):
+        """Flush the cache database."""
         await async_db_manager.aflush_db()
 
     async def aget_cache_size(self):
+        """Get the total number of cached items."""
         return await async_db_manager.aget_total_count()
+
+

crawl4ai/cache_context.py

@@ -0,0 +1,79 @@
+from enum import Enum
+
+
+class CacheMode(Enum):
+    """
+    Defines the caching behavior for web crawling operations.
+    
+    Modes:
+    - ENABLED: Normal caching behavior (read and write)
+    - DISABLED: No caching at all
+    - READ_ONLY: Only read from cache, don't write
+    - WRITE_ONLY: Only write to cache, don't read
+    - BYPASS: Bypass cache for this operation
+    """
+    ENABLED = "enabled"
+    DISABLED = "disabled"
+    READ_ONLY = "read_only"
+    WRITE_ONLY = "write_only"
+    BYPASS = "bypass"
+
+
+class CacheContext:
+    """
+    Encapsulates cache-related decisions and URL handling.
+    
+    This class centralizes all cache-related logic and URL type checking,
+    making the caching behavior more predictable and maintainable.
+    """
+    def __init__(self, url: str, cache_mode: CacheMode, always_bypass: bool = False):
+        self.url = url
+        self.cache_mode = cache_mode
+        self.always_bypass = always_bypass
+        self.is_cacheable = url.startswith(('http://', 'https://', 'file://'))
+        self.is_web_url = url.startswith(('http://', 'https://'))
+        self.is_local_file = url.startswith("file://")
+        self.is_raw_html = url.startswith("raw:")
+        self._url_display = url if not self.is_raw_html else "Raw HTML"
+    
+    def should_read(self) -> bool:
+        """Determines if cache should be read based on context."""
+        if self.always_bypass or not self.is_cacheable:
+            return False
+        return self.cache_mode in [CacheMode.ENABLED, CacheMode.READ_ONLY]
+    
+    def should_write(self) -> bool:
+        """Determines if cache should be written based on context."""
+        if self.always_bypass or not self.is_cacheable:
+            return False
+        return self.cache_mode in [CacheMode.ENABLED, CacheMode.WRITE_ONLY]
+    
+    @property
+    def display_url(self) -> str:
+        """Returns the URL in display format."""
+        return self._url_display
+
+
+def _legacy_to_cache_mode(
+    disable_cache: bool = False,
+    bypass_cache: bool = False,
+    no_cache_read: bool = False,
+    no_cache_write: bool = False
+) -> CacheMode:
+    """
+    Converts legacy cache parameters to the new CacheMode enum.
+    
+    This is an internal function to help transition from the old boolean flags
+    to the new CacheMode system.
+    """
+    if disable_cache:
+        return CacheMode.DISABLED
+    if bypass_cache:
+        return CacheMode.BYPASS
+    if no_cache_read and no_cache_write:
+        return CacheMode.DISABLED
+    if no_cache_read:
+        return CacheMode.WRITE_ONLY
+    if no_cache_write:
+        return CacheMode.READ_ONLY
+    return CacheMode.ENABLED

crawl4ai/config.py

@@ -51,3 +51,9 @@ SOCIAL_MEDIA_DOMAINS = [
 # If image format is in jpg, png or webp
 # If image is in the first half of the total images extracted from the page
 IMAGE_SCORE_THRESHOLD = 2
+
+MAX_METRICS_HISTORY = 1000
+
+NEED_MIGRATION = True
+URL_LOG_SHORTEN_LENGTH = 30
+SHOW_DEPRECATION_WARNINGS = True

crawl4ai/content_cleaning_strategy.py

@@ -1,196 +0,0 @@
-from bs4 import BeautifulSoup, Tag
-import re
-from typing import Optional
-
-class ContentCleaningStrategy:
-    def __init__(self):
-        # Precompile regex patterns for performance
-        self.negative_patterns = re.compile(r'nav|footer|header|sidebar|ads|comment', re.I)
-        self.positive_patterns = re.compile(r'content|article|main|post', re.I)
-        self.priority_tags = {'article', 'main', 'section', 'div'}
-        self.non_content_tags = {'nav', 'footer', 'header', 'aside'}
-        # Thresholds
-        self.text_density_threshold = 9.0
-        self.min_word_count = 50
-        self.link_density_threshold = 0.2
-        self.max_dom_depth = 10  # To prevent excessive DOM traversal
-
-    def clean(self, clean_html: str) -> str:
-        """
-        Main function that takes cleaned HTML and returns super cleaned HTML.
-
-        Args:
-            clean_html (str): The cleaned HTML content.
-
-        Returns:
-            str: The super cleaned HTML containing only the main content.
-        """
-        try:
-            if not clean_html or not isinstance(clean_html, str):
-                return ''
-            soup = BeautifulSoup(clean_html, 'html.parser')
-            main_content = self.extract_main_content(soup)
-            if main_content:
-                super_clean_element = self.clean_element(main_content)
-                return str(super_clean_element)
-            else:
-                return ''
-        except Exception:
-            # Handle exceptions silently or log them as needed
-            return ''
-
-    def extract_main_content(self, soup: BeautifulSoup) -> Optional[Tag]:
-        """
-        Identifies and extracts the main content element from the HTML.
-
-        Args:
-            soup (BeautifulSoup): The parsed HTML soup.
-
-        Returns:
-            Optional[Tag]: The Tag object containing the main content, or None if not found.
-        """
-        candidates = []
-        for element in soup.find_all(self.priority_tags):
-            if self.is_non_content_tag(element):
-                continue
-            if self.has_negative_class_id(element):
-                continue
-            score = self.calculate_content_score(element)
-            candidates.append((score, element))
-        
-        if not candidates:
-            return None
-
-        # Sort candidates by score in descending order
-        candidates.sort(key=lambda x: x[0], reverse=True)
-        # Select the element with the highest score
-        best_element = candidates[0][1]
-        return best_element
-
-    def calculate_content_score(self, element: Tag) -> float:
-        """
-        Calculates a score for an element based on various heuristics.
-
-        Args:
-            element (Tag): The HTML element to score.
-
-        Returns:
-            float: The content score of the element.
-        """
-        score = 0.0
-
-        if self.is_priority_tag(element):
-            score += 5.0
-        if self.has_positive_class_id(element):
-            score += 3.0
-        if self.has_negative_class_id(element):
-            score -= 3.0
-        if self.is_high_text_density(element):
-            score += 2.0
-        if self.is_low_link_density(element):
-            score += 2.0
-        if self.has_sufficient_content(element):
-            score += 2.0
-        if self.has_headings(element):
-            score += 3.0
-
-        dom_depth = self.calculate_dom_depth(element)
-        score += min(dom_depth, self.max_dom_depth) * 0.5  # Adjust weight as needed
-
-        return score
-
-    def is_priority_tag(self, element: Tag) -> bool:
-        """Checks if the element is a priority tag."""
-        return element.name in self.priority_tags
-
-    def is_non_content_tag(self, element: Tag) -> bool:
-        """Checks if the element is a non-content tag."""
-        return element.name in self.non_content_tags
-
-    def has_negative_class_id(self, element: Tag) -> bool:
-        """Checks if the element has negative indicators in its class or id."""
-        class_id = ' '.join(filter(None, [
-            self.get_attr_str(element.get('class')),
-            element.get('id', '')
-        ]))
-        return bool(self.negative_patterns.search(class_id))
-
-    def has_positive_class_id(self, element: Tag) -> bool:
-        """Checks if the element has positive indicators in its class or id."""
-        class_id = ' '.join(filter(None, [
-            self.get_attr_str(element.get('class')),
-            element.get('id', '')
-        ]))
-        return bool(self.positive_patterns.search(class_id))
-
-    @staticmethod
-    def get_attr_str(attr) -> str:
-        """Converts an attribute value to a string."""
-        if isinstance(attr, list):
-            return ' '.join(attr)
-        elif isinstance(attr, str):
-            return attr
-        else:
-            return ''
-
-    def is_high_text_density(self, element: Tag) -> bool:
-        """Determines if the element has high text density."""
-        text_density = self.calculate_text_density(element)
-        return text_density > self.text_density_threshold
-
-    def calculate_text_density(self, element: Tag) -> float:
-        """Calculates the text density of an element."""
-        text_length = len(element.get_text(strip=True))
-        tag_count = len(element.find_all())
-        tag_count = tag_count or 1  # Prevent division by zero
-        return text_length / tag_count
-
-    def is_low_link_density(self, element: Tag) -> bool:
-        """Determines if the element has low link density."""
-        link_density = self.calculate_link_density(element)
-        return link_density < self.link_density_threshold
-
-    def calculate_link_density(self, element: Tag) -> float:
-        """Calculates the link density of an element."""
-        text = element.get_text(strip=True)
-        if not text:
-            return 0.0
-        link_text = ' '.join(a.get_text(strip=True) for a in element.find_all('a'))
-        return len(link_text) / len(text) if text else 0.0
-
-    def has_sufficient_content(self, element: Tag) -> bool:
-        """Checks if the element has sufficient word count."""
-        word_count = len(element.get_text(strip=True).split())
-        return word_count >= self.min_word_count
-
-    def calculate_dom_depth(self, element: Tag) -> int:
-        """Calculates the depth of an element in the DOM tree."""
-        depth = 0
-        current_element = element
-        while current_element.parent and depth < self.max_dom_depth:
-            depth += 1
-            current_element = current_element.parent
-        return depth
-
-    def has_headings(self, element: Tag) -> bool:
-        """Checks if the element contains heading tags."""
-        return bool(element.find(['h1', 'h2', 'h3']))
-
-    def clean_element(self, element: Tag) -> Tag:
-        """
-        Cleans the selected element by removing unnecessary attributes and nested non-content elements.
-
-        Args:
-            element (Tag): The HTML element to clean.
-
-        Returns:
-            Tag: The cleaned HTML element.
-        """
-        for tag in element.find_all(['script', 'style', 'aside']):
-            tag.decompose()
-        for tag in element.find_all():
-            attrs = dict(tag.attrs)
-            for attr in attrs:
-                if attr in ['style', 'onclick', 'onmouseover', 'align', 'bgcolor']:
-                    del tag.attrs[attr]
-        return element

crawl4ai/content_filter_strategy.py

@@ -0,0 +1,502 @@
+import re
+from bs4 import BeautifulSoup, Tag
+from typing import List, Tuple, Dict
+from rank_bm25 import BM25Okapi
+from time import perf_counter
+from collections import deque
+from bs4 import BeautifulSoup, NavigableString, Tag
+from .utils import clean_tokens
+from abc import ABC, abstractmethod
+
+from snowballstemmer import stemmer
+
+
+# import regex
+# def tokenize_text(text):
+#     # Regular expression to match words or CJK (Chinese, Japanese, Korean) characters
+#     pattern = r'\p{L}+|\p{N}+|[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}ー]|[\p{P}]'
+#     return regex.findall(pattern, text)
+
+# from nltk.stem import PorterStemmer
+# ps = PorterStemmer()
+class RelevantContentFilter(ABC):
+    def __init__(self, user_query: str = None):
+        self.user_query = user_query
+        self.included_tags = {
+            # Primary structure
+            'article', 'main', 'section', 'div', 
+            # List structures
+            'ul', 'ol', 'li', 'dl', 'dt', 'dd',
+            # Text content
+            'p', 'span', 'blockquote', 'pre', 'code',
+            # Headers
+            'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+            # Tables
+            'table', 'thead', 'tbody', 'tr', 'td', 'th',
+            # Other semantic elements
+            'figure', 'figcaption', 'details', 'summary',
+            # Text formatting
+            'em', 'strong', 'b', 'i', 'mark', 'small',
+            # Rich content
+            'time', 'address', 'cite', 'q'
+        }
+        self.excluded_tags = {
+            'nav', 'footer', 'header', 'aside', 'script',
+            'style', 'form', 'iframe', 'noscript'
+        }
+        self.header_tags = {'h1', 'h2', 'h3', 'h4', 'h5', 'h6'}
+        self.negative_patterns = re.compile(
+            r'nav|footer|header|sidebar|ads|comment|promo|advert|social|share',
+            re.I
+        )
+        self.min_word_count = 2
+        
+    @abstractmethod
+    def filter_content(self, html: str) -> List[str]:
+        """Abstract method to be implemented by specific filtering strategies"""
+        pass
+    
+    def extract_page_query(self, soup: BeautifulSoup, body: Tag) -> str:
+        """Common method to extract page metadata with fallbacks"""
+        if self.user_query:
+            return self.user_query
+
+        query_parts = []
+        
+        # Title
+        try:
+            title = soup.title.string
+            if title:
+                query_parts.append(title)
+        except Exception:
+            pass
+
+        if soup.find('h1'):
+            query_parts.append(soup.find('h1').get_text())
+            
+        # Meta tags
+        temp = ""
+        for meta_name in ['keywords', 'description']:
+            meta = soup.find('meta', attrs={'name': meta_name})
+            if meta and meta.get('content'):
+                query_parts.append(meta['content'])
+                temp += meta['content']
+                
+        # If still empty, grab first significant paragraph
+        if not temp:
+            # Find the first tag P thatits text contains more than 50 characters
+            for p in body.find_all('p'):
+                if len(p.get_text()) > 150:
+                    query_parts.append(p.get_text()[:150])
+                    break        
+                                
+        return ' '.join(filter(None, query_parts))
+
+
+    def extract_text_chunks(self, body: Tag, min_word_threshold: int = None) -> List[Tuple[str, str]]:
+        """
+        Extracts text chunks from a BeautifulSoup body element while preserving order.
+        Returns list of tuples (text, tag_name) for classification.
+        
+        Args:
+            body: BeautifulSoup Tag object representing the body element
+            
+        Returns:
+            List of (text, tag_name) tuples
+        """
+        # Tags to ignore - inline elements that shouldn't break text flow
+        INLINE_TAGS = {
+            'a', 'abbr', 'acronym', 'b', 'bdo', 'big', 'br', 'button', 'cite', 'code',
+            'dfn', 'em', 'i', 'img', 'input', 'kbd', 'label', 'map', 'object', 'q',
+            'samp', 'script', 'select', 'small', 'span', 'strong', 'sub', 'sup',
+            'textarea', 'time', 'tt', 'var'
+        }
+        
+        # Tags that typically contain meaningful headers
+        HEADER_TAGS = {'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'header'}
+        
+        chunks = []
+        current_text = []
+        chunk_index = 0
+    
+        def should_break_chunk(tag: Tag) -> bool:
+            """Determine if a tag should cause a break in the current text chunk"""
+            return (
+                tag.name not in INLINE_TAGS
+                and not (tag.name == 'p' and len(current_text) == 0)
+            )
+        
+        # Use deque for efficient push/pop operations
+        stack = deque([(body, False)])
+        
+        while stack:
+            element, visited = stack.pop()
+            
+            if visited:
+                # End of block element - flush accumulated text
+                if current_text and should_break_chunk(element):
+                    text = ' '.join(''.join(current_text).split())
+                    if text:
+                        tag_type = 'header' if element.name in HEADER_TAGS else 'content'
+                        chunks.append((chunk_index, text, tag_type, element))
+                        chunk_index += 1
+                    current_text = []
+                continue
+                
+            if isinstance(element, NavigableString):
+                if str(element).strip():
+                    current_text.append(str(element).strip())
+                continue
+                
+            # Pre-allocate children to avoid multiple list operations
+            children = list(element.children)
+            if not children:
+                continue
+                
+            # Mark block for revisit after processing children
+            stack.append((element, True))
+            
+            # Add children in reverse order for correct processing
+            for child in reversed(children):
+                if isinstance(child, (Tag, NavigableString)):
+                    stack.append((child, False))
+        
+        # Handle any remaining text
+        if current_text:
+            text = ' '.join(''.join(current_text).split())
+            if text:
+                chunks.append((chunk_index, text, 'content', body))
+        
+        if min_word_threshold:
+            chunks = [chunk for chunk in chunks if len(chunk[1].split()) >= min_word_threshold]
+        
+        return chunks    
+    
+
+    def extract_text_chunks1(self, soup: BeautifulSoup) -> List[Tuple[int, str, Tag]]:
+        """Common method for extracting text chunks"""
+        _text_cache = {}
+        def fast_text(element: Tag) -> str:
+            elem_id = id(element)
+            if elem_id in _text_cache:
+                return _text_cache[elem_id]
+            texts = []
+            for content in element.contents:
+                if isinstance(content, str):
+                    text = content.strip()
+                    if text:
+                        texts.append(text)
+            result = ' '.join(texts)
+            _text_cache[elem_id] = result
+            return result
+        
+        candidates = []
+        index = 0
+        
+        def dfs(element):
+            nonlocal index
+            if isinstance(element, Tag):
+                if element.name in self.included_tags:
+                    if not self.is_excluded(element):
+                        text = fast_text(element)
+                        word_count = len(text.split())
+                        
+                        # Headers pass through with adjusted minimum
+                        if element.name in self.header_tags:
+                            if word_count >= 3:  # Minimal sanity check for headers
+                                candidates.append((index, text, element))
+                                index += 1
+                        # Regular content uses standard minimum
+                        elif word_count >= self.min_word_count:
+                            candidates.append((index, text, element))
+                            index += 1
+                            
+                for child in element.children:
+                    dfs(child)
+
+        dfs(soup.body if soup.body else soup)
+        return candidates
+
+    def is_excluded(self, tag: Tag) -> bool:
+        """Common method for exclusion logic"""
+        if tag.name in self.excluded_tags:
+            return True
+        class_id = ' '.join(filter(None, [
+            ' '.join(tag.get('class', [])),
+            tag.get('id', '')
+        ]))
+        return bool(self.negative_patterns.search(class_id))
+
+    def clean_element(self, tag: Tag) -> str:
+        """Common method for cleaning HTML elements with minimal overhead"""
+        if not tag or not isinstance(tag, Tag):
+            return ""
+            
+        unwanted_tags = {'script', 'style', 'aside', 'form', 'iframe', 'noscript'}
+        unwanted_attrs = {'style', 'onclick', 'onmouseover', 'align', 'bgcolor', 'class', 'id'}
+        
+        # Use string builder pattern for better performance
+        builder = []
+        
+        def render_tag(elem):
+            if not isinstance(elem, Tag):
+                if isinstance(elem, str):
+                    builder.append(elem.strip())
+                return
+                
+            if elem.name in unwanted_tags:
+                return
+                
+            # Start tag
+            builder.append(f'<{elem.name}')
+            
+            # Add cleaned attributes
+            attrs = {k: v for k, v in elem.attrs.items() if k not in unwanted_attrs}
+            for key, value in attrs.items():
+                builder.append(f' {key}="{value}"')
+                
+            builder.append('>')
+            
+            # Process children
+            for child in elem.children:
+                render_tag(child)
+                
+            # Close tag
+            builder.append(f'</{elem.name}>')
+        
+        try:
+            render_tag(tag)
+            return ''.join(builder)
+        except Exception:
+            return str(tag)  # Fallback to original if anything fails
+
+class BM25ContentFilter(RelevantContentFilter):
+    def __init__(self, user_query: str = None, bm25_threshold: float = 1.0, language: str = 'english'):
+        super().__init__(user_query=user_query)
+        self.bm25_threshold = bm25_threshold
+        self.priority_tags = {
+            'h1': 5.0,
+            'h2': 4.0,
+            'h3': 3.0,
+            'title': 4.0,
+            'strong': 2.0,
+            'b': 1.5,
+            'em': 1.5,
+            'blockquote': 2.0,
+            'code': 2.0,
+            'pre': 1.5,
+            'th': 1.5,  # Table headers
+        }
+        self.stemmer = stemmer(language)
+
+    def filter_content(self, html: str, min_word_threshold: int = None) -> List[str]:
+        """Implements content filtering using BM25 algorithm with priority tag handling"""
+        if not html or not isinstance(html, str):
+            return []
+
+        soup = BeautifulSoup(html, 'lxml')
+        
+        # Check if body is present
+        if not soup.body:
+            # Wrap in body tag if missing
+            soup = BeautifulSoup(f'<body>{html}</body>', 'lxml')        
+        body = soup.find('body')
+        
+        query = self.extract_page_query(soup, body)
+        
+        if not query:
+            return []
+            # return [self.clean_element(soup)]
+            
+        candidates = self.extract_text_chunks(body, min_word_threshold)
+
+        if not candidates:
+            return []
+
+        # Tokenize corpus
+        # tokenized_corpus = [chunk.lower().split() for _, chunk, _, _ in candidates]
+        # tokenized_query = query.lower().split()
+                
+        # tokenized_corpus = [[ps.stem(word) for word in chunk.lower().split()] 
+        #                 for _, chunk, _, _ in candidates]
+        # tokenized_query = [ps.stem(word) for word in query.lower().split()]        
+        
+        tokenized_corpus = [[self.stemmer.stemWord(word) for word in chunk.lower().split()] 
+                   for _, chunk, _, _ in candidates]
+        tokenized_query = [self.stemmer.stemWord(word) for word in query.lower().split()]
+
+        # tokenized_corpus = [[self.stemmer.stemWord(word) for word in tokenize_text(chunk.lower())] 
+        #            for _, chunk, _, _ in candidates]
+        # tokenized_query = [self.stemmer.stemWord(word) for word in tokenize_text(query.lower())]
+
+        # Clean from stop words and noise
+        tokenized_corpus = [clean_tokens(tokens) for tokens in tokenized_corpus]
+        tokenized_query = clean_tokens(tokenized_query)
+
+        bm25 = BM25Okapi(tokenized_corpus)
+        scores = bm25.get_scores(tokenized_query)
+
+        # Adjust scores with tag weights
+        adjusted_candidates = []
+        for score, (index, chunk, tag_type, tag) in zip(scores, candidates):
+            tag_weight = self.priority_tags.get(tag.name, 1.0)
+            adjusted_score = score * tag_weight
+            adjusted_candidates.append((adjusted_score, index, chunk, tag))
+
+        # Filter candidates by threshold
+        selected_candidates = [
+            (index, chunk, tag) for adjusted_score, index, chunk, tag in adjusted_candidates
+            if adjusted_score >= self.bm25_threshold
+        ]
+
+        if not selected_candidates:
+            return []
+
+        # Sort selected candidates by original document order
+        selected_candidates.sort(key=lambda x: x[0])
+
+        return [self.clean_element(tag) for _, _, tag in selected_candidates]
+
+
+class HeuristicContentFilter(RelevantContentFilter):
+    def __init__(self):
+        super().__init__()
+        # Weights for different heuristics
+        self.tag_weights = {
+            'article': 10,
+            'main': 8,
+            'section': 5,
+            'div': 3,
+            'p': 2,
+            'pre': 2,
+            'code': 2,
+            'blockquote': 2,
+            'li': 1,
+            'span': 1,
+        }
+        self.max_depth = 5  # Maximum depth from body to consider
+
+    def filter_content(self, html: str) -> List[str]:
+        """Implements heuristic content filtering without relying on a query."""
+        if not html or not isinstance(html, str):
+            return []
+
+        soup = BeautifulSoup(html, 'lxml')
+
+        # Ensure there is a body tag
+        if not soup.body:
+            soup = BeautifulSoup(f'<body>{html}</body>', 'lxml')
+        body = soup.body
+
+        # Extract candidate text chunks
+        candidates = self.extract_text_chunks(body)
+
+        if not candidates:
+            return []
+
+        # Score each candidate
+        scored_candidates = []
+        for index, text, tag_type, tag in candidates:
+            score = self.score_element(tag, text)
+            if score > 0:
+                scored_candidates.append((score, index, text, tag))
+
+        # Sort candidates by score and then by document order
+        scored_candidates.sort(key=lambda x: (-x[0], x[1]))
+
+        # Extract the top candidates (e.g., top 5)
+        top_candidates = scored_candidates[:5]  # Adjust the number as needed
+
+        # Sort the top candidates back to their original document order
+        top_candidates.sort(key=lambda x: x[1])
+
+        # Clean and return the content
+        return [self.clean_element(tag) for _, _, _, tag in top_candidates]
+
+    def score_element(self, tag: Tag, text: str) -> float:
+        """Compute a score for an element based on heuristics."""
+        if not text or not tag:
+            return 0
+
+        # Exclude unwanted tags
+        if self.is_excluded(tag):
+            return 0
+
+        # Text density
+        text_length = len(text.strip())
+        html_length = len(str(tag))
+        text_density = text_length / html_length if html_length > 0 else 0
+
+        # Link density
+        link_text_length = sum(len(a.get_text().strip()) for a in tag.find_all('a'))
+        link_density = link_text_length / text_length if text_length > 0 else 0
+
+        # Tag weight
+        tag_weight = self.tag_weights.get(tag.name, 1)
+
+        # Depth factor (prefer elements closer to the body tag)
+        depth = self.get_depth(tag)
+        depth_weight = max(self.max_depth - depth, 1) / self.max_depth
+
+        # Compute the final score
+        score = (text_density * tag_weight * depth_weight) / (1 + link_density)
+
+        return score
+
+    def get_depth(self, tag: Tag) -> int:
+        """Compute the depth of the tag from the body tag."""
+        depth = 0
+        current = tag
+        while current and current != current.parent and current.name != 'body':
+            current = current.parent
+            depth += 1
+        return depth
+
+    def extract_text_chunks(self, body: Tag) -> List[Tuple[int, str, str, Tag]]:
+        """
+        Extracts text chunks from the body element while preserving order.
+        Returns list of tuples (index, text, tag_type, tag) for scoring.
+        """
+        chunks = []
+        index = 0
+
+        def traverse(element):
+            nonlocal index
+            if isinstance(element, NavigableString):
+                return
+            if not isinstance(element, Tag):
+                return
+            if self.is_excluded(element):
+                return
+            # Only consider included tags
+            if element.name in self.included_tags:
+                text = element.get_text(separator=' ', strip=True)
+                if len(text.split()) >= self.min_word_count:
+                    tag_type = 'header' if element.name in self.header_tags else 'content'
+                    chunks.append((index, text, tag_type, element))
+                    index += 1
+                    # Do not traverse children of this element to prevent duplication
+                    return
+            for child in element.children:
+                traverse(child)
+
+        traverse(body)
+        return chunks
+
+    def is_excluded(self, tag: Tag) -> bool:
+        """Determine if a tag should be excluded based on heuristics."""
+        if tag.name in self.excluded_tags:
+            return True
+        class_id = ' '.join(filter(None, [
+            ' '.join(tag.get('class', [])),
+            tag.get('id', '')
+        ]))
+        if self.negative_patterns.search(class_id):
+            return True
+        # Exclude tags with high link density (e.g., navigation menus)
+        text = tag.get_text(separator=' ', strip=True)
+        link_text_length = sum(len(a.get_text(strip=True)) for a in tag.find_all('a'))
+        text_length = len(text)
+        if text_length > 0 and (link_text_length / text_length) > 0.5:
+            return True
+        return False

crawl4ai/content_scraping_strategy.py

@@ -1,5 +1,6 @@
+import re  # Point 1: Pre-Compile Regular Expressions
 from abc import ABC, abstractmethod
-from typing import Dict, Any
+from typing import Dict, Any, Optional
 from bs4 import BeautifulSoup
 from concurrent.futures import ThreadPoolExecutor
 import asyncio, requests, re, os
@@ -7,8 +8,10 @@ from .config import *
 from bs4 import element, NavigableString, Comment
 from urllib.parse import urljoin
 from requests.exceptions import InvalidSchema
-from .content_cleaning_strategy import ContentCleaningStrategy
-
+# from .content_cleaning_strategy import ContentCleaningStrategy
+from .content_filter_strategy import RelevantContentFilter, BM25ContentFilter#, HeuristicContentFilter
+from .markdown_generation_strategy import MarkdownGenerationStrategy, DefaultMarkdownGenerator
+from .models import MarkdownGenerationResult
 from .utils import (
     sanitize_input_encode,
     sanitize_html,
@@ -16,11 +19,43 @@ from .utils import (
     InvalidCSSSelectorError,
     CustomHTML2Text,
     normalize_url,
-    is_external_url
-    
+    is_external_url    
 )
+from .tools import profile_and_time
 
-class ContentScrappingStrategy(ABC):
+# Pre-compile regular expressions for Open Graph and Twitter metadata
+OG_REGEX = re.compile(r'^og:')
+TWITTER_REGEX = re.compile(r'^twitter:')
+DIMENSION_REGEX = re.compile(r"(\d+)(\D*)")
+
+# Function to parse image height/width value and units
+def parse_dimension(dimension):
+    if dimension:
+        # match = re.match(r"(\d+)(\D*)", dimension)
+        match = DIMENSION_REGEX.match(dimension)
+        if match:
+            number = int(match.group(1))
+            unit = match.group(2) or 'px'  # Default unit is 'px' if not specified
+            return number, unit
+    return None, None
+
+# Fetch image file metadata to extract size and extension
+def fetch_image_file_size(img, base_url):
+    #If src is relative path construct full URL, if not it may be CDN URL
+    img_url = urljoin(base_url,img.get('src'))
+    try:
+        response = requests.head(img_url)
+        if response.status_code == 200:
+            return response.headers.get('Content-Length',None)
+        else:
+            print(f"Failed to retrieve file size for {img_url}")
+            return None
+    except InvalidSchema as e:
+        return None
+    finally:
+        return
+
+class ContentScrapingStrategy(ABC):
     @abstractmethod
     def scrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
         pass
@@ -29,21 +64,140 @@ class ContentScrappingStrategy(ABC):
     async def ascrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
         pass
 
-class WebScrappingStrategy(ContentScrappingStrategy):
+class WebScrapingStrategy(ContentScrapingStrategy):
+    def __init__(self, logger=None):
+        self.logger = logger
+
+    def _log(self, level, message, tag="SCRAPE", **kwargs):
+        """Helper method to safely use logger."""
+        if self.logger:
+            log_method = getattr(self.logger, level)
+            log_method(message=message, tag=tag, **kwargs)
+                
     def scrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
         return self._get_content_of_website_optimized(url, html, is_async=False, **kwargs)
 
     async def ascrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
         return await asyncio.to_thread(self._get_content_of_website_optimized, url, html, **kwargs)
 
+
+    def _generate_markdown_content(self, 
+                                 cleaned_html: str,
+                                 html: str,
+                                 url: str,
+                                 success: bool,
+                                 **kwargs) -> Dict[str, Any]:
+        """Generate markdown content using either new strategy or legacy method.
+        
+        Args:
+            cleaned_html: Sanitized HTML content
+            html: Original HTML content
+            url: Base URL of the page
+            success: Whether scraping was successful
+            **kwargs: Additional options including:
+                - markdown_generator: Optional[MarkdownGenerationStrategy]
+                - html2text: Dict[str, Any] options for HTML2Text
+                - content_filter: Optional[RelevantContentFilter]
+                - fit_markdown: bool
+                - fit_markdown_user_query: Optional[str]
+                - fit_markdown_bm25_threshold: float
+        
+        Returns:
+            Dict containing markdown content in various formats
+        """
+        markdown_generator: Optional[MarkdownGenerationStrategy] = kwargs.get('markdown_generator', DefaultMarkdownGenerator())
+        
+        if markdown_generator:
+            try:
+                if kwargs.get('fit_markdown', False) and not markdown_generator.content_filter:
+                        markdown_generator.content_filter = BM25ContentFilter(
+                            user_query=kwargs.get('fit_markdown_user_query', None),
+                            bm25_threshold=kwargs.get('fit_markdown_bm25_threshold', 1.0)
+                        )
+                
+                markdown_result: MarkdownGenerationResult = markdown_generator.generate_markdown(
+                    cleaned_html=cleaned_html,
+                    base_url=url,
+                    html2text_options=kwargs.get('html2text', {})
+                )
+                
+                help_message = """"""
+                
+                return {
+                    'markdown': markdown_result.raw_markdown,  
+                    'fit_markdown': markdown_result.fit_markdown,
+                    'fit_html': markdown_result.fit_html, 
+                    'markdown_v2': markdown_result
+                }
+            except Exception as e:
+                self._log('error',
+                    message="Error using new markdown generation strategy: {error}",
+                    tag="SCRAPE",
+                    params={"error": str(e)}
+                )
+                markdown_generator = None
+                return {
+                    'markdown': f"Error using new markdown generation strategy: {str(e)}",
+                    'fit_markdown': "Set flag 'fit_markdown' to True to get cleaned HTML content.",
+                    'fit_html': "Set flag 'fit_markdown' to True to get cleaned HTML content.",
+                    'markdown_v2': None                    
+                }
+
+        # Legacy method
+        h = CustomHTML2Text()
+        h.update_params(**kwargs.get('html2text', {}))            
+        markdown = h.handle(cleaned_html)
+        markdown = markdown.replace('    ```', '```')
+        
+        fit_markdown = "Set flag 'fit_markdown' to True to get cleaned HTML content."
+        fit_html = "Set flag 'fit_markdown' to True to get cleaned HTML content."
+        
+        if kwargs.get('content_filter', None) or kwargs.get('fit_markdown', False):
+            content_filter = kwargs.get('content_filter', None)
+            if not content_filter:
+                content_filter = BM25ContentFilter(
+                    user_query=kwargs.get('fit_markdown_user_query', None),
+                    bm25_threshold=kwargs.get('fit_markdown_bm25_threshold', 1.0)
+                )
+            fit_html = content_filter.filter_content(html)
+            fit_html = '\n'.join('<div>{}</div>'.format(s) for s in fit_html)
+            fit_markdown = h.handle(fit_html)
+
+        markdown_v2 = MarkdownGenerationResult(
+            raw_markdown=markdown,
+            markdown_with_citations=markdown,
+            references_markdown=markdown,
+            fit_markdown=fit_markdown
+        )
+        
+        return {
+            'markdown': markdown,
+            'fit_markdown': fit_markdown,
+            'fit_html': fit_html,
+            'markdown_v2' : markdown_v2
+        }
+
+
     def _get_content_of_website_optimized(self, url: str, html: str, word_count_threshold: int = MIN_WORD_THRESHOLD, css_selector: str = None, **kwargs) -> Dict[str, Any]:
         success = True
         if not html:
             return None
 
-        soup = BeautifulSoup(html, 'html.parser')
+        # soup = BeautifulSoup(html, 'html.parser')
+        soup = BeautifulSoup(html, 'lxml')
         body = soup.body
         
+        try:
+            meta = extract_metadata("", soup)
+        except Exception as e:
+            self._log('error', 
+                message="Error extracting metadata: {error}",
+                tag="SCRAPE",
+                params={"error": str(e)}
+            )            
+            # print('Error extracting metadata:', str(e))
+            meta = {}
+        
         
         image_description_min_word_threshold = kwargs.get('image_description_min_word_threshold', IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD)
 
@@ -86,7 +240,9 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                             return text_content
                 return None
 
-        def process_image(img, url, index, total_images):
+        def process_image_old(img, url, index, total_images):
+                   
+            
             #Check if an image has valid display and inside undesired html elements
             def is_valid_image(img, parent, parent_classes):
                 style = img.get('style', '')
@@ -102,32 +258,6 @@ class WebScrappingStrategy(ContentScrappingStrategy):
 
             #Score an image for it's usefulness
             def score_image_for_usefulness(img, base_url, index, images_count):
-                # Function to parse image height/width value and units
-                def parse_dimension(dimension):
-                    if dimension:
-                        match = re.match(r"(\d+)(\D*)", dimension)
-                        if match:
-                            number = int(match.group(1))
-                            unit = match.group(2) or 'px'  # Default unit is 'px' if not specified
-                            return number, unit
-                    return None, None
-
-                # Fetch image file metadata to extract size and extension
-                def fetch_image_file_size(img, base_url):
-                    #If src is relative path construct full URL, if not it may be CDN URL
-                    img_url = urljoin(base_url,img.get('src'))
-                    try:
-                        response = requests.head(img_url)
-                        if response.status_code == 200:
-                            return response.headers.get('Content-Length',None)
-                        else:
-                            print(f"Failed to retrieve file size for {img_url}")
-                            return None
-                    except InvalidSchema as e:
-                        return None
-                    finally:
-                        return
-
                 image_height = img.get('height')
                 height_value, height_unit = parse_dimension(image_height)
                 image_width =  img.get('width')
@@ -161,14 +291,14 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                     score+=1
                 return score
 
-            
-            
             if not is_valid_image(img, img.parent, img.parent.get('class', [])):
                 return None
+                
             score = score_image_for_usefulness(img, url, index, total_images)
-            if score <= IMAGE_SCORE_THRESHOLD:
+            if score <= kwargs.get('image_score_threshold', IMAGE_SCORE_THRESHOLD):
                 return None
-            return {
+
+            base_result = {
                 'src': img.get('src', ''),
                 'data-src': img.get('data-src', ''),
                 'alt': img.get('alt', ''),
@@ -177,6 +307,113 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                 'type': 'image'
             }
 
+            sources = []
+            srcset = img.get('srcset', '')
+            if srcset:
+                sources = parse_srcset(srcset)
+                if sources:
+                    return [dict(base_result, src=source['url'], width=source['width']) 
+                        for source in sources]
+
+            return [base_result]  # Always return a list
+
+        def process_image(img, url, index, total_images):
+            parse_srcset = lambda s: [{'url': u.strip().split()[0], 'width': u.strip().split()[-1].rstrip('w') 
+                          if ' ' in u else None} 
+                         for u in [f"http{p}" for p in s.split("http") if p]]
+            
+            # Constants for checks
+            classes_to_check = frozenset(['button', 'icon', 'logo'])
+            tags_to_check = frozenset(['button', 'input'])
+            
+            # Pre-fetch commonly used attributes
+            style = img.get('style', '')
+            alt = img.get('alt', '')
+            src = img.get('src', '')
+            data_src = img.get('data-src', '')
+            width = img.get('width')
+            height = img.get('height')
+            parent = img.parent
+            parent_classes = parent.get('class', [])
+
+            # Quick validation checks
+            if ('display:none' in style or
+                parent.name in tags_to_check or
+                any(c in cls for c in parent_classes for cls in classes_to_check) or
+                any(c in src for c in classes_to_check) or
+                any(c in alt for c in classes_to_check)):
+                return None
+
+            # Quick score calculation
+            score = 0
+            if width and width.isdigit():
+                width_val = int(width)
+                score += 1 if width_val > 150 else 0
+            if height and height.isdigit():
+                height_val = int(height)
+                score += 1 if height_val > 150 else 0
+            if alt:
+                score += 1
+            score += index/total_images < 0.5
+            
+            image_format = ''
+            if "data:image/" in src:
+                image_format = src.split(',')[0].split(';')[0].split('/')[1].split(';')[0]
+            else:
+                image_format = os.path.splitext(src)[1].lower().strip('.').split('?')[0]
+            
+            if image_format in ('jpg', 'png', 'webp', 'avif'):
+                score += 1
+
+            if score <= kwargs.get('image_score_threshold', IMAGE_SCORE_THRESHOLD):
+                return None
+
+            # Use set for deduplication
+            unique_urls = set()
+            image_variants = []
+            
+            # Generate a unique group ID for this set of variants
+            group_id = index 
+            
+            # Base image info template
+            base_info = {
+                'alt': alt,
+                'desc': find_closest_parent_with_useful_text(img),
+                'score': score,
+                'type': 'image',
+                'group_id': group_id # Group ID for this set of variants
+            }
+
+            # Inline function for adding variants
+            def add_variant(src, width=None):
+                if src and not src.startswith('data:') and src not in unique_urls:
+                    unique_urls.add(src)
+                    image_variants.append({**base_info, 'src': src, 'width': width})
+
+            # Process all sources
+            add_variant(src)
+            add_variant(data_src)
+            
+            # Handle srcset and data-srcset in one pass
+            for attr in ('srcset', 'data-srcset'):
+                if value := img.get(attr):
+                    for source in parse_srcset(value):
+                        add_variant(source['url'], source['width'])
+
+            # Quick picture element check
+            if picture := img.find_parent('picture'):
+                for source in picture.find_all('source'):
+                    if srcset := source.get('srcset'):
+                        for src in parse_srcset(srcset):
+                            add_variant(src['url'], src['width'])
+
+            # Framework-specific attributes in one pass
+            for attr, value in img.attrs.items():
+                if attr.startswith('data-') and ('src' in attr or 'srcset' in attr) and 'http' in value:
+                    add_variant(value)
+
+            return image_variants if image_variants else None
+
         def remove_unwanted_attributes(element, important_attrs, keep_data_attributes=False):
             attrs_to_remove = []
             for attr in element.attrs:
@@ -209,7 +446,6 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                 
                 exclude_social_media_domains = SOCIAL_MEDIA_DOMAINS + kwargs.get('exclude_social_media_domains', [])
                 exclude_social_media_domains = list(set(exclude_social_media_domains))
-
                 
                 try:
                     if element.name == 'a' and element.get('href'):
@@ -329,9 +565,12 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                 try:
                     remove_unwanted_attributes(element, IMPORTANT_ATTRS, kwargs.get('keep_data_attributes', False))
                 except Exception as e:
-                    print('Error removing unwanted attributes:', str(e))
-                
-
+                    # print('Error removing unwanted attributes:', str(e))
+                    self._log('error',
+                        message="Error removing unwanted attributes: {error}",
+                        tag="SCRAPE",
+                        params={"error": str(e)}
+                    )
                 # Process children
                 for child in list(element.children):
                     if isinstance(child, NavigableString) and not isinstance(child, Comment):
@@ -352,30 +591,30 @@ class WebScrappingStrategy(ContentScrappingStrategy):
 
                 return keep_element
             except Exception as e:
-                print('Error processing element:', str(e))
+                # print('Error processing element:', str(e))
+                self._log('error',
+                    message="Error processing element: {error}",
+                    tag="SCRAPE",
+                    params={"error": str(e)}
+                )                
                 return False
-
-        #process images by filtering and extracting contextual text from the page
-        # imgs = body.find_all('img')
-        # media['images'] = [
-        #     result for result in
-        #     (process_image(img, url, i, len(imgs)) for i, img in enumerate(imgs))
-        #     if result is not None
-        # ]
-        
+       
         process_element(body)
         
         # Update the links dictionary with unique links
         links['internal'] = list(internal_links_dict.values())
         links['external'] = list(external_links_dict.values())
 
-
         # # Process images using ThreadPoolExecutor
         imgs = body.find_all('img')
         
-        with ThreadPoolExecutor() as executor:
-            image_results = list(executor.map(process_image, imgs, [url]*len(imgs), range(len(imgs)), [len(imgs)]*len(imgs)))
-        media['images'] = [result for result in image_results if result is not None]
+        # For test we use for loop instead of thread
+        media['images'] = [
+            img for result in (process_image(img, url, i, len(imgs)) 
+                            for i, img in enumerate(imgs))
+            if result is not None
+            for img in result
+        ]
 
         def flatten_nested_elements(node):
             if isinstance(node, NavigableString):
@@ -393,8 +632,9 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                 # Replace base64 data with empty string
                 img['src'] = base64_pattern.sub('', src)
                 
+        str_body = ""
         try:
-            str(body)
+            str_body = body.encode_contents().decode('utf-8')
         except Exception as e:
             # Reset body to the original HTML
             success = False
@@ -419,35 +659,26 @@ class WebScrappingStrategy(ContentScrappingStrategy):
             
             # Append the error div to the body
             body.body.append(error_div)
+            str_body = body.encode_contents().decode('utf-8')
             
             print(f"[LOG] 😧 Error: After processing the crawled HTML and removing irrelevant tags, nothing was left in the page. Check the markdown for further details.")
+            self._log('error',
+                message="After processing the crawled HTML and removing irrelevant tags, nothing was left in the page. Check the markdown for further details.",
+                tag="SCRAPE"
+            )
 
+        cleaned_html = str_body.replace('\n\n', '\n').replace('  ', ' ')
 
-        cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
-
-        try:
-            h = CustomHTML2Text()
-            h.update_params(**kwargs.get('html2text', {}))            
-            markdown = h.handle(cleaned_html)
-        except Exception as e:
-            markdown = h.handle(sanitize_html(cleaned_html))
-        markdown = markdown.replace('    ```', '```')
-
-        try:
-            meta = extract_metadata(html, soup)
-        except Exception as e:
-            print('Error extracting metadata:', str(e))
-            meta = {}
-            
-        cleaner = ContentCleaningStrategy()
-        fit_html = cleaner.clean(cleaned_html)
-        fit_markdown = h.handle(fit_html)
-
-        cleaned_html = sanitize_html(cleaned_html)
+        markdown_content = self._generate_markdown_content(
+            cleaned_html=cleaned_html,
+            html=html,
+            url=url,
+            success=success,
+            **kwargs
+        )
+        
         return {
-            'markdown': markdown,
-            'fit_markdown': fit_markdown,
-            'fit_html': fit_html,
+            **markdown_content,
             'cleaned_html': cleaned_html,
             'success': success,
             'media': media,

crawl4ai/crawler_strategy.py

@@ -132,7 +132,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
 
         # chromedriver_autoinstaller.install()
         # import chromedriver_autoinstaller
-        # crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        # crawl4ai_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
         # driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=self.options)
         # chromedriver_path = chromedriver_autoinstaller.install()
         # chromedriver_path = chromedriver_autoinstaller.utils.download_chromedriver()
@@ -205,7 +205,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
         url_hash = hashlib.md5(url.encode()).hexdigest()
         
         if self.use_cached_html:
-            cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", url_hash)
+            cache_file_path = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", url_hash)
             if os.path.exists(cache_file_path):
                 with open(cache_file_path, "r") as f:
                     return sanitize_input_encode(f.read())
@@ -275,7 +275,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
             self.driver = self.execute_hook('before_return_html', self.driver, html)
             
             # Store in cache
-            cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", url_hash)
+            cache_file_path = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", url_hash)
             with open(cache_file_path, "w", encoding="utf-8") as f:
                 f.write(html)
                 
@@ -283,7 +283,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
                 print(f"[LOG] ✅ Crawled {url} successfully!")
             
             return html
-        except InvalidArgumentException:
+        except InvalidArgumentException as e:
             if not hasattr(e, 'msg'):
                 e.msg = sanitize_input_encode(str(e))
             raise InvalidArgumentException(f"Failed to crawl {url}: {e.msg}")

crawl4ai/database.py

@@ -3,7 +3,7 @@ from pathlib import Path
 import sqlite3
 from typing import Optional, Tuple
 
-DB_PATH = os.path.join(Path.home(), ".crawl4ai")
+DB_PATH = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
 os.makedirs(DB_PATH, exist_ok=True)
 DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")
 

crawl4ai/markdown_generation_strategy.py

@@ -0,0 +1,124 @@
+from abc import ABC, abstractmethod
+from typing import Optional, Dict, Any, Tuple
+from .models import MarkdownGenerationResult
+from .utils import CustomHTML2Text
+from .content_filter_strategy import RelevantContentFilter, BM25ContentFilter
+import re
+from urllib.parse import urljoin
+
+# Pre-compile the regex pattern
+LINK_PATTERN = re.compile(r'!?\[([^\]]+)\]\(([^)]+?)(?:\s+"([^"]*)")?\)')
+
+class MarkdownGenerationStrategy(ABC):
+    """Abstract base class for markdown generation strategies."""
+    def __init__(self, content_filter: Optional[RelevantContentFilter] = None):
+        self.content_filter = content_filter
+    
+    @abstractmethod
+    def generate_markdown(self, 
+                         cleaned_html: str, 
+                         base_url: str = "",
+                         html2text_options: Optional[Dict[str, Any]] = None,
+                         content_filter: Optional[RelevantContentFilter] = None,
+                         citations: bool = True,
+                         **kwargs) -> MarkdownGenerationResult:
+        """Generate markdown from cleaned HTML."""
+        pass
+
+class DefaultMarkdownGenerator(MarkdownGenerationStrategy):
+    """Default implementation of markdown generation strategy."""
+    def __init__(self, content_filter: Optional[RelevantContentFilter] = None):
+        super().__init__(content_filter)
+    
+    def convert_links_to_citations(self, markdown: str, base_url: str = "") -> Tuple[str, str]:
+        link_map = {}
+        url_cache = {}  # Cache for URL joins
+        parts = []
+        last_end = 0
+        counter = 1
+        
+        for match in LINK_PATTERN.finditer(markdown):
+            parts.append(markdown[last_end:match.start()])
+            text, url, title = match.groups()
+            
+            # Use cached URL if available, otherwise compute and cache
+            if base_url and not url.startswith(('http://', 'https://', 'mailto:')):
+                if url not in url_cache:
+                    url_cache[url] = fast_urljoin(base_url, url)
+                url = url_cache[url]
+                
+            if url not in link_map:
+                desc = []
+                if title: desc.append(title)
+                if text and text != title: desc.append(text)
+                link_map[url] = (counter, ": " + " - ".join(desc) if desc else "")
+                counter += 1
+                
+            num = link_map[url][0]
+            parts.append(f"{text}⟨{num}⟩" if not match.group(0).startswith('!') else f"![{text}⟨{num}⟩]")
+            last_end = match.end()
+        
+        parts.append(markdown[last_end:])
+        converted_text = ''.join(parts)
+        
+        # Pre-build reference strings
+        references = ["\n\n## References\n\n"]
+        references.extend(
+            f"⟨{num}⟩ {url}{desc}\n" 
+            for url, (num, desc) in sorted(link_map.items(), key=lambda x: x[1][0])
+        )
+        
+        return converted_text, ''.join(references)
+
+    def generate_markdown(self, 
+                         cleaned_html: str, 
+                         base_url: str = "",
+                         html2text_options: Optional[Dict[str, Any]] = None,
+                         content_filter: Optional[RelevantContentFilter] = None,
+                         citations: bool = True,
+                         **kwargs) -> MarkdownGenerationResult:
+        """Generate markdown with citations from cleaned HTML."""
+        # Initialize HTML2Text with options
+        h = CustomHTML2Text()
+        if html2text_options:
+            h.update_params(**html2text_options)
+
+        # Generate raw markdown
+        raw_markdown = h.handle(cleaned_html)
+        raw_markdown = raw_markdown.replace('    ```', '```')
+
+        # Convert links to citations
+        markdown_with_citations: str = ""
+        references_markdown: str = ""
+        if citations:
+            markdown_with_citations, references_markdown = self.convert_links_to_citations(
+                raw_markdown, base_url
+            )
+
+        # Generate fit markdown if content filter is provided
+        fit_markdown: Optional[str] = ""
+        filtered_html: Optional[str] = ""
+        if content_filter or self.content_filter:
+            content_filter = content_filter or self.content_filter
+            filtered_html = content_filter.filter_content(cleaned_html)
+            filtered_html = '\n'.join('<div>{}</div>'.format(s) for s in filtered_html)
+            fit_markdown = h.handle(filtered_html)
+
+        return MarkdownGenerationResult(
+            raw_markdown=raw_markdown,
+            markdown_with_citations=markdown_with_citations,
+            references_markdown=references_markdown,
+            fit_markdown=fit_markdown,
+            fit_html=filtered_html,
+        )
+
+def fast_urljoin(base: str, url: str) -> str:
+    """Fast URL joining for common cases."""
+    if url.startswith(('http://', 'https://', 'mailto:', '//')):
+        return url
+    if url.startswith('/'):
+        # Handle absolute paths
+        if base.endswith('/'):
+            return base[:-1] + url
+        return base + url
+    return urljoin(base, url)

crawl4ai/migrations.py

@@ -0,0 +1,152 @@
+import os
+import asyncio
+import logging
+from pathlib import Path
+import aiosqlite
+from typing import Optional
+import xxhash
+import aiofiles
+import shutil
+import time
+from datetime import datetime
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+class DatabaseMigration:
+    def __init__(self, db_path: str):
+        self.db_path = db_path
+        self.content_paths = self._ensure_content_dirs(os.path.dirname(db_path))
+        
+    def _ensure_content_dirs(self, base_path: str) -> dict:
+        dirs = {
+            'html': 'html_content',
+            'cleaned': 'cleaned_html',
+            'markdown': 'markdown_content', 
+            'extracted': 'extracted_content',
+            'screenshots': 'screenshots'
+        }
+        content_paths = {}
+        for key, dirname in dirs.items():
+            path = os.path.join(base_path, dirname)
+            os.makedirs(path, exist_ok=True)
+            content_paths[key] = path
+        return content_paths
+
+    def _generate_content_hash(self, content: str) -> str:
+        x = xxhash.xxh64()
+        x.update(content.encode())
+        content_hash = x.hexdigest()
+        return content_hash
+        # return hashlib.sha256(content.encode()).hexdigest()
+
+    async def _store_content(self, content: str, content_type: str) -> str:
+        if not content:
+            return ""
+        
+        content_hash = self._generate_content_hash(content)
+        file_path = os.path.join(self.content_paths[content_type], content_hash)
+        
+        if not os.path.exists(file_path):
+            async with aiofiles.open(file_path, 'w', encoding='utf-8') as f:
+                await f.write(content)
+                
+        return content_hash
+
+    async def migrate_database(self):
+        """Migrate existing database to file-based storage"""
+        logger.info("Starting database migration...")
+        
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                # Get all rows
+                async with db.execute(
+                    '''SELECT url, html, cleaned_html, markdown, 
+                       extracted_content, screenshot FROM crawled_data'''
+                ) as cursor:
+                    rows = await cursor.fetchall()
+
+                migrated_count = 0
+                for row in rows:
+                    url, html, cleaned_html, markdown, extracted_content, screenshot = row
+                    
+                    # Store content in files and get hashes
+                    html_hash = await self._store_content(html, 'html')
+                    cleaned_hash = await self._store_content(cleaned_html, 'cleaned')
+                    markdown_hash = await self._store_content(markdown, 'markdown')
+                    extracted_hash = await self._store_content(extracted_content, 'extracted')
+                    screenshot_hash = await self._store_content(screenshot, 'screenshots')
+
+                    # Update database with hashes
+                    await db.execute('''
+                        UPDATE crawled_data 
+                        SET html = ?, 
+                            cleaned_html = ?,
+                            markdown = ?,
+                            extracted_content = ?,
+                            screenshot = ?
+                        WHERE url = ?
+                    ''', (html_hash, cleaned_hash, markdown_hash, 
+                         extracted_hash, screenshot_hash, url))
+                    
+                    migrated_count += 1
+                    if migrated_count % 100 == 0:
+                        logger.info(f"Migrated {migrated_count} records...")
+
+                await db.commit()
+                logger.info(f"Migration completed. {migrated_count} records processed.")
+
+        except Exception as e:
+            logger.error(f"Migration failed: {e}")
+            raise
+
+async def backup_database(db_path: str) -> str:
+    """Create backup of existing database"""
+    if not os.path.exists(db_path):
+        logger.info("No existing database found. Skipping backup.")
+        return None
+        
+    # Create backup with timestamp
+    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+    backup_path = f"{db_path}.backup_{timestamp}"
+    
+    try:
+        # Wait for any potential write operations to finish
+        await asyncio.sleep(1)
+        
+        # Create backup
+        shutil.copy2(db_path, backup_path)
+        logger.info(f"Database backup created at: {backup_path}")
+        return backup_path
+    except Exception as e:
+        logger.error(f"Backup failed: {e}")
+        raise
+    
+async def run_migration(db_path: Optional[str] = None):
+    """Run database migration"""
+    if db_path is None:
+        db_path = os.path.join(Path.home(), ".crawl4ai", "crawl4ai.db")
+    
+    if not os.path.exists(db_path):
+        logger.info("No existing database found. Skipping migration.")
+        return
+        
+    # Create backup first
+    backup_path = await backup_database(db_path)
+    if not backup_path:
+        return
+    
+    migration = DatabaseMigration(db_path)
+    await migration.migrate_database()
+    
+def main():
+    """CLI entry point for migration"""
+    import argparse
+    parser = argparse.ArgumentParser(description='Migrate Crawl4AI database to file-based storage')
+    parser.add_argument('--db-path', help='Custom database path')
+    args = parser.parse_args()
+    
+    asyncio.run(run_migration(args.db_path))
+
+if __name__ == "__main__":
+    main()

crawl4ai/model_loader.py

@@ -56,7 +56,7 @@ def set_model_device(model):
 
 @lru_cache()
 def get_home_folder():
-    home_folder = os.path.join(Path.home(), ".crawl4ai")
+    home_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
     os.makedirs(home_folder, exist_ok=True)
     os.makedirs(f"{home_folder}/cache", exist_ok=True)
     os.makedirs(f"{home_folder}/models", exist_ok=True)

crawl4ai/models.py

@@ -1,10 +1,19 @@
 from pydantic import BaseModel, HttpUrl
-from typing import List, Dict, Optional
+from typing import List, Dict, Optional, Callable, Awaitable, Union
+
+
 
 class UrlModel(BaseModel):
     url: HttpUrl
     forced: bool = False
 
+class MarkdownGenerationResult(BaseModel):
+    raw_markdown: str
+    markdown_with_citations: str
+    references_markdown: str
+    fit_markdown: Optional[str] = None
+    fit_html: Optional[str] = None
+
 class CrawlResult(BaseModel):
     url: str
     html: str
@@ -12,8 +21,10 @@ class CrawlResult(BaseModel):
     cleaned_html: Optional[str] = None
     media: Dict[str, List[Dict]] = {}
     links: Dict[str, List[Dict]] = {}
+    downloaded_files: Optional[List[str]] = None
     screenshot: Optional[str] = None
-    markdown: Optional[str] = None
+    markdown: Optional[Union[str, MarkdownGenerationResult]] = None
+    markdown_v2: Optional[MarkdownGenerationResult] = None
     fit_markdown: Optional[str] = None
     fit_html: Optional[str] = None
     extracted_content: Optional[str] = None
@@ -21,4 +32,17 @@ class CrawlResult(BaseModel):
     error_message: Optional[str] = None
     session_id: Optional[str] = None
     response_headers: Optional[dict] = None
-    status_code: Optional[int] = None
+    status_code: Optional[int] = None
+    
+class AsyncCrawlResponse(BaseModel):
+    html: str
+    response_headers: Dict[str, str]
+    status_code: int
+    screenshot: Optional[str] = None
+    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
+    downloaded_files: Optional[List[str]] = None
+
+    class Config:
+        arbitrary_types_allowed = True
+
+

crawl4ai/models/onnx/config.json

@@ -1,25 +0,0 @@
-{
-  "_name_or_path": "sentence-transformers/all-MiniLM-L6-v2",
-  "architectures": [
-    "BertModel"
-  ],
-  "attention_probs_dropout_prob": 0.1,
-  "classifier_dropout": null,
-  "gradient_checkpointing": false,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 384,
-  "initializer_range": 0.02,
-  "intermediate_size": 1536,
-  "layer_norm_eps": 1e-12,
-  "max_position_embeddings": 512,
-  "model_type": "bert",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 6,
-  "pad_token_id": 0,
-  "position_embedding_type": "absolute",
-  "transformers_version": "4.27.4",
-  "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
-}

crawl4ai/models/onnx/special_tokens_map.json

@@ -1,7 +0,0 @@
-{
-  "cls_token": "[CLS]",
-  "mask_token": "[MASK]",
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "unk_token": "[UNK]"
-}

crawl4ai/models/onnx/tokenizer_config.json

@@ -1,15 +0,0 @@
-{
-  "cls_token": "[CLS]",
-  "do_basic_tokenize": true,
-  "do_lower_case": true,
-  "mask_token": "[MASK]",
-  "model_max_length": 512,
-  "never_split": null,
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "special_tokens_map_file": "/Users/hammad/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125/special_tokens_map.json",
-  "strip_accents": null,
-  "tokenize_chinese_chars": true,
-  "tokenizer_class": "BertTokenizer",
-  "unk_token": "[UNK]"
-}

crawl4ai/tools.py

@@ -0,0 +1,34 @@
+import time
+import cProfile
+import pstats
+from functools import wraps
+
+def profile_and_time(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        # Start timer
+        start_time = time.perf_counter()
+        
+        # Setup profiler
+        profiler = cProfile.Profile()
+        profiler.enable()
+        
+        # Run function
+        result = func(self, *args, **kwargs)
+        
+        # Stop profiler
+        profiler.disable()
+        
+        # Calculate elapsed time
+        elapsed_time = time.perf_counter() - start_time
+        
+        # Print timing
+        print(f"[PROFILER] Scraping completed in {elapsed_time:.2f} seconds")
+        
+        # Print profiling stats
+        stats = pstats.Stats(profiler)
+        stats.sort_stats('cumulative')  # Sort by cumulative time
+        stats.print_stats(20)  # Print top 20 time-consuming functions
+        
+        return result
+    return wrapper

crawl4ai/train.py

@@ -1,146 +0,0 @@
-import spacy
-from spacy.training import Example
-import random
-import nltk
-from nltk.corpus import reuters
-import torch
-
-def save_spacy_model_as_torch(nlp, model_dir="models/reuters"):
-    # Extract the TextCategorizer component
-    textcat = nlp.get_pipe("textcat_multilabel")
-
-    # Convert the weights to a PyTorch state dictionary
-    state_dict = {name: torch.tensor(param.data) for name, param in textcat.model.named_parameters()}
-
-    # Save the state dictionary
-    torch.save(state_dict, f"{model_dir}/model_weights.pth")
-
-    # Extract and save the vocabulary
-    vocab = extract_vocab(nlp)
-    with open(f"{model_dir}/vocab.txt", "w") as vocab_file:
-        for word, idx in vocab.items():
-            vocab_file.write(f"{word}\t{idx}\n")
-    
-    print(f"Model weights and vocabulary saved to: {model_dir}")
-
-def extract_vocab(nlp):
-    # Extract vocabulary from the SpaCy model
-    vocab = {word: i for i, word in enumerate(nlp.vocab.strings)}
-    return vocab
-
-nlp = spacy.load("models/reuters")
-save_spacy_model_as_torch(nlp, model_dir="models")
-
-def train_and_save_reuters_model(model_dir="models/reuters"):
-    # Ensure the Reuters corpus is downloaded
-    nltk.download('reuters')
-    nltk.download('punkt')
-    if not reuters.fileids():
-        print("Reuters corpus not found.")
-        return
-
-    # Load a blank English spaCy model
-    nlp = spacy.blank("en")
-
-    # Create a TextCategorizer with the ensemble model for multi-label classification
-    textcat = nlp.add_pipe("textcat_multilabel")
-
-    # Add labels to text classifier
-    for label in reuters.categories():
-        textcat.add_label(label)
-
-    # Prepare training data
-    train_examples = []
-    for fileid in reuters.fileids():
-        categories = reuters.categories(fileid)
-        text = reuters.raw(fileid)
-        cats = {label: label in categories for label in reuters.categories()}
-        # Prepare spacy Example objects
-        doc = nlp.make_doc(text)
-        example = Example.from_dict(doc, {'cats': cats})
-        train_examples.append(example)
-
-    # Initialize the text categorizer with the example objects
-    nlp.initialize(lambda: train_examples)
-
-    # Train the model
-    random.seed(1)
-    spacy.util.fix_random_seed(1)
-    for i in range(5):  # Adjust iterations for better accuracy
-        random.shuffle(train_examples)
-        losses = {}
-        # Create batches of data
-        batches = spacy.util.minibatch(train_examples, size=8)
-        for batch in batches:
-            nlp.update(batch, drop=0.2, losses=losses)
-        print(f"Losses at iteration {i}: {losses}")
-
-    # Save the trained model
-    nlp.to_disk(model_dir)
-    print(f"Model saved to: {model_dir}")
-
-def train_model(model_dir, additional_epochs=0):
-    # Load the model if it exists, otherwise start with a blank model
-    try:
-        nlp = spacy.load(model_dir)
-        print("Model loaded from disk.")
-    except IOError:
-        print("No existing model found. Starting with a new model.")
-        nlp = spacy.blank("en")
-        textcat = nlp.add_pipe("textcat_multilabel")
-        for label in reuters.categories():
-            textcat.add_label(label)
-
-    # Prepare training data
-    train_examples = []
-    for fileid in reuters.fileids():
-        categories = reuters.categories(fileid)
-        text = reuters.raw(fileid)
-        cats = {label: label in categories for label in reuters.categories()}
-        doc = nlp.make_doc(text)
-        example = Example.from_dict(doc, {'cats': cats})
-        train_examples.append(example)
-
-    # Initialize the model if it was newly created
-    if 'textcat_multilabel' not in nlp.pipe_names:
-        nlp.initialize(lambda: train_examples)
-    else:
-        print("Continuing training with existing model.")
-
-    # Train the model
-    random.seed(1)
-    spacy.util.fix_random_seed(1)
-    num_epochs = 5 + additional_epochs
-    for i in range(num_epochs):
-        random.shuffle(train_examples)
-        losses = {}
-        batches = spacy.util.minibatch(train_examples, size=8)
-        for batch in batches:
-            nlp.update(batch, drop=0.2, losses=losses)
-        print(f"Losses at iteration {i}: {losses}")
-
-    # Save the trained model
-    nlp.to_disk(model_dir)
-    print(f"Model saved to: {model_dir}")
-
-def load_model_and_predict(model_dir, text, tok_k = 3):
-    # Load the trained model from the specified directory
-    nlp = spacy.load(model_dir)
-    
-    # Process the text with the loaded model
-    doc = nlp(text)
-    
-    # gee top 3 categories
-    top_categories = sorted(doc.cats.items(), key=lambda x: x[1], reverse=True)[:tok_k]
-    print(f"Top {tok_k} categories:")
-    
-    return top_categories    
-
-if __name__ == "__main__":
-    train_and_save_reuters_model()
-    train_model("models/reuters", additional_epochs=5)
-    model_directory = "reuters_model_10"
-    print(reuters.categories())
-    example_text = "Apple Inc. is reportedly buying a startup for $1 billion"
-    r =load_model_and_predict(model_directory, example_text)
-    print(r)

crawl4ai/utils.py

@@ -14,10 +14,157 @@ from typing import Dict, Any
 from urllib.parse import urljoin
 import requests
 from requests.exceptions import InvalidSchema
+import hashlib
+from typing import Optional, Tuple, Dict, Any
+import xxhash
+from colorama import Fore, Style, init
+import textwrap
 
+from .html2text import HTML2Text
+class CustomHTML2Text(HTML2Text):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.inside_pre = False
+        self.inside_code = False
+        self.preserve_tags = set()  # Set of tags to preserve
+        self.current_preserved_tag = None
+        self.preserved_content = []
+        self.preserve_depth = 0
+        
+        # Configuration options
+        self.skip_internal_links = False
+        self.single_line_break = False
+        self.mark_code = False
+        self.include_sup_sub = False
+        self.body_width = 0
+        self.ignore_mailto_links = True
+        self.ignore_links = False
+        self.escape_backslash = False
+        self.escape_dot = False
+        self.escape_plus = False
+        self.escape_dash = False
+        self.escape_snob = False
+
+    def update_params(self, **kwargs):
+        """Update parameters and set preserved tags."""
+        for key, value in kwargs.items():
+            if key == 'preserve_tags':
+                self.preserve_tags = set(value)
+            else:
+                setattr(self, key, value)
+
+    def handle_tag(self, tag, attrs, start):
+        # Handle preserved tags
+        if tag in self.preserve_tags:
+            if start:
+                if self.preserve_depth == 0:
+                    self.current_preserved_tag = tag
+                    self.preserved_content = []
+                    # Format opening tag with attributes
+                    attr_str = ''.join(f' {k}="{v}"' for k, v in attrs.items() if v is not None)
+                    self.preserved_content.append(f'<{tag}{attr_str}>')
+                self.preserve_depth += 1
+                return
+            else:
+                self.preserve_depth -= 1
+                if self.preserve_depth == 0:
+                    self.preserved_content.append(f'</{tag}>')
+                    # Output the preserved HTML block with proper spacing
+                    preserved_html = ''.join(self.preserved_content)
+                    self.o('\n' + preserved_html + '\n')
+                    self.current_preserved_tag = None
+                return
+
+        # If we're inside a preserved tag, collect all content
+        if self.preserve_depth > 0:
+            if start:
+                # Format nested tags with attributes
+                attr_str = ''.join(f' {k}="{v}"' for k, v in attrs.items() if v is not None)
+                self.preserved_content.append(f'<{tag}{attr_str}>')
+            else:
+                self.preserved_content.append(f'</{tag}>')
+            return
+
+        # Handle pre tags
+        if tag == 'pre':
+            if start:
+                self.o('```\n')
+                self.inside_pre = True
+            else:
+                self.o('\n```')
+                self.inside_pre = False
+        # elif tag in ["h1", "h2", "h3", "h4", "h5", "h6"]:
+        #     pass
+        else:
+            super().handle_tag(tag, attrs, start)
+
+    def handle_data(self, data, entity_char=False):
+        """Override handle_data to capture content within preserved tags."""
+        if self.preserve_depth > 0:
+            self.preserved_content.append(data)
+            return
+        super().handle_data(data, entity_char)
 class InvalidCSSSelectorError(Exception):
     pass
 
+
+def create_box_message(
+   message: str, 
+   type: str = "info", 
+   width: int = 80, 
+   add_newlines: bool = True,
+   double_line: bool = False
+) -> str:
+   init()
+   
+   # Define border and text colors for different types
+   styles = {
+       "warning": (Fore.YELLOW, Fore.LIGHTYELLOW_EX, "⚠"),
+       "info": (Fore.BLUE, Fore.LIGHTBLUE_EX, "ℹ"), 
+       "success": (Fore.GREEN, Fore.LIGHTGREEN_EX, "✓"),
+       "error": (Fore.RED, Fore.LIGHTRED_EX, "×"),
+   }
+   
+   border_color, text_color, prefix = styles.get(type.lower(), styles["info"])
+   
+   # Define box characters based on line style
+   box_chars = {
+       "single": ("─", "│", "┌", "┐", "└", "┘"),
+       "double": ("═", "║", "╔", "╗", "╚", "╝")
+   }
+   line_style = "double" if double_line else "single"
+   h_line, v_line, tl, tr, bl, br = box_chars[line_style]
+   
+   # Process lines with lighter text color
+   formatted_lines = []
+   raw_lines = message.split('\n')
+   
+   if raw_lines:
+       first_line = f"{prefix} {raw_lines[0].strip()}"
+       wrapped_first = textwrap.fill(first_line, width=width-4)
+       formatted_lines.extend(wrapped_first.split('\n'))
+       
+       for line in raw_lines[1:]:
+           if line.strip():
+               wrapped = textwrap.fill(f"  {line.strip()}", width=width-4)
+               formatted_lines.extend(wrapped.split('\n'))
+           else:
+               formatted_lines.append("")
+   
+   # Create the box with colored borders and lighter text
+   horizontal_line = h_line * (width - 1)
+   box = [
+       f"{border_color}{tl}{horizontal_line}{tr}",
+       *[f"{border_color}{v_line}{text_color} {line:<{width-2}}{border_color}{v_line}" for line in formatted_lines],
+       f"{border_color}{bl}{horizontal_line}{br}{Style.RESET_ALL}"
+   ]
+   
+   result = "\n".join(box)
+   if add_newlines:
+       result = f"\n{result}\n"
+   
+   return result
+
 def calculate_semaphore_count():
     cpu_count = os.cpu_count()
     memory_gb = get_system_memory() / (1024 ** 3)  # Convert to GB
@@ -60,7 +207,7 @@ def get_system_memory():
         raise OSError("Unsupported operating system")
 
 def get_home_folder():
-    home_folder = os.path.join(Path.home(), ".crawl4ai")
+    home_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())), ".crawl4ai")
     os.makedirs(home_folder, exist_ok=True)
     os.makedirs(f"{home_folder}/cache", exist_ok=True)
     os.makedirs(f"{home_folder}/models", exist_ok=True)
@@ -142,12 +289,17 @@ def sanitize_html(html):
 def sanitize_input_encode(text: str) -> str:
     """Sanitize input to handle potential encoding issues."""
     try:
-        # Attempt to encode and decode as UTF-8 to handle potential encoding issues
-        return text.encode('utf-8', errors='ignore').decode('utf-8')
-    except UnicodeEncodeError as e:
-        print(f"Warning: Encoding issue detected. Some characters may be lost. Error: {e}")
-        # Fall back to ASCII if UTF-8 fails
-        return text.encode('ascii', errors='ignore').decode('ascii')
+        try:
+            if not text:
+                return ''
+            # Attempt to encode and decode as UTF-8 to handle potential encoding issues
+            return text.encode('utf-8', errors='ignore').decode('utf-8')
+        except UnicodeEncodeError as e:
+            print(f"Warning: Encoding issue detected. Some characters may be lost. Error: {e}")
+            # Fall back to ASCII if UTF-8 fails
+            return text.encode('ascii', errors='ignore').decode('ascii')
+    except Exception as e:
+        raise ValueError(f"Error sanitizing input: {str(e)}") from e
 
 def escape_json_string(s):
     """
@@ -178,7 +330,7 @@ def escape_json_string(s):
     
     return s
 
-class CustomHTML2Text(HTML2Text):
+class CustomHTML2Text_v0(HTML2Text):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.inside_pre = False
@@ -706,9 +858,12 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
     body = flatten_nested_elements(body)
     base64_pattern = re.compile(r'data:image/[^;]+;base64,([^"]+)')
     for img in imgs:
-        src = img.get('src', '')
-        if base64_pattern.match(src):
-            img['src'] = base64_pattern.sub('', src)
+        try:
+            src = img.get('src', '')
+            if base64_pattern.match(src):
+                img['src'] = base64_pattern.sub('', src)
+        except:
+            pass        
 
     cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
     cleaned_html = sanitize_html(cleaned_html)
@@ -733,46 +888,54 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
         'metadata': meta
     }
 
-def extract_metadata(html, soup = None):
+def extract_metadata(html, soup=None):
     metadata = {}
     
-    if not html:
+    if not html and not soup:
+        return {}
+    
+    if not soup:
+        soup = BeautifulSoup(html, 'lxml')
+    
+    head = soup.head
+    if not head:
         return metadata
     
-    # Parse HTML content with BeautifulSoup
-    if not soup:
-        soup = BeautifulSoup(html, 'html.parser')
-
     # Title
-    title_tag = soup.find('title')
-    metadata['title'] = title_tag.string if title_tag else None
+    title_tag = head.find('title')
+    metadata['title'] = title_tag.string.strip() if title_tag and title_tag.string else None
 
     # Meta description
-    description_tag = soup.find('meta', attrs={'name': 'description'})
-    metadata['description'] = description_tag['content'] if description_tag else None
+    description_tag = head.find('meta', attrs={'name': 'description'})
+    metadata['description'] = description_tag.get('content', '').strip() if description_tag else None
 
     # Meta keywords
-    keywords_tag = soup.find('meta', attrs={'name': 'keywords'})
-    metadata['keywords'] = keywords_tag['content'] if keywords_tag else None
+    keywords_tag = head.find('meta', attrs={'name': 'keywords'})
+    metadata['keywords'] = keywords_tag.get('content', '').strip() if keywords_tag else None
 
     # Meta author
-    author_tag = soup.find('meta', attrs={'name': 'author'})
-    metadata['author'] = author_tag['content'] if author_tag else None
+    author_tag = head.find('meta', attrs={'name': 'author'})
+    metadata['author'] = author_tag.get('content', '').strip() if author_tag else None
 
     # Open Graph metadata
-    og_tags = soup.find_all('meta', attrs={'property': lambda value: value and value.startswith('og:')})
+    og_tags = head.find_all('meta', attrs={'property': re.compile(r'^og:')})
     for tag in og_tags:
-        property_name = tag['property']
-        metadata[property_name] = tag['content']
+        property_name = tag.get('property', '').strip()
+        content = tag.get('content', '').strip()
+        if property_name and content:
+            metadata[property_name] = content
 
     # Twitter Card metadata
-    twitter_tags = soup.find_all('meta', attrs={'name': lambda value: value and value.startswith('twitter:')})
+    twitter_tags = head.find_all('meta', attrs={'name': re.compile(r'^twitter:')})
     for tag in twitter_tags:
-        property_name = tag['name']
-        metadata[property_name] = tag['content']
-
+        property_name = tag.get('name', '').strip()
+        content = tag.get('content', '').strip()
+        if property_name and content:
+            metadata[property_name] = content
+    
     return metadata
 
+
 def extract_xml_tags(string):
     tags = re.findall(r'<(\w+)>', string)
     return list(set(tags))
@@ -977,10 +1140,68 @@ def wrap_text(draw, text, font, max_width):
     return '\n'.join(lines)
 
 def format_html(html_string):
-    soup = BeautifulSoup(html_string, 'html.parser')
+    soup = BeautifulSoup(html_string, 'lxml.parser')
     return soup.prettify()
 
+def fast_format_html(html_string):
+    """
+    A fast HTML formatter that uses string operations instead of parsing.
+    
+    Args:
+        html_string (str): The HTML string to format
+        
+    Returns:
+        str: The formatted HTML string
+    """
+    # Initialize variables
+    indent = 0
+    indent_str = "  "  # Two spaces for indentation
+    formatted = []
+    in_content = False
+    
+    # Split by < and > to separate tags and content
+    parts = html_string.replace('>', '>\n').replace('<', '\n<').split('\n')
+    
+    for part in parts:
+        if not part.strip():
+            continue
+            
+        # Handle closing tags
+        if part.startswith('</'):
+            indent -= 1
+            formatted.append(indent_str * indent + part)
+            
+        # Handle self-closing tags
+        elif part.startswith('<') and part.endswith('/>'):
+            formatted.append(indent_str * indent + part)
+            
+        # Handle opening tags
+        elif part.startswith('<'):
+            formatted.append(indent_str * indent + part)
+            indent += 1
+            
+        # Handle content between tags
+        else:
+            content = part.strip()
+            if content:
+                formatted.append(indent_str * indent + content)
+    
+    return '\n'.join(formatted)
+
 def normalize_url(href, base_url):
+    """Normalize URLs to ensure consistent format"""
+    from urllib.parse import urljoin, urlparse
+
+    # Parse base URL to get components
+    parsed_base = urlparse(base_url)
+    if not parsed_base.scheme or not parsed_base.netloc:
+        raise ValueError(f"Invalid base URL format: {base_url}")
+
+    # Use urljoin to handle all cases
+    normalized = urljoin(base_url, href.strip())
+    return normalized
+
+def normalize_url_tmp(href, base_url):
     """Normalize URLs to ensure consistent format"""
     # Extract protocol and domain from base URL
     try:
@@ -1030,3 +1251,82 @@ def is_external_url(url, base_domain):
         return False
         
     return False
+
+def clean_tokens(tokens: list[str]) -> list[str]:
+    # Set of tokens to remove
+    noise = {'ccp', 'up', '↑', '▲', '⬆️', 'a', 'an', 'at', 'by', 'in', 'of', 'on', 'to', 'the'}
+
+    STOP_WORDS = {
+        'a', 'an', 'and', 'are', 'as', 'at', 'be', 'by', 'for', 'from', 
+        'has', 'he', 'in', 'is', 'it', 'its', 'of', 'on', 'that', 'the', 
+        'to', 'was', 'were', 'will', 'with',
+        
+        # Pronouns
+        'i', 'you', 'he', 'she', 'it', 'we', 'they',
+        'me', 'him', 'her', 'us', 'them',
+        'my', 'your', 'his', 'her', 'its', 'our', 'their',
+        'mine', 'yours', 'hers', 'ours', 'theirs',
+        'myself', 'yourself', 'himself', 'herself', 'itself', 'ourselves', 'themselves',
+        
+        # Common verbs
+        'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
+        'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing',
+        
+        # Prepositions
+        'about', 'above', 'across', 'after', 'against', 'along', 'among', 'around',
+        'at', 'before', 'behind', 'below', 'beneath', 'beside', 'between', 'beyond',
+        'by', 'down', 'during', 'except', 'for', 'from', 'in', 'inside', 'into',
+        'near', 'of', 'off', 'on', 'out', 'outside', 'over', 'past', 'through',
+        'to', 'toward', 'under', 'underneath', 'until', 'up', 'upon', 'with', 'within',
+        
+        # Conjunctions
+        'and', 'but', 'or', 'nor', 'for', 'yet', 'so',
+        'although', 'because', 'since', 'unless',
+        
+        # Articles
+        'a', 'an', 'the',
+        
+        # Other common words
+        'this', 'that', 'these', 'those',
+        'what', 'which', 'who', 'whom', 'whose',
+        'when', 'where', 'why', 'how',
+        'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such',
+        'can', 'cannot', "can't", 'could', "couldn't",
+        'may', 'might', 'must', "mustn't",
+        'shall', 'should', "shouldn't",
+        'will', "won't", 'would', "wouldn't",
+        'not', "n't", 'no', 'nor', 'none'
+    }   
+   
+    # Single comprehension, more efficient than multiple passes
+    return [token for token in tokens 
+            if len(token) > 2 
+            and token not in noise 
+            and token not in STOP_WORDS
+            and not token.startswith('↑')
+            and not token.startswith('▲')
+            and not token.startswith('⬆')]
+
+
+def generate_content_hash(content: str) -> str:
+    """Generate a unique hash for content"""
+    return xxhash.xxh64(content.encode()).hexdigest()
+    # return hashlib.sha256(content.encode()).hexdigest()
+
+def ensure_content_dirs(base_path: str) -> Dict[str, str]:
+    """Create content directories if they don't exist"""
+    dirs = {
+        'html': 'html_content',
+        'cleaned': 'cleaned_html',
+        'markdown': 'markdown_content', 
+        'extracted': 'extracted_content',
+        'screenshots': 'screenshots'
+    }
+    
+    content_paths = {}
+    for key, dirname in dirs.items():
+        path = os.path.join(base_path, dirname)
+        os.makedirs(path, exist_ok=True)
+        content_paths[key] = path
+        
+    return content_paths

crawl4ai/version_manager.py

@@ -0,0 +1,30 @@
+# version_manager.py
+import os
+from pathlib import Path
+from packaging import version
+from . import __version__
+
+class VersionManager:
+    def __init__(self):
+        self.home_dir = Path.home() / ".crawl4ai"
+        self.version_file = self.home_dir / "version.txt"
+        
+    def get_installed_version(self):
+        """Get the version recorded in home directory"""
+        if not self.version_file.exists():
+            return None
+        try:
+            return version.parse(self.version_file.read_text().strip())
+        except:
+            return None
+            
+    def update_version(self):
+        """Update the version file to current library version"""
+        self.version_file.write_text(__version__.__version__)
+        
+    def needs_update(self):
+        """Check if database needs update based on version"""
+        installed = self.get_installed_version()
+        current = version.parse(__version__.__version__)
+        return installed is None or installed < current
+

crawl4ai/web_crawler.back.py

@@ -1,357 +0,0 @@
-import os, time
-os.environ["TOKENIZERS_PARALLELISM"] = "false"
-from pathlib import Path
-
-from .models import UrlModel, CrawlResult
-from .database import init_db, get_cached_url, cache_url, DB_PATH, flush_db
-from .utils import *
-from .chunking_strategy import *
-from .extraction_strategy import *
-from .crawler_strategy import *
-from typing import List
-from concurrent.futures import ThreadPoolExecutor
-from .config import *
-
-
-class WebCrawler:
-    def __init__(
-        self,
-        # db_path: str = None,
-        crawler_strategy: CrawlerStrategy = None,
-        always_by_pass_cache: bool = False,
-        verbose: bool = False,
-    ):
-        # self.db_path = db_path
-        self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose)
-        self.always_by_pass_cache = always_by_pass_cache
-
-        # Create the .crawl4ai folder in the user's home directory if it doesn't exist
-        self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
-        os.makedirs(self.crawl4ai_folder, exist_ok=True)
-        os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
-
-        # If db_path is not provided, use the default path
-        # if not db_path:
-            # self.db_path = f"{self.crawl4ai_folder}/crawl4ai.db"
-        
-        # flush_db()
-        init_db()
-        
-        self.ready = False
-        
-    def warmup(self):
-        print("[LOG] 🌤️  Warming up the WebCrawler")
-        result = self.run(
-            url='https://crawl4ai.uccode.io/',
-            word_count_threshold=5,
-            extraction_strategy= NoExtractionStrategy(),
-            bypass_cache=False,
-            verbose = False
-        )
-        self.ready = True
-        print("[LOG] 🌞 WebCrawler is ready to crawl")
-        
-    def fetch_page(
-        self,
-        url_model: UrlModel,
-        provider: str = DEFAULT_PROVIDER,
-        api_token: str = None,
-        extract_blocks_flag: bool = True,
-        word_count_threshold=MIN_WORD_THRESHOLD,
-        css_selector: str = None,
-        screenshot: bool = False,
-        use_cached_html: bool = False,
-        extraction_strategy: ExtractionStrategy = None,
-        chunking_strategy: ChunkingStrategy = RegexChunking(),
-        **kwargs,
-    ) -> CrawlResult:
-        return self.run(
-            url_model.url,
-            word_count_threshold,
-            extraction_strategy or NoExtractionStrategy(),
-            chunking_strategy,
-            bypass_cache=url_model.forced,
-            css_selector=css_selector,
-            screenshot=screenshot,
-            **kwargs,
-        )
-        pass
-
-    def run_old(
-        self,
-        url: str,
-        word_count_threshold=MIN_WORD_THRESHOLD,
-        extraction_strategy: ExtractionStrategy = None,
-        chunking_strategy: ChunkingStrategy = RegexChunking(),
-        bypass_cache: bool = False,
-        css_selector: str = None,
-        screenshot: bool = False,
-        user_agent: str = None,
-        verbose=True,
-        **kwargs,
-    ) -> CrawlResult:
-        if user_agent:
-            self.crawler_strategy.update_user_agent(user_agent)
-        extraction_strategy = extraction_strategy or NoExtractionStrategy()
-        extraction_strategy.verbose = verbose
-        # Check if extraction strategy is an instance of ExtractionStrategy if not raise an error
-        if not isinstance(extraction_strategy, ExtractionStrategy):
-            raise ValueError("Unsupported extraction strategy")
-        if not isinstance(chunking_strategy, ChunkingStrategy):
-            raise ValueError("Unsupported chunking strategy")
-        
-        # make sure word_count_threshold is not lesser than MIN_WORD_THRESHOLD
-        if word_count_threshold < MIN_WORD_THRESHOLD:
-            word_count_threshold = MIN_WORD_THRESHOLD
-
-        # Check cache first
-        if not bypass_cache and not self.always_by_pass_cache:
-            cached = get_cached_url(url)
-            if cached:
-                return CrawlResult(
-                    **{
-                        "url": cached[0],
-                        "html": cached[1],
-                        "cleaned_html": cached[2],
-                        "markdown": cached[3],
-                        "extracted_content": cached[4],
-                        "success": cached[5],
-                        "media": json.loads(cached[6] or "{}"),
-                        "links": json.loads(cached[7] or "{}"),
-                        "metadata": json.loads(cached[8] or "{}"), # "metadata": "{}
-                        "screenshot": cached[9],
-                        "error_message": "",
-                    }
-                )
-
-        # Initialize WebDriver for crawling
-        t = time.time()
-        if kwargs.get("js", None):
-            self.crawler_strategy.js_code = kwargs.get("js")
-        html = self.crawler_strategy.crawl(url)
-        base64_image = None
-        if screenshot:
-            base64_image = self.crawler_strategy.take_screenshot()
-        success = True
-        error_message = ""
-        # Extract content from HTML
-        try:
-            result = get_content_of_website(url, html, word_count_threshold, css_selector=css_selector)
-            metadata = extract_metadata(html)
-            if result is None:
-                raise ValueError(f"Failed to extract content from the website: {url}")
-        except InvalidCSSSelectorError as e:
-            raise ValueError(str(e))
-        
-        cleaned_html = result.get("cleaned_html", "")
-        markdown = result.get("markdown", "")
-        media = result.get("media", [])
-        links = result.get("links", [])
-
-        # Print a profession LOG style message, show time taken and say crawling is done
-        if verbose:
-            print(
-                f"[LOG] 🚀 Crawling done for {url}, success: {success}, time taken: {time.time() - t} seconds"
-            )
-
-        extracted_content = []
-        if verbose:
-            print(f"[LOG] 🔥 Extracting semantic blocks for {url}, Strategy: {extraction_strategy.name}")
-        t = time.time()
-        # Split markdown into sections
-        sections = chunking_strategy.chunk(markdown)
-        # sections = merge_chunks_based_on_token_threshold(sections, CHUNK_TOKEN_THRESHOLD)
-
-        extracted_content = extraction_strategy.run(
-            url, sections,
-        )
-        extracted_content = json.dumps(extracted_content)
-
-        if verbose:
-            print(
-                f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t} seconds."
-            )
-
-        # Cache the result
-        cleaned_html = beautify_html(cleaned_html)
-        cache_url(
-            url,
-            html,
-            cleaned_html,
-            markdown,
-            extracted_content,
-            success,
-            json.dumps(media),
-            json.dumps(links),
-            json.dumps(metadata),
-            screenshot=base64_image,
-        )
-
-        return CrawlResult(
-            url=url,
-            html=html,
-            cleaned_html=cleaned_html,
-            markdown=markdown,
-            media=media,
-            links=links,
-            metadata=metadata,
-            screenshot=base64_image,
-            extracted_content=extracted_content,
-            success=success,
-            error_message=error_message,
-        )
-
-    def fetch_pages(
-        self,
-        url_models: List[UrlModel],
-        provider: str = DEFAULT_PROVIDER,
-        api_token: str = None,
-        extract_blocks_flag: bool = True,
-        word_count_threshold=MIN_WORD_THRESHOLD,
-        use_cached_html: bool = False,
-        css_selector: str = None,
-        screenshot: bool = False,
-        extraction_strategy: ExtractionStrategy = None,
-        chunking_strategy: ChunkingStrategy = RegexChunking(),
-        **kwargs,
-    ) -> List[CrawlResult]:
-        extraction_strategy = extraction_strategy or NoExtractionStrategy()
-        def fetch_page_wrapper(url_model, *args, **kwargs):
-            return self.fetch_page(url_model, *args, **kwargs)
-
-        with ThreadPoolExecutor() as executor:
-            results = list(
-                executor.map(
-                    fetch_page_wrapper,
-                    url_models,
-                    [provider] * len(url_models),
-                    [api_token] * len(url_models),
-                    [extract_blocks_flag] * len(url_models),
-                    [word_count_threshold] * len(url_models),
-                    [css_selector] * len(url_models),
-                    [screenshot] * len(url_models),
-                    [use_cached_html] * len(url_models),
-                    [extraction_strategy] * len(url_models),
-                    [chunking_strategy] * len(url_models),
-                    *[kwargs] * len(url_models),
-                )
-            )
-
-        return results
-
-    def run(
-            self,
-            url: str,
-            word_count_threshold=MIN_WORD_THRESHOLD,
-            extraction_strategy: ExtractionStrategy = None,
-            chunking_strategy: ChunkingStrategy = RegexChunking(),
-            bypass_cache: bool = False,
-            css_selector: str = None,
-            screenshot: bool = False,
-            user_agent: str = None,
-            verbose=True,
-            **kwargs,
-        ) -> CrawlResult:
-            extraction_strategy = extraction_strategy or NoExtractionStrategy()
-            extraction_strategy.verbose = verbose
-            if not isinstance(extraction_strategy, ExtractionStrategy):
-                raise ValueError("Unsupported extraction strategy")
-            if not isinstance(chunking_strategy, ChunkingStrategy):
-                raise ValueError("Unsupported chunking strategy")
-            
-            if word_count_threshold < MIN_WORD_THRESHOLD:
-                word_count_threshold = MIN_WORD_THRESHOLD
-
-            # Check cache first
-            cached = None
-            extracted_content = None
-            if not bypass_cache and not self.always_by_pass_cache:
-                cached = get_cached_url(url)
-            
-            if cached:
-                html = cached[1]
-                extracted_content = cached[2]
-                if screenshot:
-                    screenshot = cached[9]
-            
-            else:
-                if user_agent:
-                    self.crawler_strategy.update_user_agent(user_agent)
-                html = self.crawler_strategy.crawl(url)
-                if screenshot:
-                    screenshot = self.crawler_strategy.take_screenshot()
-            
-            return self.process_html(url, html, extracted_content, word_count_threshold, extraction_strategy, chunking_strategy, css_selector, screenshot, verbose, bool(cached), **kwargs)
-
-    def process_html(
-            self,
-            url: str,
-            html: str,
-            extracted_content: str,
-            word_count_threshold: int,
-            extraction_strategy: ExtractionStrategy,
-            chunking_strategy: ChunkingStrategy,
-            css_selector: str,
-            screenshot: bool,
-            verbose: bool,
-            is_cached: bool,
-            **kwargs,
-        ) -> CrawlResult:
-            t = time.time()
-            # Extract content from HTML
-            try:
-                result = get_content_of_website(url, html, word_count_threshold, css_selector=css_selector)
-                metadata = extract_metadata(html)
-                if result is None:
-                    raise ValueError(f"Failed to extract content from the website: {url}")
-            except InvalidCSSSelectorError as e:
-                raise ValueError(str(e))
-            
-            cleaned_html = result.get("cleaned_html", "")
-            markdown = result.get("markdown", "")
-            media = result.get("media", [])
-            links = result.get("links", [])
-
-            if verbose:
-                print(f"[LOG] 🚀 Crawling done for {url}, success: True, time taken: {time.time() - t} seconds")
-                        
-            if extracted_content is None:
-                if verbose:
-                    print(f"[LOG] 🔥 Extracting semantic blocks for {url}, Strategy: {extraction_strategy.name}")
-
-                sections = chunking_strategy.chunk(markdown)
-                extracted_content = extraction_strategy.run(url, sections)
-                extracted_content = json.dumps(extracted_content)
-
-                if verbose:
-                    print(f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t} seconds.")
-                
-            screenshot = None if not screenshot else screenshot
-            
-            if not is_cached:
-                cache_url(
-                    url,
-                    html,
-                    cleaned_html,
-                    markdown,
-                    extracted_content,
-                    True,
-                    json.dumps(media),
-                    json.dumps(links),
-                    json.dumps(metadata),
-                    screenshot=screenshot,
-                )                
-
-            return CrawlResult(
-                url=url,
-                html=html,
-                cleaned_html=cleaned_html,
-                markdown=markdown,
-                media=media,
-                links=links,
-                metadata=metadata,
-                screenshot=screenshot,
-                extracted_content=extracted_content,
-                success=True,
-                error_message="",
-            )

crawl4ai/web_crawler.py

@@ -10,6 +10,7 @@ from .extraction_strategy import *
 from .crawler_strategy import *
 from typing import List
 from concurrent.futures import ThreadPoolExecutor
+from .content_scraping_strategy import WebScrapingStrategy
 from .config import *
 import warnings
 import json
@@ -20,7 +21,7 @@ class WebCrawler:
     def __init__(self, crawler_strategy: CrawlerStrategy = None, always_by_pass_cache: bool = False, verbose: bool = False):
         self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose)
         self.always_by_pass_cache = always_by_pass_cache
-        self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        self.crawl4ai_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
         os.makedirs(self.crawl4ai_folder, exist_ok=True)
         os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
         init_db()
@@ -181,7 +182,21 @@ class WebCrawler:
             # Extract content from HTML
             try:
                 t1 = time.time()
-                result = get_content_of_website_optimized(url, html, word_count_threshold, css_selector=css_selector, only_text=kwargs.get("only_text", False))
+                scrapping_strategy = WebScrapingStrategy()
+                extra_params = {k: v for k, v in kwargs.items() if k not in ["only_text", "image_description_min_word_threshold"]}
+                result = scrapping_strategy.scrap(
+                    url,
+                    html,
+                    word_count_threshold=word_count_threshold,
+                    css_selector=css_selector,
+                    only_text=kwargs.get("only_text", False),
+                    image_description_min_word_threshold=kwargs.get(
+                        "image_description_min_word_threshold", IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD
+                    ),
+                    **extra_params,
+                )
+                
+                # result = get_content_of_website_optimized(url, html, word_count_threshold, css_selector=css_selector, only_text=kwargs.get("only_text", False))
                 if verbose:
                     print(f"[LOG] 🚀 Content extracted for {url}, success: True, time taken: {time.time() - t1:.2f} seconds")
                 

docker-compose.yml

@@ -0,0 +1,62 @@
+services:
+  crawl4ai:
+    build:
+      context: .
+      dockerfile: Dockerfile
+      args:
+        PYTHON_VERSION: "3.10"
+        INSTALL_TYPE: ${INSTALL_TYPE:-basic}
+        ENABLE_GPU: false
+    profiles: ["local"]
+    ports:
+      - "11235:11235"
+      - "8000:8000"
+      - "9222:9222"
+      - "8080:8080"
+    environment:
+      - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
+      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
+      - CLAUDE_API_KEY=${CLAUDE_API_KEY:-}
+    volumes:
+      - /dev/shm:/dev/shm
+    deploy:
+      resources:
+        limits:
+          memory: 4G
+        reservations:
+          memory: 1G
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+
+  crawl4ai-hub:
+    image: unclecode/crawl4ai:basic
+    profiles: ["hub"]
+    ports:
+      - "11235:11235"
+      - "8000:8000"
+      - "9222:9222"
+      - "8080:8080"
+    environment:
+      - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
+      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
+      - CLAUDE_API_KEY=${CLAUDE_API_KEY:-}
+    volumes:
+      - /dev/shm:/dev/shm
+    deploy:
+      resources:
+        limits:
+          memory: 4G
+        reservations:
+          memory: 1G
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s

docs/assets/pitch-dark.svg

@@ -0,0 +1,64 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 500">
+    <!-- Background -->
+    <rect width="800" height="500" fill="#1a1a1a"/>
+    
+    <!-- Opportunities Section -->
+    <g transform="translate(50,50)">
+        <!-- Opportunity 1 Box -->
+        <rect x="0" y="0" width="300" height="150" rx="10" fill="#1a2d3d" stroke="#64b5f6" stroke-width="2"/>
+        <text x="150" y="30" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#64b5f6">Data Capitalization Opportunity</text>
+        <text x="150" y="60" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
+            <tspan x="150" dy="0">Transform digital footprints into assets</tspan>
+            <tspan x="150" dy="20">Personal data as capital</tspan>
+            <tspan x="150" dy="20">Enterprise knowledge valuation</tspan>
+            <tspan x="150" dy="20">New form of wealth creation</tspan>
+        </text>
+
+        <!-- Opportunity 2 Box -->
+        <rect x="0" y="200" width="300" height="150" rx="10" fill="#1a2d1a" stroke="#81c784" stroke-width="2"/>
+        <text x="150" y="230" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#81c784">Authentic Data Potential</text>
+        <text x="150" y="260" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
+            <tspan x="150" dy="0">Vast reservoir of real insights</tspan>
+            <tspan x="150" dy="20">Enhanced AI development</tspan>
+            <tspan x="150" dy="20">Diverse human knowledge</tspan>
+            <tspan x="150" dy="20">Willing participation model</tspan>
+        </text>
+    </g>
+
+    <!-- Development Pathway -->
+    <g transform="translate(450,50)">
+        <!-- Step 1 Box -->
+        <rect x="0" y="0" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">1. Open-Source Foundation</text>
+        <text x="150" y="65" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Data extraction engine &amp; community development</text>
+
+        <!-- Step 2 Box -->
+        <rect x="0" y="125" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="160" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">2. Data Capitalization Platform</text>
+        <text x="150" y="190" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Tools to structure &amp; value digital assets</text>
+
+        <!-- Step 3 Box -->
+        <rect x="0" y="250" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
+        <text x="150" y="285" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">3. Shared Data Marketplace</text>
+        <text x="150" y="315" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Economic platform for data exchange</text>
+    </g>
+
+    <!-- Connecting Arrows -->
+    <g transform="translate(400,125)">
+        <path d="M-20,0 L40,0" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
+        <path d="M-20,200 L40,200" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
+    </g>
+
+    <!-- Arrow Marker -->
+    <defs>
+        <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#666"/>
+        </marker>
+    </defs>
+
+    <!-- Vision Box at Bottom -->
+    <g transform="translate(200,420)">
+        <rect x="0" y="0" width="400" height="60" rx="10" fill="#2d2613" stroke="#ffd54f" stroke-width="2"/>
+        <text x="200" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ffd54f">Economic Vision: Shared Data Economy</text>
+    </g>
+</svg>

docs/examples/docker_example.py

@@ -0,0 +1,357 @@
+import requests
+import json
+import time
+import sys
+import base64
+import os
+from typing import Dict, Any
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:11235", api_token: str = None):
+        self.base_url = base_url
+        self.api_token = api_token or os.getenv('CRAWL4AI_API_TOKEN') or "test_api_code"  # Check environment variable as fallback
+        self.headers = {'Authorization': f'Bearer {self.api_token}'} if self.api_token else {}
+        
+    def submit_and_wait(self, request_data: Dict[str, Any], timeout: int = 300) -> Dict[str, Any]:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data, headers=self.headers)
+        if response.status_code == 403:
+            raise Exception("API token is invalid or missing")
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}", headers=self.headers)
+            status = result.json()
+            
+            if status["status"] == "failed":
+                print("Task failed:", status.get("error"))
+                raise Exception(f"Task failed: {status.get('error')}")
+                
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+            
+    def submit_sync(self, request_data: Dict[str, Any]) -> Dict[str, Any]:
+        response = requests.post(f"{self.base_url}/crawl_sync", json=request_data, headers=self.headers, timeout=60)
+        if response.status_code == 408:
+            raise TimeoutError("Task did not complete within server timeout")
+        response.raise_for_status()
+        return response.json()
+    
+    def crawl_direct(self, request_data: Dict[str, Any]) -> Dict[str, Any]:
+        """Directly crawl without using task queue"""
+        response = requests.post(
+            f"{self.base_url}/crawl_direct", 
+            json=request_data, 
+            headers=self.headers
+        )
+        response.raise_for_status()
+        return response.json()
+
+def test_docker_deployment(version="basic"):
+    tester = Crawl4AiTester(
+        base_url="http://localhost:11235" ,
+        # base_url="https://api.crawl4ai.com" # just for example
+        # api_token="test" # just for example
+    )
+    print(f"Testing Crawl4AI Docker {version} version")
+    
+    # Health check with timeout and retry
+    max_retries = 5
+    for i in range(max_retries):
+        try:
+            health = requests.get(f"{tester.base_url}/health", timeout=10)
+            print("Health check:", health.json())
+            break
+        except requests.exceptions.RequestException as e:
+            if i == max_retries - 1:
+                print(f"Failed to connect after {max_retries} attempts")
+                sys.exit(1)
+            print(f"Waiting for service to start (attempt {i+1}/{max_retries})...")
+            time.sleep(5)
+    
+    # Test cases based on version
+    # test_basic_crawl(tester)
+    # test_basic_crawl(tester)
+    # test_basic_crawl_sync(tester)
+    test_basic_crawl_direct(tester)
+    
+    # if version in ["full", "transformer"]:
+    #     test_cosine_extraction(tester)
+
+    # test_js_execution(tester)
+    # test_css_selector(tester)
+    # test_structured_extraction(tester)
+    # test_llm_extraction(tester)
+    # test_llm_with_ollama(tester)
+    # test_screenshot(tester)
+    
+
+def test_basic_crawl(tester: Crawl4AiTester):
+    print("\n=== Testing Basic Crawl ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10, 
+        "session_id": "test"
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+    assert len(result["result"]["markdown"]) > 0
+
+def test_basic_crawl_sync(tester: Crawl4AiTester):
+    print("\n=== Testing Basic Crawl (Sync) ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10,
+        "session_id": "test"
+    }
+    
+    result = tester.submit_sync(request)
+    print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+    assert result['status'] == 'completed'
+    assert result['result']['success']
+    assert len(result['result']['markdown']) > 0
+    
+def test_basic_crawl_direct(tester: Crawl4AiTester):
+    print("\n=== Testing Basic Crawl (Direct) ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10,
+        # "session_id": "test"
+        "cache_mode": "bypass"  # or "enabled", "disabled", "read_only", "write_only"
+    }
+    
+    result = tester.crawl_direct(request)
+    print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+    assert result['result']['success']
+    assert len(result['result']['markdown']) > 0
+    
+def test_js_execution(tester: Crawl4AiTester):
+    print("\n=== Testing JS Execution ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "js_code": [
+            "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+        ],
+        "wait_for": "article.tease-card:nth-child(10)",
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"JS execution result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_css_selector(tester: Crawl4AiTester):
+    print("\n=== Testing CSS Selector ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 7,
+        "css_selector": ".wide-tease-item__description",
+        "crawler_params": {
+            "headless": True
+        },
+        "extra": {"word_count_threshold": 10}
+        
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"CSS selector result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_structured_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Structured Extraction ===")
+    schema = {
+        "name": "Coinbase Crypto Prices",
+        "baseSelector": ".cds-tableRow-t45thuk",
+        "fields": [
+            {
+                "name": "crypto",
+                "selector": "td:nth-child(1) h2",
+                "type": "text",
+            },
+            {
+                "name": "symbol",
+                "selector": "td:nth-child(1) p",
+                "type": "text",
+            },
+            {
+                "name": "price",
+                "selector": "td:nth-child(2)",
+                "type": "text",
+            }
+        ],
+    }
+    
+    request = {
+        "urls": "https://www.coinbase.com/explore",
+        "priority": 9,
+        "extraction_config": {
+            "type": "json_css",
+            "params": {
+                "schema": schema
+            }
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    extracted = json.loads(result["result"]["extracted_content"])
+    print(f"Extracted {len(extracted)} items")
+    print("Sample item:", json.dumps(extracted[0], indent=2))
+    assert result["result"]["success"]
+    assert len(extracted) > 0
+
+def test_llm_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing LLM Extraction ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "model_name": {
+                "type": "string",
+                "description": "Name of the OpenAI model."
+            },
+            "input_fee": {
+                "type": "string",
+                "description": "Fee for input token for the OpenAI model."
+            },
+            "output_fee": {
+                "type": "string",
+                "description": "Fee for output token for the OpenAI model."
+            }
+        },
+        "required": ["model_name", "input_fee", "output_fee"]
+    }
+    
+    request = {
+        "urls": "https://openai.com/api/pricing",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "openai/gpt-4o-mini",
+                "api_token": os.getenv("OPENAI_API_KEY"),
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": """From the crawled content, extract all mentioned model names along with their fees for input and output tokens."""
+            }
+        },
+        "crawler_params": {"word_count_threshold": 1}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} model pricing entries")
+        print("Sample entry:", json.dumps(extracted[0], indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"LLM extraction test failed (might be due to missing API key): {str(e)}")
+
+def test_llm_with_ollama(tester: Crawl4AiTester):
+    print("\n=== Testing LLM with Ollama ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "article_title": {
+                "type": "string",
+                "description": "The main title of the news article"
+            },
+            "summary": {
+                "type": "string",
+                "description": "A brief summary of the article content"
+            },
+            "main_topics": {
+                "type": "array",
+                "items": {"type": "string"},
+                "description": "Main topics or themes discussed in the article"
+            }
+        }
+    }
+    
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "ollama/llama2",
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": "Extract the main article information including title, summary, and main topics."
+            }
+        },
+        "extra": {"word_count_threshold": 1},
+        "crawler_params": {"verbose": True}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print("Extracted content:", json.dumps(extracted, indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Ollama extraction test failed: {str(e)}")
+
+def test_cosine_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Cosine Extraction ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "cosine",
+            "params": {
+                "semantic_filter": "business finance economy",
+                "word_count_threshold": 10,
+                "max_dist": 0.2,
+                "top_k": 3
+            }
+        }
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} text clusters")
+        print("First cluster tags:", extracted[0]["tags"])
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Cosine extraction test failed: {str(e)}")
+
+def test_screenshot(tester: Crawl4AiTester):
+    print("\n=== Testing Screenshot ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 5,
+        "screenshot": True,
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Screenshot captured:", bool(result["result"]["screenshot"]))
+    
+    if result["result"]["screenshot"]:
+        # Save screenshot
+        screenshot_data = base64.b64decode(result["result"]["screenshot"])
+        with open("test_screenshot.jpg", "wb") as f:
+            f.write(screenshot_data)
+        print("Screenshot saved as test_screenshot.jpg")
+    
+    assert result["result"]["success"]
+
+if __name__ == "__main__":
+    version = sys.argv[1] if len(sys.argv) > 1 else "basic"
+    # version = "full"
+    test_docker_deployment(version)

docs/examples/quickstart_async.py

@@ -13,7 +13,9 @@ import re
 from typing import Dict, List
 from bs4 import BeautifulSoup
 from pydantic import BaseModel, Field
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
+from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
+from crawl4ai.content_filter_strategy import BM25ContentFilter
 from crawl4ai.extraction_strategy import (
     JsonCssExtractionStrategy,
     LLMExtractionStrategy,
@@ -51,7 +53,7 @@ async def simple_example_with_running_js_code():
             url="https://www.nbcnews.com/business",
             js_code=js_code,
             # wait_for=wait_for,
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
         )
         print(result.markdown[:500])  # Print first 500 characters
 
@@ -61,7 +63,7 @@ async def simple_example_with_css_selector():
         result = await crawler.arun(
             url="https://www.nbcnews.com/business",
             css_selector=".wide-tease-item__description",
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
         )
         print(result.markdown[:500])  # Print first 500 characters
 
@@ -71,12 +73,12 @@ async def use_proxy():
         "Note: Replace 'http://your-proxy-url:port' with a working proxy to run this example."
     )
     # Uncomment and modify the following lines to use a proxy
-    # async with AsyncWebCrawler(verbose=True, proxy="http://your-proxy-url:port") as crawler:
-    #     result = await crawler.arun(
-    #         url="https://www.nbcnews.com/business",
-    #         bypass_cache=True
-    #     )
-    #     print(result.markdown[:500])  # Print first 500 characters
+    async with AsyncWebCrawler(verbose=True, proxy="http://your-proxy-url:port") as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            bypass_cache=True
+        )
+        print(result.markdown[:500])  # Print first 500 characters
 
 async def capture_and_save_screenshot(url: str, output_path: str):
     async with AsyncWebCrawler(verbose=True) as crawler:
@@ -132,7 +134,7 @@ async def extract_structured_data_using_llm(provider: str, api_token: str = None
                 {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}.""",
                 extra_args=extra_args
             ),
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
         )
         print(result.extracted_content)
 
@@ -166,7 +168,7 @@ async def extract_structured_data_using_css_extractor():
         result = await crawler.arun(
             url="https://www.coinbase.com/explore",
             extraction_strategy=extraction_strategy,
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
         )
 
         assert result.success, "Failed to crawl the page"
@@ -213,7 +215,7 @@ async def crawl_dynamic_content_pages_method_1():
                 session_id=session_id,
                 css_selector="li.Box-sc-g0xbh4-0",
                 js=js_next_page if page > 0 else None,
-                bypass_cache=True,
+                cache_mode=CacheMode.BYPASS,
                 js_only=page > 0,
                 headless=False,
             )
@@ -282,7 +284,7 @@ async def crawl_dynamic_content_pages_method_2():
                 extraction_strategy=extraction_strategy,
                 js_code=js_next_page_and_wait if page > 0 else None,
                 js_only=page > 0,
-                bypass_cache=True,
+                cache_mode=CacheMode.BYPASS,
                 headless=False,
             )
 
@@ -343,7 +345,7 @@ async def crawl_dynamic_content_pages_method_3():
                 js_code=js_next_page if page > 0 else None,
                 wait_for=wait_for if page > 0 else None,
                 js_only=page > 0,
-                bypass_cache=True,
+                cache_mode=CacheMode.BYPASS,
                 headless=False,
             )
 
@@ -383,10 +385,11 @@ async def crawl_with_user_simultion():
     async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
         url = "YOUR-URL-HERE"
         result = await crawler.arun(
-            url=url,
-            bypass_cache=True,
-            simulate_user = True,# Causes a series of random mouse movements and clicks to simulate user interaction
-            override_navigator = True # Overrides the navigator object to make it look like a real user
+            url=url,            
+            cache_mode=CacheMode.BYPASS,
+            magic = True, # Automatically detects and removes overlays, popups, and other elements that block content
+            # simulate_user = True,# Causes a series of random mouse movements and clicks to simulate user interaction
+            # override_navigator = True # Overrides the navigator object to make it look like a real user
         )
         
         print(result.markdown)    
@@ -407,7 +410,7 @@ async def speed_comparison():
     params={'formats': ['markdown', 'html']}
     )
     end = time.time()
-    print("Firecrawl (simulated):")
+    print("Firecrawl:")
     print(f"Time taken: {end - start:.2f} seconds")
     print(f"Content length: {len(scrape_status['markdown'])} characters")
     print(f"Images found: {scrape_status['markdown'].count('cldnry.s-nbcnews.com')}")
@@ -419,7 +422,7 @@ async def speed_comparison():
         result = await crawler.arun(
             url="https://www.nbcnews.com/business",
             word_count_threshold=0,
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
             verbose=False,
         )
         end = time.time()
@@ -429,6 +432,25 @@ async def speed_comparison():
         print(f"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}")
         print()
 
+        # Crawl4AI with advanced content filtering
+        start = time.time()
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            word_count_threshold=0,
+            markdown_generator=DefaultMarkdownGenerator(
+                content_filter=BM25ContentFilter(user_query=None, bm25_threshold=1.0)
+            ),
+            cache_mode=CacheMode.BYPASS,
+            verbose=False,
+        )
+        end = time.time()
+        print("Crawl4AI (Markdown Plus):")
+        print(f"Time taken: {end - start:.2f} seconds")
+        print(f"Content length: {len(result.markdown_v2.raw_markdown)} characters")
+        print(f"Fit Markdown: {len(result.markdown_v2.fit_markdown)} characters")
+        print(f"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}")
+        print()
+
         # Crawl4AI with JavaScript execution
         start = time.time()
         result = await crawler.arun(
@@ -437,13 +459,17 @@ async def speed_comparison():
                 "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
             ],
             word_count_threshold=0,
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
+            markdown_generator=DefaultMarkdownGenerator(
+                content_filter=BM25ContentFilter(user_query=None, bm25_threshold=1.0)
+            ),
             verbose=False,
         )
         end = time.time()
         print("Crawl4AI (with JavaScript execution):")
         print(f"Time taken: {end - start:.2f} seconds")
         print(f"Content length: {len(result.markdown)} characters")
+        print(f"Fit Markdown: {len(result.markdown_v2.fit_markdown)} characters")
         print(f"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}")
 
     print("\nNote on Speed Comparison:")
@@ -482,7 +508,7 @@ async def generate_knowledge_graph():
         url = "https://paulgraham.com/love.html"
         result = await crawler.arun(
             url=url,
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
             extraction_strategy=extraction_strategy,
             # magic=True
         )
@@ -495,7 +521,7 @@ async def fit_markdown_remove_overlay():
         url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
         result = await crawler.arun(
             url=url,
-            bypass_cache=True,
+            cache_mode=CacheMode.BYPASS,
             word_count_threshold = 10,
             remove_overlay_elements=True,
             screenshot = True
@@ -516,10 +542,10 @@ async def main():
     await extract_structured_data_using_css_extractor()
 
     # LLM extraction examples
-    await extract_structured_data_using_llm()
-    await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
+    # await extract_structured_data_using_llm()
+    # await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
+    # await extract_structured_data_using_llm("ollama/llama3.2")    
     await extract_structured_data_using_llm("openai/gpt-4o", os.getenv("OPENAI_API_KEY"))
-    await extract_structured_data_using_llm("ollama/llama3.2")    
 
     # You always can pass custom headers to the extraction strategy
     custom_headers = {

docs/examples/quickstart_v0.ipynb

@@ -0,0 +1,735 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6yLvrXn7yZQI"
+      },
+      "source": [
+        "# Crawl4AI: Advanced Web Crawling and Data Extraction\n",
+        "\n",
+        "Welcome to this interactive notebook showcasing Crawl4AI, an advanced asynchronous web crawling and data extraction library.\n",
+        "\n",
+        "- GitHub Repository: [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
+        "- Twitter: [@unclecode](https://twitter.com/unclecode)\n",
+        "- Website: [https://crawl4ai.com](https://crawl4ai.com)\n",
+        "\n",
+        "Let's explore the powerful features of Crawl4AI!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KIn_9nxFyZQK"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "First, let's install Crawl4AI from GitHub:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mSnaxLf3zMog"
+      },
+      "outputs": [],
+      "source": [
+        "!sudo apt-get update && sudo apt-get install -y libwoff1 libopus0 libwebp6 libwebpdemux2 libenchant1c2a libgudev-1.0-0 libsecret-1-0 libhyphen0 libgdk-pixbuf2.0-0 libegl1 libnotify4 libxslt1.1 libevent-2.1-7 libgles2 libvpx6 libxcomposite1 libatk1.0-0 libatk-bridge2.0-0 libepoxy0 libgtk-3-0 libharfbuzz-icu0"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xlXqaRtayZQK"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install crawl4ai\n",
+        "!pip install nest-asyncio\n",
+        "!playwright install"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qKCE7TI7yZQL"
+      },
+      "source": [
+        "Now, let's import the necessary libraries:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "I67tr7aAyZQL"
+      },
+      "outputs": [],
+      "source": [
+        "import asyncio\n",
+        "import nest_asyncio\n",
+        "from crawl4ai import AsyncWebCrawler\n",
+        "from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy\n",
+        "import json\n",
+        "import time\n",
+        "from pydantic import BaseModel, Field\n",
+        "\n",
+        "nest_asyncio.apply()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "h7yR_Rt_yZQM"
+      },
+      "source": [
+        "## Basic Usage\n",
+        "\n",
+        "Let's start with a simple crawl example:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "yBh6hf4WyZQM",
+        "outputId": "0f83af5c-abba-4175-ed95-70b7512e6bcc"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.05 seconds\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.05 seconds.\n",
+            "18102\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def simple_crawl():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(url=\"https://www.nbcnews.com/business\")\n",
+        "        print(len(result.markdown))\n",
+        "await simple_crawl()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9rtkgHI28uI4"
+      },
+      "source": [
+        "💡 By default, **Crawl4AI** caches the result of every URL, so the next time you call it, you’ll get an instant result. But if you want to bypass the cache, just set `bypass_cache=True`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MzZ0zlJ9yZQM"
+      },
+      "source": [
+        "## Advanced Features\n",
+        "\n",
+        "### Executing JavaScript and Using CSS Selectors"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "gHStF86xyZQM",
+        "outputId": "34d0fb6d-4dec-4677-f76e-85a1f082829b"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
+            "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 6.06 seconds\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.10 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.11 seconds.\n",
+            "41135\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def js_and_css():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        js_code = [\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"]\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            js_code=js_code,\n",
+        "            # css_selector=\"YOUR_CSS_SELECTOR_HERE\",\n",
+        "            bypass_cache=True\n",
+        "        )\n",
+        "        print(len(result.markdown))\n",
+        "\n",
+        "await js_and_css()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cqE_W4coyZQM"
+      },
+      "source": [
+        "### Using a Proxy\n",
+        "\n",
+        "Note: You'll need to replace the proxy URL with a working proxy for this example to run successfully."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QjAyiAGqyZQM"
+      },
+      "outputs": [],
+      "source": [
+        "async def use_proxy():\n",
+        "    async with AsyncWebCrawler(verbose=True, proxy=\"http://your-proxy-url:port\") as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            bypass_cache=True\n",
+        "        )\n",
+        "        print(result.markdown[:500])  # Print first 500 characters\n",
+        "\n",
+        "# Uncomment the following line to run the proxy example\n",
+        "# await use_proxy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XTZ88lbayZQN"
+      },
+      "source": [
+        "### Extracting Structured Data with OpenAI\n",
+        "\n",
+        "Note: You'll need to set your OpenAI API key as an environment variable for this example to work."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 14,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "fIOlDayYyZQN",
+        "outputId": "cb8359cc-dee0-4762-9698-5dfdcee055b8"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://openai.com/api/pricing/ using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://openai.com/api/pricing/ successfully!\n",
+            "[LOG] 🚀 Crawling done for https://openai.com/api/pricing/, success: True, time taken: 3.77 seconds\n",
+            "[LOG] 🚀 Content extracted for https://openai.com/api/pricing/, success: True, time taken: 0.21 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://openai.com/api/pricing/, Strategy: AsyncWebCrawler\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 0\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 1\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 2\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 3\n",
+            "[LOG] Extracted 4 blocks from URL: https://openai.com/api/pricing/ block index: 3\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 4\n",
+            "[LOG] Extracted 5 blocks from URL: https://openai.com/api/pricing/ block index: 0\n",
+            "[LOG] Extracted 1 blocks from URL: https://openai.com/api/pricing/ block index: 4\n",
+            "[LOG] Extracted 8 blocks from URL: https://openai.com/api/pricing/ block index: 1\n",
+            "[LOG] Extracted 12 blocks from URL: https://openai.com/api/pricing/ block index: 2\n",
+            "[LOG] 🚀 Extraction done for https://openai.com/api/pricing/, time taken: 8.55 seconds.\n",
+            "5029\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "from google.colab import userdata\n",
+        "os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')\n",
+        "\n",
+        "class OpenAIModelFee(BaseModel):\n",
+        "    model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n",
+        "    input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n",
+        "    output_fee: str = Field(..., description=\"Fee for output token for the OpenAI model.\")\n",
+        "\n",
+        "async def extract_openai_fees():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url='https://openai.com/api/pricing/',\n",
+        "            word_count_threshold=1,\n",
+        "            extraction_strategy=LLMExtractionStrategy(\n",
+        "                provider=\"openai/gpt-4o\", api_token=os.getenv('OPENAI_API_KEY'),\n",
+        "                schema=OpenAIModelFee.schema(),\n",
+        "                extraction_type=\"schema\",\n",
+        "                instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens.\n",
+        "                Do not miss any models in the entire content. One extracted model JSON format should look like this:\n",
+        "                {\"model_name\": \"GPT-4\", \"input_fee\": \"US$10.00 / 1M tokens\", \"output_fee\": \"US$30.00 / 1M tokens\"}.\"\"\"\n",
+        "            ),\n",
+        "            bypass_cache=True,\n",
+        "        )\n",
+        "        print(len(result.extracted_content))\n",
+        "\n",
+        "# Uncomment the following line to run the OpenAI extraction example\n",
+        "await extract_openai_fees()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BypA5YxEyZQN"
+      },
+      "source": [
+        "### Advanced Multi-Page Crawling with JavaScript Execution"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tfkcVQ0b7mw-"
+      },
+      "source": [
+        "## Advanced Multi-Page Crawling with JavaScript Execution\n",
+        "\n",
+        "This example demonstrates Crawl4AI's ability to handle complex crawling scenarios, specifically extracting commits from multiple pages of a GitHub repository. The challenge here is that clicking the \"Next\" button doesn't load a new page, but instead uses asynchronous JavaScript to update the content. This is a common hurdle in modern web crawling.\n",
+        "\n",
+        "To overcome this, we use Crawl4AI's custom JavaScript execution to simulate clicking the \"Next\" button, and implement a custom hook to detect when new data has loaded. Our strategy involves comparing the first commit's text before and after \"clicking\" Next, waiting until it changes to confirm new data has rendered. This showcases Crawl4AI's flexibility in handling dynamic content and its ability to implement custom logic for even the most challenging crawling tasks."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "qUBKGpn3yZQN",
+        "outputId": "3e555b6a-ed33-42f4-cce9-499a923fbe17"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 5.16 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.28 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.28 seconds.\n",
+            "Page 1: Found 35 commits\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.78 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.90 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.90 seconds.\n",
+            "Page 2: Found 35 commits\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 2.00 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.74 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.75 seconds.\n",
+            "Page 3: Found 35 commits\n",
+            "Successfully crawled 105 commits across 3 pages\n"
+          ]
+        }
+      ],
+      "source": [
+        "import re\n",
+        "from bs4 import BeautifulSoup\n",
+        "\n",
+        "async def crawl_typescript_commits():\n",
+        "    first_commit = \"\"\n",
+        "    async def on_execution_started(page):\n",
+        "        nonlocal first_commit\n",
+        "        try:\n",
+        "            while True:\n",
+        "                await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4')\n",
+        "                commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4')\n",
+        "                commit = await commit.evaluate('(element) => element.textContent')\n",
+        "                commit = re.sub(r'\\s+', '', commit)\n",
+        "                if commit and commit != first_commit:\n",
+        "                    first_commit = commit\n",
+        "                    break\n",
+        "                await asyncio.sleep(0.5)\n",
+        "        except Exception as e:\n",
+        "            print(f\"Warning: New content didn't appear after JavaScript execution: {e}\")\n",
+        "\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started)\n",
+        "\n",
+        "        url = \"https://github.com/microsoft/TypeScript/commits/main\"\n",
+        "        session_id = \"typescript_commits_session\"\n",
+        "        all_commits = []\n",
+        "\n",
+        "        js_next_page = \"\"\"\n",
+        "        const button = document.querySelector('a[data-testid=\"pagination-next-button\"]');\n",
+        "        if (button) button.click();\n",
+        "        \"\"\"\n",
+        "\n",
+        "        for page in range(3):  # Crawl 3 pages\n",
+        "            result = await crawler.arun(\n",
+        "                url=url,\n",
+        "                session_id=session_id,\n",
+        "                css_selector=\"li.Box-sc-g0xbh4-0\",\n",
+        "                js=js_next_page if page > 0 else None,\n",
+        "                bypass_cache=True,\n",
+        "                js_only=page > 0\n",
+        "            )\n",
+        "\n",
+        "            assert result.success, f\"Failed to crawl page {page + 1}\"\n",
+        "\n",
+        "            soup = BeautifulSoup(result.cleaned_html, 'html.parser')\n",
+        "            commits = soup.select(\"li\")\n",
+        "            all_commits.extend(commits)\n",
+        "\n",
+        "            print(f\"Page {page + 1}: Found {len(commits)} commits\")\n",
+        "\n",
+        "        await crawler.crawler_strategy.kill_session(session_id)\n",
+        "        print(f\"Successfully crawled {len(all_commits)} commits across 3 pages\")\n",
+        "\n",
+        "await crawl_typescript_commits()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EJRnYsp6yZQN"
+      },
+      "source": [
+        "### Using JsonCssExtractionStrategy for Fast Structured Output"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1ZMqIzB_8SYp"
+      },
+      "source": [
+        "The JsonCssExtractionStrategy is a powerful feature of Crawl4AI that allows for precise, structured data extraction from web pages. Here's how it works:\n",
+        "\n",
+        "1. You define a schema that describes the pattern of data you're interested in extracting.\n",
+        "2. The schema includes a base selector that identifies repeating elements on the page.\n",
+        "3. Within the schema, you define fields, each with its own selector and type.\n",
+        "4. These field selectors are applied within the context of each base selector element.\n",
+        "5. The strategy supports nested structures, lists within lists, and various data types.\n",
+        "6. You can even include computed fields for more complex data manipulation.\n",
+        "\n",
+        "This approach allows for highly flexible and precise data extraction, transforming semi-structured web content into clean, structured JSON data. It's particularly useful for extracting consistent data patterns from pages like product listings, news articles, or search results.\n",
+        "\n",
+        "For more details and advanced usage, check out the full documentation on the Crawl4AI website."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "trCMR2T9yZQN",
+        "outputId": "718d36f4-cccf-40f4-8d8c-c3ba73524d16"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
+            "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 7.00 seconds\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.32 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.48 seconds.\n",
+            "Successfully extracted 11 news teasers\n",
+            "{\n",
+            "  \"category\": \"Business News\",\n",
+            "  \"headline\": \"NBC ripped up its Olympics playbook for 2024 \\u2014 so far, the new strategy paid off\",\n",
+            "  \"summary\": \"The Olympics have long been key to NBCUniversal. Paris marked the 18th Olympic Games broadcast by NBC in the U.S.\",\n",
+            "  \"time\": \"13h ago\",\n",
+            "  \"image\": {\n",
+            "    \"src\": \"https://media-cldnry.s-nbcnews.com/image/upload/t_focal-200x100,f_auto,q_auto:best/rockcms/2024-09/240903-nbc-olympics-ch-1344-c7a486.jpg\",\n",
+            "    \"alt\": \"Mike Tirico.\"\n",
+            "  },\n",
+            "  \"link\": \"https://www.nbcnews.com/business\"\n",
+            "}\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def extract_news_teasers():\n",
+        "    schema = {\n",
+        "        \"name\": \"News Teaser Extractor\",\n",
+        "        \"baseSelector\": \".wide-tease-item__wrapper\",\n",
+        "        \"fields\": [\n",
+        "            {\n",
+        "                \"name\": \"category\",\n",
+        "                \"selector\": \".unibrow span[data-testid='unibrow-text']\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"headline\",\n",
+        "                \"selector\": \".wide-tease-item__headline\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"summary\",\n",
+        "                \"selector\": \".wide-tease-item__description\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"time\",\n",
+        "                \"selector\": \"[data-testid='wide-tease-date']\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"image\",\n",
+        "                \"type\": \"nested\",\n",
+        "                \"selector\": \"picture.teasePicture img\",\n",
+        "                \"fields\": [\n",
+        "                    {\"name\": \"src\", \"type\": \"attribute\", \"attribute\": \"src\"},\n",
+        "                    {\"name\": \"alt\", \"type\": \"attribute\", \"attribute\": \"alt\"},\n",
+        "                ],\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"link\",\n",
+        "                \"selector\": \"a[href]\",\n",
+        "                \"type\": \"attribute\",\n",
+        "                \"attribute\": \"href\",\n",
+        "            },\n",
+        "        ],\n",
+        "    }\n",
+        "\n",
+        "    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n",
+        "\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            extraction_strategy=extraction_strategy,\n",
+        "            bypass_cache=True,\n",
+        "        )\n",
+        "\n",
+        "        assert result.success, \"Failed to crawl the page\"\n",
+        "\n",
+        "        news_teasers = json.loads(result.extracted_content)\n",
+        "        print(f\"Successfully extracted {len(news_teasers)} news teasers\")\n",
+        "        print(json.dumps(news_teasers[0], indent=2))\n",
+        "\n",
+        "await extract_news_teasers()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FnyVhJaByZQN"
+      },
+      "source": [
+        "## Speed Comparison\n",
+        "\n",
+        "Let's compare the speed of Crawl4AI with Firecrawl, a paid service. Note that we can't run Firecrawl in this Colab environment, so we'll simulate its performance based on previously recorded data."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "agDD186f3wig"
+      },
+      "source": [
+        "💡 **Note on Speed Comparison:**\n",
+        "\n",
+        "The speed test conducted here is running on Google Colab, where the internet speed and performance can vary and may not reflect optimal conditions. When we call Firecrawl's API, we're seeing its best performance, while Crawl4AI's performance is limited by Colab's network speed.\n",
+        "\n",
+        "For a more accurate comparison, it's recommended to run these tests on your own servers or computers with a stable and fast internet connection. Despite these limitations, Crawl4AI still demonstrates faster performance in this environment.\n",
+        "\n",
+        "If you run these tests locally, you may observe an even more significant speed advantage for Crawl4AI compared to other services."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "F7KwHv8G1LbY"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install firecrawl"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "91813zILyZQN",
+        "outputId": "663223db-ab89-4976-b233-05ceca62b19b"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Firecrawl (simulated):\n",
+            "Time taken: 4.38 seconds\n",
+            "Content length: 41967 characters\n",
+            "Images found: 49\n",
+            "\n",
+            "Crawl4AI (simple crawl):\n",
+            "Time taken: 4.22 seconds\n",
+            "Content length: 18221 characters\n",
+            "Images found: 49\n",
+            "\n",
+            "Crawl4AI (with JavaScript execution):\n",
+            "Time taken: 9.13 seconds\n",
+            "Content length: 34243 characters\n",
+            "Images found: 89\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "from google.colab import userdata\n",
+        "os.environ['FIRECRAWL_API_KEY'] = userdata.get('FIRECRAWL_API_KEY')\n",
+        "import time\n",
+        "from firecrawl import FirecrawlApp\n",
+        "\n",
+        "async def speed_comparison():\n",
+        "    # Simulated Firecrawl performance\n",
+        "    app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])\n",
+        "    start = time.time()\n",
+        "    scrape_status = app.scrape_url(\n",
+        "    'https://www.nbcnews.com/business',\n",
+        "    params={'formats': ['markdown', 'html']}\n",
+        "    )\n",
+        "    end = time.time()\n",
+        "    print(\"Firecrawl (simulated):\")\n",
+        "    print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "    print(f\"Content length: {len(scrape_status['markdown'])} characters\")\n",
+        "    print(f\"Images found: {scrape_status['markdown'].count('cldnry.s-nbcnews.com')}\")\n",
+        "    print()\n",
+        "\n",
+        "    async with AsyncWebCrawler() as crawler:\n",
+        "        # Crawl4AI simple crawl\n",
+        "        start = time.time()\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            word_count_threshold=0,\n",
+        "            bypass_cache=True,\n",
+        "            verbose=False\n",
+        "        )\n",
+        "        end = time.time()\n",
+        "        print(\"Crawl4AI (simple crawl):\")\n",
+        "        print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "        print(f\"Content length: {len(result.markdown)} characters\")\n",
+        "        print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
+        "        print()\n",
+        "\n",
+        "        # Crawl4AI with JavaScript execution\n",
+        "        start = time.time()\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            js_code=[\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"],\n",
+        "            word_count_threshold=0,\n",
+        "            bypass_cache=True,\n",
+        "            verbose=False\n",
+        "        )\n",
+        "        end = time.time()\n",
+        "        print(\"Crawl4AI (with JavaScript execution):\")\n",
+        "        print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "        print(f\"Content length: {len(result.markdown)} characters\")\n",
+        "        print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
+        "\n",
+        "await speed_comparison()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "OBFFYVJIyZQN"
+      },
+      "source": [
+        "If you run on a local machine with a proper internet speed:\n",
+        "- Simple crawl: Crawl4AI is typically over 3-4 times faster than Firecrawl.\n",
+        "- With JavaScript execution: Even when executing JavaScript to load more content (potentially doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.\n",
+        "\n",
+        "Please note that actual performance may vary depending on network conditions and the specific content being crawled."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "A6_1RK1_yZQO"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "In this notebook, we've explored the powerful features of Crawl4AI, including:\n",
+        "\n",
+        "1. Basic crawling\n",
+        "2. JavaScript execution and CSS selector usage\n",
+        "3. Proxy support\n",
+        "4. Structured data extraction with OpenAI\n",
+        "5. Advanced multi-page crawling with JavaScript execution\n",
+        "6. Fast structured output using JsonCssExtractionStrategy\n",
+        "7. Speed comparison with other services\n",
+        "\n",
+        "Crawl4AI offers a fast, flexible, and powerful solution for web crawling and data extraction tasks. Its asynchronous architecture and advanced features make it suitable for a wide range of applications, from simple web scraping to complex, multi-page data extraction scenarios.\n",
+        "\n",
+        "For more information and advanced usage, please visit the [Crawl4AI documentation](https://crawl4ai.com/mkdocs/).\n",
+        "\n",
+        "Happy crawling!"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "venv",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.13"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/examples/v0.3.74.overview.py

@@ -0,0 +1,277 @@
+import os, sys
+# append the parent directory to the sys.path
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+parent_parent_dir = os.path.dirname(parent_dir)
+sys.path.append(parent_parent_dir)
+__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
+__data__ = os.path.join(__location__, "__data")
+import asyncio
+from pathlib import Path
+import aiohttp
+import json
+from crawl4ai import AsyncWebCrawler, CacheMode
+from crawl4ai.content_filter_strategy import BM25ContentFilter
+
+# 1. File Download Processing Example
+async def download_example():
+    """Example of downloading files from Python.org"""
+    # downloads_path = os.path.join(os.getcwd(), "downloads")
+    downloads_path = os.path.join(Path.home(), ".crawl4ai", "downloads")
+    os.makedirs(downloads_path, exist_ok=True)
+    
+    print(f"Downloads will be saved to: {downloads_path}")
+    
+    async with AsyncWebCrawler(
+        accept_downloads=True,
+        downloads_path=downloads_path,
+        verbose=True
+    ) as crawler:
+        result = await crawler.arun(
+            url="https://www.python.org/downloads/",
+            js_code="""
+            // Find and click the first Windows installer link
+            const downloadLink = document.querySelector('a[href$=".exe"]');
+            if (downloadLink) {
+                console.log('Found download link:', downloadLink.href);
+                downloadLink.click();
+            } else {
+                console.log('No .exe download link found');
+            }
+            """,
+            delay_before_return_html=1,  # Wait 5 seconds to ensure download starts
+            cache_mode=CacheMode.BYPASS
+        )
+        
+        if result.downloaded_files:
+            print("\nDownload successful!")
+            print("Downloaded files:")
+            for file_path in result.downloaded_files:
+                print(f"- {file_path}")
+                print(f"  File size: {os.path.getsize(file_path) / (1024*1024):.2f} MB")
+        else:
+            print("\nNo files were downloaded")
+
+# 2. Local File and Raw HTML Processing Example
+async def local_and_raw_html_example():
+    """Example of processing local files and raw HTML"""
+    # Create a sample HTML file
+    sample_file = os.path.join(__data__, "sample.html")
+    with open(sample_file, "w") as f:
+        f.write("""
+        <html><body>
+            <h1>Test Content</h1>
+            <p>This is a test paragraph.</p>
+        </body></html>
+        """)
+    
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Process local file
+        local_result = await crawler.arun(
+            url=f"file://{os.path.abspath(sample_file)}"
+        )
+        
+        # Process raw HTML
+        raw_html = """
+        <html><body>
+            <h1>Raw HTML Test</h1>
+            <p>This is a test of raw HTML processing.</p>
+        </body></html>
+        """
+        raw_result = await crawler.arun(
+            url=f"raw:{raw_html}"
+        )
+        
+        # Clean up
+        os.remove(sample_file)
+        
+        print("Local file content:", local_result.markdown)
+        print("\nRaw HTML content:", raw_result.markdown)
+
+# 3. Enhanced Markdown Generation Example
+async def markdown_generation_example():
+    """Example of enhanced markdown generation with citations and LLM-friendly features"""
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Create a content filter (optional)
+        content_filter = BM25ContentFilter(
+            # user_query="History and cultivation",
+            bm25_threshold=1.0
+        )
+        
+        result = await crawler.arun(
+            url="https://en.wikipedia.org/wiki/Apple",
+            css_selector="main div#bodyContent",
+            content_filter=content_filter,
+            cache_mode=CacheMode.BYPASS
+        )
+        
+        from crawl4ai import AsyncWebCrawler
+        from crawl4ai.content_filter_strategy import BM25ContentFilter
+        
+        result = await crawler.arun(
+            url="https://en.wikipedia.org/wiki/Apple",
+            css_selector="main div#bodyContent",
+            content_filter=BM25ContentFilter()
+        )
+        print(result.markdown_v2.fit_markdown)
+        
+        print("\nMarkdown Generation Results:")
+        print(f"1. Original markdown length: {len(result.markdown)}")
+        print(f"2. New markdown versions (markdown_v2):")
+        print(f"   - Raw markdown length: {len(result.markdown_v2.raw_markdown)}")
+        print(f"   - Citations markdown length: {len(result.markdown_v2.markdown_with_citations)}")
+        print(f"   - References section length: {len(result.markdown_v2.references_markdown)}")
+        if result.markdown_v2.fit_markdown:
+            print(f"   - Filtered markdown length: {len(result.markdown_v2.fit_markdown)}")
+        
+        # Save examples to files
+        output_dir = os.path.join(__data__, "markdown_examples")
+        os.makedirs(output_dir, exist_ok=True)
+        
+        # Save different versions
+        with open(os.path.join(output_dir, "1_raw_markdown.md"), "w") as f:
+            f.write(result.markdown_v2.raw_markdown)
+            
+        with open(os.path.join(output_dir, "2_citations_markdown.md"), "w") as f:
+            f.write(result.markdown_v2.markdown_with_citations)
+            
+        with open(os.path.join(output_dir, "3_references.md"), "w") as f:
+            f.write(result.markdown_v2.references_markdown)
+            
+        if result.markdown_v2.fit_markdown:
+            with open(os.path.join(output_dir, "4_filtered_markdown.md"), "w") as f:
+                f.write(result.markdown_v2.fit_markdown)
+                
+        print(f"\nMarkdown examples saved to: {output_dir}")
+        
+        # Show a sample of citations and references
+        print("\nSample of markdown with citations:")
+        print(result.markdown_v2.markdown_with_citations[:500] + "...\n")
+        print("Sample of references:")
+        print('\n'.join(result.markdown_v2.references_markdown.split('\n')[:10]) + "...")
+
+# 4. Browser Management Example
+async def browser_management_example():
+    """Example of using enhanced browser management features"""
+    # Use the specified user directory path
+    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
+    os.makedirs(user_data_dir, exist_ok=True)
+    
+    print(f"Browser profile will be saved to: {user_data_dir}")
+    
+    async with AsyncWebCrawler(
+        use_managed_browser=True,
+        user_data_dir=user_data_dir,
+        headless=False,
+        verbose=True
+    ) as crawler:
+
+        result = await crawler.arun(
+            url="https://crawl4ai.com",
+            # session_id="persistent_session_1",
+            cache_mode=CacheMode.BYPASS
+        )        
+        # Use GitHub as an example - it's a good test for browser management
+        # because it requires proper browser handling
+        result = await crawler.arun(
+            url="https://github.com/trending",
+            # session_id="persistent_session_1",
+            cache_mode=CacheMode.BYPASS
+        )
+        
+        print("\nBrowser session result:", result.success)
+        if result.success:
+            print("Page title:", result.metadata.get('title', 'No title found'))
+
+# 5. API Usage Example
+async def api_example():
+    """Example of using the new API endpoints"""
+    api_token = os.getenv('CRAWL4AI_API_TOKEN') or "test_api_code"
+    headers = {'Authorization': f'Bearer {api_token}'}    
+    async with aiohttp.ClientSession() as session:
+        # Submit crawl job
+        crawl_request = {
+            "urls": ["https://news.ycombinator.com"],  # Hacker News as an example
+            "extraction_config": {
+                "type": "json_css",
+                "params": {
+                    "schema": {
+                        "name": "Hacker News Articles",
+                        "baseSelector": ".athing",
+                        "fields": [
+                            {
+                                "name": "title",
+                                "selector": ".title a",
+                                "type": "text"
+                            },
+                            {
+                                "name": "score",
+                                "selector": ".score",
+                                "type": "text"
+                            },
+                            {
+                                "name": "url",
+                                "selector": ".title a",
+                                "type": "attribute",
+                                "attribute": "href"
+                            }
+                        ]
+                    }
+                }
+            },
+            "crawler_params": {
+                "headless": True,
+                # "use_managed_browser": True
+            },
+            "cache_mode": "bypass",
+            # "screenshot": True,
+            # "magic": True
+        }
+        
+        async with session.post(
+            "http://localhost:11235/crawl",
+            json=crawl_request,
+            headers=headers
+        ) as response:
+            task_data = await response.json()
+            task_id = task_data["task_id"]
+            
+            # Check task status
+            while True:
+                async with session.get(
+                    f"http://localhost:11235/task/{task_id}",
+                    headers=headers
+                ) as status_response:
+                    result = await status_response.json()
+                    print(f"Task status: {result['status']}")
+                    
+                    if result["status"] == "completed":
+                        print("Task completed!")
+                        print("Results:")
+                        news = json.loads(result["results"][0]['extracted_content'])
+                        print(json.dumps(news[:4], indent=2))
+                        break
+                    else:
+                        await asyncio.sleep(1)
+
+# Main execution
+async def main():
+    # print("Running Crawl4AI feature examples...")
+    
+    # print("\n1. Running Download Example:")
+    # await download_example()
+    
+    # print("\n2. Running Markdown Generation Example:")
+    # await markdown_generation_example()
+    
+    # # print("\n3. Running Local and Raw HTML Example:")
+    # await local_and_raw_html_example()
+    
+    # # print("\n4. Running Browser Management Example:")
+    await browser_management_example()
+    
+    # print("\n5. Running API Example:")
+    await api_example()
+
+if __name__ == "__main__":
+    asyncio.run(main())

docs/md_v2/advanced/hooks-auth.md

@@ -18,7 +18,7 @@ Let's see how we can customize the AsyncWebCrawler using hooks! In this example,
 import asyncio
 from crawl4ai import AsyncWebCrawler
 from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
-from playwright.async_api import Page, Browser
+from playwright.async_api import Page, Browser, BrowserContext
 
 async def on_browser_created(browser: Browser):
     print("[HOOK] on_browser_created")
@@ -71,7 +71,11 @@ from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
 async def main():
     print("\n🔗 Using Crawler Hooks: Let's see how we can customize the AsyncWebCrawler using hooks!")
     
-    crawler_strategy = AsyncPlaywrightCrawlerStrategy(verbose=True)
+    initial_cookies = [
+        {"name": "sessionId", "value": "abc123", "domain": ".example.com"},
+        {"name": "userId", "value": "12345", "domain": ".example.com"}
+    ]
+    crawler_strategy = AsyncPlaywrightCrawlerStrategy(verbose=True, cookies=initial_cookies)
     crawler_strategy.set_hook('on_browser_created', on_browser_created)
     crawler_strategy.set_hook('before_goto', before_goto)
     crawler_strategy.set_hook('after_goto', after_goto)

docs/md_v2/advanced/managed_browser.md

@@ -0,0 +1,84 @@
+# Content Filtering in Crawl4AI
+
+This guide explains how to use content filtering strategies in Crawl4AI to extract the most relevant information from crawled web pages.  You'll learn how to use the built-in `BM25ContentFilter` and how to create your own custom content filtering strategies.
+
+## Relevance Content Filter
+
+The `RelevanceContentFilter` is an abstract class that provides a common interface for content filtering strategies. Specific filtering algorithms, like `BM25ContentFilter`, inherit from this class and implement the `filter_content` method. This method takes the HTML content as input and returns a list of filtered text blocks.
+
+## BM25 Algorithm
+
+The `BM25ContentFilter` uses the BM25 algorithm, a ranking function used in information retrieval to estimate the relevance of documents to a given search query. In Crawl4AI, this algorithm helps to identify and extract text chunks that are most relevant to the page's metadata or a user-specified query.
+
+### Usage
+
+To use the `BM25ContentFilter`, initialize it and then pass it as the `extraction_strategy` parameter to the `arun` method of the crawler.
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.content_filter_strategy import BM25ContentFilter
+
+async def filter_content(url, query=None):
+    async with AsyncWebCrawler() as crawler:
+        content_filter = BM25ContentFilter(user_query=query)
+        result = await crawler.arun(url=url, extraction_strategy=content_filter, fit_markdown=True) # Set fit_markdown flag to True to trigger BM25 filtering
+        if result.success:
+            print(f"Filtered Content (JSON):\n{result.extracted_content}")
+            print(f"\nFiltered Markdown:\n{result.fit_markdown}") # New field in CrawlResult object
+            print(f"\nFiltered HTML:\n{result.fit_html}") # New field in CrawlResult object. Note that raw HTML may have tags re-organized due to internal parsing.
+        else:
+            print("Error:", result.error_message)
+
+# Example usage:
+asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple", "fruit nutrition health")) # with query
+asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple")) # without query, metadata will be used as the query.
+
+```
+
+### Parameters
+
+- **`user_query`**:  (Optional) A string representing the search query. If not provided, the filter extracts relevant metadata (title, description, keywords) from the page and uses that as the query.
+- **`bm25_threshold`**: (Optional, default 1.0)  A float value that controls the threshold for relevance.  Higher values result in stricter filtering, returning only the most relevant text chunks. Lower values result in more lenient filtering.
+
+
+## Fit Markdown Flag
+
+Setting the `fit_markdown` flag to `True` in the `arun` method activates the BM25 content filtering during the crawl. The `fit_markdown` parameter instructs the scraper to extract and clean the HTML, primarily to prepare for a Large Language Model that cannot process large amounts of data. Setting this flag not only improves the quality of the extracted content but also adds the filtered content to two new attributes in the returned  `CrawlResult` object: `fit_markdown` and `fit_html`.
+
+
+## Custom Content Filtering Strategies
+
+You can create your own custom filtering strategies by inheriting from the `RelevantContentFilter` class and implementing the `filter_content` method.  This allows you to tailor the filtering logic to your specific needs.
+
+```python
+from crawl4ai.content_filter_strategy import RelevantContentFilter
+from bs4 import BeautifulSoup, Tag
+from typing import List
+
+class MyCustomFilter(RelevantContentFilter):
+    def filter_content(self, html: str) -> List[str]:
+        soup = BeautifulSoup(html, 'lxml')
+        # Implement custom filtering logic here
+        # Example: extract all paragraphs within divs with class "article-body"
+        filtered_paragraphs = []
+        for tag in soup.select("div.article-body p"):
+            if isinstance(tag, Tag):
+                filtered_paragraphs.append(str(tag)) # Add the cleaned HTML element.  
+        return filtered_paragraphs
+
+
+
+async def custom_filter_demo(url: str):
+    async with AsyncWebCrawler() as crawler:
+        custom_filter = MyCustomFilter()
+        result = await crawler.arun(url, extraction_strategy=custom_filter)
+        if result.success:
+            print(result.extracted_content)
+
+```
+
+This example demonstrates extracting paragraphs from a specific div class.  You can customize this logic to implement different filtering strategies, use regular expressions, analyze text density, or apply other relevant techniques.
+
+## Conclusion
+
+Content filtering strategies provide a powerful way to refine the output of your crawls. By using `BM25ContentFilter` or creating custom strategies, you can focus on the most pertinent information and improve the efficiency of your data processing pipeline.

docs/md_v2/advanced/session-management-advanced.md

@@ -30,7 +30,7 @@ Let's start with a basic example of session-based crawling:
 
 ```python
 import asyncio
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
 
 async def basic_session_crawl():
     async with AsyncWebCrawler(verbose=True) as crawler:
@@ -43,7 +43,7 @@ async def basic_session_crawl():
                 session_id=session_id,
                 js_code="document.querySelector('.load-more-button').click();" if page > 0 else None,
                 css_selector=".content-item",
-                bypass_cache=True
+                cache_mode=CacheMode.BYPASS
             )
             
             print(f"Page {page + 1}: Found {result.extracted_content.count('.content-item')} items")
@@ -102,7 +102,7 @@ async def advanced_session_crawl_with_hooks():
                 session_id=session_id,
                 css_selector="li.commit-item",
                 js_code=js_next_page if page > 0 else None,
-                bypass_cache=True,
+                cache_mode=CacheMode.BYPASS,
                 js_only=page > 0
             )
 
@@ -174,7 +174,7 @@ async def integrated_js_and_wait_crawl():
                 extraction_strategy=extraction_strategy,
                 js_code=js_next_page_and_wait if page > 0 else None,
                 js_only=page > 0,
-                bypass_cache=True
+                cache_mode=CacheMode.BYPASS
             )
 
             commits = json.loads(result.extracted_content)
@@ -241,7 +241,7 @@ async def wait_for_parameter_crawl():
                 js_code=js_next_page if page > 0 else None,
                 wait_for=wait_for if page > 0 else None,
                 js_only=page > 0,
-                bypass_cache=True
+                cache_mode=CacheMode.BYPASS
             )
 
             commits = json.loads(result.extracted_content)

docs/md_v2/advanced/session-management.md

@@ -75,7 +75,7 @@ async def crawl_dynamic_content():
                 js_code=js_next_page if page > 0 else None,
                 wait_for=wait_for if page > 0 else None,
                 js_only=page > 0,
-                bypass_cache=True
+                cache_mode=CacheMode.BYPASS
             )
 
             if result.success:

docs/md_v2/api/arun.md

@@ -8,11 +8,26 @@ The following parameters can be passed to the `arun()` method. They are organize
 await crawler.arun(
     url="https://example.com",   # Required: URL to crawl
     verbose=True,               # Enable detailed logging
-    bypass_cache=False,         # Skip cache for this request
+    cache_mode=CacheMode.ENABLED,  # Control cache behavior
     warmup=True                # Whether to run warmup check
 )
 ```
 
+## Cache Control
+
+```python
+from crawl4ai import CacheMode
+
+await crawler.arun(
+    cache_mode=CacheMode.ENABLED,    # Normal caching (read/write)
+    # Other cache modes:
+    # cache_mode=CacheMode.DISABLED   # No caching at all
+    # cache_mode=CacheMode.READ_ONLY  # Only read from cache
+    # cache_mode=CacheMode.WRITE_ONLY # Only write to cache
+    # cache_mode=CacheMode.BYPASS     # Skip cache for this operation
+)
+```
+
 ## Content Processing Parameters
 
 ### Text Processing
@@ -162,14 +177,13 @@ await crawler.arun(
 
 ## Parameter Interactions and Notes
 
-1. **Magic Mode Combinations**
+1. **Cache and Performance Setup**
    ```python
-   # Full anti-detection setup
+   # Optimal caching for repeated crawls
    await crawler.arun(
-       magic=True,
-       headless=False,
-       simulate_user=True,
-       override_navigator=True
+       cache_mode=CacheMode.ENABLED,
+       word_count_threshold=10,
+       process_iframes=False
    )
    ```
 
@@ -179,7 +193,8 @@ await crawler.arun(
    await crawler.arun(
        js_code="window.scrollTo(0, document.body.scrollHeight);",
        wait_for="css:.lazy-content",
-       delay_before_return_html=2.0
+       delay_before_return_html=2.0,
+       cache_mode=CacheMode.WRITE_ONLY  # Cache results after dynamic load
    )
    ```
 
@@ -192,7 +207,8 @@ await crawler.arun(
        extraction_strategy=my_strategy,
        chunking_strategy=my_chunking,
        process_iframes=True,
-       remove_overlay_elements=True
+       remove_overlay_elements=True,
+       cache_mode=CacheMode.ENABLED
    )
    ```
 
@@ -201,7 +217,7 @@ await crawler.arun(
 1. **Performance Optimization**
    ```python
    await crawler.arun(
-       bypass_cache=False,           # Use cache when possible
+       cache_mode=CacheMode.ENABLED,  # Use full caching
        word_count_threshold=10,      # Filter out noise
        process_iframes=False         # Skip iframes if not needed
    )
@@ -212,7 +228,8 @@ await crawler.arun(
    await crawler.arun(
        magic=True,                   # Enable anti-detection
        delay_before_return_html=1.0, # Wait for dynamic content
-       page_timeout=60000           # Longer timeout for slow pages
+       page_timeout=60000,          # Longer timeout for slow pages
+       cache_mode=CacheMode.WRITE_ONLY  # Cache results after successful crawl
    )
    ```
 
@@ -221,6 +238,7 @@ await crawler.arun(
    await crawler.arun(
        remove_overlay_elements=True,  # Remove popups
        excluded_tags=['nav', 'aside'],# Remove unnecessary elements
-       keep_data_attributes=False     # Remove data attributes
+       keep_data_attributes=False,    # Remove data attributes
+       cache_mode=CacheMode.ENABLED   # Use cache for faster processing
    )
    ```

docs/md_v2/api/async-webcrawler.md

@@ -13,7 +13,7 @@ AsyncWebCrawler(
     
     # Cache Settings
     always_by_pass_cache: bool = False,     # Always bypass cache
-    base_directory: str = str(Path.home()), # Base directory for cache
+    base_directory: str = str(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())), # Base directory for cache
     
     # Network Settings
     proxy: str = None,                      # Simple proxy URL

docs/md_v2/api/crawl-result.md

@@ -20,6 +20,7 @@ class CrawlResult(BaseModel):
     fit_html: Optional[str] = None          # Most relevant HTML content
     markdown: Optional[str] = None          # HTML converted to markdown
     fit_markdown: Optional[str] = None      # Most relevant markdown content
+    downloaded_files: Optional[List[str]] = None  # Downloaded files
     
     # Extracted Data
     extracted_content: Optional[str] = None  # Content from extraction strategy

docs/md_v2/api/parameters.md

@@ -0,0 +1,36 @@
+# Parameter Reference Table
+
+| File Name | Parameter Name | Code Usage | Strategy/Class | Description |
+|-----------|---------------|------------|----------------|-------------|
+| async_crawler_strategy.py | user_agent | `kwargs.get("user_agent")` | AsyncPlaywrightCrawlerStrategy | User agent string for browser identification |
+| async_crawler_strategy.py | proxy | `kwargs.get("proxy")` | AsyncPlaywrightCrawlerStrategy | Proxy server configuration for network requests |
+| async_crawler_strategy.py | proxy_config | `kwargs.get("proxy_config")` | AsyncPlaywrightCrawlerStrategy | Detailed proxy configuration including auth |
+| async_crawler_strategy.py | headless | `kwargs.get("headless", True)` | AsyncPlaywrightCrawlerStrategy | Whether to run browser in headless mode |
+| async_crawler_strategy.py | browser_type | `kwargs.get("browser_type", "chromium")` | AsyncPlaywrightCrawlerStrategy | Type of browser to use (chromium/firefox/webkit) |
+| async_crawler_strategy.py | headers | `kwargs.get("headers", {})` | AsyncPlaywrightCrawlerStrategy | Custom HTTP headers for requests |
+| async_crawler_strategy.py | verbose | `kwargs.get("verbose", False)` | AsyncPlaywrightCrawlerStrategy | Enable detailed logging output |
+| async_crawler_strategy.py | sleep_on_close | `kwargs.get("sleep_on_close", False)` | AsyncPlaywrightCrawlerStrategy | Add delay before closing browser |
+| async_crawler_strategy.py | use_managed_browser | `kwargs.get("use_managed_browser", False)` | AsyncPlaywrightCrawlerStrategy | Use managed browser instance |
+| async_crawler_strategy.py | user_data_dir | `kwargs.get("user_data_dir", None)` | AsyncPlaywrightCrawlerStrategy | Custom directory for browser profile data |
+| async_crawler_strategy.py | session_id | `kwargs.get("session_id")` | AsyncPlaywrightCrawlerStrategy | Unique identifier for browser session |
+| async_crawler_strategy.py | override_navigator | `kwargs.get("override_navigator", False)` | AsyncPlaywrightCrawlerStrategy | Override browser navigator properties |
+| async_crawler_strategy.py | simulate_user | `kwargs.get("simulate_user", False)` | AsyncPlaywrightCrawlerStrategy | Simulate human-like behavior |
+| async_crawler_strategy.py | magic | `kwargs.get("magic", False)` | AsyncPlaywrightCrawlerStrategy | Enable advanced anti-detection features |
+| async_crawler_strategy.py | log_console | `kwargs.get("log_console", False)` | AsyncPlaywrightCrawlerStrategy | Log browser console messages |
+| async_crawler_strategy.py | js_only | `kwargs.get("js_only", False)` | AsyncPlaywrightCrawlerStrategy | Only execute JavaScript without page load |
+| async_crawler_strategy.py | page_timeout | `kwargs.get("page_timeout", 60000)` | AsyncPlaywrightCrawlerStrategy | Timeout for page load in milliseconds |
+| async_crawler_strategy.py | ignore_body_visibility | `kwargs.get("ignore_body_visibility", True)` | AsyncPlaywrightCrawlerStrategy | Process page even if body is hidden |
+| async_crawler_strategy.py | js_code | `kwargs.get("js_code", kwargs.get("js", self.js_code))` | AsyncPlaywrightCrawlerStrategy | Custom JavaScript code to execute |
+| async_crawler_strategy.py | wait_for | `kwargs.get("wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait for specific element/condition |
+| async_crawler_strategy.py | process_iframes | `kwargs.get("process_iframes", False)` | AsyncPlaywrightCrawlerStrategy | Extract content from iframes |
+| async_crawler_strategy.py | delay_before_return_html | `kwargs.get("delay_before_return_html")` | AsyncPlaywrightCrawlerStrategy | Additional delay before returning HTML |
+| async_crawler_strategy.py | remove_overlay_elements | `kwargs.get("remove_overlay_elements", False)` | AsyncPlaywrightCrawlerStrategy | Remove pop-ups and overlay elements |
+| async_crawler_strategy.py | screenshot | `kwargs.get("screenshot")` | AsyncPlaywrightCrawlerStrategy | Take page screenshot |
+| async_crawler_strategy.py | screenshot_wait_for | `kwargs.get("screenshot_wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait before taking screenshot |
+| async_crawler_strategy.py | semaphore_count | `kwargs.get("semaphore_count", 5)` | AsyncPlaywrightCrawlerStrategy | Concurrent request limit |
+| async_webcrawler.py | verbose | `kwargs.get("verbose", False)` | AsyncWebCrawler | Enable detailed logging |
+| async_webcrawler.py | warmup | `kwargs.get("warmup", True)` | AsyncWebCrawler | Initialize crawler with warmup request |
+| async_webcrawler.py | session_id | `kwargs.get("session_id", None)` | AsyncWebCrawler | Session identifier for browser reuse |
+| async_webcrawler.py | only_text | `kwargs.get("only_text", False)` | AsyncWebCrawler | Extract only text content |
+| async_webcrawler.py | bypass_cache | `kwargs.get("bypass_cache", False)` | AsyncWebCrawler | Skip cache and force fresh crawl |
+| async_webcrawler.py | cache_mode | `kwargs.get("cache_mode", CacheMode.ENABLE)` | AsyncWebCrawler | Cache handling mode for request |

docs/md_v2/assets/styles.css

@@ -150,4 +150,11 @@ strong,
 .tab-content pre {
     margin: 0;
     max-height: 300px; overflow: auto; border:none;
+}
+
+ol li::before {
+    content: counters(item, ".") ". ";
+    counter-increment: item;
+    /* float: left; */
+    /* padding-right: 5px; */
 }

docs/md_v2/basic/cache-modes.md

@@ -0,0 +1,79 @@
+# Crawl4AI Cache System and Migration Guide
+
+## Overview
+Starting from version X.X.X, Crawl4AI introduces a new caching system that replaces the old boolean flags with a more intuitive `CacheMode` enum. This change simplifies cache control and makes the behavior more predictable.
+
+## Old vs New Approach
+
+### Old Way (Deprecated)
+The old system used multiple boolean flags:
+- `bypass_cache`: Skip cache entirely
+- `disable_cache`: Disable all caching
+- `no_cache_read`: Don't read from cache
+- `no_cache_write`: Don't write to cache
+
+### New Way (Recommended)
+The new system uses a single `CacheMode` enum:
+- `CacheMode.ENABLED`: Normal caching (read/write)
+- `CacheMode.DISABLED`: No caching at all
+- `CacheMode.READ_ONLY`: Only read from cache
+- `CacheMode.WRITE_ONLY`: Only write to cache
+- `CacheMode.BYPASS`: Skip cache for this operation
+
+## Migration Example
+
+### Old Code (Deprecated)
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def use_proxy():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            bypass_cache=True  # Old way
+        )
+        print(len(result.markdown))
+
+async def main():
+    await use_proxy()
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+### New Code (Recommended)
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CacheMode  # Import CacheMode
+
+async def use_proxy():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            cache_mode=CacheMode.BYPASS  # New way
+        )
+        print(len(result.markdown))
+
+async def main():
+    await use_proxy()
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## Common Migration Patterns
+
+Old Flag | New Mode
+---------|----------
+`bypass_cache=True` | `cache_mode=CacheMode.BYPASS`
+`disable_cache=True` | `cache_mode=CacheMode.DISABLED`
+`no_cache_read=True` | `cache_mode=CacheMode.WRITE_ONLY`
+`no_cache_write=True` | `cache_mode=CacheMode.READ_ONLY`
+
+## Suppressing Deprecation Warnings
+If you need time to migrate, you can temporarily suppress deprecation warnings:
+```python
+# In your config.py
+SHOW_DEPRECATION_WARNINGS = False
+```

docs/md_v2/basic/content_filtering.md

@@ -0,0 +1,84 @@
+# Content Filtering in Crawl4AI
+
+This guide explains how to use content filtering strategies in Crawl4AI to extract the most relevant information from crawled web pages.  You'll learn how to use the built-in `BM25ContentFilter` and how to create your own custom content filtering strategies.
+
+## Relevance Content Filter
+
+The `RelevanceContentFilter` is an abstract class that provides a common interface for content filtering strategies. Specific filtering algorithms, like `BM25ContentFilter`, inherit from this class and implement the `filter_content` method. This method takes the HTML content as input and returns a list of filtered text blocks.
+
+## BM25 Algorithm
+
+The `BM25ContentFilter` uses the BM25 algorithm, a ranking function used in information retrieval to estimate the relevance of documents to a given search query. In Crawl4AI, this algorithm helps to identify and extract text chunks that are most relevant to the page's metadata or a user-specified query.
+
+### Usage
+
+To use the `BM25ContentFilter`, initialize it and then pass it as the `extraction_strategy` parameter to the `arun` method of the crawler.
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.content_filter_strategy import BM25ContentFilter
+
+async def filter_content(url, query=None):
+    async with AsyncWebCrawler() as crawler:
+        content_filter = BM25ContentFilter(user_query=query)
+        result = await crawler.arun(url=url, content_filter=content_filter, fit_markdown=True) # Set fit_markdown flag to True to trigger BM25 filtering
+        if result.success:
+            print(f"Filtered Content (JSON):\n{result.extracted_content}")
+            print(f"\nFiltered Markdown:\n{result.fit_markdown}") # New field in CrawlResult object
+            print(f"\nFiltered HTML:\n{result.fit_html}") # New field in CrawlResult object. Note that raw HTML may have tags re-organized due to internal parsing.
+        else:
+            print("Error:", result.error_message)
+
+# Example usage:
+asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple", "fruit nutrition health")) # with query
+asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple")) # without query, metadata will be used as the query.
+
+```
+
+### Parameters
+
+- **`user_query`**:  (Optional) A string representing the search query. If not provided, the filter extracts relevant metadata (title, description, keywords) from the page and uses that as the query.
+- **`bm25_threshold`**: (Optional, default 1.0)  A float value that controls the threshold for relevance.  Higher values result in stricter filtering, returning only the most relevant text chunks. Lower values result in more lenient filtering.
+
+
+## Fit Markdown Flag
+
+Setting the `fit_markdown` flag to `True` in the `arun` method activates the BM25 content filtering during the crawl. The `fit_markdown` parameter instructs the scraper to extract and clean the HTML, primarily to prepare for a Large Language Model that cannot process large amounts of data. Setting this flag not only improves the quality of the extracted content but also adds the filtered content to two new attributes in the returned  `CrawlResult` object: `fit_markdown` and `fit_html`.
+
+
+## Custom Content Filtering Strategies
+
+You can create your own custom filtering strategies by inheriting from the `RelevantContentFilter` class and implementing the `filter_content` method.  This allows you to tailor the filtering logic to your specific needs.
+
+```python
+from crawl4ai.content_filter_strategy import RelevantContentFilter
+from bs4 import BeautifulSoup, Tag
+from typing import List
+
+class MyCustomFilter(RelevantContentFilter):
+    def filter_content(self, html: str) -> List[str]:
+        soup = BeautifulSoup(html, 'lxml')
+        # Implement custom filtering logic here
+        # Example: extract all paragraphs within divs with class "article-body"
+        filtered_paragraphs = []
+        for tag in soup.select("div.article-body p"):
+            if isinstance(tag, Tag):
+                filtered_paragraphs.append(str(tag)) # Add the cleaned HTML element.  
+        return filtered_paragraphs
+
+
+
+async def custom_filter_demo(url: str):
+    async with AsyncWebCrawler() as crawler:
+        custom_filter = MyCustomFilter()
+        result = await crawler.arun(url, content_filter=custom_filter)
+        if result.success:
+            print(result.extracted_content)
+
+```
+
+This example demonstrates extracting paragraphs from a specific div class.  You can customize this logic to implement different filtering strategies, use regular expressions, analyze text density, or apply other relevant techniques.
+
+## Conclusion
+
+Content filtering strategies provide a powerful way to refine the output of your crawls. By using `BM25ContentFilter` or creating custom strategies, you can focus on the most pertinent information and improve the efficiency of your data processing pipeline.

docs/md_v2/basic/docker-deploymeny.md

@@ -0,0 +1,718 @@
+# Docker Deployment
+
+Crawl4AI provides official Docker images for easy deployment and scalability. This guide covers installation, configuration, and usage of Crawl4AI in Docker environments.
+
+## Quick Start 🚀
+
+Pull and run the basic version:
+
+```bash
+# Basic run without security
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+
+# Run with API security enabled
+docker run -p 11235:11235 -e CRAWL4AI_API_TOKEN=your_secret_token unclecode/crawl4ai:basic
+```
+
+## Running with Docker Compose 🐳
+
+### Use Docker Compose (From Local Dockerfile or Docker Hub)
+
+Crawl4AI provides flexibility to use Docker Compose for managing your containerized services. You can either build the image locally from the provided `Dockerfile` or use the pre-built image from Docker Hub.
+
+### **Option 1: Using Docker Compose to Build Locally**
+If you want to build the image locally, use the provided `docker-compose.local.yml` file.
+
+```bash
+docker-compose -f docker-compose.local.yml up -d
+```
+
+This will:
+1. Build the Docker image from the provided `Dockerfile`.
+2. Start the container and expose it on `http://localhost:11235`.
+
+---
+
+### **Option 2: Using Docker Compose with Pre-Built Image from Hub**
+If you prefer using the pre-built image on Docker Hub, use the `docker-compose.hub.yml` file.
+
+```bash
+docker-compose -f docker-compose.hub.yml up -d
+```
+
+This will:
+1. Pull the pre-built image `unclecode/crawl4ai:basic` (or `all`, depending on your configuration).
+2. Start the container and expose it on `http://localhost:11235`.
+
+---
+
+### **Stopping the Running Services**
+
+To stop the services started via Docker Compose, you can use:
+
+```bash
+docker-compose -f docker-compose.local.yml down
+# OR
+docker-compose -f docker-compose.hub.yml down
+```
+
+If the containers don’t stop and the application is still running, check the running containers:
+
+```bash
+docker ps
+```
+
+Find the `CONTAINER ID` of the running service and stop it forcefully:
+
+```bash
+docker stop <CONTAINER_ID>
+```
+
+---
+
+### **Debugging with Docker Compose**
+
+- **Check Logs**: To view the container logs:
+  ```bash
+  docker-compose -f docker-compose.local.yml logs -f
+  ```
+
+- **Remove Orphaned Containers**: If the service is still running unexpectedly:
+  ```bash
+  docker-compose -f docker-compose.local.yml down --remove-orphans
+  ```
+
+- **Manually Remove Network**: If the network is still in use:
+  ```bash
+  docker network ls
+  docker network rm crawl4ai_default
+  ```
+
+---
+
+### Why Use Docker Compose?
+
+Docker Compose is the recommended way to deploy Crawl4AI because:
+1. It simplifies multi-container setups.
+2. Allows you to define environment variables, resources, and ports in a single file.
+3. Makes it easier to switch between local development and production-ready images.
+
+For example, your `docker-compose.yml` could include API keys, token settings, and memory limits, making deployment quick and consistent.
+
+
+
+
+## API Security 🔒
+
+### Understanding CRAWL4AI_API_TOKEN
+
+The `CRAWL4AI_API_TOKEN` provides optional security for your Crawl4AI instance:
+
+- If `CRAWL4AI_API_TOKEN` is set: All API endpoints (except `/health`) require authentication
+- If `CRAWL4AI_API_TOKEN` is not set: The API is publicly accessible
+
+```bash
+# Secured Instance
+docker run -p 11235:11235 -e CRAWL4AI_API_TOKEN=your_secret_token unclecode/crawl4ai:all
+
+# Unsecured Instance
+docker run -p 11235:11235 unclecode/crawl4ai:all
+```
+
+### Making API Calls
+
+For secured instances, include the token in all requests:
+
+```python
+import requests
+
+# Setup headers if token is being used
+api_token = "your_secret_token"  # Same token set in CRAWL4AI_API_TOKEN
+headers = {"Authorization": f"Bearer {api_token}"} if api_token else {}
+
+# Making authenticated requests
+response = requests.post(
+    "http://localhost:11235/crawl",
+    headers=headers,
+    json={
+        "urls": "https://example.com",
+        "priority": 10
+    }
+)
+
+# Checking task status
+task_id = response.json()["task_id"]
+status = requests.get(
+    f"http://localhost:11235/task/{task_id}",
+    headers=headers
+)
+```
+
+### Using with Docker Compose
+
+In your `docker-compose.yml`:
+```yaml
+services:
+  crawl4ai:
+    image: unclecode/crawl4ai:all
+    environment:
+      - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}  # Optional
+    # ... other configuration
+```
+
+Then either:
+1. Set in `.env` file:
+```env
+CRAWL4AI_API_TOKEN=your_secret_token
+```
+
+2. Or set via command line:
+```bash
+CRAWL4AI_API_TOKEN=your_secret_token docker-compose up
+```
+
+> **Security Note**: If you enable the API token, make sure to keep it secure and never commit it to version control. The token will be required for all API endpoints except the health check endpoint (`/health`).
+
+## Configuration Options 🔧
+
+### Environment Variables
+
+You can configure the service using environment variables:
+
+```bash
+# Basic configuration
+docker run -p 11235:11235 \
+    -e MAX_CONCURRENT_TASKS=5 \
+    unclecode/crawl4ai:all
+
+# With security and LLM support
+docker run -p 11235:11235 \
+    -e CRAWL4AI_API_TOKEN=your_secret_token \
+    -e OPENAI_API_KEY=sk-... \
+    -e ANTHROPIC_API_KEY=sk-ant-... \
+    unclecode/crawl4ai:all
+```
+
+### Using Docker Compose (Recommended) 🐳
+
+Create a `docker-compose.yml`:
+
+```yaml
+version: '3.8'
+
+services:
+  crawl4ai:
+    image: unclecode/crawl4ai:all
+    ports:
+      - "11235:11235"
+    environment:
+      - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}  # Optional API security
+      - MAX_CONCURRENT_TASKS=5
+      # LLM Provider Keys
+      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
+    volumes:
+      - /dev/shm:/dev/shm
+    deploy:
+      resources:
+        limits:
+          memory: 4G
+        reservations:
+          memory: 1G
+```
+
+You can run it in two ways:
+
+1. Using environment variables directly:
+```bash
+CRAWL4AI_API_TOKEN=secret123 OPENAI_API_KEY=sk-... docker-compose up
+```
+
+2. Using a `.env` file (recommended):
+Create a `.env` file in the same directory:
+```env
+# API Security (optional)
+CRAWL4AI_API_TOKEN=your_secret_token
+
+# LLM Provider Keys
+OPENAI_API_KEY=sk-...
+ANTHROPIC_API_KEY=sk-ant-...
+
+# Other Configuration
+MAX_CONCURRENT_TASKS=5
+```
+
+Then simply run:
+```bash
+docker-compose up
+```
+
+### Testing the Deployment 🧪
+
+```python
+import requests
+
+# For unsecured instances
+def test_unsecured():
+    # Health check
+    health = requests.get("http://localhost:11235/health")
+    print("Health check:", health.json())
+
+    # Basic crawl
+    response = requests.post(
+        "http://localhost:11235/crawl",
+        json={
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 10
+        }
+    )
+    task_id = response.json()["task_id"]
+    print("Task ID:", task_id)
+
+# For secured instances
+def test_secured(api_token):
+    headers = {"Authorization": f"Bearer {api_token}"}
+    
+    # Basic crawl with authentication
+    response = requests.post(
+        "http://localhost:11235/crawl",
+        headers=headers,
+        json={
+            "urls": "https://www.nbcnews.com/business",
+            "priority": 10
+        }
+    )
+    task_id = response.json()["task_id"]
+    print("Task ID:", task_id)
+```
+
+### LLM Extraction Example 🤖
+
+When you've configured your LLM provider keys (via environment variables or `.env`), you can use LLM extraction:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "extraction_config": {
+        "type": "llm",
+        "params": {
+            "provider": "openai/gpt-4",
+            "instruction": "Extract main topics from the page"
+        }
+    }
+}
+
+# Make the request (add headers if using API security)
+response = requests.post("http://localhost:11235/crawl", json=request)
+```
+
+> **Note**: Remember to add `.env` to your `.gitignore` to keep your API keys secure!
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+## Usage Examples 📝
+
+### Basic Crawling
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "priority": 10
+}
+
+response = requests.post("http://localhost:11235/crawl", json=request)
+task_id = response.json()["task_id"]
+
+# Get results
+result = requests.get(f"http://localhost:11235/task/{task_id}")
+```
+
+### Structured Data Extraction
+
+```python
+schema = {
+    "name": "Crypto Prices",
+    "baseSelector": ".cds-tableRow-t45thuk",
+    "fields": [
+        {
+            "name": "crypto",
+            "selector": "td:nth-child(1) h2",
+            "type": "text",
+        },
+        {
+            "name": "price",
+            "selector": "td:nth-child(2)",
+            "type": "text",
+        }
+    ],
+}
+
+request = {
+    "urls": "https://www.coinbase.com/explore",
+    "extraction_config": {
+        "type": "json_css",
+        "params": {"schema": schema}
+    }
+}
+```
+
+### Dynamic Content Handling
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "js_code": [
+        "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+    ],
+    "wait_for": "article.tease-card:nth-child(10)"
+}
+```
+
+### AI-Powered Extraction (Full Version)
+
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "extraction_config": {
+        "type": "cosine",
+        "params": {
+            "semantic_filter": "business finance economy",
+            "word_count_threshold": 10,
+            "max_dist": 0.2,
+            "top_k": 3
+        }
+    }
+}
+```
+
+## Platform-Specific Instructions 💻
+
+### macOS
+```bash
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+### Ubuntu
+```bash
+# Basic version
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+
+# With GPU support
+docker pull unclecode/crawl4ai:gpu
+docker run --gpus all -p 11235:11235 unclecode/crawl4ai:gpu
+```
+
+### Windows (PowerShell)
+```powershell
+docker pull unclecode/crawl4ai:basic
+docker run -p 11235:11235 unclecode/crawl4ai:basic
+```
+
+## Testing 🧪
+
+Save this as `test_docker.py`:
+
+```python
+import requests
+import json
+import time
+import sys
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:11235"):
+        self.base_url = base_url
+        
+    def submit_and_wait(self, request_data: dict, timeout: int = 300) -> dict:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data)
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} timeout")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}")
+            status = result.json()
+            
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+
+def test_deployment():
+    tester = Crawl4AiTester()
+    
+    # Test basic crawl
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Basic crawl successful!")
+    print(f"Content length: {len(result['result']['markdown'])}")
+
+if __name__ == "__main__":
+    test_deployment()
+```
+
+## Advanced Configuration ⚙️
+
+### Crawler Parameters
+
+The `crawler_params` field allows you to configure the browser instance and crawling behavior. Here are key parameters you can use:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        # Browser Configuration
+        "headless": True,                    # Run in headless mode
+        "browser_type": "chromium",          # chromium/firefox/webkit
+        "user_agent": "custom-agent",        # Custom user agent
+        "proxy": "http://proxy:8080",        # Proxy configuration
+        
+        # Performance & Behavior
+        "page_timeout": 30000,               # Page load timeout (ms)
+        "verbose": True,                     # Enable detailed logging
+        "semaphore_count": 5,               # Concurrent request limit
+        
+        # Anti-Detection Features
+        "simulate_user": True,               # Simulate human behavior
+        "magic": True,                       # Advanced anti-detection
+        "override_navigator": True,          # Override navigator properties
+        
+        # Session Management
+        "user_data_dir": "./browser-data",   # Browser profile location
+        "use_managed_browser": True,         # Use persistent browser
+    }
+}
+```
+
+### Extra Parameters
+
+The `extra` field allows passing additional parameters directly to the crawler's `arun` function:
+
+```python
+request = {
+    "urls": "https://example.com",
+    "extra": {
+        "word_count_threshold": 10,          # Min words per block
+        "only_text": True,                   # Extract only text
+        "bypass_cache": True,                # Force fresh crawl
+        "process_iframes": True,             # Include iframe content
+    }
+}
+```
+
+### Complete Examples
+
+1. **Advanced News Crawling**
+```python
+request = {
+    "urls": "https://www.nbcnews.com/business",
+    "crawler_params": {
+        "headless": True,
+        "page_timeout": 30000,
+        "remove_overlay_elements": True      # Remove popups
+    },
+    "extra": {
+        "word_count_threshold": 50,          # Longer content blocks
+        "bypass_cache": True                 # Fresh content
+    },
+    "css_selector": ".article-body"
+}
+```
+
+2. **Anti-Detection Configuration**
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        "simulate_user": True,
+        "magic": True,
+        "override_navigator": True,
+        "user_agent": "Mozilla/5.0 ...",
+        "headers": {
+            "Accept-Language": "en-US,en;q=0.9"
+        }
+    }
+}
+```
+
+3. **LLM Extraction with Custom Parameters**
+```python
+request = {
+    "urls": "https://openai.com/pricing",
+    "extraction_config": {
+        "type": "llm",
+        "params": {
+            "provider": "openai/gpt-4",
+            "schema": pricing_schema
+        }
+    },
+    "crawler_params": {
+        "verbose": True,
+        "page_timeout": 60000
+    },
+    "extra": {
+        "word_count_threshold": 1,
+        "only_text": True
+    }
+}
+```
+
+4. **Session-Based Dynamic Content**
+```python
+request = {
+    "urls": "https://example.com",
+    "crawler_params": {
+        "session_id": "dynamic_session",
+        "headless": False,
+        "page_timeout": 60000
+    },
+    "js_code": ["window.scrollTo(0, document.body.scrollHeight);"],
+    "wait_for": "js:() => document.querySelectorAll('.item').length > 10",
+    "extra": {
+        "delay_before_return_html": 2.0
+    }
+}
+```
+
+5. **Screenshot with Custom Timing**
+```python
+request = {
+    "urls": "https://example.com",
+    "screenshot": True,
+    "crawler_params": {
+        "headless": True,
+        "screenshot_wait_for": ".main-content"
+    },
+    "extra": {
+        "delay_before_return_html": 3.0
+    }
+}
+```
+
+### Parameter Reference Table
+
+| Category | Parameter | Type | Description |
+|----------|-----------|------|-------------|
+| Browser | headless | bool | Run browser in headless mode |
+| Browser | browser_type | str | Browser engine selection |
+| Browser | user_agent | str | Custom user agent string |
+| Network | proxy | str | Proxy server URL |
+| Network | headers | dict | Custom HTTP headers |
+| Timing | page_timeout | int | Page load timeout (ms) |
+| Timing | delay_before_return_html | float | Wait before capture |
+| Anti-Detection | simulate_user | bool | Human behavior simulation |
+| Anti-Detection | magic | bool | Advanced protection |
+| Session | session_id | str | Browser session ID |
+| Session | user_data_dir | str | Profile directory |
+| Content | word_count_threshold | int | Minimum words per block |
+| Content | only_text | bool | Text-only extraction |
+| Content | process_iframes | bool | Include iframe content |
+| Debug | verbose | bool | Detailed logging |
+| Debug | log_console | bool | Browser console logs |
+
+## Troubleshooting 🔍
+
+### Common Issues
+
+1. **Connection Refused**
+   ```
+   Error: Connection refused at localhost:11235
+   ```
+   Solution: Ensure the container is running and ports are properly mapped.
+
+2. **Resource Limits**
+   ```
+   Error: No available slots
+   ```
+   Solution: Increase MAX_CONCURRENT_TASKS or container resources.
+
+3. **GPU Access**
+   ```
+   Error: GPU not found
+   ```
+   Solution: Ensure proper NVIDIA drivers and use `--gpus all` flag.
+
+### Debug Mode
+
+Access container for debugging:
+```bash
+docker run -it --entrypoint /bin/bash unclecode/crawl4ai:all
+```
+
+View container logs:
+```bash
+docker logs [container_id]
+```
+
+## Best Practices 🌟
+
+1. **Resource Management**
+   - Set appropriate memory and CPU limits
+   - Monitor resource usage via health endpoint
+   - Use basic version for simple crawling tasks
+
+2. **Scaling**
+   - Use multiple containers for high load
+   - Implement proper load balancing
+   - Monitor performance metrics
+
+3. **Security**
+   - Use environment variables for sensitive data
+   - Implement proper network isolation
+   - Regular security updates
+
+## API Reference 📚
+
+### Health Check
+```http
+GET /health
+```
+
+### Submit Crawl Task
+```http
+POST /crawl
+Content-Type: application/json
+
+{
+    "urls": "string or array",
+    "extraction_config": {
+        "type": "basic|llm|cosine|json_css",
+        "params": {}
+    },
+    "priority": 1-10,
+    "ttl": 3600
+}
+```
+
+### Get Task Status
+```http
+GET /task/{task_id}
+```
+
+For more details, visit the [official documentation](https://crawl4ai.com/mkdocs/).

docs/md_v2/basic/file-download.md

@@ -0,0 +1,148 @@
+# Download Handling in Crawl4AI
+
+This guide explains how to use Crawl4AI to handle file downloads during crawling.  You'll learn how to trigger downloads, specify download locations, and access downloaded files.
+
+## Enabling Downloads
+
+By default, Crawl4AI does not download files. To enable downloads, set the `accept_downloads` parameter to `True` in either the `AsyncWebCrawler` constructor or the `arun` method.
+
+```python
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    async with AsyncWebCrawler(accept_downloads=True) as crawler:  # Globally enable downloads
+        # ... your crawling logic ...
+
+asyncio.run(main())
+```
+
+Or, enable it for a specific crawl:
+
+```python
+async def main():
+    async with AsyncWebCrawler() as crawler:
+        result = await crawler.arun(url="...", accept_downloads=True)
+        # ...
+```
+
+## Specifying Download Location
+
+You can specify the download directory using the `downloads_path` parameter. If not provided, Crawl4AI creates a "downloads" directory inside the `.crawl4ai` folder in your home directory.
+
+```python
+import os
+from pathlib import Path
+
+# ... inside your crawl function:
+
+downloads_path = os.path.join(os.getcwd(), "my_downloads")  # Custom download path
+os.makedirs(downloads_path, exist_ok=True)
+
+result = await crawler.arun(url="...", downloads_path=downloads_path, accept_downloads=True)
+
+# ...
+```
+
+If you are setting it globally, provide the path to the AsyncWebCrawler:
+```python
+async def crawl_with_downloads(url: str, download_path: str):
+    async with AsyncWebCrawler(
+        accept_downloads=True,
+        downloads_path=download_path, # or set it on arun
+        verbose=True
+    ) as crawler:
+        result = await crawler.arun(url=url) # you still need to enable downloads per call.
+        # ...
+```
+
+
+
+## Triggering Downloads
+
+Downloads are typically triggered by user interactions on a web page (e.g., clicking a download button).  You can simulate these actions with the `js_code` parameter, injecting JavaScript code to be executed within the browser context.  The `wait_for` parameter might also be crucial to allowing sufficient time for downloads to initiate before the crawler proceeds.
+
+```python
+result = await crawler.arun(
+    url="https://www.python.org/downloads/",
+    js_code="""
+        // Find and click the first Windows installer link
+        const downloadLink = document.querySelector('a[href$=".exe"]');
+        if (downloadLink) {
+            downloadLink.click();
+        }
+    """,
+    wait_for=5  # Wait for 5 seconds for the download to start
+)
+```
+
+## Accessing Downloaded Files
+
+Downloaded file paths are stored in the `downloaded_files` attribute of the returned  `CrawlResult`  object.  This is a list of strings, with each string representing the absolute path to a downloaded file.
+
+```python
+if result.downloaded_files:
+    print("Downloaded files:")
+    for file_path in result.downloaded_files:
+        print(f"- {file_path}")
+        # Perform operations with downloaded files, e.g., check file size
+        file_size = os.path.getsize(file_path)
+        print(f"- File size: {file_size} bytes")
+else:
+    print("No files downloaded.")
+```
+
+
+##  Example: Downloading Multiple Files
+
+```python
+import asyncio
+import os
+from pathlib import Path
+from crawl4ai import AsyncWebCrawler
+
+async def download_multiple_files(url: str, download_path: str):
+
+    async with AsyncWebCrawler(
+        accept_downloads=True,
+        downloads_path=download_path,
+        verbose=True
+    ) as crawler:
+        result = await crawler.arun(
+            url=url,
+            js_code="""
+            // Trigger multiple downloads (example)
+            const downloadLinks = document.querySelectorAll('a[download]'); // Or a more specific selector
+            for (const link of downloadLinks) {
+                link.click();
+                await new Promise(r => setTimeout(r, 2000)); // Add a small delay between clicks if needed
+            }
+            """,
+            wait_for=10 # Adjust the timeout to match the expected time for all downloads to start
+        )
+
+        if result.downloaded_files:
+            print("Downloaded files:")
+            for file in result.downloaded_files:
+                print(f"- {file}")
+        else:
+            print("No files downloaded.")
+            
+
+# Example usage
+download_path = os.path.join(Path.home(), ".crawl4ai", "downloads")
+os.makedirs(download_path, exist_ok=True) # Create directory if it doesn't exist
+
+
+asyncio.run(download_multiple_files("https://www.python.org/downloads/windows/", download_path))
+```
+
+## Important Considerations
+
+- **Browser Context:** Downloads are managed within the browser context.  Ensure your `js_code` correctly targets the download triggers on the specific web page.
+- **Waiting:**  Use `wait_for` to manage the timing of the crawl process if immediate download might not occur.
+- **Error Handling:** Implement proper error handling to gracefully manage failed downloads or incorrect file paths.
+- **Security:** Downloaded files should be scanned for potential security threats before use.
+
+
+
+This guide provides a foundation for handling downloads with Crawl4AI. You can adapt these techniques to manage downloads in various scenarios and integrate them into more complex crawling workflows.

docs/md_v2/basic/installation.md

@@ -58,6 +58,51 @@ crawl4ai-download-models
 
 This is optional but will boost the performance and speed of the crawler. You only need to do this once after installation.
 
+## Playwright Installation Note for Ubuntu
+
+If you encounter issues with Playwright installation on Ubuntu, you may need to install additional dependencies:
+
+```bash
+sudo apt-get install -y \
+    libwoff1 \
+    libopus0 \
+    libwebp7 \
+    libwebpdemux2 \
+    libenchant-2-2 \
+    libgudev-1.0-0 \
+    libsecret-1-0 \
+    libhyphen0 \
+    libgdk-pixbuf2.0-0 \
+    libegl1 \
+    libnotify4 \
+    libxslt1.1 \
+    libevent-2.1-7 \
+    libgles2 \
+    libxcomposite1 \
+    libatk1.0-0 \
+    libatk-bridge2.0-0 \
+    libepoxy0 \
+    libgtk-3-0 \
+    libharfbuzz-icu0 \
+    libgstreamer-gl1.0-0 \
+    libgstreamer-plugins-bad1.0-0 \
+    gstreamer1.0-plugins-good \
+    gstreamer1.0-plugins-bad \
+    libxt6 \
+    libxaw7 \
+    xvfb \
+    fonts-noto-color-emoji \
+    libfontconfig \
+    libfreetype6 \
+    xfonts-cyrillic \
+    xfonts-scalable \
+    fonts-liberation \
+    fonts-ipafont-gothic \
+    fonts-wqy-zenhei \
+    fonts-tlwg-loma-otf \
+    fonts-freefont-ttf
+```
+
 ## Option 2: Using Docker (Coming Soon)
 
 Docker support for Crawl4AI is currently in progress and will be available soon. This will allow you to run Crawl4AI in a containerized environment, ensuring consistency across different systems.

docs/md_v2/basic/prefix-based-input.md

@@ -0,0 +1,235 @@
+# Prefix-Based Input Handling in Crawl4AI
+
+This guide will walk you through using the Crawl4AI library to crawl web pages, local HTML files, and raw HTML strings. We'll demonstrate these capabilities using a Wikipedia page as an example.
+
+## Table of Contents
+- [Prefix-Based Input Handling in Crawl4AI](#prefix-based-input-handling-in-crawl4ai)
+  - [Table of Contents](#table-of-contents)
+    - [Crawling a Web URL](#crawling-a-web-url)
+    - [Crawling a Local HTML File](#crawling-a-local-html-file)
+    - [Crawling Raw HTML Content](#crawling-raw-html-content)
+  - [Complete Example](#complete-example)
+    - [**How It Works**](#how-it-works)
+    - [**Running the Example**](#running-the-example)
+  - [Conclusion](#conclusion)
+
+---
+
+
+### Crawling a Web URL
+
+To crawl a live web page, provide the URL starting with `http://` or `https://`.
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def crawl_web():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://en.wikipedia.org/wiki/apple", bypass_cache=True)
+        if result.success:
+            print("Markdown Content:")
+            print(result.markdown)
+        else:
+            print(f"Failed to crawl: {result.error_message}")
+
+asyncio.run(crawl_web())
+```
+
+### Crawling a Local HTML File
+
+To crawl a local HTML file, prefix the file path with `file://`.
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def crawl_local_file():
+    local_file_path = "/path/to/apple.html"  # Replace with your file path
+    file_url = f"file://{local_file_path}"
+    
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url=file_url, bypass_cache=True)
+        if result.success:
+            print("Markdown Content from Local File:")
+            print(result.markdown)
+        else:
+            print(f"Failed to crawl local file: {result.error_message}")
+
+asyncio.run(crawl_local_file())
+```
+
+### Crawling Raw HTML Content
+
+To crawl raw HTML content, prefix the HTML string with `raw:`.
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def crawl_raw_html():
+    raw_html = "<html><body><h1>Hello, World!</h1></body></html>"
+    raw_html_url = f"raw:{raw_html}"
+    
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url=raw_html_url, bypass_cache=True)
+        if result.success:
+            print("Markdown Content from Raw HTML:")
+            print(result.markdown)
+        else:
+            print(f"Failed to crawl raw HTML: {result.error_message}")
+
+asyncio.run(crawl_raw_html())
+```
+
+---
+
+## Complete Example
+
+Below is a comprehensive script that:
+1. **Crawls the Wikipedia page for "Apple".**
+2. **Saves the HTML content to a local file (`apple.html`).**
+3. **Crawls the local HTML file and verifies the markdown length matches the original crawl.**
+4. **Crawls the raw HTML content from the saved file and verifies consistency.**
+
+```python
+import os
+import sys
+import asyncio
+from pathlib import Path
+
+# Adjust the parent directory to include the crawl4ai module
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    # Define the URL to crawl
+    wikipedia_url = "https://en.wikipedia.org/wiki/apple"
+    
+    # Define the path to save the HTML file
+    # Save the file in the same directory as the script
+    script_dir = Path(__file__).parent
+    html_file_path = script_dir / "apple.html"
+    
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        print("\n=== Step 1: Crawling the Wikipedia URL ===")
+        # Crawl the Wikipedia URL
+        result = await crawler.arun(url=wikipedia_url, bypass_cache=True)
+        
+        # Check if crawling was successful
+        if not result.success:
+            print(f"Failed to crawl {wikipedia_url}: {result.error_message}")
+            return
+        
+        # Save the HTML content to a local file
+        with open(html_file_path, 'w', encoding='utf-8') as f:
+            f.write(result.html)
+        print(f"Saved HTML content to {html_file_path}")
+        
+        # Store the length of the generated markdown
+        web_crawl_length = len(result.markdown)
+        print(f"Length of markdown from web crawl: {web_crawl_length}\n")
+        
+        print("=== Step 2: Crawling from the Local HTML File ===")
+        # Construct the file URL with 'file://' prefix
+        file_url = f"file://{html_file_path.resolve()}"
+        
+        # Crawl the local HTML file
+        local_result = await crawler.arun(url=file_url, bypass_cache=True)
+        
+        # Check if crawling was successful
+        if not local_result.success:
+            print(f"Failed to crawl local file {file_url}: {local_result.error_message}")
+            return
+        
+        # Store the length of the generated markdown from local file
+        local_crawl_length = len(local_result.markdown)
+        print(f"Length of markdown from local file crawl: {local_crawl_length}")
+        
+        # Compare the lengths
+        assert web_crawl_length == local_crawl_length, (
+            f"Markdown length mismatch: Web crawl ({web_crawl_length}) != Local file crawl ({local_crawl_length})"
+        )
+        print("✅ Markdown length matches between web crawl and local file crawl.\n")
+        
+        print("=== Step 3: Crawling Using Raw HTML Content ===")
+        # Read the HTML content from the saved file
+        with open(html_file_path, 'r', encoding='utf-8') as f:
+            raw_html_content = f.read()
+        
+        # Prefix the raw HTML content with 'raw:'
+        raw_html_url = f"raw:{raw_html_content}"
+        
+        # Crawl using the raw HTML content
+        raw_result = await crawler.arun(url=raw_html_url, bypass_cache=True)
+        
+        # Check if crawling was successful
+        if not raw_result.success:
+            print(f"Failed to crawl raw HTML content: {raw_result.error_message}")
+            return
+        
+        # Store the length of the generated markdown from raw HTML
+        raw_crawl_length = len(raw_result.markdown)
+        print(f"Length of markdown from raw HTML crawl: {raw_crawl_length}")
+        
+        # Compare the lengths
+        assert web_crawl_length == raw_crawl_length, (
+            f"Markdown length mismatch: Web crawl ({web_crawl_length}) != Raw HTML crawl ({raw_crawl_length})"
+        )
+        print("✅ Markdown length matches between web crawl and raw HTML crawl.\n")
+        
+        print("All tests passed successfully!")
+        
+    # Clean up by removing the saved HTML file
+    if html_file_path.exists():
+        os.remove(html_file_path)
+        print(f"Removed the saved HTML file: {html_file_path}")
+
+# Run the main function
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+### **How It Works**
+
+1. **Step 1: Crawl the Web URL**
+   - Crawls `https://en.wikipedia.org/wiki/apple`.
+   - Saves the HTML content to `apple.html`.
+   - Records the length of the generated markdown.
+
+2. **Step 2: Crawl from the Local HTML File**
+   - Uses the `file://` prefix to crawl `apple.html`.
+   - Ensures the markdown length matches the original web crawl.
+
+3. **Step 3: Crawl Using Raw HTML Content**
+   - Reads the HTML from `apple.html`.
+   - Prefixes it with `raw:` and crawls.
+   - Verifies the markdown length matches the previous results.
+
+4. **Cleanup**
+   - Deletes the `apple.html` file after testing.
+
+### **Running the Example**
+
+1. **Save the Script:**
+   - Save the above code as `test_crawl4ai.py` in your project directory.
+
+2. **Execute the Script:**
+   - Run the script using:
+     ```bash
+     python test_crawl4ai.py
+     ```
+
+3. **Observe the Output:**
+   - The script will print logs detailing each step.
+   - Assertions ensure consistency across different crawling methods.
+   - Upon success, it confirms that all markdown lengths match.
+
+---
+
+## Conclusion
+
+With the new prefix-based input handling in **Crawl4AI**, you can effortlessly crawl web URLs, local HTML files, and raw HTML strings using a unified `url` parameter. This enhancement simplifies the API usage and provides greater flexibility for diverse crawling scenarios.
+

docs/md_v2/basic/quickstart.md

@@ -8,7 +8,7 @@ First, let's import the necessary modules and create an instance of `AsyncWebCra
 
 ```python
 import asyncio
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CasheMode
 
 async def main():
     async with AsyncWebCrawler(verbose=True) as crawler:
@@ -42,7 +42,7 @@ async def capture_and_save_screenshot(url: str, output_path: str):
         result = await crawler.arun(
             url=url,
             screenshot=True,
-            bypass_cache=True
+            cache_mode=CacheMode.BYPASS
         )
         
         if result.success and result.screenshot:
@@ -62,15 +62,15 @@ Crawl4AI supports multiple browser engines. Here's how to use different browsers
 ```python
 # Use Firefox
 async with AsyncWebCrawler(browser_type="firefox", verbose=True, headless=True) as crawler:
-    result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
+    result = await crawler.arun(url="https://www.example.com", cache_mode=CacheMode.BYPASS)
 
 # Use WebKit
 async with AsyncWebCrawler(browser_type="webkit", verbose=True, headless=True) as crawler:
-    result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
+    result = await crawler.arun(url="https://www.example.com", cache_mode=CacheMode.BYPASS)
 
 # Use Chromium (default)
 async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
-    result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
+    result = await crawler.arun(url="https://www.example.com", cache_mode=CacheMode.BYPASS)
 ```
 
 ### User Simulation 🎭
@@ -81,7 +81,7 @@ Simulate real user behavior to avoid detection:
 async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
     result = await crawler.arun(
         url="YOUR-URL-HERE",
-        bypass_cache=True,
+        cache_mode=CacheMode.BYPASS,
         simulate_user=True,  # Causes random mouse movements and clicks
         override_navigator=True  # Makes the browser appear more like a real user
     )
@@ -99,7 +99,7 @@ async def main():
         print(f"First crawl result: {result1.markdown[:100]}...")
 
         # Force to crawl again
-        result2 = await crawler.arun(url="https://www.nbcnews.com/business", bypass_cache=True)
+        result2 = await crawler.arun(url="https://www.nbcnews.com/business", cache_mode=CacheMode.BYPASS)
         print(f"Second crawl result: {result2.markdown[:100]}...")
 
 asyncio.run(main())
@@ -189,7 +189,7 @@ extraction_strategy = LLMExtractionStrategy(
 async with AsyncWebCrawler() as crawler:
     result = await crawler.arun(
         url="https://paulgraham.com/love.html",
-        bypass_cache=True,
+        cache_mode=CacheMode.BYPASS,
         extraction_strategy=extraction_strategy
     )
 ```
@@ -239,7 +239,7 @@ async def crawl_dynamic_content():
                 js_code=js_next_page if page > 0 else None,
                 wait_for=wait_for if page > 0 else None,
                 js_only=page > 0,
-                bypass_cache=True,
+                cache_mode=CacheMode.BYPASS,
                 headless=False,
             )
 
@@ -254,7 +254,7 @@ Remove overlay elements and fit content appropriately:
 async with AsyncWebCrawler(headless=False) as crawler:
     result = await crawler.arun(
         url="your-url-here",
-        bypass_cache=True,
+        cache_mode=CacheMode.BYPASS,
         word_count_threshold=10,
         remove_overlay_elements=True,
         screenshot=True
@@ -282,7 +282,7 @@ async with AsyncWebCrawler() as crawler:
     result = await crawler.arun(
         url="https://www.nbcnews.com/business",
         word_count_threshold=0,
-        bypass_cache=True,
+        cache_mode=CacheMode.BYPASS,
         verbose=False,
     )
     end = time.time()

docs/md_v2/basic/simple-crawling.md

@@ -12,7 +12,9 @@ from crawl4ai import AsyncWebCrawler
 
 async def main():
     async with AsyncWebCrawler() as crawler:
-        result = await crawler.arun(url="https://example.com")
+        result = await crawler.arun(
+            url="https://example.com"
+        )
         print(result.markdown)  # Print clean markdown content
 
 if __name__ == "__main__":
@@ -24,7 +26,7 @@ if __name__ == "__main__":
 The `arun()` method returns a `CrawlResult` object with several useful properties. Here's a quick overview (see [CrawlResult](../api/crawl-result.md) for complete details):
 
 ```python
-result = await crawler.arun(url="https://example.com")
+result = await crawler.arun(url="https://example.com", fit_markdown=True)
 
 # Different content formats
 print(result.html)         # Raw HTML
@@ -81,7 +83,7 @@ Here's a more comprehensive example showing common usage patterns:
 
 ```python
 import asyncio
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
 
 async def main():
     async with AsyncWebCrawler(verbose=True) as crawler:
@@ -97,7 +99,7 @@ async def main():
             remove_overlay_elements=True,
             
             # Cache control
-            bypass_cache=False  # Use cache if available
+            cache_mode=CacheMode.ENABLE  # Use cache if available
         )
         
         if result.success:

docs/md_v2/extraction/extraction_strategies.md

@@ -1,185 +0,0 @@
-## Extraction Strategies 🧠
-
-Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into three of the most important strategies: `CosineStrategy`, `LLMExtractionStrategy`, and the new `JsonCssExtractionStrategy`.
-
-### LLMExtractionStrategy
-
-`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
-
-#### When to Use
-- Suitable for complex extraction tasks requiring nuanced understanding.
-- Ideal for scenarios where detailed instructions can guide the extraction process.
-- Perfect for extracting specific types of information or content with precise guidelines.
-
-#### Parameters
-- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
-- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
-- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
-
-#### Example Without Instructions
-```python
-import asyncio
-import os
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define extraction strategy without instructions
-        strategy = LLMExtractionStrategy(
-            provider='openai',
-            api_token=os.getenv('OPENAI_API_KEY')
-        )
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        print(result.extracted_content)
-
-asyncio.run(main())
-```
-
-#### Example With Instructions
-```python
-import asyncio
-import os
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define extraction strategy with instructions
-        strategy = LLMExtractionStrategy(
-            provider='openai',
-            api_token=os.getenv('OPENAI_API_KEY'),
-            instruction="Extract only financial news and summarize key points."
-        )
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        print(result.extracted_content)
-
-asyncio.run(main())
-```
-
-### JsonCssExtractionStrategy
-
-`JsonCssExtractionStrategy` is a powerful tool for extracting structured data from HTML using CSS selectors. It allows you to define a schema that maps CSS selectors to specific fields, enabling precise and efficient data extraction.
-
-#### When to Use
-- Ideal for extracting structured data from websites with consistent HTML structures.
-- Perfect for scenarios where you need to extract specific elements or attributes from a webpage.
-- Suitable for creating datasets from web pages with tabular or list-based information.
-
-#### Parameters
-- `schema` (Dict[str, Any]): A dictionary defining the extraction schema, including base selector and field definitions.
-
-#### Example
-```python
-import asyncio
-import json
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define the extraction schema
-        schema = {
-            "name": "News Articles",
-            "baseSelector": "article.tease-card",
-            "fields": [
-                {
-                    "name": "title",
-                    "selector": "h2",
-                    "type": "text",
-                },
-                {
-                    "name": "summary",
-                    "selector": "div.tease-card__info",
-                    "type": "text",
-                },
-                {
-                    "name": "link",
-                    "selector": "a",
-                    "type": "attribute",
-                    "attribute": "href"
-                }
-            ],
-        }
-
-        # Create the extraction strategy
-        strategy = JsonCssExtractionStrategy(schema, verbose=True)
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        
-        # Parse and print the extracted content
-        extracted_data = json.loads(result.extracted_content)
-        print(json.dumps(extracted_data, indent=2))
-
-asyncio.run(main())
-```
-
-#### Use Cases for JsonCssExtractionStrategy
-- Extracting product information from e-commerce websites.
-- Gathering news articles and their metadata from news portals.
-- Collecting user reviews and ratings from review websites.
-- Extracting job listings from job boards.
-
-By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy`, nuanced, instruction-based extraction with `LLMExtractionStrategy`, or precise structured data extraction with `JsonCssExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨
-
-For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](./css-advanced.md).
-
-
-### CosineStrategy
-
-`CosineStrategy` uses hierarchical clustering based on cosine similarity to group text chunks into meaningful clusters. This method converts each chunk into its embedding and then clusters them to form semantical chunks.
-
-#### When to Use
-- Ideal for fast, accurate semantic segmentation of text.
-- Perfect for scenarios where LLMs might be overkill or too slow.
-- Suitable for narrowing down content based on specific queries or keywords.
-
-#### Parameters
-- `semantic_filter` (str, optional): Keywords for filtering relevant documents before clustering. Documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.
-- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.
-- `max_dist` (float, optional): Maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.
-- `linkage_method` (str, optional): Linkage method for hierarchical clustering. Default is `'ward'`.
-- `top_k` (int, optional): Number of top categories to extract. Default is `3`.
-- `model_name` (str, optional): Model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.
-
-#### Example
-```python
-import asyncio
-from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import CosineStrategy
-
-async def main():
-    async with AsyncWebCrawler(verbose=True) as crawler:
-        # Define extraction strategy
-        strategy = CosineStrategy(
-            semantic_filter="finance economy stock market",
-            word_count_threshold=10,
-            max_dist=0.2,
-            linkage_method='ward',
-            top_k=3,
-            model_name='BAAI/bge-small-en-v1.5'
-        )
-
-        # Sample URL
-        url = "https://www.nbcnews.com/business"
-
-        # Run the crawler with the extraction strategy
-        result = await crawler.arun(url=url, extraction_strategy=strategy)
-        print(result.extracted_content)
-
-asyncio.run(main())
-```

docs/md_v2/index.md

@@ -72,7 +72,7 @@ Our documentation is organized into several sections:
 ### Advanced Features
 - [Magic Mode](advanced/magic-mode.md)
 - [Session Management](advanced/session-management.md)
-- [Hooks & Authentication](advanced/hooks.md)
+- [Hooks & Authentication](advanced/hooks-auth.md)
 - [Proxy & Security](advanced/proxy-security.md)
 - [Content Processing](advanced/content-processing.md)
 

docs/md_v2/tutorial/episode_01_Introduction_to_Crawl4AI_and_Basic_Installation.md

@@ -0,0 +1,51 @@
+# Crawl4AI
+
+## Episode 1: Introduction to Crawl4AI and Basic Installation
+
+### Quick Intro
+Walk through installation from PyPI, setup, and verification. Show how to install with options like `torch` or `transformer` for advanced capabilities.
+
+Here's a condensed outline of the **Installation and Setup** video content:
+
+---
+
+1) **Introduction to Crawl4AI**: Briefly explain that Crawl4AI is a powerful tool for web scraping, data extraction, and content processing, with customizable options for various needs.
+
+2) **Installation Overview**:   
+   
+   - **Basic Install**: Run `pip install crawl4ai` and `playwright install` (to set up browser dependencies).
+ 
+   - **Optional Advanced Installs**:
+     - `pip install crawl4ai[torch]` - Adds PyTorch for clustering.
+     - `pip install crawl4ai[transformer]` - Adds support for LLM-based extraction.
+     - `pip install crawl4ai[all]` - Installs all features for complete functionality.
+
+3) **Verifying the Installation**:
+   
+   - Walk through a simple test script to confirm the setup:
+      ```python
+      import asyncio
+      from crawl4ai import AsyncWebCrawler
+      
+      async def main():
+          async with AsyncWebCrawler(verbose=True) as crawler:
+              result = await crawler.arun(url="https://www.example.com")
+              print(result.markdown[:500])  # Show first 500 characters
+
+      asyncio.run(main())
+      ```
+   - Explain that this script initializes the crawler and runs it on a test URL, displaying part of the extracted content to verify functionality.
+
+4) **Important Tips**:
+   
+   - **Run** `playwright install` **after installation** to set up dependencies.
+   - **For full performance** on text-related tasks, run `crawl4ai-download-models` after installing with `[torch]`, `[transformer]`, or `[all]` options.
+   - If you encounter issues, refer to the documentation or GitHub issues.
+
+5) **Wrap Up**:
+   
+   - Introduce the next topic in the series, which will cover Crawl4AI's browser configuration options (like choosing between `chromium`, `firefox`, and `webkit`).
+
+---
+
+This structure provides a concise, effective guide to get viewers up and running with Crawl4AI in minutes.

docs/md_v2/tutorial/episode_02_Overview_of_Advanced_Features.md

@@ -0,0 +1,78 @@
+# Crawl4AI
+
+## Episode 2: Overview of Advanced Features
+
+### Quick Intro
+A general overview of advanced features like hooks, CSS selectors, and JSON CSS extraction.
+
+Here's a condensed outline for an **Overview of Advanced Features** video covering Crawl4AI's powerful customization and extraction options:
+
+---
+
+### **Overview of Advanced Features**
+
+1) **Introduction to Advanced Features**:
+ 
+   - Briefly introduce Crawl4AI’s advanced tools, which let users go beyond basic crawling to customize and fine-tune their scraping workflows.
+
+2) **Taking Screenshots**:
+ 
+   - Explain the screenshot capability for capturing page state and verifying content.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", screenshot=True)
+      ```
+   - Mention that screenshots are saved as a base64 string in `result`, allowing easy decoding and saving.
+
+3) **Media and Link Extraction**:
+ 
+   - Demonstrate how to pull all media (images, videos) and links (internal and external) from a page for deeper analysis or content gathering.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com")
+      print("Media:", result.media)
+      print("Links:", result.links)
+      ```
+
+4) **Custom User Agent**:
+ 
+   - Show how to set a custom user agent to disguise the crawler or simulate specific devices/browsers.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
+      ```
+
+5) **Custom Hooks for Enhanced Control**:
+ 
+   - Briefly cover how to use hooks, which allow custom actions like setting headers or handling login during the crawl.
+   - **Example**: Setting a custom header with `before_get_url` hook.
+      ```python
+      async def before_get_url(page):
+          await page.set_extra_http_headers({"X-Test-Header": "test"})
+      ```
+
+6) **CSS Selectors for Targeted Extraction**:
+ 
+   - Explain the use of CSS selectors to extract specific elements, ideal for structured data like articles or product details.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", css_selector="h2")
+      print("H2 Tags:", result.extracted_content)
+      ```
+
+7) **Crawling Inside Iframes**:
+ 
+   - Mention how enabling `process_iframes=True` allows extracting content within iframes, useful for sites with embedded content or ads.
+   - **Example**:
+      ```python
+      result = await crawler.arun(url="https://www.example.com", process_iframes=True)
+      ```
+
+8) **Wrap-Up**:
+ 
+   - Summarize these advanced features and how they allow users to customize every part of their web scraping experience.
+   - Tease upcoming videos where each feature will be explored in detail.
+
+---
+
+This covers each advanced feature with a brief example, providing a useful overview to prepare viewers for the more in-depth videos.

docs/md_v2/tutorial/episode_03_Browser_Configurations_&_Headless_Crawling.md

@@ -0,0 +1,65 @@
+# Crawl4AI
+
+## Episode 3: Browser Configurations & Headless Crawling
+
+### Quick Intro
+Explain browser options (`chromium`, `firefox`, `webkit`) and settings for headless mode, caching, and verbose logging.
+
+Here’s a streamlined outline for the **Browser Configurations & Headless Crawling** video:
+
+---
+
+### **Browser Configurations & Headless Crawling**
+
+1) **Overview of Browser Options**:
+
+   - Crawl4AI supports three browser engines:
+     - **Chromium** (default) - Highly compatible.
+     - **Firefox** - Great for specialized use cases.
+     - **Webkit** - Lightweight, ideal for basic needs.
+   - **Example**:
+      ```python
+      # Using Chromium (default)
+      crawler = AsyncWebCrawler(browser_type="chromium")
+      
+      # Using Firefox
+      crawler = AsyncWebCrawler(browser_type="firefox")
+      
+      # Using WebKit
+      crawler = AsyncWebCrawler(browser_type="webkit")
+      ```
+
+2) **Headless Mode**:
+
+   - Headless mode runs the browser without a visible GUI, making it faster and less resource-intensive.
+   - To enable or disable:
+      ```python
+      # Headless mode (default is True)
+      crawler = AsyncWebCrawler(headless=True)
+      
+      # Disable headless mode for debugging
+      crawler = AsyncWebCrawler(headless=False)
+      ```
+
+3) **Verbose Logging**:
+   - Use `verbose=True` to get detailed logs for each action, useful for debugging:
+      ```python
+      crawler = AsyncWebCrawler(verbose=True)
+      ```
+
+4) **Running a Basic Crawl with Configuration**:
+   - Example of a simple crawl with custom browser settings:
+      ```python
+      async with AsyncWebCrawler(browser_type="firefox", headless=True, verbose=True) as crawler:
+          result = await crawler.arun(url="https://www.example.com")
+          print(result.markdown[:500])  # Show first 500 characters
+      ```
+   - This example uses Firefox in headless mode with logging enabled, demonstrating the flexibility of Crawl4AI’s setup.
+
+5) **Recap & Next Steps**:
+   - Recap the power of selecting different browsers and running headless mode for speed and efficiency.
+   - Tease the next video: **Proxy & Security Settings** for navigating blocked or restricted content and protecting IP identity.
+
+---
+
+This breakdown covers browser configuration essentials in Crawl4AI, providing users with practical steps to optimize their scraping setup.

docs/md_v2/tutorial/episode_04_Advanced_Proxy_and_Security_Settings.md

@@ -0,0 +1,90 @@
+# Crawl4AI
+
+## Episode 4: Advanced Proxy and Security Settings
+
+### Quick Intro
+Showcase proxy configurations (HTTP, SOCKS5, authenticated proxies). Demo: Use rotating proxies and set custom headers to avoid IP blocking and enhance security.
+
+Here’s a focused outline for the **Proxy and Security Settings** video:
+
+---
+
+### **Proxy & Security Settings**
+
+1) **Why Use Proxies in Web Crawling**:
+
+   - Proxies are essential for bypassing IP-based restrictions, improving anonymity, and managing rate limits.
+   - Crawl4AI supports simple proxies, authenticated proxies, and proxy rotation for robust web scraping.
+
+2) **Basic Proxy Setup**:
+
+   - **Using a Simple Proxy**:
+     ```python
+     # HTTP proxy
+     crawler = AsyncWebCrawler(proxy="http://proxy.example.com:8080")
+     
+     # SOCKS proxy
+     crawler = AsyncWebCrawler(proxy="socks5://proxy.example.com:1080")
+     ```
+
+3) **Authenticated Proxies**:
+
+   - Use `proxy_config` for proxies requiring a username and password:
+     ```python
+     proxy_config = {
+         "server": "http://proxy.example.com:8080",
+         "username": "user",
+         "password": "pass"
+     }
+     crawler = AsyncWebCrawler(proxy_config=proxy_config)
+     ```
+
+4) **Rotating Proxies**:
+
+   - Rotating proxies helps avoid IP bans by switching IP addresses for each request:
+     ```python
+     async def get_next_proxy():
+         # Define proxy rotation logic here
+         return {"server": "http://next.proxy.com:8080"}
+     
+     async with AsyncWebCrawler() as crawler:
+         for url in urls:
+             proxy = await get_next_proxy()
+             crawler.update_proxy(proxy)
+             result = await crawler.arun(url=url)
+     ```
+   - This setup periodically switches the proxy for enhanced security and access.
+
+5) **Custom Headers for Additional Security**:
+
+   - Set custom headers to mask the crawler’s identity and avoid detection:
+     ```python
+     headers = {
+         "X-Forwarded-For": "203.0.113.195",
+         "Accept-Language": "en-US,en;q=0.9",
+         "Cache-Control": "no-cache",
+         "Pragma": "no-cache"
+     }
+     crawler = AsyncWebCrawler(headers=headers)
+     ```
+
+6) **Combining Proxies with Magic Mode for Anti-Bot Protection**:
+
+   - For sites with aggressive bot detection, combine `proxy` settings with `magic=True`:
+     ```python
+     async with AsyncWebCrawler(proxy="http://proxy.example.com:8080", headers={"Accept-Language": "en-US"}) as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             magic=True  # Enables anti-detection features
+         )
+     ```
+   - **Magic Mode** automatically enables user simulation, random timing, and browser property masking.
+
+7) **Wrap Up & Next Steps**:
+
+   - Summarize the importance of proxies and anti-detection in accessing restricted content and avoiding bans.
+   - Tease the next video: **JavaScript Execution and Handling Dynamic Content** for working with interactive and dynamically loaded pages.
+
+---
+
+This outline provides a practical guide to setting up proxies and security configurations, empowering users to navigate restricted sites while staying undetected.

docs/md_v2/tutorial/episode_05_JavaScript_Execution_and_Dynamic_Content_Handling.md

@@ -0,0 +1,97 @@
+# Crawl4AI
+
+## Episode 5: JavaScript Execution and Dynamic Content Handling
+
+### Quick Intro
+Explain JavaScript code injection with examples (e.g., simulating scrolling, clicking ‘load more’). Demo: Extract content from a page that uses dynamic loading with lazy-loaded images.
+
+Here’s a focused outline for the **JavaScript Execution and Dynamic Content Handling** video:
+
+---
+
+### **JavaScript Execution & Dynamic Content Handling**
+
+1) **Why JavaScript Execution Matters**:
+
+   - Many modern websites load content dynamically via JavaScript, requiring special handling to access all elements.
+   - Crawl4AI can execute JavaScript on pages, enabling it to interact with elements like “load more” buttons, infinite scrolls, and content that appears only after certain actions.
+
+2) **Basic JavaScript Execution**:
+
+   - Use `js_code` to execute JavaScript commands on a page:
+     ```python
+     # Scroll to bottom of the page
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code="window.scrollTo(0, document.body.scrollHeight);"
+     )
+     ```
+   - This command scrolls to the bottom, triggering any lazy-loaded or dynamically added content.
+
+3) **Multiple Commands & Simulating Clicks**:
+
+   - Combine multiple JavaScript commands to interact with elements like “load more” buttons:
+     ```python
+     js_commands = [
+         "window.scrollTo(0, document.body.scrollHeight);",
+         "document.querySelector('.load-more').click();"
+     ]
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code=js_commands
+     )
+     ```
+   - This script scrolls down and then clicks the “load more” button, useful for loading additional content blocks.
+
+4) **Waiting for Dynamic Content**:
+
+   - Use `wait_for` to ensure the page loads specific elements before proceeding:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code="window.scrollTo(0, document.body.scrollHeight);",
+         wait_for="css:.dynamic-content"  # Wait for elements with class `.dynamic-content`
+     )
+     ```
+   - This example waits until elements with `.dynamic-content` are loaded, helping to capture content that appears after JavaScript actions.
+
+5) **Handling Complex Dynamic Content (e.g., Infinite Scroll)**:
+
+   - Combine JavaScript execution with conditional waiting to handle infinite scrolls or paginated content:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         js_code=[
+             "window.scrollTo(0, document.body.scrollHeight);",
+             "const loadMore = document.querySelector('.load-more'); if (loadMore) loadMore.click();"
+         ],
+         wait_for="js:() => document.querySelectorAll('.item').length > 10"  # Wait until 10 items are loaded
+     )
+     ```
+   - This example scrolls and clicks "load more" repeatedly, waiting each time for a specified number of items to load.
+
+6) **Complete Example: Dynamic Content Handling with Extraction**:
+
+   - Full example demonstrating a dynamic load and content extraction in one process:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             js_code=[
+                 "window.scrollTo(0, document.body.scrollHeight);",
+                 "document.querySelector('.load-more').click();"
+             ],
+             wait_for="css:.main-content",
+             css_selector=".main-content"
+         )
+         print(result.markdown[:500])  # Output the main content extracted
+     ```
+
+7) **Wrap Up & Next Steps**:
+
+   - Recap how JavaScript execution allows access to dynamic content, enabling powerful interactions.
+   - Tease the next video: **Content Cleaning and Fit Markdown** to show how Crawl4AI can extract only the most relevant content from complex pages.
+
+---
+
+This outline explains how to handle dynamic content and JavaScript-based interactions effectively, enabling users to scrape and interact with complex, modern websites.

docs/md_v2/tutorial/episode_06_Magic_Mode_and_Anti-Bot_Protection.md

@@ -0,0 +1,86 @@
+# Crawl4AI
+
+## Episode 6: Magic Mode and Anti-Bot Protection
+
+### Quick Intro
+Highlight `Magic Mode` and anti-bot features like user simulation, navigator overrides, and timing randomization. Demo: Access a site with anti-bot protection and show how `Magic Mode` seamlessly handles it.
+
+Here’s a concise outline for the **Magic Mode and Anti-Bot Protection** video:
+
+---
+
+### **Magic Mode & Anti-Bot Protection**
+
+1) **Why Anti-Bot Protection is Important**:
+
+   - Many websites use bot detection mechanisms to block automated scraping. Crawl4AI’s anti-detection features help avoid IP bans, CAPTCHAs, and access restrictions.
+   - **Magic Mode** is a one-step solution to enable a range of anti-bot features without complex configuration.
+
+2) **Enabling Magic Mode**:
+
+   - Simply set `magic=True` to activate Crawl4AI’s full anti-bot suite:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         magic=True  # Enables all anti-detection features
+     )
+     ```
+   - This enables a blend of stealth techniques, including masking automation signals, randomizing timings, and simulating real user behavior.
+
+3) **What Magic Mode Does Behind the Scenes**:
+
+   - **User Simulation**: Mimics human actions like mouse movements and scrolling.
+   - **Navigator Overrides**: Hides signals that indicate an automated browser.
+   - **Timing Randomization**: Adds random delays to simulate natural interaction patterns.
+   - **Cookie Handling**: Accepts and manages cookies dynamically to avoid triggers from cookie pop-ups.
+
+4) **Manual Anti-Bot Options (If Not Using Magic Mode)**:
+
+   - For granular control, you can configure individual settings without Magic Mode:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         simulate_user=True,        # Enables human-like behavior
+         override_navigator=True    # Hides automation fingerprints
+     )
+     ```
+   - **Use Cases**: This approach allows more specific adjustments when certain anti-bot features are needed but others are not.
+
+5) **Combining Proxies with Magic Mode**:
+
+   - To avoid rate limits or IP blocks, combine Magic Mode with a proxy:
+     ```python
+     async with AsyncWebCrawler(
+         proxy="http://proxy.example.com:8080",
+         headers={"Accept-Language": "en-US"}
+     ) as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             magic=True  # Full anti-detection
+         )
+     ```
+   - This setup maximizes stealth by pairing anti-bot detection with IP obfuscation.
+
+6) **Example of Anti-Bot Protection in Action**:
+
+   - Full example with Magic Mode and proxies to scrape a protected page:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com/protected-content",
+             magic=True,
+             proxy="http://proxy.example.com:8080",
+             wait_for="css:.content-loaded"  # Wait for the main content to load
+         )
+         print(result.markdown[:500])  # Display first 500 characters of the content
+     ```
+   - This example ensures seamless access to protected content by combining anti-detection and waiting for full content load.
+
+7) **Wrap Up & Next Steps**:
+
+   - Recap the power of Magic Mode and anti-bot features for handling restricted websites.
+   - Tease the next video: **Content Cleaning and Fit Markdown** to show how to extract clean and focused content from a page.
+
+---
+
+This outline shows users how to easily avoid bot detection and access restricted content, demonstrating both the power and simplicity of Magic Mode in Crawl4AI.

docs/md_v2/tutorial/episode_07_Content_Cleaning_and_Fit_Markdown.md

@@ -0,0 +1,89 @@
+# Crawl4AI
+
+## Episode 7: Content Cleaning and Fit Markdown
+
+### Quick Intro
+Explain content cleaning options, including `fit_markdown` to keep only the most relevant content. Demo: Extract and compare regular vs. fit markdown from a news site or blog.
+
+Here’s a streamlined outline for the **Content Cleaning and Fit Markdown** video:
+
+---
+
+### **Content Cleaning & Fit Markdown**
+
+1) **Overview of Content Cleaning in Crawl4AI**:
+
+   - Explain that web pages often include extra elements like ads, navigation bars, footers, and popups.
+   - Crawl4AI’s content cleaning features help extract only the main content, reducing noise and enhancing readability.
+
+2) **Basic Content Cleaning Options**:
+
+   - **Removing Unwanted Elements**: Exclude specific HTML tags, like forms or navigation bars:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         word_count_threshold=10,       # Filter out blocks with fewer than 10 words
+         excluded_tags=['form', 'nav'], # Exclude specific tags
+         remove_overlay_elements=True   # Remove popups and modals
+     )
+     ```
+   - This example extracts content while excluding forms, navigation, and modal overlays, ensuring clean results.
+
+3) **Fit Markdown for Main Content Extraction**:
+
+   - **What is Fit Markdown**: Uses advanced analysis to identify the most relevant content (ideal for articles, blogs, and documentation).
+   - **How it Works**: Analyzes content density, removes boilerplate elements, and maintains formatting for a clear output.
+   - **Example**:
+     ```python
+     result = await crawler.arun(url="https://example.com")
+     main_content = result.fit_markdown  # Extracted main content
+     print(main_content[:500])  # Display first 500 characters
+     ```
+   - Fit Markdown is especially helpful for long-form content like news articles or blog posts.
+
+4) **Comparing Fit Markdown with Regular Markdown**:
+
+   - **Fit Markdown** returns the primary content without extraneous elements.
+   - **Regular Markdown** includes all extracted text in markdown format.
+   - Example to show the difference:
+     ```python
+     all_content = result.markdown      # Full markdown
+     main_content = result.fit_markdown # Only the main content
+     
+     print(f"All Content Length: {len(all_content)}")
+     print(f"Main Content Length: {len(main_content)}")
+     ```
+   - This comparison shows the effectiveness of Fit Markdown in focusing on essential content.
+
+5) **Media and Metadata Handling with Content Cleaning**:
+
+   - **Media Extraction**: Crawl4AI captures images and videos with metadata like alt text, descriptions, and relevance scores:
+     ```python
+     for image in result.media["images"]:
+         print(f"Source: {image['src']}, Alt Text: {image['alt']}, Relevance Score: {image['score']}")
+     ```
+   - **Use Case**: Useful for saving only relevant images or videos from an article or content-heavy page.
+
+6) **Example of Clean Content Extraction in Action**:
+
+   - Full example extracting cleaned content and Fit Markdown:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             word_count_threshold=10,
+             excluded_tags=['nav', 'footer'],
+             remove_overlay_elements=True
+         )
+         print(result.fit_markdown[:500])  # Show main content
+     ```
+   - This example demonstrates content cleaning with settings for filtering noise and focusing on the core text.
+
+7) **Wrap Up & Next Steps**:
+
+   - Summarize the power of Crawl4AI’s content cleaning features and Fit Markdown for capturing clean, relevant content.
+   - Tease the next video: **Link Analysis and Smart Filtering** to focus on analyzing and filtering links within crawled pages.
+
+---
+
+This outline covers Crawl4AI’s content cleaning features and the unique benefits of Fit Markdown, showing users how to retrieve focused, high-quality content from web pages.

docs/md_v2/tutorial/episode_08_Media_Handling_Images_Videos_and_Audio.md

@@ -0,0 +1,116 @@
+# Crawl4AI
+
+## Episode 8: Media Handling: Images, Videos, and Audio
+
+### Quick Intro
+Showcase Crawl4AI’s media extraction capabilities, including lazy-loaded media and metadata. Demo: Crawl a multimedia page, extract images, and show metadata (alt text, context, relevance score).
+
+Here’s a clear and focused outline for the **Media Handling: Images, Videos, and Audio** video:
+
+---
+
+### **Media Handling: Images, Videos, and Audio**
+
+1) **Overview of Media Extraction in Crawl4AI**:
+
+   - Crawl4AI can detect and extract different types of media (images, videos, and audio) along with useful metadata.
+   - This functionality is essential for gathering visual content from multimedia-heavy pages like e-commerce sites, news articles, and social media feeds.
+
+2) **Image Extraction and Metadata**:
+
+   - Crawl4AI captures images with detailed metadata, including:
+     - **Source URL**: The direct URL to the image.
+     - **Alt Text**: Image description if available.
+     - **Relevance Score**: A score (0–10) indicating how relevant the image is to the main content.
+     - **Context**: Text surrounding the image on the page.
+   - **Example**:
+     ```python
+     result = await crawler.arun(url="https://example.com")
+     
+     for image in result.media["images"]:
+         print(f"Source: {image['src']}")
+         print(f"Alt Text: {image['alt']}")
+         print(f"Relevance Score: {image['score']}")
+         print(f"Context: {image['context']}")
+     ```
+   - This example shows how to access each image’s metadata, making it easy to filter for the most relevant visuals.
+
+3) **Handling Lazy-Loaded Images**:
+
+   - Crawl4AI automatically supports lazy-loaded images, which are commonly used to optimize webpage loading.
+   - **Example with Wait for Lazy-Loaded Content**:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         wait_for="css:img[data-src]",  # Wait for lazy-loaded images
+         delay_before_return_html=2.0   # Allow extra time for images to load
+     )
+     ```
+   - This setup waits for lazy-loaded images to appear, ensuring they are fully captured.
+
+4) **Video Extraction and Metadata**:
+
+   - Crawl4AI captures video elements, including:
+     - **Source URL**: The video’s direct URL.
+     - **Type**: Format of the video (e.g., MP4).
+     - **Thumbnail**: A poster or thumbnail image if available.
+     - **Duration**: Video length, if metadata is provided.
+   - **Example**:
+     ```python
+     for video in result.media["videos"]:
+         print(f"Video Source: {video['src']}")
+         print(f"Type: {video['type']}")
+         print(f"Thumbnail: {video.get('poster')}")
+         print(f"Duration: {video.get('duration')}")
+     ```
+   - This allows users to gather video content and relevant details for further processing or analysis.
+
+5) **Audio Extraction and Metadata**:
+
+   - Audio elements can also be extracted, with metadata like:
+     - **Source URL**: The audio file’s direct URL.
+     - **Type**: Format of the audio file (e.g., MP3).
+     - **Duration**: Length of the audio, if available.
+   - **Example**:
+     ```python
+     for audio in result.media["audios"]:
+         print(f"Audio Source: {audio['src']}")
+         print(f"Type: {audio['type']}")
+         print(f"Duration: {audio.get('duration')}")
+     ```
+   - Useful for sites with podcasts, sound bites, or other audio content.
+
+6) **Filtering Media by Relevance**:
+
+   - Use metadata like relevance score to filter only the most useful media content:
+     ```python
+     relevant_images = [img for img in result.media["images"] if img['score'] > 5]
+     ```
+   - This is especially helpful for content-heavy pages where you only want media directly related to the main content.
+
+7) **Example: Full Media Extraction with Content Filtering**:
+
+   - Full example extracting images, videos, and audio along with filtering by relevance:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             word_count_threshold=10,  # Filter content blocks for relevance
+             exclude_external_images=True  # Only keep internal images
+         )
+         
+         # Display media summaries
+         print(f"Relevant Images: {len(relevant_images)}")
+         print(f"Videos: {len(result.media['videos'])}")
+         print(f"Audio Clips: {len(result.media['audios'])}")
+     ```
+   - This example shows how to capture and filter various media types, focusing on what’s most relevant.
+
+8) **Wrap Up & Next Steps**:
+
+   - Recap the comprehensive media extraction capabilities, emphasizing how metadata helps users focus on relevant content.
+   - Tease the next video: **Link Analysis and Smart Filtering** to explore how Crawl4AI handles internal, external, and social media links for more focused data gathering.
+
+---
+
+This outline provides users with a complete guide to handling images, videos, and audio in Crawl4AI, using metadata to enhance relevance and precision in multimedia extraction.

docs/md_v2/tutorial/episode_09_Link_Analysis_and_Smart_Filtering.md

@@ -0,0 +1,95 @@
+# Crawl4AI
+
+## Episode 9: Link Analysis and Smart Filtering
+
+### Quick Intro
+Walk through internal and external link classification, social media link filtering, and custom domain exclusion. Demo: Analyze links on a website, focusing on internal navigation vs. external or ad links.
+
+Here’s a focused outline for the **Link Analysis and Smart Filtering** video:
+
+---
+
+### **Link Analysis & Smart Filtering**
+
+1) **Importance of Link Analysis in Web Crawling**:
+
+   - Explain that web pages often contain numerous links, including internal links, external links, social media links, and ads.
+   - Crawl4AI’s link analysis and filtering options help extract only relevant links, enabling more targeted and efficient crawls.
+
+2) **Automatic Link Classification**:
+
+   - Crawl4AI categorizes links automatically into internal, external, and social media links.
+   - **Example**:
+     ```python
+     result = await crawler.arun(url="https://example.com")
+
+     # Access internal and external links
+     internal_links = result.links["internal"]
+     external_links = result.links["external"]
+
+     # Print first few links for each type
+     print("Internal Links:", internal_links[:3])
+     print("External Links:", external_links[:3])
+     ```
+
+3) **Filtering Out Unwanted Links**:
+
+   - **Exclude External Links**: Remove all links pointing to external sites.
+   - **Exclude Social Media Links**: Filter out social media domains like Facebook or Twitter.
+   - **Example**:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         exclude_external_links=True,         # Remove external links
+         exclude_social_media_links=True      # Remove social media links
+     )
+     ```
+
+4) **Custom Domain Filtering**:
+
+   - **Exclude Specific Domains**: Filter links from particular domains, e.g., ad sites.
+   - **Custom Social Media Domains**: Add additional social media domains if needed.
+   - **Example**:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         exclude_domains=["ads.com", "trackers.com"],
+         exclude_social_media_domains=["facebook.com", "linkedin.com"]
+     )
+     ```
+
+5) **Accessing Link Context and Metadata**:
+
+   - Crawl4AI provides additional metadata for each link, including its text, type (e.g., navigation or content), and surrounding context.
+   - **Example**:
+     ```python
+     for link in result.links["internal"]:
+         print(f"Link: {link['href']}, Text: {link['text']}, Context: {link['context']}")
+     ```
+   - **Use Case**: Helps users understand the relevance of links based on where they are placed on the page (e.g., navigation vs. article content).
+
+6) **Example of Comprehensive Link Filtering and Analysis**:
+
+   - Full example combining link filtering, metadata access, and contextual information:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             exclude_external_links=True,
+             exclude_social_media_links=True,
+             exclude_domains=["ads.com"],
+             css_selector=".main-content"  # Focus only on main content area
+         )
+         for link in result.links["internal"]:
+             print(f"Internal Link: {link['href']}, Text: {link['text']}, Context: {link['context']}")
+     ```
+   - This example filters unnecessary links, keeping only internal and relevant links from the main content area.
+
+7) **Wrap Up & Next Steps**:
+
+   - Summarize the benefits of link filtering for efficient crawling and relevant content extraction.
+   - Tease the next video: **Custom Headers, Identity Management, and User Simulation** to explain how to configure identity settings and simulate user behavior for stealthier crawls.
+
+---
+
+This outline provides a practical overview of Crawl4AI’s link analysis and filtering features, helping users target only essential links while eliminating distractions.

docs/md_v2/tutorial/episode_10_Custom_Headers,_Identity,_and_User_Simulation.md

@@ -0,0 +1,93 @@
+# Crawl4AI
+
+## Episode 10: Custom Headers, Identity, and User Simulation
+
+### Quick Intro
+Teach how to use custom headers, user-agent strings, and simulate real user interactions. Demo: Set custom user-agent and headers to access a site that blocks typical crawlers.
+
+Here’s a concise outline for the **Custom Headers, Identity Management, and User Simulation** video:
+
+---
+
+### **Custom Headers, Identity Management, & User Simulation**
+
+1) **Why Customize Headers and Identity in Crawling**:
+
+   - Websites often track request headers and browser properties to detect bots. Customizing headers and managing identity help make requests appear more human, improving access to restricted sites.
+
+2) **Setting Custom Headers**:
+
+   - Customize HTTP headers to mimic genuine browser requests or meet site-specific requirements:
+     ```python
+     headers = {
+         "Accept-Language": "en-US,en;q=0.9",
+         "X-Requested-With": "XMLHttpRequest",
+         "Cache-Control": "no-cache"
+     }
+     crawler = AsyncWebCrawler(headers=headers)
+     ```
+   - **Use Case**: Customize the `Accept-Language` header to simulate local user settings, or `Cache-Control` to bypass cache for fresh content.
+
+3) **Setting a Custom User Agent**:
+
+   - Some websites block requests from common crawler user agents. Setting a custom user agent string helps bypass these restrictions:
+     ```python
+     crawler = AsyncWebCrawler(
+         user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
+     )
+     ```
+   - **Tip**: Use user-agent strings from popular browsers (e.g., Chrome, Firefox) to improve access and reduce detection risks.
+
+4) **User Simulation for Human-like Behavior**:
+
+   - Enable `simulate_user=True` to mimic natural user interactions, such as random timing and simulated mouse movements:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         simulate_user=True  # Simulates human-like behavior
+     )
+     ```
+   - **Behavioral Effects**: Adds subtle variations in interactions, making the crawler harder to detect on bot-protected sites.
+
+5) **Navigator Overrides and Magic Mode for Full Identity Masking**:
+
+   - Use `override_navigator=True` to mask automation indicators like `navigator.webdriver`, which websites check to detect bots:
+     ```python
+     result = await crawler.arun(
+         url="https://example.com",
+         override_navigator=True  # Masks bot-related signals
+     )
+     ```
+   - **Combining with Magic Mode**: For a complete anti-bot setup, combine these identity options with `magic=True` for maximum protection:
+     ```python
+     async with AsyncWebCrawler() as crawler:
+         result = await crawler.arun(
+             url="https://example.com",
+             magic=True,  # Enables all anti-bot detection features
+             user_agent="Custom-Agent",  # Custom agent with Magic Mode
+         )
+     ```
+   - This setup includes all anti-detection techniques like navigator masking, random timing, and user simulation.
+
+6) **Example: Comprehensive Setup for Identity Management**:
+
+   - A full example combining custom headers, user-agent, and user simulation for a realistic browsing profile:
+     ```python
+     async with AsyncWebCrawler(
+         headers={"Accept-Language": "en-US", "Cache-Control": "no-cache"},
+         user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/91.0",
+         simulate_user=True
+     ) as crawler:
+         result = await crawler.arun(url="https://example.com/secure-page")
+         print(result.markdown[:500])  # Display extracted content
+     ```
+   - This example enables detailed customization for evading detection and accessing protected pages smoothly.
+
+7) **Wrap Up & Next Steps**:
+
+   - Recap the value of headers, user-agent customization, and simulation in bypassing bot detection.
+   - Tease the next video: **Extraction Strategies: JSON CSS, LLM, and Cosine** to dive into structured data extraction methods for high-quality content retrieval.
+
+---
+
+This outline equips users with tools for managing crawler identity and human-like behavior, essential for accessing bot-protected or restricted websites.

docs/md_v2/tutorial/episode_11_1_Extraction_Strategies_JSON_CSS.md

@@ -0,0 +1,186 @@
+Here’s a detailed outline for the **JSON-CSS Extraction Strategy** video, covering all key aspects and supported structures in Crawl4AI:
+
+---
+
+### **10.1 JSON-CSS Extraction Strategy**
+
+#### **1. Introduction to JSON-CSS Extraction**
+   - JSON-CSS Extraction is used for pulling structured data from pages with repeated patterns, like product listings, article feeds, or directories.
+   - This strategy allows defining a schema with CSS selectors and data fields, making it easy to capture nested, list-based, or singular elements.
+
+#### **2. Basic Schema Structure**
+   - **Schema Fields**: The schema has two main components:
+     - `baseSelector`: A CSS selector to locate the main elements you want to extract (e.g., each article or product block).
+     - `fields`: Defines the data fields for each element, supporting various data types and structures.
+
+#### **3. Simple Field Extraction**
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Sample Product</h2>
+         <span class="price">$19.99</span>
+         <p class="description">This is a sample product.</p>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "price", "selector": ".price", "type": "text"},
+             {"name": "description", "selector": ".description", "type": "text"}
+         ]
+     }
+     ```
+   - **Explanation**: Each field captures text content from specified CSS selectors within each `.product` element.
+
+#### **4. Supported Field Types: Text, Attribute, HTML, Regex**
+   - **Field Type Options**:
+     - `text`: Extracts visible text.
+     - `attribute`: Captures an HTML attribute (e.g., `src`, `href`).
+     - `html`: Extracts the raw HTML of an element.
+     - `regex`: Allows regex patterns to extract part of the text.
+
+   - **Example HTML** (including an image):
+     ```html
+     <div class="product">
+         <h2 class="title">Sample Product</h2>
+         <img class="product-image" src="image.jpg" alt="Product Image">
+         <span class="price">$19.99</span>
+         <p class="description">Limited time offer.</p>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "image_url", "selector": ".product-image", "type": "attribute", "attribute": "src"},
+             {"name": "price", "selector": ".price", "type": "regex", "pattern": r"\$(\d+\.\d+)"},
+             {"name": "description_html", "selector": ".description", "type": "html"}
+         ]
+     }
+     ```
+   - **Explanation**:
+     - `attribute`: Extracts the `src` attribute from `.product-image`.
+     - `regex`: Extracts the numeric part from `$19.99`.
+     - `html`: Retrieves the full HTML of the description element.
+
+#### **5. Nested Field Extraction**
+   - **Use Case**: Useful when content contains sub-elements, such as an article with author details within it.
+   - **Example HTML**:
+     ```html
+     <div class="article">
+         <h1 class="title">Sample Article</h1>
+         <div class="author">
+             <span class="name">John Doe</span>
+             <span class="bio">Writer and editor</span>
+         </div>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".article",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "author", "type": "nested", "selector": ".author", "fields": [
+                 {"name": "name", "selector": ".name", "type": "text"},
+                 {"name": "bio", "selector": ".bio", "type": "text"}
+             ]}
+         ]
+     }
+     ```
+   - **Explanation**:
+     - `nested`: Extracts `name` and `bio` within `.author`, grouping the author details in a single `author` object.
+
+#### **6. List and Nested List Extraction**
+   - **List**: Extracts multiple elements matching the selector as a list.
+   - **Nested List**: Allows lists within lists, useful for items with sub-lists (e.g., specifications for each product).
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Product with Features</h2>
+         <ul class="features">
+             <li class="feature">Feature 1</li>
+             <li class="feature">Feature 2</li>
+             <li class="feature">Feature 3</li>
+         </ul>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text"},
+             {"name": "features", "type": "list", "selector": ".features .feature", "fields": [
+                 {"name": "feature", "type": "text"}
+             ]}
+         ]
+     }
+     ```
+   - **Explanation**:
+     - `list`: Captures each `.feature` item within `.features`, outputting an array of features under the `features` field.
+
+#### **7. Transformations for Field Values**
+   - Transformations allow you to modify extracted values (e.g., converting to lowercase).
+   - Supported transformations: `lowercase`, `uppercase`, `strip`.
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Special Product</h2>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text", "transform": "uppercase"}
+         ]
+     }
+     ```
+   - **Explanation**: The `transform` property changes the `title` to uppercase, useful for standardized outputs.
+
+#### **8. Full JSON-CSS Extraction Example**
+   - Combining all elements in a single schema example for a comprehensive crawl:
+   - **Example HTML**:
+     ```html
+     <div class="product">
+         <h2 class="title">Featured Product</h2>
+         <img class="product-image" src="product.jpg">
+         <span class="price">$99.99</span>
+         <p class="description">Best product of the year.</p>
+         <ul class="features">
+             <li class="feature">Durable</li>
+             <li class="feature">Eco-friendly</li>
+         </ul>
+     </div>
+     ```
+   - **Schema**:
+     ```python
+     schema = {
+         "baseSelector": ".product",
+         "fields": [
+             {"name": "title", "selector": ".title", "type": "text", "transform": "uppercase"},
+             {"name": "image_url", "selector": ".product-image", "type": "attribute", "attribute": "src"},
+             {"name": "price", "selector": ".price", "type": "regex", "pattern": r"\$(\d+\.\d+)"},
+             {"name": "description", "selector": ".description", "type": "html"},
+             {"name": "features", "type": "list", "selector": ".features .feature", "fields": [
+                 {"name": "feature", "type": "text"}
+             ]}
+         ]
+     }
+     ```
+   - **Explanation**: This schema captures and transforms each aspect of the product, illustrating the JSON-CSS strategy’s versatility for structured extraction.
+
+#### **9. Wrap Up & Next Steps**
+   - Summarize JSON-CSS Extraction’s flexibility for structured, pattern-based extraction.
+   - Tease the next video: **10.2 LLM Extraction Strategy**, focusing on using language models to extract data based on intelligent content analysis.
+
+---
+
+This outline covers each JSON-CSS Extraction option in Crawl4AI, with practical examples and schema configurations, making it a thorough guide for users.

docs/md_v2/tutorial/episode_11_2_Extraction_Strategies_LLM.md

@@ -0,0 +1,153 @@
+# Crawl4AI
+
+## Episode 11: Extraction Strategies: JSON CSS, LLM, and Cosine
+
+### Quick Intro
+Introduce JSON CSS Extraction Strategy for structured data, LLM Extraction Strategy for intelligent parsing, and Cosine Strategy for clustering similar content. Demo: Use JSON CSS to scrape product details from an e-commerce site.
+
+Here’s a comprehensive outline for the **LLM Extraction Strategy** video, covering key details and example applications.
+
+---
+
+### **10.2 LLM Extraction Strategy**
+
+#### **1. Introduction to LLM Extraction Strategy**
+   - The LLM Extraction Strategy leverages language models to interpret and extract structured data from complex web content.
+   - Unlike traditional CSS selectors, this strategy uses natural language instructions and schemas to guide the extraction, ideal for unstructured or diverse content.
+   - Supports **OpenAI**, **Azure OpenAI**, **HuggingFace**, and **Ollama** models, enabling flexibility with both proprietary and open-source providers.
+
+#### **2. Key Components of LLM Extraction Strategy**
+   - **Provider**: Specifies the LLM provider (e.g., OpenAI, HuggingFace, Azure).
+   - **API Token**: Required for most providers, except Ollama (local LLM model).
+   - **Instruction**: Custom extraction instructions sent to the model, providing flexibility in how the data is structured and extracted.
+   - **Schema**: Optional, defines structured fields to organize extracted data into JSON format.
+   - **Extraction Type**: Supports `"block"` for simpler text blocks or `"schema"` when a structured output format is required.
+   - **Chunking Parameters**: Breaks down large documents, with options to adjust chunk size and overlap rate for more accurate extraction across lengthy texts.
+
+#### **3. Basic Extraction Example: OpenAI Model Pricing**
+   - **Goal**: Extract model names and their input and output fees from the OpenAI pricing page.
+   - **Schema Definition**:
+     - **Model Name**: Text for model identification.
+     - **Input Fee**: Token cost for input processing.
+     - **Output Fee**: Token cost for output generation.
+
+   - **Schema**:
+     ```python
+     class OpenAIModelFee(BaseModel):
+         model_name: str = Field(..., description="Name of the OpenAI model.")
+         input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+         output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
+     ```
+
+   - **Example Code**:
+     ```python
+     async def extract_openai_pricing():
+         async with AsyncWebCrawler() as crawler:
+             result = await crawler.arun(
+                 url="https://openai.com/api/pricing/",
+                 extraction_strategy=LLMExtractionStrategy(
+                     provider="openai/gpt-4o",
+                     api_token=os.getenv("OPENAI_API_KEY"),
+                     schema=OpenAIModelFee.schema(),
+                     extraction_type="schema",
+                     instruction="Extract model names and fees for input and output tokens from the page."
+                 ),
+                 cache_mode=CacheMode.BYPASS
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - The extraction strategy combines a schema and detailed instruction to guide the LLM in capturing structured data.
+     - Each model’s name, input fee, and output fee are extracted in a JSON format.
+
+#### **4. Knowledge Graph Extraction Example**
+   - **Goal**: Extract entities and their relationships from a document for use in a knowledge graph.
+   - **Schema Definition**:
+     - **Entities**: Individual items with descriptions (e.g., people, organizations).
+     - **Relationships**: Connections between entities, including descriptions and relationship types.
+
+   - **Schema**:
+     ```python
+     class Entity(BaseModel):
+         name: str
+         description: str
+
+     class Relationship(BaseModel):
+         entity1: Entity
+         entity2: Entity
+         description: str
+         relation_type: str
+
+     class KnowledgeGraph(BaseModel):
+         entities: List[Entity]
+         relationships: List[Relationship]
+     ```
+
+   - **Example Code**:
+     ```python
+     async def extract_knowledge_graph():
+         extraction_strategy = LLMExtractionStrategy(
+             provider="azure/gpt-4o-mini",
+             api_token=os.getenv("AZURE_API_KEY"),
+             schema=KnowledgeGraph.schema(),
+             extraction_type="schema",
+             instruction="Extract entities and relationships from the content to build a knowledge graph."
+         )
+         async with AsyncWebCrawler() as crawler:
+             result = await crawler.arun(
+                 url="https://example.com/some-article",
+                 extraction_strategy=extraction_strategy,
+                 cache_mode=CacheMode.BYPASS
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - In this setup, the LLM extracts entities and their relationships based on the schema and instruction.
+     - The schema organizes results into a JSON-based knowledge graph format.
+
+#### **5. Key Settings in LLM Extraction**
+   - **Chunking Options**:
+     - For long pages, set `chunk_token_threshold` to specify maximum token count per section.
+     - Adjust `overlap_rate` to control the overlap between chunks, useful for contextual consistency.
+   - **Example**:
+     ```python
+     extraction_strategy = LLMExtractionStrategy(
+         provider="openai/gpt-4",
+         api_token=os.getenv("OPENAI_API_KEY"),
+         chunk_token_threshold=3000,
+         overlap_rate=0.2,  # 20% overlap between chunks
+         instruction="Extract key insights and relationships."
+     )
+     ```
+   - This setup ensures that longer texts are divided into manageable chunks with slight overlap, enhancing the quality of extraction.
+
+#### **6. Flexible Provider Options for LLM Extraction**
+   - **Using Proprietary Models**: OpenAI, Azure, and HuggingFace provide robust language models, often suited for complex or detailed extractions.
+   - **Using Open-Source Models**: Ollama and other open-source models can be deployed locally, suitable for offline or cost-effective extraction.
+   - **Example Call**:
+     ```python
+     await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
+     await extract_structured_data_using_llm("openai/gpt-4o", os.getenv("OPENAI_API_KEY"))
+     await extract_structured_data_using_llm("ollama/llama3.2")   
+     ```
+
+#### **7. Complete Example of LLM Extraction Setup**
+   - Code to run both the OpenAI pricing and Knowledge Graph extractions, using various providers:
+     ```python
+     async def main():
+         await extract_openai_pricing()
+         await extract_knowledge_graph()
+     
+     if __name__ == "__main__":
+         asyncio.run(main())
+     ```
+
+#### **8. Wrap Up & Next Steps**
+   - Recap the power of LLM extraction for handling unstructured or complex data extraction tasks.
+   - Tease the next video: **10.3 Cosine Similarity Strategy** for clustering similar content based on semantic similarity.
+
+---
+
+This outline explains LLM Extraction in Crawl4AI, with examples showing how to extract structured data using custom schemas and instructions. It demonstrates flexibility with multiple providers, ensuring practical application for different use cases.

docs/md_v2/tutorial/episode_11_3_Extraction_Strategies_Cosine.md

@@ -0,0 +1,136 @@
+# Crawl4AI
+
+## Episode 11: Extraction Strategies: JSON CSS, LLM, and Cosine
+
+### Quick Intro
+Introduce JSON CSS Extraction Strategy for structured data, LLM Extraction Strategy for intelligent parsing, and Cosine Strategy for clustering similar content. Demo: Use JSON CSS to scrape product details from an e-commerce site.
+
+Here’s a structured outline for the **Cosine Similarity Strategy** video, covering key concepts, configuration, and a practical example.
+
+---
+
+### **10.3 Cosine Similarity Strategy**
+
+#### **1. Introduction to Cosine Similarity Strategy**
+   - The Cosine Similarity Strategy clusters content by semantic similarity, offering an efficient alternative to LLM-based extraction, especially when speed is a priority.
+   - Ideal for grouping similar sections of text, this strategy is well-suited for pages with content sections that may need to be classified or tagged, like news articles, product descriptions, or reviews.
+
+#### **2. Key Configuration Options**
+   - **semantic_filter**: A keyword-based filter to focus on relevant content.
+   - **word_count_threshold**: Minimum number of words per cluster, filtering out shorter, less meaningful clusters.
+   - **max_dist**: Maximum allowable distance between elements in clusters, impacting cluster tightness.
+   - **linkage_method**: Method for hierarchical clustering, such as `'ward'` (for well-separated clusters).
+   - **top_k**: Specifies the number of top categories for each cluster.
+   - **model_name**: Defines the model for embeddings, such as `sentence-transformers/all-MiniLM-L6-v2`.
+   - **sim_threshold**: Minimum similarity threshold for filtering, allowing control over cluster relevance.
+
+#### **3. How Cosine Similarity Clustering Works**
+   - **Step 1**: Embeddings are generated for each text section, transforming them into vectors that capture semantic meaning.
+   - **Step 2**: Hierarchical clustering groups similar sections based on cosine similarity, forming clusters with related content.
+   - **Step 3**: Clusters are filtered based on word count, removing those below the `word_count_threshold`.
+   - **Step 4**: Each cluster is then categorized with tags, if enabled, providing context to each grouped content section.
+
+#### **4. Example Use Case: Clustering Blog Article Sections**
+   - **Goal**: Group related sections of a blog or news page to identify distinct topics or discussion areas.
+   - **Example HTML Sections**:
+     ```text
+     "The economy is showing signs of recovery, with markets up this quarter.",
+     "In the sports world, several major teams are preparing for the upcoming season.",
+     "New advancements in AI technology are reshaping the tech landscape.",
+     "Market analysts are optimistic about continued growth in tech stocks."
+     ```
+
+   - **Code Setup**:
+     ```python
+     async def extract_blog_sections():
+         extraction_strategy = CosineStrategy(
+             word_count_threshold=15,
+             max_dist=0.3,
+             sim_threshold=0.2,
+             model_name="sentence-transformers/all-MiniLM-L6-v2",
+             top_k=2
+         )
+         async with AsyncWebCrawler() as crawler:
+             url = "https://example.com/blog-page"
+             result = await crawler.arun(
+                 url=url,
+                 extraction_strategy=extraction_strategy,
+                 cache_mode=CacheMode.BYPASS
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - **word_count_threshold**: Ensures only clusters with meaningful content are included.
+     - **sim_threshold**: Filters out clusters with low similarity, focusing on closely related sections.
+     - **top_k**: Selects top tags, useful for identifying main topics.
+
+#### **5. Applying Semantic Filtering with Cosine Similarity**
+   - **Semantic Filter**: Filters sections based on relevance to a specific keyword, such as “technology” for tech articles.
+   - **Example Code**:
+     ```python
+     extraction_strategy = CosineStrategy(
+         semantic_filter="technology",
+         word_count_threshold=10,
+         max_dist=0.25,
+         model_name="sentence-transformers/all-MiniLM-L6-v2"
+     )
+     ```
+   - **Explanation**:
+     - **semantic_filter**: Only sections with high similarity to the “technology” keyword will be included in the clustering, making it easy to focus on specific topics within a mixed-content page.
+
+#### **6. Clustering Product Reviews by Similarity**
+   - **Goal**: Organize product reviews by themes, such as “price,” “quality,” or “durability.”
+   - **Example Reviews**:
+     ```text
+     "The quality of this product is outstanding and well worth the price.",
+     "I found the product to be durable but a bit overpriced.",
+     "Great value for the money and long-lasting.",
+     "The build quality is good, but I expected a lower price point."
+     ```
+
+   - **Code Setup**:
+     ```python
+     async def extract_product_reviews():
+         extraction_strategy = CosineStrategy(
+             word_count_threshold=20,
+             max_dist=0.35,
+             sim_threshold=0.25,
+             model_name="sentence-transformers/all-MiniLM-L6-v2"
+         )
+         async with AsyncWebCrawler() as crawler:
+             url = "https://example.com/product-reviews"
+             result = await crawler.arun(
+                 url=url,
+                 extraction_strategy=extraction_strategy,
+                 cache_mode=CacheMode.BYPASS
+             )
+             print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - This configuration clusters similar reviews, grouping feedback by common themes, helping businesses understand customer sentiments around particular product aspects.
+
+#### **7. Performance Advantages of Cosine Strategy**
+   - **Speed**: The Cosine Similarity Strategy is faster than LLM-based extraction, as it doesn’t rely on API calls to external LLMs.
+   - **Local Processing**: The strategy runs locally with pre-trained sentence embeddings, ideal for high-throughput scenarios where cost and latency are concerns.
+   - **Comparison**: With a well-optimized local model, this method can perform clustering on large datasets quickly, making it suitable for tasks requiring rapid, repeated analysis.
+
+#### **8. Full Code Example for Clustering News Articles**
+   - **Code**:
+     ```python
+     async def main():
+         await extract_blog_sections()
+         await extract_product_reviews()
+     
+     if __name__ == "__main__":
+         asyncio.run(main())
+     ```
+
+#### **9. Wrap Up & Next Steps**
+   - Recap the efficiency and effectiveness of Cosine Similarity for clustering related content quickly.
+   - Close with a reminder of Crawl4AI’s flexibility across extraction strategies, and prompt users to experiment with different settings to optimize clustering for their specific content.
+
+---
+
+This outline covers Cosine Similarity Strategy’s speed and effectiveness, providing examples that showcase its potential for clustering various content types efficiently.

docs/md_v2/tutorial/episode_12_Session-Based_Crawling_for_Dynamic_Websites.md

@@ -0,0 +1,140 @@
+# Crawl4AI
+
+## Episode 12: Session-Based Crawling for Dynamic Websites
+
+### Quick Intro
+Show session management for handling websites with multiple pages or actions (like “load more” buttons). Demo: Crawl a paginated content page, persisting session data across multiple requests.
+
+Here’s a detailed outline for the **Session-Based Crawling for Dynamic Websites** video, explaining why sessions are necessary, how to use them, and providing practical examples and a visual diagram to illustrate the concept.
+
+---
+
+### **11. Session-Based Crawling for Dynamic Websites**
+
+#### **1. Introduction to Session-Based Crawling**
+   - **What is Session-Based Crawling**: Session-based crawling maintains a continuous browsing session across multiple page states, allowing the crawler to interact with a page and retrieve content that loads dynamically or based on user interactions.
+   - **Why It’s Needed**:
+     - In static pages, all content is available directly from a single URL.
+     - In dynamic websites, content often loads progressively or based on user actions (e.g., clicking “load more,” submitting forms, scrolling).
+     - Session-based crawling helps simulate user actions, capturing content that is otherwise hidden until specific actions are taken.
+
+#### **2. Conceptual Diagram for Session-Based Crawling**
+
+   ```mermaid
+   graph TD
+       Start[Start Session] --> S1[Initial State (S1)]
+       S1 -->|Crawl| Content1[Extract Content S1]
+       S1 -->|Action: Click Load More| S2[State S2]
+       S2 -->|Crawl| Content2[Extract Content S2]
+       S2 -->|Action: Scroll Down| S3[State S3]
+       S3 -->|Crawl| Content3[Extract Content S3]
+       S3 -->|Action: Submit Form| S4[Final State]
+       S4 -->|Crawl| Content4[Extract Content S4]
+       Content4 --> End[End Session]
+   ```
+
+   - **Explanation of Diagram**:
+     - **Start**: Initializes the session and opens the starting URL.
+     - **State Transitions**: Each action (e.g., clicking “load more,” scrolling) transitions to a new state, where additional content becomes available.
+     - **Session Persistence**: Keeps the same browsing session active, preserving the state and allowing for a sequence of actions to unfold.
+     - **End**: After reaching the final state, the session ends, and all accumulated content has been extracted.
+
+#### **3. Key Components of Session-Based Crawling in Crawl4AI**
+   - **Session ID**: A unique identifier to maintain the state across requests, allowing the crawler to “remember” previous actions.
+   - **JavaScript Execution**: Executes JavaScript commands (e.g., clicks, scrolls) to simulate interactions.
+   - **Wait Conditions**: Ensures the crawler waits for content to load in each state before moving on.
+   - **Sequential State Transitions**: By defining actions and wait conditions between states, the crawler can navigate through the page as a user would.
+
+#### **4. Basic Session Example: Multi-Step Content Loading**
+   - **Goal**: Crawl an article feed that requires several “load more” clicks to display additional content.
+   - **Code**:
+     ```python
+     async def crawl_article_feed():
+         async with AsyncWebCrawler() as crawler:
+             session_id = "feed_session"
+             
+             for page in range(3):
+                 result = await crawler.arun(
+                     url="https://example.com/articles",
+                     session_id=session_id,
+                     js_code="document.querySelector('.load-more-button').click();" if page > 0 else None,
+                     wait_for="css:.article",
+                     css_selector=".article"  # Target article elements
+                 )
+                 print(f"Page {page + 1}: Extracted {len(result.extracted_content)} articles")
+     ```
+   - **Explanation**:
+     - **session_id**: Ensures all requests share the same browsing state.
+     - **js_code**: Clicks the “load more” button after the initial page load, expanding content on each iteration.
+     - **wait_for**: Ensures articles have loaded after each click before extraction.
+
+#### **5. Advanced Example: E-Commerce Product Search with Filter Selection**
+   - **Goal**: Interact with filters on an e-commerce page to extract products based on selected criteria.
+   - **Example Steps**:
+     1. **State 1**: Load the main product page.
+     2. **State 2**: Apply a filter (e.g., “On Sale”) by selecting a checkbox.
+     3. **State 3**: Scroll to load additional products and capture updated results.
+
+   - **Code**:
+     ```python
+     async def extract_filtered_products():
+         async with AsyncWebCrawler() as crawler:
+             session_id = "product_session"
+             
+             # Step 1: Open product page
+             result = await crawler.arun(
+                 url="https://example.com/products",
+                 session_id=session_id,
+                 wait_for="css:.product-item"
+             )
+             
+             # Step 2: Apply filter (e.g., "On Sale")
+             result = await crawler.arun(
+                 url="https://example.com/products",
+                 session_id=session_id,
+                 js_code="document.querySelector('#sale-filter-checkbox').click();",
+                 wait_for="css:.product-item"
+             )
+
+             # Step 3: Scroll to load additional products
+             for _ in range(2):  # Scroll down twice
+                 result = await crawler.arun(
+                     url="https://example.com/products",
+                     session_id=session_id,
+                     js_code="window.scrollTo(0, document.body.scrollHeight);",
+                     wait_for="css:.product-item"
+                 )
+                 print(f"Loaded {len(result.extracted_content)} products after scroll")
+     ```
+   - **Explanation**:
+     - **State Persistence**: Each action (filter selection and scroll) builds on the previous session state.
+     - **Multiple Interactions**: Combines clicking a filter with scrolling, demonstrating how the session preserves these actions.
+
+#### **6. Key Benefits of Session-Based Crawling**
+   - **Accessing Hidden Content**: Retrieves data that loads only after user actions.
+   - **Simulating User Behavior**: Handles interactive elements such as “load more” buttons, dropdowns, and filters.
+   - **Maintaining Continuity Across States**: Enables a sequential process, moving logically from one state to the next, capturing all desired content without reloading the initial state each time.
+
+#### **7. Additional Configuration Tips**
+   - **Manage Session End**: Always conclude the session after the final state to release resources.
+   - **Optimize with Wait Conditions**: Use `wait_for` to ensure complete loading before each extraction.
+   - **Handling Errors in Session-Based Crawling**: Include error handling for interactions that may fail, ensuring robustness across state transitions.
+
+#### **8. Complete Code Example: Multi-Step Session Workflow**
+   - **Example**:
+     ```python
+     async def main():
+         await crawl_article_feed()
+         await extract_filtered_products()
+     
+     if __name__ == "__main__":
+         asyncio.run(main())
+     ```
+
+#### **9. Wrap Up & Next Steps**
+   - Recap the usefulness of session-based crawling for dynamic content extraction.
+   - Tease the next video: **Hooks and Custom Workflow with AsyncWebCrawler** to cover advanced customization options for further control over the crawling process.
+
+---
+
+This outline covers session-based crawling from both a conceptual and practical perspective, helping users understand its importance, configure it effectively, and use it to handle complex dynamic content.

docs/md_v2/tutorial/episode_13_Chunking_Strategies_for_Large_Text_Processing.md

@@ -0,0 +1,138 @@
+# Crawl4AI
+
+## Episode 13: Chunking Strategies for Large Text Processing
+
+### Quick Intro
+Explain Regex, NLP, and Fixed-Length chunking, and when to use each. Demo: Chunk a large article or document for processing by topics or sentences.
+
+Here’s a structured outline for the **Chunking Strategies for Large Text Processing** video, emphasizing how chunking works within extraction and why it’s crucial for effective data aggregation.
+
+Here’s a structured outline for the **Chunking Strategies for Large Text Processing** video, explaining each strategy, when to use it, and providing examples to illustrate.
+
+---
+
+### **12. Chunking Strategies for Large Text Processing**
+
+#### **1. Introduction to Chunking in Crawl4AI**
+   - **What is Chunking**: Chunking is the process of dividing large text into manageable sections or “chunks,” enabling efficient processing in extraction tasks.
+   - **Why It’s Needed**:
+     - When processing large text, feeding it directly into an extraction function (like `F(x)`) can overwhelm memory or token limits.
+     - Chunking breaks down `x` (the text) into smaller pieces, which are processed sequentially or in parallel by the extraction function, with the final result being an aggregation of all chunks’ processed output.
+
+#### **2. Key Chunking Strategies and Use Cases**
+   - Crawl4AI offers various chunking strategies to suit different text structures, chunk sizes, and processing requirements.
+   - **Choosing a Strategy**: Select based on the type of text (e.g., articles, transcripts) and extraction needs (e.g., simple splitting or context-sensitive processing).
+
+#### **3. Strategy 1: Regex-Based Chunking**
+   - **Description**: Uses regular expressions to split text based on specified patterns (e.g., paragraphs or section breaks).
+   - **Use Case**: Ideal for dividing text by paragraphs or larger logical blocks where sections are clearly separated by line breaks or punctuation.
+   - **Example**:
+     - **Pattern**: `r'\n\n'` for double line breaks.
+     ```python
+     chunker = RegexChunking(patterns=[r'\n\n'])
+     text_chunks = chunker.chunk(long_text)
+     print(text_chunks)  # Output: List of paragraphs
+     ```
+   - **Pros**: Flexible for pattern-based chunking.
+   - **Cons**: Limited to text with consistent formatting.
+
+#### **4. Strategy 2: NLP Sentence-Based Chunking**
+   - **Description**: Uses NLP to split text by sentences, ensuring grammatically complete segments.
+   - **Use Case**: Useful for extracting individual statements, such as in news articles, quotes, or legal text.
+   - **Example**:
+     ```python
+     chunker = NlpSentenceChunking()
+     sentence_chunks = chunker.chunk(long_text)
+     print(sentence_chunks)  # Output: List of sentences
+     ```
+   - **Pros**: Maintains sentence structure, ideal for tasks needing semantic completeness.
+   - **Cons**: May create very small chunks, which could limit contextual extraction.
+
+#### **5. Strategy 3: Topic-Based Segmentation Using TextTiling**
+   - **Description**: Segments text into topics using TextTiling, identifying topic shifts and key segments.
+   - **Use Case**: Ideal for long articles, reports, or essays where each section covers a different topic.
+   - **Example**:
+     ```python
+     chunker = TopicSegmentationChunking(num_keywords=3)
+     topic_chunks = chunker.chunk_with_topics(long_text)
+     print(topic_chunks)  # Output: List of topic segments with keywords
+     ```
+   - **Pros**: Groups related content, preserving topical coherence.
+   - **Cons**: Depends on identifiable topic shifts, which may not be present in all texts.
+
+#### **6. Strategy 4: Fixed-Length Word Chunking**
+   - **Description**: Splits text into chunks based on a fixed number of words.
+   - **Use Case**: Ideal for text where exact segment size is required, such as processing word-limited documents for LLMs.
+   - **Example**:
+     ```python
+     chunker = FixedLengthWordChunking(chunk_size=100)
+     word_chunks = chunker.chunk(long_text)
+     print(word_chunks)  # Output: List of 100-word chunks
+     ```
+   - **Pros**: Ensures uniform chunk sizes, suitable for token-based extraction limits.
+   - **Cons**: May split sentences, affecting semantic coherence.
+
+#### **7. Strategy 5: Sliding Window Chunking**
+   - **Description**: Uses a fixed window size with a step, creating overlapping chunks to maintain context.
+   - **Use Case**: Useful for maintaining context across sections, as with documents where context is needed for neighboring sections.
+   - **Example**:
+     ```python
+     chunker = SlidingWindowChunking(window_size=100, step=50)
+     window_chunks = chunker.chunk(long_text)
+     print(window_chunks)  # Output: List of overlapping word chunks
+     ```
+   - **Pros**: Retains context across adjacent chunks, ideal for complex semantic extraction.
+   - **Cons**: Overlap increases data size, potentially impacting processing time.
+
+#### **8. Strategy 6: Overlapping Window Chunking**
+   - **Description**: Similar to sliding windows but with a defined overlap, allowing chunks to share content at the edges.
+   - **Use Case**: Suitable for handling long texts with essential overlapping information, like research articles or medical records.
+   - **Example**:
+     ```python
+     chunker = OverlappingWindowChunking(window_size=1000, overlap=100)
+     overlap_chunks = chunker.chunk(long_text)
+     print(overlap_chunks)  # Output: List of overlapping chunks with defined overlap
+     ```
+   - **Pros**: Allows controlled overlap for consistent content coverage across chunks.
+   - **Cons**: Redundant data in overlapping areas may increase computation.
+
+#### **9. Practical Example: Using Chunking with an Extraction Strategy**
+   - **Goal**: Combine chunking with an extraction strategy to process large text effectively.
+   - **Example Code**:
+     ```python
+     from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+     async def extract_large_text():
+         # Initialize chunker and extraction strategy
+         chunker = FixedLengthWordChunking(chunk_size=200)
+         extraction_strategy = LLMExtractionStrategy(provider="openai/gpt-4", api_token="your_api_token")
+         
+         # Split text into chunks
+         text_chunks = chunker.chunk(large_text)
+         
+         async with AsyncWebCrawler() as crawler:
+             for chunk in text_chunks:
+                 result = await crawler.arun(
+                     url="https://example.com",
+                     extraction_strategy=extraction_strategy,
+                     content=chunk
+                 )
+                 print(result.extracted_content)
+     ```
+
+   - **Explanation**:
+     - `chunker.chunk()`: Divides the `large_text` into smaller segments based on the chosen strategy.
+     - `extraction_strategy`: Processes each chunk separately, and results are then aggregated to form the final output.
+
+#### **10. Choosing the Right Chunking Strategy**
+   - **Text Structure**: If text has clear sections (e.g., paragraphs, topics), use Regex or Topic Segmentation.
+   - **Extraction Needs**: If context is crucial, consider Sliding or Overlapping Window Chunking.
+   - **Processing Constraints**: For word-limited extractions (e.g., LLMs with token limits), Fixed-Length Word Chunking is often most effective.
+
+#### **11. Wrap Up & Next Steps**
+   - Recap the benefits of each chunking strategy and when to use them in extraction workflows.
+   - Tease the next video: **Hooks and Custom Workflow with AsyncWebCrawler**, focusing on customizing crawler behavior with hooks for a fine-tuned extraction process.
+
+---
+
+This outline provides a complete understanding of chunking strategies, explaining each method’s strengths and best-use scenarios to help users process large texts effectively in Crawl4AI.

docs/md_v2/tutorial/episode_14_Hooks_and_Custom_Workflow_with_AsyncWebCrawler.md

@@ -0,0 +1,185 @@
+# Crawl4AI
+
+## Episode 14: Hooks and Custom Workflow with AsyncWebCrawler
+
+### Quick Intro
+Cover hooks (`on_browser_created`, `before_goto`, `after_goto`) to add custom workflows. Demo: Use hooks to add custom cookies or headers, log HTML, or trigger specific events on page load.
+
+Here’s a detailed outline for the **Hooks and Custom Workflow with AsyncWebCrawler** video, covering each hook’s purpose, usage, and example implementations.
+
+---
+
+### **13. Hooks and Custom Workflow with AsyncWebCrawler**
+
+#### **1. Introduction to Hooks in Crawl4AI**
+   - **What are Hooks**: Hooks are customizable entry points in the crawling process that allow users to inject custom actions or logic at specific stages.
+   - **Why Use Hooks**:
+     - They enable fine-grained control over the crawling workflow.
+     - Useful for performing additional tasks (e.g., logging, modifying headers) dynamically during the crawl.
+     - Hooks provide the flexibility to adapt the crawler to complex site structures or unique project needs.
+
+#### **2. Overview of Available Hooks**
+   - Crawl4AI offers seven key hooks to modify and control different stages in the crawling lifecycle:
+     - `on_browser_created`
+     - `on_user_agent_updated`
+     - `on_execution_started`
+     - `before_goto`
+     - `after_goto`
+     - `before_return_html`
+     - `before_retrieve_html`
+
+#### **3. Hook-by-Hook Explanation and Examples**
+
+---
+
+##### **Hook 1: `on_browser_created`**
+   - **Purpose**: Triggered right after the browser instance is created.
+   - **Use Case**:
+     - Initializing browser-specific settings or performing setup actions.
+     - Configuring browser extensions or scripts before any page is opened.
+   - **Example**:
+     ```python
+     async def log_browser_creation(browser):
+         print("Browser instance created:", browser)
+     
+     crawler.crawler_strategy.set_hook('on_browser_created', log_browser_creation)
+     ```
+   - **Explanation**: This hook logs the browser creation event, useful for tracking when a new browser instance starts.
+
+---
+
+##### **Hook 2: `on_user_agent_updated`**
+   - **Purpose**: Called whenever the user agent string is updated.
+   - **Use Case**:
+     - Modifying the user agent based on page requirements, e.g., changing to a mobile user agent for mobile-only pages.
+   - **Example**:
+     ```python
+     def update_user_agent(user_agent):
+         print(f"User Agent Updated: {user_agent}")
+     
+     crawler.crawler_strategy.set_hook('on_user_agent_updated', update_user_agent)
+     crawler.update_user_agent("Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X)")
+     ```
+   - **Explanation**: This hook provides a callback every time the user agent changes, helpful for debugging or dynamically altering user agent settings based on conditions.
+
+---
+
+##### **Hook 3: `on_execution_started`**
+   - **Purpose**: Called right before the crawler begins any interaction (e.g., JavaScript execution, clicks).
+   - **Use Case**:
+     - Performing setup actions, such as inserting cookies or initiating custom scripts.
+   - **Example**:
+     ```python
+     async def log_execution_start(page):
+         print("Execution started on page:", page.url)
+     
+     crawler.crawler_strategy.set_hook('on_execution_started', log_execution_start)
+     ```
+   - **Explanation**: Logs the start of any major interaction on the page, ideal for cases where you want to monitor each interaction.
+
+---
+
+##### **Hook 4: `before_goto`**
+   - **Purpose**: Triggered before navigating to a new URL with `page.goto()`.
+   - **Use Case**:
+     - Modifying request headers or setting up conditions right before the page loads.
+     - Adding headers or dynamically adjusting options for specific URLs.
+   - **Example**:
+     ```python
+     async def modify_headers_before_goto(page):
+         await page.set_extra_http_headers({"X-Custom-Header": "CustomValue"})
+         print("Custom headers set before navigation")
+     
+     crawler.crawler_strategy.set_hook('before_goto', modify_headers_before_goto)
+     ```
+   - **Explanation**: This hook allows injecting headers or altering settings based on the page’s needs, particularly useful for pages with custom requirements.
+
+---
+
+##### **Hook 5: `after_goto`**
+   - **Purpose**: Executed immediately after a page has loaded (after `page.goto()`).
+   - **Use Case**:
+     - Checking the loaded page state, modifying the DOM, or performing post-navigation actions (e.g., scrolling).
+   - **Example**:
+     ```python
+     async def post_navigation_scroll(page):
+         await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
+         print("Scrolled to the bottom after navigation")
+     
+     crawler.crawler_strategy.set_hook('after_goto', post_navigation_scroll)
+     ```
+   - **Explanation**: This hook scrolls to the bottom of the page after loading, which can help load dynamically added content like infinite scroll elements.
+
+---
+
+##### **Hook 6: `before_return_html`**
+   - **Purpose**: Called right before HTML content is retrieved and returned.
+   - **Use Case**:
+     - Removing overlays or cleaning up the page for a cleaner HTML extraction.
+   - **Example**:
+     ```python
+     async def remove_advertisements(page, html):
+         await page.evaluate("document.querySelectorAll('.ad-banner').forEach(el => el.remove());")
+         print("Advertisements removed before returning HTML")
+     
+     crawler.crawler_strategy.set_hook('before_return_html', remove_advertisements)
+     ```
+   - **Explanation**: The hook removes ad banners from the HTML before it’s retrieved, ensuring a cleaner data extraction.
+
+---
+
+##### **Hook 7: `before_retrieve_html`**
+   - **Purpose**: Runs right before Crawl4AI initiates HTML retrieval.
+   - **Use Case**:
+     - Finalizing any page adjustments (e.g., setting timers, waiting for specific elements).
+   - **Example**:
+     ```python
+     async def wait_for_content_before_retrieve(page):
+         await page.wait_for_selector('.main-content')
+         print("Main content loaded, ready to retrieve HTML")
+     
+     crawler.crawler_strategy.set_hook('before_retrieve_html', wait_for_content_before_retrieve)
+     ```
+   - **Explanation**: This hook waits for the main content to load before retrieving the HTML, ensuring that all essential content is captured.
+
+#### **4. Setting Hooks in Crawl4AI**
+   - **How to Set Hooks**:
+     - Use `set_hook` to define a custom function for each hook.
+     - Each hook function can be asynchronous (useful for actions like waiting or retrieving async data).
+   - **Example Setup**:
+     ```python
+     crawler.crawler_strategy.set_hook('on_browser_created', log_browser_creation)
+     crawler.crawler_strategy.set_hook('before_goto', modify_headers_before_goto)
+     crawler.crawler_strategy.set_hook('after_goto', post_navigation_scroll)
+     ```
+
+#### **5. Complete Example: Using Hooks for a Customized Crawl Workflow**
+   - **Goal**: Log each key step, set custom headers before navigation, and clean up the page before retrieving HTML.
+   - **Example Code**:
+     ```python
+     async def custom_crawl():
+         async with AsyncWebCrawler() as crawler:
+             # Set hooks for custom workflow
+             crawler.crawler_strategy.set_hook('on_browser_created', log_browser_creation)
+             crawler.crawler_strategy.set_hook('before_goto', modify_headers_before_goto)
+             crawler.crawler_strategy.set_hook('after_goto', post_navigation_scroll)
+             crawler.crawler_strategy.set_hook('before_return_html', remove_advertisements)
+             
+             # Perform the crawl
+             url = "https://example.com"
+             result = await crawler.arun(url=url)
+             print(result.html)  # Display or process HTML
+     ```
+
+#### **6. Benefits of Using Hooks in Custom Crawling Workflows**
+   - **Enhanced Control**: Hooks offer precise control over each stage, allowing adjustments based on content and structure.
+   - **Efficient Modifications**: Avoid reloading or restarting the session; hooks can alter actions dynamically.
+   - **Context-Sensitive Actions**: Hooks enable custom logic tailored to specific pages or sections, maximizing extraction quality.
+
+#### **7. Wrap Up & Next Steps**
+   - Recap how hooks empower customized workflows in Crawl4AI, enabling flexibility at every stage.
+   - Tease the next video: **Automating Post-Processing with Crawl4AI**, covering automated steps after data extraction.
+
+---
+
+This outline provides a thorough understanding of hooks, their practical applications, and examples for customizing the crawling workflow in Crawl4AI.

docs/notebooks/Crawl4AI_v0.3.72_Release_Announcement.ipynb

@@ -0,0 +1,235 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 🚀 Crawl4AI v0.3.72 Release Announcement\n",
+    "\n",
+    "Welcome to the new release of **Crawl4AI v0.3.72**! This notebook highlights the latest features and demonstrates how they work in real-time. Follow along to see each feature in action!\n",
+    "\n",
+    "### What’s New?\n",
+    "- ✨ `Fit Markdown`: Extracts only the main content from articles and blogs\n",
+    "- 🛡️ **Magic Mode**: Comprehensive anti-bot detection bypass\n",
+    "- 🌐 **Multi-browser support**: Switch between Chromium, Firefox, WebKit\n",
+    "- 🔍 **Knowledge Graph Extraction**: Generate structured graphs of entities & relationships from any URL\n",
+    "- 🤖 **Crawl4AI GPT Assistant**: Chat directly with our AI assistant for help, code generation, and faster learning (available [here](https://tinyurl.com/your-gpt-assistant-link))\n",
+    "\n",
+    "---\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 📥 Setup\n",
+    "To start, we'll install `Crawl4AI` along with Playwright and `nest_asyncio` to ensure compatibility with Colab’s asynchronous environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install Crawl4AI and dependencies\n",
+    "!pip install crawl4ai\n",
+    "!playwright install\n",
+    "!pip install nest_asyncio"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import nest_asyncio and apply it to allow asyncio in Colab\n",
+    "import nest_asyncio\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "print('Setup complete!')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## ✨ Feature 1: `Fit Markdown`\n",
+    "Extracts only the main content from articles and blog pages, removing sidebars, ads, and other distractions.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "from crawl4ai import AsyncWebCrawler\n",
+    "\n",
+    "async def fit_markdown_demo():\n",
+    "    async with AsyncWebCrawler() as crawler:\n",
+    "        result = await crawler.arun(url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\")\n",
+    "        print(result.fit_markdown)  # Shows main content in Markdown format\n",
+    "\n",
+    "# Run the demo\n",
+    "await fit_markdown_demo()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 🛡️ Feature 2: Magic Mode\n",
+    "Magic Mode bypasses anti-bot detection to make crawling more reliable on protected websites.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def magic_mode_demo():\n",
+    "    async with AsyncWebCrawler() as crawler:  # Enables anti-bot detection bypass\n",
+    "        result = await crawler.arun(\n",
+    "            url=\"https://www.reuters.com/markets/us/global-markets-view-usa-pix-2024-08-29/\",\n",
+    "            magic=True  # Enables magic mode\n",
+    "        )\n",
+    "        print(result.markdown)  # Shows the full content in Markdown format\n",
+    "\n",
+    "# Run the demo\n",
+    "await magic_mode_demo()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 🌐 Feature 3: Multi-Browser Support\n",
+    "Crawl4AI now supports Chromium, Firefox, and WebKit. Here’s how to specify Firefox for a crawl.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def multi_browser_demo():\n",
+    "    async with AsyncWebCrawler(browser_type=\"firefox\") as crawler:  # Using Firefox instead of default Chromium\n",
+    "        result = await crawler.arun(url=\"https://crawl4i.com\")\n",
+    "        print(result.markdown)  # Shows content extracted using Firefox\n",
+    "\n",
+    "# Run the demo\n",
+    "await multi_browser_demo()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## ✨ Put them all together\n",
+    "\n",
+    "Let's combine all the features to extract the main content from a blog post, bypass anti-bot detection, and generate a knowledge graph from the extracted content."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from crawl4ai.extraction_strategy import LLMExtractionStrategy\n",
+    "from pydantic import BaseModel\n",
+    "import json, os\n",
+    "from typing import List\n",
+    "\n",
+    "# Define classes for the knowledge graph structure\n",
+    "class Landmark(BaseModel):\n",
+    "    name: str\n",
+    "    description: str\n",
+    "    activities: list[str]  # E.g., visiting, sightseeing, relaxing\n",
+    "\n",
+    "class City(BaseModel):\n",
+    "    name: str\n",
+    "    description: str\n",
+    "    landmarks: list[Landmark]\n",
+    "    cultural_highlights: list[str]  # E.g., food, music, traditional crafts\n",
+    "\n",
+    "class TravelKnowledgeGraph(BaseModel):\n",
+    "    cities: list[City]  # Central Mexican cities to visit\n",
+    "\n",
+    "async def combined_demo():\n",
+    "    # Define the knowledge graph extraction strategy\n",
+    "    strategy = LLMExtractionStrategy(\n",
+    "        # provider=\"ollama/nemotron\",\n",
+    "        provider='openai/gpt-4o-mini', # Or any other provider, including Ollama and open source models\n",
+    "        pi_token=os.getenv('OPENAI_API_KEY'), # In case of Ollama just pass \"no-token\"\n",
+    "        schema=TravelKnowledgeGraph.schema(),\n",
+    "        instruction=(\n",
+    "            \"Extract cities, landmarks, and cultural highlights for places to visit in Central Mexico. \"\n",
+    "            \"For each city, list main landmarks with descriptions and activities, as well as cultural highlights.\"\n",
+    "        )\n",
+    "    )\n",
+    "\n",
+    "    # Set up the AsyncWebCrawler with multi-browser support, Magic Mode, and Fit Markdown\n",
+    "    async with AsyncWebCrawler(browser_type=\"firefox\") as crawler:\n",
+    "        result = await crawler.arun(\n",
+    "            url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\",\n",
+    "            extraction_strategy=strategy,\n",
+    "            bypass_cache=True,\n",
+    "            magic=True\n",
+    "        )\n",
+    "        \n",
+    "        # Display main article content in Fit Markdown format\n",
+    "        print(\"Extracted Main Content:\\n\", result.fit_markdown)\n",
+    "        \n",
+    "        # Display extracted knowledge graph of cities, landmarks, and cultural highlights\n",
+    "        if result.extracted_content:\n",
+    "            travel_graph = json.loads(result.extracted_content)\n",
+    "            print(\"\\nExtracted Knowledge Graph:\\n\", json.dumps(travel_graph, indent=2))\n",
+    "\n",
+    "# Run the combined demo\n",
+    "await combined_demo()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 🤖 Crawl4AI GPT Assistant\n",
+    "Chat with the Crawl4AI GPT Assistant for code generation, support, and learning Crawl4AI faster. Try it out [here](https://tinyurl.com/crawl4ai-gpt)!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

main.py

@@ -1,10 +1,6 @@
-import os
-import importlib
-import asyncio
-from functools import lru_cache
-import logging
-logging.basicConfig(level=logging.DEBUG)
-
+import asyncio, os
+from fastapi import FastAPI, HTTPException, BackgroundTasks, Request
+from fastapi.responses import JSONResponse
 from fastapi import FastAPI, HTTPException, Request
 from fastapi.responses import HTMLResponse, JSONResponse
 from fastapi.staticfiles import StaticFiles
@@ -14,102 +10,325 @@ from fastapi.exceptions import RequestValidationError
 from starlette.middleware.base import BaseHTTPMiddleware
 from starlette.responses import FileResponse
 from fastapi.responses import RedirectResponse
+from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
+from fastapi import Depends, Security
 
-from pydantic import BaseModel, HttpUrl
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import List, Optional
-
-from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.database import get_total_count, clear_db
-
+from pydantic import BaseModel, HttpUrl, Field
+from typing import Optional, List, Dict, Any, Union
+import psutil
 import time
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
+import uuid
+from collections import defaultdict
+from urllib.parse import urlparse
+import math
+import logging
+from enum import Enum
+from dataclasses import dataclass
+import json
+from crawl4ai import AsyncWebCrawler, CrawlResult, CacheMode
+from crawl4ai.config import MIN_WORD_THRESHOLD
+from crawl4ai.extraction_strategy import (
+    LLMExtractionStrategy,
+    CosineStrategy,
+    JsonCssExtractionStrategy,
+)
 
-# load .env file
-from dotenv import load_dotenv
-load_dotenv()
-
-# Configuration
 __location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
-MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
-current_requests = 0
-lock = asyncio.Lock()
 
-app = FastAPI()
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
 
-# Initialize rate limiter
-def rate_limit_key_func(request: Request):
-    access_token = request.headers.get("access-token")
-    if access_token == os.environ.get('ACCESS_TOKEN'):
-        return None
-    return get_remote_address(request)
+class TaskStatus(str, Enum):
+    PENDING = "pending"
+    PROCESSING = "processing"
+    COMPLETED = "completed"
+    FAILED = "failed"
 
-limiter = Limiter(key_func=rate_limit_key_func)
-app.state.limiter = limiter
+class CrawlerType(str, Enum):
+    BASIC = "basic"
+    LLM = "llm"
+    COSINE = "cosine"
+    JSON_CSS = "json_css"
 
-# Dictionary to store last request times for each client
-last_request_times = {}
-last_rate_limit = {}
+class ExtractionConfig(BaseModel):
+    type: CrawlerType
+    params: Dict[str, Any] = {}
 
+class ChunkingStrategy(BaseModel):
+    type: str
+    params: Dict[str, Any] = {}
 
-def get_rate_limit():
-    limit = os.environ.get('ACCESS_PER_MIN', "5")
-    return f"{limit}/minute"
+class ContentFilter(BaseModel):
+    type: str = "bm25"
+    params: Dict[str, Any] = {}
 
-# Custom rate limit exceeded handler
-async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
-    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
-        last_rate_limit[request.client.host] = time.time()
-    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
-    reset_at = time.time() + retry_after
-    return JSONResponse(
-        status_code=429,
-        content={
-            "detail": "Rate limit exceeded",
-            "limit": str(exc.limit.limit),
-            "retry_after": retry_after,
-            'reset_at': reset_at,
-            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
-        }
-    )
-    
-app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
+class CrawlRequest(BaseModel):
+    urls: Union[HttpUrl, List[HttpUrl]]
+    word_count_threshold: int = MIN_WORD_THRESHOLD
+    extraction_config: Optional[ExtractionConfig] = None
+    chunking_strategy: Optional[ChunkingStrategy] = None
+    content_filter: Optional[ContentFilter] = None
+    js_code: Optional[List[str]] = None
+    wait_for: Optional[str] = None
+    css_selector: Optional[str] = None
+    screenshot: bool = False
+    magic: bool = False
+    extra: Optional[Dict[str, Any]] = {}
+    session_id: Optional[str] = None
+    cache_mode: Optional[CacheMode] = CacheMode.ENABLED
+    priority: int = Field(default=5, ge=1, le=10)
+    ttl: Optional[int] = 3600    
+    crawler_params: Dict[str, Any] = {}
 
+@dataclass
+class TaskInfo:
+    id: str
+    status: TaskStatus
+    result: Optional[Union[CrawlResult, List[CrawlResult]]] = None
+    error: Optional[str] = None
+    created_at: float = time.time()
+    ttl: int = 3600
 
-# Middleware for token-based bypass and per-request limit
-class RateLimitMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
-        access_token = request.headers.get("access-token")
-        if access_token == os.environ.get('ACCESS_TOKEN'):
-            return await call_next(request)
-        
-        path = request.url.path
-        if path in ["/crawl", "/old"]:
-            client_ip = request.client.host
+class ResourceMonitor:
+    def __init__(self, max_concurrent_tasks: int = 10):
+        self.max_concurrent_tasks = max_concurrent_tasks
+        self.memory_threshold = 0.85
+        self.cpu_threshold = 0.90
+        self._last_check = 0
+        self._check_interval = 1  # seconds
+        self._last_available_slots = max_concurrent_tasks
+
+    async def get_available_slots(self) -> int:
+        current_time = time.time()
+        if current_time - self._last_check < self._check_interval:
+            return self._last_available_slots
+
+        mem_usage = psutil.virtual_memory().percent / 100
+        cpu_usage = psutil.cpu_percent() / 100
+
+        memory_factor = max(0, (self.memory_threshold - mem_usage) / self.memory_threshold)
+        cpu_factor = max(0, (self.cpu_threshold - cpu_usage) / self.cpu_threshold)
+
+        self._last_available_slots = math.floor(
+            self.max_concurrent_tasks * min(memory_factor, cpu_factor)
+        )
+        self._last_check = current_time
+
+        return self._last_available_slots
+
+class TaskManager:
+    def __init__(self, cleanup_interval: int = 300):
+        self.tasks: Dict[str, TaskInfo] = {}
+        self.high_priority = asyncio.PriorityQueue()
+        self.low_priority = asyncio.PriorityQueue()
+        self.cleanup_interval = cleanup_interval
+        self.cleanup_task = None
+
+    async def start(self):
+        self.cleanup_task = asyncio.create_task(self._cleanup_loop())
+
+    async def stop(self):
+        if self.cleanup_task:
+            self.cleanup_task.cancel()
+            try:
+                await self.cleanup_task
+            except asyncio.CancelledError:
+                pass
+
+    async def add_task(self, task_id: str, priority: int, ttl: int) -> None:
+        task_info = TaskInfo(id=task_id, status=TaskStatus.PENDING, ttl=ttl)
+        self.tasks[task_id] = task_info
+        queue = self.high_priority if priority > 5 else self.low_priority
+        await queue.put((-priority, task_id))  # Negative for proper priority ordering
+
+    async def get_next_task(self) -> Optional[str]:
+        try:
+            # Try high priority first
+            _, task_id = await asyncio.wait_for(self.high_priority.get(), timeout=0.1)
+            return task_id
+        except asyncio.TimeoutError:
+            try:
+                # Then try low priority
+                _, task_id = await asyncio.wait_for(self.low_priority.get(), timeout=0.1)
+                return task_id
+            except asyncio.TimeoutError:
+                return None
+
+    def update_task(self, task_id: str, status: TaskStatus, result: Any = None, error: str = None):
+        if task_id in self.tasks:
+            task_info = self.tasks[task_id]
+            task_info.status = status
+            task_info.result = result
+            task_info.error = error
+
+    def get_task(self, task_id: str) -> Optional[TaskInfo]:
+        return self.tasks.get(task_id)
+
+    async def _cleanup_loop(self):
+        while True:
+            try:
+                await asyncio.sleep(self.cleanup_interval)
+                current_time = time.time()
+                expired_tasks = [
+                    task_id
+                    for task_id, task in self.tasks.items()
+                    if current_time - task.created_at > task.ttl
+                    and task.status in [TaskStatus.COMPLETED, TaskStatus.FAILED]
+                ]
+                for task_id in expired_tasks:
+                    del self.tasks[task_id]
+            except Exception as e:
+                logger.error(f"Error in cleanup loop: {e}")
+
+class CrawlerPool:
+    def __init__(self, max_size: int = 10):
+        self.max_size = max_size
+        self.active_crawlers: Dict[AsyncWebCrawler, float] = {}
+        self._lock = asyncio.Lock()
+
+    async def acquire(self, **kwargs) -> AsyncWebCrawler:
+        async with self._lock:
+            # Clean up inactive crawlers
             current_time = time.time()
-            
-            # Check time since last request
-            if client_ip in last_request_times:
-                time_since_last_request = current_time - last_request_times[client_ip]
-                if time_since_last_request < SPAN:
-                    return JSONResponse(
-                        status_code=429,
-                        content={
-                            "detail": "Too many requests",
-                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
-                            "retry_after": max(0, SPAN - time_since_last_request),
-                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
-                        }
-                    )
-            
-            last_request_times[client_ip] = current_time
+            inactive = [
+                crawler
+                for crawler, last_used in self.active_crawlers.items()
+                if current_time - last_used > 600  # 10 minutes timeout
+            ]
+            for crawler in inactive:
+                await crawler.__aexit__(None, None, None)
+                del self.active_crawlers[crawler]
 
-        return await call_next(request)
+            # Create new crawler if needed
+            if len(self.active_crawlers) < self.max_size:
+                crawler = AsyncWebCrawler(**kwargs)
+                await crawler.__aenter__()
+                self.active_crawlers[crawler] = current_time
+                return crawler
 
-app.add_middleware(RateLimitMiddleware)
+            # Reuse least recently used crawler
+            crawler = min(self.active_crawlers.items(), key=lambda x: x[1])[0]
+            self.active_crawlers[crawler] = current_time
+            return crawler
+
+    async def release(self, crawler: AsyncWebCrawler):
+        async with self._lock:
+            if crawler in self.active_crawlers:
+                self.active_crawlers[crawler] = time.time()
+
+    async def cleanup(self):
+        async with self._lock:
+            for crawler in list(self.active_crawlers.keys()):
+                await crawler.__aexit__(None, None, None)
+            self.active_crawlers.clear()
+
+class CrawlerService:
+    def __init__(self, max_concurrent_tasks: int = 10):
+        self.resource_monitor = ResourceMonitor(max_concurrent_tasks)
+        self.task_manager = TaskManager()
+        self.crawler_pool = CrawlerPool(max_concurrent_tasks)
+        self._processing_task = None
+
+    async def start(self):
+        await self.task_manager.start()
+        self._processing_task = asyncio.create_task(self._process_queue())
+
+    async def stop(self):
+        if self._processing_task:
+            self._processing_task.cancel()
+            try:
+                await self._processing_task
+            except asyncio.CancelledError:
+                pass
+        await self.task_manager.stop()
+        await self.crawler_pool.cleanup()
+
+    def _create_extraction_strategy(self, config: ExtractionConfig):
+        if not config:
+            return None
+
+        if config.type == CrawlerType.LLM:
+            return LLMExtractionStrategy(**config.params)
+        elif config.type == CrawlerType.COSINE:
+            return CosineStrategy(**config.params)
+        elif config.type == CrawlerType.JSON_CSS:
+            return JsonCssExtractionStrategy(**config.params)
+        return None
+
+    async def submit_task(self, request: CrawlRequest) -> str:
+        task_id = str(uuid.uuid4())
+        await self.task_manager.add_task(task_id, request.priority, request.ttl or 3600)
+        
+        # Store request data with task
+        self.task_manager.tasks[task_id].request = request
+        
+        return task_id
+
+    async def _process_queue(self):
+        while True:
+            try:
+                available_slots = await self.resource_monitor.get_available_slots()
+                if False and available_slots <= 0:
+                    await asyncio.sleep(1)
+                    continue
+
+                task_id = await self.task_manager.get_next_task()
+                if not task_id:
+                    await asyncio.sleep(1)
+                    continue
+
+                task_info = self.task_manager.get_task(task_id)
+                if not task_info:
+                    continue
+
+                request = task_info.request
+                self.task_manager.update_task(task_id, TaskStatus.PROCESSING)
+
+                try:
+                    crawler = await self.crawler_pool.acquire(**request.crawler_params)
+                    
+                    extraction_strategy = self._create_extraction_strategy(request.extraction_config)
+                    
+                    if isinstance(request.urls, list):
+                        results = await crawler.arun_many(
+                            urls=[str(url) for url in request.urls],
+                            word_count_threshold=MIN_WORD_THRESHOLD,
+                            extraction_strategy=extraction_strategy,
+                            js_code=request.js_code,
+                            wait_for=request.wait_for,
+                            css_selector=request.css_selector,
+                            screenshot=request.screenshot,
+                            magic=request.magic,
+                            session_id=request.session_id,
+                            cache_mode=request.cache_mode,
+                            **request.extra,
+                        )
+                    else:
+                        results = await crawler.arun(
+                            url=str(request.urls),
+                            extraction_strategy=extraction_strategy,
+                            js_code=request.js_code,
+                            wait_for=request.wait_for,
+                            css_selector=request.css_selector,
+                            screenshot=request.screenshot,
+                            magic=request.magic,
+                            session_id=request.session_id,
+                            cache_mode=request.cache_mode,
+                            **request.extra,
+                        )
+
+                    await self.crawler_pool.release(crawler)
+                    self.task_manager.update_task(task_id, TaskStatus.COMPLETED, results)
+
+                except Exception as e:
+                    logger.error(f"Error processing task {task_id}: {str(e)}")
+                    self.task_manager.update_task(task_id, TaskStatus.FAILED, error=str(e))
+
+            except Exception as e:
+                logger.error(f"Error in queue processing: {str(e)}")
+                await asyncio.sleep(1)
+
+app = FastAPI(title="Crawl4AI API")
 
 # CORS configuration
 origins = ["*"]  # Allow all origins
@@ -123,132 +342,151 @@ app.add_middleware(
 
 # Mount the pages directory as a static directory
 app.mount("/pages", StaticFiles(directory=__location__ + "/pages"), name="pages")
-app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
+
+# API token security
+security = HTTPBearer()
+CRAWL4AI_API_TOKEN = os.getenv("CRAWL4AI_API_TOKEN") or "test_api_code"
+
+async def verify_token(credentials: HTTPAuthorizationCredentials = Security(security)):
+    if not CRAWL4AI_API_TOKEN:
+        return credentials  # No token verification if CRAWL4AI_API_TOKEN is not set
+    if credentials.credentials != CRAWL4AI_API_TOKEN:
+        raise HTTPException(status_code=401, detail="Invalid token")
+    return credentials
+
+# Helper function to conditionally apply security
+def secure_endpoint():
+    return Depends(verify_token) if CRAWL4AI_API_TOKEN else None
+
+# Check if site directory exists
+if os.path.exists(__location__ + "/site"):
+    # Mount the site directory as a static directory
+    app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
+
 site_templates = Jinja2Templates(directory=__location__ + "/site")
 templates = Jinja2Templates(directory=__location__ + "/pages")
 
-@lru_cache()
-def get_crawler():
-    # Initialize and return a WebCrawler instance
-    crawler = WebCrawler(verbose = True)
-    crawler.warmup()
-    return crawler
+crawler_service = CrawlerService()
 
-class CrawlRequest(BaseModel):
-    urls: List[str]
-    include_raw_html: Optional[bool] = False
-    bypass_cache: bool = False
-    extract_blocks: bool = True
-    word_count_threshold: Optional[int] = 5
-    extraction_strategy: Optional[str] = "NoExtractionStrategy"
-    extraction_strategy_args: Optional[dict] = {}
-    chunking_strategy: Optional[str] = "RegexChunking"
-    chunking_strategy_args: Optional[dict] = {}
-    css_selector: Optional[str] = None
-    screenshot: Optional[bool] = False
-    user_agent: Optional[str] = None
-    verbose: Optional[bool] = True
+@app.on_event("startup")
+async def startup_event():
+    await crawler_service.start()
+
+@app.on_event("shutdown")
+async def shutdown_event():
+    await crawler_service.stop()
 
 @app.get("/")
 def read_root():
-    return RedirectResponse(url="/mkdocs")
+    if os.path.exists(__location__ + "/site"):
+        return RedirectResponse(url="/mkdocs")
+    # Return a json response
+    return {"message": "Crawl4AI API service is running"}
 
-@app.get("/old", response_class=HTMLResponse)
-@limiter.limit(get_rate_limit())
-async def read_index(request: Request):
-    partials_dir = os.path.join(__location__, "pages", "partial")
-    partials = {}
 
-    for filename in os.listdir(partials_dir):
-        if filename.endswith(".html"):
-            with open(os.path.join(partials_dir, filename), "r", encoding="utf8") as file:
-                partials[filename[:-5]] = file.read()
+@app.post("/crawl", dependencies=[Depends(verify_token)])
+async def crawl(request: CrawlRequest) -> Dict[str, str]:
+    task_id = await crawler_service.submit_task(request)
+    return {"task_id": task_id}
 
-    return templates.TemplateResponse("index.html", {"request": request, **partials})
+@app.get("/task/{task_id}", dependencies=[Depends(verify_token)])
+async def get_task_status(task_id: str):
+    task_info = crawler_service.task_manager.get_task(task_id)
+    if not task_info:
+        raise HTTPException(status_code=404, detail="Task not found")
 
-@app.get("/total-count")
-async def get_total_url_count():
-    count = get_total_count()
-    return JSONResponse(content={"count": count})
+    response = {
+        "status": task_info.status,
+        "created_at": task_info.created_at,
+    }
 
-@app.get("/clear-db")
-async def clear_database():
-    # clear_db()
-    return JSONResponse(content={"message": "Database cleared."})
+    if task_info.status == TaskStatus.COMPLETED:
+        # Convert CrawlResult to dict for JSON response
+        if isinstance(task_info.result, list):
+            response["results"] = [result.dict() for result in task_info.result]
+        else:
+            response["result"] = task_info.result.dict()
+    elif task_info.status == TaskStatus.FAILED:
+        response["error"] = task_info.error
 
-def import_strategy(module_name: str, class_name: str, *args, **kwargs):
-    try:
-        module = importlib.import_module(module_name)
-        strategy_class = getattr(module, class_name)
-        return strategy_class(*args, **kwargs)
-    except ImportError:
-        print("ImportError: Module not found.")
-        raise HTTPException(status_code=400, detail=f"Module {module_name} not found.")
-    except AttributeError:
-        print("AttributeError: Class not found.")
-        raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")
+    return response
 
-@app.post("/crawl")
-@limiter.limit(get_rate_limit())
-async def crawl_urls(crawl_request: CrawlRequest, request: Request):
-    logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
-    global current_requests
-    async with lock:
-        if current_requests >= MAX_CONCURRENT_REQUESTS:
-            raise HTTPException(status_code=429, detail="Too many requests - please try again later.")
-        current_requests += 1
-
-    try:
-        logging.debug("[LOG] Loading extraction and chunking strategies...")
-        crawl_request.extraction_strategy_args['verbose'] = True
-        crawl_request.chunking_strategy_args['verbose'] = True
-        
-        extraction_strategy = import_strategy("crawl4ai.extraction_strategy", crawl_request.extraction_strategy, **crawl_request.extraction_strategy_args)
-        chunking_strategy = import_strategy("crawl4ai.chunking_strategy", crawl_request.chunking_strategy, **crawl_request.chunking_strategy_args)
-
-        # Use ThreadPoolExecutor to run the synchronous WebCrawler in async manner
-        logging.debug("[LOG] Running the WebCrawler...")
-        with ThreadPoolExecutor() as executor:
-            loop = asyncio.get_event_loop()
-            futures = [
-                loop.run_in_executor(
-                    executor, 
-                    get_crawler().run,
-                    str(url),
-                    crawl_request.word_count_threshold,
-                    extraction_strategy,
-                    chunking_strategy,
-                    crawl_request.bypass_cache,
-                    crawl_request.css_selector,
-                    crawl_request.screenshot,
-                    crawl_request.user_agent,
-                    crawl_request.verbose
-                )
-                for url in crawl_request.urls
-            ]
-            results = await asyncio.gather(*futures)
-
-        # if include_raw_html is False, remove the raw HTML content from the results
-        if not crawl_request.include_raw_html:
-            for result in results:
-                result.html = None
-
-        return {"results": [result.model_dump() for result in results]}
-    finally:
-        async with lock:
-            current_requests -= 1
+@app.post("/crawl_sync", dependencies=[Depends(verify_token)])
+async def crawl_sync(request: CrawlRequest) -> Dict[str, Any]:
+    task_id = await crawler_service.submit_task(request)
+    
+    # Wait up to 60 seconds for task completion
+    for _ in range(60):
+        task_info = crawler_service.task_manager.get_task(task_id)
+        if not task_info:
+            raise HTTPException(status_code=404, detail="Task not found")
             
-@app.get("/strategies/extraction", response_class=JSONResponse)
-async def get_extraction_strategies():
-    with open(f"{__location__}/docs/extraction_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
-
-@app.get("/strategies/chunking", response_class=JSONResponse)
-async def get_chunking_strategies():
-    with open(f"{__location__}/docs/chunking_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
+        if task_info.status == TaskStatus.COMPLETED:
+            # Return same format as /task/{task_id} endpoint
+            if isinstance(task_info.result, list):
+                return {"status": task_info.status, "results": [result.dict() for result in task_info.result]}
+            return {"status": task_info.status, "result": task_info.result.dict()}
+            
+        if task_info.status == TaskStatus.FAILED:
+            raise HTTPException(status_code=500, detail=task_info.error)
+            
+        await asyncio.sleep(1)
+    
+    # If we get here, task didn't complete within timeout
+    raise HTTPException(status_code=408, detail="Task timed out")
 
+@app.post("/crawl_direct", dependencies=[Depends(verify_token)])
+async def crawl_direct(request: CrawlRequest) -> Dict[str, Any]:
+    try:
+        crawler = await crawler_service.crawler_pool.acquire(**request.crawler_params)
+        extraction_strategy = crawler_service._create_extraction_strategy(request.extraction_config)
+        
+        try:
+            if isinstance(request.urls, list):
+                results = await crawler.arun_many(
+                    urls=[str(url) for url in request.urls],
+                    extraction_strategy=extraction_strategy,
+                    js_code=request.js_code,
+                    wait_for=request.wait_for,
+                    css_selector=request.css_selector,
+                    screenshot=request.screenshot,
+                    magic=request.magic,
+                    cache_mode=request.cache_mode,
+                    session_id=request.session_id,
+                    **request.extra,
+                )
+                return {"results": [result.dict() for result in results]}
+            else:
+                result = await crawler.arun(
+                    url=str(request.urls),
+                    extraction_strategy=extraction_strategy,
+                    js_code=request.js_code,
+                    wait_for=request.wait_for,
+                    css_selector=request.css_selector,
+                    screenshot=request.screenshot,
+                    magic=request.magic,
+                    cache_mode=request.cache_mode,
+                    session_id=request.session_id,
+                    **request.extra,
+                )
+                return {"result": result.dict()}
+        finally:
+            await crawler_service.crawler_pool.release(crawler)
+    except Exception as e:
+        logger.error(f"Error in direct crawl: {str(e)}")
+        raise HTTPException(status_code=500, detail=str(e))
+    
+@app.get("/health")
+async def health_check():
+    available_slots = await crawler_service.resource_monitor.get_available_slots()
+    memory = psutil.virtual_memory()
+    return {
+        "status": "healthy",
+        "available_slots": available_slots,
+        "memory_usage": memory.percent,
+        "cpu_usage": psutil.cpu_percent(),
+    }
 
 if __name__ == "__main__":
     import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8888)
+    uvicorn.run(app, host="0.0.0.0", port=11235)

mkdocs.yml

@@ -8,6 +8,7 @@ docs_dir: docs/md_v2
 nav:
   - Home: 'index.md'
   - 'Installation': 'basic/installation.md'
+  - 'Docker Deplotment': 'basic/docker-deploymeny.md'
   - 'Quick Start': 'basic/quickstart.md'
   
   - Basic:
@@ -16,6 +17,7 @@ nav:
     - 'Browser Configuration': 'basic/browser-config.md'
     - 'Page Interaction': 'basic/page-interaction.md'
     - 'Content Selection': 'basic/content-selection.md'
+    - 'Cache Modes': 'basic/cache-modes.md'
 
   - Advanced:
     - 'Content Processing': 'advanced/content-processing.md'
@@ -34,10 +36,29 @@ nav:
     - 'Chunking': 'extraction/chunking.md'
 
   - API Reference:
+    - 'Parameters Table': 'api/parameters.md'
     - 'AsyncWebCrawler': 'api/async-webcrawler.md'
     - 'AsyncWebCrawler.arun()': 'api/arun.md'
     - 'CrawlResult': 'api/crawl-result.md'
     - 'Strategies': 'api/strategies.md'
+    
+  - Tutorial:
+    - '1. Getting Started': 'tutorial/episode_01_Introduction_to_Crawl4AI_and_Basic_Installation.md'
+    - '2. Advanced Features': 'tutorial/episode_02_Overview_of_Advanced_Features.md'
+    - '3. Browser Setup': 'tutorial/episode_03_Browser_Configurations_&_Headless_Crawling.md'
+    - '4. Proxy Settings': 'tutorial/episode_04_Advanced_Proxy_and_Security_Settings.md'
+    - '5. Dynamic Content': 'tutorial/episode_05_JavaScript_Execution_and_Dynamic_Content_Handling.md'
+    - '6. Magic Mode': 'tutorial/episode_06_Magic_Mode_and_Anti-Bot_Protection.md'
+    - '7. Content Cleaning': 'tutorial/episode_07_Content_Cleaning_and_Fit_Markdown.md'
+    - '8. Media Handling': 'tutorial/episode_08_Media_Handling:_Images,_Videos,_and_Audio.md'
+    - '9. Link Analysis': 'tutorial/episode_09_Link_Analysis_and_Smart_Filtering.md'
+    - '10. User Simulation': 'tutorial/episode_10_Custom_Headers,_Identity,_and_User_Simulation.md'
+    - '11.1. JSON CSS': 'tutorial/episode_11_1_Extraction_Strategies:_JSON_CSS.md'
+    - '11.2. LLM Strategy': 'tutorial/episode_11_2_Extraction_Strategies:_LLM.md'
+    - '11.3. Cosine Strategy': 'tutorial/episode_11_3_Extraction_Strategies:_Cosine.md'
+    - '12. Session Crawling': 'tutorial/episode_12_Session-Based_Crawling_for_Dynamic_Websites.md'
+    - '13. Text Chunking': 'tutorial/episode_13_Chunking_Strategies_for_Large_Text_Processing.md'
+    - '14. Custom Workflows': 'tutorial/episode_14_Hooks_and_Custom_Workflow_with_AsyncWebCrawler.md'
 
 
 theme:

pages/app.css

@@ -1,131 +0,0 @@
-:root {
-    --ifm-font-size-base: 100%;
-    --ifm-line-height-base: 1.65;
-    --ifm-font-family-base: system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans, sans-serif,
-        BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji",
-        "Segoe UI Symbol";
-}
-html {
-    -webkit-font-smoothing: antialiased;
-    -webkit-text-size-adjust: 100%;
-    text-size-adjust: 100%;
-    font: var(--ifm-font-size-base) / var(--ifm-line-height-base) var(--ifm-font-family-base);
-}
-body {
-    background-color: #1a202c;
-    color: #fff;
-}
-.tab-content {
-    max-height: 400px;
-    overflow: auto;
-}
-pre {
-    white-space: pre-wrap;
-    font-size: 14px;
-}
-pre code {
-    width: 100%;
-}
-
-/* Custom styling for docs-item class and Markdown generated elements */
-.docs-item {
-    background-color: #2d3748; /* bg-gray-800 */
-    padding: 1rem; /* p-4 */
-    border-radius: 0.375rem; /* rounded */
-    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1); /* shadow-md */
-    margin-bottom: 1rem; /* space between items */
-    line-height: 1.5; /* leading-normal */
-}
-
-.docs-item h3,
-.docs-item h4 {
-    color: #ffffff; /* text-white */
-    font-size: 1.25rem; /* text-xl */
-    font-weight: 700; /* font-bold */
-    margin-bottom: 0.5rem; /* mb-2 */
-}
-.docs-item h4 {
-    font-size: 1rem; /* text-xl */
-}
-
-.docs-item p {
-    color: #e2e8f0; /* text-gray-300 */
-    margin-bottom: 0.5rem; /* mb-2 */
-}
-
-.docs-item code {
-    background-color: #1a202c; /* bg-gray-900 */
-    color: #e2e8f0; /* text-gray-300 */
-    padding: 0.25rem 0.5rem; /* px-2 py-1 */
-    border-radius: 0.25rem; /* rounded */
-    font-size: 0.875rem; /* text-sm */
-}
-
-.docs-item pre {
-    background-color: #1a202c; /* bg-gray-900 */
-    color: #e2e8f0; /* text-gray-300 */
-    padding: 0.5rem; /* p-2 */
-    border-radius: 0.375rem; /* rounded */
-    overflow: auto; /* overflow-auto */
-    margin-bottom: 0.5rem; /* mb-2 */
-}
-
-.docs-item div {
-    color: #e2e8f0; /* text-gray-300 */
-    font-size: 1rem; /* prose prose-sm */
-    line-height: 1.25rem; /* line-height for readability */
-}
-
-/* Adjustments to make prose class more suitable for dark mode */
-.prose {
-    max-width: none; /* max-w-none */
-}
-
-.prose p,
-.prose ul {
-    margin-bottom: 1rem; /* mb-4 */
-}
-
-.prose code {
-    /* background-color: #4a5568; */ /* bg-gray-700 */
-    color: #65a30d; /* text-white */
-    padding: 0.25rem 0.5rem; /* px-1 py-0.5 */
-    border-radius: 0.25rem; /* rounded */
-    display: inline-block; /* inline-block */
-}
-
-.prose pre {
-    background-color: #1a202c; /* bg-gray-900 */
-    color: #ffffff; /* text-white */
-    padding: 0.5rem; /* p-2 */
-    border-radius: 0.375rem; /* rounded */
-}
-
-.prose h3 {
-    color: #65a30d; /* text-white */
-    font-size: 1.25rem; /* text-xl */
-    font-weight: 700; /* font-bold */
-    margin-bottom: 0.5rem; /* mb-2 */
-}
-
-body {
-    background-color: #1a1a1a;
-    color: #b3ff00;
-}
-.sidebar {
-    color: #b3ff00;
-    border-right: 1px solid #333;
-}
-.sidebar a {
-    color: #b3ff00;
-    text-decoration: none;
-}
-.sidebar a:hover {
-    background-color: #555;
-}
-.content-section {
-    display: none;
-}
-.content-section.active {
-    display: block;
-}

pages/app.js

@@ -1,356 +0,0 @@
-// JavaScript to manage dynamic form changes and logic
-document.getElementById("extraction-strategy-select").addEventListener("change", function () {
-    const strategy = this.value;
-    const providerModelSelect = document.getElementById("provider-model-select");
-    const tokenInput = document.getElementById("token-input");
-    const instruction = document.getElementById("instruction");
-    const semantic_filter = document.getElementById("semantic_filter");
-    const instruction_div = document.getElementById("instruction_div");
-    const semantic_filter_div = document.getElementById("semantic_filter_div");
-    const llm_settings = document.getElementById("llm_settings");
-
-    if (strategy === "LLMExtractionStrategy") {
-        // providerModelSelect.disabled = false;
-        // tokenInput.disabled = false;
-        // semantic_filter.disabled = true;
-        // instruction.disabled = false;
-        llm_settings.classList.remove("hidden");
-        instruction_div.classList.remove("hidden");
-        semantic_filter_div.classList.add("hidden");
-    } else if (strategy === "NoExtractionStrategy") {
-        semantic_filter_div.classList.add("hidden");
-        instruction_div.classList.add("hidden");
-        llm_settings.classList.add("hidden");
-    } else {
-        // providerModelSelect.disabled = true;
-        // tokenInput.disabled = true;
-        // semantic_filter.disabled = false;
-        // instruction.disabled = true;
-        llm_settings.classList.add("hidden");
-        instruction_div.classList.add("hidden");
-        semantic_filter_div.classList.remove("hidden");
-    }
-
-
-});
-
-// Get the selected provider model and token from local storage
-const storedProviderModel = localStorage.getItem("provider_model");
-const storedToken = localStorage.getItem(storedProviderModel);
-
-if (storedProviderModel) {
-    document.getElementById("provider-model-select").value = storedProviderModel;
-}
-
-if (storedToken) {
-    document.getElementById("token-input").value = storedToken;
-}
-
-// Handle provider model dropdown change
-document.getElementById("provider-model-select").addEventListener("change", () => {
-    const selectedProviderModel = document.getElementById("provider-model-select").value;
-    const storedToken = localStorage.getItem(selectedProviderModel);
-
-    if (storedToken) {
-        document.getElementById("token-input").value = storedToken;
-    } else {
-        document.getElementById("token-input").value = "";
-    }
-});
-
-// Fetch total count from the database
-axios
-    .get("/total-count")
-    .then((response) => {
-        document.getElementById("total-count").textContent = response.data.count;
-    })
-    .catch((error) => console.error(error));
-
-// Handle crawl button click
-document.getElementById("crawl-btn").addEventListener("click", () => {
-    // validate input to have both URL and API token
-    // if selected extraction strategy is LLMExtractionStrategy, then API token is required
-    if (document.getElementById("extraction-strategy-select").value === "LLMExtractionStrategy") {
-        if (!document.getElementById("url-input").value || !document.getElementById("token-input").value) {
-            alert("Please enter both URL(s) and API token.");
-            return;
-        }
-    }
-
-    const selectedProviderModel = document.getElementById("provider-model-select").value;
-    const apiToken = document.getElementById("token-input").value;
-    const extractBlocks = document.getElementById("extract-blocks-checkbox").checked;
-    const bypassCache = document.getElementById("bypass-cache-checkbox").checked;
-
-    // Save the selected provider model and token to local storage
-    localStorage.setItem("provider_model", selectedProviderModel);
-    localStorage.setItem(selectedProviderModel, apiToken);
-
-    const urlsInput = document.getElementById("url-input").value;
-    const urls = urlsInput.split(",").map((url) => url.trim());
-    const data = {
-        urls: urls,
-        include_raw_html: true,
-        bypass_cache: bypassCache,
-        extract_blocks: extractBlocks,
-        word_count_threshold: parseInt(document.getElementById("threshold").value),
-        extraction_strategy: document.getElementById("extraction-strategy-select").value,
-        extraction_strategy_args: {
-            provider: selectedProviderModel,
-            api_token: apiToken,
-            instruction: document.getElementById("instruction").value,
-            semantic_filter: document.getElementById("semantic_filter").value,
-        },
-        chunking_strategy: document.getElementById("chunking-strategy-select").value,
-        chunking_strategy_args: {},
-        css_selector: document.getElementById("css-selector").value,
-        screenshot: document.getElementById("screenshot-checkbox").checked,
-        // instruction: document.getElementById("instruction").value,
-        // semantic_filter: document.getElementById("semantic_filter").value,
-        verbose: true,
-    };
-
-    // import requests
-
-    // data = {
-    //   "urls": [
-    //     "https://www.nbcnews.com/business"
-    //   ],
-    //   "word_count_threshold": 10,
-    //   "extraction_strategy": "NoExtractionStrategy",
-    // }
-    
-    // response = requests.post("https://crawl4ai.com/crawl", json=data) # OR local host if your run locally 
-    // print(response.json())
-
-    // save api token to local storage
-    localStorage.setItem("api_token", document.getElementById("token-input").value);
-
-    document.getElementById("loading").classList.remove("hidden");
-    document.getElementById("result").style.visibility = "hidden";
-    document.getElementById("code_help").style.visibility = "hidden";
-
-    axios
-        .post("/crawl", data)
-        .then((response) => {
-            const result = response.data.results[0];
-            const parsedJson = JSON.parse(result.extracted_content);
-            document.getElementById("json-result").textContent = JSON.stringify(parsedJson, null, 2);
-            document.getElementById("cleaned-html-result").textContent = result.cleaned_html;
-            document.getElementById("markdown-result").textContent = result.markdown;
-            document.getElementById("media-result").textContent = JSON.stringify( result.media, null, 2);
-            if (result.screenshot){
-                const imgElement = document.createElement("img");
-                // Set the src attribute with the base64 data
-                imgElement.src = `data:image/png;base64,${result.screenshot}`;
-                document.getElementById("screenshot-result").innerHTML = "";
-                document.getElementById("screenshot-result").appendChild(imgElement);
-            }
-            
-            // Update code examples dynamically
-            const extractionStrategy = data.extraction_strategy;
-            const isLLMExtraction = extractionStrategy === "LLMExtractionStrategy";
-
-            // REMOVE API TOKEN FROM CODE EXAMPLES
-            data.extraction_strategy_args.api_token = "your_api_token";
-
-            if (data.extraction_strategy === "NoExtractionStrategy") {
-                delete data.extraction_strategy_args;
-                delete data.extrac_blocks;
-            }
-
-            if (data.chunking_strategy === "RegexChunking") {
-                delete data.chunking_strategy_args;
-            }
-
-            delete data.verbose;
-
-            if (data.css_selector === "") {
-                delete data.css_selector;
-            }
-
-            if (!data.bypass_cache) {
-                delete data.bypass_cache;
-            }
-
-            if (!data.extract_blocks) {
-                delete data.extract_blocks;
-            }
-
-            if (!data.include_raw_html) {
-                delete data.include_raw_html;
-            }
-
-            document.getElementById(
-                "curl-code"
-            ).textContent = `curl -X POST -H "Content-Type: application/json" -d '${JSON.stringify({
-                ...data,
-                api_token: isLLMExtraction ? "your_api_token" : undefined,
-            }, null, 2)}' https://crawl4ai.com/crawl`;
-
-            document.getElementById("python-code").textContent = `import requests\n\ndata = ${JSON.stringify(
-                { ...data, api_token: isLLMExtraction ? "your_api_token" : undefined },
-                null,
-                2
-            )}\n\nresponse = requests.post("https://crawl4ai.com/crawl", json=data) # OR local host if your run locally \nprint(response.json())`;
-
-            document.getElementById(
-                "nodejs-code"
-            ).textContent = `const axios = require('axios');\n\nconst data = ${JSON.stringify(
-                { ...data, api_token: isLLMExtraction ? "your_api_token" : undefined },
-                null,
-                2
-            )};\n\naxios.post("https://crawl4ai.com/crawl", data) // OR local host if your run locally \n    .then(response => console.log(response.data))\n    .catch(error => console.error(error));`;
-
-            document.getElementById(
-                "library-code"
-            ).textContent = `from crawl4ai.web_crawler import WebCrawler\nfrom crawl4ai.extraction_strategy import *\nfrom crawl4ai.chunking_strategy import *\n\ncrawler = WebCrawler()\ncrawler.warmup()\n\nresult = crawler.run(\n    url='${
-                urls[0]
-            }',\n    word_count_threshold=${data.word_count_threshold},\n    extraction_strategy=${
-                isLLMExtraction
-                    ? `${extractionStrategy}(provider="${data.provider_model}", api_token="${data.api_token}")`
-                    : extractionStrategy + "()"
-            },\n    chunking_strategy=${data.chunking_strategy}(),\n    bypass_cache=${
-                data.bypass_cache
-            },\n    css_selector="${data.css_selector}"\n)\nprint(result)`;
-
-            // Highlight code syntax
-            hljs.highlightAll();
-
-            // Select JSON tab by default
-            document.querySelector('.tab-btn[data-tab="json"]').click();
-
-            document.getElementById("loading").classList.add("hidden");
-
-            document.getElementById("result").style.visibility = "visible";
-            document.getElementById("code_help").style.visibility = "visible";
-
-            // increment the total count
-            document.getElementById("total-count").textContent =
-                parseInt(document.getElementById("total-count").textContent) + 1;
-        })
-        .catch((error) => {
-            console.error(error);
-            document.getElementById("loading").classList.add("hidden");
-        });
-});
-
-// Handle tab clicks
-document.querySelectorAll(".tab-btn").forEach((btn) => {
-    btn.addEventListener("click", () => {
-        const tab = btn.dataset.tab;
-        document.querySelectorAll(".tab-btn").forEach((b) => b.classList.remove("bg-lime-700", "text-white"));
-        btn.classList.add("bg-lime-700", "text-white");
-        document.querySelectorAll(".tab-content.code pre").forEach((el) => el.classList.add("hidden"));
-        document.getElementById(`${tab}-result`).parentElement.classList.remove("hidden");
-    });
-});
-
-// Handle code tab clicks
-document.querySelectorAll(".code-tab-btn").forEach((btn) => {
-    btn.addEventListener("click", () => {
-        const tab = btn.dataset.tab;
-        document.querySelectorAll(".code-tab-btn").forEach((b) => b.classList.remove("bg-lime-700", "text-white"));
-        btn.classList.add("bg-lime-700", "text-white");
-        document.querySelectorAll(".tab-content.result pre").forEach((el) => el.classList.add("hidden"));
-        document.getElementById(`${tab}-code`).parentElement.classList.remove("hidden");
-    });
-});
-
-// Handle copy to clipboard button clicks
-
-async function copyToClipboard(text) {
-    if (navigator.clipboard && navigator.clipboard.writeText) {
-        return navigator.clipboard.writeText(text);
-    } else {
-        return fallbackCopyTextToClipboard(text);
-    }
-}
-
-function fallbackCopyTextToClipboard(text) {
-    return new Promise((resolve, reject) => {
-        const textArea = document.createElement("textarea");
-        textArea.value = text;
-
-        // Avoid scrolling to bottom
-        textArea.style.top = "0";
-        textArea.style.left = "0";
-        textArea.style.position = "fixed";
-
-        document.body.appendChild(textArea);
-        textArea.focus();
-        textArea.select();
-
-        try {
-            const successful = document.execCommand("copy");
-            if (successful) {
-                resolve();
-            } else {
-                reject();
-            }
-        } catch (err) {
-            reject(err);
-        }
-
-        document.body.removeChild(textArea);
-    });
-}
-
-document.querySelectorAll(".copy-btn").forEach((btn) => {
-    btn.addEventListener("click", () => {
-        const target = btn.dataset.target;
-        const code = document.getElementById(target).textContent;
-        //navigator.clipboard.writeText(code).then(() => {
-        copyToClipboard(code).then(() => {
-            btn.textContent = "Copied!";
-            setTimeout(() => {
-                btn.textContent = "Copy";
-            }, 2000);
-        });
-    });
-});
-
-document.addEventListener("DOMContentLoaded", async () => {
-    try {
-        const extractionResponse = await fetch("/strategies/extraction");
-        const extractionStrategies = await extractionResponse.json();
-
-        const chunkingResponse = await fetch("/strategies/chunking");
-        const chunkingStrategies = await chunkingResponse.json();
-
-        renderStrategies("extraction-strategies", extractionStrategies);
-        renderStrategies("chunking-strategies", chunkingStrategies);
-    } catch (error) {
-        console.error("Error fetching strategies:", error);
-    }
-});
-
-function renderStrategies(containerId, strategies) {
-    const container = document.getElementById(containerId);
-    container.innerHTML = ""; // Clear any existing content
-    strategies = JSON.parse(strategies);
-    Object.entries(strategies).forEach(([strategy, description]) => {
-        const strategyElement = document.createElement("div");
-        strategyElement.classList.add("bg-zinc-800", "p-4", "rounded", "shadow-md", "docs-item");
-
-        const strategyDescription = document.createElement("div");
-        strategyDescription.classList.add("text-gray-300", "prose", "prose-sm");
-        strategyDescription.innerHTML = marked.parse(description);
-
-        strategyElement.appendChild(strategyDescription);
-
-        container.appendChild(strategyElement);
-    });
-}
-document.querySelectorAll(".sidebar a").forEach((link) => {
-    link.addEventListener("click", function (event) {
-        event.preventDefault();
-        document.querySelectorAll(".content-section").forEach((section) => {
-            section.classList.remove("active");
-        });
-        const target = event.target.getAttribute("data-target");
-        document.getElementById(target).classList.add("active");
-    });
-});
-// Highlight code syntax
-hljs.highlightAll();

pages/index copy.html

@@ -1,971 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-    <head>
-        <meta charset="UTF-8" />
-        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-        <title>Crawl4AI</title>
-
-        <link rel="preconnect" href="https://fonts.googleapis.com" />
-        <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
-        <link href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap" rel="stylesheet" />
-
-        <!-- <link href="https://cdn.jsdelivr.net/npm/tailwindcss@3.4.3/dist/tailwind.min.css" rel="stylesheet" /> -->
-        <script src="https://cdn.tailwindcss.com"></script>
-        <script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
-        <link
-            rel="stylesheet"
-            href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/styles/monokai.min.css"
-        />
-        <script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
-
-        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/highlight.min.js"></script>
-        <style>
-            :root {
-                --ifm-font-size-base: 100%;
-                --ifm-line-height-base: 1.65;
-                --ifm-font-family-base: system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans,
-                    sans-serif, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji",
-                    "Segoe UI Emoji", "Segoe UI Symbol";
-            }
-            html {
-                -webkit-font-smoothing: antialiased;
-                -webkit-text-size-adjust: 100%;
-                text-size-adjust: 100%;
-                font: var(--ifm-font-size-base) / var(--ifm-line-height-base) var(--ifm-font-family-base);
-            }
-            body {
-                background-color: #1a202c;
-                color: #fff;
-            }
-            .tab-content {
-                max-height: 400px;
-                overflow: auto;
-            }
-            pre {
-                white-space: pre-wrap;
-                font-size: 14px;
-            }
-            pre code {
-                width: 100%;
-            }
-        </style>
-        <style>
-            /* Custom styling for docs-item class and Markdown generated elements */
-            .docs-item {
-                background-color: #2d3748; /* bg-gray-800 */
-                padding: 1rem; /* p-4 */
-                border-radius: 0.375rem; /* rounded */
-                box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1); /* shadow-md */
-                margin-bottom: 1rem; /* space between items */
-            }
-
-            .docs-item h3,
-            .docs-item h4 {
-                color: #ffffff; /* text-white */
-                font-size: 1.25rem; /* text-xl */
-                font-weight: 700; /* font-bold */
-                margin-bottom: 0.5rem; /* mb-2 */
-            }
-
-            .docs-item p {
-                color: #e2e8f0; /* text-gray-300 */
-                margin-bottom: 0.5rem; /* mb-2 */
-            }
-
-            .docs-item code {
-                background-color: #1a202c; /* bg-gray-900 */
-                color: #e2e8f0; /* text-gray-300 */
-                padding: 0.25rem 0.5rem; /* px-2 py-1 */
-                border-radius: 0.25rem; /* rounded */
-            }
-
-            .docs-item pre {
-                background-color: #1a202c; /* bg-gray-900 */
-                color: #e2e8f0; /* text-gray-300 */
-                padding: 0.5rem; /* p-2 */
-                border-radius: 0.375rem; /* rounded */
-                overflow: auto; /* overflow-auto */
-                margin-bottom: 0.5rem; /* mb-2 */
-            }
-
-            .docs-item div {
-                color: #e2e8f0; /* text-gray-300 */
-                font-size: 1rem; /* prose prose-sm */
-                line-height: 1.25rem; /* line-height for readability */
-            }
-
-            /* Adjustments to make prose class more suitable for dark mode */
-            .prose {
-                max-width: none; /* max-w-none */
-            }
-
-            .prose p,
-            .prose ul {
-                margin-bottom: 1rem; /* mb-4 */
-            }
-
-            .prose code {
-                /* background-color: #4a5568; */ /* bg-gray-700 */
-                color: #65a30d; /* text-white */
-                padding: 0.25rem 0.5rem; /* px-1 py-0.5 */
-                border-radius: 0.25rem; /* rounded */
-                display: inline-block; /* inline-block */
-            }
-
-            .prose pre {
-                background-color: #1a202c; /* bg-gray-900 */
-                color: #ffffff; /* text-white */
-                padding: 0.5rem; /* p-2 */
-                border-radius: 0.375rem; /* rounded */
-            }
-
-            .prose h3 {
-                color: #65a30d; /* text-white */
-                font-size: 1.25rem; /* text-xl */
-                font-weight: 700; /* font-bold */
-                margin-bottom: 0.5rem; /* mb-2 */
-            }
-        </style>
-    </head>
-    <body class="bg-black text-gray-200">
-        <header class="bg-zinc-950 text-white py-4 flex">
-            <div class="mx-auto px-4">
-                <h1 class="text-2xl font-bold">🔥🕷️ Crawl4AI: Web Data for your Thoughts</h1>
-            </div>
-            <div class="mx-auto px-4 flex font-bold text-xl gap-2">
-                <span>📊 Total Website Processed</span>
-                <span id="total-count" class="text-lime-400">2</span>
-            </div>
-        </header>
-
-        <section class="try-it py-8 px-16 pb-20">
-            <div class="container mx-auto px-4">
-                <h2 class="text-2xl font-bold mb-4">Try It Now</h2>
-                <div class="grid grid-cols-1 lg:grid-cols-3 gap-4">
-                    <div class="space-y-4">
-                        <div class="flex flex-col">
-                            <label for="url-input" class="text-lime-500 font-bold text-xs">URL(s)</label>
-                            <input
-                                type="text"
-                                id="url-input"
-                                value="https://www.nbcnews.com/business"
-                                class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-lime-500"
-                                placeholder="Enter URL(s) separated by commas"
-                            />
-                        </div>
-                        <div class="flex flex-col">
-                            <label for="threshold" class="text-lime-500 font-bold text-xs">Min Words Threshold</label>
-                            <select
-                                id="threshold"
-                                class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-lime-500"
-                            >
-                                <option value="5">5</option>
-                                <option value="10" selected>10</option>
-                                <option value="15">15</option>
-                                <option value="20">20</option>
-                                <option value="25">25</option>
-                            </select>
-                        </div>
-                        <div class="flex flex-col">
-                            <label for="css-selector" class="text-lime-500 font-bold text-xs">CSS Selector</label>
-                            <input
-                                type="text"
-                                id="css-selector"
-                                class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-lime-500"
-                                placeholder="Enter CSS Selector"
-                            />
-                        </div>
-                        <div class="flex flex-col">
-                            <label for="extraction-strategy-select" class="text-lime-500 font-bold text-xs"
-                                >Extraction Strategy</label
-                            >
-                            <select
-                                id="extraction-strategy-select"
-                                class="border border-zinc-700 rounded px-4 py-1 bg-zinc-900 text-lime-500"
-                            >
-                                <option value="CosineStrategy">CosineStrategy</option>
-                                <option value="LLMExtractionStrategy">LLMExtractionStrategy</option>
-                                <option value="NoExtractionStrategy">NoExtractionStrategy</option>
-                            </select>
-                        </div>
-                        <div class="flex flex-col">
-                            <label for="chunking-strategy-select" class="text-lime-500 font-bold text-xs"
-                                >Chunking Strategy</label
-                            >
-                            <select
-                                id="chunking-strategy-select"
-                                class="border border-zinc-700 rounded px-4 py-1 bg-zinc-900 text-lime-500"
-                            >
-                                <option value="RegexChunking">RegexChunking</option>
-                                <option value="NlpSentenceChunking">NlpSentenceChunking</option>
-                                <option value="TopicSegmentationChunking">TopicSegmentationChunking</option>
-                                <option value="FixedLengthWordChunking">FixedLengthWordChunking</option>
-                                <option value="SlidingWindowChunking">SlidingWindowChunking</option>
-                            </select>
-                        </div>
-                        <div class="flex flex-col">
-                            <label for="provider-model-select" class="text-lime-500 font-bold text-xs"
-                                >Provider Model</label
-                            >
-                            <select
-                                id="provider-model-select"
-                                class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-lime-500"
-                                disabled
-                            >
-                                <option value="groq/llama3-70b-8192">groq/llama3-70b-8192</option>
-                                <option value="groq/llama3-8b-8192">groq/llama3-8b-8192</option>
-                                <option value="openai/gpt-4-turbo">gpt-4-turbo</option>
-                                <option value="openai/gpt-3.5-turbo">gpt-3.5-turbo</option>
-                                <option value="anthropic/claude-3-haiku-20240307">claude-3-haiku</option>
-                                <option value="anthropic/claude-3-opus-20240229">claude-3-opus</option>
-                                <option value="anthropic/claude-3-sonnet-20240229">claude-3-sonnet</option>
-                            </select>
-                        </div>
-                        <div class="flex flex-col">
-                            <label for="token-input" class="text-lime-500 font-bold text-xs">API Token</label>
-                            <input
-                                type="password"
-                                id="token-input"
-                                class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-lime-500"
-                                placeholder="Enter Groq API token"
-                                disabled
-                            />
-                        </div>
-                        <div class="flex gap-3">
-                            <div class="flex items-center gap-2">
-                                <input type="checkbox" id="bypass-cache-checkbox" />
-                                <label for="bypass-cache-checkbox" class="text-lime-500 font-bold">Bypass Cache</label>
-                            </div>
-                            <div class="flex items-center gap-2">
-                                <input type="checkbox" id="extract-blocks-checkbox" checked />
-                                <label for="extract-blocks-checkbox" class="text-lime-500 font-bold"
-                                    >Extract Blocks</label
-                                >
-                            </div>
-                            <button id="crawl-btn" class="bg-lime-600 text-black font-bold px-4 py-0 rounded">
-                                Crawl
-                            </button>
-                        </div>
-                    </div>
-
-                    <div id="result" class=" ">
-                        <div id="loading" class="hidden">
-                            <p class="text-white">Loading... Please wait.</p>
-                        </div>
-                        <div class="tab-buttons flex gap-2">
-                            <button
-                                class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="json"
-                            >
-                                JSON
-                            </button>
-                            <button
-                                class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="cleaned-html"
-                            >
-                                Cleaned HTML
-                            </button>
-                            <button
-                                class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="markdown"
-                            >
-                                Markdown
-                            </button>
-                        </div>
-                        <div class="tab-content code bg-zinc-900 p-2 rounded h-full border border-zinc-700 text-sm">
-                            <pre class="h-full flex"><code id="json-result" class="language-json"></code></pre>
-                            <pre
-                                class="hidden h-full flex"
-                            ><code id="cleaned-html-result" class="language-html"></code></pre>
-                            <pre
-                                class="hidden h-full flex"
-                            ><code id="markdown-result" class="language-markdown"></code></pre>
-                        </div>
-                    </div>
-
-                    <div id="code_help" class=" ">
-                        <div class="tab-buttons flex gap-2">
-                            <button
-                                class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="curl"
-                            >
-                                cURL
-                            </button>
-                            <button
-                                class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="library"
-                            >
-                                Python Library
-                            </button>
-                            <button
-                                class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="python"
-                            >
-                                Python (Request)
-                            </button>
-                            <button
-                                class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                                data-tab="nodejs"
-                            >
-                                Node.js
-                            </button>
-                        </div>
-                        <div class="tab-content result bg-zinc-900 p-2 rounded h-full border border-zinc-700 text-sm">
-                            <pre class="h-full flex relative">
-                                <code id="curl-code" class="language-bash"></code>
-                                <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="curl-code">Copy</button>
-                            </pre>
-                            <pre class="hidden h-full flex relative">
-                                <code id="python-code" class="language-python"></code>
-                                <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="python-code">Copy</button>
-                            </pre>
-                            <pre class="hidden h-full flex relative">
-                                <code id="nodejs-code" class="language-javascript"></code>
-                                <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="nodejs-code">Copy</button>
-                            </pre>
-                            <pre class="hidden h-full flex relative">
-                                <code id="library-code" class="language-python"></code>
-                                <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="library-code">Copy</button>
-                            </pre>
-                        </div>
-                    </div>
-                </div>
-            </div>
-        </section>
-        <section class="bg-zinc-900 text-zinc-300 p-6 px-20">
-            <div class="grid grid-cols-2 gap-4 p-4 bg-zinc-900 text-lime-500">
-                <!-- Step 1 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🌟 <strong>Welcome to the Crawl4ai Quickstart Guide! Let's dive into some web crawling fun!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">
-                    First Step: Create an instance of WebCrawler and call the <code>warmup()</code> function.
-                </div>
-                <div>
-                    <pre><code class="language-python">crawler = WebCrawler()
-            crawler.warmup()</code></pre>
-                </div>
-
-                <!-- Step 2 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🧠 <strong>Understanding 'bypass_cache' and 'include_raw_html' parameters:</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">First crawl (caches the result):</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(url="https://www.nbcnews.com/business")</code></pre>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Second crawl (Force to crawl again):</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)</code></pre>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Crawl result without raw HTML content:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(url="https://www.nbcnews.com/business", include_raw_html=False)</code></pre>
-                </div>
-
-                <!-- Step 3 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    📄
-                    <strong
-                        >The 'include_raw_html' parameter, when set to True, includes the raw HTML content in the
-                        response. By default, it is set to True.</strong
-                    >
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Set <code>always_by_pass_cache</code> to True:</div>
-                <div>
-                    <pre><code class="language-python">crawler.always_by_pass_cache = True</code></pre>
-                </div>
-
-                <!-- Step 4 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🧩 <strong>Let's add a chunking strategy: RegexChunking!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using RegexChunking:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(
-                url="https://www.nbcnews.com/business",
-                chunking_strategy=RegexChunking(patterns=["\n\n"])
-            )</code></pre>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using NlpSentenceChunking:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(
-                url="https://www.nbcnews.com/business",
-                chunking_strategy=NlpSentenceChunking()
-            )</code></pre>
-                </div>
-
-                <!-- Step 5 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🧠 <strong>Let's get smarter with an extraction strategy: CosineStrategy!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using CosineStrategy:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(
-                url="https://www.nbcnews.com/business",
-                extraction_strategy=CosineStrategy(word_count_threshold=10, max_dist=0.2, linkage_method="ward", top_k=3)
-            )</code></pre>
-                </div>
-
-                <!-- Step 6 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🤖 <strong>Time to bring in the big guns: LLMExtractionStrategy without instructions!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using LLMExtractionStrategy without instructions:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(
-                url="https://www.nbcnews.com/business",
-                extraction_strategy=LLMExtractionStrategy(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY'))
-            )</code></pre>
-                </div>
-
-                <!-- Step 7 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    📜 <strong>Let's make it even more interesting: LLMExtractionStrategy with instructions!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using LLMExtractionStrategy with instructions:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(
-                url="https://www.nbcnews.com/business",
-                extraction_strategy=LLMExtractionStrategy(
-                    provider="openai/gpt-4o",
-                    api_token=os.getenv('OPENAI_API_KEY'),
-                    instruction="I am interested in only financial news"
-                )
-            )</code></pre>
-                </div>
-
-                <!-- Step 8 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🎯 <strong>Targeted extraction: Let's use a CSS selector to extract only H2 tags!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using CSS selector to extract H2 tags:</div>
-                <div>
-                    <pre><code class="language-python">result = crawler.run(
-                url="https://www.nbcnews.com/business",
-                css_selector="h2"
-            )</code></pre>
-                </div>
-
-                <!-- Step 9 -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🖱️ <strong>Let's get interactive: Passing JavaScript code to click 'Load More' button!</strong>
-                </div>
-                <div class="bg-zinc-800 p-2 rounded">Using JavaScript to click 'Load More' button:</div>
-                <div>
-                    <pre><code class="language-python">js_code = """
-            const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
-            loadMoreButton && loadMoreButton.click();
-            """
-            crawler_strategy = LocalSeleniumCrawlerStrategy(js_code=js_code)
-            crawler = WebCrawler(crawler_strategy=crawler_strategy, always_by_pass_cache=True)
-            result = crawler.run(url="https://www.nbcnews.com/business")</code></pre>
-                </div>
-
-                <!-- Conclusion -->
-                <div class="col-span-2 bg-yellow-500 p-2 rounded text-zinc-900">
-                    🎉
-                    <strong
-                        >Congratulations! You've made it through the Crawl4ai Quickstart Guide! Now go forth and crawl
-                        the web like a pro! 🕸️</strong
-                    >
-                </div>
-            </div>
-        </section>
-        <section class="bg-zinc-900 text-zinc-300 p-6 px-20">
-            <h1 class="text-3xl font-bold mb-4">Installation 💻</h1>
-            <p class="mb-4">
-                There are two ways to use Crawl4AI: as a library in your Python projects or as a standalone local
-                server.
-            </p>
-
-            <p class="mb-4">
-                You can also try Crawl4AI in a Google Colab
-                <a href="https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk"
-                    ><img
-                        src="https://colab.research.google.com/assets/colab-badge.svg"
-                        alt="Open In Colab"
-                        style="display: inline-block; width: 100px; height: 20px"
-                /></a>
-            </p>
-
-            <h2 class="text-2xl font-bold mb-2">Using Crawl4AI as a Library 📚</h2>
-            <p class="mb-4">To install Crawl4AI as a library, follow these steps:</p>
-
-            <ol class="list-decimal list-inside mb-4">
-                <li class="mb-2">
-                    Install the package from GitHub:
-                    <pre
-                        class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-                    ><code>pip install git+https://github.com/unclecode/crawl4ai.git</code></pre>
-                </li>
-                <li class="mb-2">
-                    Alternatively, you can clone the repository and install the package locally:
-                    <pre
-                        class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-                    ><code  class = "language-python bash">virtualenv venv
-source venv/bin/activate
-git clone https://github.com/unclecode/crawl4ai.git
-cd crawl4ai
-pip install -e .
-        </code></pre>
-                </li>
-                <li>
-                    Import the necessary modules in your Python script:
-                    <pre
-                        class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-                    ><code class = "language-python hljs">from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.chunking_strategy import *
-from crawl4ai.extraction_strategy import *
-import os
-
-crawler = WebCrawler()
-
-# Single page crawl
-single_url = UrlModel(url='https://www.nbcnews.com/business', forced=False)
-result = crawl4ai.fetch_page(
-    url='https://www.nbcnews.com/business',
-    word_count_threshold=5, # Minimum word count for a HTML tag to be considered as a worthy block
-    chunking_strategy= RegexChunking( patterns = ["\\n\\n"]), # Default is RegexChunking
-    extraction_strategy= CosineStrategy(word_count_threshold=10, max_dist=0.2, linkage_method='ward', top_k=3) # Default is CosineStrategy
-    # extraction_strategy= LLMExtractionStrategy(provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY')),
-    bypass_cache=False,
-    extract_blocks =True, # Whether to extract semantical blocks of text from the HTML
-    css_selector = "", # Eg: "div.article-body"
-    verbose=True,
-    include_raw_html=True, # Whether to include the raw HTML content in the response
-)
-print(result.model_dump())
-        </code></pre>
-                </li>
-            </ol>
-            <p class="mb-4">
-                For more information about how to run Crawl4AI as a local server, please refer to the
-                <a href="https://github.com/unclecode/crawl4ai" class="text-blue-400">GitHub repository</a>.
-            </p>
-            
-        </section>
-
-        <section class="bg-zinc-900 text-zinc-300 p-6 px-20">
-            <h1 class="text-3xl font-bold mb-4">📖 Parameters</h1>
-            <div class="overflow-x-auto">
-                <table class="min-w-full bg-zinc-800 border border-zinc-700">
-                    <thead>
-                        <tr>
-                            <th class="py-2 px-4 border-b border-zinc-700">Parameter</th>
-                            <th class="py-2 px-4 border-b border-zinc-700">Description</th>
-                            <th class="py-2 px-4 border-b border-zinc-700">Required</th>
-                            <th class="py-2 px-4 border-b border-zinc-700">Default Value</th>
-                        </tr>
-                    </thead>
-                    <tbody>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">urls</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                A list of URLs to crawl and extract data from.
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">Yes</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">-</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">include_raw_html</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                Whether to include the raw HTML content in the response.
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">false</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">bypass_cache</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                Whether to force a fresh crawl even if the URL has been previously crawled.
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">false</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">extract_blocks</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                Whether to extract semantical blocks of text from the HTML.
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">true</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">word_count_threshold</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                The minimum number of words a block must contain to be considered meaningful (minimum
-                                value is 5).
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">5</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">extraction_strategy</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                The strategy to use for extracting content from the HTML (e.g., "CosineStrategy").
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">CosineStrategy</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">chunking_strategy</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                The strategy to use for chunking the text before processing (e.g., "RegexChunking").
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">RegexChunking</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4 border-b border-zinc-700">css_selector</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">
-                                The CSS selector to target specific parts of the HTML for extraction.
-                            </td>
-                            <td class="py-2 px-4 border-b border-zinc-700">No</td>
-                            <td class="py-2 px-4 border-b border-zinc-700">None</td>
-                        </tr>
-                        <tr>
-                            <td class="py-2 px-4">verbose</td>
-                            <td class="py-2 px-4">Whether to enable verbose logging.</td>
-                            <td class="py-2 px-4">No</td>
-                            <td class="py-2 px-4">true</td>
-                        </tr>
-                    </tbody>
-                </table>
-            </div>
-        </section>
-
-        <section id="extraction" class="py-8 px-20">
-            <div class="overflow-x-auto mx-auto px-6">
-                <h2 class="text-2xl font-bold mb-4">Extraction Strategies</h2>
-                <div id="extraction-strategies" class="space-y-4"></div>
-            </div>
-        </section>
-
-        <section id="chunking" class="py-8 px-20">
-            <div class="overflow-x-auto mx-auto px-6">
-                <h2 class="text-2xl font-bold mb-4">Chunking Strategies</h2>
-                <div id="chunking-strategies" class="space-y-4"></div>
-            </div>
-        </section>
-
-        <section class="hero bg-zinc-900 py-8 px-20">
-            <div class="container mx-auto px-4">
-                <h2 class="text-3xl font-bold mb-4">🤔 Why building this?</h2>
-                <p class="text-lg mb-4">
-                    In recent times, we've witnessed a surge of startups emerging, riding the AI hype wave and charging
-                    for services that should rightfully be accessible to everyone. 🌍💸 One such example is scraping and
-                    crawling web pages and transforming them into a format suitable for Large Language Models (LLMs).
-                    🕸️🤖 We believe that building a business around this is not the right approach; instead, it should
-                    definitely be open-source. 🆓🌟 So, if you possess the skills to build such tools and share our
-                    philosophy, we invite you to join our "Robinhood" band and help set these products free for the
-                    benefit of all. 🤝💪
-                </p>
-            </div>
-        </section>
-
-        <section class="installation py-8 px-20">
-            <div class="container mx-auto px-4">
-                <h2 class="text-2xl font-bold mb-4">⚙️ Installation</h2>
-                <p class="mb-4">
-                    To install and run Crawl4AI as a library or a local server, please refer to the 📚
-                    <a href="https://github.com/unclecode/crawl4ai" class="text-blue-400">GitHub repository</a>.
-                </p>
-            </div>
-        </section>
-
-        <footer class="bg-zinc-900 text-white py-4">
-            <div class="container mx-auto px-4">
-                <div class="flex justify-between items-center">
-                    <p>© 2024 Crawl4AI. All rights reserved.</p>
-                    <div class="social-links">
-                        <a
-                            href="https://github.com/unclecode/crawl4ai"
-                            class="text-white hover:text-gray-300 mx-2"
-                            target="_blank"
-                            >😺 GitHub</a
-                        >
-                        <a
-                            href="https://twitter.com/unclecode"
-                            class="text-white hover:text-gray-300 mx-2"
-                            target="_blank"
-                            >🐦 Twitter</a
-                        >
-                    </div>
-                </div>
-            </div>
-        </footer>
-
-        <script>
-            // JavaScript to manage dynamic form changes and logic
-            document.getElementById("extraction-strategy-select").addEventListener("change", function () {
-                const strategy = this.value;
-                const providerModelSelect = document.getElementById("provider-model-select");
-                const tokenInput = document.getElementById("token-input");
-
-                if (strategy === "LLMExtractionStrategy") {
-                    providerModelSelect.disabled = false;
-                    tokenInput.disabled = false;
-                } else {
-                    providerModelSelect.disabled = true;
-                    tokenInput.disabled = true;
-                }
-            });
-
-            // Get the selected provider model and token from local storage
-            const storedProviderModel = localStorage.getItem("provider_model");
-            const storedToken = localStorage.getItem(storedProviderModel);
-
-            if (storedProviderModel) {
-                document.getElementById("provider-model-select").value = storedProviderModel;
-            }
-
-            if (storedToken) {
-                document.getElementById("token-input").value = storedToken;
-            }
-
-            // Handle provider model dropdown change
-            document.getElementById("provider-model-select").addEventListener("change", () => {
-                const selectedProviderModel = document.getElementById("provider-model-select").value;
-                const storedToken = localStorage.getItem(selectedProviderModel);
-
-                if (storedToken) {
-                    document.getElementById("token-input").value = storedToken;
-                } else {
-                    document.getElementById("token-input").value = "";
-                }
-            });
-
-            // Fetch total count from the database
-            axios
-                .get("/total-count")
-                .then((response) => {
-                    document.getElementById("total-count").textContent = response.data.count;
-                })
-                .catch((error) => console.error(error));
-
-            // Handle crawl button click
-            document.getElementById("crawl-btn").addEventListener("click", () => {
-                // validate input to have both URL and API token
-                if (!document.getElementById("url-input").value || !document.getElementById("token-input").value) {
-                    alert("Please enter both URL(s) and API token.");
-                    return;
-                }
-
-                const selectedProviderModel = document.getElementById("provider-model-select").value;
-                const apiToken = document.getElementById("token-input").value;
-                const extractBlocks = document.getElementById("extract-blocks-checkbox").checked;
-                const bypassCache = document.getElementById("bypass-cache-checkbox").checked;
-
-                // Save the selected provider model and token to local storage
-                localStorage.setItem("provider_model", selectedProviderModel);
-                localStorage.setItem(selectedProviderModel, apiToken);
-
-                const urlsInput = document.getElementById("url-input").value;
-                const urls = urlsInput.split(",").map((url) => url.trim());
-                const data = {
-                    urls: urls,
-                    provider_model: selectedProviderModel,
-                    api_token: apiToken,
-                    include_raw_html: true,
-                    bypass_cache: bypassCache,
-                    extract_blocks: extractBlocks,
-                    word_count_threshold: parseInt(document.getElementById("threshold").value),
-                    extraction_strategy: document.getElementById("extraction-strategy-select").value,
-                    chunking_strategy: document.getElementById("chunking-strategy-select").value,
-                    css_selector: document.getElementById("css-selector").value,
-                    verbose: true,
-                };
-
-                // save api token to local storage
-                localStorage.setItem("api_token", document.getElementById("token-input").value);
-
-                document.getElementById("loading").classList.remove("hidden");
-                //document.getElementById("result").classList.add("hidden");
-                //document.getElementById("code_help").classList.add("hidden");
-
-                axios
-                    .post("/crawl", data)
-                    .then((response) => {
-                        const result = response.data.results[0];
-                        const parsedJson = JSON.parse(result.extracted_content);
-                        document.getElementById("json-result").textContent = JSON.stringify(parsedJson, null, 2);
-                        document.getElementById("cleaned-html-result").textContent = result.cleaned_html;
-                        document.getElementById("markdown-result").textContent = result.markdown;
-
-                        // Update code examples dynamically
-                        const extractionStrategy = data.extraction_strategy;
-                        const isLLMExtraction = extractionStrategy === "LLMExtractionStrategy";
-
-                        document.getElementById(
-                            "curl-code"
-                        ).textContent = `curl -X POST -H "Content-Type: application/json" -d '${JSON.stringify({
-                            ...data,
-                            api_token: isLLMExtraction ? "your_api_token" : undefined,
-                        })}' http://crawl4ai.uccode.io/crawl`;
-
-                        document.getElementById(
-                            "python-code"
-                        ).textContent = `import requests\n\ndata = ${JSON.stringify(
-                            { ...data, api_token: isLLMExtraction ? "your_api_token" : undefined },
-                            null,
-                            2
-                        )}\n\nresponse = requests.post("http://crawl4ai.uccode.io/crawl", json=data) # OR local host if your run locally \nprint(response.json())`;
-
-                        document.getElementById(
-                            "nodejs-code"
-                        ).textContent = `const axios = require('axios');\n\nconst data = ${JSON.stringify(
-                            { ...data, api_token: isLLMExtraction ? "your_api_token" : undefined },
-                            null,
-                            2
-                        )};\n\naxios.post("http://crawl4ai.uccode.io/crawl", data) // OR local host if your run locally \n    .then(response => console.log(response.data))\n    .catch(error => console.error(error));`;
-
-                        document.getElementById(
-                            "library-code"
-                        ).textContent = `from crawl4ai.web_crawler import WebCrawler\nfrom crawl4ai.extraction_strategy import *\nfrom crawl4ai.chunking_strategy import *\n\ncrawler = WebCrawler()\ncrawler.warmup()\n\nresult = crawler.run(\n    url='${
-                            urls[0]
-                        }',\n    word_count_threshold=${data.word_count_threshold},\n    extraction_strategy=${
-                            isLLMExtraction
-                                ? `${extractionStrategy}(provider="${data.provider_model}", api_token="${data.api_token}")`
-                                : extractionStrategy + "()"
-                        },\n    chunking_strategy=${data.chunking_strategy}(),\n    bypass_cache=${
-                            data.bypass_cache
-                        },\n    css_selector="${data.css_selector}"\n)\nprint(result)`;
-
-                        // Highlight code syntax
-                        hljs.highlightAll();
-
-                        // Select JSON tab by default
-                        document.querySelector('.tab-btn[data-tab="json"]').click();
-
-                        document.getElementById("loading").classList.add("hidden");
-                        document.getElementById("result").classList.remove("hidden");
-                        document.getElementById("code_help").classList.remove("hidden");
-
-                        // increment the total count
-                        document.getElementById("total-count").textContent =
-                            parseInt(document.getElementById("total-count").textContent) + 1;
-                    })
-                    .catch((error) => {
-                        console.error(error);
-                        document.getElementById("loading").classList.add("hidden");
-                    });
-            });
-
-            // Handle tab clicks
-            document.querySelectorAll(".tab-btn").forEach((btn) => {
-                btn.addEventListener("click", () => {
-                    const tab = btn.dataset.tab;
-                    document
-                        .querySelectorAll(".tab-btn")
-                        .forEach((b) => b.classList.remove("bg-lime-700", "text-white"));
-                    btn.classList.add("bg-lime-700", "text-white");
-                    document.querySelectorAll(".tab-content.code pre").forEach((el) => el.classList.add("hidden"));
-                    document.getElementById(`${tab}-result`).parentElement.classList.remove("hidden");
-                });
-            });
-
-            // Handle code tab clicks
-            document.querySelectorAll(".code-tab-btn").forEach((btn) => {
-                btn.addEventListener("click", () => {
-                    const tab = btn.dataset.tab;
-                    document
-                        .querySelectorAll(".code-tab-btn")
-                        .forEach((b) => b.classList.remove("bg-lime-700", "text-white"));
-                    btn.classList.add("bg-lime-700", "text-white");
-                    document.querySelectorAll(".tab-content.result pre").forEach((el) => el.classList.add("hidden"));
-                    document.getElementById(`${tab}-code`).parentElement.classList.remove("hidden");
-                });
-            });
-
-            // Handle copy to clipboard button clicks
-
-            async function copyToClipboard(text) {
-                if (navigator.clipboard && navigator.clipboard.writeText) {
-                    return navigator.clipboard.writeText(text);
-                } else {
-                    return fallbackCopyTextToClipboard(text);
-                }
-            }
-
-            function fallbackCopyTextToClipboard(text) {
-                return new Promise((resolve, reject) => {
-                    const textArea = document.createElement("textarea");
-                    textArea.value = text;
-
-                    // Avoid scrolling to bottom
-                    textArea.style.top = "0";
-                    textArea.style.left = "0";
-                    textArea.style.position = "fixed";
-
-                    document.body.appendChild(textArea);
-                    textArea.focus();
-                    textArea.select();
-
-                    try {
-                        const successful = document.execCommand("copy");
-                        if (successful) {
-                            resolve();
-                        } else {
-                            reject();
-                        }
-                    } catch (err) {
-                        reject(err);
-                    }
-
-                    document.body.removeChild(textArea);
-                });
-            }
-
-            document.querySelectorAll(".copy-btn").forEach((btn) => {
-                btn.addEventListener("click", () => {
-                    const target = btn.dataset.target;
-                    const code = document.getElementById(target).textContent;
-                    //navigator.clipboard.writeText(code).then(() => {
-                    copyToClipboard(code).then(() => {
-                        btn.textContent = "Copied!";
-                        setTimeout(() => {
-                            btn.textContent = "Copy";
-                        }, 2000);
-                    });
-                });
-            });
-
-            document.addEventListener("DOMContentLoaded", async () => {
-                try {
-                    const extractionResponse = await fetch("/strategies/extraction");
-                    const extractionStrategies = await extractionResponse.json();
-
-                    const chunkingResponse = await fetch("/strategies/chunking");
-                    const chunkingStrategies = await chunkingResponse.json();
-
-                    renderStrategies("extraction-strategies", extractionStrategies);
-                    renderStrategies("chunking-strategies", chunkingStrategies);
-                } catch (error) {
-                    console.error("Error fetching strategies:", error);
-                }
-            });
-
-            function renderStrategies(containerId, strategies) {
-                const container = document.getElementById(containerId);
-                container.innerHTML = ""; // Clear any existing content
-                strategies = JSON.parse(strategies);
-                Object.entries(strategies).forEach(([strategy, description]) => {
-                    const strategyElement = document.createElement("div");
-                    strategyElement.classList.add("bg-zinc-800", "p-4", "rounded", "shadow-md", "docs-item");
-
-                    const strategyDescription = document.createElement("div");
-                    strategyDescription.classList.add("text-gray-300", "prose", "prose-sm");
-                    strategyDescription.innerHTML = marked.parse(description);
-
-                    strategyElement.appendChild(strategyDescription);
-
-                    container.appendChild(strategyElement);
-                });
-            }
-
-            // Highlight code syntax
-            hljs.highlightAll();
-        </script>
-    </body>
-</html>

pages/index.html

@@ -1,73 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-    <head>
-        <meta charset="UTF-8" />
-        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-        <title>Crawl4AI</title>
-
-        <link rel="preconnect" href="https://fonts.googleapis.com" />
-        <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
-        <link href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap" rel="stylesheet" />
-
-        <!-- <link href="https://cdn.jsdelivr.net/npm/tailwindcss@3.4.3/dist/tailwind.min.css" rel="stylesheet" /> -->
-        <script src="https://cdn.tailwindcss.com"></script>
-        <script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
-        <link rel="stylesheet" href="/pages/app.css" />
-        <link
-            rel="stylesheet"
-            href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/styles/monokai.min.css"
-        />
-        <script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
-
-        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/highlight.min.js"></script>
-    </head>
-    <body class="bg-black text-gray-200">
-        <header class="bg-zinc-950 text-lime-500 py-4 flex">
-            
-            <div class="mx-auto px-4">
-                <h1 class="text-2xl font-bold">🔥🕷️ Crawl4AI: Web Data for your Thoughts</h1>
-            </div>
-            <div class="mx-auto px-4 flex font-bold text-xl gap-2">
-                <span>📊 Total Website Processed</span>
-                <span id="total-count" class="text-lime-400">2</span>
-            </div>
-        </header>
-        
-            {{ try_it | safe }}
-
-            <div class="mx-auto p-4 bg-zinc-950 text-lime-500 min-h-screen">
-                <div class="container mx-auto">
-                <div class="flex h-full px-20">
-                    <div class="sidebar w-1/4 p-4">
-                        <h2 class="text-lg font-bold mb-4">Outline</h2>
-                        <ul>
-                            <li class="mb-2"><a href="#" data-target="installation">Installation</a></li>
-                            <li class="mb-2"><a href="#" data-target="how-to-guide">How to Guide</a></li>
-                            <li class="mb-2"><a href="#" data-target="chunking-strategies">Chunking Strategies</a></li>
-                            <li class="mb-2">
-                                <a href="#" data-target="extraction-strategies">Extraction Strategies</a>
-                            </li>
-                        </ul>
-                    </div>
-
-                    <!-- Main Content -->
-                    <div class="w-3/4 p-4">
-                        {{installation | safe}} {{how_to_guide | safe}}
-
-                        <section id="chunking-strategies" class="content-section">
-                            <h1 class="text-2xl font-bold">Chunking Strategies</h1>
-                            <p>Content for chunking strategies...</p>
-                        </section>
-                        <section id="extraction-strategies" class="content-section">
-                            <h1 class="text-2xl font-bold">Extraction Strategies</h1>
-                            <p>Content for extraction strategies...</p>
-                        </section>
-                    </div>
-                </div>
-            </div>
-            </div>
-        
-        {{ footer | safe }}
-        <script script src="/pages/app.js"></script>
-    </body>
-</html>

pages/index_pooling.html

@@ -1,425 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-    <head>
-        <meta charset="UTF-8" />
-        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-        <title>Crawl4AI</title>
-
-        <link rel="preconnect" href="https://fonts.googleapis.com" />
-        <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
-        <link href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap" rel="stylesheet" />
-
-        <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet" />
-        <script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
-        <link
-            rel="stylesheet"
-            href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/styles/vs2015.min.css"
-        />
-        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/highlight.min.js"></script>
-        <style>
-            :root {
-                --ifm-font-size-base: 100%;
-                --ifm-line-height-base: 1.65;
-                --ifm-font-family-base: system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans,
-                    sans-serif, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji",
-                    "Segoe UI Emoji", "Segoe UI Symbol";
-            }
-            html {
-                -webkit-font-smoothing: antialiased;
-                -webkit-text-size-adjust: 100%;
-                text-size-adjust: 100%;
-                font: var(--ifm-font-size-base) / var(--ifm-line-height-base) var(--ifm-font-family-base);
-            }
-            body {
-                background-color: #1a202c;
-                color: #fff;
-            }
-            .tab-content {
-                max-height: 400px;
-                overflow: auto;
-            }
-            pre {
-                white-space: pre-wrap;
-                font-size: 14px;
-            }
-            pre code {
-                width: 100%;
-            }
-        </style>
-    </head>
-    <body>
-        <header class="bg-gray-900 text-white py-4">
-            <div class="container mx-auto px-4">
-                <h1 class="text-2xl font-bold">🔥🕷️ Crawl4AI: Open-source LLM Friendly Web scraper</h1>
-            </div>
-        </header>
-
-        <section class="try-it py-8 pb-20">
-            <div class="container mx-auto px-4">
-                <h2 class="text-2xl font-bold mb-4">Try It Now</h2>
-                <div class="mb-4 flex w-full gap-2">
-                    <input
-                        type="text"
-                        id="url-input"
-                        value="https://kidocode.com"
-                        class="border border-gray-600 rounded px-4 py-2 flex-grow bg-gray-800 text-white"
-                        placeholder="Enter URL(s) separated by commas"
-                    />
-                    <select
-                        id="provider-model-select"
-                        class="border border-gray-600 rounded px-4 py-2 bg-gray-800 text-white"
-                    >
-                        <!-- Add your option values here -->
-                        <option value="groq/llama3-70b-8192">groq/llama3-70b-8192</option>
-                        <option value="groq/llama3-8b-8192">groq/llama3-8b-8192</option>
-                        <option value="openai/gpt-4-turbo">gpt-4-turbo</option>
-                        <option value="openai/gpt-3.5-turbo">gpt-3.5-turbo</option>
-                        <option value="anthropic/claude-3-haiku-20240307">claude-3-haiku</option>
-                        <option value="anthropic/claude-3-opus-20240229">claude-3-opus</option>
-                        <option value="anthropic/claude-3-sonnet-20240229">claude-3-sonnet</option>
-                    </select>
-                    <input
-                        type="password"
-                        id="token-input"
-                        class="border border-gray-600 rounded px-4 py-2 flex-grow bg-gray-800 text-white"
-                        placeholder="Enter Groq API token"
-                    />
-                    <div class="flex items-center justify-center">
-                        <input type="checkbox" id="extract-blocks-checkbox" class="mr-2" checked>
-                        <label for="extract-blocks-checkbox" class="text-white">Extract Blocks</label>
-                    </div>
-                    <button id="crawl-btn" class="bg-blue-600 text-white px-4 py-2 rounded">Crawl</button>
-                </div>
-                <div class="grid grid-cols-1 md:grid-cols-2 gap-8">
-                    <div id="loading" class="hidden mt-4">
-                        <p>Loading...</p>
-                    </div>
-                    <div id="result" class="tab-container flex-1 h-full flex-col">
-                        <div class="tab-buttons flex gap-2">
-                            <button class="tab-btn px-4 py-2 bg-gray-700 rounded-t" data-tab="json">JSON</button>
-                            <button class="tab-btn px-4 py-2 bg-gray-700 rounded-t" data-tab="cleaned-html">
-                                Cleaned HTML
-                            </button>
-                            <button class="tab-btn px-4 py-2 bg-gray-700 rounded-t" data-tab="markdown">
-                                Markdown
-                            </button>
-                        </div>
-                        <div class="tab-content code bg-gray-800 p-2 rounded  h-full  flex-1 border border-gray-600">
-                            <pre class="h-full flex"><code id="json-result" class="language-json "></code></pre>
-                            <pre
-                                class="hidden h-full flex"
-                            ><code id="cleaned-html-result" class="language-html "></code></pre>
-                            <pre
-                                class="hidden h-full flex"
-                            ><code id="markdown-result" class="language-markdown "></code></pre>
-                        </div>
-                    </div>
-                    <div id="code_help" class="tab-container flex-1 h-full">
-                        <div class="tab-buttons flex gap-2">
-                            <button class="code-tab-btn px-4 py-2 bg-gray-700 rounded-t" data-tab="curl">cURL</button>
-                            <button class="code-tab-btn px-4 py-2 bg-gray-700 rounded-t" data-tab="python">
-                                Python
-                            </button>
-                            <button class="code-tab-btn px-4 py-2 bg-gray-700 rounded-t" data-tab="nodejs">
-                                Node.js
-                            </button>
-                        </div>
-                        <div class="tab-content result bg-gray-800 p-2 rounded h-full  flex-1 border border-gray-600">
-                            <pre class="h-full flex relative">
-                                    <code id="curl-code" class="language-bash"></code>
-                                    <button class="absolute top-2 right-2 bg-gray-700 text-white px-2 py-1 rounded copy-btn" data-target="curl-code">Copy</button>
-                                </pre>
-                            <pre class="hidden h-full flex relative">
-                                    <code id="python-code" class="language-python"></code>
-                                    <button class="absolute top-2 right-2 bg-gray-700 text-white px-2 py-1 rounded copy-btn" data-target="python-code">Copy</button>
-                                </pre>
-                            <pre class="hidden h-full flex relative">
-                                    <code id="nodejs-code" class="language-javascript"></code>
-                                    <button class="absolute top-2 right-2 bg-gray-700 text-white px-2 py-1 rounded copy-btn" data-target="nodejs-code">Copy</button>
-                                </pre>
-                        </div>
-                    </div>
-                </div>
-            </div>
-        </section>
-
-        <section class="hero bg-gray-900 py-8">
-            <div class="container mx-auto px-4">
-                <h2 class="text-3xl font-bold mb-4">🤔 Why building this?</h2>
-                <p class="text-lg mb-4">
-                    In recent times, we've seen numerous startups emerging, riding the AI hype wave and charging for
-                    services that should rightfully be accessible to everyone. 🌍💸 One for example is to scrap and crawl 
-                    a web page, and transform it o a form suitable for LLM. We don't think one should build a business
-                    out of this, but definilty should be opened source. So if you possess the skills to build such things 
-                    and you have such philosphy you should join our "Robinhood" band and help set
-                    these products free. 🆓🤝
-                </p>
-            </div>
-        </section>
-
-        <section class="installation py-8">
-            <div class="container mx-auto px-4">
-                <h2 class="text-2xl font-bold mb-4">⚙️ Installation</h2>
-                <p class="mb-4">
-                    To install and run Crawl4AI locally or on your own service, the best way is to use Docker. 🐳 Follow
-                    these steps:
-                </p>
-                <ol class="list-decimal list-inside mb-4">
-                    <li>
-                        Clone the GitHub repository: 📥
-                        <code>git clone https://github.com/unclecode/crawl4ai.git</code>
-                    </li>
-                    <li>Navigate to the project directory: 📂 <code>cd crawl4ai</code></li>
-                    <li>
-                        Build the Docker image: 🛠️ <code>docker build -t crawl4ai .</code> On Mac, follow: 🍎
-                        <code>docker build --platform linux/amd64 -t crawl4ai .</code>
-                    </li>
-                    <li>Run the Docker container: ▶️ <code>docker run -p 8000:80 crawl4ai</code></li>
-                </ol>
-                <p>
-                    For more detailed instructions and advanced configuration options, please refer to the 📚
-                    <a href="https://github.com/unclecode/crawl4ai" class="text-blue-400">GitHub repository</a>.
-                </p>
-            </div>
-        </section>
-
-        <footer class="bg-gray-900 text-white py-4">
-            <div class="container mx-auto px-4">
-                <div class="flex justify-between items-center">
-                    <p>© 2024 Crawl4AI. All rights reserved.</p>
-                    <div class="social-links">
-                        <a
-                            href="https://github.com/unclecode/crawl4ai"
-                            class="text-white hover:text-gray-300 mx-2"
-                            target="_blank"
-                            >😺 GitHub</a
-                        >
-                        <a
-                            href="https://twitter.com/unclecode"
-                            class="text-white hover:text-gray-300 mx-2"
-                            target="_blank"
-                            >🐦 Twitter</a
-                        >
-                        <a
-                            href="https://discord.gg/your-invite-link"
-                            class="text-white hover:text-gray-300 mx-2"
-                            target="_blank"
-                            >💬 Discord</a
-                        >
-                    </div>
-                </div>
-            </div>
-        </footer>
-
-        <script>
-            // Get the selected provider model and token from local storage
-            const storedProviderModel = localStorage.getItem("provider_model");
-            const storedToken = localStorage.getItem(storedProviderModel);
-
-            if (storedProviderModel) {
-                document.getElementById("provider-model-select").value = storedProviderModel;
-            }
-
-            if (storedToken) {
-                document.getElementById("token-input").value = storedToken;
-            }
-
-            // Handle provider model dropdown change
-            document.getElementById("provider-model-select").addEventListener("change", () => {
-                const selectedProviderModel = document.getElementById("provider-model-select").value;
-                const storedToken = localStorage.getItem(selectedProviderModel);
-
-                if (storedToken) {
-                    document.getElementById("token-input").value = storedToken;
-                } else {
-                    document.getElementById("token-input").value = "";
-                }
-            });
-
-            // Fetch total count from the database
-            axios
-                .get("/total-count")
-                .then((response) => {
-                    document.getElementById("total-count").textContent = response.data.count;
-                })
-                .catch((error) => console.error(error));
-
-            // Handle crawl button click
-            document.getElementById("crawl-btn").addEventListener("click", () => {
-                // validate input to have both URL and API token
-                if (!document.getElementById("url-input").value || !document.getElementById("token-input").value) {
-                    alert("Please enter both URL(s) and API token.");
-                    return;
-                }
-
-                const selectedProviderModel = document.getElementById("provider-model-select").value;
-                const apiToken = document.getElementById("token-input").value;
-                const extractBlocks = document.getElementById("extract-blocks-checkbox").checked;
-
-
-                // Save the selected provider model and token to local storage
-                localStorage.setItem("provider_model", selectedProviderModel);
-                localStorage.setItem(selectedProviderModel, apiToken);
-
-                const urlsInput = document.getElementById("url-input").value;
-                const urls = urlsInput.split(",").map((url) => url.trim());
-                const data = {
-                    urls: urls,
-                    provider_model: selectedProviderModel,
-                    api_token: apiToken,
-                    include_raw_html: true,
-                    forced: false,
-                    extract_blocks: extractBlocks,
-                };
-
-                // save api token to local storage
-                localStorage.setItem("api_token", document.getElementById("token-input").value);
-
-                document.getElementById("loading").classList.remove("hidden");
-                document.getElementById("result").classList.add("hidden");
-                document.getElementById("code_help").classList.add("hidden");
-
-                axios
-                    .post("/crawl", data)
-                    .then((response) => {
-                        const result = response.data.results[0];
-                        const parsedJson = JSON.parse(result.extracted_content);
-                        document.getElementById("json-result").textContent = JSON.stringify(parsedJson, null, 2);
-                        document.getElementById("cleaned-html-result").textContent = result.cleaned_html;
-                        document.getElementById("markdown-result").textContent = result.markdown;
-
-                        // Update code examples dynamically
-                        // Update code examples dynamically
-                        document.getElementById(
-                            "curl-code"
-                        ).textContent = `curl -X POST -H "Content-Type: application/json" -d '${JSON.stringify({
-                            ...data,
-                            api_token: "your_api_token",
-                        })}' http://localhost:8000/crawl`;
-
-                        document.getElementById(
-                            "python-code"
-                        ).textContent = `import requests\n\ndata = ${JSON.stringify(
-                            { ...data, api_token: "your_api_token" },
-                            null,
-                            2
-                        )}\n\nresponse = requests.post("http://localhost:8000/crawl", json=data)\nprint(response.json())`;
-
-                        document.getElementById(
-                            "nodejs-code"
-                        ).textContent = `const axios = require('axios');\n\nconst data = ${JSON.stringify(
-                            { ...data, api_token: "your_api_token" },
-                            null,
-                            2
-                        )};\n\naxios.post("http://localhost:8000/crawl", data)\n    .then(response => console.log(response.data))\n    .catch(error => console.error(error));`;
-                        // Highlight code syntax
-                        hljs.highlightAll();
-
-                        // Select JSON tab by default
-                        document.querySelector('.tab-btn[data-tab="json"]').click();
-
-                        document.getElementById("loading").classList.add("hidden");
-                        document.getElementById("result").classList.remove("hidden");
-                        document.getElementById("code_help").classList.remove("hidden");
-                    })
-                    .catch((error) => {
-                        console.error(error);
-                        document.getElementById("loading").classList.add("hidden");
-                    });
-            });
-
-            // Handle tab clicks
-            document.querySelectorAll(".tab-btn").forEach((btn) => {
-                btn.addEventListener("click", () => {
-                    const tab = btn.dataset.tab;
-                    document
-                        .querySelectorAll(".tab-btn")
-                        .forEach((b) => b.classList.remove("bg-blue-600", "text-white"));
-                    btn.classList.add("bg-blue-600", "text-white");
-                    document.querySelectorAll(".tab-content.code pre").forEach((el) => el.classList.add("hidden"));
-                    document.getElementById(`${tab}-result`).parentElement.classList.remove("hidden");
-                });
-            });
-
-            // Handle code tab clicks
-            document.querySelectorAll(".code-tab-btn").forEach((btn) => {
-                btn.addEventListener("click", () => {
-                    const tab = btn.dataset.tab;
-                    document
-                        .querySelectorAll(".code-tab-btn")
-                        .forEach((b) => b.classList.remove("bg-blue-600", "text-white"));
-                    btn.classList.add("bg-blue-600", "text-white");
-                    document.querySelectorAll(".tab-content.result pre").forEach((el) => el.classList.add("hidden"));
-                    document.getElementById(`${tab}-code`).parentElement.classList.remove("hidden");
-                });
-            });
-
-            // Handle copy to clipboard button clicks
-            document.querySelectorAll(".copy-btn").forEach((btn) => {
-                btn.addEventListener("click", () => {
-                    const target = btn.dataset.target;
-                    const code = document.getElementById(target).textContent;
-                    navigator.clipboard.writeText(code).then(() => {
-                        btn.textContent = "Copied!";
-                        setTimeout(() => {
-                            btn.textContent = "Copy";
-                        }, 2000);
-                    });
-                });
-            });
-
-
-            document.getElementById("crawl-btn").addEventListener("click", () => {
-                const urlsInput = document.getElementById("url-input").value;
-                const urls = urlsInput.split(",").map(url => url.trim());
-                const apiToken = document.getElementById("token-input").value;
-                const selectedProviderModel = document.getElementById("provider-model-select").value;
-                const extractBlocks = document.getElementById("extract-blocks-checkbox").checked;
-            
-                const data = {
-                    urls: urls,
-                    provider_model: selectedProviderModel,
-                    api_token: apiToken,
-                    include_raw_html: true,
-                    forced: false,
-                    extract_blocks: extractBlocks
-                };
-            
-                localStorage.setItem("api_token", apiToken);
-            
-                document.getElementById("loading").classList.remove("hidden");
-                document.getElementById("result").classList.add("hidden");
-                document.getElementById("code_help").classList.add("hidden");
-            
-                axios.post("/crawl", data)
-                    .then(response => {
-                        const taskId = response.data.task_id;
-                        pollTaskStatus(taskId);
-                    })
-                    .catch(error => {
-                        console.error('Error during fetch:', error);
-                        document.getElementById("loading").classList.add("hidden");
-                    });
-            });
-            
-            function pollTaskStatus(taskId) {
-                axios.get(`/task/${taskId}`)
-                    .then(response => {
-                        const task = response.data;
-                        if (task.status === 'done') {
-                            displayResults(task.results[0]);
-                        } else if (task.status === 'pending') {
-                            setTimeout(() => pollTaskStatus(taskId), 2000);  // Poll every 2 seconds
-                        } else {
-                            console.error('Task failed:', task.error);
-                            document.getElementById("loading").classList.add("hidden");
-                        }
-                    })
-                    .catch(error => {
-                        console.error('Error polling task status:', error);
-                        document.getElementById("loading").classList.add("hidden");
-                    });
-            }
-        </script>
-    </body>
-</html>

pages/partial/footer.html

@@ -1,36 +0,0 @@
-<section class="hero bg-zinc-900 py-8 px-20 text-zinc-400">
-    <div class="container mx-auto px-4">
-        <h2 class="text-3xl font-bold mb-4">🤔 Why building this?</h2>
-        <p class="text-lg mb-4">
-            In recent times, we've witnessed a surge of startups emerging, riding the AI hype wave and charging
-            for services that should rightfully be accessible to everyone. 🌍💸 One such example is scraping and
-            crawling web pages and transforming them into a format suitable for Large Language Models (LLMs).
-            🕸️🤖 We believe that building a business around this is not the right approach; instead, it should
-            definitely be open-source. 🆓🌟 So, if you possess the skills to build such tools and share our
-            philosophy, we invite you to join our "Robinhood" band and help set these products free for the
-            benefit of all. 🤝💪
-        </p>
-    </div>
-</section>
-
-<footer class="bg-zinc-900 text-zinc-400 py-4">
-    <div class="container mx-auto px-4">
-        <div class="flex justify-between items-center">
-            <p>© 2024 Crawl4AI. All rights reserved.</p>
-            <div class="social-links">
-                <a
-                    href="https://github.com/unclecode/crawl4ai"
-                    class="text-zinc-400 hover:text-gray-300 mx-2"
-                    target="_blank"
-                    >😺 GitHub</a
-                >
-                <a
-                    href="https://twitter.com/unclecode"
-                    class="text-zinc-400 hover:text-gray-300 mx-2"
-                    target="_blank"
-                    >🐦 Twitter</a
-                >
-            </div>
-        </div>
-    </div>
-</footer>

pages/partial/how_to_guide.html

@@ -1,174 +0,0 @@
-<section id="how-to-guide" class="content-section">
-    <h1 class="text-2xl font-bold">How to Guide</h1>
-    <div class="flex flex-col gap-4 p-4 bg-zinc-900 text-lime-500">
-        <!-- Step 1 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🌟
-            <strong
-                >Welcome to the Crawl4ai Quickstart Guide! Let's dive into some web crawling
-                fun!</strong
-            >
-        </div>
-        <div class="">
-            First Step: Create an instance of WebCrawler and call the
-            <code>warmup()</code> function.
-        </div>
-        <div>
-            <pre><code class="language-python">crawler = WebCrawler()
-crawler.warmup()</code></pre>
-        </div>
-
-        <!-- Step 2 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🧠 <strong>Understanding 'bypass_cache' and 'include_raw_html' parameters:</strong>
-        </div>
-        <div class="">First crawl (caches the result):</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(url="https://www.nbcnews.com/business")</code></pre>
-        </div>
-        <div class="">Second crawl (Force to crawl again):</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)</code></pre>
-            <div class="bg-red-900 p-2 text-zinc-50">
-                ⚠️ Don't forget to set <code>`bypass_cache`</code> to True if you want to try different strategies for the same URL. Otherwise, the cached result will be returned. You can also set <code>`always_by_pass_cache`</code> in constructor to True to always bypass the cache.
-            </div>
-        </div>
-        <div class="">Crawl result without raw HTML content:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(url="https://www.nbcnews.com/business", include_raw_html=False)</code></pre>
-        </div>
-
-        <!-- Step 3 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            📄
-            <strong
-                >The 'include_raw_html' parameter, when set to True, includes the raw HTML content
-                in the response. By default, it is set to True.</strong
-            >
-        </div>
-        <div class="">Set <code>always_by_pass_cache</code> to True:</div>
-        <div>
-            <pre><code class="language-python">crawler.always_by_pass_cache = True</code></pre>
-        </div>
-        <!-- Step 3.5 Screenshot -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            📸
-            <strong>Let's take a screenshot of the page!</strong>
-        </div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    screenshot=True
-)
-with open("screenshot.png", "wb") as f:
-    f.write(base64.b64decode(result.screenshot))</code></pre>
-        </div>
-
-
-        <!-- Step 4 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🧩 <strong>Let's add a chunking strategy: RegexChunking!</strong>
-        </div>
-        <div class="">Using RegexChunking:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    chunking_strategy=RegexChunking(patterns=["\n\n"])
-)</code></pre>
-        </div>
-        <div class="">Using NlpSentenceChunking:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    chunking_strategy=NlpSentenceChunking()
-)</code></pre>
-        </div>
-
-        <!-- Step 5 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🧠 <strong>Let's get smarter with an extraction strategy: CosineStrategy!</strong>
-        </div>
-        <div class="">Using CosineStrategy:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=CosineStrategy(word_count_threshold=10, max_dist=0.2, linkage_method="ward", top_k=3)
-)</code></pre>
-        </div>
-
-        <!-- Step 6 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🤖
-            <strong
-                >Time to bring in the big guns: LLMExtractionStrategy without instructions!</strong
-            >
-        </div>
-        <div class="">Using LLMExtractionStrategy without instructions:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=LLMExtractionStrategy(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY'))
-)</code></pre>
-        </div>
-
-        <!-- Step 7 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            📜
-            <strong
-                >Let's make it even more interesting: LLMExtractionStrategy with
-                instructions!</strong
-            >
-        </div>
-        <div class="">Using LLMExtractionStrategy with instructions:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=LLMExtractionStrategy(
-    provider="openai/gpt-4o",
-    api_token=os.getenv('OPENAI_API_KEY'),
-    instruction="I am interested in only financial news"
-)
-)</code></pre>
-        </div>
-
-        <!-- Step 8 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🎯
-            <strong>Targeted extraction: Let's use a CSS selector to extract only H2 tags!</strong>
-        </div>
-        <div class="">Using CSS selector to extract H2 tags:</div>
-        <div>
-            <pre><code class="language-python">result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    css_selector="h2"
-)</code></pre>
-        </div>
-
-        <!-- Step 9 -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🖱️
-            <strong
-                >Let's get interactive: Passing JavaScript code to click 'Load More' button!</strong
-            >
-        </div>
-        <div class="">Using JavaScript to click 'Load More' button:</div>
-        <div>
-            <pre><code class="language-python">js_code = ["""
-const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
-loadMoreButton && loadMoreButton.click();
-"""]
-crawler = WebCrawler(verbos=crawler_strategy, always_by_pass_cache=True)
-result = crawler.run(url="https://www.nbcnews.com/business", js = js_code)</code></pre>
-        <div class="">Remember that you can pass multiple JavaScript code snippets in the list. They all will be executed in the order they are passed.</div>
-        </div>
-
-        <!-- Conclusion -->
-        <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-            🎉
-            <strong
-                >Congratulations! You've made it through the Crawl4ai Quickstart Guide! Now go forth
-                and crawl the web like a pro! 🕸️</strong
-            >
-        </div>
-    </div>
-</section>

pages/partial/installation.html

@@ -1,65 +0,0 @@
-<section id="installation" class="content-section active">
-    <h1 class="text-2xl font-bold">Installation 💻</h1>
-    <p class="mb-4">
-        There are three ways to use Crawl4AI: 
-        <ol class="list-decimal list-inside mb-4">
-            <li class="">
-                As a library
-            </li>
-            <li class="">
-                As a local server (Docker)
-            </li>
-            <li class="">
-                As a Google Colab notebook. <a href="https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk"
-                ><img
-                    src="https://colab.research.google.com/assets/colab-badge.svg"
-                    alt="Open In Colab"
-                    style="display: inline-block; width: 100px; height: 20px"
-            /></a>
-            </li>
-    </p>
-
-
-    <p class="my-4">To install Crawl4AI as a library, follow these steps:</p>
-
-    <ol class="list-decimal list-inside mb-4">
-        <li class="mb-4">
-            Install the package from GitHub:
-            <pre
-                class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-            ><code>virtualenv venv
-source venv/bin/activate
-pip install "crawl4ai[all] @ git+https://github.com/unclecode/crawl4ai.git"
-            </code></pre>
-        </li>
-        <li class="mb-4">
-            Run the following command to load the required models. This is optional, but it will boost the performance and speed of the crawler. You need to do this only once.
-            <pre
-                class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-            ><code>crawl4ai-download-models</code></pre>
-        </li>
-        <li class="mb-4">
-            Alternatively, you can clone the repository and install the package locally:
-            <pre
-                class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-            ><code  class = "language-python bash">virtualenv venv
-source venv/bin/activate
-git clone https://github.com/unclecode/crawl4ai.git
-cd crawl4ai
-pip install -e .[all]
-</code></pre>
-        </li>
-        <li class="">
-            Use docker to run the local server:
-            <pre
-                class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-            ><code  class = "language-python bash">docker build -t crawl4ai . 
-# docker build --platform linux/amd64 -t crawl4ai . For Mac users
-docker run -d -p 8000:80 crawl4ai</code></pre>
-        </li>
-    </ol>
-    <p class="mb-4">
-        For more information about how to run Crawl4AI as a local server, please refer to the
-        <a href="https://github.com/unclecode/crawl4ai" class="text-blue-400">GitHub repository</a>.
-    </p>
-</section>

pages/partial/try_it.html

@@ -1,217 +0,0 @@
-<section class="try-it py-8 px-16 pb-20 bg-zinc-900 overflow-hidden">
-    <div class="container mx-auto ">
-        <h2 class="text-2xl font-bold mb-4 text-lime-500">Try It Now</h2>
-        <div class="flex gap-4">
-            <div class="flex flex-col flex-1 gap-2">
-                <div class="flex flex-col">
-                    <label for="url-input" class="text-lime-500 font-bold text-xs">URL(s)</label>
-                    <input
-                        type="text"
-                        id="url-input"
-                        value="https://www.nbcnews.com/business"
-                        class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-zinc-300"
-                        placeholder="Enter URL(s) separated by commas"
-                    />
-                </div>
-                <div class="flex gap-2">
-                    <div class="flex flex-col">
-                        <label for="threshold" class="text-lime-500 font-bold text-xs">Min Words Threshold</label>
-                        <select
-                            id="threshold"
-                            class="border border-zinc-700 rounded px-4 py-1 bg-zinc-900 text-zinc-300"
-                        >
-                            <option value="1">1</option>
-                            <option value="5">5</option>
-                            <option value="10" selected>10</option>
-                            <option value="15">15</option>
-                            <option value="20">20</option>
-                            <option value="25">25</option>
-                        </select>
-                    </div>
-                    <div class="flex flex-col flex-1">
-                        <label for="css-selector" class="text-lime-500 font-bold text-xs">CSS Selector</label>
-                        <input
-                            type="text"
-                            id="css-selector"
-                            class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-zinc-300 placeholder-lime-700"
-                            placeholder="CSS Selector (e.g. .content, #main, article)"
-                        />
-                    </div>
-                </div>
-                <div class="flex gap-2">
-                    <div class="flex flex-col">
-                        <label for="extraction-strategy-select" class="text-lime-500 font-bold text-xs"
-                            >Extraction Strategy</label
-                        >
-                        <select
-                            id="extraction-strategy-select"
-                            class="border border-zinc-700 rounded px-4 py-1 bg-zinc-900 text-zinc-300"
-                        >
-                            <option value="NoExtractionStrategy" selected>NoExtractionStrategy</option>
-                            <option value="CosineStrategy">CosineStrategy</option>
-                            <option value="LLMExtractionStrategy">LLMExtractionStrategy</option>
-                        </select>
-                    </div>
-                    <div class="flex flex-col">
-                        <label for="chunking-strategy-select" class="text-lime-500 font-bold text-xs"
-                            >Chunking Strategy</label
-                        >
-                        <select
-                            id="chunking-strategy-select"
-                            class="border border-zinc-700 rounded px-4 py-1 bg-zinc-900 text-zinc-300"
-                        >
-                            <option value="RegexChunking">RegexChunking</option>
-                            <option value="NlpSentenceChunking">NlpSentenceChunking</option>
-                            <option value="TopicSegmentationChunking">TopicSegmentationChunking</option>
-                            <option value="FixedLengthWordChunking">FixedLengthWordChunking</option>
-                            <option value="SlidingWindowChunking">SlidingWindowChunking</option>
-                        </select>
-                    </div>
-                </div>
-                <div id = "llm_settings" class="flex gap-2 hidden hidden">
-                    <div class="flex flex-col">
-                        <label for="provider-model-select" class="text-lime-500 font-bold text-xs"
-                            >Provider Model</label
-                        >
-                        <select
-                            id="provider-model-select"
-                            class="border border-zinc-700 rounded px-4 py-1 bg-zinc-900 text-zinc-300"
-                        >
-                            <option value="groq/llama3-70b-8192">groq/llama3-70b-8192</option>
-                            <option value="groq/llama3-8b-8192">groq/llama3-8b-8192</option>
-                            <option value="groq/mixtral-8x7b-32768">groq/mixtral-8x7b-32768</option>
-                            <option value="openai/gpt-4-turbo">gpt-4-turbo</option>
-                            <option value="openai/gpt-3.5-turbo">gpt-3.5-turbo</option>
-                            <option value="openai/gpt-4o">gpt-4o</option>
-                            <option value="anthropic/claude-3-haiku-20240307">claude-3-haiku</option>
-                            <option value="anthropic/claude-3-opus-20240229">claude-3-opus</option>
-                            <option value="anthropic/claude-3-sonnet-20240229">claude-3-sonnet</option>
-                        </select>
-                    </div>
-                    <div class="flex flex-col flex-1">
-                        <label for="token-input" class="text-lime-500 font-bold text-xs">API Token</label>
-                        <input
-                            type="password"
-                            id="token-input"
-                            class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-zinc-300"
-                            placeholder="Enter Groq API token"
-                        />
-                    </div>
-                </div>
-                <div  class="flex gap-2">
-                    <!-- Add two textarea one for getting Keyword Filter and another one Instruction, make both grow whole with-->
-                    <div id = "semantic_filter_div" class="flex flex-col flex-1 hidden">
-                        <label for="keyword-filter" class="text-lime-500 font-bold text-xs">Keyword Filter</label>
-                        <textarea
-                            id="semantic_filter"
-                            rows="3"
-                            class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-zinc-300 placeholder-zinc-700"
-                            placeholder="Enter keywords for CosineStrategy to narrow down the content."
-                        ></textarea>
-                    </div>
-                    <div id = "instruction_div" class="flex flex-col flex-1 hidden">
-                        <label for="instruction" class="text-lime-500 font-bold text-xs">Instruction</label>
-                        <textarea
-                            id="instruction"
-                            rows="3"
-                            class="border border-zinc-700 rounded px-4 py-0 bg-zinc-900 text-zinc-300 placeholder-zinc-700"
-                            placeholder="Enter instruction for the LLMEstrategy to instruct the model."
-                        ></textarea>
-                    </div>
-                </div>
-                <div class="flex gap-3">
-                    <div class="flex items-center gap-2">
-                        <input type="checkbox" id="bypass-cache-checkbox" />
-                        <label for="bypass-cache-checkbox" class="text-lime-500 font-bold">Bypass Cache</label>
-                    </div>
-                    <div class="flex items-center gap-2">
-                        <input type="checkbox" id="screenshot-checkbox" checked />
-                        <label for="screenshot-checkbox" class="text-lime-500 font-bold">Screenshot</label>
-                    </div>
-                    <div class="flex items-center gap-2 hidden">
-                        <input type="checkbox" id="extract-blocks-checkbox" />
-                        <label for="extract-blocks-checkbox" class="text-lime-500 font-bold">Extract Blocks</label>
-                    </div>
-                    <button id="crawl-btn" class="bg-lime-600 text-black font-bold px-4 py-0 rounded">Crawl</button>
-                </div>
-            </div>
-
-            <div id="loading" class="hidden">
-                <p class="text-white">Loading... Please wait.</p>
-            </div>
-            <div id="result" class="flex-1  overflow-x-auto">
-                <div class="tab-buttons flex gap-2">
-                    <button class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500" data-tab="json">
-                        JSON
-                    </button>
-                    <button
-                        class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                        data-tab="cleaned-html"
-                    >
-                        Cleaned HTML
-                    </button>
-                    <button class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500" data-tab="markdown">
-                        Markdown
-                    </button>
-                    <button class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500" data-tab="media">
-                        Medias
-                    </button>
-                    <button class="tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500" data-tab="screenshot">
-                        Screenshot
-                    </button>
-                </div>
-                <div class="tab-content code bg-zinc-900 p-2 rounded h-full border border-zinc-700 text-sm">
-                    <pre class="h-full flex"><code id="json-result" class="language-json"></code></pre>
-                    <pre class="hidden h-full flex"><code id="cleaned-html-result" class="language-html"></code></pre>
-                    <pre class="hidden h-full flex"><code id="markdown-result" class="language-markdown"></code></pre>
-                    <pre class="hidden h-full flex"><code id="media-result" class="language-json"></code></pre>
-                    <pre class="hidden h-full flex"><code id="screenshot-result"></code></pre>
-                </div>
-            </div>
-
-            <div id="code_help" class="flex-1  overflow-x-auto">
-                <div class="tab-buttons flex gap-2">
-                    <button class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500" data-tab="curl">
-                        cURL
-                    </button>
-                    <button
-                        class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                        data-tab="library"
-                    >
-                        Python
-                    </button>
-                    <button
-                        class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                        data-tab="python"
-                    >
-                        REST API
-                    </button>
-                    <!-- <button
-                        class="code-tab-btn px-4 py-1 text-sm bg-zinc-700 rounded-t text-lime-500"
-                        data-tab="nodejs"
-                    >
-                        Node.js
-                    </button> -->
-                </div>
-                <div class="tab-content result bg-zinc-900 p-2 rounded h-full border border-zinc-700 text-sm">
-                    <pre class="h-full flex relative overflow-x-auto">
-                        <code id="curl-code" class="language-bash"></code>
-                        <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="curl-code">Copy</button>
-                    </pre>
-                    <pre class="hidden h-full flex relative overflow-x-auto">
-                        <code id="python-code" class="language-python"></code>
-                        <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="python-code">Copy</button>
-                    </pre>
-                    <pre class="hidden h-full flex relative overflow-x-auto">
-                        <code id="nodejs-code" class="language-javascript"></code>
-                        <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="nodejs-code">Copy</button>
-                    </pre>
-                    <pre class="hidden h-full flex relative overflow-x-auto">
-                        <code id="library-code" class="language-python"></code>
-                        <button class="absolute top-2 right-2 bg-zinc-700 text-white px-2 py-1 rounded copy-btn" data-target="library-code">Copy</button>
-                    </pre>
-                </div>
-            </div>
-        </div>
-    </div>
-</section>

pages/tmp.html

@@ -1,434 +0,0 @@
-<div class="w-3/4 p-4">
-    <section id="installation" class="content-section active">
-        <h1 class="text-2xl font-bold">Installation 💻</h1>
-        <p class="mb-4">There are three ways to use Crawl4AI:</p>
-        <ol class="list-decimal list-inside mb-4">
-            <li class="">As a library</li>
-            <li class="">As a local server (Docker)</li>
-            <li class="">
-                As a Google Colab notebook.
-                <a href="https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk"
-                    ><img
-                        src="https://colab.research.google.com/assets/colab-badge.svg"
-                        alt="Open In Colab"
-                        style="display: inline-block; width: 100px; height: 20px"
-                /></a>
-            </li>
-            <p></p>
-
-            <p class="my-4">To install Crawl4AI as a library, follow these steps:</p>
-
-            <ol class="list-decimal list-inside mb-4">
-                <li class="mb-4">
-                    Install the package from GitHub:
-                    <pre
-                        class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-                    ><code class="hljs language-bash">pip install git+https://github.com/unclecode/crawl4ai.git</code></pre>
-                </li>
-                <li class="mb-4">
-                    Alternatively, you can clone the repository and install the package locally:
-                    <pre
-                        class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-                    ><code class="language-python bash hljs">virtualenv venv
-source venv/<span class="hljs-built_in">bin</span>/activate
-git clone https://github.com/unclecode/crawl4ai.git
-cd crawl4ai
-pip install -e .
-</code></pre>
-                </li>
-                <li class="">
-                    Use docker to run the local server:
-                    <pre
-                        class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-                    ><code class="language-python bash hljs">docker build -t crawl4ai . 
-<span class="hljs-comment"># docker build --platform linux/amd64 -t crawl4ai . For Mac users</span>
-docker run -d -p <span class="hljs-number">8000</span>:<span class="hljs-number">80</span> crawl4ai</code></pre>
-                </li>
-            </ol>
-            <p class="mb-4">
-                For more information about how to run Crawl4AI as a local server, please refer to the
-                <a href="https://github.com/unclecode/crawl4ai" class="text-blue-400">GitHub repository</a>.
-            </p>
-        </ol>
-    </section>
-    <section id="how-to-guide" class="content-section">
-        <h1 class="text-2xl font-bold">How to Guide</h1>
-        <div class="flex flex-col gap-4 p-4 bg-zinc-900 text-lime-500">
-            <!-- Step 1 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🌟
-                <strong>Welcome to the Crawl4ai Quickstart Guide! Let's dive into some web crawling fun!</strong>
-            </div>
-            <div class="">
-                First Step: Create an instance of WebCrawler and call the
-                <code>warmup()</code> function.
-            </div>
-            <div>
-                <pre><code class="language-python hljs">crawler = WebCrawler()
-crawler.warmup()</code></pre>
-            </div>
-
-            <!-- Step 2 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🧠 <strong>Understanding 'bypass_cache' and 'include_raw_html' parameters:</strong>
-            </div>
-            <div class="">First crawl (caches the result):</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>)</code></pre>
-            </div>
-            <div class="">Second crawl (Force to crawl again):</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>, bypass_cache=<span class="hljs-literal">True</span>)</code></pre>
-                <div class="bg-red-900 p-2 text-zinc-50">
-                    ⚠️ Don't forget to set <code>`bypass_cache`</code> to True if you want to try different strategies
-                    for the same URL. Otherwise, the cached result will be returned. You can also set
-                    <code>`always_by_pass_cache`</code> in constructor to True to always bypass the cache.
-                </div>
-            </div>
-            <div class="">Crawl result without raw HTML content:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>, include_raw_html=<span class="hljs-literal">False</span>)</code></pre>
-            </div>
-
-            <!-- Step 3 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                📄
-                <strong
-                    >The 'include_raw_html' parameter, when set to True, includes the raw HTML content in the response.
-                    By default, it is set to True.</strong
-                >
-            </div>
-            <div class="">Set <code>always_by_pass_cache</code> to True:</div>
-            <div>
-                <pre><code class="language-python hljs">crawler.always_by_pass_cache = <span class="hljs-literal">True</span></code></pre>
-            </div>
-
-            <!-- Step 4 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🧩 <strong>Let's add a chunking strategy: RegexChunking!</strong>
-            </div>
-            <div class="">Using RegexChunking:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(
-url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>,
-chunking_strategy=RegexChunking(patterns=[<span class="hljs-string">"\n\n"</span>])
-)</code></pre>
-            </div>
-            <div class="">Using NlpSentenceChunking:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(
-url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>,
-chunking_strategy=NlpSentenceChunking()
-)</code></pre>
-            </div>
-
-            <!-- Step 5 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🧠 <strong>Let's get smarter with an extraction strategy: CosineStrategy!</strong>
-            </div>
-            <div class="">Using CosineStrategy:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(
-url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>,
-extraction_strategy=CosineStrategy(word_count_threshold=<span class="hljs-number">20</span>, max_dist=<span class="hljs-number">0.2</span>, linkage_method=<span class="hljs-string">"ward"</span>, top_k=<span class="hljs-number">3</span>)
-)</code></pre>
-            </div>
-
-            <!-- Step 6 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🤖
-                <strong>Time to bring in the big guns: LLMExtractionStrategy without instructions!</strong>
-            </div>
-            <div class="">Using LLMExtractionStrategy without instructions:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(
-url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>,
-extraction_strategy=LLMExtractionStrategy(provider=<span class="hljs-string">"openai/gpt-4o"</span>, api_token=os.getenv(<span class="hljs-string">'OPENAI_API_KEY'</span>))
-)</code></pre>
-            </div>
-
-            <!-- Step 7 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                📜
-                <strong>Let's make it even more interesting: LLMExtractionStrategy with instructions!</strong>
-            </div>
-            <div class="">Using LLMExtractionStrategy with instructions:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(
-url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>,
-extraction_strategy=LLMExtractionStrategy(
-provider=<span class="hljs-string">"openai/gpt-4o"</span>,
-api_token=os.getenv(<span class="hljs-string">'OPENAI_API_KEY'</span>),
-instruction=<span class="hljs-string">"I am interested in only financial news"</span>
-)
-)</code></pre>
-            </div>
-
-            <!-- Step 8 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🎯
-                <strong>Targeted extraction: Let's use a CSS selector to extract only H2 tags!</strong>
-            </div>
-            <div class="">Using CSS selector to extract H2 tags:</div>
-            <div>
-                <pre><code class="language-python hljs">result = crawler.run(
-url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>,
-css_selector=<span class="hljs-string">"h2"</span>
-)</code></pre>
-            </div>
-
-            <!-- Step 9 -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🖱️
-                <strong>Let's get interactive: Passing JavaScript code to click 'Load More' button!</strong>
-            </div>
-            <div class="">Using JavaScript to click 'Load More' button:</div>
-            <div>
-                <pre><code class="language-python hljs">js_code = <span class="hljs-string">"""
-const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button =&gt; button.textContent.includes('Load More'));
-loadMoreButton &amp;&amp; loadMoreButton.click();
-"""</span>
-crawler_strategy = LocalSeleniumCrawlerStrategy(js_code=js_code)
-crawler = WebCrawler(crawler_strategy=crawler_strategy, always_by_pass_cache=<span class="hljs-literal">True</span>)
-result = crawler.run(url=<span class="hljs-string">"https://www.nbcnews.com/business"</span>)</code></pre>
-            </div>
-
-            <!-- Conclusion -->
-            <div class="col-span-2 bg-lime-800 p-2 rounded text-zinc-50">
-                🎉
-                <strong
-                    >Congratulations! You've made it through the Crawl4ai Quickstart Guide! Now go forth and crawl the
-                    web like a pro! 🕸️</strong
-                >
-            </div>
-        </div>
-    </section>
-
-    <section id="chunking-strategies" class="content-section">
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>RegexChunking</h3>
-                <p>
-                    <code>RegexChunking</code> is a text chunking strategy that splits a given text into smaller parts
-                    using regular expressions. This is useful for preparing large texts for processing by language
-                    models, ensuring they are divided into manageable segments.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>patterns</code> (list, optional): A list of regular expression patterns used to split the
-                        text. Default is to split by double newlines (<code>['\n\n']</code>).
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">chunker = RegexChunking(patterns=[r'\n\n', r'\. '])
-chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
-</code></pre>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>NlpSentenceChunking</h3>
-                <p>
-                    <code>NlpSentenceChunking</code> uses a natural language processing model to chunk a given text into
-                    sentences. This approach leverages SpaCy to accurately split text based on sentence boundaries.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        None.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">chunker = NlpSentenceChunking()
-chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
-</code></pre>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>TopicSegmentationChunking</h3>
-                <p>
-                    <code>TopicSegmentationChunking</code> uses the TextTiling algorithm to segment a given text into
-                    topic-based chunks. This method identifies thematic boundaries in the text.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>num_keywords</code> (int, optional): The number of keywords to extract for each topic
-                        segment. Default is <code>3</code>.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">chunker = TopicSegmentationChunking(num_keywords=3)
-chunks = chunker.chunk("This is a sample text. It will be split into topic-based segments.")
-</code></pre>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>FixedLengthWordChunking</h3>
-                <p>
-                    <code>FixedLengthWordChunking</code> splits a given text into chunks of fixed length, based on the
-                    number of words.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>chunk_size</code> (int, optional): The number of words in each chunk. Default is
-                        <code>100</code>.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">chunker = FixedLengthWordChunking(chunk_size=100)
-chunks = chunker.chunk("This is a sample text. It will be split into fixed-length word chunks.")
-</code></pre>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>SlidingWindowChunking</h3>
-                <p>
-                    <code>SlidingWindowChunking</code> uses a sliding window approach to chunk a given text. Each chunk
-                    has a fixed length, and the window slides by a specified step size.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>window_size</code> (int, optional): The number of words in each chunk. Default is
-                        <code>100</code>.
-                    </li>
-                    <li>
-                        <code>step</code> (int, optional): The number of words to slide the window. Default is
-                        <code>50</code>.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">chunker = SlidingWindowChunking(window_size=100, step=50)
-chunks = chunker.chunk("This is a sample text. It will be split using a sliding window approach.")
-</code></pre>
-            </div>
-        </div>
-    </section>
-    <section id="extraction-strategies" class="content-section">
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>NoExtractionStrategy</h3>
-                <p>
-                    <code>NoExtractionStrategy</code> is a basic extraction strategy that returns the entire HTML
-                    content without any modification. It is useful for cases where no specific extraction is required.
-                    Only clean html, and amrkdown.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <p>None.</p>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">extractor = NoExtractionStrategy()
-extracted_content = extractor.extract(url, html)
-</code></pre>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>LLMExtractionStrategy</h3>
-                <p>
-                    <code>LLMExtractionStrategy</code> uses a Language Model (LLM) to extract meaningful blocks or
-                    chunks from the given HTML content. This strategy leverages an external provider for language model
-                    completions.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>provider</code> (str, optional): The provider to use for the language model completions.
-                        Default is <code>DEFAULT_PROVIDER</code> (e.g., openai/gpt-4).
-                    </li>
-                    <li>
-                        <code>api_token</code> (str, optional): The API token for the provider. If not provided, it will
-                        try to load from the environment variable <code>OPENAI_API_KEY</code>.
-                    </li>
-                    <li>
-                        <code>instruction</code> (str, optional): An instruction to guide the LLM on how to perform the
-                        extraction. This allows users to specify the type of data they are interested in or set the tone
-                        of the response. Default is <code>None</code>.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
-extracted_content = extractor.extract(url, html)
-</code></pre>
-                <p>
-                    By providing clear instructions, users can tailor the extraction process to their specific needs,
-                    enhancing the relevance and utility of the extracted content.
-                </p>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>CosineStrategy</h3>
-                <p>
-                    <code>CosineStrategy</code> uses hierarchical clustering based on cosine similarity to extract
-                    clusters of text from the given HTML content. This strategy is suitable for identifying related
-                    content sections.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>semantic_filter</code> (str, optional): A string containing keywords for filtering relevant
-                        documents before clustering. If provided, documents are filtered based on their cosine
-                        similarity to the keyword filter embedding. Default is <code>None</code>.
-                    </li>
-                    <li>
-                        <code>word_count_threshold</code> (int, optional): Minimum number of words per cluster. Default
-                        is <code>20</code>.
-                    </li>
-                    <li>
-                        <code>max_dist</code> (float, optional): The maximum cophenetic distance on the dendrogram to
-                        form clusters. Default is <code>0.2</code>.
-                    </li>
-                    <li>
-                        <code>linkage_method</code> (str, optional): The linkage method for hierarchical clustering.
-                        Default is <code>'ward'</code>.
-                    </li>
-                    <li>
-                        <code>top_k</code> (int, optional): Number of top categories to extract. Default is
-                        <code>3</code>.
-                    </li>
-                    <li>
-                        <code>model_name</code> (str, optional): The model name for embedding generation. Default is
-                        <code>'BAAI/bge-small-en-v1.5'</code>.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">extractor = CosineStrategy(semantic_filter='artificial intelligence', word_count_threshold=10, max_dist=0.2, linkage_method='ward', top_k=3, model_name='BAAI/bge-small-en-v1.5')
-extracted_content = extractor.extract(url, html)
-</code></pre>
-                <h4>Cosine Similarity Filtering</h4>
-                <p>
-                    When a <code>semantic_filter</code> is provided, the <code>CosineStrategy</code> applies an
-                    embedding-based filtering process to select relevant documents before performing hierarchical
-                    clustering.
-                </p>
-            </div>
-        </div>
-        <div class="bg-zinc-800 p-4 rounded shadow-md docs-item">
-            <div class="text-gray-300 prose prose-sm">
-                <h3>TopicExtractionStrategy</h3>
-                <p>
-                    <code>TopicExtractionStrategy</code> uses the TextTiling algorithm to segment the HTML content into
-                    topics and extracts keywords for each segment. This strategy is useful for identifying and
-                    summarizing thematic content.
-                </p>
-                <h4>Constructor Parameters:</h4>
-                <ul>
-                    <li>
-                        <code>num_keywords</code> (int, optional): Number of keywords to represent each topic segment.
-                        Default is <code>3</code>.
-                    </li>
-                </ul>
-                <h4>Example usage:</h4>
-                <pre><code class="language-python">extractor = TopicExtractionStrategy(num_keywords=3)
-extracted_content = extractor.extract(url, html)
-</code></pre>
-            </div>
-        </div>
-    </section>
-</div>

requirements-dev.txt

@@ -1,2 +0,0 @@
--r requirements.txt
-pytest