chore(git): update gitignore patterns

Add new development and tooling related patterns to gitignore:
- Add Next.js build directory (.next/)
- Add various script and documentation files
- Add local development directories (.local, .scripts, .do)
- Add tool-specific files (.codeiumignore, .windsurfrules)

Removes duplicate entries and organizes patterns more clearly.

.gitignore

@@ -199,12 +199,35 @@ test_env/
 **/.DS_Store
 
 todo.md
+todo_executor.md
 git_changes.py
 git_changes.md
 pypi_build.sh
 git_issues.py
 git_issues.md
 
+.next/
 .tests/
+# .issues/
+.docs/
 .issues/
-.docs/
+.gitboss/
+todo_executor.md
+protect-all-except-feature.sh
+manage-collab.sh
+publish.sh
+combine.sh
+combined_output.txt
+.local
+.scripts
+tree.md
+tree.md
+.scripts
+.local
+.do
+/plans
+.codeiumignore
+todo/
+
+# windsurf rules
+.windsurfrules

CHANGELOG.md

@@ -1,212 +1,5 @@
 # Changelog
 
-## [v0.3.73] - 2024-10-24
-
-### Added
-- Smart overlay removal system in AsyncPlaywrightCrawlerStrategy:
-  - Automatic removal of popups, modals, and cookie notices
-  - Detection and removal of fixed/sticky position elements
-  - Cleaning of empty block elements
-  - Configurable via `remove_overlay_elements` parameter
-- Enhanced screenshot capabilities:
-  - Added `screenshot_wait_for` parameter to control timing
-  - Improved screenshot handling with existing page context
-  - Better error handling with fallback error images
-- New URL normalization utilities:
-  - `normalize_url` function for consistent URL formatting
-  - `is_external_url` function for better link classification
-- Custom base directory support for cache storage:
-  - New `base_directory` parameter in AsyncWebCrawler
-  - Allows specifying alternative locations for `.crawl4ai` folder
-
-### Enhanced
-- Link handling improvements:
-  - Better duplicate link detection
-  - Enhanced internal/external link classification
-  - Improved handling of special URL protocols
-  - Support for anchor links and protocol-relative URLs
-- Configuration refinements:
-  - Streamlined social media domain list
-  - More focused external content filtering
-- LLM extraction strategy:
-  - Added support for separate API base URL via `api_base` parameter
-  - Better handling of base URLs in configuration
-
-### Fixed
-- Screenshot functionality:
-  - Resolved issues with screenshot timing and context
-  - Improved error handling and recovery
-- Link processing:
-  - Fixed URL normalization edge cases
-  - Better handling of invalid URLs
-  - Improved error messages for link processing failures
-
-### Developer Notes
-- The overlay removal system uses advanced JavaScript injection for better compatibility
-- URL normalization handles special cases like mailto:, tel:, and protocol-relative URLs
-- Screenshot system now reuses existing page context for better performance
-- Link processing maintains separate dictionaries for internal and external links to ensure uniqueness
-
-## [v0.3.72] - 2024-10-22
-
-### Added
-- New `ContentCleaningStrategy` class:
-  - Smart content extraction based on text density and element scoring
-  - Automatic removal of boilerplate content
-  - DOM tree analysis for better content identification
-  - Configurable thresholds for content detection
-- Advanced proxy support:
-  - Added `proxy_config` option for authenticated proxy connections
-  - Support for username/password in proxy configuration
-- New content output formats:
-  - `fit_markdown`: Optimized markdown output with main content focus
-  - `fit_html`: Clean HTML with only essential content
-
-### Enhanced
-- Image source detection:
-  - Support for multiple image source attributes (`src`, `data-src`, `srcset`, etc.)
-  - Automatic fallback through potential source attributes
-  - Smart handling of srcset attribute
-- External content handling:
-  - Made external link exclusion optional (disabled by default)
-  - Improved detection and handling of social media links
-  - Better control over external image filtering
-
-### Fixed
-- Image extraction reliability with multiple source attribute checks
-- External link and image handling logic for better accuracy
-
-### Developer Notes
-- The new `ContentCleaningStrategy` uses configurable thresholds for customization
-- Proxy configuration now supports more complex authentication scenarios
-- Content extraction process now provides both regular and optimized outputs
-
-## [v0.3.72] - 2024-10-20
-
-### Fixed
-- Added support for parsing Base64 encoded images in WebScrappingStrategy
-
-### Added
-- Forked and integrated a customized version of the html2text library for more control over Markdown generation
-- New configuration options for controlling external content:
-  - Ability to exclude all external links
-  - Option to specify domains to exclude (default includes major social media platforms)
-  - Control over excluding external images
-
-### Changed
-- Improved Markdown generation process:
-  - Added fine-grained control over character escaping in Markdown output
-  - Enhanced handling of code blocks and pre-formatted text
-- Updated `AsyncPlaywrightCrawlerStrategy.close()` method to use a shorter sleep time (0.5 seconds instead of 500)
-- Enhanced flexibility in `CosineStrategy` with a more generic `load_HF_embedding_model` function
-
-### Improved
-- Optimized content scraping and processing for better efficiency
-- Enhanced error handling and logging in various components
-
-### Developer Notes
-- The customized html2text library is now located within the crawl4ai package
-- New configuration options are available in the `config.py` file for external content handling
-- The `WebScrappingStrategy` class has been updated to accommodate new external content exclusion options
-
-## [v0.3.71] - 2024-10-19
-
-### Added
-- New chunking strategies:
-  - `OverlappingWindowChunking`: Allows for overlapping chunks of text, useful for maintaining context between chunks.
-  - Enhanced `SlidingWindowChunking`: Improved to handle edge cases and last chunks more effectively.
-
-### Changed
-- Updated `CHUNK_TOKEN_THRESHOLD` in config to 2048 tokens (2^11) for better compatibility with most LLM models.
-- Improved `AsyncPlaywrightCrawlerStrategy.close()` method to use a shorter sleep time (0.5 seconds instead of 500), significantly reducing wait time when closing the crawler.
-- Enhanced flexibility in `CosineStrategy`:
-  - Now uses a more generic `load_HF_embedding_model` function, allowing for easier swapping of embedding models.
-- Updated `JsonCssExtractionStrategy` and `JsonXPATHExtractionStrategy` for better JSON-based extraction.
-
-### Fixed
-- Addressed potential issues with the sliding window chunking strategy to ensure all text is properly chunked.
-
-### Developer Notes
-- Added more comprehensive docstrings to chunking strategies for better code documentation.
-- Removed hardcoded device setting in `CosineStrategy`, now using the automatically detected device.
-- Added a new example in `quickstart_async.py` for generating a knowledge graph from crawled content.
-
-These updates aim to provide more flexibility in text processing, improve performance, and enhance the overall capabilities of the crawl4ai library. The new chunking strategies, in particular, offer more options for handling large texts in various scenarios.
-
-## [v0.3.71] - 2024-10-18
-
-### Changes
-1. **Version Update**:
-   - Updated version number from 0.3.7 to 0.3.71.
-
-2. **Crawler Enhancements**:
-   - Added `sleep_on_close` option to AsyncPlaywrightCrawlerStrategy for delayed browser closure.
-   - Improved context creation with additional options:
-     - Enabled `accept_downloads` and `java_script_enabled`.
-     - Added a cookie to enable cookies by default.
-
-3. **Error Handling Improvements**:
-   - Enhanced error messages in AsyncWebCrawler's `arun` method.
-   - Updated error reporting format for better visibility and consistency.
-
-4. **Performance Optimization**:
-   - Commented out automatic page and context closure in `crawl` method to potentially improve performance in certain scenarios.
-
-### Documentation
-- Updated quickstart notebook:
-  - Changed installation command to use the released package instead of GitHub repository.
-  - Updated kernel display name.
-
-### Developer Notes
-- Minor code refactoring and cleanup.
-
-## [v0.3.7] - 2024-10-17
-
-### New Features
-1. **Enhanced Browser Stealth**: 
-   - Implemented `playwright_stealth` for improved bot detection avoidance.
-   - Added `StealthConfig` for fine-tuned control over stealth parameters.
-
-2. **User Simulation**:
-   - New `simulate_user` option to mimic human-like interactions (mouse movements, clicks, keyboard presses).
-
-3. **Navigator Override**:
-   - Added `override_navigator` option to modify navigator properties, further improving bot detection evasion.
-
-4. **Improved iframe Handling**:
-   - New `process_iframes` parameter to extract and integrate iframe content into the main page.
-
-5. **Flexible Browser Selection**:
-   - Support for choosing between Chromium, Firefox, and WebKit browsers.
-
-6. **Include Links in Markdown**:
-    - Added support for including links in Markdown content, by definin g a new flag `include_links_on_markdown` in `crawl` method.   
-
-### Improvements
-1. **Better Error Handling**:
-   - Enhanced error reporting in WebScrappingStrategy with detailed error messages and suggestions.
-   - Added console message and error logging for better debugging.
-
-2. **Image Processing Enhancements**:
-   - Improved image dimension updating and filtering logic.
-
-3. **Crawling Flexibility**:
-   - Added support for custom viewport sizes.
-   - Implemented delayed content retrieval with `delay_before_return_html` parameter.
-
-4. **Performance Optimization**:
-   - Adjusted default semaphore count for parallel crawling.
-
-### Bug Fixes
-- Fixed an issue where the HTML content could be empty after processing.
-
-### Examples
-- Added new example `crawl_with_user_simulation()` demonstrating the use of user simulation and navigator override features.
-
-### Developer Notes
-- Refactored code for better maintainability and readability.
-- Updated browser launch arguments for improved compatibility and performance.
-
 ## [v0.3.6] - 2024-10-12 
 
 ### 1. Improved Crawling Control

README.md

@@ -8,14 +8,16 @@
 
 Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
 
-## New in 0.3.72 ✨
+> Looking for the synchronous version? Check out [README.sync.md](./README.sync.md). You can also access the previous version in the branch [V0.2.76](https://github.com/unclecode/crawl4ai/blob/v0.2.76).
 
-- 📄 Fit markdown generation for extracting main article content.
-- 🪄 Magic mode for comprehensive anti-bot detection bypass.
-- 🌐 Enhanced multi-browser support with seamless switching (Chromium, Firefox, WebKit)
-- 📚 New chunking strategies(Sliding window, Overlapping window, Flexible size control)
-- 💾 Improved caching system for better performance
-- ⚡ Optimized batch processing with automatic rate limiting
+## New update 0.3.6
+- 🌐 Multi-browser support (Chromium, Firefox, WebKit)
+- 🖼️ Improved image processing with lazy-loading detection
+- 🔧 Custom page timeout parameter for better control over crawling behavior
+- 🕰️ Enhanced handling of delayed content loading
+- 🔑 Custom headers support for LLM interactions
+- 🖼️ iframe content extraction for comprehensive page analysis
+- ⏱️ Flexible timeout and delayed content retrieval options
 
 ## Try it Now!
 
@@ -28,28 +30,22 @@ Crawl4AI simplifies asynchronous web crawling and data extraction, making it acc
 - 🆓 Completely free and open-source
 - 🚀 Blazing fast performance, outperforming many paid services
 - 🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
-- 🌐 Multi-browser support (Chromium, Firefox, WebKit)
 - 🌍 Supports crawling multiple URLs simultaneously
 - 🎨 Extracts and returns all media tags (Images, Audio, and Video)
 - 🔗 Extracts all external and internal links
 - 📚 Extracts metadata from the page
-- 🔄 Custom hooks for authentication, headers, and page modifications
+- 🔄 Custom hooks for authentication, headers, and page modifications before crawling
 - 🕵️ User-agent customization
-- 🖼️ Takes screenshots of pages with enhanced error handling
+- 🖼️ Takes screenshots of the page
 - 📜 Executes multiple custom JavaScripts before crawling
 - 📊 Generates structured output without LLM using JsonCssExtractionStrategy
 - 📚 Various chunking strategies: topic-based, regex, sentence, and more
 - 🧠 Advanced extraction strategies: cosine clustering, LLM, and more
 - 🎯 CSS selector support for precise data extraction
 - 📝 Passes instructions/keywords to refine extraction
-- 🔒 Proxy support with authentication for enhanced access
-- 🔄 Session management for complex multi-page crawling
-- 🌐 Asynchronous architecture for improved performance
-- 🖼️ Improved image processing with lazy-loading detection
-- 🕰️ Enhanced handling of delayed content loading
-- 🔑 Custom headers support for LLM interactions
-- 🖼️ iframe content extraction for comprehensive analysis
-- ⏱️ Flexible timeout and delayed content retrieval options
+- 🔒 Proxy support for enhanced privacy and access
+- 🔄 Session management for complex multi-page crawling scenarios
+- 🌐 Asynchronous architecture for improved performance and scalability
 
 ## Installation 🛠️
 

crawl4ai/__init__.py

@@ -3,7 +3,7 @@
 from .async_webcrawler import AsyncWebCrawler
 from .models import CrawlResult
 
-__version__ = "0.3.72"
+__version__ = "0.3.6"
 
 __all__ = [
     "AsyncWebCrawler",

crawl4ai/async_crawler_strategy copy.py

@@ -1,558 +0,0 @@
-import asyncio
-import base64
-import time
-from abc import ABC, abstractmethod
-from typing import Callable, Dict, Any, List, Optional, Awaitable
-import os
-from playwright.async_api import async_playwright, Page, Browser, Error
-from io import BytesIO
-from PIL import Image, ImageDraw, ImageFont
-from pathlib import Path
-from playwright.async_api import ProxySettings
-from pydantic import BaseModel
-import hashlib
-import json
-import uuid
-from playwright_stealth import stealth_async
-
-class AsyncCrawlResponse(BaseModel):
-    html: str
-    response_headers: Dict[str, str]
-    status_code: int
-    screenshot: Optional[str] = None
-    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
-
-    class Config:
-        arbitrary_types_allowed = True
-
-class AsyncCrawlerStrategy(ABC):
-    @abstractmethod
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        pass
-    
-    @abstractmethod
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        pass
-    
-    @abstractmethod
-    async def take_screenshot(self, url: str) -> str:
-        pass
-    
-    @abstractmethod
-    def update_user_agent(self, user_agent: str):
-        pass
-    
-    @abstractmethod
-    def set_hook(self, hook_type: str, hook: Callable):
-        pass
-
-class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
-    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
-        self.use_cached_html = use_cached_html
-        self.user_agent = kwargs.get(
-            "user_agent",
-            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
-            "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
-        )
-        self.proxy = kwargs.get("proxy")
-        self.headless = kwargs.get("headless", True)
-        self.browser_type = kwargs.get("browser_type", "chromium")
-        self.headers = kwargs.get("headers", {})
-        self.sessions = {}
-        self.session_ttl = 1800 
-        self.js_code = js_code
-        self.verbose = kwargs.get("verbose", False)
-        self.playwright = None
-        self.browser = None
-        self.hooks = {
-            'on_browser_created': None,
-            'on_user_agent_updated': None,
-            'on_execution_started': None,
-            'before_goto': None,
-            'after_goto': None,
-            'before_return_html': None,
-            'before_retrieve_html': None
-        }
-
-    async def __aenter__(self):
-        await self.start()
-        return self
-
-    async def __aexit__(self, exc_type, exc_val, exc_tb):
-        await self.close()
-
-    async def start(self):
-        if self.playwright is None:
-            self.playwright = await async_playwright().start()
-        if self.browser is None:
-            browser_args = {
-                "headless": self.headless,
-                "args": [
-                    "--disable-gpu",
-                    "--no-sandbox",
-                    "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
-                ]
-            }
-            
-            # Add proxy settings if a proxy is specified
-            if self.proxy:
-                proxy_settings = ProxySettings(server=self.proxy)
-                browser_args["proxy"] = proxy_settings
-                
-            # Select the appropriate browser based on the browser_type
-            if self.browser_type == "firefox":
-                self.browser = await self.playwright.firefox.launch(**browser_args)
-            elif self.browser_type == "webkit":
-                self.browser = await self.playwright.webkit.launch(**browser_args)
-            else:
-                self.browser = await self.playwright.chromium.launch(**browser_args)
-
-            await self.execute_hook('on_browser_created', self.browser)
-
-    async def close(self):
-        if self.browser:
-            await self.browser.close()
-            self.browser = None
-        if self.playwright:
-            await self.playwright.stop()
-            self.playwright = None
-
-    def __del__(self):
-        if self.browser or self.playwright:
-            asyncio.get_event_loop().run_until_complete(self.close())
-
-    def set_hook(self, hook_type: str, hook: Callable):
-        if hook_type in self.hooks:
-            self.hooks[hook_type] = hook
-        else:
-            raise ValueError(f"Invalid hook type: {hook_type}")
-
-    async def execute_hook(self, hook_type: str, *args):
-        hook = self.hooks.get(hook_type)
-        if hook:
-            if asyncio.iscoroutinefunction(hook):
-                return await hook(*args)
-            else:
-                return hook(*args)
-        return args[0] if args else None
-
-    def update_user_agent(self, user_agent: str):
-        self.user_agent = user_agent
-
-    def set_custom_headers(self, headers: Dict[str, str]):
-        self.headers = headers
-
-    async def kill_session(self, session_id: str):
-        if session_id in self.sessions:
-            context, page, _ = self.sessions[session_id]
-            await page.close()
-            await context.close()
-            del self.sessions[session_id]
-
-    def _cleanup_expired_sessions(self):
-        current_time = time.time()
-        expired_sessions = [
-            sid for sid, (_, _, last_used) in self.sessions.items() 
-            if current_time - last_used > self.session_ttl
-        ]
-        for sid in expired_sessions:
-            asyncio.create_task(self.kill_session(sid))
-            
-    async def smart_wait(self, page: Page, wait_for: str, timeout: float = 30000):
-        wait_for = wait_for.strip()
-        
-        if wait_for.startswith('js:'):
-            # Explicitly specified JavaScript
-            js_code = wait_for[3:].strip()
-            return await self.csp_compliant_wait(page, js_code, timeout)
-        elif wait_for.startswith('css:'):
-            # Explicitly specified CSS selector
-            css_selector = wait_for[4:].strip()
-            try:
-                await page.wait_for_selector(css_selector, timeout=timeout)
-            except Error as e:
-                if 'Timeout' in str(e):
-                    raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{css_selector}'")
-                else:
-                    raise ValueError(f"Invalid CSS selector: '{css_selector}'")
-        else:
-            # Auto-detect based on content
-            if wait_for.startswith('()') or wait_for.startswith('function'):
-                # It's likely a JavaScript function
-                return await self.csp_compliant_wait(page, wait_for, timeout)
-            else:
-                # Assume it's a CSS selector first
-                try:
-                    await page.wait_for_selector(wait_for, timeout=timeout)
-                except Error as e:
-                    if 'Timeout' in str(e):
-                        raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{wait_for}'")
-                    else:
-                        # If it's not a timeout error, it might be an invalid selector
-                        # Let's try to evaluate it as a JavaScript function as a fallback
-                        try:
-                            return await self.csp_compliant_wait(page, f"() => {{{wait_for}}}", timeout)
-                        except Error:
-                            raise ValueError(f"Invalid wait_for parameter: '{wait_for}'. "
-                                             "It should be either a valid CSS selector, a JavaScript function, "
-                                             "or explicitly prefixed with 'js:' or 'css:'.")
-    
-    async def csp_compliant_wait(self, page: Page, user_wait_function: str, timeout: float = 30000):
-        wrapper_js = f"""
-        async () => {{
-            const userFunction = {user_wait_function};
-            const startTime = Date.now();
-            while (true) {{
-                if (await userFunction()) {{
-                    return true;
-                }}
-                if (Date.now() - startTime > {timeout}) {{
-                    throw new Error('Timeout waiting for condition');
-                }}
-                await new Promise(resolve => setTimeout(resolve, 100));
-            }}
-        }}
-        """
-        
-        try:
-            await page.evaluate(wrapper_js)
-        except TimeoutError:
-            raise TimeoutError(f"Timeout after {timeout}ms waiting for condition")
-        except Exception as e:
-            raise RuntimeError(f"Error in wait condition: {str(e)}")
-
-    async def process_iframes(self, page):
-        # Find all iframes
-        iframes = await page.query_selector_all('iframe')
-        
-        for i, iframe in enumerate(iframes):
-            try:
-                # Add a unique identifier to the iframe
-                await iframe.evaluate(f'(element) => element.id = "iframe-{i}"')
-                
-                # Get the frame associated with this iframe
-                frame = await iframe.content_frame()
-                
-                if frame:
-                    # Wait for the frame to load
-                    await frame.wait_for_load_state('load', timeout=30000)  # 30 seconds timeout
-                    
-                    # Extract the content of the iframe's body
-                    iframe_content = await frame.evaluate('() => document.body.innerHTML')
-                    
-                    # Generate a unique class name for this iframe
-                    class_name = f'extracted-iframe-content-{i}'
-                    
-                    # Replace the iframe with a div containing the extracted content
-                    _iframe = iframe_content.replace('`', '\\`')
-                    await page.evaluate(f"""
-                        () => {{
-                            const iframe = document.getElementById('iframe-{i}');
-                            const div = document.createElement('div');
-                            div.innerHTML = `{_iframe}`;
-                            div.className = '{class_name}';
-                            iframe.replaceWith(div);
-                        }}
-                    """)
-                else:
-                    print(f"Warning: Could not access content frame for iframe {i}")
-            except Exception as e:
-                print(f"Error processing iframe {i}: {str(e)}")
-
-        # Return the page object
-        return page  
-    
-    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
-        response_headers = {}
-        status_code = None
-        
-        self._cleanup_expired_sessions()
-        session_id = kwargs.get("session_id")
-        if session_id:
-            context, page, _ = self.sessions.get(session_id, (None, None, None))
-            if not context:
-                context = await self.browser.new_context(
-                    user_agent=self.user_agent,
-                    viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None
-                )
-                await context.set_extra_http_headers(self.headers)
-                page = await context.new_page()
-                self.sessions[session_id] = (context, page, time.time())
-        else:
-            context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
-            )
-            await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
-            page = await context.new_page()
-
-        try:
-            if self.verbose:
-                print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                if os.path.exists(cache_file_path):
-                    html = ""
-                    with open(cache_file_path, "r") as f:
-                        html = f.read()
-                    # retrieve response headers and status code from cache
-                    with open(cache_file_path + ".meta", "r") as f:
-                        meta = json.load(f)
-                        response_headers = meta.get("response_headers", {})
-                        status_code = meta.get("status_code")
-                    response = AsyncCrawlResponse(
-                        html=html, response_headers=response_headers, status_code=status_code
-                    )
-                    return response
-
-            if not kwargs.get("js_only", False):
-                await self.execute_hook('before_goto', page)
-                
-                response = await page.goto("about:blank")
-                await stealth_async(page)
-                response = await page.goto(
-                    url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000)
-                )
-                
-                # await stealth_async(page)
-                # response = await page.goto("about:blank")
-                # await stealth_async(page)
-                # await page.evaluate(f"window.location.href = '{url}'")
-                
-                await self.execute_hook('after_goto', page)
-                
-                # Get status code and headers
-                status_code = response.status
-                response_headers = response.headers
-            else:
-                status_code = 200
-                response_headers = {}
-
-            await page.wait_for_selector('body')
-            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
-
-            js_code = kwargs.get("js_code", kwargs.get("js", self.js_code))
-            if js_code:
-                if isinstance(js_code, str):
-                    await page.evaluate(js_code)
-                elif isinstance(js_code, list):
-                    for js in js_code:
-                        await page.evaluate(js)
-                
-                await page.wait_for_load_state('networkidle')
-                # Check for on execution event
-                await self.execute_hook('on_execution_started', page)
-                
-            if kwargs.get("simulate_user", False):
-                # Simulate user interactions
-                await page.mouse.move(100, 100)
-                await page.mouse.down()
-                await page.mouse.up()
-                await page.keyboard.press('ArrowDown')
-
-            # Handle the wait_for parameter
-            wait_for = kwargs.get("wait_for")
-            if wait_for:
-                try:
-                    await self.smart_wait(page, wait_for, timeout=kwargs.get("page_timeout", 60000))
-                except Exception as e:
-                    raise RuntimeError(f"Wait condition failed: {str(e)}")
-
-
-            
-            # Update image dimensions
-            update_image_dimensions_js = """
-            () => {
-                return new Promise((resolve) => {
-                    const filterImage = (img) => {
-                        // Filter out images that are too small
-                        if (img.width < 100 && img.height < 100) return false;
-                        
-                        // Filter out images that are not visible
-                        const rect = img.getBoundingClientRect();
-                        if (rect.width === 0 || rect.height === 0) return false;
-                        
-                        // Filter out images with certain class names (e.g., icons, thumbnails)
-                        if (img.classList.contains('icon') || img.classList.contains('thumbnail')) return false;
-                        
-                        // Filter out images with certain patterns in their src (e.g., placeholder images)
-                        if (img.src.includes('placeholder') || img.src.includes('icon')) return false;
-                        
-                        return true;
-                    };
-
-                    const images = Array.from(document.querySelectorAll('img')).filter(filterImage);
-                    let imagesLeft = images.length;
-                    
-                    if (imagesLeft === 0) {
-                        resolve();
-                        return;
-                    }
-
-                    const checkImage = (img) => {
-                        if (img.complete && img.naturalWidth !== 0) {
-                            img.setAttribute('width', img.naturalWidth);
-                            img.setAttribute('height', img.naturalHeight);
-                            imagesLeft--;
-                            if (imagesLeft === 0) resolve();
-                        }
-                    };
-
-                    images.forEach(img => {
-                        checkImage(img);
-                        if (!img.complete) {
-                            img.onload = () => {
-                                checkImage(img);
-                            };
-                            img.onerror = () => {
-                                imagesLeft--;
-                                if (imagesLeft === 0) resolve();
-                            };
-                        }
-                    });
-
-                    // Fallback timeout of 5 seconds
-                    setTimeout(() => resolve(), 5000);
-                });
-            }
-            """
-            await page.evaluate(update_image_dimensions_js)
-
-            # Wait a bit for any onload events to complete
-            await page.wait_for_timeout(100)
-
-            # Process iframes
-            if kwargs.get("process_iframes", False):
-                page = await self.process_iframes(page)
-            
-            await self.execute_hook('before_retrieve_html', page)
-            # Check if delay_before_return_html is set then wait for that time
-            delay_before_return_html = kwargs.get("delay_before_return_html")
-            if delay_before_return_html:
-                await asyncio.sleep(delay_before_return_html)
-                
-            html = await page.content()
-            await self.execute_hook('before_return_html', page, html)
-            
-            # Check if kwargs has screenshot=True then take screenshot
-            screenshot_data = None
-            if kwargs.get("screenshot"):
-                screenshot_data = await self.take_screenshot(url)            
-
-            if self.verbose:
-                print(f"[LOG] ✅ Crawled {url} successfully!")
-
-            if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
-                with open(cache_file_path, "w", encoding="utf-8") as f:
-                    f.write(html)
-                # store response headers and status code in cache
-                with open(cache_file_path + ".meta", "w", encoding="utf-8") as f:
-                    json.dump({
-                        "response_headers": response_headers,
-                        "status_code": status_code
-                    }, f)
-
-            async def get_delayed_content(delay: float = 5.0) -> str:
-                if self.verbose:
-                    print(f"[LOG] Waiting for {delay} seconds before retrieving content for {url}")
-                await asyncio.sleep(delay)
-                return await page.content()
-                
-            response = AsyncCrawlResponse(
-                html=html, 
-                response_headers=response_headers, 
-                status_code=status_code,
-                screenshot=screenshot_data,
-                get_delayed_content=get_delayed_content
-            )
-            return response
-        except Error as e:
-            raise Error(f"Failed to crawl {url}: {str(e)}")
-        finally:
-            if not session_id:
-                await page.close()
-                await context.close()
-
-    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        semaphore_count = kwargs.get('semaphore_count', 5)  # Adjust as needed
-        semaphore = asyncio.Semaphore(semaphore_count)
-
-        async def crawl_with_semaphore(url):
-            async with semaphore:
-                return await self.crawl(url, **kwargs)
-
-        tasks = [crawl_with_semaphore(url) for url in urls]
-        results = await asyncio.gather(*tasks, return_exceptions=True)
-        return [result if not isinstance(result, Exception) else str(result) for result in results]
-
-    async def take_screenshot(self, url: str, wait_time=1000) -> str:
-        async with await self.browser.new_context(user_agent=self.user_agent) as context:
-            page = await context.new_page()
-            try:
-                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
-                # Wait for a specified time (default is 1 second)
-                await page.wait_for_timeout(wait_time)
-                screenshot = await page.screenshot(full_page=True)
-                return base64.b64encode(screenshot).decode('utf-8')
-            except Exception as e:
-                error_message = f"Failed to take screenshot: {str(e)}"
-                print(error_message)
-
-                # Generate an error image
-                img = Image.new('RGB', (800, 600), color='black')
-                draw = ImageDraw.Draw(img)
-                font = ImageFont.load_default()
-                draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
-                
-                buffered = BytesIO()
-                img.save(buffered, format="JPEG")
-                return base64.b64encode(buffered.getvalue()).decode('utf-8')
-            finally:
-                await page.close()
-

crawl4ai/async_crawler_strategy.py

@@ -1,35 +1,17 @@
 import asyncio
-import base64
-import time
+import base64, time
 from abc import ABC, abstractmethod
 from typing import Callable, Dict, Any, List, Optional, Awaitable
 import os
 from playwright.async_api import async_playwright, Page, Browser, Error
 from io import BytesIO
 from PIL import Image, ImageDraw, ImageFont
+from .utils import sanitize_input_encode, calculate_semaphore_count
+import json, uuid
+import hashlib
 from pathlib import Path
 from playwright.async_api import ProxySettings
 from pydantic import BaseModel
-import hashlib
-import json
-import uuid
-from playwright_stealth import StealthConfig, stealth_async
-
-stealth_config = StealthConfig(
-    webdriver=True,
-    chrome_app=True,
-    chrome_csi=True,
-    chrome_load_times=True,
-    chrome_runtime=True,
-    navigator_languages=True,
-    navigator_plugins=True,
-    navigator_permissions=True,
-    webgl_vendor=True,
-    outerdimensions=True,
-    navigator_hardware_concurrency=True,
-    media_codecs=True,
-)
-
 
 class AsyncCrawlResponse(BaseModel):
     html: str
@@ -51,7 +33,7 @@ class AsyncCrawlerStrategy(ABC):
         pass
     
     @abstractmethod
-    async def take_screenshot(self, **kwargs) -> str:
+    async def take_screenshot(self, url: str) -> str:
         pass
     
     @abstractmethod
@@ -65,15 +47,10 @@ class AsyncCrawlerStrategy(ABC):
 class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
     def __init__(self, use_cached_html=False, js_code=None, **kwargs):
         self.use_cached_html = use_cached_html
-        self.user_agent = kwargs.get(
-            "user_agent",
-            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
-            "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
-        )
+        self.user_agent = kwargs.get("user_agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
         self.proxy = kwargs.get("proxy")
-        self.proxy_config = kwargs.get("proxy_config")
         self.headless = kwargs.get("headless", True)
-        self.browser_type = kwargs.get("browser_type", "chromium")
+        self.browser_type = kwargs.get("browser_type", "chromium")  # New parameter
         self.headers = kwargs.get("headers", {})
         self.sessions = {}
         self.session_ttl = 1800 
@@ -81,7 +58,6 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         self.verbose = kwargs.get("verbose", False)
         self.playwright = None
         self.browser = None
-        self.sleep_on_close = kwargs.get("sleep_on_close", False)
         self.hooks = {
             'on_browser_created': None,
             'on_user_agent_updated': None,
@@ -107,14 +83,9 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 "headless": self.headless,
                 "args": [
                     "--disable-gpu",
-                    "--no-sandbox",
                     "--disable-dev-shm-usage",
-                    "--disable-blink-features=AutomationControlled",
-                    "--disable-infobars",
-                    "--window-position=0,0",
-                    "--ignore-certificate-errors",
-                    "--ignore-certificate-errors-spki-list",
-                    # "--headless=new",  # Use the new headless mode
+                    "--disable-setuid-sandbox",
+                    "--no-sandbox",
                 ]
             }
             
@@ -122,9 +93,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             if self.proxy:
                 proxy_settings = ProxySettings(server=self.proxy)
                 browser_args["proxy"] = proxy_settings
-            elif self.proxy_config:
-                proxy_settings = ProxySettings(server=self.proxy_config.get("server"), username=self.proxy_config.get("username"), password=self.proxy_config.get("password"))
-                browser_args["proxy"] = proxy_settings
+                
                 
             # Select the appropriate browser based on the browser_type
             if self.browser_type == "firefox":
@@ -137,8 +106,6 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             await self.execute_hook('on_browser_created', self.browser)
 
     async def close(self):
-        if self.sleep_on_close:
-            await asyncio.sleep(0.5)
         if self.browser:
             await self.browser.close()
             self.browser = None
@@ -180,10 +147,8 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
 
     def _cleanup_expired_sessions(self):
         current_time = time.time()
-        expired_sessions = [
-            sid for sid, (_, _, last_used) in self.sessions.items() 
-            if current_time - last_used > self.session_ttl
-        ]
+        expired_sessions = [sid for sid, (_, _, last_used) in self.sessions.items() 
+                            if current_time - last_used > self.session_ttl]
         for sid in expired_sessions:
             asyncio.create_task(self.kill_session(sid))
             
@@ -223,8 +188,8 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                             return await self.csp_compliant_wait(page, f"() => {{{wait_for}}}", timeout)
                         except Error:
                             raise ValueError(f"Invalid wait_for parameter: '{wait_for}'. "
-                                             "It should be either a valid CSS selector, a JavaScript function, "
-                                             "or explicitly prefixed with 'js:' or 'css:'.")
+                                            "It should be either a valid CSS selector, a JavaScript function, "
+                                            "or explicitly prefixed with 'js:' or 'css:'.")
     
     async def csp_compliant_wait(self, page: Page, user_wait_function: str, timeout: float = 30000):
         wrapper_js = f"""
@@ -289,7 +254,8 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 print(f"Error processing iframe {i}: {str(e)}")
 
         # Return the page object
-        return page  
+        return page
+    
     
     async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
         response_headers = {}
@@ -302,70 +268,25 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             if not context:
                 context = await self.browser.new_context(
                     user_agent=self.user_agent,
-                    viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None,
-                    accept_downloads=True,
-                    java_script_enabled=True
+                    proxy={"server": self.proxy} if self.proxy else None
                 )
-                await context.add_cookies([{"name": "cookiesEnabled", "value": "true", "url": url}])
                 await context.set_extra_http_headers(self.headers)
                 page = await context.new_page()
                 self.sessions[session_id] = (context, page, time.time())
         else:
             context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
+                    user_agent=self.user_agent,
+                    proxy={"server": self.proxy} if self.proxy else None
             )
             await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False) or kwargs.get("simulate_user", False) or kwargs.get("magic", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
             page = await context.new_page()
-            # await stealth_async(page) #, stealth_config)
 
-        # Add console message and error logging
-        if kwargs.get("log_console", False):
-            page.on("console", lambda msg: print(f"Console: {msg.text}"))
-            page.on("pageerror", lambda exc: print(f"Page Error: {exc}"))
-        
         try:
             if self.verbose:
                 print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
 
             if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
+                cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest())
                 if os.path.exists(cache_file_path):
                     html = ""
                     with open(cache_file_path, "r") as f:
@@ -375,21 +296,12 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                         meta = json.load(f)
                         response_headers = meta.get("response_headers", {})
                         status_code = meta.get("status_code")
-                    response = AsyncCrawlResponse(
-                        html=html, response_headers=response_headers, status_code=status_code
-                    )
+                    response = AsyncCrawlResponse(html=html, response_headers=response_headers, status_code=status_code)
                     return response
 
             if not kwargs.get("js_only", False):
                 await self.execute_hook('before_goto', page)
-                
-                response = await page.goto(
-                    url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000)
-                )
-                
-                # response = await page.goto("about:blank")
-                # await page.evaluate(f"window.location.href = '{url}'")
-                
+                response = await page.goto(url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000))
                 await self.execute_hook('after_goto', page)
                 
                 # Get status code and headers
@@ -399,30 +311,37 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 status_code = 200
                 response_headers = {}
 
+
             await page.wait_for_selector('body')
-            
             await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
 
             js_code = kwargs.get("js_code", kwargs.get("js", self.js_code))
             if js_code:
                 if isinstance(js_code, str):
-                    await page.evaluate(js_code)
+                    r = await page.evaluate(js_code)
                 elif isinstance(js_code, list):
                     for js in js_code:
                         await page.evaluate(js)
                 
+                # await page.wait_for_timeout(100)
                 await page.wait_for_load_state('networkidle')
-                # Check for on execution event
+                # Check for on execution even
                 await self.execute_hook('on_execution_started', page)
                 
-            if kwargs.get("simulate_user", False) or kwargs.get("magic", False):
-                # Simulate user interactions
-                await page.mouse.move(100, 100)
-                await page.mouse.down()
-                await page.mouse.up()
-                await page.keyboard.press('ArrowDown')
-
-            # Handle the wait_for parameter
+            # New code to handle the wait_for parameter
+            # Example usage:
+            # await crawler.crawl(
+            #     url,
+            #     js_code="// some JavaScript code",
+            #     wait_for="""() => {
+            #         return document.querySelector('#my-element') !== null;
+            #     }"""
+            # )
+            # Example of using a CSS selector:
+            # await crawler.crawl(
+            #     url,
+            #     wait_for="#my-element"
+            # )
             wait_for = kwargs.get("wait_for")
             if wait_for:
                 try:
@@ -430,7 +349,13 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 except Exception as e:
                     raise RuntimeError(f"Wait condition failed: {str(e)}")
 
-            # Update image dimensions
+            # Check if kwargs has screenshot=True then take screenshot
+            screenshot_data = None
+            if kwargs.get("screenshot"):
+                screenshot_data = await self.take_screenshot(url)
+            
+            
+            # New code to update image dimensions
             update_image_dimensions_js = """
             () => {
                 return new Promise((resolve) => {
@@ -482,8 +407,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                     });
 
                     // Fallback timeout of 5 seconds
-                    // setTimeout(() => resolve(), 5000);
-                    resolve();
+                    setTimeout(() => resolve(), 5000);
                 });
             }
             """
@@ -502,29 +426,14 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             if delay_before_return_html:
                 await asyncio.sleep(delay_before_return_html)
                 
-            # Check for remove_overlay_elements parameter
-            if kwargs.get("remove_overlay_elements", False):
-                await self.remove_overlay_elements(page)
-            
             html = await page.content()
             await self.execute_hook('before_return_html', page, html)
-            
-            # Check if kwargs has screenshot=True then take screenshot
-            screenshot_data = None
-            if kwargs.get("screenshot"):
-                # Check we have screenshot_wait_for parameter, if we have simply wait for that time
-                screenshot_wait_for = kwargs.get("screenshot_wait_for")
-                if screenshot_wait_for:
-                    await asyncio.sleep(screenshot_wait_for)
-                screenshot_data = await self.take_screenshot(page)          
 
             if self.verbose:
                 print(f"[LOG] ✅ Crawled {url} successfully!")
 
             if self.use_cached_html:
-                cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
-                )
+                cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest())
                 with open(cache_file_path, "w", encoding="utf-8") as f:
                     f.write(html)
                 # store response headers and status code in cache
@@ -534,6 +443,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                         "status_code": status_code
                     }, f)
 
+            
             async def get_delayed_content(delay: float = 5.0) -> str:
                 if self.verbose:
                     print(f"[LOG] Waiting for {delay} seconds before retrieving content for {url}")
@@ -549,14 +459,63 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
             )
             return response
         except Error as e:
-            raise Error(f"[ERROR] 🚫 crawl(): Failed to crawl {url}: {str(e)}")
-        # finally:
-        #     if not session_id:
-        #         await page.close()
-        #         await context.close()
+            raise Error(f"Failed to crawl {url}: {str(e)}")
+        finally:
+            if not session_id:
+                await page.close()
 
+        # try:
+        #     html = await _crawl()
+        #     return sanitize_input_encode(html)
+        # except Error as e:
+        #     raise Error(f"Failed to crawl {url}: {str(e)}")
+        # except Exception as e:
+        #     raise Exception(f"Failed to crawl {url}: {str(e)}")
+
+    async def execute_js(self, session_id: str, js_code: str, wait_for_js: str = None, wait_for_css: str = None) -> AsyncCrawlResponse:
+        """
+        Execute JavaScript code in a specific session and optionally wait for a condition.
+        
+        :param session_id: The ID of the session to execute the JS code in.
+        :param js_code: The JavaScript code to execute.
+        :param wait_for_js: JavaScript condition to wait for after execution.
+        :param wait_for_css: CSS selector to wait for after execution.
+        :return: AsyncCrawlResponse containing the page's HTML and other information.
+        :raises ValueError: If the session does not exist.
+        """
+        if not session_id:
+            raise ValueError("Session ID must be provided")
+        
+        if session_id not in self.sessions:
+            raise ValueError(f"No active session found for session ID: {session_id}")
+        
+        context, page, last_used = self.sessions[session_id]
+        
+        try:
+            await page.evaluate(js_code)
+            
+            if wait_for_js:
+                await page.wait_for_function(wait_for_js)
+            
+            if wait_for_css:
+                await page.wait_for_selector(wait_for_css)
+            
+            # Get the updated HTML content
+            html = await page.content()
+            
+            # Get response headers and status code (assuming these are available)
+            response_headers = await page.evaluate("() => JSON.stringify(performance.getEntriesByType('resource')[0].responseHeaders)")
+            status_code = await page.evaluate("() => performance.getEntriesByType('resource')[0].responseStatus")
+            
+            # Update the last used time for this session
+            self.sessions[session_id] = (context, page, time.time())
+            
+            return AsyncCrawlResponse(html=html, response_headers=response_headers, status_code=status_code)
+        except Error as e:
+            raise Error(f"Failed to execute JavaScript or wait for condition in session {session_id}: {str(e)}")
+    
     async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
-        semaphore_count = kwargs.get('semaphore_count', 5)  # Adjust as needed
+        semaphore_count = kwargs.get('semaphore_count', calculate_semaphore_count())
         semaphore = asyncio.Semaphore(semaphore_count)
 
         async def crawl_with_semaphore(url):
@@ -567,156 +526,27 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         results = await asyncio.gather(*tasks, return_exceptions=True)
         return [result if not isinstance(result, Exception) else str(result) for result in results]
 
-    async def remove_overlay_elements(self, page: Page) -> None:
-        """
-        Removes popup overlays, modals, cookie notices, and other intrusive elements from the page.
-        
-        Args:
-            page (Page): The Playwright page instance
-        """
-        remove_overlays_js = """
-        async () => {
-            // Function to check if element is visible
-            const isVisible = (elem) => {
-                const style = window.getComputedStyle(elem);
-                return style.display !== 'none' && 
-                       style.visibility !== 'hidden' && 
-                       style.opacity !== '0';
-            };
+    async def take_screenshot(self, url: str, wait_time = 1000) -> str:
+        async with await self.browser.new_context(user_agent=self.user_agent) as context:
+            page = await context.new_page()
+            try:
+                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
+                # Wait for a specified time (default is 1 second)
+                await page.wait_for_timeout(wait_time)
+                screenshot = await page.screenshot(full_page=True)
+                return base64.b64encode(screenshot).decode('utf-8')
+            except Exception as e:
+                error_message = f"Failed to take screenshot: {str(e)}"
+                print(error_message)
 
-            // Common selectors for popups and overlays
-            const commonSelectors = [
-                // Close buttons first
-                'button[class*="close" i]', 'button[class*="dismiss" i]', 
-                'button[aria-label*="close" i]', 'button[title*="close" i]',
-                'a[class*="close" i]', 'span[class*="close" i]',
+                # Generate an error image
+                img = Image.new('RGB', (800, 600), color='black')
+                draw = ImageDraw.Draw(img)
+                font = ImageFont.load_default()
+                draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
                 
-                // Cookie notices
-                '[class*="cookie-banner" i]', '[id*="cookie-banner" i]',
-                '[class*="cookie-consent" i]', '[id*="cookie-consent" i]',
-                
-                // Newsletter/subscription dialogs
-                '[class*="newsletter" i]', '[class*="subscribe" i]',
-                
-                // Generic popups/modals
-                '[class*="popup" i]', '[class*="modal" i]', 
-                '[class*="overlay" i]', '[class*="dialog" i]',
-                '[role="dialog"]', '[role="alertdialog"]'
-            ];
-
-            // Try to click close buttons first
-            for (const selector of commonSelectors.slice(0, 6)) {
-                const closeButtons = document.querySelectorAll(selector);
-                for (const button of closeButtons) {
-                    if (isVisible(button)) {
-                        try {
-                            button.click();
-                            await new Promise(resolve => setTimeout(resolve, 100));
-                        } catch (e) {
-                            console.log('Error clicking button:', e);
-                        }
-                    }
-                }
-            }
-
-            // Remove remaining overlay elements
-            const removeOverlays = () => {
-                // Find elements with high z-index
-                const allElements = document.querySelectorAll('*');
-                for (const elem of allElements) {
-                    const style = window.getComputedStyle(elem);
-                    const zIndex = parseInt(style.zIndex);
-                    const position = style.position;
-                    
-                    if (
-                        isVisible(elem) && 
-                        (zIndex > 999 || position === 'fixed' || position === 'absolute') &&
-                        (
-                            elem.offsetWidth > window.innerWidth * 0.5 ||
-                            elem.offsetHeight > window.innerHeight * 0.5 ||
-                            style.backgroundColor.includes('rgba') ||
-                            parseFloat(style.opacity) < 1
-                        )
-                    ) {
-                        elem.remove();
-                    }
-                }
-
-                // Remove elements matching common selectors
-                for (const selector of commonSelectors) {
-                    const elements = document.querySelectorAll(selector);
-                    elements.forEach(elem => {
-                        if (isVisible(elem)) {
-                            elem.remove();
-                        }
-                    });
-                }
-            };
-
-            // Remove overlay elements
-            removeOverlays();
-
-            // Remove any fixed/sticky position elements at the top/bottom
-            const removeFixedElements = () => {
-                const elements = document.querySelectorAll('*');
-                elements.forEach(elem => {
-                    const style = window.getComputedStyle(elem);
-                    if (
-                        (style.position === 'fixed' || style.position === 'sticky') &&
-                        isVisible(elem)
-                    ) {
-                        elem.remove();
-                    }
-                });
-            };
-
-            removeFixedElements();
-            
-            // Remove empty block elements as: div, p, span, etc.
-            const removeEmptyBlockElements = () => {
-                const blockElements = document.querySelectorAll('div, p, span, section, article, header, footer, aside, nav, main, ul, ol, li, dl, dt, dd, h1, h2, h3, h4, h5, h6');
-                blockElements.forEach(elem => {
-                    if (elem.innerText.trim() === '') {
-                        elem.remove();
-                    }
-                });
-            };
-
-            // Remove margin-right and padding-right from body (often added by modal scripts)
-            document.body.style.marginRight = '0px';
-            document.body.style.paddingRight = '0px';
-            document.body.style.overflow = 'auto';
-
-            // Wait a bit for any animations to complete
-            await new Promise(resolve => setTimeout(resolve, 100));
-        }
-        """
-        
-        try:
-            await page.evaluate(remove_overlays_js)
-            await page.wait_for_timeout(500)  # Wait for any animations to complete
-        except Exception as e:
-            if self.verbose:
-                print(f"Warning: Failed to remove overlay elements: {str(e)}")
-
-    async def take_screenshot(self, page: Page) -> str:
-        try:
-            # The page is already loaded, just take the screenshot
-            screenshot = await page.screenshot(full_page=True)
-            return base64.b64encode(screenshot).decode('utf-8')
-        except Exception as e:
-            error_message = f"Failed to take screenshot: {str(e)}"
-            print(error_message)
-
-            # Generate an error image
-            img = Image.new('RGB', (800, 600), color='black')
-            draw = ImageDraw.Draw(img)
-            font = ImageFont.load_default()
-            draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
-            
-            buffered = BytesIO()
-            img.save(buffered, format="JPEG")
-            return base64.b64encode(buffered.getvalue()).decode('utf-8')
-        finally:
-            await page.close()
-
+                buffered = BytesIO()
+                img.save(buffered, format="JPEG")
+                return base64.b64encode(buffered.getvalue()).decode('utf-8')
+            finally:
+                await page.close()

crawl4ai/async_webcrawler.py

@@ -23,15 +23,13 @@ class AsyncWebCrawler:
         self,
         crawler_strategy: Optional[AsyncCrawlerStrategy] = None,
         always_by_pass_cache: bool = False,
-        base_directory: str = str(Path.home()),
         **kwargs,
     ):
         self.crawler_strategy = crawler_strategy or AsyncPlaywrightCrawlerStrategy(
             **kwargs
         )
         self.always_by_pass_cache = always_by_pass_cache
-        # self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
-        self.crawl4ai_folder = os.path.join(base_directory, ".crawl4ai")
+        self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
         os.makedirs(self.crawl4ai_folder, exist_ok=True)
         os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
         self.ready = False
@@ -135,8 +133,8 @@ class AsyncWebCrawler:
         except Exception as e:
             if not hasattr(e, "msg"):
                 e.msg = str(e)
-            print(f"[ERROR] 🚫 arun(): Failed to crawl {url}, error: {e.msg}")
-            return CrawlResult(url=url, html="", markdown = f"[ERROR] 🚫 arun(): Failed to crawl {url}, error: {e.msg}", success=False, error_message=e.msg)
+            print(f"[ERROR] 🚫 Failed to crawl {url}, error: {e.msg}")
+            return CrawlResult(url=url, html="", success=False, error_message=e.msg)
 
     async def arun_many(
         self,
@@ -188,8 +186,7 @@ class AsyncWebCrawler:
         try:
             t1 = time.time()
             scrapping_strategy = WebScrappingStrategy()
-            # result = await scrapping_strategy.ascrap(
-            result = scrapping_strategy.scrap(
+            result = await scrapping_strategy.ascrap(
                 url,
                 html,
                 word_count_threshold=word_count_threshold,
@@ -198,7 +195,6 @@ class AsyncWebCrawler:
                 image_description_min_word_threshold=kwargs.get(
                     "image_description_min_word_threshold", IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD
                 ),
-                **kwargs,
             )
             if verbose:
                 print(
@@ -214,8 +210,6 @@ class AsyncWebCrawler:
 
         cleaned_html = sanitize_input_encode(result.get("cleaned_html", ""))
         markdown = sanitize_input_encode(result.get("markdown", ""))
-        fit_markdown = sanitize_input_encode(result.get("fit_markdown", ""))
-        fit_html = sanitize_input_encode(result.get("fit_html", ""))
         media = result.get("media", [])
         links = result.get("links", [])
         metadata = result.get("metadata", {})
@@ -262,8 +256,6 @@ class AsyncWebCrawler:
             html=html,
             cleaned_html=format_html(cleaned_html),
             markdown=markdown,
-            fit_markdown=fit_markdown,
-            fit_html= fit_html,
             media=media,
             links=links,
             metadata=metadata,

crawl4ai/chunking_strategy.py

@@ -84,12 +84,6 @@ class TopicSegmentationChunking(ChunkingStrategy):
 # Fixed-length word chunks
 class FixedLengthWordChunking(ChunkingStrategy):
     def __init__(self, chunk_size=100, **kwargs):
-        """
-        Initialize the fixed-length word chunking strategy with the given chunk size.
-        
-        Args:
-            chunk_size (int): The size of each chunk in words.
-        """
         self.chunk_size = chunk_size
 
     def chunk(self, text: str) -> list:
@@ -99,64 +93,14 @@ class FixedLengthWordChunking(ChunkingStrategy):
 # Sliding window chunking
 class SlidingWindowChunking(ChunkingStrategy):
     def __init__(self, window_size=100, step=50, **kwargs):
-        """
-        Initialize the sliding window chunking strategy with the given window size and
-        step size.
-        
-        Args:
-            window_size (int): The size of the sliding window in words.
-            step (int): The step size for sliding the window in words.
-        """
         self.window_size = window_size
         self.step = step
 
     def chunk(self, text: str) -> list:
         words = text.split()
         chunks = []
-        
-        if len(words) <= self.window_size:
-            return [text]
-        
-        for i in range(0, len(words) - self.window_size + 1, self.step):
-            chunk = ' '.join(words[i:i + self.window_size])
-            chunks.append(chunk)
-        
-        # Handle the last chunk if it doesn't align perfectly
-        if i + self.window_size < len(words):
-            chunks.append(' '.join(words[-self.window_size:]))
-        
+        for i in range(0, len(words), self.step):
+            chunks.append(' '.join(words[i:i + self.window_size]))
         return chunks
     
 
-class OverlappingWindowChunking(ChunkingStrategy):
-    def __init__(self, window_size=1000, overlap=100, **kwargs):
-        """
-        Initialize the overlapping window chunking strategy with the given window size and
-        overlap size.
-        
-        Args:
-            window_size (int): The size of the window in words.
-            overlap (int): The size of the overlap between consecutive chunks in words.
-        """
-        self.window_size = window_size
-        self.overlap = overlap
-
-    def chunk(self, text: str) -> list:
-        words = text.split()
-        chunks = []
-        
-        if len(words) <= self.window_size:
-            return [text]
-        
-        start = 0
-        while start < len(words):
-            end = start + self.window_size
-            chunk = ' '.join(words[start:end])
-            chunks.append(chunk)
-            
-            if end >= len(words):
-                break
-            
-            start = end - self.overlap
-        
-        return chunks

crawl4ai/config.py

@@ -4,23 +4,24 @@ from dotenv import load_dotenv
 load_dotenv()  # Load environment variables from .env file
 
 # Default provider, ONLY used when the extraction strategy is LLMExtractionStrategy
-DEFAULT_PROVIDER = "openai/gpt-4o-mini"
+DEFAULT_PROVIDER = "openai/gpt-4-turbo"
 MODEL_REPO_BRANCH = "new-release-0.0.2"
 # Provider-model dictionary, ONLY used when the extraction strategy is LLMExtractionStrategy
 PROVIDER_MODELS = {
     "ollama/llama3": "no-token-needed", # Any model from Ollama no need for API token
     "groq/llama3-70b-8192": os.getenv("GROQ_API_KEY"),
     "groq/llama3-8b-8192": os.getenv("GROQ_API_KEY"),
-    "openai/gpt-4o-mini": os.getenv("OPENAI_API_KEY"),
+    "openai/gpt-3.5-turbo": os.getenv("OPENAI_API_KEY"),
+    "openai/gpt-4-turbo": os.getenv("OPENAI_API_KEY"),
     "openai/gpt-4o": os.getenv("OPENAI_API_KEY"),
     "anthropic/claude-3-haiku-20240307": os.getenv("ANTHROPIC_API_KEY"),
     "anthropic/claude-3-opus-20240229": os.getenv("ANTHROPIC_API_KEY"),
     "anthropic/claude-3-sonnet-20240229": os.getenv("ANTHROPIC_API_KEY"),
-    "anthropic/claude-3-5-sonnet-20240620": os.getenv("ANTHROPIC_API_KEY"),
 }
 
+
 # Chunk token threshold
-CHUNK_TOKEN_THRESHOLD = 2 ** 11 # 2048 tokens
+CHUNK_TOKEN_THRESHOLD = 500
 OVERLAP_RATE = 0.1
 WORD_TOKEN_RATE = 1.3
 
@@ -28,20 +29,6 @@ WORD_TOKEN_RATE = 1.3
 MIN_WORD_THRESHOLD = 1
 IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD = 1
 
-IMPORTANT_ATTRS = ['src', 'href', 'alt', 'title', 'width', 'height'] 
-ONLY_TEXT_ELIGIBLE_TAGS = ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']
-SOCIAL_MEDIA_DOMAINS = [
-                            'facebook.com',
-                            'twitter.com',
-                            'x.com',
-                            'linkedin.com',
-                            'instagram.com',
-                            'pinterest.com',
-                            'tiktok.com',
-                            'snapchat.com',
-                            'reddit.com',
-                        ]
-
 # Threshold for the Image extraction - Range is 1 to 6
 # Images are scored based on point based system, to filter based on usefulness. Points are assigned
 # to each image based on the following aspects.

crawl4ai/content_cleaning_strategy.py

@@ -1,196 +0,0 @@
-from bs4 import BeautifulSoup, Tag
-import re
-from typing import Optional
-
-class ContentCleaningStrategy:
-    def __init__(self):
-        # Precompile regex patterns for performance
-        self.negative_patterns = re.compile(r'nav|footer|header|sidebar|ads|comment', re.I)
-        self.positive_patterns = re.compile(r'content|article|main|post', re.I)
-        self.priority_tags = {'article', 'main', 'section', 'div'}
-        self.non_content_tags = {'nav', 'footer', 'header', 'aside'}
-        # Thresholds
-        self.text_density_threshold = 9.0
-        self.min_word_count = 50
-        self.link_density_threshold = 0.2
-        self.max_dom_depth = 10  # To prevent excessive DOM traversal
-
-    def clean(self, clean_html: str) -> str:
-        """
-        Main function that takes cleaned HTML and returns super cleaned HTML.
-
-        Args:
-            clean_html (str): The cleaned HTML content.
-
-        Returns:
-            str: The super cleaned HTML containing only the main content.
-        """
-        try:
-            if not clean_html or not isinstance(clean_html, str):
-                return ''
-            soup = BeautifulSoup(clean_html, 'html.parser')
-            main_content = self.extract_main_content(soup)
-            if main_content:
-                super_clean_element = self.clean_element(main_content)
-                return str(super_clean_element)
-            else:
-                return ''
-        except Exception:
-            # Handle exceptions silently or log them as needed
-            return ''
-
-    def extract_main_content(self, soup: BeautifulSoup) -> Optional[Tag]:
-        """
-        Identifies and extracts the main content element from the HTML.
-
-        Args:
-            soup (BeautifulSoup): The parsed HTML soup.
-
-        Returns:
-            Optional[Tag]: The Tag object containing the main content, or None if not found.
-        """
-        candidates = []
-        for element in soup.find_all(self.priority_tags):
-            if self.is_non_content_tag(element):
-                continue
-            if self.has_negative_class_id(element):
-                continue
-            score = self.calculate_content_score(element)
-            candidates.append((score, element))
-        
-        if not candidates:
-            return None
-
-        # Sort candidates by score in descending order
-        candidates.sort(key=lambda x: x[0], reverse=True)
-        # Select the element with the highest score
-        best_element = candidates[0][1]
-        return best_element
-
-    def calculate_content_score(self, element: Tag) -> float:
-        """
-        Calculates a score for an element based on various heuristics.
-
-        Args:
-            element (Tag): The HTML element to score.
-
-        Returns:
-            float: The content score of the element.
-        """
-        score = 0.0
-
-        if self.is_priority_tag(element):
-            score += 5.0
-        if self.has_positive_class_id(element):
-            score += 3.0
-        if self.has_negative_class_id(element):
-            score -= 3.0
-        if self.is_high_text_density(element):
-            score += 2.0
-        if self.is_low_link_density(element):
-            score += 2.0
-        if self.has_sufficient_content(element):
-            score += 2.0
-        if self.has_headings(element):
-            score += 3.0
-
-        dom_depth = self.calculate_dom_depth(element)
-        score += min(dom_depth, self.max_dom_depth) * 0.5  # Adjust weight as needed
-
-        return score
-
-    def is_priority_tag(self, element: Tag) -> bool:
-        """Checks if the element is a priority tag."""
-        return element.name in self.priority_tags
-
-    def is_non_content_tag(self, element: Tag) -> bool:
-        """Checks if the element is a non-content tag."""
-        return element.name in self.non_content_tags
-
-    def has_negative_class_id(self, element: Tag) -> bool:
-        """Checks if the element has negative indicators in its class or id."""
-        class_id = ' '.join(filter(None, [
-            self.get_attr_str(element.get('class')),
-            element.get('id', '')
-        ]))
-        return bool(self.negative_patterns.search(class_id))
-
-    def has_positive_class_id(self, element: Tag) -> bool:
-        """Checks if the element has positive indicators in its class or id."""
-        class_id = ' '.join(filter(None, [
-            self.get_attr_str(element.get('class')),
-            element.get('id', '')
-        ]))
-        return bool(self.positive_patterns.search(class_id))
-
-    @staticmethod
-    def get_attr_str(attr) -> str:
-        """Converts an attribute value to a string."""
-        if isinstance(attr, list):
-            return ' '.join(attr)
-        elif isinstance(attr, str):
-            return attr
-        else:
-            return ''
-
-    def is_high_text_density(self, element: Tag) -> bool:
-        """Determines if the element has high text density."""
-        text_density = self.calculate_text_density(element)
-        return text_density > self.text_density_threshold
-
-    def calculate_text_density(self, element: Tag) -> float:
-        """Calculates the text density of an element."""
-        text_length = len(element.get_text(strip=True))
-        tag_count = len(element.find_all())
-        tag_count = tag_count or 1  # Prevent division by zero
-        return text_length / tag_count
-
-    def is_low_link_density(self, element: Tag) -> bool:
-        """Determines if the element has low link density."""
-        link_density = self.calculate_link_density(element)
-        return link_density < self.link_density_threshold
-
-    def calculate_link_density(self, element: Tag) -> float:
-        """Calculates the link density of an element."""
-        text = element.get_text(strip=True)
-        if not text:
-            return 0.0
-        link_text = ' '.join(a.get_text(strip=True) for a in element.find_all('a'))
-        return len(link_text) / len(text) if text else 0.0
-
-    def has_sufficient_content(self, element: Tag) -> bool:
-        """Checks if the element has sufficient word count."""
-        word_count = len(element.get_text(strip=True).split())
-        return word_count >= self.min_word_count
-
-    def calculate_dom_depth(self, element: Tag) -> int:
-        """Calculates the depth of an element in the DOM tree."""
-        depth = 0
-        current_element = element
-        while current_element.parent and depth < self.max_dom_depth:
-            depth += 1
-            current_element = current_element.parent
-        return depth
-
-    def has_headings(self, element: Tag) -> bool:
-        """Checks if the element contains heading tags."""
-        return bool(element.find(['h1', 'h2', 'h3']))
-
-    def clean_element(self, element: Tag) -> Tag:
-        """
-        Cleans the selected element by removing unnecessary attributes and nested non-content elements.
-
-        Args:
-            element (Tag): The HTML element to clean.
-
-        Returns:
-            Tag: The cleaned HTML element.
-        """
-        for tag in element.find_all(['script', 'style', 'aside']):
-            tag.decompose()
-        for tag in element.find_all():
-            attrs = dict(tag.attrs)
-            for attr in attrs:
-                if attr in ['style', 'onclick', 'onmouseover', 'align', 'bgcolor']:
-                    del tag.attrs[attr]
-        return element

crawl4ai/content_scrapping_strategy.py

@@ -7,17 +7,13 @@ from .config import *
 from bs4 import element, NavigableString, Comment
 from urllib.parse import urljoin
 from requests.exceptions import InvalidSchema
-from .content_cleaning_strategy import ContentCleaningStrategy
 
 from .utils import (
     sanitize_input_encode,
     sanitize_html,
     extract_metadata,
     InvalidCSSSelectorError,
-    CustomHTML2Text,
-    normalize_url,
-    is_external_url
-    
+    CustomHTML2Text
 )
 
 class ContentScrappingStrategy(ABC):
@@ -37,14 +33,12 @@ class WebScrappingStrategy(ContentScrappingStrategy):
         return await asyncio.to_thread(self._get_content_of_website_optimized, url, html, **kwargs)
 
     def _get_content_of_website_optimized(self, url: str, html: str, word_count_threshold: int = MIN_WORD_THRESHOLD, css_selector: str = None, **kwargs) -> Dict[str, Any]:
-        success = True
         if not html:
             return None
 
         soup = BeautifulSoup(html, 'html.parser')
         body = soup.body
         
-        
         image_description_min_word_threshold = kwargs.get('image_description_min_word_threshold', IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD)
 
         for tag in kwargs.get('excluded_tags', []) or []:
@@ -70,8 +64,6 @@ class WebScrappingStrategy(ContentScrappingStrategy):
 
         links = {'internal': [], 'external': []}
         media = {'images': [], 'videos': [], 'audios': []}
-        internal_links_dict = {}
-        external_links_dict = {}
 
         # Extract meaningful text for media files from closest parent
         def find_closest_parent_with_useful_text(tag):
@@ -133,11 +125,7 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                 image_width =  img.get('width')
                 width_value, width_unit = parse_dimension(image_width)
                 image_size = 0 #int(fetch_image_file_size(img,base_url) or 0)
-                image_src = img.get('src','')
-                if "data:image/" in image_src:
-                    image_format = image_src.split(',')[0].split(';')[0].split('/')[1]
-                else:
-                    image_format = os.path.splitext(img.get('src',''))[1].lower()
+                image_format = os.path.splitext(img.get('src',''))[1].lower()
                 # Remove . from format
                 image_format = image_format.strip('.').split('?')[0]
                 score = 0
@@ -161,8 +149,6 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                     score+=1
                 return score
 
-            
-            
             if not is_valid_image(img, img.parent, img.parent.get('class', [])):
                 return None
             score = score_image_for_usefulness(img, url, index, total_images)
@@ -177,19 +163,6 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                 'type': 'image'
             }
 
-        def remove_unwanted_attributes(element, important_attrs, keep_data_attributes=False):
-            attrs_to_remove = []
-            for attr in element.attrs:
-                if attr not in important_attrs:
-                    if keep_data_attributes:
-                        if not attr.startswith('data-'):
-                            attrs_to_remove.append(attr)
-                    else:
-                        attrs_to_remove.append(attr)
-            
-            for attr in attrs_to_remove:
-                del element[attr]
-        
         def process_element(element: element.PageElement) -> bool:
             try:
                 if isinstance(element, NavigableString):
@@ -206,106 +179,21 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                     return False
 
                 keep_element = False
-                
-                exclude_social_media_domains = SOCIAL_MEDIA_DOMAINS + kwargs.get('exclude_social_media_domains', [])
-                exclude_social_media_domains = list(set(exclude_social_media_domains))
 
-                
-                try:
-                    if element.name == 'a' and element.get('href'):
-                        href = element.get('href', '').strip()
-                        if not href:  # Skip empty hrefs
-                            return False
-                            
-                        url_base = url.split('/')[2]
-                        
-                        # Normalize the URL
-                        try:
-                            normalized_href = normalize_url(href, url)
-                        except ValueError as e:
-                            # logging.warning(f"Invalid URL format: {href}, Error: {str(e)}")
-                            return False
-                            
-                        link_data = {
-                            'href': normalized_href,
-                            'text': element.get_text().strip(),
-                            'title': element.get('title', '').strip()
-                        }
-                        
-                        # Check for duplicates and add to appropriate dictionary
-                        is_external = is_external_url(normalized_href, url_base)
-                        if is_external:
-                            if normalized_href not in external_links_dict:
-                                external_links_dict[normalized_href] = link_data
-                        else:
-                            if normalized_href not in internal_links_dict:
-                                internal_links_dict[normalized_href] = link_data
-                                
-                        keep_element = True
-                        
-                        # Handle external link exclusions
-                        if is_external:
-                            if kwargs.get('exclude_external_links', False):
-                                element.decompose()
-                                return False
-                            elif kwargs.get('exclude_social_media_links', False):
-                                if any(domain in normalized_href.lower() for domain in exclude_social_media_domains):
-                                    element.decompose()
-                                    return False
-                            elif kwargs.get('exclude_domains', []):
-                                if any(domain in normalized_href.lower() for domain in kwargs.get('exclude_domains', [])):
-                                    element.decompose()
-                                    return False
-                                    
-                except Exception as e:
-                    raise Exception(f"Error processing links: {str(e)}")
+                if element.name == 'a' and element.get('href'):
+                    href = element['href']
+                    url_base = url.split('/')[2]
+                    link_data = {'href': href, 'text': element.get_text()}
+                    if href.startswith('http') and url_base not in href:
+                        links['external'].append(link_data)
+                    else:
+                        links['internal'].append(link_data)
+                    keep_element = True
 
-                try:
-                    if element.name == 'img':
-                        potential_sources = ['src', 'data-src', 'srcset' 'data-lazy-src', 'data-original']
-                        src = element.get('src', '')
-                        while not src and potential_sources:
-                            src = element.get(potential_sources.pop(0), '')
-                        if not src:
-                            element.decompose()
-                            return False
-                        
-                        # If it is srcset pick up the first image
-                        if 'srcset' in element.attrs:
-                            src = element.attrs['srcset'].split(',')[0].split(' ')[0]
-                            
-                        # Check flag if we should remove external images
-                        if kwargs.get('exclude_external_images', False):
-                            src_url_base = src.split('/')[2]
-                            url_base = url.split('/')[2]
-                            if url_base not in src_url_base:
-                                element.decompose()
-                                return False
-                            
-                        if not kwargs.get('exclude_external_images', False) and kwargs.get('exclude_social_media_links', False):
-                            src_url_base = src.split('/')[2]
-                            url_base = url.split('/')[2]
-                            if any(domain in src for domain in exclude_social_media_domains):
-                                element.decompose()
-                                return False
-                            
-                        # Handle exclude domains
-                        if kwargs.get('exclude_domains', []):
-                            if any(domain in src for domain in kwargs.get('exclude_domains', [])):
-                                element.decompose()
-                                return False
-                        
-                        return True  # Always keep image elements
-                except Exception as e:
-                    raise "Error processing images"
-                
-                
-                # Check if flag to remove all forms is set
-                if kwargs.get('remove_forms', False) and element.name == 'form':
-                    element.decompose()
-                    return False
-                
-                if element.name in ['video', 'audio']:
+                elif element.name == 'img':
+                    return True  # Always keep image elements
+
+                elif element.name in ['video', 'audio']:
                     media[f"{element.name}s"].append({
                         'src': element.get('src'),
                         'alt': element.get('alt'),
@@ -322,15 +210,14 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                     })
                     return True  # Always keep video and audio elements
 
-                if element.name in ONLY_TEXT_ELIGIBLE_TAGS:
-                    if kwargs.get('only_text', False):
-                        element.replace_with(element.get_text())
-
-                try:
-                    remove_unwanted_attributes(element, IMPORTANT_ATTRS, kwargs.get('keep_data_attributes', False))
-                except Exception as e:
-                    print('Error removing unwanted attributes:', str(e))
-                
+                if element.name != 'pre':
+                    if element.name in ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']:
+                        if kwargs.get('only_text', False):
+                            element.replace_with(element.get_text())
+                        else:
+                            element.unwrap()
+                    elif element.name != 'img':
+                        element.attrs = {}
 
                 # Process children
                 for child in list(element.children):
@@ -364,15 +251,9 @@ class WebScrappingStrategy(ContentScrappingStrategy):
         # ]
         
         process_element(body)
-        
-        # Update the links dictionary with unique links
-        links['internal'] = list(internal_links_dict.values())
-        links['external'] = list(external_links_dict.values())
-
 
         # # Process images using ThreadPoolExecutor
         imgs = body.find_all('img')
-        
         with ThreadPoolExecutor() as executor:
             image_results = list(executor.map(process_image, imgs, [url]*len(imgs), range(len(imgs)), [len(imgs)]*len(imgs)))
         media['images'] = [result for result in image_results if result is not None]
@@ -392,42 +273,12 @@ class WebScrappingStrategy(ContentScrappingStrategy):
             if base64_pattern.match(src):
                 # Replace base64 data with empty string
                 img['src'] = base64_pattern.sub('', src)
-                
-        try:
-            str(body)
-        except Exception as e:
-            # Reset body to the original HTML
-            success = False
-            body = BeautifulSoup(html, 'html.parser')
-            
-            # Create a new div with a special ID
-            error_div = body.new_tag('div', id='crawl4ai_error_message')
-            error_div.string = '''
-            Crawl4AI Error: This page is not fully supported.
-            
-            Possible reasons:
-            1. The page may have restrictions that prevent crawling.
-            2. The page might not be fully loaded.
-            
-            Suggestions:
-            - Try calling the crawl function with these parameters:
-            magic=True,
-            - Set headless=False to visualize what's happening on the page.
-            
-            If the issue persists, please check the page's structure and any potential anti-crawling measures.
-            '''
-            
-            # Append the error div to the body
-            body.body.append(error_div)
-            
-            print(f"[LOG] 😧 Error: After processing the crawled HTML and removing irrelevant tags, nothing was left in the page. Check the markdown for further details.")
-
-
         cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
 
+        h = CustomHTML2Text()
+        h.ignore_links = True
+        h.body_width = 0
         try:
-            h = CustomHTML2Text()
-            h.update_params(**kwargs.get('html2text', {}))            
             markdown = h.handle(cleaned_html)
         except Exception as e:
             markdown = h.handle(sanitize_html(cleaned_html))
@@ -438,18 +289,12 @@ class WebScrappingStrategy(ContentScrappingStrategy):
         except Exception as e:
             print('Error extracting metadata:', str(e))
             meta = {}
-            
-        cleaner = ContentCleaningStrategy()
-        fit_html = cleaner.clean(cleaned_html)
-        fit_markdown = h.handle(fit_html)
 
         cleaned_html = sanitize_html(cleaned_html)
         return {
             'markdown': markdown,
-            'fit_markdown': fit_markdown,
-            'fit_html': fit_html,
             'cleaned_html': cleaned_html,
-            'success': success,
+            'success': True,
             'media': media,
             'links': links,
             'metadata': meta

crawl4ai/extraction_strategy.py

@@ -68,7 +68,7 @@ class LLMExtractionStrategy(ExtractionStrategy):
         """
         super().__init__() 
         self.provider = provider
-        self.api_token = api_token or PROVIDER_MODELS.get(provider, "no-token") or os.getenv("OPENAI_API_KEY")
+        self.api_token = api_token or PROVIDER_MODELS.get(provider, None) or os.getenv("OPENAI_API_KEY")
         self.instruction = instruction
         self.extract_type = extraction_type
         self.schema = schema
@@ -80,7 +80,6 @@ class LLMExtractionStrategy(ExtractionStrategy):
         self.word_token_rate = kwargs.get("word_token_rate", WORD_TOKEN_RATE)
         self.apply_chunking = kwargs.get("apply_chunking", True)
         self.base_url = kwargs.get("base_url", None)
-        self.api_base = kwargs.get("api_base", kwargs.get("base_url", None))
         self.extra_args = kwargs.get("extra_args", {})
         if not self.apply_chunking:
             self.chunk_token_threshold = 1e9
@@ -117,7 +116,7 @@ class LLMExtractionStrategy(ExtractionStrategy):
             self.provider, 
             prompt_with_variables, 
             self.api_token, 
-            base_url=self.api_base or self.base_url,
+            base_url=self.base_url,
             extra_args = self.extra_args
             ) # , json_response=self.extract_type == "schema")
         try:
@@ -235,12 +234,11 @@ class CosineStrategy(ExtractionStrategy):
         """
         Initialize the strategy with clustering parameters.
 
-        Args:
-            semantic_filter (str): A keyword filter for document filtering.
-            word_count_threshold (int): Minimum number of words per cluster.
-            max_dist (float): The maximum cophenetic distance on the dendrogram to form clusters.
-            linkage_method (str): The linkage method for hierarchical clustering.
-            top_k (int): Number of top categories to extract.
+        :param semantic_filter: A keyword filter for document filtering.
+        :param word_count_threshold: Minimum number of words per cluster.
+        :param max_dist: The maximum cophenetic distance on the dendrogram to form clusters.
+        :param linkage_method: The linkage method for hierarchical clustering.
+        :param top_k: Number of top categories to extract.
         """
         super().__init__()
         
@@ -259,8 +257,8 @@ class CosineStrategy(ExtractionStrategy):
         self.get_embedding_method = "direct"
         
         self.device = get_device()
-        # import torch
-        # self.device = torch.device('cpu')
+        import torch
+        self.device = torch.device('cpu')
         
         self.default_batch_size = calculate_batch_size(self.device)
 
@@ -273,7 +271,7 @@ class CosineStrategy(ExtractionStrategy):
         #     self.get_embedding_method = "direct"
         # else:
 
-        self.tokenizer, self.model = load_HF_embedding_model(model_name)
+        self.tokenizer, self.model = load_bge_small_en_v1_5()
         self.model.to(self.device)
         self.model.eval()  
         
@@ -740,6 +738,7 @@ class JsonCssExtractionStrategy(ExtractionStrategy):
         combined_html = self.DEL.join(sections)
         return self.extract(url, combined_html, **kwargs)
     
+
 class JsonXPATHExtractionStrategy(ExtractionStrategy):
     def __init__(self, schema: Dict[str, Any], **kwargs):
         super().__init__(**kwargs)

crawl4ai/html2text/__main__.py

@@ -1,3 +0,0 @@
-from .cli import main
-
-main()

crawl4ai/html2text/_typing.py

@@ -1,2 +0,0 @@
-class OutCallback:
-    def __call__(self, s: str) -> None: ...

crawl4ai/html2text/cli.py

@@ -1,330 +0,0 @@
-import argparse
-import sys
-
-from . import HTML2Text, __version__, config
-
-
-def main() -> None:
-    baseurl = ""
-
-    class bcolors:
-        HEADER = "\033[95m"
-        OKBLUE = "\033[94m"
-        OKGREEN = "\033[92m"
-        WARNING = "\033[93m"
-        FAIL = "\033[91m"
-        ENDC = "\033[0m"
-        BOLD = "\033[1m"
-        UNDERLINE = "\033[4m"
-
-    p = argparse.ArgumentParser()
-    p.add_argument(
-        "--default-image-alt",
-        dest="default_image_alt",
-        default=config.DEFAULT_IMAGE_ALT,
-        help="The default alt string for images with missing ones",
-    )
-    p.add_argument(
-        "--pad-tables",
-        dest="pad_tables",
-        action="store_true",
-        default=config.PAD_TABLES,
-        help="pad the cells to equal column width in tables",
-    )
-    p.add_argument(
-        "--no-wrap-links",
-        dest="wrap_links",
-        action="store_false",
-        default=config.WRAP_LINKS,
-        help="don't wrap links during conversion",
-    )
-    p.add_argument(
-        "--wrap-list-items",
-        dest="wrap_list_items",
-        action="store_true",
-        default=config.WRAP_LIST_ITEMS,
-        help="wrap list items during conversion",
-    )
-    p.add_argument(
-        "--wrap-tables",
-        dest="wrap_tables",
-        action="store_true",
-        default=config.WRAP_TABLES,
-        help="wrap tables",
-    )
-    p.add_argument(
-        "--ignore-emphasis",
-        dest="ignore_emphasis",
-        action="store_true",
-        default=config.IGNORE_EMPHASIS,
-        help="don't include any formatting for emphasis",
-    )
-    p.add_argument(
-        "--reference-links",
-        dest="inline_links",
-        action="store_false",
-        default=config.INLINE_LINKS,
-        help="use reference style links instead of inline links",
-    )
-    p.add_argument(
-        "--ignore-links",
-        dest="ignore_links",
-        action="store_true",
-        default=config.IGNORE_ANCHORS,
-        help="don't include any formatting for links",
-    )
-    p.add_argument(
-        "--ignore-mailto-links",
-        action="store_true",
-        dest="ignore_mailto_links",
-        default=config.IGNORE_MAILTO_LINKS,
-        help="don't include mailto: links",
-    )
-    p.add_argument(
-        "--protect-links",
-        dest="protect_links",
-        action="store_true",
-        default=config.PROTECT_LINKS,
-        help="protect links from line breaks surrounding them with angle brackets",
-    )
-    p.add_argument(
-        "--ignore-images",
-        dest="ignore_images",
-        action="store_true",
-        default=config.IGNORE_IMAGES,
-        help="don't include any formatting for images",
-    )
-    p.add_argument(
-        "--images-as-html",
-        dest="images_as_html",
-        action="store_true",
-        default=config.IMAGES_AS_HTML,
-        help=(
-            "Always write image tags as raw html; preserves `height`, `width` and "
-            "`alt` if possible."
-        ),
-    )
-    p.add_argument(
-        "--images-to-alt",
-        dest="images_to_alt",
-        action="store_true",
-        default=config.IMAGES_TO_ALT,
-        help="Discard image data, only keep alt text",
-    )
-    p.add_argument(
-        "--images-with-size",
-        dest="images_with_size",
-        action="store_true",
-        default=config.IMAGES_WITH_SIZE,
-        help=(
-            "Write image tags with height and width attrs as raw html to retain "
-            "dimensions"
-        ),
-    )
-    p.add_argument(
-        "-g",
-        "--google-doc",
-        action="store_true",
-        dest="google_doc",
-        default=False,
-        help="convert an html-exported Google Document",
-    )
-    p.add_argument(
-        "-d",
-        "--dash-unordered-list",
-        action="store_true",
-        dest="ul_style_dash",
-        default=False,
-        help="use a dash rather than a star for unordered list items",
-    )
-    p.add_argument(
-        "-e",
-        "--asterisk-emphasis",
-        action="store_true",
-        dest="em_style_asterisk",
-        default=False,
-        help="use an asterisk rather than an underscore for emphasized text",
-    )
-    p.add_argument(
-        "-b",
-        "--body-width",
-        dest="body_width",
-        type=int,
-        default=config.BODY_WIDTH,
-        help="number of characters per output line, 0 for no wrap",
-    )
-    p.add_argument(
-        "-i",
-        "--google-list-indent",
-        dest="list_indent",
-        type=int,
-        default=config.GOOGLE_LIST_INDENT,
-        help="number of pixels Google indents nested lists",
-    )
-    p.add_argument(
-        "-s",
-        "--hide-strikethrough",
-        action="store_true",
-        dest="hide_strikethrough",
-        default=False,
-        help="hide strike-through text. only relevant when -g is " "specified as well",
-    )
-    p.add_argument(
-        "--escape-all",
-        action="store_true",
-        dest="escape_snob",
-        default=False,
-        help=(
-            "Escape all special characters.  Output is less readable, but avoids "
-            "corner case formatting issues."
-        ),
-    )
-    p.add_argument(
-        "--bypass-tables",
-        action="store_true",
-        dest="bypass_tables",
-        default=config.BYPASS_TABLES,
-        help="Format tables in HTML rather than Markdown syntax.",
-    )
-    p.add_argument(
-        "--ignore-tables",
-        action="store_true",
-        dest="ignore_tables",
-        default=config.IGNORE_TABLES,
-        help="Ignore table-related tags (table, th, td, tr) " "while keeping rows.",
-    )
-    p.add_argument(
-        "--single-line-break",
-        action="store_true",
-        dest="single_line_break",
-        default=config.SINGLE_LINE_BREAK,
-        help=(
-            "Use a single line break after a block element rather than two line "
-            "breaks. NOTE: Requires --body-width=0"
-        ),
-    )
-    p.add_argument(
-        "--unicode-snob",
-        action="store_true",
-        dest="unicode_snob",
-        default=config.UNICODE_SNOB,
-        help="Use unicode throughout document",
-    )
-    p.add_argument(
-        "--no-automatic-links",
-        action="store_false",
-        dest="use_automatic_links",
-        default=config.USE_AUTOMATIC_LINKS,
-        help="Do not use automatic links wherever applicable",
-    )
-    p.add_argument(
-        "--no-skip-internal-links",
-        action="store_false",
-        dest="skip_internal_links",
-        default=config.SKIP_INTERNAL_LINKS,
-        help="Do not skip internal links",
-    )
-    p.add_argument(
-        "--links-after-para",
-        action="store_true",
-        dest="links_each_paragraph",
-        default=config.LINKS_EACH_PARAGRAPH,
-        help="Put links after each paragraph instead of document",
-    )
-    p.add_argument(
-        "--mark-code",
-        action="store_true",
-        dest="mark_code",
-        default=config.MARK_CODE,
-        help="Mark program code blocks with [code]...[/code]",
-    )
-    p.add_argument(
-        "--decode-errors",
-        dest="decode_errors",
-        default=config.DECODE_ERRORS,
-        help=(
-            "What to do in case of decode errors.'ignore', 'strict' and 'replace' are "
-            "acceptable values"
-        ),
-    )
-    p.add_argument(
-        "--open-quote",
-        dest="open_quote",
-        default=config.OPEN_QUOTE,
-        help="The character used to open quotes",
-    )
-    p.add_argument(
-        "--close-quote",
-        dest="close_quote",
-        default=config.CLOSE_QUOTE,
-        help="The character used to close quotes",
-    )
-    p.add_argument(
-        "--version", action="version", version=".".join(map(str, __version__))
-    )
-    p.add_argument("filename", nargs="?")
-    p.add_argument("encoding", nargs="?", default="utf-8")
-    p.add_argument(
-        "--include-sup-sub",
-        dest="include_sup_sub",
-        action="store_true",
-        default=config.INCLUDE_SUP_SUB,
-        help="Include the sup and sub tags",
-    )
-    args = p.parse_args()
-
-    if args.filename and args.filename != "-":
-        with open(args.filename, "rb") as fp:
-            data = fp.read()
-    else:
-        data = sys.stdin.buffer.read()
-
-    try:
-        html = data.decode(args.encoding, args.decode_errors)
-    except UnicodeDecodeError as err:
-        warning = bcolors.WARNING + "Warning:" + bcolors.ENDC
-        warning += " Use the " + bcolors.OKGREEN
-        warning += "--decode-errors=ignore" + bcolors.ENDC + " flag."
-        print(warning)
-        raise err
-
-    h = HTML2Text(baseurl=baseurl)
-    # handle options
-    if args.ul_style_dash:
-        h.ul_item_mark = "-"
-    if args.em_style_asterisk:
-        h.emphasis_mark = "*"
-        h.strong_mark = "__"
-
-    h.body_width = args.body_width
-    h.google_list_indent = args.list_indent
-    h.ignore_emphasis = args.ignore_emphasis
-    h.ignore_links = args.ignore_links
-    h.ignore_mailto_links = args.ignore_mailto_links
-    h.protect_links = args.protect_links
-    h.ignore_images = args.ignore_images
-    h.images_as_html = args.images_as_html
-    h.images_to_alt = args.images_to_alt
-    h.images_with_size = args.images_with_size
-    h.google_doc = args.google_doc
-    h.hide_strikethrough = args.hide_strikethrough
-    h.escape_snob = args.escape_snob
-    h.bypass_tables = args.bypass_tables
-    h.ignore_tables = args.ignore_tables
-    h.single_line_break = args.single_line_break
-    h.inline_links = args.inline_links
-    h.unicode_snob = args.unicode_snob
-    h.use_automatic_links = args.use_automatic_links
-    h.skip_internal_links = args.skip_internal_links
-    h.links_each_paragraph = args.links_each_paragraph
-    h.mark_code = args.mark_code
-    h.wrap_links = args.wrap_links
-    h.wrap_list_items = args.wrap_list_items
-    h.wrap_tables = args.wrap_tables
-    h.pad_tables = args.pad_tables
-    h.default_image_alt = args.default_image_alt
-    h.open_quote = args.open_quote
-    h.close_quote = args.close_quote
-    h.include_sup_sub = args.include_sup_sub
-
-    sys.stdout.write(h.handle(html))

crawl4ai/html2text/config.py

@@ -1,172 +0,0 @@
-import re
-
-# Use Unicode characters instead of their ascii pseudo-replacements
-UNICODE_SNOB = False
-
-# Marker to use for marking tables for padding post processing
-TABLE_MARKER_FOR_PAD = "special_marker_for_table_padding"
-# Escape all special characters.  Output is less readable, but avoids
-# corner case formatting issues.
-ESCAPE_SNOB = False
-ESCAPE_BACKSLASH = False
-ESCAPE_DOT = False
-ESCAPE_PLUS = False
-ESCAPE_DASH = False
-
-# Put the links after each paragraph instead of at the end.
-LINKS_EACH_PARAGRAPH = False
-
-# Wrap long lines at position. 0 for no wrapping.
-BODY_WIDTH = 78
-
-# Don't show internal links (href="#local-anchor") -- corresponding link
-# targets won't be visible in the plain text file anyway.
-SKIP_INTERNAL_LINKS = True
-
-# Use inline, rather than reference, formatting for images and links
-INLINE_LINKS = True
-
-# Protect links from line breaks surrounding them with angle brackets (in
-# addition to their square brackets)
-PROTECT_LINKS = False
-# WRAP_LINKS = True
-WRAP_LINKS = True
-
-# Wrap list items.
-WRAP_LIST_ITEMS = False
-
-# Wrap tables
-WRAP_TABLES = False
-
-# Number of pixels Google indents nested lists
-GOOGLE_LIST_INDENT = 36
-
-# Values Google and others may use to indicate bold text
-BOLD_TEXT_STYLE_VALUES = ("bold", "700", "800", "900")
-
-IGNORE_ANCHORS = False
-IGNORE_MAILTO_LINKS = False
-IGNORE_IMAGES = False
-IMAGES_AS_HTML = False
-IMAGES_TO_ALT = False
-IMAGES_WITH_SIZE = False
-IGNORE_EMPHASIS = False
-MARK_CODE = False
-DECODE_ERRORS = "strict"
-DEFAULT_IMAGE_ALT = ""
-PAD_TABLES = False
-
-# Convert links with same href and text to <href> format
-# if they are absolute links
-USE_AUTOMATIC_LINKS = True
-
-# For checking space-only lines on line 771
-RE_SPACE = re.compile(r"\s\+")
-
-RE_ORDERED_LIST_MATCHER = re.compile(r"\d+\.\s")
-RE_UNORDERED_LIST_MATCHER = re.compile(r"[-\*\+]\s")
-RE_MD_CHARS_MATCHER = re.compile(r"([\\\[\]\(\)])")
-RE_MD_CHARS_MATCHER_ALL = re.compile(r"([`\*_{}\[\]\(\)#!])")
-
-# to find links in the text
-RE_LINK = re.compile(r"(\[.*?\] ?\(.*?\))|(\[.*?\]:.*?)")
-
-# to find table separators
-RE_TABLE = re.compile(r" \| ")
-
-RE_MD_DOT_MATCHER = re.compile(
-    r"""
-    ^             # start of line
-    (\s*\d+)      # optional whitespace and a number
-    (\.)          # dot
-    (?=\s)        # lookahead assert whitespace
-    """,
-    re.MULTILINE | re.VERBOSE,
-)
-RE_MD_PLUS_MATCHER = re.compile(
-    r"""
-    ^
-    (\s*)
-    (\+)
-    (?=\s)
-    """,
-    flags=re.MULTILINE | re.VERBOSE,
-)
-RE_MD_DASH_MATCHER = re.compile(
-    r"""
-    ^
-    (\s*)
-    (-)
-    (?=\s|\-)     # followed by whitespace (bullet list, or spaced out hr)
-                  # or another dash (header or hr)
-    """,
-    flags=re.MULTILINE | re.VERBOSE,
-)
-RE_SLASH_CHARS = r"\`*_{}[]()#+-.!"
-RE_MD_BACKSLASH_MATCHER = re.compile(
-    r"""
-    (\\)          # match one slash
-    (?=[%s])      # followed by a char that requires escaping
-    """
-    % re.escape(RE_SLASH_CHARS),
-    flags=re.VERBOSE,
-)
-
-UNIFIABLE = {
-    "rsquo": "'",
-    "lsquo": "'",
-    "rdquo": '"',
-    "ldquo": '"',
-    "copy": "(C)",
-    "mdash": "--",
-    "nbsp": " ",
-    "rarr": "->",
-    "larr": "<-",
-    "middot": "*",
-    "ndash": "-",
-    "oelig": "oe",
-    "aelig": "ae",
-    "agrave": "a",
-    "aacute": "a",
-    "acirc": "a",
-    "atilde": "a",
-    "auml": "a",
-    "aring": "a",
-    "egrave": "e",
-    "eacute": "e",
-    "ecirc": "e",
-    "euml": "e",
-    "igrave": "i",
-    "iacute": "i",
-    "icirc": "i",
-    "iuml": "i",
-    "ograve": "o",
-    "oacute": "o",
-    "ocirc": "o",
-    "otilde": "o",
-    "ouml": "o",
-    "ugrave": "u",
-    "uacute": "u",
-    "ucirc": "u",
-    "uuml": "u",
-    "lrm": "",
-    "rlm": "",
-}
-
-# Format tables in HTML rather than Markdown syntax
-BYPASS_TABLES = False
-# Ignore table-related tags (table, th, td, tr) while keeping rows
-IGNORE_TABLES = False
-
-
-# Use a single line break after a block element rather than two line breaks.
-# NOTE: Requires body width setting to be 0.
-SINGLE_LINE_BREAK = False
-
-
-# Use double quotation marks when converting the <q> tag.
-OPEN_QUOTE = '"'
-CLOSE_QUOTE = '"'
-
-# Include the <sup> and <sub> tags
-INCLUDE_SUP_SUB = False

crawl4ai/html2text/elements.py

@@ -1,18 +0,0 @@
-from typing import Dict, Optional
-
-
-class AnchorElement:
-    __slots__ = ["attrs", "count", "outcount"]
-
-    def __init__(self, attrs: Dict[str, Optional[str]], count: int, outcount: int):
-        self.attrs = attrs
-        self.count = count
-        self.outcount = outcount
-
-
-class ListElement:
-    __slots__ = ["name", "num"]
-
-    def __init__(self, name: str, num: int):
-        self.name = name
-        self.num = num

crawl4ai/html2text/utils.py

@@ -1,303 +0,0 @@
-import html.entities
-from typing import Dict, List, Optional
-
-from . import config
-
-unifiable_n = {
-    html.entities.name2codepoint[k]: v
-    for k, v in config.UNIFIABLE.items()
-    if k != "nbsp"
-}
-
-
-def hn(tag: str) -> int:
-    if tag[0] == "h" and len(tag) == 2:
-        n = tag[1]
-        if "0" < n <= "9":
-            return int(n)
-    return 0
-
-
-def dumb_property_dict(style: str) -> Dict[str, str]:
-    """
-    :returns: A hash of css attributes
-    """
-    return {
-        x.strip().lower(): y.strip().lower()
-        for x, y in [z.split(":", 1) for z in style.split(";") if ":" in z]
-    }
-
-
-def dumb_css_parser(data: str) -> Dict[str, Dict[str, str]]:
-    """
-    :type data: str
-
-    :returns: A hash of css selectors, each of which contains a hash of
-    css attributes.
-    :rtype: dict
-    """
-    # remove @import sentences
-    data += ";"
-    importIndex = data.find("@import")
-    while importIndex != -1:
-        data = data[0:importIndex] + data[data.find(";", importIndex) + 1 :]
-        importIndex = data.find("@import")
-
-    # parse the css. reverted from dictionary comprehension in order to
-    # support older pythons
-    pairs = [x.split("{") for x in data.split("}") if "{" in x.strip()]
-    try:
-        elements = {a.strip(): dumb_property_dict(b) for a, b in pairs}
-    except ValueError:
-        elements = {}  # not that important
-
-    return elements
-
-
-def element_style(
-    attrs: Dict[str, Optional[str]],
-    style_def: Dict[str, Dict[str, str]],
-    parent_style: Dict[str, str],
-) -> Dict[str, str]:
-    """
-    :type attrs: dict
-    :type style_def: dict
-    :type style_def: dict
-
-    :returns: A hash of the 'final' style attributes of the element
-    :rtype: dict
-    """
-    style = parent_style.copy()
-    if "class" in attrs:
-        assert attrs["class"] is not None
-        for css_class in attrs["class"].split():
-            css_style = style_def.get("." + css_class, {})
-            style.update(css_style)
-    if "style" in attrs:
-        assert attrs["style"] is not None
-        immediate_style = dumb_property_dict(attrs["style"])
-        style.update(immediate_style)
-
-    return style
-
-
-def google_list_style(style: Dict[str, str]) -> str:
-    """
-    Finds out whether this is an ordered or unordered list
-
-    :type style: dict
-
-    :rtype: str
-    """
-    if "list-style-type" in style:
-        list_style = style["list-style-type"]
-        if list_style in ["disc", "circle", "square", "none"]:
-            return "ul"
-
-    return "ol"
-
-
-def google_has_height(style: Dict[str, str]) -> bool:
-    """
-    Check if the style of the element has the 'height' attribute
-    explicitly defined
-
-    :type style: dict
-
-    :rtype: bool
-    """
-    return "height" in style
-
-
-def google_text_emphasis(style: Dict[str, str]) -> List[str]:
-    """
-    :type style: dict
-
-    :returns: A list of all emphasis modifiers of the element
-    :rtype: list
-    """
-    emphasis = []
-    if "text-decoration" in style:
-        emphasis.append(style["text-decoration"])
-    if "font-style" in style:
-        emphasis.append(style["font-style"])
-    if "font-weight" in style:
-        emphasis.append(style["font-weight"])
-
-    return emphasis
-
-
-def google_fixed_width_font(style: Dict[str, str]) -> bool:
-    """
-    Check if the css of the current element defines a fixed width font
-
-    :type style: dict
-
-    :rtype: bool
-    """
-    font_family = ""
-    if "font-family" in style:
-        font_family = style["font-family"]
-    return "courier new" == font_family or "consolas" == font_family
-
-
-def list_numbering_start(attrs: Dict[str, Optional[str]]) -> int:
-    """
-    Extract numbering from list element attributes
-
-    :type attrs: dict
-
-    :rtype: int or None
-    """
-    if "start" in attrs:
-        assert attrs["start"] is not None
-        try:
-            return int(attrs["start"]) - 1
-        except ValueError:
-            pass
-
-    return 0
-
-
-def skipwrap(
-    para: str, wrap_links: bool, wrap_list_items: bool, wrap_tables: bool
-) -> bool:
-    # If it appears to contain a link
-    # don't wrap
-    if not wrap_links and config.RE_LINK.search(para):
-        return True
-    # If the text begins with four spaces or one tab, it's a code block;
-    # don't wrap
-    if para[0:4] == "    " or para[0] == "\t":
-        return True
-
-    # If the text begins with only two "--", possibly preceded by
-    # whitespace, that's an emdash; so wrap.
-    stripped = para.lstrip()
-    if stripped[0:2] == "--" and len(stripped) > 2 and stripped[2] != "-":
-        return False
-
-    # I'm not sure what this is for; I thought it was to detect lists,
-    # but there's a <br>-inside-<span> case in one of the tests that
-    # also depends upon it.
-    if stripped[0:1] in ("-", "*") and not stripped[0:2] == "**":
-        return not wrap_list_items
-
-    # If text contains a pipe character it is likely a table
-    if not wrap_tables and config.RE_TABLE.search(para):
-        return True
-
-    # If the text begins with a single -, *, or +, followed by a space,
-    # or an integer, followed by a ., followed by a space (in either
-    # case optionally proceeded by whitespace), it's a list; don't wrap.
-    return bool(
-        config.RE_ORDERED_LIST_MATCHER.match(stripped)
-        or config.RE_UNORDERED_LIST_MATCHER.match(stripped)
-    )
-
-
-def escape_md(text: str) -> str:
-    """
-    Escapes markdown-sensitive characters within other markdown
-    constructs.
-    """
-    return config.RE_MD_CHARS_MATCHER.sub(r"\\\1", text)
-
-
-def escape_md_section(
-    text: str,
-    escape_backslash: bool = True,
-    snob: bool = False,
-    escape_dot: bool = True,
-    escape_plus: bool = True,
-    escape_dash: bool = True
-) -> str:
-    """
-    Escapes markdown-sensitive characters across whole document sections.
-    Each escaping operation can be controlled individually.
-    """
-    if escape_backslash:
-        text = config.RE_MD_BACKSLASH_MATCHER.sub(r"\\\1", text)
-
-    if snob:
-        text = config.RE_MD_CHARS_MATCHER_ALL.sub(r"\\\1", text)
-
-    if escape_dot:
-        text = config.RE_MD_DOT_MATCHER.sub(r"\1\\\2", text)
-
-    if escape_plus:
-        text = config.RE_MD_PLUS_MATCHER.sub(r"\1\\\2", text)
-
-    if escape_dash:
-        text = config.RE_MD_DASH_MATCHER.sub(r"\1\\\2", text)
-
-    return text
-
-def reformat_table(lines: List[str], right_margin: int) -> List[str]:
-    """
-    Given the lines of a table
-    padds the cells and returns the new lines
-    """
-    # find the maximum width of the columns
-    max_width = [len(x.rstrip()) + right_margin for x in lines[0].split("|")]
-    max_cols = len(max_width)
-    for line in lines:
-        cols = [x.rstrip() for x in line.split("|")]
-        num_cols = len(cols)
-
-        # don't drop any data if colspan attributes result in unequal lengths
-        if num_cols < max_cols:
-            cols += [""] * (max_cols - num_cols)
-        elif max_cols < num_cols:
-            max_width += [len(x) + right_margin for x in cols[-(num_cols - max_cols) :]]
-            max_cols = num_cols
-
-        max_width = [
-            max(len(x) + right_margin, old_len) for x, old_len in zip(cols, max_width)
-        ]
-
-    # reformat
-    new_lines = []
-    for line in lines:
-        cols = [x.rstrip() for x in line.split("|")]
-        if set(line.strip()) == set("-|"):
-            filler = "-"
-            new_cols = [
-                x.rstrip() + (filler * (M - len(x.rstrip())))
-                for x, M in zip(cols, max_width)
-            ]
-            new_lines.append("|-" + "|".join(new_cols) + "|")
-        else:
-            filler = " "
-            new_cols = [
-                x.rstrip() + (filler * (M - len(x.rstrip())))
-                for x, M in zip(cols, max_width)
-            ]
-            new_lines.append("| " + "|".join(new_cols) + "|")
-    return new_lines
-
-
-def pad_tables_in_text(text: str, right_margin: int = 1) -> str:
-    """
-    Provide padding for tables in the text
-    """
-    lines = text.split("\n")
-    table_buffer = []  # type: List[str]
-    table_started = False
-    new_lines = []
-    for line in lines:
-        # Toggle table started
-        if config.TABLE_MARKER_FOR_PAD in line:
-            table_started = not table_started
-            if not table_started:
-                table = reformat_table(table_buffer, right_margin)
-                new_lines.extend(table)
-                table_buffer = []
-                new_lines.append("")
-            continue
-        # Process lines
-        if table_started:
-            table_buffer.append(line)
-        else:
-            new_lines.append(line)
-    return "\n".join(new_lines)

crawl4ai/model_loader.py

@@ -72,18 +72,10 @@ def load_bert_base_uncased():
     return tokenizer, model
 
 @lru_cache()
-def load_HF_embedding_model(model_name="BAAI/bge-small-en-v1.5") -> tuple:
-    """Load the Hugging Face model for embedding.
-    
-    Args:
-        model_name (str, optional): The model name to load. Defaults to "BAAI/bge-small-en-v1.5".
-        
-    Returns:
-        tuple: The tokenizer and model.
-    """
+def load_bge_small_en_v1_5():
     from transformers import BertTokenizer, BertModel, AutoTokenizer, AutoModel
-    tokenizer = AutoTokenizer.from_pretrained(model_name, resume_download=None)
-    model = AutoModel.from_pretrained(model_name, resume_download=None)
+    tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-small-en-v1.5', resume_download=None)
+    model = AutoModel.from_pretrained('BAAI/bge-small-en-v1.5', resume_download=None)
     model.eval()
     model, device = set_model_device(model)
     return tokenizer, model

crawl4ai/models.py

@@ -14,8 +14,6 @@ class CrawlResult(BaseModel):
     links: Dict[str, List[Dict]] = {}
     screenshot: Optional[str] = None
     markdown: Optional[str] = None
-    fit_markdown: Optional[str] = None
-    fit_html: Optional[str] = None
     extracted_content: Optional[str] = None
     metadata: Optional[dict] = None
     error_message: Optional[str] = None

crawl4ai/scraper/__init__.py

@@ -0,0 +1,2 @@
+from .async_web_scraper import AsyncWebScraper
+from .bfs_scraper_strategy import BFSScraperStrategy

crawl4ai/scraper/async_web_scraper.py

@@ -0,0 +1,33 @@
+from .scraper_strategy import ScraperStrategy
+from .models import ScraperResult, CrawlResult
+from ..async_webcrawler import AsyncWebCrawler
+from typing import Union, AsyncGenerator
+
+class AsyncWebScraper:
+    def __init__(self, crawler: AsyncWebCrawler, strategy: ScraperStrategy):
+        self.crawler = crawler
+        self.strategy = strategy
+
+    async def ascrape(self, url: str, parallel_processing: bool = True, stream: bool = False) -> Union[AsyncGenerator[CrawlResult, None], ScraperResult]:
+        if stream:
+            return self._ascrape_yielding(url, parallel_processing)
+        else:
+            return await self._ascrape_collecting(url, parallel_processing)
+
+    async def _ascrape_yielding(self, url: str, parallel_processing: bool) -> AsyncGenerator[CrawlResult, None]:
+        result_generator = self.strategy.ascrape(url, self.crawler, parallel_processing)
+        async for res in result_generator:  # Consume the async generator
+            yield res  # Yielding individual results
+
+    async def _ascrape_collecting(self, url: str, parallel_processing: bool) -> ScraperResult:
+        extracted_data = {}
+        result_generator = self.strategy.ascrape(url, self.crawler, parallel_processing)
+        async for res in result_generator:  # Consume the async generator
+            extracted_data[res.url] = res
+
+        # Return a final ScraperResult
+        return ScraperResult(
+            url=url,
+            crawled_urls=list(extracted_data.keys()),
+            extracted_data=extracted_data
+        )

crawl4ai/scraper/bfs_scraper_strategy.py

@@ -0,0 +1,139 @@
+from .scraper_strategy import ScraperStrategy
+from .filters import FilterChain
+from .scorers import URLScorer
+from ..models import CrawlResult
+from ..async_webcrawler import AsyncWebCrawler
+import asyncio
+import validators
+from urllib.parse import urljoin,urlparse,urlunparse
+from urllib.robotparser import RobotFileParser
+import time
+from aiolimiter import AsyncLimiter
+from tenacity import retry, stop_after_attempt, wait_exponential
+from collections import defaultdict
+import logging
+from typing import Dict, AsyncGenerator
+logging.basicConfig(level=logging.DEBUG)
+
+rate_limiter = AsyncLimiter(1, 1)  # 1 request per second
+
+class BFSScraperStrategy(ScraperStrategy):
+    def __init__(self, max_depth: int, filter_chain: FilterChain, url_scorer: URLScorer, max_concurrent: int = 5, min_crawl_delay: int=1):
+        self.max_depth = max_depth
+        self.filter_chain = filter_chain
+        self.url_scorer = url_scorer
+        self.max_concurrent = max_concurrent
+        # For Crawl Politeness
+        self.last_crawl_time = defaultdict(float)
+        self.min_crawl_delay = min_crawl_delay  # 1 second delay between requests to the same domain
+        # For Robots.txt Compliance
+        self.robot_parsers = {}
+
+    # Robots.txt Parser
+    def get_robot_parser(self, url: str) -> RobotFileParser:
+        domain = urlparse(url)
+        scheme = domain.scheme if domain.scheme else 'http'  # Default to 'http' if no scheme provided
+        netloc = domain.netloc
+        if netloc not in self.robot_parsers:
+            rp = RobotFileParser()
+            rp.set_url(f"{scheme}://{netloc}/robots.txt")
+            try:
+                rp.read()
+            except Exception as e:
+                # Log the type of error, message, and the URL
+                logging.warning(f"Error {type(e).__name__} occurred while fetching robots.txt for {netloc}: {e}")
+                return None
+            self.robot_parsers[netloc] = rp
+        return self.robot_parsers[netloc]
+
+    
+    # Retry with exponential backoff
+    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
+    async def retry_crawl(self, crawler: AsyncWebCrawler, url: str) -> CrawlResult:
+        return await crawler.arun(url)
+    
+    async def process_url(self, url: str, depth: int, crawler: AsyncWebCrawler, queue: asyncio.PriorityQueue, visited: set, depths: Dict[str, int]) -> AsyncGenerator[CrawlResult, None]:
+        def normalize_url(url: str) -> str:
+            parsed = urlparse(url)
+            return urlunparse(parsed._replace(fragment=""))
+        
+        # URL Validation
+        if not validators.url(url):
+            logging.warning(f"Invalid URL: {url}")
+            return None
+        
+        # Robots.txt Compliance
+        robot_parser = self.get_robot_parser(url)
+        if robot_parser is None:
+            logging.info(f"Could not retrieve robots.txt for {url}, hence proceeding with crawl.")
+        else:
+            # If robots.txt was fetched, check if crawling is allowed
+            if not robot_parser.can_fetch(crawler.crawler_strategy.user_agent, url):
+                logging.info(f"Skipping {url} as per robots.txt")
+                return None
+    
+        # Crawl Politeness
+        domain = urlparse(url).netloc
+        time_since_last_crawl = time.time() - self.last_crawl_time[domain]
+        if time_since_last_crawl < self.min_crawl_delay:
+            await asyncio.sleep(self.min_crawl_delay - time_since_last_crawl)
+        self.last_crawl_time[domain] = time.time()
+
+        # Rate Limiting
+        async with rate_limiter:
+            # Error Handling
+            try:
+                crawl_result = await self.retry_crawl(crawler, url)
+            except Exception as e:
+                logging.error(f"Error crawling {url}: {str(e)}")
+                crawl_result = CrawlResult(url=url, html="", success=False, status_code=0, error_message=str(e))
+        
+        if not crawl_result.success:
+            # Logging and Monitoring
+            logging.error(f"Failed to crawl URL: {url}. Error: {crawl_result.error_message}")
+            return crawl_result
+
+        # Process links
+        for link_type in ["internal", "external"]:
+            for link in crawl_result.links[link_type]:
+                absolute_link = urljoin(url, link['href'])
+                normalized_link = normalize_url(absolute_link)
+                if self.filter_chain.apply(normalized_link) and normalized_link not in visited:
+                    new_depth = depths[url] + 1
+                    if new_depth <= self.max_depth:
+                        # URL Scoring
+                        score = self.url_scorer.score(normalized_link)
+                        await queue.put((score, new_depth, normalized_link))
+                        depths[normalized_link] = new_depth
+        return crawl_result
+
+    async def ascrape(self, start_url: str, crawler: AsyncWebCrawler, parallel_processing:bool = True) -> AsyncGenerator[CrawlResult,None]:
+        queue = asyncio.PriorityQueue()
+        queue.put_nowait((0, 0, start_url))
+        visited = set()
+        depths = {start_url: 0}
+        pending_tasks = set()
+
+        while not queue.empty() or pending_tasks:
+            while not queue.empty() and len(pending_tasks) < self.max_concurrent:
+                _, depth, url = await queue.get()
+                if url not in visited:
+                    # Adding URL to the visited set here itself, (instead of after result generation)
+                    # so that other tasks are not queued for same URL, found at different depth before
+                    # crawling and extraction of this task is completed.
+                    visited.add(url)
+                    if parallel_processing:
+                        task = asyncio.create_task(self.process_url(url, depth, crawler, queue, visited, depths))
+                        pending_tasks.add(task)
+                    else:
+                        result = await self.process_url(url, depth, crawler, queue, visited, depths)
+                        if result:
+                            yield result 
+
+            # Wait for the first task to complete and yield results incrementally as each task is completed
+            if pending_tasks:
+                done, pending_tasks = await asyncio.wait(pending_tasks, return_when=asyncio.FIRST_COMPLETED)
+                for task in done:
+                    result = await task
+                    if result:
+                        yield result

crawl4ai/scraper/filters/__init__.py

@@ -0,0 +1,3 @@
+from .url_filter import URLFilter, FilterChain
+from .content_type_filter import ContentTypeFilter
+from .url_pattern_filter import URLPatternFilter

crawl4ai/scraper/filters/content_type_filter.py

@@ -0,0 +1,8 @@
+from .url_filter import URLFilter
+
+class ContentTypeFilter(URLFilter):
+    def __init__(self, contentType: str):
+        self.contentType = contentType
+    def apply(self, url: str) -> bool:
+        #TODO: This is a stub. Will implement this later
+        return True

crawl4ai/scraper/filters/url_filter.py

@@ -0,0 +1,16 @@
+from abc import ABC, abstractmethod
+
+class URLFilter(ABC):
+    @abstractmethod
+    def apply(self, url: str) -> bool:
+        pass
+
+class FilterChain:
+    def __init__(self):
+        self.filters = []
+
+    def add_filter(self, filter: URLFilter):
+        self.filters.append(filter)
+
+    def apply(self, url: str) -> bool:
+        return all(filter.apply(url) for filter in self.filters)

crawl4ai/scraper/filters/url_pattern_filter.py

@@ -0,0 +1,9 @@
+from .url_filter import URLFilter
+from re import Pattern
+
+class URLPatternFilter(URLFilter):
+    def __init__(self, pattern: Pattern):
+        self.pattern = pattern
+    def apply(self, url: str) -> bool:
+        #TODO: This is a stub. Will implement this later.
+        return True

crawl4ai/scraper/models.py

@@ -0,0 +1,8 @@
+from pydantic import BaseModel
+from typing import List, Dict
+from ..models import CrawlResult
+
+class ScraperResult(BaseModel):
+    url: str
+    crawled_urls: List[str]
+    extracted_data: Dict[str,CrawlResult]

crawl4ai/scraper/scorers/__init__.py

@@ -0,0 +1,2 @@
+from .url_scorer import URLScorer
+from .keyword_relevance_scorer import KeywordRelevanceScorer

crawl4ai/scraper/scorers/keyword_relevance_scorer.py

@@ -0,0 +1,9 @@
+from .url_scorer import URLScorer
+from typing import List
+
+class KeywordRelevanceScorer(URLScorer):
+    def __init__(self,keywords: List[str]):
+        self.keyworkds = keywords
+    def score(self, url: str) -> float:
+        #TODO: This is a stub. Will implement this later.
+        return 1

crawl4ai/scraper/scorers/url_scorer.py

@@ -0,0 +1,6 @@
+from abc import ABC, abstractmethod
+
+class URLScorer(ABC):
+    @abstractmethod
+    def score(self, url: str) -> float:
+        pass

crawl4ai/scraper/scraper_strategy.py

@@ -0,0 +1,26 @@
+from abc import ABC, abstractmethod
+from .models import ScraperResult, CrawlResult
+from ..models import CrawlResult
+from ..async_webcrawler import AsyncWebCrawler
+from typing import Union, AsyncGenerator
+
+class ScraperStrategy(ABC):
+    @abstractmethod
+    async def ascrape(self, url: str, crawler: AsyncWebCrawler, parallel_processing: bool = True, stream: bool = False) -> Union[AsyncGenerator[CrawlResult, None], ScraperResult]:
+        """Scrape the given URL using the specified crawler.
+
+        Args:
+            url (str): The starting URL for the scrape.
+            crawler (AsyncWebCrawler): The web crawler instance.
+            parallel_processing (bool): Whether to use parallel processing. Defaults to True.
+            stream (bool): If True, yields individual crawl results as they are ready; 
+                                if False, accumulates results and returns a final ScraperResult.
+
+        Yields:
+            CrawlResult: Individual crawl results if stream is True.
+
+        Returns:
+            ScraperResult: A summary of the scrape results containing the final extracted data 
+            and the list of crawled URLs if stream is False.
+        """
+        pass

crawl4ai/utils.py

@@ -1,12 +1,13 @@
 import time
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from bs4 import BeautifulSoup, Comment, element, Tag, NavigableString
+import html2text
 import json
 import html
 import re
 import os
 import platform
-from .html2text import HTML2Text
+from html2text import HTML2Text
 from .prompts import PROMPT_EXTRACT_BLOCKS
 from .config import *
 from pathlib import Path
@@ -181,22 +182,9 @@ def escape_json_string(s):
 class CustomHTML2Text(HTML2Text):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
+        self.ignore_links = True
         self.inside_pre = False
         self.inside_code = False
-        
-        self.skip_internal_links = False
-        self.single_line_break = False
-        self.mark_code = False
-        self.include_sup_sub = False
-        self.body_width = 0
-        self.ignore_mailto_links = True
-        self.ignore_links = False
-        self.escape_backslash = False
-        self.escape_dot = False
-        self.escape_plus = False
-        self.escape_dash = False
-        self.escape_snob = False
-
 
     def handle_tag(self, tag, attrs, start):
         if tag == 'pre':
@@ -206,10 +194,6 @@ class CustomHTML2Text(HTML2Text):
             else:
                 self.o('\n```')
                 self.inside_pre = False
-        elif tag in ["h1", "h2", "h3", "h4", "h5", "h6"]:
-            pass
-
-
         # elif tag == 'code' and not self.inside_pre:
         #     if start:
         #         if not self.inside_pre:
@@ -708,8 +692,8 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
     for img in imgs:
         src = img.get('src', '')
         if base64_pattern.match(src):
+            # Replace base64 data with empty string
             img['src'] = base64_pattern.sub('', src)
-
     cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
     cleaned_html = sanitize_html(cleaned_html)
 
@@ -980,53 +964,4 @@ def format_html(html_string):
     soup = BeautifulSoup(html_string, 'html.parser')
     return soup.prettify()
 
-def normalize_url(href, base_url):
-    """Normalize URLs to ensure consistent format"""
-    # Extract protocol and domain from base URL
-    try:
-        base_parts = base_url.split('/')
-        protocol = base_parts[0]
-        domain = base_parts[2]
-    except IndexError:
-        raise ValueError(f"Invalid base URL format: {base_url}")
-    
-    # Handle special protocols
-    special_protocols = {'mailto:', 'tel:', 'ftp:', 'file:', 'data:', 'javascript:'}
-    if any(href.lower().startswith(proto) for proto in special_protocols):
-        return href.strip()
-        
-    # Handle anchor links
-    if href.startswith('#'):
-        return f"{base_url}{href}"
-        
-    # Handle protocol-relative URLs
-    if href.startswith('//'):
-        return f"{protocol}{href}"
-        
-    # Handle root-relative URLs
-    if href.startswith('/'):
-        return f"{protocol}//{domain}{href}"
-        
-    # Handle relative URLs
-    if not href.startswith(('http://', 'https://')):
-        # Remove leading './' if present
-        href = href.lstrip('./')
-        return f"{protocol}//{domain}/{href}"
-        
-    return href.strip()
 
-def is_external_url(url, base_domain):
-    """Determine if a URL is external"""
-    special_protocols = {'mailto:', 'tel:', 'ftp:', 'file:', 'data:', 'javascript:'}
-    if any(url.lower().startswith(proto) for proto in special_protocols):
-        return True
-        
-    try:
-        # Handle URLs with protocol
-        if url.startswith(('http://', 'https://')):
-            url_domain = url.split('/')[2]
-            return base_domain.lower() not in url_domain.lower()
-    except IndexError:
-        return False
-        
-    return False

docs/chunking_strategies.json

@@ -0,0 +1,12 @@
+{
+    "RegexChunking": "### RegexChunking\n\n`RegexChunking` is a text chunking strategy that splits a given text into smaller parts using regular expressions.\nThis is useful for preparing large texts for processing by language models, ensuring they are divided into manageable segments.\n\n#### Constructor Parameters:\n- `patterns` (list, optional): A list of regular expression patterns used to split the text. Default is to split by double newlines (`['\\n\\n']`).\n\n#### Example usage:\n```python\nchunker = RegexChunking(patterns=[r'\\n\\n', r'\\. '])\nchunks = chunker.chunk(\"This is a sample text. It will be split into chunks.\")\n```",
+    
+    "NlpSentenceChunking": "### NlpSentenceChunking\n\n`NlpSentenceChunking` uses a natural language processing model to chunk a given text into sentences. This approach leverages SpaCy to accurately split text based on sentence boundaries.\n\n#### Constructor Parameters:\n- None.\n\n#### Example usage:\n```python\nchunker = NlpSentenceChunking()\nchunks = chunker.chunk(\"This is a sample text. It will be split into sentences.\")\n```",
+    
+    "TopicSegmentationChunking": "### TopicSegmentationChunking\n\n`TopicSegmentationChunking` uses the TextTiling algorithm to segment a given text into topic-based chunks. This method identifies thematic boundaries in the text.\n\n#### Constructor Parameters:\n- `num_keywords` (int, optional): The number of keywords to extract for each topic segment. Default is `3`.\n\n#### Example usage:\n```python\nchunker = TopicSegmentationChunking(num_keywords=3)\nchunks = chunker.chunk(\"This is a sample text. It will be split into topic-based segments.\")\n```",
+    
+    "FixedLengthWordChunking": "### FixedLengthWordChunking\n\n`FixedLengthWordChunking` splits a given text into chunks of fixed length, based on the number of words.\n\n#### Constructor Parameters:\n- `chunk_size` (int, optional): The number of words in each chunk. Default is `100`.\n\n#### Example usage:\n```python\nchunker = FixedLengthWordChunking(chunk_size=100)\nchunks = chunker.chunk(\"This is a sample text. It will be split into fixed-length word chunks.\")\n```",
+    
+    "SlidingWindowChunking": "### SlidingWindowChunking\n\n`SlidingWindowChunking` uses a sliding window approach to chunk a given text. Each chunk has a fixed length, and the window slides by a specified step size.\n\n#### Constructor Parameters:\n- `window_size` (int, optional): The number of words in each chunk. Default is `100`.\n- `step` (int, optional): The number of words to slide the window. Default is `50`.\n\n#### Example usage:\n```python\nchunker = SlidingWindowChunking(window_size=100, step=50)\nchunks = chunker.chunk(\"This is a sample text. It will be split using a sliding window approach.\")\n```"
+  }
+  

docs/examples/quickstart.ipynb

@@ -47,7 +47,8 @@
       },
       "outputs": [],
       "source": [
-        "!pip install crawl4ai\n",
+        "# !pip install \"crawl4ai @ git+https://github.com/unclecode/crawl4ai.git\"\n",
+        "!pip install \"crawl4ai @ git+https://github.com/unclecode/crawl4ai.git@staging\"\n",
         "!pip install nest-asyncio\n",
         "!playwright install"
       ]
@@ -713,7 +714,7 @@
       "provenance": []
     },
     "kernelspec": {
-      "display_name": "venv",
+      "display_name": "Python 3",
       "language": "python",
       "name": "python3"
     },

docs/examples/quickstart_async.py

@@ -10,7 +10,7 @@ import time
 import json
 import os
 import re
-from typing import Dict, List
+from typing import Dict
 from bs4 import BeautifulSoup
 from pydantic import BaseModel, Field
 from crawl4ai import AsyncWebCrawler
@@ -379,18 +379,6 @@ async def crawl_custom_browser_type():
         print(result.markdown[:500])
         print("Time taken: ", time.time() - start)
 
-async def crawl_with_user_simultion():
-    async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
-        url = "YOUR-URL-HERE"
-        result = await crawler.arun(
-            url=url,
-            bypass_cache=True,
-            simulate_user = True,# Causes a series of random mouse movements and clicks to simulate user interaction
-            override_navigator = True # Overrides the navigator object to make it look like a real user
-        )
-        
-        print(result.markdown)    
-
 async def speed_comparison():
     # print("\n--- Speed Comparison ---")
     # print("Firecrawl (simulated):")
@@ -456,57 +444,6 @@ async def speed_comparison():
     print("If you run these tests in an environment with better network conditions,")
     print("you may observe an even more significant speed advantage for Crawl4AI.")
 
-async def generate_knowledge_graph():
-    class Entity(BaseModel):
-        name: str
-        description: str
-        
-    class Relationship(BaseModel):
-        entity1: Entity
-        entity2: Entity
-        description: str
-        relation_type: str
-
-    class KnowledgeGraph(BaseModel):
-        entities: List[Entity]
-        relationships: List[Relationship]
-
-    extraction_strategy = LLMExtractionStrategy(
-            provider='openai/gpt-4o-mini', # Or any other provider, including Ollama and open source models
-            api_token=os.getenv('OPENAI_API_KEY'), # In case of Ollama just pass "no-token"
-            schema=KnowledgeGraph.model_json_schema(),
-            extraction_type="schema",
-            instruction="""Extract entities and relationships from the given text."""
-    )
-    async with AsyncWebCrawler() as crawler:
-        url = "https://paulgraham.com/love.html"
-        result = await crawler.arun(
-            url=url,
-            bypass_cache=True,
-            extraction_strategy=extraction_strategy,
-            # magic=True
-        )
-        # print(result.extracted_content)
-        with open(os.path.join(__location__, "kb.json"), "w") as f:
-            f.write(result.extracted_content)
-
-async def fit_markdown_remove_overlay():
-    async with AsyncWebCrawler(headless = False) as crawler:
-        url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
-        result = await crawler.arun(
-            url=url,
-            bypass_cache=True,
-            word_count_threshold = 10,
-            remove_overlay_elements=True,
-            screenshot = True
-        )
-        # Save markdown to file
-        with open(os.path.join(__location__, "mexico_places.md"), "w") as f:
-            f.write(result.fit_markdown)
-
-    print("Done")
-
-
 async def main():
     await simple_crawl()
     await simple_example_with_running_js_code()
@@ -518,7 +455,7 @@ async def main():
     # LLM extraction examples
     await extract_structured_data_using_llm()
     await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
-    await extract_structured_data_using_llm("openai/gpt-4o", os.getenv("OPENAI_API_KEY"))
+    await extract_structured_data_using_llm("openai/gpt-4", os.getenv("OPENAI_API_KEY"))
     await extract_structured_data_using_llm("ollama/llama3.2")    
 
     # You always can pass custom headers to the extraction strategy

docs/extraction_strategies.json

@@ -0,0 +1,10 @@
+{
+    "NoExtractionStrategy": "### NoExtractionStrategy\n\n`NoExtractionStrategy` is a basic extraction strategy that returns the entire HTML content without any modification. It is useful for cases where no specific extraction is required. Only clean html, and amrkdown.\n\n#### Constructor Parameters:\nNone.\n\n#### Example usage:\n```python\nextractor = NoExtractionStrategy()\nextracted_content = extractor.extract(url, html)\n```",
+    
+    "LLMExtractionStrategy": "### LLMExtractionStrategy\n\n`LLMExtractionStrategy` uses a Language Model (LLM) to extract meaningful blocks or chunks from the given HTML content. This strategy leverages an external provider for language model completions.\n\n#### Constructor Parameters:\n- `provider` (str, optional): The provider to use for the language model completions. Default is `DEFAULT_PROVIDER` (e.g., openai/gpt-4).\n- `api_token` (str, optional): The API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.\n- `instruction` (str, optional): An instruction to guide the LLM on how to perform the extraction. This allows users to specify the type of data they are interested in or set the tone of the response. Default is `None`.\n\n#### Example usage:\n```python\nextractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')\nextracted_content = extractor.extract(url, html)\n```\n\nBy providing clear instructions, users can tailor the extraction process to their specific needs, enhancing the relevance and utility of the extracted content.",
+    
+    "CosineStrategy": "### CosineStrategy\n\n`CosineStrategy` uses hierarchical clustering based on cosine similarity to extract clusters of text from the given HTML content. This strategy is suitable for identifying related content sections.\n\n#### Constructor Parameters:\n- `semantic_filter` (str, optional): A string containing keywords for filtering relevant documents before clustering. If provided, documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.\n- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.\n- `max_dist` (float, optional): The maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.\n- `linkage_method` (str, optional): The linkage method for hierarchical clustering. Default is `'ward'`.\n- `top_k` (int, optional): Number of top categories to extract. Default is `3`.\n- `model_name` (str, optional): The model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.\n\n#### Example usage:\n```python\nextractor = CosineStrategy(semantic_filter='artificial intelligence', word_count_threshold=10, max_dist=0.2, linkage_method='ward', top_k=3, model_name='BAAI/bge-small-en-v1.5')\nextracted_content = extractor.extract(url, html)\n```\n\n#### Cosine Similarity Filtering\n\nWhen a `semantic_filter` is provided, the `CosineStrategy` applies an embedding-based filtering process to select relevant documents before performing hierarchical clustering.",
+    
+    "TopicExtractionStrategy": "### TopicExtractionStrategy\n\n`TopicExtractionStrategy` uses the TextTiling algorithm to segment the HTML content into topics and extracts keywords for each segment. This strategy is useful for identifying and summarizing thematic content.\n\n#### Constructor Parameters:\n- `num_keywords` (int, optional): Number of keywords to represent each topic segment. Default is `3`.\n\n#### Example usage:\n```python\nextractor = TopicExtractionStrategy(num_keywords=3)\nextracted_content = extractor.extract(url, html)\n```"
+  }
+  

docs/md _sync/api/core_classes_and_functions.md

@@ -0,0 +1,141 @@
+# Core Classes and Functions
+
+## Overview
+
+In this section, we will delve into the core classes and functions that make up the Crawl4AI library. This includes the `WebCrawler` class, various `CrawlerStrategy` classes, `ChunkingStrategy` classes, and `ExtractionStrategy` classes. Understanding these core components will help you leverage the full power of Crawl4AI for your web crawling and data extraction needs.
+
+## WebCrawler Class
+
+The `WebCrawler` class is the main class you'll interact with. It provides the interface for crawling web pages and extracting data.
+
+### Initialization
+
+```python
+from crawl4ai import WebCrawler
+
+# Create an instance of WebCrawler
+crawler = WebCrawler()
+```
+
+### Methods
+
+- **`warmup()`**: Prepares the crawler for use, such as loading necessary models.
+- **`run(url: str, **kwargs)`**: Runs the crawler on the specified URL with optional parameters for customization.
+
+```python
+crawler.warmup()
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(result)
+```
+
+## CrawlerStrategy Classes
+
+The `CrawlerStrategy` classes define how the web crawling is executed. The base class is `CrawlerStrategy`, which is extended by specific implementations like `LocalSeleniumCrawlerStrategy`.
+
+### CrawlerStrategy Base Class
+
+An abstract base class that defines the interface for different crawler strategies.
+
+```python
+from abc import ABC, abstractmethod
+
+class CrawlerStrategy(ABC):
+    @abstractmethod
+    def crawl(self, url: str, **kwargs) -> str:
+        pass
+    
+    @abstractmethod
+    def take_screenshot(self, save_path: str):
+        pass
+    
+    @abstractmethod
+    def update_user_agent(self, user_agent: str):
+        pass
+    
+    @abstractmethod
+    def set_hook(self, hook_type: str, hook: Callable):
+        pass
+```
+
+### LocalSeleniumCrawlerStrategy Class
+
+A concrete implementation of `CrawlerStrategy` that uses Selenium to crawl web pages.
+
+#### Initialization
+
+```python
+from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
+
+strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
+```
+
+#### Methods
+
+- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+```python
+result = strategy.crawl("https://www.example.com")
+strategy.take_screenshot("screenshot.png")
+strategy.update_user_agent("Mozilla/5.0")
+strategy.set_hook("before_get_url", lambda: print("About to get URL"))
+```
+
+## ChunkingStrategy Classes
+
+The `ChunkingStrategy` classes define how the text from a web page is divided into chunks. Here are a few examples:
+
+### RegexChunking Class
+
+Splits text using regular expressions.
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+chunker = RegexChunking(patterns=[r'\n\n'])
+chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
+```
+
+### NlpSentenceChunking Class
+
+Uses NLP to split text into sentences.
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
+```
+
+## ExtractionStrategy Classes
+
+The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks. Here are a few examples:
+
+### CosineStrategy Class
+
+Clusters text chunks based on cosine similarity.
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### LLMExtractionStrategy Class
+
+Uses a Language Model to extract meaningful blocks from HTML.
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+## Conclusion
+
+By understanding these core classes and functions, you can customize and extend Crawl4AI to suit your specific web crawling and data extraction needs. Happy crawling! 🕷️🤖
+

docs/md _sync/api/detailed_api_documentation.md

@@ -0,0 +1,338 @@
+# Detailed API Documentation
+
+## Overview
+
+This section provides comprehensive documentation for the Crawl4AI API, covering all classes, methods, and their parameters. This guide will help you understand how to utilize the API to its full potential, enabling efficient web crawling and data extraction.
+
+## WebCrawler Class
+
+The `WebCrawler` class is the primary interface for crawling web pages and extracting data.
+
+### Initialization
+
+```python
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+```
+
+### Methods
+
+#### `warmup()`
+
+Prepares the crawler for use, such as loading necessary models.
+
+```python
+crawler.warmup()
+```
+
+#### `run(url: str, **kwargs) -> CrawlResult`
+
+Crawls the specified URL and returns the result.
+
+- **Parameters:**
+  - `url` (str): The URL to crawl.
+  - `**kwargs`: Additional parameters for customization.
+
+- **Returns:**
+  - `CrawlResult`: An object containing the crawl result.
+
+- **Example:**
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(result)
+```
+
+### CrawlResult Class
+
+Represents the result of a crawl operation.
+
+- **Attributes:**
+  - `url` (str): The URL of the crawled page.
+  - `html` (str): The raw HTML of the page.
+  - `success` (bool): Whether the crawl was successful.
+  - `cleaned_html` (Optional[str]): The cleaned HTML.
+  - `media` (Dict[str, List[Dict]]): Media tags in the page (images, audio, video).
+  - `links` (Dict[str, List[Dict]]): Links in the page (external, internal).
+  - `screenshot` (Optional[str]): Base64 encoded screenshot.
+  - `markdown` (Optional[str]): Extracted content in Markdown format.
+  - `extracted_content` (Optional[str]): Extracted meaningful content.
+  - `metadata` (Optional[dict]): Metadata from the page.
+  - `error_message` (Optional[str]): Error message if any.
+
+## CrawlerStrategy Classes
+
+The `CrawlerStrategy` classes define how the web crawling is executed.
+
+### CrawlerStrategy Base Class
+
+An abstract base class for different crawler strategies.
+
+#### Methods
+
+- **`crawl(url: str, **kwargs) -> str`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+### LocalSeleniumCrawlerStrategy Class
+
+Uses Selenium to crawl web pages.
+
+#### Initialization
+
+```python
+from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
+
+strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
+```
+
+#### Methods
+
+- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+#### Example
+
+```python
+result = strategy.crawl("https://www.example.com")
+strategy.take_screenshot("screenshot.png")
+strategy.update_user_agent("Mozilla/5.0")
+strategy.set_hook("before_get_url", lambda: print("About to get URL"))
+```
+
+## ChunkingStrategy Classes
+
+The `ChunkingStrategy` classes define how the text from a web page is divided into chunks.
+
+### RegexChunking Class
+
+Splits text using regular expressions.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+chunker = RegexChunking(patterns=[r'\n\n'])
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into chunks.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
+```
+
+### NlpSentenceChunking Class
+
+Uses NLP to split text into sentences.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into sentences.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
+```
+
+### TopicSegmentationChunking Class
+
+Uses the TextTiling algorithm to segment text into topics.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import TopicSegmentationChunking
+
+chunker = TopicSegmentationChunking(num_keywords=3)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into topic-based segments.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into topic-based segments.")
+```
+
+### FixedLengthWordChunking Class
+
+Splits text into chunks of fixed length based on the number of words.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import FixedLengthWordChunking
+
+chunker = FixedLengthWordChunking(chunk_size=100)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into fixed-length word chunks.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into fixed-length word chunks.")
+```
+
+### SlidingWindowChunking Class
+
+Uses a sliding window approach to chunk text.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import SlidingWindowChunking
+
+chunker = SlidingWindowChunking(window_size=100, step=50)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text using a sliding window approach.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split using a sliding window approach.")
+```
+
+## ExtractionStrategy Classes
+
+The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks.
+
+### NoExtractionStrategy Class
+
+Returns the entire HTML content without any modification.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import NoExtractionStrategy
+
+extractor = NoExtractionStrategy()
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Returns the HTML content.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### LLMExtractionStrategy Class
+
+Uses a Language Model to extract meaningful blocks from HTML.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts meaningful content using the LLM.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### CosineStrategy Class
+
+Clusters text chunks based on cosine similarity.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts clusters of text based on cosine similarity.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### TopicExtractionStrategy Class
+
+Uses the TextTiling algorithm to segment HTML content into topics and extract keywords.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import TopicExtractionStrategy
+
+extractor = TopicExtractionStrategy(num_keywords=3)
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts topic-based segments and keywords.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+## Parameters
+
+Here are the common parameters used across various classes and methods:
+
+- **`url`** (str): The URL to crawl.
+- **`html`** (str): The HTML content of the page.
+- **`user_agent`** (str): The user agent for the HTTP requests.
+- **`patterns`** (list): A list of regular expression patterns for chunking.
+- **`num_keywords`** (int): Number of keywords for topic extraction.
+- **`chunk_size`** (int): Number of words in each chunk.
+- **`window_size`** (int): Number of words in the sliding window.
+- **`step`** (int): Step size for the sliding window.
+- **`semantic_filter`** (str): Keywords for filtering relevant documents.
+- **`word_count_threshold`** (int): Minimum number of words per cluster.
+- **`max_dist`** (float): Maximum cophenetic distance for clustering.
+- **`linkage_method`** (str): Linkage method for hierarchical clustering.
+- **`top_k`** (int): Number of top categories to extract.
+- **`provider`** (
+
+str): Provider for language model completions.
+- **`api_token`** (str): API token for the provider.
+- **`instruction`** (str): Instruction to guide the LLM extraction.
+
+## Conclusion
+
+This detailed API documentation provides a thorough understanding of the classes, methods, and parameters in the Crawl4AI library. With this knowledge, you can effectively use the API to perform advanced web crawling and data extraction tasks.

docs/md _sync/changelog.md

@@ -0,0 +1,102 @@
+# Changelog
+
+## [v0.2.77] - 2024-08-04
+
+Significant improvements in text processing and performance:
+
+- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
+- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
+- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
+- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
+
+These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
+
+## [v0.2.76] - 2024-08-02
+
+Major improvements in functionality, performance, and cross-platform compatibility! 🚀
+
+- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
+- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
+- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
+- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
+- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
+
+A big shoutout to our amazing community contributors:
+- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
+- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
+- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
+
+Your contributions are driving Crawl4AI forward! 🙌
+
+## [v0.2.75] - 2024-07-19
+
+Minor improvements for a more maintainable codebase:
+
+- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
+- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
+
+These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
+
+
+## v0.2.74 - 2024-07-08
+A slew of exciting updates to improve the crawler's stability and robustness! 🎉
+
+- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
+- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
+- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
+- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
+
+## [v0.2.73] - 2024-07-03
+
+💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
+
+* Supporting website need "with-head" mode to crawl the website with head.
+* Fixing the installation issues for setup.py and dockerfile.
+* Resolve multiple issues.
+
+## [v0.2.72] - 2024-06-30
+
+This release brings exciting updates and improvements to our project! 🎉
+
+* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
+* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
+* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
+* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
+
+These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
+
+## [0.2.71] - 2024-06-26
+
+**Improved Error Handling and Performance** 🚧
+
+* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
+* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
+* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
+* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
+
+These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
+
+## [0.2.71] - 2024-06-25
+### Fixed
+- Speed up twice the extraction function.
+
+## [0.2.6] - 2024-06-22
+### Fixed
+- Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
+
+## [0.2.5] - 2024-06-18
+### Added
+- Added five important hooks to the crawler:
+  - on_driver_created: Called when the driver is ready for initializations.
+  - before_get_url: Called right before Selenium fetches the URL.
+  - after_get_url: Called after Selenium fetches the URL.
+  - before_return_html: Called when the data is parsed and ready.
+  - on_user_agent_updated: Called when the user changes the user_agent, causing the driver to reinitialize.
+- Added an example in `quickstart.py` in the example folder under the docs.
+- Enhancement issue #24: Replaced inline HTML tags (e.g., DEL, INS, SUB, ABBR) with textual format for better context handling in LLM.
+- Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
+- Updated Dockerfile to ensure compatibility across multiple platforms (Hopefully!).
+
+## [0.2.4] - 2024-06-17
+### Fixed
+- Fix issue #22: Use MD5 hash for caching HTML files to handle long URLs

docs/md _sync/contact.md

@@ -0,0 +1,25 @@
+# Contact
+If you have any questions, suggestions, or feedback, please feel free to reach out to us:
+
+- GitHub: [unclecode](https://github.com/unclecode)
+- Twitter: [@unclecode](https://twitter.com/unclecode)
+- Website: [crawl4ai.com](https://crawl4ai.com)
+
+
+## Contributing 🤝
+
+We welcome contributions from the open-source community to help improve Crawl4AI and make it even more valuable for AI enthusiasts and developers. To contribute, please follow these steps:
+
+1. Fork the repository.
+2. Create a new branch for your feature or bug fix.
+3. Make your changes and commit them with descriptive messages.
+4. Push your changes to your forked repository.
+5. Submit a pull request to the main repository.
+
+For more information on contributing, please see our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md).
+
+## License 📄
+
+Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
+
+Let's work together to make the web more accessible and useful for AI applications! 💪🌐🤖

docs/md _sync/demo.md

@@ -0,0 +1,231 @@
+# Interactive Demo for Crowler
+<div id="demo">
+    <form id="crawlForm" class="terminal-form">
+        <fieldset>
+            <legend>Enter URL and Options</legend>
+            <div class="form-group">
+                <label for="url">Enter URL:</label>
+                <input type="text" id="url" name="url" required>
+            </div>
+            <div class="form-group">
+                <label for="screenshot">Get Screenshot:</label>
+                <input type="checkbox" id="screenshot" name="screenshot">
+            </div>
+            <div class="form-group">
+                <button class="btn btn-default" type="submit">Submit</button>
+            </div>
+
+        </fieldset>
+    </form>
+
+    <div id="loading" class="loading-message">
+        <div class="terminal-alert terminal-alert-primary">Loading... Please wait.</div>
+    </div>
+
+    <section id="response" class="response-section">
+        <h2>Response</h2>
+        <div class="tabs">
+            <ul class="tab-list">
+                <li class="tab-item" onclick="showTab('markdown')">Markdown</li>
+                <li class="tab-item" onclick="showTab('cleanedHtml')">Cleaned HTML</li>
+                <li class="tab-item" onclick="showTab('media')">Media</li>
+                <li class="tab-item" onclick="showTab('extractedContent')">Extracted Content</li>
+                <li class="tab-item" onclick="showTab('screenshot')">Screenshot</li>
+                <li class="tab-item" onclick="showTab('pythonCode')">Python Code</li>
+            </ul>
+            <div class="tab-content" id="tab-markdown">
+                <header>
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('markdownContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('markdownContent', 'markdown.md')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="markdownContent" class="language-markdown hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-cleanedHtml" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('cleanedHtmlContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('cleanedHtmlContent', 'cleaned.html')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="cleanedHtmlContent" class="language-html hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-media" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('mediaContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('mediaContent', 'media.json')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="mediaContent" class="language-json hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-extractedContent" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('extractedContentContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('extractedContentContent', 'extracted_content.json')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="extractedContentContent" class="language-json hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-screenshot" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadImage('screenshotContent', 'screenshot.png')">Download</button>
+                    </div>
+                </header>
+                <pre><img id="screenshotContent" /></pre>
+            </div>
+
+            <div class="tab-content" id="tab-pythonCode" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('pythonCode')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('pythonCode', 'example.py')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="pythonCode" class="language-python hljs"></code></pre>
+            </div>
+        </div>
+    </section>
+
+    <div id="error" class="error-message" style="display: none; margin-top:1em;">
+        <div class="terminal-alert terminal-alert-error"></div>
+    </div>
+
+    <script>
+        function showTab(tabId) {
+            const tabs = document.querySelectorAll('.tab-content');
+            tabs.forEach(tab => tab.style.display = 'none');
+            document.getElementById(`tab-${tabId}`).style.display = 'block';
+        }
+
+        function redo(codeBlock, codeText){
+            codeBlock.classList.remove('hljs');
+            codeBlock.removeAttribute('data-highlighted');
+
+            // Set new code and re-highlight
+            codeBlock.textContent = codeText;
+            hljs.highlightBlock(codeBlock);
+        }
+
+        function copyToClipboard(elementId) {
+            const content = document.getElementById(elementId).textContent;
+            navigator.clipboard.writeText(content).then(() => {
+                alert('Copied to clipboard');
+            });
+        }
+
+        function downloadContent(elementId, filename) {
+            const content = document.getElementById(elementId).textContent;
+            const blob = new Blob([content], { type: 'text/plain' });
+            const url = window.URL.createObjectURL(blob);
+            const a = document.createElement('a');
+            a.style.display = 'none';
+            a.href = url;
+            a.download = filename;
+            document.body.appendChild(a);
+            a.click();
+            window.URL.revokeObjectURL(url);
+            document.body.removeChild(a);
+        }
+
+        function downloadImage(elementId, filename) {
+            const content = document.getElementById(elementId).src;
+            const a = document.createElement('a');
+            a.style.display = 'none';
+            a.href = content;
+            a.download = filename;
+            document.body.appendChild(a);
+            a.click();
+            document.body.removeChild(a);
+        }
+
+        document.getElementById('crawlForm').addEventListener('submit', function(event) {
+            event.preventDefault();
+            document.getElementById('loading').style.display = 'block';
+            document.getElementById('response').style.display = 'none';
+
+            const url = document.getElementById('url').value;
+            const screenshot = document.getElementById('screenshot').checked;
+            const data = {
+                urls: [url],
+                bypass_cache: false,
+                word_count_threshold: 5,
+                screenshot: screenshot
+            };
+
+            fetch('/crawl', {
+                method: 'POST',
+                headers: {
+                    'Content-Type': 'application/json'
+                },
+                body: JSON.stringify(data)
+            })
+            .then(response => {
+                if (!response.ok) {
+                    if (response.status === 429) {
+                        return response.json().then(err => { 
+                            throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
+                        });
+                    }
+                    throw new Error('Network response was not ok');
+                }
+                return response.json();
+            })
+            .then(data => {
+                data = data.results[0]; // Only one URL is requested
+                document.getElementById('loading').style.display = 'none';
+                document.getElementById('response').style.display = 'block';
+                redo(document.getElementById('markdownContent'), data.markdown);
+                redo(document.getElementById('cleanedHtmlContent'), data.cleaned_html);
+                redo(document.getElementById('mediaContent'), JSON.stringify(data.media, null, 2));
+                redo(document.getElementById('extractedContentContent'), data.extracted_content);
+                if (screenshot) {
+                    document.getElementById('screenshotContent').src = `data:image/png;base64,${data.screenshot}`;
+                }
+                const pythonCode = `
+from crawl4ai.web_crawler import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+    url='${url}',
+    screenshot=${screenshot}
+)
+print(result)
+                `;
+                redo(document.getElementById('pythonCode'), pythonCode);
+                document.getElementById('error').style.display = 'none';
+            })
+            .catch(error => {
+                document.getElementById('loading').style.display = 'none';
+                document.getElementById('error').style.display = 'block';
+                let errorMessage = 'An unexpected error occurred. Please try again later.';
+                
+                if (error.status === 429) {
+                    const details = error.details;
+                    if (details.retry_after) {
+                        errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
+                    } else if (details.reset_at) {
+                        const resetTime = new Date(details.reset_at);
+                        const waitTime = Math.ceil((resetTime - new Date()) / 1000);
+                        errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
+                    } else {
+                        errorMessage = `Rate limit exceeded. Please try again later.`;
+                    }
+                } else if (error.message) {
+                    errorMessage = error.message;
+                }
+                
+                document.querySelector('#error .terminal-alert').textContent = errorMessage;
+            });
+        });
+    </script>
+</div>

docs/md _sync/examples/hooks_auth.md

@@ -0,0 +1,100 @@
+# Hooks & Auth
+
+Crawl4AI allows you to customize the behavior of the web crawler using hooks. Hooks are functions that are called at specific points in the crawling process, allowing you to modify the crawler's behavior or perform additional actions. This example demonstrates how to use various hooks to customize the crawling process.
+
+## Example: Using Crawler Hooks
+
+Let's see how we can customize the crawler using hooks! In this example, we'll:
+
+1. Maximize the browser window and log in to a website when the driver is created.
+2. Add a custom header before fetching the URL.
+3. Log the current URL after fetching it.
+4. Log the length of the HTML before returning it.
+
+### Hook Definitions
+
+```python
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.crawler_strategy import *
+
+def on_driver_created(driver):
+    print("[HOOK] on_driver_created")
+    # Example customization: maximize the window
+    driver.maximize_window()
+    
+    # Example customization: logging in to a hypothetical website
+    driver.get('https://example.com/login')
+    
+    from selenium.webdriver.support.ui import WebDriverWait
+    from selenium.webdriver.common.by import By
+    from selenium.webdriver.support import expected_conditions as EC
+    
+    WebDriverWait(driver, 10).until(
+        EC.presence_of_element_located((By.NAME, 'username'))
+    )
+    driver.find_element(By.NAME, 'username').send_keys('testuser')
+    driver.find_element(By.NAME, 'password').send_keys('password123')
+    driver.find_element(By.NAME, 'login').click()
+    WebDriverWait(driver, 10).until(
+        EC.presence_of_element_located((By.ID, 'welcome'))
+    )
+    # Add a custom cookie
+    driver.add_cookie({'name': 'test_cookie', 'value': 'cookie_value'})
+    return driver        
+    
+
+def before_get_url(driver):
+    print("[HOOK] before_get_url")
+    # Example customization: add a custom header
+    # Enable Network domain for sending headers
+    driver.execute_cdp_cmd('Network.enable', {})
+    # Add a custom header
+    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
+    return driver
+
+def after_get_url(driver):
+    print("[HOOK] after_get_url")
+    # Example customization: log the URL
+    print(driver.current_url)
+    return driver
+
+def before_return_html(driver, html):
+    print("[HOOK] before_return_html")
+    # Example customization: log the HTML
+    print(len(html))
+    return driver
+```
+
+### Using the Hooks with the WebCrawler
+
+```python
+print("\n🔗 [bold cyan]Using Crawler Hooks: Let's see how we can customize the crawler using hooks![/bold cyan]", True)
+crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
+crawler_strategy.set_hook('on_driver_created', on_driver_created)
+crawler_strategy.set_hook('before_get_url', before_get_url)
+crawler_strategy.set_hook('after_get_url', after_get_url)
+crawler_strategy.set_hook('before_return_html', before_return_html)
+crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
+crawler.warmup()
+
+result = crawler.run(url="https://example.com")
+
+print("[LOG] 📦 [bold yellow]Crawler Hooks result:[/bold yellow]")
+print(result)
+```
+
+### Explanation
+
+- `on_driver_created`: This hook is called when the Selenium driver is created. In this example, it maximizes the window, logs in to a website, and adds a custom cookie.
+- `before_get_url`: This hook is called right before Selenium fetches the URL. In this example, it adds a custom HTTP header.
+- `after_get_url`: This hook is called after Selenium fetches the URL. In this example, it logs the current URL.
+- `before_return_html`: This hook is called before returning the HTML content. In this example, it logs the length of the HTML content.
+
+### Additional Ideas
+
+- **Add custom headers to requests**: You can add custom headers to the requests using the `before_get_url` hook.
+- **Perform safety checks**: Use the hooks to perform safety checks before the crawling process starts.
+- **Modify the HTML content**: Use the `before_return_html` hook to modify the HTML content before it is returned.
+- **Log additional information**: Use the hooks to log additional information for debugging or monitoring purposes.
+
+By using these hooks, you can customize the behavior of the crawler to suit your specific needs.

docs/md _sync/examples/index.md

@@ -0,0 +1,29 @@
+# Examples
+
+Welcome to the examples section of Crawl4AI documentation! In this section, you will find practical examples demonstrating how to use Crawl4AI for various web crawling and data extraction tasks. Each example is designed to showcase different features and capabilities of the library.
+
+## Examples Index
+
+### [LLM Extraction](llm_extraction.md)
+
+This example demonstrates how to use Crawl4AI to extract information using Large Language Models (LLMs). You will learn how to configure the `LLMExtractionStrategy` to get structured data from web pages.
+
+### [JS Execution & CSS Filtering](js_execution_css_filtering.md)
+
+Learn how to execute custom JavaScript code and filter data using CSS selectors. This example shows how to perform complex web interactions and extract specific content from web pages.
+
+### [Hooks & Auth](hooks_auth.md)
+
+This example covers the use of custom hooks for authentication and other pre-crawling tasks. You will see how to set up hooks to modify headers, authenticate sessions, and perform other preparatory actions before crawling.
+
+### [Summarization](summarization.md)
+
+Discover how to use Crawl4AI to summarize web page content. This example demonstrates the summarization capabilities of the library, helping you extract concise information from lengthy web pages.
+
+### [Research Assistant](research_assistant.md)
+
+In this example, Crawl4AI is used as a research assistant to gather and organize information from multiple sources. You will learn how to use various extraction and chunking strategies to compile a comprehensive report.
+
+---
+
+Each example includes detailed explanations and code snippets to help you understand and implement the features in your projects. Click on the links to explore each example and start making the most of Crawl4AI!

docs/md _sync/examples/js_execution_css_filtering.md

@@ -0,0 +1,44 @@
+# JS Execution & CSS Filtering
+
+In this example, we'll demonstrate how to use Crawl4AI to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.
+
+## Example: Extracting Structured Data
+
+```python
+# Import necessary modules
+from crawl4ai import WebCrawler
+from crawl4ai.chunking_strategy import *
+from crawl4ai.extraction_strategy import *
+from crawl4ai.crawler_strategy import *
+
+# Define the JavaScript code to click the "Load More" button
+js_code = ["""
+const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+loadMoreButton && loadMoreButton.click();
+"""]
+
+crawler = WebCrawler(verbose=True)
+crawler.warmup()
+# Run the crawler with keyword filtering and CSS selector
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    js=js_code,
+    css_selector="p",
+    extraction_strategy=CosineStrategy(
+        semantic_filter="technology",
+    ),
+)
+
+# Display the extracted result
+print(result)
+```
+
+### Explanation
+
+1. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button. This is useful for loading additional content dynamically.
+2. **CSS Selector**: The `css_selector="p"` parameter ensures that only paragraph (`<p>`) tags are extracted from the web page.
+3. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
+
+## Try It Yourself
+
+This example demonstrates the power and flexibility of Crawl4AI in handling complex web interactions and extracting meaningful data. You can customize the JavaScript code, CSS selectors, and extraction strategies to suit your specific requirements.

docs/md _sync/examples/llm_extraction.md

@@ -0,0 +1,90 @@
+# LLM Extraction
+
+Crawl4AI allows you to use Language Models (LLMs) to extract structured data or relevant content from web pages. Below are two examples demonstrating how to use LLMExtractionStrategy for different purposes.
+
+## Example 1: Extract Structured Data
+
+In this example, we use the `LLMExtractionStrategy` to extract structured data (model names and their fees) from the OpenAI pricing page.
+
+```python
+import os
+import time
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.chunking_strategy import *
+from crawl4ai.extraction_strategy import *
+from crawl4ai.crawler_strategy import *
+
+url = r'https://openai.com/api/pricing/'
+
+crawler = WebCrawler()
+crawler.warmup()
+
+from pydantic import BaseModel, Field
+
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
+
+result = crawler.run(
+    url=url,
+    word_count_threshold=1,
+    extraction_strategy= LLMExtractionStrategy(
+        provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
+        schema=OpenAIModelFee.model_json_schema(),
+        extraction_type="schema",
+        instruction="From the crawled content, extract all mentioned model names along with their "\
+            "fees for input and output tokens. Make sure not to miss anything in the entire content. "\
+            'One extracted model JSON format should look like this: '\
+            '{ "model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens" }'
+    ),
+    bypass_cache=True,
+)
+
+model_fees = json.loads(result.extracted_content)
+
+print(len(model_fees))
+
+with open(".data/data.json", "w", encoding="utf-8") as f:
+    f.write(result.extracted_content)
+```
+
+## Example 2: Extract Relevant Content
+
+In this example, we instruct the LLM to extract only content related to technology from the NBC News business page.
+
+```python
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+        url="https://www.nbcnews.com/business",
+        extraction_strategy=LLMExtractionStrategy(
+            provider="openai/gpt-4o",
+            api_token=os.getenv('OPENAI_API_KEY'),
+            instruction="Extract only content related to technology"
+        ),
+    bypass_cache=True,
+    )
+
+model_fees = json.loads(result.extracted_content)
+
+print(len(model_fees))
+
+with open(".data/data.json", "w", encoding="utf-8") as f:
+    f.write(result.extracted_content)
+```
+
+## Customizing LLM Provider
+
+Under the hood, Crawl4AI uses the `litellm` library, which allows you to use any LLM provider you want. Just pass the correct model name and API token.
+
+```python
+extraction_strategy=LLMExtractionStrategy(
+    provider="your_llm_provider/model_name",
+    api_token="your_api_token",
+    instruction="Your extraction instruction"
+)
+```
+
+This flexibility allows you to integrate with various LLM providers and tailor the extraction process to your specific needs.

docs/md _sync/examples/research_assistant.md

@@ -0,0 +1,248 @@
+## Research Assistant Example
+
+This example demonstrates how to build a research assistant using `Chainlit` and `Crawl4AI`. The assistant will be capable of crawling web pages for information and answering questions based on the crawled content. Additionally, it integrates speech-to-text functionality for audio inputs.
+
+### Step-by-Step Guide
+
+1. **Install Required Packages**
+
+    Ensure you have the necessary packages installed. You need `chainlit`, `groq`, `requests`, and `openai`.
+
+    ```bash
+    pip install chainlit groq requests openai
+    ```
+
+2. **Import Libraries**
+
+    Import all the necessary modules and initialize the OpenAI client.
+
+    ```python
+    import os
+    import time
+    from openai import AsyncOpenAI
+    import chainlit as cl
+    import re
+    import requests
+    from io import BytesIO
+    from chainlit.element import ElementBased
+    from groq import Groq
+
+    from concurrent.futures import ThreadPoolExecutor
+
+    client = AsyncOpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.getenv("GROQ_API_KEY"))
+
+    # Instrument the OpenAI client
+    cl.instrument_openai()
+    ```
+
+3. **Set Configuration**
+
+    Define the model settings for the assistant.
+
+    ```python
+    settings = {
+        "model": "llama3-8b-8192",
+        "temperature": 0.5,
+        "max_tokens": 500,
+        "top_p": 1,
+        "frequency_penalty": 0,
+        "presence_penalty": 0,
+    }
+    ```
+
+4. **Define Utility Functions**
+
+    - **Extract URLs from Text**: Use regex to find URLs in messages.
+
+        ```python
+        def extract_urls(text):
+            url_pattern = re.compile(r'(https?://\S+)')
+            return url_pattern.findall(text)
+        ```
+
+    - **Crawl URL**: Send a request to `Crawl4AI` to fetch the content of a URL.
+
+        ```python
+        def crawl_url(url):
+            data = {
+                "urls": [url],
+                "include_raw_html": True,
+                "word_count_threshold": 10,
+                "extraction_strategy": "NoExtractionStrategy",
+                "chunking_strategy": "RegexChunking"
+            }
+            response = requests.post("https://crawl4ai.com/crawl", json=data)
+            response_data = response.json()
+            response_data = response_data['results'][0]
+            return response_data['markdown']
+        ```
+
+5. **Initialize Chat Start Event**
+
+    Set up the initial chat message and user session.
+
+    ```python
+    @cl.on_chat_start
+    async def on_chat_start():
+        cl.user_session.set("session", {
+            "history": [],
+            "context": {}
+        })  
+        await cl.Message(
+            content="Welcome to the chat! How can I assist you today?"
+        ).send()
+    ```
+
+6. **Handle Incoming Messages**
+
+    Process user messages, extract URLs, and crawl them concurrently. Update the chat history and system message.
+
+    ```python
+    @cl.on_message
+    async def on_message(message: cl.Message):
+        user_session = cl.user_session.get("session")
+
+        # Extract URLs from the user's message
+        urls = extract_urls(message.content)
+
+        futures = []
+        with ThreadPoolExecutor() as executor:
+            for url in urls:
+                futures.append(executor.submit(crawl_url, url))
+
+        results = [future.result() for future in futures]
+
+        for url, result in zip(urls, results):
+            ref_number = f"REF_{len(user_session['context']) + 1}"
+            user_session["context"][ref_number] = {
+                "url": url,
+                "content": result
+            }    
+
+        user_session["history"].append({
+            "role": "user",
+            "content": message.content
+        })
+
+        # Create a system message that includes the context
+        context_messages = [
+            f'<appendix ref="{ref}">\n{data["content"]}\n</appendix>'
+            for ref, data in user_session["context"].items()
+        ]
+        if context_messages:
+            system_message = {
+                "role": "system",
+                "content": (
+                    "You are a helpful bot. Use the following context for answering questions. "
+                    "Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
+                    "If the question requires any information from the provided appendices or context, refer to the sources. "
+                    "If not, there is no need to add a references section. "
+                    "At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
+                    "\n\n".join(context_messages)
+                )
+            }
+        else:
+            system_message = {
+                "role": "system",
+                "content": "You are a helpful assistant."
+            }
+
+        msg = cl.Message(content="")
+        await msg.send()
+
+        # Get response from the LLM
+        stream = await client.chat.completions.create(
+            messages=[
+                system_message,
+                *user_session["history"]
+            ],
+            stream=True,
+            **settings
+        )
+
+        assistant_response = ""
+        async for part in stream:
+            if token := part.choices[0].delta.content:
+                assistant_response += token
+                await msg.stream_token(token)
+
+        # Add assistant message to the history
+        user_session["history"].append({
+            "role": "assistant",
+            "content": assistant_response
+        })
+        await msg.update()
+
+        # Append the reference section to the assistant's response
+        reference_section = "\n\nReferences:\n"
+        for ref, data in user_session["context"].items():
+            reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
+
+        msg.content += reference_section
+        await msg.update()
+    ```
+
+7. **Handle Audio Input**
+
+    Capture and transcribe audio input. Store the audio buffer and transcribe it when the audio ends.
+
+    ```python
+    @cl.on_audio_chunk
+    async def on_audio_chunk(chunk: cl.AudioChunk):
+        if chunk.isStart:
+            buffer = BytesIO()
+            buffer.name = f"input_audio.{chunk.mimeType.split('/')[1]}"
+            cl.user_session.set("audio_buffer", buffer)
+            cl.user_session.set("audio_mime_type", chunk.mimeType)
+
+        cl.user_session.get("audio_buffer").write(chunk.data)
+
+    @cl.step(type="tool")
+    async def speech_to_text(audio_file):
+        cli = Groq()
+        response = await client.audio.transcriptions.create(
+            model="whisper-large-v3", file=audio_file
+        )
+        return response.text
+
+    @cl.on_audio_end
+    async def on_audio_end(elements: list[ElementBased]):
+        audio_buffer: BytesIO = cl.user_session.get("audio_buffer")
+        audio_buffer.seek(0)
+        audio_file = audio_buffer.read()
+        audio_mime_type: str = cl.user_session.get("audio_mime_type")
+        
+        start_time = time.time()
+        transcription = await speech_to_text((audio_buffer.name, audio_file, audio_mime_type))
+        end_time = time.time()
+        print(f"Transcription took {end_time - start_time} seconds")
+        
+        user_msg = cl.Message(
+            author="You", 
+            type="user_message",
+            content=transcription
+        )
+        await user_msg.send()
+        await on_message(user_msg)
+    ```
+
+8. **Run the Chat Application**
+
+    Start the Chainlit application.
+
+    ```python
+    if __name__ == "__main__":
+        from chainlit.cli import run_chainlit
+        run_chainlit(__file__)
+    ```
+
+### Explanation
+
+- **Libraries and Configuration**: Import necessary libraries and configure the OpenAI client.
+- **Utility Functions**: Define functions to extract URLs and crawl them.
+- **Chat Start Event**: Initialize chat session and welcome message.
+- **Message Handling**: Extract URLs, crawl them concurrently, and update chat history and context.
+- **Audio Handling**: Capture, buffer, and transcribe audio input, then process the transcription as text.
+- **Running the Application**: Start the Chainlit server to interact with the assistant.
+
+This example showcases how to create an interactive research assistant that can fetch, process, and summarize web content, along with handling audio inputs for a seamless user experience.

docs/md _sync/examples/summarization.md

@@ -0,0 +1,108 @@
+## Summarization Example
+
+This example demonstrates how to use `Crawl4AI` to extract a summary from a web page. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.
+
+### Step-by-Step Guide
+
+1. **Import Necessary Modules**
+
+    First, import the necessary modules and classes.
+
+    ```python
+    import os
+    import time
+    import json
+    from crawl4ai.web_crawler import WebCrawler
+    from crawl4ai.chunking_strategy import *
+    from crawl4ai.extraction_strategy import *
+    from crawl4ai.crawler_strategy import *
+    from pydantic import BaseModel, Field
+    ```
+
+2. **Define the URL to be Crawled**
+
+    Set the URL of the web page you want to summarize.
+
+    ```python
+    url = r'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
+    ```
+
+3. **Initialize the WebCrawler**
+
+    Create an instance of the `WebCrawler` and call the `warmup` method.
+
+    ```python
+    crawler = WebCrawler()
+    crawler.warmup()
+    ```
+
+4. **Define the Data Model**
+
+    Use Pydantic to define the structure of the extracted data.
+
+    ```python
+    class PageSummary(BaseModel):
+        title: str = Field(..., description="Title of the page.")
+        summary: str = Field(..., description="Summary of the page.")
+        brief_summary: str = Field(..., description="Brief summary of the page.")
+        keywords: list = Field(..., description="Keywords assigned to the page.")
+    ```
+
+5. **Run the Crawler**
+
+    Set up and run the crawler with the `LLMExtractionStrategy`. Provide the necessary parameters, including the schema for the extracted data and the instruction for the LLM.
+
+    ```python
+    result = crawler.run(
+        url=url,
+        word_count_threshold=1,
+        extraction_strategy=LLMExtractionStrategy(
+            provider="openai/gpt-4o", 
+            api_token=os.getenv('OPENAI_API_KEY'), 
+            schema=PageSummary.model_json_schema(),
+            extraction_type="schema",
+            apply_chunking=False,
+            instruction=(
+                "From the crawled content, extract the following details: "
+                "1. Title of the page "
+                "2. Summary of the page, which is a detailed summary "
+                "3. Brief summary of the page, which is a paragraph text "
+                "4. Keywords assigned to the page, which is a list of keywords. "
+                'The extracted JSON format should look like this: '
+                '{ "title": "Page Title", "summary": "Detailed summary of the page.", '
+                '"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
+            )
+        ),
+        bypass_cache=True,
+    )
+    ```
+
+6. **Process the Extracted Data**
+
+    Load the extracted content into a JSON object and print it.
+
+    ```python
+    page_summary = json.loads(result.extracted_content)
+    print(page_summary)
+    ```
+
+7. **Save the Extracted Data**
+
+    Save the extracted data to a file for further use.
+
+    ```python
+    with open(".data/page_summary.json", "w", encoding="utf-8") as f:
+        f.write(result.extracted_content)
+    ```
+
+### Explanation
+
+- **Importing Modules**: Import the necessary modules, including `WebCrawler` and `LLMExtractionStrategy` from `Crawl4AI`.
+- **URL Definition**: Set the URL of the web page you want to crawl and summarize.
+- **WebCrawler Initialization**: Create an instance of `WebCrawler` and call the `warmup` method to prepare the crawler.
+- **Data Model Definition**: Define the structure of the data you want to extract using Pydantic's `BaseModel`.
+- **Crawler Execution**: Run the crawler with the `LLMExtractionStrategy`, providing the schema and detailed instructions for the extraction process.
+- **Data Processing**: Load the extracted content into a JSON object and print it to verify the results.
+- **Data Saving**: Save the extracted data to a file for further use.
+
+This example demonstrates how to harness the power of `Crawl4AI` to perform advanced web crawling and data extraction tasks with minimal code.

docs/md _sync/full_details/advanced_features.md

@@ -0,0 +1,138 @@
+# Advanced Features
+
+Crawl4AI offers a range of advanced features that allow you to fine-tune your web crawling and data extraction process. This section will cover some of these advanced features, including taking screenshots, extracting media and links, customizing the user agent, using custom hooks, and leveraging CSS selectors.
+
+## Taking Screenshots 📸
+
+One of the cool features of Crawl4AI is the ability to take screenshots of the web pages you're crawling. This can be particularly useful for visual verification or for capturing the state of dynamic content.
+
+Here's how you can take a screenshot:
+
+```python
+from crawl4ai import WebCrawler
+import base64
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with the screenshot parameter
+result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
+
+# Save the screenshot to a file
+with open("screenshot.png", "wb") as f:
+    f.write(base64.b64decode(result.screenshot))
+
+print("Screenshot saved to 'screenshot.png'!")
+```
+
+In this example, we create a `WebCrawler` instance, warm it up, and then run it with the `screenshot` parameter set to `True`. The screenshot is saved as a base64 encoded string in the result, which we then decode and save as a PNG file.
+
+## Extracting Media and Links 🎨🔗
+
+Crawl4AI can extract all media tags (images, audio, and video) and links (both internal and external) from a web page. This feature is useful for collecting multimedia content or analyzing link structures.
+
+Here's an example:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler
+result = crawler.run(url="https://www.nbcnews.com/business")
+
+print("Extracted media:", result.media)
+print("Extracted links:", result.links)
+```
+
+In this example, the `result` object contains dictionaries for media and links, which you can access and use as needed.
+
+## Customizing the User Agent 🕵️‍♂️
+
+Crawl4AI allows you to set a custom user agent for your HTTP requests. This can help you avoid detection by web servers or simulate different browsing environments.
+
+Here's how to set a custom user agent:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with a custom user agent
+result = crawler.run(url="https://www.nbcnews.com/business", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
+
+print("Crawl result:", result)
+```
+
+In this example, we specify a custom user agent string when running the crawler.
+
+## Using Custom Hooks 🪝
+
+Hooks are a powerful feature in Crawl4AI that allow you to customize the crawling process at various stages. You can define hooks for actions such as driver initialization, before and after URL fetching, and before returning the HTML.
+
+Here's an example of using hooks:
+
+```python
+from crawl4ai import WebCrawler
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+# Define the hooks
+def on_driver_created(driver):
+    driver.maximize_window()
+    driver.get('https://example.com/login')
+    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.NAME, 'username'))).send_keys('testuser')
+    driver.find_element(By.NAME, 'password').send_keys('password123')
+    driver.find_element(By.NAME, 'login').click()
+    return driver
+
+def before_get_url(driver):
+    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
+    return driver
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Set the hooks
+crawler.set_hook('on_driver_created', on_driver_created)
+crawler.set_hook('before_get_url', before_get_url)
+
+# Run the crawler
+result = crawler.run(url="https://example.com")
+
+print("Crawl result:", result)
+```
+
+In this example, we define hooks to handle driver initialization and custom headers before fetching the URL.
+
+## Using CSS Selectors 🎯
+
+CSS selectors allow you to target specific elements on a web page for extraction. This can be useful for scraping structured content, such as articles or product details.
+
+Here's an example of using a CSS selector:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with a CSS selector to extract only H2 tags
+result = crawler.run(url="https://www.nbcnews.com/business", css_selector="h2")
+
+print("Extracted H2 tags:", result.extracted_content)
+```
+
+In this example, we use the `css_selector` parameter to extract only the H2 tags from the web page.
+
+---
+
+With these advanced features, you can leverage Crawl4AI to perform sophisticated web crawling and data extraction tasks. Whether you need to take screenshots, extract specific elements, customize the crawling process, or set custom headers, Crawl4AI provides the flexibility and power to meet your needs. Happy crawling! 🕷️🚀

docs/md _sync/full_details/crawl_request_parameters.md

@@ -0,0 +1,130 @@
+# Crawl Request Parameters
+
+The `run` function in Crawl4AI is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `run` function, along with their descriptions, possible values, and examples.
+
+## Parameters
+
+### url (str)
+**Description:** The URL of the webpage to crawl.
+**Required:** Yes
+**Example:**
+```python
+url = "https://www.nbcnews.com/business"
+```
+
+### word_count_threshold (int)
+**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is `5`.
+**Required:** No
+**Default Value:** `5`
+**Example:**
+```python
+word_count_threshold = 10
+```
+
+### extraction_strategy (ExtractionStrategy)
+**Description:** The strategy to use for extracting content from the HTML. It must be an instance of `ExtractionStrategy`. If not provided, the default is `NoExtractionStrategy`.
+**Required:** No
+**Default Value:** `NoExtractionStrategy()`
+**Example:**
+```python
+extraction_strategy = CosineStrategy(semantic_filter="finance")
+```
+
+### chunking_strategy (ChunkingStrategy)
+**Description:** The strategy to use for chunking the text before processing. It must be an instance of `ChunkingStrategy`. The default value is `RegexChunking()`.
+**Required:** No
+**Default Value:** `RegexChunking()`
+**Example:**
+```python
+chunking_strategy = NlpSentenceChunking()
+```
+
+### bypass_cache (bool)
+**Description:** Whether to force a fresh crawl even if the URL has been previously crawled. The default value is `False`.
+**Required:** No
+**Default Value:** `False`
+**Example:**
+```python
+bypass_cache = True
+```
+
+### css_selector (str)
+**Description:** The CSS selector to target specific parts of the HTML for extraction. If not provided, the entire HTML will be processed.
+**Required:** No
+**Default Value:** `None`
+**Example:**
+```python
+css_selector = "div.article-content"
+```
+
+### screenshot (bool)
+**Description:** Whether to take screenshots of the page. The default value is `False`.
+**Required:** No
+**Default Value:** `False`
+**Example:**
+```python
+screenshot = True
+```
+
+### user_agent (str)
+**Description:** The user agent to use for the HTTP requests. If not provided, a default user agent will be used.
+**Required:** No
+**Default Value:** `None`
+**Example:**
+```python
+user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
+```
+
+### verbose (bool)
+**Description:** Whether to enable verbose logging. The default value is `True`.
+**Required:** No
+**Default Value:** `True`
+**Example:**
+```python
+verbose = True
+```
+
+### **kwargs
+Additional keyword arguments that can be passed to customize the crawling process further. Some notable options include:
+
+- **only_text (bool):** Whether to extract only text content, excluding HTML tags. Default is `False`.
+
+**Example:**
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    css_selector="p",
+    only_text=True
+)
+```
+
+## Example Usage
+
+Here's an example of how to use the `run` function with various parameters:
+
+```python
+from crawl4ai import WebCrawler
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+# Create the WebCrawler instance 
+crawler = WebCrawler() 
+
+# Run the crawler with custom parameters
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    word_count_threshold=10,
+    extraction_strategy=CosineStrategy(semantic_filter="finance"),
+    chunking_strategy=NlpSentenceChunking(),
+    bypass_cache=True,
+    css_selector="div.article-content",
+    screenshot=True,
+    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
+    verbose=True,
+    only_text=True
+)
+
+print(result)
+```
+
+This example demonstrates how to configure various parameters to customize the crawling and extraction process using Crawl4AI.

docs/md _sync/full_details/crawl_result_class.md

@@ -0,0 +1,120 @@
+# Crawl Result
+
+The `CrawlResult` class is the heart of Crawl4AI's output, encapsulating all the data extracted from a crawling session. This class contains various fields that store the results of the web crawling and extraction process. Let's break down each field and see what it holds. 🎉
+
+## Class Definition
+
+```python
+class CrawlResult(BaseModel):
+    url: str
+    html: str
+    success: bool
+    cleaned_html: Optional[str] = None
+    media: Dict[str, List[Dict]] = {}
+    links: Dict[str, List[Dict]] = {}
+    screenshot: Optional[str] = None
+    markdown: Optional[str] = None
+    extracted_content: Optional[str] = None
+    metadata: Optional[dict] = None
+    error_message: Optional[str] = None
+```
+
+## Fields Explanation
+
+### `url: str`
+The URL that was crawled. This field simply stores the URL of the web page that was processed.
+
+### `html: str`
+The raw HTML content of the web page. This is the unprocessed HTML source as retrieved by the crawler.
+
+### `success: bool`
+A flag indicating whether the crawling and extraction were successful. If any error occurs during the process, this will be `False`.
+
+### `cleaned_html: Optional[str]`
+The cleaned HTML content of the web page. This field holds the HTML after removing unwanted tags like `<script>`, `<style>`, and others that do not contribute to the useful content.
+
+### `media: Dict[str, List[Dict]]`
+A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here’s how they are structured:
+
+- **Images**: Each image is represented as a dictionary with `src` (source URL) and `alt` (alternate text).
+- **Videos**: Each video is represented similarly with `src` and `alt`.
+- **Audios**: Each audio is represented with `src` and `alt`.
+
+```python
+media = {
+    'images': [
+        {'src': 'image_url1', 'alt': 'description1', "type": "image"},
+        {'src': 'image_url2', 'alt': 'description2', "type": "image"}
+    ],
+    'videos': [
+        {'src': 'video_url1', 'alt': 'description1', "type": "video"}
+    ],
+    'audios': [
+        {'src': 'audio_url1', 'alt': 'description1', "type": "audio"}
+    ]
+}
+```
+
+### `links: Dict[str, List[Dict]]`
+A dictionary containing lists of internal and external links extracted from the web page. Each link is represented as a dictionary with `href` (URL) and `text` (link text).
+
+- **Internal Links**: Links pointing to the same domain.
+- **External Links**: Links pointing to different domains.
+
+```python
+links = {
+    'internal': [
+        {'href': 'internal_link1', 'text': 'link_text1'},
+        {'href': 'internal_link2', 'text': 'link_text2'}
+    ],
+    'external': [
+        {'href': 'external_link1', 'text': 'link_text1'}
+    ]
+}
+```
+
+### `screenshot: Optional[str]`
+A base64-encoded screenshot of the web page. This field stores the screenshot data if the crawling was configured to take a screenshot.
+
+### `markdown: Optional[str]`
+The content of the web page converted to Markdown format. This is useful for generating clean, readable text that retains the structure of the original HTML.
+
+### `extracted_content: Optional[str]`
+The content extracted based on the specified extraction strategy. This field holds the meaningful content blocks extracted from the web page, ready for your AI and data processing needs.
+
+### `metadata: Optional[dict]`
+A dictionary containing metadata extracted from the web page, such as title, description, keywords, and other meta tags.
+
+### `error_message: Optional[str]`
+If an error occurs during crawling, this field will contain the error message, helping you debug and understand what went wrong. 🚨
+
+## Example Usage
+
+Here's a quick example to illustrate how you might use the `CrawlResult` in your code:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+
+# Run the crawler on a URL
+result = crawler.run(url="https://www.example.com")
+
+# Check if the crawl was successful
+if result.success:
+    print("Crawl succeeded!")
+    print("URL:", result.url)
+    print("HTML:", result.html[:100])  # Print the first 100 characters of the HTML
+    print("Cleaned HTML:", result.cleaned_html[:100])
+    print("Media:", result.media)
+    print("Links:", result.links)
+    print("Screenshot:", result.screenshot)
+    print("Markdown:", result.markdown[:100])
+    print("Extracted Content:", result.extracted_content)
+    print("Metadata:", result.metadata)
+else:
+    print("Crawl failed with error:", result.error_message)
+```
+
+With this setup, you can easily access all the valuable data extracted from the web page and integrate it into your applications. Happy crawling! 🕷️🤖

docs/md _sync/full_details/extraction_strategies.md

@@ -0,0 +1,116 @@
+## Extraction Strategies 🧠
+
+Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into two of the most important strategies: `CosineStrategy` and `LLMExtractionStrategy`.
+
+### CosineStrategy
+
+`CosineStrategy` uses hierarchical clustering based on cosine similarity to group text chunks into meaningful clusters. This method converts each chunk into its embedding and then clusters them to form semantical chunks.
+
+#### When to Use
+- Ideal for fast, accurate semantic segmentation of text.
+- Perfect for scenarios where LLMs might be overkill or too slow.
+- Suitable for narrowing down content based on specific queries or keywords.
+
+#### Parameters
+- `semantic_filter` (str, optional): Keywords for filtering relevant documents before clustering. Documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.
+- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.
+- `max_dist` (float, optional): Maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.
+- `linkage_method` (str, optional): Linkage method for hierarchical clustering. Default is `'ward'`.
+- `top_k` (int, optional): Number of top categories to extract. Default is `3`.
+- `model_name` (str, optional): Model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.
+
+#### Example
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+# Define extraction strategy
+strategy = CosineStrategy(
+    semantic_filter="finance economy stock market",
+    word_count_threshold=10,
+    max_dist=0.2,
+    linkage_method='ward',
+    top_k=3,
+    model_name='BAAI/bge-small-en-v1.5'
+)
+
+# Sample URL
+url = "https://www.nbcnews.com/business"
+
+# Run the crawler with the extraction strategy
+result = crawler.run(url=url, extraction_strategy=strategy)
+print(result.extracted_content)
+```
+
+### LLMExtractionStrategy
+
+`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
+
+#### When to Use
+- Suitable for complex extraction tasks requiring nuanced understanding.
+- Ideal for scenarios where detailed instructions can guide the extraction process.
+- Perfect for extracting specific types of information or content with precise guidelines.
+
+#### Parameters
+- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
+- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
+- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
+
+#### Example Without Instructions
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+# Define extraction strategy without instructions
+strategy = LLMExtractionStrategy(
+    provider='openai',
+    api_token='your_api_token'
+)
+
+# Sample URL
+url = "https://www.nbcnews.com/business"
+
+# Run the crawler with the extraction strategy
+result = crawler.run(url=url, extraction_strategy=strategy)
+print(result.extracted_content)
+```
+
+#### Example With Instructions
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+# Define extraction strategy with instructions
+strategy = LLMExtractionStrategy(
+    provider='openai',
+    api_token='your_api_token',
+    instruction="Extract only financial news and summarize key points."
+)
+
+# Sample URL
+url = "https://www.nbcnews.com/business"
+
+# Run the crawler with the extraction strategy
+result = crawler.run(url=url, extraction_strategy=strategy)
+print(result.extracted_content)
+```
+
+#### Use Cases for LLMExtractionStrategy
+- Extracting specific data types from structured or semi-structured content.
+- Generating summaries, extracting key information, or transforming content into different formats.
+- Performing detailed extractions based on custom instructions.
+
+For more detailed examples, please refer to the [Examples section](../examples/index.md) of the documentation.
+
+---
+
+By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy` or nuanced, instruction-based extraction with `LLMExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨

docs/md _sync/index.md

@@ -0,0 +1,101 @@
+# Crawl4AI v0.2.77
+
+Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
+
+
+## Try the [Demo](demo.md)
+
+Just try it now and crawl different pages to see how it works. You can set the links, see the structures of the output, and also view the Python sample code on how to run it. The old demo is available at [/old_demo](/old) where you can see more details.
+
+## Introduction
+
+Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution.
+
+## Quick Start
+
+Here's a quick example to show you how easy it is to use Crawl4AI:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create an instance of WebCrawler
+crawler = WebCrawler()
+
+# Warm up the crawler (load necessary models)
+crawler.warmup()
+
+# Run the crawler on a URL
+result = crawler.run(url="https://www.nbcnews.com/business")
+
+# Print the extracted content
+print(result.extracted_content)
+```
+
+### Explanation
+
+1. **Importing the Library**: We start by importing the `WebCrawler` class from the `crawl4ai` library.
+2. **Creating an Instance**: An instance of `WebCrawler` is created.
+3. **Warming Up**: The `warmup()` method prepares the crawler by loading necessary models and settings.
+4. **Running the Crawler**: The `run()` method is used to crawl the specified URL and extract meaningful content.
+5. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
+
+## Documentation Structure
+
+This documentation is organized into several sections to help you navigate and find the information you need quickly:
+
+### [Home](index.md)
+
+An introduction to Crawl4AI, including a quick start guide and an overview of the documentation structure.
+
+### [Installation](installation.md)
+
+Instructions on how to install Crawl4AI and its dependencies.
+
+### [Introduction](introduction.md)
+
+A detailed introduction to Crawl4AI, its features, and how it can be used for various web crawling and data extraction tasks.
+
+### [Quick Start](quickstart.md)
+
+A step-by-step guide to get you up and running with Crawl4AI, including installation instructions and basic usage examples.
+
+### [Examples](examples/index.md)
+
+This section contains practical examples demonstrating different use cases of Crawl4AI:
+
+- [LLM Extraction](examples/llm_extraction.md)
+- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
+- [Hooks & Auth](examples/hooks_auth.md)
+- [Summarization](examples/summarization.md)
+- [Research Assistant](examples/research_assistant.md)
+
+### [Full Details of Using Crawler](full_details/crawl_request_parameters.md)
+
+Comprehensive details on using the crawler, including:
+
+- [Crawl Request Parameters](full_details/crawl_request_parameters.md)
+- [Crawl Result Class](full_details/crawl_result_class.md)
+- [Advanced Features](full_details/advanced_features.md)
+- [Chunking Strategies](full_details/chunking_strategies.md)
+- [Extraction Strategies](full_details/extraction_strategies.md)
+
+### [API Reference](api/core_classes_and_functions.md)
+
+Detailed documentation of the API, covering:
+
+- [Core Classes and Functions](api/core_classes_and_functions.md)
+- [Detailed API Documentation](api/detailed_api_documentation.md)
+
+### [Change Log](changelog.md)
+
+A log of all changes, updates, and improvements made to Crawl4AI.
+
+### [Contact](contact.md)
+
+Information on how to get in touch with the developers, report issues, and contribute to the project.
+
+## Get Started
+
+To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier and more efficient.
+
+Happy Crawling! 🕸️🚀

docs/md _sync/installation.md

@@ -0,0 +1,193 @@
+# Installation 💻
+
+There are three ways to use Crawl4AI:
+
+1. As a library (Recommended).
+2. As a local server (Docker) or using the REST API.
+3. As a local server (Docker) using the pre-built image from Docker Hub.
+
+## Option 1: Library Installation
+
+You can try this Colab for a quick start: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sJPAmeLj5PMrg2VgOwMJ2ubGIcK0cJeX#scrollTo=g1RrmI4W_rPk)
+
+Crawl4AI offers flexible installation options to suit various use cases. Choose the option that best fits your needs:
+
+- **Default Installation** (Basic functionality):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai @ git+https://github.com/unclecode/crawl4ai.git"
+```
+Use this for basic web crawling and scraping tasks.
+
+- **Installation with PyTorch** (For advanced text clustering):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai[torch] @ git+https://github.com/unclecode/crawl4ai.git"
+```
+Choose this if you need the CosineSimilarity cluster strategy.
+
+- **Installation with Transformers** (For summarization and Hugging Face models):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai[transformer] @ git+https://github.com/unclecode/crawl4ai.git"
+```
+Opt for this if you require text summarization or plan to use Hugging Face models.
+
+- **Full Installation** (All features):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai[all] @ git+https://github.com/unclecode/crawl4ai.git"
+```
+This installs all dependencies for full functionality.
+
+- **Development Installation** (For contributors):
+```bash
+virtualenv venv
+source venv/bin/activate
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+pip install -e ".[all]"
+```
+Use this if you plan to modify the source code.
+
+💡 After installation, if you have used "torch", "transformer" or "all", it's recommended to run the following CLI command to load the required models. This is optional but will boost the performance and speed of the crawler. You need to do this only once, this is only for when you install using []
+```bash
+crawl4ai-download-models
+```
+
+## Option 2: Using Docker for Local Server
+
+Crawl4AI can be run as a local server using Docker. The Dockerfile supports different installation options to cater to various use cases. Here's how you can build and run the Docker image:
+
+### Default Installation
+
+The default installation includes the basic Crawl4AI package without additional dependencies or pre-downloaded models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 -t crawl4ai .
+
+# For other users
+docker build -t crawl4ai .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai
+```
+
+### Full Installation (All Dependencies and Models)
+
+This option installs all dependencies and downloads the models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 --build-arg INSTALL_OPTION=all -t crawl4ai:all .
+
+# For other users
+docker build --build-arg INSTALL_OPTION=all -t crawl4ai:all .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai:all
+```
+
+### Torch Installation
+
+This option installs torch-related dependencies and downloads the models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
+
+# For other users
+docker build --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai:torch
+```
+
+### Transformer Installation
+
+This option installs transformer-related dependencies and downloads the models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
+
+# For other users
+docker build --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai:transformer
+```
+
+### Notes
+
+- The `--platform linux/amd64` flag is necessary for Mac users with M1/M2 chips to ensure compatibility.
+- The `-t` flag tags the image with a name (and optionally a tag in the 'name:tag' format).
+- The `-d` flag runs the container in detached mode.
+- The `-p 8000:80` flag maps port 8000 on the host to port 80 in the container.
+
+Choose the installation option that best suits your needs. The default installation is suitable for basic usage, while the other options provide additional capabilities for more advanced use cases.
+
+## Option 3: Using the Pre-built Image from Docker Hub
+
+You can use pre-built Crawl4AI images from Docker Hub, which are available for all platforms (Mac, Linux, Windows). We have official images as well as a community-contributed image (Thanks to https://github.com/FractalMind):
+
+### Default Installation
+
+```bash
+
+# Pull the image
+
+docker pull unclecode/crawl4ai:latest
+
+# Run the container
+
+docker run -d -p 8000:80 unclecode/crawl4ai:latest
+
+```
+
+### Community-Contributed Image
+
+A stable version of Crawl4AI is also available, created and maintained by a community member:
+
+```bash
+
+# Pull the community-contributed image
+
+docker pull ryser007/crawl4ai:stable
+
+# Run the container
+
+docker run -d -p 8000:80 ryser007/crawl4ai:stable
+
+```
+
+We'd like to express our gratitude to GitHub user [@FractalMind](https://github.com/FractalMind) for creating and maintaining this stable version of the Crawl4AI Docker image. Community contributions like this are invaluable to the project.
+
+
+### Testing the Installation
+
+After running the container, you can test if it's working correctly:
+
+- On Mac and Linux:
+
+  ```bash
+
+  curl http://localhost:8000
+
+  ```
+
+- On Windows (PowerShell):
+
+  ```powershell
+
+  Invoke-WebRequest -Uri http://localhost:8000
+
+  ```
+
+  Or open a web browser and navigate to http://localhost:8000
+

docs/md _sync/interactive_content.html

@@ -0,0 +1,28 @@
+<h1>Try Our Library</h1>
+<form id="apiForm">
+    <label for="inputField">Enter some input:</label>
+    <input type="text" id="inputField" name="inputField" required>
+    <button type="submit">Submit</button>
+</form>
+<div id="result"></div>
+
+<script>
+    document.getElementById('apiForm').addEventListener('submit', function(event) {
+        event.preventDefault();
+        const input = document.getElementById('inputField').value;
+        fetch('https://your-api-endpoint.com/api', {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json'
+            },
+            body: JSON.stringify({ input: input })
+        })
+        .then(response => response.json())
+        .then(data => {
+            document.getElementById('result').textContent = JSON.stringify(data);
+        })
+        .catch(error => {
+            document.getElementById('result').textContent = 'Error: ' + error;
+        });
+    });
+</script>

docs/md _sync/introduction.md

@@ -0,0 +1,29 @@
+# Introduction
+
+Welcome to the documentation for Crawl4AI v0.2.5! 🕷️🤖
+
+Crawl4AI is designed to simplify the process of crawling web pages and extracting useful information for large language models (LLMs) and AI applications. Whether you're using it as a REST API, a Python library, or through a Google Colab notebook, Crawl4AI provides powerful features to make web data extraction easier and more efficient.
+
+## Key Features ✨
+
+- **🆓 Completely Free and Open-Source**: Crawl4AI is free to use and open-source, making it accessible for everyone.
+- **🤖 LLM-Friendly Output Formats**: Supports JSON, cleaned HTML, and markdown formats.
+- **🌍 Concurrent Crawling**: Crawl multiple URLs simultaneously to save time.
+- **🎨 Media Extraction**: Extract all media tags including images, audio, and video.
+- **🔗 Link Extraction**: Extract all external and internal links from web pages.
+- **📚 Metadata Extraction**: Extract metadata from web pages for additional context.
+- **🔄 Custom Hooks**: Define custom hooks for authentication, headers, and page modifications before crawling.
+- **🕵️ User Agent Support**: Customize the user agent for HTTP requests.
+- **🖼️ Screenshot Capability**: Take screenshots of web pages during crawling.
+- **📜 JavaScript Execution**: Execute custom JavaScripts before crawling.
+- **📚 Advanced Chunking and Extraction Strategies**: Utilize topic-based, regex, sentence chunking, cosine clustering, and LLM extraction strategies.
+- **🎯 CSS Selector Support**: Extract specific content using CSS selectors.
+- **📝 Instruction/Keyword Refinement**: Pass instructions or keywords to refine the extraction process.
+
+Check the [Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for more details.
+
+## Power and Simplicity of Crawl4AI 🚀
+
+Crawl4AI provides an easy way to crawl and extract data from web pages without installing any library. You can use the REST API on our server or run the local server on your machine. For more advanced control, use the Python library to customize your crawling and extraction strategies.
+
+Explore the documentation to learn more about the features, installation process, usage examples, and how to contribute to Crawl4AI. Let's make the web more accessible and useful for AI applications! 💪🌐🤖

docs/md _sync/quickstart.md

@@ -0,0 +1,204 @@
+# Quick Start Guide 🚀
+
+Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies. Let's dive in! 🌟
+
+## Getting Started 🛠️
+
+First, let's create an instance of `WebCrawler` and call the `warmup()` function. This might take a few seconds the first time you run Crawl4AI, as it loads the required model files.
+
+```python
+from crawl4ai import WebCrawler
+
+def create_crawler():
+    crawler = WebCrawler(verbose=True)
+    crawler.warmup()
+    return crawler
+
+crawler = create_crawler()
+```
+
+### Basic Usage
+
+Simply provide a URL and let Crawl4AI do the magic!
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(f"Basic crawl result: {result}")
+```
+
+### Taking Screenshots 📸
+
+Let's take a screenshot of the page!
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
+with open("screenshot.png", "wb") as f:
+    f.write(base64.b64decode(result.screenshot))
+print("Screenshot saved to 'screenshot.png'!")
+```
+
+### Understanding Parameters 🧠
+
+By default, Crawl4AI caches the results of your crawls. This means that subsequent crawls of the same URL will be much faster! Let's see this in action.
+
+First crawl (caches the result):
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(f"First crawl result: {result}")
+```
+
+Force to crawl again:
+```python
+result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
+print(f"Second crawl result: {result}")
+```
+
+### Adding a Chunking Strategy 🧩
+
+Let's add a chunking strategy: `RegexChunking`! This strategy splits the text based on a given regex pattern.
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    chunking_strategy=RegexChunking(patterns=["\n\n"])
+)
+print(f"RegexChunking result: {result}")
+```
+
+You can also use `NlpSentenceChunking` which splits the text into sentences using NLP techniques.
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    chunking_strategy=NlpSentenceChunking()
+)
+print(f"NlpSentenceChunking result: {result}")
+```
+
+### Adding an Extraction Strategy 🧠
+
+Let's get smarter with an extraction strategy: `CosineStrategy`! This strategy uses cosine similarity to extract semantically similar blocks of text.
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=CosineStrategy(
+        word_count_threshold=10, 
+        max_dist=0.2, 
+        linkage_method="ward", 
+        top_k=3
+    )
+)
+print(f"CosineStrategy result: {result}")
+```
+
+You can also pass other parameters like `semantic_filter` to extract specific content.
+
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=CosineStrategy(
+        semantic_filter="inflation rent prices"
+    )
+)
+print(f"CosineStrategy result with semantic filter: {result}")
+```
+
+### Using LLMExtractionStrategy 🤖
+
+Time to bring in the big guns: `LLMExtractionStrategy` without instructions! This strategy uses a large language model to extract relevant information from the web page.
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+import os
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=LLMExtractionStrategy(
+        provider="openai/gpt-4o", 
+        api_token=os.getenv('OPENAI_API_KEY')
+    )
+)
+print(f"LLMExtractionStrategy (no instructions) result: {result}")
+```
+
+You can also provide specific instructions to guide the extraction.
+
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=LLMExtractionStrategy(
+        provider="openai/gpt-4o",
+        api_token=os.getenv('OPENAI_API_KEY'),
+        instruction="I am interested in only financial news"
+    )
+)
+print(f"LLMExtractionStrategy (with instructions) result: {result}")
+```
+
+### Targeted Extraction 🎯
+
+Let's use a CSS selector to extract only H2 tags!
+
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    css_selector="h2"
+)
+print(f"CSS Selector (H2 tags) result: {result}")
+```
+
+### Interactive Extraction 🖱️
+
+Passing JavaScript code to click the 'Load More' button!
+
+```python
+js_code = """
+const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+loadMoreButton && loadMoreButton.click();
+"""
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    js=js_code
+)
+print(f"JavaScript Code (Load More button) result: {result}")
+```
+
+### Using Crawler Hooks 🔗
+
+Let's see how we can customize the crawler using hooks!
+
+```python
+import time
+
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.crawler_strategy import *
+
+def delay(driver):
+    print("Delaying for 5 seconds...")
+    time.sleep(5)
+    print("Resuming...")
+    
+def create_crawler():
+    crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
+    crawler_strategy.set_hook('after_get_url', delay)
+    crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
+    crawler.warmup()
+    return crawler
+
+crawler = create_crawler()
+result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
+```
+
+check [Hooks](examples/hooks_auth.md) for more examples.
+
+## Congratulations! 🎉
+
+You've made it through the Crawl4AI Quickstart Guide! Now go forth and crawl the web like a pro! 🕸️

docs/md/api/core_classes_and_functions.md

@@ -0,0 +1,141 @@
+# Core Classes and Functions
+
+## Overview
+
+In this section, we will delve into the core classes and functions that make up the Crawl4AI library. This includes the `WebCrawler` class, various `CrawlerStrategy` classes, `ChunkingStrategy` classes, and `ExtractionStrategy` classes. Understanding these core components will help you leverage the full power of Crawl4AI for your web crawling and data extraction needs.
+
+## WebCrawler Class
+
+The `WebCrawler` class is the main class you'll interact with. It provides the interface for crawling web pages and extracting data.
+
+### Initialization
+
+```python
+from crawl4ai import WebCrawler
+
+# Create an instance of WebCrawler
+crawler = WebCrawler()
+```
+
+### Methods
+
+- **`warmup()`**: Prepares the crawler for use, such as loading necessary models.
+- **`run(url: str, **kwargs)`**: Runs the crawler on the specified URL with optional parameters for customization.
+
+```python
+crawler.warmup()
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(result)
+```
+
+## CrawlerStrategy Classes
+
+The `CrawlerStrategy` classes define how the web crawling is executed. The base class is `CrawlerStrategy`, which is extended by specific implementations like `LocalSeleniumCrawlerStrategy`.
+
+### CrawlerStrategy Base Class
+
+An abstract base class that defines the interface for different crawler strategies.
+
+```python
+from abc import ABC, abstractmethod
+
+class CrawlerStrategy(ABC):
+    @abstractmethod
+    def crawl(self, url: str, **kwargs) -> str:
+        pass
+    
+    @abstractmethod
+    def take_screenshot(self, save_path: str):
+        pass
+    
+    @abstractmethod
+    def update_user_agent(self, user_agent: str):
+        pass
+    
+    @abstractmethod
+    def set_hook(self, hook_type: str, hook: Callable):
+        pass
+```
+
+### LocalSeleniumCrawlerStrategy Class
+
+A concrete implementation of `CrawlerStrategy` that uses Selenium to crawl web pages.
+
+#### Initialization
+
+```python
+from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
+
+strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
+```
+
+#### Methods
+
+- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+```python
+result = strategy.crawl("https://www.example.com")
+strategy.take_screenshot("screenshot.png")
+strategy.update_user_agent("Mozilla/5.0")
+strategy.set_hook("before_get_url", lambda: print("About to get URL"))
+```
+
+## ChunkingStrategy Classes
+
+The `ChunkingStrategy` classes define how the text from a web page is divided into chunks. Here are a few examples:
+
+### RegexChunking Class
+
+Splits text using regular expressions.
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+chunker = RegexChunking(patterns=[r'\n\n'])
+chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
+```
+
+### NlpSentenceChunking Class
+
+Uses NLP to split text into sentences.
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
+```
+
+## ExtractionStrategy Classes
+
+The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks. Here are a few examples:
+
+### CosineStrategy Class
+
+Clusters text chunks based on cosine similarity.
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### LLMExtractionStrategy Class
+
+Uses a Language Model to extract meaningful blocks from HTML.
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+## Conclusion
+
+By understanding these core classes and functions, you can customize and extend Crawl4AI to suit your specific web crawling and data extraction needs. Happy crawling! 🕷️🤖
+

docs/md/api/detailed_api_documentation.md

@@ -0,0 +1,338 @@
+# Detailed API Documentation
+
+## Overview
+
+This section provides comprehensive documentation for the Crawl4AI API, covering all classes, methods, and their parameters. This guide will help you understand how to utilize the API to its full potential, enabling efficient web crawling and data extraction.
+
+## WebCrawler Class
+
+The `WebCrawler` class is the primary interface for crawling web pages and extracting data.
+
+### Initialization
+
+```python
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+```
+
+### Methods
+
+#### `warmup()`
+
+Prepares the crawler for use, such as loading necessary models.
+
+```python
+crawler.warmup()
+```
+
+#### `run(url: str, **kwargs) -> CrawlResult`
+
+Crawls the specified URL and returns the result.
+
+- **Parameters:**
+  - `url` (str): The URL to crawl.
+  - `**kwargs`: Additional parameters for customization.
+
+- **Returns:**
+  - `CrawlResult`: An object containing the crawl result.
+
+- **Example:**
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(result)
+```
+
+### CrawlResult Class
+
+Represents the result of a crawl operation.
+
+- **Attributes:**
+  - `url` (str): The URL of the crawled page.
+  - `html` (str): The raw HTML of the page.
+  - `success` (bool): Whether the crawl was successful.
+  - `cleaned_html` (Optional[str]): The cleaned HTML.
+  - `media` (Dict[str, List[Dict]]): Media tags in the page (images, audio, video).
+  - `links` (Dict[str, List[Dict]]): Links in the page (external, internal).
+  - `screenshot` (Optional[str]): Base64 encoded screenshot.
+  - `markdown` (Optional[str]): Extracted content in Markdown format.
+  - `extracted_content` (Optional[str]): Extracted meaningful content.
+  - `metadata` (Optional[dict]): Metadata from the page.
+  - `error_message` (Optional[str]): Error message if any.
+
+## CrawlerStrategy Classes
+
+The `CrawlerStrategy` classes define how the web crawling is executed.
+
+### CrawlerStrategy Base Class
+
+An abstract base class for different crawler strategies.
+
+#### Methods
+
+- **`crawl(url: str, **kwargs) -> str`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+### LocalSeleniumCrawlerStrategy Class
+
+Uses Selenium to crawl web pages.
+
+#### Initialization
+
+```python
+from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
+
+strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
+```
+
+#### Methods
+
+- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+#### Example
+
+```python
+result = strategy.crawl("https://www.example.com")
+strategy.take_screenshot("screenshot.png")
+strategy.update_user_agent("Mozilla/5.0")
+strategy.set_hook("before_get_url", lambda: print("About to get URL"))
+```
+
+## ChunkingStrategy Classes
+
+The `ChunkingStrategy` classes define how the text from a web page is divided into chunks.
+
+### RegexChunking Class
+
+Splits text using regular expressions.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+chunker = RegexChunking(patterns=[r'\n\n'])
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into chunks.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
+```
+
+### NlpSentenceChunking Class
+
+Uses NLP to split text into sentences.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into sentences.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
+```
+
+### TopicSegmentationChunking Class
+
+Uses the TextTiling algorithm to segment text into topics.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import TopicSegmentationChunking
+
+chunker = TopicSegmentationChunking(num_keywords=3)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into topic-based segments.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into topic-based segments.")
+```
+
+### FixedLengthWordChunking Class
+
+Splits text into chunks of fixed length based on the number of words.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import FixedLengthWordChunking
+
+chunker = FixedLengthWordChunking(chunk_size=100)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into fixed-length word chunks.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into fixed-length word chunks.")
+```
+
+### SlidingWindowChunking Class
+
+Uses a sliding window approach to chunk text.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import SlidingWindowChunking
+
+chunker = SlidingWindowChunking(window_size=100, step=50)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text using a sliding window approach.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split using a sliding window approach.")
+```
+
+## ExtractionStrategy Classes
+
+The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks.
+
+### NoExtractionStrategy Class
+
+Returns the entire HTML content without any modification.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import NoExtractionStrategy
+
+extractor = NoExtractionStrategy()
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Returns the HTML content.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### LLMExtractionStrategy Class
+
+Uses a Language Model to extract meaningful blocks from HTML.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts meaningful content using the LLM.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### CosineStrategy Class
+
+Clusters text chunks based on cosine similarity.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts clusters of text based on cosine similarity.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### TopicExtractionStrategy Class
+
+Uses the TextTiling algorithm to segment HTML content into topics and extract keywords.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import TopicExtractionStrategy
+
+extractor = TopicExtractionStrategy(num_keywords=3)
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts topic-based segments and keywords.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+## Parameters
+
+Here are the common parameters used across various classes and methods:
+
+- **`url`** (str): The URL to crawl.
+- **`html`** (str): The HTML content of the page.
+- **`user_agent`** (str): The user agent for the HTTP requests.
+- **`patterns`** (list): A list of regular expression patterns for chunking.
+- **`num_keywords`** (int): Number of keywords for topic extraction.
+- **`chunk_size`** (int): Number of words in each chunk.
+- **`window_size`** (int): Number of words in the sliding window.
+- **`step`** (int): Step size for the sliding window.
+- **`semantic_filter`** (str): Keywords for filtering relevant documents.
+- **`word_count_threshold`** (int): Minimum number of words per cluster.
+- **`max_dist`** (float): Maximum cophenetic distance for clustering.
+- **`linkage_method`** (str): Linkage method for hierarchical clustering.
+- **`top_k`** (int): Number of top categories to extract.
+- **`provider`** (
+
+str): Provider for language model completions.
+- **`api_token`** (str): API token for the provider.
+- **`instruction`** (str): Instruction to guide the LLM extraction.
+
+## Conclusion
+
+This detailed API documentation provides a thorough understanding of the classes, methods, and parameters in the Crawl4AI library. With this knowledge, you can effectively use the API to perform advanced web crawling and data extraction tasks.

docs/md/assets/highlight_init.js

@@ -0,0 +1,6 @@
+document.addEventListener('DOMContentLoaded', (event) => {
+    document.querySelectorAll('pre code').forEach((block) => {
+      hljs.highlightBlock(block);
+    });
+  });
+  

docs/md/assets/styles.css

@@ -0,0 +1,153 @@
+@font-face {
+    font-family: "Monaco";
+    font-style: normal;
+    font-weight: normal;
+    src: local("Monaco"), url("Monaco.woff") format("woff");
+}
+
+:root {
+    --global-font-size: 16px;
+    --global-line-height: 1.5em;
+    --global-space: 10px;
+    --font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
+        Courier New, monospace, serif;
+    --font-stack: dm, Monaco, Courier New, monospace, serif;
+    --mono-font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
+        Courier New, monospace, serif;
+
+    --background-color: #151515; /* Dark background */
+    --font-color: #eaeaea; /* Light font color for contrast */
+    --invert-font-color: #151515; /* Dark color for inverted elements */
+    --primary-color: #1a95e0; /* Primary color can remain the same or be adjusted for better contrast */
+    --secondary-color: #727578; /* Secondary color for less important text */
+    --error-color: #ff5555; /* Bright color for errors */
+    --progress-bar-background: #444; /* Darker background for progress bar */
+    --progress-bar-fill: #1a95e0; /* Bright color for progress bar fill */
+    --code-bg-color: #1e1e1e; /* Darker background for code blocks */
+    --input-style: solid; /* Keeping input style solid */
+    --block-background-color: #202020; /* Darker background for block elements */
+    --global-font-color: #eaeaea; /* Light font color for global elements */
+
+    --background-color: #222225;
+
+    --background-color: #070708;
+    --page-width: 70em;
+    --font-color: #e8e9ed;
+    --invert-font-color: #222225;
+    --secondary-color: #a3abba;
+    --secondary-color: #d5cec0;
+    --tertiary-color: #a3abba;
+    --primary-color: #09b5a5; /* Updated to the brand color */
+    --primary-color: #50ffff; /* Updated to the brand color */
+    --error-color: #ff3c74;
+    --progress-bar-background: #3f3f44;
+    --progress-bar-fill: #09b5a5; /* Updated to the brand color */
+    --code-bg-color: #3f3f44;
+    --input-style: solid;
+    --display-h1-decoration: none;
+
+    --display-h1-decoration: none;
+}
+
+/* body {
+    background-color: var(--background-color);
+    color: var(--font-color);
+}
+
+a {
+    color: var(--primary-color);
+}
+
+a:hover {
+    background-color: var(--primary-color);
+    color: var(--invert-font-color);
+}
+
+blockquote::after {
+    color: #444; 
+}
+
+pre, code {
+    background-color: var(--code-bg-color);
+    color: var(--font-color);
+}
+
+.terminal-nav:first-child {
+    border-bottom: 1px dashed var(--secondary-color);
+} */
+
+.terminal-mkdocs-main-content {
+    line-height: var(--global-line-height);
+}
+
+strong,
+.highlight {
+    /* background: url(//s2.svgbox.net/pen-brushes.svg?ic=brush-1&color=50ffff); */
+    background-color: #50ffff33;
+}
+
+.terminal-card > header {
+    color: var(--font-color);
+    text-align: center;
+    background-color: var(--progress-bar-background);
+    padding: 0.3em 0.5em;
+}
+.btn.btn-sm {
+    color: var(--font-color);
+    padding: 0.2em 0.5em;
+    font-size: 0.8em;
+}
+
+.loading-message {
+    display: none;
+    margin-top: 20px;
+}
+
+.response-section {
+    display: none;
+    padding-top: 20px;
+}
+
+.tabs {
+    display: flex;
+    flex-direction: column;
+}
+.tab-list {
+    display: flex;
+    padding: 0;
+    margin: 0;
+    list-style-type: none;
+    border-bottom: 1px solid var(--font-color);
+}
+.tab-item {
+    cursor: pointer;
+    padding: 10px;
+    border: 1px solid var(--font-color);
+    margin-right: -1px;
+    border-bottom: none;
+}
+.tab-item:hover,
+.tab-item:focus,
+.tab-item:active {
+    background-color: var(--progress-bar-background);
+}
+.tab-content {
+    display: none;
+    border: 1px solid var(--font-color);
+    border-top: none;
+}
+.tab-content:first-of-type {
+    display: block;
+}
+
+.tab-content header {
+    padding: 0.5em;
+    display: flex; 
+    justify-content: end; 
+    align-items: center;
+    background-color: var(--progress-bar-background);
+}
+.tab-content pre {
+    margin: 0;
+    max-height: 300px; overflow: auto; border:none;
+}

docs/md/changelog.md

@@ -0,0 +1,102 @@
+# Changelog
+
+## [v0.2.77] - 2024-08-04
+
+Significant improvements in text processing and performance:
+
+- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
+- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
+- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
+- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
+
+These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
+
+## [v0.2.76] - 2024-08-02
+
+Major improvements in functionality, performance, and cross-platform compatibility! 🚀
+
+- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
+- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
+- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
+- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
+- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
+
+A big shoutout to our amazing community contributors:
+- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
+- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
+- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
+
+Your contributions are driving Crawl4AI forward! 🙌
+
+## [v0.2.75] - 2024-07-19
+
+Minor improvements for a more maintainable codebase:
+
+- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
+- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
+
+These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
+
+
+## v0.2.74 - 2024-07-08
+A slew of exciting updates to improve the crawler's stability and robustness! 🎉
+
+- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
+- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
+- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
+- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
+
+## [v0.2.73] - 2024-07-03
+
+💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
+
+* Supporting website need "with-head" mode to crawl the website with head.
+* Fixing the installation issues for setup.py and dockerfile.
+* Resolve multiple issues.
+
+## [v0.2.72] - 2024-06-30
+
+This release brings exciting updates and improvements to our project! 🎉
+
+* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
+* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
+* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
+* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
+
+These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
+
+## [0.2.71] - 2024-06-26
+
+**Improved Error Handling and Performance** 🚧
+
+* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
+* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
+* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
+* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
+
+These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
+
+## [0.2.71] - 2024-06-25
+### Fixed
+- Speed up twice the extraction function.
+
+## [0.2.6] - 2024-06-22
+### Fixed
+- Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
+
+## [0.2.5] - 2024-06-18
+### Added
+- Added five important hooks to the crawler:
+  - on_driver_created: Called when the driver is ready for initializations.
+  - before_get_url: Called right before Selenium fetches the URL.
+  - after_get_url: Called after Selenium fetches the URL.
+  - before_return_html: Called when the data is parsed and ready.
+  - on_user_agent_updated: Called when the user changes the user_agent, causing the driver to reinitialize.
+- Added an example in `quickstart.py` in the example folder under the docs.
+- Enhancement issue #24: Replaced inline HTML tags (e.g., DEL, INS, SUB, ABBR) with textual format for better context handling in LLM.
+- Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
+- Updated Dockerfile to ensure compatibility across multiple platforms (Hopefully!).
+
+## [0.2.4] - 2024-06-17
+### Fixed
+- Fix issue #22: Use MD5 hash for caching HTML files to handle long URLs

docs/md/contact.md

@@ -0,0 +1,25 @@
+# Contact
+If you have any questions, suggestions, or feedback, please feel free to reach out to us:
+
+- GitHub: [unclecode](https://github.com/unclecode)
+- Twitter: [@unclecode](https://twitter.com/unclecode)
+- Website: [crawl4ai.com](https://crawl4ai.com)
+
+
+## Contributing 🤝
+
+We welcome contributions from the open-source community to help improve Crawl4AI and make it even more valuable for AI enthusiasts and developers. To contribute, please follow these steps:
+
+1. Fork the repository.
+2. Create a new branch for your feature or bug fix.
+3. Make your changes and commit them with descriptive messages.
+4. Push your changes to your forked repository.
+5. Submit a pull request to the main repository.
+
+For more information on contributing, please see our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md).
+
+## License 📄
+
+Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
+
+Let's work together to make the web more accessible and useful for AI applications! 💪🌐🤖

docs/md/demo.md

@@ -0,0 +1,231 @@
+# Interactive Demo for Crowler
+<div id="demo">
+    <form id="crawlForm" class="terminal-form">
+        <fieldset>
+            <legend>Enter URL and Options</legend>
+            <div class="form-group">
+                <label for="url">Enter URL:</label>
+                <input type="text" id="url" name="url" required>
+            </div>
+            <div class="form-group">
+                <label for="screenshot">Get Screenshot:</label>
+                <input type="checkbox" id="screenshot" name="screenshot">
+            </div>
+            <div class="form-group">
+                <button class="btn btn-default" type="submit">Submit</button>
+            </div>
+
+        </fieldset>
+    </form>
+
+    <div id="loading" class="loading-message">
+        <div class="terminal-alert terminal-alert-primary">Loading... Please wait.</div>
+    </div>
+
+    <section id="response" class="response-section">
+        <h2>Response</h2>
+        <div class="tabs">
+            <ul class="tab-list">
+                <li class="tab-item" onclick="showTab('markdown')">Markdown</li>
+                <li class="tab-item" onclick="showTab('cleanedHtml')">Cleaned HTML</li>
+                <li class="tab-item" onclick="showTab('media')">Media</li>
+                <li class="tab-item" onclick="showTab('extractedContent')">Extracted Content</li>
+                <li class="tab-item" onclick="showTab('screenshot')">Screenshot</li>
+                <li class="tab-item" onclick="showTab('pythonCode')">Python Code</li>
+            </ul>
+            <div class="tab-content" id="tab-markdown">
+                <header>
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('markdownContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('markdownContent', 'markdown.md')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="markdownContent" class="language-markdown hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-cleanedHtml" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('cleanedHtmlContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('cleanedHtmlContent', 'cleaned.html')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="cleanedHtmlContent" class="language-html hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-media" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('mediaContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('mediaContent', 'media.json')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="mediaContent" class="language-json hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-extractedContent" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('extractedContentContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('extractedContentContent', 'extracted_content.json')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="extractedContentContent" class="language-json hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-screenshot" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadImage('screenshotContent', 'screenshot.png')">Download</button>
+                    </div>
+                </header>
+                <pre><img id="screenshotContent" /></pre>
+            </div>
+
+            <div class="tab-content" id="tab-pythonCode" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('pythonCode')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('pythonCode', 'example.py')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="pythonCode" class="language-python hljs"></code></pre>
+            </div>
+        </div>
+    </section>
+
+    <div id="error" class="error-message" style="display: none; margin-top:1em;">
+        <div class="terminal-alert terminal-alert-error"></div>
+    </div>
+
+    <script>
+        function showTab(tabId) {
+            const tabs = document.querySelectorAll('.tab-content');
+            tabs.forEach(tab => tab.style.display = 'none');
+            document.getElementById(`tab-${tabId}`).style.display = 'block';
+        }
+
+        function redo(codeBlock, codeText){
+            codeBlock.classList.remove('hljs');
+            codeBlock.removeAttribute('data-highlighted');
+
+            // Set new code and re-highlight
+            codeBlock.textContent = codeText;
+            hljs.highlightBlock(codeBlock);
+        }
+
+        function copyToClipboard(elementId) {
+            const content = document.getElementById(elementId).textContent;
+            navigator.clipboard.writeText(content).then(() => {
+                alert('Copied to clipboard');
+            });
+        }
+
+        function downloadContent(elementId, filename) {
+            const content = document.getElementById(elementId).textContent;
+            const blob = new Blob([content], { type: 'text/plain' });
+            const url = window.URL.createObjectURL(blob);
+            const a = document.createElement('a');
+            a.style.display = 'none';
+            a.href = url;
+            a.download = filename;
+            document.body.appendChild(a);
+            a.click();
+            window.URL.revokeObjectURL(url);
+            document.body.removeChild(a);
+        }
+
+        function downloadImage(elementId, filename) {
+            const content = document.getElementById(elementId).src;
+            const a = document.createElement('a');
+            a.style.display = 'none';
+            a.href = content;
+            a.download = filename;
+            document.body.appendChild(a);
+            a.click();
+            document.body.removeChild(a);
+        }
+
+        document.getElementById('crawlForm').addEventListener('submit', function(event) {
+            event.preventDefault();
+            document.getElementById('loading').style.display = 'block';
+            document.getElementById('response').style.display = 'none';
+
+            const url = document.getElementById('url').value;
+            const screenshot = document.getElementById('screenshot').checked;
+            const data = {
+                urls: [url],
+                bypass_cache: false,
+                word_count_threshold: 5,
+                screenshot: screenshot
+            };
+
+            fetch('/crawl', {
+                method: 'POST',
+                headers: {
+                    'Content-Type': 'application/json'
+                },
+                body: JSON.stringify(data)
+            })
+            .then(response => {
+                if (!response.ok) {
+                    if (response.status === 429) {
+                        return response.json().then(err => { 
+                            throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
+                        });
+                    }
+                    throw new Error('Network response was not ok');
+                }
+                return response.json();
+            })
+            .then(data => {
+                data = data.results[0]; // Only one URL is requested
+                document.getElementById('loading').style.display = 'none';
+                document.getElementById('response').style.display = 'block';
+                redo(document.getElementById('markdownContent'), data.markdown);
+                redo(document.getElementById('cleanedHtmlContent'), data.cleaned_html);
+                redo(document.getElementById('mediaContent'), JSON.stringify(data.media, null, 2));
+                redo(document.getElementById('extractedContentContent'), data.extracted_content);
+                if (screenshot) {
+                    document.getElementById('screenshotContent').src = `data:image/png;base64,${data.screenshot}`;
+                }
+                const pythonCode = `
+from crawl4ai.web_crawler import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+    url='${url}',
+    screenshot=${screenshot}
+)
+print(result)
+                `;
+                redo(document.getElementById('pythonCode'), pythonCode);
+                document.getElementById('error').style.display = 'none';
+            })
+            .catch(error => {
+                document.getElementById('loading').style.display = 'none';
+                document.getElementById('error').style.display = 'block';
+                let errorMessage = 'An unexpected error occurred. Please try again later.';
+                
+                if (error.status === 429) {
+                    const details = error.details;
+                    if (details.retry_after) {
+                        errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
+                    } else if (details.reset_at) {
+                        const resetTime = new Date(details.reset_at);
+                        const waitTime = Math.ceil((resetTime - new Date()) / 1000);
+                        errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
+                    } else {
+                        errorMessage = `Rate limit exceeded. Please try again later.`;
+                    }
+                } else if (error.message) {
+                    errorMessage = error.message;
+                }
+                
+                document.querySelector('#error .terminal-alert').textContent = errorMessage;
+            });
+        });
+    </script>
+</div>

docs/md/examples/index.md

@@ -0,0 +1,33 @@
+# Examples
+
+Welcome to the examples section of Crawl4AI documentation! In this section, you will find practical examples demonstrating how to use Crawl4AI for various web crawling and data extraction tasks. Each example is designed to showcase different features and capabilities of the library.
+
+## Examples Index
+
+### [LLM Extraction](llm_extraction.md)
+
+This example demonstrates how to use Crawl4AI to extract information using Large Language Models (LLMs). You will learn how to configure the `LLMExtractionStrategy` to get structured data from web pages.
+
+### [JSON CSS Extraction](json_css_extraction.md)
+
+This example demonstrates how to use Crawl4AI to extract structured data without using LLM, and just focusing on page structure. You will learn how to use the `JsonCssExtractionStrategy` to extract data using CSS selectors.
+
+### [JS Execution & CSS Filtering](js_execution_css_filtering.md)
+
+Learn how to execute custom JavaScript code and filter data using CSS selectors. This example shows how to perform complex web interactions and extract specific content from web pages.
+
+### [Hooks & Auth](hooks_auth.md)
+
+This example covers the use of custom hooks for authentication and other pre-crawling tasks. You will see how to set up hooks to modify headers, authenticate sessions, and perform other preparatory actions before crawling.
+
+### [Summarization](summarization.md)
+
+Discover how to use Crawl4AI to summarize web page content. This example demonstrates the summarization capabilities of the library, helping you extract concise information from lengthy web pages.
+
+### [Research Assistant](research_assistant.md)
+
+In this example, Crawl4AI is used as a research assistant to gather and organize information from multiple sources. You will learn how to use various extraction and chunking strategies to compile a comprehensive report.
+
+---
+
+Each example includes detailed explanations and code snippets to help you understand and implement the features in your projects. Click on the links to explore each example and start making the most of Crawl4AI!

docs/md/examples/js_execution_css_filtering.md

@@ -0,0 +1,104 @@
+# JS Execution & CSS Filtering with AsyncWebCrawler
+
+In this example, we'll demonstrate how to use Crawl4AI's AsyncWebCrawler to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.
+
+## Example: Extracting Structured Data Asynchronously
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.chunking_strategy import RegexChunking
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
+
+async def main():
+    # Define the JavaScript code to click the "Load More" button
+    js_code = """
+    const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+    if (loadMoreButton) {
+        loadMoreButton.click();
+        // Wait for new content to load
+        await new Promise(resolve => setTimeout(resolve, 2000));
+    }
+    """
+
+    # Define a wait_for function to ensure content is loaded
+    wait_for = """
+    () => {
+        const articles = document.querySelectorAll('article.tease-card');
+        return articles.length > 10;
+    }
+    """
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Run the crawler with keyword filtering and CSS selector
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            wait_for=wait_for,
+            css_selector="article.tease-card",
+            extraction_strategy=CosineStrategy(
+                semantic_filter="technology",
+            ),
+            chunking_strategy=RegexChunking(),
+        )
+
+    # Display the extracted result
+    print(result.extracted_content)
+
+# Run the async function
+asyncio.run(main())
+```
+
+### Explanation
+
+1. **Asynchronous Execution**: We use `AsyncWebCrawler` with async/await syntax for non-blocking execution.
+
+2. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button and waits for new content to load.
+
+3. **Wait Condition**: The `wait_for` function ensures that the page has loaded more than 10 articles before proceeding with the extraction.
+
+4. **CSS Selector**: The `css_selector="article.tease-card"` parameter ensures that only article cards are extracted from the web page.
+
+5. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
+
+6. **Chunking Strategy**: We use `RegexChunking()` to split the content into manageable chunks for processing.
+
+## Advanced Usage: Custom Session and Multiple Requests
+
+For more complex scenarios where you need to maintain state across multiple requests or execute additional JavaScript after the initial page load, you can use a custom session:
+
+```python
+async def advanced_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Initial crawl with custom session
+        result1 = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            wait_for=wait_for,
+            css_selector="article.tease-card",
+            session_id="business_session"
+        )
+
+        # Execute additional JavaScript in the same session
+        result2 = await crawler.crawler_strategy.execute_js(
+            session_id="business_session",
+            js_code="window.scrollTo(0, document.body.scrollHeight);",
+            wait_for_js="() => window.innerHeight + window.scrollY >= document.body.offsetHeight"
+        )
+
+        # Process results
+        print("Initial crawl result:", result1.extracted_content)
+        print("Additional JS execution result:", result2.html)
+
+asyncio.run(advanced_crawl())
+```
+
+This advanced example demonstrates how to:
+1. Use a custom session to maintain state across requests.
+2. Execute additional JavaScript after the initial page load.
+3. Wait for specific conditions using JavaScript functions.
+
+## Try It Yourself
+
+These examples demonstrate the power and flexibility of Crawl4AI's AsyncWebCrawler in handling complex web interactions and extracting meaningful data asynchronously. You can customize the JavaScript code, CSS selectors, extraction strategies, and waiting conditions to suit your specific requirements.

docs/md/examples/json_css_extraction.md

@@ -139,4 +139,4 @@ This advanced example demonstrates how to:
 
 By mastering the `JsonCssExtractionStrategy`, you can efficiently extract structured data from a wide variety of web pages, making it a valuable tool in your web scraping toolkit.
 
-For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](./css-advanced.md).
+For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](../full_details/advanced_jsoncss_extraction.md).

docs/md/examples/llm_extraction.md

@@ -27,7 +27,7 @@ async def extract_openai_fees():
             url=url,
             word_count_threshold=1,
             extraction_strategy=LLMExtractionStrategy(
-                provider="openai/gpt-4o", # Or use ollama like provider="ollama/nemotron"
+                provider="openai/gpt-4o",
                 api_token=os.getenv('OPENAI_API_KEY'),
                 schema=OpenAIModelFee.model_json_schema(),
                 extraction_type="schema",

docs/md/examples/research_assistant.md

@@ -0,0 +1,220 @@
+# Research Assistant Example with AsyncWebCrawler
+
+This example demonstrates how to build an advanced research assistant using `Chainlit`, `Crawl4AI`'s `AsyncWebCrawler`, and various AI services. The assistant can crawl web pages asynchronously, answer questions based on the crawled content, and handle audio inputs.
+
+## Step-by-Step Guide
+
+1. **Install Required Packages**
+
+    Ensure you have the necessary packages installed:
+
+    ```bash
+    pip install chainlit groq openai crawl4ai
+    ```
+
+2. **Import Libraries**
+
+    ```python
+    import os
+    import time
+    import asyncio
+    from openai import AsyncOpenAI
+    import chainlit as cl
+    import re
+    from io import BytesIO
+    from chainlit.element import ElementBased
+    from groq import Groq
+    from crawl4ai import AsyncWebCrawler
+    from crawl4ai.extraction_strategy import NoExtractionStrategy
+    from crawl4ai.chunking_strategy import RegexChunking
+
+    client = AsyncOpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.getenv("GROQ_API_KEY"))
+
+    # Instrument the OpenAI client
+    cl.instrument_openai()
+    ```
+
+3. **Set Configuration**
+
+    ```python
+    settings = {
+        "model": "llama3-8b-8192",
+        "temperature": 0.5,
+        "max_tokens": 500,
+        "top_p": 1,
+        "frequency_penalty": 0,
+        "presence_penalty": 0,
+    }
+    ```
+
+4. **Define Utility Functions**
+
+    ```python
+    def extract_urls(text):
+        url_pattern = re.compile(r'(https?://\S+)')
+        return url_pattern.findall(text)
+
+    async def crawl_urls(urls):
+        async with AsyncWebCrawler(verbose=True) as crawler:
+            results = await crawler.arun_many(
+                urls=urls,
+                word_count_threshold=10,
+                extraction_strategy=NoExtractionStrategy(),
+                chunking_strategy=RegexChunking(),
+                bypass_cache=True
+            )
+        return [result.markdown for result in results if result.success]
+    ```
+
+5. **Initialize Chat Start Event**
+
+    ```python
+    @cl.on_chat_start
+    async def on_chat_start():
+        cl.user_session.set("session", {
+            "history": [],
+            "context": {}
+        })  
+        await cl.Message(content="Welcome to the chat! How can I assist you today?").send()
+    ```
+
+6. **Handle Incoming Messages**
+
+    ```python
+    @cl.on_message
+    async def on_message(message: cl.Message):
+        user_session = cl.user_session.get("session")
+
+        # Extract URLs from the user's message
+        urls = extract_urls(message.content)
+
+        if urls:
+            crawled_contents = await crawl_urls(urls)
+            for url, content in zip(urls, crawled_contents):
+                ref_number = f"REF_{len(user_session['context']) + 1}"
+                user_session["context"][ref_number] = {
+                    "url": url,
+                    "content": content
+                }
+
+        user_session["history"].append({
+            "role": "user",
+            "content": message.content
+        })
+
+        # Create a system message that includes the context
+        context_messages = [
+            f'<appendix ref="{ref}">\n{data["content"]}\n</appendix>'
+            for ref, data in user_session["context"].items()
+        ]
+        system_message = {
+            "role": "system",
+            "content": (
+                "You are a helpful bot. Use the following context for answering questions. "
+                "Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
+                "If the question requires any information from the provided appendices or context, refer to the sources. "
+                "If not, there is no need to add a references section. "
+                "At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
+                "\n\n".join(context_messages)
+            ) if context_messages else "You are a helpful assistant."
+        }
+
+        msg = cl.Message(content="")
+        await msg.send()
+
+        # Get response from the LLM
+        stream = await client.chat.completions.create(
+            messages=[system_message, *user_session["history"]],
+            stream=True,
+            **settings
+        )
+
+        assistant_response = ""
+        async for part in stream:
+            if token := part.choices[0].delta.content:
+                assistant_response += token
+                await msg.stream_token(token)
+
+        # Add assistant message to the history
+        user_session["history"].append({
+            "role": "assistant",
+            "content": assistant_response
+        })
+        await msg.update()
+
+        # Append the reference section to the assistant's response
+        if user_session["context"]:
+            reference_section = "\n\nReferences:\n"
+            for ref, data in user_session["context"].items():
+                reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
+            msg.content += reference_section
+            await msg.update()
+    ```
+
+7. **Handle Audio Input**
+
+    ```python
+    @cl.on_audio_chunk
+    async def on_audio_chunk(chunk: cl.AudioChunk):
+        if chunk.isStart:
+            buffer = BytesIO()
+            buffer.name = f"input_audio.{chunk.mimeType.split('/')[1]}"
+            cl.user_session.set("audio_buffer", buffer)
+            cl.user_session.set("audio_mime_type", chunk.mimeType)
+        cl.user_session.get("audio_buffer").write(chunk.data)
+
+    @cl.step(type="tool")
+    async def speech_to_text(audio_file):
+        response = await client.audio.transcriptions.create(
+            model="whisper-large-v3", file=audio_file
+        )
+        return response.text
+
+    @cl.on_audio_end
+    async def on_audio_end(elements: list[ElementBased]):
+        audio_buffer: BytesIO = cl.user_session.get("audio_buffer")
+        audio_buffer.seek(0)
+        audio_file = audio_buffer.read()
+        audio_mime_type: str = cl.user_session.get("audio_mime_type")
+        
+        start_time = time.time()
+        transcription = await speech_to_text((audio_buffer.name, audio_file, audio_mime_type))
+        end_time = time.time()
+        print(f"Transcription took {end_time - start_time} seconds")
+        
+        user_msg = cl.Message(author="You", type="user_message", content=transcription)
+        await user_msg.send()
+        await on_message(user_msg)
+    ```
+
+8. **Run the Chat Application**
+
+    ```python
+    if __name__ == "__main__":
+        from chainlit.cli import run_chainlit
+        run_chainlit(__file__)
+    ```
+
+## Explanation
+
+- **Libraries and Configuration**: We import necessary libraries, including `AsyncWebCrawler` from `crawl4ai`.
+- **Utility Functions**: 
+  - `extract_urls`: Uses regex to find URLs in messages.
+  - `crawl_urls`: An asynchronous function that uses `AsyncWebCrawler` to fetch content from multiple URLs concurrently.
+- **Chat Start Event**: Initializes the chat session and sends a welcome message.
+- **Message Handling**: 
+  - Extracts URLs from user messages.
+  - Asynchronously crawls the URLs using `AsyncWebCrawler`.
+  - Updates chat history and context with crawled content.
+  - Generates a response using the LLM, incorporating the crawled context.
+- **Audio Handling**: Captures, buffers, and transcribes audio input, then processes the transcription as text.
+- **Running the Application**: Starts the Chainlit server for interaction with the assistant.
+
+## Key Improvements
+
+1. **Asynchronous Web Crawling**: Using `AsyncWebCrawler` allows for efficient, concurrent crawling of multiple URLs.
+2. **Improved Context Management**: The assistant now maintains a context of crawled content, allowing for more informed responses.
+3. **Dynamic Reference System**: The assistant can refer to specific sources in its responses and provide a reference section.
+4. **Seamless Audio Integration**: The ability to handle audio inputs makes the assistant more versatile and user-friendly.
+
+This updated Research Assistant showcases how to create a powerful, interactive tool that can efficiently fetch and process web content, handle various input types, and provide informed responses based on the gathered information.

docs/md/examples/summarization.md

@@ -0,0 +1,153 @@
+# Summarization Example with AsyncWebCrawler
+
+This example demonstrates how to use Crawl4AI's `AsyncWebCrawler` to extract a summary from a web page asynchronously. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.
+
+## Step-by-Step Guide
+
+1. **Import Necessary Modules**
+
+    First, import the necessary modules and classes:
+
+    ```python
+    import os
+    import json
+    import asyncio
+    from crawl4ai import AsyncWebCrawler
+    from crawl4ai.extraction_strategy import LLMExtractionStrategy
+    from crawl4ai.chunking_strategy import RegexChunking
+    from pydantic import BaseModel, Field
+    ```
+
+2. **Define the URL to be Crawled**
+
+    Set the URL of the web page you want to summarize:
+
+    ```python
+    url = 'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
+    ```
+
+3. **Define the Data Model**
+
+    Use Pydantic to define the structure of the extracted data:
+
+    ```python
+    class PageSummary(BaseModel):
+        title: str = Field(..., description="Title of the page.")
+        summary: str = Field(..., description="Summary of the page.")
+        brief_summary: str = Field(..., description="Brief summary of the page.")
+        keywords: list = Field(..., description="Keywords assigned to the page.")
+    ```
+
+4. **Create the Extraction Strategy**
+
+    Set up the `LLMExtractionStrategy` with the necessary parameters:
+
+    ```python
+    extraction_strategy = LLMExtractionStrategy(
+        provider="openai/gpt-4o", 
+        api_token=os.getenv('OPENAI_API_KEY'), 
+        schema=PageSummary.model_json_schema(),
+        extraction_type="schema",
+        apply_chunking=False,
+        instruction=(
+            "From the crawled content, extract the following details: "
+            "1. Title of the page "
+            "2. Summary of the page, which is a detailed summary "
+            "3. Brief summary of the page, which is a paragraph text "
+            "4. Keywords assigned to the page, which is a list of keywords. "
+            'The extracted JSON format should look like this: '
+            '{ "title": "Page Title", "summary": "Detailed summary of the page.", '
+            '"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
+        )
+    )
+    ```
+
+5. **Define the Async Crawl Function**
+
+    Create an asynchronous function to run the crawler:
+
+    ```python
+    async def crawl_and_summarize(url):
+        async with AsyncWebCrawler(verbose=True) as crawler:
+            result = await crawler.arun(
+                url=url,
+                word_count_threshold=1,
+                extraction_strategy=extraction_strategy,
+                chunking_strategy=RegexChunking(),
+                bypass_cache=True,
+            )
+            return result
+    ```
+
+6. **Run the Crawler and Process Results**
+
+    Use asyncio to run the crawler and process the results:
+
+    ```python
+    async def main():
+        result = await crawl_and_summarize(url)
+        
+        if result.success:
+            page_summary = json.loads(result.extracted_content)
+            print("Extracted Page Summary:")
+            print(json.dumps(page_summary, indent=2))
+            
+            # Save the extracted data
+            with open(".data/page_summary.json", "w", encoding="utf-8") as f:
+                json.dump(page_summary, f, indent=2)
+            print("Page summary saved to .data/page_summary.json")
+        else:
+            print(f"Failed to crawl and summarize the page. Error: {result.error_message}")
+
+    # Run the async main function
+    asyncio.run(main())
+    ```
+
+## Explanation
+
+- **Importing Modules**: We import the necessary modules, including `AsyncWebCrawler` and `LLMExtractionStrategy` from Crawl4AI.
+- **URL Definition**: We set the URL of the web page to crawl and summarize.
+- **Data Model Definition**: We define the structure of the data to extract using Pydantic's `BaseModel`.
+- **Extraction Strategy Setup**: We create an instance of `LLMExtractionStrategy` with the schema and detailed instructions for the extraction process.
+- **Async Crawl Function**: We define an asynchronous function `crawl_and_summarize` that uses `AsyncWebCrawler` to perform the crawling and extraction.
+- **Main Execution**: In the `main` function, we run the crawler, process the results, and save the extracted data.
+
+## Advanced Usage: Crawling Multiple URLs
+
+To demonstrate the power of `AsyncWebCrawler`, here's how you can summarize multiple pages concurrently:
+
+```python
+async def crawl_multiple_urls(urls):
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        tasks = [crawler.arun(
+            url=url,
+            word_count_threshold=1,
+            extraction_strategy=extraction_strategy,
+            chunking_strategy=RegexChunking(),
+            bypass_cache=True
+        ) for url in urls]
+        results = await asyncio.gather(*tasks)
+    return results
+
+async def main():
+    urls = [
+        'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot',
+        'https://marketplace.visualstudio.com/items?itemName=GitHub.copilot',
+        'https://marketplace.visualstudio.com/items?itemName=ms-python.python'
+    ]
+    results = await crawl_multiple_urls(urls)
+    
+    for i, result in enumerate(results):
+        if result.success:
+            page_summary = json.loads(result.extracted_content)
+            print(f"\nSummary for URL {i+1}:")
+            print(json.dumps(page_summary, indent=2))
+        else:
+            print(f"\nFailed to summarize URL {i+1}. Error: {result.error_message}")
+
+asyncio.run(main())
+```
+
+This advanced example shows how to use `AsyncWebCrawler` to efficiently summarize multiple web pages concurrently, significantly reducing the total processing time compared to sequential crawling.
+
+By leveraging the asynchronous capabilities of Crawl4AI, you can perform advanced web crawling and data extraction tasks with improved efficiency and scalability.

docs/md/full_details/advanced_features.md

@@ -0,0 +1,138 @@
+# Advanced Features
+
+Crawl4AI offers a range of advanced features that allow you to fine-tune your web crawling and data extraction process. This section will cover some of these advanced features, including taking screenshots, extracting media and links, customizing the user agent, using custom hooks, and leveraging CSS selectors.
+
+## Taking Screenshots 📸
+
+One of the cool features of Crawl4AI is the ability to take screenshots of the web pages you're crawling. This can be particularly useful for visual verification or for capturing the state of dynamic content.
+
+Here's how you can take a screenshot:
+
+```python
+from crawl4ai import WebCrawler
+import base64
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with the screenshot parameter
+result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
+
+# Save the screenshot to a file
+with open("screenshot.png", "wb") as f:
+    f.write(base64.b64decode(result.screenshot))
+
+print("Screenshot saved to 'screenshot.png'!")
+```
+
+In this example, we create a `WebCrawler` instance, warm it up, and then run it with the `screenshot` parameter set to `True`. The screenshot is saved as a base64 encoded string in the result, which we then decode and save as a PNG file.
+
+## Extracting Media and Links 🎨🔗
+
+Crawl4AI can extract all media tags (images, audio, and video) and links (both internal and external) from a web page. This feature is useful for collecting multimedia content or analyzing link structures.
+
+Here's an example:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler
+result = crawler.run(url="https://www.nbcnews.com/business")
+
+print("Extracted media:", result.media)
+print("Extracted links:", result.links)
+```
+
+In this example, the `result` object contains dictionaries for media and links, which you can access and use as needed.
+
+## Customizing the User Agent 🕵️‍♂️
+
+Crawl4AI allows you to set a custom user agent for your HTTP requests. This can help you avoid detection by web servers or simulate different browsing environments.
+
+Here's how to set a custom user agent:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with a custom user agent
+result = crawler.run(url="https://www.nbcnews.com/business", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
+
+print("Crawl result:", result)
+```
+
+In this example, we specify a custom user agent string when running the crawler.
+
+## Using Custom Hooks 🪝
+
+Hooks are a powerful feature in Crawl4AI that allow you to customize the crawling process at various stages. You can define hooks for actions such as driver initialization, before and after URL fetching, and before returning the HTML.
+
+Here's an example of using hooks:
+
+```python
+from crawl4ai import WebCrawler
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+# Define the hooks
+def on_driver_created(driver):
+    driver.maximize_window()
+    driver.get('https://example.com/login')
+    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.NAME, 'username'))).send_keys('testuser')
+    driver.find_element(By.NAME, 'password').send_keys('password123')
+    driver.find_element(By.NAME, 'login').click()
+    return driver
+
+def before_get_url(driver):
+    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
+    return driver
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Set the hooks
+crawler.set_hook('on_driver_created', on_driver_created)
+crawler.set_hook('before_get_url', before_get_url)
+
+# Run the crawler
+result = crawler.run(url="https://example.com")
+
+print("Crawl result:", result)
+```
+
+In this example, we define hooks to handle driver initialization and custom headers before fetching the URL.
+
+## Using CSS Selectors 🎯
+
+CSS selectors allow you to target specific elements on a web page for extraction. This can be useful for scraping structured content, such as articles or product details.
+
+Here's an example of using a CSS selector:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with a CSS selector to extract only H2 tags
+result = crawler.run(url="https://www.nbcnews.com/business", css_selector="h2")
+
+print("Extracted H2 tags:", result.extracted_content)
+```
+
+In this example, we use the `css_selector` parameter to extract only the H2 tags from the web page.
+
+---
+
+With these advanced features, you can leverage Crawl4AI to perform sophisticated web crawling and data extraction tasks. Whether you need to take screenshots, extract specific elements, customize the crawling process, or set custom headers, Crawl4AI provides the flexibility and power to meet your needs. Happy crawling! 🕷️🚀

docs/md/full_details/chunking_strategies.md

@@ -0,0 +1,133 @@
+## Chunking Strategies 📚
+
+Crawl4AI provides several powerful chunking strategies to divide text into manageable parts for further processing. Each strategy has unique characteristics and is suitable for different scenarios. Let's explore them one by one.
+
+### RegexChunking
+
+`RegexChunking` splits text using regular expressions. This is ideal for creating chunks based on specific patterns like paragraphs or sentences.
+
+#### When to Use
+- Great for structured text with consistent delimiters.
+- Suitable for documents where specific patterns (e.g., double newlines, periods) indicate logical chunks.
+
+#### Parameters
+- `patterns` (list, optional): Regular expressions used to split the text. Default is to split by double newlines (`['\n\n']`).
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+# Define patterns for splitting text
+patterns = [r'\n\n', r'\. ']
+chunker = RegexChunking(patterns=patterns)
+
+# Sample text
+text = "This is a sample text. It will be split into chunks.\n\nThis is another paragraph."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### NlpSentenceChunking
+
+`NlpSentenceChunking` uses NLP models to split text into sentences, ensuring accurate sentence boundaries.
+
+#### When to Use
+- Ideal for texts where sentence boundaries are crucial.
+- Useful for creating chunks that preserve grammatical structures.
+
+#### Parameters
+- None.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+
+# Sample text
+text = "This is a sample text. It will be split into sentences. Here's another sentence."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### TopicSegmentationChunking
+
+`TopicSegmentationChunking` employs the TextTiling algorithm to segment text into topic-based chunks. This method identifies thematic boundaries.
+
+#### When to Use
+- Perfect for long documents with distinct topics.
+- Useful when preserving topic continuity is more important than maintaining text order.
+
+#### Parameters
+- `num_keywords` (int, optional): Number of keywords for each topic segment. Default is `3`.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import TopicSegmentationChunking
+
+chunker = TopicSegmentationChunking(num_keywords=3)
+
+# Sample text
+text = "This document contains several topics. Topic one discusses AI. Topic two covers machine learning."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### FixedLengthWordChunking
+
+`FixedLengthWordChunking` splits text into chunks based on a fixed number of words. This ensures each chunk has approximately the same length.
+
+#### When to Use
+- Suitable for processing large texts where uniform chunk size is important.
+- Useful when the number of words per chunk needs to be controlled.
+
+#### Parameters
+- `chunk_size` (int, optional): Number of words per chunk. Default is `100`.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import FixedLengthWordChunking
+
+chunker = FixedLengthWordChunking(chunk_size=10)
+
+# Sample text
+text = "This is a sample text. It will be split into chunks of fixed length."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### SlidingWindowChunking
+
+`SlidingWindowChunking` uses a sliding window approach to create overlapping chunks. Each chunk has a fixed length, and the window slides by a specified step size.
+
+#### When to Use
+- Ideal for creating overlapping chunks to preserve context.
+- Useful for tasks where context from adjacent chunks is needed.
+
+#### Parameters
+- `window_size` (int, optional): Number of words in each chunk. Default is `100`.
+- `step` (int, optional): Number of words to slide the window. Default is `50`.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import SlidingWindowChunking
+
+chunker = SlidingWindowChunking(window_size=10, step=5)
+
+# Sample text
+text = "This is a sample text. It will be split using a sliding window approach to preserve context."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+With these chunking strategies, you can choose the best method to divide your text based on your specific needs. Whether you need precise sentence boundaries, topic-based segmentation, or uniform chunk sizes, Crawl4AI has you covered. Happy chunking! 📝✨

docs/md/full_details/crawl_request_parameters.md

@@ -0,0 +1,179 @@
+# Crawl Request Parameters for AsyncWebCrawler
+
+The `arun` method in Crawl4AI's `AsyncWebCrawler` is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `arun` method, along with their descriptions, possible values, and examples.
+
+## Parameters
+
+### url (str)
+**Description:** The URL of the webpage to crawl.
+**Required:** Yes
+**Example:**
+```python
+url = "https://www.nbcnews.com/business"
+```
+
+### word_count_threshold (int)
+**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is defined by `MIN_WORD_THRESHOLD`.
+**Required:** No
+**Default Value:** `MIN_WORD_THRESHOLD`
+**Example:**
+```python
+word_count_threshold = 10
+```
+
+### extraction_strategy (ExtractionStrategy)
+**Description:** The strategy to use for extracting content from the HTML. It must be an instance of `ExtractionStrategy`. If not provided, the default is `NoExtractionStrategy`.
+**Required:** No
+**Default Value:** `NoExtractionStrategy()`
+**Example:**
+```python
+extraction_strategy = CosineStrategy(semantic_filter="finance")
+```
+
+### chunking_strategy (ChunkingStrategy)
+**Description:** The strategy to use for chunking the text before processing. It must be an instance of `ChunkingStrategy`. The default value is `RegexChunking()`.
+**Required:** No
+**Default Value:** `RegexChunking()`
+**Example:**
+```python
+chunking_strategy = NlpSentenceChunking()
+```
+
+### bypass_cache (bool)
+**Description:** Whether to force a fresh crawl even if the URL has been previously crawled. The default value is `False`.
+**Required:** No
+**Default Value:** `False`
+**Example:**
+```python
+bypass_cache = True
+```
+
+### css_selector (str)
+**Description:** The CSS selector to target specific parts of the HTML for extraction. If not provided, the entire HTML will be processed.
+**Required:** No
+**Default Value:** `None`
+**Example:**
+```python
+css_selector = "div.article-content"
+```
+
+### screenshot (bool)
+**Description:** Whether to take screenshots of the page. The default value is `False`.
+**Required:** No
+**Default Value:** `False`
+**Example:**
+```python
+screenshot = True
+```
+
+### user_agent (str)
+**Description:** The user agent to use for the HTTP requests. If not provided, a default user agent will be used.
+**Required:** No
+**Default Value:** `None`
+**Example:**
+```python
+user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
+```
+
+### verbose (bool)
+**Description:** Whether to enable verbose logging. The default value is `True`.
+**Required:** No
+**Default Value:** `True`
+**Example:**
+```python
+verbose = True
+```
+
+### **kwargs
+Additional keyword arguments that can be passed to customize the crawling process further. Some notable options include:
+
+- **only_text (bool):** Whether to extract only text content, excluding HTML tags. Default is `False`.
+- **session_id (str):** A unique identifier for the crawling session. This is useful for maintaining state across multiple requests.
+- **js_code (str or list):** JavaScript code to be executed on the page before extraction.
+- **wait_for (str):** A CSS selector or JavaScript function to wait for before considering the page load complete.
+
+**Example:**
+```python
+result = await crawler.arun(
+    url="https://www.nbcnews.com/business",
+    css_selector="p",
+    only_text=True,
+    session_id="unique_session_123",
+    js_code="window.scrollTo(0, document.body.scrollHeight);",
+    wait_for="article.main-article"
+)
+```
+
+## Example Usage
+
+Here's an example of how to use the `arun` method with various parameters:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+async def main():
+    # Create the AsyncWebCrawler instance 
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Run the crawler with custom parameters
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            word_count_threshold=10,
+            extraction_strategy=CosineStrategy(semantic_filter="finance"),
+            chunking_strategy=NlpSentenceChunking(),
+            bypass_cache=True,
+            css_selector="div.article-content",
+            screenshot=True,
+            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
+            verbose=True,
+            only_text=True,
+            session_id="business_news_session",
+            js_code="window.scrollTo(0, document.body.scrollHeight);",
+            wait_for="footer"
+        )
+
+        print(result)
+
+# Run the async function
+asyncio.run(main())
+```
+
+This example demonstrates how to configure various parameters to customize the crawling and extraction process using the asynchronous version of Crawl4AI.
+
+## Additional Asynchronous Methods
+
+The `AsyncWebCrawler` class also provides other useful asynchronous methods:
+
+### arun_many
+**Description:** Crawl multiple URLs concurrently.
+**Example:**
+```python
+urls = ["https://example1.com", "https://example2.com", "https://example3.com"]
+results = await crawler.arun_many(urls, word_count_threshold=10, bypass_cache=True)
+```
+
+### aclear_cache
+**Description:** Clear the crawler's cache.
+**Example:**
+```python
+await crawler.aclear_cache()
+```
+
+### aflush_cache
+**Description:** Completely flush the crawler's cache.
+**Example:**
+```python
+await crawler.aflush_cache()
+```
+
+### aget_cache_size
+**Description:** Get the current size of the cache.
+**Example:**
+```python
+cache_size = await crawler.aget_cache_size()
+print(f"Current cache size: {cache_size}")
+```
+
+These asynchronous methods allow for efficient and flexible use of the AsyncWebCrawler in various scenarios.

docs/md/full_details/crawl_result_class.md

@@ -0,0 +1,104 @@
+# Crawl Result
+
+The `CrawlResult` class is the heart of Crawl4AI's output, encapsulating all the data extracted from a crawling session. This class contains various fields that store the results of the web crawling and extraction process. Let's break down each field and see what it holds. 🎉
+
+## Class Definition
+
+```python
+from pydantic import BaseModel
+from typing import Dict, List, Optional
+
+class CrawlResult(BaseModel):
+    url: str
+    html: str
+    success: bool
+    cleaned_html: Optional[str] = None
+    media: Dict[str, List[Dict]] = {}
+    links: Dict[str, List[Dict]] = {}
+    screenshot: Optional[str] = None
+    markdown: Optional[str] = None
+    extracted_content: Optional[str] = None
+    metadata: Optional[dict] = None
+    error_message: Optional[str] = None
+    session_id: Optional[str] = None
+    responser_headers: Optional[dict] = None
+    status_code: Optional[int] = None
+```
+
+## Fields Explanation
+
+### `url: str`
+The URL that was crawled. This field simply stores the URL of the web page that was processed.
+
+### `html: str`
+The raw HTML content of the web page. This is the unprocessed HTML source as retrieved by the crawler.
+
+### `success: bool`
+A flag indicating whether the crawling and extraction were successful. If any error occurs during the process, this will be `False`.
+
+### `cleaned_html: Optional[str]`
+The cleaned HTML content of the web page. This field holds the HTML after removing unwanted tags like `<script>`, `<style>`, and others that do not contribute to the useful content.
+
+### `media: Dict[str, List[Dict]]`
+A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here's how they are structured:
+
+- **Images**: Each image is represented as a dictionary with `src` (source URL) and `alt` (alternate text).
+- **Videos**: Each video is represented similarly with `src` and `alt`.
+- **Audios**: Each audio is represented with `src` and `alt`.
+
+```python
+media = {
+    'images': [
+        {'src': 'image_url1', 'alt': 'description1', "type": "image"},
+        {'src': 'image_url2', 'alt': 'description2', "type": "image"}
+    ],
+    'videos': [
+        {'src': 'video_url1', 'alt': 'description1', "type": "video"}
+    ],
+    'audios': [
+        {'src': 'audio_url1', 'alt': 'description1', "type": "audio"}
+    ]
+}
+```
+
+### `links: Dict[str, List[Dict]]`
+A dictionary containing lists of internal and external links extracted from the web page. Each link is represented as a dictionary with `href` (URL) and `text` (link text).
+
+- **Internal Links**: Links pointing to the same domain.
+- **External Links**: Links pointing to different domains.
+
+```python
+links = {
+    'internal': [
+        {'href': 'internal_link1', 'text': 'link_text1'},
+        {'href': 'internal_link2', 'text': 'link_text2'}
+    ],
+    'external': [
+        {'href': 'external_link1', 'text': 'link_text1'}
+    ]
+}
+```
+
+### `screenshot: Optional[str]`
+A base64-encoded screenshot of the web page. This field stores the screenshot data if the crawling was configured to take a screenshot.
+
+### `markdown: Optional[str]`
+The content of the web page converted to Markdown format. This is useful for generating clean, readable text that retains the structure of the original HTML.
+
+### `extracted_content: Optional[str]`
+The content extracted based on the specified extraction strategy. This field holds the meaningful content blocks extracted from the web page, ready for your AI and data processing needs.
+
+### `metadata: Optional[dict]`
+A dictionary containing metadata extracted from the web page, such as title, description, keywords, and other meta tags.
+
+### `error_message: Optional[str]`
+If an error occurs during crawling, this field will contain the error message, helping you debug and understand what went wrong. 🚨
+
+### `session_id: Optional[str]`
+A unique identifier for the crawling session. This can be useful for tracking and managing multiple crawling sessions.
+
+### `responser_headers: Optional[dict]`
+A dictionary containing the response headers from the web server. This can provide additional information about the server and the response.
+
+### `status_code: Optional[int]`
+The HTTP status code of the response. This indicates the success or failure of the HTTP request (e.g., 200 for success, 404 for not found, etc.).

docs/md/full_details/extraction_strategies.md

@@ -136,7 +136,7 @@ asyncio.run(main())
 
 By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy`, nuanced, instruction-based extraction with `LLMExtractionStrategy`, or precise structured data extraction with `JsonCssExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨
 
-For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](./css-advanced.md).
+For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](../full_details/advanced_jsoncss_extraction.md).
 
 
 ### CosineStrategy

docs/md/index.md

@@ -0,0 +1,93 @@
+# Crawl4AI
+
+Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
+
+## Introduction
+
+Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution with full asynchronous support.
+
+## Quick Start
+
+Here's a quick example to show you how easy it is to use Crawl4AI with its new asynchronous capabilities:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    # Create an instance of AsyncWebCrawler
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Run the crawler on a URL
+        result = await crawler.arun(url="https://www.nbcnews.com/business")
+
+        # Print the extracted content
+        print(result.markdown)
+
+# Run the async main function
+asyncio.run(main())
+```
+
+### Explanation
+
+1. **Importing the Library**: We start by importing the `AsyncWebCrawler` class from the `crawl4ai` library and the `asyncio` module.
+2. **Creating an Async Context**: We use an async context manager to create an instance of `AsyncWebCrawler`.
+3. **Running the Crawler**: The `arun()` method is used to asynchronously crawl the specified URL and extract meaningful content.
+4. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
+5. **Running the Async Function**: We use `asyncio.run()` to execute our async main function.
+
+## Documentation Structure
+
+This documentation is organized into several sections to help you navigate and find the information you need quickly:
+
+### [Home](index.md)
+
+An introduction to Crawl4AI, including a quick start guide and an overview of the documentation structure.
+
+### [Installation](installation.md)
+
+Instructions on how to install Crawl4AI and its dependencies.
+
+### [Introduction](introduction.md)
+
+A detailed introduction to Crawl4AI, its features, and how it can be used for various web crawling and data extraction tasks.
+
+### [Quick Start](quickstart.md)
+
+A step-by-step guide to get you up and running with Crawl4AI, including installation instructions and basic usage examples.
+
+### [Examples](examples/index.md)
+
+This section contains practical examples demonstrating different use cases of Crawl4AI:
+
+- [Structured Data Extraction](examples/json_css_extraction.md)
+- [LLM Extraction](examples/llm_extraction.md)
+- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
+- [Hooks & Auth](examples/hooks_auth.md)
+- [Summarization](examples/summarization.md)
+- [Research Assistant](examples/research_assistant.md)
+
+### [Full Details of Using Crawler](full_details/crawl_request_parameters.md)
+
+Comprehensive details on using the crawler, including:
+
+- [Crawl Request Parameters](full_details/crawl_request_parameters.md)
+- [Crawl Result Class](full_details/crawl_result_class.md)
+- [Session Based Crawling](full_details/session_based_crawling.md)
+- [Advanced Structured Data Extraction JsonCssExtraction](full_details/advanced_jsoncss_extraction.md)
+- [Advanced Features](full_details/advanced_features.md)
+- [Chunking Strategies](full_details/chunking_strategies.md)
+- [Extraction Strategies](full_details/extraction_strategies.md)
+
+### [Change Log](changelog.md)
+
+A log of all changes, updates, and improvements made to Crawl4AI.
+
+### [Contact](contact.md)
+
+Information on how to get in touch with the developers, report issues, and contribute to the project.
+
+## Get Started
+
+To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier, more efficient, and now fully asynchronous.
+
+Happy Crawling! 🕸️🚀

docs/md/interactive_content.html

@@ -0,0 +1,28 @@
+<h1>Try Our Library</h1>
+<form id="apiForm">
+    <label for="inputField">Enter some input:</label>
+    <input type="text" id="inputField" name="inputField" required>
+    <button type="submit">Submit</button>
+</form>
+<div id="result"></div>
+
+<script>
+    document.getElementById('apiForm').addEventListener('submit', function(event) {
+        event.preventDefault();
+        const input = document.getElementById('inputField').value;
+        fetch('https://your-api-endpoint.com/api', {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json'
+            },
+            body: JSON.stringify({ input: input })
+        })
+        .then(response => response.json())
+        .then(data => {
+            document.getElementById('result').textContent = JSON.stringify(data);
+        })
+        .catch(error => {
+            document.getElementById('result').textContent = 'Error: ' + error;
+        });
+    });
+</script>