Update Documentation

2024-10-27 19:24:46 +08:00
parent 38474bd66a
commit 4239654722
111 changed files with 7680 additions and 53 deletions
--- a/README.md
+++ b/README.md
@@ -8,16 +8,14 @@

 Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐

-> Looking for the synchronous version? Check out [README.sync.md](./README.sync.md). You can also access the previous version in the branch [V0.2.76](https://github.com/unclecode/crawl4ai/blob/v0.2.76).
+## New in 0.3.72 ✨

-## New update 0.3.6
- 🌐 Multi-browser support (Chromium, Firefox, WebKit)
- 🖼️ Improved image processing with lazy-loading detection
- 🔧 Custom page timeout parameter for better control over crawling behavior
- 🕰️ Enhanced handling of delayed content loading
- 🔑 Custom headers support for LLM interactions
- 🖼️ iframe content extraction for comprehensive page analysis
- ⏱️ Flexible timeout and delayed content retrieval options
+- 📄 Fit markdown generation for extracting main article content.
+- 🪄 Magic mode for comprehensive anti-bot detection bypass.
+- 🌐 Enhanced multi-browser support with seamless switching (Chromium, Firefox, WebKit)
+- 📚 New chunking strategies(Sliding window, Overlapping window, Flexible size control)
+- 💾 Improved caching system for better performance
+- ⚡ Optimized batch processing with automatic rate limiting

 ## Try it Now!

@@ -30,22 +28,28 @@ Crawl4AI simplifies asynchronous web crawling and data extraction, making it acc
 - 🆓 Completely free and open-source
 - 🚀 Blazing fast performance, outperforming many paid services
 - 🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
+- 🌐 Multi-browser support (Chromium, Firefox, WebKit)
 - 🌍 Supports crawling multiple URLs simultaneously
 - 🎨 Extracts and returns all media tags (Images, Audio, and Video)
 - 🔗 Extracts all external and internal links
 - 📚 Extracts metadata from the page
- 🔄 Custom hooks for authentication, headers, and page modifications before crawling
+- 🔄 Custom hooks for authentication, headers, and page modifications
 - 🕵️ User-agent customization
- 🖼️ Takes screenshots of the page
+- 🖼️ Takes screenshots of pages with enhanced error handling
 - 📜 Executes multiple custom JavaScripts before crawling
 - 📊 Generates structured output without LLM using JsonCssExtractionStrategy
 - 📚 Various chunking strategies: topic-based, regex, sentence, and more
 - 🧠 Advanced extraction strategies: cosine clustering, LLM, and more
 - 🎯 CSS selector support for precise data extraction
 - 📝 Passes instructions/keywords to refine extraction
- 🔒 Proxy support for enhanced privacy and access
- 🔄 Session management for complex multi-page crawling scenarios
- 🌐 Asynchronous architecture for improved performance and scalability
+- 🔒 Proxy support with authentication for enhanced access
+- 🔄 Session management for complex multi-page crawling
+- 🌐 Asynchronous architecture for improved performance
+- 🖼️ Improved image processing with lazy-loading detection
+- 🕰️ Enhanced handling of delayed content loading
+- 🔑 Custom headers support for LLM interactions
+- 🖼️ iframe content extraction for comprehensive analysis
+- ⏱️ Flexible timeout and delayed content retrieval options

 ## Installation 🛠️

--- a/crawl4ai/content_scrapping_strategy.py
+++ b/crawl4ai/content_scrapping_strategy.py
@@ -207,8 +207,8 @@ class WebScrappingStrategy(ContentScrappingStrategy):

                keep_element = False
                
-                social_media_domains = SOCIAL_MEDIA_DOMAINS + kwargs.get('social_media_domains', [])
-                social_media_domains = list(set(social_media_domains))
+                exclude_social_media_domains = SOCIAL_MEDIA_DOMAINS + kwargs.get('exclude_social_media_domains', [])
+                exclude_social_media_domains = list(set(exclude_social_media_domains))

                
                try:
@@ -249,7 +249,7 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                                element.decompose()
                                return False
                            elif kwargs.get('exclude_social_media_links', False):
-                                if any(domain in normalized_href.lower() for domain in social_media_domains):
+                                if any(domain in normalized_href.lower() for domain in exclude_social_media_domains):
                                    element.decompose()
                                    return False
                            elif kwargs.get('exclude_domains', []):
@@ -285,7 +285,7 @@ class WebScrappingStrategy(ContentScrappingStrategy):
                        if not kwargs.get('exclude_external_images', False) and kwargs.get('exclude_social_media_links', False):
                            src_url_base = src.split('/')[2]
                            url_base = url.split('/')[2]
-                            if any(domain in src for domain in social_media_domains):
+                            if any(domain in src for domain in exclude_social_media_domains):
                                element.decompose()
                                return False
                            
--- a/docs/details/extraction.md
+++ b/docs/details/extraction.md
@@ -0,0 +1,157 @@
+### Extraction Strategies
+
+#### 1. LLMExtractionStrategy
+```python
+LLMExtractionStrategy(
+    # Core Parameters
+    provider: str = DEFAULT_PROVIDER,  # LLM provider (e.g., "openai/gpt-4", "huggingface/...", "ollama/...")
+    api_token: Optional[str] = None,  # API token for the provider
+    instruction: str = None,  # Custom instruction for extraction
+    schema: Dict = None,  # Pydantic model schema for structured extraction
+    extraction_type: str = "block",  # Type of extraction: "block" or "schema"
+    
+    # Chunking Parameters
+    chunk_token_threshold: int = CHUNK_TOKEN_THRESHOLD,  # Maximum tokens per chunk
+    overlap_rate: float = OVERLAP_RATE,  # Overlap between chunks
+    word_token_rate: float = WORD_TOKEN_RATE,  # Conversion rate from words to tokens
+    apply_chunking: bool = True,  # Whether to apply text chunking
+    
+    # API Configuration
+    base_url: str = None,  # Base URL for API calls
+    api_base: str = None,  # Alternative base URL
+    extra_args: Dict = {},  # Additional provider-specific arguments
+    
+    verbose: bool = False  # Enable verbose logging
+)
+```
+
+Usage Example:
+```python
+class NewsArticle(BaseModel):
+    title: str
+    content: str
+
+strategy = LLMExtractionStrategy(
+    provider="ollama/nemotron",
+    api_token="your-token",
+    schema=NewsArticle.schema(),
+    instruction="Extract news article content with title and main text"
+)
+
+result = await crawler.arun(url="https://example.com", extraction_strategy=strategy)
+```
+
+#### 2. JsonCssExtractionStrategy
+```python
+JsonCssExtractionStrategy(
+    schema: Dict[str, Any],  # Schema defining extraction rules
+    verbose: bool = False  # Enable verbose logging
+)
+
+# Schema Structure
+schema = {
+    "name": str,  # Name of the extraction schema
+    "baseSelector": str,  # CSS selector for base elements
+    "fields": [
+        {
+            "name": str,  # Field name
+            "selector": str,  # CSS selector
+            "type": str,  # Field type: "text", "attribute", "html", "regex", "nested", "list", "nested_list"
+            "attribute": str,  # For type="attribute"
+            "pattern": str,  # For type="regex"
+            "transform": str,  # Optional: "lowercase", "uppercase", "strip"
+            "default": Any,  # Default value if extraction fails
+            "fields": List[Dict],  # For nested/list types
+        }
+    ]
+}
+```
+
+Usage Example:
+```python
+schema = {
+    "name": "News Articles",
+    "baseSelector": "article.news-item",
+    "fields": [
+        {
+            "name": "title",
+            "selector": "h1",
+            "type": "text",
+            "transform": "strip"
+        },
+        {
+            "name": "date",
+            "selector": ".date",
+            "type": "attribute",
+            "attribute": "datetime"
+        }
+    ]
+}
+
+strategy = JsonCssExtractionStrategy(schema)
+result = await crawler.arun(url="https://example.com", extraction_strategy=strategy)
+```
+
+#### 3. CosineStrategy
+```python
+CosineStrategy(
+    # Content Filtering
+    semantic_filter: str = None,  # Keyword filter for document filtering
+    word_count_threshold: int = 10,  # Minimum words per cluster
+    sim_threshold: float = 0.3,  # Similarity threshold for filtering
+    
+    # Clustering Parameters
+    max_dist: float = 0.2,  # Maximum distance for clustering
+    linkage_method: str = 'ward',  # Clustering linkage method
+    top_k: int = 3,  # Number of top categories to extract
+    
+    # Model Configuration
+    model_name: str = 'sentence-transformers/all-MiniLM-L6-v2',  # Embedding model
+    
+    verbose: bool = False  # Enable verbose logging
+)
+```
+
+### Chunking Strategies
+
+#### 1. RegexChunking
+```python
+RegexChunking(
+    patterns: List[str] = None  # List of regex patterns for splitting text
+    # Default pattern: [r'\n\n']
+)
+```
+
+Usage Example:
+```python
+chunker = RegexChunking(patterns=[r'\n\n', r'\.\s+'])  # Split on double newlines and sentences
+chunks = chunker.chunk(text)
+```
+
+#### 2. SlidingWindowChunking
+```python
+SlidingWindowChunking(
+    window_size: int = 100,  # Size of the window in words
+    step: int = 50,  # Number of words to slide the window
+)
+```
+
+Usage Example:
+```python
+chunker = SlidingWindowChunking(window_size=200, step=100)
+chunks = chunker.chunk(text)  # Creates overlapping chunks of 200 words, moving 100 words at a time
+```
+
+#### 3. OverlappingWindowChunking
+```python
+OverlappingWindowChunking(
+    window_size: int = 1000,  # Size of each chunk in words
+    overlap: int = 100  # Number of words to overlap between chunks
+)
+```
+
+Usage Example:
+```python
+chunker = OverlappingWindowChunking(window_size=500, overlap=50)
+chunks = chunker.chunk(text)  # Creates 500-word chunks with 50-word overlap
+```
--- a/docs/details/feature_lists.md
+++ b/docs/details/feature_lists.md
@@ -0,0 +1,175 @@
+# Features
+
+## Current Features
+1. Async-first architecture for high-performance web crawling
+2. Built-in anti-bot detection bypass ("magic mode")
+3. Multiple browser engine support (Chromium, Firefox, WebKit)
+4. Smart session management with automatic cleanup
+5. Automatic content cleaning and relevance scoring
+6. Built-in markdown generation with formatting preservation
+7. Intelligent image scoring and filtering
+8. Automatic popup and overlay removal
+9. Smart wait conditions (CSS/JavaScript based)
+10. Multi-provider LLM integration (OpenAI, HuggingFace, Ollama)
+11. Schema-based structured data extraction
+12. Automated iframe content processing
+13. Intelligent link categorization (internal/external)
+14. Multiple chunking strategies for large content
+15. Real-time HTML cleaning and sanitization
+16. Automatic screenshot capabilities
+17. Social media link filtering
+18. Semantic similarity-based content clustering
+19. Human behavior simulation for anti-bot bypass
+20. Proxy support with authentication
+21. Automatic resource cleanup
+22. Custom CSS selector-based extraction
+23. Automatic content relevance scoring ("fit" content)
+24. Recursive website crawling capabilities
+25. Flexible hook system for customization
+26. Built-in caching system
+27. Domain-based content filtering
+28. Dynamic content handling with JavaScript execution
+29. Automatic media content extraction and classification
+30. Metadata extraction and processing
+31. Customizable HTML to Markdown conversion
+32. Token-aware content chunking for LLM processing
+33. Automatic response header and status code handling
+34. Browser fingerprint customization
+35. Multiple extraction strategies (LLM, CSS, Cosine, XPATH)
+36. Automatic error image generation for failed screenshots
+37. Smart content overlap handling for large texts
+38. Built-in rate limiting for batch processing
+39. Automatic cookie handling
+40. Browser Console logging and debugging capabilities
+
+## Feature Techs
+• Browser Management
+  - Asynchronous browser control
+  - Multi-browser support (Chromium, Firefox, WebKit)
+  - Headless mode support
+  - Browser cleanup and resource management
+  - Custom browser arguments and configuration
+  - Context management with `__aenter__` and `__aexit__`
+
+• Session Handling
+  - Session management with TTL (Time To Live)
+  - Session reuse capabilities
+  - Session cleanup for expired sessions
+  - Session-based context preservation
+
+• Stealth Features
+  - Playwright stealth configuration
+  - Navigator properties override
+  - WebDriver detection evasion
+  - Chrome app simulation
+  - Plugin simulation
+  - Language preferences simulation
+  - Hardware concurrency simulation
+  - Media codecs simulation
+
+• Network Features
+  - Proxy support with authentication
+  - Custom headers management
+  - Cookie handling
+  - Response header capture
+  - Status code tracking
+  - Network idle detection
+
+• Page Interaction
+  - Smart wait functionality for multiple conditions
+  - CSS selector-based waiting
+  - JavaScript condition waiting
+  - Custom JavaScript execution
+  - User interaction simulation (mouse/keyboard)
+  - Page scrolling
+  - Timeout management
+  - Load state monitoring
+
+• Content Processing
+  - HTML content extraction
+  - Iframe processing and content extraction
+  - Delayed content retrieval
+  - Content caching
+  - Cache file management
+  - HTML cleaning and processing
+
+• Image Handling
+  - Screenshot capabilities (full page)
+  - Base64 encoding of screenshots
+  - Image dimension updating
+  - Image filtering (size/visibility)
+  - Error image generation
+  - Natural width/height preservation
+
+• Overlay Management
+  - Popup removal
+  - Cookie notice removal
+  - Newsletter dialog removal
+  - Modal removal
+  - Fixed position element removal
+  - Z-index based overlay detection
+  - Visibility checking
+
+• Hook System
+  - Browser creation hooks
+  - User agent update hooks
+  - Execution start hooks
+  - Navigation hooks (before/after goto)
+  - HTML retrieval hooks
+  - HTML return hooks
+
+• Error Handling
+  - Browser error catching
+  - Network error handling
+  - Timeout handling
+  - Screenshot error recovery
+  - Invalid selector handling
+  - General exception management
+
+• Performance Features
+  - Concurrent URL processing
+  - Semaphore-based rate limiting
+  - Async gathering of results
+  - Resource cleanup
+  - Memory management
+
+• Debug Features
+  - Console logging
+  - Page error logging
+  - Verbose mode
+  - Error message generation
+  - Warning system
+
+• Security Features
+  - Certificate error handling
+  - Sandbox configuration
+  - GPU handling
+  - CSP (Content Security Policy) compliant waiting
+
+• Configuration
+  - User agent customization
+  - Viewport configuration
+  - Timeout configuration
+  - Browser type selection
+  - Proxy configuration
+  - Header configuration
+
+• Data Models
+  - Pydantic model for responses
+  - Type hints throughout code
+  - Structured response format
+  - Optional response fields
+
+• File System Integration
+  - Cache directory management
+  - File path handling
+  - Cache metadata storage
+  - File read/write operations
+
+• Metadata Handling
+  - Response headers capture
+  - Status code tracking
+  - Cache metadata
+  - Session tracking
+  - Timestamp management
+
--- a/docs/details/features.md
+++ b/docs/details/features.md
@@ -0,0 +1,150 @@
+### 1. Basic Web Crawling
+```python
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(url="https://example.com")
+    print(result.markdown)  # Get clean markdown content
+    print(result.html)      # Get raw HTML
+    print(result.cleaned_html)  # Get cleaned HTML
+```
+
+### 2. Browser Control Options
+- Multiple Browser Support
+```python
+# Choose between different browser engines
+crawler = AsyncWebCrawler(browser_type="firefox")  # or "chromium", "webkit"
+crawler = AsyncWebCrawler(headless=False)  # For visible browser
+```
+
+- Proxy Configuration
+```python
+crawler = AsyncWebCrawler(proxy="http://proxy.example.com:8080")
+# Or with authentication
+crawler = AsyncWebCrawler(proxy_config={
+    "server": "http://proxy.example.com:8080",
+    "username": "user",
+    "password": "pass"
+})
+```
+
+### 3. Content Selection & Filtering
+- CSS Selector Support
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    css_selector=".main-content"  # Extract specific content
+)
+```
+
+- Content Filtering Options
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    word_count_threshold=10,  # Minimum words per block
+    excluded_tags=['form', 'header'],  # Tags to exclude
+    exclude_external_links=True,  # Remove external links
+    exclude_social_media_links=True,  # Remove social media links
+    exclude_external_images=True  # Remove external images
+)
+```
+
+### 4. Dynamic Content Handling
+- JavaScript Execution
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    js_code="window.scrollTo(0, document.body.scrollHeight)"  # Execute custom JS
+)
+```
+
+- Wait Conditions
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    wait_for="css:.my-element",  # Wait for element
+    wait_for="js:() => document.readyState === 'complete'"  # Wait for condition
+)
+```
+
+### 5. Anti-Bot Protection Handling
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    simulate_user=True,  # Simulate human behavior
+    override_navigator=True,  # Mask automation signals
+    magic=True  # Enable all anti-detection features
+)
+```
+
+### 6. Session Management
+```python
+session_id = "my_session"
+result1 = await crawler.arun(url="https://example.com/page1", session_id=session_id)
+result2 = await crawler.arun(url="https://example.com/page2", session_id=session_id)
+await crawler.crawler_strategy.kill_session(session_id)
+```
+
+### 7. Media Handling
+- Screenshot Capture
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    screenshot=True
+)
+base64_screenshot = result.screenshot
+```
+
+- Media Extraction
+```python
+result = await crawler.arun(url="https://example.com")
+print(result.media['images'])  # List of images
+print(result.media['videos'])  # List of videos
+print(result.media['audios'])  # List of audio files
+```
+
+### 8. Structured Data Extraction
+- CSS-based Extraction
+```python
+schema = {
+    "name": "News Articles",
+    "baseSelector": "article",
+    "fields": [
+        {"name": "title", "selector": "h1", "type": "text"},
+        {"name": "date", "selector": ".date", "type": "text"}
+    ]
+}
+extraction_strategy = JsonCssExtractionStrategy(schema)
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=extraction_strategy
+)
+structured_data = json.loads(result.extracted_content)
+```
+
+- LLM-based Extraction (Multiple Providers)
+```python
+class NewsArticle(BaseModel):
+    title: str
+    summary: str
+
+strategy = LLMExtractionStrategy(
+    provider="ollama/nemotron",  # or "huggingface/...", "ollama/..."
+    api_token="your-token",
+    schema=NewsArticle.schema(),
+    instruction="Extract news article details..."
+)
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=strategy
+)
+```
+
+### 9. Content Cleaning & Processing
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    remove_overlay_elements=True,  # Remove popups/modals
+    process_iframes=True,  # Process iframe content
+)
+print(result.fit_markdown)  # Get most relevant content
+print(result.fit_html)     # Get cleaned HTML
+```
--- a/docs/details/features_details.md
+++ b/docs/details/features_details.md
@@ -0,0 +1,457 @@
+I'll expand the outline with detailed descriptions and examples based on all the provided files. I'll start with the first few sections:
+
+### 1. Basic Web Crawling
+Basic web crawling provides the foundation for extracting content from websites. The library supports both simple single-page crawling and recursive website crawling.
+
+```python
+# Simple page crawling
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(url="https://example.com")
+    print(result.html)        # Raw HTML
+    print(result.markdown)    # Cleaned markdown
+    print(result.cleaned_html)  # Cleaned HTML
+
+# Recursive website crawling
+class SimpleWebsiteScraper:
+    def __init__(self, crawler: AsyncWebCrawler):
+        self.crawler = crawler
+
+    async def scrape(self, start_url: str, max_depth: int):
+        results = await self.scrape_recursive(start_url, max_depth)
+        return results
+
+# Usage
+async with AsyncWebCrawler() as crawler:
+    scraper = SimpleWebsiteScraper(crawler)
+    results = await scraper.scrape("https://example.com", depth=2)
+```
+
+### 2. Browser Control Options
+The library provides extensive control over browser behavior, allowing customization of browser type, headless mode, and proxy settings.
+
+```python
+# Browser Type Selection
+async with AsyncWebCrawler(
+    browser_type="firefox",  # Options: "chromium", "firefox", "webkit"
+    headless=False,         # For visible browser
+    verbose=True           # Enable logging
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+
+# Proxy Configuration
+async with AsyncWebCrawler(
+    proxy_config={
+        "server": "http://proxy.example.com:8080",
+        "username": "user",
+        "password": "pass"
+    },
+    headers={
+        "User-Agent": "Custom User Agent",
+        "Accept-Language": "en-US,en;q=0.9"
+    }
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+### 3. Content Selection & Filtering
+The library offers multiple ways to select and filter content, from CSS selectors to word count thresholds.
+
+```python
+# CSS Selector and Content Filtering
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        css_selector="article.main-content",  # Extract specific content
+        word_count_threshold=10,              # Minimum words per block
+        excluded_tags=['form', 'header'],     # Tags to exclude
+        exclude_external_links=True,          # Remove external links
+        exclude_social_media_links=True,      # Remove social media links
+        exclude_domains=["pinterest.com", "facebook.com"]  # Exclude specific domains
+    )
+
+# Custom HTML to Text Options
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        html2text={
+            "escape_dot": False,
+            "links_each_paragraph": True,
+            "protect_links": True
+        }
+    )
+```
+
+### 4. Dynamic Content Handling
+The library provides sophisticated handling of dynamic content with JavaScript execution and wait conditions.
+
+```python
+# JavaScript Execution and Wait Conditions
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        js_code=[
+            "window.scrollTo(0, document.body.scrollHeight);",
+            "document.querySelector('.load-more').click();"
+        ],
+        wait_for="css:.dynamic-content",  # Wait for element
+        delay_before_return_html=2.0      # Wait after JS execution
+    )
+
+# Smart Wait Conditions
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        wait_for="""() => {
+            return document.querySelectorAll('.item').length > 10;
+        }""",
+        page_timeout=60000  # 60 seconds timeout
+    )
+```
+
+### 5. Advanced Link Analysis
+The library provides comprehensive link analysis capabilities, distinguishing between internal and external links, with options for filtering and processing.
+
+```python
+# Basic Link Analysis
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(url="https://example.com")
+    
+    # Access internal and external links
+    for internal_link in result.links['internal']:
+        print(f"Internal: {internal_link['href']} - {internal_link['text']}")
+    
+    for external_link in result.links['external']:
+        print(f"External: {external_link['href']} - {external_link['text']}")
+
+# Advanced Link Filtering
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        exclude_external_links=True,          # Remove all external links
+        exclude_social_media_links=True,      # Remove social media links
+        exclude_social_media_domains=[                # Custom social media domains
+            "facebook.com", "twitter.com", "instagram.com"
+        ],
+        exclude_domains=["pinterest.com"]     # Specific domains to exclude
+    )
+```
+
+### 6. Anti-Bot Protection Handling
+The library includes sophisticated anti-detection mechanisms to handle websites with bot protection.
+
+```python
+# Basic Anti-Detection
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        simulate_user=True,        # Simulate human behavior
+        override_navigator=True    # Override navigator properties
+    )
+
+# Advanced Anti-Detection with Magic Mode
+async with AsyncWebCrawler(headless=False) as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        magic=True,               # Enable all anti-detection features
+        remove_overlay_elements=True,  # Remove popups/modals automatically
+        # Custom navigator properties
+        js_code="""
+        Object.defineProperty(navigator, 'webdriver', {
+            get: () => undefined
+        });
+        """
+    )
+```
+
+### 7. Session Management
+Session management allows maintaining state across multiple requests and handling cookies.
+
+```python
+# Basic Session Management
+async with AsyncWebCrawler() as crawler:
+    session_id = "my_session"
+    
+    # Login
+    login_result = await crawler.arun(
+        url="https://example.com/login",
+        session_id=session_id,
+        js_code="document.querySelector('form').submit();"
+    )
+    
+    # Use same session for subsequent requests
+    protected_result = await crawler.arun(
+        url="https://example.com/protected",
+        session_id=session_id
+    )
+    
+    # Clean up session
+    await crawler.crawler_strategy.kill_session(session_id)
+
+# Advanced Session with Custom Cookies
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        session_id="custom_session",
+        cookies=[{
+            "name": "sessionId",
+            "value": "abc123",
+            "domain": "example.com"
+        }]
+    )
+```
+
+### 8. Screenshot and Media Handling
+The library provides comprehensive media handling capabilities, including screenshots and media content extraction.
+
+```python
+# Screenshot Capture
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        screenshot=True,
+        screenshot_wait_for=2.0  # Wait before taking screenshot
+    )
+    
+    # Save screenshot
+    if result.screenshot:
+        with open("screenshot.png", "wb") as f:
+            f.write(base64.b64decode(result.screenshot))
+
+# Media Extraction
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(url="https://example.com")
+    
+    # Process images with metadata
+    for image in result.media['images']:
+        print(f"Image: {image['src']}")
+        print(f"Alt text: {image['alt']}")
+        print(f"Context: {image['desc']}")
+        print(f"Relevance score: {image['score']}")
+    
+    # Process videos and audio
+    for video in result.media['videos']:
+        print(f"Video: {video['src']}")
+    for audio in result.media['audios']:
+        print(f"Audio: {audio['src']}")
+```
+
+### 9. Structured Data Extraction & Chunking
+The library supports multiple strategies for structured data extraction and content chunking.
+
+```python
+# LLM-based Extraction
+class NewsArticle(BaseModel):
+    title: str
+    content: str
+    author: str
+
+extraction_strategy = LLMExtractionStrategy(
+    provider='openai/gpt-4',
+    api_token="your-token",
+    schema=NewsArticle.schema(),
+    instruction="Extract news article details",
+    chunk_token_threshold=1000,
+    overlap_rate=0.1
+)
+
+# CSS-based Extraction
+schema = {
+    "name": "Product Listing",
+    "baseSelector": ".product-card",
+    "fields": [
+        {
+            "name": "title",
+            "selector": "h2",
+            "type": "text"
+        },
+        {
+            "name": "price",
+            "selector": ".price",
+            "type": "text",
+            "transform": "strip"
+        }
+    ]
+}
+
+css_strategy = JsonCssExtractionStrategy(schema)
+
+# Text Chunking
+from crawl4ai.chunking_strategy import OverlappingWindowChunking
+
+chunking_strategy = OverlappingWindowChunking(
+    window_size=1000,
+    overlap=100
+)
+
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        extraction_strategy=extraction_strategy,
+        chunking_strategy=chunking_strategy
+    )
+```
+
+
+### 10. Content Cleaning & Processing
+The library provides extensive content cleaning and processing capabilities, ensuring high-quality output in various formats.
+
+```python
+# Basic Content Cleaning
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        remove_overlay_elements=True,  # Remove popups/modals
+        process_iframes=True,          # Process iframe content
+        word_count_threshold=10        # Minimum words per block
+    )
+    
+    print(result.cleaned_html)    # Clean HTML
+    print(result.fit_html)        # Most relevant HTML content
+    print(result.fit_markdown)    # Most relevant markdown content
+
+# Advanced Content Processing
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        excluded_tags=['form', 'header', 'footer', 'nav'],
+        html2text={
+            "escape_dot": False,
+            "body_width": 0,
+            "protect_links": True,
+            "unicode_snob": True,
+            "ignore_links": False,
+            "ignore_images": False,
+            "ignore_emphasis": False,
+            "bypass_tables": False,
+            "ignore_tables": False
+        }
+    )
+```
+
+### Advanced Usage Patterns
+
+#### 1. Combining Multiple Features
+```python
+async with AsyncWebCrawler(
+    browser_type="chromium",
+    headless=False,
+    verbose=True
+) as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        # Anti-bot measures
+        magic=True,
+        simulate_user=True,
+        
+        # Content selection
+        css_selector="article.main",
+        word_count_threshold=10,
+        
+        # Dynamic content handling
+        js_code="window.scrollTo(0, document.body.scrollHeight);",
+        wait_for="css:.dynamic-content",
+        
+        # Content filtering
+        exclude_external_links=True,
+        exclude_social_media_links=True,
+        
+        # Media handling
+        screenshot=True,
+        process_iframes=True,
+        
+        # Content cleaning
+        remove_overlay_elements=True
+    )
+```
+
+#### 2. Custom Extraction Pipeline
+```python
+# Define custom schemas and strategies
+class Article(BaseModel):
+    title: str
+    content: str
+    date: str
+
+# CSS extraction for initial content
+css_schema = {
+    "name": "Article Extraction",
+    "baseSelector": "article",
+    "fields": [
+        {"name": "title", "selector": "h1", "type": "text"},
+        {"name": "content", "selector": ".content", "type": "html"},
+        {"name": "date", "selector": ".date", "type": "text"}
+    ]
+}
+
+# LLM processing for semantic analysis
+llm_strategy = LLMExtractionStrategy(
+    provider="ollama/nemotron",
+    api_token="your-token",
+    schema=Article.schema(),
+    instruction="Extract and clean article content"
+)
+
+# Chunking strategy for large content
+chunking = OverlappingWindowChunking(window_size=1000, overlap=100)
+
+async with AsyncWebCrawler() as crawler:
+    # First pass: Extract structure
+    css_result = await crawler.arun(
+        url="https://example.com",
+        extraction_strategy=JsonCssExtractionStrategy(css_schema)
+    )
+    
+    # Second pass: Semantic processing
+    llm_result = await crawler.arun(
+        url="https://example.com",
+        extraction_strategy=llm_strategy,
+        chunking_strategy=chunking
+    )
+```
+
+#### 3. Website Crawling with Custom Processing
+```python
+class CustomWebsiteCrawler:
+    def __init__(self, crawler: AsyncWebCrawler):
+        self.crawler = crawler
+        self.results = {}
+
+    async def process_page(self, url: str) -> Dict:
+        result = await self.crawler.arun(
+            url=url,
+            magic=True,
+            word_count_threshold=10,
+            exclude_external_links=True,
+            process_iframes=True,
+            remove_overlay_elements=True
+        )
+        
+        # Process internal links
+        internal_links = [
+            link['href'] for link in result.links['internal']
+            if self._is_valid_link(link['href'])
+        ]
+        
+        # Extract media
+        media_urls = [img['src'] for img in result.media['images']]
+        
+        return {
+            'content': result.markdown,
+            'links': internal_links,
+            'media': media_urls,
+            'metadata': result.metadata
+        }
+
+    async def crawl_website(self, start_url: str, max_depth: int = 2):
+        visited = set()
+        queue = [(start_url, 0)]
+        
+        while queue:
+            url, depth = queue.pop(0)
+            if depth > max_depth or url in visited:
+                continue
+                
+            visited.add(url)
+            self.results[url] = await self.process_page(url)
+```
+
--- a/docs/details/input_output.md
+++ b/docs/details/input_output.md
@@ -0,0 +1,282 @@
+### AsyncWebCrawler Constructor Parameters
+```python
+AsyncWebCrawler(
+    # Core Browser Settings
+    browser_type: str = "chromium",  # Options: "chromium", "firefox", "webkit"
+    headless: bool = True,  # Whether to run browser in headless mode
+    verbose: bool = False,  # Enable verbose logging
+    
+    # Cache Settings
+    always_by_pass_cache: bool = False,  # Always bypass cache regardless of run settings
+    base_directory: str = str(Path.home()),  # Base directory for cache storage
+    
+    # Network Settings
+    proxy: str = None,  # Simple proxy URL (e.g., "http://proxy.example.com:8080")
+    proxy_config: Dict = None,  # Advanced proxy settings with auth: {"server": str, "username": str, "password": str}
+    
+    # Browser Behavior
+    sleep_on_close: bool = False,  # Wait before closing browser
+    
+    # Other Settings passed to AsyncPlaywrightCrawlerStrategy
+    user_agent: str = None,  # Custom user agent string
+    headers: Dict[str, str] = {},  # Custom HTTP headers
+    js_code: Union[str, List[str]] = None,  # Default JavaScript to execute
+)
+```
+
+### arun() Method Parameters
+```python
+arun(
+    # Core Parameters
+    url: str,  # Required: URL to crawl
+    
+    # Content Selection
+    css_selector: str = None,  # CSS selector to extract specific content
+    word_count_threshold: int = MIN_WORD_THRESHOLD,  # Minimum words for content blocks
+    
+    # Cache Control
+    bypass_cache: bool = False,  # Bypass cache for this request
+    
+    # Session Management
+    session_id: str = None,  # Session identifier for persistent browsing
+    
+    # Screenshot Options
+    screenshot: bool = False,  # Take page screenshot
+    screenshot_wait_for: float = None,  # Wait time before screenshot
+    
+    # Content Processing
+    process_iframes: bool = False,  # Process iframe content
+    remove_overlay_elements: bool = False,  # Remove popups/modals
+    
+    # Anti-Bot/Detection
+    simulate_user: bool = False,  # Simulate human-like behavior
+    override_navigator: bool = False,  # Override navigator properties
+    magic: bool = False,  # Enable all anti-detection features
+    
+    # Content Filtering
+    excluded_tags: List[str] = None,  # HTML tags to exclude
+    exclude_external_links: bool = False,  # Remove external links
+    exclude_social_media_links: bool = False,  # Remove social media links
+    exclude_external_images: bool = False,  # Remove external images
+    exclude_social_media_domains: List[str] = None,  # Additional social media domains to exclude
+    remove_forms: bool = False,  # Remove all form elements
+    
+    # JavaScript Handling
+    js_code: Union[str, List[str]] = None,  # JavaScript to execute
+    js_only: bool = False,  # Only execute JavaScript without reloading page
+    wait_for: str = None,  # Wait condition (CSS selector or JS function)
+    
+    # Page Loading
+    page_timeout: int = 60000,  # Page load timeout in milliseconds
+    delay_before_return_html: float = None,  # Wait before returning HTML
+    
+    # Debug Options
+    log_console: bool = False,  # Log browser console messages
+    
+    # Content Format Control
+    only_text: bool = False,  # Extract only text content
+    keep_data_attributes: bool = False,  # Keep data-* attributes in HTML
+    
+    # Markdown Options
+    include_links_on_markdown: bool = False,  # Include links in markdown output
+    html2text: Dict = {},  # HTML to text conversion options
+    
+    # Extraction Strategy
+    extraction_strategy: ExtractionStrategy = None,  # Strategy for structured data extraction
+    
+    # Advanced Browser Control
+    user_agent: str = None,  # Override user agent for this request
+)
+```
+
+### Extraction Strategy Parameters
+```python
+# JsonCssExtractionStrategy
+{
+    "name": str,  # Name of extraction schema
+    "baseSelector": str,  # Base CSS selector
+    "fields": [
+        {
+            "name": str,  # Field name
+            "selector": str,  # CSS selector
+            "type": str,  # Data type ("text", etc.)
+            "transform": str = None  # Optional transformation
+        }
+    ]
+}
+
+# LLMExtractionStrategy
+{
+    "provider": str,  # LLM provider (e.g., "openai/gpt-4", "huggingface/...", "ollama/...")
+    "api_token": str,  # API token
+    "schema": dict,  # Pydantic model schema
+    "extraction_type": str,  # Type of extraction ("schema", etc.)
+    "instruction": str,  # Extraction instruction
+    "extra_args": dict = None,  # Additional provider-specific arguments
+    "extra_headers": dict = None  # Additional HTTP headers
+}
+```
+
+### HTML to Text Conversion Options (html2text parameter)
+```python
+{
+    "escape_dot": bool = True,  # Escape dots in text
+    # Other html2text library options
+}
+```
+
+
+### CrawlResult Fields
+
+```python
+class CrawlResult(BaseModel):
+    # Basic Information
+    url: str  # The crawled URL
+    # Example: "https://example.com"
+    
+    success: bool  # Whether the crawl was successful
+    # Example: True/False
+    
+    status_code: Optional[int]  # HTTP status code
+    # Example: 200, 404, 500
+    
+    # Content Fields
+    html: str  # Raw HTML content
+    # Example: "<html><body>...</body></html>"
+    
+    cleaned_html: Optional[str]  # HTML after cleaning and processing
+    # Example: "<article><p>Clean content...</p></article>"
+    
+    fit_html: Optional[str]  # Most relevant HTML content after content cleaning strategy
+    # Example: "<div><p>Most relevant content...</p></div>"
+    
+    markdown: Optional[str]  # HTML converted to markdown
+    # Example: "# Title\n\nContent paragraph..."
+    
+    fit_markdown: Optional[str]  # Most relevant content in markdown
+    # Example: "# Main Article\n\nKey content..."
+    
+    # Media Content
+    media: Dict[str, List[Dict]] = {}  # Extracted media information
+    # Example: {
+    #     "images": [
+    #         {
+    #             "src": "https://example.com/image.jpg",
+    #             "alt": "Image description",
+    #             "desc": "Contextual description",
+    #             "score": 5,  # Relevance score
+    #             "type": "image"
+    #         }
+    #     ],
+    #     "videos": [
+    #         {
+    #             "src": "https://example.com/video.mp4",
+    #             "alt": "Video title",
+    #             "type": "video",
+    #             "description": "Video context"
+    #         }
+    #     ],
+    #     "audios": [
+    #         {
+    #             "src": "https://example.com/audio.mp3",
+    #             "alt": "Audio title",
+    #             "type": "audio",
+    #             "description": "Audio context"
+    #         }
+    #     ]
+    # }
+    
+    # Link Information
+    links: Dict[str, List[Dict]] = {}  # Extracted links
+    # Example: {
+    #     "internal": [
+    #         {
+    #             "href": "https://example.com/page",
+    #             "text": "Link text",
+    #             "title": "Link title"
+    #         }
+    #     ],
+    #     "external": [
+    #         {
+    #             "href": "https://external.com",
+    #             "text": "External link text",
+    #             "title": "External link title"
+    #         }
+    #     ]
+    # }
+    
+    # Extraction Results
+    extracted_content: Optional[str]  # Content from extraction strategy
+    # Example for JsonCssExtractionStrategy:
+    # '[{"title": "Article 1", "date": "2024-03-20"}, ...]'
+    # Example for LLMExtractionStrategy:
+    # '{"entities": [...], "relationships": [...]}'
+    
+    # Additional Information
+    metadata: Optional[dict] = None  # Page metadata
+    # Example: {
+    #     "title": "Page Title",
+    #     "description": "Meta description",
+    #     "keywords": ["keyword1", "keyword2"],
+    #     "author": "Author Name",
+    #     "published_date": "2024-03-20"
+    # }
+    
+    screenshot: Optional[str] = None  # Base64 encoded screenshot
+    # Example: "iVBORw0KGgoAAAANSUhEUgAA..."
+    
+    error_message: Optional[str] = None  # Error message if crawl failed
+    # Example: "Failed to load page: timeout"
+    
+    session_id: Optional[str] = None  # Session identifier
+    # Example: "session_123456"
+    
+    response_headers: Optional[dict] = None  # HTTP response headers
+    # Example: {
+    #     "content-type": "text/html",
+    #     "server": "nginx/1.18.0",
+    #     "date": "Wed, 20 Mar 2024 12:00:00 GMT"
+    # }
+```
+
+### Common Usage Patterns:
+
+1. Basic Content Extraction:
+```python
+result = await crawler.arun(url="https://example.com")
+print(result.markdown)  # Clean, readable content
+print(result.cleaned_html)  # Cleaned HTML
+```
+
+2. Media Analysis:
+```python
+result = await crawler.arun(url="https://example.com")
+for image in result.media["images"]:
+    if image["score"] > 3:  # High-relevance images
+        print(f"High-quality image: {image['src']}")
+```
+
+3. Link Analysis:
+```python
+result = await crawler.arun(url="https://example.com")
+internal_links = [link["href"] for link in result.links["internal"]]
+external_links = [link["href"] for link in result.links["external"]]
+```
+
+4. Structured Data Extraction:
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=my_strategy
+)
+structured_data = json.loads(result.extracted_content)
+```
+
+5. Error Handling:
+```python
+result = await crawler.arun(url="https://example.com")
+if not result.success:
+    print(f"Crawl failed: {result.error_message}")
+    print(f"Status code: {result.status_code}")
+```
+
--- a/docs/details/realworld_examples.md
+++ b/docs/details/realworld_examples.md
@@ -0,0 +1,67 @@
+1. **E-commerce Product Monitor**
+   - Scraping product details from multiple e-commerce sites
+   - Price tracking with structured data extraction
+   - Handling dynamic content and anti-bot measures
+   - Features: JsonCssExtraction, session management, anti-bot
+
+2. **News Aggregator & Summarizer**
+   - Crawling news websites
+   - Content extraction and summarization
+   - Topic classification
+   - Features: LLMExtraction, CosineStrategy, content cleaning
+
+3. **Academic Paper Research Assistant**
+   - Crawling research papers from academic sites
+   - Extracting citations and references
+   - Building knowledge graphs
+   - Features: structured extraction, link analysis, chunking
+
+4. **Social Media Content Analyzer**
+   - Handling JavaScript-heavy sites
+   - Dynamic content loading
+   - Sentiment analysis integration
+   - Features: dynamic content handling, session management
+
+5. **Real Estate Market Analyzer**
+   - Scraping property listings
+   - Processing image galleries
+   - Geolocation data extraction
+   - Features: media handling, structured data extraction
+
+6. **Documentation Site Generator**
+   - Recursive website crawling
+   - Markdown generation
+   - Link validation
+   - Features: website crawling, content cleaning
+
+7. **Job Board Aggregator**
+   - Handling pagination
+   - Structured job data extraction
+   - Filtering and categorization
+   - Features: session management, JsonCssExtraction
+
+8. **Recipe Database Builder**
+   - Schema-based extraction
+   - Image processing
+   - Ingredient parsing
+   - Features: structured extraction, media handling
+
+9. **Travel Blog Content Analyzer**
+   - Location extraction
+   - Image and map processing
+   - Content categorization
+   - Features: CosineStrategy, media handling
+
+10. **Technical Documentation Scraper**
+    - API documentation extraction
+    - Code snippet processing
+    - Version tracking
+    - Features: content cleaning, structured extraction
+
+Each example will include:
+- Problem description
+- Technical requirements
+- Complete implementation
+- Error handling
+- Output processing
+- Performance considerations
--- a/docs/examples/quickstart_async.py
+++ b/docs/examples/quickstart_async.py
@@ -456,7 +456,6 @@ async def speed_comparison():
    print("If you run these tests in an environment with better network conditions,")
    print("you may observe an even more significant speed advantage for Crawl4AI.")

-
 async def generate_knowledge_graph():
    class Entity(BaseModel):
        name: str
@@ -473,8 +472,8 @@ async def generate_knowledge_graph():
        relationships: List[Relationship]

    extraction_strategy = LLMExtractionStrategy(
-            provider='openai/gpt-4o-mini',
-            api_token=os.getenv('OPENAI_API_KEY'),
+            provider='openai/gpt-4o-mini', # Or any other provider, including Ollama and open source models
+            api_token=os.getenv('OPENAI_API_KEY'), # In case of Ollama just pass "no-token"
            schema=KnowledgeGraph.model_json_schema(),
            extraction_type="schema",
            instruction="""Extract entities and relationships from the given text."""
@@ -491,6 +490,23 @@ async def generate_knowledge_graph():
        with open(os.path.join(__location__, "kb.json"), "w") as f:
            f.write(result.extracted_content)

+async def fit_markdown_remove_overlay():
+    async with AsyncWebCrawler(headless = False) as crawler:
+        url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
+        result = await crawler.arun(
+            url=url,
+            bypass_cache=True,
+            word_count_threshold = 10,
+            remove_overlay_elements=True,
+            screenshot = True
+        )
+        # Save markdown to file
+        with open(os.path.join(__location__, "mexico_places.md"), "w") as f:
+            f.write(result.fit_markdown)
+
+    print("Done")
+
+
 async def main():
    await simple_crawl()
    await simple_example_with_running_js_code()
--- a/docs/md_v0/api/core_classes_and_functions.md
+++ b/docs/md_v0/api/core_classes_and_functions.md
--- a/docs/md_v0/api/detailed_api_documentation.md
+++ b/docs/md_v0/api/detailed_api_documentation.md
--- a/docs/md_v0/assets/DankMono-Bold.woff2
+++ b/docs/md_v0/assets/DankMono-Bold.woff2
--- a/docs/md_v0/assets/DankMono-Italic.woff2
+++ b/docs/md_v0/assets/DankMono-Italic.woff2
--- a/docs/md_v0/assets/DankMono-Regular.woff2
+++ b/docs/md_v0/assets/DankMono-Regular.woff2
--- a/docs/md_v0/assets/Monaco.woff
+++ b/docs/md_v0/assets/Monaco.woff
--- a/docs/md_v0/assets/dmvendor.css
+++ b/docs/md_v0/assets/dmvendor.css
--- a/docs/md_v0/assets/highlight.css
+++ b/docs/md_v0/assets/highlight.css
--- a/docs/md_v0/assets/highlight.min.js
+++ b/docs/md_v0/assets/highlight.min.js
--- a/docs/md_v0/assets/highlight_init.js
+++ b/docs/md_v0/assets/highlight_init.js
--- a/docs/md_v0/assets/styles.css
+++ b/docs/md_v0/assets/styles.css
--- a/docs/md_v0/changelog.md
+++ b/docs/md_v0/changelog.md
--- a/docs/md_v0/chunking_strategies.json
+++ b/docs/md_v0/chunking_strategies.json
--- a/docs/md_v0/contact.md
+++ b/docs/md_v0/contact.md
--- a/docs/md_v0/demo.md
+++ b/docs/md_v0/demo.md
--- a/docs/md_v0/examples/hooks_auth.md
+++ b/docs/md_v0/examples/hooks_auth.md
--- a/docs/md_v0/examples/index.md
+++ b/docs/md_v0/examples/index.md
--- a/docs/md_v0/examples/js_execution_css_filtering.md
+++ b/docs/md_v0/examples/js_execution_css_filtering.md
--- a/docs/md_v0/examples/llm_extraction.md
+++ b/docs/md_v0/examples/llm_extraction.md
--- a/docs/md_v0/examples/research_assistant.md
+++ b/docs/md_v0/examples/research_assistant.md
--- a/docs/md_v0/examples/summarization.md
+++ b/docs/md_v0/examples/summarization.md
--- a/docs/md_v0/extraction_strategies.json
+++ b/docs/md_v0/extraction_strategies.json
--- a/docs/md_v0/full_details/advanced_features.md
+++ b/docs/md_v0/full_details/advanced_features.md
--- a/docs/md_v0/full_details/chunking_strategies.md
+++ b/docs/md_v0/full_details/chunking_strategies.md
--- a/docs/md_v0/full_details/crawl_request_parameters.md
+++ b/docs/md_v0/full_details/crawl_request_parameters.md
--- a/docs/md_v0/full_details/crawl_result_class.md
+++ b/docs/md_v0/full_details/crawl_result_class.md
--- a/docs/md_v0/full_details/extraction_strategies.md
+++ b/docs/md_v0/full_details/extraction_strategies.md
--- a/docs/md_v0/index.md
+++ b/docs/md_v0/index.md
--- a/docs/md_v0/installation.md
+++ b/docs/md_v0/installation.md
--- a/docs/md_v0/interactive_content.html
+++ b/docs/md_v0/interactive_content.html
--- a/docs/md_v0/introduction.md
+++ b/docs/md_v0/introduction.md
--- a/docs/md_v0/quickstart.md
+++ b/docs/md_v0/quickstart.md
--- a/docs/md_v1/api/core_classes_and_functions.md
+++ b/docs/md_v1/api/core_classes_and_functions.md
--- a/docs/md_v1/api/detailed_api_documentation.md
+++ b/docs/md_v1/api/detailed_api_documentation.md
--- a/docs/md_v1/assets/DankMono-Bold.woff2
+++ b/docs/md_v1/assets/DankMono-Bold.woff2
--- a/docs/md_v1/assets/DankMono-Italic.woff2
+++ b/docs/md_v1/assets/DankMono-Italic.woff2
--- a/docs/md_v1/assets/DankMono-Regular.woff2
+++ b/docs/md_v1/assets/DankMono-Regular.woff2
--- a/docs/md_v1/assets/Monaco.woff
+++ b/docs/md_v1/assets/Monaco.woff
--- a/docs/md_v1/assets/dmvendor.css
+++ b/docs/md_v1/assets/dmvendor.css
--- a/docs/md_v1/assets/highlight.css
+++ b/docs/md_v1/assets/highlight.css
--- a/docs/md_v1/assets/highlight.min.js
+++ b/docs/md_v1/assets/highlight.min.js
--- a/docs/md_v1/assets/highlight_init.js
+++ b/docs/md_v1/assets/highlight_init.js
--- a/docs/md_v1/assets/styles.css
+++ b/docs/md_v1/assets/styles.css
--- a/docs/md_v1/changelog.md
+++ b/docs/md_v1/changelog.md
--- a/docs/md_v1/contact.md
+++ b/docs/md_v1/contact.md
--- a/docs/md_v1/demo.md
+++ b/docs/md_v1/demo.md
--- a/docs/md_v1/examples/hooks_auth.md
+++ b/docs/md_v1/examples/hooks_auth.md
--- a/docs/md_v1/examples/index.md
+++ b/docs/md_v1/examples/index.md
--- a/docs/md_v1/examples/js_execution_css_filtering.md
+++ b/docs/md_v1/examples/js_execution_css_filtering.md
--- a/docs/md_v1/examples/json_css_extraction.md
+++ b/docs/md_v1/examples/json_css_extraction.md
--- a/docs/md_v1/examples/llm_extraction.md
+++ b/docs/md_v1/examples/llm_extraction.md
--- a/docs/md_v1/examples/research_assistant.md
+++ b/docs/md_v1/examples/research_assistant.md
--- a/docs/md_v1/examples/summarization.md
+++ b/docs/md_v1/examples/summarization.md
--- a/docs/md_v1/full_details/advanced_features.md
+++ b/docs/md_v1/full_details/advanced_features.md
--- a/docs/md_v1/full_details/advanced_jsoncss_extraction.md
+++ b/docs/md_v1/full_details/advanced_jsoncss_extraction.md
--- a/docs/md_v1/full_details/chunking_strategies.md
+++ b/docs/md_v1/full_details/chunking_strategies.md
--- a/docs/md_v1/full_details/crawl_request_parameters.md
+++ b/docs/md_v1/full_details/crawl_request_parameters.md
--- a/docs/md_v1/full_details/crawl_result_class.md
+++ b/docs/md_v1/full_details/crawl_result_class.md
--- a/docs/md_v1/full_details/extraction_strategies.md
+++ b/docs/md_v1/full_details/extraction_strategies.md
--- a/docs/md_v1/full_details/session_based_crawling.md
+++ b/docs/md_v1/full_details/session_based_crawling.md
--- a/docs/md_v1/index.md
+++ b/docs/md_v1/index.md
--- a/docs/md_v1/installation.md
+++ b/docs/md_v1/installation.md
--- a/docs/md_v1/interactive_content.html
+++ b/docs/md_v1/interactive_content.html
--- a/docs/md_v1/introduction.md
+++ b/docs/md_v1/introduction.md
--- a/docs/md_v1/mkdocs.yml
+++ b/docs/md_v1/mkdocs.yml
@@ -0,0 +1,45 @@
+site_name: Crawl4AI Documentation
+site_description: 🔥🕷️ Crawl4AI, Open-source LLM Friendly Web Crawler & Scrapper
+site_url: https://docs.crawl4ai.com
+repo_url: https://github.com/unclecode/crawl4ai
+repo_name: unclecode/crawl4ai
+docs_dir: docs/md
+nav:
+  - Home: index.md
+  - First Steps:
+      - Introduction: introduction.md
+      - Installation: installation.md
+      - Quick Start: quickstart.md
+  - Examples:
+      - Intro: examples/index.md
+      - Structured Data Extraction: examples/json_css_extraction.md
+      - LLM Extraction: examples/llm_extraction.md
+      - JS Execution & CSS Filtering: examples/js_execution_css_filtering.md
+      - Hooks & Auth: examples/hooks_auth.md
+      - Summarization: examples/summarization.md
+      - Research Assistant: examples/research_assistant.md
+  - Full Details of Using Crawler:
+      - Crawl Request Parameters: full_details/crawl_request_parameters.md
+      - Crawl Result Class: full_details/crawl_result_class.md
+      - Session Based Crawling: full_details/session_based_crawling.md
+      - Advanced Features: full_details/advanced_features.md
+      - Advanced JsonCssExtraction: full_details/advanced_jsoncss_extraction.md
+      - Chunking Strategies: full_details/chunking_strategies.md
+      - Extraction Strategies: full_details/extraction_strategies.md
+  - Miscellaneous:
+      - Change Log: changelog.md
+      - Contact: contact.md
+
+theme:
+  name: terminal
+  palette: dark
+
+# Add the css/extra.css
+extra_css:
+  - assets/styles.css
+  - assets/highlight.css
+  - assets/dmvendor.css
+
+extra_javascript:
+  - assets/highlight.min.js
+  - assets/highlight_init.js
--- a/docs/md_v1/quickstart.md
+++ b/docs/md_v1/quickstart.md
@@ -147,8 +147,8 @@ async def main():
            url="https://openai.com/api/pricing/",
            word_count_threshold=1,
            extraction_strategy=LLMExtractionStrategy(
-                provider="openai/gpt-4o",
-                api_token=os.getenv("OPENAI_API_KEY"),
+                provider="openai/gpt-4o", # Or use open source model like "ollama/nemotron"
+                api_token=os.getenv("OPENAI_API_KEY"), # Pass "no-token" if using Ollama
                schema=OpenAIModelFee.schema(),
                extraction_type="schema",
                instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
--- a/docs/md_v2/advanced/content-processing.md
+++ b/docs/md_v2/advanced/content-processing.md
@@ -0,0 +1,223 @@
+# Content Processing
+
+Crawl4AI provides powerful content processing capabilities that help you extract clean, relevant content from web pages. This guide covers content cleaning, media handling, link analysis, and metadata extraction.
+
+## Content Cleaning
+
+### Understanding Clean Content
+When crawling web pages, you often encounter a lot of noise - advertisements, navigation menus, footers, popups, and other irrelevant content. Crawl4AI automatically cleans this noise using several approaches:
+
+1. **Basic Cleaning**: Removes unwanted HTML elements and attributes
+2. **Content Relevance**: Identifies and preserves meaningful content blocks
+3. **Layout Analysis**: Understands page structure to identify main content areas
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    word_count_threshold=10,        # Remove blocks with fewer words
+    excluded_tags=['form', 'nav'],  # Remove specific HTML tags
+    remove_overlay_elements=True    # Remove popups/modals
+)
+
+# Get clean content
+print(result.cleaned_html)  # Cleaned HTML
+print(result.markdown)      # Clean markdown version
+```
+
+### Fit Markdown: Smart Content Extraction
+One of Crawl4AI's most powerful features is `fit_markdown`. This feature uses advanced heuristics to identify and extract the main content from a webpage while excluding irrelevant elements.
+
+#### How Fit Markdown Works
+- Analyzes content density and distribution
+- Identifies content patterns and structures
+- Removes boilerplate content (headers, footers, sidebars)
+- Preserves the most relevant content blocks
+- Maintains content hierarchy and formatting
+
+#### Perfect For:
+- Blog posts and articles
+- News content
+- Documentation pages
+- Any page with a clear main content area
+
+#### Not Recommended For:
+- E-commerce product listings
+- Search results pages
+- Social media feeds
+- Pages with multiple equal-weight content sections
+
+```python
+result = await crawler.arun(url="https://example.com")
+
+# Get the most relevant content
+main_content = result.fit_markdown
+
+# Compare with regular markdown
+all_content = result.markdown
+
+print(f"Fit Markdown Length: {len(main_content)}")
+print(f"Regular Markdown Length: {len(all_content)}")
+```
+
+#### Example Use Case
+```python
+async def extract_article_content(url: str) -> str:
+    """Extract main article content from a blog or news site."""
+    async with AsyncWebCrawler() as crawler:
+        result = await crawler.arun(url=url)
+        
+        # fit_markdown will focus on the article content,
+        # excluding navigation, ads, and other distractions
+        return result.fit_markdown
+```
+
+## Media Processing
+
+Crawl4AI provides comprehensive media extraction and analysis capabilities. It automatically detects and processes various types of media elements while maintaining their context and relevance.
+
+### Image Processing
+The library handles various image scenarios, including:
+- Regular images
+- Lazy-loaded images
+- Background images
+- Responsive images
+- Image metadata and context
+
+```python
+result = await crawler.arun(url="https://example.com")
+
+for image in result.media["images"]:
+    # Each image includes rich metadata
+    print(f"Source: {image['src']}")
+    print(f"Alt text: {image['alt']}")
+    print(f"Description: {image['desc']}")
+    print(f"Context: {image['context']}")  # Surrounding text
+    print(f"Relevance score: {image['score']}")  # 0-10 score
+```
+
+### Handling Lazy-Loaded Content
+Crawl4aai already handles lazy loading for media elements. You can also customize the wait time for lazy-loaded content:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    wait_for="css:img[data-src]",  # Wait for lazy images
+    delay_before_return_html=2.0   # Additional wait time
+)
+```
+
+### Video and Audio Content
+The library extracts video and audio elements with their metadata:
+
+```python
+# Process videos
+for video in result.media["videos"]:
+    print(f"Video source: {video['src']}")
+    print(f"Type: {video['type']}")
+    print(f"Duration: {video.get('duration')}")
+    print(f"Thumbnail: {video.get('poster')}")
+
+# Process audio
+for audio in result.media["audios"]:
+    print(f"Audio source: {audio['src']}")
+    print(f"Type: {audio['type']}")
+    print(f"Duration: {audio.get('duration')}")
+```
+
+## Link Analysis
+
+Crawl4AI provides sophisticated link analysis capabilities, helping you understand the relationship between pages and identify important navigation patterns.
+
+### Link Classification
+The library automatically categorizes links into:
+- Internal links (same domain)
+- External links (different domains)
+- Social media links
+- Navigation links
+- Content links
+
+```python
+result = await crawler.arun(url="https://example.com")
+
+# Analyze internal links
+for link in result.links["internal"]:
+    print(f"Internal: {link['href']}")
+    print(f"Link text: {link['text']}")
+    print(f"Context: {link['context']}")  # Surrounding text
+    print(f"Type: {link['type']}")  # nav, content, etc.
+
+# Analyze external links
+for link in result.links["external"]:
+    print(f"External: {link['href']}")
+    print(f"Domain: {link['domain']}")
+    print(f"Type: {link['type']}")
+```
+
+### Smart Link Filtering
+Control which links are included in the results:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    exclude_external_links=True,          # Remove external links
+    exclude_social_media_links=True,      # Remove social media links
+    exclude_social_media_domains=[                # Custom social media domains
+        "facebook.com", "twitter.com", "instagram.com"
+    ],
+    exclude_domains=["ads.example.com"]   # Exclude specific domains
+)
+```
+
+## Metadata Extraction
+
+Crawl4AI automatically extracts and processes page metadata, providing valuable information about the content:
+
+```python
+result = await crawler.arun(url="https://example.com")
+
+metadata = result.metadata
+print(f"Title: {metadata['title']}")
+print(f"Description: {metadata['description']}")
+print(f"Keywords: {metadata['keywords']}")
+print(f"Author: {metadata['author']}")
+print(f"Published Date: {metadata['published_date']}")
+print(f"Modified Date: {metadata['modified_date']}")
+print(f"Language: {metadata['language']}")
+```
+
+## Best Practices
+
+1. **Use Fit Markdown for Articles**
+   ```python
+   # Perfect for blog posts, news articles, documentation
+   content = result.fit_markdown
+   ```
+
+2. **Handle Media Appropriately**
+   ```python
+   # Filter by relevance score
+   relevant_images = [
+       img for img in result.media["images"]
+       if img['score'] > 5
+   ]
+   ```
+
+3. **Combine Link Analysis with Content**
+   ```python
+   # Get content links with context
+   content_links = [
+       link for link in result.links["internal"]
+       if link['type'] == 'content'
+   ]
+   ```
+
+4. **Clean Content with Purpose**
+   ```python
+   # Customize cleaning based on your needs
+   result = await crawler.arun(
+       url=url,
+       word_count_threshold=20,      # Adjust based on content type
+       keep_data_attributes=False,   # Remove data attributes
+       process_iframes=True         # Include iframe content
+   )
+   ```
--- a/docs/md_v2/advanced/hooks-auth.md
+++ b/docs/md_v2/advanced/hooks-auth.md
@@ -0,0 +1,110 @@
+# Hooks & Auth for AsyncWebCrawler
+
+Crawl4AI's AsyncWebCrawler allows you to customize the behavior of the web crawler using hooks. Hooks are asynchronous functions that are called at specific points in the crawling process, allowing you to modify the crawler's behavior or perform additional actions. This example demonstrates how to use various hooks to customize the asynchronous crawling process.
+
+## Example: Using Crawler Hooks with AsyncWebCrawler
+
+Let's see how we can customize the AsyncWebCrawler using hooks! In this example, we'll:
+
+1. Configure the browser when it's created.
+2. Add custom headers before navigating to the URL.
+3. Log the current URL after navigation.
+4. Perform actions after JavaScript execution.
+5. Log the length of the HTML before returning it.
+
+### Hook Definitions
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
+from playwright.async_api import Page, Browser
+
+async def on_browser_created(browser: Browser):
+    print("[HOOK] on_browser_created")
+    # Example customization: set browser viewport size
+    context = await browser.new_context(viewport={'width': 1920, 'height': 1080})
+    page = await context.new_page()
+    
+    # Example customization: logging in to a hypothetical website
+    await page.goto('https://example.com/login')
+    await page.fill('input[name="username"]', 'testuser')
+    await page.fill('input[name="password"]', 'password123')
+    await page.click('button[type="submit"]')
+    await page.wait_for_selector('#welcome')
+    
+    # Add a custom cookie
+    await context.add_cookies([{'name': 'test_cookie', 'value': 'cookie_value', 'url': 'https://example.com'}])
+    
+    await page.close()
+    await context.close()
+
+async def before_goto(page: Page):
+    print("[HOOK] before_goto")
+    # Example customization: add custom headers
+    await page.set_extra_http_headers({'X-Test-Header': 'test'})
+
+async def after_goto(page: Page):
+    print("[HOOK] after_goto")
+    # Example customization: log the URL
+    print(f"Current URL: {page.url}")
+
+async def on_execution_started(page: Page):
+    print("[HOOK] on_execution_started")
+    # Example customization: perform actions after JS execution
+    await page.evaluate("console.log('Custom JS executed')")
+
+async def before_return_html(page: Page, html: str):
+    print("[HOOK] before_return_html")
+    # Example customization: log the HTML length
+    print(f"HTML length: {len(html)}")
+    return page
+```
+
+### Using the Hooks with the AsyncWebCrawler
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
+
+async def main():
+    print("\n🔗 Using Crawler Hooks: Let's see how we can customize the AsyncWebCrawler using hooks!")
+    
+    crawler_strategy = AsyncPlaywrightCrawlerStrategy(verbose=True)
+    crawler_strategy.set_hook('on_browser_created', on_browser_created)
+    crawler_strategy.set_hook('before_goto', before_goto)
+    crawler_strategy.set_hook('after_goto', after_goto)
+    crawler_strategy.set_hook('on_execution_started', on_execution_started)
+    crawler_strategy.set_hook('before_return_html', before_return_html)
+    
+    async with AsyncWebCrawler(verbose=True, crawler_strategy=crawler_strategy) as crawler:
+        result = await crawler.arun(
+            url="https://example.com",
+            js_code="window.scrollTo(0, document.body.scrollHeight);",
+            wait_for="footer"
+        )
+
+    print("📦 Crawler Hooks result:")
+    print(result)
+
+asyncio.run(main())
+```
+
+### Explanation
+
+- `on_browser_created`: This hook is called when the Playwright browser is created. It sets up the browser context, logs in to a website, and adds a custom cookie.
+- `before_goto`: This hook is called right before Playwright navigates to the URL. It adds custom HTTP headers.
+- `after_goto`: This hook is called after Playwright navigates to the URL. It logs the current URL.
+- `on_execution_started`: This hook is called after any custom JavaScript is executed. It performs additional JavaScript actions.
+- `before_return_html`: This hook is called before returning the HTML content. It logs the length of the HTML content.
+
+### Additional Ideas
+
+- **Handling authentication**: Use the `on_browser_created` hook to handle login processes or set authentication tokens.
+- **Dynamic header modification**: Modify headers based on the target URL or other conditions in the `before_goto` hook.
+- **Content verification**: Use the `after_goto` hook to verify that the expected content is present on the page.
+- **Custom JavaScript injection**: Inject and execute custom JavaScript using the `on_execution_started` hook.
+- **Content preprocessing**: Modify or analyze the HTML content in the `before_return_html` hook before it's returned.
+
+By using these hooks, you can customize the behavior of the AsyncWebCrawler to suit your specific needs, including handling authentication, modifying requests, and preprocessing content.
--- a/docs/md_v2/advanced/hooks.md
+++ b/docs/md_v2/advanced/hooks.md
--- a/docs/md_v2/advanced/magic-mode.md
+++ b/docs/md_v2/advanced/magic-mode.md
@@ -0,0 +1,52 @@
+# Magic Mode & Anti-Bot Protection
+
+Crawl4AI provides powerful anti-detection capabilities, with Magic Mode being the simplest and most comprehensive solution.
+
+## Magic Mode
+
+The easiest way to bypass anti-bot protections:
+
+```python
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        magic=True  # Enables all anti-detection features
+    )
+```
+
+Magic Mode automatically:
+- Masks browser automation signals
+- Simulates human-like behavior
+- Overrides navigator properties
+- Handles cookie consent popups
+- Manages browser fingerprinting
+- Randomizes timing patterns
+
+## Manual Anti-Bot Options
+
+While Magic Mode is recommended, you can also configure individual anti-detection features:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    simulate_user=True,        # Simulate human behavior
+    override_navigator=True    # Mask automation signals
+)
+```
+
+Note: When `magic=True` is used, you don't need to set these individual options.
+
+## Example: Handling Protected Sites
+
+```python
+async def crawl_protected_site(url: str):
+    async with AsyncWebCrawler(headless=True) as crawler:
+        result = await crawler.arun(
+            url=url,
+            magic=True,
+            remove_overlay_elements=True,  # Remove popups/modals
+            page_timeout=60000            # Increased timeout for protection checks
+        )
+        
+        return result.markdown if result.success else None
+```
--- a/docs/md_v2/advanced/proxy-security.md
+++ b/docs/md_v2/advanced/proxy-security.md
@@ -0,0 +1,84 @@
+# Proxy & Security
+
+Configure proxy settings and enhance security features in Crawl4AI for reliable data extraction.
+
+## Basic Proxy Setup
+
+Simple proxy configuration:
+
+```python
+# Using proxy URL
+async with AsyncWebCrawler(
+    proxy="http://proxy.example.com:8080"
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+
+# Using SOCKS proxy
+async with AsyncWebCrawler(
+    proxy="socks5://proxy.example.com:1080"
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Authenticated Proxy
+
+Use proxy with authentication:
+
+```python
+proxy_config = {
+    "server": "http://proxy.example.com:8080",
+    "username": "user",
+    "password": "pass"
+}
+
+async with AsyncWebCrawler(proxy_config=proxy_config) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Rotating Proxies
+
+Example using a proxy rotation service:
+
+```python
+async def get_next_proxy():
+    # Your proxy rotation logic here
+    return {"server": "http://next.proxy.com:8080"}
+
+async with AsyncWebCrawler() as crawler:
+    # Update proxy for each request
+    for url in urls:
+        proxy = await get_next_proxy()
+        crawler.update_proxy(proxy)
+        result = await crawler.arun(url=url)
+```
+
+## Custom Headers
+
+Add security-related headers:
+
+```python
+headers = {
+    "X-Forwarded-For": "203.0.113.195",
+    "Accept-Language": "en-US,en;q=0.9",
+    "Cache-Control": "no-cache",
+    "Pragma": "no-cache"
+}
+
+async with AsyncWebCrawler(headers=headers) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Combining with Magic Mode
+
+For maximum protection, combine proxy with Magic Mode:
+
+```python
+async with AsyncWebCrawler(
+    proxy="http://proxy.example.com:8080",
+    headers={"Accept-Language": "en-US"}
+) as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        magic=True  # Enable all anti-detection features
+    )
+```
--- a/docs/md_v2/advanced/session-management-advanced.md
+++ b/docs/md_v2/advanced/session-management-advanced.md
@@ -0,0 +1,276 @@
+# Session-Based Crawling for Dynamic Content
+
+In modern web applications, content is often loaded dynamically without changing the URL. Examples include "Load More" buttons, infinite scrolling, or paginated content that updates via JavaScript. To effectively crawl such websites, Crawl4AI provides powerful session-based crawling capabilities.
+
+This guide will explore advanced techniques for crawling dynamic content using Crawl4AI's session management features.
+
+## Understanding Session-Based Crawling
+
+Session-based crawling allows you to maintain a persistent browser session across multiple requests. This is crucial when:
+
+1. The content changes dynamically without URL changes
+2. You need to interact with the page (e.g., clicking buttons) between requests
+3. The site requires authentication or maintains state across pages
+
+Crawl4AI's `AsyncWebCrawler` class supports session-based crawling through the `session_id` parameter and related methods.
+
+## Basic Concepts
+
+Before diving into examples, let's review some key concepts:
+
+- **Session ID**: A unique identifier for a browsing session. Use the same `session_id` across multiple `arun` calls to maintain state.
+- **JavaScript Execution**: Use the `js_code` parameter to execute JavaScript on the page, such as clicking a "Load More" button.
+- **CSS Selectors**: Use these to target specific elements for extraction or interaction.
+- **Extraction Strategy**: Define how to extract structured data from the page.
+- **Wait Conditions**: Specify conditions to wait for before considering the page loaded.
+
+## Example 1: Basic Session-Based Crawling
+
+Let's start with a basic example of session-based crawling:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def basic_session_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        session_id = "my_session"
+        url = "https://example.com/dynamic-content"
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                js_code="document.querySelector('.load-more-button').click();" if page > 0 else None,
+                css_selector=".content-item",
+                bypass_cache=True
+            )
+            
+            print(f"Page {page + 1}: Found {result.extracted_content.count('.content-item')} items")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+
+asyncio.run(basic_session_crawl())
+```
+
+This example demonstrates:
+1. Using a consistent `session_id` across multiple `arun` calls
+2. Executing JavaScript to load more content after the first page
+3. Using a CSS selector to extract specific content
+4. Properly closing the session after crawling
+
+## Advanced Technique 1: Custom Execution Hooks
+
+Crawl4AI allows you to set custom hooks that execute at different stages of the crawling process. This is particularly useful for handling complex loading scenarios.
+
+Here's an example that waits for new content to appear before proceeding:
+
+```python
+async def advanced_session_crawl_with_hooks():
+    first_commit = ""
+
+    async def on_execution_started(page):
+        nonlocal first_commit
+        try:
+            while True:
+                await page.wait_for_selector("li.commit-item h4")
+                commit = await page.query_selector("li.commit-item h4")
+                commit = await commit.evaluate("(element) => element.textContent")
+                commit = commit.strip()
+                if commit and commit != first_commit:
+                    first_commit = commit
+                    break
+                await asyncio.sleep(0.5)
+        except Exception as e:
+            print(f"Warning: New content didn't appear after JavaScript execution: {e}")
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started)
+
+        url = "https://github.com/example/repo/commits/main"
+        session_id = "commit_session"
+        all_commits = []
+
+        js_next_page = """
+        const button = document.querySelector('a.pagination-next');
+        if (button) button.click();
+        """
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.commit-item",
+                js_code=js_next_page if page > 0 else None,
+                bypass_cache=True,
+                js_only=page > 0
+            )
+
+            commits = result.extracted_content.select("li.commit-item")
+            all_commits.extend(commits)
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(advanced_session_crawl_with_hooks())
+```
+
+This technique uses a custom `on_execution_started` hook to ensure new content has loaded before proceeding to the next step.
+
+## Advanced Technique 2: Integrated JavaScript Execution and Waiting
+
+Instead of using separate hooks, you can integrate the waiting logic directly into your JavaScript execution. This approach can be more concise and easier to manage for some scenarios.
+
+Here's an example:
+
+```python
+async def integrated_js_and_wait_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/example/repo/commits/main"
+        session_id = "integrated_session"
+        all_commits = []
+
+        js_next_page_and_wait = """
+        (async () => {
+            const getCurrentCommit = () => {
+                const commits = document.querySelectorAll('li.commit-item h4');
+                return commits.length > 0 ? commits[0].textContent.trim() : null;
+            };
+
+            const initialCommit = getCurrentCommit();
+            const button = document.querySelector('a.pagination-next');
+            if (button) button.click();
+
+            while (true) {
+                await new Promise(resolve => setTimeout(resolve, 100));
+                const newCommit = getCurrentCommit();
+                if (newCommit && newCommit !== initialCommit) {
+                    break;
+                }
+            }
+        })();
+        """
+
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.commit-item",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.commit-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.commit-item",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page_and_wait if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True
+            )
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(integrated_js_and_wait_crawl())
+```
+
+This approach combines the JavaScript for clicking the "next" button and waiting for new content to load into a single script.
+
+## Advanced Technique 3: Using the `wait_for` Parameter
+
+Crawl4AI provides a `wait_for` parameter that allows you to specify a condition to wait for before considering the page fully loaded. This can be particularly useful for dynamic content.
+
+Here's an example:
+
+```python
+async def wait_for_parameter_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/example/repo/commits/main"
+        session_id = "wait_for_session"
+        all_commits = []
+
+        js_next_page = """
+        const commits = document.querySelectorAll('li.commit-item h4');
+        if (commits.length > 0) {
+            window.lastCommit = commits[0].textContent.trim();
+        }
+        const button = document.querySelector('a.pagination-next');
+        if (button) button.click();
+        """
+
+        wait_for = """() => {
+            const commits = document.querySelectorAll('li.commit-item h4');
+            if (commits.length === 0) return false;
+            const firstCommit = commits[0].textContent.trim();
+            return firstCommit !== window.lastCommit;
+        }"""
+        
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.commit-item",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.commit-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.commit-item",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page if page > 0 else None,
+                wait_for=wait_for if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True
+            )
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(wait_for_parameter_crawl())
+```
+
+This technique separates the JavaScript execution (clicking the "next" button) from the waiting condition, providing more flexibility and clarity in some scenarios.
+
+## Best Practices for Session-Based Crawling
+
+1. **Use Unique Session IDs**: Ensure each crawling session has a unique `session_id` to prevent conflicts.
+2. **Close Sessions**: Always close sessions using `kill_session` when you're done to free up resources.
+3. **Handle Errors**: Implement proper error handling to deal with unexpected situations during crawling.
+4. **Respect Website Terms**: Ensure your crawling adheres to the website's terms of service and robots.txt file.
+5. **Implement Delays**: Add appropriate delays between requests to avoid overwhelming the target server.
+6. **Use Extraction Strategies**: Leverage `JsonCssExtractionStrategy` or other extraction strategies for structured data extraction.
+7. **Optimize JavaScript**: Keep your JavaScript execution concise and efficient to improve crawling speed.
+8. **Monitor Performance**: Keep an eye on memory usage and crawling speed, especially for long-running sessions.
+
+## Conclusion
+
+Session-based crawling with Crawl4AI provides powerful capabilities for handling dynamic content and complex web applications. By leveraging session management, JavaScript execution, and waiting strategies, you can effectively crawl and extract data from a wide range of modern websites.
+
+Remember to use these techniques responsibly and in compliance with website policies and ethical web scraping practices.
+
+For more advanced usage and API details, refer to the Crawl4AI API documentation.
--- a/docs/md_v2/advanced/session-management.md
+++ b/docs/md_v2/advanced/session-management.md
@@ -0,0 +1,133 @@
+# Session Management
+
+Session management in Crawl4AI allows you to maintain state across multiple requests and handle complex multi-page crawling tasks, particularly useful for dynamic websites.
+
+## Basic Session Usage
+
+Use `session_id` to maintain state between requests:
+
+```python
+async with AsyncWebCrawler() as crawler:
+    session_id = "my_session"
+    
+    # First request
+    result1 = await crawler.arun(
+        url="https://example.com/page1",
+        session_id=session_id
+    )
+    
+    # Subsequent request using same session
+    result2 = await crawler.arun(
+        url="https://example.com/page2",
+        session_id=session_id
+    )
+    
+    # Clean up when done
+    await crawler.crawler_strategy.kill_session(session_id)
+```
+
+## Dynamic Content with Sessions
+
+Here's a real-world example of crawling GitHub commits across multiple pages:
+
+```python
+async def crawl_dynamic_content():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/microsoft/TypeScript/commits/main"
+        session_id = "typescript_commits_session"
+        all_commits = []
+
+        # Define navigation JavaScript
+        js_next_page = """
+        const button = document.querySelector('a[data-testid="pagination-next-button"]');
+        if (button) button.click();
+        """
+
+        # Define wait condition
+        wait_for = """() => {
+            const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
+            if (commits.length === 0) return false;
+            const firstCommit = commits[0].textContent.trim();
+            return firstCommit !== window.firstCommit;
+        }"""
+        
+        # Define extraction schema
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.Box-sc-g0xbh4-0",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.markdown-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema)
+
+        # Crawl multiple pages
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page if page > 0 else None,
+                wait_for=wait_for if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True
+            )
+
+            if result.success:
+                commits = json.loads(result.extracted_content)
+                all_commits.extend(commits)
+                print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        # Clean up session
+        await crawler.crawler_strategy.kill_session(session_id)
+        return all_commits
+```
+
+## Session Best Practices
+
+1. **Session Naming**:
+```python
+# Use descriptive session IDs
+session_id = "login_flow_session"
+session_id = "product_catalog_session"
+```
+
+2. **Resource Management**:
+```python
+try:
+    # Your crawling code
+    pass
+finally:
+    # Always clean up sessions
+    await crawler.crawler_strategy.kill_session(session_id)
+```
+
+3. **State Management**:
+```python
+# First page: login
+result = await crawler.arun(
+    url="https://example.com/login",
+    session_id=session_id,
+    js_code="document.querySelector('form').submit();"
+)
+
+# Second page: verify login success
+result = await crawler.arun(
+    url="https://example.com/dashboard",
+    session_id=session_id,
+    wait_for="css:.user-profile"  # Wait for authenticated content
+)
+```
+
+## Common Use Cases
+
+1. **Authentication Flows**
+2. **Pagination Handling**
+3. **Form Submissions**
+4. **Multi-step Processes**
+5. **Dynamic Content Navigation**
--- a/docs/md_v2/api/arun.md
+++ b/docs/md_v2/api/arun.md
@@ -0,0 +1,226 @@
+# Complete Parameter Guide for arun()
+
+The following parameters can be passed to the `arun()` method. They are organized by their primary usage context and functionality.
+
+## Core Parameters
+
+```python
+await crawler.arun(
+    url="https://example.com",   # Required: URL to crawl
+    verbose=True,               # Enable detailed logging
+    bypass_cache=False,         # Skip cache for this request
+    warmup=True                # Whether to run warmup check
+)
+```
+
+## Content Processing Parameters
+
+### Text Processing
+```python
+await crawler.arun(
+    word_count_threshold=10,                # Minimum words per content block
+    image_description_min_word_threshold=5,  # Minimum words for image descriptions
+    only_text=False,                        # Extract only text content
+    excluded_tags=['form', 'nav'],          # HTML tags to exclude
+    keep_data_attributes=False,             # Preserve data-* attributes
+)
+```
+
+### Content Selection
+```python
+await crawler.arun(
+    css_selector=".main-content",  # CSS selector for content extraction
+    remove_forms=True,             # Remove all form elements
+    remove_overlay_elements=True,  # Remove popups/modals/overlays
+)
+```
+
+### Link Handling
+```python
+await crawler.arun(
+    exclude_external_links=True,          # Remove external links
+    exclude_social_media_links=True,      # Remove social media links
+    exclude_external_images=True,         # Remove external images
+    exclude_domains=["ads.example.com"],  # Specific domains to exclude
+    social_media_domains=[               # Additional social media domains
+        "facebook.com",
+        "twitter.com",
+        "instagram.com"
+    ]
+)
+```
+
+## Browser Control Parameters
+
+### Basic Browser Settings
+```python
+await crawler.arun(
+    headless=True,                # Run browser in headless mode
+    browser_type="chromium",      # Browser engine: "chromium", "firefox", "webkit"
+    page_timeout=60000,          # Page load timeout in milliseconds
+    user_agent="custom-agent",    # Custom user agent
+)
+```
+
+### Navigation and Waiting
+```python
+await crawler.arun(
+    wait_for="css:.dynamic-content",  # Wait for element/condition
+    delay_before_return_html=2.0,     # Wait before returning HTML (seconds)
+)
+```
+
+### JavaScript Execution
+```python
+await crawler.arun(
+    js_code=[                     # JavaScript to execute (string or list)
+        "window.scrollTo(0, document.body.scrollHeight);",
+        "document.querySelector('.load-more').click();"
+    ],
+    js_only=False,               # Only execute JavaScript without reloading page
+)
+```
+
+### Anti-Bot Features
+```python
+await crawler.arun(
+    magic=True,              # Enable all anti-detection features
+    simulate_user=True,      # Simulate human behavior
+    override_navigator=True  # Override navigator properties
+)
+```
+
+### Session Management
+```python
+await crawler.arun(
+    session_id="my_session",  # Session identifier for persistent browsing
+)
+```
+
+### Screenshot Options
+```python
+await crawler.arun(
+    screenshot=True,              # Take page screenshot
+    screenshot_wait_for=2.0,      # Wait before screenshot (seconds)
+)
+```
+
+### Proxy Configuration
+```python
+await crawler.arun(
+    proxy="http://proxy.example.com:8080",     # Simple proxy URL
+    proxy_config={                             # Advanced proxy settings
+        "server": "http://proxy.example.com:8080",
+        "username": "user",
+        "password": "pass"
+    }
+)
+```
+
+## Content Extraction Parameters
+
+### Extraction Strategy
+```python
+await crawler.arun(
+    extraction_strategy=LLMExtractionStrategy(
+        provider="ollama/llama2",
+        schema=MySchema.schema(),
+        instruction="Extract specific data"
+    )
+)
+```
+
+### Chunking Strategy
+```python
+await crawler.arun(
+    chunking_strategy=RegexChunking(
+        patterns=[r'\n\n', r'\.\s+']
+    )
+)
+```
+
+### HTML to Text Options
+```python
+await crawler.arun(
+    html2text={
+        "ignore_links": False,
+        "ignore_images": False,
+        "escape_dot": False,
+        "body_width": 0,
+        "protect_links": True,
+        "unicode_snob": True
+    }
+)
+```
+
+## Debug Options
+```python
+await crawler.arun(
+    log_console=True,   # Log browser console messages
+)
+```
+
+## Parameter Interactions and Notes
+
+1. **Magic Mode Combinations**
+   ```python
+   # Full anti-detection setup
+   await crawler.arun(
+       magic=True,
+       headless=False,
+       simulate_user=True,
+       override_navigator=True
+   )
+   ```
+
+2. **Dynamic Content Handling**
+   ```python
+   # Handle lazy-loaded content
+   await crawler.arun(
+       js_code="window.scrollTo(0, document.body.scrollHeight);",
+       wait_for="css:.lazy-content",
+       delay_before_return_html=2.0
+   )
+   ```
+
+3. **Content Extraction Pipeline**
+   ```python
+   # Complete extraction setup
+   await crawler.arun(
+       css_selector=".main-content",
+       word_count_threshold=20,
+       extraction_strategy=my_strategy,
+       chunking_strategy=my_chunking,
+       process_iframes=True,
+       remove_overlay_elements=True
+   )
+   ```
+
+## Best Practices
+
+1. **Performance Optimization**
+   ```python
+   await crawler.arun(
+       bypass_cache=False,           # Use cache when possible
+       word_count_threshold=10,      # Filter out noise
+       process_iframes=False         # Skip iframes if not needed
+   )
+   ```
+
+2. **Reliable Scraping**
+   ```python
+   await crawler.arun(
+       magic=True,                   # Enable anti-detection
+       delay_before_return_html=1.0, # Wait for dynamic content
+       page_timeout=60000           # Longer timeout for slow pages
+   )
+   ```
+
+3. **Clean Content**
+   ```python
+   await crawler.arun(
+       remove_overlay_elements=True,  # Remove popups
+       excluded_tags=['nav', 'aside'],# Remove unnecessary elements
+       keep_data_attributes=False     # Remove data attributes
+   )
+   ```
--- a/docs/md_v2/api/async-webcrawler.md
+++ b/docs/md_v2/api/async-webcrawler.md
@@ -0,0 +1,320 @@
+# AsyncWebCrawler
+
+The `AsyncWebCrawler` class is the main interface for web crawling operations. It provides asynchronous web crawling capabilities with extensive configuration options.
+
+## Constructor
+
+```python
+AsyncWebCrawler(
+    # Browser Settings
+    browser_type: str = "chromium",         # Options: "chromium", "firefox", "webkit"
+    headless: bool = True,                  # Run browser in headless mode
+    verbose: bool = False,                  # Enable verbose logging
+    
+    # Cache Settings
+    always_by_pass_cache: bool = False,     # Always bypass cache
+    base_directory: str = str(Path.home()), # Base directory for cache
+    
+    # Network Settings
+    proxy: str = None,                      # Simple proxy URL
+    proxy_config: Dict = None,              # Advanced proxy configuration
+    
+    # Browser Behavior
+    sleep_on_close: bool = False,           # Wait before closing browser
+    
+    # Custom Settings
+    user_agent: str = None,                 # Custom user agent
+    headers: Dict[str, str] = {},           # Custom HTTP headers
+    js_code: Union[str, List[str]] = None,  # Default JavaScript to execute
+)
+```
+
+### Parameters in Detail
+
+#### Browser Settings
+
+- **browser_type** (str, optional)
+  - Default: `"chromium"`
+  - Options: `"chromium"`, `"firefox"`, `"webkit"`
+  - Controls which browser engine to use
+  ```python
+  # Example: Using Firefox
+  crawler = AsyncWebCrawler(browser_type="firefox")
+  ```
+
+- **headless** (bool, optional)
+  - Default: `True`
+  - When `True`, browser runs without GUI
+  - Set to `False` for debugging
+  ```python
+  # Visible browser for debugging
+  crawler = AsyncWebCrawler(headless=False)
+  ```
+
+- **verbose** (bool, optional)
+  - Default: `False`
+  - Enables detailed logging
+  ```python
+  # Enable detailed logging
+  crawler = AsyncWebCrawler(verbose=True)
+  ```
+
+#### Cache Settings
+
+- **always_by_pass_cache** (bool, optional)
+  - Default: `False`
+  - When `True`, always fetches fresh content
+  ```python
+  # Always fetch fresh content
+  crawler = AsyncWebCrawler(always_by_pass_cache=True)
+  ```
+
+- **base_directory** (str, optional)
+  - Default: User's home directory
+  - Base path for cache storage
+  ```python
+  # Custom cache directory
+  crawler = AsyncWebCrawler(base_directory="/path/to/cache")
+  ```
+
+#### Network Settings
+
+- **proxy** (str, optional)
+  - Simple proxy URL
+  ```python
+  # Using simple proxy
+  crawler = AsyncWebCrawler(proxy="http://proxy.example.com:8080")
+  ```
+
+- **proxy_config** (Dict, optional)
+  - Advanced proxy configuration with authentication
+  ```python
+  # Advanced proxy with auth
+  crawler = AsyncWebCrawler(proxy_config={
+      "server": "http://proxy.example.com:8080",
+      "username": "user",
+      "password": "pass"
+  })
+  ```
+
+#### Browser Behavior
+
+- **sleep_on_close** (bool, optional)
+  - Default: `False`
+  - Adds delay before closing browser
+  ```python
+  # Wait before closing
+  crawler = AsyncWebCrawler(sleep_on_close=True)
+  ```
+
+#### Custom Settings
+
+- **user_agent** (str, optional)
+  - Custom user agent string
+  ```python
+  # Custom user agent
+  crawler = AsyncWebCrawler(
+      user_agent="Mozilla/5.0 (Custom Agent) Chrome/90.0"
+  )
+  ```
+
+- **headers** (Dict[str, str], optional)
+  - Custom HTTP headers
+  ```python
+  # Custom headers
+  crawler = AsyncWebCrawler(
+      headers={
+          "Accept-Language": "en-US",
+          "Custom-Header": "Value"
+      }
+  )
+  ```
+
+- **js_code** (Union[str, List[str]], optional)
+  - Default JavaScript to execute on each page
+  ```python
+  # Default JavaScript
+  crawler = AsyncWebCrawler(
+      js_code=[
+          "window.scrollTo(0, document.body.scrollHeight);",
+          "document.querySelector('.load-more').click();"
+      ]
+  )
+  ```
+
+## Methods
+
+### arun()
+
+The primary method for crawling web pages.
+
+```python
+async def arun(
+    # Required
+    url: str,                              # URL to crawl
+    
+    # Content Selection
+    css_selector: str = None,              # CSS selector for content
+    word_count_threshold: int = 10,        # Minimum words per block
+    
+    # Cache Control
+    bypass_cache: bool = False,            # Bypass cache for this request
+    
+    # Session Management
+    session_id: str = None,                # Session identifier
+    
+    # Screenshot Options
+    screenshot: bool = False,              # Take screenshot
+    screenshot_wait_for: float = None,     # Wait before screenshot
+    
+    # Content Processing
+    process_iframes: bool = False,         # Process iframe content
+    remove_overlay_elements: bool = False, # Remove popups/modals
+    
+    # Anti-Bot Settings
+    simulate_user: bool = False,           # Simulate human behavior
+    override_navigator: bool = False,      # Override navigator properties
+    magic: bool = False,                   # Enable all anti-detection
+    
+    # Content Filtering
+    excluded_tags: List[str] = None,       # HTML tags to exclude
+    exclude_external_links: bool = False,  # Remove external links
+    exclude_social_media_links: bool = False, # Remove social media links
+    
+    # JavaScript Handling
+    js_code: Union[str, List[str]] = None, # JavaScript to execute
+    wait_for: str = None,                  # Wait condition
+    
+    # Page Loading
+    page_timeout: int = 60000,            # Page load timeout (ms)
+    delay_before_return_html: float = None, # Wait before return
+    
+    # Extraction
+    extraction_strategy: ExtractionStrategy = None  # Extraction strategy
+) -> CrawlResult:
+```
+
+### Usage Examples
+
+#### Basic Crawling
+```python
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+#### Advanced Crawling
+```python
+async with AsyncWebCrawler(
+    browser_type="firefox",
+    verbose=True,
+    headers={"Custom-Header": "Value"}
+) as crawler:
+    result = await crawler.arun(
+        url="https://example.com",
+        css_selector=".main-content",
+        word_count_threshold=20,
+        process_iframes=True,
+        magic=True,
+        wait_for="css:.dynamic-content",
+        screenshot=True
+    )
+```
+
+#### Session Management
+```python
+async with AsyncWebCrawler() as crawler:
+    # First request
+    result1 = await crawler.arun(
+        url="https://example.com/login",
+        session_id="my_session"
+    )
+    
+    # Subsequent request using same session
+    result2 = await crawler.arun(
+        url="https://example.com/protected",
+        session_id="my_session"
+    )
+```
+
+## Context Manager
+
+AsyncWebCrawler implements the async context manager protocol:
+
+```python
+async def __aenter__(self) -> 'AsyncWebCrawler':
+    # Initialize browser and resources
+    return self
+
+async def __aexit__(self, *args):
+    # Cleanup resources
+    pass
+```
+
+Always use AsyncWebCrawler with async context manager:
+```python
+async with AsyncWebCrawler() as crawler:
+    # Your crawling code here
+    pass
+```
+
+## Best Practices
+
+1. **Resource Management**
+```python
+# Always use context manager
+async with AsyncWebCrawler() as crawler:
+    # Crawler will be properly cleaned up
+    pass
+```
+
+2. **Error Handling**
+```python
+try:
+    async with AsyncWebCrawler() as crawler:
+        result = await crawler.arun(url="https://example.com")
+        if not result.success:
+            print(f"Crawl failed: {result.error_message}")
+except Exception as e:
+    print(f"Error: {str(e)}")
+```
+
+3. **Performance Optimization**
+```python
+# Enable caching for better performance
+crawler = AsyncWebCrawler(
+    always_by_pass_cache=False,
+    verbose=True
+)
+```
+
+4. **Anti-Detection**
+```python
+# Maximum stealth
+crawler = AsyncWebCrawler(
+    headless=True,
+    user_agent="Mozilla/5.0...",
+    headers={"Accept-Language": "en-US"}
+)
+result = await crawler.arun(
+    url="https://example.com",
+    magic=True,
+    simulate_user=True
+)
+```
+
+## Note on Browser Types
+
+Each browser type has its characteristics:
+
+- **chromium**: Best overall compatibility
+- **firefox**: Good for specific use cases
+- **webkit**: Lighter weight, good for basic crawling
+
+Choose based on your specific needs:
+```python
+# High compatibility
+crawler = AsyncWebCrawler(browser_type="chromium")
+
+# Memory efficient
+crawler = AsyncWebCrawler(browser_type="webkit")
+```
--- a/docs/md_v2/api/crawl-result.md
+++ b/docs/md_v2/api/crawl-result.md
@@ -0,0 +1,301 @@
+# CrawlResult
+
+The `CrawlResult` class represents the result of a web crawling operation. It provides access to various forms of extracted content and metadata from the crawled webpage.
+
+## Class Definition
+
+```python
+class CrawlResult(BaseModel):
+    """Result of a web crawling operation."""
+    
+    # Basic Information
+    url: str                                # Crawled URL
+    success: bool                           # Whether crawl succeeded
+    status_code: Optional[int] = None       # HTTP status code
+    error_message: Optional[str] = None     # Error message if failed
+    
+    # Content
+    html: str                              # Raw HTML content
+    cleaned_html: Optional[str] = None      # Cleaned HTML
+    fit_html: Optional[str] = None          # Most relevant HTML content
+    markdown: Optional[str] = None          # HTML converted to markdown
+    fit_markdown: Optional[str] = None      # Most relevant markdown content
+    
+    # Extracted Data
+    extracted_content: Optional[str] = None  # Content from extraction strategy
+    media: Dict[str, List[Dict]] = {}       # Extracted media information
+    links: Dict[str, List[Dict]] = {}       # Extracted links
+    metadata: Optional[dict] = None         # Page metadata
+    
+    # Additional Data
+    screenshot: Optional[str] = None         # Base64 encoded screenshot
+    session_id: Optional[str] = None         # Session identifier
+    response_headers: Optional[dict] = None  # HTTP response headers
+```
+
+## Properties and Their Data Structures
+
+### Basic Information
+
+```python
+# Access basic information
+result = await crawler.arun(url="https://example.com")
+
+print(result.url)          # "https://example.com"
+print(result.success)      # True/False
+print(result.status_code)  # 200, 404, etc.
+print(result.error_message)  # Error details if failed
+```
+
+### Content Properties
+
+#### HTML Content
+```python
+# Raw HTML
+html_content = result.html
+
+# Cleaned HTML (removed ads, popups, etc.)
+clean_content = result.cleaned_html
+
+# Most relevant HTML content
+main_content = result.fit_html
+```
+
+#### Markdown Content
+```python
+# Full markdown version
+markdown_content = result.markdown
+
+# Most relevant markdown content
+main_content = result.fit_markdown
+```
+
+### Media Content
+
+The media dictionary contains organized media elements:
+
+```python
+# Structure
+media = {
+    "images": [
+        {
+            "src": str,           # Image URL
+            "alt": str,           # Alt text
+            "desc": str,          # Contextual description
+            "score": float,       # Relevance score (0-10)
+            "type": str,          # "image"
+            "width": int,         # Image width (if available)
+            "height": int,        # Image height (if available)
+            "context": str,       # Surrounding text
+            "lazy": bool          # Whether image was lazy-loaded
+        }
+    ],
+    "videos": [
+        {
+            "src": str,           # Video URL
+            "type": str,          # "video"
+            "title": str,         # Video title
+            "poster": str,        # Thumbnail URL
+            "duration": str,      # Video duration
+            "description": str    # Video description
+        }
+    ],
+    "audios": [
+        {
+            "src": str,           # Audio URL
+            "type": str,          # "audio"
+            "title": str,         # Audio title
+            "duration": str,      # Audio duration
+            "description": str    # Audio description
+        }
+    ]
+}
+
+# Example usage
+for image in result.media["images"]:
+    if image["score"] > 5:  # High-relevance images
+        print(f"High-quality image: {image['src']}")
+        print(f"Context: {image['context']}")
+```
+
+### Link Analysis
+
+The links dictionary organizes discovered links:
+
+```python
+# Structure
+links = {
+    "internal": [
+        {
+            "href": str,          # URL
+            "text": str,          # Link text
+            "title": str,         # Title attribute
+            "type": str,          # Link type (nav, content, etc.)
+            "context": str,       # Surrounding text
+            "score": float        # Relevance score
+        }
+    ],
+    "external": [
+        {
+            "href": str,          # External URL
+            "text": str,          # Link text
+            "title": str,         # Title attribute
+            "domain": str,        # Domain name
+            "type": str,          # Link type
+            "context": str        # Surrounding text
+        }
+    ]
+}
+
+# Example usage
+for link in result.links["internal"]:
+    print(f"Internal link: {link['href']}")
+    print(f"Context: {link['context']}")
+```
+
+### Metadata
+
+The metadata dictionary contains page information:
+
+```python
+# Structure
+metadata = {
+    "title": str,                # Page title
+    "description": str,          # Meta description
+    "keywords": List[str],       # Meta keywords
+    "author": str,              # Author information
+    "published_date": str,      # Publication date
+    "modified_date": str,       # Last modified date
+    "language": str,            # Page language
+    "canonical_url": str,       # Canonical URL
+    "og_data": Dict,           # Open Graph data
+    "twitter_data": Dict       # Twitter card data
+}
+
+# Example usage
+if result.metadata:
+    print(f"Title: {result.metadata['title']}")
+    print(f"Author: {result.metadata.get('author', 'Unknown')}")
+```
+
+### Extracted Content
+
+Content from extraction strategies:
+
+```python
+# For LLM or CSS extraction strategies
+if result.extracted_content:
+    structured_data = json.loads(result.extracted_content)
+    print(structured_data)
+```
+
+### Screenshot
+
+Base64 encoded screenshot:
+
+```python
+# Save screenshot if available
+if result.screenshot:
+    import base64
+    
+    # Decode and save
+    with open("screenshot.png", "wb") as f:
+        f.write(base64.b64decode(result.screenshot))
+```
+
+## Usage Examples
+
+### Basic Content Access
+```python
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(url="https://example.com")
+    
+    if result.success:
+        # Get clean content
+        print(result.fit_markdown)
+        
+        # Process images
+        for image in result.media["images"]:
+            if image["score"] > 7:
+                print(f"High-quality image: {image['src']}")
+```
+
+### Complete Data Processing
+```python
+async def process_webpage(url: str) -> Dict:
+    async with AsyncWebCrawler() as crawler:
+        result = await crawler.arun(url=url)
+        
+        if not result.success:
+            raise Exception(f"Crawl failed: {result.error_message}")
+        
+        return {
+            "content": result.fit_markdown,
+            "images": [
+                img for img in result.media["images"]
+                if img["score"] > 5
+            ],
+            "internal_links": [
+                link["href"] for link in result.links["internal"]
+            ],
+            "metadata": result.metadata,
+            "status": result.status_code
+        }
+```
+
+### Error Handling
+```python
+async def safe_crawl(url: str) -> Dict:
+    async with AsyncWebCrawler() as crawler:
+        try:
+            result = await crawler.arun(url=url)
+            
+            if not result.success:
+                return {
+                    "success": False,
+                    "error": result.error_message,
+                    "status": result.status_code
+                }
+            
+            return {
+                "success": True,
+                "content": result.fit_markdown,
+                "status": result.status_code
+            }
+            
+        except Exception as e:
+            return {
+                "success": False,
+                "error": str(e),
+                "status": None
+            }
+```
+
+## Best Practices
+
+1. **Always Check Success**
+```python
+if not result.success:
+    print(f"Error: {result.error_message}")
+    return
+```
+
+2. **Use fit_markdown for Articles**
+```python
+# Better for article content
+content = result.fit_markdown if result.fit_markdown else result.markdown
+```
+
+3. **Filter Media by Score**
+```python
+relevant_images = [
+    img for img in result.media["images"]
+    if img["score"] > 5
+]
+```
+
+4. **Handle Missing Data**
+```python
+metadata = result.metadata or {}
+title = metadata.get('title', 'Unknown Title')
+```
--- a/docs/md_v2/api/strategies.md
+++ b/docs/md_v2/api/strategies.md
@@ -0,0 +1,255 @@
+# Extraction & Chunking Strategies API
+
+This documentation covers the API reference for extraction and chunking strategies in Crawl4AI.
+
+## Extraction Strategies
+
+All extraction strategies inherit from the base `ExtractionStrategy` class and implement two key methods:
+- `extract(url: str, html: str) -> List[Dict[str, Any]]`
+- `run(url: str, sections: List[str]) -> List[Dict[str, Any]]`
+
+### LLMExtractionStrategy
+
+Used for extracting structured data using Language Models.
+
+```python
+LLMExtractionStrategy(
+    # Required Parameters
+    provider: str = DEFAULT_PROVIDER,     # LLM provider (e.g., "ollama/llama2")
+    api_token: Optional[str] = None,      # API token
+    
+    # Extraction Configuration
+    instruction: str = None,              # Custom extraction instruction
+    schema: Dict = None,                  # Pydantic model schema for structured data
+    extraction_type: str = "block",       # "block" or "schema"
+    
+    # Chunking Parameters
+    chunk_token_threshold: int = 4000,    # Maximum tokens per chunk
+    overlap_rate: float = 0.1,           # Overlap between chunks
+    word_token_rate: float = 0.75,       # Word to token conversion rate
+    apply_chunking: bool = True,         # Enable/disable chunking
+    
+    # API Configuration
+    base_url: str = None,                # Base URL for API
+    extra_args: Dict = {},               # Additional provider arguments
+    verbose: bool = False                # Enable verbose logging
+)
+```
+
+### CosineStrategy
+
+Used for content similarity-based extraction and clustering.
+
+```python
+CosineStrategy(
+    # Content Filtering
+    semantic_filter: str = None,        # Topic/keyword filter
+    word_count_threshold: int = 10,     # Minimum words per cluster
+    sim_threshold: float = 0.3,         # Similarity threshold
+    
+    # Clustering Parameters
+    max_dist: float = 0.2,             # Maximum cluster distance
+    linkage_method: str = 'ward',       # Clustering method
+    top_k: int = 3,                    # Top clusters to return
+    
+    # Model Configuration
+    model_name: str = 'sentence-transformers/all-MiniLM-L6-v2',  # Embedding model
+    
+    verbose: bool = False              # Enable verbose logging
+)
+```
+
+### JsonCssExtractionStrategy
+
+Used for CSS selector-based structured data extraction.
+
+```python
+JsonCssExtractionStrategy(
+    schema: Dict[str, Any],    # Extraction schema
+    verbose: bool = False      # Enable verbose logging
+)
+
+# Schema Structure
+schema = {
+    "name": str,              # Schema name
+    "baseSelector": str,      # Base CSS selector
+    "fields": [               # List of fields to extract
+        {
+            "name": str,      # Field name
+            "selector": str,  # CSS selector
+            "type": str,     # Field type: "text", "attribute", "html", "regex"
+            "attribute": str, # For type="attribute"
+            "pattern": str,  # For type="regex"
+            "transform": str, # Optional: "lowercase", "uppercase", "strip"
+            "default": Any    # Default value if extraction fails
+        }
+    ]
+}
+```
+
+## Chunking Strategies
+
+All chunking strategies inherit from `ChunkingStrategy` and implement the `chunk(text: str) -> list` method.
+
+### RegexChunking
+
+Splits text based on regex patterns.
+
+```python
+RegexChunking(
+    patterns: List[str] = None  # Regex patterns for splitting
+                               # Default: [r'\n\n']
+)
+```
+
+### SlidingWindowChunking
+
+Creates overlapping chunks with a sliding window approach.
+
+```python
+SlidingWindowChunking(
+    window_size: int = 100,    # Window size in words
+    step: int = 50             # Step size between windows
+)
+```
+
+### OverlappingWindowChunking
+
+Creates chunks with specified overlap.
+
+```python
+OverlappingWindowChunking(
+    window_size: int = 1000,   # Chunk size in words
+    overlap: int = 100         # Overlap size in words
+)
+```
+
+## Usage Examples
+
+### LLM Extraction
+
+```python
+from pydantic import BaseModel
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+# Define schema
+class Article(BaseModel):
+    title: str
+    content: str
+    author: str
+
+# Create strategy
+strategy = LLMExtractionStrategy(
+    provider="ollama/llama2",
+    schema=Article.schema(),
+    instruction="Extract article details"
+)
+
+# Use with crawler
+result = await crawler.arun(
+    url="https://example.com/article",
+    extraction_strategy=strategy
+)
+
+# Access extracted data
+data = json.loads(result.extracted_content)
+```
+
+### CSS Extraction
+
+```python
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+# Define schema
+schema = {
+    "name": "Product List",
+    "baseSelector": ".product-card",
+    "fields": [
+        {
+            "name": "title",
+            "selector": "h2.title",
+            "type": "text"
+        },
+        {
+            "name": "price",
+            "selector": ".price",
+            "type": "text",
+            "transform": "strip"
+        },
+        {
+            "name": "image",
+            "selector": "img",
+            "type": "attribute",
+            "attribute": "src"
+        }
+    ]
+}
+
+# Create and use strategy
+strategy = JsonCssExtractionStrategy(schema)
+result = await crawler.arun(
+    url="https://example.com/products",
+    extraction_strategy=strategy
+)
+```
+
+### Content Chunking
+
+```python
+from crawl4ai.chunking_strategy import OverlappingWindowChunking
+
+# Create chunking strategy
+chunker = OverlappingWindowChunking(
+    window_size=500,  # 500 words per chunk
+    overlap=50        # 50 words overlap
+)
+
+# Use with extraction strategy
+strategy = LLMExtractionStrategy(
+    provider="ollama/llama2",
+    chunking_strategy=chunker
+)
+
+result = await crawler.arun(
+    url="https://example.com/long-article",
+    extraction_strategy=strategy
+)
+```
+
+## Best Practices
+
+1. **Choose the Right Strategy**
+   - Use `LLMExtractionStrategy` for complex, unstructured content
+   - Use `JsonCssExtractionStrategy` for well-structured HTML
+   - Use `CosineStrategy` for content similarity and clustering
+
+2. **Optimize Chunking**
+   ```python
+   # For long documents
+   strategy = LLMExtractionStrategy(
+       chunk_token_threshold=2000,  # Smaller chunks
+       overlap_rate=0.1           # 10% overlap
+   )
+   ```
+
+3. **Handle Errors**
+   ```python
+   try:
+       result = await crawler.arun(
+           url="https://example.com",
+           extraction_strategy=strategy
+       )
+       if result.success:
+           content = json.loads(result.extracted_content)
+   except Exception as e:
+       print(f"Extraction failed: {e}")
+   ```
+
+4. **Monitor Performance**
+   ```python
+   strategy = CosineStrategy(
+       verbose=True,  # Enable logging
+       word_count_threshold=20,  # Filter short content
+       top_k=5  # Limit results
+   )
+   ```
--- a/docs/md_v2/assets/DankMono-Bold.woff2
+++ b/docs/md_v2/assets/DankMono-Bold.woff2
--- a/docs/md_v2/assets/DankMono-Italic.woff2
+++ b/docs/md_v2/assets/DankMono-Italic.woff2
--- a/docs/md_v2/assets/DankMono-Regular.woff2
+++ b/docs/md_v2/assets/DankMono-Regular.woff2
--- a/docs/md_v2/assets/Monaco.woff
+++ b/docs/md_v2/assets/Monaco.woff
--- a/docs/md_v2/assets/dmvendor.css
+++ b/docs/md_v2/assets/dmvendor.css
--- a/docs/md_v2/assets/highlight.css
+++ b/docs/md_v2/assets/highlight.css
--- a/docs/md_v2/assets/highlight.min.js
+++ b/docs/md_v2/assets/highlight.min.js
--- a/docs/md_v2/assets/highlight_init.js
+++ b/docs/md_v2/assets/highlight_init.js
@@ -0,0 +1,6 @@
+document.addEventListener('DOMContentLoaded', (event) => {
+    document.querySelectorAll('pre code').forEach((block) => {
+      hljs.highlightBlock(block);
+    });
+  });
+  
--- a/docs/md_v2/assets/styles.css
+++ b/docs/md_v2/assets/styles.css
@@ -0,0 +1,153 @@
+@font-face {
+    font-family: "Monaco";
+    font-style: normal;
+    font-weight: normal;
+    src: local("Monaco"), url("Monaco.woff") format("woff");
+}
+
+:root {
+    --global-font-size: 16px;
+    --global-line-height: 1.5em;
+    --global-space: 10px;
+    --font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
+        Courier New, monospace, serif;
+    --font-stack: dm, Monaco, Courier New, monospace, serif;
+    --mono-font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
+        Courier New, monospace, serif;
+
+    --background-color: #151515; /* Dark background */
+    --font-color: #eaeaea; /* Light font color for contrast */
+    --invert-font-color: #151515; /* Dark color for inverted elements */
+    --primary-color: #1a95e0; /* Primary color can remain the same or be adjusted for better contrast */
+    --secondary-color: #727578; /* Secondary color for less important text */
+    --error-color: #ff5555; /* Bright color for errors */
+    --progress-bar-background: #444; /* Darker background for progress bar */
+    --progress-bar-fill: #1a95e0; /* Bright color for progress bar fill */
+    --code-bg-color: #1e1e1e; /* Darker background for code blocks */
+    --input-style: solid; /* Keeping input style solid */
+    --block-background-color: #202020; /* Darker background for block elements */
+    --global-font-color: #eaeaea; /* Light font color for global elements */
+
+    --background-color: #222225;
+
+    --background-color: #070708;
+    --page-width: 70em;
+    --font-color: #e8e9ed;
+    --invert-font-color: #222225;
+    --secondary-color: #a3abba;
+    --secondary-color: #d5cec0;
+    --tertiary-color: #a3abba;
+    --primary-color: #09b5a5; /* Updated to the brand color */
+    --primary-color: #50ffff; /* Updated to the brand color */
+    --error-color: #ff3c74;
+    --progress-bar-background: #3f3f44;
+    --progress-bar-fill: #09b5a5; /* Updated to the brand color */
+    --code-bg-color: #3f3f44;
+    --input-style: solid;
+    --display-h1-decoration: none;
+
+    --display-h1-decoration: none;
+}
+
+/* body {
+    background-color: var(--background-color);
+    color: var(--font-color);
+}
+
+a {
+    color: var(--primary-color);
+}
+
+a:hover {
+    background-color: var(--primary-color);
+    color: var(--invert-font-color);
+}
+
+blockquote::after {
+    color: #444; 
+}
+
+pre, code {
+    background-color: var(--code-bg-color);
+    color: var(--font-color);
+}
+
+.terminal-nav:first-child {
+    border-bottom: 1px dashed var(--secondary-color);
+} */
+
+.terminal-mkdocs-main-content {
+    line-height: var(--global-line-height);
+}
+
+strong,
+.highlight {
+    /* background: url(//s2.svgbox.net/pen-brushes.svg?ic=brush-1&color=50ffff); */
+    background-color: #50ffff33;
+}
+
+.terminal-card > header {
+    color: var(--font-color);
+    text-align: center;
+    background-color: var(--progress-bar-background);
+    padding: 0.3em 0.5em;
+}
+.btn.btn-sm {
+    color: var(--font-color);
+    padding: 0.2em 0.5em;
+    font-size: 0.8em;
+}
+
+.loading-message {
+    display: none;
+    margin-top: 20px;
+}
+
+.response-section {
+    display: none;
+    padding-top: 20px;
+}
+
+.tabs {
+    display: flex;
+    flex-direction: column;
+}
+.tab-list {
+    display: flex;
+    padding: 0;
+    margin: 0;
+    list-style-type: none;
+    border-bottom: 1px solid var(--font-color);
+}
+.tab-item {
+    cursor: pointer;
+    padding: 10px;
+    border: 1px solid var(--font-color);
+    margin-right: -1px;
+    border-bottom: none;
+}
+.tab-item:hover,
+.tab-item:focus,
+.tab-item:active {
+    background-color: var(--progress-bar-background);
+}
+.tab-content {
+    display: none;
+    border: 1px solid var(--font-color);
+    border-top: none;
+}
+.tab-content:first-of-type {
+    display: block;
+}
+
+.tab-content header {
+    padding: 0.5em;
+    display: flex; 
+    justify-content: end; 
+    align-items: center;
+    background-color: var(--progress-bar-background);
+}
+.tab-content pre {
+    margin: 0;
+    max-height: 300px; overflow: auto; border:none;
+}
--- a/docs/md_v2/basic/browser-config.md
+++ b/docs/md_v2/basic/browser-config.md
@@ -0,0 +1,208 @@
+# Browser Configuration
+
+Crawl4AI supports multiple browser engines and offers extensive configuration options for browser behavior.
+
+## Browser Types
+
+Choose from three browser engines:
+
+```python
+# Chromium (default)
+async with AsyncWebCrawler(browser_type="chromium") as crawler:
+    result = await crawler.arun(url="https://example.com")
+
+# Firefox
+async with AsyncWebCrawler(browser_type="firefox") as crawler:
+    result = await crawler.arun(url="https://example.com")
+
+# WebKit
+async with AsyncWebCrawler(browser_type="webkit") as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Basic Configuration
+
+Common browser settings:
+
+```python
+async with AsyncWebCrawler(
+    headless=True,           # Run in headless mode (no GUI)
+    verbose=True,           # Enable detailed logging
+    sleep_on_close=False    # No delay when closing browser
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Identity Management
+
+Control how your crawler appears to websites:
+
+```python
+# Custom user agent
+async with AsyncWebCrawler(
+    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+
+# Custom headers
+headers = {
+    "Accept-Language": "en-US,en;q=0.9",
+    "Cache-Control": "no-cache"
+}
+async with AsyncWebCrawler(headers=headers) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Screenshot Capabilities
+
+Capture page screenshots with enhanced error handling:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    screenshot=True,                # Enable screenshot
+    screenshot_wait_for=2.0        # Wait 2 seconds before capture
+)
+
+if result.screenshot:  # Base64 encoded image
+    import base64
+    with open("screenshot.png", "wb") as f:
+        f.write(base64.b64decode(result.screenshot))
+```
+
+## Timeouts and Waiting
+
+Control page loading behavior:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    page_timeout=60000,              # Page load timeout (ms)
+    delay_before_return_html=2.0,    # Wait before content capture
+    wait_for="css:.dynamic-content"  # Wait for specific element
+)
+```
+
+## JavaScript Execution
+
+Execute custom JavaScript before crawling:
+
+```python
+# Single JavaScript command
+result = await crawler.arun(
+    url="https://example.com",
+    js_code="window.scrollTo(0, document.body.scrollHeight);"
+)
+
+# Multiple commands
+js_commands = [
+    "window.scrollTo(0, document.body.scrollHeight);",
+    "document.querySelector('.load-more').click();"
+]
+result = await crawler.arun(
+    url="https://example.com",
+    js_code=js_commands
+)
+```
+
+## Proxy Configuration
+
+Use proxies for enhanced access:
+
+```python
+# Simple proxy
+async with AsyncWebCrawler(
+    proxy="http://proxy.example.com:8080"
+) as crawler:
+    result = await crawler.arun(url="https://example.com")
+
+# Proxy with authentication
+proxy_config = {
+    "server": "http://proxy.example.com:8080",
+    "username": "user",
+    "password": "pass"
+}
+async with AsyncWebCrawler(proxy_config=proxy_config) as crawler:
+    result = await crawler.arun(url="https://example.com")
+```
+
+## Anti-Detection Features
+
+Enable stealth features to avoid bot detection:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    simulate_user=True,        # Simulate human behavior
+    override_navigator=True,   # Mask automation signals
+    magic=True               # Enable all anti-detection features
+)
+```
+
+## Handling Dynamic Content
+
+Configure browser to handle dynamic content:
+
+```python
+# Wait for dynamic content
+result = await crawler.arun(
+    url="https://example.com",
+    wait_for="js:() => document.querySelector('.content').children.length > 10",
+    process_iframes=True     # Process iframe content
+)
+
+# Handle lazy-loaded images
+result = await crawler.arun(
+    url="https://example.com",
+    js_code="window.scrollTo(0, document.body.scrollHeight);",
+    delay_before_return_html=2.0  # Wait for images to load
+)
+```
+
+## Comprehensive Example
+
+Here's how to combine various browser configurations:
+
+```python
+async def crawl_with_advanced_config(url: str):
+    async with AsyncWebCrawler(
+        # Browser setup
+        browser_type="chromium",
+        headless=True,
+        verbose=True,
+        
+        # Identity
+        user_agent="Custom User Agent",
+        headers={"Accept-Language": "en-US"},
+        
+        # Proxy setup
+        proxy="http://proxy.example.com:8080"
+    ) as crawler:
+        result = await crawler.arun(
+            url=url,
+            # Content handling
+            process_iframes=True,
+            screenshot=True,
+            
+            # Timing
+            page_timeout=60000,
+            delay_before_return_html=2.0,
+            
+            # Anti-detection
+            magic=True,
+            simulate_user=True,
+            
+            # Dynamic content
+            js_code=[
+                "window.scrollTo(0, document.body.scrollHeight);",
+                "document.querySelector('.load-more')?.click();"
+            ],
+            wait_for="css:.dynamic-content"
+        )
+        
+        return {
+            "content": result.markdown,
+            "screenshot": result.screenshot,
+            "success": result.success
+        }
+```
--- a/docs/md_v2/basic/content-selection.md
+++ b/docs/md_v2/basic/content-selection.md
@@ -0,0 +1,199 @@
+# Content Selection
+
+Crawl4AI provides multiple ways to select and filter specific content from webpages. Learn how to precisely target the content you need.
+
+## CSS Selectors
+
+The simplest way to extract specific content:
+
+```python
+# Extract specific content using CSS selector
+result = await crawler.arun(
+    url="https://example.com",
+    css_selector=".main-article"  # Target main article content
+)
+
+# Multiple selectors
+result = await crawler.arun(
+    url="https://example.com",
+    css_selector="article h1, article .content"  # Target heading and content
+)
+```
+
+## Content Filtering
+
+Control what content is included or excluded:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    # Content thresholds
+    word_count_threshold=10,        # Minimum words per block
+    
+    # Tag exclusions
+    excluded_tags=['form', 'header', 'footer', 'nav'],
+    
+    # Link filtering
+    exclude_external_links=True,    # Remove external links
+    exclude_social_media_links=True,  # Remove social media links
+    
+    # Media filtering
+    exclude_external_images=True   # Remove external images
+)
+```
+
+## Iframe Content
+
+Process content inside iframes:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    process_iframes=True,  # Extract iframe content
+    remove_overlay_elements=True  # Remove popups/modals that might block iframes
+)
+```
+
+## Structured Content Selection
+
+### Using LLMs for Smart Selection
+
+Use LLMs to intelligently extract specific types of content:
+
+```python
+from pydantic import BaseModel
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+class ArticleContent(BaseModel):
+    title: str
+    main_points: List[str]
+    conclusion: str
+
+strategy = LLMExtractionStrategy(
+    provider="ollama/nemotron",  # Works with any supported LLM
+    schema=ArticleContent.schema(),
+    instruction="Extract the main article title, key points, and conclusion"
+)
+
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=strategy
+)
+article = json.loads(result.extracted_content)
+```
+
+### Pattern-Based Selection
+
+For repeated content patterns (like product listings, news feeds):
+
+```python
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+schema = {
+    "name": "News Articles",
+    "baseSelector": "article.news-item",  # Repeated element
+    "fields": [
+        {"name": "headline", "selector": "h2", "type": "text"},
+        {"name": "summary", "selector": ".summary", "type": "text"},
+        {"name": "category", "selector": ".category", "type": "text"},
+        {
+            "name": "metadata",
+            "type": "nested",
+            "fields": [
+                {"name": "author", "selector": ".author", "type": "text"},
+                {"name": "date", "selector": ".date", "type": "text"}
+            ]
+        }
+    ]
+}
+
+strategy = JsonCssExtractionStrategy(schema)
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=strategy
+)
+articles = json.loads(result.extracted_content)
+```
+
+## Domain-Based Filtering
+
+Control content based on domains:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    exclude_domains=["ads.com", "tracker.com"],
+    exclude_social_media_domains=["facebook.com", "twitter.com"],  # Custom social media domains to exclude
+    exclude_social_media_links=True
+)
+```
+
+## Media Selection
+
+Select specific types of media:
+
+```python
+result = await crawler.arun(url="https://example.com")
+
+# Access different media types
+images = result.media["images"]  # List of image details
+videos = result.media["videos"]  # List of video details
+audios = result.media["audios"]  # List of audio details
+
+# Image with metadata
+for image in images:
+    print(f"URL: {image['src']}")
+    print(f"Alt text: {image['alt']}")
+    print(f"Description: {image['desc']}")
+    print(f"Relevance score: {image['score']}")
+```
+
+## Comprehensive Example
+
+Here's how to combine different selection methods:
+
+```python
+async def extract_article_content(url: str):
+    # Define structured extraction
+    article_schema = {
+        "name": "Article",
+        "baseSelector": "article.main",
+        "fields": [
+            {"name": "title", "selector": "h1", "type": "text"},
+            {"name": "content", "selector": ".content", "type": "text"}
+        ]
+    }
+    
+    # Define LLM extraction
+    class ArticleAnalysis(BaseModel):
+        key_points: List[str]
+        sentiment: str
+        category: str
+
+    async with AsyncWebCrawler() as crawler:
+        # Get structured content
+        pattern_result = await crawler.arun(
+            url=url,
+            extraction_strategy=JsonCssExtractionStrategy(article_schema),
+            word_count_threshold=10,
+            excluded_tags=['nav', 'footer'],
+            exclude_external_links=True
+        )
+        
+        # Get semantic analysis
+        analysis_result = await crawler.arun(
+            url=url,
+            extraction_strategy=LLMExtractionStrategy(
+                provider="ollama/nemotron",
+                schema=ArticleAnalysis.schema(),
+                instruction="Analyze the article content"
+            )
+        )
+        
+        # Combine results
+        return {
+            "article": json.loads(pattern_result.extracted_content),
+            "analysis": json.loads(analysis_result.extracted_content),
+            "media": pattern_result.media
+        }
+```
--- a/docs/md_v2/basic/installation.md
+++ b/docs/md_v2/basic/installation.md
@@ -0,0 +1,92 @@
+# Installation 💻
+
+Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package, use it with Docker, or run it as a local server.
+
+## Option 1: Python Package Installation (Recommended)
+
+Crawl4AI is now available on PyPI, making installation easier than ever. Choose the option that best fits your needs:
+
+### Basic Installation
+
+For basic web crawling and scraping tasks:
+
+```bash
+pip install crawl4ai
+playwright install # Install Playwright dependencies
+```
+
+### Installation with PyTorch
+
+For advanced text clustering (includes CosineSimilarity cluster strategy):
+
+```bash
+pip install crawl4ai[torch]
+```
+
+### Installation with Transformers
+
+For text summarization and Hugging Face models:
+
+```bash
+pip install crawl4ai[transformer]
+```
+
+### Full Installation
+
+For all features:
+
+```bash
+pip install crawl4ai[all]
+```
+
+### Development Installation
+
+For contributors who plan to modify the source code:
+
+```bash
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+pip install -e ".[all]"
+playwright install # Install Playwright dependencies
+```
+
+💡 After installation with "torch", "transformer", or "all" options, it's recommended to run the following CLI command to load the required models:
+
+```bash
+crawl4ai-download-models
+```
+
+This is optional but will boost the performance and speed of the crawler. You only need to do this once after installation.
+
+## Option 2: Using Docker (Coming Soon)
+
+Docker support for Crawl4AI is currently in progress and will be available soon. This will allow you to run Crawl4AI in a containerized environment, ensuring consistency across different systems.
+
+## Option 3: Local Server Installation
+
+For those who prefer to run Crawl4AI as a local server, instructions will be provided once the Docker implementation is complete.
+
+## Verifying Your Installation
+
+After installation, you can verify that Crawl4AI is working correctly by running a simple Python script:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.example.com")
+        print(result.markdown[:500])  # Print first 500 characters
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+This script should successfully crawl the example website and print the first 500 characters of the extracted content.
+
+## Getting Help
+
+If you encounter any issues during installation or usage, please check the [documentation](https://crawl4ai.com/mkdocs/) or raise an issue on the [GitHub repository](https://github.com/unclecode/crawl4ai/issues).
+
+Happy crawling! 🕷️🤖
--- a/docs/md_v2/basic/output-formats.md
+++ b/docs/md_v2/basic/output-formats.md
@@ -0,0 +1,195 @@
+# Output Formats
+
+Crawl4AI provides multiple output formats to suit different needs, from raw HTML to structured data using LLM or pattern-based extraction.
+
+## Basic Formats
+
+```python
+result = await crawler.arun(url="https://example.com")
+
+# Access different formats
+raw_html = result.html           # Original HTML
+clean_html = result.cleaned_html # Sanitized HTML
+markdown = result.markdown       # Standard markdown
+fit_md = result.fit_markdown    # Most relevant content in markdown
+```
+
+## Raw HTML
+
+Original, unmodified HTML from the webpage. Useful when you need to:
+- Preserve the exact page structure
+- Process HTML with your own tools
+- Debug page issues
+
+```python
+result = await crawler.arun(url="https://example.com")
+print(result.html)  # Complete HTML including headers, scripts, etc.
+```
+
+## Cleaned HTML
+
+Sanitized HTML with unnecessary elements removed. Automatically:
+- Removes scripts and styles
+- Cleans up formatting
+- Preserves semantic structure
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    excluded_tags=['form', 'header', 'footer'],  # Additional tags to remove
+    keep_data_attributes=False  # Remove data-* attributes
+)
+print(result.cleaned_html)
+```
+
+## Standard Markdown
+
+HTML converted to clean markdown format. Great for:
+- Content analysis
+- Documentation
+- Readability
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    include_links_on_markdown=True  # Include links in markdown
+)
+print(result.markdown)
+```
+
+## Fit Markdown
+
+Most relevant content extracted and converted to markdown. Ideal for:
+- Article extraction
+- Main content focus
+- Removing boilerplate
+
+```python
+result = await crawler.arun(url="https://example.com")
+print(result.fit_markdown)  # Only the main content
+```
+
+## Structured Data Extraction
+
+Crawl4AI offers two powerful approaches for structured data extraction:
+
+### 1. LLM-Based Extraction
+
+Use any LLM (OpenAI, HuggingFace, Ollama, etc.) to extract structured data with high accuracy:
+
+```python
+from pydantic import BaseModel
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+class KnowledgeGraph(BaseModel):
+    entities: List[dict]
+    relationships: List[dict]
+
+strategy = LLMExtractionStrategy(
+    provider="ollama/nemotron",  # or "huggingface/...", "ollama/..."
+    api_token="your-token",   # not needed for Ollama
+    schema=KnowledgeGraph.schema(),
+    instruction="Extract entities and relationships from the content"
+)
+
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=strategy
+)
+knowledge_graph = json.loads(result.extracted_content)
+```
+
+### 2. Pattern-Based Extraction
+
+For pages with repetitive patterns (e.g., product listings, article feeds), use JsonCssExtractionStrategy:
+
+```python
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+schema = {
+    "name": "Product Listing",
+    "baseSelector": ".product-card",  # Repeated element
+    "fields": [
+        {"name": "title", "selector": "h2", "type": "text"},
+        {"name": "price", "selector": ".price", "type": "text"},
+        {"name": "description", "selector": ".desc", "type": "text"}
+    ]
+}
+
+strategy = JsonCssExtractionStrategy(schema)
+result = await crawler.arun(
+    url="https://example.com",
+    extraction_strategy=strategy
+)
+products = json.loads(result.extracted_content)
+```
+
+## Content Customization
+
+### HTML to Text Options
+
+Configure markdown conversion:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    html2text={
+        "escape_dot": False,
+        "body_width": 0,
+        "protect_links": True,
+        "unicode_snob": True
+    }
+)
+```
+
+### Content Filters
+
+Control what content is included:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    word_count_threshold=10,        # Minimum words per block
+    exclude_external_links=True,    # Remove external links
+    exclude_external_images=True,   # Remove external images
+    excluded_tags=['form', 'nav']   # Remove specific HTML tags
+)
+```
+
+## Comprehensive Example
+
+Here's how to use multiple output formats together:
+
+```python
+async def crawl_content(url: str):
+    async with AsyncWebCrawler() as crawler:
+        # Extract main content with fit markdown
+        result = await crawler.arun(
+            url=url,
+            word_count_threshold=10,
+            exclude_external_links=True
+        )
+        
+        # Get structured data using LLM
+        llm_result = await crawler.arun(
+            url=url,
+            extraction_strategy=LLMExtractionStrategy(
+                provider="ollama/nemotron",
+                schema=YourSchema.schema(),
+                instruction="Extract key information"
+            )
+        )
+        
+        # Get repeated patterns (if any)
+        pattern_result = await crawler.arun(
+            url=url,
+            extraction_strategy=JsonCssExtractionStrategy(your_schema)
+        )
+        
+        return {
+            "main_content": result.fit_markdown,
+            "structured_data": json.loads(llm_result.extracted_content),
+            "pattern_data": json.loads(pattern_result.extracted_content),
+            "media": result.media
+        }
+```
--- a/docs/md_v2/basic/page-interaction.md
+++ b/docs/md_v2/basic/page-interaction.md
@@ -0,0 +1,207 @@
+# Page Interaction
+
+Crawl4AI provides powerful features for interacting with dynamic webpages, handling JavaScript execution, and managing page events.
+
+## JavaScript Execution
+
+### Basic Execution
+
+```python
+# Single JavaScript command
+result = await crawler.arun(
+    url="https://example.com",
+    js_code="window.scrollTo(0, document.body.scrollHeight);"
+)
+
+# Multiple commands
+js_commands = [
+    "window.scrollTo(0, document.body.scrollHeight);",
+    "document.querySelector('.load-more').click();",
+    "document.querySelector('#consent-button').click();"
+]
+result = await crawler.arun(
+    url="https://example.com",
+    js_code=js_commands
+)
+```
+
+## Wait Conditions
+
+### CSS-Based Waiting
+
+Wait for elements to appear:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    wait_for="css:.dynamic-content"  # Wait for element with class 'dynamic-content'
+)
+```
+
+### JavaScript-Based Waiting
+
+Wait for custom conditions:
+
+```python
+# Wait for number of elements
+wait_condition = """() => {
+    return document.querySelectorAll('.item').length > 10;
+}"""
+
+result = await crawler.arun(
+    url="https://example.com",
+    wait_for=f"js:{wait_condition}"
+)
+
+# Wait for dynamic content to load
+wait_for_content = """() => {
+    const content = document.querySelector('.content');
+    return content && content.innerText.length > 100;
+}"""
+
+result = await crawler.arun(
+    url="https://example.com",
+    wait_for=f"js:{wait_for_content}"
+)
+```
+
+## Handling Dynamic Content
+
+### Load More Content
+
+Handle infinite scroll or load more buttons:
+
+```python
+# Scroll and wait pattern
+result = await crawler.arun(
+    url="https://example.com",
+    js_code=[
+        # Scroll to bottom
+        "window.scrollTo(0, document.body.scrollHeight);",
+        # Click load more if exists
+        "const loadMore = document.querySelector('.load-more'); if(loadMore) loadMore.click();"
+    ],
+    # Wait for new content
+    wait_for="js:() => document.querySelectorAll('.item').length > previousCount"
+)
+```
+
+### Form Interaction
+
+Handle forms and inputs:
+
+```python
+js_form_interaction = """
+    // Fill form fields
+    document.querySelector('#search').value = 'search term';
+    // Submit form
+    document.querySelector('form').submit();
+"""
+
+result = await crawler.arun(
+    url="https://example.com",
+    js_code=js_form_interaction,
+    wait_for="css:.results"  # Wait for results to load
+)
+```
+
+## Timing Control
+
+### Delays and Timeouts
+
+Control timing of interactions:
+
+```python
+result = await crawler.arun(
+    url="https://example.com",
+    page_timeout=60000,              # Page load timeout (ms)
+    delay_before_return_html=2.0,    # Wait before capturing content
+)
+```
+
+## Complex Interactions Example
+
+Here's an example of handling a dynamic page with multiple interactions:
+
+```python
+async def crawl_dynamic_content():
+    async with AsyncWebCrawler() as crawler:
+        # Initial page load
+        result = await crawler.arun(
+            url="https://example.com",
+            # Handle cookie consent
+            js_code="document.querySelector('.cookie-accept')?.click();",
+            wait_for="css:.main-content"
+        )
+
+        # Load more content
+        session_id = "dynamic_session"  # Keep session for multiple interactions
+        
+        for page in range(3):  # Load 3 pages of content
+            result = await crawler.arun(
+                url="https://example.com",
+                session_id=session_id,
+                js_code=[
+                    # Scroll to bottom
+                    "window.scrollTo(0, document.body.scrollHeight);",
+                    # Store current item count
+                    "window.previousCount = document.querySelectorAll('.item').length;",
+                    # Click load more
+                    "document.querySelector('.load-more')?.click();"
+                ],
+                # Wait for new items
+                wait_for="""() => {
+                    const currentCount = document.querySelectorAll('.item').length;
+                    return currentCount > window.previousCount;
+                }""",
+                # Only execute JS without reloading page
+                js_only=True if page > 0 else False
+            )
+            
+            # Process content after each load
+            print(f"Page {page + 1} items:", len(result.cleaned_html))
+            
+        # Clean up session
+        await crawler.crawler_strategy.kill_session(session_id)
+```
+
+## Using with Extraction Strategies
+
+Combine page interaction with structured extraction:
+
+```python
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy
+
+# Pattern-based extraction after interaction
+schema = {
+    "name": "Dynamic Items",
+    "baseSelector": ".item",
+    "fields": [
+        {"name": "title", "selector": "h2", "type": "text"},
+        {"name": "description", "selector": ".desc", "type": "text"}
+    ]
+}
+
+result = await crawler.arun(
+    url="https://example.com",
+    js_code="window.scrollTo(0, document.body.scrollHeight);",
+    wait_for="css:.item:nth-child(10)",  # Wait for 10 items
+    extraction_strategy=JsonCssExtractionStrategy(schema)
+)
+
+# Or use LLM to analyze dynamic content
+class ContentAnalysis(BaseModel):
+    topics: List[str]
+    summary: str
+
+result = await crawler.arun(
+    url="https://example.com",
+    js_code="document.querySelector('.show-more').click();",
+    wait_for="css:.full-content",
+    extraction_strategy=LLMExtractionStrategy(
+        provider="ollama/nemotron",
+        schema=ContentAnalysis.schema(),
+        instruction="Analyze the full content"
+    )
+)
+```
--- a/Show More
+++ b/Show More