Update README

Implements a responsive hamburger menu for mobile devices with the following changes:
- Add new mobile_menu.js for handling mobile navigation
- Update layout.css with mobile-specific styles and animations
- Enhance README with updated geolocation example
- Register mobile_menu.js in mkdocs.yml

The mobile menu includes:
- Hamburger button animation
- Slide-out sidebar
- Backdrop overlay
- Touch-friendly navigation
- Proper event handling

CHANGELOG.md

@@ -5,6 +5,53 @@ All notable changes to Crawl4AI will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0rc1] ‑ 2025‑04‑22
+
+### Added
+- Browser pooling with page pre‑warming and fine‑grained **geolocation, locale, and timezone** controls  
+- Crawler pool manager (SDK + Docker API) for smarter resource allocation  
+- Network & console log capture plus MHTML snapshot export  
+- **Table extractor**: turn HTML `<table>`s into DataFrames or CSV with one flag  
+- High‑volume stress‑test framework in `tests/memory` and API load scripts  
+- MCP protocol endpoints with socket & SSE support; playground UI scaffold  
+- Docs v2 revamp: TOC, GitHub badge, copy‑code buttons, Docker API demo  
+- “Ask AI” helper button *(work‑in‑progress, shipping soon)*  
+- New examples: geo‑location usage, network/console capture, Docker API, markdown source selection, crypto analysis  
+- Expanded automated test suites for browser, Docker, MCP and memory benchmarks  
+
+### Changed
+- Consolidated and renamed browser strategies; legacy docker strategy modules removed  
+- `ProxyConfig` moved to `async_configs`  
+- Server migrated to pool‑based crawler management  
+- FastAPI validators replace custom query validation  
+- Docker build now uses Chromium base image  
+- Large‑scale repo tidy‑up (≈36 k insertions, ≈5 k deletions)  
+
+### Fixed
+- Async crawler session leak, duplicate‑visit handling, URL normalisation  
+- Target‑element regressions in scraping strategies  
+- Logged‑URL readability, encoded‑URL decoding, middle truncation for long URLs  
+- Closed issues: #701, #733, #756, #774, #804, #822, #839, #841, #842, #843, #867, #902, #911  
+
+### Removed
+- Obsolete modules under `crawl4ai/browser/*` superseded by the new pooled browser layer  
+
+### Deprecated
+- Old markdown generator names now alias `DefaultMarkdownGenerator` and emit warnings  
+
+---
+
+#### Upgrade notes
+1. Update any direct imports from `crawl4ai/browser/*` to the new pooled browser modules  
+2. If you override `AsyncPlaywrightCrawlerStrategy.get_page`, adopt the new signature  
+3. Rebuild Docker images to pull the new Chromium layer  
+4. Switch to `DefaultMarkdownGenerator` (or silence the deprecation warning)  
+
+---
+
+`121 files changed, ≈36 223 insertions, ≈4 975 deletions` :contentReference[oaicite:0]{index=0}&#8203;:contentReference[oaicite:1]{index=1}
+
+
 ### [Feature] 2025-04-21
 - Implemented MCP protocol for machine-to-machine communication
   - Added WebSocket and SSE transport for MCP server

Dockerfile

@@ -1,4 +1,9 @@
-FROM python:3.10-slim
+FROM python:3.12-slim-bookworm AS build
+
+# C4ai version
+ARG C4AI_VER=0.6.0
+ENV C4AI_VERSION=$C4AI_VER
+LABEL c4ai.version=$C4AI_VER
 
 # Set build arguments
 ARG APP_HOME=/app
@@ -17,7 +22,7 @@ ENV PYTHONFAULTHANDLER=1 \
     REDIS_HOST=localhost \
     REDIS_PORT=6379
 
-ARG PYTHON_VERSION=3.10
+ARG PYTHON_VERSION=3.12
 ARG INSTALL_TYPE=default
 ARG ENABLE_GPU=false
 ARG TARGETARCH
@@ -66,6 +71,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && apt-get clean \ 
     && rm -rf /var/lib/apt/lists/*
 
+RUN apt-get update && apt-get dist-upgrade -y \
+    && rm -rf /var/lib/apt/lists/*
+
 RUN if [ "$ENABLE_GPU" = "true" ] && [ "$TARGETARCH" = "amd64" ] ; then \
     apt-get update && apt-get install -y --no-install-recommends \
     nvidia-cuda-toolkit \

README.md

@@ -21,9 +21,9 @@
 
 Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.  
 
-[✨ Check out latest update v0.5.0](#-recent-updates)
+[✨ Check out latest update v0.6.0rc1](#-recent-updates)
 
-🎉 **Version 0.5.0 is out!** This major release introduces Deep Crawling with BFS/DFS/BestFirst strategies, Memory-Adaptive Dispatcher, Multiple Crawling Strategies (Playwright and HTTP), Docker Deployment with FastAPI, Command-Line Interface (CLI), and more! [Read the release notes →](https://docs.crawl4ai.com/blog)
+🎉 **Version 0.6.0rc1 is now available!** This release candidate introduces World-aware Crawling with geolocation and locale settings, Table-to-DataFrame extraction, Browser pooling with pre-warming, Network and console traffic capture, MCP integration for AI tools, and a completely revamped Docker deployment! [Read the release notes →](https://docs.crawl4ai.com/blog)
 
 <details>
 <summary>🤓 <strong>My Personal Story</strong></summary>
@@ -253,24 +253,29 @@ pip install -e ".[all]"             # Install all optional features
 <details>
 <summary>🐳 <strong>Docker Deployment</strong></summary>
 
-> 🚀 **Major Changes Coming!** We're developing a completely new Docker implementation that will make deployment even more efficient and seamless. The current Docker setup is being deprecated in favor of this new solution.
+> 🚀 **Now Available!** Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.
 
-### Current Docker Support
+### New Docker Features
 
-The existing Docker implementation is being deprecated and will be replaced soon. If you still need to use Docker with the current version:
+The new Docker implementation includes:
+- **Browser pooling** with page pre-warming for faster response times
+- **Interactive playground** to test and generate request code
+- **MCP integration** for direct connection to AI tools like Claude Code
+- **Comprehensive API endpoints** including HTML extraction, screenshots, PDF generation, and JavaScript execution
+- **Multi-architecture support** with automatic detection (AMD64/ARM64)
+- **Optimized resources** with improved memory management
 
-- 📚 [Deprecated Docker Setup](./docs/deprecated/docker-deployment.md) - Instructions for the current Docker implementation
-- ⚠️ Note: This setup will be replaced in the next major release
+### Getting Started
 
-### What's Coming Next?
+```bash
+# Pull and run the latest release candidate
+docker pull unclecode/crawl4ai:0.6.0-rN # Use your favorite revision number
+docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:0.6.0-rN # Use your favorite revision number
 
-Our new Docker implementation will bring:
-- Improved performance and resource efficiency
-- Streamlined deployment process
-- Better integration with Crawl4AI features
-- Enhanced scalability options
+# Visit the playground at http://localhost:11235/playground
+```
 
-Stay connected with our [GitHub repository](https://github.com/unclecode/crawl4ai) for updates!
+For complete documentation, see our [Docker Deployment Guide](https://docs.crawl4ai.com/core/docker-deployment/).
 
 </details>
 
@@ -500,31 +505,92 @@ async def test_news_crawl():
 
 ## ✨ Recent Updates
 
-### Version 0.5.0 Major Release Highlights
+### Version 0.6.0rc1 Release Highlights
 
--   **🚀 Deep Crawling System**: Explore websites beyond initial URLs with three strategies:
-    -   **BFS Strategy**: Breadth-first search explores websites level by level
-    -   **DFS Strategy**: Depth-first search explores each branch deeply before backtracking
-    -   **BestFirst Strategy**: Uses scoring functions to prioritize which URLs to crawl next
-    -   **Page Limiting**: Control the maximum number of pages to crawl with `max_pages` parameter
-    -   **Score Thresholds**: Filter URLs based on relevance scores
--   **⚡ Memory-Adaptive Dispatcher**: Dynamically adjusts concurrency based on system memory with built-in rate limiting
--   **🔄 Multiple Crawling Strategies**:
-    -   **AsyncPlaywrightCrawlerStrategy**: Browser-based crawling with JavaScript support (Default)
-    -   **AsyncHTTPCrawlerStrategy**: Fast, lightweight HTTP-only crawler for simple tasks
--   **🐳 Docker Deployment**: Easy deployment with FastAPI server and streaming/non-streaming endpoints
--   **💻 Command-Line Interface**: New `crwl` CLI provides convenient terminal access to all features with intuitive commands and configuration options
--   **👤 Browser Profiler**: Create and manage persistent browser profiles to save authentication states, cookies, and settings for seamless crawling of protected content
--   **🧠 Crawl4AI Coding Assistant**: AI-powered coding assistant to answer your question for Crawl4ai, and generate proper code for crawling.
--   **🏎️ LXML Scraping Mode**: Fast HTML parsing using the `lxml` library for improved performance
--   **🌐 Proxy Rotation**: Built-in support for proxy switching with `RoundRobinProxyStrategy`
+- **🌎 World-aware Crawling**: Set geolocation, language, and timezone for authentic locale-specific content:
+  ```python
+    crun_cfg = CrawlerRunConfig(
+        url="https://browserleaks.com/geo",          # test page that shows your location
+        locale="en-US",                              # Accept-Language & UI locale
+        timezone_id="America/Los_Angeles",           # JS Date()/Intl timezone
+        geolocation=GeolocationConfig(                 # override GPS coords
+            latitude=34.0522,
+            longitude=-118.2437,
+            accuracy=10.0,
+        )
+    )
+  ```
+
+- **📊 Table-to-DataFrame Extraction**: Extract HTML tables directly to CSV or pandas DataFrames:
+  ```python
+    crawler = AsyncWebCrawler(config=browser_config)
+    await crawler.start()
+
+    try:
+        # Set up scraping parameters
+        crawl_config = CrawlerRunConfig(
+            table_score_threshold=8,  # Strict table detection
+        )
+
+        # Execute market data extraction
+        results: List[CrawlResult] = await crawler.arun(
+            url="https://coinmarketcap.com/?page=1", config=crawl_config
+        )
+
+        # Process results
+        raw_df = pd.DataFrame()
+        for result in results:
+            if result.success and result.media["tables"]:
+                raw_df = pd.DataFrame(
+                    result.media["tables"][0]["rows"],
+                    columns=result.media["tables"][0]["headers"],
+                )
+                break
+        print(raw_df.head())
+
+    finally:
+        await crawler.stop()
+  ```
+
+- **🚀 Browser Pooling**: Pages launch hot with pre-warmed browser instances for lower latency and memory usage
+
+- **🕸️ Network and Console Capture**: Full traffic logs and MHTML snapshots for debugging:
+  ```python
+  crawler_config = CrawlerRunConfig(
+      capture_network=True,
+      capture_console=True,
+      mhtml=True
+  )
+  ```
+
+- **🔌 MCP Integration**: Connect to AI tools like Claude Code through the Model Context Protocol
+  ```bash
+  # Add Crawl4AI to Claude Code
+  claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse
+  ```
+
+- **🖥️ Interactive Playground**: Test configurations and generate API requests with the built-in web interface at `http://localhost:11235//playground`
+
+- **🐳 Revamped Docker Deployment**: Streamlined multi-architecture Docker image with improved resource efficiency
+
+- **📱 Multi-stage Build System**: Optimized Dockerfile with platform-specific performance enhancements
+
+Read the full details in our [0.6.0rc1 Release Notes](https://docs.crawl4ai.com/blog/releases/0.6.0.html) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md).
+
+### Previous Version: 0.5.0 Major Release Highlights
+
+-   **🚀 Deep Crawling System**: Explore websites beyond initial URLs with BFS, DFS, and BestFirst strategies
+-   **⚡ Memory-Adaptive Dispatcher**: Dynamically adjusts concurrency based on system memory
+-   **🔄 Multiple Crawling Strategies**: Browser-based and lightweight HTTP-only crawlers
+-   **💻 Command-Line Interface**: New `crwl` CLI provides convenient terminal access
+-   **👤 Browser Profiler**: Create and manage persistent browser profiles
+-   **🧠 Crawl4AI Coding Assistant**: AI-powered coding assistant
+-   **🏎️ LXML Scraping Mode**: Fast HTML parsing using the `lxml` library
+-   **🌐 Proxy Rotation**: Built-in support for proxy switching
 -   **🤖 LLM Content Filter**: Intelligent markdown generation using LLMs
 -   **📄 PDF Processing**: Extract text, images, and metadata from PDF files
--   **🔗 URL Redirection Tracking**: Automatically follow and record HTTP redirects
--   **🤖 LLM Schema Generation**: Easily create extraction schemas with LLM assistance
--   **🔍 robots.txt Compliance**: Respect website crawling rules
 
-Read the full details in our [0.5.0 Release Notes](https://docs.crawl4ai.com/blog/releases/0.5.0.html) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md).
+Read the full details in our [0.5.0 Release Notes](https://docs.crawl4ai.com/blog/releases/0.5.0.html).
 
 ## Version Numbering in Crawl4AI
 

crawl4ai/__version__.py

@@ -1,2 +1,3 @@
 # crawl4ai/_version.py
-__version__ = "0.5.0.post8"
+__version__ = "0.6.0"
+

deploy/docker/README-new.md

@@ -1,644 +0,0 @@
-# Crawl4AI Docker Guide 🐳
-
-## Table of Contents
-- [Prerequisites](#prerequisites)
-- [Installation](#installation)
-  - [Option 1: Using Docker Compose (Recommended)](#option-1-using-docker-compose-recommended)
-  - [Option 2: Manual Local Build & Run](#option-2-manual-local-build--run)
-  - [Option 3: Using Pre-built Docker Hub Images](#option-3-using-pre-built-docker-hub-images)
-- [Dockerfile Parameters](#dockerfile-parameters)
-- [Using the API](#using-the-api)
-  - [Understanding Request Schema](#understanding-request-schema)
-  - [REST API Examples](#rest-api-examples)
-  - [Python SDK](#python-sdk)
-- [Metrics & Monitoring](#metrics--monitoring)
-- [Deployment Scenarios](#deployment-scenarios)
-- [Complete Examples](#complete-examples)
-- [Server Configuration](#server-configuration)
-  - [Understanding config.yml](#understanding-configyml)
-  - [JWT Authentication](#jwt-authentication)
-  - [Configuration Tips and Best Practices](#configuration-tips-and-best-practices)
-  - [Customizing Your Configuration](#customizing-your-configuration)
-  - [Configuration Recommendations](#configuration-recommendations)
-- [Getting Help](#getting-help)
-
-## Prerequisites
-
-Before we dive in, make sure you have:
-- Docker installed and running (version 20.10.0 or higher), including `docker compose` (usually bundled with Docker Desktop).
-- `git` for cloning the repository.
-- At least 4GB of RAM available for the container (more recommended for heavy use).
-- Python 3.10+ (if using the Python SDK).
-- Node.js 16+ (if using the Node.js examples).
-
-> 💡 **Pro tip**: Run `docker info` to check your Docker installation and available resources.
-
-## Installation
-
-We offer several ways to get the Crawl4AI server running. Docker Compose is the easiest way to manage local builds and runs.
-
-### Option 1: Using Docker Compose (Recommended)
-
-Docker Compose simplifies building and running the service, especially for local development and testing across different platforms.
-
-#### 1. Clone Repository
-
-```bash
-git clone https://github.com/unclecode/crawl4ai.git
-cd crawl4ai
-```
-
-#### 2. Environment Setup (API Keys)
-
-If you plan to use LLMs, copy the example environment file and add your API keys. This file should be in the **project root directory**.
-
-```bash
-# Make sure you are in the 'crawl4ai' root directory
-cp deploy/docker/.llm.env.example .llm.env
-
-# Now edit .llm.env and add your API keys
-# Example content:
-# OPENAI_API_KEY=sk-your-key
-# ANTHROPIC_API_KEY=your-anthropic-key
-# ...
-```
-> 🔑 **Note**: Keep your API keys secure! Never commit `.llm.env` to version control.
-
-#### 3. Build and Run with Compose
-
-The `docker-compose.yml` file in the project root defines services for different scenarios using **profiles**.
-
-*   **Build and Run Locally (AMD64):**
-    ```bash
-    # Builds the image locally using Dockerfile and runs it
-    docker compose --profile local-amd64 up --build -d
-    ```
-
-*   **Build and Run Locally (ARM64):**
-    ```bash
-    # Builds the image locally using Dockerfile and runs it
-    docker compose --profile local-arm64 up --build -d
-    ```
-
-*   **Run Pre-built Image from Docker Hub (AMD64):**
-    ```bash
-    # Pulls and runs the specified AMD64 image from Docker Hub
-    # (Set VERSION env var for specific tags, e.g., VERSION=0.5.1-d1)
-    docker compose --profile hub-amd64 up -d
-    ```
-
-*   **Run Pre-built Image from Docker Hub (ARM64):**
-    ```bash
-    # Pulls and runs the specified ARM64 image from Docker Hub
-    docker compose --profile hub-arm64 up -d
-    ```
-
-> The server will be available at `http://localhost:11235`.
-
-#### 4. Stopping Compose Services
-
-```bash
-# Stop the service(s) associated with a profile (e.g., local-amd64)
-docker compose --profile local-amd64 down
-```
-
-### Option 2: Manual Local Build & Run
-
-If you prefer not to use Docker Compose for local builds.
-
-#### 1. Clone Repository & Setup Environment
-
-Follow steps 1 and 2 from the Docker Compose section above (clone repo, `cd crawl4ai`, create `.llm.env` in the root).
-
-#### 2. Build the Image (Multi-Arch)
-
-Use `docker buildx` to build the image. This example builds for multiple platforms and loads the image matching your host architecture into the local Docker daemon.
-
-```bash
-# Make sure you are in the 'crawl4ai' root directory
-docker buildx build --platform linux/amd64,linux/arm64 -t crawl4ai-local:latest --load .
-```
-
-#### 3. Run the Container
-
-*   **Basic run (no LLM support):**
-    ```bash
-    # Replace --platform if your host is ARM64
-    docker run -d \
-      -p 11235:11235 \
-      --name crawl4ai-standalone \
-      --shm-size=1g \
-      --platform linux/amd64 \
-      crawl4ai-local:latest
-    ```
-
-*   **With LLM support:**
-    ```bash
-    # Make sure .llm.env is in the current directory (project root)
-    # Replace --platform if your host is ARM64
-    docker run -d \
-      -p 11235:11235 \
-      --name crawl4ai-standalone \
-      --env-file .llm.env \
-      --shm-size=1g \
-      --platform linux/amd64 \
-      crawl4ai-local:latest
-    ```
-
-> The server will be available at `http://localhost:11235`.
-
-#### 4. Stopping the Manual Container
-
-```bash
-docker stop crawl4ai-standalone && docker rm crawl4ai-standalone
-```
-
-### Option 3: Using Pre-built Docker Hub Images
-
-Pull and run images directly from Docker Hub without building locally.
-
-#### 1. Pull the Image
-
-We use a versioning scheme like `LIBRARY_VERSION-dREVISION` (e.g., `0.5.1-d1`). The `latest` tag points to the most recent stable release. Images are built with multi-arch manifests, so Docker usually pulls the correct version for your system automatically.
-
-```bash
-# Pull a specific version (recommended for stability)
-docker pull unclecode/crawl4ai:0.5.1-d1
-
-# Or pull the latest stable version
-docker pull unclecode/crawl4ai:latest
-```
-
-#### 2. Setup Environment (API Keys)
-
-If using LLMs, create the `.llm.env` file in a directory of your choice, similar to Step 2 in the Compose section.
-
-#### 3. Run the Container
-
-*   **Basic run:**
-    ```bash
-    docker run -d \
-      -p 11235:11235 \
-      --name crawl4ai-hub \
-      --shm-size=1g \
-      unclecode/crawl4ai:0.5.1-d1 # Or use :latest
-    ```
-
-*   **With LLM support:**
-    ```bash
-    # Make sure .llm.env is in the current directory you are running docker from
-    docker run -d \
-      -p 11235:11235 \
-      --name crawl4ai-hub \
-      --env-file .llm.env \
-      --shm-size=1g \
-      unclecode/crawl4ai:0.5.1-d1 # Or use :latest
-    ```
-
-> The server will be available at `http://localhost:11235`.
-
-#### 4. Stopping the Hub Container
-
-```bash
-docker stop crawl4ai-hub && docker rm crawl4ai-hub
-```
-
-#### Docker Hub Versioning Explained
-
-*   **Image Name:** `unclecode/crawl4ai`
-*   **Tag Format:** `LIBRARY_VERSION-dREVISION`
-    *   `LIBRARY_VERSION`: The Semantic Version of the core `crawl4ai` Python library included (e.g., `0.5.1`).
-    *   `dREVISION`: An incrementing number (starting at `d1`) for Docker build changes made *without* changing the library version (e.g., base image updates, dependency fixes). Resets to `d1` for each new `LIBRARY_VERSION`.
-*   **Example:** `unclecode/crawl4ai:0.5.1-d1`
-*   **`latest` Tag:** Points to the most recent stable `LIBRARY_VERSION-dREVISION`.
-*   **Multi-Arch:** Images support `linux/amd64` and `linux/arm64`. Docker automatically selects the correct architecture.
-
----
-
-*(Rest of the document remains largely the same, but with key updates below)*
-
----
-
-## Dockerfile Parameters
-
-You can customize the image build process using build arguments (`--build-arg`). These are typically used via `docker buildx build` or within the `docker-compose.yml` file.
-
-```bash
-# Example: Build with 'all' features using buildx
-docker buildx build \
-  --platform linux/amd64,linux/arm64 \
-  --build-arg INSTALL_TYPE=all \
-  -t yourname/crawl4ai-all:latest \
-  --load \
-  . # Build from root context
-```
-
-### Build Arguments Explained
-
-| Argument     | Description                              | Default   | Options                            |
-| :----------- | :--------------------------------------- | :-------- | :--------------------------------- |
-| INSTALL_TYPE | Feature set                              | `default` | `default`, `all`, `torch`, `transformer` |
-| ENABLE_GPU   | GPU support (CUDA for AMD64)           | `false`   | `true`, `false`                    |
-| APP_HOME     | Install path inside container (advanced) | `/app`    | any valid path                   |
-| USE_LOCAL    | Install library from local source        | `true`    | `true`, `false`                    |
-| GITHUB_REPO  | Git repo to clone if USE_LOCAL=false   | *(see Dockerfile)* | any git URL                  |
-| GITHUB_BRANCH| Git branch to clone if USE_LOCAL=false   | `main`    | any branch name                  |
-
-*(Note: PYTHON_VERSION is fixed by the `FROM` instruction in the Dockerfile)*
-
-### Build Best Practices
-
-1.  **Choose the Right Install Type**
-    *   `default`: Basic installation, smallest image size. Suitable for most standard web scraping and markdown generation.
-    *   `all`: Full features including `torch` and `transformers` for advanced extraction strategies (e.g., CosineStrategy, certain LLM filters). Significantly larger image. Ensure you need these extras.
-2.  **Platform Considerations**
-    *   Use `buildx` for building multi-architecture images, especially for pushing to registries.
-    *   Use `docker compose` profiles (`local-amd64`, `local-arm64`) for easy platform-specific local builds.
-3.  **Performance Optimization**
-    *   The image automatically includes platform-specific optimizations (OpenMP for AMD64, OpenBLAS for ARM64).
-
----
-
-## Using the API
-
-Communicate with the running Docker server via its REST API (defaulting to `http://localhost:11235`). You can use the Python SDK or make direct HTTP requests.
-
-### Python SDK
-
-Install the SDK: `pip install crawl4ai`
-
-```python
-import asyncio
-from crawl4ai.docker_client import Crawl4aiDockerClient
-from crawl4ai import BrowserConfig, CrawlerRunConfig, CacheMode # Assuming you have crawl4ai installed
-
-async def main():
-    # Point to the correct server port
-    async with Crawl4aiDockerClient(base_url="http://localhost:11235", verbose=True) as client:
-        # If JWT is enabled on the server, authenticate first:
-        # await client.authenticate("user@example.com") # See Server Configuration section
-
-        # Example Non-streaming crawl
-        print("--- Running Non-Streaming Crawl ---")
-        results = await client.crawl(
-            ["https://httpbin.org/html"],
-            browser_config=BrowserConfig(headless=True), # Use library classes for config aid
-            crawler_config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
-        )
-        if results: # client.crawl returns None on failure
-          print(f"Non-streaming results success: {results.success}")
-          if results.success:
-              for result in results: # Iterate through the CrawlResultContainer
-                  print(f"URL: {result.url}, Success: {result.success}")
-        else:
-            print("Non-streaming crawl failed.")
-
-
-        # Example Streaming crawl
-        print("\n--- Running Streaming Crawl ---")
-        stream_config = CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS)
-        try:
-            async for result in await client.crawl( # client.crawl returns an async generator for streaming
-                ["https://httpbin.org/html", "https://httpbin.org/links/5/0"],
-                browser_config=BrowserConfig(headless=True),
-                crawler_config=stream_config
-            ):
-                print(f"Streamed result: URL: {result.url}, Success: {result.success}")
-        except Exception as e:
-            print(f"Streaming crawl failed: {e}")
-
-
-        # Example Get schema
-        print("\n--- Getting Schema ---")
-        schema = await client.get_schema()
-        print(f"Schema received: {bool(schema)}") # Print whether schema was received
-
-if __name__ == "__main__":
-    asyncio.run(main())
-```
-
-*(SDK parameters like timeout, verify_ssl etc. remain the same)*
-
-### Second Approach: Direct API Calls
-
-Crucially, when sending configurations directly via JSON, they **must** follow the `{"type": "ClassName", "params": {...}}` structure for any non-primitive value (like config objects or strategies). Dictionaries must be wrapped as `{"type": "dict", "value": {...}}`.
-
-*(Keep the detailed explanation of Configuration Structure, Basic Pattern, Simple vs Complex, Strategy Pattern, Complex Nested Example, Quick Grammar Overview, Important Rules, Pro Tip)*
-
-#### More Examples *(Ensure Schema example uses type/value wrapper)*
-
-**Advanced Crawler Configuration**
-*(Keep example, ensure cache_mode uses valid enum value like "bypass")*
-
-**Extraction Strategy**
-```json
-{
-    "crawler_config": {
-        "type": "CrawlerRunConfig",
-        "params": {
-            "extraction_strategy": {
-                "type": "JsonCssExtractionStrategy",
-                "params": {
-                    "schema": {
-                        "type": "dict",
-                        "value": {
-                           "baseSelector": "article.post",
-                           "fields": [
-                               {"name": "title", "selector": "h1", "type": "text"},
-                               {"name": "content", "selector": ".content", "type": "html"}
-                           ]
-                         }
-                    }
-                }
-            }
-        }
-    }
-}
-```
-
-**LLM Extraction Strategy** *(Keep example, ensure schema uses type/value wrapper)*
-*(Keep Deep Crawler Example)*
-
-### REST API Examples
-
-Update URLs to use port `11235`.
-
-#### Simple Crawl
-
-```python
-import requests
-
-# Configuration objects converted to the required JSON structure
-browser_config_payload = {
-    "type": "BrowserConfig",
-    "params": {"headless": True}
-}
-crawler_config_payload = {
-    "type": "CrawlerRunConfig",
-    "params": {"stream": False, "cache_mode": "bypass"} # Use string value of enum
-}
-
-crawl_payload = {
-    "urls": ["https://httpbin.org/html"],
-    "browser_config": browser_config_payload,
-    "crawler_config": crawler_config_payload
-}
-response = requests.post(
-    "http://localhost:11235/crawl", # Updated port
-    # headers={"Authorization": f"Bearer {token}"},  # If JWT is enabled
-    json=crawl_payload
-)
-print(f"Status Code: {response.status_code}")
-if response.ok:
-    print(response.json())
-else:
-    print(f"Error: {response.text}")
-
-```
-
-#### Streaming Results
-
-```python
-import json
-import httpx # Use httpx for async streaming example
-
-async def test_stream_crawl(token: str = None): # Made token optional
-    """Test the /crawl/stream endpoint with multiple URLs."""
-    url = "http://localhost:11235/crawl/stream" # Updated port
-    payload = {
-        "urls": [
-            "https://httpbin.org/html",
-            "https://httpbin.org/links/5/0",
-        ],
-        "browser_config": {
-            "type": "BrowserConfig",
-            "params": {"headless": True, "viewport": {"type": "dict", "value": {"width": 1200, "height": 800}}} # Viewport needs type:dict
-        },
-        "crawler_config": {
-            "type": "CrawlerRunConfig",
-            "params": {"stream": True, "cache_mode": "bypass"}
-        }
-    }
-
-    headers = {}
-    # if token:
-    #    headers = {"Authorization": f"Bearer {token}"} # If JWT is enabled
-
-    try:
-        async with httpx.AsyncClient() as client:
-            async with client.stream("POST", url, json=payload, headers=headers, timeout=120.0) as response:
-                print(f"Status: {response.status_code} (Expected: 200)")
-                response.raise_for_status() # Raise exception for bad status codes
-
-                # Read streaming response line-by-line (NDJSON)
-                async for line in response.aiter_lines():
-                    if line:
-                        try:
-                            data = json.loads(line)
-                            # Check for completion marker
-                            if data.get("status") == "completed":
-                                print("Stream completed.")
-                                break
-                            print(f"Streamed Result: {json.dumps(data, indent=2)}")
-                        except json.JSONDecodeError:
-                            print(f"Warning: Could not decode JSON line: {line}")
-
-    except httpx.HTTPStatusError as e:
-         print(f"HTTP error occurred: {e.response.status_code} - {e.response.text}")
-    except Exception as e:
-        print(f"Error in streaming crawl test: {str(e)}")
-
-# To run this example:
-# import asyncio
-# asyncio.run(test_stream_crawl())
-```
-
----
-
-## Metrics & Monitoring
-
-Keep an eye on your crawler with these endpoints:
-
-- `/health` - Quick health check
-- `/metrics` - Detailed Prometheus metrics
-- `/schema` - Full API schema
-
-Example health check:
-```bash
-curl http://localhost:11235/health
-```
-
----
-
-*(Deployment Scenarios and Complete Examples sections remain the same, maybe update links if examples moved)*
-
----
-
-## Server Configuration
-
-The server's behavior can be customized through the `config.yml` file.
-
-### Understanding config.yml
-
-The configuration file is loaded from `/app/config.yml` inside the container. By default, the file from `deploy/docker/config.yml` in the repository is copied there during the build.
-
-Here's a detailed breakdown of the configuration options (using defaults from `deploy/docker/config.yml`):
-
-```yaml
-# Application Configuration
-app:
-  title: "Crawl4AI API"
-  version: "1.0.0" # Consider setting this to match library version, e.g., "0.5.1"
-  host: "0.0.0.0"
-  port: 8020 # NOTE: This port is used ONLY when running server.py directly. Gunicorn overrides this (see supervisord.conf).
-  reload: False # Default set to False - suitable for production
-  timeout_keep_alive: 300
-
-# Default LLM Configuration
-llm:
-  provider: "openai/gpt-4o-mini"
-  api_key_env: "OPENAI_API_KEY"
-  # api_key: sk-...  # If you pass the API key directly then api_key_env will be ignored
-
-# Redis Configuration (Used by internal Redis server managed by supervisord)
-redis:
-  host: "localhost"
-  port: 6379
-  db: 0
-  password: ""
-  # ... other redis options ...
-
-# Rate Limiting Configuration
-rate_limiting:
-  enabled: True
-  default_limit: "1000/minute"
-  trusted_proxies: []
-  storage_uri: "memory://"  # Use "redis://localhost:6379" if you need persistent/shared limits
-
-# Security Configuration
-security:
-  enabled: false # Master toggle for security features
-  jwt_enabled: false # Enable JWT authentication (requires security.enabled=true)
-  https_redirect: false # Force HTTPS (requires security.enabled=true)
-  trusted_hosts: ["*"] # Allowed hosts (use specific domains in production)
-  headers: # Security headers (applied if security.enabled=true)
-    x_content_type_options: "nosniff"
-    x_frame_options: "DENY"
-    content_security_policy: "default-src 'self'"
-    strict_transport_security: "max-age=63072000; includeSubDomains"
-
-# Crawler Configuration
-crawler:
-  memory_threshold_percent: 95.0
-  rate_limiter:
-    base_delay: [1.0, 2.0] # Min/max delay between requests in seconds for dispatcher
-  timeouts:
-    stream_init: 30.0  # Timeout for stream initialization
-    batch_process: 300.0 # Timeout for non-streaming /crawl processing
-
-# Logging Configuration
-logging:
-  level: "INFO"
-  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
-
-# Observability Configuration
-observability:
-  prometheus:
-    enabled: True
-    endpoint: "/metrics"
-  health_check:
-    endpoint: "/health"
-```
-
-*(JWT Authentication section remains the same, just note the default port is now 11235 for requests)*
-
-*(Configuration Tips and Best Practices remain the same)*
-
-### Customizing Your Configuration
-
-You can override the default `config.yml`.
-
-#### Method 1: Modify Before Build
-
-1.  Edit the `deploy/docker/config.yml` file in your local repository clone.
-2.  Build the image using `docker buildx` or `docker compose --profile local-... up --build`. The modified file will be copied into the image.
-
-#### Method 2: Runtime Mount (Recommended for Custom Deploys)
-
-1.  Create your custom configuration file, e.g., `my-custom-config.yml` locally. Ensure it contains all necessary sections.
-2.  Mount it when running the container:
-
-    *   **Using `docker run`:**
-        ```bash
-        # Assumes my-custom-config.yml is in the current directory
-        docker run -d -p 11235:11235 \
-          --name crawl4ai-custom-config \
-          --env-file .llm.env \
-          --shm-size=1g \
-          -v $(pwd)/my-custom-config.yml:/app/config.yml \
-          unclecode/crawl4ai:latest # Or your specific tag
-        ```
-
-    *   **Using `docker-compose.yml`:** Add a `volumes` section to the service definition:
-        ```yaml
-        services:
-          crawl4ai-hub-amd64: # Or your chosen service
-            image: unclecode/crawl4ai:latest
-            profiles: ["hub-amd64"]
-            <<: *base-config
-            volumes:
-              # Mount local custom config over the default one in the container
-              - ./my-custom-config.yml:/app/config.yml
-              # Keep the shared memory volume from base-config
-              - /dev/shm:/dev/shm
-        ```
-        *(Note: Ensure `my-custom-config.yml` is in the same directory as `docker-compose.yml`)*
-
-> 💡 When mounting, your custom file *completely replaces* the default one. Ensure it's a valid and complete configuration.
-
-### Configuration Recommendations
-
-1. **Security First** 🔒
-   - Always enable security in production
-   - Use specific trusted_hosts instead of wildcards
-   - Set up proper rate limiting to protect your server
-   - Consider your environment before enabling HTTPS redirect
-
-2. **Resource Management** 💻
-   - Adjust memory_threshold_percent based on available RAM
-   - Set timeouts according to your content size and network conditions
-   - Use Redis for rate limiting in multi-container setups
-
-3. **Monitoring** 📊
-   - Enable Prometheus if you need metrics
-   - Set DEBUG logging in development, INFO in production
-   - Regular health check monitoring is crucial
-
-4. **Performance Tuning** ⚡
-   - Start with conservative rate limiter delays
-   - Increase batch_process timeout for large content
-   - Adjust stream_init timeout based on initial response times
-
-## Getting Help
-
-We're here to help you succeed with Crawl4AI! Here's how to get support:
-
-- 📖 Check our [full documentation](https://docs.crawl4ai.com)
-- 🐛 Found a bug? [Open an issue](https://github.com/unclecode/crawl4ai/issues)
-- 💬 Join our [Discord community](https://discord.gg/crawl4ai)
-- ⭐ Star us on GitHub to show support!
-
-## Summary
-
-In this guide, we've covered everything you need to get started with Crawl4AI's Docker deployment:
-- Building and running the Docker container
-- Configuring the environment
-- Making API requests with proper typing
-- Using the Python SDK
-- Monitoring your deployment
-
-Remember, the examples in the `examples` folder are your friends - they show real-world usage patterns that you can adapt for your needs.
-
-Keep exploring, and don't hesitate to reach out if you need help! We're building something amazing together. 🚀
-
-Happy crawling! 🕷️

deploy/docker/config.yml

@@ -3,9 +3,9 @@ app:
   title: "Crawl4AI API"
   version: "1.0.0"
   host: "0.0.0.0"
-  port: 8020
+  port: 11235
   reload: False
-  workers: 4
+  workers: 1
   timeout_keep_alive: 300
 
 # Default LLM Configuration

deploy/docker/requirements.txt

@@ -1,5 +1,5 @@
-fastapi==0.115.12
-uvicorn==0.34.2
+fastapi>=0.115.12
+uvicorn>=0.34.2
 gunicorn>=23.0.0
 slowapi==0.1.9
 prometheus-fastapi-instrumentator>=7.1.0
@@ -8,8 +8,9 @@ jwt>=1.3.1
 dnspython>=2.7.0
 email-validator==2.2.0
 sse-starlette==2.2.1
-pydantic==2.11
+pydantic>=2.11
 rank-bm25==0.2.2
 anyio==4.9.0
 PyJWT==2.10.1
-
+mcp>=1.6.0
+websockets>=15.0.1

deploy/docker/server.py

@@ -629,6 +629,7 @@ async def get_context(
     
 
 # attach MCP layer (adds /mcp/ws, /mcp/sse, /mcp/schema)
+print(f"MCP server running on {config['app']['host']}:{config['app']['port']}")
 attach_mcp(
     app,
     base_url=f"http://{config['app']['host']}:{config['app']['port']}"

deploy/docker/static/playground/index.html

@@ -536,10 +536,14 @@
 
             const endpointMap = {
                 crawl: '/crawl',
+            };
+
+            /*const endpointMap = {
+                crawl: '/crawl',
                 crawl_stream: '/crawl/stream',
                 md: '/md',
                 llm: '/llm'
-            };
+            };*/
 
             const api = endpointMap[endpoint];
             const payload = {

deploy/docker/supervisord.conf

@@ -14,7 +14,7 @@ stderr_logfile=/dev/stderr      ; Redirect redis stderr to container stderr
 stderr_logfile_maxbytes=0
 
 [program:gunicorn]
-command=/usr/local/bin/gunicorn --bind 0.0.0.0:11235 --workers 2 --threads 2 --timeout 120 --graceful-timeout 30 --keep-alive 60 --log-level info --worker-class uvicorn.workers.UvicornWorker server:app
+command=/usr/local/bin/gunicorn --bind 0.0.0.0:11235 --workers 1 --threads 4 --timeout 1800 --graceful-timeout 30 --keep-alive 300 --log-level info --worker-class uvicorn.workers.UvicornWorker server:app
 directory=/app                  ; Working directory for the app
 user=appuser                    ; Run gunicorn as our non-root user
 autorestart=true

docker-compose.yml

@@ -1,19 +1,11 @@
-# docker-compose.yml
+version: '3.8'
 
-# Base configuration anchor for reusability
+# Shared configuration for all environments
 x-base-config: &base-config
   ports:
-    # Map host port 11235 to container port 11235 (where Gunicorn will listen)
-    - "11235:11235"
-    # - "8080:8080" # Uncomment if needed
-
-  # Load API keys primarily from .llm.env file
-  # Create .llm.env in the root directory .llm.env.example
+    - "11235:11235"  # Gunicorn port
   env_file:
-    - .llm.env
-
-  # Define environment variables, allowing overrides from host environment
-  # Syntax ${VAR:-} uses host env var 'VAR' if set, otherwise uses value from .llm.env
+    - .llm.env       # API keys (create from .llm.env.example)
   environment:
     - OPENAI_API_KEY=${OPENAI_API_KEY:-}
     - DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY:-}
@@ -22,10 +14,8 @@ x-base-config: &base-config
     - TOGETHER_API_KEY=${TOGETHER_API_KEY:-}
     - MISTRAL_API_KEY=${MISTRAL_API_KEY:-}
     - GEMINI_API_TOKEN=${GEMINI_API_TOKEN:-}
-
   volumes:
-    # Mount /dev/shm for Chromium/Playwright performance
-    - /dev/shm:/dev/shm
+    - /dev/shm:/dev/shm  # Chromium performance
   deploy:
     resources:
       limits:
@@ -34,47 +24,26 @@ x-base-config: &base-config
         memory: 1G
   restart: unless-stopped
   healthcheck:
-    # IMPORTANT: Ensure Gunicorn binds to 11235 in supervisord.conf
     test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
     interval: 30s
     timeout: 10s
     retries: 3
-    start_period: 40s # Give the server time to start
-  # Run the container as the non-root user defined in the Dockerfile
+    start_period: 40s
   user: "appuser"
 
 services:
-  # --- Local Build Services ---
-  crawl4ai-local-amd64:
+  crawl4ai:
+    # 1. Default: Pull multi-platform test image from Docker Hub
+    # 2. Override with local image via: IMAGE=local-test docker compose up
+    image: ${IMAGE:-unclecode/crawl4ai:${TAG:-latest}}
+    
+    # Local build config (used with --build)
     build:
-      context: . # Build context is the root directory
-      dockerfile: Dockerfile # Dockerfile is in the root directory
+      context: .
+      dockerfile: Dockerfile
       args:
         INSTALL_TYPE: ${INSTALL_TYPE:-default}
         ENABLE_GPU: ${ENABLE_GPU:-false}
-        # PYTHON_VERSION arg is omitted as it's fixed by 'FROM python:3.10-slim' in Dockerfile
-    platform: linux/amd64
-    profiles: ["local-amd64"]
-    <<: *base-config # Inherit base configuration
-
-  crawl4ai-local-arm64:
-    build:
-      context: . # Build context is the root directory
-      dockerfile: Dockerfile # Dockerfile is in the root directory
-      args:
-        INSTALL_TYPE: ${INSTALL_TYPE:-default}
-        ENABLE_GPU: ${ENABLE_GPU:-false}
-    platform: linux/arm64
-    profiles: ["local-arm64"]
-    <<: *base-config
-
-  # --- Docker Hub Image Services ---
-  crawl4ai-hub-amd64:
-    image: unclecode/crawl4ai:${VERSION:-latest}-amd64
-    profiles: ["hub-amd64"]
-    <<: *base-config
-
-  crawl4ai-hub-arm64:
-    image: unclecode/crawl4ai:${VERSION:-latest}-arm64
-    profiles: ["hub-arm64"]
+    
+    # Inherit shared config
     <<: *base-config

docs/examples/crypto_analysis_example.py

@@ -383,29 +383,29 @@ async def main():
             scroll_delay=0.2,
         )
 
-        # # Execute market data extraction
-        # results: List[CrawlResult] = await crawler.arun(
-        #     url="https://coinmarketcap.com/?page=1", config=crawl_config
-        # )
+        # Execute market data extraction
+        results: List[CrawlResult] = await crawler.arun(
+            url="https://coinmarketcap.com/?page=1", config=crawl_config
+        )
 
-        # # Process results
-        # raw_df = pd.DataFrame()
-        # for result in results:
-        #     if result.success and result.media["tables"]:
-        #         # Extract primary market table
-        #         # DataFrame
-        #         raw_df = pd.DataFrame(
-        #             result.media["tables"][0]["rows"],
-        #             columns=result.media["tables"][0]["headers"],
-        #         )
-        #         break
+        # Process results
+        raw_df = pd.DataFrame()
+        for result in results:
+            if result.success and result.media["tables"]:
+                # Extract primary market table
+                # DataFrame
+                raw_df = pd.DataFrame(
+                    result.media["tables"][0]["rows"],
+                    columns=result.media["tables"][0]["headers"],
+                )
+                break
 
 
         # This is for debugging only
         # ////// Remove this in production from here..
         # Save raw data for debugging
-        # raw_df.to_csv(f"{__current_dir__}/tmp/raw_crypto_data.csv", index=False)
-        # print("🔍 Raw data saved to 'raw_crypto_data.csv'")
+        raw_df.to_csv(f"{__current_dir__}/tmp/raw_crypto_data.csv", index=False)
+        print("🔍 Raw data saved to 'raw_crypto_data.csv'")
 
         # Read from file for debugging
         raw_df = pd.read_csv(f"{__current_dir__}/tmp/raw_crypto_data.csv")

docs/examples/docker/demo_docker_api.py

@@ -398,7 +398,6 @@ async def demo_param_js_execution(client: httpx.AsyncClient):
     elif results:
          console.print("[yellow]JS Execution Result not found in response.[/]")
 
-
 async def demo_param_screenshot(client: httpx.AsyncClient):
     payload = {
         "urls": [SIMPLE_URL],
@@ -430,8 +429,6 @@ async def demo_param_ssl_fetch(client: httpx.AsyncClient):
     elif results:
          console.print("[yellow]SSL Certificate data not found in response.[/]")
 
-
-
 async def demo_param_proxy(client: httpx.AsyncClient):
     proxy_params_list = load_proxies_from_env() # Get the list of parameter dicts
     if not proxy_params_list:

docs/md_v2/ask_ai/ask-ai.js

@@ -361,8 +361,10 @@ A code snippet: \`crawler.run()\`. Check the [quickstart](/core/quickstart).`;
         chatMessages.innerHTML = ""; // Start with clean slate for query
         if (!isFromQuery) {
             // Show welcome only if manually started
+            // chatMessages.innerHTML =
+            //     '<div class="message ai-message welcome-message">Started a new chat! Ask me anything about Crawl4AI.</div>';
             chatMessages.innerHTML =
-                '<div class="message ai-message welcome-message">Started a new chat! Ask me anything about Crawl4AI.</div>';
+                '<div class="message ai-message welcome-message">We will launch this feature very soon.</div>';
         }
         addCitations([]); // Clear citations
         updateCitationsDisplay(); // Clear UI
@@ -504,8 +506,10 @@ A code snippet: \`crawler.run()\`. Check the [quickstart](/core/quickstart).`;
             addMessageToChat(message, false);
         });
         if (messages.length === 0) {
+            // chatMessages.innerHTML =
+            //     '<div class="message ai-message welcome-message">Chat history loaded. Ask a question!</div>';
             chatMessages.innerHTML =
-                '<div class="message ai-message welcome-message">Chat history loaded. Ask a question!</div>';
+                '<div class="message ai-message welcome-message">We will launch this feature very soon.</div>';
         }
         // Scroll to bottom after loading messages
         scrollToBottom();

docs/md_v2/ask_ai/index.html

@@ -36,7 +36,7 @@
             <div id="chat-input-area">
                 <!-- Loading indicator for general waiting (optional) -->
                 <!-- <div class="loading-indicator" style="display: none;">Thinking...</div> -->
-                <textarea id="chat-input" placeholder="Ask about Crawl4AI..." rows="2"></textarea> 
+                <textarea id="chat-input" placeholder="We will roll out this feature very soon." rows="2" disabled></textarea> 
                 <button id="send-button">Send</button>
             </div>
         </main>

docs/md_v2/assets/layout.css

@@ -64,7 +64,7 @@ body {
     /* Apply side padding within the centered block */
     padding-left: calc(var(--global-space) * 2);
     padding-right: calc(var(--global-space) * 2);
-    /* Add margin-left to clear the fixed sidebar */
+    /* Add margin-left to clear the fixed sidebar - ONLY ON DESKTOP */
     margin-left: var(--sidebar-width);
 }
 
@@ -81,7 +81,7 @@ body {
     z-index: 900;
     padding: 1em calc(var(--global-space) * 2);
     padding-bottom: 2em;
-    /* transition: left var(--layout-transition-speed) ease-in-out; */
+    transition: left var(--layout-transition-speed) ease-in-out;
 }
 
 /* --- 2. Main Content Area (Within Centered Grid) --- */
@@ -188,21 +188,133 @@ footer {
     }
 }
 
+/* --- Mobile Menu Styles --- */
+.mobile-menu-toggle {
+    display: none; /* Hidden by default, shown in mobile */
+    background: none;
+    border: none;
+    padding: 10px;
+    cursor: pointer;
+    z-index: 1200;
+    margin-right: 10px;
+    position: absolute;
+    left: 10px;
+    top: 50%;
+    transform: translateY(-50%);
+    /* Make sure it doesn't get moved */
+    min-width: 30px;
+    min-height: 30px;
+}
+
+.hamburger-line {
+    display: block;
+    width: 22px;
+    height: 2px;
+    margin: 5px 0;
+    background-color: var(--font-color);
+    transition: transform 0.3s, opacity 0.3s;
+}
+
+/* Hamburger animation */
+.mobile-menu-toggle.is-active .hamburger-line:nth-child(1) {
+    transform: translateY(7px) rotate(45deg);
+}
+
+.mobile-menu-toggle.is-active .hamburger-line:nth-child(2) {
+    opacity: 0;
+}
+
+.mobile-menu-toggle.is-active .hamburger-line:nth-child(3) {
+    transform: translateY(-7px) rotate(-45deg);
+}
+
+.mobile-menu-close {
+    display: none; /* Hidden by default, shown in mobile */
+    position: absolute;
+    top: 10px;
+    right: 10px;
+    background: none;
+    border: none;
+    color: var(--font-color);
+    font-size: 24px;
+    cursor: pointer;
+    z-index: 1200;
+    padding: 5px 10px;
+}
+
+.mobile-menu-backdrop {
+    position: fixed;
+    top: 0;
+    left: 0;
+    right: 0;
+    bottom: 0;
+    background-color: rgba(0, 0, 0, 0.7);
+    z-index: 1050;
+}
+
 /* --- Small screens: Hide left sidebar, full width content & footer --- */
 @media screen and (max-width: 768px) {
+    /* Hide the terminal-menu from theme */
+    .terminal-menu {
+        display: none !important;
+    }
+    
+    /* Add padding to site name to prevent hamburger overlap */
+    .terminal-mkdocs-site-name,
+    .terminal-logo a,
+    .terminal-nav .logo {
+        padding-left: 40px !important;
+        white-space: nowrap;
+        overflow: hidden;
+        text-overflow: ellipsis;
+    }
+    
+    /* Show mobile menu toggle button */
+    .mobile-menu-toggle {
+        display: block;
+    }
+    
+    /* Show mobile menu close button */
+    .mobile-menu-close {
+        display: block;
+    }
 
     #terminal-mkdocs-side-panel {
-        left: calc(-1 * var(--sidebar-width));
+        left: -100%; /* Hide completely off-screen */
         z-index: 1100;
         box-shadow: 2px 0 10px rgba(0,0,0,0.3);
+        top: 0; /* Start from top edge */
+        height: 100%; /* Full height */
+        transition: left 0.3s ease-in-out;
+        padding-top: 50px; /* Space for close button */
+        overflow-y: auto;
+        width: 85%; /* Wider on mobile */
+        max-width: 320px; /* Maximum width */
+        background-color: var(--background-color); /* Ensure solid background */
     }
+    
     #terminal-mkdocs-side-panel.sidebar-visible {
         left: 0;
     }
+    
+    /* Make navigation links more touch-friendly */
+    #terminal-mkdocs-side-panel a {
+        padding: 6px 15px;
+        display: block;
+        /* No border as requested */
+    }
+    
+    #terminal-mkdocs-side-panel ul {
+        padding-left: 0;
+    }
+    
+    #terminal-mkdocs-side-panel ul ul a {
+        padding-left: 10px;
+    }
 
     .terminal-mkdocs-main-grid {
         /* Grid now takes full width (minus body padding) */
-        margin-left: 0; /* Override sidebar margin */
+        margin-left: 0 !important; /* Override sidebar margin with !important */
         margin-right: 0; /* Override auto margin */
         max-width: 100%; /* Allow full width */
         padding-left: var(--global-space); /* Reduce padding */
@@ -224,7 +336,6 @@ footer {
          text-align: center;
          gap: 0.5em;
     }
-    /* Remember JS for toggle button & overlay */
 }
 
 

docs/md_v2/assets/mobile_menu.js

@@ -0,0 +1,106 @@
+// mobile_menu.js - Hamburger menu for mobile view
+document.addEventListener('DOMContentLoaded', () => {
+    // Get references to key elements
+    const sidePanel = document.getElementById('terminal-mkdocs-side-panel');
+    const mainHeader = document.querySelector('.terminal .container:first-child');
+    
+    if (!sidePanel || !mainHeader) {
+        console.warn('Mobile menu: Required elements not found');
+        return;
+    }
+    
+    // Force hide sidebar on mobile
+    const checkMobile = () => {
+        if (window.innerWidth <= 768) {
+            // Force with !important-like priority
+            sidePanel.style.setProperty('left', '-100%', 'important');
+            // Also hide terminal-menu from the theme
+            const terminalMenu = document.querySelector('.terminal-menu');
+            if (terminalMenu) {
+                terminalMenu.style.setProperty('display', 'none', 'important');
+            }
+        } else {
+            sidePanel.style.removeProperty('left');
+            // Restore terminal-menu if it exists
+            const terminalMenu = document.querySelector('.terminal-menu');
+            if (terminalMenu) {
+                terminalMenu.style.removeProperty('display');
+            }
+        }
+    };
+    
+    // Run on initial load
+    checkMobile();
+    
+    // Also run on resize
+    window.addEventListener('resize', checkMobile);
+    
+    // Create hamburger button
+    const hamburgerBtn = document.createElement('button');
+    hamburgerBtn.className = 'mobile-menu-toggle';
+    hamburgerBtn.setAttribute('aria-label', 'Toggle navigation menu');
+    hamburgerBtn.innerHTML = `
+        <span class="hamburger-line"></span>
+        <span class="hamburger-line"></span>
+        <span class="hamburger-line"></span>
+    `;
+    
+    // Create backdrop overlay
+    const menuBackdrop = document.createElement('div');
+    menuBackdrop.className = 'mobile-menu-backdrop';
+    menuBackdrop.style.display = 'none';
+    document.body.appendChild(menuBackdrop);
+    
+    // Make sure it's properly hidden on page load
+    if (window.innerWidth <= 768) {
+        menuBackdrop.style.display = 'none';
+    }
+    
+    // Insert hamburger button into header
+    mainHeader.insertBefore(hamburgerBtn, mainHeader.firstChild);
+    
+    // Add menu close button to side panel
+    const closeBtn = document.createElement('button');
+    closeBtn.className = 'mobile-menu-close';
+    closeBtn.setAttribute('aria-label', 'Close navigation menu');
+    closeBtn.innerHTML = `&times;`;
+    sidePanel.insertBefore(closeBtn, sidePanel.firstChild);
+    
+    // Toggle function
+    function toggleMobileMenu() {
+        const isOpen = sidePanel.classList.toggle('sidebar-visible');
+        
+        // Toggle backdrop
+        menuBackdrop.style.display = isOpen ? 'block' : 'none';
+        
+        // Toggle aria-expanded
+        hamburgerBtn.setAttribute('aria-expanded', isOpen ? 'true' : 'false');
+        
+        // Toggle hamburger animation class
+        hamburgerBtn.classList.toggle('is-active');
+        
+        // Force sidebar visibility setting
+        if (isOpen) {
+            sidePanel.style.setProperty('left', '0', 'important');
+        } else {
+            sidePanel.style.setProperty('left', '-100%', 'important');
+        }
+        
+        // Prevent body scrolling when menu is open
+        document.body.style.overflow = isOpen ? 'hidden' : '';
+    }
+    
+    // Event listeners
+    hamburgerBtn.addEventListener('click', toggleMobileMenu);
+    closeBtn.addEventListener('click', toggleMobileMenu);
+    menuBackdrop.addEventListener('click', toggleMobileMenu);
+    
+    // Close menu on window resize to desktop
+    window.addEventListener('resize', () => {
+        if (window.innerWidth > 768 && sidePanel.classList.contains('sidebar-visible')) {
+            toggleMobileMenu();
+        }
+    });
+    
+    console.log('Mobile menu initialized');
+});

docs/md_v2/assets/styles.css

@@ -268,3 +268,6 @@ div.badges a > img {
 }
 
 
+table td, table th {
+    border: 1px solid var(--code-bg-color) !important;
+}

docs/md_v2/blog/releases/0.6.0.md

@@ -0,0 +1,51 @@
+# Crawl4AI 0.6.0
+
+*Release date: 2025‑04‑22*
+
+0.6.0 is the **biggest jump** since the 0.5 series, packing a smarter browser core, pool‑based crawlers, and a ton of DX candy. Expect faster runs, lower RAM burn, and richer diagnostics.
+
+---
+
+## 🚀 Key upgrades
+
+| Area | What changed |
+|------|--------------|
+| **Browser** | New **Browser** management with pooling, page pre‑warm, geolocation + locale + timezone switches |
+| **Crawler** | Console and network log capture, MHTML snapshots, safer `get_page` API |
+| **Server & API** | **Crawler Pool Manager** endpoint, MCP socket + SSE support |
+| **Docs** | v2 layout, floating Ask‑AI helper, GitHub stats badge, copy‑code buttons, Docker API demo |
+| **Tests** | Memory + load benchmarks, 90+ new cases covering MCP and Docker |
+
+---
+
+## ⚠️ Breaking changes
+
+1. **`get_page` signature** – returns `(html, metadata)` instead of plain html.
+2. **Docker** – new Chromium base layer, rebuild images.
+
+---
+
+## How to upgrade
+
+```bash
+pip install -U crawl4ai==0.6.0
+```
+
+---
+
+## Full changelog
+
+The diff between `main` and `next` spans **36 k insertions, 4.9 k deletions** over 121 files. Read the [compare view](https://github.com/unclecode/crawl4ai/compare/0.5.0.post8...0.6.0) or see `CHANGELOG.md` for the granular list.
+
+---
+
+## Upgrade tips
+
+* Using the Docker API? Pull `unclecode/crawl4ai:0.6.0`, new args are documented in `/deploy/docker/README.md`.
+* Stress‑test your stack with `tests/memory/run_benchmark.py` before production rollout.
+* Markdown generators renamed but aliased, update when convenient, warnings will remind you.
+
+---
+
+Happy crawling, ping `@unclecode` on X for questions or memes.
+

docs/md_v2/core/examples.md

@@ -0,0 +1,115 @@
+# Code Examples
+
+This page provides a comprehensive list of example scripts that demonstrate various features and capabilities of Crawl4AI. Each example is designed to showcase specific functionality, making it easier for you to understand how to implement these features in your own projects.
+
+## Getting Started Examples
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Hello World | A simple introductory example demonstrating basic usage of AsyncWebCrawler with JavaScript execution and content filtering. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/hello_world.py) |
+| Quickstart | A comprehensive collection of examples showcasing various features including basic crawling, content cleaning, link analysis, JavaScript execution, CSS selectors, media handling, custom hooks, proxy configuration, screenshots, and multiple extraction strategies. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/quickstart.py) |
+| Quickstart Set 1 | Basic examples for getting started with Crawl4AI. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/quickstart_examples_set_1.py) |
+| Quickstart Set 2 | More advanced examples for working with Crawl4AI. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/quickstart_examples_set_2.py) |
+
+## Browser & Crawling Features
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Built-in Browser | Demonstrates how to use the built-in browser capabilities. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/builtin_browser_example.py) |
+| Browser Optimization | Focuses on browser performance optimization techniques. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/browser_optimization_example.py) |
+| arun vs arun_many | Compares the `arun` and `arun_many` methods for single vs. multiple URL crawling. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/arun_vs_arun_many.py) |
+| Multiple URLs | Shows how to crawl multiple URLs asynchronously. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/async_webcrawler_multiple_urls_example.py) |
+| Page Interaction | Guide on interacting with dynamic elements through clicks. | [View Guide](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/tutorial_dynamic_clicks.md) |
+| Crawler Monitor | Shows how to monitor the crawler's activities and status. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/crawler_monitor_example.py) |
+| Full Page Screenshot & PDF | Guide on capturing full-page screenshots and PDFs from massive webpages. | [View Guide](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/full_page_screenshot_and_pdf_export.md) |
+
+## Advanced Crawling & Deep Crawling
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Deep Crawling | An extensive tutorial on deep crawling capabilities, demonstrating BFS and BestFirst strategies, stream vs. non-stream execution, filters, scorers, and advanced configurations. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/deepcrawl_example.py) |
+| Dispatcher | Shows how to use the crawl dispatcher for advanced workload management. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/dispatcher_example.py) |
+| Storage State | Tutorial on managing browser storage state for persistence. | [View Guide](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/storage_state_tutorial.md) |
+| Network Console Capture | Demonstrates how to capture and analyze network requests and console logs. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/network_console_capture_example.py) |
+
+## Extraction Strategies
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Extraction Strategies | Demonstrates different extraction strategies with various input formats (markdown, HTML, fit_markdown) and JSON-based extractors (CSS and XPath). | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/extraction_strategies_examples.py) |
+| Scraping Strategies | Compares the performance of different scraping strategies. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/scraping_strategies_performance.py) |
+| LLM Extraction | Demonstrates LLM-based extraction specifically for OpenAI pricing data. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/llm_extraction_openai_pricing.py) |
+| LLM Markdown | Shows how to use LLMs to generate markdown from crawled content. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/llm_markdown_generator.py) |
+| Summarize Page | Shows how to summarize web page content. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/summarize_page.py) |
+
+## E-commerce & Specialized Crawling
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Amazon Product Extraction | Demonstrates how to extract structured product data from Amazon search results using CSS selectors. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/amazon_product_extraction_direct_url.py) |
+| Amazon with Hooks | Shows how to use hooks with Amazon product extraction. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/amazon_product_extraction_using_hooks.py) |
+| Amazon with JavaScript | Demonstrates using custom JavaScript for Amazon product extraction. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/amazon_product_extraction_using_use_javascript.py) |
+| Crypto Analysis | Demonstrates how to crawl and analyze cryptocurrency data. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/crypto_analysis_example.py) |
+| SERP API | Demonstrates using Crawl4AI with search engine result pages. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/serp_api_project_11_feb.py) |
+
+## Customization & Security
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Hooks | Illustrates how to use hooks at different stages of the crawling process for advanced customization. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/hooks_example.py) |
+| Identity-Based Browsing | Illustrates identity-based browsing configurations for authentic browsing experiences. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/identity_based_browsing.py) |
+| Proxy Rotation | Shows how to use proxy rotation for web scraping and avoiding IP blocks. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/proxy_rotation_demo.py) |
+| SSL Certificate | Illustrates SSL certificate handling and verification. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/ssl_example.py) |
+| Language Support | Shows how to handle different languages during crawling. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/language_support_example.py) |
+| Geolocation | Demonstrates how to use geolocation features. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/use_geo_location.py) |
+
+## Docker & Deployment
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Docker Config | Demonstrates how to create and use Docker configuration objects. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_config_obj.py) |
+| Docker Basic | A test suite for Docker deployment, showcasing various functionalities through the Docker API. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_example.py) |
+| Docker REST API | Shows how to interact with Crawl4AI Docker using REST API calls. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_python_rest_api.py) |
+| Docker SDK | Demonstrates using the Python SDK for Crawl4AI Docker. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_python_sdk.py) |
+
+## Application Examples
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Research Assistant | Demonstrates how to build a research assistant using Crawl4AI. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/research_assistant.py) |
+| REST Call | Shows how to make REST API calls with Crawl4AI. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/rest_call.py) |
+| Chainlit Integration | Shows how to integrate Crawl4AI with Chainlit. | [View Guide](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/chainlit.md) |
+| Crawl4AI vs FireCrawl | Compares Crawl4AI with the FireCrawl library. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/crawlai_vs_firecrawl.py) |
+
+## Content Generation & Markdown
+
+| Example | Description | Link |
+|---------|-------------|------|
+| Content Source | Demonstrates how to work with different content sources in markdown generation. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/markdown/content_source_example.py) |
+| Content Source (Short) | A simplified version of content source usage. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/markdown/content_source_short_example.py) |
+| Built-in Browser Guide | Guide for using the built-in browser capabilities. | [View Guide](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/README_BUILTIN_BROWSER.md) |
+
+## Running the Examples
+
+To run any of these examples, you'll need to have Crawl4AI installed:
+
+```bash
+pip install crawl4ai
+```
+
+Then, you can run an example script like this:
+
+```bash
+python -m docs.examples.hello_world
+```
+
+For examples that require additional dependencies or environment variables, refer to the comments at the top of each file.
+
+Some examples may require:
+- API keys (for LLM-based examples)
+- Docker setup (for Docker-related examples)
+- Additional dependencies (specified in the example files)
+
+## Contributing New Examples
+
+If you've created an interesting example that demonstrates a unique use case or feature of Crawl4AI, we encourage you to contribute it to our examples collection. Please see our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md) for more information.

docs/md_v2/index.md

@@ -72,6 +72,14 @@ asyncio.run(main())
 
 ---
 
+## Video Tutorial
+
+<div align="center">
+  <iframe width="560" height="315" src="https://www.youtube.com/embed/xo3qK6Hg9AA?start=15" title="Crawl4AI Tutorial" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+---
+
 ## What Does Crawl4AI Do?
 
 Crawl4AI is a feature-rich crawler and scraper that aims to:

mkdocs.yml

@@ -1,4 +1,4 @@
-site_name: Crawl4AI Documentation (v0.5.x)
+site_name: Crawl4AI Documentation (v0.6.x)
 site_description:  🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper
 site_url: https://docs.crawl4ai.com
 repo_url: https://github.com/unclecode/crawl4ai
@@ -9,6 +9,7 @@ nav:
   - Home: 'index.md'
   - "Ask AI": "core/ask-ai.md"
   - "Quick Start": "core/quickstart.md"
+  - "Code Examples": "core/examples.md"
   - Setup & Installation:
     - "Installation": "core/installation.md"
     - "Docker Deployment": "core/docker-deployment.md"
@@ -90,4 +91,5 @@ extra_javascript:
   - assets/github_stats.js 
   - assets/selection_ask_ai.js
   - assets/copy_code.js
-  - assets/floating_ask_ai_button.js
+  - assets/floating_ask_ai_button.js
+  - assets/mobile_menu.js

pyproject.toml

@@ -8,7 +8,7 @@ dynamic = ["version"]
 description = "🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper"
 readme = "README.md"
 requires-python = ">=3.9"
-license = {text = "MIT"}
+license = "Apache-2.0"
 authors = [
     {name = "Unclecode", email = "unclecode@kidocode.com"}
 ]
@@ -48,7 +48,6 @@ dependencies = [
 classifiers = [
     "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
-    "License :: OSI Approved :: Apache Software License",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",

setup.py

@@ -49,13 +49,12 @@ setup(
     url="https://github.com/unclecode/crawl4ai",
     author="Unclecode",
     author_email="unclecode@kidocode.com",
-    license="MIT",
+    license="Apache-2.0",
     packages=find_packages(),
     package_data={"crawl4ai": ["js_snippet/*.js"]},
     classifiers=[
         "Development Status :: 3 - Alpha",
         "Intended Audience :: Developers",
-        "License :: OSI Approved :: Apache Software License",
         "Programming Language :: Python :: 3",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",

tests/mcp/test_mcp_socket.py

@@ -101,19 +101,19 @@ async def test_context(s: ClientSession):
 
 
 async def main() -> None:
-    async with websocket_client("ws://localhost:8020/mcp/ws") as (r, w):
+    async with websocket_client("ws://localhost:11235/mcp/ws") as (r, w):
         async with ClientSession(r, w) as s:
             await s.initialize()                       # handshake
             tools = (await s.list_tools()).tools
             print("tools:", [t.name for t in tools])
 
             # await test_list()
-            # await test_crawl(s)
-            # await test_md(s)
-            # await test_screenshot(s)
-            # await test_pdf(s)
-            # await test_execute_js(s)
-            # await test_html(s)
+            await test_crawl(s)
+            await test_md(s)
+            await test_screenshot(s)
+            await test_pdf(s)
+            await test_execute_js(s)
+            await test_html(s)
             await test_context(s)
 
 anyio.run(main)