Update gitignore

Merge branch 'main' of https://github.com/unclecode/crawl4ai
Merge pull request #194 from IdrisHanafi/feat/customize-crawl-base-directory
2024-10-27 19:29:04 +08:00 · 2024-10-27 19:28:05 +08:00 · 2024-10-24 13:09:27 +02:00 · 2024-10-21 17:58:39 -04:00 · 2024-10-17 21:38:48 +08:00 · 2024-10-14 22:58:27 +08:00
105 changed files with 11505 additions and 1221 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -165,6 +165,8 @@ Crawl4AI.egg-info/
 Crawl4AI.egg-info/*
 crawler_data.db
 .vscode/
+.tests/
+.test_pads/
 test_pad.py
 test_pad*.py
 .data/
@@ -185,4 +187,24 @@ local/

 a.txt
 .lambda_function.py
-ec2*
+ec2*
+
+update_changelog.sh
+
+.DS_Store
+docs/.DS_Store
+tmp/
+test_env/
+**/.DS_Store
+**/.DS_Store
+
+todo.md
+git_changes.py
+git_changes.md
+pypi_build.sh
+git_issues.py
+git_issues.md
+
+.tests/
+
+.issues/
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,173 @@
 # Changelog

+## [v0.3.6] - 2024-10-12 
+
+### 1. Improved Crawling Control
+- **New Hook**: Added `before_retrieve_html` hook in `AsyncPlaywrightCrawlerStrategy`.
+- **Delayed HTML Retrieval**: Introduced `delay_before_return_html` parameter to allow waiting before retrieving HTML content.
+  - Useful for pages with delayed content loading.
+- **Flexible Timeout**: `smart_wait` function now uses `page_timeout` (default 60 seconds) instead of a fixed 30-second timeout.
+  - Provides better handling for slow-loading pages.
+- **How to use**: Set `page_timeout=your_desired_timeout` (in milliseconds) when calling `crawler.arun()`.
+
+### 2. Browser Type Selection
+- Added support for different browser types (Chromium, Firefox, WebKit).
+- Users can now specify the browser type when initializing AsyncWebCrawler.
+- **How to use**: Set `browser_type="firefox"` or `browser_type="webkit"` when initializing AsyncWebCrawler.
+
+### 3. Screenshot Capture
+- Added ability to capture screenshots during crawling.
+- Useful for debugging and content verification.
+- **How to use**: Set `screenshot=True` when calling `crawler.arun()`.
+
+### 4. Enhanced LLM Extraction Strategy
+- Added support for multiple LLM providers (OpenAI, Hugging Face, Ollama).
+- **Custom Arguments**: Added support for passing extra arguments to LLM providers via `extra_args` parameter.
+- **Custom Headers**: Users can now pass custom headers to the extraction strategy.
+- **How to use**: Specify the desired provider and custom arguments when using `LLMExtractionStrategy`.
+
+### 5. iframe Content Extraction
+- New feature to process and extract content from iframes.
+- **How to use**: Set `process_iframes=True` in the crawl method.
+
+### 6. Delayed Content Retrieval
+- Introduced `get_delayed_content` method in `AsyncCrawlResponse`.
+- Allows retrieval of content after a specified delay, useful for dynamically loaded content.
+- **How to use**: Access `result.get_delayed_content(delay_in_seconds)` after crawling.
+
+## Improvements and Optimizations
+
+### 1. AsyncWebCrawler Enhancements
+- **Flexible Initialization**: Now accepts arbitrary keyword arguments, passed directly to the crawler strategy.
+- Allows for more customized setups.
+
+### 2. Image Processing Optimization
+- Enhanced image handling in WebScrappingStrategy.
+- Added filtering for small, invisible, or irrelevant images.
+- Improved image scoring system for better content relevance.
+- Implemented JavaScript-based image dimension updating for more accurate representation.
+
+### 3. Database Schema Auto-updates
+- Automatic database schema updates ensure compatibility with the latest version.
+
+### 4. Enhanced Error Handling and Logging
+- Improved error messages and logging for easier debugging.
+
+### 5. Content Extraction Refinements
+- Refined HTML sanitization process.
+- Improved handling of base64 encoded images.
+- Enhanced Markdown conversion process.
+- Optimized content extraction algorithms.
+
+### 6. Utility Function Enhancements
+- `perform_completion_with_backoff` function now supports additional arguments for more customized API calls to LLM providers.
+
+## Bug Fixes
+- Fixed an issue where image tags were being prematurely removed during content extraction.
+
+## Examples and Documentation
+- Updated `quickstart_async.py` with examples of:
+  - Using custom headers in LLM extraction.
+  - Different LLM provider usage (OpenAI, Hugging Face, Ollama).
+  - Custom browser type usage.
+
+## Developer Notes
+- Refactored code for better maintainability, flexibility, and performance.
+- Enhanced type hinting throughout the codebase for improved development experience.
+- Expanded error handling for more robust operation.
+
+These updates significantly enhance the flexibility, accuracy, and robustness of crawl4ai, providing users with more control and options for their web crawling and content extraction tasks.
+
+## [v0.3.5] - 2024-09-02
+
+Enhance AsyncWebCrawler with smart waiting and screenshot capabilities
+
+- Implement smart_wait function in AsyncPlaywrightCrawlerStrategy
+- Add screenshot support to AsyncCrawlResponse and AsyncWebCrawler
+- Improve error handling and timeout management in crawling process
+- Fix typo in CrawlResult model (responser_headers -> response_headers)
+
+## [v0.2.77] - 2024-08-04
+
+Significant improvements in text processing and performance:
+
+- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
+- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
+- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
+- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
+
+These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
+
+## [v0.2.76] - 2024-08-02
+
+Major improvements in functionality, performance, and cross-platform compatibility! 🚀
+
+- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
+- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
+- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
+- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
+- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
+
+A big shoutout to our amazing community contributors:
+- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
+- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
+- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
+
+Your contributions are driving Crawl4AI forward! 🙌
+
+## [v0.2.75] - 2024-07-19
+
+Minor improvements for a more maintainable codebase:
+
+- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
+- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
+
+These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
+
+## [v0.2.74] - 2024-07-08
+A slew of exciting updates to improve the crawler's stability and robustness! 🎉
+
+- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
+- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
+- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
+- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
+
+
+## [v0.2.73] - 2024-07-03
+
+💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
+
+* Supporting website need "with-head" mode to crawl the website with head.
+* Fixing the installation issues for setup.py and dockerfile.
+* Resolve multiple issues.
+
+## [v0.2.72] - 2024-06-30
+
+This release brings exciting updates and improvements to our project! 🎉
+
+* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
+* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
+* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
+* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
+
+These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
+
+## [0.2.71] - 2024-06-26
+
+**Improved Error Handling and Performance** 🚧
+
+* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
+* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
+* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
+* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
+
+These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
+
+## [0.2.71] - 2024-06-25
+### Fixed
+- Speed up twice the extraction function.
+
+
 ## [0.2.6] - 2024-06-22
 ### Fixed
 - Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
--- a/CONTRIBUTORS.md
+++ b/CONTRIBUTORS.md
@@ -0,0 +1,32 @@
+# Contributors to Crawl4AI
+
+We would like to thank the following people for their contributions to Crawl4AI:
+
+## Core Team
+
+- [Unclecode](https://github.com/unclecode) - Project Creator and Main Developer
+- [Nasrin](https://github.com/ntohidi) - Project Manager and Developer
+- [Aravind Karnam](https://github.com/aravindkarnam) - Developer
+
+## Community Contributors
+
+- [FractalMind](https://github.com/FractalMind) - Created the first official Docker Hub image and fixed Dockerfile errors
+- [ketonkss4](https://github.com/ketonkss4) - Identified Selenium's new capabilities, helping reduce dependencies
+- [jonymusky](https://github.com/jonymusky) - Javascript execution documentation, and wait_for
+- [datehoer](https://github.com/datehoer) - Add browser prxy support
+
+## Other Contributors
+
+- [Gokhan](https://github.com/gkhngyk) 
+- [Shiv Kumar](https://github.com/shivkumar0757)
+- [QIN2DIM](https://github.com/QIN2DIM)
+
+## Acknowledgements
+
+We also want to thank all the users who have reported bugs, suggested features, or helped in any other way to make Crawl4AI better.
+
+---
+
+If you've contributed to Crawl4AI and your name isn't on this list, please [open a pull request](https://github.com/unclecode/crawl4ai/pulls) with your name, link, and contribution, and we'll review it promptly.
+
+Thank you all for your contributions!
--- a/67
+++ b/67
@@ -1,67 +0,0 @@
-# First stage: Build and install dependencies
-FROM python:3.10-slim-bookworm
-
-# Set the working directory in the container
-WORKDIR /usr/src/app
-
-# Install build dependencies
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-    wget \
-    git \
-    curl \
-    unzip \
-    gnupg \
-    xvfb \
-    ca-certificates \
-    apt-transport-https \
-    software-properties-common && \
-    rm -rf /var/lib/apt/lists/*    
-
-# Install Python dependencies
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt && \
-    pip install --no-cache-dir spacy torch onnxruntime uvicorn && \
-    python -m spacy download en_core_web_sm
-    # pip install --no-cache-dir spacy torch torchvision torchaudio onnxruntime uvicorn && \
-
-# Install Google Chrome and ChromeDriver
-RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
-    sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' && \
-    apt-get update && \
-    apt-get install -y google-chrome-stable && \
-    wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip && \
-    unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
-
-# Copy the rest of the application code
-COPY . .
-
-# Set environment to use Chrome and ChromeDriver properly
-ENV CHROME_BIN=/usr/bin/google-chrome \
-    CHROMEDRIVER=/usr/local/bin/chromedriver \
-    DISPLAY=:99 \
-    DBUS_SESSION_BUS_ADDRESS=/dev/null \
-    PYTHONUNBUFFERED=1
-
-#  pip install -e .[all]
-RUN pip install --no-cache-dir -e .[all]
-
-# Ensure the PATH environment variable includes the location of the installed packages
-ENV PATH /opt/conda/bin:$PATH   
-
-# Make port 80 available to the world outside this container
-EXPOSE 80
-
-# Download models call cli "crawl4ai-download-models"
-RUN crawl4ai-download-models
-
-# Instakk mkdocs
-RUN pip install mkdocs mkdocs-terminal
-
-# Call mkdocs to build the documentation
-RUN mkdocs build
-
-# Run uvicorn
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80", "--workers", "4"]
-
-
--- a/44
+++ b/44
@@ -1,44 +0,0 @@
-# Use an official Python runtime as a parent image
-FROM python:3.10-slim
-
-# Set the working directory in the container
-WORKDIR /usr/src/app
-
-# Copy the current directory contents into the container at /usr/src/app
-COPY . .
-
-# Install any needed packages specified in requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt
-
-# Install dependencies for Chrome and ChromeDriver
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    wget \
-    xvfb \
-    unzip \
-    curl \
-    gnupg2 \
-    ca-certificates \
-    apt-transport-https \
-    software-properties-common \
-    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
-    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \
-    && apt-get update \
-    && apt-get install -y google-chrome-stable \
-    && rm -rf /var/lib/apt/lists/* \
-    && apt install chromium-chromedriver -y
-
-# Install spacy library using pip
-RUN pip install spacy
-
-# Set display port and dbus env to avoid hanging
-ENV DISPLAY=:99
-ENV DBUS_SESSION_BUS_ADDRESS=/dev/null
-
-# Make port 80 available to the world outside this container
-EXPOSE 80
-
-# Define environment variable
-ENV PYTHONUNBUFFERED 1
-
-# Run uvicorn
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80", "--workers", "4"]
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -0,0 +1 @@
+include requirements.txt
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# Crawl4AI v0.2.7 🕷️🤖
+# Crawl4AI (Async Version) 🕷️🤖

 [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
 [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
@@ -6,18 +6,29 @@
 [![GitHub Pull Requests](https://img.shields.io/github/issues-pr/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/pulls)
 [![License](https://img.shields.io/github/license/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE)

-Crawl4AI simplifies web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
+Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
+
+> Looking for the synchronous version? Check out [README.sync.md](./README.sync.md). You can also access the previous version in the branch [V0.2.76](https://github.com/unclecode/crawl4ai/blob/v0.2.76).
+
+## New update 0.3.6
+- 🌐 Multi-browser support (Chromium, Firefox, WebKit)
+- 🖼️ Improved image processing with lazy-loading detection
+- 🔧 Custom page timeout parameter for better control over crawling behavior
+- 🕰️ Enhanced handling of delayed content loading
+- 🔑 Custom headers support for LLM interactions
+- 🖼️ iframe content extraction for comprehensive page analysis
+- ⏱️ Flexible timeout and delayed content retrieval options

 ## Try it Now!

- Use as REST API: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zODYjhemJ5bUmYceWpVoBMVpd0ofzNBZ?usp=sharing)
- Use as Python library: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
+✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1REChY6fXQf-EaVYLv0eHEWvzlYxGm0pd?usp=sharing)

-✨ visit our [Documentation Website](https://crawl4ai.com/mkdocs/)
+✨ Visit our [Documentation Website](https://crawl4ai.com/mkdocs/)

 ## Features ✨

 - 🆓 Completely free and open-source
+- 🚀 Blazing fast performance, outperforming many paid services
 - 🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
 - 🌍 Supports crawling multiple URLs simultaneously
 - 🎨 Extracts and returns all media tags (Images, Audio, and Video)
@@ -27,77 +38,334 @@ Crawl4AI simplifies web crawling and data extraction, making it accessible for l
 - 🕵️ User-agent customization
 - 🖼️ Takes screenshots of the page
 - 📜 Executes multiple custom JavaScripts before crawling
+- 📊 Generates structured output without LLM using JsonCssExtractionStrategy
 - 📚 Various chunking strategies: topic-based, regex, sentence, and more
 - 🧠 Advanced extraction strategies: cosine clustering, LLM, and more
- 🎯 CSS selector support
+- 🎯 CSS selector support for precise data extraction
 - 📝 Passes instructions/keywords to refine extraction
+- 🔒 Proxy support for enhanced privacy and access
+- 🔄 Session management for complex multi-page crawling scenarios
+- 🌐 Asynchronous architecture for improved performance and scalability

-## Cool Examples 🚀
+## Installation 🛠️

-### Quick Start
+Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

-```python
-from crawl4ai import WebCrawler
+### Using pip 🐍

-# Create an instance of WebCrawler
-crawler = WebCrawler()
+Choose the installation option that best fits your needs:

-# Warm up the crawler (load necessary models)
-crawler.warmup()
+#### Basic Installation

-# Run the crawler on a URL
-result = crawler.run(url="https://www.nbcnews.com/business")
+For basic web crawling and scraping tasks:

-# Print the extracted content
-print(result.markdown)
+```bash
+pip install crawl4ai
 ```

-### Extract Structured Data from Web Pages 📊
+By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

-Crawl all OpenAI models and their fees from the official page.
+👉 Note: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
+
+1. Through the command line:
+   ```bash
+   playwright install
+   ```
+
+2. If the above doesn't work, try this more specific command:
+   ```bash
+   python -m playwright install chromium
+   ```
+
+This second method has proven to be more reliable in some cases.
+
+#### Installation with Synchronous Version
+
+If you need the synchronous version using Selenium:
+
+```bash
+pip install crawl4ai[sync]
+```
+
+#### Development Installation
+
+For contributors who plan to modify the source code:
+
+```bash
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+pip install -e .
+```
+
+### Using Docker 🐳
+
+We're in the process of creating Docker images and pushing them to Docker Hub. This will provide an easy way to run Crawl4AI in a containerized environment. Stay tuned for updates!
+
+For more detailed installation instructions and options, please refer to our [Installation Guide](https://crawl4ai.com/mkdocs/installation).
+
+## Quick Start 🚀
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.nbcnews.com/business")
+        print(result.markdown)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## Advanced Usage 🔬
+
+### Executing JavaScript and Using CSS Selectors
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        js_code = ["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"]
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            css_selector=".wide-tease-item__description",
+            bypass_cache=True
+        )
+        print(result.extracted_content)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+### Using a Proxy
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    async with AsyncWebCrawler(verbose=True, proxy="http://127.0.0.1:7890") as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            bypass_cache=True
+        )
+        print(result.markdown)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+### Extracting Structured Data without LLM
+
+The `JsonCssExtractionStrategy` allows for precise extraction of structured data from web pages using CSS selectors.
+
+```python
+import asyncio
+import json
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+async def extract_news_teasers():
+    schema = {
+        "name": "News Teaser Extractor",
+        "baseSelector": ".wide-tease-item__wrapper",
+        "fields": [
+            {
+                "name": "category",
+                "selector": ".unibrow span[data-testid='unibrow-text']",
+                "type": "text",
+            },
+            {
+                "name": "headline",
+                "selector": ".wide-tease-item__headline",
+                "type": "text",
+            },
+            {
+                "name": "summary",
+                "selector": ".wide-tease-item__description",
+                "type": "text",
+            },
+            {
+                "name": "time",
+                "selector": "[data-testid='wide-tease-date']",
+                "type": "text",
+            },
+            {
+                "name": "image",
+                "type": "nested",
+                "selector": "picture.teasePicture img",
+                "fields": [
+                    {"name": "src", "type": "attribute", "attribute": "src"},
+                    {"name": "alt", "type": "attribute", "attribute": "alt"},
+                ],
+            },
+            {
+                "name": "link",
+                "selector": "a[href]",
+                "type": "attribute",
+                "attribute": "href",
+            },
+        ],
+    }
+
+    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            extraction_strategy=extraction_strategy,
+            bypass_cache=True,
+        )
+
+        assert result.success, "Failed to crawl the page"
+
+        news_teasers = json.loads(result.extracted_content)
+        print(f"Successfully extracted {len(news_teasers)} news teasers")
+        print(json.dumps(news_teasers[0], indent=2))
+
+if __name__ == "__main__":
+    asyncio.run(extract_news_teasers())
+```
+
+For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/advanced_jsoncss_extraction.md) section in the documentation.
+
+### Extracting Structured Data with OpenAI

 ```python
 import os
-from crawl4ai import WebCrawler
+import asyncio
+from crawl4ai import AsyncWebCrawler
 from crawl4ai.extraction_strategy import LLMExtractionStrategy
+from pydantic import BaseModel, Field

-url = 'https://openai.com/api/pricing/'
-crawler = WebCrawler()
-crawler.warmup()
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")

-result = crawler.run(
-    url=url,
-    extraction_strategy=LLMExtractionStrategy(
-        provider="openai/gpt-4",
-        api_token=os.getenv('OPENAI_API_KEY'),
-        instruction="Extract all model names and their fees for input and output tokens."
-    ),
-)
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url='https://openai.com/api/pricing/',
+            word_count_threshold=1,
+            extraction_strategy=LLMExtractionStrategy(
+                provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY'), 
+                schema=OpenAIModelFee.schema(),
+                extraction_type="schema",
+                instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
+                Do not miss any models in the entire content. One extracted model JSON format should look like this: 
+                {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
+            ),            
+            bypass_cache=True,
+        )
+        print(result.extracted_content)

-print(result.extracted_content)
+if __name__ == "__main__":
+    asyncio.run(main())
 ```

-### Execute JS, Filter Data with CSS Selector, and Clustering
+### Session Management and Dynamic Content Crawling
+
+Crawl4AI excels at handling complex scenarios, such as crawling multiple pages with dynamic content loaded via JavaScript. Here's an example of crawling GitHub commits across multiple pages:

 ```python
-from crawl4ai import WebCrawler
-from crawl4ai.chunking_strategy import CosineStrategy
+import asyncio
+import re
+from bs4 import BeautifulSoup
+from crawl4ai import AsyncWebCrawler

-js_code = ["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"]
+async def crawl_typescript_commits():
+    first_commit = ""
+    async def on_execution_started(page):
+        nonlocal first_commit 
+        try:
+            while True:
+                await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4')
+                commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4')
+                commit = await commit.evaluate('(element) => element.textContent')
+                commit = re.sub(r'\s+', '', commit)
+                if commit and commit != first_commit:
+                    first_commit = commit
+                    break
+                await asyncio.sleep(0.5)
+        except Exception as e:
+            print(f"Warning: New content didn't appear after JavaScript execution: {e}")

-crawler = WebCrawler()
-crawler.warmup()
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started)

-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    js=js_code,
-    css_selector="p",
-    extraction_strategy=CosineStrategy(semantic_filter="technology")
-)
+        url = "https://github.com/microsoft/TypeScript/commits/main"
+        session_id = "typescript_commits_session"
+        all_commits = []

-print(result.extracted_content)
+        js_next_page = """
+        const button = document.querySelector('a[data-testid="pagination-next-button"]');
+        if (button) button.click();
+        """
+
+        for page in range(3):  # Crawl 3 pages
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.Box-sc-g0xbh4-0",
+                js=js_next_page if page > 0 else None,
+                bypass_cache=True,
+                js_only=page > 0
+            )
+
+            assert result.success, f"Failed to crawl page {page + 1}"
+
+            soup = BeautifulSoup(result.cleaned_html, 'html.parser')
+            commits = soup.select("li")
+            all_commits.extend(commits)
+
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+if __name__ == "__main__":
+    asyncio.run(crawl_typescript_commits())
 ```

+This example demonstrates Crawl4AI's ability to handle complex scenarios where content is loaded asynchronously. It crawls multiple pages of GitHub commits, executing JavaScript to load new content and using custom hooks to ensure data is loaded before proceeding.
+
+For more advanced usage examples, check out our [Examples](https://crawl4ai.com/mkdocs/full_details/session_based_crawling.md) section in the documentation.
+
+
+## Speed Comparison 🚀
+
+Crawl4AI is designed with speed as a primary focus. Our goal is to provide the fastest possible response with high-quality data extraction, minimizing abstractions between the data and the user.
+
+We've conducted a speed comparison between Crawl4AI and Firecrawl, a paid service. The results demonstrate Crawl4AI's superior performance:
+
+```
+Firecrawl:
+Time taken: 7.02 seconds
+Content length: 42074 characters
+Images found: 49
+
+Crawl4AI (simple crawl):
+Time taken: 1.60 seconds
+Content length: 18238 characters
+Images found: 49
+
+Crawl4AI (with JavaScript execution):
+Time taken: 4.64 seconds
+Content length: 40869 characters
+Images found: 89
+```
+
+As you can see, Crawl4AI outperforms Firecrawl significantly:
+- Simple crawl: Crawl4AI is over 4 times faster than Firecrawl.
+- With JavaScript execution: Even when executing JavaScript to load more content (doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.
+
+You can find the full comparison code in our repository at `docs/examples/crawl4ai_vs_firecrawl.py`.
+
 ## Documentation 📚

 For detailed documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://crawl4ai.com/mkdocs/).
@@ -119,3 +387,7 @@ For questions, suggestions, or feedback, feel free to reach out:
 - Website: [crawl4ai.com](https://crawl4ai.com)

 Happy Crawling! 🕸️🚀
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)
--- a/README.sync.md
+++ b/README.sync.md
@@ -0,0 +1,244 @@
+# Crawl4AI v0.2.77 🕷️🤖
+
+[![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
+[![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
+[![GitHub Issues](https://img.shields.io/github/issues/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/issues)
+[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/pulls)
+[![License](https://img.shields.io/github/license/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE)
+
+Crawl4AI simplifies web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
+
+#### [v0.2.77] - 2024-08-02
+
+Major improvements in functionality, performance, and cross-platform compatibility! 🚀
+
+- 🐳 **Docker enhancements**:
+  - Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
+- 🌐 **Official Docker Hub image**:
+  - Launched our first official image on Docker Hub for streamlined deployment (unclecode/crawl4ai).
+- 🔧 **Selenium upgrade**:
+  - Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
+- 🖼️ **Image description**:
+  - Implemented ability to generate textual descriptions for extracted images from web pages.
+- ⚡ **Performance boost**:
+  - Various improvements to enhance overall speed and performance.
+  
+## Try it Now!
+
+✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sJPAmeLj5PMrg2VgOwMJ2ubGIcK0cJeX?usp=sharing)
+
+✨ visit our [Documentation Website](https://crawl4ai.com/mkdocs/)
+
+✨ Check [Demo](https://crawl4ai.com/mkdocs/demo)
+
+## Features ✨
+
+- 🆓 Completely free and open-source
+- 🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
+- 🌍 Supports crawling multiple URLs simultaneously
+- 🎨 Extracts and returns all media tags (Images, Audio, and Video)
+- 🔗 Extracts all external and internal links
+- 📚 Extracts metadata from the page
+- 🔄 Custom hooks for authentication, headers, and page modifications before crawling
+- 🕵️ User-agent customization
+- 🖼️ Takes screenshots of the page
+- 📜 Executes multiple custom JavaScripts before crawling
+- 📚 Various chunking strategies: topic-based, regex, sentence, and more
+- 🧠 Advanced extraction strategies: cosine clustering, LLM, and more
+- 🎯 CSS selector support
+- 📝 Passes instructions/keywords to refine extraction
+
+# Crawl4AI
+
+## 🌟 Shoutout to Contributors of v0.2.77!
+
+A big thank you to the amazing contributors who've made this release possible:
+
+- [@aravindkarnam](https://github.com/aravindkarnam) for the new image description feature
+- [@FractalMind](https://github.com/FractalMind) for our official Docker Hub image
+- [@ketonkss4](https://github.com/ketonkss4) for helping streamline our Selenium setup
+
+Your contributions are driving Crawl4AI forward! 🚀
+
+## Cool Examples 🚀
+
+### Quick Start
+
+```python
+from crawl4ai import WebCrawler
+
+# Create an instance of WebCrawler
+crawler = WebCrawler()
+
+# Warm up the crawler (load necessary models)
+crawler.warmup()
+
+# Run the crawler on a URL
+result = crawler.run(url="https://www.nbcnews.com/business")
+
+# Print the extracted content
+print(result.markdown)
+```
+
+## How to install 🛠 
+
+### Using pip 🐍
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai @ git+https://github.com/unclecode/crawl4ai.git"
+```
+
+### Using Docker 🐳
+
+```bash
+# For Mac users (M1/M2)
+# docker build --platform linux/amd64 -t crawl4ai .
+docker build -t crawl4ai .
+docker run -d -p 8000:80 crawl4ai
+```
+
+### Using Docker Hub 🐳
+
+```bash
+docker pull unclecode/crawl4ai:latest
+docker run -d -p 8000:80 unclecode/crawl4ai:latest
+```
+
+
+## Speed-First Design 🚀
+
+Perhaps the most important design principle for this library is speed. We need to ensure it can handle many links and resources in parallel as quickly as possible. By combining this speed with fast LLMs like Groq, the results will be truly amazing.
+
+```python
+import time
+from crawl4ai.web_crawler import WebCrawler
+crawler = WebCrawler()
+crawler.warmup()
+
+start = time.time()
+url = r"https://www.nbcnews.com/business"
+result = crawler.run( url, word_count_threshold=10, bypass_cache=True)
+end = time.time()
+print(f"Time taken: {end - start}")
+```
+
+Let's take a look the calculated time for the above code snippet:
+
+```bash
+[LOG] 🚀 Crawling done, success: True, time taken: 1.3623387813568115 seconds
+[LOG] 🚀 Content extracted, success: True, time taken: 0.05715131759643555 seconds
+[LOG] 🚀 Extraction, time taken: 0.05750393867492676 seconds.
+Time taken: 1.439958095550537
+```
+Fetching the content from the page took 1.3623 seconds, and extracting the content took 0.0575 seconds. 🚀
+
+### Extract Structured Data from Web Pages 📊
+
+Crawl all OpenAI models and their fees from the official page.
+
+```python
+import os
+from crawl4ai import WebCrawler
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+from pydantic import BaseModel, Field
+
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(..., description="Fee for output token ßfor the OpenAI model.")
+
+url = 'https://openai.com/api/pricing/'
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+        url=url,
+        word_count_threshold=1,
+        extraction_strategy= LLMExtractionStrategy(
+            provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
+            schema=OpenAIModelFee.schema(),
+            extraction_type="schema",
+            instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
+            Do not miss any models in the entire content. One extracted model JSON format should look like this: 
+            {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
+        ),            
+        bypass_cache=True,
+    )
+
+print(result.extracted_content)
+```
+
+### Execute JS, Filter Data with CSS Selector, and Clustering
+
+```python
+from crawl4ai import WebCrawler
+from crawl4ai.chunking_strategy import CosineStrategy
+
+js_code = ["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"]
+
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    js=js_code,
+    css_selector="p",
+    extraction_strategy=CosineStrategy(semantic_filter="technology")
+)
+
+print(result.extracted_content)
+```
+
+### Extract Structured Data from Web Pages With Proxy and BaseUrl
+
+```python
+from crawl4ai import WebCrawler
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+def create_crawler():
+    crawler = WebCrawler(verbose=True, proxy="http://127.0.0.1:7890")
+    crawler.warmup()
+    return crawler
+
+crawler = create_crawler()
+
+crawler.warmup()
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=LLMExtractionStrategy(
+        provider="openai/gpt-4o",
+        api_token="sk-",
+        base_url="https://api.openai.com/v1"
+    )
+)
+
+print(result.markdown)
+```
+
+## Documentation 📚
+
+For detailed documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://crawl4ai.com/mkdocs/).
+
+## Contributing 🤝
+
+We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md) for more information.
+
+## License 📄
+
+Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
+
+## Contact 📧
+
+For questions, suggestions, or feedback, feel free to reach out:
+
+- GitHub: [unclecode](https://github.com/unclecode)
+- Twitter: [@unclecode](https://twitter.com/unclecode)
+- Website: [crawl4ai.com](https://crawl4ai.com)
+
+Happy Crawling! 🕸️🚀
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)
--- a/crawl4ai/init.py
+++ b/crawl4ai/init.py
@@ -1 +1,30 @@
-from .web_crawler import WebCrawler
+# __init__.py
+
+from .async_webcrawler import AsyncWebCrawler
+from .models import CrawlResult
+
+__version__ = "0.3.6"
+
+__all__ = [
+    "AsyncWebCrawler",
+    "CrawlResult",
+]
+
+def is_sync_version_installed():
+    try:
+        import selenium
+        return True
+    except ImportError:
+        return False
+
+if is_sync_version_installed():
+    try:
+        from .web_crawler import WebCrawler
+        __all__.append("WebCrawler")
+    except ImportError:
+        import warnings
+        print("Warning: Failed to import WebCrawler even though selenium is installed. This might be due to other missing dependencies.")
+else:
+    WebCrawler = None
+    import warnings
+    print("Warning: Synchronous WebCrawler is not available. Install crawl4ai[sync] for synchronous support. However, please note that the synchronous version will be deprecated soon.")
--- a/crawl4ai/async_crawler_strategy.py
+++ b/crawl4ai/async_crawler_strategy.py
@@ -0,0 +1,552 @@
+import asyncio
+import base64, time
+from abc import ABC, abstractmethod
+from typing import Callable, Dict, Any, List, Optional, Awaitable
+import os
+from playwright.async_api import async_playwright, Page, Browser, Error
+from io import BytesIO
+from PIL import Image, ImageDraw, ImageFont
+from .utils import sanitize_input_encode, calculate_semaphore_count
+import json, uuid
+import hashlib
+from pathlib import Path
+from playwright.async_api import ProxySettings
+from pydantic import BaseModel
+
+class AsyncCrawlResponse(BaseModel):
+    html: str
+    response_headers: Dict[str, str]
+    status_code: int
+    screenshot: Optional[str] = None
+    get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None
+
+    class Config:
+        arbitrary_types_allowed = True
+
+class AsyncCrawlerStrategy(ABC):
+    @abstractmethod
+    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
+        pass
+    
+    @abstractmethod
+    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
+        pass
+    
+    @abstractmethod
+    async def take_screenshot(self, url: str) -> str:
+        pass
+    
+    @abstractmethod
+    def update_user_agent(self, user_agent: str):
+        pass
+    
+    @abstractmethod
+    def set_hook(self, hook_type: str, hook: Callable):
+        pass
+
+class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
+    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
+        self.use_cached_html = use_cached_html
+        self.user_agent = kwargs.get("user_agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
+        self.proxy = kwargs.get("proxy")
+        self.headless = kwargs.get("headless", True)
+        self.browser_type = kwargs.get("browser_type", "chromium")  # New parameter
+        self.headers = kwargs.get("headers", {})
+        self.sessions = {}
+        self.session_ttl = 1800 
+        self.js_code = js_code
+        self.verbose = kwargs.get("verbose", False)
+        self.playwright = None
+        self.browser = None
+        self.hooks = {
+            'on_browser_created': None,
+            'on_user_agent_updated': None,
+            'on_execution_started': None,
+            'before_goto': None,
+            'after_goto': None,
+            'before_return_html': None,
+            'before_retrieve_html': None
+        }
+
+    async def __aenter__(self):
+        await self.start()
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        await self.close()
+
+    async def start(self):
+        if self.playwright is None:
+            self.playwright = await async_playwright().start()
+        if self.browser is None:
+            browser_args = {
+                "headless": self.headless,
+                "args": [
+                    "--disable-gpu",
+                    "--disable-dev-shm-usage",
+                    "--disable-setuid-sandbox",
+                    "--no-sandbox",
+                ]
+            }
+            
+            # Add proxy settings if a proxy is specified
+            if self.proxy:
+                proxy_settings = ProxySettings(server=self.proxy)
+                browser_args["proxy"] = proxy_settings
+                
+                
+            # Select the appropriate browser based on the browser_type
+            if self.browser_type == "firefox":
+                self.browser = await self.playwright.firefox.launch(**browser_args)
+            elif self.browser_type == "webkit":
+                self.browser = await self.playwright.webkit.launch(**browser_args)
+            else:
+                self.browser = await self.playwright.chromium.launch(**browser_args)
+
+            await self.execute_hook('on_browser_created', self.browser)
+
+    async def close(self):
+        if self.browser:
+            await self.browser.close()
+            self.browser = None
+        if self.playwright:
+            await self.playwright.stop()
+            self.playwright = None
+
+    def __del__(self):
+        if self.browser or self.playwright:
+            asyncio.get_event_loop().run_until_complete(self.close())
+
+    def set_hook(self, hook_type: str, hook: Callable):
+        if hook_type in self.hooks:
+            self.hooks[hook_type] = hook
+        else:
+            raise ValueError(f"Invalid hook type: {hook_type}")
+
+    async def execute_hook(self, hook_type: str, *args):
+        hook = self.hooks.get(hook_type)
+        if hook:
+            if asyncio.iscoroutinefunction(hook):
+                return await hook(*args)
+            else:
+                return hook(*args)
+        return args[0] if args else None
+
+    def update_user_agent(self, user_agent: str):
+        self.user_agent = user_agent
+
+    def set_custom_headers(self, headers: Dict[str, str]):
+        self.headers = headers
+
+    async def kill_session(self, session_id: str):
+        if session_id in self.sessions:
+            context, page, _ = self.sessions[session_id]
+            await page.close()
+            await context.close()
+            del self.sessions[session_id]
+
+    def _cleanup_expired_sessions(self):
+        current_time = time.time()
+        expired_sessions = [sid for sid, (_, _, last_used) in self.sessions.items() 
+                            if current_time - last_used > self.session_ttl]
+        for sid in expired_sessions:
+            asyncio.create_task(self.kill_session(sid))
+            
+    async def smart_wait(self, page: Page, wait_for: str, timeout: float = 30000):
+        wait_for = wait_for.strip()
+        
+        if wait_for.startswith('js:'):
+            # Explicitly specified JavaScript
+            js_code = wait_for[3:].strip()
+            return await self.csp_compliant_wait(page, js_code, timeout)
+        elif wait_for.startswith('css:'):
+            # Explicitly specified CSS selector
+            css_selector = wait_for[4:].strip()
+            try:
+                await page.wait_for_selector(css_selector, timeout=timeout)
+            except Error as e:
+                if 'Timeout' in str(e):
+                    raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{css_selector}'")
+                else:
+                    raise ValueError(f"Invalid CSS selector: '{css_selector}'")
+        else:
+            # Auto-detect based on content
+            if wait_for.startswith('()') or wait_for.startswith('function'):
+                # It's likely a JavaScript function
+                return await self.csp_compliant_wait(page, wait_for, timeout)
+            else:
+                # Assume it's a CSS selector first
+                try:
+                    await page.wait_for_selector(wait_for, timeout=timeout)
+                except Error as e:
+                    if 'Timeout' in str(e):
+                        raise TimeoutError(f"Timeout after {timeout}ms waiting for selector '{wait_for}'")
+                    else:
+                        # If it's not a timeout error, it might be an invalid selector
+                        # Let's try to evaluate it as a JavaScript function as a fallback
+                        try:
+                            return await self.csp_compliant_wait(page, f"() => {{{wait_for}}}", timeout)
+                        except Error:
+                            raise ValueError(f"Invalid wait_for parameter: '{wait_for}'. "
+                                            "It should be either a valid CSS selector, a JavaScript function, "
+                                            "or explicitly prefixed with 'js:' or 'css:'.")
+    
+    async def csp_compliant_wait(self, page: Page, user_wait_function: str, timeout: float = 30000):
+        wrapper_js = f"""
+        async () => {{
+            const userFunction = {user_wait_function};
+            const startTime = Date.now();
+            while (true) {{
+                if (await userFunction()) {{
+                    return true;
+                }}
+                if (Date.now() - startTime > {timeout}) {{
+                    throw new Error('Timeout waiting for condition');
+                }}
+                await new Promise(resolve => setTimeout(resolve, 100));
+            }}
+        }}
+        """
+        
+        try:
+            await page.evaluate(wrapper_js)
+        except TimeoutError:
+            raise TimeoutError(f"Timeout after {timeout}ms waiting for condition")
+        except Exception as e:
+            raise RuntimeError(f"Error in wait condition: {str(e)}")
+
+    async def process_iframes(self, page):
+        # Find all iframes
+        iframes = await page.query_selector_all('iframe')
+        
+        for i, iframe in enumerate(iframes):
+            try:
+                # Add a unique identifier to the iframe
+                await iframe.evaluate(f'(element) => element.id = "iframe-{i}"')
+                
+                # Get the frame associated with this iframe
+                frame = await iframe.content_frame()
+                
+                if frame:
+                    # Wait for the frame to load
+                    await frame.wait_for_load_state('load', timeout=30000)  # 30 seconds timeout
+                    
+                    # Extract the content of the iframe's body
+                    iframe_content = await frame.evaluate('() => document.body.innerHTML')
+                    
+                    # Generate a unique class name for this iframe
+                    class_name = f'extracted-iframe-content-{i}'
+                    
+                    # Replace the iframe with a div containing the extracted content
+                    _iframe = iframe_content.replace('`', '\\`')
+                    await page.evaluate(f"""
+                        () => {{
+                            const iframe = document.getElementById('iframe-{i}');
+                            const div = document.createElement('div');
+                            div.innerHTML = `{_iframe}`;
+                            div.className = '{class_name}';
+                            iframe.replaceWith(div);
+                        }}
+                    """)
+                else:
+                    print(f"Warning: Could not access content frame for iframe {i}")
+            except Exception as e:
+                print(f"Error processing iframe {i}: {str(e)}")
+
+        # Return the page object
+        return page
+    
+    
+    async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse:
+        response_headers = {}
+        status_code = None
+        
+        self._cleanup_expired_sessions()
+        session_id = kwargs.get("session_id")
+        if session_id:
+            context, page, _ = self.sessions.get(session_id, (None, None, None))
+            if not context:
+                context = await self.browser.new_context(
+                    user_agent=self.user_agent,
+                    proxy={"server": self.proxy} if self.proxy else None
+                )
+                await context.set_extra_http_headers(self.headers)
+                page = await context.new_page()
+                self.sessions[session_id] = (context, page, time.time())
+        else:
+            context = await self.browser.new_context(
+                    user_agent=self.user_agent,
+                    proxy={"server": self.proxy} if self.proxy else None
+            )
+            await context.set_extra_http_headers(self.headers)
+            page = await context.new_page()
+
+        try:
+            if self.verbose:
+                print(f"[LOG] 🕸️ Crawling {url} using AsyncPlaywrightCrawlerStrategy...")
+
+            if self.use_cached_html:
+                cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest())
+                if os.path.exists(cache_file_path):
+                    html = ""
+                    with open(cache_file_path, "r") as f:
+                        html = f.read()
+                    # retrieve response headers and status code from cache
+                    with open(cache_file_path + ".meta", "r") as f:
+                        meta = json.load(f)
+                        response_headers = meta.get("response_headers", {})
+                        status_code = meta.get("status_code")
+                    response = AsyncCrawlResponse(html=html, response_headers=response_headers, status_code=status_code)
+                    return response
+
+            if not kwargs.get("js_only", False):
+                await self.execute_hook('before_goto', page)
+                response = await page.goto(url, wait_until="domcontentloaded", timeout=kwargs.get("page_timeout", 60000))
+                await self.execute_hook('after_goto', page)
+                
+                # Get status code and headers
+                status_code = response.status
+                response_headers = response.headers
+            else:
+                status_code = 200
+                response_headers = {}
+
+
+            await page.wait_for_selector('body')
+            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
+
+            js_code = kwargs.get("js_code", kwargs.get("js", self.js_code))
+            if js_code:
+                if isinstance(js_code, str):
+                    r = await page.evaluate(js_code)
+                elif isinstance(js_code, list):
+                    for js in js_code:
+                        await page.evaluate(js)
+                
+                # await page.wait_for_timeout(100)
+                await page.wait_for_load_state('networkidle')
+                # Check for on execution even
+                await self.execute_hook('on_execution_started', page)
+                
+            # New code to handle the wait_for parameter
+            # Example usage:
+            # await crawler.crawl(
+            #     url,
+            #     js_code="// some JavaScript code",
+            #     wait_for="""() => {
+            #         return document.querySelector('#my-element') !== null;
+            #     }"""
+            # )
+            # Example of using a CSS selector:
+            # await crawler.crawl(
+            #     url,
+            #     wait_for="#my-element"
+            # )
+            wait_for = kwargs.get("wait_for")
+            if wait_for:
+                try:
+                    await self.smart_wait(page, wait_for, timeout=kwargs.get("page_timeout", 60000))
+                except Exception as e:
+                    raise RuntimeError(f"Wait condition failed: {str(e)}")
+
+            # Check if kwargs has screenshot=True then take screenshot
+            screenshot_data = None
+            if kwargs.get("screenshot"):
+                screenshot_data = await self.take_screenshot(url)
+            
+            
+            # New code to update image dimensions
+            update_image_dimensions_js = """
+            () => {
+                return new Promise((resolve) => {
+                    const filterImage = (img) => {
+                        // Filter out images that are too small
+                        if (img.width < 100 && img.height < 100) return false;
+                        
+                        // Filter out images that are not visible
+                        const rect = img.getBoundingClientRect();
+                        if (rect.width === 0 || rect.height === 0) return false;
+                        
+                        // Filter out images with certain class names (e.g., icons, thumbnails)
+                        if (img.classList.contains('icon') || img.classList.contains('thumbnail')) return false;
+                        
+                        // Filter out images with certain patterns in their src (e.g., placeholder images)
+                        if (img.src.includes('placeholder') || img.src.includes('icon')) return false;
+                        
+                        return true;
+                    };
+
+                    const images = Array.from(document.querySelectorAll('img')).filter(filterImage);
+                    let imagesLeft = images.length;
+                    
+                    if (imagesLeft === 0) {
+                        resolve();
+                        return;
+                    }
+
+                    const checkImage = (img) => {
+                        if (img.complete && img.naturalWidth !== 0) {
+                            img.setAttribute('width', img.naturalWidth);
+                            img.setAttribute('height', img.naturalHeight);
+                            imagesLeft--;
+                            if (imagesLeft === 0) resolve();
+                        }
+                    };
+
+                    images.forEach(img => {
+                        checkImage(img);
+                        if (!img.complete) {
+                            img.onload = () => {
+                                checkImage(img);
+                            };
+                            img.onerror = () => {
+                                imagesLeft--;
+                                if (imagesLeft === 0) resolve();
+                            };
+                        }
+                    });
+
+                    // Fallback timeout of 5 seconds
+                    setTimeout(() => resolve(), 5000);
+                });
+            }
+            """
+            await page.evaluate(update_image_dimensions_js)
+
+            # Wait a bit for any onload events to complete
+            await page.wait_for_timeout(100)
+
+            # Process iframes
+            if kwargs.get("process_iframes", False):
+                page = await self.process_iframes(page)
+            
+            await self.execute_hook('before_retrieve_html', page)
+            # Check if delay_before_return_html is set then wait for that time
+            delay_before_return_html = kwargs.get("delay_before_return_html")
+            if delay_before_return_html:
+                await asyncio.sleep(delay_before_return_html)
+                
+            html = await page.content()
+            await self.execute_hook('before_return_html', page, html)
+
+            if self.verbose:
+                print(f"[LOG] ✅ Crawled {url} successfully!")
+
+            if self.use_cached_html:
+                cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest())
+                with open(cache_file_path, "w", encoding="utf-8") as f:
+                    f.write(html)
+                # store response headers and status code in cache
+                with open(cache_file_path + ".meta", "w", encoding="utf-8") as f:
+                    json.dump({
+                        "response_headers": response_headers,
+                        "status_code": status_code
+                    }, f)
+
+            
+            async def get_delayed_content(delay: float = 5.0) -> str:
+                if self.verbose:
+                    print(f"[LOG] Waiting for {delay} seconds before retrieving content for {url}")
+                await asyncio.sleep(delay)
+                return await page.content()
+                
+            response = AsyncCrawlResponse(
+                html=html, 
+                response_headers=response_headers, 
+                status_code=status_code,
+                screenshot=screenshot_data,
+                get_delayed_content=get_delayed_content
+            )
+            return response
+        except Error as e:
+            raise Error(f"Failed to crawl {url}: {str(e)}")
+        finally:
+            if not session_id:
+                await page.close()
+
+        # try:
+        #     html = await _crawl()
+        #     return sanitize_input_encode(html)
+        # except Error as e:
+        #     raise Error(f"Failed to crawl {url}: {str(e)}")
+        # except Exception as e:
+        #     raise Exception(f"Failed to crawl {url}: {str(e)}")
+
+    async def execute_js(self, session_id: str, js_code: str, wait_for_js: str = None, wait_for_css: str = None) -> AsyncCrawlResponse:
+        """
+        Execute JavaScript code in a specific session and optionally wait for a condition.
+        
+        :param session_id: The ID of the session to execute the JS code in.
+        :param js_code: The JavaScript code to execute.
+        :param wait_for_js: JavaScript condition to wait for after execution.
+        :param wait_for_css: CSS selector to wait for after execution.
+        :return: AsyncCrawlResponse containing the page's HTML and other information.
+        :raises ValueError: If the session does not exist.
+        """
+        if not session_id:
+            raise ValueError("Session ID must be provided")
+        
+        if session_id not in self.sessions:
+            raise ValueError(f"No active session found for session ID: {session_id}")
+        
+        context, page, last_used = self.sessions[session_id]
+        
+        try:
+            await page.evaluate(js_code)
+            
+            if wait_for_js:
+                await page.wait_for_function(wait_for_js)
+            
+            if wait_for_css:
+                await page.wait_for_selector(wait_for_css)
+            
+            # Get the updated HTML content
+            html = await page.content()
+            
+            # Get response headers and status code (assuming these are available)
+            response_headers = await page.evaluate("() => JSON.stringify(performance.getEntriesByType('resource')[0].responseHeaders)")
+            status_code = await page.evaluate("() => performance.getEntriesByType('resource')[0].responseStatus")
+            
+            # Update the last used time for this session
+            self.sessions[session_id] = (context, page, time.time())
+            
+            return AsyncCrawlResponse(html=html, response_headers=response_headers, status_code=status_code)
+        except Error as e:
+            raise Error(f"Failed to execute JavaScript or wait for condition in session {session_id}: {str(e)}")
+    
+    async def crawl_many(self, urls: List[str], **kwargs) -> List[AsyncCrawlResponse]:
+        semaphore_count = kwargs.get('semaphore_count', calculate_semaphore_count())
+        semaphore = asyncio.Semaphore(semaphore_count)
+
+        async def crawl_with_semaphore(url):
+            async with semaphore:
+                return await self.crawl(url, **kwargs)
+
+        tasks = [crawl_with_semaphore(url) for url in urls]
+        results = await asyncio.gather(*tasks, return_exceptions=True)
+        return [result if not isinstance(result, Exception) else str(result) for result in results]
+
+    async def take_screenshot(self, url: str, wait_time = 1000) -> str:
+        async with await self.browser.new_context(user_agent=self.user_agent) as context:
+            page = await context.new_page()
+            try:
+                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
+                # Wait for a specified time (default is 1 second)
+                await page.wait_for_timeout(wait_time)
+                screenshot = await page.screenshot(full_page=True)
+                return base64.b64encode(screenshot).decode('utf-8')
+            except Exception as e:
+                error_message = f"Failed to take screenshot: {str(e)}"
+                print(error_message)
+
+                # Generate an error image
+                img = Image.new('RGB', (800, 600), color='black')
+                draw = ImageDraw.Draw(img)
+                font = ImageFont.load_default()
+                draw.text((10, 10), error_message, fill=(255, 255, 255), font=font)
+                
+                buffered = BytesIO()
+                img.save(buffered, format="JPEG")
+                return base64.b64encode(buffered.getvalue()).decode('utf-8')
+            finally:
+                await page.close()
--- a/crawl4ai/async_database.py
+++ b/crawl4ai/async_database.py
@@ -0,0 +1,114 @@
+import os
+from pathlib import Path
+import aiosqlite
+import asyncio
+from typing import Optional, Tuple
+
+DB_PATH = os.path.join(Path.home(), ".crawl4ai")
+os.makedirs(DB_PATH, exist_ok=True)
+DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")
+
+class AsyncDatabaseManager:
+    def __init__(self):
+        self.db_path = DB_PATH
+
+    async def ainit_db(self):
+        async with aiosqlite.connect(self.db_path) as db:
+            await db.execute('''
+                CREATE TABLE IF NOT EXISTS crawled_data (
+                    url TEXT PRIMARY KEY,
+                    html TEXT,
+                    cleaned_html TEXT,
+                    markdown TEXT,
+                    extracted_content TEXT,
+                    success BOOLEAN,
+                    media TEXT DEFAULT "{}",
+                    links TEXT DEFAULT "{}",
+                    metadata TEXT DEFAULT "{}",
+                    screenshot TEXT DEFAULT ""
+                )
+            ''')
+            await db.commit()
+        await self.update_db_schema()
+
+    async def update_db_schema(self):
+        async with aiosqlite.connect(self.db_path) as db:
+            # Check if the 'media' column exists
+            cursor = await db.execute("PRAGMA table_info(crawled_data)")
+            columns = await cursor.fetchall()
+            column_names = [column[1] for column in columns]
+            
+            if 'media' not in column_names:
+                await self.aalter_db_add_column('media')
+            
+            # Check for other missing columns and add them if necessary
+            for column in ['links', 'metadata', 'screenshot']:
+                if column not in column_names:
+                    await self.aalter_db_add_column(column)
+
+    async def aalter_db_add_column(self, new_column: str):
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                await db.execute(f'ALTER TABLE crawled_data ADD COLUMN {new_column} TEXT DEFAULT ""')
+                await db.commit()
+            print(f"Added column '{new_column}' to the database.")
+        except Exception as e:
+            print(f"Error altering database to add {new_column} column: {e}")
+
+    async def aget_cached_url(self, url: str) -> Optional[Tuple[str, str, str, str, str, str, str, bool, str]]:
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                async with db.execute('SELECT url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot FROM crawled_data WHERE url = ?', (url,)) as cursor:
+                    return await cursor.fetchone()
+        except Exception as e:
+            print(f"Error retrieving cached URL: {e}")
+            return None
+
+    async def acache_url(self, url: str, html: str, cleaned_html: str, markdown: str, extracted_content: str, success: bool, media: str = "{}", links: str = "{}", metadata: str = "{}", screenshot: str = ""):
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                await db.execute('''
+                    INSERT INTO crawled_data (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot)
+                    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                    ON CONFLICT(url) DO UPDATE SET
+                        html = excluded.html,
+                        cleaned_html = excluded.cleaned_html,
+                        markdown = excluded.markdown,
+                        extracted_content = excluded.extracted_content,
+                        success = excluded.success,
+                        media = excluded.media,      
+                        links = excluded.links,    
+                        metadata = excluded.metadata,      
+                        screenshot = excluded.screenshot
+                ''', (url, html, cleaned_html, markdown, extracted_content, success, media, links, metadata, screenshot))
+                await db.commit()
+        except Exception as e:
+            print(f"Error caching URL: {e}")
+
+    async def aget_total_count(self) -> int:
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                async with db.execute('SELECT COUNT(*) FROM crawled_data') as cursor:
+                    result = await cursor.fetchone()
+                    return result[0] if result else 0
+        except Exception as e:
+            print(f"Error getting total count: {e}")
+            return 0
+
+    async def aclear_db(self):
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                await db.execute('DELETE FROM crawled_data')
+                await db.commit()
+        except Exception as e:
+            print(f"Error clearing database: {e}")
+
+    async def aflush_db(self):
+        try:
+            async with aiosqlite.connect(self.db_path) as db:
+                await db.execute('DROP TABLE IF EXISTS crawled_data')
+                await db.commit()
+        except Exception as e:
+            print(f"Error flushing database: {e}")
+
+async_db_manager = AsyncDatabaseManager()
--- a/crawl4ai/async_webcrawler.py
+++ b/crawl4ai/async_webcrawler.py
@@ -0,0 +1,276 @@
+import os
+import time
+from pathlib import Path
+from typing import Optional
+import json
+import asyncio
+from .models import CrawlResult
+from .async_database import async_db_manager
+from .chunking_strategy import *
+from .extraction_strategy import *
+from .async_crawler_strategy import AsyncCrawlerStrategy, AsyncPlaywrightCrawlerStrategy, AsyncCrawlResponse
+from .content_scrapping_strategy import WebScrappingStrategy
+from .config import MIN_WORD_THRESHOLD, IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD
+from .utils import (
+    sanitize_input_encode,
+    InvalidCSSSelectorError,
+    format_html
+)
+
+
+class AsyncWebCrawler:
+    def __init__(
+        self,
+        crawler_strategy: Optional[AsyncCrawlerStrategy] = None,
+        always_by_pass_cache: bool = False,
+        base_directory: str = str(Path.home()),
+        **kwargs,
+    ):
+        self.crawler_strategy = crawler_strategy or AsyncPlaywrightCrawlerStrategy(
+            **kwargs
+        )
+        self.always_by_pass_cache = always_by_pass_cache
+        self.crawl4ai_folder = os.path.join(base_directory, ".crawl4ai")
+        os.makedirs(self.crawl4ai_folder, exist_ok=True)
+        os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
+        self.ready = False
+        self.verbose = kwargs.get("verbose", False)
+
+    async def __aenter__(self):
+        await self.crawler_strategy.__aenter__()
+        await self.awarmup()
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        await self.crawler_strategy.__aexit__(exc_type, exc_val, exc_tb)
+
+    async def awarmup(self):
+        if self.verbose:
+            print("[LOG] 🌤️  Warming up the AsyncWebCrawler")
+        await async_db_manager.ainit_db()
+        await self.arun(
+            url="https://google.com/",
+            word_count_threshold=5,
+            bypass_cache=False,
+            verbose=False,
+        )
+        self.ready = True
+        if self.verbose:
+            print("[LOG] 🌞 AsyncWebCrawler is ready to crawl")
+
+    async def arun(
+        self,
+        url: str,
+        word_count_threshold=MIN_WORD_THRESHOLD,
+        extraction_strategy: ExtractionStrategy = None,
+        chunking_strategy: ChunkingStrategy = RegexChunking(),
+        bypass_cache: bool = False,
+        css_selector: str = None,
+        screenshot: bool = False,
+        user_agent: str = None,
+        verbose=True,
+        **kwargs,
+    ) -> CrawlResult:
+        try:
+            extraction_strategy = extraction_strategy or NoExtractionStrategy()
+            extraction_strategy.verbose = verbose
+            if not isinstance(extraction_strategy, ExtractionStrategy):
+                raise ValueError("Unsupported extraction strategy")
+            if not isinstance(chunking_strategy, ChunkingStrategy):
+                raise ValueError("Unsupported chunking strategy")
+            
+            word_count_threshold = max(word_count_threshold, MIN_WORD_THRESHOLD)
+
+            async_response: AsyncCrawlResponse = None
+            cached = None
+            screenshot_data = None
+            extracted_content = None
+            if not bypass_cache and not self.always_by_pass_cache:
+                cached = await async_db_manager.aget_cached_url(url)
+
+            if kwargs.get("warmup", True) and not self.ready:
+                return None
+
+            if cached:
+                html = sanitize_input_encode(cached[1])
+                extracted_content = sanitize_input_encode(cached[4])
+                if screenshot:
+                    screenshot_data = cached[9]
+                    if not screenshot_data:
+                        cached = None
+
+            if not cached or not html:
+                t1 = time.time()
+                if user_agent:
+                    self.crawler_strategy.update_user_agent(user_agent)
+                async_response: AsyncCrawlResponse = await self.crawler_strategy.crawl(url, screenshot=screenshot, **kwargs)
+                html = sanitize_input_encode(async_response.html)
+                screenshot_data = async_response.screenshot
+                t2 = time.time()
+                if verbose:
+                    print(
+                        f"[LOG] 🚀 Crawling done for {url}, success: {bool(html)}, time taken: {t2 - t1:.2f} seconds"
+                    )
+
+            crawl_result = await self.aprocess_html(
+                url,
+                html,
+                extracted_content,
+                word_count_threshold,
+                extraction_strategy,
+                chunking_strategy,
+                css_selector,
+                screenshot_data,
+                verbose,
+                bool(cached),
+                async_response=async_response,
+                **kwargs,
+            )
+            crawl_result.status_code = async_response.status_code if async_response else 200
+            crawl_result.response_headers = async_response.response_headers if async_response else {}
+            crawl_result.success = bool(html)
+            crawl_result.session_id = kwargs.get("session_id", None)
+            return crawl_result
+        except Exception as e:
+            if not hasattr(e, "msg"):
+                e.msg = str(e)
+            print(f"[ERROR] 🚫 Failed to crawl {url}, error: {e.msg}")
+            return CrawlResult(url=url, html="", success=False, error_message=e.msg)
+
+    async def arun_many(
+        self,
+        urls: List[str],
+        word_count_threshold=MIN_WORD_THRESHOLD,
+        extraction_strategy: ExtractionStrategy = None,
+        chunking_strategy: ChunkingStrategy = RegexChunking(),
+        bypass_cache: bool = False,
+        css_selector: str = None,
+        screenshot: bool = False,
+        user_agent: str = None,
+        verbose=True,
+        **kwargs,
+    ) -> List[CrawlResult]:
+        tasks = [
+            self.arun(
+                url,
+                word_count_threshold,
+                extraction_strategy,
+                chunking_strategy,
+                bypass_cache,
+                css_selector,
+                screenshot,
+                user_agent,
+                verbose,
+                **kwargs
+            )
+            for url in urls
+        ]
+        return await asyncio.gather(*tasks)
+
+
+    async def aprocess_html(
+        self,
+        url: str,
+        html: str,
+        extracted_content: str,
+        word_count_threshold: int,
+        extraction_strategy: ExtractionStrategy,
+        chunking_strategy: ChunkingStrategy,
+        css_selector: str,
+        screenshot: str,
+        verbose: bool,
+        is_cached: bool,
+        **kwargs,
+    ) -> CrawlResult:
+        t = time.time()
+        # Extract content from HTML
+        try:
+            t1 = time.time()
+            scrapping_strategy = WebScrappingStrategy()
+            result = await scrapping_strategy.ascrap(
+                url,
+                html,
+                word_count_threshold=word_count_threshold,
+                css_selector=css_selector,
+                only_text=kwargs.get("only_text", False),
+                image_description_min_word_threshold=kwargs.get(
+                    "image_description_min_word_threshold", IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD
+                ),
+            )
+            if verbose:
+                print(
+                    f"[LOG] 🚀 Content extracted for {url}, success: True, time taken: {time.time() - t1:.2f} seconds"
+                )
+
+            if result is None:
+                raise ValueError(f"Process HTML, Failed to extract content from the website: {url}")
+        except InvalidCSSSelectorError as e:
+            raise ValueError(str(e))
+        except Exception as e:
+            raise ValueError(f"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}")
+
+        cleaned_html = sanitize_input_encode(result.get("cleaned_html", ""))
+        markdown = sanitize_input_encode(result.get("markdown", ""))
+        media = result.get("media", [])
+        links = result.get("links", [])
+        metadata = result.get("metadata", {})
+
+        if extracted_content is None and extraction_strategy and chunking_strategy:
+            if verbose:
+                print(
+                    f"[LOG] 🔥 Extracting semantic blocks for {url}, Strategy: {self.__class__.__name__}"
+                )
+
+            # Check if extraction strategy is type of JsonCssExtractionStrategy
+            if isinstance(extraction_strategy, JsonCssExtractionStrategy) or isinstance(extraction_strategy, JsonCssExtractionStrategy):
+                extraction_strategy.verbose = verbose
+                extracted_content = extraction_strategy.run(url, [html])
+                extracted_content = json.dumps(extracted_content, indent=4, default=str, ensure_ascii=False)
+            else:
+                sections = chunking_strategy.chunk(markdown)
+                extracted_content = extraction_strategy.run(url, sections)
+                extracted_content = json.dumps(extracted_content, indent=4, default=str, ensure_ascii=False)
+
+        if verbose:
+            print(
+                f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t:.2f} seconds."
+            )
+
+        screenshot = None if not screenshot else screenshot
+
+        if not is_cached:
+            await async_db_manager.acache_url(
+                url,
+                html,
+                cleaned_html,
+                markdown,
+                extracted_content,
+                True,
+                json.dumps(media),
+                json.dumps(links),
+                json.dumps(metadata),
+                screenshot=screenshot,
+            )
+
+        return CrawlResult(
+            url=url,
+            html=html,
+            cleaned_html=format_html(cleaned_html),
+            markdown=markdown,
+            media=media,
+            links=links,
+            metadata=metadata,
+            screenshot=screenshot,
+            extracted_content=extracted_content,
+            success=True,
+            error_message="",
+        )
+
+    async def aclear_cache(self):
+        await async_db_manager.aclear_db()
+
+    async def aflush_cache(self):
+        await async_db_manager.aflush_db()
+
+    async def aget_cache_size(self):
+        return await async_db_manager.aget_total_count()
--- a/crawl4ai/chunking_strategy.py
+++ b/crawl4ai/chunking_strategy.py
@@ -3,6 +3,7 @@ import re
 from collections import Counter
 import string
 from .model_loader import load_nltk_punkt
+from .utils import *

 # Define the abstract base class for chunking strategies
 class ChunkingStrategy(ABC):
@@ -54,7 +55,7 @@ class TopicSegmentationChunking(ChunkingStrategy):
    
    def __init__(self, num_keywords=3, **kwargs):
        import nltk as nl
-        self.tokenizer = nl.toknize.TextTilingTokenizer()
+        self.tokenizer = nl.tokenize.TextTilingTokenizer()
        self.num_keywords = num_keywords

    def chunk(self, text: str) -> list:
--- a/crawl4ai/config.py
+++ b/crawl4ai/config.py
@@ -27,3 +27,14 @@ WORD_TOKEN_RATE = 1.3

 # Threshold for the minimum number of word in a HTML tag to be considered 
 MIN_WORD_THRESHOLD = 1
+IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD = 1
+
+# Threshold for the Image extraction - Range is 1 to 6
+# Images are scored based on point based system, to filter based on usefulness. Points are assigned
+# to each image based on the following aspects.
+# If either height or width exceeds 150px
+# If image size is greater than 10Kb
+# If alt property is set
+# If image format is in jpg, png or webp
+# If image is in the first half of the total images extracted from the page
+IMAGE_SCORE_THRESHOLD = 2
--- a/crawl4ai/content_scrapping_strategy.py
+++ b/crawl4ai/content_scrapping_strategy.py
@@ -0,0 +1,301 @@
+from abc import ABC, abstractmethod
+from typing import Dict, Any
+from bs4 import BeautifulSoup
+from concurrent.futures import ThreadPoolExecutor
+import asyncio, requests, re, os
+from .config import *
+from bs4 import element, NavigableString, Comment
+from urllib.parse import urljoin
+from requests.exceptions import InvalidSchema
+
+from .utils import (
+    sanitize_input_encode,
+    sanitize_html,
+    extract_metadata,
+    InvalidCSSSelectorError,
+    CustomHTML2Text
+)
+
+class ContentScrappingStrategy(ABC):
+    @abstractmethod
+    def scrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
+        pass
+
+    @abstractmethod
+    async def ascrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
+        pass
+
+class WebScrappingStrategy(ContentScrappingStrategy):
+    def scrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
+        return self._get_content_of_website_optimized(url, html, is_async=False, **kwargs)
+
+    async def ascrap(self, url: str, html: str, **kwargs) -> Dict[str, Any]:
+        return await asyncio.to_thread(self._get_content_of_website_optimized, url, html, **kwargs)
+
+    def _get_content_of_website_optimized(self, url: str, html: str, word_count_threshold: int = MIN_WORD_THRESHOLD, css_selector: str = None, **kwargs) -> Dict[str, Any]:
+        if not html:
+            return None
+
+        soup = BeautifulSoup(html, 'html.parser')
+        body = soup.body
+        
+        image_description_min_word_threshold = kwargs.get('image_description_min_word_threshold', IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD)
+
+        for tag in kwargs.get('excluded_tags', []) or []:
+            for el in body.select(tag):
+                el.decompose()
+        
+        if css_selector:
+            selected_elements = body.select(css_selector)
+            if not selected_elements:
+                return {
+                    'markdown': '',
+                    'cleaned_html': '',
+                    'success': True,
+                    'media': {'images': [], 'videos': [], 'audios': []},
+                    'links': {'internal': [], 'external': []},
+                    'metadata': {},
+                    'message': f"No elements found for CSS selector: {css_selector}"
+                }
+                # raise InvalidCSSSelectorError(f"Invalid CSS selector, No elements found for CSS selector: {css_selector}")
+            body = soup.new_tag('div')
+            for el in selected_elements:
+                body.append(el)
+
+        links = {'internal': [], 'external': []}
+        media = {'images': [], 'videos': [], 'audios': []}
+
+        # Extract meaningful text for media files from closest parent
+        def find_closest_parent_with_useful_text(tag):
+                current_tag = tag
+                while current_tag:
+                    current_tag = current_tag.parent
+                    # Get the text content of the parent tag
+                    if current_tag:
+                        text_content = current_tag.get_text(separator=' ',strip=True)
+                        # Check if the text content has at least word_count_threshold
+                        if len(text_content.split()) >= image_description_min_word_threshold:
+                            return text_content
+                return None
+
+        def process_image(img, url, index, total_images):
+            #Check if an image has valid display and inside undesired html elements
+            def is_valid_image(img, parent, parent_classes):
+                style = img.get('style', '')
+                src = img.get('src', '')
+                classes_to_check = ['button', 'icon', 'logo']
+                tags_to_check = ['button', 'input']
+                return all([
+                    'display:none' not in style,
+                    src,
+                    not any(s in var for var in [src, img.get('alt', ''), *parent_classes] for s in classes_to_check),
+                    parent.name not in tags_to_check
+                ])
+
+            #Score an image for it's usefulness
+            def score_image_for_usefulness(img, base_url, index, images_count):
+                # Function to parse image height/width value and units
+                def parse_dimension(dimension):
+                    if dimension:
+                        match = re.match(r"(\d+)(\D*)", dimension)
+                        if match:
+                            number = int(match.group(1))
+                            unit = match.group(2) or 'px'  # Default unit is 'px' if not specified
+                            return number, unit
+                    return None, None
+
+                # Fetch image file metadata to extract size and extension
+                def fetch_image_file_size(img, base_url):
+                    #If src is relative path construct full URL, if not it may be CDN URL
+                    img_url = urljoin(base_url,img.get('src'))
+                    try:
+                        response = requests.head(img_url)
+                        if response.status_code == 200:
+                            return response.headers.get('Content-Length',None)
+                        else:
+                            print(f"Failed to retrieve file size for {img_url}")
+                            return None
+                    except InvalidSchema as e:
+                        return None
+                    finally:
+                        return
+
+                image_height = img.get('height')
+                height_value, height_unit = parse_dimension(image_height)
+                image_width =  img.get('width')
+                width_value, width_unit = parse_dimension(image_width)
+                image_size = 0 #int(fetch_image_file_size(img,base_url) or 0)
+                image_format = os.path.splitext(img.get('src',''))[1].lower()
+                # Remove . from format
+                image_format = image_format.strip('.').split('?')[0]
+                score = 0
+                if height_value:
+                    if height_unit == 'px' and height_value > 150:
+                        score += 1
+                    if height_unit in ['%','vh','vmin','vmax'] and height_value >30:
+                        score += 1
+                if width_value:
+                    if width_unit == 'px' and width_value > 150:
+                        score += 1
+                    if width_unit in ['%','vh','vmin','vmax'] and width_value >30:
+                        score += 1
+                if image_size > 10000:
+                    score += 1
+                if img.get('alt') != '':
+                    score+=1
+                if any(image_format==format for format in ['jpg','png','webp']):
+                    score+=1
+                if index/images_count<0.5:
+                    score+=1
+                return score
+
+            if not is_valid_image(img, img.parent, img.parent.get('class', [])):
+                return None
+            score = score_image_for_usefulness(img, url, index, total_images)
+            if score <= IMAGE_SCORE_THRESHOLD:
+                return None
+            return {
+                'src': img.get('src', ''),
+                'data-src': img.get('data-src', ''),
+                'alt': img.get('alt', ''),
+                'desc': find_closest_parent_with_useful_text(img),
+                'score': score,
+                'type': 'image'
+            }
+
+        def process_element(element: element.PageElement) -> bool:
+            try:
+                if isinstance(element, NavigableString):
+                    if isinstance(element, Comment):
+                        element.extract()
+                    return False
+                
+                # if element.name == 'img':
+                #     process_image(element, url, 0, 1)
+                #     return True
+
+                if element.name in ['script', 'style', 'link', 'meta', 'noscript']:
+                    element.decompose()
+                    return False
+
+                keep_element = False
+
+                if element.name == 'a' and element.get('href'):
+                    href = element['href']
+                    url_base = url.split('/')[2]
+                    link_data = {'href': href, 'text': element.get_text()}
+                    if href.startswith('http') and url_base not in href:
+                        links['external'].append(link_data)
+                    else:
+                        links['internal'].append(link_data)
+                    keep_element = True
+
+                elif element.name == 'img':
+                    return True  # Always keep image elements
+
+                elif element.name in ['video', 'audio']:
+                    media[f"{element.name}s"].append({
+                        'src': element.get('src'),
+                        'alt': element.get('alt'),
+                        'type': element.name,
+                        'description': find_closest_parent_with_useful_text(element)
+                    })
+                    source_tags = element.find_all('source')
+                    for source_tag in source_tags:
+                        media[f"{element.name}s"].append({
+                        'src': source_tag.get('src'),
+                        'alt': element.get('alt'),
+                        'type': element.name,
+                        'description': find_closest_parent_with_useful_text(element)
+                    })
+                    return True  # Always keep video and audio elements
+
+                if element.name != 'pre':
+                    if element.name in ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']:
+                        if kwargs.get('only_text', False):
+                            element.replace_with(element.get_text())
+                        else:
+                            element.unwrap()
+                    elif element.name != 'img':
+                        element.attrs = {}
+
+                # Process children
+                for child in list(element.children):
+                    if isinstance(child, NavigableString) and not isinstance(child, Comment):
+                        if len(child.strip()) > 0:
+                            keep_element = True
+                    else:
+                        if process_element(child):
+                            keep_element = True
+                    
+
+                # Check word count
+                if not keep_element:
+                    word_count = len(element.get_text(strip=True).split())
+                    keep_element = word_count >= word_count_threshold
+
+                if not keep_element:
+                    element.decompose()
+
+                return keep_element
+            except Exception as e:
+                print('Error processing element:', str(e))
+                return False
+
+        #process images by filtering and extracting contextual text from the page
+        # imgs = body.find_all('img')
+        # media['images'] = [
+        #     result for result in
+        #     (process_image(img, url, i, len(imgs)) for i, img in enumerate(imgs))
+        #     if result is not None
+        # ]
+        
+        process_element(body)
+
+        # # Process images using ThreadPoolExecutor
+        imgs = body.find_all('img')
+        with ThreadPoolExecutor() as executor:
+            image_results = list(executor.map(process_image, imgs, [url]*len(imgs), range(len(imgs)), [len(imgs)]*len(imgs)))
+        media['images'] = [result for result in image_results if result is not None]
+
+        def flatten_nested_elements(node):
+            if isinstance(node, NavigableString):
+                return node
+            if len(node.contents) == 1 and isinstance(node.contents[0], element.Tag) and node.contents[0].name == node.name:
+                return flatten_nested_elements(node.contents[0])
+            node.contents = [flatten_nested_elements(child) for child in node.contents]
+            return node
+
+        body = flatten_nested_elements(body)
+        base64_pattern = re.compile(r'data:image/[^;]+;base64,([^"]+)')
+        for img in imgs:
+            src = img.get('src', '')
+            if base64_pattern.match(src):
+                # Replace base64 data with empty string
+                img['src'] = base64_pattern.sub('', src)
+        cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
+
+        h = CustomHTML2Text()
+        h.ignore_links = True
+        h.body_width = 0
+        try:
+            markdown = h.handle(cleaned_html)
+        except Exception as e:
+            markdown = h.handle(sanitize_html(cleaned_html))
+        markdown = markdown.replace('    ```', '```')
+
+        try:
+            meta = extract_metadata(html, soup)
+        except Exception as e:
+            print('Error extracting metadata:', str(e))
+            meta = {}
+
+        cleaned_html = sanitize_html(cleaned_html)
+        return {
+            'markdown': markdown,
+            'cleaned_html': cleaned_html,
+            'success': True,
+            'media': media,
+            'links': links,
+            'metadata': meta
+        }
--- a/crawl4ai/crawler_strategy.py
+++ b/crawl4ai/crawler_strategy.py
@@ -5,8 +5,13 @@ from selenium.webdriver.common.by import By
 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.support import expected_conditions as EC
 from selenium.webdriver.chrome.options import Options
-from selenium.common.exceptions import InvalidArgumentException
-import logging
+from selenium.common.exceptions import InvalidArgumentException, WebDriverException
+# from selenium.webdriver.chrome.service import Service as ChromeService
+# from webdriver_manager.chrome import ChromeDriverManager
+# from urllib3.exceptions import MaxRetryError
+
+from .config import *
+import logging, time
 import base64
 from PIL import Image, ImageDraw, ImageFont
 from io import BytesIO
@@ -14,7 +19,7 @@ from typing import List, Callable
 import requests
 import os
 from pathlib import Path
-from .utils import wrap_text
+from .utils import *

 logger = logging.getLogger('selenium.webdriver.remote.remote_connection')
 logger.setLevel(logging.WARNING)
@@ -69,7 +74,7 @@ class CloudCrawlerStrategy(CrawlerStrategy):
        response = requests.post("http://crawl4ai.uccode.io/crawl", json=data)
        response = response.json()
        html = response["results"][0]["html"]
-        return html
+        return sanitize_input_encode(html)

 class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
    def __init__(self, use_cached_html=False, js_code=None, **kwargs):
@@ -77,17 +82,25 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
        print("[LOG] 🚀 Initializing LocalSeleniumCrawlerStrategy")
        self.options = Options()
        self.options.headless = True
+        if kwargs.get("proxy"):
+            self.options.add_argument("--proxy-server={}".format(kwargs.get("proxy")))
        if kwargs.get("user_agent"):
            self.options.add_argument("--user-agent=" + kwargs.get("user_agent"))
        else:
-            # Set user agent
            user_agent = kwargs.get("user_agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
-            self.options.add_argument(f"--user-agent={user_agent}")          
-            
-        self.options.add_argument("--no-sandbox")
+            self.options.add_argument(f"--user-agent={user_agent}")
+            self.options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
+                  
        self.options.headless = kwargs.get("headless", True)
        if self.options.headless:
            self.options.add_argument("--headless")
+        
+        self.options.add_argument("--disable-gpu")  
+        self.options.add_argument("--window-size=1920,1080")
+        self.options.add_argument("--no-sandbox")
+        self.options.add_argument("--disable-dev-shm-usage")
+        self.options.add_argument("--disable-blink-features=AutomationControlled")     
+        
        # self.options.add_argument("--disable-dev-shm-usage")
        self.options.add_argument("--disable-gpu")
        # self.options.add_argument("--disable-extensions")
@@ -118,13 +131,23 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
        }

        # chromedriver_autoinstaller.install()
-        import chromedriver_autoinstaller
-        crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
-        chromedriver_path = chromedriver_autoinstaller.utils.download_chromedriver(crawl4ai_folder, False)
+        # import chromedriver_autoinstaller
+        # crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        # driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=self.options)
+        # chromedriver_path = chromedriver_autoinstaller.install()
+        # chromedriver_path = chromedriver_autoinstaller.utils.download_chromedriver()
        # self.service = Service(chromedriver_autoinstaller.install())
-        self.service = Service(chromedriver_path)
-        self.service.log_path = "NUL"
-        self.driver = webdriver.Chrome(service=self.service, options=self.options)
+        
+        
+        # chromedriver_path = ChromeDriverManager().install()
+        # self.service = Service(chromedriver_path)
+        # self.service.log_path = "NUL"
+        # self.driver = webdriver.Chrome(service=self.service, options=self.options)
+        
+        # Use selenium-manager (built into Selenium 4.10.0+)
+        self.service = Service()
+        self.driver = webdriver.Chrome(options=self.options)
+        
        self.driver = self.execute_hook('on_driver_created', self.driver)
        
        if kwargs.get("cookies"):
@@ -163,8 +186,20 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
        # Set extra HTTP headers
        self.driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': headers})

+    def _ensure_page_load(self,  max_checks=6, check_interval=0.01):
+        initial_length = len(self.driver.page_source)
+        
+        for ix in range(max_checks):
+            # print(f"Checking page load: {ix}")
+            time.sleep(check_interval)
+            current_length = len(self.driver.page_source)
+            
+            if current_length != initial_length:
+                break

-    def crawl(self, url: str) -> str:
+        return self.driver.page_source
+    
+    def crawl(self, url: str, **kwargs) -> str:
        # Create md5 hash of the URL
        import hashlib
        url_hash = hashlib.md5(url.encode()).hexdigest()
@@ -173,19 +208,43 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
            cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", url_hash)
            if os.path.exists(cache_file_path):
                with open(cache_file_path, "r") as f:
-                    return f.read()
+                    return sanitize_input_encode(f.read())

        try:
            self.driver = self.execute_hook('before_get_url', self.driver)
            if self.verbose:
                print(f"[LOG] 🕸️ Crawling {url} using LocalSeleniumCrawlerStrategy...")
-            self.driver.get(url)
-            WebDriverWait(self.driver, 10).until(
-                EC.presence_of_all_elements_located((By.TAG_NAME, "html"))
+            self.driver.get(url) #<html><head></head><body></body></html>
+            
+            WebDriverWait(self.driver, 20).until(
+                lambda d: d.execute_script('return document.readyState') == 'complete'
            )
+            WebDriverWait(self.driver, 10).until(
+                EC.presence_of_all_elements_located((By.TAG_NAME, "body"))
+            )
+            
+            self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
+            
            self.driver = self.execute_hook('after_get_url', self.driver)
+            html = sanitize_input_encode(self._ensure_page_load()) # self.driver.page_source                                        
+            can_not_be_done_headless = False # Look at my creativity for naming variables
+            
+            # TODO: Very ugly approach, but promise to change it!
+            if kwargs.get('bypass_headless', False) or html == "<html><head></head><body></body></html>":
+                print("[LOG] 🙌 Page could not be loaded in headless mode. Trying non-headless mode...")
+                can_not_be_done_headless = True
+                options = Options()
+                options.headless = False
+                # set window size very small
+                options.add_argument("--window-size=5,5")
+                driver = webdriver.Chrome(service=self.service, options=options)
+                driver.get(url)
+                self.driver = self.execute_hook('after_get_url', driver)
+                html = sanitize_input_encode(driver.page_source)
+                driver.quit()
            
            # Execute JS code if provided
+            self.js_code = kwargs.get("js_code", self.js_code)
            if self.js_code and type(self.js_code) == str:
                self.driver.execute_script(self.js_code)
                # Optionally, wait for some condition after executing the JS code
@@ -199,12 +258,25 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
                        lambda driver: driver.execute_script("return document.readyState") == "complete"
                    )
            
-            html = self.driver.page_source
+            # Optionally, wait for some condition after executing the JS code : Contributed by (https://github.com/jonymusky)
+            wait_for = kwargs.get('wait_for', False)
+            if wait_for:
+                if callable(wait_for):
+                    print("[LOG] 🔄 Waiting for condition...")
+                    WebDriverWait(self.driver, 20).until(wait_for)
+                else:
+                    print("[LOG] 🔄 Waiting for condition...")
+                    WebDriverWait(self.driver, 20).until(
+                        EC.presence_of_element_located((By.CSS_SELECTOR, wait_for))
+                    ) 
+            
+            if not can_not_be_done_headless:
+                html = sanitize_input_encode(self.driver.page_source)
            self.driver = self.execute_hook('before_return_html', self.driver, html)
            
            # Store in cache
            cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", url_hash)
-            with open(cache_file_path, "w") as f:
+            with open(cache_file_path, "w", encoding="utf-8") as f:
                f.write(html)
                
            if self.verbose:
@@ -212,9 +284,18 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
            
            return html
        except InvalidArgumentException:
-            raise InvalidArgumentException(f"Invalid URL {url}")
+            if not hasattr(e, 'msg'):
+                e.msg = sanitize_input_encode(str(e))
+            raise InvalidArgumentException(f"Failed to crawl {url}: {e.msg}")
+        except WebDriverException as e:
+            # If e does nlt have msg attribute create it and set it to str(e)
+            if not hasattr(e, 'msg'):
+                e.msg = sanitize_input_encode(str(e))
+            raise WebDriverException(f"Failed to crawl {url}: {e.msg}")  
        except Exception as e:
-            raise Exception(f"Failed to crawl {url}: {str(e)}")
+            if not hasattr(e, 'msg'):
+                e.msg = sanitize_input_encode(str(e))
+            raise Exception(f"Failed to crawl {url}: {e.msg}")

    def take_screenshot(self) -> str:
        try:
@@ -231,18 +312,20 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
            # Open the screenshot with PIL
            image = Image.open(BytesIO(screenshot))

+            # Convert image to RGB mode (this will handle both RGB and RGBA images)
+            rgb_image = image.convert('RGB')
+
            # Convert to JPEG and compress
            buffered = BytesIO()
-            image.save(buffered, format="JPEG", quality=85)
+            rgb_image.save(buffered, format="JPEG", quality=85)
            img_base64 = base64.b64encode(buffered.getvalue()).decode('utf-8')

            if self.verbose:
                print(f"[LOG] 📸 Screenshot taken and converted to base64")

            return img_base64
-
        except Exception as e:
-            error_message = f"Failed to take screenshot: {str(e)}"
+            error_message = sanitize_input_encode(f"Failed to take screenshot: {str(e)}")
            print(error_message)

            # Generate an image with black background
@@ -253,7 +336,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
            try:
                font = ImageFont.truetype("arial.ttf", 40)
            except IOError:
-                font = ImageFont.load_default(size=40)
+                font = ImageFont.load_default()

            # Define text color and wrap the text
            text_color = (255, 255, 255)
@@ -272,6 +355,6 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
            img_base64 = base64.b64encode(buffered.getvalue()).decode('utf-8')

            return img_base64
-
+        
    def quit(self):
-        self.driver.quit()
+        self.driver.quit()
--- a/crawl4ai/database.py
+++ b/crawl4ai/database.py
@@ -20,7 +20,7 @@ def init_db():
            extracted_content TEXT,
            success BOOLEAN,
            media TEXT DEFAULT "{}",
-            link TEXT DEFAULT "{}",
+            links TEXT DEFAULT "{}",
            metadata TEXT DEFAULT "{}",
            screenshot TEXT DEFAULT ""
        )
@@ -127,6 +127,9 @@ def update_existing_records(new_column: str = "media", default_value: str = "{}"
        print(f"Error updating existing records: {e}")

 if __name__ == "__main__":
-    init_db()  # Initialize the database if not already initialized
-    alter_db_add_screenshot("metadata")  # Add the new column to the table
-    update_existing_records("metadata")  # Update existing records to set the new column to an empty string
+    # Delete the existing database file
+    if os.path.exists(DB_PATH):
+        os.remove(DB_PATH)
+    init_db()  
+    # alter_db_add_screenshot("COL_NAME")
+    
--- a/crawl4ai/extraction_strategy.py
+++ b/crawl4ai/extraction_strategy.py
@@ -9,8 +9,9 @@ from .utils import *
 from functools import partial
 from .model_loader import *
 import math
-
 import numpy as np
+from lxml import etree
+
 class ExtractionStrategy(ABC):
    """
    Abstract base class for all extraction strategies.
@@ -78,6 +79,8 @@ class LLMExtractionStrategy(ExtractionStrategy):
        self.overlap_rate = kwargs.get("overlap_rate", OVERLAP_RATE)
        self.word_token_rate = kwargs.get("word_token_rate", WORD_TOKEN_RATE)
        self.apply_chunking = kwargs.get("apply_chunking", True)
+        self.base_url = kwargs.get("base_url", None)
+        self.extra_args = kwargs.get("extra_args", {})
        if not self.apply_chunking:
            self.chunk_token_threshold = 1e9
        
@@ -100,8 +103,8 @@ class LLMExtractionStrategy(ExtractionStrategy):
            variable_values["REQUEST"] = self.instruction
            prompt_with_variables = PROMPT_EXTRACT_BLOCKS_WITH_INSTRUCTION
            
-        if self.extract_type == "schema":
-            variable_values["SCHEMA"] = json.dumps(self.schema)
+        if self.extract_type == "schema" and self.schema:
+            variable_values["SCHEMA"] = json.dumps(self.schema, indent=2)
            prompt_with_variables = PROMPT_EXTRACT_SCHEMA_WITH_INSTRUCTION

        for variable in variable_values:
@@ -109,14 +112,19 @@ class LLMExtractionStrategy(ExtractionStrategy):
                "{" + variable + "}", variable_values[variable]
            )
        
-        response = perform_completion_with_backoff(self.provider, prompt_with_variables, self.api_token)
+        response = perform_completion_with_backoff(
+            self.provider, 
+            prompt_with_variables, 
+            self.api_token, 
+            base_url=self.base_url,
+            extra_args = self.extra_args
+            ) # , json_response=self.extract_type == "schema")
        try:
            blocks = extract_xml_data(["blocks"], response.choices[0].message.content)['blocks']
            blocks = json.loads(blocks)
            for block in blocks:
                block['error'] = False
        except Exception as e:
-            print("Error extracting blocks:", str(e))
            parsed, unparsed = split_and_parse_json_objects(response.choices[0].message.content)
            blocks = parsed
            if unparsed:
@@ -192,16 +200,31 @@ class LLMExtractionStrategy(ExtractionStrategy):
            # Sequential processing with a delay
            for ix, section in enumerate(merged_sections):
                extract_func = partial(self.extract, url)
-                extracted_content.extend(extract_func(ix, section))
+                extracted_content.extend(extract_func(ix, sanitize_input_encode(section)))
                time.sleep(0.5)  # 500 ms delay between each processing
        else:
            # Parallel processing using ThreadPoolExecutor
+            # extract_func = partial(self.extract, url)
+            # for ix, section in enumerate(merged_sections):
+            #     extracted_content.append(extract_func(ix, section))            
+            
            with ThreadPoolExecutor(max_workers=4) as executor:
                extract_func = partial(self.extract, url)
-                futures = [executor.submit(extract_func, ix, section) for ix, section in enumerate(merged_sections)]
+                futures = [executor.submit(extract_func, ix, sanitize_input_encode(section)) for ix, section in enumerate(merged_sections)]
                
                for future in as_completed(futures):
-                    extracted_content.extend(future.result())
+                    try:
+                        extracted_content.extend(future.result())
+                    except Exception as e:
+                        if self.verbose:
+                            print(f"Error in thread execution: {e}")
+                        # Add error information to extracted_content
+                        extracted_content.append({
+                            "index": 0,
+                            "error": True,
+                            "tags": ["error"],
+                            "content": str(e)
+                        })

        
        return extracted_content        
@@ -219,6 +242,8 @@ class CosineStrategy(ExtractionStrategy):
        """
        super().__init__()
        
+        import numpy as np
+        
        self.semantic_filter = semantic_filter
        self.word_count_threshold = word_count_threshold
        self.max_dist = max_dist
@@ -232,6 +257,9 @@ class CosineStrategy(ExtractionStrategy):
        self.get_embedding_method = "direct"
        
        self.device = get_device()
+        import torch
+        self.device = torch.device('cpu')
+        
        self.default_batch_size = calculate_batch_size(self.device)

        if self.verbose:
@@ -244,7 +272,9 @@ class CosineStrategy(ExtractionStrategy):
        # else:

        self.tokenizer, self.model = load_bge_small_en_v1_5()
+        self.model.to(self.device)
        self.model.eval()  
+        
        self.get_embedding_method = "batch"
        
        self.buffer_embeddings = np.array([])
@@ -266,7 +296,7 @@ class CosineStrategy(ExtractionStrategy):
        if self.verbose:
            print(f"[LOG] Loading Multilabel Classifier for {self.device.type} device.")
            
-        self.nlp, self.device = load_text_multilabel_classifier()
+        self.nlp, _ = load_text_multilabel_classifier()
        # self.default_batch_size = 16 if self.device.type == 'cpu' else 64
        
        if self.verbose:
@@ -437,21 +467,21 @@ class CosineStrategy(ExtractionStrategy):
        if self.verbose:
            print(f"[LOG] 🚀 Assign tags using {self.device}")
        
-        if self.device.type in ["gpu", "cuda", "mps"]:
+        if self.device.type in ["gpu", "cuda", "mps", "cpu"]:
            labels = self.nlp([cluster['content'] for cluster in cluster_list])
            
            for cluster, label in zip(cluster_list, labels):
                cluster['tags'] = label
-        elif self.device == "cpu":
-            # Process the text with the loaded model
-            texts = [cluster['content'] for cluster in cluster_list]
-            # Batch process texts
-            docs = self.nlp.pipe(texts, disable=["tagger", "parser", "ner", "lemmatizer"])
+        # elif self.device.type == "cpu":
+        #     # Process the text with the loaded model
+        #     texts = [cluster['content'] for cluster in cluster_list]
+        #     # Batch process texts
+        #     docs = self.nlp.pipe(texts, disable=["tagger", "parser", "ner", "lemmatizer"])

-            for doc, cluster in zip(docs, cluster_list):
-                tok_k = self.top_k
-                top_categories = sorted(doc.cats.items(), key=lambda x: x[1], reverse=True)[:tok_k]
-                cluster['tags'] = [cat for cat, _ in top_categories]
+        #     for doc, cluster in zip(docs, cluster_list):
+        #         tok_k = self.top_k
+        #         top_categories = sorted(doc.cats.items(), key=lambda x: x[1], reverse=True)[:tok_k]
+        #         cluster['tags'] = [cat for cat, _ in top_categories]
                            
            # for cluster in  cluster_list:
            #     doc = self.nlp(cluster['content'])
@@ -600,3 +630,241 @@ class ContentSummarizationStrategy(ExtractionStrategy):
        # Sort summaries by the original section index to maintain order
        summaries.sort(key=lambda x: x[0])
        return [summary for _, summary in summaries]
+  
+class JsonCssExtractionStrategy(ExtractionStrategy):
+    def __init__(self, schema: Dict[str, Any], **kwargs):
+        super().__init__(**kwargs)
+        self.schema = schema
+
+    def extract(self, url: str, html: str, *q, **kwargs) -> List[Dict[str, Any]]:
+        soup = BeautifulSoup(html, 'html.parser')
+        base_elements = soup.select(self.schema['baseSelector'])
+        
+        results = []
+        for element in base_elements:
+            item = self._extract_item(element, self.schema['fields'])
+            if item:
+                results.append(item)
+        
+        return results
+
+    
+
+    def _extract_field(self, element, field):
+        try:
+            if field['type'] == 'nested':
+                nested_element = element.select_one(field['selector'])
+                return self._extract_item(nested_element, field['fields']) if nested_element else {}
+            
+            if field['type'] == 'list':
+                elements = element.select(field['selector'])
+                return [self._extract_list_item(el, field['fields']) for el in elements]
+            
+            if field['type'] == 'nested_list':
+                elements = element.select(field['selector'])
+                return [self._extract_item(el, field['fields']) for el in elements]
+            
+            return self._extract_single_field(element, field)
+        except Exception as e:
+            if self.verbose:
+                print(f"Error extracting field {field['name']}: {str(e)}")
+            return field.get('default')
+
+    def _extract_list_item(self, element, fields):
+        item = {}
+        for field in fields:
+            value = self._extract_single_field(element, field)
+            if value is not None:
+                item[field['name']] = value
+        return item
+    
+    def _extract_single_field(self, element, field):
+        if 'selector' in field:
+            selected = element.select_one(field['selector'])
+            if not selected:
+                return field.get('default')
+        else:
+            selected = element
+
+        value = None
+        if field['type'] == 'text':
+            value = selected.get_text(strip=True)
+        elif field['type'] == 'attribute':
+            value = selected.get(field['attribute'])
+        elif field['type'] == 'html':
+            value = str(selected)
+        elif field['type'] == 'regex':
+            text = selected.get_text(strip=True)
+            match = re.search(field['pattern'], text)
+            value = match.group(1) if match else None
+
+        if 'transform' in field:
+            value = self._apply_transform(value, field['transform'])
+
+        return value if value is not None else field.get('default')
+
+    def _extract_item(self, element, fields):
+        item = {}
+        for field in fields:
+            if field['type'] == 'computed':
+                value = self._compute_field(item, field)
+            else:
+                value = self._extract_field(element, field)
+            if value is not None:
+                item[field['name']] = value
+        return item
+    
+    def _apply_transform(self, value, transform):
+        if transform == 'lowercase':
+            return value.lower()
+        elif transform == 'uppercase':
+            return value.upper()
+        elif transform == 'strip':
+            return value.strip()
+        return value
+
+    def _compute_field(self, item, field):
+        try:
+            if 'expression' in field:
+                return eval(field['expression'], {}, item)
+            elif 'function' in field:
+                return field['function'](item)
+        except Exception as e:
+            if self.verbose:
+                print(f"Error computing field {field['name']}: {str(e)}")
+            return field.get('default')
+
+    def run(self, url: str, sections: List[str], *q, **kwargs) -> List[Dict[str, Any]]:
+        combined_html = self.DEL.join(sections)
+        return self.extract(url, combined_html, **kwargs)
+    
+
+class JsonXPATHExtractionStrategy(ExtractionStrategy):
+    def __init__(self, schema: Dict[str, Any], **kwargs):
+        super().__init__(**kwargs)
+        self.schema = schema
+        self.use_cssselect = self._check_cssselect()
+
+    def _check_cssselect(self):
+        try:
+            import cssselect
+            return True
+        except ImportError:
+            print("Warning: cssselect is not installed. Falling back to XPath for all selectors.")
+            return False
+
+    def extract(self, url: str, html: str, *q, **kwargs) -> List[Dict[str, Any]]:
+        self.soup = BeautifulSoup(html, 'lxml')
+        self.tree = etree.HTML(str(self.soup))
+        
+        selector_type = 'xpath' if not self.use_cssselect else self.schema.get('selectorType', 'css')
+        base_selector = self.schema.get('baseXPath' if selector_type == 'xpath' else 'baseSelector')
+        base_elements = self._select_elements(base_selector, selector_type)
+        
+        results = []
+        for element in base_elements:
+            item = self._extract_item(element, self.schema['fields'])
+            if item:
+                results.append(item)
+        
+        return results
+
+    def _select_elements(self, selector, selector_type, element=None):
+        if selector_type == 'xpath' or not self.use_cssselect:
+            return self.tree.xpath(selector) if element is None else element.xpath(selector)
+        else:  # CSS
+            return self.tree.cssselect(selector) if element is None else element.cssselect(selector)
+
+    def _extract_field(self, element, field):
+        try:
+            selector_type = 'xpath' if not self.use_cssselect else field.get('selectorType', 'css')
+            selector = field.get('xpathSelector' if selector_type == 'xpath' else 'selector')
+            
+            if field['type'] == 'nested':
+                nested_element = self._select_elements(selector, selector_type, element)
+                return self._extract_item(nested_element[0], field['fields']) if nested_element else {}
+            
+            if field['type'] == 'list':
+                elements = self._select_elements(selector, selector_type, element)
+                return [self._extract_list_item(el, field['fields']) for el in elements]
+            
+            if field['type'] == 'nested_list':
+                elements = self._select_elements(selector, selector_type, element)
+                return [self._extract_item(el, field['fields']) for el in elements]
+            
+            return self._extract_single_field(element, field)
+        except Exception as e:
+            if self.verbose:
+                print(f"Error extracting field {field['name']}: {str(e)}")
+            return field.get('default')
+
+    def _extract_list_item(self, element, fields):
+        item = {}
+        for field in fields:
+            value = self._extract_single_field(element, field)
+            if value is not None:
+                item[field['name']] = value
+        return item
+    
+    def _extract_single_field(self, element, field):
+        selector_type = field.get('selectorType', 'css')
+        
+        if 'selector' in field:
+            selected = self._select_elements(field['selector'], selector_type, element)
+            if not selected:
+                return field.get('default')
+            selected = selected[0]
+        else:
+            selected = element
+
+        value = None
+        if field['type'] == 'text':
+            value = selected.text_content().strip() if hasattr(selected, 'text_content') else selected.text.strip()
+        elif field['type'] == 'attribute':
+            value = selected.get(field['attribute'])
+        elif field['type'] == 'html':
+            value = etree.tostring(selected, encoding='unicode')
+        elif field['type'] == 'regex':
+            text = selected.text_content().strip() if hasattr(selected, 'text_content') else selected.text.strip()
+            match = re.search(field['pattern'], text)
+            value = match.group(1) if match else None
+
+        if 'transform' in field:
+            value = self._apply_transform(value, field['transform'])
+
+        return value if value is not None else field.get('default')
+
+    def _extract_item(self, element, fields):
+        item = {}
+        for field in fields:
+            if field['type'] == 'computed':
+                value = self._compute_field(item, field)
+            else:
+                value = self._extract_field(element, field)
+            if value is not None:
+                item[field['name']] = value
+        return item
+    
+    def _apply_transform(self, value, transform):
+        if transform == 'lowercase':
+            return value.lower()
+        elif transform == 'uppercase':
+            return value.upper()
+        elif transform == 'strip':
+            return value.strip()
+        return value
+
+    def _compute_field(self, item, field):
+        try:
+            if 'expression' in field:
+                return eval(field['expression'], {}, item)
+            elif 'function' in field:
+                return field['function'](item)
+        except Exception as e:
+            if self.verbose:
+                print(f"Error computing field {field['name']}: {str(e)}")
+            return field.get('default')
+
+    def run(self, url: str, sections: List[str], *q, **kwargs) -> List[Dict[str, Any]]:
+        combined_html = self.DEL.join(sections)
+        return self.extract(url, combined_html, **kwargs)
--- a/crawl4ai/model_loader.py
+++ b/crawl4ai/model_loader.py
@@ -3,9 +3,10 @@ from pathlib import Path
 import subprocess, os
 import shutil
 import tarfile
-from crawl4ai.config import MODEL_REPO_BRANCH
+from .model_loader import *
 import argparse
 import urllib.request
+from crawl4ai.config import MODEL_REPO_BRANCH
 __location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))

@lru_cache()
@@ -79,48 +80,6 @@ def load_bge_small_en_v1_5():
    model, device = set_model_device(model)
    return tokenizer, model

-@lru_cache()
-def load_onnx_all_MiniLM_l6_v2():
-    from crawl4ai.onnx_embedding import DefaultEmbeddingModel
-
-    model_path = "models/onnx.tar.gz"
-    model_url = "https://unclecode-files.s3.us-west-2.amazonaws.com/onnx.tar.gz"
-    __location__ = os.path.realpath(
-        os.path.join(os.getcwd(), os.path.dirname(__file__)))
-    download_path = os.path.join(__location__, model_path)
-    onnx_dir = os.path.join(__location__, "models/onnx")
-    
-    # Create the models directory if it does not exist
-    os.makedirs(os.path.dirname(download_path), exist_ok=True)
-
-    # Download the tar.gz file if it does not exist
-    if not os.path.exists(download_path):
-        def download_with_progress(url, filename):
-            def reporthook(block_num, block_size, total_size):
-                downloaded = block_num * block_size
-                percentage = 100 * downloaded / total_size
-                if downloaded < total_size:
-                    print(f"\rDownloading: {percentage:.2f}% ({downloaded / (1024 * 1024):.2f} MB of {total_size / (1024 * 1024):.2f} MB)", end='')
-                else:
-                    print("\rDownload complete!")
-
-            urllib.request.urlretrieve(url, filename, reporthook)
-
-        download_with_progress(model_url, download_path)
-
-    # Extract the tar.gz file if the onnx directory does not exist
-    if not os.path.exists(onnx_dir):
-        with tarfile.open(download_path, "r:gz") as tar:
-            tar.extractall(path=os.path.join(__location__, "models"))
-        
-        # remove the tar.gz file
-        os.remove(download_path)
-    
-    
-    
-    model = DefaultEmbeddingModel()
-    return model
-
@lru_cache()
 def load_text_classifier():
    from transformers import AutoTokenizer, AutoModelForSequenceClassification
@@ -141,14 +100,15 @@ def load_text_multilabel_classifier():
    from scipy.special import expit
    import torch

-    # Check for available device: CUDA, MPS (for Apple Silicon), or CPU
-    if torch.cuda.is_available():
-        device = torch.device("cuda")
-    elif torch.backends.mps.is_available():
-        device = torch.device("mps")
-    else:
-        return load_spacy_model(), torch.device("cpu")
-
+    # # Check for available device: CUDA, MPS (for Apple Silicon), or CPU
+    # if torch.cuda.is_available():
+    #     device = torch.device("cuda")
+    # elif torch.backends.mps.is_available():
+    #     device = torch.device("mps")
+    # else:
+    #     device = torch.device("cpu")
+    #     # return load_spacy_model(), torch.device("cpu")
+    

    MODEL = "cardiffnlp/tweet-topic-21-multi"
    tokenizer = AutoTokenizer.from_pretrained(MODEL, resume_download=None)
@@ -186,57 +146,66 @@ def load_nltk_punkt():
        nltk.download('punkt')
    return nltk.data.find('tokenizers/punkt')

-
@lru_cache()
 def load_spacy_model():
    import spacy
    name = "models/reuters"
    home_folder = get_home_folder()
-    model_folder = os.path.join(home_folder, name)
+    model_folder = Path(home_folder) / name
    
    # Check if the model directory already exists
-    if not (Path(model_folder).exists() and any(Path(model_folder).iterdir())):
+    if not (model_folder.exists() and any(model_folder.iterdir())):
        repo_url = "https://github.com/unclecode/crawl4ai.git"
-        # branch = "main"
        branch = MODEL_REPO_BRANCH 
-        repo_folder = os.path.join(home_folder, "crawl4ai")
-        model_folder = os.path.join(home_folder, name)
-
-        # print("[LOG] ⏬ Downloading Spacy model for the first time...")
+        repo_folder = Path(home_folder) / "crawl4ai"
+        
+        print("[LOG] ⏬ Downloading Spacy model for the first time...")

        # Remove existing repo folder if it exists
-        if Path(repo_folder).exists():
-            shutil.rmtree(repo_folder)
-            shutil.rmtree(model_folder)
+        if repo_folder.exists():
+            try:
+                shutil.rmtree(repo_folder)
+                if model_folder.exists():
+                    shutil.rmtree(model_folder)
+            except PermissionError:
+                print("[WARNING] Unable to remove existing folders. Please manually delete the following folders and try again:")
+                print(f"- {repo_folder}")
+                print(f"- {model_folder}")
+                return None

        try:
            # Clone the repository
            subprocess.run(
-                ["git", "clone", "-b", branch, repo_url, repo_folder],
+                ["git", "clone", "-b", branch, repo_url, str(repo_folder)],
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
                check=True
            )

            # Create the models directory if it doesn't exist
-            models_folder = os.path.join(home_folder, "models")
-            os.makedirs(models_folder, exist_ok=True)
+            models_folder = Path(home_folder) / "models"
+            models_folder.mkdir(parents=True, exist_ok=True)

            # Copy the reuters model folder to the models directory
-            source_folder = os.path.join(repo_folder, "models/reuters")
+            source_folder = repo_folder / "models" / "reuters"
            shutil.copytree(source_folder, model_folder)

            # Remove the cloned repository
            shutil.rmtree(repo_folder)

-            # Print completion message
-            # print("[LOG] ✅ Spacy Model downloaded successfully")
+            print("[LOG] ✅ Spacy Model downloaded successfully")
        except subprocess.CalledProcessError as e:
            print(f"An error occurred while cloning the repository: {e}")
+            return None
        except Exception as e:
            print(f"An error occurred: {e}")
+            return None

-    return spacy.load(model_folder)
+    try:
+        return spacy.load(str(model_folder))
+    except Exception as e:
+        print(f"Error loading spacy model: {e}")
+        return None

 def download_all_models(remove_existing=False):
    """Download all models required for Crawl4AI."""
--- a/crawl4ai/models.py
+++ b/crawl4ai/models.py
@@ -16,4 +16,7 @@ class CrawlResult(BaseModel):
    markdown: Optional[str] = None
    extracted_content: Optional[str] = None
    metadata: Optional[dict] = None
-    error_message: Optional[str] = None
+    error_message: Optional[str] = None
+    session_id: Optional[str] = None
+    response_headers: Optional[dict] = None
+    status_code: Optional[int] = None
--- a/crawl4ai/onnx_embedding.py
+++ b/crawl4ai/onnx_embedding.py
@@ -1,50 +0,0 @@
-# A dependency-light way to run the onnx model
-
-
-import numpy as np
-from typing import List
-import os
-
-__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
-MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"
-
-def normalize(v):
-    norm = np.linalg.norm(v, axis=1)
-    norm[norm == 0] = 1e-12
-    return v / norm[:, np.newaxis]
-
-# Sampel implementation of the default sentence-transformers model using ONNX
-class DefaultEmbeddingModel():
-
-    def __init__(self):
-        from tokenizers import Tokenizer
-        import onnxruntime as ort
-        # max_seq_length = 256, for some reason sentence-transformers uses 256 even though the HF config has a max length of 128
-        # https://github.com/UKPLab/sentence-transformers/blob/3e1929fddef16df94f8bc6e3b10598a98f46e62d/docs/_static/html/models_en_sentence_embeddings.html#LL480
-        self.tokenizer = Tokenizer.from_file(os.path.join(__location__, "models/onnx/tokenizer.json"))
-        self.tokenizer.enable_truncation(max_length=256)
-        self.tokenizer.enable_padding(pad_id=0, pad_token="[PAD]", length=256)
-        self.model = ort.InferenceSession(os.path.join(__location__,"models/onnx/model.onnx"))
-        
-
-    def __call__(self, documents: List[str], batch_size: int = 32):
-        all_embeddings = []
-        for i in range(0, len(documents), batch_size):
-            batch = documents[i:i + batch_size]
-            encoded = [self.tokenizer.encode(d) for d in batch]
-            input_ids = np.array([e.ids for e in encoded])
-            attention_mask = np.array([e.attention_mask for e in encoded])
-            onnx_input = {
-                "input_ids": np.array(input_ids, dtype=np.int64),
-                "attention_mask": np.array(attention_mask, dtype=np.int64),
-                "token_type_ids": np.array([np.zeros(len(e), dtype=np.int64) for e in input_ids], dtype=np.int64),
-            }
-            model_output = self.model.run(None, onnx_input)
-            last_hidden_state = model_output[0]
-            # Perform mean pooling with attention weighting
-            input_mask_expanded = np.broadcast_to(np.expand_dims(attention_mask, -1), last_hidden_state.shape)
-            embeddings = np.sum(last_hidden_state * input_mask_expanded, 1) / np.clip(input_mask_expanded.sum(1), a_min=1e-9, a_max=None)
-            embeddings = normalize(embeddings).astype(np.float32)
-            all_embeddings.append(embeddings)
-        return np.concatenate(all_embeddings)
-
--- a/crawl4ai/prompts.py
+++ b/crawl4ai/prompts.py
@@ -1,4 +1,4 @@
-PROMPT_EXTRACT_BLOCKS = """YHere is the URL of the webpage:
+PROMPT_EXTRACT_BLOCKS = """Here is the URL of the webpage:
 <url>{URL}</url>

 And here is the cleaned HTML content of that webpage:
@@ -29,7 +29,7 @@ To generate the JSON objects:

 5. Make sure the generated JSON is complete and parsable, with no errors or omissions.

-6. Make sur to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
+6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.

 Please provide your output within <blocks> tags, like this:

@@ -79,7 +79,7 @@ To generate the JSON objects:
 2. For each block:
   a. Assign it an index based on its order in the content.
   b. Analyze the content and generate ONE semantic tag that describe what the block is about.
-   c. Extract the text content, EXACTLY SAME AS GIVE DATA, clean it up if needed, and store it as a list of strings in the "content" field.
+   c. Extract the text content, EXACTLY SAME AS THE GIVE DATA, clean it up if needed, and store it as a list of strings in the "content" field.

 3. Ensure that the order of the JSON objects matches the order of the blocks as they appear in the original HTML content.

@@ -87,7 +87,7 @@ To generate the JSON objects:

 5. Make sure the generated JSON is complete and parsable, with no errors or omissions.

-6. Make sur to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
+6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.

 7. Never alter the extracted content, just copy and paste it as it is.

@@ -142,7 +142,7 @@ To generate the JSON objects:

 5. Make sure the generated JSON is complete and parsable, with no errors or omissions.

-6. Make sur to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
+6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.

 7. Never alter the extracted content, just copy and paste it as it is.

@@ -186,7 +186,7 @@ The user has made the following request for what information to extract from the
 Please carefully read the URL content and the user's request. If the user provided a desired JSON schema in the <schema_block> above, extract the requested information from the URL content according to that schema. If no schema was provided, infer an appropriate JSON schema based on the user's request that will best capture the key information they are looking for.

 Extraction instructions:
-Return the extracted information as a list of JSON objects, with each object in the list corresponding to a block of content from the URL, in the same order as it appears on the page. Wrap the entire JSON list in <blocks> tags.
+Return the extracted information as a list of JSON objects, with each object in the list corresponding to a block of content from the URL, in the same order as it appears on the page. Wrap the entire JSON list in <blocks>...</blocks> XML tags.

 Quality Reflection:
 Before outputting your final answer, double check that the JSON you are returning is complete, containing all the information requested by the user, and is valid JSON that could be parsed by json.loads() with no errors or omissions. The outputted JSON objects should fully match the schema, either provided or inferred.
@@ -194,5 +194,11 @@ Before outputting your final answer, double check that the JSON you are returnin
 Quality Score:
 After reflecting, score the quality and completeness of the JSON data you are about to return on a scale of 1 to 5. Write the score inside <score> tags.

+Avoid Common Mistakes:
+- Do NOT add any comments using "//" or "#" in the JSON output. It causes parsing errors.
+- Make sure the JSON is properly formatted with curly braces, square brackets, and commas in the right places.
+- Do not miss closing </blocks> tag at the end of the JSON output.
+- Do not generate the Python coee show me how to do the task, this is your task to extract the information and return it in JSON format.
+
 Result
-Output the final list of JSON objects, wrapped in <blocks> tags."""
+Output the final list of JSON objects, wrapped in <blocks>...</blocks> XML tags. Make sure to close the tag properly."""
--- a/crawl4ai/utils.py
+++ b/crawl4ai/utils.py
@@ -6,15 +6,59 @@ import json
 import html
 import re
 import os
+import platform
 from html2text import HTML2Text
 from .prompts import PROMPT_EXTRACT_BLOCKS
 from .config import *
 from pathlib import Path
 from typing import Dict, Any
+from urllib.parse import urljoin
+import requests
+from requests.exceptions import InvalidSchema

 class InvalidCSSSelectorError(Exception):
    pass

+def calculate_semaphore_count():
+    cpu_count = os.cpu_count()
+    memory_gb = get_system_memory() / (1024 ** 3)  # Convert to GB
+    base_count = max(1, cpu_count // 2)
+    memory_based_cap = int(memory_gb / 2)  # Assume 2GB per instance
+    return min(base_count, memory_based_cap)
+
+def get_system_memory():
+    system = platform.system()
+    if system == "Linux":
+        with open('/proc/meminfo', 'r') as mem:
+            for line in mem:
+                if line.startswith('MemTotal:'):
+                    return int(line.split()[1]) * 1024  # Convert KB to bytes
+    elif system == "Darwin":  # macOS
+        import subprocess
+        output = subprocess.check_output(['sysctl', '-n', 'hw.memsize']).decode('utf-8')
+        return int(output.strip())
+    elif system == "Windows":
+        import ctypes
+        kernel32 = ctypes.windll.kernel32
+        c_ulonglong = ctypes.c_ulonglong
+        class MEMORYSTATUSEX(ctypes.Structure):
+            _fields_ = [
+                ('dwLength', ctypes.c_ulong),
+                ('dwMemoryLoad', ctypes.c_ulong),
+                ('ullTotalPhys', c_ulonglong),
+                ('ullAvailPhys', c_ulonglong),
+                ('ullTotalPageFile', c_ulonglong),
+                ('ullAvailPageFile', c_ulonglong),
+                ('ullTotalVirtual', c_ulonglong),
+                ('ullAvailVirtual', c_ulonglong),
+                ('ullAvailExtendedVirtual', c_ulonglong),
+            ]
+        memoryStatus = MEMORYSTATUSEX()
+        memoryStatus.dwLength = ctypes.sizeof(MEMORYSTATUSEX)
+        kernel32.GlobalMemoryStatusEx(ctypes.byref(memoryStatus))
+        return memoryStatus.ullTotalPhys
+    else:
+        raise OSError("Unsupported operating system")

 def get_home_folder():
    home_folder = os.path.join(Path.home(), ".crawl4ai")
@@ -87,7 +131,7 @@ def split_and_parse_json_objects(json_string):
    return parsed_objects, unparsed_segments

 def sanitize_html(html):
-    # Replace all weird and special characters with an empty string
+    # Replace all unwanted and special characters with an empty string
    sanitized_html = html
    # sanitized_html = re.sub(r'[^\w\s.,;:!?=\[\]{}()<>\/\\\-"]', '', html)

@@ -96,6 +140,16 @@ def sanitize_html(html):

    return sanitized_html

+def sanitize_input_encode(text: str) -> str:
+    """Sanitize input to handle potential encoding issues."""
+    try:
+        # Attempt to encode and decode as UTF-8 to handle potential encoding issues
+        return text.encode('utf-8', errors='ignore').decode('utf-8')
+    except UnicodeEncodeError as e:
+        print(f"Warning: Encoding issue detected. Some characters may be lost. Error: {e}")
+        # Fall back to ASCII if UTF-8 fails
+        return text.encode('ascii', errors='ignore').decode('ascii')
+
 def escape_json_string(s):
    """
    Escapes characters in a string to be JSON safe.
@@ -247,7 +301,7 @@ def get_content_of_website(url, html, word_count_threshold = MIN_WORD_THRESHOLD,
            if tag.name != 'img':
                tag.attrs = {}

-        # Extract all img tgas inti [{src: '', alt: ''}]
+        # Extract all img tgas int0 [{src: '', alt: ''}]
        media = {
            'images': [],
            'videos': [],
@@ -285,7 +339,7 @@ def get_content_of_website(url, html, word_count_threshold = MIN_WORD_THRESHOLD,
                img.decompose()


-        # Create a function that replace content of all"pre" tage with its inner text
+        # Create a function that replace content of all"pre" tag with its inner text
        def replace_pre_tags_with_text(node):
            for child in node.find_all('pre'):
                # set child inner html to its text
@@ -419,14 +473,19 @@ def get_content_of_website(url, html, word_count_threshold = MIN_WORD_THRESHOLD,
        print('Error processing HTML content:', str(e))
        raise InvalidCSSSelectorError(f"Invalid CSS selector: {css_selector}") from e

-
 def get_content_of_website_optimized(url: str, html: str, word_count_threshold: int = MIN_WORD_THRESHOLD, css_selector: str = None, **kwargs) -> Dict[str, Any]:
    if not html:
        return None

    soup = BeautifulSoup(html, 'html.parser')
    body = soup.body
+    
+    image_description_min_word_threshold = kwargs.get('image_description_min_word_threshold', IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD)

+    for tag in kwargs.get('excluded_tags', []) or []:
+        for el in body.select(tag):
+            el.decompose()
+        
    if css_selector:
        selected_elements = body.select(css_selector)
        if not selected_elements:
@@ -438,67 +497,185 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
    links = {'internal': [], 'external': []}
    media = {'images': [], 'videos': [], 'audios': []}

-    def process_element(element: element.PageElement) -> None:
-        if isinstance(element, NavigableString):
-            if isinstance(element, Comment):
-                element.extract()
-            return
+    # Extract meaningful text for media files from closest parent
+    def find_closest_parent_with_useful_text(tag):
+            current_tag = tag
+            while current_tag:
+                current_tag = current_tag.parent
+                # Get the text content from the parent tag
+                if current_tag:
+                    text_content = current_tag.get_text(separator=' ',strip=True)
+                    # Check if the text content has at least word_count_threshold
+                    if len(text_content.split()) >= image_description_min_word_threshold:
+                        return text_content
+            return None

-        # if not isinstance(element, element.Tag):
-        #     return
+    def process_image(img, url, index, total_images):
+        #Check if an image has valid display and inside undesired html elements
+        def is_valid_image(img, parent, parent_classes):
+            style = img.get('style', '')
+            src = img.get('src', '')
+            classes_to_check = ['button', 'icon', 'logo']
+            tags_to_check = ['button', 'input']
+            return all([
+                'display:none' not in style,
+                src,
+                not any(s in var for var in [src, img.get('alt', ''), *parent_classes] for s in classes_to_check),
+                parent.name not in tags_to_check
+            ])

-        if element.name in ['script', 'style', 'link', 'meta', 'noscript']:
-            element.decompose()
-            return
+        #Score an image for it's usefulness
+        def score_image_for_usefulness(img, base_url, index, images_count):
+            # Function to parse image height/width value and units
+            def parse_dimension(dimension):
+                if dimension:
+                    match = re.match(r"(\d+)(\D*)", dimension)
+                    if match:
+                        number = int(match.group(1))
+                        unit = match.group(2) or 'px'  # Default unit is 'px' if not specified
+                        return number, unit
+                return None, None

-        if element.name == 'a' and element.get('href'):
-            href = element['href']
-            url_base = url.split('/')[2]
-            link_data = {'href': href, 'text': element.get_text()}
-            if href.startswith('http') and url_base not in href:
-                links['external'].append(link_data)
-            else:
-                links['internal'].append(link_data)
+            # Fetch image file metadata to extract size and extension
+            def fetch_image_file_size(img, base_url):
+                #If src is relative path construct full URL, if not it may be CDN URL
+                img_url = urljoin(base_url,img.get('src'))
+                try:
+                    response = requests.head(img_url)
+                    if response.status_code == 200:
+                        return response.headers.get('Content-Length',None)
+                    else:
+                        print(f"Failed to retrieve file size for {img_url}")
+                        return None
+                except InvalidSchema as e:
+                    return None
+                finally:
+                    return

-        elif element.name == 'img':
-            media['images'].append({
-                'src': element.get('src'),
-                'alt': element.get('alt'),
-                'type': 'image'
-            })
-            alt_text = element.get('alt')
-            if alt_text:
-                element.replace_with(soup.new_string(alt_text))
-            else:
+            image_height = img.get('height')
+            height_value, height_unit = parse_dimension(image_height)
+            image_width =  img.get('width')
+            width_value, width_unit = parse_dimension(image_width)
+            image_size = 0 #int(fetch_image_file_size(img,base_url) or 0)
+            image_format = os.path.splitext(img.get('src',''))[1].lower()
+            # Remove . from format
+            image_format = image_format.strip('.')
+            score = 0
+            if height_value:
+                if height_unit == 'px' and height_value > 150:
+                    score += 1
+                if height_unit in ['%','vh','vmin','vmax'] and height_value >30:
+                    score += 1
+            if width_value:
+                if width_unit == 'px' and width_value > 150:
+                    score += 1
+                if width_unit in ['%','vh','vmin','vmax'] and width_value >30:
+                    score += 1
+            if image_size > 10000:
+                score += 1
+            if img.get('alt') != '':
+                score+=1
+            if any(image_format==format for format in ['jpg','png','webp']):
+                score+=1
+            if index/images_count<0.5:
+                score+=1
+            return score
+
+        if not is_valid_image(img, img.parent, img.parent.get('class', [])):
+            return None
+        score = score_image_for_usefulness(img, url, index, total_images)
+        if score <= IMAGE_SCORE_THRESHOLD:
+            return None
+        return {
+            'src': img.get('src', '').replace('\\"', '"').strip(),
+            'alt': img.get('alt', ''),
+            'desc': find_closest_parent_with_useful_text(img),
+            'score': score,
+            'type': 'image'
+        }
+
+    def process_element(element: element.PageElement) -> bool:
+        try:
+            if isinstance(element, NavigableString):
+                if isinstance(element, Comment):
+                    element.extract()
+                return False
+
+            if element.name in ['script', 'style', 'link', 'meta', 'noscript']:
                element.decompose()
-            return
+                return False

-        elif element.name in ['video', 'audio']:
-            media[f"{element.name}s"].append({
-                'src': element.get('src'),
-                'alt': element.get('alt'),
-                'type': element.name
-            })
+            keep_element = False

-        if element.name != 'pre':
-            if element.name in ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']:
-                if kwargs.get('only_text', False):
-                    element.replace_with(element.get_text())
+            if element.name == 'a' and element.get('href'):
+                href = element['href']
+                url_base = url.split('/')[2]
+                link_data = {'href': href, 'text': element.get_text()}
+                if href.startswith('http') and url_base not in href:
+                    links['external'].append(link_data)
                else:
-                    element.unwrap()
-            elif element.name != 'img':
-                element.attrs = {}
+                    links['internal'].append(link_data)
+                keep_element = True

-        word_count = len(element.get_text(strip=True).split())
-        if word_count < word_count_threshold:
-            element.decompose()
-            return
+            elif element.name == 'img':
+                return True  # Always keep image elements

-        for child in list(element.children):
-            process_element(child)
+            elif element.name in ['video', 'audio']:
+                media[f"{element.name}s"].append({
+                    'src': element.get('src'),
+                    'alt': element.get('alt'),
+                    'type': element.name,
+                    'description': find_closest_parent_with_useful_text(element)
+                })
+                source_tags = element.find_all('source')
+                for source_tag in source_tags:
+                    media[f"{element.name}s"].append({
+                    'src': source_tag.get('src'),
+                    'alt': element.get('alt'),
+                    'type': element.name,
+                    'description': find_closest_parent_with_useful_text(element)
+                })
+                return True  # Always keep video and audio elements

-        if not element.contents and not element.get_text(strip=True):
-            element.decompose()
+            if element.name != 'pre':
+                if element.name in ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']:
+                    if kwargs.get('only_text', False):
+                        element.replace_with(element.get_text())
+                    else:
+                        element.unwrap()
+                elif element.name != 'img':
+                    element.attrs = {}
+
+            # Process children
+            for child in list(element.children):
+                if isinstance(child, NavigableString) and not isinstance(child, Comment):
+                    if len(child.strip()) > 0:
+                        keep_element = True
+                else:
+                    if process_element(child):
+                        keep_element = True
+                
+
+            # Check word count
+            if not keep_element:
+                word_count = len(element.get_text(strip=True).split())
+                keep_element = word_count >= word_count_threshold
+
+            if not keep_element:
+                element.decompose()
+
+            return keep_element
+        except Exception as e:
+            print('Error processing element:', str(e))
+            return False
+
+    #process images by filtering and extracting contextual text from the page
+    imgs = body.find_all('img')
+    media['images'] = [
+        result for result in
+        (process_image(img, url, i, len(imgs)) for i, img in enumerate(imgs))
+        if result is not None
+    ]

    process_element(body)

@@ -511,7 +688,12 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
        return node

    body = flatten_nested_elements(body)
-
+    base64_pattern = re.compile(r'data:image/[^;]+;base64,([^"]+)')
+    for img in imgs:
+        src = img.get('src', '')
+        if base64_pattern.match(src):
+            # Replace base64 data with empty string
+            img['src'] = base64_pattern.sub('', src)
    cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
    cleaned_html = sanitize_html(cleaned_html)

@@ -535,7 +717,6 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
        'metadata': meta
    }

-
 def extract_metadata(html, soup = None):
    metadata = {}
    
@@ -594,12 +775,26 @@ def extract_xml_data(tags, string):
    return data
    
 # Function to perform the completion with exponential backoff
-def perform_completion_with_backoff(provider, prompt_with_variables, api_token):
+def perform_completion_with_backoff(
+    provider, 
+    prompt_with_variables, 
+    api_token, 
+    json_response = False, 
+    base_url=None,
+    **kwargs
+    ):
    from litellm import completion 
    from litellm.exceptions import RateLimitError
    max_attempts = 3
    base_delay = 2  # Base delay in seconds, you can adjust this based on your needs
    
+    extra_args = {}
+    if json_response:
+        extra_args["response_format"] = { "type": "json_object" }
+        
+    if kwargs.get("extra_args"):
+        extra_args.update(kwargs["extra_args"])
+    
    for attempt in range(max_attempts):
        try:
            response =completion(
@@ -608,7 +803,9 @@ def perform_completion_with_backoff(provider, prompt_with_variables, api_token):
                    {"role": "user", "content": prompt_with_variables}
                ],
                temperature=0.01,
-                api_key=api_token
+                api_key=api_token,
+                base_url=base_url,
+                **extra_args
            )
            return response  # Return the successful response
        except RateLimitError as e:
@@ -628,7 +825,7 @@ def perform_completion_with_backoff(provider, prompt_with_variables, api_token):
                    "content": ["Rate limit error. Please try again later."]
                }]
    
-def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None):
+def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None, base_url = None):
    # api_token = os.getenv('GROQ_API_KEY', None) if not api_token else api_token
    api_token = PROVIDER_MODELS.get(provider, None) if not api_token else api_token
    
@@ -643,7 +840,7 @@ def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None):
            "{" + variable + "}", variable_values[variable]
        )
        
-    response = perform_completion_with_backoff(provider, prompt_with_variables, api_token)
+    response = perform_completion_with_backoff(provider, prompt_with_variables, api_token, base_url=base_url)
        
    try:
        blocks = extract_xml_data(["blocks"], response.choices[0].message.content)['blocks']
@@ -652,7 +849,6 @@ def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None):
        for block in blocks:
            block['error'] = False
    except Exception as e:
-        print("Error extracting blocks:", str(e))
        parsed, unparsed = split_and_parse_json_objects(response.choices[0].message.content)
        blocks = parsed
        # Append all unparsed segments as onr error block and content is list of unparsed segments
@@ -698,7 +894,6 @@ def extract_blocks_batch(batch_data, provider = "groq/llama3-70b-8192", api_toke
            blocks = json.loads(blocks)

        except Exception as e:
-            print("Error extracting blocks:", str(e))
            blocks = [{
                "index": 0,
                "tags": ["error"],
@@ -709,7 +904,6 @@ def extract_blocks_batch(batch_data, provider = "groq/llama3-70b-8192", api_toke
    
    return sum(all_blocks, [])

-
 def merge_chunks_based_on_token_threshold(chunks, token_threshold):
    """
    Merges small chunks into larger ones based on the total token threshold.
@@ -739,23 +933,22 @@ def merge_chunks_based_on_token_threshold(chunks, token_threshold):

    return merged_sections

-def process_sections(url: str, sections: list, provider: str, api_token: str) -> list:
+def process_sections(url: str, sections: list, provider: str, api_token: str, base_url=None) -> list:
    extracted_content = []
    if provider.startswith("groq/"):
        # Sequential processing with a delay
        for section in sections:
-            extracted_content.extend(extract_blocks(url, section, provider, api_token))
+            extracted_content.extend(extract_blocks(url, section, provider, api_token, base_url=base_url))
            time.sleep(0.5)  # 500 ms delay between each processing
    else:
        # Parallel processing using ThreadPoolExecutor
        with ThreadPoolExecutor() as executor:
-            futures = [executor.submit(extract_blocks, url, section, provider, api_token) for section in sections]
+            futures = [executor.submit(extract_blocks, url, section, provider, api_token, base_url=base_url) for section in sections]
            for future in as_completed(futures):
                extracted_content.extend(future.result())
    
    return extracted_content

-
 def wrap_text(draw, text, font, max_width):
    # Wrap the text to fit within the specified width
    lines = []
@@ -767,7 +960,8 @@ def wrap_text(draw, text, font, max_width):
        lines.append(line)
    return '\n'.join(lines)

-
 def format_html(html_string):
    soup = BeautifulSoup(html_string, 'html.parser')
-    return soup.prettify()
+    return soup.prettify()
+
+
--- a/crawl4ai/web_crawler.py
+++ b/crawl4ai/web_crawler.py
@@ -11,43 +11,29 @@ from .crawler_strategy import *
 from typing import List
 from concurrent.futures import ThreadPoolExecutor
 from .config import *
+import warnings
+import json
+warnings.filterwarnings("ignore", message='Field "model_name" has conflict with protected namespace "model_".')


 class WebCrawler:
-    def __init__(
-        self,
-        # db_path: str = None,
-        crawler_strategy: CrawlerStrategy = None,
-        always_by_pass_cache: bool = False,
-        verbose: bool = False,
-    ):
-        # self.db_path = db_path
+    def __init__(self, crawler_strategy: CrawlerStrategy = None, always_by_pass_cache: bool = False, verbose: bool = False):
        self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose)
        self.always_by_pass_cache = always_by_pass_cache
-
-        # Create the .crawl4ai folder in the user's home directory if it doesn't exist
        self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
        os.makedirs(self.crawl4ai_folder, exist_ok=True)
        os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
-
-        # If db_path is not provided, use the default path
-        # if not db_path:
-            # self.db_path = f"{self.crawl4ai_folder}/crawl4ai.db"
-        
-        # flush_db()
        init_db()
-        
        self.ready = False
        
    def warmup(self):
        print("[LOG] 🌤️  Warming up the WebCrawler")
-        result = self.run(
+        self.run(
            url='https://google.com/',
            word_count_threshold=5,
-            extraction_strategy= NoExtractionStrategy(),
+            extraction_strategy=NoExtractionStrategy(),
            bypass_cache=False,
-            verbose = False,
-            warmup=True
+            verbose=False
        )
        self.ready = True
        print("[LOG] 🌞 WebCrawler is ready to crawl")
@@ -129,43 +115,53 @@ class WebCrawler:
            verbose=True,
            **kwargs,
        ) -> CrawlResult:
-            extraction_strategy = extraction_strategy or NoExtractionStrategy()
-            extraction_strategy.verbose = verbose
-            if not isinstance(extraction_strategy, ExtractionStrategy):
-                raise ValueError("Unsupported extraction strategy")
-            if not isinstance(chunking_strategy, ChunkingStrategy):
-                raise ValueError("Unsupported chunking strategy")
-            
-            if word_count_threshold < MIN_WORD_THRESHOLD:
-                word_count_threshold = MIN_WORD_THRESHOLD
+            try:
+                extraction_strategy = extraction_strategy or NoExtractionStrategy()
+                extraction_strategy.verbose = verbose
+                if not isinstance(extraction_strategy, ExtractionStrategy):
+                    raise ValueError("Unsupported extraction strategy")
+                if not isinstance(chunking_strategy, ChunkingStrategy):
+                    raise ValueError("Unsupported chunking strategy")
+                
+                word_count_threshold = max(word_count_threshold, MIN_WORD_THRESHOLD)

-            # Check cache first
-            cached = None
-            screenshot_data = None
-            extracted_content = None
-            if not bypass_cache and not self.always_by_pass_cache:
-                cached = get_cached_url(url)
-            
-            if kwargs.get("warmup", True) and not self.ready:
-                return None
-            
-            if cached:
-                html = cached[1]
-                extracted_content = cached[4]
-                if screenshot:
-                    screenshot_data = cached[9]
-                    if not screenshot_data:
-                        cached = None
-            
-            if not cached or not html:
-                if user_agent:
-                    self.crawler_strategy.update_user_agent(user_agent)
-                html = self.crawler_strategy.crawl(url)
-                if screenshot:
-                    screenshot_data = self.crawler_strategy.take_screenshot()
+                cached = None
+                screenshot_data = None
+                extracted_content = None
+                if not bypass_cache and not self.always_by_pass_cache:
+                    cached = get_cached_url(url)
+                
+                if kwargs.get("warmup", True) and not self.ready:
+                    return None
+                
+                if cached:
+                    html = sanitize_input_encode(cached[1])
+                    extracted_content = sanitize_input_encode(cached[4])
+                    if screenshot:
+                        screenshot_data = cached[9]
+                        if not screenshot_data:
+                            cached = None
+                
+                if not cached or not html:
+                    if user_agent:
+                        self.crawler_strategy.update_user_agent(user_agent)
+                    t1 = time.time()
+                    html = sanitize_input_encode(self.crawler_strategy.crawl(url, **kwargs))
+                    t2 = time.time()
+                    if verbose:
+                        print(f"[LOG] 🚀 Crawling done for {url}, success: {bool(html)}, time taken: {t2 - t1:.2f} seconds")
+                    if screenshot:
+                        screenshot_data = self.crawler_strategy.take_screenshot()

-            
-            return self.process_html(url, html, extracted_content, word_count_threshold, extraction_strategy, chunking_strategy, css_selector, screenshot_data, verbose, bool(cached), **kwargs)
+                
+                crawl_result = self.process_html(url, html, extracted_content, word_count_threshold, extraction_strategy, chunking_strategy, css_selector, screenshot_data, verbose, bool(cached), **kwargs)
+                crawl_result.success = bool(html)
+                return crawl_result
+            except Exception as e:
+                if not hasattr(e, "msg"):
+                    e.msg = str(e)
+                print(f"[ERROR] 🚫 Failed to crawl {url}, error: {e.msg}")    
+                return CrawlResult(url=url, html="", success=False, error_message=e.msg)

    def process_html(
            self,
@@ -184,26 +180,21 @@ class WebCrawler:
            t = time.time()
            # Extract content from HTML
            try:
-                # t1 = time.time()
-                # result = get_content_of_website(url, html, word_count_threshold, css_selector=css_selector, only_text=kwargs.get("only_text", False))
-                # print(f"[LOG] 🚀 Crawling done for {url}, success: True, time taken: {time.time() - t1} seconds")
                t1 = time.time()
                result = get_content_of_website_optimized(url, html, word_count_threshold, css_selector=css_selector, only_text=kwargs.get("only_text", False))
-                print(f"[LOG] 🚀 Crawling done for {url}, success: True, time taken: {time.time() - t1} seconds")
+                if verbose:
+                    print(f"[LOG] 🚀 Content extracted for {url}, success: True, time taken: {time.time() - t1:.2f} seconds")
                
                if result is None:
                    raise ValueError(f"Failed to extract content from the website: {url}")
            except InvalidCSSSelectorError as e:
                raise ValueError(str(e))
            
-            cleaned_html = result.get("cleaned_html", "")
-            markdown = result.get("markdown", "")
+            cleaned_html = sanitize_input_encode(result.get("cleaned_html", ""))
+            markdown = sanitize_input_encode(result.get("markdown", ""))
            media = result.get("media", [])
            links = result.get("links", [])
            metadata = result.get("metadata", {})
-
-            if verbose:
-                print(f"[LOG] 🚀 Crawling done for {url}, success: True, time taken: {time.time() - t} seconds")
                        
            if extracted_content is None:
                if verbose:
@@ -211,10 +202,10 @@ class WebCrawler:

                sections = chunking_strategy.chunk(markdown)
                extracted_content = extraction_strategy.run(url, sections)
-                extracted_content = json.dumps(extracted_content, indent=4, default=str)
+                extracted_content = json.dumps(extracted_content, indent=4, default=str, ensure_ascii=False)

                if verbose:
-                    print(f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t} seconds.")
+                    print(f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t:.2f} seconds.")
                
            screenshot = None if not screenshot else screenshot
            
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,10 +0,0 @@
-version: '3.8'
-
-services:
-  web:
-    build: .
-    command: uvicorn main:app --host 0.0.0.0 --port 80 --workers $(nproc)
-    ports:
-      - "80:80"
-    environment:
-      - PYTHONUNBUFFERED=1
--- a/docs/.DS_Store
+++ b/docs/.DS_Store
--- a/docs/examples/async_webcrawler_multiple_urls_example.py
+++ b/docs/examples/async_webcrawler_multiple_urls_example.py
@@ -0,0 +1,48 @@
+# File: async_webcrawler_multiple_urls_example.py
+import os, sys
+# append 2 parent directories to sys.path to import crawl4ai
+parent_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+sys.path.append(parent_dir)
+
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    # Initialize the AsyncWebCrawler
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # List of URLs to crawl
+        urls = [
+            "https://example.com",
+            "https://python.org",
+            "https://github.com",
+            "https://stackoverflow.com",
+            "https://news.ycombinator.com"
+        ]
+
+        # Set up crawling parameters
+        word_count_threshold = 100
+
+        # Run the crawling process for multiple URLs
+        results = await crawler.arun_many(
+            urls=urls,
+            word_count_threshold=word_count_threshold,
+            bypass_cache=True,
+            verbose=True
+        )
+
+        # Process the results
+        for result in results:
+            if result.success:
+                print(f"Successfully crawled: {result.url}")
+                print(f"Title: {result.metadata.get('title', 'N/A')}")
+                print(f"Word count: {len(result.markdown.split())}")
+                print(f"Number of links: {len(result.links.get('internal', [])) + len(result.links.get('external', []))}")
+                print(f"Number of images: {len(result.media.get('images', []))}")
+                print("---")
+            else:
+                print(f"Failed to crawl: {result.url}")
+                print(f"Error: {result.error_message}")
+                print("---")
+
+if __name__ == "__main__":
+    asyncio.run(main())
--- a/docs/examples/crawlai_vs_firecrawl.py
+++ b/docs/examples/crawlai_vs_firecrawl.py
@@ -0,0 +1,67 @@
+import os, time
+# append the path to the root of the project
+import sys
+import asyncio
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..'))
+from firecrawl import FirecrawlApp
+from crawl4ai import AsyncWebCrawler
+__data__ = os.path.join(os.path.dirname(__file__), '..', '..') + '/.data'
+
+async def compare():
+    app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
+
+    # Tet Firecrawl with a simple crawl
+    start = time.time()
+    scrape_status = app.scrape_url(
+    'https://www.nbcnews.com/business',
+    params={'formats': ['markdown', 'html']}
+    )
+    end = time.time()
+    print(f"Time taken: {end - start} seconds")
+    print(len(scrape_status['markdown']))
+    # save the markdown content with provider name
+    with open(f"{__data__}/firecrawl_simple.md", "w") as f:
+        f.write(scrape_status['markdown'])
+    # Count how many "cldnry.s-nbcnews.com" are in the markdown
+    print(scrape_status['markdown'].count("cldnry.s-nbcnews.com"))
+    
+
+
+    async with AsyncWebCrawler() as crawler:
+        start = time.time()
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            # js_code=["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"],
+            word_count_threshold=0,
+            bypass_cache=True, 
+            verbose=False
+        )
+        end = time.time()
+        print(f"Time taken: {end - start} seconds")
+        print(len(result.markdown))
+        # save the markdown content with provider name  
+        with open(f"{__data__}/crawl4ai_simple.md", "w") as f:
+            f.write(result.markdown)
+        # count how many "cldnry.s-nbcnews.com" are in the markdown
+        print(result.markdown.count("cldnry.s-nbcnews.com"))
+
+        start = time.time()
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"],
+            word_count_threshold=0,
+            bypass_cache=True, 
+            verbose=False
+        )
+        end = time.time()
+        print(f"Time taken: {end - start} seconds")
+        print(len(result.markdown))
+        # save the markdown content with provider name
+        with open(f"{__data__}/crawl4ai_js.md", "w") as f:
+            f.write(result.markdown)
+        # count how many "cldnry.s-nbcnews.com" are in the markdown
+        print(result.markdown.count("cldnry.s-nbcnews.com"))
+        
+if __name__ == "__main__":
+    asyncio.run(compare())
+    
--- a/docs/examples/language_support_example.py
+++ b/docs/examples/language_support_example.py
@@ -0,0 +1,45 @@
+import asyncio
+from crawl4ai import AsyncWebCrawler, AsyncPlaywrightCrawlerStrategy
+
+async def main():
+    # Example 1: Setting language when creating the crawler
+    crawler1 = AsyncWebCrawler(
+        crawler_strategy=AsyncPlaywrightCrawlerStrategy(
+            headers={"Accept-Language": "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"}
+        )
+    )
+    result1 = await crawler1.arun("https://www.example.com")
+    print("Example 1 result:", result1.extracted_content[:100])  # Print first 100 characters
+
+    # Example 2: Setting language before crawling
+    crawler2 = AsyncWebCrawler()
+    crawler2.crawler_strategy.headers["Accept-Language"] = "es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7"
+    result2 = await crawler2.arun("https://www.example.com")
+    print("Example 2 result:", result2.extracted_content[:100])
+
+    # Example 3: Setting language when calling arun method
+    crawler3 = AsyncWebCrawler()
+    result3 = await crawler3.arun(
+        "https://www.example.com",
+        headers={"Accept-Language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}
+    )
+    print("Example 3 result:", result3.extracted_content[:100])
+
+    # Example 4: Crawling multiple pages with different languages
+    urls = [
+        ("https://www.example.com", "fr-FR,fr;q=0.9"),
+        ("https://www.example.org", "es-ES,es;q=0.9"),
+        ("https://www.example.net", "de-DE,de;q=0.9"),
+    ]
+    
+    crawler4 = AsyncWebCrawler()
+    results = await asyncio.gather(*[
+        crawler4.arun(url, headers={"Accept-Language": lang})
+        for url, lang in urls
+    ])
+    
+    for url, result in zip([u for u, _ in urls], results):
+        print(f"Result for {url}:", result.extracted_content[:100])
+
+if __name__ == "__main__":
+    asyncio.run(main())
--- a/docs/examples/llm_extraction_openai_pricing.py
+++ b/docs/examples/llm_extraction_openai_pricing.py
@@ -21,7 +21,8 @@ result = crawler.run(
    url=url,
    word_count_threshold=1,
    extraction_strategy= LLMExtractionStrategy(
-        provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
+        # provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
+        provider= "groq/llama-3.1-70b-versatile", api_token = os.getenv('GROQ_API_KEY'), 
        schema=OpenAIModelFee.model_json_schema(),
        extraction_type="schema",
        instruction="From the crawled content, extract all mentioned model names along with their "\
@@ -36,5 +37,5 @@ model_fees = json.loads(result.extracted_content)

 print(len(model_fees))

-with open(".data/data.json", "w") as f:
+with open(".data/data.json", "w", encoding="utf-8") as f:
    f.write(result.extracted_content)
--- a/docs/examples/quickstart.ipynb
+++ b/docs/examples/quickstart.ipynb
@@ -0,0 +1,736 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6yLvrXn7yZQI"
+      },
+      "source": [
+        "# Crawl4AI: Advanced Web Crawling and Data Extraction\n",
+        "\n",
+        "Welcome to this interactive notebook showcasing Crawl4AI, an advanced asynchronous web crawling and data extraction library.\n",
+        "\n",
+        "- GitHub Repository: [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
+        "- Twitter: [@unclecode](https://twitter.com/unclecode)\n",
+        "- Website: [https://crawl4ai.com](https://crawl4ai.com)\n",
+        "\n",
+        "Let's explore the powerful features of Crawl4AI!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KIn_9nxFyZQK"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "First, let's install Crawl4AI from GitHub:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mSnaxLf3zMog"
+      },
+      "outputs": [],
+      "source": [
+        "!sudo apt-get update && sudo apt-get install -y libwoff1 libopus0 libwebp6 libwebpdemux2 libenchant1c2a libgudev-1.0-0 libsecret-1-0 libhyphen0 libgdk-pixbuf2.0-0 libegl1 libnotify4 libxslt1.1 libevent-2.1-7 libgles2 libvpx6 libxcomposite1 libatk1.0-0 libatk-bridge2.0-0 libepoxy0 libgtk-3-0 libharfbuzz-icu0"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xlXqaRtayZQK"
+      },
+      "outputs": [],
+      "source": [
+        "# !pip install \"crawl4ai @ git+https://github.com/unclecode/crawl4ai.git\"\n",
+        "!pip install \"crawl4ai @ git+https://github.com/unclecode/crawl4ai.git@staging\"\n",
+        "!pip install nest-asyncio\n",
+        "!playwright install"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qKCE7TI7yZQL"
+      },
+      "source": [
+        "Now, let's import the necessary libraries:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "I67tr7aAyZQL"
+      },
+      "outputs": [],
+      "source": [
+        "import asyncio\n",
+        "import nest_asyncio\n",
+        "from crawl4ai import AsyncWebCrawler\n",
+        "from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy\n",
+        "import json\n",
+        "import time\n",
+        "from pydantic import BaseModel, Field\n",
+        "\n",
+        "nest_asyncio.apply()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "h7yR_Rt_yZQM"
+      },
+      "source": [
+        "## Basic Usage\n",
+        "\n",
+        "Let's start with a simple crawl example:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "yBh6hf4WyZQM",
+        "outputId": "0f83af5c-abba-4175-ed95-70b7512e6bcc"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.05 seconds\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.05 seconds.\n",
+            "18102\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def simple_crawl():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(url=\"https://www.nbcnews.com/business\")\n",
+        "        print(len(result.markdown))\n",
+        "await simple_crawl()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9rtkgHI28uI4"
+      },
+      "source": [
+        "💡 By default, **Crawl4AI** caches the result of every URL, so the next time you call it, you’ll get an instant result. But if you want to bypass the cache, just set `bypass_cache=True`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MzZ0zlJ9yZQM"
+      },
+      "source": [
+        "## Advanced Features\n",
+        "\n",
+        "### Executing JavaScript and Using CSS Selectors"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "gHStF86xyZQM",
+        "outputId": "34d0fb6d-4dec-4677-f76e-85a1f082829b"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
+            "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 6.06 seconds\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.10 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.11 seconds.\n",
+            "41135\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def js_and_css():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        js_code = [\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"]\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            js_code=js_code,\n",
+        "            # css_selector=\"YOUR_CSS_SELECTOR_HERE\",\n",
+        "            bypass_cache=True\n",
+        "        )\n",
+        "        print(len(result.markdown))\n",
+        "\n",
+        "await js_and_css()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cqE_W4coyZQM"
+      },
+      "source": [
+        "### Using a Proxy\n",
+        "\n",
+        "Note: You'll need to replace the proxy URL with a working proxy for this example to run successfully."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QjAyiAGqyZQM"
+      },
+      "outputs": [],
+      "source": [
+        "async def use_proxy():\n",
+        "    async with AsyncWebCrawler(verbose=True, proxy=\"http://your-proxy-url:port\") as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            bypass_cache=True\n",
+        "        )\n",
+        "        print(result.markdown[:500])  # Print first 500 characters\n",
+        "\n",
+        "# Uncomment the following line to run the proxy example\n",
+        "# await use_proxy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XTZ88lbayZQN"
+      },
+      "source": [
+        "### Extracting Structured Data with OpenAI\n",
+        "\n",
+        "Note: You'll need to set your OpenAI API key as an environment variable for this example to work."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 14,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "fIOlDayYyZQN",
+        "outputId": "cb8359cc-dee0-4762-9698-5dfdcee055b8"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://openai.com/api/pricing/ using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://openai.com/api/pricing/ successfully!\n",
+            "[LOG] 🚀 Crawling done for https://openai.com/api/pricing/, success: True, time taken: 3.77 seconds\n",
+            "[LOG] 🚀 Content extracted for https://openai.com/api/pricing/, success: True, time taken: 0.21 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://openai.com/api/pricing/, Strategy: AsyncWebCrawler\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 0\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 1\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 2\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 3\n",
+            "[LOG] Extracted 4 blocks from URL: https://openai.com/api/pricing/ block index: 3\n",
+            "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 4\n",
+            "[LOG] Extracted 5 blocks from URL: https://openai.com/api/pricing/ block index: 0\n",
+            "[LOG] Extracted 1 blocks from URL: https://openai.com/api/pricing/ block index: 4\n",
+            "[LOG] Extracted 8 blocks from URL: https://openai.com/api/pricing/ block index: 1\n",
+            "[LOG] Extracted 12 blocks from URL: https://openai.com/api/pricing/ block index: 2\n",
+            "[LOG] 🚀 Extraction done for https://openai.com/api/pricing/, time taken: 8.55 seconds.\n",
+            "5029\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "from google.colab import userdata\n",
+        "os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')\n",
+        "\n",
+        "class OpenAIModelFee(BaseModel):\n",
+        "    model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n",
+        "    input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n",
+        "    output_fee: str = Field(..., description=\"Fee for output token for the OpenAI model.\")\n",
+        "\n",
+        "async def extract_openai_fees():\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url='https://openai.com/api/pricing/',\n",
+        "            word_count_threshold=1,\n",
+        "            extraction_strategy=LLMExtractionStrategy(\n",
+        "                provider=\"openai/gpt-4o\", api_token=os.getenv('OPENAI_API_KEY'),\n",
+        "                schema=OpenAIModelFee.schema(),\n",
+        "                extraction_type=\"schema\",\n",
+        "                instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens.\n",
+        "                Do not miss any models in the entire content. One extracted model JSON format should look like this:\n",
+        "                {\"model_name\": \"GPT-4\", \"input_fee\": \"US$10.00 / 1M tokens\", \"output_fee\": \"US$30.00 / 1M tokens\"}.\"\"\"\n",
+        "            ),\n",
+        "            bypass_cache=True,\n",
+        "        )\n",
+        "        print(len(result.extracted_content))\n",
+        "\n",
+        "# Uncomment the following line to run the OpenAI extraction example\n",
+        "await extract_openai_fees()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BypA5YxEyZQN"
+      },
+      "source": [
+        "### Advanced Multi-Page Crawling with JavaScript Execution"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tfkcVQ0b7mw-"
+      },
+      "source": [
+        "## Advanced Multi-Page Crawling with JavaScript Execution\n",
+        "\n",
+        "This example demonstrates Crawl4AI's ability to handle complex crawling scenarios, specifically extracting commits from multiple pages of a GitHub repository. The challenge here is that clicking the \"Next\" button doesn't load a new page, but instead uses asynchronous JavaScript to update the content. This is a common hurdle in modern web crawling.\n",
+        "\n",
+        "To overcome this, we use Crawl4AI's custom JavaScript execution to simulate clicking the \"Next\" button, and implement a custom hook to detect when new data has loaded. Our strategy involves comparing the first commit's text before and after \"clicking\" Next, waiting until it changes to confirm new data has rendered. This showcases Crawl4AI's flexibility in handling dynamic content and its ability to implement custom logic for even the most challenging crawling tasks."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "qUBKGpn3yZQN",
+        "outputId": "3e555b6a-ed33-42f4-cce9-499a923fbe17"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 5.16 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.28 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.28 seconds.\n",
+            "Page 1: Found 35 commits\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.78 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.90 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.90 seconds.\n",
+            "Page 2: Found 35 commits\n",
+            "[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
+            "[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 2.00 seconds\n",
+            "[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.74 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.75 seconds.\n",
+            "Page 3: Found 35 commits\n",
+            "Successfully crawled 105 commits across 3 pages\n"
+          ]
+        }
+      ],
+      "source": [
+        "import re\n",
+        "from bs4 import BeautifulSoup\n",
+        "\n",
+        "async def crawl_typescript_commits():\n",
+        "    first_commit = \"\"\n",
+        "    async def on_execution_started(page):\n",
+        "        nonlocal first_commit\n",
+        "        try:\n",
+        "            while True:\n",
+        "                await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4')\n",
+        "                commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4')\n",
+        "                commit = await commit.evaluate('(element) => element.textContent')\n",
+        "                commit = re.sub(r'\\s+', '', commit)\n",
+        "                if commit and commit != first_commit:\n",
+        "                    first_commit = commit\n",
+        "                    break\n",
+        "                await asyncio.sleep(0.5)\n",
+        "        except Exception as e:\n",
+        "            print(f\"Warning: New content didn't appear after JavaScript execution: {e}\")\n",
+        "\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started)\n",
+        "\n",
+        "        url = \"https://github.com/microsoft/TypeScript/commits/main\"\n",
+        "        session_id = \"typescript_commits_session\"\n",
+        "        all_commits = []\n",
+        "\n",
+        "        js_next_page = \"\"\"\n",
+        "        const button = document.querySelector('a[data-testid=\"pagination-next-button\"]');\n",
+        "        if (button) button.click();\n",
+        "        \"\"\"\n",
+        "\n",
+        "        for page in range(3):  # Crawl 3 pages\n",
+        "            result = await crawler.arun(\n",
+        "                url=url,\n",
+        "                session_id=session_id,\n",
+        "                css_selector=\"li.Box-sc-g0xbh4-0\",\n",
+        "                js=js_next_page if page > 0 else None,\n",
+        "                bypass_cache=True,\n",
+        "                js_only=page > 0\n",
+        "            )\n",
+        "\n",
+        "            assert result.success, f\"Failed to crawl page {page + 1}\"\n",
+        "\n",
+        "            soup = BeautifulSoup(result.cleaned_html, 'html.parser')\n",
+        "            commits = soup.select(\"li\")\n",
+        "            all_commits.extend(commits)\n",
+        "\n",
+        "            print(f\"Page {page + 1}: Found {len(commits)} commits\")\n",
+        "\n",
+        "        await crawler.crawler_strategy.kill_session(session_id)\n",
+        "        print(f\"Successfully crawled {len(all_commits)} commits across 3 pages\")\n",
+        "\n",
+        "await crawl_typescript_commits()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EJRnYsp6yZQN"
+      },
+      "source": [
+        "### Using JsonCssExtractionStrategy for Fast Structured Output"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1ZMqIzB_8SYp"
+      },
+      "source": [
+        "The JsonCssExtractionStrategy is a powerful feature of Crawl4AI that allows for precise, structured data extraction from web pages. Here's how it works:\n",
+        "\n",
+        "1. You define a schema that describes the pattern of data you're interested in extracting.\n",
+        "2. The schema includes a base selector that identifies repeating elements on the page.\n",
+        "3. Within the schema, you define fields, each with its own selector and type.\n",
+        "4. These field selectors are applied within the context of each base selector element.\n",
+        "5. The strategy supports nested structures, lists within lists, and various data types.\n",
+        "6. You can even include computed fields for more complex data manipulation.\n",
+        "\n",
+        "This approach allows for highly flexible and precise data extraction, transforming semi-structured web content into clean, structured JSON data. It's particularly useful for extracting consistent data patterns from pages like product listings, news articles, or search results.\n",
+        "\n",
+        "For more details and advanced usage, check out the full documentation on the Crawl4AI website."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "trCMR2T9yZQN",
+        "outputId": "718d36f4-cccf-40f4-8d8c-c3ba73524d16"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
+            "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
+            "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
+            "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
+            "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 7.00 seconds\n",
+            "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.32 seconds\n",
+            "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
+            "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.48 seconds.\n",
+            "Successfully extracted 11 news teasers\n",
+            "{\n",
+            "  \"category\": \"Business News\",\n",
+            "  \"headline\": \"NBC ripped up its Olympics playbook for 2024 \\u2014 so far, the new strategy paid off\",\n",
+            "  \"summary\": \"The Olympics have long been key to NBCUniversal. Paris marked the 18th Olympic Games broadcast by NBC in the U.S.\",\n",
+            "  \"time\": \"13h ago\",\n",
+            "  \"image\": {\n",
+            "    \"src\": \"https://media-cldnry.s-nbcnews.com/image/upload/t_focal-200x100,f_auto,q_auto:best/rockcms/2024-09/240903-nbc-olympics-ch-1344-c7a486.jpg\",\n",
+            "    \"alt\": \"Mike Tirico.\"\n",
+            "  },\n",
+            "  \"link\": \"https://www.nbcnews.com/business\"\n",
+            "}\n"
+          ]
+        }
+      ],
+      "source": [
+        "async def extract_news_teasers():\n",
+        "    schema = {\n",
+        "        \"name\": \"News Teaser Extractor\",\n",
+        "        \"baseSelector\": \".wide-tease-item__wrapper\",\n",
+        "        \"fields\": [\n",
+        "            {\n",
+        "                \"name\": \"category\",\n",
+        "                \"selector\": \".unibrow span[data-testid='unibrow-text']\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"headline\",\n",
+        "                \"selector\": \".wide-tease-item__headline\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"summary\",\n",
+        "                \"selector\": \".wide-tease-item__description\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"time\",\n",
+        "                \"selector\": \"[data-testid='wide-tease-date']\",\n",
+        "                \"type\": \"text\",\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"image\",\n",
+        "                \"type\": \"nested\",\n",
+        "                \"selector\": \"picture.teasePicture img\",\n",
+        "                \"fields\": [\n",
+        "                    {\"name\": \"src\", \"type\": \"attribute\", \"attribute\": \"src\"},\n",
+        "                    {\"name\": \"alt\", \"type\": \"attribute\", \"attribute\": \"alt\"},\n",
+        "                ],\n",
+        "            },\n",
+        "            {\n",
+        "                \"name\": \"link\",\n",
+        "                \"selector\": \"a[href]\",\n",
+        "                \"type\": \"attribute\",\n",
+        "                \"attribute\": \"href\",\n",
+        "            },\n",
+        "        ],\n",
+        "    }\n",
+        "\n",
+        "    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n",
+        "\n",
+        "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            extraction_strategy=extraction_strategy,\n",
+        "            bypass_cache=True,\n",
+        "        )\n",
+        "\n",
+        "        assert result.success, \"Failed to crawl the page\"\n",
+        "\n",
+        "        news_teasers = json.loads(result.extracted_content)\n",
+        "        print(f\"Successfully extracted {len(news_teasers)} news teasers\")\n",
+        "        print(json.dumps(news_teasers[0], indent=2))\n",
+        "\n",
+        "await extract_news_teasers()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FnyVhJaByZQN"
+      },
+      "source": [
+        "## Speed Comparison\n",
+        "\n",
+        "Let's compare the speed of Crawl4AI with Firecrawl, a paid service. Note that we can't run Firecrawl in this Colab environment, so we'll simulate its performance based on previously recorded data."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "agDD186f3wig"
+      },
+      "source": [
+        "💡 **Note on Speed Comparison:**\n",
+        "\n",
+        "The speed test conducted here is running on Google Colab, where the internet speed and performance can vary and may not reflect optimal conditions. When we call Firecrawl's API, we're seeing its best performance, while Crawl4AI's performance is limited by Colab's network speed.\n",
+        "\n",
+        "For a more accurate comparison, it's recommended to run these tests on your own servers or computers with a stable and fast internet connection. Despite these limitations, Crawl4AI still demonstrates faster performance in this environment.\n",
+        "\n",
+        "If you run these tests locally, you may observe an even more significant speed advantage for Crawl4AI compared to other services."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "F7KwHv8G1LbY"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install firecrawl"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "91813zILyZQN",
+        "outputId": "663223db-ab89-4976-b233-05ceca62b19b"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Firecrawl (simulated):\n",
+            "Time taken: 4.38 seconds\n",
+            "Content length: 41967 characters\n",
+            "Images found: 49\n",
+            "\n",
+            "Crawl4AI (simple crawl):\n",
+            "Time taken: 4.22 seconds\n",
+            "Content length: 18221 characters\n",
+            "Images found: 49\n",
+            "\n",
+            "Crawl4AI (with JavaScript execution):\n",
+            "Time taken: 9.13 seconds\n",
+            "Content length: 34243 characters\n",
+            "Images found: 89\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "from google.colab import userdata\n",
+        "os.environ['FIRECRAWL_API_KEY'] = userdata.get('FIRECRAWL_API_KEY')\n",
+        "import time\n",
+        "from firecrawl import FirecrawlApp\n",
+        "\n",
+        "async def speed_comparison():\n",
+        "    # Simulated Firecrawl performance\n",
+        "    app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])\n",
+        "    start = time.time()\n",
+        "    scrape_status = app.scrape_url(\n",
+        "    'https://www.nbcnews.com/business',\n",
+        "    params={'formats': ['markdown', 'html']}\n",
+        "    )\n",
+        "    end = time.time()\n",
+        "    print(\"Firecrawl (simulated):\")\n",
+        "    print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "    print(f\"Content length: {len(scrape_status['markdown'])} characters\")\n",
+        "    print(f\"Images found: {scrape_status['markdown'].count('cldnry.s-nbcnews.com')}\")\n",
+        "    print()\n",
+        "\n",
+        "    async with AsyncWebCrawler() as crawler:\n",
+        "        # Crawl4AI simple crawl\n",
+        "        start = time.time()\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            word_count_threshold=0,\n",
+        "            bypass_cache=True,\n",
+        "            verbose=False\n",
+        "        )\n",
+        "        end = time.time()\n",
+        "        print(\"Crawl4AI (simple crawl):\")\n",
+        "        print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "        print(f\"Content length: {len(result.markdown)} characters\")\n",
+        "        print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
+        "        print()\n",
+        "\n",
+        "        # Crawl4AI with JavaScript execution\n",
+        "        start = time.time()\n",
+        "        result = await crawler.arun(\n",
+        "            url=\"https://www.nbcnews.com/business\",\n",
+        "            js_code=[\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"],\n",
+        "            word_count_threshold=0,\n",
+        "            bypass_cache=True,\n",
+        "            verbose=False\n",
+        "        )\n",
+        "        end = time.time()\n",
+        "        print(\"Crawl4AI (with JavaScript execution):\")\n",
+        "        print(f\"Time taken: {end - start:.2f} seconds\")\n",
+        "        print(f\"Content length: {len(result.markdown)} characters\")\n",
+        "        print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
+        "\n",
+        "await speed_comparison()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "OBFFYVJIyZQN"
+      },
+      "source": [
+        "If you run on a local machine with a proper internet speed:\n",
+        "- Simple crawl: Crawl4AI is typically over 3-4 times faster than Firecrawl.\n",
+        "- With JavaScript execution: Even when executing JavaScript to load more content (potentially doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.\n",
+        "\n",
+        "Please note that actual performance may vary depending on network conditions and the specific content being crawled."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "A6_1RK1_yZQO"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "In this notebook, we've explored the powerful features of Crawl4AI, including:\n",
+        "\n",
+        "1. Basic crawling\n",
+        "2. JavaScript execution and CSS selector usage\n",
+        "3. Proxy support\n",
+        "4. Structured data extraction with OpenAI\n",
+        "5. Advanced multi-page crawling with JavaScript execution\n",
+        "6. Fast structured output using JsonCssExtractionStrategy\n",
+        "7. Speed comparison with other services\n",
+        "\n",
+        "Crawl4AI offers a fast, flexible, and powerful solution for web crawling and data extraction tasks. Its asynchronous architecture and advanced features make it suitable for a wide range of applications, from simple web scraping to complex, multi-page data extraction scenarios.\n",
+        "\n",
+        "For more information and advanced usage, please visit the [Crawl4AI documentation](https://crawl4ai.com/mkdocs/).\n",
+        "\n",
+        "Happy crawling!"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.13"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}
--- a/docs/examples/quickstart_async.py
+++ b/docs/examples/quickstart_async.py
@@ -0,0 +1,478 @@
+import os, sys
+# append parent directory to system path
+sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))); os.environ['FIRECRAWL_API_KEY'] = "fc-84b370ccfad44beabc686b38f1769692";
+
+import asyncio
+# import nest_asyncio
+# nest_asyncio.apply()
+
+import time
+import json
+import os
+import re
+from typing import Dict
+from bs4 import BeautifulSoup
+from pydantic import BaseModel, Field
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import (
+    JsonCssExtractionStrategy,
+    LLMExtractionStrategy,
+)
+
+__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
+
+print("Crawl4AI: Advanced Web Crawling and Data Extraction")
+print("GitHub Repository: https://github.com/unclecode/crawl4ai")
+print("Twitter: @unclecode")
+print("Website: https://crawl4ai.com")
+
+
+async def simple_crawl():
+    print("\n--- Basic Usage ---")
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.nbcnews.com/business")
+        print(result.markdown[:500])  # Print first 500 characters
+
+async def simple_example_with_running_js_code():
+    print("\n--- Executing JavaScript and Using CSS Selectors ---")
+    # New code to handle the wait_for parameter
+    wait_for = """() => {
+        return Array.from(document.querySelectorAll('article.tease-card')).length > 10;
+    }"""
+
+    # wait_for can be also just a css selector
+    # wait_for = "article.tease-card:nth-child(10)"
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        js_code = [
+            "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+        ]
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            # wait_for=wait_for,
+            bypass_cache=True,
+        )
+        print(result.markdown[:500])  # Print first 500 characters
+
+async def simple_example_with_css_selector():
+    print("\n--- Using CSS Selectors ---")
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            css_selector=".wide-tease-item__description",
+            bypass_cache=True,
+        )
+        print(result.markdown[:500])  # Print first 500 characters
+
+async def use_proxy():
+    print("\n--- Using a Proxy ---")
+    print(
+        "Note: Replace 'http://your-proxy-url:port' with a working proxy to run this example."
+    )
+    # Uncomment and modify the following lines to use a proxy
+    # async with AsyncWebCrawler(verbose=True, proxy="http://your-proxy-url:port") as crawler:
+    #     result = await crawler.arun(
+    #         url="https://www.nbcnews.com/business",
+    #         bypass_cache=True
+    #     )
+    #     print(result.markdown[:500])  # Print first 500 characters
+
+async def capture_and_save_screenshot(url: str, output_path: str):
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url=url,
+            screenshot=True,
+            bypass_cache=True
+        )
+        
+        if result.success and result.screenshot:
+            import base64
+            
+            # Decode the base64 screenshot data
+            screenshot_data = base64.b64decode(result.screenshot)
+            
+            # Save the screenshot as a JPEG file
+            with open(output_path, 'wb') as f:
+                f.write(screenshot_data)
+            
+            print(f"Screenshot saved successfully to {output_path}")
+        else:
+            print("Failed to capture screenshot")
+
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(
+        ..., description="Fee for output token for the OpenAI model."
+    )
+
+async def extract_structured_data_using_llm(provider: str, api_token: str = None, extra_headers: Dict[str, str] = None):
+    print(f"\n--- Extracting Structured Data with {provider} ---")
+    
+    if api_token is None and provider != "ollama":
+        print(f"API token is required for {provider}. Skipping this example.")
+        return
+
+    extra_args = {}
+    if extra_headers:
+        extra_args["extra_headers"] = extra_headers
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://openai.com/api/pricing/",
+            word_count_threshold=1,
+            extraction_strategy=LLMExtractionStrategy(
+                provider=provider,
+                api_token=api_token,
+                schema=OpenAIModelFee.schema(),
+                extraction_type="schema",
+                instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
+                Do not miss any models in the entire content. One extracted model JSON format should look like this: 
+                {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}.""",
+                extra_args=extra_args
+            ),
+            bypass_cache=True,
+        )
+        print(result.extracted_content)
+
+async def extract_structured_data_using_css_extractor():
+    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
+    schema = {
+        "name": "Coinbase Crypto Prices",
+        "baseSelector": ".cds-tableRow-t45thuk",
+        "fields": [
+            {
+                "name": "crypto",
+                "selector": "td:nth-child(1) h2",
+                "type": "text",
+            },
+            {
+                "name": "symbol",
+                "selector": "td:nth-child(1) p",
+                "type": "text",
+            },
+            {
+                "name": "price",
+                "selector": "td:nth-child(2)",
+                "type": "text",
+            }
+        ],
+    }
+
+    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.coinbase.com/explore",
+            extraction_strategy=extraction_strategy,
+            bypass_cache=True,
+        )
+
+        assert result.success, "Failed to crawl the page"
+
+        news_teasers = json.loads(result.extracted_content)
+        print(f"Successfully extracted {len(news_teasers)} news teasers")
+        print(json.dumps(news_teasers[0], indent=2))
+
+# Advanced Session-Based Crawling with Dynamic Content 🔄
+async def crawl_dynamic_content_pages_method_1():
+    print("\n--- Advanced Multi-Page Crawling with JavaScript Execution ---")
+    first_commit = ""
+
+    async def on_execution_started(page):
+        nonlocal first_commit
+        try:
+            while True:
+                await page.wait_for_selector("li.Box-sc-g0xbh4-0 h4")
+                commit = await page.query_selector("li.Box-sc-g0xbh4-0 h4")
+                commit = await commit.evaluate("(element) => element.textContent")
+                commit = re.sub(r"\s+", "", commit)
+                if commit and commit != first_commit:
+                    first_commit = commit
+                    break
+                await asyncio.sleep(0.5)
+        except Exception as e:
+            print(f"Warning: New content didn't appear after JavaScript execution: {e}")
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started)
+
+        url = "https://github.com/microsoft/TypeScript/commits/main"
+        session_id = "typescript_commits_session"
+        all_commits = []
+
+        js_next_page = """
+        const button = document.querySelector('a[data-testid="pagination-next-button"]');
+        if (button) button.click();
+        """
+
+        for page in range(3):  # Crawl 3 pages
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.Box-sc-g0xbh4-0",
+                js=js_next_page if page > 0 else None,
+                bypass_cache=True,
+                js_only=page > 0,
+                headless=False,
+            )
+
+            assert result.success, f"Failed to crawl page {page + 1}"
+
+            soup = BeautifulSoup(result.cleaned_html, "html.parser")
+            commits = soup.select("li")
+            all_commits.extend(commits)
+
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+async def crawl_dynamic_content_pages_method_2():
+    print("\n--- Advanced Multi-Page Crawling with JavaScript Execution ---")
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/microsoft/TypeScript/commits/main"
+        session_id = "typescript_commits_session"
+        all_commits = []
+        last_commit = ""
+
+        js_next_page_and_wait = """
+        (async () => {
+            const getCurrentCommit = () => {
+                const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
+                return commits.length > 0 ? commits[0].textContent.trim() : null;
+            };
+
+            const initialCommit = getCurrentCommit();
+            const button = document.querySelector('a[data-testid="pagination-next-button"]');
+            if (button) button.click();
+
+            // Poll for changes
+            while (true) {
+                await new Promise(resolve => setTimeout(resolve, 100)); // Wait 100ms
+                const newCommit = getCurrentCommit();
+                if (newCommit && newCommit !== initialCommit) {
+                    break;
+                }
+            }
+        })();
+        """
+
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.Box-sc-g0xbh4-0",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.markdown-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        for page in range(3):  # Crawl 3 pages
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.Box-sc-g0xbh4-0",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page_and_wait if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True,
+                headless=False,
+            )
+
+            assert result.success, f"Failed to crawl page {page + 1}"
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+async def crawl_dynamic_content_pages_method_3():
+    print("\n--- Advanced Multi-Page Crawling with JavaScript Execution using `wait_for` ---")
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/microsoft/TypeScript/commits/main"
+        session_id = "typescript_commits_session"
+        all_commits = []
+
+        js_next_page = """
+        const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
+        if (commits.length > 0) {
+            window.firstCommit = commits[0].textContent.trim();
+        }
+        const button = document.querySelector('a[data-testid="pagination-next-button"]');
+        if (button) button.click();
+        """
+
+        wait_for = """() => {
+            const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
+            if (commits.length === 0) return false;
+            const firstCommit = commits[0].textContent.trim();
+            return firstCommit !== window.firstCommit;
+        }"""
+        
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.Box-sc-g0xbh4-0",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.markdown-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        for page in range(3):  # Crawl 3 pages
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.Box-sc-g0xbh4-0",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page if page > 0 else None,
+                wait_for=wait_for if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True,
+                headless=False,
+            )
+
+            assert result.success, f"Failed to crawl page {page + 1}"
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+async def crawl_custom_browser_type():
+    # Use Firefox
+    start = time.time()
+    async with AsyncWebCrawler(browser_type="firefox", verbose=True, headless = True) as crawler:
+        result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
+        print(result.markdown[:500])
+        print("Time taken: ", time.time() - start)
+
+    # Use WebKit
+    start = time.time()
+    async with AsyncWebCrawler(browser_type="webkit", verbose=True, headless = True) as crawler:
+        result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
+        print(result.markdown[:500])
+        print("Time taken: ", time.time() - start)
+
+    # Use Chromium (default)
+    start = time.time()
+    async with AsyncWebCrawler(verbose=True, headless = True) as crawler:
+        result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
+        print(result.markdown[:500])
+        print("Time taken: ", time.time() - start)
+
+async def speed_comparison():
+    # print("\n--- Speed Comparison ---")
+    # print("Firecrawl (simulated):")
+    # print("Time taken: 7.02 seconds")
+    # print("Content length: 42074 characters")
+    # print("Images found: 49")
+    # print()
+    # Simulated Firecrawl performance
+    from firecrawl import FirecrawlApp
+    app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
+    start = time.time()
+    scrape_status = app.scrape_url(
+    'https://www.nbcnews.com/business',
+    params={'formats': ['markdown', 'html']}
+    )
+    end = time.time()
+    print("Firecrawl (simulated):")
+    print(f"Time taken: {end - start:.2f} seconds")
+    print(f"Content length: {len(scrape_status['markdown'])} characters")
+    print(f"Images found: {scrape_status['markdown'].count('cldnry.s-nbcnews.com')}")
+    print()    
+
+    async with AsyncWebCrawler() as crawler:
+        # Crawl4AI simple crawl
+        start = time.time()
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            word_count_threshold=0,
+            bypass_cache=True,
+            verbose=False,
+        )
+        end = time.time()
+        print("Crawl4AI (simple crawl):")
+        print(f"Time taken: {end - start:.2f} seconds")
+        print(f"Content length: {len(result.markdown)} characters")
+        print(f"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}")
+        print()
+
+        # Crawl4AI with JavaScript execution
+        start = time.time()
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=[
+                "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+            ],
+            word_count_threshold=0,
+            bypass_cache=True,
+            verbose=False,
+        )
+        end = time.time()
+        print("Crawl4AI (with JavaScript execution):")
+        print(f"Time taken: {end - start:.2f} seconds")
+        print(f"Content length: {len(result.markdown)} characters")
+        print(f"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}")
+
+    print("\nNote on Speed Comparison:")
+    print("The speed test conducted here may not reflect optimal conditions.")
+    print("When we call Firecrawl's API, we're seeing its best performance,")
+    print("while Crawl4AI's performance is limited by the local network speed.")
+    print("For a more accurate comparison, it's recommended to run these tests")
+    print("on servers with a stable and fast internet connection.")
+    print("Despite these limitations, Crawl4AI still demonstrates faster performance.")
+    print("If you run these tests in an environment with better network conditions,")
+    print("you may observe an even more significant speed advantage for Crawl4AI.")
+
+async def main():
+    await simple_crawl()
+    await simple_example_with_running_js_code()
+    await simple_example_with_css_selector()
+    await use_proxy()
+    await capture_and_save_screenshot("https://www.example.com", os.path.join(__location__, "tmp/example_screenshot.jpg"))
+    await extract_structured_data_using_css_extractor()
+
+    # LLM extraction examples
+    await extract_structured_data_using_llm()
+    await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
+    await extract_structured_data_using_llm("openai/gpt-4", os.getenv("OPENAI_API_KEY"))
+    await extract_structured_data_using_llm("ollama/llama3.2")    
+
+    # You always can pass custom headers to the extraction strategy
+    custom_headers = {
+        "Authorization": "Bearer your-custom-token",
+        "X-Custom-Header": "Some-Value"
+    }
+    await extract_structured_data_using_llm(extra_headers=custom_headers)
+    
+    # await crawl_dynamic_content_pages_method_1()
+    # await crawl_dynamic_content_pages_method_2()
+    await crawl_dynamic_content_pages_method_3()
+    
+    await crawl_custom_browser_type()
+    
+    await speed_comparison()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
--- a/docs/examples/quickstart_sync.py
+++ b/docs/examples/quickstart_sync.py
@@ -249,15 +249,40 @@ def using_crawler_hooks(crawler):
    
    cprint("\n🔗 [bold cyan]Using Crawler Hooks: Let's see how we can customize the crawler using hooks![/bold cyan]", True)
    
-    crawler.set_hook('on_driver_created', on_driver_created)
-    crawler.set_hook('before_get_url', before_get_url)
-    crawler.set_hook('after_get_url', after_get_url)
-    crawler.set_hook('before_return_html', before_return_html)
+    crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
+    crawler_strategy.set_hook('on_driver_created', on_driver_created)
+    crawler_strategy.set_hook('before_get_url', before_get_url)
+    crawler_strategy.set_hook('after_get_url', after_get_url)
+    crawler_strategy.set_hook('before_return_html', before_return_html)
    
+    crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
+    crawler.warmup()    
    result = crawler.run(url="https://example.com")
    
    cprint("[LOG] 📦 [bold yellow]Crawler Hooks result:[/bold yellow]")
    print_result(result= result)
+    
+def using_crawler_hooks_dleay_example(crawler):
+    def delay(driver):
+        print("Delaying for 5 seconds...")
+        time.sleep(5)
+        print("Resuming...")
+        
+    def create_crawler():
+        crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
+        crawler_strategy.set_hook('after_get_url', delay)
+        crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
+        crawler.warmup()
+        return crawler
+
+    cprint("\n🔗 [bold cyan]Using Crawler Hooks: Let's add a delay after fetching the url to make sure entire page is fetched.[/bold cyan]")
+    crawler = create_crawler()
+    result = crawler.run(url="https://google.com", bypass_cache=True)    
+    
+    cprint("[LOG] 📦 [bold yellow]Crawler Hooks result:[/bold yellow]")
+    print_result(result)
+    
+    

 def main():
    cprint("🌟 [bold green]Welcome to the Crawl4ai Quickstart Guide! Let's dive into some web crawling fun! 🌐[/bold green]")
--- a/docs/examples/research_assistant.py
+++ b/docs/examples/research_assistant.py
@@ -1,4 +1,4 @@
-# Make sur to install the required packageschainlit and groq
+# Make sure to install the required packageschainlit and groq
 import os, time
 from openai import AsyncOpenAI
 import chainlit as cl
--- a/docs/examples/sample_ecommerce.html
+++ b/docs/examples/sample_ecommerce.html
@@ -0,0 +1,106 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Sample E-commerce Page for JsonCssExtractionStrategy Testing</title>
+    <style>
+        body { font-family: Arial, sans-serif; line-height: 1.6; padding: 20px; }
+        .category { border: 1px solid #ddd; margin-bottom: 20px; padding: 10px; }
+        .product { border: 1px solid #eee; margin: 10px 0; padding: 10px; }
+        .product-details, .product-reviews, .related-products { margin-top: 10px; }
+        .review { background-color: #f9f9f9; margin: 5px 0; padding: 5px; }
+    </style>
+</head>
+<body>
+    <h1>Sample E-commerce Product Catalog</h1>
+    <div id="catalog"></div>
+
+    <script>
+        const categories = ['Electronics', 'Home & Kitchen', 'Books'];
+        const products = [
+            {
+                name: 'Smartphone X',
+                price: '$999',
+                brand: 'TechCorp',
+                model: 'X-2000',
+                features: ['5G capable', '6.5" OLED screen', '128GB storage'],
+                reviews: [
+                    { reviewer: 'John D.', rating: '4.5', text: 'Great phone, love the camera!' },
+                    { reviewer: 'Jane S.', rating: '5', text: 'Best smartphone I\'ve ever owned.' }
+                ],
+                related: [
+                    { name: 'Phone Case', price: '$29.99' },
+                    { name: 'Screen Protector', price: '$9.99' }
+                ]
+            },
+            {
+                name: 'Laptop Pro',
+                price: '$1499',
+                brand: 'TechMaster',
+                model: 'LT-3000',
+                features: ['Intel i7 processor', '16GB RAM', '512GB SSD'],
+                reviews: [
+                    { reviewer: 'Alice W.', rating: '4', text: 'Powerful machine, but a bit heavy.' },
+                    { reviewer: 'Bob M.', rating: '5', text: 'Perfect for my development work!' }
+                ],
+                related: [
+                    { name: 'Laptop Bag', price: '$49.99' },
+                    { name: 'Wireless Mouse', price: '$24.99' }
+                ]
+            }
+        ];
+
+        function createProductHTML(product) {
+            return `
+                <div class="product">
+                    <h3 class="product-name">${product.name}</h3>
+                    <p class="product-price">${product.price}</p>
+                    <div class="product-details">
+                        <span class="brand">${product.brand}</span>
+                        <span class="model">${product.model}</span>
+                    </div>
+                    <ul class="product-features">
+                        ${product.features.map(feature => `<li>${feature}</li>`).join('')}
+                    </ul>
+                    <div class="product-reviews">
+                        ${product.reviews.map(review => `
+                            <div class="review">
+                                <span class="reviewer">${review.reviewer}</span>
+                                <span class="rating">${review.rating}</span>
+                                <p class="review-text">${review.text}</p>
+                            </div>
+                        `).join('')}
+                    </div>
+                    <ul class="related-products">
+                        ${product.related.map(item => `
+                            <li>
+                                <span class="related-name">${item.name}</span>
+                                <span class="related-price">${item.price}</span>
+                            </li>
+                        `).join('')}
+                    </ul>
+                </div>
+            `;
+        }
+
+        function createCategoryHTML(category, products) {
+            return `
+                <div class="category">
+                    <h2 class="category-name">${category}</h2>
+                    ${products.map(createProductHTML).join('')}
+                </div>
+            `;
+        }
+
+        function populateCatalog() {
+            const catalog = document.getElementById('catalog');
+            categories.forEach(category => {
+                catalog.innerHTML += createCategoryHTML(category, products);
+            });
+        }
+
+        populateCatalog();
+    </script>
+</body>
+</html>
--- a/docs/examples/summarize_page.py
+++ b/docs/examples/summarize_page.py
@@ -42,5 +42,5 @@ page_summary = json.loads(result.extracted_content)

 print(page_summary)

-with open(".data/page_summary.json", "w") as f:
+with open(".data/page_summary.json", "w", encoding="utf-8") as f:
    f.write(result.extracted_content)
--- a/docs/examples/tmp/research_assistant_audio_not_completed.py
+++ b/docs/examples/tmp/research_assistant_audio_not_completed.py
@@ -1,4 +1,4 @@
-# Make sur to install the required packageschainlit and groq
+# Make sure to install the required packageschainlit and groq
 import os, time
 from openai import AsyncOpenAI
 import chainlit as cl
--- a/_sync/api/core_classes_and_functions.md
+++ b/_sync/api/core_classes_and_functions.md
@@ -0,0 +1,141 @@
+# Core Classes and Functions
+
+## Overview
+
+In this section, we will delve into the core classes and functions that make up the Crawl4AI library. This includes the `WebCrawler` class, various `CrawlerStrategy` classes, `ChunkingStrategy` classes, and `ExtractionStrategy` classes. Understanding these core components will help you leverage the full power of Crawl4AI for your web crawling and data extraction needs.
+
+## WebCrawler Class
+
+The `WebCrawler` class is the main class you'll interact with. It provides the interface for crawling web pages and extracting data.
+
+### Initialization
+
+```python
+from crawl4ai import WebCrawler
+
+# Create an instance of WebCrawler
+crawler = WebCrawler()
+```
+
+### Methods
+
+- **`warmup()`**: Prepares the crawler for use, such as loading necessary models.
+- **`run(url: str, **kwargs)`**: Runs the crawler on the specified URL with optional parameters for customization.
+
+```python
+crawler.warmup()
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(result)
+```
+
+## CrawlerStrategy Classes
+
+The `CrawlerStrategy` classes define how the web crawling is executed. The base class is `CrawlerStrategy`, which is extended by specific implementations like `LocalSeleniumCrawlerStrategy`.
+
+### CrawlerStrategy Base Class
+
+An abstract base class that defines the interface for different crawler strategies.
+
+```python
+from abc import ABC, abstractmethod
+
+class CrawlerStrategy(ABC):
+    @abstractmethod
+    def crawl(self, url: str, **kwargs) -> str:
+        pass
+    
+    @abstractmethod
+    def take_screenshot(self, save_path: str):
+        pass
+    
+    @abstractmethod
+    def update_user_agent(self, user_agent: str):
+        pass
+    
+    @abstractmethod
+    def set_hook(self, hook_type: str, hook: Callable):
+        pass
+```
+
+### LocalSeleniumCrawlerStrategy Class
+
+A concrete implementation of `CrawlerStrategy` that uses Selenium to crawl web pages.
+
+#### Initialization
+
+```python
+from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
+
+strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
+```
+
+#### Methods
+
+- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+```python
+result = strategy.crawl("https://www.example.com")
+strategy.take_screenshot("screenshot.png")
+strategy.update_user_agent("Mozilla/5.0")
+strategy.set_hook("before_get_url", lambda: print("About to get URL"))
+```
+
+## ChunkingStrategy Classes
+
+The `ChunkingStrategy` classes define how the text from a web page is divided into chunks. Here are a few examples:
+
+### RegexChunking Class
+
+Splits text using regular expressions.
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+chunker = RegexChunking(patterns=[r'\n\n'])
+chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
+```
+
+### NlpSentenceChunking Class
+
+Uses NLP to split text into sentences.
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
+```
+
+## ExtractionStrategy Classes
+
+The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks. Here are a few examples:
+
+### CosineStrategy Class
+
+Clusters text chunks based on cosine similarity.
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### LLMExtractionStrategy Class
+
+Uses a Language Model to extract meaningful blocks from HTML.
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+## Conclusion
+
+By understanding these core classes and functions, you can customize and extend Crawl4AI to suit your specific web crawling and data extraction needs. Happy crawling! 🕷️🤖
+
--- a/_sync/api/detailed_api_documentation.md
+++ b/_sync/api/detailed_api_documentation.md
@@ -0,0 +1,338 @@
+# Detailed API Documentation
+
+## Overview
+
+This section provides comprehensive documentation for the Crawl4AI API, covering all classes, methods, and their parameters. This guide will help you understand how to utilize the API to its full potential, enabling efficient web crawling and data extraction.
+
+## WebCrawler Class
+
+The `WebCrawler` class is the primary interface for crawling web pages and extracting data.
+
+### Initialization
+
+```python
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+```
+
+### Methods
+
+#### `warmup()`
+
+Prepares the crawler for use, such as loading necessary models.
+
+```python
+crawler.warmup()
+```
+
+#### `run(url: str, **kwargs) -> CrawlResult`
+
+Crawls the specified URL and returns the result.
+
+- **Parameters:**
+  - `url` (str): The URL to crawl.
+  - `**kwargs`: Additional parameters for customization.
+
+- **Returns:**
+  - `CrawlResult`: An object containing the crawl result.
+
+- **Example:**
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(result)
+```
+
+### CrawlResult Class
+
+Represents the result of a crawl operation.
+
+- **Attributes:**
+  - `url` (str): The URL of the crawled page.
+  - `html` (str): The raw HTML of the page.
+  - `success` (bool): Whether the crawl was successful.
+  - `cleaned_html` (Optional[str]): The cleaned HTML.
+  - `media` (Dict[str, List[Dict]]): Media tags in the page (images, audio, video).
+  - `links` (Dict[str, List[Dict]]): Links in the page (external, internal).
+  - `screenshot` (Optional[str]): Base64 encoded screenshot.
+  - `markdown` (Optional[str]): Extracted content in Markdown format.
+  - `extracted_content` (Optional[str]): Extracted meaningful content.
+  - `metadata` (Optional[dict]): Metadata from the page.
+  - `error_message` (Optional[str]): Error message if any.
+
+## CrawlerStrategy Classes
+
+The `CrawlerStrategy` classes define how the web crawling is executed.
+
+### CrawlerStrategy Base Class
+
+An abstract base class for different crawler strategies.
+
+#### Methods
+
+- **`crawl(url: str, **kwargs) -> str`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+### LocalSeleniumCrawlerStrategy Class
+
+Uses Selenium to crawl web pages.
+
+#### Initialization
+
+```python
+from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
+
+strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
+```
+
+#### Methods
+
+- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
+- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
+- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
+- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
+
+#### Example
+
+```python
+result = strategy.crawl("https://www.example.com")
+strategy.take_screenshot("screenshot.png")
+strategy.update_user_agent("Mozilla/5.0")
+strategy.set_hook("before_get_url", lambda: print("About to get URL"))
+```
+
+## ChunkingStrategy Classes
+
+The `ChunkingStrategy` classes define how the text from a web page is divided into chunks.
+
+### RegexChunking Class
+
+Splits text using regular expressions.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+chunker = RegexChunking(patterns=[r'\n\n'])
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into chunks.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
+```
+
+### NlpSentenceChunking Class
+
+Uses NLP to split text into sentences.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into sentences.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
+```
+
+### TopicSegmentationChunking Class
+
+Uses the TextTiling algorithm to segment text into topics.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import TopicSegmentationChunking
+
+chunker = TopicSegmentationChunking(num_keywords=3)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into topic-based segments.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into topic-based segments.")
+```
+
+### FixedLengthWordChunking Class
+
+Splits text into chunks of fixed length based on the number of words.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import FixedLengthWordChunking
+
+chunker = FixedLengthWordChunking(chunk_size=100)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text into fixed-length word chunks.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split into fixed-length word chunks.")
+```
+
+### SlidingWindowChunking Class
+
+Uses a sliding window approach to chunk text.
+
+#### Initialization
+
+```python
+from crawl4ai.chunking_strategy import SlidingWindowChunking
+
+chunker = SlidingWindowChunking(window_size=100, step=50)
+```
+
+#### Methods
+
+- **`chunk(text: str) -> List[str]`**: Splits the text using a sliding window approach.
+
+#### Example
+
+```python
+chunks = chunker.chunk("This is a sample text. It will be split using a sliding window approach.")
+```
+
+## ExtractionStrategy Classes
+
+The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks.
+
+### NoExtractionStrategy Class
+
+Returns the entire HTML content without any modification.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import NoExtractionStrategy
+
+extractor = NoExtractionStrategy()
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Returns the HTML content.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### LLMExtractionStrategy Class
+
+Uses a Language Model to extract meaningful blocks from HTML.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts meaningful content using the LLM.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### CosineStrategy Class
+
+Clusters text chunks based on cosine similarity.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts clusters of text based on cosine similarity.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+### TopicExtractionStrategy Class
+
+Uses the TextTiling algorithm to segment HTML content into topics and extract keywords.
+
+#### Initialization
+
+```python
+from crawl4ai.extraction_strategy import TopicExtractionStrategy
+
+extractor = TopicExtractionStrategy(num_keywords=3)
+```
+
+#### Methods
+
+- **`extract(url: str, html: str) -> str`**: Extracts topic-based segments and keywords.
+
+#### Example
+
+```python
+extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
+```
+
+## Parameters
+
+Here are the common parameters used across various classes and methods:
+
+- **`url`** (str): The URL to crawl.
+- **`html`** (str): The HTML content of the page.
+- **`user_agent`** (str): The user agent for the HTTP requests.
+- **`patterns`** (list): A list of regular expression patterns for chunking.
+- **`num_keywords`** (int): Number of keywords for topic extraction.
+- **`chunk_size`** (int): Number of words in each chunk.
+- **`window_size`** (int): Number of words in the sliding window.
+- **`step`** (int): Step size for the sliding window.
+- **`semantic_filter`** (str): Keywords for filtering relevant documents.
+- **`word_count_threshold`** (int): Minimum number of words per cluster.
+- **`max_dist`** (float): Maximum cophenetic distance for clustering.
+- **`linkage_method`** (str): Linkage method for hierarchical clustering.
+- **`top_k`** (int): Number of top categories to extract.
+- **`provider`** (
+
+str): Provider for language model completions.
+- **`api_token`** (str): API token for the provider.
+- **`instruction`** (str): Instruction to guide the LLM extraction.
+
+## Conclusion
+
+This detailed API documentation provides a thorough understanding of the classes, methods, and parameters in the Crawl4AI library. With this knowledge, you can effectively use the API to perform advanced web crawling and data extraction tasks.
--- a/_sync/assets/DankMono-Bold.woff2
+++ b/_sync/assets/DankMono-Bold.woff2
--- a/_sync/assets/DankMono-Italic.woff2
+++ b/_sync/assets/DankMono-Italic.woff2
--- a/_sync/assets/DankMono-Regular.woff2
+++ b/_sync/assets/DankMono-Regular.woff2
--- a/_sync/assets/Monaco.woff
+++ b/_sync/assets/Monaco.woff
--- a/_sync/assets/dmvendor.css
+++ b/_sync/assets/dmvendor.css
--- a/_sync/assets/highlight.css
+++ b/_sync/assets/highlight.css
--- a/_sync/assets/highlight.min.js
+++ b/_sync/assets/highlight.min.js
--- a/_sync/assets/highlight_init.js
+++ b/_sync/assets/highlight_init.js
@@ -0,0 +1,6 @@
+document.addEventListener('DOMContentLoaded', (event) => {
+    document.querySelectorAll('pre code').forEach((block) => {
+      hljs.highlightBlock(block);
+    });
+  });
+  
--- a/_sync/assets/styles.css
+++ b/_sync/assets/styles.css
@@ -0,0 +1,153 @@
+@font-face {
+    font-family: "Monaco";
+    font-style: normal;
+    font-weight: normal;
+    src: local("Monaco"), url("Monaco.woff") format("woff");
+}
+
+:root {
+    --global-font-size: 16px;
+    --global-line-height: 1.5em;
+    --global-space: 10px;
+    --font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
+        Courier New, monospace, serif;
+    --font-stack: dm, Monaco, Courier New, monospace, serif;
+    --mono-font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
+        Courier New, monospace, serif;
+
+    --background-color: #151515; /* Dark background */
+    --font-color: #eaeaea; /* Light font color for contrast */
+    --invert-font-color: #151515; /* Dark color for inverted elements */
+    --primary-color: #1a95e0; /* Primary color can remain the same or be adjusted for better contrast */
+    --secondary-color: #727578; /* Secondary color for less important text */
+    --error-color: #ff5555; /* Bright color for errors */
+    --progress-bar-background: #444; /* Darker background for progress bar */
+    --progress-bar-fill: #1a95e0; /* Bright color for progress bar fill */
+    --code-bg-color: #1e1e1e; /* Darker background for code blocks */
+    --input-style: solid; /* Keeping input style solid */
+    --block-background-color: #202020; /* Darker background for block elements */
+    --global-font-color: #eaeaea; /* Light font color for global elements */
+
+    --background-color: #222225;
+
+    --background-color: #070708;
+    --page-width: 70em;
+    --font-color: #e8e9ed;
+    --invert-font-color: #222225;
+    --secondary-color: #a3abba;
+    --secondary-color: #d5cec0;
+    --tertiary-color: #a3abba;
+    --primary-color: #09b5a5; /* Updated to the brand color */
+    --primary-color: #50ffff; /* Updated to the brand color */
+    --error-color: #ff3c74;
+    --progress-bar-background: #3f3f44;
+    --progress-bar-fill: #09b5a5; /* Updated to the brand color */
+    --code-bg-color: #3f3f44;
+    --input-style: solid;
+    --display-h1-decoration: none;
+
+    --display-h1-decoration: none;
+}
+
+/* body {
+    background-color: var(--background-color);
+    color: var(--font-color);
+}
+
+a {
+    color: var(--primary-color);
+}
+
+a:hover {
+    background-color: var(--primary-color);
+    color: var(--invert-font-color);
+}
+
+blockquote::after {
+    color: #444; 
+}
+
+pre, code {
+    background-color: var(--code-bg-color);
+    color: var(--font-color);
+}
+
+.terminal-nav:first-child {
+    border-bottom: 1px dashed var(--secondary-color);
+} */
+
+.terminal-mkdocs-main-content {
+    line-height: var(--global-line-height);
+}
+
+strong,
+.highlight {
+    /* background: url(//s2.svgbox.net/pen-brushes.svg?ic=brush-1&color=50ffff); */
+    background-color: #50ffff33;
+}
+
+.terminal-card > header {
+    color: var(--font-color);
+    text-align: center;
+    background-color: var(--progress-bar-background);
+    padding: 0.3em 0.5em;
+}
+.btn.btn-sm {
+    color: var(--font-color);
+    padding: 0.2em 0.5em;
+    font-size: 0.8em;
+}
+
+.loading-message {
+    display: none;
+    margin-top: 20px;
+}
+
+.response-section {
+    display: none;
+    padding-top: 20px;
+}
+
+.tabs {
+    display: flex;
+    flex-direction: column;
+}
+.tab-list {
+    display: flex;
+    padding: 0;
+    margin: 0;
+    list-style-type: none;
+    border-bottom: 1px solid var(--font-color);
+}
+.tab-item {
+    cursor: pointer;
+    padding: 10px;
+    border: 1px solid var(--font-color);
+    margin-right: -1px;
+    border-bottom: none;
+}
+.tab-item:hover,
+.tab-item:focus,
+.tab-item:active {
+    background-color: var(--progress-bar-background);
+}
+.tab-content {
+    display: none;
+    border: 1px solid var(--font-color);
+    border-top: none;
+}
+.tab-content:first-of-type {
+    display: block;
+}
+
+.tab-content header {
+    padding: 0.5em;
+    display: flex; 
+    justify-content: end; 
+    align-items: center;
+    background-color: var(--progress-bar-background);
+}
+.tab-content pre {
+    margin: 0;
+    max-height: 300px; overflow: auto; border:none;
+}
--- a/_sync/changelog.md
+++ b/_sync/changelog.md
@@ -0,0 +1,102 @@
+# Changelog
+
+## [v0.2.77] - 2024-08-04
+
+Significant improvements in text processing and performance:
+
+- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
+- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
+- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
+- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
+
+These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
+
+## [v0.2.76] - 2024-08-02
+
+Major improvements in functionality, performance, and cross-platform compatibility! 🚀
+
+- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
+- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
+- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
+- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
+- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
+
+A big shoutout to our amazing community contributors:
+- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
+- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
+- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
+
+Your contributions are driving Crawl4AI forward! 🙌
+
+## [v0.2.75] - 2024-07-19
+
+Minor improvements for a more maintainable codebase:
+
+- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
+- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
+
+These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
+
+
+## v0.2.74 - 2024-07-08
+A slew of exciting updates to improve the crawler's stability and robustness! 🎉
+
+- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
+- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
+- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
+- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
+
+## [v0.2.73] - 2024-07-03
+
+💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
+
+* Supporting website need "with-head" mode to crawl the website with head.
+* Fixing the installation issues for setup.py and dockerfile.
+* Resolve multiple issues.
+
+## [v0.2.72] - 2024-06-30
+
+This release brings exciting updates and improvements to our project! 🎉
+
+* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
+* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
+* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
+* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
+
+These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
+
+## [0.2.71] - 2024-06-26
+
+**Improved Error Handling and Performance** 🚧
+
+* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
+* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
+* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
+* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
+
+These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
+
+## [0.2.71] - 2024-06-25
+### Fixed
+- Speed up twice the extraction function.
+
+## [0.2.6] - 2024-06-22
+### Fixed
+- Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
+
+## [0.2.5] - 2024-06-18
+### Added
+- Added five important hooks to the crawler:
+  - on_driver_created: Called when the driver is ready for initializations.
+  - before_get_url: Called right before Selenium fetches the URL.
+  - after_get_url: Called after Selenium fetches the URL.
+  - before_return_html: Called when the data is parsed and ready.
+  - on_user_agent_updated: Called when the user changes the user_agent, causing the driver to reinitialize.
+- Added an example in `quickstart.py` in the example folder under the docs.
+- Enhancement issue #24: Replaced inline HTML tags (e.g., DEL, INS, SUB, ABBR) with textual format for better context handling in LLM.
+- Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
+- Updated Dockerfile to ensure compatibility across multiple platforms (Hopefully!).
+
+## [0.2.4] - 2024-06-17
+### Fixed
+- Fix issue #22: Use MD5 hash for caching HTML files to handle long URLs
--- a/_sync/contact.md
+++ b/_sync/contact.md
@@ -0,0 +1,25 @@
+# Contact
+If you have any questions, suggestions, or feedback, please feel free to reach out to us:
+
+- GitHub: [unclecode](https://github.com/unclecode)
+- Twitter: [@unclecode](https://twitter.com/unclecode)
+- Website: [crawl4ai.com](https://crawl4ai.com)
+
+
+## Contributing 🤝
+
+We welcome contributions from the open-source community to help improve Crawl4AI and make it even more valuable for AI enthusiasts and developers. To contribute, please follow these steps:
+
+1. Fork the repository.
+2. Create a new branch for your feature or bug fix.
+3. Make your changes and commit them with descriptive messages.
+4. Push your changes to your forked repository.
+5. Submit a pull request to the main repository.
+
+For more information on contributing, please see our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md).
+
+## License 📄
+
+Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
+
+Let's work together to make the web more accessible and useful for AI applications! 💪🌐🤖
--- a/_sync/demo.md
+++ b/_sync/demo.md
@@ -0,0 +1,231 @@
+# Interactive Demo for Crowler
+<div id="demo">
+    <form id="crawlForm" class="terminal-form">
+        <fieldset>
+            <legend>Enter URL and Options</legend>
+            <div class="form-group">
+                <label for="url">Enter URL:</label>
+                <input type="text" id="url" name="url" required>
+            </div>
+            <div class="form-group">
+                <label for="screenshot">Get Screenshot:</label>
+                <input type="checkbox" id="screenshot" name="screenshot">
+            </div>
+            <div class="form-group">
+                <button class="btn btn-default" type="submit">Submit</button>
+            </div>
+
+        </fieldset>
+    </form>
+
+    <div id="loading" class="loading-message">
+        <div class="terminal-alert terminal-alert-primary">Loading... Please wait.</div>
+    </div>
+
+    <section id="response" class="response-section">
+        <h2>Response</h2>
+        <div class="tabs">
+            <ul class="tab-list">
+                <li class="tab-item" onclick="showTab('markdown')">Markdown</li>
+                <li class="tab-item" onclick="showTab('cleanedHtml')">Cleaned HTML</li>
+                <li class="tab-item" onclick="showTab('media')">Media</li>
+                <li class="tab-item" onclick="showTab('extractedContent')">Extracted Content</li>
+                <li class="tab-item" onclick="showTab('screenshot')">Screenshot</li>
+                <li class="tab-item" onclick="showTab('pythonCode')">Python Code</li>
+            </ul>
+            <div class="tab-content" id="tab-markdown">
+                <header>
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('markdownContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('markdownContent', 'markdown.md')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="markdownContent" class="language-markdown hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-cleanedHtml" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('cleanedHtmlContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('cleanedHtmlContent', 'cleaned.html')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="cleanedHtmlContent" class="language-html hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-media" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('mediaContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('mediaContent', 'media.json')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="mediaContent" class="language-json hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-extractedContent" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('extractedContentContent')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('extractedContentContent', 'extracted_content.json')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="extractedContentContent" class="language-json hljs"></code></pre>
+            </div>
+
+            <div class="tab-content" id="tab-screenshot" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadImage('screenshotContent', 'screenshot.png')">Download</button>
+                    </div>
+                </header>
+                <pre><img id="screenshotContent" /></pre>
+            </div>
+
+            <div class="tab-content" id="tab-pythonCode" style="display: none;">
+                <header >
+                    <div>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('pythonCode')">Copy</button>
+                        <button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('pythonCode', 'example.py')">Download</button>
+                    </div>
+                </header>
+                <pre><code id="pythonCode" class="language-python hljs"></code></pre>
+            </div>
+        </div>
+    </section>
+
+    <div id="error" class="error-message" style="display: none; margin-top:1em;">
+        <div class="terminal-alert terminal-alert-error"></div>
+    </div>
+
+    <script>
+        function showTab(tabId) {
+            const tabs = document.querySelectorAll('.tab-content');
+            tabs.forEach(tab => tab.style.display = 'none');
+            document.getElementById(`tab-${tabId}`).style.display = 'block';
+        }
+
+        function redo(codeBlock, codeText){
+            codeBlock.classList.remove('hljs');
+            codeBlock.removeAttribute('data-highlighted');
+
+            // Set new code and re-highlight
+            codeBlock.textContent = codeText;
+            hljs.highlightBlock(codeBlock);
+        }
+
+        function copyToClipboard(elementId) {
+            const content = document.getElementById(elementId).textContent;
+            navigator.clipboard.writeText(content).then(() => {
+                alert('Copied to clipboard');
+            });
+        }
+
+        function downloadContent(elementId, filename) {
+            const content = document.getElementById(elementId).textContent;
+            const blob = new Blob([content], { type: 'text/plain' });
+            const url = window.URL.createObjectURL(blob);
+            const a = document.createElement('a');
+            a.style.display = 'none';
+            a.href = url;
+            a.download = filename;
+            document.body.appendChild(a);
+            a.click();
+            window.URL.revokeObjectURL(url);
+            document.body.removeChild(a);
+        }
+
+        function downloadImage(elementId, filename) {
+            const content = document.getElementById(elementId).src;
+            const a = document.createElement('a');
+            a.style.display = 'none';
+            a.href = content;
+            a.download = filename;
+            document.body.appendChild(a);
+            a.click();
+            document.body.removeChild(a);
+        }
+
+        document.getElementById('crawlForm').addEventListener('submit', function(event) {
+            event.preventDefault();
+            document.getElementById('loading').style.display = 'block';
+            document.getElementById('response').style.display = 'none';
+
+            const url = document.getElementById('url').value;
+            const screenshot = document.getElementById('screenshot').checked;
+            const data = {
+                urls: [url],
+                bypass_cache: false,
+                word_count_threshold: 5,
+                screenshot: screenshot
+            };
+
+            fetch('/crawl', {
+                method: 'POST',
+                headers: {
+                    'Content-Type': 'application/json'
+                },
+                body: JSON.stringify(data)
+            })
+            .then(response => {
+                if (!response.ok) {
+                    if (response.status === 429) {
+                        return response.json().then(err => { 
+                            throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
+                        });
+                    }
+                    throw new Error('Network response was not ok');
+                }
+                return response.json();
+            })
+            .then(data => {
+                data = data.results[0]; // Only one URL is requested
+                document.getElementById('loading').style.display = 'none';
+                document.getElementById('response').style.display = 'block';
+                redo(document.getElementById('markdownContent'), data.markdown);
+                redo(document.getElementById('cleanedHtmlContent'), data.cleaned_html);
+                redo(document.getElementById('mediaContent'), JSON.stringify(data.media, null, 2));
+                redo(document.getElementById('extractedContentContent'), data.extracted_content);
+                if (screenshot) {
+                    document.getElementById('screenshotContent').src = `data:image/png;base64,${data.screenshot}`;
+                }
+                const pythonCode = `
+from crawl4ai.web_crawler import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+    url='${url}',
+    screenshot=${screenshot}
+)
+print(result)
+                `;
+                redo(document.getElementById('pythonCode'), pythonCode);
+                document.getElementById('error').style.display = 'none';
+            })
+            .catch(error => {
+                document.getElementById('loading').style.display = 'none';
+                document.getElementById('error').style.display = 'block';
+                let errorMessage = 'An unexpected error occurred. Please try again later.';
+                
+                if (error.status === 429) {
+                    const details = error.details;
+                    if (details.retry_after) {
+                        errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
+                    } else if (details.reset_at) {
+                        const resetTime = new Date(details.reset_at);
+                        const waitTime = Math.ceil((resetTime - new Date()) / 1000);
+                        errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
+                    } else {
+                        errorMessage = `Rate limit exceeded. Please try again later.`;
+                    }
+                } else if (error.message) {
+                    errorMessage = error.message;
+                }
+                
+                document.querySelector('#error .terminal-alert').textContent = errorMessage;
+            });
+        });
+    </script>
+</div>
--- a/_sync/examples/hooks_auth.md
+++ b/_sync/examples/hooks_auth.md
@@ -0,0 +1,100 @@
+# Hooks & Auth
+
+Crawl4AI allows you to customize the behavior of the web crawler using hooks. Hooks are functions that are called at specific points in the crawling process, allowing you to modify the crawler's behavior or perform additional actions. This example demonstrates how to use various hooks to customize the crawling process.
+
+## Example: Using Crawler Hooks
+
+Let's see how we can customize the crawler using hooks! In this example, we'll:
+
+1. Maximize the browser window and log in to a website when the driver is created.
+2. Add a custom header before fetching the URL.
+3. Log the current URL after fetching it.
+4. Log the length of the HTML before returning it.
+
+### Hook Definitions
+
+```python
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.crawler_strategy import *
+
+def on_driver_created(driver):
+    print("[HOOK] on_driver_created")
+    # Example customization: maximize the window
+    driver.maximize_window()
+    
+    # Example customization: logging in to a hypothetical website
+    driver.get('https://example.com/login')
+    
+    from selenium.webdriver.support.ui import WebDriverWait
+    from selenium.webdriver.common.by import By
+    from selenium.webdriver.support import expected_conditions as EC
+    
+    WebDriverWait(driver, 10).until(
+        EC.presence_of_element_located((By.NAME, 'username'))
+    )
+    driver.find_element(By.NAME, 'username').send_keys('testuser')
+    driver.find_element(By.NAME, 'password').send_keys('password123')
+    driver.find_element(By.NAME, 'login').click()
+    WebDriverWait(driver, 10).until(
+        EC.presence_of_element_located((By.ID, 'welcome'))
+    )
+    # Add a custom cookie
+    driver.add_cookie({'name': 'test_cookie', 'value': 'cookie_value'})
+    return driver        
+    
+
+def before_get_url(driver):
+    print("[HOOK] before_get_url")
+    # Example customization: add a custom header
+    # Enable Network domain for sending headers
+    driver.execute_cdp_cmd('Network.enable', {})
+    # Add a custom header
+    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
+    return driver
+
+def after_get_url(driver):
+    print("[HOOK] after_get_url")
+    # Example customization: log the URL
+    print(driver.current_url)
+    return driver
+
+def before_return_html(driver, html):
+    print("[HOOK] before_return_html")
+    # Example customization: log the HTML
+    print(len(html))
+    return driver
+```
+
+### Using the Hooks with the WebCrawler
+
+```python
+print("\n🔗 [bold cyan]Using Crawler Hooks: Let's see how we can customize the crawler using hooks![/bold cyan]", True)
+crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
+crawler_strategy.set_hook('on_driver_created', on_driver_created)
+crawler_strategy.set_hook('before_get_url', before_get_url)
+crawler_strategy.set_hook('after_get_url', after_get_url)
+crawler_strategy.set_hook('before_return_html', before_return_html)
+crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
+crawler.warmup()
+
+result = crawler.run(url="https://example.com")
+
+print("[LOG] 📦 [bold yellow]Crawler Hooks result:[/bold yellow]")
+print(result)
+```
+
+### Explanation
+
+- `on_driver_created`: This hook is called when the Selenium driver is created. In this example, it maximizes the window, logs in to a website, and adds a custom cookie.
+- `before_get_url`: This hook is called right before Selenium fetches the URL. In this example, it adds a custom HTTP header.
+- `after_get_url`: This hook is called after Selenium fetches the URL. In this example, it logs the current URL.
+- `before_return_html`: This hook is called before returning the HTML content. In this example, it logs the length of the HTML content.
+
+### Additional Ideas
+
+- **Add custom headers to requests**: You can add custom headers to the requests using the `before_get_url` hook.
+- **Perform safety checks**: Use the hooks to perform safety checks before the crawling process starts.
+- **Modify the HTML content**: Use the `before_return_html` hook to modify the HTML content before it is returned.
+- **Log additional information**: Use the hooks to log additional information for debugging or monitoring purposes.
+
+By using these hooks, you can customize the behavior of the crawler to suit your specific needs.
--- a/_sync/examples/index.md
+++ b/_sync/examples/index.md
@@ -0,0 +1,29 @@
+# Examples
+
+Welcome to the examples section of Crawl4AI documentation! In this section, you will find practical examples demonstrating how to use Crawl4AI for various web crawling and data extraction tasks. Each example is designed to showcase different features and capabilities of the library.
+
+## Examples Index
+
+### [LLM Extraction](llm_extraction.md)
+
+This example demonstrates how to use Crawl4AI to extract information using Large Language Models (LLMs). You will learn how to configure the `LLMExtractionStrategy` to get structured data from web pages.
+
+### [JS Execution & CSS Filtering](js_execution_css_filtering.md)
+
+Learn how to execute custom JavaScript code and filter data using CSS selectors. This example shows how to perform complex web interactions and extract specific content from web pages.
+
+### [Hooks & Auth](hooks_auth.md)
+
+This example covers the use of custom hooks for authentication and other pre-crawling tasks. You will see how to set up hooks to modify headers, authenticate sessions, and perform other preparatory actions before crawling.
+
+### [Summarization](summarization.md)
+
+Discover how to use Crawl4AI to summarize web page content. This example demonstrates the summarization capabilities of the library, helping you extract concise information from lengthy web pages.
+
+### [Research Assistant](research_assistant.md)
+
+In this example, Crawl4AI is used as a research assistant to gather and organize information from multiple sources. You will learn how to use various extraction and chunking strategies to compile a comprehensive report.
+
+---
+
+Each example includes detailed explanations and code snippets to help you understand and implement the features in your projects. Click on the links to explore each example and start making the most of Crawl4AI!
--- a/_sync/examples/js_execution_css_filtering.md
+++ b/_sync/examples/js_execution_css_filtering.md
@@ -0,0 +1,44 @@
+# JS Execution & CSS Filtering
+
+In this example, we'll demonstrate how to use Crawl4AI to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.
+
+## Example: Extracting Structured Data
+
+```python
+# Import necessary modules
+from crawl4ai import WebCrawler
+from crawl4ai.chunking_strategy import *
+from crawl4ai.extraction_strategy import *
+from crawl4ai.crawler_strategy import *
+
+# Define the JavaScript code to click the "Load More" button
+js_code = ["""
+const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+loadMoreButton && loadMoreButton.click();
+"""]
+
+crawler = WebCrawler(verbose=True)
+crawler.warmup()
+# Run the crawler with keyword filtering and CSS selector
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    js=js_code,
+    css_selector="p",
+    extraction_strategy=CosineStrategy(
+        semantic_filter="technology",
+    ),
+)
+
+# Display the extracted result
+print(result)
+```
+
+### Explanation
+
+1. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button. This is useful for loading additional content dynamically.
+2. **CSS Selector**: The `css_selector="p"` parameter ensures that only paragraph (`<p>`) tags are extracted from the web page.
+3. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
+
+## Try It Yourself
+
+This example demonstrates the power and flexibility of Crawl4AI in handling complex web interactions and extracting meaningful data. You can customize the JavaScript code, CSS selectors, and extraction strategies to suit your specific requirements.
--- a/_sync/examples/llm_extraction.md
+++ b/_sync/examples/llm_extraction.md
@@ -0,0 +1,90 @@
+# LLM Extraction
+
+Crawl4AI allows you to use Language Models (LLMs) to extract structured data or relevant content from web pages. Below are two examples demonstrating how to use LLMExtractionStrategy for different purposes.
+
+## Example 1: Extract Structured Data
+
+In this example, we use the `LLMExtractionStrategy` to extract structured data (model names and their fees) from the OpenAI pricing page.
+
+```python
+import os
+import time
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.chunking_strategy import *
+from crawl4ai.extraction_strategy import *
+from crawl4ai.crawler_strategy import *
+
+url = r'https://openai.com/api/pricing/'
+
+crawler = WebCrawler()
+crawler.warmup()
+
+from pydantic import BaseModel, Field
+
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
+
+result = crawler.run(
+    url=url,
+    word_count_threshold=1,
+    extraction_strategy= LLMExtractionStrategy(
+        provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
+        schema=OpenAIModelFee.model_json_schema(),
+        extraction_type="schema",
+        instruction="From the crawled content, extract all mentioned model names along with their "\
+            "fees for input and output tokens. Make sure not to miss anything in the entire content. "\
+            'One extracted model JSON format should look like this: '\
+            '{ "model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens" }'
+    ),
+    bypass_cache=True,
+)
+
+model_fees = json.loads(result.extracted_content)
+
+print(len(model_fees))
+
+with open(".data/data.json", "w", encoding="utf-8") as f:
+    f.write(result.extracted_content)
+```
+
+## Example 2: Extract Relevant Content
+
+In this example, we instruct the LLM to extract only content related to technology from the NBC News business page.
+
+```python
+crawler = WebCrawler()
+crawler.warmup()
+
+result = crawler.run(
+        url="https://www.nbcnews.com/business",
+        extraction_strategy=LLMExtractionStrategy(
+            provider="openai/gpt-4o",
+            api_token=os.getenv('OPENAI_API_KEY'),
+            instruction="Extract only content related to technology"
+        ),
+    bypass_cache=True,
+    )
+
+model_fees = json.loads(result.extracted_content)
+
+print(len(model_fees))
+
+with open(".data/data.json", "w", encoding="utf-8") as f:
+    f.write(result.extracted_content)
+```
+
+## Customizing LLM Provider
+
+Under the hood, Crawl4AI uses the `litellm` library, which allows you to use any LLM provider you want. Just pass the correct model name and API token.
+
+```python
+extraction_strategy=LLMExtractionStrategy(
+    provider="your_llm_provider/model_name",
+    api_token="your_api_token",
+    instruction="Your extraction instruction"
+)
+```
+
+This flexibility allows you to integrate with various LLM providers and tailor the extraction process to your specific needs.
--- a/_sync/examples/research_assistant.md
+++ b/_sync/examples/research_assistant.md
@@ -0,0 +1,248 @@
+## Research Assistant Example
+
+This example demonstrates how to build a research assistant using `Chainlit` and `Crawl4AI`. The assistant will be capable of crawling web pages for information and answering questions based on the crawled content. Additionally, it integrates speech-to-text functionality for audio inputs.
+
+### Step-by-Step Guide
+
+1. **Install Required Packages**
+
+    Ensure you have the necessary packages installed. You need `chainlit`, `groq`, `requests`, and `openai`.
+
+    ```bash
+    pip install chainlit groq requests openai
+    ```
+
+2. **Import Libraries**
+
+    Import all the necessary modules and initialize the OpenAI client.
+
+    ```python
+    import os
+    import time
+    from openai import AsyncOpenAI
+    import chainlit as cl
+    import re
+    import requests
+    from io import BytesIO
+    from chainlit.element import ElementBased
+    from groq import Groq
+
+    from concurrent.futures import ThreadPoolExecutor
+
+    client = AsyncOpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.getenv("GROQ_API_KEY"))
+
+    # Instrument the OpenAI client
+    cl.instrument_openai()
+    ```
+
+3. **Set Configuration**
+
+    Define the model settings for the assistant.
+
+    ```python
+    settings = {
+        "model": "llama3-8b-8192",
+        "temperature": 0.5,
+        "max_tokens": 500,
+        "top_p": 1,
+        "frequency_penalty": 0,
+        "presence_penalty": 0,
+    }
+    ```
+
+4. **Define Utility Functions**
+
+    - **Extract URLs from Text**: Use regex to find URLs in messages.
+
+        ```python
+        def extract_urls(text):
+            url_pattern = re.compile(r'(https?://\S+)')
+            return url_pattern.findall(text)
+        ```
+
+    - **Crawl URL**: Send a request to `Crawl4AI` to fetch the content of a URL.
+
+        ```python
+        def crawl_url(url):
+            data = {
+                "urls": [url],
+                "include_raw_html": True,
+                "word_count_threshold": 10,
+                "extraction_strategy": "NoExtractionStrategy",
+                "chunking_strategy": "RegexChunking"
+            }
+            response = requests.post("https://crawl4ai.com/crawl", json=data)
+            response_data = response.json()
+            response_data = response_data['results'][0]
+            return response_data['markdown']
+        ```
+
+5. **Initialize Chat Start Event**
+
+    Set up the initial chat message and user session.
+
+    ```python
+    @cl.on_chat_start
+    async def on_chat_start():
+        cl.user_session.set("session", {
+            "history": [],
+            "context": {}
+        })  
+        await cl.Message(
+            content="Welcome to the chat! How can I assist you today?"
+        ).send()
+    ```
+
+6. **Handle Incoming Messages**
+
+    Process user messages, extract URLs, and crawl them concurrently. Update the chat history and system message.
+
+    ```python
+    @cl.on_message
+    async def on_message(message: cl.Message):
+        user_session = cl.user_session.get("session")
+
+        # Extract URLs from the user's message
+        urls = extract_urls(message.content)
+
+        futures = []
+        with ThreadPoolExecutor() as executor:
+            for url in urls:
+                futures.append(executor.submit(crawl_url, url))
+
+        results = [future.result() for future in futures]
+
+        for url, result in zip(urls, results):
+            ref_number = f"REF_{len(user_session['context']) + 1}"
+            user_session["context"][ref_number] = {
+                "url": url,
+                "content": result
+            }    
+
+        user_session["history"].append({
+            "role": "user",
+            "content": message.content
+        })
+
+        # Create a system message that includes the context
+        context_messages = [
+            f'<appendix ref="{ref}">\n{data["content"]}\n</appendix>'
+            for ref, data in user_session["context"].items()
+        ]
+        if context_messages:
+            system_message = {
+                "role": "system",
+                "content": (
+                    "You are a helpful bot. Use the following context for answering questions. "
+                    "Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
+                    "If the question requires any information from the provided appendices or context, refer to the sources. "
+                    "If not, there is no need to add a references section. "
+                    "At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
+                    "\n\n".join(context_messages)
+                )
+            }
+        else:
+            system_message = {
+                "role": "system",
+                "content": "You are a helpful assistant."
+            }
+
+        msg = cl.Message(content="")
+        await msg.send()
+
+        # Get response from the LLM
+        stream = await client.chat.completions.create(
+            messages=[
+                system_message,
+                *user_session["history"]
+            ],
+            stream=True,
+            **settings
+        )
+
+        assistant_response = ""
+        async for part in stream:
+            if token := part.choices[0].delta.content:
+                assistant_response += token
+                await msg.stream_token(token)
+
+        # Add assistant message to the history
+        user_session["history"].append({
+            "role": "assistant",
+            "content": assistant_response
+        })
+        await msg.update()
+
+        # Append the reference section to the assistant's response
+        reference_section = "\n\nReferences:\n"
+        for ref, data in user_session["context"].items():
+            reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
+
+        msg.content += reference_section
+        await msg.update()
+    ```
+
+7. **Handle Audio Input**
+
+    Capture and transcribe audio input. Store the audio buffer and transcribe it when the audio ends.
+
+    ```python
+    @cl.on_audio_chunk
+    async def on_audio_chunk(chunk: cl.AudioChunk):
+        if chunk.isStart:
+            buffer = BytesIO()
+            buffer.name = f"input_audio.{chunk.mimeType.split('/')[1]}"
+            cl.user_session.set("audio_buffer", buffer)
+            cl.user_session.set("audio_mime_type", chunk.mimeType)
+
+        cl.user_session.get("audio_buffer").write(chunk.data)
+
+    @cl.step(type="tool")
+    async def speech_to_text(audio_file):
+        cli = Groq()
+        response = await client.audio.transcriptions.create(
+            model="whisper-large-v3", file=audio_file
+        )
+        return response.text
+
+    @cl.on_audio_end
+    async def on_audio_end(elements: list[ElementBased]):
+        audio_buffer: BytesIO = cl.user_session.get("audio_buffer")
+        audio_buffer.seek(0)
+        audio_file = audio_buffer.read()
+        audio_mime_type: str = cl.user_session.get("audio_mime_type")
+        
+        start_time = time.time()
+        transcription = await speech_to_text((audio_buffer.name, audio_file, audio_mime_type))
+        end_time = time.time()
+        print(f"Transcription took {end_time - start_time} seconds")
+        
+        user_msg = cl.Message(
+            author="You", 
+            type="user_message",
+            content=transcription
+        )
+        await user_msg.send()
+        await on_message(user_msg)
+    ```
+
+8. **Run the Chat Application**
+
+    Start the Chainlit application.
+
+    ```python
+    if __name__ == "__main__":
+        from chainlit.cli import run_chainlit
+        run_chainlit(__file__)
+    ```
+
+### Explanation
+
+- **Libraries and Configuration**: Import necessary libraries and configure the OpenAI client.
+- **Utility Functions**: Define functions to extract URLs and crawl them.
+- **Chat Start Event**: Initialize chat session and welcome message.
+- **Message Handling**: Extract URLs, crawl them concurrently, and update chat history and context.
+- **Audio Handling**: Capture, buffer, and transcribe audio input, then process the transcription as text.
+- **Running the Application**: Start the Chainlit server to interact with the assistant.
+
+This example showcases how to create an interactive research assistant that can fetch, process, and summarize web content, along with handling audio inputs for a seamless user experience.
--- a/_sync/examples/summarization.md
+++ b/_sync/examples/summarization.md
@@ -0,0 +1,108 @@
+## Summarization Example
+
+This example demonstrates how to use `Crawl4AI` to extract a summary from a web page. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.
+
+### Step-by-Step Guide
+
+1. **Import Necessary Modules**
+
+    First, import the necessary modules and classes.
+
+    ```python
+    import os
+    import time
+    import json
+    from crawl4ai.web_crawler import WebCrawler
+    from crawl4ai.chunking_strategy import *
+    from crawl4ai.extraction_strategy import *
+    from crawl4ai.crawler_strategy import *
+    from pydantic import BaseModel, Field
+    ```
+
+2. **Define the URL to be Crawled**
+
+    Set the URL of the web page you want to summarize.
+
+    ```python
+    url = r'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
+    ```
+
+3. **Initialize the WebCrawler**
+
+    Create an instance of the `WebCrawler` and call the `warmup` method.
+
+    ```python
+    crawler = WebCrawler()
+    crawler.warmup()
+    ```
+
+4. **Define the Data Model**
+
+    Use Pydantic to define the structure of the extracted data.
+
+    ```python
+    class PageSummary(BaseModel):
+        title: str = Field(..., description="Title of the page.")
+        summary: str = Field(..., description="Summary of the page.")
+        brief_summary: str = Field(..., description="Brief summary of the page.")
+        keywords: list = Field(..., description="Keywords assigned to the page.")
+    ```
+
+5. **Run the Crawler**
+
+    Set up and run the crawler with the `LLMExtractionStrategy`. Provide the necessary parameters, including the schema for the extracted data and the instruction for the LLM.
+
+    ```python
+    result = crawler.run(
+        url=url,
+        word_count_threshold=1,
+        extraction_strategy=LLMExtractionStrategy(
+            provider="openai/gpt-4o", 
+            api_token=os.getenv('OPENAI_API_KEY'), 
+            schema=PageSummary.model_json_schema(),
+            extraction_type="schema",
+            apply_chunking=False,
+            instruction=(
+                "From the crawled content, extract the following details: "
+                "1. Title of the page "
+                "2. Summary of the page, which is a detailed summary "
+                "3. Brief summary of the page, which is a paragraph text "
+                "4. Keywords assigned to the page, which is a list of keywords. "
+                'The extracted JSON format should look like this: '
+                '{ "title": "Page Title", "summary": "Detailed summary of the page.", '
+                '"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
+            )
+        ),
+        bypass_cache=True,
+    )
+    ```
+
+6. **Process the Extracted Data**
+
+    Load the extracted content into a JSON object and print it.
+
+    ```python
+    page_summary = json.loads(result.extracted_content)
+    print(page_summary)
+    ```
+
+7. **Save the Extracted Data**
+
+    Save the extracted data to a file for further use.
+
+    ```python
+    with open(".data/page_summary.json", "w", encoding="utf-8") as f:
+        f.write(result.extracted_content)
+    ```
+
+### Explanation
+
+- **Importing Modules**: Import the necessary modules, including `WebCrawler` and `LLMExtractionStrategy` from `Crawl4AI`.
+- **URL Definition**: Set the URL of the web page you want to crawl and summarize.
+- **WebCrawler Initialization**: Create an instance of `WebCrawler` and call the `warmup` method to prepare the crawler.
+- **Data Model Definition**: Define the structure of the data you want to extract using Pydantic's `BaseModel`.
+- **Crawler Execution**: Run the crawler with the `LLMExtractionStrategy`, providing the schema and detailed instructions for the extraction process.
+- **Data Processing**: Load the extracted content into a JSON object and print it to verify the results.
+- **Data Saving**: Save the extracted data to a file for further use.
+
+This example demonstrates how to harness the power of `Crawl4AI` to perform advanced web crawling and data extraction tasks with minimal code.
--- a/_sync/full_details/advanced_features.md
+++ b/_sync/full_details/advanced_features.md
@@ -0,0 +1,138 @@
+# Advanced Features
+
+Crawl4AI offers a range of advanced features that allow you to fine-tune your web crawling and data extraction process. This section will cover some of these advanced features, including taking screenshots, extracting media and links, customizing the user agent, using custom hooks, and leveraging CSS selectors.
+
+## Taking Screenshots 📸
+
+One of the cool features of Crawl4AI is the ability to take screenshots of the web pages you're crawling. This can be particularly useful for visual verification or for capturing the state of dynamic content.
+
+Here's how you can take a screenshot:
+
+```python
+from crawl4ai import WebCrawler
+import base64
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with the screenshot parameter
+result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
+
+# Save the screenshot to a file
+with open("screenshot.png", "wb") as f:
+    f.write(base64.b64decode(result.screenshot))
+
+print("Screenshot saved to 'screenshot.png'!")
+```
+
+In this example, we create a `WebCrawler` instance, warm it up, and then run it with the `screenshot` parameter set to `True`. The screenshot is saved as a base64 encoded string in the result, which we then decode and save as a PNG file.
+
+## Extracting Media and Links 🎨🔗
+
+Crawl4AI can extract all media tags (images, audio, and video) and links (both internal and external) from a web page. This feature is useful for collecting multimedia content or analyzing link structures.
+
+Here's an example:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler
+result = crawler.run(url="https://www.nbcnews.com/business")
+
+print("Extracted media:", result.media)
+print("Extracted links:", result.links)
+```
+
+In this example, the `result` object contains dictionaries for media and links, which you can access and use as needed.
+
+## Customizing the User Agent 🕵️‍♂️
+
+Crawl4AI allows you to set a custom user agent for your HTTP requests. This can help you avoid detection by web servers or simulate different browsing environments.
+
+Here's how to set a custom user agent:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with a custom user agent
+result = crawler.run(url="https://www.nbcnews.com/business", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
+
+print("Crawl result:", result)
+```
+
+In this example, we specify a custom user agent string when running the crawler.
+
+## Using Custom Hooks 🪝
+
+Hooks are a powerful feature in Crawl4AI that allow you to customize the crawling process at various stages. You can define hooks for actions such as driver initialization, before and after URL fetching, and before returning the HTML.
+
+Here's an example of using hooks:
+
+```python
+from crawl4ai import WebCrawler
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+# Define the hooks
+def on_driver_created(driver):
+    driver.maximize_window()
+    driver.get('https://example.com/login')
+    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.NAME, 'username'))).send_keys('testuser')
+    driver.find_element(By.NAME, 'password').send_keys('password123')
+    driver.find_element(By.NAME, 'login').click()
+    return driver
+
+def before_get_url(driver):
+    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
+    return driver
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Set the hooks
+crawler.set_hook('on_driver_created', on_driver_created)
+crawler.set_hook('before_get_url', before_get_url)
+
+# Run the crawler
+result = crawler.run(url="https://example.com")
+
+print("Crawl result:", result)
+```
+
+In this example, we define hooks to handle driver initialization and custom headers before fetching the URL.
+
+## Using CSS Selectors 🎯
+
+CSS selectors allow you to target specific elements on a web page for extraction. This can be useful for scraping structured content, such as articles or product details.
+
+Here's an example of using a CSS selector:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+crawler.warmup()
+
+# Run the crawler with a CSS selector to extract only H2 tags
+result = crawler.run(url="https://www.nbcnews.com/business", css_selector="h2")
+
+print("Extracted H2 tags:", result.extracted_content)
+```
+
+In this example, we use the `css_selector` parameter to extract only the H2 tags from the web page.
+
+---
+
+With these advanced features, you can leverage Crawl4AI to perform sophisticated web crawling and data extraction tasks. Whether you need to take screenshots, extract specific elements, customize the crawling process, or set custom headers, Crawl4AI provides the flexibility and power to meet your needs. Happy crawling! 🕷️🚀
--- a/_sync/full_details/chunking_strategies.md
+++ b/_sync/full_details/chunking_strategies.md
@@ -0,0 +1,133 @@
+## Chunking Strategies 📚
+
+Crawl4AI provides several powerful chunking strategies to divide text into manageable parts for further processing. Each strategy has unique characteristics and is suitable for different scenarios. Let's explore them one by one.
+
+### RegexChunking
+
+`RegexChunking` splits text using regular expressions. This is ideal for creating chunks based on specific patterns like paragraphs or sentences.
+
+#### When to Use
+- Great for structured text with consistent delimiters.
+- Suitable for documents where specific patterns (e.g., double newlines, periods) indicate logical chunks.
+
+#### Parameters
+- `patterns` (list, optional): Regular expressions used to split the text. Default is to split by double newlines (`['\n\n']`).
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+# Define patterns for splitting text
+patterns = [r'\n\n', r'\. ']
+chunker = RegexChunking(patterns=patterns)
+
+# Sample text
+text = "This is a sample text. It will be split into chunks.\n\nThis is another paragraph."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### NlpSentenceChunking
+
+`NlpSentenceChunking` uses NLP models to split text into sentences, ensuring accurate sentence boundaries.
+
+#### When to Use
+- Ideal for texts where sentence boundaries are crucial.
+- Useful for creating chunks that preserve grammatical structures.
+
+#### Parameters
+- None.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+chunker = NlpSentenceChunking()
+
+# Sample text
+text = "This is a sample text. It will be split into sentences. Here's another sentence."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### TopicSegmentationChunking
+
+`TopicSegmentationChunking` employs the TextTiling algorithm to segment text into topic-based chunks. This method identifies thematic boundaries.
+
+#### When to Use
+- Perfect for long documents with distinct topics.
+- Useful when preserving topic continuity is more important than maintaining text order.
+
+#### Parameters
+- `num_keywords` (int, optional): Number of keywords for each topic segment. Default is `3`.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import TopicSegmentationChunking
+
+chunker = TopicSegmentationChunking(num_keywords=3)
+
+# Sample text
+text = "This document contains several topics. Topic one discusses AI. Topic two covers machine learning."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### FixedLengthWordChunking
+
+`FixedLengthWordChunking` splits text into chunks based on a fixed number of words. This ensures each chunk has approximately the same length.
+
+#### When to Use
+- Suitable for processing large texts where uniform chunk size is important.
+- Useful when the number of words per chunk needs to be controlled.
+
+#### Parameters
+- `chunk_size` (int, optional): Number of words per chunk. Default is `100`.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import FixedLengthWordChunking
+
+chunker = FixedLengthWordChunking(chunk_size=10)
+
+# Sample text
+text = "This is a sample text. It will be split into chunks of fixed length."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+### SlidingWindowChunking
+
+`SlidingWindowChunking` uses a sliding window approach to create overlapping chunks. Each chunk has a fixed length, and the window slides by a specified step size.
+
+#### When to Use
+- Ideal for creating overlapping chunks to preserve context.
+- Useful for tasks where context from adjacent chunks is needed.
+
+#### Parameters
+- `window_size` (int, optional): Number of words in each chunk. Default is `100`.
+- `step` (int, optional): Number of words to slide the window. Default is `50`.
+
+#### Example
+```python
+from crawl4ai.chunking_strategy import SlidingWindowChunking
+
+chunker = SlidingWindowChunking(window_size=10, step=5)
+
+# Sample text
+text = "This is a sample text. It will be split using a sliding window approach to preserve context."
+
+# Chunk the text
+chunks = chunker.chunk(text)
+print(chunks)
+```
+
+With these chunking strategies, you can choose the best method to divide your text based on your specific needs. Whether you need precise sentence boundaries, topic-based segmentation, or uniform chunk sizes, Crawl4AI has you covered. Happy chunking! 📝✨
--- a/_sync/full_details/crawl_request_parameters.md
+++ b/_sync/full_details/crawl_request_parameters.md
@@ -0,0 +1,130 @@
+# Crawl Request Parameters
+
+The `run` function in Crawl4AI is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `run` function, along with their descriptions, possible values, and examples.
+
+## Parameters
+
+### url (str)
+**Description:** The URL of the webpage to crawl.
+**Required:** Yes
+**Example:**
+```python
+url = "https://www.nbcnews.com/business"
+```
+
+### word_count_threshold (int)
+**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is `5`.
+**Required:** No
+**Default Value:** `5`
+**Example:**
+```python
+word_count_threshold = 10
+```
+
+### extraction_strategy (ExtractionStrategy)
+**Description:** The strategy to use for extracting content from the HTML. It must be an instance of `ExtractionStrategy`. If not provided, the default is `NoExtractionStrategy`.
+**Required:** No
+**Default Value:** `NoExtractionStrategy()`
+**Example:**
+```python
+extraction_strategy = CosineStrategy(semantic_filter="finance")
+```
+
+### chunking_strategy (ChunkingStrategy)
+**Description:** The strategy to use for chunking the text before processing. It must be an instance of `ChunkingStrategy`. The default value is `RegexChunking()`.
+**Required:** No
+**Default Value:** `RegexChunking()`
+**Example:**
+```python
+chunking_strategy = NlpSentenceChunking()
+```
+
+### bypass_cache (bool)
+**Description:** Whether to force a fresh crawl even if the URL has been previously crawled. The default value is `False`.
+**Required:** No
+**Default Value:** `False`
+**Example:**
+```python
+bypass_cache = True
+```
+
+### css_selector (str)
+**Description:** The CSS selector to target specific parts of the HTML for extraction. If not provided, the entire HTML will be processed.
+**Required:** No
+**Default Value:** `None`
+**Example:**
+```python
+css_selector = "div.article-content"
+```
+
+### screenshot (bool)
+**Description:** Whether to take screenshots of the page. The default value is `False`.
+**Required:** No
+**Default Value:** `False`
+**Example:**
+```python
+screenshot = True
+```
+
+### user_agent (str)
+**Description:** The user agent to use for the HTTP requests. If not provided, a default user agent will be used.
+**Required:** No
+**Default Value:** `None`
+**Example:**
+```python
+user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
+```
+
+### verbose (bool)
+**Description:** Whether to enable verbose logging. The default value is `True`.
+**Required:** No
+**Default Value:** `True`
+**Example:**
+```python
+verbose = True
+```
+
+### **kwargs
+Additional keyword arguments that can be passed to customize the crawling process further. Some notable options include:
+
+- **only_text (bool):** Whether to extract only text content, excluding HTML tags. Default is `False`.
+
+**Example:**
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    css_selector="p",
+    only_text=True
+)
+```
+
+## Example Usage
+
+Here's an example of how to use the `run` function with various parameters:
+
+```python
+from crawl4ai import WebCrawler
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+# Create the WebCrawler instance 
+crawler = WebCrawler() 
+
+# Run the crawler with custom parameters
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    word_count_threshold=10,
+    extraction_strategy=CosineStrategy(semantic_filter="finance"),
+    chunking_strategy=NlpSentenceChunking(),
+    bypass_cache=True,
+    css_selector="div.article-content",
+    screenshot=True,
+    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
+    verbose=True,
+    only_text=True
+)
+
+print(result)
+```
+
+This example demonstrates how to configure various parameters to customize the crawling and extraction process using Crawl4AI.
--- a/_sync/full_details/crawl_result_class.md
+++ b/_sync/full_details/crawl_result_class.md
@@ -0,0 +1,120 @@
+# Crawl Result
+
+The `CrawlResult` class is the heart of Crawl4AI's output, encapsulating all the data extracted from a crawling session. This class contains various fields that store the results of the web crawling and extraction process. Let's break down each field and see what it holds. 🎉
+
+## Class Definition
+
+```python
+class CrawlResult(BaseModel):
+    url: str
+    html: str
+    success: bool
+    cleaned_html: Optional[str] = None
+    media: Dict[str, List[Dict]] = {}
+    links: Dict[str, List[Dict]] = {}
+    screenshot: Optional[str] = None
+    markdown: Optional[str] = None
+    extracted_content: Optional[str] = None
+    metadata: Optional[dict] = None
+    error_message: Optional[str] = None
+```
+
+## Fields Explanation
+
+### `url: str`
+The URL that was crawled. This field simply stores the URL of the web page that was processed.
+
+### `html: str`
+The raw HTML content of the web page. This is the unprocessed HTML source as retrieved by the crawler.
+
+### `success: bool`
+A flag indicating whether the crawling and extraction were successful. If any error occurs during the process, this will be `False`.
+
+### `cleaned_html: Optional[str]`
+The cleaned HTML content of the web page. This field holds the HTML after removing unwanted tags like `<script>`, `<style>`, and others that do not contribute to the useful content.
+
+### `media: Dict[str, List[Dict]]`
+A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here’s how they are structured:
+
+- **Images**: Each image is represented as a dictionary with `src` (source URL) and `alt` (alternate text).
+- **Videos**: Each video is represented similarly with `src` and `alt`.
+- **Audios**: Each audio is represented with `src` and `alt`.
+
+```python
+media = {
+    'images': [
+        {'src': 'image_url1', 'alt': 'description1', "type": "image"},
+        {'src': 'image_url2', 'alt': 'description2', "type": "image"}
+    ],
+    'videos': [
+        {'src': 'video_url1', 'alt': 'description1', "type": "video"}
+    ],
+    'audios': [
+        {'src': 'audio_url1', 'alt': 'description1', "type": "audio"}
+    ]
+}
+```
+
+### `links: Dict[str, List[Dict]]`
+A dictionary containing lists of internal and external links extracted from the web page. Each link is represented as a dictionary with `href` (URL) and `text` (link text).
+
+- **Internal Links**: Links pointing to the same domain.
+- **External Links**: Links pointing to different domains.
+
+```python
+links = {
+    'internal': [
+        {'href': 'internal_link1', 'text': 'link_text1'},
+        {'href': 'internal_link2', 'text': 'link_text2'}
+    ],
+    'external': [
+        {'href': 'external_link1', 'text': 'link_text1'}
+    ]
+}
+```
+
+### `screenshot: Optional[str]`
+A base64-encoded screenshot of the web page. This field stores the screenshot data if the crawling was configured to take a screenshot.
+
+### `markdown: Optional[str]`
+The content of the web page converted to Markdown format. This is useful for generating clean, readable text that retains the structure of the original HTML.
+
+### `extracted_content: Optional[str]`
+The content extracted based on the specified extraction strategy. This field holds the meaningful content blocks extracted from the web page, ready for your AI and data processing needs.
+
+### `metadata: Optional[dict]`
+A dictionary containing metadata extracted from the web page, such as title, description, keywords, and other meta tags.
+
+### `error_message: Optional[str]`
+If an error occurs during crawling, this field will contain the error message, helping you debug and understand what went wrong. 🚨
+
+## Example Usage
+
+Here's a quick example to illustrate how you might use the `CrawlResult` in your code:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create the WebCrawler instance
+crawler = WebCrawler()
+
+# Run the crawler on a URL
+result = crawler.run(url="https://www.example.com")
+
+# Check if the crawl was successful
+if result.success:
+    print("Crawl succeeded!")
+    print("URL:", result.url)
+    print("HTML:", result.html[:100])  # Print the first 100 characters of the HTML
+    print("Cleaned HTML:", result.cleaned_html[:100])
+    print("Media:", result.media)
+    print("Links:", result.links)
+    print("Screenshot:", result.screenshot)
+    print("Markdown:", result.markdown[:100])
+    print("Extracted Content:", result.extracted_content)
+    print("Metadata:", result.metadata)
+else:
+    print("Crawl failed with error:", result.error_message)
+```
+
+With this setup, you can easily access all the valuable data extracted from the web page and integrate it into your applications. Happy crawling! 🕷️🤖
--- a/_sync/full_details/extraction_strategies.md
+++ b/_sync/full_details/extraction_strategies.md
@@ -0,0 +1,116 @@
+## Extraction Strategies 🧠
+
+Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into two of the most important strategies: `CosineStrategy` and `LLMExtractionStrategy`.
+
+### CosineStrategy
+
+`CosineStrategy` uses hierarchical clustering based on cosine similarity to group text chunks into meaningful clusters. This method converts each chunk into its embedding and then clusters them to form semantical chunks.
+
+#### When to Use
+- Ideal for fast, accurate semantic segmentation of text.
+- Perfect for scenarios where LLMs might be overkill or too slow.
+- Suitable for narrowing down content based on specific queries or keywords.
+
+#### Parameters
+- `semantic_filter` (str, optional): Keywords for filtering relevant documents before clustering. Documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.
+- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.
+- `max_dist` (float, optional): Maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.
+- `linkage_method` (str, optional): Linkage method for hierarchical clustering. Default is `'ward'`.
+- `top_k` (int, optional): Number of top categories to extract. Default is `3`.
+- `model_name` (str, optional): Model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.
+
+#### Example
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+# Define extraction strategy
+strategy = CosineStrategy(
+    semantic_filter="finance economy stock market",
+    word_count_threshold=10,
+    max_dist=0.2,
+    linkage_method='ward',
+    top_k=3,
+    model_name='BAAI/bge-small-en-v1.5'
+)
+
+# Sample URL
+url = "https://www.nbcnews.com/business"
+
+# Run the crawler with the extraction strategy
+result = crawler.run(url=url, extraction_strategy=strategy)
+print(result.extracted_content)
+```
+
+### LLMExtractionStrategy
+
+`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
+
+#### When to Use
+- Suitable for complex extraction tasks requiring nuanced understanding.
+- Ideal for scenarios where detailed instructions can guide the extraction process.
+- Perfect for extracting specific types of information or content with precise guidelines.
+
+#### Parameters
+- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
+- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
+- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
+
+#### Example Without Instructions
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+# Define extraction strategy without instructions
+strategy = LLMExtractionStrategy(
+    provider='openai',
+    api_token='your_api_token'
+)
+
+# Sample URL
+url = "https://www.nbcnews.com/business"
+
+# Run the crawler with the extraction strategy
+result = crawler.run(url=url, extraction_strategy=strategy)
+print(result.extracted_content)
+```
+
+#### Example With Instructions
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+from crawl4ai import WebCrawler
+
+crawler = WebCrawler()
+crawler.warmup()
+
+# Define extraction strategy with instructions
+strategy = LLMExtractionStrategy(
+    provider='openai',
+    api_token='your_api_token',
+    instruction="Extract only financial news and summarize key points."
+)
+
+# Sample URL
+url = "https://www.nbcnews.com/business"
+
+# Run the crawler with the extraction strategy
+result = crawler.run(url=url, extraction_strategy=strategy)
+print(result.extracted_content)
+```
+
+#### Use Cases for LLMExtractionStrategy
+- Extracting specific data types from structured or semi-structured content.
+- Generating summaries, extracting key information, or transforming content into different formats.
+- Performing detailed extractions based on custom instructions.
+
+For more detailed examples, please refer to the [Examples section](../examples/index.md) of the documentation.
+
+---
+
+By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy` or nuanced, instruction-based extraction with `LLMExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨
--- a/_sync/index.md
+++ b/_sync/index.md
@@ -0,0 +1,101 @@
+# Crawl4AI v0.2.77
+
+Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
+
+
+## Try the [Demo](demo.md)
+
+Just try it now and crawl different pages to see how it works. You can set the links, see the structures of the output, and also view the Python sample code on how to run it. The old demo is available at [/old_demo](/old) where you can see more details.
+
+## Introduction
+
+Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution.
+
+## Quick Start
+
+Here's a quick example to show you how easy it is to use Crawl4AI:
+
+```python
+from crawl4ai import WebCrawler
+
+# Create an instance of WebCrawler
+crawler = WebCrawler()
+
+# Warm up the crawler (load necessary models)
+crawler.warmup()
+
+# Run the crawler on a URL
+result = crawler.run(url="https://www.nbcnews.com/business")
+
+# Print the extracted content
+print(result.extracted_content)
+```
+
+### Explanation
+
+1. **Importing the Library**: We start by importing the `WebCrawler` class from the `crawl4ai` library.
+2. **Creating an Instance**: An instance of `WebCrawler` is created.
+3. **Warming Up**: The `warmup()` method prepares the crawler by loading necessary models and settings.
+4. **Running the Crawler**: The `run()` method is used to crawl the specified URL and extract meaningful content.
+5. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
+
+## Documentation Structure
+
+This documentation is organized into several sections to help you navigate and find the information you need quickly:
+
+### [Home](index.md)
+
+An introduction to Crawl4AI, including a quick start guide and an overview of the documentation structure.
+
+### [Installation](installation.md)
+
+Instructions on how to install Crawl4AI and its dependencies.
+
+### [Introduction](introduction.md)
+
+A detailed introduction to Crawl4AI, its features, and how it can be used for various web crawling and data extraction tasks.
+
+### [Quick Start](quickstart.md)
+
+A step-by-step guide to get you up and running with Crawl4AI, including installation instructions and basic usage examples.
+
+### [Examples](examples/index.md)
+
+This section contains practical examples demonstrating different use cases of Crawl4AI:
+
+- [LLM Extraction](examples/llm_extraction.md)
+- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
+- [Hooks & Auth](examples/hooks_auth.md)
+- [Summarization](examples/summarization.md)
+- [Research Assistant](examples/research_assistant.md)
+
+### [Full Details of Using Crawler](full_details/crawl_request_parameters.md)
+
+Comprehensive details on using the crawler, including:
+
+- [Crawl Request Parameters](full_details/crawl_request_parameters.md)
+- [Crawl Result Class](full_details/crawl_result_class.md)
+- [Advanced Features](full_details/advanced_features.md)
+- [Chunking Strategies](full_details/chunking_strategies.md)
+- [Extraction Strategies](full_details/extraction_strategies.md)
+
+### [API Reference](api/core_classes_and_functions.md)
+
+Detailed documentation of the API, covering:
+
+- [Core Classes and Functions](api/core_classes_and_functions.md)
+- [Detailed API Documentation](api/detailed_api_documentation.md)
+
+### [Change Log](changelog.md)
+
+A log of all changes, updates, and improvements made to Crawl4AI.
+
+### [Contact](contact.md)
+
+Information on how to get in touch with the developers, report issues, and contribute to the project.
+
+## Get Started
+
+To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier and more efficient.
+
+Happy Crawling! 🕸️🚀
--- a/_sync/installation.md
+++ b/_sync/installation.md
@@ -0,0 +1,193 @@
+# Installation 💻
+
+There are three ways to use Crawl4AI:
+
+1. As a library (Recommended).
+2. As a local server (Docker) or using the REST API.
+3. As a local server (Docker) using the pre-built image from Docker Hub.
+
+## Option 1: Library Installation
+
+You can try this Colab for a quick start: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sJPAmeLj5PMrg2VgOwMJ2ubGIcK0cJeX#scrollTo=g1RrmI4W_rPk)
+
+Crawl4AI offers flexible installation options to suit various use cases. Choose the option that best fits your needs:
+
+- **Default Installation** (Basic functionality):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai @ git+https://github.com/unclecode/crawl4ai.git"
+```
+Use this for basic web crawling and scraping tasks.
+
+- **Installation with PyTorch** (For advanced text clustering):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai[torch] @ git+https://github.com/unclecode/crawl4ai.git"
+```
+Choose this if you need the CosineSimilarity cluster strategy.
+
+- **Installation with Transformers** (For summarization and Hugging Face models):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai[transformer] @ git+https://github.com/unclecode/crawl4ai.git"
+```
+Opt for this if you require text summarization or plan to use Hugging Face models.
+
+- **Full Installation** (All features):
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install "crawl4ai[all] @ git+https://github.com/unclecode/crawl4ai.git"
+```
+This installs all dependencies for full functionality.
+
+- **Development Installation** (For contributors):
+```bash
+virtualenv venv
+source venv/bin/activate
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+pip install -e ".[all]"
+```
+Use this if you plan to modify the source code.
+
+💡 After installation, if you have used "torch", "transformer" or "all", it's recommended to run the following CLI command to load the required models. This is optional but will boost the performance and speed of the crawler. You need to do this only once, this is only for when you install using []
+```bash
+crawl4ai-download-models
+```
+
+## Option 2: Using Docker for Local Server
+
+Crawl4AI can be run as a local server using Docker. The Dockerfile supports different installation options to cater to various use cases. Here's how you can build and run the Docker image:
+
+### Default Installation
+
+The default installation includes the basic Crawl4AI package without additional dependencies or pre-downloaded models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 -t crawl4ai .
+
+# For other users
+docker build -t crawl4ai .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai
+```
+
+### Full Installation (All Dependencies and Models)
+
+This option installs all dependencies and downloads the models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 --build-arg INSTALL_OPTION=all -t crawl4ai:all .
+
+# For other users
+docker build --build-arg INSTALL_OPTION=all -t crawl4ai:all .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai:all
+```
+
+### Torch Installation
+
+This option installs torch-related dependencies and downloads the models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
+
+# For other users
+docker build --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai:torch
+```
+
+### Transformer Installation
+
+This option installs transformer-related dependencies and downloads the models.
+
+```bash
+# For Mac users (M1/M2)
+docker build --platform linux/amd64 --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
+
+# For other users
+docker build --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
+
+# Run the container
+docker run -d -p 8000:80 crawl4ai:transformer
+```
+
+### Notes
+
+- The `--platform linux/amd64` flag is necessary for Mac users with M1/M2 chips to ensure compatibility.
+- The `-t` flag tags the image with a name (and optionally a tag in the 'name:tag' format).
+- The `-d` flag runs the container in detached mode.
+- The `-p 8000:80` flag maps port 8000 on the host to port 80 in the container.
+
+Choose the installation option that best suits your needs. The default installation is suitable for basic usage, while the other options provide additional capabilities for more advanced use cases.
+
+## Option 3: Using the Pre-built Image from Docker Hub
+
+You can use pre-built Crawl4AI images from Docker Hub, which are available for all platforms (Mac, Linux, Windows). We have official images as well as a community-contributed image (Thanks to https://github.com/FractalMind):
+
+### Default Installation
+
+```bash
+
+# Pull the image
+
+docker pull unclecode/crawl4ai:latest
+
+# Run the container
+
+docker run -d -p 8000:80 unclecode/crawl4ai:latest
+
+```
+
+### Community-Contributed Image
+
+A stable version of Crawl4AI is also available, created and maintained by a community member:
+
+```bash
+
+# Pull the community-contributed image
+
+docker pull ryser007/crawl4ai:stable
+
+# Run the container
+
+docker run -d -p 8000:80 ryser007/crawl4ai:stable
+
+```
+
+We'd like to express our gratitude to GitHub user [@FractalMind](https://github.com/FractalMind) for creating and maintaining this stable version of the Crawl4AI Docker image. Community contributions like this are invaluable to the project.
+
+
+### Testing the Installation
+
+After running the container, you can test if it's working correctly:
+
+- On Mac and Linux:
+
+  ```bash
+
+  curl http://localhost:8000
+
+  ```
+
+- On Windows (PowerShell):
+
+  ```powershell
+
+  Invoke-WebRequest -Uri http://localhost:8000
+
+  ```
+
+  Or open a web browser and navigate to http://localhost:8000
+
--- a/_sync/interactive_content.html
+++ b/_sync/interactive_content.html
@@ -0,0 +1,28 @@
+<h1>Try Our Library</h1>
+<form id="apiForm">
+    <label for="inputField">Enter some input:</label>
+    <input type="text" id="inputField" name="inputField" required>
+    <button type="submit">Submit</button>
+</form>
+<div id="result"></div>
+
+<script>
+    document.getElementById('apiForm').addEventListener('submit', function(event) {
+        event.preventDefault();
+        const input = document.getElementById('inputField').value;
+        fetch('https://your-api-endpoint.com/api', {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json'
+            },
+            body: JSON.stringify({ input: input })
+        })
+        .then(response => response.json())
+        .then(data => {
+            document.getElementById('result').textContent = JSON.stringify(data);
+        })
+        .catch(error => {
+            document.getElementById('result').textContent = 'Error: ' + error;
+        });
+    });
+</script>
--- a/_sync/introduction.md
+++ b/_sync/introduction.md
@@ -0,0 +1,29 @@
+# Introduction
+
+Welcome to the documentation for Crawl4AI v0.2.5! 🕷️🤖
+
+Crawl4AI is designed to simplify the process of crawling web pages and extracting useful information for large language models (LLMs) and AI applications. Whether you're using it as a REST API, a Python library, or through a Google Colab notebook, Crawl4AI provides powerful features to make web data extraction easier and more efficient.
+
+## Key Features ✨
+
+- **🆓 Completely Free and Open-Source**: Crawl4AI is free to use and open-source, making it accessible for everyone.
+- **🤖 LLM-Friendly Output Formats**: Supports JSON, cleaned HTML, and markdown formats.
+- **🌍 Concurrent Crawling**: Crawl multiple URLs simultaneously to save time.
+- **🎨 Media Extraction**: Extract all media tags including images, audio, and video.
+- **🔗 Link Extraction**: Extract all external and internal links from web pages.
+- **📚 Metadata Extraction**: Extract metadata from web pages for additional context.
+- **🔄 Custom Hooks**: Define custom hooks for authentication, headers, and page modifications before crawling.
+- **🕵️ User Agent Support**: Customize the user agent for HTTP requests.
+- **🖼️ Screenshot Capability**: Take screenshots of web pages during crawling.
+- **📜 JavaScript Execution**: Execute custom JavaScripts before crawling.
+- **📚 Advanced Chunking and Extraction Strategies**: Utilize topic-based, regex, sentence chunking, cosine clustering, and LLM extraction strategies.
+- **🎯 CSS Selector Support**: Extract specific content using CSS selectors.
+- **📝 Instruction/Keyword Refinement**: Pass instructions or keywords to refine the extraction process.
+
+Check the [Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for more details.
+
+## Power and Simplicity of Crawl4AI 🚀
+
+Crawl4AI provides an easy way to crawl and extract data from web pages without installing any library. You can use the REST API on our server or run the local server on your machine. For more advanced control, use the Python library to customize your crawling and extraction strategies.
+
+Explore the documentation to learn more about the features, installation process, usage examples, and how to contribute to Crawl4AI. Let's make the web more accessible and useful for AI applications! 💪🌐🤖
--- a/_sync/quickstart.md
+++ b/_sync/quickstart.md
@@ -0,0 +1,204 @@
+# Quick Start Guide 🚀
+
+Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies. Let's dive in! 🌟
+
+## Getting Started 🛠️
+
+First, let's create an instance of `WebCrawler` and call the `warmup()` function. This might take a few seconds the first time you run Crawl4AI, as it loads the required model files.
+
+```python
+from crawl4ai import WebCrawler
+
+def create_crawler():
+    crawler = WebCrawler(verbose=True)
+    crawler.warmup()
+    return crawler
+
+crawler = create_crawler()
+```
+
+### Basic Usage
+
+Simply provide a URL and let Crawl4AI do the magic!
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(f"Basic crawl result: {result}")
+```
+
+### Taking Screenshots 📸
+
+Let's take a screenshot of the page!
+
+```python
+result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
+with open("screenshot.png", "wb") as f:
+    f.write(base64.b64decode(result.screenshot))
+print("Screenshot saved to 'screenshot.png'!")
+```
+
+### Understanding Parameters 🧠
+
+By default, Crawl4AI caches the results of your crawls. This means that subsequent crawls of the same URL will be much faster! Let's see this in action.
+
+First crawl (caches the result):
+```python
+result = crawler.run(url="https://www.nbcnews.com/business")
+print(f"First crawl result: {result}")
+```
+
+Force to crawl again:
+```python
+result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
+print(f"Second crawl result: {result}")
+```
+
+### Adding a Chunking Strategy 🧩
+
+Let's add a chunking strategy: `RegexChunking`! This strategy splits the text based on a given regex pattern.
+
+```python
+from crawl4ai.chunking_strategy import RegexChunking
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    chunking_strategy=RegexChunking(patterns=["\n\n"])
+)
+print(f"RegexChunking result: {result}")
+```
+
+You can also use `NlpSentenceChunking` which splits the text into sentences using NLP techniques.
+
+```python
+from crawl4ai.chunking_strategy import NlpSentenceChunking
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    chunking_strategy=NlpSentenceChunking()
+)
+print(f"NlpSentenceChunking result: {result}")
+```
+
+### Adding an Extraction Strategy 🧠
+
+Let's get smarter with an extraction strategy: `CosineStrategy`! This strategy uses cosine similarity to extract semantically similar blocks of text.
+
+```python
+from crawl4ai.extraction_strategy import CosineStrategy
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=CosineStrategy(
+        word_count_threshold=10, 
+        max_dist=0.2, 
+        linkage_method="ward", 
+        top_k=3
+    )
+)
+print(f"CosineStrategy result: {result}")
+```
+
+You can also pass other parameters like `semantic_filter` to extract specific content.
+
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=CosineStrategy(
+        semantic_filter="inflation rent prices"
+    )
+)
+print(f"CosineStrategy result with semantic filter: {result}")
+```
+
+### Using LLMExtractionStrategy 🤖
+
+Time to bring in the big guns: `LLMExtractionStrategy` without instructions! This strategy uses a large language model to extract relevant information from the web page.
+
+```python
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+import os
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=LLMExtractionStrategy(
+        provider="openai/gpt-4o", 
+        api_token=os.getenv('OPENAI_API_KEY')
+    )
+)
+print(f"LLMExtractionStrategy (no instructions) result: {result}")
+```
+
+You can also provide specific instructions to guide the extraction.
+
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    extraction_strategy=LLMExtractionStrategy(
+        provider="openai/gpt-4o",
+        api_token=os.getenv('OPENAI_API_KEY'),
+        instruction="I am interested in only financial news"
+    )
+)
+print(f"LLMExtractionStrategy (with instructions) result: {result}")
+```
+
+### Targeted Extraction 🎯
+
+Let's use a CSS selector to extract only H2 tags!
+
+```python
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    css_selector="h2"
+)
+print(f"CSS Selector (H2 tags) result: {result}")
+```
+
+### Interactive Extraction 🖱️
+
+Passing JavaScript code to click the 'Load More' button!
+
+```python
+js_code = """
+const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+loadMoreButton && loadMoreButton.click();
+"""
+
+result = crawler.run(
+    url="https://www.nbcnews.com/business",
+    js=js_code
+)
+print(f"JavaScript Code (Load More button) result: {result}")
+```
+
+### Using Crawler Hooks 🔗
+
+Let's see how we can customize the crawler using hooks!
+
+```python
+import time
+
+from crawl4ai.web_crawler import WebCrawler
+from crawl4ai.crawler_strategy import *
+
+def delay(driver):
+    print("Delaying for 5 seconds...")
+    time.sleep(5)
+    print("Resuming...")
+    
+def create_crawler():
+    crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
+    crawler_strategy.set_hook('after_get_url', delay)
+    crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
+    crawler.warmup()
+    return crawler
+
+crawler = create_crawler()
+result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
+```
+
+check [Hooks](examples/hooks_auth.md) for more examples.
+
+## Congratulations! 🎉
+
+You've made it through the Crawl4AI Quickstart Guide! Now go forth and crawl the web like a pro! 🕸️
--- a/docs/md/changelog.md
+++ b/docs/md/changelog.md
@@ -1,6 +1,82 @@
 # Changelog

-## [0.2.7] - 2024-06-27
+## [v0.2.77] - 2024-08-04
+
+Significant improvements in text processing and performance:
+
+- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
+- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
+- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
+- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
+
+These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
+
+## [v0.2.76] - 2024-08-02
+
+Major improvements in functionality, performance, and cross-platform compatibility! 🚀
+
+- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
+- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
+- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
+- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
+- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
+
+A big shoutout to our amazing community contributors:
+- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
+- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
+- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
+
+Your contributions are driving Crawl4AI forward! 🙌
+
+## [v0.2.75] - 2024-07-19
+
+Minor improvements for a more maintainable codebase:
+
+- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
+- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
+
+These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
+
+
+## v0.2.74 - 2024-07-08
+A slew of exciting updates to improve the crawler's stability and robustness! 🎉
+
+- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
+- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
+- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
+- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
+
+## [v0.2.73] - 2024-07-03
+
+💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
+
+* Supporting website need "with-head" mode to crawl the website with head.
+* Fixing the installation issues for setup.py and dockerfile.
+* Resolve multiple issues.
+
+## [v0.2.72] - 2024-06-30
+
+This release brings exciting updates and improvements to our project! 🎉
+
+* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
+* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
+* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
+* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
+
+These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
+
+## [0.2.71] - 2024-06-26
+
+**Improved Error Handling and Performance** 🚧
+
+* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
+* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
+* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
+* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
+
+These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
+
+## [0.2.71] - 2024-06-25
 ### Fixed
 - Speed up twice the extraction function.

--- a/docs/md/demo.md
+++ b/docs/md/demo.md
@@ -14,6 +14,7 @@
            <div class="form-group">
                <button class="btn btn-default" type="submit">Submit</button>
            </div>
+
        </fieldset>
    </form>

@@ -93,6 +94,10 @@
        </div>
    </section>

+    <div id="error" class="error-message" style="display: none; margin-top:1em;">
+        <div class="terminal-alert terminal-alert-error"></div>
+    </div>
+
    <script>
        function showTab(tabId) {
            const tabs = document.querySelectorAll('.tab-content');
@@ -162,7 +167,17 @@
                },
                body: JSON.stringify(data)
            })
-            .then(response => response.json())
+            .then(response => {
+                if (!response.ok) {
+                    if (response.status === 429) {
+                        return response.json().then(err => { 
+                            throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
+                        });
+                    }
+                    throw new Error('Network response was not ok');
+                }
+                return response.json();
+            })
            .then(data => {
                data = data.results[0]; // Only one URL is requested
                document.getElementById('loading').style.display = 'none';
@@ -187,11 +202,29 @@ result = crawler.run(
 print(result)
                `;
                redo(document.getElementById('pythonCode'), pythonCode);
+                document.getElementById('error').style.display = 'none';
            })
            .catch(error => {
                document.getElementById('loading').style.display = 'none';
-                document.getElementById('response').style.display = 'block';
-                document.getElementById('markdownContent').textContent = 'Error: ' + error;
+                document.getElementById('error').style.display = 'block';
+                let errorMessage = 'An unexpected error occurred. Please try again later.';
+                
+                if (error.status === 429) {
+                    const details = error.details;
+                    if (details.retry_after) {
+                        errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
+                    } else if (details.reset_at) {
+                        const resetTime = new Date(details.reset_at);
+                        const waitTime = Math.ceil((resetTime - new Date()) / 1000);
+                        errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
+                    } else {
+                        errorMessage = `Rate limit exceeded. Please try again later.`;
+                    }
+                } else if (error.message) {
+                    errorMessage = error.message;
+                }
+                
+                document.querySelector('#error .terminal-alert').textContent = errorMessage;
            });
        });
    </script>
--- a/docs/md/examples/hooks_auth.md
+++ b/docs/md/examples/hooks_auth.md
@@ -1,96 +1,110 @@
-# Hooks & Auth
+# Hooks & Auth for AsyncWebCrawler

-Crawl4AI allows you to customize the behavior of the web crawler using hooks. Hooks are functions that are called at specific points in the crawling process, allowing you to modify the crawler's behavior or perform additional actions. This example demonstrates how to use various hooks to customize the crawling process.
+Crawl4AI's AsyncWebCrawler allows you to customize the behavior of the web crawler using hooks. Hooks are asynchronous functions that are called at specific points in the crawling process, allowing you to modify the crawler's behavior or perform additional actions. This example demonstrates how to use various hooks to customize the asynchronous crawling process.

-## Example: Using Crawler Hooks
+## Example: Using Crawler Hooks with AsyncWebCrawler

-Let's see how we can customize the crawler using hooks! In this example, we'll:
+Let's see how we can customize the AsyncWebCrawler using hooks! In this example, we'll:

-1. Maximize the browser window and log in to a website when the driver is created.
-2. Add a custom header before fetching the URL.
-3. Log the current URL after fetching it.
-4. Log the length of the HTML before returning it.
+1. Configure the browser when it's created.
+2. Add custom headers before navigating to the URL.
+3. Log the current URL after navigation.
+4. Perform actions after JavaScript execution.
+5. Log the length of the HTML before returning it.

 ### Hook Definitions

 ```python
-def on_driver_created(driver):
-    print("[HOOK] on_driver_created")
-    # Example customization: maximize the window
-    driver.maximize_window()
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
+from playwright.async_api import Page, Browser
+
+async def on_browser_created(browser: Browser):
+    print("[HOOK] on_browser_created")
+    # Example customization: set browser viewport size
+    context = await browser.new_context(viewport={'width': 1920, 'height': 1080})
+    page = await context.new_page()
    
    # Example customization: logging in to a hypothetical website
-    driver.get('https://example.com/login')
+    await page.goto('https://example.com/login')
+    await page.fill('input[name="username"]', 'testuser')
+    await page.fill('input[name="password"]', 'password123')
+    await page.click('button[type="submit"]')
+    await page.wait_for_selector('#welcome')
    
-    from selenium.webdriver.support.ui import WebDriverWait
-    from selenium.webdriver.common.by import By
-    from selenium.webdriver.support import expected_conditions as EC
-    
-    WebDriverWait(driver, 10).until(
-        EC.presence_of_element_located((By.NAME, 'username'))
-    )
-    driver.find_element(By.NAME, 'username').send_keys('testuser')
-    driver.find_element(By.NAME, 'password').send_keys('password123')
-    driver.find_element(By.NAME, 'login').click()
-    WebDriverWait(driver, 10).until(
-        EC.presence_of_element_located((By.ID, 'welcome'))
-    )
    # Add a custom cookie
-    driver.add_cookie({'name': 'test_cookie', 'value': 'cookie_value'})
-    return driver        
+    await context.add_cookies([{'name': 'test_cookie', 'value': 'cookie_value', 'url': 'https://example.com'}])
    
+    await page.close()
+    await context.close()

-def before_get_url(driver):
-    print("[HOOK] before_get_url")
-    # Example customization: add a custom header
-    # Enable Network domain for sending headers
-    driver.execute_cdp_cmd('Network.enable', {})
-    # Add a custom header
-    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
-    return driver
+async def before_goto(page: Page):
+    print("[HOOK] before_goto")
+    # Example customization: add custom headers
+    await page.set_extra_http_headers({'X-Test-Header': 'test'})

-def after_get_url(driver):
-    print("[HOOK] after_get_url")
+async def after_goto(page: Page):
+    print("[HOOK] after_goto")
    # Example customization: log the URL
-    print(driver.current_url)
-    return driver
+    print(f"Current URL: {page.url}")

-def before_return_html(driver, html):
+async def on_execution_started(page: Page):
+    print("[HOOK] on_execution_started")
+    # Example customization: perform actions after JS execution
+    await page.evaluate("console.log('Custom JS executed')")
+
+async def before_return_html(page: Page, html: str):
    print("[HOOK] before_return_html")
-    # Example customization: log the HTML
-    print(len(html))
-    return driver
+    # Example customization: log the HTML length
+    print(f"HTML length: {len(html)}")
+    return page
 ```

-### Using the Hooks with the WebCrawler
+### Using the Hooks with the AsyncWebCrawler

 ```python
-print("\n🔗 [bold cyan]Using Crawler Hooks: Let's see how we can customize the crawler using hooks![/bold cyan]", True)
-crawler = WebCrawler(verbose=True)
-crawler.warmup()
-crawler.set_hook('on_driver_created', on_driver_created)
-crawler.set_hook('before_get_url', before_get_url)
-crawler.set_hook('after_get_url', after_get_url)
-crawler.set_hook('before_return_html', before_return_html)
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

-result = crawler.run(url="https://example.com")
+async def main():
+    print("\n🔗 Using Crawler Hooks: Let's see how we can customize the AsyncWebCrawler using hooks!")
+    
+    crawler_strategy = AsyncPlaywrightCrawlerStrategy(verbose=True)
+    crawler_strategy.set_hook('on_browser_created', on_browser_created)
+    crawler_strategy.set_hook('before_goto', before_goto)
+    crawler_strategy.set_hook('after_goto', after_goto)
+    crawler_strategy.set_hook('on_execution_started', on_execution_started)
+    crawler_strategy.set_hook('before_return_html', before_return_html)
+    
+    async with AsyncWebCrawler(verbose=True, crawler_strategy=crawler_strategy) as crawler:
+        result = await crawler.arun(
+            url="https://example.com",
+            js_code="window.scrollTo(0, document.body.scrollHeight);",
+            wait_for="footer"
+        )

-print("[LOG] 📦 [bold yellow]Crawler Hooks result:[/bold yellow]")
-print(result)
+    print("📦 Crawler Hooks result:")
+    print(result)
+
+asyncio.run(main())
 ```

 ### Explanation

- `on_driver_created`: This hook is called when the Selenium driver is created. In this example, it maximizes the window, logs in to a website, and adds a custom cookie.
- `before_get_url`: This hook is called right before Selenium fetches the URL. In this example, it adds a custom HTTP header.
- `after_get_url`: This hook is called after Selenium fetches the URL. In this example, it logs the current URL.
- `before_return_html`: This hook is called before returning the HTML content. In this example, it logs the length of the HTML content.
+- `on_browser_created`: This hook is called when the Playwright browser is created. It sets up the browser context, logs in to a website, and adds a custom cookie.
+- `before_goto`: This hook is called right before Playwright navigates to the URL. It adds custom HTTP headers.
+- `after_goto`: This hook is called after Playwright navigates to the URL. It logs the current URL.
+- `on_execution_started`: This hook is called after any custom JavaScript is executed. It performs additional JavaScript actions.
+- `before_return_html`: This hook is called before returning the HTML content. It logs the length of the HTML content.

 ### Additional Ideas

- **Add custom headers to requests**: You can add custom headers to the requests using the `before_get_url` hook.
- **Perform safety checks**: Use the hooks to perform safety checks before the crawling process starts.
- **Modify the HTML content**: Use the `before_return_html` hook to modify the HTML content before it is returned.
- **Log additional information**: Use the hooks to log additional information for debugging or monitoring purposes.
+- **Handling authentication**: Use the `on_browser_created` hook to handle login processes or set authentication tokens.
+- **Dynamic header modification**: Modify headers based on the target URL or other conditions in the `before_goto` hook.
+- **Content verification**: Use the `after_goto` hook to verify that the expected content is present on the page.
+- **Custom JavaScript injection**: Inject and execute custom JavaScript using the `on_execution_started` hook.
+- **Content preprocessing**: Modify or analyze the HTML content in the `before_return_html` hook before it's returned.

-By using these hooks, you can customize the behavior of the crawler to suit your specific needs.
+By using these hooks, you can customize the behavior of the AsyncWebCrawler to suit your specific needs, including handling authentication, modifying requests, and preprocessing content.
--- a/docs/md/examples/index.md
+++ b/docs/md/examples/index.md
@@ -8,6 +8,10 @@ Welcome to the examples section of Crawl4AI documentation! In this section, you

 This example demonstrates how to use Crawl4AI to extract information using Large Language Models (LLMs). You will learn how to configure the `LLMExtractionStrategy` to get structured data from web pages.

+### [JSON CSS Extraction](json_css_extraction.md)
+
+This example demonstrates how to use Crawl4AI to extract structured data without using LLM, and just focusing on page structure. You will learn how to use the `JsonCssExtractionStrategy` to extract data using CSS selectors.
+
 ### [JS Execution & CSS Filtering](js_execution_css_filtering.md)

 Learn how to execute custom JavaScript code and filter data using CSS selectors. This example shows how to perform complex web interactions and extract specific content from web pages.
--- a/docs/md/examples/js_execution_css_filtering.md
+++ b/docs/md/examples/js_execution_css_filtering.md
@@ -1,44 +1,104 @@
-# JS Execution & CSS Filtering
+# JS Execution & CSS Filtering with AsyncWebCrawler

-In this example, we'll demonstrate how to use Crawl4AI to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.
+In this example, we'll demonstrate how to use Crawl4AI's AsyncWebCrawler to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.

-## Example: Extracting Structured Data
+## Example: Extracting Structured Data Asynchronously

 ```python
-# Import necessary modules
-from crawl4ai import WebCrawler
-from crawl4ai.chunking_strategy import *
-from crawl4ai.extraction_strategy import *
-from crawl4ai.crawler_strategy import *
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.chunking_strategy import RegexChunking
+from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

-# Define the JavaScript code to click the "Load More" button
-js_code = ["""
-const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
-loadMoreButton && loadMoreButton.click();
-"""]
+async def main():
+    # Define the JavaScript code to click the "Load More" button
+    js_code = """
+    const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+    if (loadMoreButton) {
+        loadMoreButton.click();
+        // Wait for new content to load
+        await new Promise(resolve => setTimeout(resolve, 2000));
+    }
+    """

-crawler = WebCrawler(verbose=True)
-crawler.warmup()
-# Run the crawler with keyword filtering and CSS selector
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    js=js_code,
-    css_selector="p",
-    extraction_strategy=CosineStrategy(
-        semantic_filter="technology",
-    ),
-)
+    # Define a wait_for function to ensure content is loaded
+    wait_for = """
+    () => {
+        const articles = document.querySelectorAll('article.tease-card');
+        return articles.length > 10;
+    }
+    """

-# Display the extracted result
-print(result)
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Run the crawler with keyword filtering and CSS selector
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            wait_for=wait_for,
+            css_selector="article.tease-card",
+            extraction_strategy=CosineStrategy(
+                semantic_filter="technology",
+            ),
+            chunking_strategy=RegexChunking(),
+        )
+
+    # Display the extracted result
+    print(result.extracted_content)
+
+# Run the async function
+asyncio.run(main())
 ```

 ### Explanation

-1. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button. This is useful for loading additional content dynamically.
-2. **CSS Selector**: The `css_selector="p"` parameter ensures that only paragraph (`<p>`) tags are extracted from the web page.
-3. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
+1. **Asynchronous Execution**: We use `AsyncWebCrawler` with async/await syntax for non-blocking execution.
+
+2. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button and waits for new content to load.
+
+3. **Wait Condition**: The `wait_for` function ensures that the page has loaded more than 10 articles before proceeding with the extraction.
+
+4. **CSS Selector**: The `css_selector="article.tease-card"` parameter ensures that only article cards are extracted from the web page.
+
+5. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
+
+6. **Chunking Strategy**: We use `RegexChunking()` to split the content into manageable chunks for processing.
+
+## Advanced Usage: Custom Session and Multiple Requests
+
+For more complex scenarios where you need to maintain state across multiple requests or execute additional JavaScript after the initial page load, you can use a custom session:
+
+```python
+async def advanced_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Initial crawl with custom session
+        result1 = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            wait_for=wait_for,
+            css_selector="article.tease-card",
+            session_id="business_session"
+        )
+
+        # Execute additional JavaScript in the same session
+        result2 = await crawler.crawler_strategy.execute_js(
+            session_id="business_session",
+            js_code="window.scrollTo(0, document.body.scrollHeight);",
+            wait_for_js="() => window.innerHeight + window.scrollY >= document.body.offsetHeight"
+        )
+
+        # Process results
+        print("Initial crawl result:", result1.extracted_content)
+        print("Additional JS execution result:", result2.html)
+
+asyncio.run(advanced_crawl())
+```
+
+This advanced example demonstrates how to:
+1. Use a custom session to maintain state across requests.
+2. Execute additional JavaScript after the initial page load.
+3. Wait for specific conditions using JavaScript functions.

 ## Try It Yourself

-This example demonstrates the power and flexibility of Crawl4AI in handling complex web interactions and extracting meaningful data. You can customize the JavaScript code, CSS selectors, and extraction strategies to suit your specific requirements.
+These examples demonstrate the power and flexibility of Crawl4AI's AsyncWebCrawler in handling complex web interactions and extracting meaningful data asynchronously. You can customize the JavaScript code, CSS selectors, extraction strategies, and waiting conditions to suit your specific requirements.
--- a/docs/md/examples/json_css_extraction.md
+++ b/docs/md/examples/json_css_extraction.md
@@ -0,0 +1,142 @@
+# JSON CSS Extraction Strategy with AsyncWebCrawler
+
+The `JsonCssExtractionStrategy` is a powerful feature of Crawl4AI that allows you to extract structured data from web pages using CSS selectors. This method is particularly useful when you need to extract specific data points from a consistent HTML structure, such as tables or repeated elements. Here's how to use it with the AsyncWebCrawler.
+
+## Overview
+
+The `JsonCssExtractionStrategy` works by defining a schema that specifies:
+1. A base CSS selector for the repeating elements
+2. Fields to extract from each element, each with its own CSS selector
+
+This strategy is fast and efficient, as it doesn't rely on external services like LLMs for extraction.
+
+## Example: Extracting Cryptocurrency Prices from Coinbase
+
+Let's look at an example that extracts cryptocurrency prices from the Coinbase explore page.
+
+```python
+import json
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+async def extract_structured_data_using_css_extractor():
+    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
+    
+    # Define the extraction schema
+    schema = {
+        "name": "Coinbase Crypto Prices",
+        "baseSelector": ".cds-tableRow-t45thuk",
+        "fields": [
+            {
+                "name": "crypto",
+                "selector": "td:nth-child(1) h2",
+                "type": "text",
+            },
+            {
+                "name": "symbol",
+                "selector": "td:nth-child(1) p",
+                "type": "text",
+            },
+            {
+                "name": "price",
+                "selector": "td:nth-child(2)",
+                "type": "text",
+            }
+        ],
+    }
+
+    # Create the extraction strategy
+    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+    # Use the AsyncWebCrawler with the extraction strategy
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.coinbase.com/explore",
+            extraction_strategy=extraction_strategy,
+            bypass_cache=True,
+        )
+
+        assert result.success, "Failed to crawl the page"
+
+        # Parse the extracted content
+        crypto_prices = json.loads(result.extracted_content)
+        print(f"Successfully extracted {len(crypto_prices)} cryptocurrency prices")
+        print(json.dumps(crypto_prices[0], indent=2))
+
+    return crypto_prices
+
+# Run the async function
+asyncio.run(extract_structured_data_using_css_extractor())
+```
+
+## Explanation of the Schema
+
+The schema defines how to extract the data:
+
+- `name`: A descriptive name for the extraction task.
+- `baseSelector`: The CSS selector for the repeating elements (in this case, table rows).
+- `fields`: An array of fields to extract from each element:
+  - `name`: The name to give the extracted data.
+  - `selector`: The CSS selector to find the specific data within the base element.
+  - `type`: The type of data to extract (usually "text" for textual content).
+
+## Advantages of JsonCssExtractionStrategy
+
+1. **Speed**: CSS selectors are fast to execute, making this method efficient for large datasets.
+2. **Precision**: You can target exactly the elements you need.
+3. **Structured Output**: The result is already structured as JSON, ready for further processing.
+4. **No External Dependencies**: Unlike LLM-based strategies, this doesn't require any API calls to external services.
+
+## Tips for Using JsonCssExtractionStrategy
+
+1. **Inspect the Page**: Use browser developer tools to identify the correct CSS selectors.
+2. **Test Selectors**: Verify your selectors in the browser console before using them in the script.
+3. **Handle Dynamic Content**: If the page uses JavaScript to load content, you may need to combine this with JS execution (see the Advanced Usage section).
+4. **Error Handling**: Always check the `result.success` flag and handle potential failures.
+
+## Advanced Usage: Combining with JavaScript Execution
+
+For pages that load data dynamically, you can combine the `JsonCssExtractionStrategy` with JavaScript execution:
+
+```python
+async def extract_dynamic_structured_data():
+    schema = {
+        "name": "Dynamic Crypto Prices",
+        "baseSelector": ".crypto-row",
+        "fields": [
+            {"name": "name", "selector": ".crypto-name", "type": "text"},
+            {"name": "price", "selector": ".crypto-price", "type": "text"},
+        ]
+    }
+
+    js_code = """
+    window.scrollTo(0, document.body.scrollHeight);
+    await new Promise(resolve => setTimeout(resolve, 2000));  // Wait for 2 seconds
+    """
+
+    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://example.com/crypto-prices",
+            extraction_strategy=extraction_strategy,
+            js_code=js_code,
+            wait_for=".crypto-row:nth-child(20)",  # Wait for 20 rows to load
+            bypass_cache=True,
+        )
+
+        crypto_data = json.loads(result.extracted_content)
+        print(f"Extracted {len(crypto_data)} cryptocurrency entries")
+
+asyncio.run(extract_dynamic_structured_data())
+```
+
+This advanced example demonstrates how to:
+1. Execute JavaScript to trigger dynamic content loading.
+2. Wait for a specific condition (20 rows loaded) before extraction.
+3. Extract data from the dynamically loaded content.
+
+By mastering the `JsonCssExtractionStrategy`, you can efficiently extract structured data from a wide variety of web pages, making it a valuable tool in your web scraping toolkit.
+
+For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](../full_details/advanced_jsoncss_extraction.md).
--- a/docs/md/examples/llm_extraction.md
+++ b/docs/md/examples/llm_extraction.md
@@ -1,6 +1,6 @@
-# LLM Extraction
+# LLM Extraction with AsyncWebCrawler

-Crawl4AI allows you to use Language Models (LLMs) to extract structured data or relevant content from web pages. Below are two examples demonstrating how to use LLMExtractionStrategy for different purposes.
+Crawl4AI's AsyncWebCrawler allows you to use Language Models (LLMs) to extract structured data or relevant content from web pages asynchronously. Below are two examples demonstrating how to use `LLMExtractionStrategy` for different purposes with the AsyncWebCrawler.

 ## Example 1: Extract Structured Data

@@ -8,17 +8,10 @@ In this example, we use the `LLMExtractionStrategy` to extract structured data (

 ```python
 import os
-import time
-from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.chunking_strategy import *
-from crawl4ai.extraction_strategy import *
-from crawl4ai.crawler_strategy import *
-
-url = r'https://openai.com/api/pricing/'
-
-crawler = WebCrawler()
-crawler.warmup()
-
+import json
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
 from pydantic import BaseModel, Field

 class OpenAIModelFee(BaseModel):
@@ -26,27 +19,33 @@ class OpenAIModelFee(BaseModel):
    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")

-result = crawler.run(
-    url=url,
-    word_count_threshold=1,
-    extraction_strategy= LLMExtractionStrategy(
-        provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
-        schema=OpenAIModelFee.model_json_schema(),
-        extraction_type="schema",
-        instruction="From the crawled content, extract all mentioned model names along with their "\
-            "fees for input and output tokens. Make sure not to miss anything in the entire content. "\
-            'One extracted model JSON format should look like this: '\
-            '{ "model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens" }'
-    ),
-    bypass_cache=True,
-)
+async def extract_openai_fees():
+    url = 'https://openai.com/api/pricing/'

-model_fees = json.loads(result.extracted_content)
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url=url,
+            word_count_threshold=1,
+            extraction_strategy=LLMExtractionStrategy(
+                provider="openai/gpt-4o",
+                api_token=os.getenv('OPENAI_API_KEY'),
+                schema=OpenAIModelFee.model_json_schema(),
+                extraction_type="schema",
+                instruction="From the crawled content, extract all mentioned model names along with their "
+                            "fees for input and output tokens. Make sure not to miss anything in the entire content. "
+                            'One extracted model JSON format should look like this: '
+                            '{ "model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens" }'
+            ),
+            bypass_cache=True,
+        )

-print(len(model_fees))
+    model_fees = json.loads(result.extracted_content)
+    print(f"Number of models extracted: {len(model_fees)}")

-with open(".data/data.json", "w") as f:
-    f.write(result.extracted_content)
+    with open(".data/openai_fees.json", "w", encoding="utf-8") as f:
+        json.dump(model_fees, f, indent=2)
+
+asyncio.run(extract_openai_fees())
 ```

 ## Example 2: Extract Relevant Content
@@ -54,30 +53,80 @@ with open(".data/data.json", "w") as f:
 In this example, we instruct the LLM to extract only content related to technology from the NBC News business page.

 ```python
-crawler = WebCrawler()
-crawler.warmup()
+import os
+import json
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import LLMExtractionStrategy

-result = crawler.run(
-        url="https://www.nbcnews.com/business",
-        extraction_strategy=LLMExtractionStrategy(
-            provider="openai/gpt-4o",
-            api_token=os.getenv('OPENAI_API_KEY'),
-            instruction="Extract only content related to technology"
-        ),
-    bypass_cache=True,
-    )
+async def extract_tech_content():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            extraction_strategy=LLMExtractionStrategy(
+                provider="openai/gpt-4o",
+                api_token=os.getenv('OPENAI_API_KEY'),
+                instruction="Extract only content related to technology"
+            ),
+            bypass_cache=True,
+        )

-model_fees = json.loads(result.extracted_content)
+    tech_content = json.loads(result.extracted_content)
+    print(f"Number of tech-related items extracted: {len(tech_content)}")

-print(len(model_fees))
+    with open(".data/tech_content.json", "w", encoding="utf-8") as f:
+        json.dump(tech_content, f, indent=2)

-with open(".data/data.json", "w") as f:
-    f.write(result.extracted_content)
+asyncio.run(extract_tech_content())
+```
+
+## Advanced Usage: Combining JS Execution with LLM Extraction
+
+This example demonstrates how to combine JavaScript execution with LLM extraction to handle dynamic content:
+
+```python
+async def extract_dynamic_content():
+    js_code = """
+    const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+    if (loadMoreButton) {
+        loadMoreButton.click();
+        await new Promise(resolve => setTimeout(resolve, 2000));
+    }
+    """
+
+    wait_for = """
+    () => {
+        const articles = document.querySelectorAll('article.tease-card');
+        return articles.length > 10;
+    }
+    """
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            wait_for=wait_for,
+            css_selector="article.tease-card",
+            extraction_strategy=LLMExtractionStrategy(
+                provider="openai/gpt-4o",
+                api_token=os.getenv('OPENAI_API_KEY'),
+                instruction="Summarize each article, focusing on technology-related content"
+            ),
+            bypass_cache=True,
+        )
+
+    summaries = json.loads(result.extracted_content)
+    print(f"Number of summarized articles: {len(summaries)}")
+
+    with open(".data/tech_summaries.json", "w", encoding="utf-8") as f:
+        json.dump(summaries, f, indent=2)
+
+asyncio.run(extract_dynamic_content())
 ```

 ## Customizing LLM Provider

-Under the hood, Crawl4AI uses the `litellm` library, which allows you to use any LLM provider you want. Just pass the correct model name and API token.
+Crawl4AI uses the `litellm` library under the hood, which allows you to use any LLM provider you want. Just pass the correct model name and API token:

 ```python
 extraction_strategy=LLMExtractionStrategy(
@@ -88,3 +137,43 @@ extraction_strategy=LLMExtractionStrategy(
 ```

 This flexibility allows you to integrate with various LLM providers and tailor the extraction process to your specific needs.
+
+## Error Handling and Retries
+
+When working with external LLM APIs, it's important to handle potential errors and implement retry logic. Here's an example of how you might do this:
+
+```python
+import asyncio
+from tenacity import retry, stop_after_attempt, wait_exponential
+
+class LLMExtractionError(Exception):
+    pass
+
+@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
+async def extract_with_retry(crawler, url, extraction_strategy):
+    try:
+        result = await crawler.arun(url=url, extraction_strategy=extraction_strategy, bypass_cache=True)
+        return json.loads(result.extracted_content)
+    except Exception as e:
+        raise LLMExtractionError(f"Failed to extract content: {str(e)}")
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        try:
+            content = await extract_with_retry(
+                crawler,
+                "https://www.example.com",
+                LLMExtractionStrategy(
+                    provider="openai/gpt-4o",
+                    api_token=os.getenv('OPENAI_API_KEY'),
+                    instruction="Extract and summarize main points"
+                )
+            )
+            print("Extracted content:", content)
+        except LLMExtractionError as e:
+            print(f"Extraction failed after retries: {e}")
+
+asyncio.run(main())
+```
+
+This example uses the `tenacity` library to implement a retry mechanism with exponential backoff, which can help handle temporary failures or rate limiting from the LLM API.
--- a/docs/md/examples/research_assistant.md
+++ b/docs/md/examples/research_assistant.md
@@ -1,33 +1,32 @@
-## Research Assistant Example
+# Research Assistant Example with AsyncWebCrawler

-This example demonstrates how to build a research assistant using `Chainlit` and `Crawl4AI`. The assistant will be capable of crawling web pages for information and answering questions based on the crawled content. Additionally, it integrates speech-to-text functionality for audio inputs.
+This example demonstrates how to build an advanced research assistant using `Chainlit`, `Crawl4AI`'s `AsyncWebCrawler`, and various AI services. The assistant can crawl web pages asynchronously, answer questions based on the crawled content, and handle audio inputs.

-### Step-by-Step Guide
+## Step-by-Step Guide

 1. **Install Required Packages**

-    Ensure you have the necessary packages installed. You need `chainlit`, `groq`, `requests`, and `openai`.
+    Ensure you have the necessary packages installed:

    ```bash
-    pip install chainlit groq requests openai
+    pip install chainlit groq openai crawl4ai
    ```

 2. **Import Libraries**

-    Import all the necessary modules and initialize the OpenAI client.
-
    ```python
    import os
    import time
+    import asyncio
    from openai import AsyncOpenAI
    import chainlit as cl
    import re
-    import requests
    from io import BytesIO
    from chainlit.element import ElementBased
    from groq import Groq
-
-    from concurrent.futures import ThreadPoolExecutor
+    from crawl4ai import AsyncWebCrawler
+    from crawl4ai.extraction_strategy import NoExtractionStrategy
+    from crawl4ai.chunking_strategy import RegexChunking

    client = AsyncOpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.getenv("GROQ_API_KEY"))

@@ -37,8 +36,6 @@ This example demonstrates how to build a research assistant using `Chainlit` and

 3. **Set Configuration**

-    Define the model settings for the assistant.
-
    ```python
    settings = {
        "model": "llama3-8b-8192",
@@ -52,35 +49,25 @@ This example demonstrates how to build a research assistant using `Chainlit` and

 4. **Define Utility Functions**

-    - **Extract URLs from Text**: Use regex to find URLs in messages.
+    ```python
+    def extract_urls(text):
+        url_pattern = re.compile(r'(https?://\S+)')
+        return url_pattern.findall(text)

-        ```python
-        def extract_urls(text):
-            url_pattern = re.compile(r'(https?://\S+)')
-            return url_pattern.findall(text)
-        ```
-
-    - **Crawl URL**: Send a request to `Crawl4AI` to fetch the content of a URL.
-
-        ```python
-        def crawl_url(url):
-            data = {
-                "urls": [url],
-                "include_raw_html": True,
-                "word_count_threshold": 10,
-                "extraction_strategy": "NoExtractionStrategy",
-                "chunking_strategy": "RegexChunking"
-            }
-            response = requests.post("https://crawl4ai.com/crawl", json=data)
-            response_data = response.json()
-            response_data = response_data['results'][0]
-            return response_data['markdown']
-        ```
+    async def crawl_urls(urls):
+        async with AsyncWebCrawler(verbose=True) as crawler:
+            results = await crawler.arun_many(
+                urls=urls,
+                word_count_threshold=10,
+                extraction_strategy=NoExtractionStrategy(),
+                chunking_strategy=RegexChunking(),
+                bypass_cache=True
+            )
+        return [result.markdown for result in results if result.success]
+    ```

 5. **Initialize Chat Start Event**

-    Set up the initial chat message and user session.
-
    ```python
    @cl.on_chat_start
    async def on_chat_start():
@@ -88,15 +75,11 @@ This example demonstrates how to build a research assistant using `Chainlit` and
            "history": [],
            "context": {}
        })  
-        await cl.Message(
-            content="Welcome to the chat! How can I assist you today?"
-        ).send()
+        await cl.Message(content="Welcome to the chat! How can I assist you today?").send()
    ```

 6. **Handle Incoming Messages**

-    Process user messages, extract URLs, and crawl them concurrently. Update the chat history and system message.
-
    ```python
    @cl.on_message
    async def on_message(message: cl.Message):
@@ -105,19 +88,14 @@ This example demonstrates how to build a research assistant using `Chainlit` and
        # Extract URLs from the user's message
        urls = extract_urls(message.content)

-        futures = []
-        with ThreadPoolExecutor() as executor:
-            for url in urls:
-                futures.append(executor.submit(crawl_url, url))
-
-        results = [future.result() for future in futures]
-
-        for url, result in zip(urls, results):
-            ref_number = f"REF_{len(user_session['context']) + 1}"
-            user_session["context"][ref_number] = {
-                "url": url,
-                "content": result
-            }    
+        if urls:
+            crawled_contents = await crawl_urls(urls)
+            for url, content in zip(urls, crawled_contents):
+                ref_number = f"REF_{len(user_session['context']) + 1}"
+                user_session["context"][ref_number] = {
+                    "url": url,
+                    "content": content
+                }

        user_session["history"].append({
            "role": "user",
@@ -129,33 +107,24 @@ This example demonstrates how to build a research assistant using `Chainlit` and
            f'<appendix ref="{ref}">\n{data["content"]}\n</appendix>'
            for ref, data in user_session["context"].items()
        ]
-        if context_messages:
-            system_message = {
-                "role": "system",
-                "content": (
-                    "You are a helpful bot. Use the following context for answering questions. "
-                    "Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
-                    "If the question requires any information from the provided appendices or context, refer to the sources. "
-                    "If not, there is no need to add a references section. "
-                    "At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
-                    "\n\n".join(context_messages)
-                )
-            }
-        else:
-            system_message = {
-                "role": "system",
-                "content": "You are a helpful assistant."
-            }
+        system_message = {
+            "role": "system",
+            "content": (
+                "You are a helpful bot. Use the following context for answering questions. "
+                "Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
+                "If the question requires any information from the provided appendices or context, refer to the sources. "
+                "If not, there is no need to add a references section. "
+                "At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
+                "\n\n".join(context_messages)
+            ) if context_messages else "You are a helpful assistant."
+        }

        msg = cl.Message(content="")
        await msg.send()

        # Get response from the LLM
        stream = await client.chat.completions.create(
-            messages=[
-                system_message,
-                *user_session["history"]
-            ],
+            messages=[system_message, *user_session["history"]],
            stream=True,
            **settings
        )
@@ -174,18 +143,16 @@ This example demonstrates how to build a research assistant using `Chainlit` and
        await msg.update()

        # Append the reference section to the assistant's response
-        reference_section = "\n\nReferences:\n"
-        for ref, data in user_session["context"].items():
-            reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
-
-        msg.content += reference_section
-        await msg.update()
+        if user_session["context"]:
+            reference_section = "\n\nReferences:\n"
+            for ref, data in user_session["context"].items():
+                reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
+            msg.content += reference_section
+            await msg.update()
    ```

 7. **Handle Audio Input**

-    Capture and transcribe audio input. Store the audio buffer and transcribe it when the audio ends.
-
    ```python
    @cl.on_audio_chunk
    async def on_audio_chunk(chunk: cl.AudioChunk):
@@ -194,12 +161,10 @@ This example demonstrates how to build a research assistant using `Chainlit` and
            buffer.name = f"input_audio.{chunk.mimeType.split('/')[1]}"
            cl.user_session.set("audio_buffer", buffer)
            cl.user_session.set("audio_mime_type", chunk.mimeType)
-
        cl.user_session.get("audio_buffer").write(chunk.data)

    @cl.step(type="tool")
    async def speech_to_text(audio_file):
-        cli = Groq()
        response = await client.audio.transcriptions.create(
            model="whisper-large-v3", file=audio_file
        )
@@ -217,32 +182,39 @@ This example demonstrates how to build a research assistant using `Chainlit` and
        end_time = time.time()
        print(f"Transcription took {end_time - start_time} seconds")
        
-        user_msg = cl.Message(
-            author="You", 
-            type="user_message",
-            content=transcription
-        )
+        user_msg = cl.Message(author="You", type="user_message", content=transcription)
        await user_msg.send()
        await on_message(user_msg)
    ```

 8. **Run the Chat Application**

-    Start the Chainlit application.
-
    ```python
    if __name__ == "__main__":
        from chainlit.cli import run_chainlit
        run_chainlit(__file__)
    ```

-### Explanation
+## Explanation

- **Libraries and Configuration**: Import necessary libraries and configure the OpenAI client.
- **Utility Functions**: Define functions to extract URLs and crawl them.
- **Chat Start Event**: Initialize chat session and welcome message.
- **Message Handling**: Extract URLs, crawl them concurrently, and update chat history and context.
- **Audio Handling**: Capture, buffer, and transcribe audio input, then process the transcription as text.
- **Running the Application**: Start the Chainlit server to interact with the assistant.
+- **Libraries and Configuration**: We import necessary libraries, including `AsyncWebCrawler` from `crawl4ai`.
+- **Utility Functions**: 
+  - `extract_urls`: Uses regex to find URLs in messages.
+  - `crawl_urls`: An asynchronous function that uses `AsyncWebCrawler` to fetch content from multiple URLs concurrently.
+- **Chat Start Event**: Initializes the chat session and sends a welcome message.
+- **Message Handling**: 
+  - Extracts URLs from user messages.
+  - Asynchronously crawls the URLs using `AsyncWebCrawler`.
+  - Updates chat history and context with crawled content.
+  - Generates a response using the LLM, incorporating the crawled context.
+- **Audio Handling**: Captures, buffers, and transcribes audio input, then processes the transcription as text.
+- **Running the Application**: Starts the Chainlit server for interaction with the assistant.

-This example showcases how to create an interactive research assistant that can fetch, process, and summarize web content, along with handling audio inputs for a seamless user experience.
+## Key Improvements
+
+1. **Asynchronous Web Crawling**: Using `AsyncWebCrawler` allows for efficient, concurrent crawling of multiple URLs.
+2. **Improved Context Management**: The assistant now maintains a context of crawled content, allowing for more informed responses.
+3. **Dynamic Reference System**: The assistant can refer to specific sources in its responses and provide a reference section.
+4. **Seamless Audio Integration**: The ability to handle audio inputs makes the assistant more versatile and user-friendly.
+
+This updated Research Assistant showcases how to create a powerful, interactive tool that can efficiently fetch and process web content, handle various input types, and provide informed responses based on the gathered information.
--- a/docs/md/examples/summarization.md
+++ b/docs/md/examples/summarization.md
@@ -1,44 +1,34 @@
-## Summarization Example
+# Summarization Example with AsyncWebCrawler

-This example demonstrates how to use `Crawl4AI` to extract a summary from a web page. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.
+This example demonstrates how to use Crawl4AI's `AsyncWebCrawler` to extract a summary from a web page asynchronously. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.

-### Step-by-Step Guide
+## Step-by-Step Guide

 1. **Import Necessary Modules**

-    First, import the necessary modules and classes.
+    First, import the necessary modules and classes:

    ```python
    import os
-    import time
    import json
-    from crawl4ai.web_crawler import WebCrawler
-    from crawl4ai.chunking_strategy import *
-    from crawl4ai.extraction_strategy import *
-    from crawl4ai.crawler_strategy import *
+    import asyncio
+    from crawl4ai import AsyncWebCrawler
+    from crawl4ai.extraction_strategy import LLMExtractionStrategy
+    from crawl4ai.chunking_strategy import RegexChunking
    from pydantic import BaseModel, Field
    ```

 2. **Define the URL to be Crawled**

-    Set the URL of the web page you want to summarize.
+    Set the URL of the web page you want to summarize:

    ```python
-    url = r'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
+    url = 'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
    ```

-3. **Initialize the WebCrawler**
+3. **Define the Data Model**

-    Create an instance of the `WebCrawler` and call the `warmup` method.
-
-    ```python
-    crawler = WebCrawler()
-    crawler.warmup()
-    ```
-
-4. **Define the Data Model**
-
-    Use Pydantic to define the structure of the extracted data.
+    Use Pydantic to define the structure of the extracted data:

    ```python
    class PageSummary(BaseModel):
@@ -48,61 +38,116 @@ This example demonstrates how to use `Crawl4AI` to extract a summary from a web
        keywords: list = Field(..., description="Keywords assigned to the page.")
    ```

-5. **Run the Crawler**
+4. **Create the Extraction Strategy**

-    Set up and run the crawler with the `LLMExtractionStrategy`. Provide the necessary parameters, including the schema for the extracted data and the instruction for the LLM.
+    Set up the `LLMExtractionStrategy` with the necessary parameters:

    ```python
-    result = crawler.run(
-        url=url,
-        word_count_threshold=1,
-        extraction_strategy=LLMExtractionStrategy(
-            provider="openai/gpt-4o", 
-            api_token=os.getenv('OPENAI_API_KEY'), 
-            schema=PageSummary.model_json_schema(),
-            extraction_type="schema",
-            apply_chunking=False,
-            instruction=(
-                "From the crawled content, extract the following details: "
-                "1. Title of the page "
-                "2. Summary of the page, which is a detailed summary "
-                "3. Brief summary of the page, which is a paragraph text "
-                "4. Keywords assigned to the page, which is a list of keywords. "
-                'The extracted JSON format should look like this: '
-                '{ "title": "Page Title", "summary": "Detailed summary of the page.", '
-                '"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
-            )
-        ),
-        bypass_cache=True,
+    extraction_strategy = LLMExtractionStrategy(
+        provider="openai/gpt-4o", 
+        api_token=os.getenv('OPENAI_API_KEY'), 
+        schema=PageSummary.model_json_schema(),
+        extraction_type="schema",
+        apply_chunking=False,
+        instruction=(
+            "From the crawled content, extract the following details: "
+            "1. Title of the page "
+            "2. Summary of the page, which is a detailed summary "
+            "3. Brief summary of the page, which is a paragraph text "
+            "4. Keywords assigned to the page, which is a list of keywords. "
+            'The extracted JSON format should look like this: '
+            '{ "title": "Page Title", "summary": "Detailed summary of the page.", '
+            '"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
+        )
    )
    ```

-6. **Process the Extracted Data**
+5. **Define the Async Crawl Function**

-    Load the extracted content into a JSON object and print it.
+    Create an asynchronous function to run the crawler:

    ```python
-    page_summary = json.loads(result.extracted_content)
-    print(page_summary)
+    async def crawl_and_summarize(url):
+        async with AsyncWebCrawler(verbose=True) as crawler:
+            result = await crawler.arun(
+                url=url,
+                word_count_threshold=1,
+                extraction_strategy=extraction_strategy,
+                chunking_strategy=RegexChunking(),
+                bypass_cache=True,
+            )
+            return result
    ```

-7. **Save the Extracted Data**
+6. **Run the Crawler and Process Results**

-    Save the extracted data to a file for further use.
+    Use asyncio to run the crawler and process the results:

    ```python
-    with open(".data/page_summary.json", "w") as f:
-        f.write(result.extracted_content)
+    async def main():
+        result = await crawl_and_summarize(url)
+        
+        if result.success:
+            page_summary = json.loads(result.extracted_content)
+            print("Extracted Page Summary:")
+            print(json.dumps(page_summary, indent=2))
+            
+            # Save the extracted data
+            with open(".data/page_summary.json", "w", encoding="utf-8") as f:
+                json.dump(page_summary, f, indent=2)
+            print("Page summary saved to .data/page_summary.json")
+        else:
+            print(f"Failed to crawl and summarize the page. Error: {result.error_message}")
+
+    # Run the async main function
+    asyncio.run(main())
    ```

-### Explanation
+## Explanation

- **Importing Modules**: Import the necessary modules, including `WebCrawler` and `LLMExtractionStrategy` from `Crawl4AI`.
- **URL Definition**: Set the URL of the web page you want to crawl and summarize.
- **WebCrawler Initialization**: Create an instance of `WebCrawler` and call the `warmup` method to prepare the crawler.
- **Data Model Definition**: Define the structure of the data you want to extract using Pydantic's `BaseModel`.
- **Crawler Execution**: Run the crawler with the `LLMExtractionStrategy`, providing the schema and detailed instructions for the extraction process.
- **Data Processing**: Load the extracted content into a JSON object and print it to verify the results.
- **Data Saving**: Save the extracted data to a file for further use.
+- **Importing Modules**: We import the necessary modules, including `AsyncWebCrawler` and `LLMExtractionStrategy` from Crawl4AI.
+- **URL Definition**: We set the URL of the web page to crawl and summarize.
+- **Data Model Definition**: We define the structure of the data to extract using Pydantic's `BaseModel`.
+- **Extraction Strategy Setup**: We create an instance of `LLMExtractionStrategy` with the schema and detailed instructions for the extraction process.
+- **Async Crawl Function**: We define an asynchronous function `crawl_and_summarize` that uses `AsyncWebCrawler` to perform the crawling and extraction.
+- **Main Execution**: In the `main` function, we run the crawler, process the results, and save the extracted data.

-This example demonstrates how to harness the power of `Crawl4AI` to perform advanced web crawling and data extraction tasks with minimal code.
+## Advanced Usage: Crawling Multiple URLs
+
+To demonstrate the power of `AsyncWebCrawler`, here's how you can summarize multiple pages concurrently:
+
+```python
+async def crawl_multiple_urls(urls):
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        tasks = [crawler.arun(
+            url=url,
+            word_count_threshold=1,
+            extraction_strategy=extraction_strategy,
+            chunking_strategy=RegexChunking(),
+            bypass_cache=True
+        ) for url in urls]
+        results = await asyncio.gather(*tasks)
+    return results
+
+async def main():
+    urls = [
+        'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot',
+        'https://marketplace.visualstudio.com/items?itemName=GitHub.copilot',
+        'https://marketplace.visualstudio.com/items?itemName=ms-python.python'
+    ]
+    results = await crawl_multiple_urls(urls)
+    
+    for i, result in enumerate(results):
+        if result.success:
+            page_summary = json.loads(result.extracted_content)
+            print(f"\nSummary for URL {i+1}:")
+            print(json.dumps(page_summary, indent=2))
+        else:
+            print(f"\nFailed to summarize URL {i+1}. Error: {result.error_message}")
+
+asyncio.run(main())
+```
+
+This advanced example shows how to use `AsyncWebCrawler` to efficiently summarize multiple web pages concurrently, significantly reducing the total processing time compared to sequential crawling.
+
+By leveraging the asynchronous capabilities of Crawl4AI, you can perform advanced web crawling and data extraction tasks with improved efficiency and scalability.
--- a/docs/md/full_details/advanced_jsoncss_extraction.md
+++ b/docs/md/full_details/advanced_jsoncss_extraction.md
@@ -0,0 +1,282 @@
+# Advanced Usage of JsonCssExtractionStrategy
+
+While the basic usage of JsonCssExtractionStrategy is powerful for simple structures, its true potential shines when dealing with complex, nested HTML structures. This section will explore advanced usage scenarios, demonstrating how to extract nested objects, lists, and nested lists.
+
+## Hypothetical Website Example
+
+Let's consider a hypothetical e-commerce website that displays product categories, each containing multiple products. Each product has details, reviews, and related items. This complex structure will allow us to demonstrate various advanced features of JsonCssExtractionStrategy.
+
+Assume the HTML structure looks something like this:
+
+```html
+<div class="category">
+  <h2 class="category-name">Electronics</h2>
+  <div class="product">
+    <h3 class="product-name">Smartphone X</h3>
+    <p class="product-price">$999</p>
+    <div class="product-details">
+      <span class="brand">TechCorp</span>
+      <span class="model">X-2000</span>
+    </div>
+    <ul class="product-features">
+      <li>5G capable</li>
+      <li>6.5" OLED screen</li>
+      <li>128GB storage</li>
+    </ul>
+    <div class="product-reviews">
+      <div class="review">
+        <span class="reviewer">John D.</span>
+        <span class="rating">4.5</span>
+        <p class="review-text">Great phone, love the camera!</p>
+      </div>
+      <div class="review">
+        <span class="reviewer">Jane S.</span>
+        <span class="rating">5</span>
+        <p class="review-text">Best smartphone I've ever owned.</p>
+      </div>
+    </div>
+    <ul class="related-products">
+      <li>
+        <span class="related-name">Phone Case</span>
+        <span class="related-price">$29.99</span>
+      </li>
+      <li>
+        <span class="related-name">Screen Protector</span>
+        <span class="related-price">$9.99</span>
+      </li>
+    </ul>
+  </div>
+  <!-- More products... -->
+</div>
+```
+
+Now, let's create a schema to extract this complex structure:
+
+```python
+schema = {
+    "name": "E-commerce Product Catalog",
+    "baseSelector": "div.category",
+    "fields": [
+        {
+            "name": "category_name",
+            "selector": "h2.category-name",
+            "type": "text"
+        },
+        {
+            "name": "products",
+            "selector": "div.product",
+            "type": "nested_list",
+            "fields": [
+                {
+                    "name": "name",
+                    "selector": "h3.product-name",
+                    "type": "text"
+                },
+                {
+                    "name": "price",
+                    "selector": "p.product-price",
+                    "type": "text"
+                },
+                {
+                    "name": "details",
+                    "selector": "div.product-details",
+                    "type": "nested",
+                    "fields": [
+                        {
+                            "name": "brand",
+                            "selector": "span.brand",
+                            "type": "text"
+                        },
+                        {
+                            "name": "model",
+                            "selector": "span.model",
+                            "type": "text"
+                        }
+                    ]
+                },
+                {
+                    "name": "features",
+                    "selector": "ul.product-features li",
+                    "type": "list",
+                    "fields": [
+                        {
+                            "name": "feature",
+                            "type": "text"
+                        }
+                    ]
+                },
+                {
+                    "name": "reviews",
+                    "selector": "div.review",
+                    "type": "nested_list",
+                    "fields": [
+                        {
+                            "name": "reviewer",
+                            "selector": "span.reviewer",
+                            "type": "text"
+                        },
+                        {
+                            "name": "rating",
+                            "selector": "span.rating",
+                            "type": "text"
+                        },
+                        {
+                            "name": "comment",
+                            "selector": "p.review-text",
+                            "type": "text"
+                        }
+                    ]
+                },
+                {
+                    "name": "related_products",
+                    "selector": "ul.related-products li",
+                    "type": "list",
+                    "fields": [
+                        {
+                            "name": "name",
+                            "selector": "span.related-name",
+                            "type": "text"
+                        },
+                        {
+                            "name": "price",
+                            "selector": "span.related-price",
+                            "type": "text"
+                        }
+                    ]
+                }
+            ]
+        }
+    ]
+}
+```
+
+This schema demonstrates several advanced features:
+
+1. **Nested Objects**: The `details` field is a nested object within each product.
+2. **Simple Lists**: The `features` field is a simple list of text items.
+3. **Nested Lists**: The `products` field is a nested list, where each item is a complex object.
+4. **Lists of Objects**: The `reviews` and `related_products` fields are lists of objects.
+
+Let's break down the key concepts:
+
+### Nested Objects
+
+To create a nested object, use `"type": "nested"` and provide a `fields` array for the nested structure:
+
+```python
+{
+    "name": "details",
+    "selector": "div.product-details",
+    "type": "nested",
+    "fields": [
+        {
+            "name": "brand",
+            "selector": "span.brand",
+            "type": "text"
+        },
+        {
+            "name": "model",
+            "selector": "span.model",
+            "type": "text"
+        }
+    ]
+}
+```
+
+### Simple Lists
+
+For a simple list of identical items, use `"type": "list"`:
+
+```python
+{
+    "name": "features",
+    "selector": "ul.product-features li",
+    "type": "list",
+    "fields": [
+        {
+            "name": "feature",
+            "type": "text"
+        }
+    ]
+}
+```
+
+### Nested Lists
+
+For a list of complex objects, use `"type": "nested_list"`:
+
+```python
+{
+    "name": "products",
+    "selector": "div.product",
+    "type": "nested_list",
+    "fields": [
+        // ... fields for each product
+    ]
+}
+```
+
+### Lists of Objects
+
+Similar to nested lists, but typically used for simpler objects within the list:
+
+```python
+{
+    "name": "related_products",
+    "selector": "ul.related-products li",
+    "type": "list",
+    "fields": [
+        {
+            "name": "name",
+            "selector": "span.related-name",
+            "type": "text"
+        },
+        {
+            "name": "price",
+            "selector": "span.related-price",
+            "type": "text"
+        }
+    ]
+}
+```
+
+## Using the Advanced Schema
+
+To use this advanced schema with AsyncWebCrawler:
+
+```python
+import json
+import asyncio
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+async def extract_complex_product_data():
+    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html",
+            extraction_strategy=extraction_strategy,
+            bypass_cache=True,
+        )
+
+        assert result.success, "Failed to crawl the page"
+
+        product_data = json.loads(result.extracted_content)
+        print(json.dumps(product_data, indent=2))
+
+asyncio.run(extract_complex_product_data())
+```
+
+This will produce a structured JSON output that captures the complex hierarchy of the product catalog, including nested objects, lists, and nested lists.
+
+## Tips for Advanced Usage
+
+1. **Start Simple**: Begin with a basic schema and gradually add complexity.
+2. **Test Incrementally**: Test each part of your schema separately before combining them.
+3. **Use Chrome DevTools**: The Element Inspector is invaluable for identifying the correct selectors.
+4. **Handle Missing Data**: Use the `default` key in your field definitions to handle cases where data might be missing.
+5. **Leverage Transforms**: Use the `transform` key to clean or format extracted data (e.g., converting prices to numbers).
+6. **Consider Performance**: Very complex schemas might slow down extraction. Balance complexity with performance needs.
+
+By mastering these advanced techniques, you can use JsonCssExtractionStrategy to extract highly structured data from even the most complex web pages, making it a powerful tool for web scraping and data analysis tasks.
--- a/docs/md/full_details/crawl_request_parameters.md
+++ b/docs/md/full_details/crawl_request_parameters.md
@@ -1,6 +1,6 @@
-# Crawl Request Parameters
+# Crawl Request Parameters for AsyncWebCrawler

-The `run` function in Crawl4AI is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `run` function, along with their descriptions, possible values, and examples.
+The `arun` method in Crawl4AI's `AsyncWebCrawler` is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `arun` method, along with their descriptions, possible values, and examples.

 ## Parameters

@@ -13,9 +13,9 @@ url = "https://www.nbcnews.com/business"
 ```

 ### word_count_threshold (int)
-**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is `5`.
+**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is defined by `MIN_WORD_THRESHOLD`.
 **Required:** No
-**Default Value:** `5`
+**Default Value:** `MIN_WORD_THRESHOLD`
 **Example:**
 ```python
 word_count_threshold = 10
@@ -88,43 +88,92 @@ verbose = True
 Additional keyword arguments that can be passed to customize the crawling process further. Some notable options include:

 - **only_text (bool):** Whether to extract only text content, excluding HTML tags. Default is `False`.
+- **session_id (str):** A unique identifier for the crawling session. This is useful for maintaining state across multiple requests.
+- **js_code (str or list):** JavaScript code to be executed on the page before extraction.
+- **wait_for (str):** A CSS selector or JavaScript function to wait for before considering the page load complete.

 **Example:**
 ```python
-result = crawler.run(
+result = await crawler.arun(
    url="https://www.nbcnews.com/business",
    css_selector="p",
-    only_text=True
+    only_text=True,
+    session_id="unique_session_123",
+    js_code="window.scrollTo(0, document.body.scrollHeight);",
+    wait_for="article.main-article"
 )
 ```

 ## Example Usage

-Here's an example of how to use the `run` function with various parameters:
+Here's an example of how to use the `arun` method with various parameters:

 ```python
-from crawl4ai import WebCrawler
+import asyncio
+from crawl4ai import AsyncWebCrawler
 from crawl4ai.extraction_strategy import CosineStrategy
 from crawl4ai.chunking_strategy import NlpSentenceChunking

-# Create the WebCrawler instance 
-crawler = WebCrawler() 
+async def main():
+    # Create the AsyncWebCrawler instance 
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Run the crawler with custom parameters
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            word_count_threshold=10,
+            extraction_strategy=CosineStrategy(semantic_filter="finance"),
+            chunking_strategy=NlpSentenceChunking(),
+            bypass_cache=True,
+            css_selector="div.article-content",
+            screenshot=True,
+            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
+            verbose=True,
+            only_text=True,
+            session_id="business_news_session",
+            js_code="window.scrollTo(0, document.body.scrollHeight);",
+            wait_for="footer"
+        )

-# Run the crawler with custom parameters
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    word_count_threshold=10,
-    extraction_strategy=CosineStrategy(semantic_filter="finance"),
-    chunking_strategy=NlpSentenceChunking(),
-    bypass_cache=True,
-    css_selector="div.article-content",
-    screenshot=True,
-    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
-    verbose=True,
-    only_text=True
-)
+        print(result)

-print(result)
+# Run the async function
+asyncio.run(main())
 ```

-This example demonstrates how to configure various parameters to customize the crawling and extraction process using Crawl4AI.
+This example demonstrates how to configure various parameters to customize the crawling and extraction process using the asynchronous version of Crawl4AI.
+
+## Additional Asynchronous Methods
+
+The `AsyncWebCrawler` class also provides other useful asynchronous methods:
+
+### arun_many
+**Description:** Crawl multiple URLs concurrently.
+**Example:**
+```python
+urls = ["https://example1.com", "https://example2.com", "https://example3.com"]
+results = await crawler.arun_many(urls, word_count_threshold=10, bypass_cache=True)
+```
+
+### aclear_cache
+**Description:** Clear the crawler's cache.
+**Example:**
+```python
+await crawler.aclear_cache()
+```
+
+### aflush_cache
+**Description:** Completely flush the crawler's cache.
+**Example:**
+```python
+await crawler.aflush_cache()
+```
+
+### aget_cache_size
+**Description:** Get the current size of the cache.
+**Example:**
+```python
+cache_size = await crawler.aget_cache_size()
+print(f"Current cache size: {cache_size}")
+```
+
+These asynchronous methods allow for efficient and flexible use of the AsyncWebCrawler in various scenarios.
--- a/docs/md/full_details/crawl_result_class.md
+++ b/docs/md/full_details/crawl_result_class.md
@@ -5,6 +5,9 @@ The `CrawlResult` class is the heart of Crawl4AI's output, encapsulating all the
 ## Class Definition

 ```python
+from pydantic import BaseModel
+from typing import Dict, List, Optional
+
 class CrawlResult(BaseModel):
    url: str
    html: str
@@ -17,6 +20,9 @@ class CrawlResult(BaseModel):
    extracted_content: Optional[str] = None
    metadata: Optional[dict] = None
    error_message: Optional[str] = None
+    session_id: Optional[str] = None
+    responser_headers: Optional[dict] = None
+    status_code: Optional[int] = None
 ```

 ## Fields Explanation
@@ -34,7 +40,7 @@ A flag indicating whether the crawling and extraction were successful. If any er
 The cleaned HTML content of the web page. This field holds the HTML after removing unwanted tags like `<script>`, `<style>`, and others that do not contribute to the useful content.

 ### `media: Dict[str, List[Dict]]`
-A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here’s how they are structured:
+A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here's how they are structured:

 - **Images**: Each image is represented as a dictionary with `src` (source URL) and `alt` (alternate text).
 - **Videos**: Each video is represented similarly with `src` and `alt`.
@@ -88,33 +94,11 @@ A dictionary containing metadata extracted from the web page, such as title, des
 ### `error_message: Optional[str]`
 If an error occurs during crawling, this field will contain the error message, helping you debug and understand what went wrong. 🚨

-## Example Usage
+### `session_id: Optional[str]`
+A unique identifier for the crawling session. This can be useful for tracking and managing multiple crawling sessions.

-Here's a quick example to illustrate how you might use the `CrawlResult` in your code:
+### `responser_headers: Optional[dict]`
+A dictionary containing the response headers from the web server. This can provide additional information about the server and the response.

-```python
-from crawl4ai import WebCrawler
-
-# Create the WebCrawler instance
-crawler = WebCrawler()
-
-# Run the crawler on a URL
-result = crawler.run(url="https://www.example.com")
-
-# Check if the crawl was successful
-if result.success:
-    print("Crawl succeeded!")
-    print("URL:", result.url)
-    print("HTML:", result.html[:100])  # Print the first 100 characters of the HTML
-    print("Cleaned HTML:", result.cleaned_html[:100])
-    print("Media:", result.media)
-    print("Links:", result.links)
-    print("Screenshot:", result.screenshot)
-    print("Markdown:", result.markdown[:100])
-    print("Extracted Content:", result.extracted_content)
-    print("Metadata:", result.metadata)
-else:
-    print("Crawl failed with error:", result.error_message)
-```
-
-With this setup, you can easily access all the valuable data extracted from the web page and integrate it into your applications. Happy crawling! 🕷️🤖
+### `status_code: Optional[int]`
+The HTTP status code of the response. This indicates the success or failure of the HTTP request (e.g., 200 for success, 404 for not found, etc.).
--- a/docs/md/full_details/extraction_strategies.md
+++ b/docs/md/full_details/extraction_strategies.md
@@ -1,6 +1,143 @@
 ## Extraction Strategies 🧠

-Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into two of the most important strategies: `CosineStrategy` and `LLMExtractionStrategy`.
+Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into three of the most important strategies: `CosineStrategy`, `LLMExtractionStrategy`, and the new `JsonCssExtractionStrategy`.
+
+### LLMExtractionStrategy
+
+`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
+
+#### When to Use
+- Suitable for complex extraction tasks requiring nuanced understanding.
+- Ideal for scenarios where detailed instructions can guide the extraction process.
+- Perfect for extracting specific types of information or content with precise guidelines.
+
+#### Parameters
+- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
+- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
+- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
+
+#### Example Without Instructions
+```python
+import asyncio
+import os
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Define extraction strategy without instructions
+        strategy = LLMExtractionStrategy(
+            provider='openai',
+            api_token=os.getenv('OPENAI_API_KEY')
+        )
+
+        # Sample URL
+        url = "https://www.nbcnews.com/business"
+
+        # Run the crawler with the extraction strategy
+        result = await crawler.arun(url=url, extraction_strategy=strategy)
+        print(result.extracted_content)
+
+asyncio.run(main())
+```
+
+#### Example With Instructions
+```python
+import asyncio
+import os
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import LLMExtractionStrategy
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Define extraction strategy with instructions
+        strategy = LLMExtractionStrategy(
+            provider='openai',
+            api_token=os.getenv('OPENAI_API_KEY'),
+            instruction="Extract only financial news and summarize key points."
+        )
+
+        # Sample URL
+        url = "https://www.nbcnews.com/business"
+
+        # Run the crawler with the extraction strategy
+        result = await crawler.arun(url=url, extraction_strategy=strategy)
+        print(result.extracted_content)
+
+asyncio.run(main())
+```
+
+### JsonCssExtractionStrategy
+
+`JsonCssExtractionStrategy` is a powerful tool for extracting structured data from HTML using CSS selectors. It allows you to define a schema that maps CSS selectors to specific fields, enabling precise and efficient data extraction.
+
+#### When to Use
+- Ideal for extracting structured data from websites with consistent HTML structures.
+- Perfect for scenarios where you need to extract specific elements or attributes from a webpage.
+- Suitable for creating datasets from web pages with tabular or list-based information.
+
+#### Parameters
+- `schema` (Dict[str, Any]): A dictionary defining the extraction schema, including base selector and field definitions.
+
+#### Example
+```python
+import asyncio
+import json
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Define the extraction schema
+        schema = {
+            "name": "News Articles",
+            "baseSelector": "article.tease-card",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h2",
+                    "type": "text",
+                },
+                {
+                    "name": "summary",
+                    "selector": "div.tease-card__info",
+                    "type": "text",
+                },
+                {
+                    "name": "link",
+                    "selector": "a",
+                    "type": "attribute",
+                    "attribute": "href"
+                }
+            ],
+        }
+
+        # Create the extraction strategy
+        strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        # Sample URL
+        url = "https://www.nbcnews.com/business"
+
+        # Run the crawler with the extraction strategy
+        result = await crawler.arun(url=url, extraction_strategy=strategy)
+        
+        # Parse and print the extracted content
+        extracted_data = json.loads(result.extracted_content)
+        print(json.dumps(extracted_data, indent=2))
+
+asyncio.run(main())
+```
+
+#### Use Cases for JsonCssExtractionStrategy
+- Extracting product information from e-commerce websites.
+- Gathering news articles and their metadata from news portals.
+- Collecting user reviews and ratings from review websites.
+- Extracting job listings from job boards.
+
+By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy`, nuanced, instruction-based extraction with `LLMExtractionStrategy`, or precise structured data extraction with `JsonCssExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨
+
+For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](../full_details/advanced_jsoncss_extraction.md).
+

 ### CosineStrategy

@@ -21,96 +158,28 @@ Crawl4AI offers powerful extraction strategies to derive meaningful information

 #### Example
 ```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
 from crawl4ai.extraction_strategy import CosineStrategy
-from crawl4ai import WebCrawler

-crawler = WebCrawler()
-crawler.warmup()
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Define extraction strategy
+        strategy = CosineStrategy(
+            semantic_filter="finance economy stock market",
+            word_count_threshold=10,
+            max_dist=0.2,
+            linkage_method='ward',
+            top_k=3,
+            model_name='BAAI/bge-small-en-v1.5'
+        )

-# Define extraction strategy
-strategy = CosineStrategy(
-    semantic_filter="finance economy stock market",
-    word_count_threshold=10,
-    max_dist=0.2,
-    linkage_method='ward',
-    top_k=3,
-    model_name='BAAI/bge-small-en-v1.5'
-)
+        # Sample URL
+        url = "https://www.nbcnews.com/business"

-# Sample URL
-url = "https://www.nbcnews.com/business"
+        # Run the crawler with the extraction strategy
+        result = await crawler.arun(url=url, extraction_strategy=strategy)
+        print(result.extracted_content)

-# Run the crawler with the extraction strategy
-result = crawler.run(url=url, extraction_strategy=strategy)
-print(result.extracted_content)
+asyncio.run(main())
 ```
-
-### LLMExtractionStrategy
-
-`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
-
-#### When to Use
- Suitable for complex extraction tasks requiring nuanced understanding.
- Ideal for scenarios where detailed instructions can guide the extraction process.
- Perfect for extracting specific types of information or content with precise guidelines.
-
-#### Parameters
- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
-
-#### Example Without Instructions
-```python
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-from crawl4ai import WebCrawler
-
-crawler = WebCrawler()
-crawler.warmup()
-
-# Define extraction strategy without instructions
-strategy = LLMExtractionStrategy(
-    provider='openai',
-    api_token='your_api_token'
-)
-
-# Sample URL
-url = "https://www.nbcnews.com/business"
-
-# Run the crawler with the extraction strategy
-result = crawler.run(url=url, extraction_strategy=strategy)
-print(result.extracted_content)
-```
-
-#### Example With Instructions
-```python
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-from crawl4ai import WebCrawler
-
-crawler = WebCrawler()
-crawler.warmup()
-
-# Define extraction strategy with instructions
-strategy = LLMExtractionStrategy(
-    provider='openai',
-    api_token='your_api_token',
-    instruction="Extract only financial news and summarize key points."
-)
-
-# Sample URL
-url = "https://www.nbcnews.com/business"
-
-# Run the crawler with the extraction strategy
-result = crawler.run(url=url, extraction_strategy=strategy)
-print(result.extracted_content)
-```
-
-#### Use Cases for LLMExtractionStrategy
- Extracting specific data types from structured or semi-structured content.
- Generating summaries, extracting key information, or transforming content into different formats.
- Performing detailed extractions based on custom instructions.
-
-For more detailed examples, please refer to the [Examples section](../examples/index.md) of the documentation.
-
---
-
-By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy` or nuanced, instruction-based extraction with `LLMExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️‍♂️✨
--- a/docs/md/full_details/session_based_crawling.md
+++ b/docs/md/full_details/session_based_crawling.md
@@ -0,0 +1,276 @@
+# Session-Based Crawling for Dynamic Content
+
+In modern web applications, content is often loaded dynamically without changing the URL. Examples include "Load More" buttons, infinite scrolling, or paginated content that updates via JavaScript. To effectively crawl such websites, Crawl4AI provides powerful session-based crawling capabilities.
+
+This guide will explore advanced techniques for crawling dynamic content using Crawl4AI's session management features.
+
+## Understanding Session-Based Crawling
+
+Session-based crawling allows you to maintain a persistent browser session across multiple requests. This is crucial when:
+
+1. The content changes dynamically without URL changes
+2. You need to interact with the page (e.g., clicking buttons) between requests
+3. The site requires authentication or maintains state across pages
+
+Crawl4AI's `AsyncWebCrawler` class supports session-based crawling through the `session_id` parameter and related methods.
+
+## Basic Concepts
+
+Before diving into examples, let's review some key concepts:
+
+- **Session ID**: A unique identifier for a browsing session. Use the same `session_id` across multiple `arun` calls to maintain state.
+- **JavaScript Execution**: Use the `js_code` parameter to execute JavaScript on the page, such as clicking a "Load More" button.
+- **CSS Selectors**: Use these to target specific elements for extraction or interaction.
+- **Extraction Strategy**: Define how to extract structured data from the page.
+- **Wait Conditions**: Specify conditions to wait for before considering the page loaded.
+
+## Example 1: Basic Session-Based Crawling
+
+Let's start with a basic example of session-based crawling:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def basic_session_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        session_id = "my_session"
+        url = "https://example.com/dynamic-content"
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                js_code="document.querySelector('.load-more-button').click();" if page > 0 else None,
+                css_selector=".content-item",
+                bypass_cache=True
+            )
+            
+            print(f"Page {page + 1}: Found {result.extracted_content.count('.content-item')} items")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+
+asyncio.run(basic_session_crawl())
+```
+
+This example demonstrates:
+1. Using a consistent `session_id` across multiple `arun` calls
+2. Executing JavaScript to load more content after the first page
+3. Using a CSS selector to extract specific content
+4. Properly closing the session after crawling
+
+## Advanced Technique 1: Custom Execution Hooks
+
+Crawl4AI allows you to set custom hooks that execute at different stages of the crawling process. This is particularly useful for handling complex loading scenarios.
+
+Here's an example that waits for new content to appear before proceeding:
+
+```python
+async def advanced_session_crawl_with_hooks():
+    first_commit = ""
+
+    async def on_execution_started(page):
+        nonlocal first_commit
+        try:
+            while True:
+                await page.wait_for_selector("li.commit-item h4")
+                commit = await page.query_selector("li.commit-item h4")
+                commit = await commit.evaluate("(element) => element.textContent")
+                commit = commit.strip()
+                if commit and commit != first_commit:
+                    first_commit = commit
+                    break
+                await asyncio.sleep(0.5)
+        except Exception as e:
+            print(f"Warning: New content didn't appear after JavaScript execution: {e}")
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started)
+
+        url = "https://github.com/example/repo/commits/main"
+        session_id = "commit_session"
+        all_commits = []
+
+        js_next_page = """
+        const button = document.querySelector('a.pagination-next');
+        if (button) button.click();
+        """
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.commit-item",
+                js_code=js_next_page if page > 0 else None,
+                bypass_cache=True,
+                js_only=page > 0
+            )
+
+            commits = result.extracted_content.select("li.commit-item")
+            all_commits.extend(commits)
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(advanced_session_crawl_with_hooks())
+```
+
+This technique uses a custom `on_execution_started` hook to ensure new content has loaded before proceeding to the next step.
+
+## Advanced Technique 2: Integrated JavaScript Execution and Waiting
+
+Instead of using separate hooks, you can integrate the waiting logic directly into your JavaScript execution. This approach can be more concise and easier to manage for some scenarios.
+
+Here's an example:
+
+```python
+async def integrated_js_and_wait_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/example/repo/commits/main"
+        session_id = "integrated_session"
+        all_commits = []
+
+        js_next_page_and_wait = """
+        (async () => {
+            const getCurrentCommit = () => {
+                const commits = document.querySelectorAll('li.commit-item h4');
+                return commits.length > 0 ? commits[0].textContent.trim() : null;
+            };
+
+            const initialCommit = getCurrentCommit();
+            const button = document.querySelector('a.pagination-next');
+            if (button) button.click();
+
+            while (true) {
+                await new Promise(resolve => setTimeout(resolve, 100));
+                const newCommit = getCurrentCommit();
+                if (newCommit && newCommit !== initialCommit) {
+                    break;
+                }
+            }
+        })();
+        """
+
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.commit-item",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.commit-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.commit-item",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page_and_wait if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True
+            )
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(integrated_js_and_wait_crawl())
+```
+
+This approach combines the JavaScript for clicking the "next" button and waiting for new content to load into a single script.
+
+## Advanced Technique 3: Using the `wait_for` Parameter
+
+Crawl4AI provides a `wait_for` parameter that allows you to specify a condition to wait for before considering the page fully loaded. This can be particularly useful for dynamic content.
+
+Here's an example:
+
+```python
+async def wait_for_parameter_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/example/repo/commits/main"
+        session_id = "wait_for_session"
+        all_commits = []
+
+        js_next_page = """
+        const commits = document.querySelectorAll('li.commit-item h4');
+        if (commits.length > 0) {
+            window.lastCommit = commits[0].textContent.trim();
+        }
+        const button = document.querySelector('a.pagination-next');
+        if (button) button.click();
+        """
+
+        wait_for = """() => {
+            const commits = document.querySelectorAll('li.commit-item h4');
+            if (commits.length === 0) return false;
+            const firstCommit = commits[0].textContent.trim();
+            return firstCommit !== window.lastCommit;
+        }"""
+        
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.commit-item",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.commit-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
+
+        for page in range(3):
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.commit-item",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page if page > 0 else None,
+                wait_for=wait_for if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True
+            )
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(wait_for_parameter_crawl())
+```
+
+This technique separates the JavaScript execution (clicking the "next" button) from the waiting condition, providing more flexibility and clarity in some scenarios.
+
+## Best Practices for Session-Based Crawling
+
+1. **Use Unique Session IDs**: Ensure each crawling session has a unique `session_id` to prevent conflicts.
+2. **Close Sessions**: Always close sessions using `kill_session` when you're done to free up resources.
+3. **Handle Errors**: Implement proper error handling to deal with unexpected situations during crawling.
+4. **Respect Website Terms**: Ensure your crawling adheres to the website's terms of service and robots.txt file.
+5. **Implement Delays**: Add appropriate delays between requests to avoid overwhelming the target server.
+6. **Use Extraction Strategies**: Leverage `JsonCssExtractionStrategy` or other extraction strategies for structured data extraction.
+7. **Optimize JavaScript**: Keep your JavaScript execution concise and efficient to improve crawling speed.
+8. **Monitor Performance**: Keep an eye on memory usage and crawling speed, especially for long-running sessions.
+
+## Conclusion
+
+Session-based crawling with Crawl4AI provides powerful capabilities for handling dynamic content and complex web applications. By leveraging session management, JavaScript execution, and waiting strategies, you can effectively crawl and extract data from a wide range of modern websites.
+
+Remember to use these techniques responsibly and in compliance with website policies and ethical web scraping practices.
+
+For more advanced usage and API details, refer to the Crawl4AI API documentation.
--- a/docs/md/index.md
+++ b/docs/md/index.md
@@ -1,43 +1,39 @@
-# Crawl4AI v0.2.7
+# Crawl4AI

 Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.

-
-## Try the [Demo](demo.md)
-
-Just try it now and crawl different pages to see how it works. You can set the links, see the structures of the output, and also view the Python sample code on how to run it. The old demo is available at [/old_demo](/old) where you can see more details.
-
 ## Introduction

-Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution.
+Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution with full asynchronous support.

 ## Quick Start

-Here's a quick example to show you how easy it is to use Crawl4AI:
+Here's a quick example to show you how easy it is to use Crawl4AI with its new asynchronous capabilities:

 ```python
-from crawl4ai import WebCrawler
+import asyncio
+from crawl4ai import AsyncWebCrawler

-# Create an instance of WebCrawler
-crawler = WebCrawler()
+async def main():
+    # Create an instance of AsyncWebCrawler
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # Run the crawler on a URL
+        result = await crawler.arun(url="https://www.nbcnews.com/business")

-# Warm up the crawler (load necessary models)
-crawler.warmup()
+        # Print the extracted content
+        print(result.markdown)

-# Run the crawler on a URL
-result = crawler.run(url="https://www.nbcnews.com/business")
-
-# Print the extracted content
-print(result.extracted_content)
+# Run the async main function
+asyncio.run(main())
 ```

 ### Explanation

-1. **Importing the Library**: We start by importing the `WebCrawler` class from the `crawl4ai` library.
-2. **Creating an Instance**: An instance of `WebCrawler` is created.
-3. **Warming Up**: The `warmup()` method prepares the crawler by loading necessary models and settings.
-4. **Running the Crawler**: The `run()` method is used to crawl the specified URL and extract meaningful content.
-5. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
+1. **Importing the Library**: We start by importing the `AsyncWebCrawler` class from the `crawl4ai` library and the `asyncio` module.
+2. **Creating an Async Context**: We use an async context manager to create an instance of `AsyncWebCrawler`.
+3. **Running the Crawler**: The `arun()` method is used to asynchronously crawl the specified URL and extract meaningful content.
+4. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
+5. **Running the Async Function**: We use `asyncio.run()` to execute our async main function.

 ## Documentation Structure

@@ -63,6 +59,7 @@ A step-by-step guide to get you up and running with Crawl4AI, including installa

 This section contains practical examples demonstrating different use cases of Crawl4AI:

+- [Structured Data Extraction](examples/json_css_extraction.md)
 - [LLM Extraction](examples/llm_extraction.md)
 - [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
 - [Hooks & Auth](examples/hooks_auth.md)
@@ -75,17 +72,12 @@ Comprehensive details on using the crawler, including:

 - [Crawl Request Parameters](full_details/crawl_request_parameters.md)
 - [Crawl Result Class](full_details/crawl_result_class.md)
+- [Session Based Crawling](full_details/session_based_crawling.md)
+- [Advanced Structured Data Extraction JsonCssExtraction](full_details/advanced_jsoncss_extraction.md)
 - [Advanced Features](full_details/advanced_features.md)
 - [Chunking Strategies](full_details/chunking_strategies.md)
 - [Extraction Strategies](full_details/extraction_strategies.md)

-### [API Reference](api/core_classes_and_functions.md)
-
-Detailed documentation of the API, covering:
-
- [Core Classes and Functions](api/core_classes_and_functions.md)
- [Detailed API Documentation](api/detailed_api_documentation.md)
-
 ### [Change Log](changelog.md)

 A log of all changes, updates, and improvements made to Crawl4AI.
@@ -96,6 +88,6 @@ Information on how to get in touch with the developers, report issues, and contr

 ## Get Started

-To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier and more efficient.
+To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier, more efficient, and now fully asynchronous.

-Happy Crawling! 🕸️🚀
+Happy Crawling! 🕸️🚀
--- a/docs/md/installation.md
+++ b/docs/md/installation.md
@@ -1,46 +1,92 @@
 # Installation 💻

-There are three ways to use Crawl4AI:
-1. As a library (Recommended)
-2. As a local server (Docker) or using the REST API
-3. As a Google Colab notebook. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
+Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package, use it with Docker, or run it as a local server.

-## Library Installation
+## Option 1: Python Package Installation (Recommended)

-To install Crawl4AI as a library, follow these steps:
+Crawl4AI is now available on PyPI, making installation easier than ever. Choose the option that best fits your needs:

-1. Install the package from GitHub:
-```
-virtualenv venv
-source venv/bin/activate
-pip install "crawl4ai[all] @ git+https://github.com/unclecode/crawl4ai.git"
+### Basic Installation
+
+For basic web crawling and scraping tasks:
+
+```bash
+pip install crawl4ai
+playwright install # Install Playwright dependencies
 ```

-💡 Better to run the following CLI-command to load the required models. This is optional, but it will boost the performance and speed of the crawler. You need to do this only once.
+### Installation with PyTorch
+
+For advanced text clustering (includes CosineSimilarity cluster strategy):
+
+```bash
+pip install crawl4ai[torch]
 ```
+
+### Installation with Transformers
+
+For text summarization and Hugging Face models:
+
+```bash
+pip install crawl4ai[transformer]
+```
+
+### Full Installation
+
+For all features:
+
+```bash
+pip install crawl4ai[all]
+```
+
+### Development Installation
+
+For contributors who plan to modify the source code:
+
+```bash
+git clone https://github.com/unclecode/crawl4ai.git
+cd crawl4ai
+pip install -e ".[all]"
+playwright install # Install Playwright dependencies
+```
+
+💡 After installation with "torch", "transformer", or "all" options, it's recommended to run the following CLI command to load the required models:
+
+```bash
 crawl4ai-download-models
 ```

-2. Alternatively, you can clone the repository and install the package locally:
-```
-virtualenv venv
-source venv/bin/activate
-git clone https://github.com/unclecode/crawl4ai.git
-cd crawl4ai
-pip install -e .[all]
+This is optional but will boost the performance and speed of the crawler. You only need to do this once after installation.
+
+## Option 2: Using Docker (Coming Soon)
+
+Docker support for Crawl4AI is currently in progress and will be available soon. This will allow you to run Crawl4AI in a containerized environment, ensuring consistency across different systems.
+
+## Option 3: Local Server Installation
+
+For those who prefer to run Crawl4AI as a local server, instructions will be provided once the Docker implementation is complete.
+
+## Verifying Your Installation
+
+After installation, you can verify that Crawl4AI is working correctly by running a simple Python script:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.example.com")
+        print(result.markdown[:500])  # Print first 500 characters
+
+if __name__ == "__main__":
+    asyncio.run(main())
 ```

-## Using Docker for Local Server
+This script should successfully crawl the example website and print the first 500 characters of the extracted content.

-3. Use Docker to run the local server:
-```
-# For Mac users
-# docker build --platform linux/amd64 -t crawl4ai .
-# For other users
-# docker build -t crawl4ai .
-docker run -d -p 8000:80 crawl4ai
-```
+## Getting Help

-## Using Google Colab
+If you encounter any issues during installation or usage, please check the [documentation](https://crawl4ai.com/mkdocs/) or raise an issue on the [GitHub repository](https://github.com/unclecode/crawl4ai/issues).

-You can also use Crawl4AI in a Google Colab notebook for easy setup and experimentation. Simply open the following Colab notebook and follow the instructions: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
+Happy crawling! 🕷️🤖
--- a/docs/md/introduction.md
+++ b/docs/md/introduction.md
@@ -20,18 +20,6 @@ Crawl4AI is designed to simplify the process of crawling web pages and extractin
 - **🎯 CSS Selector Support**: Extract specific content using CSS selectors.
 - **📝 Instruction/Keyword Refinement**: Pass instructions or keywords to refine the extraction process.

-## Recent Changes (v0.2.5) 🌟
-
- **New Hooks**: Added six important hooks to the crawler:
-  - 🟢 `on_driver_created`: Called when the driver is ready for initializations.
-  - 🔵 `before_get_url`: Called right before Selenium fetches the URL.
-  - 🟣 `after_get_url`: Called after Selenium fetches the URL.
-  - 🟠 `before_return_html`: Called when the data is parsed and ready.
-  - 🟡 `on_user_agent_updated`: Called when the user changes the user agent, causing the driver to reinitialize.
- **New Example**: Added an example in [`quickstart.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/quickstart.py) in the example folder under the docs.
- **Improved Semantic Context**: Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
- **Dockerfile Update**: Updated Dockerfile to ensure compatibility across multiple platforms.
-
 Check the [Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for more details.

 ## Power and Simplicity of Crawl4AI 🚀
--- a/docs/md/quickstart.md
+++ b/docs/md/quickstart.md
@@ -1,20 +1,22 @@
 # Quick Start Guide 🚀

-Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies. Let's dive in! 🌟
+Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies, all with the power of asynchronous programming. Let's dive in! 🌟

 ## Getting Started 🛠️

-First, let's create an instance of `WebCrawler` and call the `warmup()` function. This might take a few seconds the first time you run Crawl4AI, as it loads the required model files.
+First, let's import the necessary modules and create an instance of `AsyncWebCrawler`. We'll use an async context manager, which handles the setup and teardown of the crawler for us.

 ```python
-from crawl4ai import WebCrawler
+import asyncio
+from crawl4ai import AsyncWebCrawler

-def create_crawler():
-    crawler = WebCrawler(verbose=True)
-    crawler.warmup()
-    return crawler
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # We'll add our crawling code here
+        pass

-crawler = create_crawler()
+if __name__ == "__main__":
+    asyncio.run(main())
 ```

 ### Basic Usage
@@ -22,8 +24,12 @@ crawler = create_crawler()
 Simply provide a URL and let Crawl4AI do the magic!

 ```python
-result = crawler.run(url="https://www.nbcnews.com/business")
-print(f"Basic crawl result: {result}")
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.nbcnews.com/business")
+        print(f"Basic crawl result: {result.markdown[:500]}")  # Print first 500 characters
+
+asyncio.run(main())
 ```

 ### Taking Screenshots 📸
@@ -31,26 +37,34 @@ print(f"Basic crawl result: {result}")
 Let's take a screenshot of the page!

 ```python
-result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
-with open("screenshot.png", "wb") as f:
-    f.write(base64.b64decode(result.screenshot))
-print("Screenshot saved to 'screenshot.png'!")
+import base64
+
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(url="https://www.nbcnews.com/business", screenshot=True)
+        with open("screenshot.png", "wb") as f:
+            f.write(base64.b64decode(result.screenshot))
+        print("Screenshot saved to 'screenshot.png'!")
+
+asyncio.run(main())
 ```

 ### Understanding Parameters 🧠

 By default, Crawl4AI caches the results of your crawls. This means that subsequent crawls of the same URL will be much faster! Let's see this in action.

-First crawl (caches the result):
 ```python
-result = crawler.run(url="https://www.nbcnews.com/business")
-print(f"First crawl result: {result}")
-```
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        # First crawl (caches the result)
+        result1 = await crawler.arun(url="https://www.nbcnews.com/business")
+        print(f"First crawl result: {result1.markdown[:100]}...")

-Force to crawl again:
-```python
-result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
-print(f"Second crawl result: {result}")
+        # Force to crawl again
+        result2 = await crawler.arun(url="https://www.nbcnews.com/business", bypass_cache=True)
+        print(f"Second crawl result: {result2.markdown[:100]}...")
+
+asyncio.run(main())
 ```

 ### Adding a Chunking Strategy 🧩
@@ -60,157 +74,212 @@ Let's add a chunking strategy: `RegexChunking`! This strategy splits the text ba
 ```python
 from crawl4ai.chunking_strategy import RegexChunking

-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    chunking_strategy=RegexChunking(patterns=["\n\n"])
-)
-print(f"RegexChunking result: {result}")
-```
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            chunking_strategy=RegexChunking(patterns=["\n\n"])
+        )
+        print(f"RegexChunking result: {result.extracted_content[:200]}...")

-You can also use `NlpSentenceChunking` which splits the text into sentences using NLP techniques.
-
-```python
-from crawl4ai.chunking_strategy import NlpSentenceChunking
-
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    chunking_strategy=NlpSentenceChunking()
-)
-print(f"NlpSentenceChunking result: {result}")
+asyncio.run(main())
 ```

 ### Adding an Extraction Strategy 🧠

-Let's get smarter with an extraction strategy: `CosineStrategy`! This strategy uses cosine similarity to extract semantically similar blocks of text.
+Let's get smarter with an extraction strategy: `JsonCssExtractionStrategy`! This strategy extracts structured data from HTML using CSS selectors.

 ```python
-from crawl4ai.extraction_strategy import CosineStrategy
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+import json

-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=CosineStrategy(
-        word_count_threshold=10, 
-        max_dist=0.2, 
-        linkage_method="ward", 
-        top_k=3
-    )
-)
-print(f"CosineStrategy result: {result}")
-```
+async def main():
+    schema = {
+        "name": "News Articles",
+        "baseSelector": "article.tease-card",
+        "fields": [
+            {
+                "name": "title",
+                "selector": "h2",
+                "type": "text",
+            },
+            {
+                "name": "summary",
+                "selector": "div.tease-card__info",
+                "type": "text",
+            }
+        ],
+    }

-You can also pass other parameters like `semantic_filter` to extract specific content.
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            extraction_strategy=JsonCssExtractionStrategy(schema, verbose=True)
+        )
+        extracted_data = json.loads(result.extracted_content)
+        print(f"Extracted {len(extracted_data)} articles")
+        print(json.dumps(extracted_data[0], indent=2))

-```python
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=CosineStrategy(
-        semantic_filter="inflation rent prices"
-    )
-)
-print(f"CosineStrategy result with semantic filter: {result}")
+asyncio.run(main())
 ```

 ### Using LLMExtractionStrategy 🤖

-Time to bring in the big guns: `LLMExtractionStrategy` without instructions! This strategy uses a large language model to extract relevant information from the web page.
+Time to bring in the big guns: `LLMExtractionStrategy`! This strategy uses a large language model to extract relevant information from the web page.

 ```python
 from crawl4ai.extraction_strategy import LLMExtractionStrategy
 import os
+from pydantic import BaseModel, Field

-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=LLMExtractionStrategy(
-        provider="openai/gpt-4o", 
-        api_token=os.getenv('OPENAI_API_KEY')
-    )
-)
-print(f"LLMExtractionStrategy (no instructions) result: {result}")
-```
+class OpenAIModelFee(BaseModel):
+    model_name: str = Field(..., description="Name of the OpenAI model.")
+    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
+    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")

-You can also provide specific instructions to guide the extraction.
+async def main():
+    if not os.getenv("OPENAI_API_KEY"):
+        print("OpenAI API key not found. Skipping this example.")
+        return

-```python
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=LLMExtractionStrategy(
-        provider="openai/gpt-4o",
-        api_token=os.getenv('OPENAI_API_KEY'),
-        instruction="I am interested in only financial news"
-    )
-)
-print(f"LLMExtractionStrategy (with instructions) result: {result}")
-```
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://openai.com/api/pricing/",
+            word_count_threshold=1,
+            extraction_strategy=LLMExtractionStrategy(
+                provider="openai/gpt-4o",
+                api_token=os.getenv("OPENAI_API_KEY"),
+                schema=OpenAIModelFee.schema(),
+                extraction_type="schema",
+                instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
+                Do not miss any models in the entire content. One extracted model JSON format should look like this: 
+                {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}.""",
+            ),
+            bypass_cache=True,
+        )
+        print(result.extracted_content)

-### Targeted Extraction 🎯
-
-Let's use a CSS selector to extract only H2 tags!
-
-```python
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    css_selector="h2"
-)
-print(f"CSS Selector (H2 tags) result: {result}")
+asyncio.run(main())
 ```

 ### Interactive Extraction 🖱️

-Passing JavaScript code to click the 'Load More' button!
+Let's use JavaScript to interact with the page before extraction!

 ```python
-js_code = """
-const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
-loadMoreButton && loadMoreButton.click();
-"""
+async def main():
+    js_code = """
+    const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
+    loadMoreButton && loadMoreButton.click();
+    """

-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    js=js_code
-)
-print(f"JavaScript Code (Load More button) result: {result}")
+    wait_for = """() => {
+        return Array.from(document.querySelectorAll('article.tease-card')).length > 10;
+    }"""
+
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        result = await crawler.arun(
+            url="https://www.nbcnews.com/business",
+            js_code=js_code,
+            wait_for=wait_for,
+            css_selector="article.tease-card",
+            bypass_cache=True,
+        )
+        print(f"JavaScript interaction result: {result.extracted_content[:500]}")
+
+asyncio.run(main())
 ```

-### Using Crawler Hooks 🔗
+### Advanced Session-Based Crawling with Dynamic Content 🔄

-Let's see how we can customize the crawler using hooks!
+In modern web applications, content is often loaded dynamically without changing the URL. This is common in single-page applications (SPAs) or websites using infinite scrolling. Traditional crawling methods that rely on URL changes won't work here. That's where Crawl4AI's advanced session-based crawling comes in handy!
+
+Here's what makes this approach powerful:
+
+1. **Session Preservation**: By using a `session_id`, we can maintain the state of our crawling session across multiple interactions with the page. This is crucial for navigating through dynamically loaded content.
+
+2. **Asynchronous JavaScript Execution**: We can execute custom JavaScript to trigger content loading or navigation. In this example, we'll click a "Load More" button to fetch the next page of commits.
+
+3. **Dynamic Content Waiting**: The `wait_for` parameter allows us to specify a condition that must be met before considering the page load complete. This ensures we don't extract data before the new content is fully loaded.
+
+Let's see how this works with a real-world example: crawling multiple pages of commits on a GitHub repository. The URL doesn't change as we load more commits, so we'll use these advanced techniques to navigate and extract data.

 ```python
-def on_driver_created(driver):
-    print("[HOOK] on_driver_created")
-    driver.maximize_window()
-    driver.get('https://example.com/login')
-    driver.find_element(By.NAME, 'username').send_keys('testuser')
-    driver.find_element(By.NAME, 'password').send_keys('password123')
-    driver.find_element(By.NAME, 'login').click()
-    driver.add_cookie({'name': 'test_cookie', 'value': 'cookie_value'})
-    return driver        
+import json
+from bs4 import BeautifulSoup
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

-def before_get_url(driver):
-    print("[HOOK] before_get_url")
-    driver.execute_cdp_cmd('Network.enable', {})
-    driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
-    return driver
+async def main():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://github.com/microsoft/TypeScript/commits/main"
+        session_id = "typescript_commits_session"
+        all_commits = []

-def after_get_url(driver):
-    print("[HOOK] after_get_url")
-    print(driver.current_url)
-    return driver
+        js_next_page = """
+        const button = document.querySelector('a[data-testid="pagination-next-button"]');
+        if (button) button.click();
+        """

-def before_return_html(driver, html):
-    print("[HOOK] before_return_html")
-    print(len(html))
-    return driver
+        wait_for = """() => {
+            const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
+            if (commits.length === 0) return false;
+            const firstCommit = commits[0].textContent.trim();
+            return firstCommit !== window.lastCommit;
+        }"""

-crawler.set_hook('on_driver_created', on_driver_created)
-crawler.set_hook('before_get_url', before_get_url)
-crawler.set_hook('after_get_url', after_get_url)
-crawler.set_hook('before_return_html', before_return_html)
+        schema = {
+            "name": "Commit Extractor",
+            "baseSelector": "li.Box-sc-g0xbh4-0",
+            "fields": [
+                {
+                    "name": "title",
+                    "selector": "h4.markdown-title",
+                    "type": "text",
+                    "transform": "strip",
+                },
+            ],
+        }
+        extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

-result = crawler.run(url="https://example.com")
-print(f"Crawler Hooks result: {result}")
+        for page in range(3):  # Crawl 3 pages
+            result = await crawler.arun(
+                url=url,
+                session_id=session_id,
+                css_selector="li.Box-sc-g0xbh4-0",
+                extraction_strategy=extraction_strategy,
+                js_code=js_next_page if page > 0 else None,
+                wait_for=wait_for if page > 0 else None,
+                js_only=page > 0,
+                bypass_cache=True,
+                headless=False,
+            )
+
+            assert result.success, f"Failed to crawl page {page + 1}"
+
+            commits = json.loads(result.extracted_content)
+            all_commits.extend(commits)
+
+            print(f"Page {page + 1}: Found {len(commits)} commits")
+
+        await crawler.crawler_strategy.kill_session(session_id)
+        print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
+
+asyncio.run(main())
 ```

+In this example, we're crawling multiple pages of commits from a GitHub repository. The URL doesn't change as we load more commits, so we use JavaScript to click the "Load More" button and a `wait_for` condition to ensure the new content is loaded before extraction. This powerful combination allows us to navigate and extract data from complex, dynamically-loaded web applications with ease!
+
 ## Congratulations! 🎉

-You've made it through the Crawl4AI Quickstart Guide! Now go forth and crawl the web like a pro! 🕸️
+You've made it through the Crawl4AI Quickstart Guide! Now go forth and crawl the web asynchronously like a pro! 🕸️
+
+Remember, these are just a few examples of what Crawl4AI can do. For more advanced usage, check out our other documentation pages:
+
+- [LLM Extraction](examples/llm_extraction.md)
+- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
+- [Hooks & Auth](examples/hooks_auth.md)
+- [Summarization](examples/summarization.md)
+- [Research Assistant](examples/research_assistant.md)
+
+Happy crawling! 🚀
--- a/main.py
+++ b/main.py
@@ -22,6 +22,15 @@ from typing import List, Optional
 from crawl4ai.web_crawler import WebCrawler
 from crawl4ai.database import get_total_count, clear_db

+import time
+from slowapi import Limiter, _rate_limit_exceeded_handler
+from slowapi.util import get_remote_address
+from slowapi.errors import RateLimitExceeded
+
+# load .env file
+from dotenv import load_dotenv
+load_dotenv()
+
 # Configuration
 __location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
 MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
@@ -30,6 +39,78 @@ lock = asyncio.Lock()

 app = FastAPI()

+# Initialize rate limiter
+def rate_limit_key_func(request: Request):
+    access_token = request.headers.get("access-token")
+    if access_token == os.environ.get('ACCESS_TOKEN'):
+        return None
+    return get_remote_address(request)
+
+limiter = Limiter(key_func=rate_limit_key_func)
+app.state.limiter = limiter
+
+# Dictionary to store last request times for each client
+last_request_times = {}
+last_rate_limit = {}
+
+
+def get_rate_limit():
+    limit = os.environ.get('ACCESS_PER_MIN', "5")
+    return f"{limit}/minute"
+
+# Custom rate limit exceeded handler
+async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
+    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
+        last_rate_limit[request.client.host] = time.time()
+    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
+    reset_at = time.time() + retry_after
+    return JSONResponse(
+        status_code=429,
+        content={
+            "detail": "Rate limit exceeded",
+            "limit": str(exc.limit.limit),
+            "retry_after": retry_after,
+            'reset_at': reset_at,
+            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
+        }
+    )
+    
+app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
+
+
+# Middleware for token-based bypass and per-request limit
+class RateLimitMiddleware(BaseHTTPMiddleware):
+    async def dispatch(self, request: Request, call_next):
+        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
+        access_token = request.headers.get("access-token")
+        if access_token == os.environ.get('ACCESS_TOKEN'):
+            return await call_next(request)
+        
+        path = request.url.path
+        if path in ["/crawl", "/old"]:
+            client_ip = request.client.host
+            current_time = time.time()
+            
+            # Check time since last request
+            if client_ip in last_request_times:
+                time_since_last_request = current_time - last_request_times[client_ip]
+                if time_since_last_request < SPAN:
+                    return JSONResponse(
+                        status_code=429,
+                        content={
+                            "detail": "Too many requests",
+                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
+                            "retry_after": max(0, SPAN - time_since_last_request),
+                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
+                        }
+                    )
+            
+            last_request_times[client_ip] = current_time
+
+        return await call_next(request)
+
+app.add_middleware(RateLimitMiddleware)
+
 # CORS configuration
 origins = ["*"]  # Allow all origins
 app.add_middleware(
@@ -49,7 +130,9 @@ templates = Jinja2Templates(directory=__location__ + "/pages")
@lru_cache()
 def get_crawler():
    # Initialize and return a WebCrawler instance
-    return WebCrawler(verbose = True)
+    crawler = WebCrawler(verbose = True)
+    crawler.warmup()
+    return crawler

 class CrawlRequest(BaseModel):
    urls: List[str]
@@ -71,6 +154,7 @@ def read_root():
    return RedirectResponse(url="/mkdocs")

@app.get("/old", response_class=HTMLResponse)
+@limiter.limit(get_rate_limit())
 async def read_index(request: Request):
    partials_dir = os.path.join(__location__, "pages", "partial")
    partials = {}
@@ -105,6 +189,7 @@ def import_strategy(module_name: str, class_name: str, *args, **kwargs):
        raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")

@app.post("/crawl")
+@limiter.limit(get_rate_limit())
 async def crawl_urls(crawl_request: CrawlRequest, request: Request):
    logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
    global current_requests
@@ -166,4 +251,4 @@ async def get_chunking_strategies():

 if __name__ == "__main__":
    import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8080)
+    uvicorn.run(app, host="0.0.0.0", port=8888)
--- a/middlewares.py
+++ b/middlewares.py
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -2,13 +2,13 @@ site_name: Crawl4AI Documentation
 docs_dir: docs/md
 nav:
  - Home: index.md
-  - Demo: demo.md  # Add this line
  - First Steps:
      - Introduction: introduction.md
      - Installation: installation.md
      - Quick Start: quickstart.md
  - Examples:
      - Intro: examples/index.md
+      - Structured Data Extraction: examples/json_css_extraction.md
      - LLM Extraction: examples/llm_extraction.md
      - JS Execution & CSS Filtering: examples/js_execution_css_filtering.md
      - Hooks & Auth: examples/hooks_auth.md
@@ -17,12 +17,11 @@ nav:
  - Full Details of Using Crawler:
      - Crawl Request Parameters: full_details/crawl_request_parameters.md
      - Crawl Result Class: full_details/crawl_result_class.md
+      - Session Based Crawling: full_details/session_based_crawling.md
      - Advanced Features: full_details/advanced_features.md
+      - Advanced JsonCssExtraction: full_details/advanced_jsoncss_extraction.md
      - Chunking Strategies: full_details/chunking_strategies.md
      - Extraction Strategies: full_details/extraction_strategies.md
-  - API Reference:
-      - Core Classes and Functions: api/core_classes_and_functions.md
-      - Detailed API Documentation: api/detailed_api_documentation.md
  - Miscellaneous:
      - Change Log: changelog.md
      - Contact: contact.md
--- a/pages/index_pooling.html
+++ b/pages/index_pooling.html
@@ -50,7 +50,7 @@
    <body>
        <header class="bg-gray-900 text-white py-4">
            <div class="container mx-auto px-4">
-                <h1 class="text-2xl font-bold">🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Scrapper</h1>
+                <h1 class="text-2xl font-bold">🔥🕷️ Crawl4AI: Open-source LLM Friendly Web scraper</h1>
            </div>
        </header>

--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -0,0 +1,2 @@
+-r requirements.txt
+pytest
--- a/requirements.crawl.txt
+++ b/requirements.crawl.txt
@@ -1,13 +0,0 @@
-aiohttp
-aiosqlite
-bs4
-fastapi
-html2text
-httpx
-pydantic
-python-dotenv
-requests
-rich
-selenium
-uvicorn
-chromedriver-autoinstaller
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,22 +1,10 @@
-numpy==1.25.0
-aiohttp==3.9.5
 aiosqlite==0.20.0
-beautifulsoup4==4.12.3
-fastapi==0.111.0
 html2text==2024.2.26
-httpx==0.27.0
-litellm==1.40.17
-nltk==3.8.1
-pydantic==2.7.4
+lxml==5.3.0
+litellm==1.48.0
+numpy>=1.26.0,<2.1.1
+pillow==10.4.0
+playwright==1.47.0
 python-dotenv==1.0.1
-requests==2.32.3
-rich==13.7.1
-scikit-learn==1.5.0
-selenium==4.21.0
-uvicorn==0.30.1
-transformers==4.41.2
-chromedriver-autoinstaller==0.6.4
-torch==2.3.1
-onnxruntime==1.18.0
-tokenizers==0.19.1
-pillow==10.3.0
+requests>=2.26.0,<2.32.3
+beautifulsoup4==4.12.3
--- a/setup.cfg
+++ b/setup.cfg
@@ -0,0 +1,2 @@
+[options]
+include_package_data = True
--- a/setup.py
+++ b/setup.py
@@ -1,55 +1,76 @@
 from setuptools import setup, find_packages
-import os
-import sys
-from pathlib import Path
-import subprocess
 from setuptools.command.install import install
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys

 # Create the .crawl4ai folder in the user's home directory if it doesn't exist
-crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
-os.makedirs(crawl4ai_folder, exist_ok=True)
-os.makedirs(f"{crawl4ai_folder}/cache", exist_ok=True)
+# If the folder already exists, remove the cache folder
+crawl4ai_folder = Path.home() / ".crawl4ai"
+cache_folder = crawl4ai_folder / "cache"
+
+if cache_folder.exists():
+    shutil.rmtree(cache_folder)
+
+crawl4ai_folder.mkdir(exist_ok=True)
+cache_folder.mkdir(exist_ok=True)

 # Read the requirements from requirements.txt
-with open("requirements.txt") as f:
+__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
+with open(os.path.join(__location__, "requirements.txt")) as f:
    requirements = f.read().splitlines()
-
-# Read the requirements from requirements.txt
-with open("requirements.crawl.txt") as f:
-    requirements_crawl_only = f.read().splitlines()
+    
+# Read version from __init__.py
+with open("crawl4ai/__init__.py") as f:
+    for line in f:
+        if line.startswith("__version__"):
+            version = line.split("=")[1].strip().strip('"')
+            break

 # Define the requirements for different environments
-requirements_without_torch = [req for req in requirements if not req.startswith("torch")]
-requirements_without_transformers = [req for req in requirements if not req.startswith("transformers")]
-requirements_without_nltk = [req for req in requirements if not req.startswith("nltk")]
-requirements_without_torch_transformers_nlkt = [req for req in requirements if not req.startswith("torch") and not req.startswith("transformers") and not req.startswith("nltk")]
-requirements_crawl_only = [req for req in requirements if not req.startswith("torch") and not req.startswith("transformers") and not req.startswith("nltk")]
+default_requirements = requirements
+torch_requirements = ["torch", "nltk", "spacy", "scikit-learn"]
+transformer_requirements = ["transformers", "tokenizers", "onnxruntime"]
+cosine_similarity_requirements = ["torch", "transformers", "nltk", "spacy"]
+sync_requirements = ["selenium"]

-class CustomInstallCommand(install):
-    """Customized setuptools install command to install spacy without dependencies."""
+def install_playwright():
+    print("Installing Playwright browsers...")
+    try:
+        subprocess.check_call([sys.executable, "-m", "playwright", "install"])
+        print("Playwright installation completed successfully.")
+    except subprocess.CalledProcessError as e:
+        print(f"Error during Playwright installation: {e}")
+        print("Please run 'python -m playwright install' manually after the installation.")
+    except Exception as e:
+        print(f"Unexpected error during Playwright installation: {e}")
+        print("Please run 'python -m playwright install' manually after the installation.")
+
+class PostInstallCommand(install):
    def run(self):
        install.run(self)
-        subprocess.check_call([os.sys.executable, '-m', 'pip', 'install', 'spacy', '--no-deps'])
+        install_playwright()

 setup(
    name="Crawl4AI",
-    version="0.2.7",
-    description="🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper",
-    long_description=open("README.md").read(),
+    version=version,
+    description="🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & scraper",
+    long_description=open("README.md", encoding="utf-8").read(),
    long_description_content_type="text/markdown",
    url="https://github.com/unclecode/crawl4ai",
    author="Unclecode",
    author_email="unclecode@kidocode.com",
    license="MIT",
    packages=find_packages(),
-    install_requires=requirements_without_torch_transformers_nlkt,
+    install_requires=default_requirements + ["playwright"],  # Add playwright to default requirements
    extras_require={
-        "all": requirements,  # Include all requirements
-        "colab": requirements_without_torch,  # Exclude torch for Colab
-        "crawl": requirements_crawl_only,  # Include only crawl requirements
-    },
-    cmdclass={
-        'install': CustomInstallCommand,
+        "torch": torch_requirements,
+        "transformer": transformer_requirements,
+        "cosine": cosine_similarity_requirements,
+        "sync": sync_requirements,
+        "all": default_requirements + torch_requirements + transformer_requirements + cosine_similarity_requirements + sync_requirements,
    },
    entry_points={
        'console_scripts': [
@@ -67,4 +88,7 @@ setup(
        "Programming Language :: Python :: 3.10",
    ],
    python_requires=">=3.7",
-)
+    cmdclass={
+        'install': PostInstallCommand,
+    },
+)
--- a/tests/async/test_basic_crawling.py
+++ b/tests/async/test_basic_crawling.py
@@ -0,0 +1,81 @@
+import os
+import sys
+import pytest
+import asyncio
+import time
+
+# Add the parent directory to the Python path
+parent_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+sys.path.append(parent_dir)
+
+from crawl4ai.async_webcrawler import AsyncWebCrawler
+
+@pytest.mark.asyncio
+async def test_successful_crawl():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert result.url == url
+        assert result.html
+        assert result.markdown
+        assert result.cleaned_html
+
+@pytest.mark.asyncio
+async def test_invalid_url():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.invalidurl12345.com"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert not result.success
+        assert result.error_message
+
+@pytest.mark.asyncio
+async def test_multiple_urls():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        urls = [
+            "https://www.nbcnews.com/business",
+            "https://www.example.com",
+            "https://www.python.org"
+        ]
+        results = await crawler.arun_many(urls=urls, bypass_cache=True)
+        assert len(results) == len(urls)
+        assert all(result.success for result in results)
+        assert all(result.html for result in results)
+
+@pytest.mark.asyncio
+async def test_javascript_execution():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        js_code = "document.body.innerHTML = '<h1>Modified by JS</h1>';"
+        url = "https://www.example.com"
+        result = await crawler.arun(url=url, bypass_cache=True, js_code=js_code)
+        assert result.success
+        assert "<h1>Modified by JS</h1>" in result.html
+
+@pytest.mark.asyncio
+async def test_concurrent_crawling_performance():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        urls = [
+            "https://www.nbcnews.com/business",
+            "https://www.example.com",
+            "https://www.python.org",
+            "https://www.github.com",
+            "https://www.stackoverflow.com"
+        ]
+        
+        start_time = time.time()
+        results = await crawler.arun_many(urls=urls, bypass_cache=True)
+        end_time = time.time()
+        
+        total_time = end_time - start_time
+        print(f"Total time for concurrent crawling: {total_time:.2f} seconds")
+        
+        assert all(result.success for result in results)
+        assert len(results) == len(urls)
+        
+        # Assert that concurrent crawling is faster than sequential
+        # This multiplier may need adjustment based on the number of URLs and their complexity
+        assert total_time < len(urls) * 5, f"Concurrent crawling not significantly faster: {total_time:.2f} seconds"
+
+# Entry point for debugging
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
--- a/tests/async/test_caching.py
+++ b/tests/async/test_caching.py
@@ -0,0 +1,82 @@
+import os
+import sys
+import pytest
+import asyncio
+
+# Add the parent directory to the Python path
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+
+from crawl4ai.async_webcrawler import AsyncWebCrawler
+
+@pytest.mark.asyncio
+async def test_caching():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        
+        # First crawl (should not use cache)
+        start_time = asyncio.get_event_loop().time()
+        result1 = await crawler.arun(url=url, bypass_cache=True)
+        end_time = asyncio.get_event_loop().time()
+        time_taken1 = end_time - start_time
+        
+        assert result1.success
+        
+        # Second crawl (should use cache)
+        start_time = asyncio.get_event_loop().time()
+        result2 = await crawler.arun(url=url, bypass_cache=False)
+        end_time = asyncio.get_event_loop().time()
+        time_taken2 = end_time - start_time
+        
+        assert result2.success
+        assert time_taken2 < time_taken1  # Cached result should be faster
+
+@pytest.mark.asyncio
+async def test_bypass_cache():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        
+        # First crawl
+        result1 = await crawler.arun(url=url, bypass_cache=False)
+        assert result1.success
+        
+        # Second crawl with bypass_cache=True
+        result2 = await crawler.arun(url=url, bypass_cache=True)
+        assert result2.success
+        
+        # Content should be different (or at least, not guaranteed to be the same)
+        assert result1.html != result2.html or result1.markdown != result2.markdown
+
+@pytest.mark.asyncio
+async def test_clear_cache():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        
+        # Crawl and cache
+        await crawler.arun(url=url, bypass_cache=False)
+        
+        # Clear cache
+        await crawler.aclear_cache()
+        
+        # Check cache size
+        cache_size = await crawler.aget_cache_size()
+        assert cache_size == 0
+
+@pytest.mark.asyncio
+async def test_flush_cache():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        
+        # Crawl and cache
+        await crawler.arun(url=url, bypass_cache=False)
+        
+        # Flush cache
+        await crawler.aflush_cache()
+        
+        # Check cache size
+        cache_size = await crawler.aget_cache_size()
+        assert cache_size == 0
+
+# Entry point for debugging
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
--- a/tests/async/test_chunking_and_extraction_strategies.py
+++ b/tests/async/test_chunking_and_extraction_strategies.py
@@ -0,0 +1,87 @@
+import os
+import sys
+import pytest
+import asyncio
+import json
+
+# Add the parent directory to the Python path
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+
+from crawl4ai.async_webcrawler import AsyncWebCrawler
+from crawl4ai.chunking_strategy import RegexChunking, NlpSentenceChunking
+from crawl4ai.extraction_strategy import CosineStrategy, LLMExtractionStrategy
+
+@pytest.mark.asyncio
+async def test_regex_chunking():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        chunking_strategy = RegexChunking(patterns=["\n\n"])
+        result = await crawler.arun(
+            url=url,
+            chunking_strategy=chunking_strategy,
+            bypass_cache=True
+        )
+        assert result.success
+        assert result.extracted_content
+        chunks = json.loads(result.extracted_content)
+        assert len(chunks) > 1  # Ensure multiple chunks were created
+
+# @pytest.mark.asyncio
+# async def test_cosine_strategy():
+#     async with AsyncWebCrawler(verbose=True) as crawler:
+#         url = "https://www.nbcnews.com/business"
+#         extraction_strategy = CosineStrategy(word_count_threshold=10, max_dist=0.2, linkage_method="ward", top_k=3, sim_threshold=0.3)
+#         result = await crawler.arun(
+#             url=url,
+#             extraction_strategy=extraction_strategy,
+#             bypass_cache=True
+#         )
+#         assert result.success
+#         assert result.extracted_content
+#         extracted_data = json.loads(result.extracted_content)
+#         assert len(extracted_data) > 0
+#         assert all('tags' in item for item in extracted_data)
+
+@pytest.mark.asyncio
+async def test_llm_extraction_strategy():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        extraction_strategy = LLMExtractionStrategy(
+            provider="openai/gpt-4o-mini",
+            api_token=os.getenv('OPENAI_API_KEY'),
+            instruction="Extract only content related to technology"
+        )
+        result = await crawler.arun(
+            url=url,
+            extraction_strategy=extraction_strategy,
+            bypass_cache=True
+        )
+        assert result.success
+        assert result.extracted_content
+        extracted_data = json.loads(result.extracted_content)
+        assert len(extracted_data) > 0
+        assert all('content' in item for item in extracted_data)
+
+# @pytest.mark.asyncio
+# async def test_combined_chunking_and_extraction():
+#     async with AsyncWebCrawler(verbose=True) as crawler:
+#         url = "https://www.nbcnews.com/business"
+#         chunking_strategy = RegexChunking(patterns=["\n\n"])
+#         extraction_strategy = CosineStrategy(word_count_threshold=10, max_dist=0.2, linkage_method="ward", top_k=3, sim_threshold=0.3)
+#         result = await crawler.arun(
+#             url=url,
+#             chunking_strategy=chunking_strategy,
+#             extraction_strategy=extraction_strategy,
+#             bypass_cache=True
+#         )
+#         assert result.success
+#         assert result.extracted_content
+#         extracted_data = json.loads(result.extracted_content)
+#         assert len(extracted_data) > 0
+#         assert all('tags' in item for item in extracted_data)
+#         assert all('content' in item for item in extracted_data)
+
+# Entry point for debugging
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
--- a/tests/async/test_content_extraction.py
+++ b/tests/async/test_content_extraction.py
@@ -0,0 +1,90 @@
+import os
+import sys
+import pytest
+import asyncio
+import json
+
+# Add the parent directory to the Python path
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+
+from crawl4ai.async_webcrawler import AsyncWebCrawler
+
+@pytest.mark.asyncio
+async def test_extract_markdown():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert result.markdown
+        assert isinstance(result.markdown, str)
+        assert len(result.markdown) > 0
+
+@pytest.mark.asyncio
+async def test_extract_cleaned_html():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert result.cleaned_html
+        assert isinstance(result.cleaned_html, str)
+        assert len(result.cleaned_html) > 0
+
+@pytest.mark.asyncio
+async def test_extract_media():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert result.media
+        media = result.media
+        assert isinstance(media, dict)
+        assert "images" in media
+        assert isinstance(media["images"], list)
+        for image in media["images"]:
+            assert "src" in image
+            assert "alt" in image
+            assert "type" in image
+
+@pytest.mark.asyncio
+async def test_extract_links():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert result.links
+        links = result.links
+        assert isinstance(links, dict)
+        assert "internal" in links
+        assert "external" in links
+        assert isinstance(links["internal"], list)
+        assert isinstance(links["external"], list)
+        for link in links["internal"] + links["external"]:
+            assert "href" in link
+            assert "text" in link
+
+@pytest.mark.asyncio
+async def test_extract_metadata():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert result.metadata
+        metadata = result.metadata
+        assert isinstance(metadata, dict)
+        assert "title" in metadata
+        assert isinstance(metadata["title"], str)
+
+@pytest.mark.asyncio
+async def test_css_selector_extraction():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.nbcnews.com/business"
+        css_selector = "h1, h2, h3"
+        result = await crawler.arun(url=url, bypass_cache=True, css_selector=css_selector)
+        assert result.success
+        assert result.markdown
+        assert all(heading in result.markdown for heading in ["#", "##", "###"])
+
+# Entry point for debugging
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
--- a/tests/async/test_crawler_strategy.py
+++ b/tests/async/test_crawler_strategy.py
@@ -0,0 +1,68 @@
+import os
+import sys
+import pytest
+import asyncio
+
+# Add the parent directory to the Python path
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+
+from crawl4ai.async_webcrawler import AsyncWebCrawler
+from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
+
+@pytest.mark.asyncio
+async def test_custom_user_agent():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        custom_user_agent = "MyCustomUserAgent/1.0"
+        crawler.crawler_strategy.update_user_agent(custom_user_agent)
+        url = "https://httpbin.org/user-agent"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert custom_user_agent in result.html
+
+@pytest.mark.asyncio
+async def test_custom_headers():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        custom_headers = {"X-Test-Header": "TestValue"}
+        crawler.crawler_strategy.set_custom_headers(custom_headers)
+        url = "https://httpbin.org/headers"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert "X-Test-Header" in result.html
+        assert "TestValue" in result.html
+
+@pytest.mark.asyncio
+async def test_javascript_execution():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        js_code = "document.body.innerHTML = '<h1>Modified by JS</h1>';"
+        url = "https://www.example.com"
+        result = await crawler.arun(url=url, bypass_cache=True, js_code=js_code)
+        assert result.success
+        assert "<h1>Modified by JS</h1>" in result.html
+
+@pytest.mark.asyncio
+async def test_hook_execution():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        async def test_hook(page):
+            await page.evaluate("document.body.style.backgroundColor = 'red';")
+            return page
+
+        crawler.crawler_strategy.set_hook('after_goto', test_hook)
+        url = "https://www.example.com"
+        result = await crawler.arun(url=url, bypass_cache=True)
+        assert result.success
+        assert "background-color: red" in result.html
+
+@pytest.mark.asyncio
+async def test_screenshot():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.example.com"
+        result = await crawler.arun(url=url, bypass_cache=True, screenshot=True)
+        assert result.success
+        assert result.screenshot
+        assert isinstance(result.screenshot, str)
+        assert len(result.screenshot) > 0
+
+# Entry point for debugging
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
--- a/tests/async/test_database_operations.py
+++ b/tests/async/test_database_operations.py
@@ -0,0 +1,82 @@
+import os
+import sys
+import pytest
+import asyncio
+import json
+
+# Add the parent directory to the Python path
+parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(parent_dir)
+
+from crawl4ai.async_webcrawler import AsyncWebCrawler
+
+@pytest.mark.asyncio
+async def test_cache_url():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.example.com"
+        # First run to cache the URL
+        result1 = await crawler.arun(url=url, bypass_cache=True)
+        assert result1.success
+
+        # Second run to retrieve from cache
+        result2 = await crawler.arun(url=url, bypass_cache=False)
+        assert result2.success
+        assert result2.html == result1.html
+
+@pytest.mark.asyncio
+async def test_bypass_cache():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.python.org"
+        # First run to cache the URL
+        result1 = await crawler.arun(url=url, bypass_cache=True)
+        assert result1.success
+
+        # Second run bypassing cache
+        result2 = await crawler.arun(url=url, bypass_cache=True)
+        assert result2.success
+        assert result2.html != result1.html  # Content might be different due to dynamic nature of websites
+
+@pytest.mark.asyncio
+async def test_cache_size():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        initial_size = await crawler.aget_cache_size()
+        
+        url = "https://www.nbcnews.com/business"
+        await crawler.arun(url=url, bypass_cache=True)
+        
+        new_size = await crawler.aget_cache_size()
+        assert new_size == initial_size + 1
+
+@pytest.mark.asyncio
+async def test_clear_cache():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.example.org"
+        await crawler.arun(url=url, bypass_cache=True)
+        
+        initial_size = await crawler.aget_cache_size()
+        assert initial_size > 0
+
+        await crawler.aclear_cache()
+        new_size = await crawler.aget_cache_size()
+        assert new_size == 0
+
+@pytest.mark.asyncio
+async def test_flush_cache():
+    async with AsyncWebCrawler(verbose=True) as crawler:
+        url = "https://www.example.net"
+        await crawler.arun(url=url, bypass_cache=True)
+        
+        initial_size = await crawler.aget_cache_size()
+        assert initial_size > 0
+
+        await crawler.aflush_cache()
+        new_size = await crawler.aget_cache_size()
+        assert new_size == 0
+
+        # Try to retrieve the previously cached URL
+        result = await crawler.arun(url=url, bypass_cache=False)
+        assert result.success  # The crawler should still succeed, but it will fetch the content anew
+
+# Entry point for debugging
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
UncleCode	ac9d83c72f	Update gitignore	2024-10-27 19:29:04 +08:00
UncleCode	ff9149b5c9	Merge branch 'main' of https://github.com/unclecode/crawl4ai	2024-10-27 19:28:05 +08:00
UncleCode	32f57c49d6	Merge pull request #194 from IdrisHanafi/feat/customize-crawl-base-directory Support for custom crawl base directory	2024-10-24 13:09:27 +02:00
Idris Hanafi	a5f627ba1a	feat: customize crawl base directory	2024-10-21 17:58:39 -04:00
UncleCode	dbb587d681	Update gitignore	2024-10-17 21:38:48 +08:00
unclecode	9ffa34b697	Update README	2024-10-14 22:58:27 +08:00
unclecode	740802c491	Merge branch '0.3.6'	2024-10-14 22:55:24 +08:00
unclecode	b9ac96c332	Merge branch 'main' of https://github.com/unclecode/crawl4ai	2024-10-14 22:54:23 +08:00
unclecode	d06535388a	Update gitignore	2024-10-14 22:53:56 +08:00
unclecode	2b73bdf6b0	Update changelog	2024-10-14 21:04:02 +08:00
unclecode	6aa803d712	Update gitignore	2024-10-14 21:03:40 +08:00
unclecode	320afdea64	feat: Enhance crawler flexibility and LLM extraction capabilities - Add browser type selection (Chromium, Firefox, WebKit) - Implement iframe content extraction - Improve image processing and dimension updates - Add custom headers support in AsyncPlaywrightCrawlerStrategy - Enhance delayed content retrieval with new parameter - Optimize HTML sanitization and Markdown conversion - Update examples in quickstart_async.py for new features	2024-10-14 21:03:28 +08:00
UncleCode	ccbe72cfc1	Merge pull request #135 from hitesh22rana/fix/docs-example docs: fixed css_selector for example	2024-10-13 14:39:07 +08:00
unclecode	b9bbd42373	Update Quickstart examples	2024-10-13 14:37:45 +08:00
unclecode	68e9144ce3	feat: Enhance crawling control and LLM extraction flexibility - Add before_retrieve_html hook and delay_before_return_html option - Implement flexible page_timeout for smart_wait function - Support extra_args and custom headers in LLM extraction - Allow arbitrary kwargs in AsyncWebCrawler initialization - Improve perform_completion_with_backoff for custom API calls - Update examples with new features and diverse LLM providers	2024-10-12 14:48:22 +08:00
unclecode	9b2b267820	CHANGELOG UPDATE	2024-10-12 13:42:56 +08:00
unclecode	ff3524d9b1	feat(v0.3.6): Add screenshot capture, delayed content, and custom timeouts - Implement screenshot capture functionality - Add delayed content retrieval method - Introduce custom page timeout parameter - Enhance LLM support with multiple providers - Improve database schema auto-updates - Optimize image processing in WebScrappingStrategy - Update error handling and logging - Expand examples in quickstart_async.py	2024-10-12 13:42:42 +08:00
unclecode	b99d20b725	Add pypi_build.sh to .gitignore	2024-10-08 18:10:57 +08:00
hitesh22rana	768b93140f	docs: fixed css_selector for example	2024-10-05 00:25:41 +09:00
unclecode	4750810a67	Enhance AsyncWebCrawler with smart waiting and screenshot capabilities - Implement smart_wait function in AsyncPlaywrightCrawlerStrategy - Add screenshot support to AsyncCrawlResponse and AsyncWebCrawler - Improve error handling and timeout management in crawling process - Fix typo in CrawlResult model (responser_headers -> response_headers) - Update .gitignore to exclude additional files - Adjust import path in test_basic_crawling.py	2024-10-02 17:34:56 +08:00
unclecode	e0e0db4247	Bump version to 0.3.4	2024-09-29 17:07:52 +08:00
unclecode	bccadec887	Remove dependency on psutil, PyYaml, and extend requests version range	2024-09-29 17:07:06 +08:00
unclecode	0759503e50	Extend numpy version range to support Python 3.9	2024-09-29 00:08:02 +08:00
unclecode	7f1c020746	Update README to add link to previous version in branch V0.2.76	2024-09-28 00:31:53 +08:00
unclecode	5d4e92db7d	Update quickstart_async.py to improve performance and add Firecrawl simulation	2024-09-28 00:11:39 +08:00
unclecode	8b6e88c85c	Update .gitignore to ignore temporary and test directories	2024-09-26 15:09:49 +08:00
unclecode	64190dd0c4	Update README	2024-09-25 17:26:13 +08:00
unclecode	7100bcdf04	Add session based crawling documentation	2024-09-25 17:16:55 +08:00
unclecode	10cdad039d	Update documents and README	2024-09-25 16:52:11 +08:00
unclecode	f1eee09cf4	Update README, add manifest, make selenium optional library	2024-09-25 16:35:14 +08:00
unclecode	4d48bd31ca	Push async version last changes for merge to main branch	2024-09-24 20:52:08 +08:00
unclecode	d628bc4034	Refactor content_scrapping_strategy.py to remove excluded tags	2024-09-12 17:35:45 +08:00
unclecode	b179aa9b6f	Refactor website content and setup.py descriptions for consistent terminology	2024-09-12 16:50:52 +08:00
unclecode	30807f5535	Remove excluded tags from website content	2024-09-12 16:11:20 +08:00
unclecode	396f430022	Refactor AsyncCrawlerStrategy to return AsyncCrawlResponse This commit refactors the AsyncCrawlerStrategy class in the async_crawler_strategy.py file to modify the return types of the crawl and crawl_many methods. Instead of returning strings, these methods now return instances of the AsyncCrawlResponse class from the pydantic module. The AsyncCrawlResponse class contains the crawled HTML, response headers, and status code. This change improves the clarity and consistency of the code.	2024-09-12 15:49:49 +08:00
unclecode	eb131bebdf	Create series of quickstart files.	2024-09-04 15:33:24 +08:00
unclecode	5c15837677	chore: Update README, generate new notbook for quickstart	2024-09-04 14:46:22 +08:00
unclecode	2fada16abb	chore: Update crawl4ai package with AsyncWebCrawler and JsonCssExtractionStrategy	2024-09-03 23:32:27 +08:00
unclecode	c37614cbc8	Add Async Version, JsonCss Extrator	2024-09-03 01:27:00 +08:00
unclecode	3116f95c1a	Merge branch 'pull-84' into staging	2024-09-01 16:44:06 +08:00
unclecode	b0e8b66666	Merge branch 'proxy-support' into staging	2024-09-01 16:35:14 +08:00
unclecode	3caf48c9be	refactor: Update LocalSeleniumCrawlerStrategy to execute JS code if provided	2024-09-01 16:34:51 +08:00
Umut CAN	3c6ebb73ae	Update web_crawler.py Improve code efficiency, readability, and maintainability in web_crawler.py	2024-08-30 15:30:06 +03:00
UncleCode	0d9b638636	Merge pull request #75 from aravindkarnam/main Added support to source tags wrapped inside video and audio tags. Ext…	2024-08-30 12:54:15 +02:00
datehoer	2ba70b9501	add use proxy and llm baseurl examples	2024-08-27 10:14:54 +08:00
datehoer	16f98cebc0	replace base64 image url to ''	2024-08-27 09:44:35 +08:00
datehoer	fe9ff498ce	add proxy and add ai base_url	2024-08-26 16:12:49 +08:00
Datehoer	eba831ca30	fix spelling mistake	2024-08-26 15:29:23 +08:00
unclecode	dec3d44224	refactor: Update extraction strategy to handle schema extraction with non-empty schema This code change updates the `LLMExtractionStrategy` class to handle schema extraction when the schema is non-empty. Previously, the schema extraction was only triggered when the `extract_type` was set to "schema", regardless of whether a schema was provided. With this update, the schema extraction will only be performed if the `extract_type` is "schema" and a non-empty schema is provided. This ensures that the extraction strategy behaves correctly and avoids unnecessary schema extraction when not needed. Also "numpy" is removed from default installation mode.	2024-08-19 15:37:07 +08:00
Aravind Karnam	9ed1551125	Added support to source tags wrapped inside video and audio tags. Extended the text extraction to video and audio elements in media. https://github.com/unclecode/crawl4ai/issues/71	2024-08-14 11:07:26 +05:30
unclecode	e5e6a34e80	## [v0.2.77] - 2024-08-04 Significant improvements in text processing and performance: - 🚀 Dependency reduction: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy. - 🤖 Transformer upgrade: Implemented text sequence classification using a transformer model for labeling text chunks. - ⚡ Performance enhancement: Improved model loading speed due to removal of spaCy dependency. - 🔧 Future-proofing: Laid groundwork for potential complete removal of spaCy dependency in future versions. These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.	2024-08-04 14:54:18 +08:00
unclecode	897e766728	Update README	2024-08-02 16:04:14 +08:00
unclecode	9200a6731d	## [v0.2.76] - 2024-08-02 Major improvements in functionality, performance, and cross-platform compatibility! 🚀 - 🐳 Docker enhancements: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows. - 🌐 Official Docker Hub image: Launched our first official image on Docker Hub for streamlined deployment (unclecode/crawl4ai). - 🔧 Selenium upgrade: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility. - 🖼️ Image description: Implemented ability to generate textual descriptions for extracted images from web pages. - ⚡ Performance boost: Various improvements to enhance overall speed and performance.	2024-08-02 16:02:42 +08:00
unclecode	61c166ab19	refactor: Update Crawl4AI version to v0.2.76 This commit updates the Crawl4AI version from v0.2.7765 to v0.2.76. The version number is updated in the README.md file. This change ensures consistency and reflects the correct version of the software.	2024-08-02 15:55:53 +08:00
unclecode	659c8cd953	refactor: Update image description minimum word threshold in get_content_of_website_optimized	2024-08-02 15:55:32 +08:00
unclecode	9ee988753d	refactor: Update image description minimum word threshold in get_content_of_website_optimized	2024-08-02 14:53:11 +08:00
unclecode	8ae6c43ca4	refactor: Update Dockerfile to install Crawl4AI with specified options	2024-08-01 20:13:06 +08:00
unclecode	b6713870ef	refactor: Update Dockerfile to install Crawl4AI with specified options This commit updates the Dockerfile to install Crawl4AI with the specified options. The `INSTALL_OPTION` build argument is used to determine which additional packages to install. If the option is set to "all", all models will be downloaded. If the option is set to "torch", only torch models will be downloaded. If the option is set to "transformer", only transformer models will be downloaded. If no option is specified, the default installation will be used. This change improves the flexibility and customization of the Crawl4AI installation process.	2024-08-01 17:56:19 +08:00
unclecode	40477493d3	refactor: Remove image format dot in get_content_of_website_optimized The code change removes the dot from the image format in the `get_content_of_website_optimized` function. This change ensures consistency in the image format and improves the functionality.	2024-07-31 16:15:55 +08:00
Kevin Moturi	efcf3ac6eb	Update LocalSeleniumCrawlerStrategy to resolve ChromeDriver version mismatch issue This resolves the following error: `selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 114` Windows users are getting.	2024-07-31 13:33:09 +08:00
unclecode	9e43f7beda	refactor: Temporarily disable fetching image file size in get_content_of_website_optimized Set the `image_size` variable to 0 in the `get_content_of_website_optimized` function to temporarily disable fetching the image file size. This change addresses performance issues and will be improved in a future update. Update Dockerfile for linuz users	2024-07-31 13:29:23 +08:00
unclecode	aa9412e1b4	refactor: Set image_size to 0 in get_content_of_website_optimized The code change sets the `image_size` variable to 0 in the `get_content_of_website_optimized` function. This change is made to temporarily disable fetching the image file size, which was causing performance issues. The image size will be fetched in a future update to improve the functionality.	2024-07-23 13:08:53 +08:00
Aravind Karnam	cf6c835e18	moved score threshold to config.py & replaced the separator for tag.get_text in find_closest_parent_with_useful_text fn from period(.) to space( ) to keep the text more neutral.	2024-07-21 15:18:23 +05:30
Aravind Karnam	e5ecf291f3	Implemented filtering for images and grabbing the contextual text from nearest parent	2024-07-21 15:03:17 +05:30
Aravind Karnam	9d0cafcfa6	fixed import error in model_loader.py	2024-07-21 14:55:58 +05:30
unclecode	7715623430	chore: Fix typos and update .gitignore These changes fix typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability. Additionally, the `.test_pads/` directory is removed from the `.gitignore` file to keep the repository clean and organized.	2024-07-19 17:42:39 +08:00
unclecode	f5a4e80e2c	chore: Fix typo in chunking_strategy.py and crawler_strategy.py The commit fixes a typo in the `chunking_strategy.py` file where `nl.toknize.TextTilingTokenizer()` was corrected to `nl.tokenize.TextTilingTokenizer()`. Additionally, in the `crawler_strategy.py` file, the commit converts the screenshot image to RGB mode before saving it as a JPEG. This ensures consistent image quality and compression.	2024-07-19 17:40:31 +08:00
unclecode	8463aabedf	chore: Remove .test_pads/ directory from .gitignore	2024-07-19 17:09:29 +08:00
unclecode	7f30144ef2	chore: Remove .tests/ directory from .gitignore	2024-07-09 15:10:18 +08:00
unclecode	fa5516aad6	chore: Refactor setup.py to use pathlib and shutil for folder creation and removal, to remove cache folder in cross platform manner.	2024-07-09 13:25:00 +08:00
unclecode	ca0336af9e	feat: Add error handling for rate limit exceeded in form submission This commit adds error handling for rate limit exceeded in the form submission process. If the server returns a 429 status code, the client will display an error message indicating the rate limit has been exceeded and provide information on when the user can try again. This improves the user experience by providing clear feedback and guidance when rate limits are reached.	2024-07-08 20:24:00 +08:00
unclecode	65ed1aeade	feat: Add rate limiting functionality with custom handlers	2024-07-08 20:02:12 +08:00
unclecode	4d283ab386	## [v0.2.74] - 2024-07-08 A slew of exciting updates to improve the crawler's stability and robustness! 🎉 - 💻 UTF encoding fix: Resolved the Windows \"charmap\" error by adding UTF encoding. - 🛡️ Error handling: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy. - 🧹 Input sanitization: Improved input sanitization and handled encoding issues in LLMExtractionStrategy. - 🚮 Database cleanup: Removed existing database file and initialized a new one.	2024-07-08 16:33:25 +08:00
unclecode	3ff2a0d0e7	Merge branch 'main' of https://github.com/unclecode/crawl4ai	2024-07-03 15:26:47 +08:00
unclecode	3cd1b3719f	Bump version to v0.2.73, update documentation, and resolve installation issues	2024-07-03 15:26:43 +08:00
unclecode	9926eb9f95	feat: Bump version to v0.2.73 and update documentation This commit updates the version number to v0.2.73 and makes corresponding changes in the README.md and Dockerfile. Docker file install the default mode, this resolve many of installation issues. Additionally, the installation instructions are updated to include support for different modes. Setup.py doesn't have anymore dependancy on Spacy. The change log is also updated to reflect these changes. Supporting websites need with-head browser.	2024-07-03 15:19:22 +08:00
UncleCode	3abaa82501	Merge pull request #37 from shivkumar0757/fix-readme-encoding @shivkumar0757 Great work! I value your contribution and have merged your pull request. You will be credited in the upcoming change-log. Thank you for your continuous support in advancing this library, to democratize an open access crawler to everyone.	2024-07-01 07:31:07 +02:00
unclecode	88d8cd8650	feat: Add page load check for LocalSeleniumCrawlerStrategy This commit adds a page load check for the LocalSeleniumCrawlerStrategy in the `crawl` method. The `_ensure_page_load` method is introduced to ensure that the page has finished loading before proceeding. This helps to prevent issues with incomplete page sources and improves the reliability of the crawler.	2024-07-01 00:07:32 +08:00
shiv	a08f21d66c	Fix UnicodeDecodeError by reading README.md with UTF-8 encoding	2024-06-30 20:27:33 +05:30
unclecode	d58286989c	UPDATE DOCUMENTS	2024-06-30 00:34:02 +08:00
unclecode	b58af3349c	chore: Update installation instructions with support for different modes	2024-06-30 00:22:17 +08:00
unclecode	940df4631f	Update ChangeLog	2024-06-30 00:18:40 +08:00
unclecode	685706e0aa	Update version, and change log	2024-06-30 00:17:43 +08:00
unclecode	7b0979e134	Update Redme and Docker file	2024-06-30 00:15:43 +08:00
unclecode	61ae2de841	1/Update setup.py to support following modes: - default (most frequent mode) - torch - transformers - all 2/ Update Docker file 3/ Update documentation as well.	2024-06-30 00:15:29 +08:00
unclecode	5b28eed2c0	Add a temporary solution for when we can't crawl websites in headless mode.	2024-06-29 23:25:50 +08:00
unclecode	f8a11779fe	Update change log	2024-06-26 16:48:36 +08:00
unclecode	d11a83c232	## [0.2.71] 2024-06-26 • Refactored `crawler_strategy.py` to handle exceptions and improve error messages • Improved `get_content_of_website_optimized` function in `utils.py` for better performance • Updated `utils.py` with latest changes • Migrated to `ChromeDriverManager` for resolving Chrome driver download issues	2024-06-26 15:34:15 +08:00
unclecode	3255c7a3fa	Update CHANGELOG.md with recent commits	2024-06-26 15:20:34 +08:00
unclecode	4756d0a532	Refactor crawler_strategy.py to handle exceptions and improve error messages	2024-06-26 15:04:33 +08:00
unclecode	7ba2142363	chore: Refactor get_content_of_website_optimized function in utils.py	2024-06-26 14:43:09 +08:00
unclecode	96d1eb0d0d	Some updated ins utils.py	2024-06-26 13:03:03 +08:00
unclecode	144cfa0eda	Switch to ChromeDriverManager due some issues with download the chrome driver	2024-06-26 13:00:17 +08:00
unclecode	a0dff192ae	Update README for speed example	2024-06-24 23:06:12 +08:00
unclecode	1fffeeedd2	Update Readme: Showcase the speed	2024-06-24 23:02:08 +08:00
unclecode	f51b078042	Update reame example.	2024-06-24 22:54:29 +08:00
unclecode	b6023a51fb	Add star chart	2024-06-24 22:47:46 +08:00