chore: Update version to 0.2.74 in setup.py

This commit adds a delay of 5 seconds after fetching the URL in the `after_get_url` hook of the crawler hooks. The delay is implemented using the `time.sleep()` function. This change ensures that the entire page is fetched before proceeding with further actions.

.gitignore

@@ -165,8 +165,6 @@ Crawl4AI.egg-info/
 Crawl4AI.egg-info/*
 crawler_data.db
 .vscode/
-.tests/
-.test_pads/
 test_pad.py
 test_pad*.py
 .data/

CHANGELOG.md

@@ -1,42 +1,5 @@
 # Changelog
 
-## [v0.2.77] - 2024-08-04
-
-Significant improvements in text processing and performance:
-
-- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
-- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
-- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
-- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
-
-These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
-
-## [v0.2.76] - 2024-08-02
-
-Major improvements in functionality, performance, and cross-platform compatibility! 🚀
-
-- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
-- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
-- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
-- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
-- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
-
-A big shoutout to our amazing community contributors:
-- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
-- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
-- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
-
-Your contributions are driving Crawl4AI forward! 🙌
-
-## [v0.2.75] - 2024-07-19
-
-Minor improvements for a more maintainable codebase:
-
-- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
-- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
-
-These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
-
 ## [v0.2.74] - 2024-07-08
 A slew of exciting updates to improve the crawler's stability and robustness! 🎉
 

CONTRIBUTORS.md

@@ -1,31 +0,0 @@
-# Contributors to Crawl4AI
-
-We would like to thank the following people for their contributions to Crawl4AI:
-
-## Core Team
-
-- [Unclecode](https://github.com/unclecode) - Project Creator and Main Developer
-- [Nasrin](https://github.com/ntohidi) - Project Manager and Developer
-
-## Community Contributors
-
-- [Aravind Karnam](https://github.com/aravindkarnam) - Developed textual description extraction feature
-- [FractalMind](https://github.com/FractalMind) - Created the first official Docker Hub image and fixed Dockerfile errors
-- [ketonkss4](https://github.com/ketonkss4) - Identified Selenium's new capabilities, helping reduce dependencies
-
-## Other Contributors
-
-- [Gokhan](https://github.com/gkhngyk) 
-- [Shiv Kumar](https://github.com/shivkumar0757)
-- [QIN2DIM](https://github.com/QIN2DIM)
-
-
-## Acknowledgements
-
-We also want to thank all the users who have reported bugs, suggested features, or helped in any other way to make Crawl4AI better.
-
----
-
-If you've contributed to Crawl4AI and your name isn't on this list, please [open a pull request](https://github.com/unclecode/crawl4ai/pulls) with your name, link, and contribution, and we'll review it promptly.
-
-Thank you all for your contributions!

Dockerfile

@@ -4,9 +4,6 @@ FROM python:3.10-slim-bookworm
 # Set the working directory in the container
 WORKDIR /usr/src/app
 
-# Define build arguments
-ARG INSTALL_OPTION=default
-
 # Install build dependencies
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
@@ -24,39 +21,33 @@ RUN apt-get update && \
 # Copy the application code
 COPY . .
 
-# Install Crawl4AI using the local setup.py with the specified option
-# and download models only for torch, transformer, or all options
-RUN if [ "$INSTALL_OPTION" = "all" ]; then \
-        pip install --no-cache-dir .[all] && \
-        crawl4ai-download-models; \
-    elif [ "$INSTALL_OPTION" = "torch" ]; then \
-        pip install --no-cache-dir .[torch] && \
-        crawl4ai-download-models; \
-    elif [ "$INSTALL_OPTION" = "transformer" ]; then \
-        pip install --no-cache-dir .[transformer] && \
-        crawl4ai-download-models; \
-    else \
-        pip install --no-cache-dir .; \
-    fi
+# Install Crawl4AI using the local setup.py (which will use the default installation)
+RUN pip install --no-cache-dir .
 
-# Install Google Chrome
+# Install Google Chrome and ChromeDriver
 RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
     sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' && \
     apt-get update && \
-    apt-get install -y google-chrome-stable
+    apt-get install -y google-chrome-stable && \
+    wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip && \
+    unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
 
-# Set environment to use Chrome properly
+# Set environment to use Chrome and ChromeDriver properly
 ENV CHROME_BIN=/usr/bin/google-chrome \
+    CHROMEDRIVER=/usr/local/bin/chromedriver \
     DISPLAY=:99 \
     DBUS_SESSION_BUS_ADDRESS=/dev/null \
     PYTHONUNBUFFERED=1
 
 # Ensure the PATH environment variable includes the location of the installed packages
-ENV PATH=/opt/conda/bin:$PATH   
+ENV PATH /opt/conda/bin:$PATH   
 
 # Make port 80 available to the world outside this container
 EXPOSE 80
 
+# Download models call cli "crawl4ai-download-models"
+# RUN crawl4ai-download-models
+
 # Install mkdocs
 RUN pip install mkdocs mkdocs-terminal
 

README.md

@@ -1,4 +1,4 @@
-# Crawl4AI v0.2.77 🕷️🤖
+# Crawl4AI v0.2.74 🕷️🤖
 
 [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
 [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
@@ -8,29 +8,13 @@
 
 Crawl4AI simplifies web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
 
-#### [v0.2.77] - 2024-08-02
-
-Major improvements in functionality, performance, and cross-platform compatibility! 🚀
-
-- 🐳 **Docker enhancements**:
-  - Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
-- 🌐 **Official Docker Hub image**:
-  - Launched our first official image on Docker Hub for streamlined deployment (unclecode/crawl4ai).
-- 🔧 **Selenium upgrade**:
-  - Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
-- 🖼️ **Image description**:
-  - Implemented ability to generate textual descriptions for extracted images from web pages.
-- ⚡ **Performance boost**:
-  - Various improvements to enhance overall speed and performance.
-  
 ## Try it Now!
 
-✨ Play around with this [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sJPAmeLj5PMrg2VgOwMJ2ubGIcK0cJeX?usp=sharing)
+- Use as REST API: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zODYjhemJ5bUmYceWpVoBMVpd0ofzNBZ?usp=sharing)
+- Use as Python library: This collab is a bit outdated. I'm updating it with the newest versions, so please refer to the website for the latest documentation. This will be updated in a few days, and you'll have the latest version here. Thank you so much. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
 
 ✨ visit our [Documentation Website](https://crawl4ai.com/mkdocs/)
 
-✨ Check [Demo](https://crawl4ai.com/mkdocs/demo)
-
 ## Features ✨
 
 - 🆓 Completely free and open-source
@@ -48,18 +32,6 @@ Major improvements in functionality, performance, and cross-platform compatibili
 - 🎯 CSS selector support
 - 📝 Passes instructions/keywords to refine extraction
 
-# Crawl4AI
-
-## 🌟 Shoutout to Contributors of v0.2.77!
-
-A big thank you to the amazing contributors who've made this release possible:
-
-- [@aravindkarnam](https://github.com/aravindkarnam) for the new image description feature
-- [@FractalMind](https://github.com/FractalMind) for our official Docker Hub image
-- [@ketonkss4](https://github.com/ketonkss4) for helping streamline our Selenium setup
-
-Your contributions are driving Crawl4AI forward! 🚀
-
 ## Cool Examples 🚀
 
 ### Quick Start
@@ -80,33 +52,14 @@ result = crawler.run(url="https://www.nbcnews.com/business")
 print(result.markdown)
 ```
 
-## How to install 🛠 
-
-### Using pip 🐍
+## How to install 🛠    
 ```bash
 virtualenv venv
 source venv/bin/activate
 pip install "crawl4ai @ git+https://github.com/unclecode/crawl4ai.git"
-```
+```️
 
-### Using Docker 🐳
-
-```bash
-# For Mac users (M1/M2)
-# docker build --platform linux/amd64 -t crawl4ai .
-docker build -t crawl4ai .
-docker run -d -p 8000:80 crawl4ai
-```
-
-### Using Docker Hub 🐳
-
-```bash
-docker pull unclecode/crawl4ai:latest
-docker run -d -p 8000:80 unclecode/crawl4ai:latest
-```
-
-
-## Speed-First Design 🚀
+### Speed-First Design 🚀
 
 Perhaps the most important design principle for this library is speed. We need to ensure it can handle many links and resources in parallel as quickly as possible. By combining this speed with fast LLMs like Groq, the results will be truly amazing.
 
@@ -190,33 +143,6 @@ result = crawler.run(
 print(result.extracted_content)
 ```
 
-### Extract Structured Data from Web Pages With Proxy and BaseUrl
-
-```python
-from crawl4ai import WebCrawler
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
-
-def create_crawler():
-    crawler = WebCrawler(verbose=True, proxy="http://127.0.0.1:7890")
-    crawler.warmup()
-    return crawler
-
-crawler = create_crawler()
-
-crawler.warmup()
-
-result = crawler.run(
-    url="https://www.nbcnews.com/business",
-    extraction_strategy=LLMExtractionStrategy(
-        provider="openai/gpt-4o",
-        api_token="sk-",
-        base_url="https://api.openai.com/v1"
-    )
-)
-
-print(result.markdown)
-```
-
 ## Documentation 📚
 
 For detailed documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://crawl4ai.com/mkdocs/).

crawl4ai/chunking_strategy.py

@@ -55,7 +55,7 @@ class TopicSegmentationChunking(ChunkingStrategy):
     
     def __init__(self, num_keywords=3, **kwargs):
         import nltk as nl
-        self.tokenizer = nl.tokenize.TextTilingTokenizer()
+        self.tokenizer = nl.toknize.TextTilingTokenizer()
         self.num_keywords = num_keywords
 
     def chunk(self, text: str) -> list:

crawl4ai/config.py

@@ -27,14 +27,3 @@ WORD_TOKEN_RATE = 1.3
 
 # Threshold for the minimum number of word in a HTML tag to be considered 
 MIN_WORD_THRESHOLD = 1
-IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD = 1
-
-# Threshold for the Image extraction - Range is 1 to 6
-# Images are scored based on point based system, to filter based on usefulness. Points are assigned
-# to each image based on the following aspects.
-# If either height or width exceeds 150px
-# If image size is greater than 10Kb
-# If alt property is set
-# If image format is in jpg, png or webp
-# If image is in the first half of the total images extracted from the page
-IMAGE_SCORE_THRESHOLD = 2

crawl4ai/crawler_strategy.py

@@ -6,9 +6,9 @@ from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.support import expected_conditions as EC
 from selenium.webdriver.chrome.options import Options
 from selenium.common.exceptions import InvalidArgumentException, WebDriverException
-# from selenium.webdriver.chrome.service import Service as ChromeService
-# from webdriver_manager.chrome import ChromeDriverManager
-# from urllib3.exceptions import MaxRetryError
+from selenium.webdriver.chrome.service import Service as ChromeService
+from webdriver_manager.chrome import ChromeDriverManager
+from urllib3.exceptions import MaxRetryError
 
 from .config import *
 import logging, time
@@ -82,8 +82,6 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
         print("[LOG] 🚀 Initializing LocalSeleniumCrawlerStrategy")
         self.options = Options()
         self.options.headless = True
-        if kwargs.get("proxy"):
-            self.options.add_argument("--proxy-server={}".format(kwargs.get("proxy")))
         if kwargs.get("user_agent"):
             self.options.add_argument("--user-agent=" + kwargs.get("user_agent"))
         else:
@@ -139,15 +137,10 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
         # self.service = Service(chromedriver_autoinstaller.install())
         
         
-        # chromedriver_path = ChromeDriverManager().install()
-        # self.service = Service(chromedriver_path)
-        # self.service.log_path = "NUL"
-        # self.driver = webdriver.Chrome(service=self.service, options=self.options)
-        
-        # Use selenium-manager (built into Selenium 4.10.0+)
-        self.service = Service()
-        self.driver = webdriver.Chrome(options=self.options)
-        
+        chromedriver_path = ChromeDriverManager().install()
+        self.service = Service(chromedriver_path)
+        self.service.log_path = "NUL"
+        self.driver = webdriver.Chrome(service=self.service, options=self.options)
         self.driver = self.execute_hook('on_driver_created', self.driver)
         
         if kwargs.get("cookies"):
@@ -244,7 +237,6 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
                 driver.quit()
             
             # Execute JS code if provided
-            self.js_code = kwargs.get("js_code", self.js_code)
             if self.js_code and type(self.js_code) == str:
                 self.driver.execute_script(self.js_code)
                 # Optionally, wait for some condition after executing the JS code
@@ -300,18 +292,16 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
             # Open the screenshot with PIL
             image = Image.open(BytesIO(screenshot))
 
-            # Convert image to RGB mode (this will handle both RGB and RGBA images)
-            rgb_image = image.convert('RGB')
-
             # Convert to JPEG and compress
             buffered = BytesIO()
-            rgb_image.save(buffered, format="JPEG", quality=85)
+            image.save(buffered, format="JPEG", quality=85)
             img_base64 = base64.b64encode(buffered.getvalue()).decode('utf-8')
 
             if self.verbose:
                 print(f"[LOG] 📸 Screenshot taken and converted to base64")
 
             return img_base64
+
         except Exception as e:
             error_message = sanitize_input_encode(f"Failed to take screenshot: {str(e)}")
             print(error_message)
@@ -324,7 +314,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
             try:
                 font = ImageFont.truetype("arial.ttf", 40)
             except IOError:
-                font = ImageFont.load_default()
+                font = ImageFont.load_default(size=40)
 
             # Define text color and wrap the text
             text_color = (255, 255, 255)
@@ -343,6 +333,6 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
             img_base64 = base64.b64encode(buffered.getvalue()).decode('utf-8')
 
             return img_base64
-        
+
     def quit(self):
-        self.driver.quit()
+        self.driver.quit()

crawl4ai/extraction_strategy.py

@@ -9,7 +9,6 @@ from .utils import *
 from functools import partial
 from .model_loader import *
 import math
-import numpy as np
 
 
 class ExtractionStrategy(ABC):
@@ -79,7 +78,6 @@ class LLMExtractionStrategy(ExtractionStrategy):
         self.overlap_rate = kwargs.get("overlap_rate", OVERLAP_RATE)
         self.word_token_rate = kwargs.get("word_token_rate", WORD_TOKEN_RATE)
         self.apply_chunking = kwargs.get("apply_chunking", True)
-        self.base_url = kwargs.get("base_url", None)
         if not self.apply_chunking:
             self.chunk_token_threshold = 1e9
         
@@ -111,7 +109,7 @@ class LLMExtractionStrategy(ExtractionStrategy):
                 "{" + variable + "}", variable_values[variable]
             )
         
-        response = perform_completion_with_backoff(self.provider, prompt_with_variables, self.api_token, base_url=self.base_url) # , json_response=self.extract_type == "schema")
+        response = perform_completion_with_backoff(self.provider, prompt_with_variables, self.api_token) # , json_response=self.extract_type == "schema")
         try:
             blocks = extract_xml_data(["blocks"], response.choices[0].message.content)['blocks']
             blocks = json.loads(blocks)
@@ -250,9 +248,6 @@ class CosineStrategy(ExtractionStrategy):
         self.get_embedding_method = "direct"
         
         self.device = get_device()
-        import torch
-        self.device = torch.device('cpu')
-        
         self.default_batch_size = calculate_batch_size(self.device)
 
         if self.verbose:
@@ -265,9 +260,7 @@ class CosineStrategy(ExtractionStrategy):
         # else:
 
         self.tokenizer, self.model = load_bge_small_en_v1_5()
-        self.model.to(self.device)
         self.model.eval()  
-        
         self.get_embedding_method = "batch"
         
         self.buffer_embeddings = np.array([])
@@ -289,7 +282,7 @@ class CosineStrategy(ExtractionStrategy):
         if self.verbose:
             print(f"[LOG] Loading Multilabel Classifier for {self.device.type} device.")
             
-        self.nlp, _ = load_text_multilabel_classifier()
+        self.nlp, self.device = load_text_multilabel_classifier()
         # self.default_batch_size = 16 if self.device.type == 'cpu' else 64
         
         if self.verbose:
@@ -460,21 +453,21 @@ class CosineStrategy(ExtractionStrategy):
         if self.verbose:
             print(f"[LOG] 🚀 Assign tags using {self.device}")
         
-        if self.device.type in ["gpu", "cuda", "mps", "cpu"]:
+        if self.device.type in ["gpu", "cuda", "mps"]:
             labels = self.nlp([cluster['content'] for cluster in cluster_list])
             
             for cluster, label in zip(cluster_list, labels):
                 cluster['tags'] = label
-        # elif self.device.type == "cpu":
-        #     # Process the text with the loaded model
-        #     texts = [cluster['content'] for cluster in cluster_list]
-        #     # Batch process texts
-        #     docs = self.nlp.pipe(texts, disable=["tagger", "parser", "ner", "lemmatizer"])
+        elif self.device == "cpu":
+            # Process the text with the loaded model
+            texts = [cluster['content'] for cluster in cluster_list]
+            # Batch process texts
+            docs = self.nlp.pipe(texts, disable=["tagger", "parser", "ner", "lemmatizer"])
 
-        #     for doc, cluster in zip(docs, cluster_list):
-        #         tok_k = self.top_k
-        #         top_categories = sorted(doc.cats.items(), key=lambda x: x[1], reverse=True)[:tok_k]
-        #         cluster['tags'] = [cat for cat, _ in top_categories]
+            for doc, cluster in zip(docs, cluster_list):
+                tok_k = self.top_k
+                top_categories = sorted(doc.cats.items(), key=lambda x: x[1], reverse=True)[:tok_k]
+                cluster['tags'] = [cat for cat, _ in top_categories]
                             
             # for cluster in  cluster_list:
             #     doc = self.nlp(cluster['content'])

crawl4ai/model_loader.py

@@ -3,10 +3,9 @@ from pathlib import Path
 import subprocess, os
 import shutil
 import tarfile
-from .model_loader import *
+from crawl4ai.config import MODEL_REPO_BRANCH
 import argparse
 import urllib.request
-from crawl4ai.config import MODEL_REPO_BRANCH
 __location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
 
 @lru_cache()
@@ -142,15 +141,14 @@ def load_text_multilabel_classifier():
     from scipy.special import expit
     import torch
 
-    # # Check for available device: CUDA, MPS (for Apple Silicon), or CPU
-    # if torch.cuda.is_available():
-    #     device = torch.device("cuda")
-    # elif torch.backends.mps.is_available():
-    #     device = torch.device("mps")
-    # else:
-    #     device = torch.device("cpu")
-    #     # return load_spacy_model(), torch.device("cpu")
-    
+    # Check for available device: CUDA, MPS (for Apple Silicon), or CPU
+    if torch.cuda.is_available():
+        device = torch.device("cuda")
+    elif torch.backends.mps.is_available():
+        device = torch.device("mps")
+    else:
+        return load_spacy_model(), torch.device("cpu")
+
 
     MODEL = "cardiffnlp/tweet-topic-21-multi"
     tokenizer = AutoTokenizer.from_pretrained(MODEL, resume_download=None)
@@ -194,61 +192,51 @@ def load_spacy_model():
     import spacy
     name = "models/reuters"
     home_folder = get_home_folder()
-    model_folder = Path(home_folder) / name
+    model_folder = os.path.join(home_folder, name)
     
     # Check if the model directory already exists
-    if not (model_folder.exists() and any(model_folder.iterdir())):
+    if not (Path(model_folder).exists() and any(Path(model_folder).iterdir())):
         repo_url = "https://github.com/unclecode/crawl4ai.git"
+        # branch = "main"
         branch = MODEL_REPO_BRANCH 
-        repo_folder = Path(home_folder) / "crawl4ai"
-        
-        print("[LOG] ⏬ Downloading Spacy model for the first time...")
+        repo_folder = os.path.join(home_folder, "crawl4ai")
+        model_folder = os.path.join(home_folder, name)
+
+        # print("[LOG] ⏬ Downloading Spacy model for the first time...")
 
         # Remove existing repo folder if it exists
-        if repo_folder.exists():
-            try:
-                shutil.rmtree(repo_folder)
-                if model_folder.exists():
-                    shutil.rmtree(model_folder)
-            except PermissionError:
-                print("[WARNING] Unable to remove existing folders. Please manually delete the following folders and try again:")
-                print(f"- {repo_folder}")
-                print(f"- {model_folder}")
-                return None
+        if Path(repo_folder).exists():
+            shutil.rmtree(repo_folder)
+            shutil.rmtree(model_folder)
 
         try:
             # Clone the repository
             subprocess.run(
-                ["git", "clone", "-b", branch, repo_url, str(repo_folder)],
+                ["git", "clone", "-b", branch, repo_url, repo_folder],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
                 check=True
             )
 
             # Create the models directory if it doesn't exist
-            models_folder = Path(home_folder) / "models"
-            models_folder.mkdir(parents=True, exist_ok=True)
+            models_folder = os.path.join(home_folder, "models")
+            os.makedirs(models_folder, exist_ok=True)
 
             # Copy the reuters model folder to the models directory
-            source_folder = repo_folder / "models" / "reuters"
+            source_folder = os.path.join(repo_folder, "models/reuters")
             shutil.copytree(source_folder, model_folder)
 
             # Remove the cloned repository
             shutil.rmtree(repo_folder)
 
-            print("[LOG] ✅ Spacy Model downloaded successfully")
+            # Print completion message
+            # print("[LOG] ✅ Spacy Model downloaded successfully")
         except subprocess.CalledProcessError as e:
             print(f"An error occurred while cloning the repository: {e}")
-            return None
         except Exception as e:
             print(f"An error occurred: {e}")
-            return None
 
-    try:
-        return spacy.load(str(model_folder))
-    except Exception as e:
-        print(f"Error loading spacy model: {e}")
-        return None
+    return spacy.load(model_folder)
 
 def download_all_models(remove_existing=False):
     """Download all models required for Crawl4AI."""

crawl4ai/prompts.py

@@ -29,7 +29,7 @@ To generate the JSON objects:
 
 5. Make sure the generated JSON is complete and parsable, with no errors or omissions.
 
-6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
+6. Make sur to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
 
 Please provide your output within <blocks> tags, like this:
 
@@ -87,7 +87,7 @@ To generate the JSON objects:
 
 5. Make sure the generated JSON is complete and parsable, with no errors or omissions.
 
-6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
+6. Make sur to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
 
 7. Never alter the extracted content, just copy and paste it as it is.
 
@@ -142,7 +142,7 @@ To generate the JSON objects:
 
 5. Make sure the generated JSON is complete and parsable, with no errors or omissions.
 
-6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
+6. Make sur to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
 
 7. Never alter the extracted content, just copy and paste it as it is.
 
@@ -201,4 +201,4 @@ Avoid Common Mistakes:
 - Do not generate the Python coee show me how to do the task, this is your task to extract the information and return it in JSON format.
 
 Result
-Output the final list of JSON objects, wrapped in <blocks>...</blocks> XML tags. Make sure to close the tag properly."""
+Output the final list of JSON objects, wrapped in <blocks>...</blocks> XML tags. Make sure to close the tag properly."""

crawl4ai/utils.py

@@ -11,9 +11,6 @@ from .prompts import PROMPT_EXTRACT_BLOCKS
 from .config import *
 from pathlib import Path
 from typing import Dict, Any
-from urllib.parse import urljoin
-import requests
-from requests.exceptions import InvalidSchema
 
 class InvalidCSSSelectorError(Exception):
     pass
@@ -438,8 +435,6 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
 
     soup = BeautifulSoup(html, 'html.parser')
     body = soup.body
-    
-    image_description_min_word_threshold = kwargs.get('image_description_min_word_threshold', IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD)
 
     if css_selector:
         selected_elements = body.select(css_selector)
@@ -452,103 +447,6 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
     links = {'internal': [], 'external': []}
     media = {'images': [], 'videos': [], 'audios': []}
 
-    def process_image(img, url, index, total_images):
-            #Check if an image has valid display and inside undesired html elements
-            def is_valid_image(img, parent, parent_classes):
-                style = img.get('style', '')
-                src = img.get('src', '')
-                classes_to_check = ['button', 'icon', 'logo']
-                tags_to_check = ['button', 'input']
-                return all([
-                    'display:none' not in style,
-                    src,
-                    not any(s in var for var in [src, img.get('alt', ''), *parent_classes] for s in classes_to_check),
-                    parent.name not in tags_to_check
-                ])
-
-            #Score an image for it's usefulness
-            def score_image_for_usefulness(img, base_url, index, images_count):
-                # Function to parse image height/width value and units
-                def parse_dimension(dimension):
-                    if dimension:
-                        match = re.match(r"(\d+)(\D*)", dimension)
-                        if match:
-                            number = int(match.group(1))
-                            unit = match.group(2) or 'px'  # Default unit is 'px' if not specified
-                            return number, unit
-                    return None, None
-
-                # Fetch image file metadata to extract size and extension
-                def fetch_image_file_size(img, base_url):
-                    #If src is relative path construct full URL, if not it may be CDN URL
-                    img_url = urljoin(base_url,img.get('src'))
-                    try:
-                        response = requests.head(img_url)
-                        if response.status_code == 200:
-                            return response.headers.get('Content-Length',None)
-                        else:
-                            print(f"Failed to retrieve file size for {img_url}")
-                            return None
-                    except InvalidSchema as e:
-                        return None
-                    finally:
-                        return
-
-                image_height = img.get('height')
-                height_value, height_unit = parse_dimension(image_height)
-                image_width =  img.get('width')
-                width_value, width_unit = parse_dimension(image_width)
-                image_size = 0 #int(fetch_image_file_size(img,base_url) or 0)
-                image_format = os.path.splitext(img.get('src',''))[1].lower()
-                # Remove . from format
-                image_format = image_format.strip('.')
-                score = 0
-                if height_value:
-                    if height_unit == 'px' and height_value > 150:
-                        score += 1
-                    if height_unit in ['%','vh','vmin','vmax'] and height_value >30:
-                        score += 1
-                if width_value:
-                    if width_unit == 'px' and width_value > 150:
-                        score += 1
-                    if width_unit in ['%','vh','vmin','vmax'] and width_value >30:
-                        score += 1
-                if image_size > 10000:
-                    score += 1
-                if img.get('alt') != '':
-                    score+=1
-                if any(image_format==format for format in ['jpg','png','webp']):
-                    score+=1
-                if index/images_count<0.5:
-                    score+=1
-                return score
-
-            # Extract meaningful text for images from closest parent
-            def find_closest_parent_with_useful_text(tag):
-                current_tag = tag
-                while current_tag:
-                    current_tag = current_tag.parent
-                    # Get the text content of the parent tag
-                    if current_tag:
-                        text_content = current_tag.get_text(separator=' ',strip=True)
-                        # Check if the text content has at least word_count_threshold
-                        if len(text_content.split()) >= image_description_min_word_threshold:
-                            return text_content
-                return None
-
-            if not is_valid_image(img, img.parent, img.parent.get('class', [])):
-                return None
-            score = score_image_for_usefulness(img, url, index, total_images)
-            if score <= IMAGE_SCORE_THRESHOLD:
-                return None
-            return {
-                'src': img.get('src', ''),
-                'alt': img.get('alt', ''),
-                'desc': find_closest_parent_with_useful_text(img),
-                'score': score,
-                'type': 'image'
-            }
-
     def process_element(element: element.PageElement) -> bool:
         try:
             if isinstance(element, NavigableString):
@@ -573,6 +471,11 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
                 keep_element = True
 
             elif element.name == 'img':
+                media['images'].append({
+                    'src': element.get('src'),
+                    'alt': element.get('alt'),
+                    'type': 'image'
+                })
                 return True  # Always keep image elements
 
             elif element.name in ['video', 'audio']:
@@ -615,14 +518,6 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
             print('Error processing element:', str(e))
             return False
 
-    #process images by filtering and extracting contextual text from the page
-    imgs = body.find_all('img')
-    media['images'] = [
-        result for result in
-        (process_image(img, url, i, len(imgs)) for i, img in enumerate(imgs))
-        if result is not None
-    ]
-
     process_element(body)
 
     def flatten_nested_elements(node):
@@ -634,12 +529,7 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
         return node
 
     body = flatten_nested_elements(body)
-    base64_pattern = re.compile(r'data:image/[^;]+;base64,([^"]+)')
-    for img in imgs:
-        src = img.get('src', '')
-        if base64_pattern.match(src):
-            # Replace base64 data with empty string
-            img['src'] = base64_pattern.sub('', src)
+
     cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
     cleaned_html = sanitize_html(cleaned_html)
 
@@ -721,7 +611,7 @@ def extract_xml_data(tags, string):
     return data
     
 # Function to perform the completion with exponential backoff
-def perform_completion_with_backoff(provider, prompt_with_variables, api_token, json_response = False, base_url=None):
+def perform_completion_with_backoff(provider, prompt_with_variables, api_token, json_response = False):
     from litellm import completion 
     from litellm.exceptions import RateLimitError
     max_attempts = 3
@@ -740,7 +630,6 @@ def perform_completion_with_backoff(provider, prompt_with_variables, api_token,
                 ],
                 temperature=0.01,
                 api_key=api_token,
-                base_url=base_url,
                 **extra_args
             )
             return response  # Return the successful response
@@ -761,7 +650,7 @@ def perform_completion_with_backoff(provider, prompt_with_variables, api_token,
                     "content": ["Rate limit error. Please try again later."]
                 }]
     
-def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None, base_url = None):
+def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None):
     # api_token = os.getenv('GROQ_API_KEY', None) if not api_token else api_token
     api_token = PROVIDER_MODELS.get(provider, None) if not api_token else api_token
     
@@ -776,7 +665,7 @@ def extract_blocks(url, html, provider = DEFAULT_PROVIDER, api_token = None, bas
             "{" + variable + "}", variable_values[variable]
         )
         
-    response = perform_completion_with_backoff(provider, prompt_with_variables, api_token, base_url=base_url)
+    response = perform_completion_with_backoff(provider, prompt_with_variables, api_token)
         
     try:
         blocks = extract_xml_data(["blocks"], response.choices[0].message.content)['blocks']
@@ -870,17 +759,17 @@ def merge_chunks_based_on_token_threshold(chunks, token_threshold):
 
     return merged_sections
 
-def process_sections(url: str, sections: list, provider: str, api_token: str, base_url=None) -> list:
+def process_sections(url: str, sections: list, provider: str, api_token: str) -> list:
     extracted_content = []
     if provider.startswith("groq/"):
         # Sequential processing with a delay
         for section in sections:
-            extracted_content.extend(extract_blocks(url, section, provider, api_token, base_url=base_url))
+            extracted_content.extend(extract_blocks(url, section, provider, api_token))
             time.sleep(0.5)  # 500 ms delay between each processing
     else:
         # Parallel processing using ThreadPoolExecutor
         with ThreadPoolExecutor() as executor:
-            futures = [executor.submit(extract_blocks, url, section, provider, api_token, base_url=base_url) for section in sections]
+            futures = [executor.submit(extract_blocks, url, section, provider, api_token) for section in sections]
             for future in as_completed(futures):
                 extracted_content.extend(future.result())
     

crawl4ai/web_crawler.py

@@ -22,10 +22,9 @@ class WebCrawler:
         crawler_strategy: CrawlerStrategy = None,
         always_by_pass_cache: bool = False,
         verbose: bool = False,
-        proxy: str = None,
     ):
         # self.db_path = db_path
-        self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose, proxy=proxy)
+        self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose)
         self.always_by_pass_cache = always_by_pass_cache
 
         # Create the .crawl4ai folder in the user's home directory if it doesn't exist

docs/examples/llm_extraction_openai_pricing.py

@@ -21,8 +21,7 @@ result = crawler.run(
     url=url,
     word_count_threshold=1,
     extraction_strategy= LLMExtractionStrategy(
-        # provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
-        provider= "groq/llama-3.1-70b-versatile", api_token = os.getenv('GROQ_API_KEY'), 
+        provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'), 
         schema=OpenAIModelFee.model_json_schema(),
         extraction_type="schema",
         instruction="From the crawled content, extract all mentioned model names along with their "\

docs/md/changelog.md

@@ -1,43 +1,5 @@
 # Changelog
 
-## [v0.2.77] - 2024-08-04
-
-Significant improvements in text processing and performance:
-
-- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
-- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
-- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
-- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
-
-These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
-
-## [v0.2.76] - 2024-08-02
-
-Major improvements in functionality, performance, and cross-platform compatibility! 🚀
-
-- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
-- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
-- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
-- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
-- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
-
-A big shoutout to our amazing community contributors:
-- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
-- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
-- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
-
-Your contributions are driving Crawl4AI forward! 🙌
-
-## [v0.2.75] - 2024-07-19
-
-Minor improvements for a more maintainable codebase:
-
-- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
-- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
-
-These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
-
-
 ## v0.2.74 - 2024-07-08
 A slew of exciting updates to improve the crawler's stability and robustness! 🎉
 

docs/md/demo.md

@@ -14,7 +14,6 @@
             <div class="form-group">
                 <button class="btn btn-default" type="submit">Submit</button>
             </div>
-
         </fieldset>
     </form>
 
@@ -94,10 +93,6 @@
         </div>
     </section>
 
-    <div id="error" class="error-message" style="display: none; margin-top:1em;">
-        <div class="terminal-alert terminal-alert-error"></div>
-    </div>
-
     <script>
         function showTab(tabId) {
             const tabs = document.querySelectorAll('.tab-content');
@@ -167,17 +162,7 @@
                 },
                 body: JSON.stringify(data)
             })
-            .then(response => {
-                if (!response.ok) {
-                    if (response.status === 429) {
-                        return response.json().then(err => { 
-                            throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
-                        });
-                    }
-                    throw new Error('Network response was not ok');
-                }
-                return response.json();
-            })
+            .then(response => response.json())
             .then(data => {
                 data = data.results[0]; // Only one URL is requested
                 document.getElementById('loading').style.display = 'none';
@@ -202,29 +187,11 @@ result = crawler.run(
 print(result)
                 `;
                 redo(document.getElementById('pythonCode'), pythonCode);
-                document.getElementById('error').style.display = 'none';
             })
             .catch(error => {
                 document.getElementById('loading').style.display = 'none';
-                document.getElementById('error').style.display = 'block';
-                let errorMessage = 'An unexpected error occurred. Please try again later.';
-                
-                if (error.status === 429) {
-                    const details = error.details;
-                    if (details.retry_after) {
-                        errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
-                    } else if (details.reset_at) {
-                        const resetTime = new Date(details.reset_at);
-                        const waitTime = Math.ceil((resetTime - new Date()) / 1000);
-                        errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
-                    } else {
-                        errorMessage = `Rate limit exceeded. Please try again later.`;
-                    }
-                } else if (error.message) {
-                    errorMessage = error.message;
-                }
-                
-                document.querySelector('#error .terminal-alert').textContent = errorMessage;
+                document.getElementById('response').style.display = 'block';
+                document.getElementById('markdownContent').textContent = 'Error: ' + error;
             });
         });
     </script>

docs/md/index.md

@@ -1,4 +1,4 @@
-# Crawl4AI v0.2.77
+# Crawl4AI v0.2.74
 
 Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
 

docs/md/installation.md

@@ -2,13 +2,11 @@
 
 There are three ways to use Crawl4AI:
 
-1. As a library (Recommended).
-2. As a local server (Docker) or using the REST API.
-3. As a local server (Docker) using the pre-built image from Docker Hub.
+1. As a library (Recommended)
+2. As a local server (Docker) or using the REST API
+3. As a Google Colab notebook.    
 
-## Option 1: Library Installation
-
-You can try this Colab for a quick start: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sJPAmeLj5PMrg2VgOwMJ2ubGIcK0cJeX#scrollTo=g1RrmI4W_rPk)
+## Library Installation
 
 Crawl4AI offers flexible installation options to suit various use cases. Choose the option that best fits your needs:
 
@@ -59,135 +57,23 @@ Use this if you plan to modify the source code.
 crawl4ai-download-models
 ```
 
-## Option 2: Using Docker for Local Server
+## Using Docker for Local Server
 
-Crawl4AI can be run as a local server using Docker. The Dockerfile supports different installation options to cater to various use cases. Here's how you can build and run the Docker image:
-
-### Default Installation
-
-The default installation includes the basic Crawl4AI package without additional dependencies or pre-downloaded models.
+To run Crawl4AI as a local server using Docker:
 
 ```bash
-# For Mac users (M1/M2)
-docker build --platform linux/amd64 -t crawl4ai .
-
+# For Mac users
+# docker build --platform linux/amd64 -t crawl4ai .
 # For other users
-docker build -t crawl4ai .
-
-# Run the container
+# docker build -t crawl4ai .
 docker run -d -p 8000:80 crawl4ai
 ```
 
-### Full Installation (All Dependencies and Models)
-
-This option installs all dependencies and downloads the models.
-
-```bash
-# For Mac users (M1/M2)
-docker build --platform linux/amd64 --build-arg INSTALL_OPTION=all -t crawl4ai:all .
-
-# For other users
-docker build --build-arg INSTALL_OPTION=all -t crawl4ai:all .
-
-# Run the container
-docker run -d -p 8000:80 crawl4ai:all
-```
-
-### Torch Installation
-
-This option installs torch-related dependencies and downloads the models.
-
-```bash
-# For Mac users (M1/M2)
-docker build --platform linux/amd64 --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
-
-# For other users
-docker build --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
-
-# Run the container
-docker run -d -p 8000:80 crawl4ai:torch
-```
-
-### Transformer Installation
-
-This option installs transformer-related dependencies and downloads the models.
-
-```bash
-# For Mac users (M1/M2)
-docker build --platform linux/amd64 --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
-
-# For other users
-docker build --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
-
-# Run the container
-docker run -d -p 8000:80 crawl4ai:transformer
-```
-
-### Notes
-
-- The `--platform linux/amd64` flag is necessary for Mac users with M1/M2 chips to ensure compatibility.
-- The `-t` flag tags the image with a name (and optionally a tag in the 'name:tag' format).
-- The `-d` flag runs the container in detached mode.
-- The `-p 8000:80` flag maps port 8000 on the host to port 80 in the container.
-
-Choose the installation option that best suits your needs. The default installation is suitable for basic usage, while the other options provide additional capabilities for more advanced use cases.
-
-## Option 3: Using the Pre-built Image from Docker Hub
-
-You can use pre-built Crawl4AI images from Docker Hub, which are available for all platforms (Mac, Linux, Windows). We have official images as well as a community-contributed image (Thanks to https://github.com/FractalMind):
-
-### Default Installation
-
-```bash
-
-# Pull the image
-
-docker pull unclecode/crawl4ai:latest
-
-# Run the container
-
-docker run -d -p 8000:80 unclecode/crawl4ai:latest
-
-```
-
-### Community-Contributed Image
-
-A stable version of Crawl4AI is also available, created and maintained by a community member:
-
-```bash
-
-# Pull the community-contributed image
-
-docker pull ryser007/crawl4ai:stable
-
-# Run the container
-
-docker run -d -p 8000:80 ryser007/crawl4ai:stable
-
-```
-
-We'd like to express our gratitude to GitHub user [@FractalMind](https://github.com/FractalMind) for creating and maintaining this stable version of the Crawl4AI Docker image. Community contributions like this are invaluable to the project.
+## Using Google Colab
 
 
-### Testing the Installation
+You can also use Crawl4AI in a Google Colab notebook for easy setup and experimentation. Simply open the following Colab notebook and follow the instructions: 
 
-After running the container, you can test if it's working correctly:
-
-- On Mac and Linux:
-
-  ```bash
-
-  curl http://localhost:8000
-
-  ```
-
-- On Windows (PowerShell):
-
-  ```powershell
-
-  Invoke-WebRequest -Uri http://localhost:8000
-
-  ```
-
-  Or open a web browser and navigate to http://localhost:8000
+    ⚠️ This collab is a bit outdated. I'm updating it with the newest versions, so please refer to the website for the latest documentation. This will be updated in a few days, and you'll have the latest version here. Thank you so much.
 
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)

docs/md/introduction.md

@@ -20,6 +20,18 @@ Crawl4AI is designed to simplify the process of crawling web pages and extractin
 - **🎯 CSS Selector Support**: Extract specific content using CSS selectors.
 - **📝 Instruction/Keyword Refinement**: Pass instructions or keywords to refine the extraction process.
 
+## Recent Changes (v0.2.5) 🌟
+
+- **New Hooks**: Added six important hooks to the crawler:
+  - 🟢 `on_driver_created`: Called when the driver is ready for initializations.
+  - 🔵 `before_get_url`: Called right before Selenium fetches the URL.
+  - 🟣 `after_get_url`: Called after Selenium fetches the URL.
+  - 🟠 `before_return_html`: Called when the data is parsed and ready.
+  - 🟡 `on_user_agent_updated`: Called when the user changes the user agent, causing the driver to reinitialize.
+- **New Example**: Added an example in [`quickstart.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/quickstart.py) in the example folder under the docs.
+- **Improved Semantic Context**: Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
+- **Dockerfile Update**: Updated Dockerfile to ensure compatibility across multiple platforms.
+
 Check the [Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for more details.
 
 ## Power and Simplicity of Crawl4AI 🚀

main.py

@@ -22,15 +22,6 @@ from typing import List, Optional
 from crawl4ai.web_crawler import WebCrawler
 from crawl4ai.database import get_total_count, clear_db
 
-import time
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
-
-# load .env file
-from dotenv import load_dotenv
-load_dotenv()
-
 # Configuration
 __location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
 MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
@@ -39,78 +30,6 @@ lock = asyncio.Lock()
 
 app = FastAPI()
 
-# Initialize rate limiter
-def rate_limit_key_func(request: Request):
-    access_token = request.headers.get("access-token")
-    if access_token == os.environ.get('ACCESS_TOKEN'):
-        return None
-    return get_remote_address(request)
-
-limiter = Limiter(key_func=rate_limit_key_func)
-app.state.limiter = limiter
-
-# Dictionary to store last request times for each client
-last_request_times = {}
-last_rate_limit = {}
-
-
-def get_rate_limit():
-    limit = os.environ.get('ACCESS_PER_MIN', "5")
-    return f"{limit}/minute"
-
-# Custom rate limit exceeded handler
-async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
-    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
-        last_rate_limit[request.client.host] = time.time()
-    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
-    reset_at = time.time() + retry_after
-    return JSONResponse(
-        status_code=429,
-        content={
-            "detail": "Rate limit exceeded",
-            "limit": str(exc.limit.limit),
-            "retry_after": retry_after,
-            'reset_at': reset_at,
-            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
-        }
-    )
-    
-app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
-
-
-# Middleware for token-based bypass and per-request limit
-class RateLimitMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
-        access_token = request.headers.get("access-token")
-        if access_token == os.environ.get('ACCESS_TOKEN'):
-            return await call_next(request)
-        
-        path = request.url.path
-        if path in ["/crawl", "/old"]:
-            client_ip = request.client.host
-            current_time = time.time()
-            
-            # Check time since last request
-            if client_ip in last_request_times:
-                time_since_last_request = current_time - last_request_times[client_ip]
-                if time_since_last_request < SPAN:
-                    return JSONResponse(
-                        status_code=429,
-                        content={
-                            "detail": "Too many requests",
-                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
-                            "retry_after": max(0, SPAN - time_since_last_request),
-                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
-                        }
-                    )
-            
-            last_request_times[client_ip] = current_time
-
-        return await call_next(request)
-
-app.add_middleware(RateLimitMiddleware)
-
 # CORS configuration
 origins = ["*"]  # Allow all origins
 app.add_middleware(
@@ -154,7 +73,6 @@ def read_root():
     return RedirectResponse(url="/mkdocs")
 
 @app.get("/old", response_class=HTMLResponse)
-@limiter.limit(get_rate_limit())
 async def read_index(request: Request):
     partials_dir = os.path.join(__location__, "pages", "partial")
     partials = {}
@@ -189,7 +107,6 @@ def import_strategy(module_name: str, class_name: str, *args, **kwargs):
         raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")
 
 @app.post("/crawl")
-@limiter.limit(get_rate_limit())
 async def crawl_urls(crawl_request: CrawlRequest, request: Request):
     logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
     global current_requests

requirements.txt

@@ -12,13 +12,12 @@ python-dotenv==1.0.1
 requests==2.32.3
 rich==13.7.1
 scikit-learn==1.5.0
-selenium==4.23.1
+selenium==4.21.0
 uvicorn==0.30.1
 transformers==4.41.2
-# webdriver-manager==4.0.1
-# chromedriver-autoinstaller==0.6.4
+chromedriver-autoinstaller==0.6.4
 torch==2.3.1
 onnxruntime==1.18.0
 tokenizers==0.19.1
 pillow==10.3.0
-slowapi==0.1.9
+webdriver-manager==4.0.1

setup.py

@@ -1,18 +1,18 @@
 from setuptools import setup, find_packages
 import os
 from pathlib import Path
-import shutil
+import subprocess
+from setuptools.command.install import install
 
 # Create the .crawl4ai folder in the user's home directory if it doesn't exist
 # If the folder already exists, remove the cache folder
-crawl4ai_folder = Path.home() / ".crawl4ai"
-cache_folder = crawl4ai_folder / "cache"
+crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+if os.path.exists(f"{crawl4ai_folder}/cache"):
+    subprocess.run(["rm", "-rf", f"{crawl4ai_folder}/cache"])
+os.makedirs(crawl4ai_folder, exist_ok=True)
+os.makedirs(f"{crawl4ai_folder}/cache", exist_ok=True)
 
-if cache_folder.exists():
-    shutil.rmtree(cache_folder)
 
-crawl4ai_folder.mkdir(exist_ok=True)
-cache_folder.mkdir(exist_ok=True)
 
 # Read the requirements from requirements.txt
 with open("requirements.txt") as f:
@@ -25,7 +25,7 @@ transformer_requirements = [req for req in requirements if req.startswith(("tran
 
 setup(
     name="Crawl4AI",
-    version="0.2.77",
+    version="0.2.74",
     description="🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper",
     long_description=open("README.md", encoding="utf-8").read(),
     long_description_content_type="text/markdown",