Merge branch 'main' of https://github.com/unclecode/crawl4ai

Bump version to v0.2.73, update documentation, and resolve installation issues
feat: Bump version to v0.2.73 and update documentation
2024-07-03 15:26:47 +08:00 · 2024-07-03 15:26:43 +08:00 · 2024-07-03 15:19:22 +08:00 · 2024-07-01 07:31:07 +02:00 · 2024-07-01 00:07:32 +08:00 · 2024-06-30 20:27:33 +05:30
12 changed files with 139 additions and 88 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 ## [v0.2.73] - 2024-07-03
 💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
 * Supporting website need "with-head" mode to crawl the website with head.
 * Fixing the installation issues for setup.py and dockerfile.
 * Resolve multiple issues.
 ## [v0.2.72] - 2024-06-30
 This release brings exciting updates and improvements to our project! 🎉
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# Crawl4AI v0.2.72 🕷️🤖
+# Crawl4AI v0.2.73 🕷️🤖
 [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
 [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)
@@ -11,7 +11,7 @@ Crawl4AI simplifies web crawling and data extraction, making it accessible for l
 ## Try it Now!
 - Use as REST API: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zODYjhemJ5bUmYceWpVoBMVpd0ofzNBZ?usp=sharing)
- Use as Python library: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
+- Use as Python library: This collab is a bit outdated. I'm updating it with the newest versions, so please refer to the website for the latest documentation. This will be updated in a few days, and you'll have the latest version here. Thank you so much. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
 ✨ visit our [Documentation Website](https://crawl4ai.com/mkdocs/)
--- a/crawl4ai/crawler_strategy.py
+++ b/crawl4ai/crawler_strategy.py
@@ -9,7 +9,8 @@ from selenium.common.exceptions import InvalidArgumentException, WebDriverExcept
 from selenium.webdriver.chrome.service import Service as ChromeService
 from webdriver_manager.chrome import ChromeDriverManager
-import logging
+from .config import *
 import logging, time
 import base64
 from PIL import Image, ImageDraw, ImageFont
 from io import BytesIO
@@ -177,8 +178,20 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
        # Set extra HTTP headers
        self.driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': headers})
    def _ensure_page_load(self,  max_checks=6, check_interval=0.01):
        initial_length = len(self.driver.page_source)
        for ix in range(max_checks):
            # print(f"Checking page load: {ix}")
            time.sleep(check_interval)
            current_length = len(self.driver.page_source)
            if current_length != initial_length:
                break
-    def crawl(self, url: str) -> str:
+        return self.driver.page_source
    def crawl(self, url: str, **kwargs) -> str:
        # Create md5 hash of the URL
        import hashlib
        url_hash = hashlib.md5(url.encode()).hexdigest()
@@ -194,18 +207,24 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
            if self.verbose:
                print(f"[LOG] 🕸️ Crawling {url} using LocalSeleniumCrawlerStrategy...")
            self.driver.get(url) #<html><head></head><body></body></html>
-            html = self.driver.page_source                
+            
            WebDriverWait(self.driver, 20).until(
                lambda d: d.execute_script('return document.readyState') == 'complete'
            )
            WebDriverWait(self.driver, 10).until(
                EC.presence_of_all_elements_located((By.TAG_NAME, "body"))
            )
            self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            html = self._ensure_page_load() # self.driver.page_source                
            can_not_be_done_headless = False # Look at my creativity for naming variables
            # TODO: Very ugly way for now but it works
-            if html == "<html><head></head><body></body></html>":
+            if not kwargs.get('bypass_headless', False) and html == "<html><head></head><body></body></html>":
                print("[LOG] 🙌 Page could not be loaded in headless mode. Trying non-headless mode...")
                can_not_be_done_headless = True
                options = Options()
                options.headless = False
                # set window size very small
-                options.add_argument("--window-size=10,10")
+                options.add_argument("--window-size=5,5")
                driver = webdriver.Chrome(service=self.service, options=options)
                driver.get(url)
                html = driver.page_source
--- a/crawl4ai/extraction_strategy.py
+++ b/crawl4ai/extraction_strategy.py
@@ -101,7 +101,7 @@ class LLMExtractionStrategy(ExtractionStrategy):
            prompt_with_variables = PROMPT_EXTRACT_BLOCKS_WITH_INSTRUCTION
        if self.extract_type == "schema":
-            variable_values["SCHEMA"] = json.dumps(self.schema)
+            variable_values["SCHEMA"] = json.dumps(self.schema, indent=2)
            prompt_with_variables = PROMPT_EXTRACT_SCHEMA_WITH_INSTRUCTION
        for variable in variable_values:
@@ -109,7 +109,7 @@ class LLMExtractionStrategy(ExtractionStrategy):
                "{" + variable + "}", variable_values[variable]
            )
-        response = perform_completion_with_backoff(self.provider, prompt_with_variables, self.api_token)
+        response = perform_completion_with_backoff(self.provider, prompt_with_variables, self.api_token) # , json_response=self.extract_type == "schema")
        try:
            blocks = extract_xml_data(["blocks"], response.choices[0].message.content)['blocks']
            blocks = json.loads(blocks)
@@ -196,6 +196,10 @@ class LLMExtractionStrategy(ExtractionStrategy):
                time.sleep(0.5)  # 500 ms delay between each processing
        else:
            # Parallel processing using ThreadPoolExecutor
            # extract_func = partial(self.extract, url)
            # for ix, section in enumerate(merged_sections):
            #     extracted_content.append(extract_func(ix, section))            
            with ThreadPoolExecutor(max_workers=4) as executor:
                extract_func = partial(self.extract, url)
                futures = [executor.submit(extract_func, ix, section) for ix, section in enumerate(merged_sections)]
--- a/crawl4ai/prompts.py
+++ b/crawl4ai/prompts.py
@@ -186,7 +186,7 @@ The user has made the following request for what information to extract from the
 Please carefully read the URL content and the user's request. If the user provided a desired JSON schema in the <schema_block> above, extract the requested information from the URL content according to that schema. If no schema was provided, infer an appropriate JSON schema based on the user's request that will best capture the key information they are looking for.
 Extraction instructions:
-Return the extracted information as a list of JSON objects, with each object in the list corresponding to a block of content from the URL, in the same order as it appears on the page. Wrap the entire JSON list in <blocks> tags.
+Return the extracted information as a list of JSON objects, with each object in the list corresponding to a block of content from the URL, in the same order as it appears on the page. Wrap the entire JSON list in <blocks>...</blocks> XML tags.
 Quality Reflection:
 Before outputting your final answer, double check that the JSON you are returning is complete, containing all the information requested by the user, and is valid JSON that could be parsed by json.loads() with no errors or omissions. The outputted JSON objects should fully match the schema, either provided or inferred.
@@ -194,5 +194,11 @@ Before outputting your final answer, double check that the JSON you are returnin
 Quality Score:
 After reflecting, score the quality and completeness of the JSON data you are about to return on a scale of 1 to 5. Write the score inside <score> tags.
 Avoid Common Mistakes:
 - Do NOT add any comments using "//" or "#" in the JSON output. It causes parsing errors.
 - Make sure the JSON is properly formatted with curly braces, square brackets, and commas in the right places.
 - Do not miss closing </blocks> tag at the end of the JSON output.
 - Do not generate the Python coee show me how to do the task, this is your task to extract the information and return it in JSON format.
 Result
-Output the final list of JSON objects, wrapped in <blocks> tags."""
+Output the final list of JSON objects, wrapped in <blocks>...</blocks> XML tags. Make sure to close the tag properly."""
--- a/crawl4ai/utils.py
+++ b/crawl4ai/utils.py
@@ -419,7 +419,6 @@ def get_content_of_website(url, html, word_count_threshold = MIN_WORD_THRESHOLD,
        print('Error processing HTML content:', str(e))
        raise InvalidCSSSelectorError(f"Invalid CSS selector: {css_selector}") from e
 def get_content_of_website_optimized(url: str, html: str, word_count_threshold: int = MIN_WORD_THRESHOLD, css_selector: str = None, **kwargs) -> Dict[str, Any]:
    if not html:
        return None
@@ -439,71 +438,75 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
    media = {'images': [], 'videos': [], 'audios': []}
    def process_element(element: element.PageElement) -> bool:
-        if isinstance(element, NavigableString):
+        try:
-            if isinstance(element, Comment):
+            if isinstance(element, NavigableString):
-                element.extract()
+                if isinstance(element, Comment):
-            return False
+                    element.extract()
                return False
-        if element.name in ['script', 'style', 'link', 'meta', 'noscript']:
+            if element.name in ['script', 'style', 'link', 'meta', 'noscript']:
-            element.decompose()
+                element.decompose()
-            return False
+                return False
-        keep_element = False
+            keep_element = False
-        if element.name == 'a' and element.get('href'):
+            if element.name == 'a' and element.get('href'):
-            href = element['href']
+                href = element['href']
-            url_base = url.split('/')[2]
+                url_base = url.split('/')[2]
-            link_data = {'href': href, 'text': element.get_text()}
+                link_data = {'href': href, 'text': element.get_text()}
-            if href.startswith('http') and url_base not in href:
+                if href.startswith('http') and url_base not in href:
-                links['external'].append(link_data)
+                    links['external'].append(link_data)
            else:
                links['internal'].append(link_data)
            keep_element = True
        elif element.name == 'img':
            media['images'].append({
                'src': element.get('src'),
                'alt': element.get('alt'),
                'type': 'image'
            })
            return True  # Always keep image elements
        elif element.name in ['video', 'audio']:
            media[f"{element.name}s"].append({
                'src': element.get('src'),
                'alt': element.get('alt'),
                'type': element.name
            })
            return True  # Always keep video and audio elements
        if element.name != 'pre':
            if element.name in ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']:
                if kwargs.get('only_text', False):
                    element.replace_with(element.get_text())
                else:
-                    element.unwrap()
+                    links['internal'].append(link_data)
-            elif element.name != 'img':
+                keep_element = True
                element.attrs = {}
-        # Process children
+            elif element.name == 'img':
-        for child in list(element.children):
+                media['images'].append({
-            if isinstance(child, NavigableString) and not isinstance(child, Comment):
+                    'src': element.get('src'),
-                if len(child.strip()) > 0:
+                    'alt': element.get('alt'),
-                    keep_element = True
+                    'type': 'image'
-            else:
+                })
-                if process_element(child):
+                return True  # Always keep image elements
                    keep_element = True
-        # Check word count
+            elif element.name in ['video', 'audio']:
-        if not keep_element:
+                media[f"{element.name}s"].append({
-            word_count = len(element.get_text(strip=True).split())
+                    'src': element.get('src'),
-            keep_element = word_count >= word_count_threshold
+                    'alt': element.get('alt'),
                    'type': element.name
                })
                return True  # Always keep video and audio elements
-        if not keep_element:
+            if element.name != 'pre':
-            element.decompose()
+                if element.name in ['b', 'i', 'u', 'span', 'del', 'ins', 'sub', 'sup', 'strong', 'em', 'code', 'kbd', 'var', 's', 'q', 'abbr', 'cite', 'dfn', 'time', 'small', 'mark']:
                    if kwargs.get('only_text', False):
                        element.replace_with(element.get_text())
                    else:
                        element.unwrap()
                elif element.name != 'img':
                    element.attrs = {}
-        return keep_element
+            # Process children
            for child in list(element.children):
                if isinstance(child, NavigableString) and not isinstance(child, Comment):
                    if len(child.strip()) > 0:
                        keep_element = True
                else:
                    if process_element(child):
                        keep_element = True
            # Check word count
            if not keep_element:
                word_count = len(element.get_text(strip=True).split())
                keep_element = word_count >= word_count_threshold
            if not keep_element:
                element.decompose()
            return keep_element
        except Exception as e:
            print('Error processing element:', str(e))
            return False
    process_element(body)
@@ -540,7 +543,6 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
        'metadata': meta
    }
 def extract_metadata(html, soup = None):
    metadata = {}
@@ -599,12 +601,16 @@ def extract_xml_data(tags, string):
    return data
 # Function to perform the completion with exponential backoff
-def perform_completion_with_backoff(provider, prompt_with_variables, api_token):
+def perform_completion_with_backoff(provider, prompt_with_variables, api_token, json_response = False):
    from litellm import completion 
    from litellm.exceptions import RateLimitError
    max_attempts = 3
    base_delay = 2  # Base delay in seconds, you can adjust this based on your needs
    extra_args = {}
    if json_response:
        extra_args["response_format"] = { "type": "json_object" }
    for attempt in range(max_attempts):
        try:
            response =completion(
@@ -613,7 +619,8 @@ def perform_completion_with_backoff(provider, prompt_with_variables, api_token):
                    {"role": "user", "content": prompt_with_variables}
                ],
                temperature=0.01,
-                api_key=api_token
+                api_key=api_token,
                **extra_args
            )
            return response  # Return the successful response
        except RateLimitError as e:
--- a/crawl4ai/web_crawler.py
+++ b/crawl4ai/web_crawler.py
@@ -11,6 +11,8 @@ from .crawler_strategy import *
 from typing import List
 from concurrent.futures import ThreadPoolExecutor
 from .config import *
 import warnings
 warnings.filterwarnings("ignore", message='Field "model_name" has conflict with protected namespace "model_".')
 class WebCrawler:
@@ -164,7 +166,7 @@ class WebCrawler:
                    if user_agent:
                        self.crawler_strategy.update_user_agent(user_agent)
                    t1 = time.time()
-                    html = self.crawler_strategy.crawl(url)
+                    html = self.crawler_strategy.crawl(url, **kwargs)
                    t2 = time.time()
                    if verbose:
                        print(f"[LOG] 🚀 Crawling done for {url}, success: {bool(html)}, time taken: {t2 - t1} seconds")
--- a/docs/md/changelog.md
+++ b/docs/md/changelog.md
@@ -1,5 +1,13 @@
 # Changelog
 ## [v0.2.73] - 2024-07-03
 💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
 * Supporting website need "with-head" mode to crawl the website with head.
 * Fixing the installation issues for setup.py and dockerfile.
 * Resolve multiple issues.
 ## [v0.2.72] - 2024-06-30
 This release brings exciting updates and improvements to our project! 🎉
--- a/docs/md/index.md
+++ b/docs/md/index.md
@@ -1,4 +1,4 @@
-# Crawl4AI v0.2.72
+# Crawl4AI v0.2.73
 Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
--- a/docs/md/installation.md
+++ b/docs/md/installation.md
@@ -1,9 +1,10 @@
 # Installation 💻
 There are three ways to use Crawl4AI:
 1. As a library (Recommended)
 2. As a local server (Docker) or using the REST API
-3. As a Google Colab notebook. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
+3. As a Google Colab notebook.    
 ## Library Installation
@@ -70,4 +71,9 @@ docker run -d -p 8000:80 crawl4ai
 ## Using Google Colab
-You can also use Crawl4AI in a Google Colab notebook for easy setup and experimentation. Simply open the following Colab notebook and follow the instructions: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
+
 You can also use Crawl4AI in a Google Colab notebook for easy setup and experimentation. Simply open the following Colab notebook and follow the instructions: 
    ⚠️ This collab is a bit outdated. I'm updating it with the newest versions, so please refer to the website for the latest documentation. This will be updated in a few days, and you'll have the latest version here. Thank you so much.
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz8u30rvbq6Scodye9AGCw8Qg_Z8QGsk)
--- a/main.py
+++ b/main.py
@@ -168,4 +168,4 @@ async def get_chunking_strategies():
 if __name__ == "__main__":
    import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8080)
+    uvicorn.run(app, host="0.0.0.0", port=8888)
--- a/setup.py
+++ b/setup.py
@@ -18,17 +18,11 @@ default_requirements = [req for req in requirements if not req.startswith(("torc
 torch_requirements = [req for req in requirements if req.startswith(("torch", "nltk", "spacy", "scikit-learn", "numpy"))]
 transformer_requirements = [req for req in requirements if req.startswith(("transformers", "tokenizers", "onnxruntime"))]
 class CustomInstallCommand(install):
    """Customized setuptools install command to install spacy without dependencies."""
    def run(self):
        install.run(self)
        subprocess.check_call([os.sys.executable, '-m', 'pip', 'install', 'spacy', '--no-deps'])
 setup(
    name="Crawl4AI",
-    version="0.2.72",
+    version="0.2.73",
    description="🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper",
-    long_description=open("README.md").read(),
+    long_description=open("README.md", encoding="utf-8").read(),
    long_description_content_type="text/markdown",
    url="https://github.com/unclecode/crawl4ai",
    author="Unclecode",
@@ -41,9 +35,6 @@ setup(
        "transformer": transformer_requirements,
        "all": requirements,
    },
    cmdclass={
        'install': CustomInstallCommand,
    },
    entry_points={
        'console_scripts': [
            'crawl4ai-download-models=crawl4ai.model_loader:main',
Author	SHA1	Message	Date
unclecode	3ff2a0d0e7	Merge branch 'main' of https://github.com/unclecode/crawl4ai	2024-07-03 15:26:47 +08:00
unclecode	3cd1b3719f	Bump version to v0.2.73, update documentation, and resolve installation issues	2024-07-03 15:26:43 +08:00
unclecode	9926eb9f95	feat: Bump version to v0.2.73 and update documentation This commit updates the version number to v0.2.73 and makes corresponding changes in the README.md and Dockerfile. Docker file install the default mode, this resolve many of installation issues. Additionally, the installation instructions are updated to include support for different modes. Setup.py doesn't have anymore dependancy on Spacy. The change log is also updated to reflect these changes. Supporting websites need with-head browser.	2024-07-03 15:19:22 +08:00
UncleCode	3abaa82501	Merge pull request #37 from shivkumar0757/fix-readme-encoding @shivkumar0757 Great work! I value your contribution and have merged your pull request. You will be credited in the upcoming change-log. Thank you for your continuous support in advancing this library, to democratize an open access crawler to everyone.	2024-07-01 07:31:07 +02:00
unclecode	88d8cd8650	feat: Add page load check for LocalSeleniumCrawlerStrategy This commit adds a page load check for the LocalSeleniumCrawlerStrategy in the `crawl` method. The `_ensure_page_load` method is introduced to ensure that the page has finished loading before proceeding. This helps to prevent issues with incomplete page sources and improves the reliability of the crawler.	2024-07-01 00:07:32 +08:00
shiv	a08f21d66c	Fix UnicodeDecodeError by reading README.md with UTF-8 encoding	2024-06-30 20:27:33 +05:30
unclecode	d58286989c	UPDATE DOCUMENTS	2024-06-30 00:34:02 +08:00
`@@ -1,4 +1,4 @@`
	`# Crawl4AI v0.2.72`	`# Crawl4AI v0.2.73`

	`Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.`	`Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.`