Merge pull request #7 from aravindkarnam/main
pulling the main branch into scraper-uc
This commit is contained in:
aravind
BIN
docs/.DS_Store
vendored
BIN
docs/.DS_Store
vendored
Binary file not shown.
BIN
docs/assets/pitch-dark.png
Normal file
BIN
docs/assets/pitch-dark.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
64
docs/assets/pitch-dark.svg
Normal file
64
docs/assets/pitch-dark.svg
Normal file
@@ -0,0 +1,64 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 500">
|
||||
<!-- Background -->
|
||||
<rect width="800" height="500" fill="#1a1a1a"/>
|
||||
|
||||
<!-- Opportunities Section -->
|
||||
<g transform="translate(50,50)">
|
||||
<!-- Opportunity 1 Box -->
|
||||
<rect x="0" y="0" width="300" height="150" rx="10" fill="#1a2d3d" stroke="#64b5f6" stroke-width="2"/>
|
||||
<text x="150" y="30" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#64b5f6">Data Capitalization Opportunity</text>
|
||||
<text x="150" y="60" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
|
||||
<tspan x="150" dy="0">Transform digital footprints into assets</tspan>
|
||||
<tspan x="150" dy="20">Personal data as capital</tspan>
|
||||
<tspan x="150" dy="20">Enterprise knowledge valuation</tspan>
|
||||
<tspan x="150" dy="20">New form of wealth creation</tspan>
|
||||
</text>
|
||||
|
||||
<!-- Opportunity 2 Box -->
|
||||
<rect x="0" y="200" width="300" height="150" rx="10" fill="#1a2d1a" stroke="#81c784" stroke-width="2"/>
|
||||
<text x="150" y="230" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#81c784">Authentic Data Potential</text>
|
||||
<text x="150" y="260" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
|
||||
<tspan x="150" dy="0">Vast reservoir of real insights</tspan>
|
||||
<tspan x="150" dy="20">Enhanced AI development</tspan>
|
||||
<tspan x="150" dy="20">Diverse human knowledge</tspan>
|
||||
<tspan x="150" dy="20">Willing participation model</tspan>
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<!-- Development Pathway -->
|
||||
<g transform="translate(450,50)">
|
||||
<!-- Step 1 Box -->
|
||||
<rect x="0" y="0" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
|
||||
<text x="150" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">1. Open-Source Foundation</text>
|
||||
<text x="150" y="65" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Data extraction engine & community development</text>
|
||||
|
||||
<!-- Step 2 Box -->
|
||||
<rect x="0" y="125" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
|
||||
<text x="150" y="160" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">2. Data Capitalization Platform</text>
|
||||
<text x="150" y="190" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Tools to structure & value digital assets</text>
|
||||
|
||||
<!-- Step 3 Box -->
|
||||
<rect x="0" y="250" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
|
||||
<text x="150" y="285" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">3. Shared Data Marketplace</text>
|
||||
<text x="150" y="315" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Economic platform for data exchange</text>
|
||||
</g>
|
||||
|
||||
<!-- Connecting Arrows -->
|
||||
<g transform="translate(400,125)">
|
||||
<path d="M-20,0 L40,0" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
|
||||
<path d="M-20,200 L40,200" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
|
||||
</g>
|
||||
|
||||
<!-- Arrow Marker -->
|
||||
<defs>
|
||||
<marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
|
||||
<polygon points="0 0, 10 3.5, 0 7" fill="#666"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- Vision Box at Bottom -->
|
||||
<g transform="translate(200,420)">
|
||||
<rect x="0" y="0" width="400" height="60" rx="10" fill="#2d2613" stroke="#ffd54f" stroke-width="2"/>
|
||||
<text x="200" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ffd54f">Economic Vision: Shared Data Economy</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"RegexChunking": "### RegexChunking\n\n`RegexChunking` is a text chunking strategy that splits a given text into smaller parts using regular expressions.\nThis is useful for preparing large texts for processing by language models, ensuring they are divided into manageable segments.\n\n#### Constructor Parameters:\n- `patterns` (list, optional): A list of regular expression patterns used to split the text. Default is to split by double newlines (`['\\n\\n']`).\n\n#### Example usage:\n```python\nchunker = RegexChunking(patterns=[r'\\n\\n', r'\\. '])\nchunks = chunker.chunk(\"This is a sample text. It will be split into chunks.\")\n```",
|
||||
|
||||
"NlpSentenceChunking": "### NlpSentenceChunking\n\n`NlpSentenceChunking` uses a natural language processing model to chunk a given text into sentences. This approach leverages SpaCy to accurately split text based on sentence boundaries.\n\n#### Constructor Parameters:\n- None.\n\n#### Example usage:\n```python\nchunker = NlpSentenceChunking()\nchunks = chunker.chunk(\"This is a sample text. It will be split into sentences.\")\n```",
|
||||
|
||||
"TopicSegmentationChunking": "### TopicSegmentationChunking\n\n`TopicSegmentationChunking` uses the TextTiling algorithm to segment a given text into topic-based chunks. This method identifies thematic boundaries in the text.\n\n#### Constructor Parameters:\n- `num_keywords` (int, optional): The number of keywords to extract for each topic segment. Default is `3`.\n\n#### Example usage:\n```python\nchunker = TopicSegmentationChunking(num_keywords=3)\nchunks = chunker.chunk(\"This is a sample text. It will be split into topic-based segments.\")\n```",
|
||||
|
||||
"FixedLengthWordChunking": "### FixedLengthWordChunking\n\n`FixedLengthWordChunking` splits a given text into chunks of fixed length, based on the number of words.\n\n#### Constructor Parameters:\n- `chunk_size` (int, optional): The number of words in each chunk. Default is `100`.\n\n#### Example usage:\n```python\nchunker = FixedLengthWordChunking(chunk_size=100)\nchunks = chunker.chunk(\"This is a sample text. It will be split into fixed-length word chunks.\")\n```",
|
||||
|
||||
"SlidingWindowChunking": "### SlidingWindowChunking\n\n`SlidingWindowChunking` uses a sliding window approach to chunk a given text. Each chunk has a fixed length, and the window slides by a specified step size.\n\n#### Constructor Parameters:\n- `window_size` (int, optional): The number of words in each chunk. Default is `100`.\n- `step` (int, optional): The number of words to slide the window. Default is `50`.\n\n#### Example usage:\n```python\nchunker = SlidingWindowChunking(window_size=100, step=50)\nchunks = chunker.chunk(\"This is a sample text. It will be split using a sliding window approach.\")\n```"
|
||||
}
|
||||
|
||||
300
docs/examples/docker_example.py
Normal file
300
docs/examples/docker_example.py
Normal file
@@ -0,0 +1,300 @@
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import sys
|
||||
import base64
|
||||
import os
|
||||
from typing import Dict, Any
|
||||
|
||||
class Crawl4AiTester:
|
||||
def __init__(self, base_url: str = "http://localhost:11235"):
|
||||
self.base_url = base_url
|
||||
|
||||
def submit_and_wait(self, request_data: Dict[str, Any], timeout: int = 300) -> Dict[str, Any]:
|
||||
# Submit crawl job
|
||||
response = requests.post(f"{self.base_url}/crawl", json=request_data)
|
||||
task_id = response.json()["task_id"]
|
||||
print(f"Task ID: {task_id}")
|
||||
|
||||
# Poll for result
|
||||
start_time = time.time()
|
||||
while True:
|
||||
if time.time() - start_time > timeout:
|
||||
raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
|
||||
|
||||
result = requests.get(f"{self.base_url}/task/{task_id}")
|
||||
status = result.json()
|
||||
|
||||
if status["status"] == "failed":
|
||||
print("Task failed:", status.get("error"))
|
||||
raise Exception(f"Task failed: {status.get('error')}")
|
||||
|
||||
if status["status"] == "completed":
|
||||
return status
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
def test_docker_deployment(version="basic"):
|
||||
tester = Crawl4AiTester()
|
||||
print(f"Testing Crawl4AI Docker {version} version")
|
||||
|
||||
# Health check with timeout and retry
|
||||
max_retries = 5
|
||||
for i in range(max_retries):
|
||||
try:
|
||||
health = requests.get(f"{tester.base_url}/health", timeout=10)
|
||||
print("Health check:", health.json())
|
||||
break
|
||||
except requests.exceptions.RequestException as e:
|
||||
if i == max_retries - 1:
|
||||
print(f"Failed to connect after {max_retries} attempts")
|
||||
sys.exit(1)
|
||||
print(f"Waiting for service to start (attempt {i+1}/{max_retries})...")
|
||||
time.sleep(5)
|
||||
|
||||
# Test cases based on version
|
||||
test_basic_crawl(tester)
|
||||
|
||||
# if version in ["full", "transformer"]:
|
||||
# test_cosine_extraction(tester)
|
||||
|
||||
# test_js_execution(tester)
|
||||
# test_css_selector(tester)
|
||||
# test_structured_extraction(tester)
|
||||
# test_llm_extraction(tester)
|
||||
# test_llm_with_ollama(tester)
|
||||
# test_screenshot(tester)
|
||||
|
||||
|
||||
def test_basic_crawl(tester: Crawl4AiTester):
|
||||
print("\n=== Testing Basic Crawl ===")
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 10
|
||||
}
|
||||
|
||||
result = tester.submit_and_wait(request)
|
||||
print(f"Basic crawl result length: {len(result['result']['markdown'])}")
|
||||
assert result["result"]["success"]
|
||||
assert len(result["result"]["markdown"]) > 0
|
||||
|
||||
def test_js_execution(tester: Crawl4AiTester):
|
||||
print("\n=== Testing JS Execution ===")
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 8,
|
||||
"js_code": [
|
||||
"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
|
||||
],
|
||||
"wait_for": "article.tease-card:nth-child(10)",
|
||||
"crawler_params": {
|
||||
"headless": True
|
||||
}
|
||||
}
|
||||
|
||||
result = tester.submit_and_wait(request)
|
||||
print(f"JS execution result length: {len(result['result']['markdown'])}")
|
||||
assert result["result"]["success"]
|
||||
|
||||
def test_css_selector(tester: Crawl4AiTester):
|
||||
print("\n=== Testing CSS Selector ===")
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 7,
|
||||
"css_selector": ".wide-tease-item__description",
|
||||
"crawler_params": {
|
||||
"headless": True
|
||||
},
|
||||
"extra": {"word_count_threshold": 10}
|
||||
|
||||
}
|
||||
|
||||
result = tester.submit_and_wait(request)
|
||||
print(f"CSS selector result length: {len(result['result']['markdown'])}")
|
||||
assert result["result"]["success"]
|
||||
|
||||
def test_structured_extraction(tester: Crawl4AiTester):
|
||||
print("\n=== Testing Structured Extraction ===")
|
||||
schema = {
|
||||
"name": "Coinbase Crypto Prices",
|
||||
"baseSelector": ".cds-tableRow-t45thuk",
|
||||
"fields": [
|
||||
{
|
||||
"name": "crypto",
|
||||
"selector": "td:nth-child(1) h2",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "symbol",
|
||||
"selector": "td:nth-child(1) p",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": "td:nth-child(2)",
|
||||
"type": "text",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
request = {
|
||||
"urls": "https://www.coinbase.com/explore",
|
||||
"priority": 9,
|
||||
"extraction_config": {
|
||||
"type": "json_css",
|
||||
"params": {
|
||||
"schema": schema
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result = tester.submit_and_wait(request)
|
||||
extracted = json.loads(result["result"]["extracted_content"])
|
||||
print(f"Extracted {len(extracted)} items")
|
||||
print("Sample item:", json.dumps(extracted[0], indent=2))
|
||||
assert result["result"]["success"]
|
||||
assert len(extracted) > 0
|
||||
|
||||
def test_llm_extraction(tester: Crawl4AiTester):
|
||||
print("\n=== Testing LLM Extraction ===")
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"model_name": {
|
||||
"type": "string",
|
||||
"description": "Name of the OpenAI model."
|
||||
},
|
||||
"input_fee": {
|
||||
"type": "string",
|
||||
"description": "Fee for input token for the OpenAI model."
|
||||
},
|
||||
"output_fee": {
|
||||
"type": "string",
|
||||
"description": "Fee for output token for the OpenAI model."
|
||||
}
|
||||
},
|
||||
"required": ["model_name", "input_fee", "output_fee"]
|
||||
}
|
||||
|
||||
request = {
|
||||
"urls": "https://openai.com/api/pricing",
|
||||
"priority": 8,
|
||||
"extraction_config": {
|
||||
"type": "llm",
|
||||
"params": {
|
||||
"provider": "openai/gpt-4o-mini",
|
||||
"api_token": os.getenv("OPENAI_API_KEY"),
|
||||
"schema": schema,
|
||||
"extraction_type": "schema",
|
||||
"instruction": """From the crawled content, extract all mentioned model names along with their fees for input and output tokens."""
|
||||
}
|
||||
},
|
||||
"crawler_params": {"word_count_threshold": 1}
|
||||
}
|
||||
|
||||
try:
|
||||
result = tester.submit_and_wait(request)
|
||||
extracted = json.loads(result["result"]["extracted_content"])
|
||||
print(f"Extracted {len(extracted)} model pricing entries")
|
||||
print("Sample entry:", json.dumps(extracted[0], indent=2))
|
||||
assert result["result"]["success"]
|
||||
except Exception as e:
|
||||
print(f"LLM extraction test failed (might be due to missing API key): {str(e)}")
|
||||
|
||||
def test_llm_with_ollama(tester: Crawl4AiTester):
|
||||
print("\n=== Testing LLM with Ollama ===")
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"article_title": {
|
||||
"type": "string",
|
||||
"description": "The main title of the news article"
|
||||
},
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"description": "A brief summary of the article content"
|
||||
},
|
||||
"main_topics": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Main topics or themes discussed in the article"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 8,
|
||||
"extraction_config": {
|
||||
"type": "llm",
|
||||
"params": {
|
||||
"provider": "ollama/llama2",
|
||||
"schema": schema,
|
||||
"extraction_type": "schema",
|
||||
"instruction": "Extract the main article information including title, summary, and main topics."
|
||||
}
|
||||
},
|
||||
"extra": {"word_count_threshold": 1},
|
||||
"crawler_params": {"verbose": True}
|
||||
}
|
||||
|
||||
try:
|
||||
result = tester.submit_and_wait(request)
|
||||
extracted = json.loads(result["result"]["extracted_content"])
|
||||
print("Extracted content:", json.dumps(extracted, indent=2))
|
||||
assert result["result"]["success"]
|
||||
except Exception as e:
|
||||
print(f"Ollama extraction test failed: {str(e)}")
|
||||
|
||||
def test_cosine_extraction(tester: Crawl4AiTester):
|
||||
print("\n=== Testing Cosine Extraction ===")
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 8,
|
||||
"extraction_config": {
|
||||
"type": "cosine",
|
||||
"params": {
|
||||
"semantic_filter": "business finance economy",
|
||||
"word_count_threshold": 10,
|
||||
"max_dist": 0.2,
|
||||
"top_k": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
result = tester.submit_and_wait(request)
|
||||
extracted = json.loads(result["result"]["extracted_content"])
|
||||
print(f"Extracted {len(extracted)} text clusters")
|
||||
print("First cluster tags:", extracted[0]["tags"])
|
||||
assert result["result"]["success"]
|
||||
except Exception as e:
|
||||
print(f"Cosine extraction test failed: {str(e)}")
|
||||
|
||||
def test_screenshot(tester: Crawl4AiTester):
|
||||
print("\n=== Testing Screenshot ===")
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 5,
|
||||
"screenshot": True,
|
||||
"crawler_params": {
|
||||
"headless": True
|
||||
}
|
||||
}
|
||||
|
||||
result = tester.submit_and_wait(request)
|
||||
print("Screenshot captured:", bool(result["result"]["screenshot"]))
|
||||
|
||||
if result["result"]["screenshot"]:
|
||||
# Save screenshot
|
||||
screenshot_data = base64.b64decode(result["result"]["screenshot"])
|
||||
with open("test_screenshot.jpg", "wb") as f:
|
||||
f.write(screenshot_data)
|
||||
print("Screenshot saved as test_screenshot.jpg")
|
||||
|
||||
assert result["result"]["success"]
|
||||
|
||||
if __name__ == "__main__":
|
||||
version = sys.argv[1] if len(sys.argv) > 1 else "basic"
|
||||
# version = "full"
|
||||
test_docker_deployment(version)
|
||||
File diff suppressed because one or more lines are too long
@@ -10,7 +10,7 @@ import time
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from typing import Dict
|
||||
from typing import Dict, List
|
||||
from bs4 import BeautifulSoup
|
||||
from pydantic import BaseModel, Field
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
@@ -379,6 +379,19 @@ async def crawl_custom_browser_type():
|
||||
print(result.markdown[:500])
|
||||
print("Time taken: ", time.time() - start)
|
||||
|
||||
async def crawl_with_user_simultion():
|
||||
async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
|
||||
url = "YOUR-URL-HERE"
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
bypass_cache=True,
|
||||
magic = True, # Automatically detects and removes overlays, popups, and other elements that block content
|
||||
# simulate_user = True,# Causes a series of random mouse movements and clicks to simulate user interaction
|
||||
# override_navigator = True # Overrides the navigator object to make it look like a real user
|
||||
)
|
||||
|
||||
print(result.markdown)
|
||||
|
||||
async def speed_comparison():
|
||||
# print("\n--- Speed Comparison ---")
|
||||
# print("Firecrawl (simulated):")
|
||||
@@ -444,6 +457,57 @@ async def speed_comparison():
|
||||
print("If you run these tests in an environment with better network conditions,")
|
||||
print("you may observe an even more significant speed advantage for Crawl4AI.")
|
||||
|
||||
async def generate_knowledge_graph():
|
||||
class Entity(BaseModel):
|
||||
name: str
|
||||
description: str
|
||||
|
||||
class Relationship(BaseModel):
|
||||
entity1: Entity
|
||||
entity2: Entity
|
||||
description: str
|
||||
relation_type: str
|
||||
|
||||
class KnowledgeGraph(BaseModel):
|
||||
entities: List[Entity]
|
||||
relationships: List[Relationship]
|
||||
|
||||
extraction_strategy = LLMExtractionStrategy(
|
||||
provider='openai/gpt-4o-mini', # Or any other provider, including Ollama and open source models
|
||||
api_token=os.getenv('OPENAI_API_KEY'), # In case of Ollama just pass "no-token"
|
||||
schema=KnowledgeGraph.model_json_schema(),
|
||||
extraction_type="schema",
|
||||
instruction="""Extract entities and relationships from the given text."""
|
||||
)
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
url = "https://paulgraham.com/love.html"
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
bypass_cache=True,
|
||||
extraction_strategy=extraction_strategy,
|
||||
# magic=True
|
||||
)
|
||||
# print(result.extracted_content)
|
||||
with open(os.path.join(__location__, "kb.json"), "w") as f:
|
||||
f.write(result.extracted_content)
|
||||
|
||||
async def fit_markdown_remove_overlay():
|
||||
async with AsyncWebCrawler(headless = False) as crawler:
|
||||
url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
bypass_cache=True,
|
||||
word_count_threshold = 10,
|
||||
remove_overlay_elements=True,
|
||||
screenshot = True
|
||||
)
|
||||
# Save markdown to file
|
||||
with open(os.path.join(__location__, "mexico_places.md"), "w") as f:
|
||||
f.write(result.fit_markdown)
|
||||
|
||||
print("Done")
|
||||
|
||||
|
||||
async def main():
|
||||
await simple_crawl()
|
||||
await simple_example_with_running_js_code()
|
||||
@@ -455,7 +519,7 @@ async def main():
|
||||
# LLM extraction examples
|
||||
await extract_structured_data_using_llm()
|
||||
await extract_structured_data_using_llm("huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct", os.getenv("HUGGINGFACE_API_KEY"))
|
||||
await extract_structured_data_using_llm("openai/gpt-4", os.getenv("OPENAI_API_KEY"))
|
||||
await extract_structured_data_using_llm("openai/gpt-4o", os.getenv("OPENAI_API_KEY"))
|
||||
await extract_structured_data_using_llm("ollama/llama3.2")
|
||||
|
||||
# You always can pass custom headers to the extraction strategy
|
||||
|
||||
735
docs/examples/quickstart_v0.ipynb
Normal file
735
docs/examples/quickstart_v0.ipynb
Normal file
@@ -0,0 +1,735 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "6yLvrXn7yZQI"
|
||||
},
|
||||
"source": [
|
||||
"# Crawl4AI: Advanced Web Crawling and Data Extraction\n",
|
||||
"\n",
|
||||
"Welcome to this interactive notebook showcasing Crawl4AI, an advanced asynchronous web crawling and data extraction library.\n",
|
||||
"\n",
|
||||
"- GitHub Repository: [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
|
||||
"- Twitter: [@unclecode](https://twitter.com/unclecode)\n",
|
||||
"- Website: [https://crawl4ai.com](https://crawl4ai.com)\n",
|
||||
"\n",
|
||||
"Let's explore the powerful features of Crawl4AI!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "KIn_9nxFyZQK"
|
||||
},
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"First, let's install Crawl4AI from GitHub:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "mSnaxLf3zMog"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!sudo apt-get update && sudo apt-get install -y libwoff1 libopus0 libwebp6 libwebpdemux2 libenchant1c2a libgudev-1.0-0 libsecret-1-0 libhyphen0 libgdk-pixbuf2.0-0 libegl1 libnotify4 libxslt1.1 libevent-2.1-7 libgles2 libvpx6 libxcomposite1 libatk1.0-0 libatk-bridge2.0-0 libepoxy0 libgtk-3-0 libharfbuzz-icu0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "xlXqaRtayZQK"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install crawl4ai\n",
|
||||
"!pip install nest-asyncio\n",
|
||||
"!playwright install"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "qKCE7TI7yZQL"
|
||||
},
|
||||
"source": [
|
||||
"Now, let's import the necessary libraries:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"id": "I67tr7aAyZQL"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import asyncio\n",
|
||||
"import nest_asyncio\n",
|
||||
"from crawl4ai import AsyncWebCrawler\n",
|
||||
"from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy\n",
|
||||
"import json\n",
|
||||
"import time\n",
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "h7yR_Rt_yZQM"
|
||||
},
|
||||
"source": [
|
||||
"## Basic Usage\n",
|
||||
"\n",
|
||||
"Let's start with a simple crawl example:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "yBh6hf4WyZQM",
|
||||
"outputId": "0f83af5c-abba-4175-ed95-70b7512e6bcc"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[LOG] 🌤️ Warming up the AsyncWebCrawler\n",
|
||||
"[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
|
||||
"[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.05 seconds\n",
|
||||
"[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.05 seconds.\n",
|
||||
"18102\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"async def simple_crawl():\n",
|
||||
" async with AsyncWebCrawler(verbose=True) as crawler:\n",
|
||||
" result = await crawler.arun(url=\"https://www.nbcnews.com/business\")\n",
|
||||
" print(len(result.markdown))\n",
|
||||
"await simple_crawl()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "9rtkgHI28uI4"
|
||||
},
|
||||
"source": [
|
||||
"💡 By default, **Crawl4AI** caches the result of every URL, so the next time you call it, you’ll get an instant result. But if you want to bypass the cache, just set `bypass_cache=True`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "MzZ0zlJ9yZQM"
|
||||
},
|
||||
"source": [
|
||||
"## Advanced Features\n",
|
||||
"\n",
|
||||
"### Executing JavaScript and Using CSS Selectors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "gHStF86xyZQM",
|
||||
"outputId": "34d0fb6d-4dec-4677-f76e-85a1f082829b"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[LOG] 🌤️ Warming up the AsyncWebCrawler\n",
|
||||
"[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
|
||||
"[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
|
||||
"[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
|
||||
"[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 6.06 seconds\n",
|
||||
"[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.10 seconds\n",
|
||||
"[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
|
||||
"[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.11 seconds.\n",
|
||||
"41135\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"async def js_and_css():\n",
|
||||
" async with AsyncWebCrawler(verbose=True) as crawler:\n",
|
||||
" js_code = [\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"]\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://www.nbcnews.com/business\",\n",
|
||||
" js_code=js_code,\n",
|
||||
" # css_selector=\"YOUR_CSS_SELECTOR_HERE\",\n",
|
||||
" bypass_cache=True\n",
|
||||
" )\n",
|
||||
" print(len(result.markdown))\n",
|
||||
"\n",
|
||||
"await js_and_css()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "cqE_W4coyZQM"
|
||||
},
|
||||
"source": [
|
||||
"### Using a Proxy\n",
|
||||
"\n",
|
||||
"Note: You'll need to replace the proxy URL with a working proxy for this example to run successfully."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "QjAyiAGqyZQM"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"async def use_proxy():\n",
|
||||
" async with AsyncWebCrawler(verbose=True, proxy=\"http://your-proxy-url:port\") as crawler:\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://www.nbcnews.com/business\",\n",
|
||||
" bypass_cache=True\n",
|
||||
" )\n",
|
||||
" print(result.markdown[:500]) # Print first 500 characters\n",
|
||||
"\n",
|
||||
"# Uncomment the following line to run the proxy example\n",
|
||||
"# await use_proxy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XTZ88lbayZQN"
|
||||
},
|
||||
"source": [
|
||||
"### Extracting Structured Data with OpenAI\n",
|
||||
"\n",
|
||||
"Note: You'll need to set your OpenAI API key as an environment variable for this example to work."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "fIOlDayYyZQN",
|
||||
"outputId": "cb8359cc-dee0-4762-9698-5dfdcee055b8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[LOG] 🌤️ Warming up the AsyncWebCrawler\n",
|
||||
"[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
|
||||
"[LOG] 🕸️ Crawling https://openai.com/api/pricing/ using AsyncPlaywrightCrawlerStrategy...\n",
|
||||
"[LOG] ✅ Crawled https://openai.com/api/pricing/ successfully!\n",
|
||||
"[LOG] 🚀 Crawling done for https://openai.com/api/pricing/, success: True, time taken: 3.77 seconds\n",
|
||||
"[LOG] 🚀 Content extracted for https://openai.com/api/pricing/, success: True, time taken: 0.21 seconds\n",
|
||||
"[LOG] 🔥 Extracting semantic blocks for https://openai.com/api/pricing/, Strategy: AsyncWebCrawler\n",
|
||||
"[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 0\n",
|
||||
"[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 1\n",
|
||||
"[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 2\n",
|
||||
"[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 3\n",
|
||||
"[LOG] Extracted 4 blocks from URL: https://openai.com/api/pricing/ block index: 3\n",
|
||||
"[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 4\n",
|
||||
"[LOG] Extracted 5 blocks from URL: https://openai.com/api/pricing/ block index: 0\n",
|
||||
"[LOG] Extracted 1 blocks from URL: https://openai.com/api/pricing/ block index: 4\n",
|
||||
"[LOG] Extracted 8 blocks from URL: https://openai.com/api/pricing/ block index: 1\n",
|
||||
"[LOG] Extracted 12 blocks from URL: https://openai.com/api/pricing/ block index: 2\n",
|
||||
"[LOG] 🚀 Extraction done for https://openai.com/api/pricing/, time taken: 8.55 seconds.\n",
|
||||
"5029\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from google.colab import userdata\n",
|
||||
"os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')\n",
|
||||
"\n",
|
||||
"class OpenAIModelFee(BaseModel):\n",
|
||||
" model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n",
|
||||
" input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n",
|
||||
" output_fee: str = Field(..., description=\"Fee for output token for the OpenAI model.\")\n",
|
||||
"\n",
|
||||
"async def extract_openai_fees():\n",
|
||||
" async with AsyncWebCrawler(verbose=True) as crawler:\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url='https://openai.com/api/pricing/',\n",
|
||||
" word_count_threshold=1,\n",
|
||||
" extraction_strategy=LLMExtractionStrategy(\n",
|
||||
" provider=\"openai/gpt-4o\", api_token=os.getenv('OPENAI_API_KEY'),\n",
|
||||
" schema=OpenAIModelFee.schema(),\n",
|
||||
" extraction_type=\"schema\",\n",
|
||||
" instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens.\n",
|
||||
" Do not miss any models in the entire content. One extracted model JSON format should look like this:\n",
|
||||
" {\"model_name\": \"GPT-4\", \"input_fee\": \"US$10.00 / 1M tokens\", \"output_fee\": \"US$30.00 / 1M tokens\"}.\"\"\"\n",
|
||||
" ),\n",
|
||||
" bypass_cache=True,\n",
|
||||
" )\n",
|
||||
" print(len(result.extracted_content))\n",
|
||||
"\n",
|
||||
"# Uncomment the following line to run the OpenAI extraction example\n",
|
||||
"await extract_openai_fees()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "BypA5YxEyZQN"
|
||||
},
|
||||
"source": [
|
||||
"### Advanced Multi-Page Crawling with JavaScript Execution"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tfkcVQ0b7mw-"
|
||||
},
|
||||
"source": [
|
||||
"## Advanced Multi-Page Crawling with JavaScript Execution\n",
|
||||
"\n",
|
||||
"This example demonstrates Crawl4AI's ability to handle complex crawling scenarios, specifically extracting commits from multiple pages of a GitHub repository. The challenge here is that clicking the \"Next\" button doesn't load a new page, but instead uses asynchronous JavaScript to update the content. This is a common hurdle in modern web crawling.\n",
|
||||
"\n",
|
||||
"To overcome this, we use Crawl4AI's custom JavaScript execution to simulate clicking the \"Next\" button, and implement a custom hook to detect when new data has loaded. Our strategy involves comparing the first commit's text before and after \"clicking\" Next, waiting until it changes to confirm new data has rendered. This showcases Crawl4AI's flexibility in handling dynamic content and its ability to implement custom logic for even the most challenging crawling tasks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "qUBKGpn3yZQN",
|
||||
"outputId": "3e555b6a-ed33-42f4-cce9-499a923fbe17"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[LOG] 🌤️ Warming up the AsyncWebCrawler\n",
|
||||
"[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
|
||||
"[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
|
||||
"[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
|
||||
"[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 5.16 seconds\n",
|
||||
"[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.28 seconds\n",
|
||||
"[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
|
||||
"[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.28 seconds.\n",
|
||||
"Page 1: Found 35 commits\n",
|
||||
"[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
|
||||
"[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
|
||||
"[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.78 seconds\n",
|
||||
"[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.90 seconds\n",
|
||||
"[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
|
||||
"[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.90 seconds.\n",
|
||||
"Page 2: Found 35 commits\n",
|
||||
"[LOG] 🕸️ Crawling https://github.com/microsoft/TypeScript/commits/main using AsyncPlaywrightCrawlerStrategy...\n",
|
||||
"[LOG] ✅ Crawled https://github.com/microsoft/TypeScript/commits/main successfully!\n",
|
||||
"[LOG] 🚀 Crawling done for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 2.00 seconds\n",
|
||||
"[LOG] 🚀 Content extracted for https://github.com/microsoft/TypeScript/commits/main, success: True, time taken: 0.74 seconds\n",
|
||||
"[LOG] 🔥 Extracting semantic blocks for https://github.com/microsoft/TypeScript/commits/main, Strategy: AsyncWebCrawler\n",
|
||||
"[LOG] 🚀 Extraction done for https://github.com/microsoft/TypeScript/commits/main, time taken: 0.75 seconds.\n",
|
||||
"Page 3: Found 35 commits\n",
|
||||
"Successfully crawled 105 commits across 3 pages\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import re\n",
|
||||
"from bs4 import BeautifulSoup\n",
|
||||
"\n",
|
||||
"async def crawl_typescript_commits():\n",
|
||||
" first_commit = \"\"\n",
|
||||
" async def on_execution_started(page):\n",
|
||||
" nonlocal first_commit\n",
|
||||
" try:\n",
|
||||
" while True:\n",
|
||||
" await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4')\n",
|
||||
" commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4')\n",
|
||||
" commit = await commit.evaluate('(element) => element.textContent')\n",
|
||||
" commit = re.sub(r'\\s+', '', commit)\n",
|
||||
" if commit and commit != first_commit:\n",
|
||||
" first_commit = commit\n",
|
||||
" break\n",
|
||||
" await asyncio.sleep(0.5)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\"Warning: New content didn't appear after JavaScript execution: {e}\")\n",
|
||||
"\n",
|
||||
" async with AsyncWebCrawler(verbose=True) as crawler:\n",
|
||||
" crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started)\n",
|
||||
"\n",
|
||||
" url = \"https://github.com/microsoft/TypeScript/commits/main\"\n",
|
||||
" session_id = \"typescript_commits_session\"\n",
|
||||
" all_commits = []\n",
|
||||
"\n",
|
||||
" js_next_page = \"\"\"\n",
|
||||
" const button = document.querySelector('a[data-testid=\"pagination-next-button\"]');\n",
|
||||
" if (button) button.click();\n",
|
||||
" \"\"\"\n",
|
||||
"\n",
|
||||
" for page in range(3): # Crawl 3 pages\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=url,\n",
|
||||
" session_id=session_id,\n",
|
||||
" css_selector=\"li.Box-sc-g0xbh4-0\",\n",
|
||||
" js=js_next_page if page > 0 else None,\n",
|
||||
" bypass_cache=True,\n",
|
||||
" js_only=page > 0\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" assert result.success, f\"Failed to crawl page {page + 1}\"\n",
|
||||
"\n",
|
||||
" soup = BeautifulSoup(result.cleaned_html, 'html.parser')\n",
|
||||
" commits = soup.select(\"li\")\n",
|
||||
" all_commits.extend(commits)\n",
|
||||
"\n",
|
||||
" print(f\"Page {page + 1}: Found {len(commits)} commits\")\n",
|
||||
"\n",
|
||||
" await crawler.crawler_strategy.kill_session(session_id)\n",
|
||||
" print(f\"Successfully crawled {len(all_commits)} commits across 3 pages\")\n",
|
||||
"\n",
|
||||
"await crawl_typescript_commits()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "EJRnYsp6yZQN"
|
||||
},
|
||||
"source": [
|
||||
"### Using JsonCssExtractionStrategy for Fast Structured Output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "1ZMqIzB_8SYp"
|
||||
},
|
||||
"source": [
|
||||
"The JsonCssExtractionStrategy is a powerful feature of Crawl4AI that allows for precise, structured data extraction from web pages. Here's how it works:\n",
|
||||
"\n",
|
||||
"1. You define a schema that describes the pattern of data you're interested in extracting.\n",
|
||||
"2. The schema includes a base selector that identifies repeating elements on the page.\n",
|
||||
"3. Within the schema, you define fields, each with its own selector and type.\n",
|
||||
"4. These field selectors are applied within the context of each base selector element.\n",
|
||||
"5. The strategy supports nested structures, lists within lists, and various data types.\n",
|
||||
"6. You can even include computed fields for more complex data manipulation.\n",
|
||||
"\n",
|
||||
"This approach allows for highly flexible and precise data extraction, transforming semi-structured web content into clean, structured JSON data. It's particularly useful for extracting consistent data patterns from pages like product listings, news articles, or search results.\n",
|
||||
"\n",
|
||||
"For more details and advanced usage, check out the full documentation on the Crawl4AI website."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "trCMR2T9yZQN",
|
||||
"outputId": "718d36f4-cccf-40f4-8d8c-c3ba73524d16"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[LOG] 🌤️ Warming up the AsyncWebCrawler\n",
|
||||
"[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
|
||||
"[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
|
||||
"[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
|
||||
"[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 7.00 seconds\n",
|
||||
"[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.32 seconds\n",
|
||||
"[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
|
||||
"[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.48 seconds.\n",
|
||||
"Successfully extracted 11 news teasers\n",
|
||||
"{\n",
|
||||
" \"category\": \"Business News\",\n",
|
||||
" \"headline\": \"NBC ripped up its Olympics playbook for 2024 \\u2014 so far, the new strategy paid off\",\n",
|
||||
" \"summary\": \"The Olympics have long been key to NBCUniversal. Paris marked the 18th Olympic Games broadcast by NBC in the U.S.\",\n",
|
||||
" \"time\": \"13h ago\",\n",
|
||||
" \"image\": {\n",
|
||||
" \"src\": \"https://media-cldnry.s-nbcnews.com/image/upload/t_focal-200x100,f_auto,q_auto:best/rockcms/2024-09/240903-nbc-olympics-ch-1344-c7a486.jpg\",\n",
|
||||
" \"alt\": \"Mike Tirico.\"\n",
|
||||
" },\n",
|
||||
" \"link\": \"https://www.nbcnews.com/business\"\n",
|
||||
"}\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"async def extract_news_teasers():\n",
|
||||
" schema = {\n",
|
||||
" \"name\": \"News Teaser Extractor\",\n",
|
||||
" \"baseSelector\": \".wide-tease-item__wrapper\",\n",
|
||||
" \"fields\": [\n",
|
||||
" {\n",
|
||||
" \"name\": \"category\",\n",
|
||||
" \"selector\": \".unibrow span[data-testid='unibrow-text']\",\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"name\": \"headline\",\n",
|
||||
" \"selector\": \".wide-tease-item__headline\",\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"name\": \"summary\",\n",
|
||||
" \"selector\": \".wide-tease-item__description\",\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"name\": \"time\",\n",
|
||||
" \"selector\": \"[data-testid='wide-tease-date']\",\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"name\": \"image\",\n",
|
||||
" \"type\": \"nested\",\n",
|
||||
" \"selector\": \"picture.teasePicture img\",\n",
|
||||
" \"fields\": [\n",
|
||||
" {\"name\": \"src\", \"type\": \"attribute\", \"attribute\": \"src\"},\n",
|
||||
" {\"name\": \"alt\", \"type\": \"attribute\", \"attribute\": \"alt\"},\n",
|
||||
" ],\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"name\": \"link\",\n",
|
||||
" \"selector\": \"a[href]\",\n",
|
||||
" \"type\": \"attribute\",\n",
|
||||
" \"attribute\": \"href\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n",
|
||||
"\n",
|
||||
" async with AsyncWebCrawler(verbose=True) as crawler:\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://www.nbcnews.com/business\",\n",
|
||||
" extraction_strategy=extraction_strategy,\n",
|
||||
" bypass_cache=True,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" assert result.success, \"Failed to crawl the page\"\n",
|
||||
"\n",
|
||||
" news_teasers = json.loads(result.extracted_content)\n",
|
||||
" print(f\"Successfully extracted {len(news_teasers)} news teasers\")\n",
|
||||
" print(json.dumps(news_teasers[0], indent=2))\n",
|
||||
"\n",
|
||||
"await extract_news_teasers()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "FnyVhJaByZQN"
|
||||
},
|
||||
"source": [
|
||||
"## Speed Comparison\n",
|
||||
"\n",
|
||||
"Let's compare the speed of Crawl4AI with Firecrawl, a paid service. Note that we can't run Firecrawl in this Colab environment, so we'll simulate its performance based on previously recorded data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "agDD186f3wig"
|
||||
},
|
||||
"source": [
|
||||
"💡 **Note on Speed Comparison:**\n",
|
||||
"\n",
|
||||
"The speed test conducted here is running on Google Colab, where the internet speed and performance can vary and may not reflect optimal conditions. When we call Firecrawl's API, we're seeing its best performance, while Crawl4AI's performance is limited by Colab's network speed.\n",
|
||||
"\n",
|
||||
"For a more accurate comparison, it's recommended to run these tests on your own servers or computers with a stable and fast internet connection. Despite these limitations, Crawl4AI still demonstrates faster performance in this environment.\n",
|
||||
"\n",
|
||||
"If you run these tests locally, you may observe an even more significant speed advantage for Crawl4AI compared to other services."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "F7KwHv8G1LbY"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install firecrawl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "91813zILyZQN",
|
||||
"outputId": "663223db-ab89-4976-b233-05ceca62b19b"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Firecrawl (simulated):\n",
|
||||
"Time taken: 4.38 seconds\n",
|
||||
"Content length: 41967 characters\n",
|
||||
"Images found: 49\n",
|
||||
"\n",
|
||||
"Crawl4AI (simple crawl):\n",
|
||||
"Time taken: 4.22 seconds\n",
|
||||
"Content length: 18221 characters\n",
|
||||
"Images found: 49\n",
|
||||
"\n",
|
||||
"Crawl4AI (with JavaScript execution):\n",
|
||||
"Time taken: 9.13 seconds\n",
|
||||
"Content length: 34243 characters\n",
|
||||
"Images found: 89\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from google.colab import userdata\n",
|
||||
"os.environ['FIRECRAWL_API_KEY'] = userdata.get('FIRECRAWL_API_KEY')\n",
|
||||
"import time\n",
|
||||
"from firecrawl import FirecrawlApp\n",
|
||||
"\n",
|
||||
"async def speed_comparison():\n",
|
||||
" # Simulated Firecrawl performance\n",
|
||||
" app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])\n",
|
||||
" start = time.time()\n",
|
||||
" scrape_status = app.scrape_url(\n",
|
||||
" 'https://www.nbcnews.com/business',\n",
|
||||
" params={'formats': ['markdown', 'html']}\n",
|
||||
" )\n",
|
||||
" end = time.time()\n",
|
||||
" print(\"Firecrawl (simulated):\")\n",
|
||||
" print(f\"Time taken: {end - start:.2f} seconds\")\n",
|
||||
" print(f\"Content length: {len(scrape_status['markdown'])} characters\")\n",
|
||||
" print(f\"Images found: {scrape_status['markdown'].count('cldnry.s-nbcnews.com')}\")\n",
|
||||
" print()\n",
|
||||
"\n",
|
||||
" async with AsyncWebCrawler() as crawler:\n",
|
||||
" # Crawl4AI simple crawl\n",
|
||||
" start = time.time()\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://www.nbcnews.com/business\",\n",
|
||||
" word_count_threshold=0,\n",
|
||||
" bypass_cache=True,\n",
|
||||
" verbose=False\n",
|
||||
" )\n",
|
||||
" end = time.time()\n",
|
||||
" print(\"Crawl4AI (simple crawl):\")\n",
|
||||
" print(f\"Time taken: {end - start:.2f} seconds\")\n",
|
||||
" print(f\"Content length: {len(result.markdown)} characters\")\n",
|
||||
" print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
|
||||
" print()\n",
|
||||
"\n",
|
||||
" # Crawl4AI with JavaScript execution\n",
|
||||
" start = time.time()\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://www.nbcnews.com/business\",\n",
|
||||
" js_code=[\"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"],\n",
|
||||
" word_count_threshold=0,\n",
|
||||
" bypass_cache=True,\n",
|
||||
" verbose=False\n",
|
||||
" )\n",
|
||||
" end = time.time()\n",
|
||||
" print(\"Crawl4AI (with JavaScript execution):\")\n",
|
||||
" print(f\"Time taken: {end - start:.2f} seconds\")\n",
|
||||
" print(f\"Content length: {len(result.markdown)} characters\")\n",
|
||||
" print(f\"Images found: {result.markdown.count('cldnry.s-nbcnews.com')}\")\n",
|
||||
"\n",
|
||||
"await speed_comparison()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "OBFFYVJIyZQN"
|
||||
},
|
||||
"source": [
|
||||
"If you run on a local machine with a proper internet speed:\n",
|
||||
"- Simple crawl: Crawl4AI is typically over 3-4 times faster than Firecrawl.\n",
|
||||
"- With JavaScript execution: Even when executing JavaScript to load more content (potentially doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.\n",
|
||||
"\n",
|
||||
"Please note that actual performance may vary depending on network conditions and the specific content being crawled."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "A6_1RK1_yZQO"
|
||||
},
|
||||
"source": [
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"In this notebook, we've explored the powerful features of Crawl4AI, including:\n",
|
||||
"\n",
|
||||
"1. Basic crawling\n",
|
||||
"2. JavaScript execution and CSS selector usage\n",
|
||||
"3. Proxy support\n",
|
||||
"4. Structured data extraction with OpenAI\n",
|
||||
"5. Advanced multi-page crawling with JavaScript execution\n",
|
||||
"6. Fast structured output using JsonCssExtractionStrategy\n",
|
||||
"7. Speed comparison with other services\n",
|
||||
"\n",
|
||||
"Crawl4AI offers a fast, flexible, and powerful solution for web crawling and data extraction tasks. Its asynchronous architecture and advanced features make it suitable for a wide range of applications, from simple web scraping to complex, multi-page data extraction scenarios.\n",
|
||||
"\n",
|
||||
"For more information and advanced usage, please visit the [Crawl4AI documentation](https://crawl4ai.com/mkdocs/).\n",
|
||||
"\n",
|
||||
"Happy crawling!"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": []
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "venv",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.13"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
@@ -1,10 +0,0 @@
|
||||
{
|
||||
"NoExtractionStrategy": "### NoExtractionStrategy\n\n`NoExtractionStrategy` is a basic extraction strategy that returns the entire HTML content without any modification. It is useful for cases where no specific extraction is required. Only clean html, and amrkdown.\n\n#### Constructor Parameters:\nNone.\n\n#### Example usage:\n```python\nextractor = NoExtractionStrategy()\nextracted_content = extractor.extract(url, html)\n```",
|
||||
|
||||
"LLMExtractionStrategy": "### LLMExtractionStrategy\n\n`LLMExtractionStrategy` uses a Language Model (LLM) to extract meaningful blocks or chunks from the given HTML content. This strategy leverages an external provider for language model completions.\n\n#### Constructor Parameters:\n- `provider` (str, optional): The provider to use for the language model completions. Default is `DEFAULT_PROVIDER` (e.g., openai/gpt-4).\n- `api_token` (str, optional): The API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.\n- `instruction` (str, optional): An instruction to guide the LLM on how to perform the extraction. This allows users to specify the type of data they are interested in or set the tone of the response. Default is `None`.\n\n#### Example usage:\n```python\nextractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')\nextracted_content = extractor.extract(url, html)\n```\n\nBy providing clear instructions, users can tailor the extraction process to their specific needs, enhancing the relevance and utility of the extracted content.",
|
||||
|
||||
"CosineStrategy": "### CosineStrategy\n\n`CosineStrategy` uses hierarchical clustering based on cosine similarity to extract clusters of text from the given HTML content. This strategy is suitable for identifying related content sections.\n\n#### Constructor Parameters:\n- `semantic_filter` (str, optional): A string containing keywords for filtering relevant documents before clustering. If provided, documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.\n- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.\n- `max_dist` (float, optional): The maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.\n- `linkage_method` (str, optional): The linkage method for hierarchical clustering. Default is `'ward'`.\n- `top_k` (int, optional): Number of top categories to extract. Default is `3`.\n- `model_name` (str, optional): The model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.\n\n#### Example usage:\n```python\nextractor = CosineStrategy(semantic_filter='artificial intelligence', word_count_threshold=10, max_dist=0.2, linkage_method='ward', top_k=3, model_name='BAAI/bge-small-en-v1.5')\nextracted_content = extractor.extract(url, html)\n```\n\n#### Cosine Similarity Filtering\n\nWhen a `semantic_filter` is provided, the `CosineStrategy` applies an embedding-based filtering process to select relevant documents before performing hierarchical clustering.",
|
||||
|
||||
"TopicExtractionStrategy": "### TopicExtractionStrategy\n\n`TopicExtractionStrategy` uses the TextTiling algorithm to segment the HTML content into topics and extracts keywords for each segment. This strategy is useful for identifying and summarizing thematic content.\n\n#### Constructor Parameters:\n- `num_keywords` (int, optional): Number of keywords to represent each topic segment. Default is `3`.\n\n#### Example usage:\n```python\nextractor = TopicExtractionStrategy(num_keywords=3)\nextracted_content = extractor.extract(url, html)\n```"
|
||||
}
|
||||
|
||||
@@ -1,141 +0,0 @@
|
||||
# Core Classes and Functions
|
||||
|
||||
## Overview
|
||||
|
||||
In this section, we will delve into the core classes and functions that make up the Crawl4AI library. This includes the `WebCrawler` class, various `CrawlerStrategy` classes, `ChunkingStrategy` classes, and `ExtractionStrategy` classes. Understanding these core components will help you leverage the full power of Crawl4AI for your web crawling and data extraction needs.
|
||||
|
||||
## WebCrawler Class
|
||||
|
||||
The `WebCrawler` class is the main class you'll interact with. It provides the interface for crawling web pages and extracting data.
|
||||
|
||||
### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create an instance of WebCrawler
|
||||
crawler = WebCrawler()
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
- **`warmup()`**: Prepares the crawler for use, such as loading necessary models.
|
||||
- **`run(url: str, **kwargs)`**: Runs the crawler on the specified URL with optional parameters for customization.
|
||||
|
||||
```python
|
||||
crawler.warmup()
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
print(result)
|
||||
```
|
||||
|
||||
## CrawlerStrategy Classes
|
||||
|
||||
The `CrawlerStrategy` classes define how the web crawling is executed. The base class is `CrawlerStrategy`, which is extended by specific implementations like `LocalSeleniumCrawlerStrategy`.
|
||||
|
||||
### CrawlerStrategy Base Class
|
||||
|
||||
An abstract base class that defines the interface for different crawler strategies.
|
||||
|
||||
```python
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
class CrawlerStrategy(ABC):
|
||||
@abstractmethod
|
||||
def crawl(self, url: str, **kwargs) -> str:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def take_screenshot(self, save_path: str):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def update_user_agent(self, user_agent: str):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def set_hook(self, hook_type: str, hook: Callable):
|
||||
pass
|
||||
```
|
||||
|
||||
### LocalSeleniumCrawlerStrategy Class
|
||||
|
||||
A concrete implementation of `CrawlerStrategy` that uses Selenium to crawl web pages.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
|
||||
|
||||
strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
|
||||
- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
|
||||
- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
|
||||
- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
|
||||
|
||||
```python
|
||||
result = strategy.crawl("https://www.example.com")
|
||||
strategy.take_screenshot("screenshot.png")
|
||||
strategy.update_user_agent("Mozilla/5.0")
|
||||
strategy.set_hook("before_get_url", lambda: print("About to get URL"))
|
||||
```
|
||||
|
||||
## ChunkingStrategy Classes
|
||||
|
||||
The `ChunkingStrategy` classes define how the text from a web page is divided into chunks. Here are a few examples:
|
||||
|
||||
### RegexChunking Class
|
||||
|
||||
Splits text using regular expressions.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
chunker = RegexChunking(patterns=[r'\n\n'])
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
|
||||
```
|
||||
|
||||
### NlpSentenceChunking Class
|
||||
|
||||
Uses NLP to split text into sentences.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
chunker = NlpSentenceChunking()
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
|
||||
```
|
||||
|
||||
## ExtractionStrategy Classes
|
||||
|
||||
The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks. Here are a few examples:
|
||||
|
||||
### CosineStrategy Class
|
||||
|
||||
Clusters text chunks based on cosine similarity.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### LLMExtractionStrategy Class
|
||||
|
||||
Uses a Language Model to extract meaningful blocks from HTML.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
By understanding these core classes and functions, you can customize and extend Crawl4AI to suit your specific web crawling and data extraction needs. Happy crawling! 🕷️🤖
|
||||
|
||||
@@ -1,338 +0,0 @@
|
||||
# Detailed API Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This section provides comprehensive documentation for the Crawl4AI API, covering all classes, methods, and their parameters. This guide will help you understand how to utilize the API to its full potential, enabling efficient web crawling and data extraction.
|
||||
|
||||
## WebCrawler Class
|
||||
|
||||
The `WebCrawler` class is the primary interface for crawling web pages and extracting data.
|
||||
|
||||
### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
#### `warmup()`
|
||||
|
||||
Prepares the crawler for use, such as loading necessary models.
|
||||
|
||||
```python
|
||||
crawler.warmup()
|
||||
```
|
||||
|
||||
#### `run(url: str, **kwargs) -> CrawlResult`
|
||||
|
||||
Crawls the specified URL and returns the result.
|
||||
|
||||
- **Parameters:**
|
||||
- `url` (str): The URL to crawl.
|
||||
- `**kwargs`: Additional parameters for customization.
|
||||
|
||||
- **Returns:**
|
||||
- `CrawlResult`: An object containing the crawl result.
|
||||
|
||||
- **Example:**
|
||||
|
||||
```python
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
print(result)
|
||||
```
|
||||
|
||||
### CrawlResult Class
|
||||
|
||||
Represents the result of a crawl operation.
|
||||
|
||||
- **Attributes:**
|
||||
- `url` (str): The URL of the crawled page.
|
||||
- `html` (str): The raw HTML of the page.
|
||||
- `success` (bool): Whether the crawl was successful.
|
||||
- `cleaned_html` (Optional[str]): The cleaned HTML.
|
||||
- `media` (Dict[str, List[Dict]]): Media tags in the page (images, audio, video).
|
||||
- `links` (Dict[str, List[Dict]]): Links in the page (external, internal).
|
||||
- `screenshot` (Optional[str]): Base64 encoded screenshot.
|
||||
- `markdown` (Optional[str]): Extracted content in Markdown format.
|
||||
- `extracted_content` (Optional[str]): Extracted meaningful content.
|
||||
- `metadata` (Optional[dict]): Metadata from the page.
|
||||
- `error_message` (Optional[str]): Error message if any.
|
||||
|
||||
## CrawlerStrategy Classes
|
||||
|
||||
The `CrawlerStrategy` classes define how the web crawling is executed.
|
||||
|
||||
### CrawlerStrategy Base Class
|
||||
|
||||
An abstract base class for different crawler strategies.
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`crawl(url: str, **kwargs) -> str`**: Crawls the specified URL.
|
||||
- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
|
||||
- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
|
||||
- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
|
||||
|
||||
### LocalSeleniumCrawlerStrategy Class
|
||||
|
||||
Uses Selenium to crawl web pages.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
|
||||
|
||||
strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
|
||||
- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
|
||||
- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
|
||||
- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
result = strategy.crawl("https://www.example.com")
|
||||
strategy.take_screenshot("screenshot.png")
|
||||
strategy.update_user_agent("Mozilla/5.0")
|
||||
strategy.set_hook("before_get_url", lambda: print("About to get URL"))
|
||||
```
|
||||
|
||||
## ChunkingStrategy Classes
|
||||
|
||||
The `ChunkingStrategy` classes define how the text from a web page is divided into chunks.
|
||||
|
||||
### RegexChunking Class
|
||||
|
||||
Splits text using regular expressions.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
chunker = RegexChunking(patterns=[r'\n\n'])
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into chunks.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
|
||||
```
|
||||
|
||||
### NlpSentenceChunking Class
|
||||
|
||||
Uses NLP to split text into sentences.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
chunker = NlpSentenceChunking()
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into sentences.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
|
||||
```
|
||||
|
||||
### TopicSegmentationChunking Class
|
||||
|
||||
Uses the TextTiling algorithm to segment text into topics.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import TopicSegmentationChunking
|
||||
|
||||
chunker = TopicSegmentationChunking(num_keywords=3)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into topic-based segments.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into topic-based segments.")
|
||||
```
|
||||
|
||||
### FixedLengthWordChunking Class
|
||||
|
||||
Splits text into chunks of fixed length based on the number of words.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import FixedLengthWordChunking
|
||||
|
||||
chunker = FixedLengthWordChunking(chunk_size=100)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into fixed-length word chunks.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into fixed-length word chunks.")
|
||||
```
|
||||
|
||||
### SlidingWindowChunking Class
|
||||
|
||||
Uses a sliding window approach to chunk text.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import SlidingWindowChunking
|
||||
|
||||
chunker = SlidingWindowChunking(window_size=100, step=50)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text using a sliding window approach.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split using a sliding window approach.")
|
||||
```
|
||||
|
||||
## ExtractionStrategy Classes
|
||||
|
||||
The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks.
|
||||
|
||||
### NoExtractionStrategy Class
|
||||
|
||||
Returns the entire HTML content without any modification.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import NoExtractionStrategy
|
||||
|
||||
extractor = NoExtractionStrategy()
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Returns the HTML content.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### LLMExtractionStrategy Class
|
||||
|
||||
Uses a Language Model to extract meaningful blocks from HTML.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Extracts meaningful content using the LLM.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### CosineStrategy Class
|
||||
|
||||
Clusters text chunks based on cosine similarity.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Extracts clusters of text based on cosine similarity.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### TopicExtractionStrategy Class
|
||||
|
||||
Uses the TextTiling algorithm to segment HTML content into topics and extract keywords.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import TopicExtractionStrategy
|
||||
|
||||
extractor = TopicExtractionStrategy(num_keywords=3)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Extracts topic-based segments and keywords.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
Here are the common parameters used across various classes and methods:
|
||||
|
||||
- **`url`** (str): The URL to crawl.
|
||||
- **`html`** (str): The HTML content of the page.
|
||||
- **`user_agent`** (str): The user agent for the HTTP requests.
|
||||
- **`patterns`** (list): A list of regular expression patterns for chunking.
|
||||
- **`num_keywords`** (int): Number of keywords for topic extraction.
|
||||
- **`chunk_size`** (int): Number of words in each chunk.
|
||||
- **`window_size`** (int): Number of words in the sliding window.
|
||||
- **`step`** (int): Step size for the sliding window.
|
||||
- **`semantic_filter`** (str): Keywords for filtering relevant documents.
|
||||
- **`word_count_threshold`** (int): Minimum number of words per cluster.
|
||||
- **`max_dist`** (float): Maximum cophenetic distance for clustering.
|
||||
- **`linkage_method`** (str): Linkage method for hierarchical clustering.
|
||||
- **`top_k`** (int): Number of top categories to extract.
|
||||
- **`provider`** (
|
||||
|
||||
str): Provider for language model completions.
|
||||
- **`api_token`** (str): API token for the provider.
|
||||
- **`instruction`** (str): Instruction to guide the LLM extraction.
|
||||
|
||||
## Conclusion
|
||||
|
||||
This detailed API documentation provides a thorough understanding of the classes, methods, and parameters in the Crawl4AI library. With this knowledge, you can effectively use the API to perform advanced web crawling and data extraction tasks.
|
||||
@@ -1,102 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
## [v0.2.77] - 2024-08-04
|
||||
|
||||
Significant improvements in text processing and performance:
|
||||
|
||||
- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
|
||||
- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
|
||||
- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
|
||||
- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
|
||||
|
||||
These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
|
||||
|
||||
## [v0.2.76] - 2024-08-02
|
||||
|
||||
Major improvements in functionality, performance, and cross-platform compatibility! 🚀
|
||||
|
||||
- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
|
||||
- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
|
||||
- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
|
||||
- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
|
||||
- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
|
||||
|
||||
A big shoutout to our amazing community contributors:
|
||||
- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
|
||||
- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
|
||||
- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
|
||||
|
||||
Your contributions are driving Crawl4AI forward! 🙌
|
||||
|
||||
## [v0.2.75] - 2024-07-19
|
||||
|
||||
Minor improvements for a more maintainable codebase:
|
||||
|
||||
- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
|
||||
- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
|
||||
|
||||
These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
|
||||
|
||||
|
||||
## v0.2.74 - 2024-07-08
|
||||
A slew of exciting updates to improve the crawler's stability and robustness! 🎉
|
||||
|
||||
- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
|
||||
- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
|
||||
- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
|
||||
- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
|
||||
|
||||
## [v0.2.73] - 2024-07-03
|
||||
|
||||
💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
|
||||
|
||||
* Supporting website need "with-head" mode to crawl the website with head.
|
||||
* Fixing the installation issues for setup.py and dockerfile.
|
||||
* Resolve multiple issues.
|
||||
|
||||
## [v0.2.72] - 2024-06-30
|
||||
|
||||
This release brings exciting updates and improvements to our project! 🎉
|
||||
|
||||
* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
|
||||
* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
|
||||
* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
|
||||
* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
|
||||
|
||||
These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
|
||||
|
||||
## [0.2.71] - 2024-06-26
|
||||
|
||||
**Improved Error Handling and Performance** 🚧
|
||||
|
||||
* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
|
||||
* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
|
||||
* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
|
||||
* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
|
||||
|
||||
These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
|
||||
|
||||
## [0.2.71] - 2024-06-25
|
||||
### Fixed
|
||||
- Speed up twice the extraction function.
|
||||
|
||||
## [0.2.6] - 2024-06-22
|
||||
### Fixed
|
||||
- Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
|
||||
|
||||
## [0.2.5] - 2024-06-18
|
||||
### Added
|
||||
- Added five important hooks to the crawler:
|
||||
- on_driver_created: Called when the driver is ready for initializations.
|
||||
- before_get_url: Called right before Selenium fetches the URL.
|
||||
- after_get_url: Called after Selenium fetches the URL.
|
||||
- before_return_html: Called when the data is parsed and ready.
|
||||
- on_user_agent_updated: Called when the user changes the user_agent, causing the driver to reinitialize.
|
||||
- Added an example in `quickstart.py` in the example folder under the docs.
|
||||
- Enhancement issue #24: Replaced inline HTML tags (e.g., DEL, INS, SUB, ABBR) with textual format for better context handling in LLM.
|
||||
- Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
|
||||
- Updated Dockerfile to ensure compatibility across multiple platforms (Hopefully!).
|
||||
|
||||
## [0.2.4] - 2024-06-17
|
||||
### Fixed
|
||||
- Fix issue #22: Use MD5 hash for caching HTML files to handle long URLs
|
||||
@@ -1,25 +0,0 @@
|
||||
# Contact
|
||||
If you have any questions, suggestions, or feedback, please feel free to reach out to us:
|
||||
|
||||
- GitHub: [unclecode](https://github.com/unclecode)
|
||||
- Twitter: [@unclecode](https://twitter.com/unclecode)
|
||||
- Website: [crawl4ai.com](https://crawl4ai.com)
|
||||
|
||||
|
||||
## Contributing 🤝
|
||||
|
||||
We welcome contributions from the open-source community to help improve Crawl4AI and make it even more valuable for AI enthusiasts and developers. To contribute, please follow these steps:
|
||||
|
||||
1. Fork the repository.
|
||||
2. Create a new branch for your feature or bug fix.
|
||||
3. Make your changes and commit them with descriptive messages.
|
||||
4. Push your changes to your forked repository.
|
||||
5. Submit a pull request to the main repository.
|
||||
|
||||
For more information on contributing, please see our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md).
|
||||
|
||||
## License 📄
|
||||
|
||||
Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
|
||||
|
||||
Let's work together to make the web more accessible and useful for AI applications! 💪🌐🤖
|
||||
@@ -1,231 +0,0 @@
|
||||
# Interactive Demo for Crowler
|
||||
<div id="demo">
|
||||
<form id="crawlForm" class="terminal-form">
|
||||
<fieldset>
|
||||
<legend>Enter URL and Options</legend>
|
||||
<div class="form-group">
|
||||
<label for="url">Enter URL:</label>
|
||||
<input type="text" id="url" name="url" required>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="screenshot">Get Screenshot:</label>
|
||||
<input type="checkbox" id="screenshot" name="screenshot">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<button class="btn btn-default" type="submit">Submit</button>
|
||||
</div>
|
||||
|
||||
</fieldset>
|
||||
</form>
|
||||
|
||||
<div id="loading" class="loading-message">
|
||||
<div class="terminal-alert terminal-alert-primary">Loading... Please wait.</div>
|
||||
</div>
|
||||
|
||||
<section id="response" class="response-section">
|
||||
<h2>Response</h2>
|
||||
<div class="tabs">
|
||||
<ul class="tab-list">
|
||||
<li class="tab-item" onclick="showTab('markdown')">Markdown</li>
|
||||
<li class="tab-item" onclick="showTab('cleanedHtml')">Cleaned HTML</li>
|
||||
<li class="tab-item" onclick="showTab('media')">Media</li>
|
||||
<li class="tab-item" onclick="showTab('extractedContent')">Extracted Content</li>
|
||||
<li class="tab-item" onclick="showTab('screenshot')">Screenshot</li>
|
||||
<li class="tab-item" onclick="showTab('pythonCode')">Python Code</li>
|
||||
</ul>
|
||||
<div class="tab-content" id="tab-markdown">
|
||||
<header>
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('markdownContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('markdownContent', 'markdown.md')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="markdownContent" class="language-markdown hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-cleanedHtml" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('cleanedHtmlContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('cleanedHtmlContent', 'cleaned.html')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="cleanedHtmlContent" class="language-html hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-media" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('mediaContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('mediaContent', 'media.json')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="mediaContent" class="language-json hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-extractedContent" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('extractedContentContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('extractedContentContent', 'extracted_content.json')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="extractedContentContent" class="language-json hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-screenshot" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadImage('screenshotContent', 'screenshot.png')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><img id="screenshotContent" /></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-pythonCode" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('pythonCode')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('pythonCode', 'example.py')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="pythonCode" class="language-python hljs"></code></pre>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<div id="error" class="error-message" style="display: none; margin-top:1em;">
|
||||
<div class="terminal-alert terminal-alert-error"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
function showTab(tabId) {
|
||||
const tabs = document.querySelectorAll('.tab-content');
|
||||
tabs.forEach(tab => tab.style.display = 'none');
|
||||
document.getElementById(`tab-${tabId}`).style.display = 'block';
|
||||
}
|
||||
|
||||
function redo(codeBlock, codeText){
|
||||
codeBlock.classList.remove('hljs');
|
||||
codeBlock.removeAttribute('data-highlighted');
|
||||
|
||||
// Set new code and re-highlight
|
||||
codeBlock.textContent = codeText;
|
||||
hljs.highlightBlock(codeBlock);
|
||||
}
|
||||
|
||||
function copyToClipboard(elementId) {
|
||||
const content = document.getElementById(elementId).textContent;
|
||||
navigator.clipboard.writeText(content).then(() => {
|
||||
alert('Copied to clipboard');
|
||||
});
|
||||
}
|
||||
|
||||
function downloadContent(elementId, filename) {
|
||||
const content = document.getElementById(elementId).textContent;
|
||||
const blob = new Blob([content], { type: 'text/plain' });
|
||||
const url = window.URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.style.display = 'none';
|
||||
a.href = url;
|
||||
a.download = filename;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
window.URL.revokeObjectURL(url);
|
||||
document.body.removeChild(a);
|
||||
}
|
||||
|
||||
function downloadImage(elementId, filename) {
|
||||
const content = document.getElementById(elementId).src;
|
||||
const a = document.createElement('a');
|
||||
a.style.display = 'none';
|
||||
a.href = content;
|
||||
a.download = filename;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
}
|
||||
|
||||
document.getElementById('crawlForm').addEventListener('submit', function(event) {
|
||||
event.preventDefault();
|
||||
document.getElementById('loading').style.display = 'block';
|
||||
document.getElementById('response').style.display = 'none';
|
||||
|
||||
const url = document.getElementById('url').value;
|
||||
const screenshot = document.getElementById('screenshot').checked;
|
||||
const data = {
|
||||
urls: [url],
|
||||
bypass_cache: false,
|
||||
word_count_threshold: 5,
|
||||
screenshot: screenshot
|
||||
};
|
||||
|
||||
fetch('/crawl', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify(data)
|
||||
})
|
||||
.then(response => {
|
||||
if (!response.ok) {
|
||||
if (response.status === 429) {
|
||||
return response.json().then(err => {
|
||||
throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
|
||||
});
|
||||
}
|
||||
throw new Error('Network response was not ok');
|
||||
}
|
||||
return response.json();
|
||||
})
|
||||
.then(data => {
|
||||
data = data.results[0]; // Only one URL is requested
|
||||
document.getElementById('loading').style.display = 'none';
|
||||
document.getElementById('response').style.display = 'block';
|
||||
redo(document.getElementById('markdownContent'), data.markdown);
|
||||
redo(document.getElementById('cleanedHtmlContent'), data.cleaned_html);
|
||||
redo(document.getElementById('mediaContent'), JSON.stringify(data.media, null, 2));
|
||||
redo(document.getElementById('extractedContentContent'), data.extracted_content);
|
||||
if (screenshot) {
|
||||
document.getElementById('screenshotContent').src = `data:image/png;base64,${data.screenshot}`;
|
||||
}
|
||||
const pythonCode = `
|
||||
from crawl4ai.web_crawler import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
result = crawler.run(
|
||||
url='${url}',
|
||||
screenshot=${screenshot}
|
||||
)
|
||||
print(result)
|
||||
`;
|
||||
redo(document.getElementById('pythonCode'), pythonCode);
|
||||
document.getElementById('error').style.display = 'none';
|
||||
})
|
||||
.catch(error => {
|
||||
document.getElementById('loading').style.display = 'none';
|
||||
document.getElementById('error').style.display = 'block';
|
||||
let errorMessage = 'An unexpected error occurred. Please try again later.';
|
||||
|
||||
if (error.status === 429) {
|
||||
const details = error.details;
|
||||
if (details.retry_after) {
|
||||
errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
|
||||
} else if (details.reset_at) {
|
||||
const resetTime = new Date(details.reset_at);
|
||||
const waitTime = Math.ceil((resetTime - new Date()) / 1000);
|
||||
errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
|
||||
} else {
|
||||
errorMessage = `Rate limit exceeded. Please try again later.`;
|
||||
}
|
||||
} else if (error.message) {
|
||||
errorMessage = error.message;
|
||||
}
|
||||
|
||||
document.querySelector('#error .terminal-alert').textContent = errorMessage;
|
||||
});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
@@ -1,100 +0,0 @@
|
||||
# Hooks & Auth
|
||||
|
||||
Crawl4AI allows you to customize the behavior of the web crawler using hooks. Hooks are functions that are called at specific points in the crawling process, allowing you to modify the crawler's behavior or perform additional actions. This example demonstrates how to use various hooks to customize the crawling process.
|
||||
|
||||
## Example: Using Crawler Hooks
|
||||
|
||||
Let's see how we can customize the crawler using hooks! In this example, we'll:
|
||||
|
||||
1. Maximize the browser window and log in to a website when the driver is created.
|
||||
2. Add a custom header before fetching the URL.
|
||||
3. Log the current URL after fetching it.
|
||||
4. Log the length of the HTML before returning it.
|
||||
|
||||
### Hook Definitions
|
||||
|
||||
```python
|
||||
from crawl4ai.web_crawler import WebCrawler
|
||||
from crawl4ai.crawler_strategy import *
|
||||
|
||||
def on_driver_created(driver):
|
||||
print("[HOOK] on_driver_created")
|
||||
# Example customization: maximize the window
|
||||
driver.maximize_window()
|
||||
|
||||
# Example customization: logging in to a hypothetical website
|
||||
driver.get('https://example.com/login')
|
||||
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
|
||||
WebDriverWait(driver, 10).until(
|
||||
EC.presence_of_element_located((By.NAME, 'username'))
|
||||
)
|
||||
driver.find_element(By.NAME, 'username').send_keys('testuser')
|
||||
driver.find_element(By.NAME, 'password').send_keys('password123')
|
||||
driver.find_element(By.NAME, 'login').click()
|
||||
WebDriverWait(driver, 10).until(
|
||||
EC.presence_of_element_located((By.ID, 'welcome'))
|
||||
)
|
||||
# Add a custom cookie
|
||||
driver.add_cookie({'name': 'test_cookie', 'value': 'cookie_value'})
|
||||
return driver
|
||||
|
||||
|
||||
def before_get_url(driver):
|
||||
print("[HOOK] before_get_url")
|
||||
# Example customization: add a custom header
|
||||
# Enable Network domain for sending headers
|
||||
driver.execute_cdp_cmd('Network.enable', {})
|
||||
# Add a custom header
|
||||
driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
|
||||
return driver
|
||||
|
||||
def after_get_url(driver):
|
||||
print("[HOOK] after_get_url")
|
||||
# Example customization: log the URL
|
||||
print(driver.current_url)
|
||||
return driver
|
||||
|
||||
def before_return_html(driver, html):
|
||||
print("[HOOK] before_return_html")
|
||||
# Example customization: log the HTML
|
||||
print(len(html))
|
||||
return driver
|
||||
```
|
||||
|
||||
### Using the Hooks with the WebCrawler
|
||||
|
||||
```python
|
||||
print("\n🔗 [bold cyan]Using Crawler Hooks: Let's see how we can customize the crawler using hooks![/bold cyan]", True)
|
||||
crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
|
||||
crawler_strategy.set_hook('on_driver_created', on_driver_created)
|
||||
crawler_strategy.set_hook('before_get_url', before_get_url)
|
||||
crawler_strategy.set_hook('after_get_url', after_get_url)
|
||||
crawler_strategy.set_hook('before_return_html', before_return_html)
|
||||
crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
|
||||
crawler.warmup()
|
||||
|
||||
result = crawler.run(url="https://example.com")
|
||||
|
||||
print("[LOG] 📦 [bold yellow]Crawler Hooks result:[/bold yellow]")
|
||||
print(result)
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
- `on_driver_created`: This hook is called when the Selenium driver is created. In this example, it maximizes the window, logs in to a website, and adds a custom cookie.
|
||||
- `before_get_url`: This hook is called right before Selenium fetches the URL. In this example, it adds a custom HTTP header.
|
||||
- `after_get_url`: This hook is called after Selenium fetches the URL. In this example, it logs the current URL.
|
||||
- `before_return_html`: This hook is called before returning the HTML content. In this example, it logs the length of the HTML content.
|
||||
|
||||
### Additional Ideas
|
||||
|
||||
- **Add custom headers to requests**: You can add custom headers to the requests using the `before_get_url` hook.
|
||||
- **Perform safety checks**: Use the hooks to perform safety checks before the crawling process starts.
|
||||
- **Modify the HTML content**: Use the `before_return_html` hook to modify the HTML content before it is returned.
|
||||
- **Log additional information**: Use the hooks to log additional information for debugging or monitoring purposes.
|
||||
|
||||
By using these hooks, you can customize the behavior of the crawler to suit your specific needs.
|
||||
@@ -1,29 +0,0 @@
|
||||
# Examples
|
||||
|
||||
Welcome to the examples section of Crawl4AI documentation! In this section, you will find practical examples demonstrating how to use Crawl4AI for various web crawling and data extraction tasks. Each example is designed to showcase different features and capabilities of the library.
|
||||
|
||||
## Examples Index
|
||||
|
||||
### [LLM Extraction](llm_extraction.md)
|
||||
|
||||
This example demonstrates how to use Crawl4AI to extract information using Large Language Models (LLMs). You will learn how to configure the `LLMExtractionStrategy` to get structured data from web pages.
|
||||
|
||||
### [JS Execution & CSS Filtering](js_execution_css_filtering.md)
|
||||
|
||||
Learn how to execute custom JavaScript code and filter data using CSS selectors. This example shows how to perform complex web interactions and extract specific content from web pages.
|
||||
|
||||
### [Hooks & Auth](hooks_auth.md)
|
||||
|
||||
This example covers the use of custom hooks for authentication and other pre-crawling tasks. You will see how to set up hooks to modify headers, authenticate sessions, and perform other preparatory actions before crawling.
|
||||
|
||||
### [Summarization](summarization.md)
|
||||
|
||||
Discover how to use Crawl4AI to summarize web page content. This example demonstrates the summarization capabilities of the library, helping you extract concise information from lengthy web pages.
|
||||
|
||||
### [Research Assistant](research_assistant.md)
|
||||
|
||||
In this example, Crawl4AI is used as a research assistant to gather and organize information from multiple sources. You will learn how to use various extraction and chunking strategies to compile a comprehensive report.
|
||||
|
||||
---
|
||||
|
||||
Each example includes detailed explanations and code snippets to help you understand and implement the features in your projects. Click on the links to explore each example and start making the most of Crawl4AI!
|
||||
@@ -1,44 +0,0 @@
|
||||
# JS Execution & CSS Filtering
|
||||
|
||||
In this example, we'll demonstrate how to use Crawl4AI to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.
|
||||
|
||||
## Example: Extracting Structured Data
|
||||
|
||||
```python
|
||||
# Import necessary modules
|
||||
from crawl4ai import WebCrawler
|
||||
from crawl4ai.chunking_strategy import *
|
||||
from crawl4ai.extraction_strategy import *
|
||||
from crawl4ai.crawler_strategy import *
|
||||
|
||||
# Define the JavaScript code to click the "Load More" button
|
||||
js_code = ["""
|
||||
const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
|
||||
loadMoreButton && loadMoreButton.click();
|
||||
"""]
|
||||
|
||||
crawler = WebCrawler(verbose=True)
|
||||
crawler.warmup()
|
||||
# Run the crawler with keyword filtering and CSS selector
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
js=js_code,
|
||||
css_selector="p",
|
||||
extraction_strategy=CosineStrategy(
|
||||
semantic_filter="technology",
|
||||
),
|
||||
)
|
||||
|
||||
# Display the extracted result
|
||||
print(result)
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
1. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button. This is useful for loading additional content dynamically.
|
||||
2. **CSS Selector**: The `css_selector="p"` parameter ensures that only paragraph (`<p>`) tags are extracted from the web page.
|
||||
3. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
This example demonstrates the power and flexibility of Crawl4AI in handling complex web interactions and extracting meaningful data. You can customize the JavaScript code, CSS selectors, and extraction strategies to suit your specific requirements.
|
||||
@@ -1,90 +0,0 @@
|
||||
# LLM Extraction
|
||||
|
||||
Crawl4AI allows you to use Language Models (LLMs) to extract structured data or relevant content from web pages. Below are two examples demonstrating how to use LLMExtractionStrategy for different purposes.
|
||||
|
||||
## Example 1: Extract Structured Data
|
||||
|
||||
In this example, we use the `LLMExtractionStrategy` to extract structured data (model names and their fees) from the OpenAI pricing page.
|
||||
|
||||
```python
|
||||
import os
|
||||
import time
|
||||
from crawl4ai.web_crawler import WebCrawler
|
||||
from crawl4ai.chunking_strategy import *
|
||||
from crawl4ai.extraction_strategy import *
|
||||
from crawl4ai.crawler_strategy import *
|
||||
|
||||
url = r'https://openai.com/api/pricing/'
|
||||
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
class OpenAIModelFee(BaseModel):
|
||||
model_name: str = Field(..., description="Name of the OpenAI model.")
|
||||
input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
|
||||
output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
|
||||
|
||||
result = crawler.run(
|
||||
url=url,
|
||||
word_count_threshold=1,
|
||||
extraction_strategy= LLMExtractionStrategy(
|
||||
provider= "openai/gpt-4o", api_token = os.getenv('OPENAI_API_KEY'),
|
||||
schema=OpenAIModelFee.model_json_schema(),
|
||||
extraction_type="schema",
|
||||
instruction="From the crawled content, extract all mentioned model names along with their "\
|
||||
"fees for input and output tokens. Make sure not to miss anything in the entire content. "\
|
||||
'One extracted model JSON format should look like this: '\
|
||||
'{ "model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens" }'
|
||||
),
|
||||
bypass_cache=True,
|
||||
)
|
||||
|
||||
model_fees = json.loads(result.extracted_content)
|
||||
|
||||
print(len(model_fees))
|
||||
|
||||
with open(".data/data.json", "w", encoding="utf-8") as f:
|
||||
f.write(result.extracted_content)
|
||||
```
|
||||
|
||||
## Example 2: Extract Relevant Content
|
||||
|
||||
In this example, we instruct the LLM to extract only content related to technology from the NBC News business page.
|
||||
|
||||
```python
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
instruction="Extract only content related to technology"
|
||||
),
|
||||
bypass_cache=True,
|
||||
)
|
||||
|
||||
model_fees = json.loads(result.extracted_content)
|
||||
|
||||
print(len(model_fees))
|
||||
|
||||
with open(".data/data.json", "w", encoding="utf-8") as f:
|
||||
f.write(result.extracted_content)
|
||||
```
|
||||
|
||||
## Customizing LLM Provider
|
||||
|
||||
Under the hood, Crawl4AI uses the `litellm` library, which allows you to use any LLM provider you want. Just pass the correct model name and API token.
|
||||
|
||||
```python
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="your_llm_provider/model_name",
|
||||
api_token="your_api_token",
|
||||
instruction="Your extraction instruction"
|
||||
)
|
||||
```
|
||||
|
||||
This flexibility allows you to integrate with various LLM providers and tailor the extraction process to your specific needs.
|
||||
@@ -1,248 +0,0 @@
|
||||
## Research Assistant Example
|
||||
|
||||
This example demonstrates how to build a research assistant using `Chainlit` and `Crawl4AI`. The assistant will be capable of crawling web pages for information and answering questions based on the crawled content. Additionally, it integrates speech-to-text functionality for audio inputs.
|
||||
|
||||
### Step-by-Step Guide
|
||||
|
||||
1. **Install Required Packages**
|
||||
|
||||
Ensure you have the necessary packages installed. You need `chainlit`, `groq`, `requests`, and `openai`.
|
||||
|
||||
```bash
|
||||
pip install chainlit groq requests openai
|
||||
```
|
||||
|
||||
2. **Import Libraries**
|
||||
|
||||
Import all the necessary modules and initialize the OpenAI client.
|
||||
|
||||
```python
|
||||
import os
|
||||
import time
|
||||
from openai import AsyncOpenAI
|
||||
import chainlit as cl
|
||||
import re
|
||||
import requests
|
||||
from io import BytesIO
|
||||
from chainlit.element import ElementBased
|
||||
from groq import Groq
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
client = AsyncOpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.getenv("GROQ_API_KEY"))
|
||||
|
||||
# Instrument the OpenAI client
|
||||
cl.instrument_openai()
|
||||
```
|
||||
|
||||
3. **Set Configuration**
|
||||
|
||||
Define the model settings for the assistant.
|
||||
|
||||
```python
|
||||
settings = {
|
||||
"model": "llama3-8b-8192",
|
||||
"temperature": 0.5,
|
||||
"max_tokens": 500,
|
||||
"top_p": 1,
|
||||
"frequency_penalty": 0,
|
||||
"presence_penalty": 0,
|
||||
}
|
||||
```
|
||||
|
||||
4. **Define Utility Functions**
|
||||
|
||||
- **Extract URLs from Text**: Use regex to find URLs in messages.
|
||||
|
||||
```python
|
||||
def extract_urls(text):
|
||||
url_pattern = re.compile(r'(https?://\S+)')
|
||||
return url_pattern.findall(text)
|
||||
```
|
||||
|
||||
- **Crawl URL**: Send a request to `Crawl4AI` to fetch the content of a URL.
|
||||
|
||||
```python
|
||||
def crawl_url(url):
|
||||
data = {
|
||||
"urls": [url],
|
||||
"include_raw_html": True,
|
||||
"word_count_threshold": 10,
|
||||
"extraction_strategy": "NoExtractionStrategy",
|
||||
"chunking_strategy": "RegexChunking"
|
||||
}
|
||||
response = requests.post("https://crawl4ai.com/crawl", json=data)
|
||||
response_data = response.json()
|
||||
response_data = response_data['results'][0]
|
||||
return response_data['markdown']
|
||||
```
|
||||
|
||||
5. **Initialize Chat Start Event**
|
||||
|
||||
Set up the initial chat message and user session.
|
||||
|
||||
```python
|
||||
@cl.on_chat_start
|
||||
async def on_chat_start():
|
||||
cl.user_session.set("session", {
|
||||
"history": [],
|
||||
"context": {}
|
||||
})
|
||||
await cl.Message(
|
||||
content="Welcome to the chat! How can I assist you today?"
|
||||
).send()
|
||||
```
|
||||
|
||||
6. **Handle Incoming Messages**
|
||||
|
||||
Process user messages, extract URLs, and crawl them concurrently. Update the chat history and system message.
|
||||
|
||||
```python
|
||||
@cl.on_message
|
||||
async def on_message(message: cl.Message):
|
||||
user_session = cl.user_session.get("session")
|
||||
|
||||
# Extract URLs from the user's message
|
||||
urls = extract_urls(message.content)
|
||||
|
||||
futures = []
|
||||
with ThreadPoolExecutor() as executor:
|
||||
for url in urls:
|
||||
futures.append(executor.submit(crawl_url, url))
|
||||
|
||||
results = [future.result() for future in futures]
|
||||
|
||||
for url, result in zip(urls, results):
|
||||
ref_number = f"REF_{len(user_session['context']) + 1}"
|
||||
user_session["context"][ref_number] = {
|
||||
"url": url,
|
||||
"content": result
|
||||
}
|
||||
|
||||
user_session["history"].append({
|
||||
"role": "user",
|
||||
"content": message.content
|
||||
})
|
||||
|
||||
# Create a system message that includes the context
|
||||
context_messages = [
|
||||
f'<appendix ref="{ref}">\n{data["content"]}\n</appendix>'
|
||||
for ref, data in user_session["context"].items()
|
||||
]
|
||||
if context_messages:
|
||||
system_message = {
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful bot. Use the following context for answering questions. "
|
||||
"Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
|
||||
"If the question requires any information from the provided appendices or context, refer to the sources. "
|
||||
"If not, there is no need to add a references section. "
|
||||
"At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
|
||||
"\n\n".join(context_messages)
|
||||
)
|
||||
}
|
||||
else:
|
||||
system_message = {
|
||||
"role": "system",
|
||||
"content": "You are a helpful assistant."
|
||||
}
|
||||
|
||||
msg = cl.Message(content="")
|
||||
await msg.send()
|
||||
|
||||
# Get response from the LLM
|
||||
stream = await client.chat.completions.create(
|
||||
messages=[
|
||||
system_message,
|
||||
*user_session["history"]
|
||||
],
|
||||
stream=True,
|
||||
**settings
|
||||
)
|
||||
|
||||
assistant_response = ""
|
||||
async for part in stream:
|
||||
if token := part.choices[0].delta.content:
|
||||
assistant_response += token
|
||||
await msg.stream_token(token)
|
||||
|
||||
# Add assistant message to the history
|
||||
user_session["history"].append({
|
||||
"role": "assistant",
|
||||
"content": assistant_response
|
||||
})
|
||||
await msg.update()
|
||||
|
||||
# Append the reference section to the assistant's response
|
||||
reference_section = "\n\nReferences:\n"
|
||||
for ref, data in user_session["context"].items():
|
||||
reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
|
||||
|
||||
msg.content += reference_section
|
||||
await msg.update()
|
||||
```
|
||||
|
||||
7. **Handle Audio Input**
|
||||
|
||||
Capture and transcribe audio input. Store the audio buffer and transcribe it when the audio ends.
|
||||
|
||||
```python
|
||||
@cl.on_audio_chunk
|
||||
async def on_audio_chunk(chunk: cl.AudioChunk):
|
||||
if chunk.isStart:
|
||||
buffer = BytesIO()
|
||||
buffer.name = f"input_audio.{chunk.mimeType.split('/')[1]}"
|
||||
cl.user_session.set("audio_buffer", buffer)
|
||||
cl.user_session.set("audio_mime_type", chunk.mimeType)
|
||||
|
||||
cl.user_session.get("audio_buffer").write(chunk.data)
|
||||
|
||||
@cl.step(type="tool")
|
||||
async def speech_to_text(audio_file):
|
||||
cli = Groq()
|
||||
response = await client.audio.transcriptions.create(
|
||||
model="whisper-large-v3", file=audio_file
|
||||
)
|
||||
return response.text
|
||||
|
||||
@cl.on_audio_end
|
||||
async def on_audio_end(elements: list[ElementBased]):
|
||||
audio_buffer: BytesIO = cl.user_session.get("audio_buffer")
|
||||
audio_buffer.seek(0)
|
||||
audio_file = audio_buffer.read()
|
||||
audio_mime_type: str = cl.user_session.get("audio_mime_type")
|
||||
|
||||
start_time = time.time()
|
||||
transcription = await speech_to_text((audio_buffer.name, audio_file, audio_mime_type))
|
||||
end_time = time.time()
|
||||
print(f"Transcription took {end_time - start_time} seconds")
|
||||
|
||||
user_msg = cl.Message(
|
||||
author="You",
|
||||
type="user_message",
|
||||
content=transcription
|
||||
)
|
||||
await user_msg.send()
|
||||
await on_message(user_msg)
|
||||
```
|
||||
|
||||
8. **Run the Chat Application**
|
||||
|
||||
Start the Chainlit application.
|
||||
|
||||
```python
|
||||
if __name__ == "__main__":
|
||||
from chainlit.cli import run_chainlit
|
||||
run_chainlit(__file__)
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
- **Libraries and Configuration**: Import necessary libraries and configure the OpenAI client.
|
||||
- **Utility Functions**: Define functions to extract URLs and crawl them.
|
||||
- **Chat Start Event**: Initialize chat session and welcome message.
|
||||
- **Message Handling**: Extract URLs, crawl them concurrently, and update chat history and context.
|
||||
- **Audio Handling**: Capture, buffer, and transcribe audio input, then process the transcription as text.
|
||||
- **Running the Application**: Start the Chainlit server to interact with the assistant.
|
||||
|
||||
This example showcases how to create an interactive research assistant that can fetch, process, and summarize web content, along with handling audio inputs for a seamless user experience.
|
||||
@@ -1,108 +0,0 @@
|
||||
## Summarization Example
|
||||
|
||||
This example demonstrates how to use `Crawl4AI` to extract a summary from a web page. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.
|
||||
|
||||
### Step-by-Step Guide
|
||||
|
||||
1. **Import Necessary Modules**
|
||||
|
||||
First, import the necessary modules and classes.
|
||||
|
||||
```python
|
||||
import os
|
||||
import time
|
||||
import json
|
||||
from crawl4ai.web_crawler import WebCrawler
|
||||
from crawl4ai.chunking_strategy import *
|
||||
from crawl4ai.extraction_strategy import *
|
||||
from crawl4ai.crawler_strategy import *
|
||||
from pydantic import BaseModel, Field
|
||||
```
|
||||
|
||||
2. **Define the URL to be Crawled**
|
||||
|
||||
Set the URL of the web page you want to summarize.
|
||||
|
||||
```python
|
||||
url = r'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
|
||||
```
|
||||
|
||||
3. **Initialize the WebCrawler**
|
||||
|
||||
Create an instance of the `WebCrawler` and call the `warmup` method.
|
||||
|
||||
```python
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
```
|
||||
|
||||
4. **Define the Data Model**
|
||||
|
||||
Use Pydantic to define the structure of the extracted data.
|
||||
|
||||
```python
|
||||
class PageSummary(BaseModel):
|
||||
title: str = Field(..., description="Title of the page.")
|
||||
summary: str = Field(..., description="Summary of the page.")
|
||||
brief_summary: str = Field(..., description="Brief summary of the page.")
|
||||
keywords: list = Field(..., description="Keywords assigned to the page.")
|
||||
```
|
||||
|
||||
5. **Run the Crawler**
|
||||
|
||||
Set up and run the crawler with the `LLMExtractionStrategy`. Provide the necessary parameters, including the schema for the extracted data and the instruction for the LLM.
|
||||
|
||||
```python
|
||||
result = crawler.run(
|
||||
url=url,
|
||||
word_count_threshold=1,
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
schema=PageSummary.model_json_schema(),
|
||||
extraction_type="schema",
|
||||
apply_chunking=False,
|
||||
instruction=(
|
||||
"From the crawled content, extract the following details: "
|
||||
"1. Title of the page "
|
||||
"2. Summary of the page, which is a detailed summary "
|
||||
"3. Brief summary of the page, which is a paragraph text "
|
||||
"4. Keywords assigned to the page, which is a list of keywords. "
|
||||
'The extracted JSON format should look like this: '
|
||||
'{ "title": "Page Title", "summary": "Detailed summary of the page.", '
|
||||
'"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
|
||||
)
|
||||
),
|
||||
bypass_cache=True,
|
||||
)
|
||||
```
|
||||
|
||||
6. **Process the Extracted Data**
|
||||
|
||||
Load the extracted content into a JSON object and print it.
|
||||
|
||||
```python
|
||||
page_summary = json.loads(result.extracted_content)
|
||||
print(page_summary)
|
||||
```
|
||||
|
||||
7. **Save the Extracted Data**
|
||||
|
||||
Save the extracted data to a file for further use.
|
||||
|
||||
```python
|
||||
with open(".data/page_summary.json", "w", encoding="utf-8") as f:
|
||||
f.write(result.extracted_content)
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
- **Importing Modules**: Import the necessary modules, including `WebCrawler` and `LLMExtractionStrategy` from `Crawl4AI`.
|
||||
- **URL Definition**: Set the URL of the web page you want to crawl and summarize.
|
||||
- **WebCrawler Initialization**: Create an instance of `WebCrawler` and call the `warmup` method to prepare the crawler.
|
||||
- **Data Model Definition**: Define the structure of the data you want to extract using Pydantic's `BaseModel`.
|
||||
- **Crawler Execution**: Run the crawler with the `LLMExtractionStrategy`, providing the schema and detailed instructions for the extraction process.
|
||||
- **Data Processing**: Load the extracted content into a JSON object and print it to verify the results.
|
||||
- **Data Saving**: Save the extracted data to a file for further use.
|
||||
|
||||
This example demonstrates how to harness the power of `Crawl4AI` to perform advanced web crawling and data extraction tasks with minimal code.
|
||||
@@ -1,138 +0,0 @@
|
||||
# Advanced Features
|
||||
|
||||
Crawl4AI offers a range of advanced features that allow you to fine-tune your web crawling and data extraction process. This section will cover some of these advanced features, including taking screenshots, extracting media and links, customizing the user agent, using custom hooks, and leveraging CSS selectors.
|
||||
|
||||
## Taking Screenshots 📸
|
||||
|
||||
One of the cool features of Crawl4AI is the ability to take screenshots of the web pages you're crawling. This can be particularly useful for visual verification or for capturing the state of dynamic content.
|
||||
|
||||
Here's how you can take a screenshot:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
import base64
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler with the screenshot parameter
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
|
||||
|
||||
# Save the screenshot to a file
|
||||
with open("screenshot.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
|
||||
print("Screenshot saved to 'screenshot.png'!")
|
||||
```
|
||||
|
||||
In this example, we create a `WebCrawler` instance, warm it up, and then run it with the `screenshot` parameter set to `True`. The screenshot is saved as a base64 encoded string in the result, which we then decode and save as a PNG file.
|
||||
|
||||
## Extracting Media and Links 🎨🔗
|
||||
|
||||
Crawl4AI can extract all media tags (images, audio, and video) and links (both internal and external) from a web page. This feature is useful for collecting multimedia content or analyzing link structures.
|
||||
|
||||
Here's an example:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
|
||||
print("Extracted media:", result.media)
|
||||
print("Extracted links:", result.links)
|
||||
```
|
||||
|
||||
In this example, the `result` object contains dictionaries for media and links, which you can access and use as needed.
|
||||
|
||||
## Customizing the User Agent 🕵️♂️
|
||||
|
||||
Crawl4AI allows you to set a custom user agent for your HTTP requests. This can help you avoid detection by web servers or simulate different browsing environments.
|
||||
|
||||
Here's how to set a custom user agent:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler with a custom user agent
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
|
||||
|
||||
print("Crawl result:", result)
|
||||
```
|
||||
|
||||
In this example, we specify a custom user agent string when running the crawler.
|
||||
|
||||
## Using Custom Hooks 🪝
|
||||
|
||||
Hooks are a powerful feature in Crawl4AI that allow you to customize the crawling process at various stages. You can define hooks for actions such as driver initialization, before and after URL fetching, and before returning the HTML.
|
||||
|
||||
Here's an example of using hooks:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
|
||||
# Define the hooks
|
||||
def on_driver_created(driver):
|
||||
driver.maximize_window()
|
||||
driver.get('https://example.com/login')
|
||||
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.NAME, 'username'))).send_keys('testuser')
|
||||
driver.find_element(By.NAME, 'password').send_keys('password123')
|
||||
driver.find_element(By.NAME, 'login').click()
|
||||
return driver
|
||||
|
||||
def before_get_url(driver):
|
||||
driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
|
||||
return driver
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Set the hooks
|
||||
crawler.set_hook('on_driver_created', on_driver_created)
|
||||
crawler.set_hook('before_get_url', before_get_url)
|
||||
|
||||
# Run the crawler
|
||||
result = crawler.run(url="https://example.com")
|
||||
|
||||
print("Crawl result:", result)
|
||||
```
|
||||
|
||||
In this example, we define hooks to handle driver initialization and custom headers before fetching the URL.
|
||||
|
||||
## Using CSS Selectors 🎯
|
||||
|
||||
CSS selectors allow you to target specific elements on a web page for extraction. This can be useful for scraping structured content, such as articles or product details.
|
||||
|
||||
Here's an example of using a CSS selector:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler with a CSS selector to extract only H2 tags
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", css_selector="h2")
|
||||
|
||||
print("Extracted H2 tags:", result.extracted_content)
|
||||
```
|
||||
|
||||
In this example, we use the `css_selector` parameter to extract only the H2 tags from the web page.
|
||||
|
||||
---
|
||||
|
||||
With these advanced features, you can leverage Crawl4AI to perform sophisticated web crawling and data extraction tasks. Whether you need to take screenshots, extract specific elements, customize the crawling process, or set custom headers, Crawl4AI provides the flexibility and power to meet your needs. Happy crawling! 🕷️🚀
|
||||
@@ -1,130 +0,0 @@
|
||||
# Crawl Request Parameters
|
||||
|
||||
The `run` function in Crawl4AI is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `run` function, along with their descriptions, possible values, and examples.
|
||||
|
||||
## Parameters
|
||||
|
||||
### url (str)
|
||||
**Description:** The URL of the webpage to crawl.
|
||||
**Required:** Yes
|
||||
**Example:**
|
||||
```python
|
||||
url = "https://www.nbcnews.com/business"
|
||||
```
|
||||
|
||||
### word_count_threshold (int)
|
||||
**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is `5`.
|
||||
**Required:** No
|
||||
**Default Value:** `5`
|
||||
**Example:**
|
||||
```python
|
||||
word_count_threshold = 10
|
||||
```
|
||||
|
||||
### extraction_strategy (ExtractionStrategy)
|
||||
**Description:** The strategy to use for extracting content from the HTML. It must be an instance of `ExtractionStrategy`. If not provided, the default is `NoExtractionStrategy`.
|
||||
**Required:** No
|
||||
**Default Value:** `NoExtractionStrategy()`
|
||||
**Example:**
|
||||
```python
|
||||
extraction_strategy = CosineStrategy(semantic_filter="finance")
|
||||
```
|
||||
|
||||
### chunking_strategy (ChunkingStrategy)
|
||||
**Description:** The strategy to use for chunking the text before processing. It must be an instance of `ChunkingStrategy`. The default value is `RegexChunking()`.
|
||||
**Required:** No
|
||||
**Default Value:** `RegexChunking()`
|
||||
**Example:**
|
||||
```python
|
||||
chunking_strategy = NlpSentenceChunking()
|
||||
```
|
||||
|
||||
### bypass_cache (bool)
|
||||
**Description:** Whether to force a fresh crawl even if the URL has been previously crawled. The default value is `False`.
|
||||
**Required:** No
|
||||
**Default Value:** `False`
|
||||
**Example:**
|
||||
```python
|
||||
bypass_cache = True
|
||||
```
|
||||
|
||||
### css_selector (str)
|
||||
**Description:** The CSS selector to target specific parts of the HTML for extraction. If not provided, the entire HTML will be processed.
|
||||
**Required:** No
|
||||
**Default Value:** `None`
|
||||
**Example:**
|
||||
```python
|
||||
css_selector = "div.article-content"
|
||||
```
|
||||
|
||||
### screenshot (bool)
|
||||
**Description:** Whether to take screenshots of the page. The default value is `False`.
|
||||
**Required:** No
|
||||
**Default Value:** `False`
|
||||
**Example:**
|
||||
```python
|
||||
screenshot = True
|
||||
```
|
||||
|
||||
### user_agent (str)
|
||||
**Description:** The user agent to use for the HTTP requests. If not provided, a default user agent will be used.
|
||||
**Required:** No
|
||||
**Default Value:** `None`
|
||||
**Example:**
|
||||
```python
|
||||
user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
|
||||
```
|
||||
|
||||
### verbose (bool)
|
||||
**Description:** Whether to enable verbose logging. The default value is `True`.
|
||||
**Required:** No
|
||||
**Default Value:** `True`
|
||||
**Example:**
|
||||
```python
|
||||
verbose = True
|
||||
```
|
||||
|
||||
### **kwargs
|
||||
Additional keyword arguments that can be passed to customize the crawling process further. Some notable options include:
|
||||
|
||||
- **only_text (bool):** Whether to extract only text content, excluding HTML tags. Default is `False`.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
css_selector="p",
|
||||
only_text=True
|
||||
)
|
||||
```
|
||||
|
||||
## Example Usage
|
||||
|
||||
Here's an example of how to use the `run` function with various parameters:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
|
||||
# Run the crawler with custom parameters
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
word_count_threshold=10,
|
||||
extraction_strategy=CosineStrategy(semantic_filter="finance"),
|
||||
chunking_strategy=NlpSentenceChunking(),
|
||||
bypass_cache=True,
|
||||
css_selector="div.article-content",
|
||||
screenshot=True,
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
|
||||
verbose=True,
|
||||
only_text=True
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
This example demonstrates how to configure various parameters to customize the crawling and extraction process using Crawl4AI.
|
||||
@@ -1,120 +0,0 @@
|
||||
# Crawl Result
|
||||
|
||||
The `CrawlResult` class is the heart of Crawl4AI's output, encapsulating all the data extracted from a crawling session. This class contains various fields that store the results of the web crawling and extraction process. Let's break down each field and see what it holds. 🎉
|
||||
|
||||
## Class Definition
|
||||
|
||||
```python
|
||||
class CrawlResult(BaseModel):
|
||||
url: str
|
||||
html: str
|
||||
success: bool
|
||||
cleaned_html: Optional[str] = None
|
||||
media: Dict[str, List[Dict]] = {}
|
||||
links: Dict[str, List[Dict]] = {}
|
||||
screenshot: Optional[str] = None
|
||||
markdown: Optional[str] = None
|
||||
extracted_content: Optional[str] = None
|
||||
metadata: Optional[dict] = None
|
||||
error_message: Optional[str] = None
|
||||
```
|
||||
|
||||
## Fields Explanation
|
||||
|
||||
### `url: str`
|
||||
The URL that was crawled. This field simply stores the URL of the web page that was processed.
|
||||
|
||||
### `html: str`
|
||||
The raw HTML content of the web page. This is the unprocessed HTML source as retrieved by the crawler.
|
||||
|
||||
### `success: bool`
|
||||
A flag indicating whether the crawling and extraction were successful. If any error occurs during the process, this will be `False`.
|
||||
|
||||
### `cleaned_html: Optional[str]`
|
||||
The cleaned HTML content of the web page. This field holds the HTML after removing unwanted tags like `<script>`, `<style>`, and others that do not contribute to the useful content.
|
||||
|
||||
### `media: Dict[str, List[Dict]]`
|
||||
A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here’s how they are structured:
|
||||
|
||||
- **Images**: Each image is represented as a dictionary with `src` (source URL) and `alt` (alternate text).
|
||||
- **Videos**: Each video is represented similarly with `src` and `alt`.
|
||||
- **Audios**: Each audio is represented with `src` and `alt`.
|
||||
|
||||
```python
|
||||
media = {
|
||||
'images': [
|
||||
{'src': 'image_url1', 'alt': 'description1', "type": "image"},
|
||||
{'src': 'image_url2', 'alt': 'description2', "type": "image"}
|
||||
],
|
||||
'videos': [
|
||||
{'src': 'video_url1', 'alt': 'description1', "type": "video"}
|
||||
],
|
||||
'audios': [
|
||||
{'src': 'audio_url1', 'alt': 'description1', "type": "audio"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### `links: Dict[str, List[Dict]]`
|
||||
A dictionary containing lists of internal and external links extracted from the web page. Each link is represented as a dictionary with `href` (URL) and `text` (link text).
|
||||
|
||||
- **Internal Links**: Links pointing to the same domain.
|
||||
- **External Links**: Links pointing to different domains.
|
||||
|
||||
```python
|
||||
links = {
|
||||
'internal': [
|
||||
{'href': 'internal_link1', 'text': 'link_text1'},
|
||||
{'href': 'internal_link2', 'text': 'link_text2'}
|
||||
],
|
||||
'external': [
|
||||
{'href': 'external_link1', 'text': 'link_text1'}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### `screenshot: Optional[str]`
|
||||
A base64-encoded screenshot of the web page. This field stores the screenshot data if the crawling was configured to take a screenshot.
|
||||
|
||||
### `markdown: Optional[str]`
|
||||
The content of the web page converted to Markdown format. This is useful for generating clean, readable text that retains the structure of the original HTML.
|
||||
|
||||
### `extracted_content: Optional[str]`
|
||||
The content extracted based on the specified extraction strategy. This field holds the meaningful content blocks extracted from the web page, ready for your AI and data processing needs.
|
||||
|
||||
### `metadata: Optional[dict]`
|
||||
A dictionary containing metadata extracted from the web page, such as title, description, keywords, and other meta tags.
|
||||
|
||||
### `error_message: Optional[str]`
|
||||
If an error occurs during crawling, this field will contain the error message, helping you debug and understand what went wrong. 🚨
|
||||
|
||||
## Example Usage
|
||||
|
||||
Here's a quick example to illustrate how you might use the `CrawlResult` in your code:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
|
||||
# Run the crawler on a URL
|
||||
result = crawler.run(url="https://www.example.com")
|
||||
|
||||
# Check if the crawl was successful
|
||||
if result.success:
|
||||
print("Crawl succeeded!")
|
||||
print("URL:", result.url)
|
||||
print("HTML:", result.html[:100]) # Print the first 100 characters of the HTML
|
||||
print("Cleaned HTML:", result.cleaned_html[:100])
|
||||
print("Media:", result.media)
|
||||
print("Links:", result.links)
|
||||
print("Screenshot:", result.screenshot)
|
||||
print("Markdown:", result.markdown[:100])
|
||||
print("Extracted Content:", result.extracted_content)
|
||||
print("Metadata:", result.metadata)
|
||||
else:
|
||||
print("Crawl failed with error:", result.error_message)
|
||||
```
|
||||
|
||||
With this setup, you can easily access all the valuable data extracted from the web page and integrate it into your applications. Happy crawling! 🕷️🤖
|
||||
@@ -1,116 +0,0 @@
|
||||
## Extraction Strategies 🧠
|
||||
|
||||
Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into two of the most important strategies: `CosineStrategy` and `LLMExtractionStrategy`.
|
||||
|
||||
### CosineStrategy
|
||||
|
||||
`CosineStrategy` uses hierarchical clustering based on cosine similarity to group text chunks into meaningful clusters. This method converts each chunk into its embedding and then clusters them to form semantical chunks.
|
||||
|
||||
#### When to Use
|
||||
- Ideal for fast, accurate semantic segmentation of text.
|
||||
- Perfect for scenarios where LLMs might be overkill or too slow.
|
||||
- Suitable for narrowing down content based on specific queries or keywords.
|
||||
|
||||
#### Parameters
|
||||
- `semantic_filter` (str, optional): Keywords for filtering relevant documents before clustering. Documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.
|
||||
- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.
|
||||
- `max_dist` (float, optional): Maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.
|
||||
- `linkage_method` (str, optional): Linkage method for hierarchical clustering. Default is `'ward'`.
|
||||
- `top_k` (int, optional): Number of top categories to extract. Default is `3`.
|
||||
- `model_name` (str, optional): Model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Define extraction strategy
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="finance economy stock market",
|
||||
word_count_threshold=10,
|
||||
max_dist=0.2,
|
||||
linkage_method='ward',
|
||||
top_k=3,
|
||||
model_name='BAAI/bge-small-en-v1.5'
|
||||
)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = crawler.run(url=url, extraction_strategy=strategy)
|
||||
print(result.extracted_content)
|
||||
```
|
||||
|
||||
### LLMExtractionStrategy
|
||||
|
||||
`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
|
||||
|
||||
#### When to Use
|
||||
- Suitable for complex extraction tasks requiring nuanced understanding.
|
||||
- Ideal for scenarios where detailed instructions can guide the extraction process.
|
||||
- Perfect for extracting specific types of information or content with precise guidelines.
|
||||
|
||||
#### Parameters
|
||||
- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
|
||||
- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
|
||||
- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
|
||||
|
||||
#### Example Without Instructions
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Define extraction strategy without instructions
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider='openai',
|
||||
api_token='your_api_token'
|
||||
)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = crawler.run(url=url, extraction_strategy=strategy)
|
||||
print(result.extracted_content)
|
||||
```
|
||||
|
||||
#### Example With Instructions
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Define extraction strategy with instructions
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider='openai',
|
||||
api_token='your_api_token',
|
||||
instruction="Extract only financial news and summarize key points."
|
||||
)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = crawler.run(url=url, extraction_strategy=strategy)
|
||||
print(result.extracted_content)
|
||||
```
|
||||
|
||||
#### Use Cases for LLMExtractionStrategy
|
||||
- Extracting specific data types from structured or semi-structured content.
|
||||
- Generating summaries, extracting key information, or transforming content into different formats.
|
||||
- Performing detailed extractions based on custom instructions.
|
||||
|
||||
For more detailed examples, please refer to the [Examples section](../examples/index.md) of the documentation.
|
||||
|
||||
---
|
||||
|
||||
By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy` or nuanced, instruction-based extraction with `LLMExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️♂️✨
|
||||
@@ -1,101 +0,0 @@
|
||||
# Crawl4AI v0.2.77
|
||||
|
||||
Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
|
||||
|
||||
|
||||
## Try the [Demo](demo.md)
|
||||
|
||||
Just try it now and crawl different pages to see how it works. You can set the links, see the structures of the output, and also view the Python sample code on how to run it. The old demo is available at [/old_demo](/old) where you can see more details.
|
||||
|
||||
## Introduction
|
||||
|
||||
Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Here's a quick example to show you how easy it is to use Crawl4AI:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create an instance of WebCrawler
|
||||
crawler = WebCrawler()
|
||||
|
||||
# Warm up the crawler (load necessary models)
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler on a URL
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
|
||||
# Print the extracted content
|
||||
print(result.extracted_content)
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
1. **Importing the Library**: We start by importing the `WebCrawler` class from the `crawl4ai` library.
|
||||
2. **Creating an Instance**: An instance of `WebCrawler` is created.
|
||||
3. **Warming Up**: The `warmup()` method prepares the crawler by loading necessary models and settings.
|
||||
4. **Running the Crawler**: The `run()` method is used to crawl the specified URL and extract meaningful content.
|
||||
5. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
This documentation is organized into several sections to help you navigate and find the information you need quickly:
|
||||
|
||||
### [Home](index.md)
|
||||
|
||||
An introduction to Crawl4AI, including a quick start guide and an overview of the documentation structure.
|
||||
|
||||
### [Installation](installation.md)
|
||||
|
||||
Instructions on how to install Crawl4AI and its dependencies.
|
||||
|
||||
### [Introduction](introduction.md)
|
||||
|
||||
A detailed introduction to Crawl4AI, its features, and how it can be used for various web crawling and data extraction tasks.
|
||||
|
||||
### [Quick Start](quickstart.md)
|
||||
|
||||
A step-by-step guide to get you up and running with Crawl4AI, including installation instructions and basic usage examples.
|
||||
|
||||
### [Examples](examples/index.md)
|
||||
|
||||
This section contains practical examples demonstrating different use cases of Crawl4AI:
|
||||
|
||||
- [LLM Extraction](examples/llm_extraction.md)
|
||||
- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
|
||||
- [Hooks & Auth](examples/hooks_auth.md)
|
||||
- [Summarization](examples/summarization.md)
|
||||
- [Research Assistant](examples/research_assistant.md)
|
||||
|
||||
### [Full Details of Using Crawler](full_details/crawl_request_parameters.md)
|
||||
|
||||
Comprehensive details on using the crawler, including:
|
||||
|
||||
- [Crawl Request Parameters](full_details/crawl_request_parameters.md)
|
||||
- [Crawl Result Class](full_details/crawl_result_class.md)
|
||||
- [Advanced Features](full_details/advanced_features.md)
|
||||
- [Chunking Strategies](full_details/chunking_strategies.md)
|
||||
- [Extraction Strategies](full_details/extraction_strategies.md)
|
||||
|
||||
### [API Reference](api/core_classes_and_functions.md)
|
||||
|
||||
Detailed documentation of the API, covering:
|
||||
|
||||
- [Core Classes and Functions](api/core_classes_and_functions.md)
|
||||
- [Detailed API Documentation](api/detailed_api_documentation.md)
|
||||
|
||||
### [Change Log](changelog.md)
|
||||
|
||||
A log of all changes, updates, and improvements made to Crawl4AI.
|
||||
|
||||
### [Contact](contact.md)
|
||||
|
||||
Information on how to get in touch with the developers, report issues, and contribute to the project.
|
||||
|
||||
## Get Started
|
||||
|
||||
To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier and more efficient.
|
||||
|
||||
Happy Crawling! 🕸️🚀
|
||||
@@ -1,193 +0,0 @@
|
||||
# Installation 💻
|
||||
|
||||
There are three ways to use Crawl4AI:
|
||||
|
||||
1. As a library (Recommended).
|
||||
2. As a local server (Docker) or using the REST API.
|
||||
3. As a local server (Docker) using the pre-built image from Docker Hub.
|
||||
|
||||
## Option 1: Library Installation
|
||||
|
||||
You can try this Colab for a quick start: [](https://colab.research.google.com/drive/1sJPAmeLj5PMrg2VgOwMJ2ubGIcK0cJeX#scrollTo=g1RrmI4W_rPk)
|
||||
|
||||
Crawl4AI offers flexible installation options to suit various use cases. Choose the option that best fits your needs:
|
||||
|
||||
- **Default Installation** (Basic functionality):
|
||||
```bash
|
||||
virtualenv venv
|
||||
source venv/bin/activate
|
||||
pip install "crawl4ai @ git+https://github.com/unclecode/crawl4ai.git"
|
||||
```
|
||||
Use this for basic web crawling and scraping tasks.
|
||||
|
||||
- **Installation with PyTorch** (For advanced text clustering):
|
||||
```bash
|
||||
virtualenv venv
|
||||
source venv/bin/activate
|
||||
pip install "crawl4ai[torch] @ git+https://github.com/unclecode/crawl4ai.git"
|
||||
```
|
||||
Choose this if you need the CosineSimilarity cluster strategy.
|
||||
|
||||
- **Installation with Transformers** (For summarization and Hugging Face models):
|
||||
```bash
|
||||
virtualenv venv
|
||||
source venv/bin/activate
|
||||
pip install "crawl4ai[transformer] @ git+https://github.com/unclecode/crawl4ai.git"
|
||||
```
|
||||
Opt for this if you require text summarization or plan to use Hugging Face models.
|
||||
|
||||
- **Full Installation** (All features):
|
||||
```bash
|
||||
virtualenv venv
|
||||
source venv/bin/activate
|
||||
pip install "crawl4ai[all] @ git+https://github.com/unclecode/crawl4ai.git"
|
||||
```
|
||||
This installs all dependencies for full functionality.
|
||||
|
||||
- **Development Installation** (For contributors):
|
||||
```bash
|
||||
virtualenv venv
|
||||
source venv/bin/activate
|
||||
git clone https://github.com/unclecode/crawl4ai.git
|
||||
cd crawl4ai
|
||||
pip install -e ".[all]"
|
||||
```
|
||||
Use this if you plan to modify the source code.
|
||||
|
||||
💡 After installation, if you have used "torch", "transformer" or "all", it's recommended to run the following CLI command to load the required models. This is optional but will boost the performance and speed of the crawler. You need to do this only once, this is only for when you install using []
|
||||
```bash
|
||||
crawl4ai-download-models
|
||||
```
|
||||
|
||||
## Option 2: Using Docker for Local Server
|
||||
|
||||
Crawl4AI can be run as a local server using Docker. The Dockerfile supports different installation options to cater to various use cases. Here's how you can build and run the Docker image:
|
||||
|
||||
### Default Installation
|
||||
|
||||
The default installation includes the basic Crawl4AI package without additional dependencies or pre-downloaded models.
|
||||
|
||||
```bash
|
||||
# For Mac users (M1/M2)
|
||||
docker build --platform linux/amd64 -t crawl4ai .
|
||||
|
||||
# For other users
|
||||
docker build -t crawl4ai .
|
||||
|
||||
# Run the container
|
||||
docker run -d -p 8000:80 crawl4ai
|
||||
```
|
||||
|
||||
### Full Installation (All Dependencies and Models)
|
||||
|
||||
This option installs all dependencies and downloads the models.
|
||||
|
||||
```bash
|
||||
# For Mac users (M1/M2)
|
||||
docker build --platform linux/amd64 --build-arg INSTALL_OPTION=all -t crawl4ai:all .
|
||||
|
||||
# For other users
|
||||
docker build --build-arg INSTALL_OPTION=all -t crawl4ai:all .
|
||||
|
||||
# Run the container
|
||||
docker run -d -p 8000:80 crawl4ai:all
|
||||
```
|
||||
|
||||
### Torch Installation
|
||||
|
||||
This option installs torch-related dependencies and downloads the models.
|
||||
|
||||
```bash
|
||||
# For Mac users (M1/M2)
|
||||
docker build --platform linux/amd64 --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
|
||||
|
||||
# For other users
|
||||
docker build --build-arg INSTALL_OPTION=torch -t crawl4ai:torch .
|
||||
|
||||
# Run the container
|
||||
docker run -d -p 8000:80 crawl4ai:torch
|
||||
```
|
||||
|
||||
### Transformer Installation
|
||||
|
||||
This option installs transformer-related dependencies and downloads the models.
|
||||
|
||||
```bash
|
||||
# For Mac users (M1/M2)
|
||||
docker build --platform linux/amd64 --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
|
||||
|
||||
# For other users
|
||||
docker build --build-arg INSTALL_OPTION=transformer -t crawl4ai:transformer .
|
||||
|
||||
# Run the container
|
||||
docker run -d -p 8000:80 crawl4ai:transformer
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- The `--platform linux/amd64` flag is necessary for Mac users with M1/M2 chips to ensure compatibility.
|
||||
- The `-t` flag tags the image with a name (and optionally a tag in the 'name:tag' format).
|
||||
- The `-d` flag runs the container in detached mode.
|
||||
- The `-p 8000:80` flag maps port 8000 on the host to port 80 in the container.
|
||||
|
||||
Choose the installation option that best suits your needs. The default installation is suitable for basic usage, while the other options provide additional capabilities for more advanced use cases.
|
||||
|
||||
## Option 3: Using the Pre-built Image from Docker Hub
|
||||
|
||||
You can use pre-built Crawl4AI images from Docker Hub, which are available for all platforms (Mac, Linux, Windows). We have official images as well as a community-contributed image (Thanks to https://github.com/FractalMind):
|
||||
|
||||
### Default Installation
|
||||
|
||||
```bash
|
||||
|
||||
# Pull the image
|
||||
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run the container
|
||||
|
||||
docker run -d -p 8000:80 unclecode/crawl4ai:latest
|
||||
|
||||
```
|
||||
|
||||
### Community-Contributed Image
|
||||
|
||||
A stable version of Crawl4AI is also available, created and maintained by a community member:
|
||||
|
||||
```bash
|
||||
|
||||
# Pull the community-contributed image
|
||||
|
||||
docker pull ryser007/crawl4ai:stable
|
||||
|
||||
# Run the container
|
||||
|
||||
docker run -d -p 8000:80 ryser007/crawl4ai:stable
|
||||
|
||||
```
|
||||
|
||||
We'd like to express our gratitude to GitHub user [@FractalMind](https://github.com/FractalMind) for creating and maintaining this stable version of the Crawl4AI Docker image. Community contributions like this are invaluable to the project.
|
||||
|
||||
|
||||
### Testing the Installation
|
||||
|
||||
After running the container, you can test if it's working correctly:
|
||||
|
||||
- On Mac and Linux:
|
||||
|
||||
```bash
|
||||
|
||||
curl http://localhost:8000
|
||||
|
||||
```
|
||||
|
||||
- On Windows (PowerShell):
|
||||
|
||||
```powershell
|
||||
|
||||
Invoke-WebRequest -Uri http://localhost:8000
|
||||
|
||||
```
|
||||
|
||||
Or open a web browser and navigate to http://localhost:8000
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
<h1>Try Our Library</h1>
|
||||
<form id="apiForm">
|
||||
<label for="inputField">Enter some input:</label>
|
||||
<input type="text" id="inputField" name="inputField" required>
|
||||
<button type="submit">Submit</button>
|
||||
</form>
|
||||
<div id="result"></div>
|
||||
|
||||
<script>
|
||||
document.getElementById('apiForm').addEventListener('submit', function(event) {
|
||||
event.preventDefault();
|
||||
const input = document.getElementById('inputField').value;
|
||||
fetch('https://your-api-endpoint.com/api', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({ input: input })
|
||||
})
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
document.getElementById('result').textContent = JSON.stringify(data);
|
||||
})
|
||||
.catch(error => {
|
||||
document.getElementById('result').textContent = 'Error: ' + error;
|
||||
});
|
||||
});
|
||||
</script>
|
||||
@@ -1,29 +0,0 @@
|
||||
# Introduction
|
||||
|
||||
Welcome to the documentation for Crawl4AI v0.2.5! 🕷️🤖
|
||||
|
||||
Crawl4AI is designed to simplify the process of crawling web pages and extracting useful information for large language models (LLMs) and AI applications. Whether you're using it as a REST API, a Python library, or through a Google Colab notebook, Crawl4AI provides powerful features to make web data extraction easier and more efficient.
|
||||
|
||||
## Key Features ✨
|
||||
|
||||
- **🆓 Completely Free and Open-Source**: Crawl4AI is free to use and open-source, making it accessible for everyone.
|
||||
- **🤖 LLM-Friendly Output Formats**: Supports JSON, cleaned HTML, and markdown formats.
|
||||
- **🌍 Concurrent Crawling**: Crawl multiple URLs simultaneously to save time.
|
||||
- **🎨 Media Extraction**: Extract all media tags including images, audio, and video.
|
||||
- **🔗 Link Extraction**: Extract all external and internal links from web pages.
|
||||
- **📚 Metadata Extraction**: Extract metadata from web pages for additional context.
|
||||
- **🔄 Custom Hooks**: Define custom hooks for authentication, headers, and page modifications before crawling.
|
||||
- **🕵️ User Agent Support**: Customize the user agent for HTTP requests.
|
||||
- **🖼️ Screenshot Capability**: Take screenshots of web pages during crawling.
|
||||
- **📜 JavaScript Execution**: Execute custom JavaScripts before crawling.
|
||||
- **📚 Advanced Chunking and Extraction Strategies**: Utilize topic-based, regex, sentence chunking, cosine clustering, and LLM extraction strategies.
|
||||
- **🎯 CSS Selector Support**: Extract specific content using CSS selectors.
|
||||
- **📝 Instruction/Keyword Refinement**: Pass instructions or keywords to refine the extraction process.
|
||||
|
||||
Check the [Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for more details.
|
||||
|
||||
## Power and Simplicity of Crawl4AI 🚀
|
||||
|
||||
Crawl4AI provides an easy way to crawl and extract data from web pages without installing any library. You can use the REST API on our server or run the local server on your machine. For more advanced control, use the Python library to customize your crawling and extraction strategies.
|
||||
|
||||
Explore the documentation to learn more about the features, installation process, usage examples, and how to contribute to Crawl4AI. Let's make the web more accessible and useful for AI applications! 💪🌐🤖
|
||||
@@ -1,204 +0,0 @@
|
||||
# Quick Start Guide 🚀
|
||||
|
||||
Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies. Let's dive in! 🌟
|
||||
|
||||
## Getting Started 🛠️
|
||||
|
||||
First, let's create an instance of `WebCrawler` and call the `warmup()` function. This might take a few seconds the first time you run Crawl4AI, as it loads the required model files.
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
def create_crawler():
|
||||
crawler = WebCrawler(verbose=True)
|
||||
crawler.warmup()
|
||||
return crawler
|
||||
|
||||
crawler = create_crawler()
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simply provide a URL and let Crawl4AI do the magic!
|
||||
|
||||
```python
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
print(f"Basic crawl result: {result}")
|
||||
```
|
||||
|
||||
### Taking Screenshots 📸
|
||||
|
||||
Let's take a screenshot of the page!
|
||||
|
||||
```python
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
|
||||
with open("screenshot.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
print("Screenshot saved to 'screenshot.png'!")
|
||||
```
|
||||
|
||||
### Understanding Parameters 🧠
|
||||
|
||||
By default, Crawl4AI caches the results of your crawls. This means that subsequent crawls of the same URL will be much faster! Let's see this in action.
|
||||
|
||||
First crawl (caches the result):
|
||||
```python
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
print(f"First crawl result: {result}")
|
||||
```
|
||||
|
||||
Force to crawl again:
|
||||
```python
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
|
||||
print(f"Second crawl result: {result}")
|
||||
```
|
||||
|
||||
### Adding a Chunking Strategy 🧩
|
||||
|
||||
Let's add a chunking strategy: `RegexChunking`! This strategy splits the text based on a given regex pattern.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
chunking_strategy=RegexChunking(patterns=["\n\n"])
|
||||
)
|
||||
print(f"RegexChunking result: {result}")
|
||||
```
|
||||
|
||||
You can also use `NlpSentenceChunking` which splits the text into sentences using NLP techniques.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
chunking_strategy=NlpSentenceChunking()
|
||||
)
|
||||
print(f"NlpSentenceChunking result: {result}")
|
||||
```
|
||||
|
||||
### Adding an Extraction Strategy 🧠
|
||||
|
||||
Let's get smarter with an extraction strategy: `CosineStrategy`! This strategy uses cosine similarity to extract semantically similar blocks of text.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
extraction_strategy=CosineStrategy(
|
||||
word_count_threshold=10,
|
||||
max_dist=0.2,
|
||||
linkage_method="ward",
|
||||
top_k=3
|
||||
)
|
||||
)
|
||||
print(f"CosineStrategy result: {result}")
|
||||
```
|
||||
|
||||
You can also pass other parameters like `semantic_filter` to extract specific content.
|
||||
|
||||
```python
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
extraction_strategy=CosineStrategy(
|
||||
semantic_filter="inflation rent prices"
|
||||
)
|
||||
)
|
||||
print(f"CosineStrategy result with semantic filter: {result}")
|
||||
```
|
||||
|
||||
### Using LLMExtractionStrategy 🤖
|
||||
|
||||
Time to bring in the big guns: `LLMExtractionStrategy` without instructions! This strategy uses a large language model to extract relevant information from the web page.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
import os
|
||||
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
api_token=os.getenv('OPENAI_API_KEY')
|
||||
)
|
||||
)
|
||||
print(f"LLMExtractionStrategy (no instructions) result: {result}")
|
||||
```
|
||||
|
||||
You can also provide specific instructions to guide the extraction.
|
||||
|
||||
```python
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
instruction="I am interested in only financial news"
|
||||
)
|
||||
)
|
||||
print(f"LLMExtractionStrategy (with instructions) result: {result}")
|
||||
```
|
||||
|
||||
### Targeted Extraction 🎯
|
||||
|
||||
Let's use a CSS selector to extract only H2 tags!
|
||||
|
||||
```python
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
css_selector="h2"
|
||||
)
|
||||
print(f"CSS Selector (H2 tags) result: {result}")
|
||||
```
|
||||
|
||||
### Interactive Extraction 🖱️
|
||||
|
||||
Passing JavaScript code to click the 'Load More' button!
|
||||
|
||||
```python
|
||||
js_code = """
|
||||
const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
|
||||
loadMoreButton && loadMoreButton.click();
|
||||
"""
|
||||
|
||||
result = crawler.run(
|
||||
url="https://www.nbcnews.com/business",
|
||||
js=js_code
|
||||
)
|
||||
print(f"JavaScript Code (Load More button) result: {result}")
|
||||
```
|
||||
|
||||
### Using Crawler Hooks 🔗
|
||||
|
||||
Let's see how we can customize the crawler using hooks!
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
from crawl4ai.web_crawler import WebCrawler
|
||||
from crawl4ai.crawler_strategy import *
|
||||
|
||||
def delay(driver):
|
||||
print("Delaying for 5 seconds...")
|
||||
time.sleep(5)
|
||||
print("Resuming...")
|
||||
|
||||
def create_crawler():
|
||||
crawler_strategy = LocalSeleniumCrawlerStrategy(verbose=True)
|
||||
crawler_strategy.set_hook('after_get_url', delay)
|
||||
crawler = WebCrawler(verbose=True, crawler_strategy=crawler_strategy)
|
||||
crawler.warmup()
|
||||
return crawler
|
||||
|
||||
crawler = create_crawler()
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", bypass_cache=True)
|
||||
```
|
||||
|
||||
check [Hooks](examples/hooks_auth.md) for more examples.
|
||||
|
||||
## Congratulations! 🎉
|
||||
|
||||
You've made it through the Crawl4AI Quickstart Guide! Now go forth and crawl the web like a pro! 🕸️
|
||||
@@ -1,141 +0,0 @@
|
||||
# Core Classes and Functions
|
||||
|
||||
## Overview
|
||||
|
||||
In this section, we will delve into the core classes and functions that make up the Crawl4AI library. This includes the `WebCrawler` class, various `CrawlerStrategy` classes, `ChunkingStrategy` classes, and `ExtractionStrategy` classes. Understanding these core components will help you leverage the full power of Crawl4AI for your web crawling and data extraction needs.
|
||||
|
||||
## WebCrawler Class
|
||||
|
||||
The `WebCrawler` class is the main class you'll interact with. It provides the interface for crawling web pages and extracting data.
|
||||
|
||||
### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create an instance of WebCrawler
|
||||
crawler = WebCrawler()
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
- **`warmup()`**: Prepares the crawler for use, such as loading necessary models.
|
||||
- **`run(url: str, **kwargs)`**: Runs the crawler on the specified URL with optional parameters for customization.
|
||||
|
||||
```python
|
||||
crawler.warmup()
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
print(result)
|
||||
```
|
||||
|
||||
## CrawlerStrategy Classes
|
||||
|
||||
The `CrawlerStrategy` classes define how the web crawling is executed. The base class is `CrawlerStrategy`, which is extended by specific implementations like `LocalSeleniumCrawlerStrategy`.
|
||||
|
||||
### CrawlerStrategy Base Class
|
||||
|
||||
An abstract base class that defines the interface for different crawler strategies.
|
||||
|
||||
```python
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
class CrawlerStrategy(ABC):
|
||||
@abstractmethod
|
||||
def crawl(self, url: str, **kwargs) -> str:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def take_screenshot(self, save_path: str):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def update_user_agent(self, user_agent: str):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def set_hook(self, hook_type: str, hook: Callable):
|
||||
pass
|
||||
```
|
||||
|
||||
### LocalSeleniumCrawlerStrategy Class
|
||||
|
||||
A concrete implementation of `CrawlerStrategy` that uses Selenium to crawl web pages.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
|
||||
|
||||
strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
|
||||
- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
|
||||
- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
|
||||
- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
|
||||
|
||||
```python
|
||||
result = strategy.crawl("https://www.example.com")
|
||||
strategy.take_screenshot("screenshot.png")
|
||||
strategy.update_user_agent("Mozilla/5.0")
|
||||
strategy.set_hook("before_get_url", lambda: print("About to get URL"))
|
||||
```
|
||||
|
||||
## ChunkingStrategy Classes
|
||||
|
||||
The `ChunkingStrategy` classes define how the text from a web page is divided into chunks. Here are a few examples:
|
||||
|
||||
### RegexChunking Class
|
||||
|
||||
Splits text using regular expressions.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
chunker = RegexChunking(patterns=[r'\n\n'])
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
|
||||
```
|
||||
|
||||
### NlpSentenceChunking Class
|
||||
|
||||
Uses NLP to split text into sentences.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
chunker = NlpSentenceChunking()
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
|
||||
```
|
||||
|
||||
## ExtractionStrategy Classes
|
||||
|
||||
The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks. Here are a few examples:
|
||||
|
||||
### CosineStrategy Class
|
||||
|
||||
Clusters text chunks based on cosine similarity.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### LLMExtractionStrategy Class
|
||||
|
||||
Uses a Language Model to extract meaningful blocks from HTML.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
By understanding these core classes and functions, you can customize and extend Crawl4AI to suit your specific web crawling and data extraction needs. Happy crawling! 🕷️🤖
|
||||
|
||||
@@ -1,338 +0,0 @@
|
||||
# Detailed API Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This section provides comprehensive documentation for the Crawl4AI API, covering all classes, methods, and their parameters. This guide will help you understand how to utilize the API to its full potential, enabling efficient web crawling and data extraction.
|
||||
|
||||
## WebCrawler Class
|
||||
|
||||
The `WebCrawler` class is the primary interface for crawling web pages and extracting data.
|
||||
|
||||
### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
#### `warmup()`
|
||||
|
||||
Prepares the crawler for use, such as loading necessary models.
|
||||
|
||||
```python
|
||||
crawler.warmup()
|
||||
```
|
||||
|
||||
#### `run(url: str, **kwargs) -> CrawlResult`
|
||||
|
||||
Crawls the specified URL and returns the result.
|
||||
|
||||
- **Parameters:**
|
||||
- `url` (str): The URL to crawl.
|
||||
- `**kwargs`: Additional parameters for customization.
|
||||
|
||||
- **Returns:**
|
||||
- `CrawlResult`: An object containing the crawl result.
|
||||
|
||||
- **Example:**
|
||||
|
||||
```python
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
print(result)
|
||||
```
|
||||
|
||||
### CrawlResult Class
|
||||
|
||||
Represents the result of a crawl operation.
|
||||
|
||||
- **Attributes:**
|
||||
- `url` (str): The URL of the crawled page.
|
||||
- `html` (str): The raw HTML of the page.
|
||||
- `success` (bool): Whether the crawl was successful.
|
||||
- `cleaned_html` (Optional[str]): The cleaned HTML.
|
||||
- `media` (Dict[str, List[Dict]]): Media tags in the page (images, audio, video).
|
||||
- `links` (Dict[str, List[Dict]]): Links in the page (external, internal).
|
||||
- `screenshot` (Optional[str]): Base64 encoded screenshot.
|
||||
- `markdown` (Optional[str]): Extracted content in Markdown format.
|
||||
- `extracted_content` (Optional[str]): Extracted meaningful content.
|
||||
- `metadata` (Optional[dict]): Metadata from the page.
|
||||
- `error_message` (Optional[str]): Error message if any.
|
||||
|
||||
## CrawlerStrategy Classes
|
||||
|
||||
The `CrawlerStrategy` classes define how the web crawling is executed.
|
||||
|
||||
### CrawlerStrategy Base Class
|
||||
|
||||
An abstract base class for different crawler strategies.
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`crawl(url: str, **kwargs) -> str`**: Crawls the specified URL.
|
||||
- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
|
||||
- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
|
||||
- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
|
||||
|
||||
### LocalSeleniumCrawlerStrategy Class
|
||||
|
||||
Uses Selenium to crawl web pages.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.crawler_strategy import LocalSeleniumCrawlerStrategy
|
||||
|
||||
strategy = LocalSeleniumCrawlerStrategy(js_code=["console.log('Hello, world!');"])
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`crawl(url: str, **kwargs)`**: Crawls the specified URL.
|
||||
- **`take_screenshot(save_path: str)`**: Takes a screenshot of the current page.
|
||||
- **`update_user_agent(user_agent: str)`**: Updates the user agent for the browser.
|
||||
- **`set_hook(hook_type: str, hook: Callable)`**: Sets a hook for various events.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
result = strategy.crawl("https://www.example.com")
|
||||
strategy.take_screenshot("screenshot.png")
|
||||
strategy.update_user_agent("Mozilla/5.0")
|
||||
strategy.set_hook("before_get_url", lambda: print("About to get URL"))
|
||||
```
|
||||
|
||||
## ChunkingStrategy Classes
|
||||
|
||||
The `ChunkingStrategy` classes define how the text from a web page is divided into chunks.
|
||||
|
||||
### RegexChunking Class
|
||||
|
||||
Splits text using regular expressions.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
chunker = RegexChunking(patterns=[r'\n\n'])
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into chunks.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into chunks.")
|
||||
```
|
||||
|
||||
### NlpSentenceChunking Class
|
||||
|
||||
Uses NLP to split text into sentences.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
chunker = NlpSentenceChunking()
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into sentences.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into sentences.")
|
||||
```
|
||||
|
||||
### TopicSegmentationChunking Class
|
||||
|
||||
Uses the TextTiling algorithm to segment text into topics.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import TopicSegmentationChunking
|
||||
|
||||
chunker = TopicSegmentationChunking(num_keywords=3)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into topic-based segments.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into topic-based segments.")
|
||||
```
|
||||
|
||||
### FixedLengthWordChunking Class
|
||||
|
||||
Splits text into chunks of fixed length based on the number of words.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import FixedLengthWordChunking
|
||||
|
||||
chunker = FixedLengthWordChunking(chunk_size=100)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text into fixed-length word chunks.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split into fixed-length word chunks.")
|
||||
```
|
||||
|
||||
### SlidingWindowChunking Class
|
||||
|
||||
Uses a sliding window approach to chunk text.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import SlidingWindowChunking
|
||||
|
||||
chunker = SlidingWindowChunking(window_size=100, step=50)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`chunk(text: str) -> List[str]`**: Splits the text using a sliding window approach.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
chunks = chunker.chunk("This is a sample text. It will be split using a sliding window approach.")
|
||||
```
|
||||
|
||||
## ExtractionStrategy Classes
|
||||
|
||||
The `ExtractionStrategy` classes define how meaningful content is extracted from the chunks.
|
||||
|
||||
### NoExtractionStrategy Class
|
||||
|
||||
Returns the entire HTML content without any modification.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import NoExtractionStrategy
|
||||
|
||||
extractor = NoExtractionStrategy()
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Returns the HTML content.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### LLMExtractionStrategy Class
|
||||
|
||||
Uses a Language Model to extract meaningful blocks from HTML.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
extractor = LLMExtractionStrategy(provider='openai', api_token='your_api_token', instruction='Extract only news about AI.')
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Extracts meaningful content using the LLM.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### CosineStrategy Class
|
||||
|
||||
Clusters text chunks based on cosine similarity.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
extractor = CosineStrategy(semantic_filter="finance", word_count_threshold=10)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Extracts clusters of text based on cosine similarity.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
### TopicExtractionStrategy Class
|
||||
|
||||
Uses the TextTiling algorithm to segment HTML content into topics and extract keywords.
|
||||
|
||||
#### Initialization
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import TopicExtractionStrategy
|
||||
|
||||
extractor = TopicExtractionStrategy(num_keywords=3)
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
- **`extract(url: str, html: str) -> str`**: Extracts topic-based segments and keywords.
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
extracted_content = extractor.extract(url="https://www.example.com", html="<html>...</html>")
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
Here are the common parameters used across various classes and methods:
|
||||
|
||||
- **`url`** (str): The URL to crawl.
|
||||
- **`html`** (str): The HTML content of the page.
|
||||
- **`user_agent`** (str): The user agent for the HTTP requests.
|
||||
- **`patterns`** (list): A list of regular expression patterns for chunking.
|
||||
- **`num_keywords`** (int): Number of keywords for topic extraction.
|
||||
- **`chunk_size`** (int): Number of words in each chunk.
|
||||
- **`window_size`** (int): Number of words in the sliding window.
|
||||
- **`step`** (int): Step size for the sliding window.
|
||||
- **`semantic_filter`** (str): Keywords for filtering relevant documents.
|
||||
- **`word_count_threshold`** (int): Minimum number of words per cluster.
|
||||
- **`max_dist`** (float): Maximum cophenetic distance for clustering.
|
||||
- **`linkage_method`** (str): Linkage method for hierarchical clustering.
|
||||
- **`top_k`** (int): Number of top categories to extract.
|
||||
- **`provider`** (
|
||||
|
||||
str): Provider for language model completions.
|
||||
- **`api_token`** (str): API token for the provider.
|
||||
- **`instruction`** (str): Instruction to guide the LLM extraction.
|
||||
|
||||
## Conclusion
|
||||
|
||||
This detailed API documentation provides a thorough understanding of the classes, methods, and parameters in the Crawl4AI library. With this knowledge, you can effectively use the API to perform advanced web crawling and data extraction tasks.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because one or more lines are too long
1213
docs/md/assets/highlight.min.js
vendored
1213
docs/md/assets/highlight.min.js
vendored
File diff suppressed because one or more lines are too long
@@ -1,6 +0,0 @@
|
||||
document.addEventListener('DOMContentLoaded', (event) => {
|
||||
document.querySelectorAll('pre code').forEach((block) => {
|
||||
hljs.highlightBlock(block);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,153 +0,0 @@
|
||||
@font-face {
|
||||
font-family: "Monaco";
|
||||
font-style: normal;
|
||||
font-weight: normal;
|
||||
src: local("Monaco"), url("Monaco.woff") format("woff");
|
||||
}
|
||||
|
||||
:root {
|
||||
--global-font-size: 16px;
|
||||
--global-line-height: 1.5em;
|
||||
--global-space: 10px;
|
||||
--font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
|
||||
Courier New, monospace, serif;
|
||||
--font-stack: dm, Monaco, Courier New, monospace, serif;
|
||||
--mono-font-stack: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono,
|
||||
Courier New, monospace, serif;
|
||||
|
||||
--background-color: #151515; /* Dark background */
|
||||
--font-color: #eaeaea; /* Light font color for contrast */
|
||||
--invert-font-color: #151515; /* Dark color for inverted elements */
|
||||
--primary-color: #1a95e0; /* Primary color can remain the same or be adjusted for better contrast */
|
||||
--secondary-color: #727578; /* Secondary color for less important text */
|
||||
--error-color: #ff5555; /* Bright color for errors */
|
||||
--progress-bar-background: #444; /* Darker background for progress bar */
|
||||
--progress-bar-fill: #1a95e0; /* Bright color for progress bar fill */
|
||||
--code-bg-color: #1e1e1e; /* Darker background for code blocks */
|
||||
--input-style: solid; /* Keeping input style solid */
|
||||
--block-background-color: #202020; /* Darker background for block elements */
|
||||
--global-font-color: #eaeaea; /* Light font color for global elements */
|
||||
|
||||
--background-color: #222225;
|
||||
|
||||
--background-color: #070708;
|
||||
--page-width: 70em;
|
||||
--font-color: #e8e9ed;
|
||||
--invert-font-color: #222225;
|
||||
--secondary-color: #a3abba;
|
||||
--secondary-color: #d5cec0;
|
||||
--tertiary-color: #a3abba;
|
||||
--primary-color: #09b5a5; /* Updated to the brand color */
|
||||
--primary-color: #50ffff; /* Updated to the brand color */
|
||||
--error-color: #ff3c74;
|
||||
--progress-bar-background: #3f3f44;
|
||||
--progress-bar-fill: #09b5a5; /* Updated to the brand color */
|
||||
--code-bg-color: #3f3f44;
|
||||
--input-style: solid;
|
||||
--display-h1-decoration: none;
|
||||
|
||||
--display-h1-decoration: none;
|
||||
}
|
||||
|
||||
/* body {
|
||||
background-color: var(--background-color);
|
||||
color: var(--font-color);
|
||||
}
|
||||
|
||||
a {
|
||||
color: var(--primary-color);
|
||||
}
|
||||
|
||||
a:hover {
|
||||
background-color: var(--primary-color);
|
||||
color: var(--invert-font-color);
|
||||
}
|
||||
|
||||
blockquote::after {
|
||||
color: #444;
|
||||
}
|
||||
|
||||
pre, code {
|
||||
background-color: var(--code-bg-color);
|
||||
color: var(--font-color);
|
||||
}
|
||||
|
||||
.terminal-nav:first-child {
|
||||
border-bottom: 1px dashed var(--secondary-color);
|
||||
} */
|
||||
|
||||
.terminal-mkdocs-main-content {
|
||||
line-height: var(--global-line-height);
|
||||
}
|
||||
|
||||
strong,
|
||||
.highlight {
|
||||
/* background: url(//s2.svgbox.net/pen-brushes.svg?ic=brush-1&color=50ffff); */
|
||||
background-color: #50ffff33;
|
||||
}
|
||||
|
||||
.terminal-card > header {
|
||||
color: var(--font-color);
|
||||
text-align: center;
|
||||
background-color: var(--progress-bar-background);
|
||||
padding: 0.3em 0.5em;
|
||||
}
|
||||
.btn.btn-sm {
|
||||
color: var(--font-color);
|
||||
padding: 0.2em 0.5em;
|
||||
font-size: 0.8em;
|
||||
}
|
||||
|
||||
.loading-message {
|
||||
display: none;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
.response-section {
|
||||
display: none;
|
||||
padding-top: 20px;
|
||||
}
|
||||
|
||||
.tabs {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
.tab-list {
|
||||
display: flex;
|
||||
padding: 0;
|
||||
margin: 0;
|
||||
list-style-type: none;
|
||||
border-bottom: 1px solid var(--font-color);
|
||||
}
|
||||
.tab-item {
|
||||
cursor: pointer;
|
||||
padding: 10px;
|
||||
border: 1px solid var(--font-color);
|
||||
margin-right: -1px;
|
||||
border-bottom: none;
|
||||
}
|
||||
.tab-item:hover,
|
||||
.tab-item:focus,
|
||||
.tab-item:active {
|
||||
background-color: var(--progress-bar-background);
|
||||
}
|
||||
.tab-content {
|
||||
display: none;
|
||||
border: 1px solid var(--font-color);
|
||||
border-top: none;
|
||||
}
|
||||
.tab-content:first-of-type {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.tab-content header {
|
||||
padding: 0.5em;
|
||||
display: flex;
|
||||
justify-content: end;
|
||||
align-items: center;
|
||||
background-color: var(--progress-bar-background);
|
||||
}
|
||||
.tab-content pre {
|
||||
margin: 0;
|
||||
max-height: 300px; overflow: auto; border:none;
|
||||
}
|
||||
@@ -1,102 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
## [v0.2.77] - 2024-08-04
|
||||
|
||||
Significant improvements in text processing and performance:
|
||||
|
||||
- 🚀 **Dependency reduction**: Removed dependency on spaCy model for text chunk labeling in cosine extraction strategy.
|
||||
- 🤖 **Transformer upgrade**: Implemented text sequence classification using a transformer model for labeling text chunks.
|
||||
- ⚡ **Performance enhancement**: Improved model loading speed due to removal of spaCy dependency.
|
||||
- 🔧 **Future-proofing**: Laid groundwork for potential complete removal of spaCy dependency in future versions.
|
||||
|
||||
These changes address issue #68 and provide a foundation for faster, more efficient text processing in Crawl4AI.
|
||||
|
||||
## [v0.2.76] - 2024-08-02
|
||||
|
||||
Major improvements in functionality, performance, and cross-platform compatibility! 🚀
|
||||
|
||||
- 🐳 **Docker enhancements**: Significantly improved Dockerfile for easy installation on Linux, Mac, and Windows.
|
||||
- 🌐 **Official Docker Hub image**: Launched our first official image on Docker Hub for streamlined deployment.
|
||||
- 🔧 **Selenium upgrade**: Removed dependency on ChromeDriver, now using Selenium's built-in capabilities for better compatibility.
|
||||
- 🖼️ **Image description**: Implemented ability to generate textual descriptions for extracted images from web pages.
|
||||
- ⚡ **Performance boost**: Various improvements to enhance overall speed and performance.
|
||||
|
||||
A big shoutout to our amazing community contributors:
|
||||
- [@aravindkarnam](https://github.com/aravindkarnam) for developing the textual description extraction feature.
|
||||
- [@FractalMind](https://github.com/FractalMind) for creating the first official Docker Hub image and fixing Dockerfile errors.
|
||||
- [@ketonkss4](https://github.com/ketonkss4) for identifying Selenium's new capabilities, helping us reduce dependencies.
|
||||
|
||||
Your contributions are driving Crawl4AI forward! 🙌
|
||||
|
||||
## [v0.2.75] - 2024-07-19
|
||||
|
||||
Minor improvements for a more maintainable codebase:
|
||||
|
||||
- 🔄 Fixed typos in `chunking_strategy.py` and `crawler_strategy.py` to improve code readability
|
||||
- 🔄 Removed `.test_pads/` directory from `.gitignore` to keep our repository clean and organized
|
||||
|
||||
These changes may seem small, but they contribute to a more stable and sustainable codebase. By fixing typos and updating our `.gitignore` settings, we're ensuring that our code is easier to maintain and scale in the long run.
|
||||
|
||||
|
||||
## v0.2.74 - 2024-07-08
|
||||
A slew of exciting updates to improve the crawler's stability and robustness! 🎉
|
||||
|
||||
- 💻 **UTF encoding fix**: Resolved the Windows \"charmap\" error by adding UTF encoding.
|
||||
- 🛡️ **Error handling**: Implemented MaxRetryError exception handling in LocalSeleniumCrawlerStrategy.
|
||||
- 🧹 **Input sanitization**: Improved input sanitization and handled encoding issues in LLMExtractionStrategy.
|
||||
- 🚮 **Database cleanup**: Removed existing database file and initialized a new one.
|
||||
|
||||
## [v0.2.73] - 2024-07-03
|
||||
|
||||
💡 In this release, we've bumped the version to v0.2.73 and refreshed our documentation to ensure you have the best experience with our project.
|
||||
|
||||
* Supporting website need "with-head" mode to crawl the website with head.
|
||||
* Fixing the installation issues for setup.py and dockerfile.
|
||||
* Resolve multiple issues.
|
||||
|
||||
## [v0.2.72] - 2024-06-30
|
||||
|
||||
This release brings exciting updates and improvements to our project! 🎉
|
||||
|
||||
* 📚 **Documentation Updates**: Our documentation has been revamped to reflect the latest changes and additions.
|
||||
* 🚀 **New Modes in setup.py**: We've added support for three new modes in setup.py: default, torch, and transformers. This enhances the project's flexibility and usability.
|
||||
* 🐳 **Docker File Updates**: The Docker file has been updated to ensure seamless compatibility with the new modes and improvements.
|
||||
* 🕷️ **Temporary Solution for Headless Crawling**: We've implemented a temporary solution to overcome issues with crawling websites in headless mode.
|
||||
|
||||
These changes aim to improve the overall user experience, provide more flexibility, and enhance the project's performance. We're thrilled to share these updates with you and look forward to continuing to evolve and improve our project!
|
||||
|
||||
## [0.2.71] - 2024-06-26
|
||||
|
||||
**Improved Error Handling and Performance** 🚧
|
||||
|
||||
* 🚫 Refactored `crawler_strategy.py` to handle exceptions and provide better error messages, making it more robust and reliable.
|
||||
* 💻 Optimized the `get_content_of_website_optimized` function in `utils.py` for improved performance, reducing potential bottlenecks.
|
||||
* 💻 Updated `utils.py` with the latest changes, ensuring consistency and accuracy.
|
||||
* 🚫 Migrated to `ChromeDriverManager` to resolve Chrome driver download issues, providing a smoother user experience.
|
||||
|
||||
These changes focus on refining the existing codebase, resulting in a more stable, efficient, and user-friendly experience. With these improvements, you can expect fewer errors and better performance in the crawler strategy and utility functions.
|
||||
|
||||
## [0.2.71] - 2024-06-25
|
||||
### Fixed
|
||||
- Speed up twice the extraction function.
|
||||
|
||||
## [0.2.6] - 2024-06-22
|
||||
### Fixed
|
||||
- Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
|
||||
|
||||
## [0.2.5] - 2024-06-18
|
||||
### Added
|
||||
- Added five important hooks to the crawler:
|
||||
- on_driver_created: Called when the driver is ready for initializations.
|
||||
- before_get_url: Called right before Selenium fetches the URL.
|
||||
- after_get_url: Called after Selenium fetches the URL.
|
||||
- before_return_html: Called when the data is parsed and ready.
|
||||
- on_user_agent_updated: Called when the user changes the user_agent, causing the driver to reinitialize.
|
||||
- Added an example in `quickstart.py` in the example folder under the docs.
|
||||
- Enhancement issue #24: Replaced inline HTML tags (e.g., DEL, INS, SUB, ABBR) with textual format for better context handling in LLM.
|
||||
- Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
|
||||
- Updated Dockerfile to ensure compatibility across multiple platforms (Hopefully!).
|
||||
|
||||
## [0.2.4] - 2024-06-17
|
||||
### Fixed
|
||||
- Fix issue #22: Use MD5 hash for caching HTML files to handle long URLs
|
||||
@@ -1,25 +0,0 @@
|
||||
# Contact
|
||||
If you have any questions, suggestions, or feedback, please feel free to reach out to us:
|
||||
|
||||
- GitHub: [unclecode](https://github.com/unclecode)
|
||||
- Twitter: [@unclecode](https://twitter.com/unclecode)
|
||||
- Website: [crawl4ai.com](https://crawl4ai.com)
|
||||
|
||||
|
||||
## Contributing 🤝
|
||||
|
||||
We welcome contributions from the open-source community to help improve Crawl4AI and make it even more valuable for AI enthusiasts and developers. To contribute, please follow these steps:
|
||||
|
||||
1. Fork the repository.
|
||||
2. Create a new branch for your feature or bug fix.
|
||||
3. Make your changes and commit them with descriptive messages.
|
||||
4. Push your changes to your forked repository.
|
||||
5. Submit a pull request to the main repository.
|
||||
|
||||
For more information on contributing, please see our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md).
|
||||
|
||||
## License 📄
|
||||
|
||||
Crawl4AI is released under the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE).
|
||||
|
||||
Let's work together to make the web more accessible and useful for AI applications! 💪🌐🤖
|
||||
231
docs/md/demo.md
231
docs/md/demo.md
@@ -1,231 +0,0 @@
|
||||
# Interactive Demo for Crowler
|
||||
<div id="demo">
|
||||
<form id="crawlForm" class="terminal-form">
|
||||
<fieldset>
|
||||
<legend>Enter URL and Options</legend>
|
||||
<div class="form-group">
|
||||
<label for="url">Enter URL:</label>
|
||||
<input type="text" id="url" name="url" required>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="screenshot">Get Screenshot:</label>
|
||||
<input type="checkbox" id="screenshot" name="screenshot">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<button class="btn btn-default" type="submit">Submit</button>
|
||||
</div>
|
||||
|
||||
</fieldset>
|
||||
</form>
|
||||
|
||||
<div id="loading" class="loading-message">
|
||||
<div class="terminal-alert terminal-alert-primary">Loading... Please wait.</div>
|
||||
</div>
|
||||
|
||||
<section id="response" class="response-section">
|
||||
<h2>Response</h2>
|
||||
<div class="tabs">
|
||||
<ul class="tab-list">
|
||||
<li class="tab-item" onclick="showTab('markdown')">Markdown</li>
|
||||
<li class="tab-item" onclick="showTab('cleanedHtml')">Cleaned HTML</li>
|
||||
<li class="tab-item" onclick="showTab('media')">Media</li>
|
||||
<li class="tab-item" onclick="showTab('extractedContent')">Extracted Content</li>
|
||||
<li class="tab-item" onclick="showTab('screenshot')">Screenshot</li>
|
||||
<li class="tab-item" onclick="showTab('pythonCode')">Python Code</li>
|
||||
</ul>
|
||||
<div class="tab-content" id="tab-markdown">
|
||||
<header>
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('markdownContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('markdownContent', 'markdown.md')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="markdownContent" class="language-markdown hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-cleanedHtml" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('cleanedHtmlContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('cleanedHtmlContent', 'cleaned.html')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="cleanedHtmlContent" class="language-html hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-media" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('mediaContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('mediaContent', 'media.json')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="mediaContent" class="language-json hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-extractedContent" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('extractedContentContent')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('extractedContentContent', 'extracted_content.json')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="extractedContentContent" class="language-json hljs"></code></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-screenshot" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadImage('screenshotContent', 'screenshot.png')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><img id="screenshotContent" /></pre>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="tab-pythonCode" style="display: none;">
|
||||
<header >
|
||||
<div>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="copyToClipboard('pythonCode')">Copy</button>
|
||||
<button class="btn btn-default btn-ghost btn-sm" onclick="downloadContent('pythonCode', 'example.py')">Download</button>
|
||||
</div>
|
||||
</header>
|
||||
<pre><code id="pythonCode" class="language-python hljs"></code></pre>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<div id="error" class="error-message" style="display: none; margin-top:1em;">
|
||||
<div class="terminal-alert terminal-alert-error"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
function showTab(tabId) {
|
||||
const tabs = document.querySelectorAll('.tab-content');
|
||||
tabs.forEach(tab => tab.style.display = 'none');
|
||||
document.getElementById(`tab-${tabId}`).style.display = 'block';
|
||||
}
|
||||
|
||||
function redo(codeBlock, codeText){
|
||||
codeBlock.classList.remove('hljs');
|
||||
codeBlock.removeAttribute('data-highlighted');
|
||||
|
||||
// Set new code and re-highlight
|
||||
codeBlock.textContent = codeText;
|
||||
hljs.highlightBlock(codeBlock);
|
||||
}
|
||||
|
||||
function copyToClipboard(elementId) {
|
||||
const content = document.getElementById(elementId).textContent;
|
||||
navigator.clipboard.writeText(content).then(() => {
|
||||
alert('Copied to clipboard');
|
||||
});
|
||||
}
|
||||
|
||||
function downloadContent(elementId, filename) {
|
||||
const content = document.getElementById(elementId).textContent;
|
||||
const blob = new Blob([content], { type: 'text/plain' });
|
||||
const url = window.URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.style.display = 'none';
|
||||
a.href = url;
|
||||
a.download = filename;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
window.URL.revokeObjectURL(url);
|
||||
document.body.removeChild(a);
|
||||
}
|
||||
|
||||
function downloadImage(elementId, filename) {
|
||||
const content = document.getElementById(elementId).src;
|
||||
const a = document.createElement('a');
|
||||
a.style.display = 'none';
|
||||
a.href = content;
|
||||
a.download = filename;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
}
|
||||
|
||||
document.getElementById('crawlForm').addEventListener('submit', function(event) {
|
||||
event.preventDefault();
|
||||
document.getElementById('loading').style.display = 'block';
|
||||
document.getElementById('response').style.display = 'none';
|
||||
|
||||
const url = document.getElementById('url').value;
|
||||
const screenshot = document.getElementById('screenshot').checked;
|
||||
const data = {
|
||||
urls: [url],
|
||||
bypass_cache: false,
|
||||
word_count_threshold: 5,
|
||||
screenshot: screenshot
|
||||
};
|
||||
|
||||
fetch('/crawl', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify(data)
|
||||
})
|
||||
.then(response => {
|
||||
if (!response.ok) {
|
||||
if (response.status === 429) {
|
||||
return response.json().then(err => {
|
||||
throw Object.assign(new Error('Rate limit exceeded'), { status: 429, details: err });
|
||||
});
|
||||
}
|
||||
throw new Error('Network response was not ok');
|
||||
}
|
||||
return response.json();
|
||||
})
|
||||
.then(data => {
|
||||
data = data.results[0]; // Only one URL is requested
|
||||
document.getElementById('loading').style.display = 'none';
|
||||
document.getElementById('response').style.display = 'block';
|
||||
redo(document.getElementById('markdownContent'), data.markdown);
|
||||
redo(document.getElementById('cleanedHtmlContent'), data.cleaned_html);
|
||||
redo(document.getElementById('mediaContent'), JSON.stringify(data.media, null, 2));
|
||||
redo(document.getElementById('extractedContentContent'), data.extracted_content);
|
||||
if (screenshot) {
|
||||
document.getElementById('screenshotContent').src = `data:image/png;base64,${data.screenshot}`;
|
||||
}
|
||||
const pythonCode = `
|
||||
from crawl4ai.web_crawler import WebCrawler
|
||||
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
result = crawler.run(
|
||||
url='${url}',
|
||||
screenshot=${screenshot}
|
||||
)
|
||||
print(result)
|
||||
`;
|
||||
redo(document.getElementById('pythonCode'), pythonCode);
|
||||
document.getElementById('error').style.display = 'none';
|
||||
})
|
||||
.catch(error => {
|
||||
document.getElementById('loading').style.display = 'none';
|
||||
document.getElementById('error').style.display = 'block';
|
||||
let errorMessage = 'An unexpected error occurred. Please try again later.';
|
||||
|
||||
if (error.status === 429) {
|
||||
const details = error.details;
|
||||
if (details.retry_after) {
|
||||
errorMessage = `Rate limit exceeded. Please wait ${parseFloat(details.retry_after).toFixed(1)} seconds before trying again.`;
|
||||
} else if (details.reset_at) {
|
||||
const resetTime = new Date(details.reset_at);
|
||||
const waitTime = Math.ceil((resetTime - new Date()) / 1000);
|
||||
errorMessage = `Rate limit exceeded. Please try again after ${waitTime} seconds.`;
|
||||
} else {
|
||||
errorMessage = `Rate limit exceeded. Please try again later.`;
|
||||
}
|
||||
} else if (error.message) {
|
||||
errorMessage = error.message;
|
||||
}
|
||||
|
||||
document.querySelector('#error .terminal-alert').textContent = errorMessage;
|
||||
});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
@@ -1,33 +0,0 @@
|
||||
# Examples
|
||||
|
||||
Welcome to the examples section of Crawl4AI documentation! In this section, you will find practical examples demonstrating how to use Crawl4AI for various web crawling and data extraction tasks. Each example is designed to showcase different features and capabilities of the library.
|
||||
|
||||
## Examples Index
|
||||
|
||||
### [LLM Extraction](llm_extraction.md)
|
||||
|
||||
This example demonstrates how to use Crawl4AI to extract information using Large Language Models (LLMs). You will learn how to configure the `LLMExtractionStrategy` to get structured data from web pages.
|
||||
|
||||
### [JSON CSS Extraction](json_css_extraction.md)
|
||||
|
||||
This example demonstrates how to use Crawl4AI to extract structured data without using LLM, and just focusing on page structure. You will learn how to use the `JsonCssExtractionStrategy` to extract data using CSS selectors.
|
||||
|
||||
### [JS Execution & CSS Filtering](js_execution_css_filtering.md)
|
||||
|
||||
Learn how to execute custom JavaScript code and filter data using CSS selectors. This example shows how to perform complex web interactions and extract specific content from web pages.
|
||||
|
||||
### [Hooks & Auth](hooks_auth.md)
|
||||
|
||||
This example covers the use of custom hooks for authentication and other pre-crawling tasks. You will see how to set up hooks to modify headers, authenticate sessions, and perform other preparatory actions before crawling.
|
||||
|
||||
### [Summarization](summarization.md)
|
||||
|
||||
Discover how to use Crawl4AI to summarize web page content. This example demonstrates the summarization capabilities of the library, helping you extract concise information from lengthy web pages.
|
||||
|
||||
### [Research Assistant](research_assistant.md)
|
||||
|
||||
In this example, Crawl4AI is used as a research assistant to gather and organize information from multiple sources. You will learn how to use various extraction and chunking strategies to compile a comprehensive report.
|
||||
|
||||
---
|
||||
|
||||
Each example includes detailed explanations and code snippets to help you understand and implement the features in your projects. Click on the links to explore each example and start making the most of Crawl4AI!
|
||||
@@ -1,104 +0,0 @@
|
||||
# JS Execution & CSS Filtering with AsyncWebCrawler
|
||||
|
||||
In this example, we'll demonstrate how to use Crawl4AI's AsyncWebCrawler to execute JavaScript, filter data with CSS selectors, and use a cosine similarity strategy to extract relevant content. This approach is particularly useful when you need to interact with dynamic content on web pages, such as clicking "Load More" buttons.
|
||||
|
||||
## Example: Extracting Structured Data Asynchronously
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
|
||||
|
||||
async def main():
|
||||
# Define the JavaScript code to click the "Load More" button
|
||||
js_code = """
|
||||
const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
|
||||
if (loadMoreButton) {
|
||||
loadMoreButton.click();
|
||||
// Wait for new content to load
|
||||
await new Promise(resolve => setTimeout(resolve, 2000));
|
||||
}
|
||||
"""
|
||||
|
||||
# Define a wait_for function to ensure content is loaded
|
||||
wait_for = """
|
||||
() => {
|
||||
const articles = document.querySelectorAll('article.tease-card');
|
||||
return articles.length > 10;
|
||||
}
|
||||
"""
|
||||
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Run the crawler with keyword filtering and CSS selector
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
js_code=js_code,
|
||||
wait_for=wait_for,
|
||||
css_selector="article.tease-card",
|
||||
extraction_strategy=CosineStrategy(
|
||||
semantic_filter="technology",
|
||||
),
|
||||
chunking_strategy=RegexChunking(),
|
||||
)
|
||||
|
||||
# Display the extracted result
|
||||
print(result.extracted_content)
|
||||
|
||||
# Run the async function
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
1. **Asynchronous Execution**: We use `AsyncWebCrawler` with async/await syntax for non-blocking execution.
|
||||
|
||||
2. **JavaScript Execution**: The `js_code` variable contains JavaScript code that simulates clicking a "Load More" button and waits for new content to load.
|
||||
|
||||
3. **Wait Condition**: The `wait_for` function ensures that the page has loaded more than 10 articles before proceeding with the extraction.
|
||||
|
||||
4. **CSS Selector**: The `css_selector="article.tease-card"` parameter ensures that only article cards are extracted from the web page.
|
||||
|
||||
5. **Extraction Strategy**: The `CosineStrategy` is used with a semantic filter for "technology" to extract relevant content based on cosine similarity.
|
||||
|
||||
6. **Chunking Strategy**: We use `RegexChunking()` to split the content into manageable chunks for processing.
|
||||
|
||||
## Advanced Usage: Custom Session and Multiple Requests
|
||||
|
||||
For more complex scenarios where you need to maintain state across multiple requests or execute additional JavaScript after the initial page load, you can use a custom session:
|
||||
|
||||
```python
|
||||
async def advanced_crawl():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Initial crawl with custom session
|
||||
result1 = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
js_code=js_code,
|
||||
wait_for=wait_for,
|
||||
css_selector="article.tease-card",
|
||||
session_id="business_session"
|
||||
)
|
||||
|
||||
# Execute additional JavaScript in the same session
|
||||
result2 = await crawler.crawler_strategy.execute_js(
|
||||
session_id="business_session",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for_js="() => window.innerHeight + window.scrollY >= document.body.offsetHeight"
|
||||
)
|
||||
|
||||
# Process results
|
||||
print("Initial crawl result:", result1.extracted_content)
|
||||
print("Additional JS execution result:", result2.html)
|
||||
|
||||
asyncio.run(advanced_crawl())
|
||||
```
|
||||
|
||||
This advanced example demonstrates how to:
|
||||
1. Use a custom session to maintain state across requests.
|
||||
2. Execute additional JavaScript after the initial page load.
|
||||
3. Wait for specific conditions using JavaScript functions.
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
These examples demonstrate the power and flexibility of Crawl4AI's AsyncWebCrawler in handling complex web interactions and extracting meaningful data asynchronously. You can customize the JavaScript code, CSS selectors, extraction strategies, and waiting conditions to suit your specific requirements.
|
||||
@@ -1,220 +0,0 @@
|
||||
# Research Assistant Example with AsyncWebCrawler
|
||||
|
||||
This example demonstrates how to build an advanced research assistant using `Chainlit`, `Crawl4AI`'s `AsyncWebCrawler`, and various AI services. The assistant can crawl web pages asynchronously, answer questions based on the crawled content, and handle audio inputs.
|
||||
|
||||
## Step-by-Step Guide
|
||||
|
||||
1. **Install Required Packages**
|
||||
|
||||
Ensure you have the necessary packages installed:
|
||||
|
||||
```bash
|
||||
pip install chainlit groq openai crawl4ai
|
||||
```
|
||||
|
||||
2. **Import Libraries**
|
||||
|
||||
```python
|
||||
import os
|
||||
import time
|
||||
import asyncio
|
||||
from openai import AsyncOpenAI
|
||||
import chainlit as cl
|
||||
import re
|
||||
from io import BytesIO
|
||||
from chainlit.element import ElementBased
|
||||
from groq import Groq
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import NoExtractionStrategy
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
client = AsyncOpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.getenv("GROQ_API_KEY"))
|
||||
|
||||
# Instrument the OpenAI client
|
||||
cl.instrument_openai()
|
||||
```
|
||||
|
||||
3. **Set Configuration**
|
||||
|
||||
```python
|
||||
settings = {
|
||||
"model": "llama3-8b-8192",
|
||||
"temperature": 0.5,
|
||||
"max_tokens": 500,
|
||||
"top_p": 1,
|
||||
"frequency_penalty": 0,
|
||||
"presence_penalty": 0,
|
||||
}
|
||||
```
|
||||
|
||||
4. **Define Utility Functions**
|
||||
|
||||
```python
|
||||
def extract_urls(text):
|
||||
url_pattern = re.compile(r'(https?://\S+)')
|
||||
return url_pattern.findall(text)
|
||||
|
||||
async def crawl_urls(urls):
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=urls,
|
||||
word_count_threshold=10,
|
||||
extraction_strategy=NoExtractionStrategy(),
|
||||
chunking_strategy=RegexChunking(),
|
||||
bypass_cache=True
|
||||
)
|
||||
return [result.markdown for result in results if result.success]
|
||||
```
|
||||
|
||||
5. **Initialize Chat Start Event**
|
||||
|
||||
```python
|
||||
@cl.on_chat_start
|
||||
async def on_chat_start():
|
||||
cl.user_session.set("session", {
|
||||
"history": [],
|
||||
"context": {}
|
||||
})
|
||||
await cl.Message(content="Welcome to the chat! How can I assist you today?").send()
|
||||
```
|
||||
|
||||
6. **Handle Incoming Messages**
|
||||
|
||||
```python
|
||||
@cl.on_message
|
||||
async def on_message(message: cl.Message):
|
||||
user_session = cl.user_session.get("session")
|
||||
|
||||
# Extract URLs from the user's message
|
||||
urls = extract_urls(message.content)
|
||||
|
||||
if urls:
|
||||
crawled_contents = await crawl_urls(urls)
|
||||
for url, content in zip(urls, crawled_contents):
|
||||
ref_number = f"REF_{len(user_session['context']) + 1}"
|
||||
user_session["context"][ref_number] = {
|
||||
"url": url,
|
||||
"content": content
|
||||
}
|
||||
|
||||
user_session["history"].append({
|
||||
"role": "user",
|
||||
"content": message.content
|
||||
})
|
||||
|
||||
# Create a system message that includes the context
|
||||
context_messages = [
|
||||
f'<appendix ref="{ref}">\n{data["content"]}\n</appendix>'
|
||||
for ref, data in user_session["context"].items()
|
||||
]
|
||||
system_message = {
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful bot. Use the following context for answering questions. "
|
||||
"Refer to the sources using the REF number in square brackets, e.g., [1], only if the source is given in the appendices below.\n\n"
|
||||
"If the question requires any information from the provided appendices or context, refer to the sources. "
|
||||
"If not, there is no need to add a references section. "
|
||||
"At the end of your response, provide a reference section listing the URLs and their REF numbers only if sources from the appendices were used.\n\n"
|
||||
"\n\n".join(context_messages)
|
||||
) if context_messages else "You are a helpful assistant."
|
||||
}
|
||||
|
||||
msg = cl.Message(content="")
|
||||
await msg.send()
|
||||
|
||||
# Get response from the LLM
|
||||
stream = await client.chat.completions.create(
|
||||
messages=[system_message, *user_session["history"]],
|
||||
stream=True,
|
||||
**settings
|
||||
)
|
||||
|
||||
assistant_response = ""
|
||||
async for part in stream:
|
||||
if token := part.choices[0].delta.content:
|
||||
assistant_response += token
|
||||
await msg.stream_token(token)
|
||||
|
||||
# Add assistant message to the history
|
||||
user_session["history"].append({
|
||||
"role": "assistant",
|
||||
"content": assistant_response
|
||||
})
|
||||
await msg.update()
|
||||
|
||||
# Append the reference section to the assistant's response
|
||||
if user_session["context"]:
|
||||
reference_section = "\n\nReferences:\n"
|
||||
for ref, data in user_session["context"].items():
|
||||
reference_section += f"[{ref.split('_')[1]}]: {data['url']}\n"
|
||||
msg.content += reference_section
|
||||
await msg.update()
|
||||
```
|
||||
|
||||
7. **Handle Audio Input**
|
||||
|
||||
```python
|
||||
@cl.on_audio_chunk
|
||||
async def on_audio_chunk(chunk: cl.AudioChunk):
|
||||
if chunk.isStart:
|
||||
buffer = BytesIO()
|
||||
buffer.name = f"input_audio.{chunk.mimeType.split('/')[1]}"
|
||||
cl.user_session.set("audio_buffer", buffer)
|
||||
cl.user_session.set("audio_mime_type", chunk.mimeType)
|
||||
cl.user_session.get("audio_buffer").write(chunk.data)
|
||||
|
||||
@cl.step(type="tool")
|
||||
async def speech_to_text(audio_file):
|
||||
response = await client.audio.transcriptions.create(
|
||||
model="whisper-large-v3", file=audio_file
|
||||
)
|
||||
return response.text
|
||||
|
||||
@cl.on_audio_end
|
||||
async def on_audio_end(elements: list[ElementBased]):
|
||||
audio_buffer: BytesIO = cl.user_session.get("audio_buffer")
|
||||
audio_buffer.seek(0)
|
||||
audio_file = audio_buffer.read()
|
||||
audio_mime_type: str = cl.user_session.get("audio_mime_type")
|
||||
|
||||
start_time = time.time()
|
||||
transcription = await speech_to_text((audio_buffer.name, audio_file, audio_mime_type))
|
||||
end_time = time.time()
|
||||
print(f"Transcription took {end_time - start_time} seconds")
|
||||
|
||||
user_msg = cl.Message(author="You", type="user_message", content=transcription)
|
||||
await user_msg.send()
|
||||
await on_message(user_msg)
|
||||
```
|
||||
|
||||
8. **Run the Chat Application**
|
||||
|
||||
```python
|
||||
if __name__ == "__main__":
|
||||
from chainlit.cli import run_chainlit
|
||||
run_chainlit(__file__)
|
||||
```
|
||||
|
||||
## Explanation
|
||||
|
||||
- **Libraries and Configuration**: We import necessary libraries, including `AsyncWebCrawler` from `crawl4ai`.
|
||||
- **Utility Functions**:
|
||||
- `extract_urls`: Uses regex to find URLs in messages.
|
||||
- `crawl_urls`: An asynchronous function that uses `AsyncWebCrawler` to fetch content from multiple URLs concurrently.
|
||||
- **Chat Start Event**: Initializes the chat session and sends a welcome message.
|
||||
- **Message Handling**:
|
||||
- Extracts URLs from user messages.
|
||||
- Asynchronously crawls the URLs using `AsyncWebCrawler`.
|
||||
- Updates chat history and context with crawled content.
|
||||
- Generates a response using the LLM, incorporating the crawled context.
|
||||
- **Audio Handling**: Captures, buffers, and transcribes audio input, then processes the transcription as text.
|
||||
- **Running the Application**: Starts the Chainlit server for interaction with the assistant.
|
||||
|
||||
## Key Improvements
|
||||
|
||||
1. **Asynchronous Web Crawling**: Using `AsyncWebCrawler` allows for efficient, concurrent crawling of multiple URLs.
|
||||
2. **Improved Context Management**: The assistant now maintains a context of crawled content, allowing for more informed responses.
|
||||
3. **Dynamic Reference System**: The assistant can refer to specific sources in its responses and provide a reference section.
|
||||
4. **Seamless Audio Integration**: The ability to handle audio inputs makes the assistant more versatile and user-friendly.
|
||||
|
||||
This updated Research Assistant showcases how to create a powerful, interactive tool that can efficiently fetch and process web content, handle various input types, and provide informed responses based on the gathered information.
|
||||
@@ -1,153 +0,0 @@
|
||||
# Summarization Example with AsyncWebCrawler
|
||||
|
||||
This example demonstrates how to use Crawl4AI's `AsyncWebCrawler` to extract a summary from a web page asynchronously. The goal is to obtain the title, a detailed summary, a brief summary, and a list of keywords from the given page.
|
||||
|
||||
## Step-by-Step Guide
|
||||
|
||||
1. **Import Necessary Modules**
|
||||
|
||||
First, import the necessary modules and classes:
|
||||
|
||||
```python
|
||||
import os
|
||||
import json
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
from pydantic import BaseModel, Field
|
||||
```
|
||||
|
||||
2. **Define the URL to be Crawled**
|
||||
|
||||
Set the URL of the web page you want to summarize:
|
||||
|
||||
```python
|
||||
url = 'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot'
|
||||
```
|
||||
|
||||
3. **Define the Data Model**
|
||||
|
||||
Use Pydantic to define the structure of the extracted data:
|
||||
|
||||
```python
|
||||
class PageSummary(BaseModel):
|
||||
title: str = Field(..., description="Title of the page.")
|
||||
summary: str = Field(..., description="Summary of the page.")
|
||||
brief_summary: str = Field(..., description="Brief summary of the page.")
|
||||
keywords: list = Field(..., description="Keywords assigned to the page.")
|
||||
```
|
||||
|
||||
4. **Create the Extraction Strategy**
|
||||
|
||||
Set up the `LLMExtractionStrategy` with the necessary parameters:
|
||||
|
||||
```python
|
||||
extraction_strategy = LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
schema=PageSummary.model_json_schema(),
|
||||
extraction_type="schema",
|
||||
apply_chunking=False,
|
||||
instruction=(
|
||||
"From the crawled content, extract the following details: "
|
||||
"1. Title of the page "
|
||||
"2. Summary of the page, which is a detailed summary "
|
||||
"3. Brief summary of the page, which is a paragraph text "
|
||||
"4. Keywords assigned to the page, which is a list of keywords. "
|
||||
'The extracted JSON format should look like this: '
|
||||
'{ "title": "Page Title", "summary": "Detailed summary of the page.", '
|
||||
'"brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }'
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
5. **Define the Async Crawl Function**
|
||||
|
||||
Create an asynchronous function to run the crawler:
|
||||
|
||||
```python
|
||||
async def crawl_and_summarize(url):
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
word_count_threshold=1,
|
||||
extraction_strategy=extraction_strategy,
|
||||
chunking_strategy=RegexChunking(),
|
||||
bypass_cache=True,
|
||||
)
|
||||
return result
|
||||
```
|
||||
|
||||
6. **Run the Crawler and Process Results**
|
||||
|
||||
Use asyncio to run the crawler and process the results:
|
||||
|
||||
```python
|
||||
async def main():
|
||||
result = await crawl_and_summarize(url)
|
||||
|
||||
if result.success:
|
||||
page_summary = json.loads(result.extracted_content)
|
||||
print("Extracted Page Summary:")
|
||||
print(json.dumps(page_summary, indent=2))
|
||||
|
||||
# Save the extracted data
|
||||
with open(".data/page_summary.json", "w", encoding="utf-8") as f:
|
||||
json.dump(page_summary, f, indent=2)
|
||||
print("Page summary saved to .data/page_summary.json")
|
||||
else:
|
||||
print(f"Failed to crawl and summarize the page. Error: {result.error_message}")
|
||||
|
||||
# Run the async main function
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Explanation
|
||||
|
||||
- **Importing Modules**: We import the necessary modules, including `AsyncWebCrawler` and `LLMExtractionStrategy` from Crawl4AI.
|
||||
- **URL Definition**: We set the URL of the web page to crawl and summarize.
|
||||
- **Data Model Definition**: We define the structure of the data to extract using Pydantic's `BaseModel`.
|
||||
- **Extraction Strategy Setup**: We create an instance of `LLMExtractionStrategy` with the schema and detailed instructions for the extraction process.
|
||||
- **Async Crawl Function**: We define an asynchronous function `crawl_and_summarize` that uses `AsyncWebCrawler` to perform the crawling and extraction.
|
||||
- **Main Execution**: In the `main` function, we run the crawler, process the results, and save the extracted data.
|
||||
|
||||
## Advanced Usage: Crawling Multiple URLs
|
||||
|
||||
To demonstrate the power of `AsyncWebCrawler`, here's how you can summarize multiple pages concurrently:
|
||||
|
||||
```python
|
||||
async def crawl_multiple_urls(urls):
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
tasks = [crawler.arun(
|
||||
url=url,
|
||||
word_count_threshold=1,
|
||||
extraction_strategy=extraction_strategy,
|
||||
chunking_strategy=RegexChunking(),
|
||||
bypass_cache=True
|
||||
) for url in urls]
|
||||
results = await asyncio.gather(*tasks)
|
||||
return results
|
||||
|
||||
async def main():
|
||||
urls = [
|
||||
'https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot',
|
||||
'https://marketplace.visualstudio.com/items?itemName=GitHub.copilot',
|
||||
'https://marketplace.visualstudio.com/items?itemName=ms-python.python'
|
||||
]
|
||||
results = await crawl_multiple_urls(urls)
|
||||
|
||||
for i, result in enumerate(results):
|
||||
if result.success:
|
||||
page_summary = json.loads(result.extracted_content)
|
||||
print(f"\nSummary for URL {i+1}:")
|
||||
print(json.dumps(page_summary, indent=2))
|
||||
else:
|
||||
print(f"\nFailed to summarize URL {i+1}. Error: {result.error_message}")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
This advanced example shows how to use `AsyncWebCrawler` to efficiently summarize multiple web pages concurrently, significantly reducing the total processing time compared to sequential crawling.
|
||||
|
||||
By leveraging the asynchronous capabilities of Crawl4AI, you can perform advanced web crawling and data extraction tasks with improved efficiency and scalability.
|
||||
@@ -1,138 +0,0 @@
|
||||
# Advanced Features
|
||||
|
||||
Crawl4AI offers a range of advanced features that allow you to fine-tune your web crawling and data extraction process. This section will cover some of these advanced features, including taking screenshots, extracting media and links, customizing the user agent, using custom hooks, and leveraging CSS selectors.
|
||||
|
||||
## Taking Screenshots 📸
|
||||
|
||||
One of the cool features of Crawl4AI is the ability to take screenshots of the web pages you're crawling. This can be particularly useful for visual verification or for capturing the state of dynamic content.
|
||||
|
||||
Here's how you can take a screenshot:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
import base64
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler with the screenshot parameter
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", screenshot=True)
|
||||
|
||||
# Save the screenshot to a file
|
||||
with open("screenshot.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
|
||||
print("Screenshot saved to 'screenshot.png'!")
|
||||
```
|
||||
|
||||
In this example, we create a `WebCrawler` instance, warm it up, and then run it with the `screenshot` parameter set to `True`. The screenshot is saved as a base64 encoded string in the result, which we then decode and save as a PNG file.
|
||||
|
||||
## Extracting Media and Links 🎨🔗
|
||||
|
||||
Crawl4AI can extract all media tags (images, audio, and video) and links (both internal and external) from a web page. This feature is useful for collecting multimedia content or analyzing link structures.
|
||||
|
||||
Here's an example:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler
|
||||
result = crawler.run(url="https://www.nbcnews.com/business")
|
||||
|
||||
print("Extracted media:", result.media)
|
||||
print("Extracted links:", result.links)
|
||||
```
|
||||
|
||||
In this example, the `result` object contains dictionaries for media and links, which you can access and use as needed.
|
||||
|
||||
## Customizing the User Agent 🕵️♂️
|
||||
|
||||
Crawl4AI allows you to set a custom user agent for your HTTP requests. This can help you avoid detection by web servers or simulate different browsing environments.
|
||||
|
||||
Here's how to set a custom user agent:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler with a custom user agent
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
|
||||
|
||||
print("Crawl result:", result)
|
||||
```
|
||||
|
||||
In this example, we specify a custom user agent string when running the crawler.
|
||||
|
||||
## Using Custom Hooks 🪝
|
||||
|
||||
Hooks are a powerful feature in Crawl4AI that allow you to customize the crawling process at various stages. You can define hooks for actions such as driver initialization, before and after URL fetching, and before returning the HTML.
|
||||
|
||||
Here's an example of using hooks:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
|
||||
# Define the hooks
|
||||
def on_driver_created(driver):
|
||||
driver.maximize_window()
|
||||
driver.get('https://example.com/login')
|
||||
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.NAME, 'username'))).send_keys('testuser')
|
||||
driver.find_element(By.NAME, 'password').send_keys('password123')
|
||||
driver.find_element(By.NAME, 'login').click()
|
||||
return driver
|
||||
|
||||
def before_get_url(driver):
|
||||
driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Test-Header': 'test'}})
|
||||
return driver
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Set the hooks
|
||||
crawler.set_hook('on_driver_created', on_driver_created)
|
||||
crawler.set_hook('before_get_url', before_get_url)
|
||||
|
||||
# Run the crawler
|
||||
result = crawler.run(url="https://example.com")
|
||||
|
||||
print("Crawl result:", result)
|
||||
```
|
||||
|
||||
In this example, we define hooks to handle driver initialization and custom headers before fetching the URL.
|
||||
|
||||
## Using CSS Selectors 🎯
|
||||
|
||||
CSS selectors allow you to target specific elements on a web page for extraction. This can be useful for scraping structured content, such as articles or product details.
|
||||
|
||||
Here's an example of using a CSS selector:
|
||||
|
||||
```python
|
||||
from crawl4ai import WebCrawler
|
||||
|
||||
# Create the WebCrawler instance
|
||||
crawler = WebCrawler()
|
||||
crawler.warmup()
|
||||
|
||||
# Run the crawler with a CSS selector to extract only H2 tags
|
||||
result = crawler.run(url="https://www.nbcnews.com/business", css_selector="h2")
|
||||
|
||||
print("Extracted H2 tags:", result.extracted_content)
|
||||
```
|
||||
|
||||
In this example, we use the `css_selector` parameter to extract only the H2 tags from the web page.
|
||||
|
||||
---
|
||||
|
||||
With these advanced features, you can leverage Crawl4AI to perform sophisticated web crawling and data extraction tasks. Whether you need to take screenshots, extract specific elements, customize the crawling process, or set custom headers, Crawl4AI provides the flexibility and power to meet your needs. Happy crawling! 🕷️🚀
|
||||
@@ -1,133 +0,0 @@
|
||||
## Chunking Strategies 📚
|
||||
|
||||
Crawl4AI provides several powerful chunking strategies to divide text into manageable parts for further processing. Each strategy has unique characteristics and is suitable for different scenarios. Let's explore them one by one.
|
||||
|
||||
### RegexChunking
|
||||
|
||||
`RegexChunking` splits text using regular expressions. This is ideal for creating chunks based on specific patterns like paragraphs or sentences.
|
||||
|
||||
#### When to Use
|
||||
- Great for structured text with consistent delimiters.
|
||||
- Suitable for documents where specific patterns (e.g., double newlines, periods) indicate logical chunks.
|
||||
|
||||
#### Parameters
|
||||
- `patterns` (list, optional): Regular expressions used to split the text. Default is to split by double newlines (`['\n\n']`).
|
||||
|
||||
#### Example
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
# Define patterns for splitting text
|
||||
patterns = [r'\n\n', r'\. ']
|
||||
chunker = RegexChunking(patterns=patterns)
|
||||
|
||||
# Sample text
|
||||
text = "This is a sample text. It will be split into chunks.\n\nThis is another paragraph."
|
||||
|
||||
# Chunk the text
|
||||
chunks = chunker.chunk(text)
|
||||
print(chunks)
|
||||
```
|
||||
|
||||
### NlpSentenceChunking
|
||||
|
||||
`NlpSentenceChunking` uses NLP models to split text into sentences, ensuring accurate sentence boundaries.
|
||||
|
||||
#### When to Use
|
||||
- Ideal for texts where sentence boundaries are crucial.
|
||||
- Useful for creating chunks that preserve grammatical structures.
|
||||
|
||||
#### Parameters
|
||||
- None.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
chunker = NlpSentenceChunking()
|
||||
|
||||
# Sample text
|
||||
text = "This is a sample text. It will be split into sentences. Here's another sentence."
|
||||
|
||||
# Chunk the text
|
||||
chunks = chunker.chunk(text)
|
||||
print(chunks)
|
||||
```
|
||||
|
||||
### TopicSegmentationChunking
|
||||
|
||||
`TopicSegmentationChunking` employs the TextTiling algorithm to segment text into topic-based chunks. This method identifies thematic boundaries.
|
||||
|
||||
#### When to Use
|
||||
- Perfect for long documents with distinct topics.
|
||||
- Useful when preserving topic continuity is more important than maintaining text order.
|
||||
|
||||
#### Parameters
|
||||
- `num_keywords` (int, optional): Number of keywords for each topic segment. Default is `3`.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import TopicSegmentationChunking
|
||||
|
||||
chunker = TopicSegmentationChunking(num_keywords=3)
|
||||
|
||||
# Sample text
|
||||
text = "This document contains several topics. Topic one discusses AI. Topic two covers machine learning."
|
||||
|
||||
# Chunk the text
|
||||
chunks = chunker.chunk(text)
|
||||
print(chunks)
|
||||
```
|
||||
|
||||
### FixedLengthWordChunking
|
||||
|
||||
`FixedLengthWordChunking` splits text into chunks based on a fixed number of words. This ensures each chunk has approximately the same length.
|
||||
|
||||
#### When to Use
|
||||
- Suitable for processing large texts where uniform chunk size is important.
|
||||
- Useful when the number of words per chunk needs to be controlled.
|
||||
|
||||
#### Parameters
|
||||
- `chunk_size` (int, optional): Number of words per chunk. Default is `100`.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import FixedLengthWordChunking
|
||||
|
||||
chunker = FixedLengthWordChunking(chunk_size=10)
|
||||
|
||||
# Sample text
|
||||
text = "This is a sample text. It will be split into chunks of fixed length."
|
||||
|
||||
# Chunk the text
|
||||
chunks = chunker.chunk(text)
|
||||
print(chunks)
|
||||
```
|
||||
|
||||
### SlidingWindowChunking
|
||||
|
||||
`SlidingWindowChunking` uses a sliding window approach to create overlapping chunks. Each chunk has a fixed length, and the window slides by a specified step size.
|
||||
|
||||
#### When to Use
|
||||
- Ideal for creating overlapping chunks to preserve context.
|
||||
- Useful for tasks where context from adjacent chunks is needed.
|
||||
|
||||
#### Parameters
|
||||
- `window_size` (int, optional): Number of words in each chunk. Default is `100`.
|
||||
- `step` (int, optional): Number of words to slide the window. Default is `50`.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import SlidingWindowChunking
|
||||
|
||||
chunker = SlidingWindowChunking(window_size=10, step=5)
|
||||
|
||||
# Sample text
|
||||
text = "This is a sample text. It will be split using a sliding window approach to preserve context."
|
||||
|
||||
# Chunk the text
|
||||
chunks = chunker.chunk(text)
|
||||
print(chunks)
|
||||
```
|
||||
|
||||
With these chunking strategies, you can choose the best method to divide your text based on your specific needs. Whether you need precise sentence boundaries, topic-based segmentation, or uniform chunk sizes, Crawl4AI has you covered. Happy chunking! 📝✨
|
||||
@@ -1,179 +0,0 @@
|
||||
# Crawl Request Parameters for AsyncWebCrawler
|
||||
|
||||
The `arun` method in Crawl4AI's `AsyncWebCrawler` is designed to be highly configurable, allowing you to customize the crawling and extraction process to suit your needs. Below are the parameters you can use with the `arun` method, along with their descriptions, possible values, and examples.
|
||||
|
||||
## Parameters
|
||||
|
||||
### url (str)
|
||||
**Description:** The URL of the webpage to crawl.
|
||||
**Required:** Yes
|
||||
**Example:**
|
||||
```python
|
||||
url = "https://www.nbcnews.com/business"
|
||||
```
|
||||
|
||||
### word_count_threshold (int)
|
||||
**Description:** The minimum number of words a block must contain to be considered meaningful. The default value is defined by `MIN_WORD_THRESHOLD`.
|
||||
**Required:** No
|
||||
**Default Value:** `MIN_WORD_THRESHOLD`
|
||||
**Example:**
|
||||
```python
|
||||
word_count_threshold = 10
|
||||
```
|
||||
|
||||
### extraction_strategy (ExtractionStrategy)
|
||||
**Description:** The strategy to use for extracting content from the HTML. It must be an instance of `ExtractionStrategy`. If not provided, the default is `NoExtractionStrategy`.
|
||||
**Required:** No
|
||||
**Default Value:** `NoExtractionStrategy()`
|
||||
**Example:**
|
||||
```python
|
||||
extraction_strategy = CosineStrategy(semantic_filter="finance")
|
||||
```
|
||||
|
||||
### chunking_strategy (ChunkingStrategy)
|
||||
**Description:** The strategy to use for chunking the text before processing. It must be an instance of `ChunkingStrategy`. The default value is `RegexChunking()`.
|
||||
**Required:** No
|
||||
**Default Value:** `RegexChunking()`
|
||||
**Example:**
|
||||
```python
|
||||
chunking_strategy = NlpSentenceChunking()
|
||||
```
|
||||
|
||||
### bypass_cache (bool)
|
||||
**Description:** Whether to force a fresh crawl even if the URL has been previously crawled. The default value is `False`.
|
||||
**Required:** No
|
||||
**Default Value:** `False`
|
||||
**Example:**
|
||||
```python
|
||||
bypass_cache = True
|
||||
```
|
||||
|
||||
### css_selector (str)
|
||||
**Description:** The CSS selector to target specific parts of the HTML for extraction. If not provided, the entire HTML will be processed.
|
||||
**Required:** No
|
||||
**Default Value:** `None`
|
||||
**Example:**
|
||||
```python
|
||||
css_selector = "div.article-content"
|
||||
```
|
||||
|
||||
### screenshot (bool)
|
||||
**Description:** Whether to take screenshots of the page. The default value is `False`.
|
||||
**Required:** No
|
||||
**Default Value:** `False`
|
||||
**Example:**
|
||||
```python
|
||||
screenshot = True
|
||||
```
|
||||
|
||||
### user_agent (str)
|
||||
**Description:** The user agent to use for the HTTP requests. If not provided, a default user agent will be used.
|
||||
**Required:** No
|
||||
**Default Value:** `None`
|
||||
**Example:**
|
||||
```python
|
||||
user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
|
||||
```
|
||||
|
||||
### verbose (bool)
|
||||
**Description:** Whether to enable verbose logging. The default value is `True`.
|
||||
**Required:** No
|
||||
**Default Value:** `True`
|
||||
**Example:**
|
||||
```python
|
||||
verbose = True
|
||||
```
|
||||
|
||||
### **kwargs
|
||||
Additional keyword arguments that can be passed to customize the crawling process further. Some notable options include:
|
||||
|
||||
- **only_text (bool):** Whether to extract only text content, excluding HTML tags. Default is `False`.
|
||||
- **session_id (str):** A unique identifier for the crawling session. This is useful for maintaining state across multiple requests.
|
||||
- **js_code (str or list):** JavaScript code to be executed on the page before extraction.
|
||||
- **wait_for (str):** A CSS selector or JavaScript function to wait for before considering the page load complete.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
css_selector="p",
|
||||
only_text=True,
|
||||
session_id="unique_session_123",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="article.main-article"
|
||||
)
|
||||
```
|
||||
|
||||
## Example Usage
|
||||
|
||||
Here's an example of how to use the `arun` method with various parameters:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
from crawl4ai.chunking_strategy import NlpSentenceChunking
|
||||
|
||||
async def main():
|
||||
# Create the AsyncWebCrawler instance
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Run the crawler with custom parameters
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
word_count_threshold=10,
|
||||
extraction_strategy=CosineStrategy(semantic_filter="finance"),
|
||||
chunking_strategy=NlpSentenceChunking(),
|
||||
bypass_cache=True,
|
||||
css_selector="div.article-content",
|
||||
screenshot=True,
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
|
||||
verbose=True,
|
||||
only_text=True,
|
||||
session_id="business_news_session",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="footer"
|
||||
)
|
||||
|
||||
print(result)
|
||||
|
||||
# Run the async function
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
This example demonstrates how to configure various parameters to customize the crawling and extraction process using the asynchronous version of Crawl4AI.
|
||||
|
||||
## Additional Asynchronous Methods
|
||||
|
||||
The `AsyncWebCrawler` class also provides other useful asynchronous methods:
|
||||
|
||||
### arun_many
|
||||
**Description:** Crawl multiple URLs concurrently.
|
||||
**Example:**
|
||||
```python
|
||||
urls = ["https://example1.com", "https://example2.com", "https://example3.com"]
|
||||
results = await crawler.arun_many(urls, word_count_threshold=10, bypass_cache=True)
|
||||
```
|
||||
|
||||
### aclear_cache
|
||||
**Description:** Clear the crawler's cache.
|
||||
**Example:**
|
||||
```python
|
||||
await crawler.aclear_cache()
|
||||
```
|
||||
|
||||
### aflush_cache
|
||||
**Description:** Completely flush the crawler's cache.
|
||||
**Example:**
|
||||
```python
|
||||
await crawler.aflush_cache()
|
||||
```
|
||||
|
||||
### aget_cache_size
|
||||
**Description:** Get the current size of the cache.
|
||||
**Example:**
|
||||
```python
|
||||
cache_size = await crawler.aget_cache_size()
|
||||
print(f"Current cache size: {cache_size}")
|
||||
```
|
||||
|
||||
These asynchronous methods allow for efficient and flexible use of the AsyncWebCrawler in various scenarios.
|
||||
@@ -1,104 +0,0 @@
|
||||
# Crawl Result
|
||||
|
||||
The `CrawlResult` class is the heart of Crawl4AI's output, encapsulating all the data extracted from a crawling session. This class contains various fields that store the results of the web crawling and extraction process. Let's break down each field and see what it holds. 🎉
|
||||
|
||||
## Class Definition
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
class CrawlResult(BaseModel):
|
||||
url: str
|
||||
html: str
|
||||
success: bool
|
||||
cleaned_html: Optional[str] = None
|
||||
media: Dict[str, List[Dict]] = {}
|
||||
links: Dict[str, List[Dict]] = {}
|
||||
screenshot: Optional[str] = None
|
||||
markdown: Optional[str] = None
|
||||
extracted_content: Optional[str] = None
|
||||
metadata: Optional[dict] = None
|
||||
error_message: Optional[str] = None
|
||||
session_id: Optional[str] = None
|
||||
responser_headers: Optional[dict] = None
|
||||
status_code: Optional[int] = None
|
||||
```
|
||||
|
||||
## Fields Explanation
|
||||
|
||||
### `url: str`
|
||||
The URL that was crawled. This field simply stores the URL of the web page that was processed.
|
||||
|
||||
### `html: str`
|
||||
The raw HTML content of the web page. This is the unprocessed HTML source as retrieved by the crawler.
|
||||
|
||||
### `success: bool`
|
||||
A flag indicating whether the crawling and extraction were successful. If any error occurs during the process, this will be `False`.
|
||||
|
||||
### `cleaned_html: Optional[str]`
|
||||
The cleaned HTML content of the web page. This field holds the HTML after removing unwanted tags like `<script>`, `<style>`, and others that do not contribute to the useful content.
|
||||
|
||||
### `media: Dict[str, List[Dict]]`
|
||||
A dictionary containing lists of extracted media elements from the web page. The media elements are categorized into images, videos, and audios. Here's how they are structured:
|
||||
|
||||
- **Images**: Each image is represented as a dictionary with `src` (source URL) and `alt` (alternate text).
|
||||
- **Videos**: Each video is represented similarly with `src` and `alt`.
|
||||
- **Audios**: Each audio is represented with `src` and `alt`.
|
||||
|
||||
```python
|
||||
media = {
|
||||
'images': [
|
||||
{'src': 'image_url1', 'alt': 'description1', "type": "image"},
|
||||
{'src': 'image_url2', 'alt': 'description2', "type": "image"}
|
||||
],
|
||||
'videos': [
|
||||
{'src': 'video_url1', 'alt': 'description1', "type": "video"}
|
||||
],
|
||||
'audios': [
|
||||
{'src': 'audio_url1', 'alt': 'description1', "type": "audio"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### `links: Dict[str, List[Dict]]`
|
||||
A dictionary containing lists of internal and external links extracted from the web page. Each link is represented as a dictionary with `href` (URL) and `text` (link text).
|
||||
|
||||
- **Internal Links**: Links pointing to the same domain.
|
||||
- **External Links**: Links pointing to different domains.
|
||||
|
||||
```python
|
||||
links = {
|
||||
'internal': [
|
||||
{'href': 'internal_link1', 'text': 'link_text1'},
|
||||
{'href': 'internal_link2', 'text': 'link_text2'}
|
||||
],
|
||||
'external': [
|
||||
{'href': 'external_link1', 'text': 'link_text1'}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### `screenshot: Optional[str]`
|
||||
A base64-encoded screenshot of the web page. This field stores the screenshot data if the crawling was configured to take a screenshot.
|
||||
|
||||
### `markdown: Optional[str]`
|
||||
The content of the web page converted to Markdown format. This is useful for generating clean, readable text that retains the structure of the original HTML.
|
||||
|
||||
### `extracted_content: Optional[str]`
|
||||
The content extracted based on the specified extraction strategy. This field holds the meaningful content blocks extracted from the web page, ready for your AI and data processing needs.
|
||||
|
||||
### `metadata: Optional[dict]`
|
||||
A dictionary containing metadata extracted from the web page, such as title, description, keywords, and other meta tags.
|
||||
|
||||
### `error_message: Optional[str]`
|
||||
If an error occurs during crawling, this field will contain the error message, helping you debug and understand what went wrong. 🚨
|
||||
|
||||
### `session_id: Optional[str]`
|
||||
A unique identifier for the crawling session. This can be useful for tracking and managing multiple crawling sessions.
|
||||
|
||||
### `responser_headers: Optional[dict]`
|
||||
A dictionary containing the response headers from the web server. This can provide additional information about the server and the response.
|
||||
|
||||
### `status_code: Optional[int]`
|
||||
The HTTP status code of the response. This indicates the success or failure of the HTTP request (e.g., 200 for success, 404 for not found, etc.).
|
||||
@@ -1,185 +0,0 @@
|
||||
## Extraction Strategies 🧠
|
||||
|
||||
Crawl4AI offers powerful extraction strategies to derive meaningful information from web content. Let's dive into three of the most important strategies: `CosineStrategy`, `LLMExtractionStrategy`, and the new `JsonCssExtractionStrategy`.
|
||||
|
||||
### LLMExtractionStrategy
|
||||
|
||||
`LLMExtractionStrategy` leverages a Language Model (LLM) to extract meaningful content from HTML. This strategy uses an external provider for LLM completions to perform extraction based on instructions.
|
||||
|
||||
#### When to Use
|
||||
- Suitable for complex extraction tasks requiring nuanced understanding.
|
||||
- Ideal for scenarios where detailed instructions can guide the extraction process.
|
||||
- Perfect for extracting specific types of information or content with precise guidelines.
|
||||
|
||||
#### Parameters
|
||||
- `provider` (str, optional): Provider for language model completions (e.g., openai/gpt-4). Default is `DEFAULT_PROVIDER`.
|
||||
- `api_token` (str, optional): API token for the provider. If not provided, it will try to load from the environment variable `OPENAI_API_KEY`.
|
||||
- `instruction` (str, optional): Instructions to guide the LLM on how to perform the extraction. Default is `None`.
|
||||
|
||||
#### Example Without Instructions
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Define extraction strategy without instructions
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider='openai',
|
||||
api_token=os.getenv('OPENAI_API_KEY')
|
||||
)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = await crawler.arun(url=url, extraction_strategy=strategy)
|
||||
print(result.extracted_content)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
#### Example With Instructions
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Define extraction strategy with instructions
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider='openai',
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
instruction="Extract only financial news and summarize key points."
|
||||
)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = await crawler.arun(url=url, extraction_strategy=strategy)
|
||||
print(result.extracted_content)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### JsonCssExtractionStrategy
|
||||
|
||||
`JsonCssExtractionStrategy` is a powerful tool for extracting structured data from HTML using CSS selectors. It allows you to define a schema that maps CSS selectors to specific fields, enabling precise and efficient data extraction.
|
||||
|
||||
#### When to Use
|
||||
- Ideal for extracting structured data from websites with consistent HTML structures.
|
||||
- Perfect for scenarios where you need to extract specific elements or attributes from a webpage.
|
||||
- Suitable for creating datasets from web pages with tabular or list-based information.
|
||||
|
||||
#### Parameters
|
||||
- `schema` (Dict[str, Any]): A dictionary defining the extraction schema, including base selector and field definitions.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
import asyncio
|
||||
import json
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Define the extraction schema
|
||||
schema = {
|
||||
"name": "News Articles",
|
||||
"baseSelector": "article.tease-card",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h2",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "summary",
|
||||
"selector": "div.tease-card__info",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "link",
|
||||
"selector": "a",
|
||||
"type": "attribute",
|
||||
"attribute": "href"
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
# Create the extraction strategy
|
||||
strategy = JsonCssExtractionStrategy(schema, verbose=True)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = await crawler.arun(url=url, extraction_strategy=strategy)
|
||||
|
||||
# Parse and print the extracted content
|
||||
extracted_data = json.loads(result.extracted_content)
|
||||
print(json.dumps(extracted_data, indent=2))
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
#### Use Cases for JsonCssExtractionStrategy
|
||||
- Extracting product information from e-commerce websites.
|
||||
- Gathering news articles and their metadata from news portals.
|
||||
- Collecting user reviews and ratings from review websites.
|
||||
- Extracting job listings from job boards.
|
||||
|
||||
By choosing the right extraction strategy, you can effectively extract the most relevant and useful information from web content. Whether you need fast, accurate semantic segmentation with `CosineStrategy`, nuanced, instruction-based extraction with `LLMExtractionStrategy`, or precise structured data extraction with `JsonCssExtractionStrategy`, Crawl4AI has you covered. Happy extracting! 🕵️♂️✨
|
||||
|
||||
For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](../full_details/advanced_jsoncss_extraction.md).
|
||||
|
||||
|
||||
### CosineStrategy
|
||||
|
||||
`CosineStrategy` uses hierarchical clustering based on cosine similarity to group text chunks into meaningful clusters. This method converts each chunk into its embedding and then clusters them to form semantical chunks.
|
||||
|
||||
#### When to Use
|
||||
- Ideal for fast, accurate semantic segmentation of text.
|
||||
- Perfect for scenarios where LLMs might be overkill or too slow.
|
||||
- Suitable for narrowing down content based on specific queries or keywords.
|
||||
|
||||
#### Parameters
|
||||
- `semantic_filter` (str, optional): Keywords for filtering relevant documents before clustering. Documents are filtered based on their cosine similarity to the keyword filter embedding. Default is `None`.
|
||||
- `word_count_threshold` (int, optional): Minimum number of words per cluster. Default is `20`.
|
||||
- `max_dist` (float, optional): Maximum cophenetic distance on the dendrogram to form clusters. Default is `0.2`.
|
||||
- `linkage_method` (str, optional): Linkage method for hierarchical clustering. Default is `'ward'`.
|
||||
- `top_k` (int, optional): Number of top categories to extract. Default is `3`.
|
||||
- `model_name` (str, optional): Model name for embedding generation. Default is `'BAAI/bge-small-en-v1.5'`.
|
||||
|
||||
#### Example
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Define extraction strategy
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="finance economy stock market",
|
||||
word_count_threshold=10,
|
||||
max_dist=0.2,
|
||||
linkage_method='ward',
|
||||
top_k=3,
|
||||
model_name='BAAI/bge-small-en-v1.5'
|
||||
)
|
||||
|
||||
# Sample URL
|
||||
url = "https://www.nbcnews.com/business"
|
||||
|
||||
# Run the crawler with the extraction strategy
|
||||
result = await crawler.arun(url=url, extraction_strategy=strategy)
|
||||
print(result.extracted_content)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
@@ -1,93 +0,0 @@
|
||||
# Crawl4AI
|
||||
|
||||
Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
|
||||
|
||||
## Introduction
|
||||
|
||||
Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution with full asynchronous support.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Here's a quick example to show you how easy it is to use Crawl4AI with its new asynchronous capabilities:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
# Create an instance of AsyncWebCrawler
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Run the crawler on a URL
|
||||
result = await crawler.arun(url="https://www.nbcnews.com/business")
|
||||
|
||||
# Print the extracted content
|
||||
print(result.markdown)
|
||||
|
||||
# Run the async main function
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Explanation
|
||||
|
||||
1. **Importing the Library**: We start by importing the `AsyncWebCrawler` class from the `crawl4ai` library and the `asyncio` module.
|
||||
2. **Creating an Async Context**: We use an async context manager to create an instance of `AsyncWebCrawler`.
|
||||
3. **Running the Crawler**: The `arun()` method is used to asynchronously crawl the specified URL and extract meaningful content.
|
||||
4. **Printing the Result**: The extracted content is printed, showcasing the data extracted from the web page.
|
||||
5. **Running the Async Function**: We use `asyncio.run()` to execute our async main function.
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
This documentation is organized into several sections to help you navigate and find the information you need quickly:
|
||||
|
||||
### [Home](index.md)
|
||||
|
||||
An introduction to Crawl4AI, including a quick start guide and an overview of the documentation structure.
|
||||
|
||||
### [Installation](installation.md)
|
||||
|
||||
Instructions on how to install Crawl4AI and its dependencies.
|
||||
|
||||
### [Introduction](introduction.md)
|
||||
|
||||
A detailed introduction to Crawl4AI, its features, and how it can be used for various web crawling and data extraction tasks.
|
||||
|
||||
### [Quick Start](quickstart.md)
|
||||
|
||||
A step-by-step guide to get you up and running with Crawl4AI, including installation instructions and basic usage examples.
|
||||
|
||||
### [Examples](examples/index.md)
|
||||
|
||||
This section contains practical examples demonstrating different use cases of Crawl4AI:
|
||||
|
||||
- [Structured Data Extraction](examples/json_css_extraction.md)
|
||||
- [LLM Extraction](examples/llm_extraction.md)
|
||||
- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
|
||||
- [Hooks & Auth](examples/hooks_auth.md)
|
||||
- [Summarization](examples/summarization.md)
|
||||
- [Research Assistant](examples/research_assistant.md)
|
||||
|
||||
### [Full Details of Using Crawler](full_details/crawl_request_parameters.md)
|
||||
|
||||
Comprehensive details on using the crawler, including:
|
||||
|
||||
- [Crawl Request Parameters](full_details/crawl_request_parameters.md)
|
||||
- [Crawl Result Class](full_details/crawl_result_class.md)
|
||||
- [Session Based Crawling](full_details/session_based_crawling.md)
|
||||
- [Advanced Structured Data Extraction JsonCssExtraction](full_details/advanced_jsoncss_extraction.md)
|
||||
- [Advanced Features](full_details/advanced_features.md)
|
||||
- [Chunking Strategies](full_details/chunking_strategies.md)
|
||||
- [Extraction Strategies](full_details/extraction_strategies.md)
|
||||
|
||||
### [Change Log](changelog.md)
|
||||
|
||||
A log of all changes, updates, and improvements made to Crawl4AI.
|
||||
|
||||
### [Contact](contact.md)
|
||||
|
||||
Information on how to get in touch with the developers, report issues, and contribute to the project.
|
||||
|
||||
## Get Started
|
||||
|
||||
To get started with Crawl4AI, follow the quick start guide above or explore the detailed sections of this documentation. Whether you are a beginner or an advanced user, Crawl4AI has something to offer to make your web crawling and data extraction tasks easier, more efficient, and now fully asynchronous.
|
||||
|
||||
Happy Crawling! 🕸️🚀
|
||||
@@ -1,28 +0,0 @@
|
||||
<h1>Try Our Library</h1>
|
||||
<form id="apiForm">
|
||||
<label for="inputField">Enter some input:</label>
|
||||
<input type="text" id="inputField" name="inputField" required>
|
||||
<button type="submit">Submit</button>
|
||||
</form>
|
||||
<div id="result"></div>
|
||||
|
||||
<script>
|
||||
document.getElementById('apiForm').addEventListener('submit', function(event) {
|
||||
event.preventDefault();
|
||||
const input = document.getElementById('inputField').value;
|
||||
fetch('https://your-api-endpoint.com/api', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({ input: input })
|
||||
})
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
document.getElementById('result').textContent = JSON.stringify(data);
|
||||
})
|
||||
.catch(error => {
|
||||
document.getElementById('result').textContent = 'Error: ' + error;
|
||||
});
|
||||
});
|
||||
</script>
|
||||
@@ -1,29 +0,0 @@
|
||||
# Introduction
|
||||
|
||||
Welcome to the documentation for Crawl4AI v0.2.5! 🕷️🤖
|
||||
|
||||
Crawl4AI is designed to simplify the process of crawling web pages and extracting useful information for large language models (LLMs) and AI applications. Whether you're using it as a REST API, a Python library, or through a Google Colab notebook, Crawl4AI provides powerful features to make web data extraction easier and more efficient.
|
||||
|
||||
## Key Features ✨
|
||||
|
||||
- **🆓 Completely Free and Open-Source**: Crawl4AI is free to use and open-source, making it accessible for everyone.
|
||||
- **🤖 LLM-Friendly Output Formats**: Supports JSON, cleaned HTML, and markdown formats.
|
||||
- **🌍 Concurrent Crawling**: Crawl multiple URLs simultaneously to save time.
|
||||
- **🎨 Media Extraction**: Extract all media tags including images, audio, and video.
|
||||
- **🔗 Link Extraction**: Extract all external and internal links from web pages.
|
||||
- **📚 Metadata Extraction**: Extract metadata from web pages for additional context.
|
||||
- **🔄 Custom Hooks**: Define custom hooks for authentication, headers, and page modifications before crawling.
|
||||
- **🕵️ User Agent Support**: Customize the user agent for HTTP requests.
|
||||
- **🖼️ Screenshot Capability**: Take screenshots of web pages during crawling.
|
||||
- **📜 JavaScript Execution**: Execute custom JavaScripts before crawling.
|
||||
- **📚 Advanced Chunking and Extraction Strategies**: Utilize topic-based, regex, sentence chunking, cosine clustering, and LLM extraction strategies.
|
||||
- **🎯 CSS Selector Support**: Extract specific content using CSS selectors.
|
||||
- **📝 Instruction/Keyword Refinement**: Pass instructions or keywords to refine the extraction process.
|
||||
|
||||
Check the [Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for more details.
|
||||
|
||||
## Power and Simplicity of Crawl4AI 🚀
|
||||
|
||||
Crawl4AI provides an easy way to crawl and extract data from web pages without installing any library. You can use the REST API on our server or run the local server on your machine. For more advanced control, use the Python library to customize your crawling and extraction strategies.
|
||||
|
||||
Explore the documentation to learn more about the features, installation process, usage examples, and how to contribute to Crawl4AI. Let's make the web more accessible and useful for AI applications! 💪🌐🤖
|
||||
@@ -1,285 +0,0 @@
|
||||
# Quick Start Guide 🚀
|
||||
|
||||
Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies, all with the power of asynchronous programming. Let's dive in! 🌟
|
||||
|
||||
## Getting Started 🛠️
|
||||
|
||||
First, let's import the necessary modules and create an instance of `AsyncWebCrawler`. We'll use an async context manager, which handles the setup and teardown of the crawler for us.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# We'll add our crawling code here
|
||||
pass
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simply provide a URL and let Crawl4AI do the magic!
|
||||
|
||||
```python
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.nbcnews.com/business")
|
||||
print(f"Basic crawl result: {result.markdown[:500]}") # Print first 500 characters
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Taking Screenshots 📸
|
||||
|
||||
Let's take a screenshot of the page!
|
||||
|
||||
```python
|
||||
import base64
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.nbcnews.com/business", screenshot=True)
|
||||
with open("screenshot.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
print("Screenshot saved to 'screenshot.png'!")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Understanding Parameters 🧠
|
||||
|
||||
By default, Crawl4AI caches the results of your crawls. This means that subsequent crawls of the same URL will be much faster! Let's see this in action.
|
||||
|
||||
```python
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# First crawl (caches the result)
|
||||
result1 = await crawler.arun(url="https://www.nbcnews.com/business")
|
||||
print(f"First crawl result: {result1.markdown[:100]}...")
|
||||
|
||||
# Force to crawl again
|
||||
result2 = await crawler.arun(url="https://www.nbcnews.com/business", bypass_cache=True)
|
||||
print(f"Second crawl result: {result2.markdown[:100]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Adding a Chunking Strategy 🧩
|
||||
|
||||
Let's add a chunking strategy: `RegexChunking`! This strategy splits the text based on a given regex pattern.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
chunking_strategy=RegexChunking(patterns=["\n\n"])
|
||||
)
|
||||
print(f"RegexChunking result: {result.extracted_content[:200]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Adding an Extraction Strategy 🧠
|
||||
|
||||
Let's get smarter with an extraction strategy: `JsonCssExtractionStrategy`! This strategy extracts structured data from HTML using CSS selectors.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
import json
|
||||
|
||||
async def main():
|
||||
schema = {
|
||||
"name": "News Articles",
|
||||
"baseSelector": "article.tease-card",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h2",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "summary",
|
||||
"selector": "div.tease-card__info",
|
||||
"type": "text",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
extraction_strategy=JsonCssExtractionStrategy(schema, verbose=True)
|
||||
)
|
||||
extracted_data = json.loads(result.extracted_content)
|
||||
print(f"Extracted {len(extracted_data)} articles")
|
||||
print(json.dumps(extracted_data[0], indent=2))
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Using LLMExtractionStrategy 🤖
|
||||
|
||||
Time to bring in the big guns: `LLMExtractionStrategy`! This strategy uses a large language model to extract relevant information from the web page.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
import os
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
class OpenAIModelFee(BaseModel):
|
||||
model_name: str = Field(..., description="Name of the OpenAI model.")
|
||||
input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
|
||||
output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
|
||||
|
||||
async def main():
|
||||
if not os.getenv("OPENAI_API_KEY"):
|
||||
print("OpenAI API key not found. Skipping this example.")
|
||||
return
|
||||
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://openai.com/api/pricing/",
|
||||
word_count_threshold=1,
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
api_token=os.getenv("OPENAI_API_KEY"),
|
||||
schema=OpenAIModelFee.schema(),
|
||||
extraction_type="schema",
|
||||
instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens.
|
||||
Do not miss any models in the entire content. One extracted model JSON format should look like this:
|
||||
{"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}.""",
|
||||
),
|
||||
bypass_cache=True,
|
||||
)
|
||||
print(result.extracted_content)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Interactive Extraction 🖱️
|
||||
|
||||
Let's use JavaScript to interact with the page before extraction!
|
||||
|
||||
```python
|
||||
async def main():
|
||||
js_code = """
|
||||
const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More'));
|
||||
loadMoreButton && loadMoreButton.click();
|
||||
"""
|
||||
|
||||
wait_for = """() => {
|
||||
return Array.from(document.querySelectorAll('article.tease-card')).length > 10;
|
||||
}"""
|
||||
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
js_code=js_code,
|
||||
wait_for=wait_for,
|
||||
css_selector="article.tease-card",
|
||||
bypass_cache=True,
|
||||
)
|
||||
print(f"JavaScript interaction result: {result.extracted_content[:500]}")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Advanced Session-Based Crawling with Dynamic Content 🔄
|
||||
|
||||
In modern web applications, content is often loaded dynamically without changing the URL. This is common in single-page applications (SPAs) or websites using infinite scrolling. Traditional crawling methods that rely on URL changes won't work here. That's where Crawl4AI's advanced session-based crawling comes in handy!
|
||||
|
||||
Here's what makes this approach powerful:
|
||||
|
||||
1. **Session Preservation**: By using a `session_id`, we can maintain the state of our crawling session across multiple interactions with the page. This is crucial for navigating through dynamically loaded content.
|
||||
|
||||
2. **Asynchronous JavaScript Execution**: We can execute custom JavaScript to trigger content loading or navigation. In this example, we'll click a "Load More" button to fetch the next page of commits.
|
||||
|
||||
3. **Dynamic Content Waiting**: The `wait_for` parameter allows us to specify a condition that must be met before considering the page load complete. This ensures we don't extract data before the new content is fully loaded.
|
||||
|
||||
Let's see how this works with a real-world example: crawling multiple pages of commits on a GitHub repository. The URL doesn't change as we load more commits, so we'll use these advanced techniques to navigate and extract data.
|
||||
|
||||
```python
|
||||
import json
|
||||
from bs4 import BeautifulSoup
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
url = "https://github.com/microsoft/TypeScript/commits/main"
|
||||
session_id = "typescript_commits_session"
|
||||
all_commits = []
|
||||
|
||||
js_next_page = """
|
||||
const button = document.querySelector('a[data-testid="pagination-next-button"]');
|
||||
if (button) button.click();
|
||||
"""
|
||||
|
||||
wait_for = """() => {
|
||||
const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
|
||||
if (commits.length === 0) return false;
|
||||
const firstCommit = commits[0].textContent.trim();
|
||||
return firstCommit !== window.lastCommit;
|
||||
}"""
|
||||
|
||||
schema = {
|
||||
"name": "Commit Extractor",
|
||||
"baseSelector": "li.Box-sc-g0xbh4-0",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h4.markdown-title",
|
||||
"type": "text",
|
||||
"transform": "strip",
|
||||
},
|
||||
],
|
||||
}
|
||||
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
|
||||
|
||||
for page in range(3): # Crawl 3 pages
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
session_id=session_id,
|
||||
css_selector="li.Box-sc-g0xbh4-0",
|
||||
extraction_strategy=extraction_strategy,
|
||||
js_code=js_next_page if page > 0 else None,
|
||||
wait_for=wait_for if page > 0 else None,
|
||||
js_only=page > 0,
|
||||
bypass_cache=True,
|
||||
headless=False,
|
||||
)
|
||||
|
||||
assert result.success, f"Failed to crawl page {page + 1}"
|
||||
|
||||
commits = json.loads(result.extracted_content)
|
||||
all_commits.extend(commits)
|
||||
|
||||
print(f"Page {page + 1}: Found {len(commits)} commits")
|
||||
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
In this example, we're crawling multiple pages of commits from a GitHub repository. The URL doesn't change as we load more commits, so we use JavaScript to click the "Load More" button and a `wait_for` condition to ensure the new content is loaded before extraction. This powerful combination allows us to navigate and extract data from complex, dynamically-loaded web applications with ease!
|
||||
|
||||
## Congratulations! 🎉
|
||||
|
||||
You've made it through the Crawl4AI Quickstart Guide! Now go forth and crawl the web asynchronously like a pro! 🕸️
|
||||
|
||||
Remember, these are just a few examples of what Crawl4AI can do. For more advanced usage, check out our other documentation pages:
|
||||
|
||||
- [LLM Extraction](examples/llm_extraction.md)
|
||||
- [JS Execution & CSS Filtering](examples/js_execution_css_filtering.md)
|
||||
- [Hooks & Auth](examples/hooks_auth.md)
|
||||
- [Summarization](examples/summarization.md)
|
||||
- [Research Assistant](examples/research_assistant.md)
|
||||
|
||||
Happy crawling! 🚀
|
||||
223
docs/md_v2/advanced/content-processing.md
Normal file
223
docs/md_v2/advanced/content-processing.md
Normal file
@@ -0,0 +1,223 @@
|
||||
# Content Processing
|
||||
|
||||
Crawl4AI provides powerful content processing capabilities that help you extract clean, relevant content from web pages. This guide covers content cleaning, media handling, link analysis, and metadata extraction.
|
||||
|
||||
## Content Cleaning
|
||||
|
||||
### Understanding Clean Content
|
||||
When crawling web pages, you often encounter a lot of noise - advertisements, navigation menus, footers, popups, and other irrelevant content. Crawl4AI automatically cleans this noise using several approaches:
|
||||
|
||||
1. **Basic Cleaning**: Removes unwanted HTML elements and attributes
|
||||
2. **Content Relevance**: Identifies and preserves meaningful content blocks
|
||||
3. **Layout Analysis**: Understands page structure to identify main content areas
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
word_count_threshold=10, # Remove blocks with fewer words
|
||||
excluded_tags=['form', 'nav'], # Remove specific HTML tags
|
||||
remove_overlay_elements=True # Remove popups/modals
|
||||
)
|
||||
|
||||
# Get clean content
|
||||
print(result.cleaned_html) # Cleaned HTML
|
||||
print(result.markdown) # Clean markdown version
|
||||
```
|
||||
|
||||
### Fit Markdown: Smart Content Extraction
|
||||
One of Crawl4AI's most powerful features is `fit_markdown`. This feature uses advanced heuristics to identify and extract the main content from a webpage while excluding irrelevant elements.
|
||||
|
||||
#### How Fit Markdown Works
|
||||
- Analyzes content density and distribution
|
||||
- Identifies content patterns and structures
|
||||
- Removes boilerplate content (headers, footers, sidebars)
|
||||
- Preserves the most relevant content blocks
|
||||
- Maintains content hierarchy and formatting
|
||||
|
||||
#### Perfect For:
|
||||
- Blog posts and articles
|
||||
- News content
|
||||
- Documentation pages
|
||||
- Any page with a clear main content area
|
||||
|
||||
#### Not Recommended For:
|
||||
- E-commerce product listings
|
||||
- Search results pages
|
||||
- Social media feeds
|
||||
- Pages with multiple equal-weight content sections
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Get the most relevant content
|
||||
main_content = result.fit_markdown
|
||||
|
||||
# Compare with regular markdown
|
||||
all_content = result.markdown
|
||||
|
||||
print(f"Fit Markdown Length: {len(main_content)}")
|
||||
print(f"Regular Markdown Length: {len(all_content)}")
|
||||
```
|
||||
|
||||
#### Example Use Case
|
||||
```python
|
||||
async def extract_article_content(url: str) -> str:
|
||||
"""Extract main article content from a blog or news site."""
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url=url)
|
||||
|
||||
# fit_markdown will focus on the article content,
|
||||
# excluding navigation, ads, and other distractions
|
||||
return result.fit_markdown
|
||||
```
|
||||
|
||||
## Media Processing
|
||||
|
||||
Crawl4AI provides comprehensive media extraction and analysis capabilities. It automatically detects and processes various types of media elements while maintaining their context and relevance.
|
||||
|
||||
### Image Processing
|
||||
The library handles various image scenarios, including:
|
||||
- Regular images
|
||||
- Lazy-loaded images
|
||||
- Background images
|
||||
- Responsive images
|
||||
- Image metadata and context
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
for image in result.media["images"]:
|
||||
# Each image includes rich metadata
|
||||
print(f"Source: {image['src']}")
|
||||
print(f"Alt text: {image['alt']}")
|
||||
print(f"Description: {image['desc']}")
|
||||
print(f"Context: {image['context']}") # Surrounding text
|
||||
print(f"Relevance score: {image['score']}") # 0-10 score
|
||||
```
|
||||
|
||||
### Handling Lazy-Loaded Content
|
||||
Crawl4aai already handles lazy loading for media elements. You can also customize the wait time for lazy-loaded content:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
wait_for="css:img[data-src]", # Wait for lazy images
|
||||
delay_before_return_html=2.0 # Additional wait time
|
||||
)
|
||||
```
|
||||
|
||||
### Video and Audio Content
|
||||
The library extracts video and audio elements with their metadata:
|
||||
|
||||
```python
|
||||
# Process videos
|
||||
for video in result.media["videos"]:
|
||||
print(f"Video source: {video['src']}")
|
||||
print(f"Type: {video['type']}")
|
||||
print(f"Duration: {video.get('duration')}")
|
||||
print(f"Thumbnail: {video.get('poster')}")
|
||||
|
||||
# Process audio
|
||||
for audio in result.media["audios"]:
|
||||
print(f"Audio source: {audio['src']}")
|
||||
print(f"Type: {audio['type']}")
|
||||
print(f"Duration: {audio.get('duration')}")
|
||||
```
|
||||
|
||||
## Link Analysis
|
||||
|
||||
Crawl4AI provides sophisticated link analysis capabilities, helping you understand the relationship between pages and identify important navigation patterns.
|
||||
|
||||
### Link Classification
|
||||
The library automatically categorizes links into:
|
||||
- Internal links (same domain)
|
||||
- External links (different domains)
|
||||
- Social media links
|
||||
- Navigation links
|
||||
- Content links
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Analyze internal links
|
||||
for link in result.links["internal"]:
|
||||
print(f"Internal: {link['href']}")
|
||||
print(f"Link text: {link['text']}")
|
||||
print(f"Context: {link['context']}") # Surrounding text
|
||||
print(f"Type: {link['type']}") # nav, content, etc.
|
||||
|
||||
# Analyze external links
|
||||
for link in result.links["external"]:
|
||||
print(f"External: {link['href']}")
|
||||
print(f"Domain: {link['domain']}")
|
||||
print(f"Type: {link['type']}")
|
||||
```
|
||||
|
||||
### Smart Link Filtering
|
||||
Control which links are included in the results:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
exclude_external_links=True, # Remove external links
|
||||
exclude_social_media_links=True, # Remove social media links
|
||||
exclude_social_media_domains=[ # Custom social media domains
|
||||
"facebook.com", "twitter.com", "instagram.com"
|
||||
],
|
||||
exclude_domains=["ads.example.com"] # Exclude specific domains
|
||||
)
|
||||
```
|
||||
|
||||
## Metadata Extraction
|
||||
|
||||
Crawl4AI automatically extracts and processes page metadata, providing valuable information about the content:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
metadata = result.metadata
|
||||
print(f"Title: {metadata['title']}")
|
||||
print(f"Description: {metadata['description']}")
|
||||
print(f"Keywords: {metadata['keywords']}")
|
||||
print(f"Author: {metadata['author']}")
|
||||
print(f"Published Date: {metadata['published_date']}")
|
||||
print(f"Modified Date: {metadata['modified_date']}")
|
||||
print(f"Language: {metadata['language']}")
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use Fit Markdown for Articles**
|
||||
```python
|
||||
# Perfect for blog posts, news articles, documentation
|
||||
content = result.fit_markdown
|
||||
```
|
||||
|
||||
2. **Handle Media Appropriately**
|
||||
```python
|
||||
# Filter by relevance score
|
||||
relevant_images = [
|
||||
img for img in result.media["images"]
|
||||
if img['score'] > 5
|
||||
]
|
||||
```
|
||||
|
||||
3. **Combine Link Analysis with Content**
|
||||
```python
|
||||
# Get content links with context
|
||||
content_links = [
|
||||
link for link in result.links["internal"]
|
||||
if link['type'] == 'content'
|
||||
]
|
||||
```
|
||||
|
||||
4. **Clean Content with Purpose**
|
||||
```python
|
||||
# Customize cleaning based on your needs
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
word_count_threshold=20, # Adjust based on content type
|
||||
keep_data_attributes=False, # Remove data attributes
|
||||
process_iframes=True # Include iframe content
|
||||
)
|
||||
```
|
||||
52
docs/md_v2/advanced/magic-mode.md
Normal file
52
docs/md_v2/advanced/magic-mode.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# Magic Mode & Anti-Bot Protection
|
||||
|
||||
Crawl4AI provides powerful anti-detection capabilities, with Magic Mode being the simplest and most comprehensive solution.
|
||||
|
||||
## Magic Mode
|
||||
|
||||
The easiest way to bypass anti-bot protections:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
magic=True # Enables all anti-detection features
|
||||
)
|
||||
```
|
||||
|
||||
Magic Mode automatically:
|
||||
- Masks browser automation signals
|
||||
- Simulates human-like behavior
|
||||
- Overrides navigator properties
|
||||
- Handles cookie consent popups
|
||||
- Manages browser fingerprinting
|
||||
- Randomizes timing patterns
|
||||
|
||||
## Manual Anti-Bot Options
|
||||
|
||||
While Magic Mode is recommended, you can also configure individual anti-detection features:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
simulate_user=True, # Simulate human behavior
|
||||
override_navigator=True # Mask automation signals
|
||||
)
|
||||
```
|
||||
|
||||
Note: When `magic=True` is used, you don't need to set these individual options.
|
||||
|
||||
## Example: Handling Protected Sites
|
||||
|
||||
```python
|
||||
async def crawl_protected_site(url: str):
|
||||
async with AsyncWebCrawler(headless=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
magic=True,
|
||||
remove_overlay_elements=True, # Remove popups/modals
|
||||
page_timeout=60000 # Increased timeout for protection checks
|
||||
)
|
||||
|
||||
return result.markdown if result.success else None
|
||||
```
|
||||
84
docs/md_v2/advanced/proxy-security.md
Normal file
84
docs/md_v2/advanced/proxy-security.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Proxy & Security
|
||||
|
||||
Configure proxy settings and enhance security features in Crawl4AI for reliable data extraction.
|
||||
|
||||
## Basic Proxy Setup
|
||||
|
||||
Simple proxy configuration:
|
||||
|
||||
```python
|
||||
# Using proxy URL
|
||||
async with AsyncWebCrawler(
|
||||
proxy="http://proxy.example.com:8080"
|
||||
) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Using SOCKS proxy
|
||||
async with AsyncWebCrawler(
|
||||
proxy="socks5://proxy.example.com:1080"
|
||||
) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Authenticated Proxy
|
||||
|
||||
Use proxy with authentication:
|
||||
|
||||
```python
|
||||
proxy_config = {
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
}
|
||||
|
||||
async with AsyncWebCrawler(proxy_config=proxy_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Rotating Proxies
|
||||
|
||||
Example using a proxy rotation service:
|
||||
|
||||
```python
|
||||
async def get_next_proxy():
|
||||
# Your proxy rotation logic here
|
||||
return {"server": "http://next.proxy.com:8080"}
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Update proxy for each request
|
||||
for url in urls:
|
||||
proxy = await get_next_proxy()
|
||||
crawler.update_proxy(proxy)
|
||||
result = await crawler.arun(url=url)
|
||||
```
|
||||
|
||||
## Custom Headers
|
||||
|
||||
Add security-related headers:
|
||||
|
||||
```python
|
||||
headers = {
|
||||
"X-Forwarded-For": "203.0.113.195",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Cache-Control": "no-cache",
|
||||
"Pragma": "no-cache"
|
||||
}
|
||||
|
||||
async with AsyncWebCrawler(headers=headers) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Combining with Magic Mode
|
||||
|
||||
For maximum protection, combine proxy with Magic Mode:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler(
|
||||
proxy="http://proxy.example.com:8080",
|
||||
headers={"Accept-Language": "en-US"}
|
||||
) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
magic=True # Enable all anti-detection features
|
||||
)
|
||||
```
|
||||
133
docs/md_v2/advanced/session-management.md
Normal file
133
docs/md_v2/advanced/session-management.md
Normal file
@@ -0,0 +1,133 @@
|
||||
# Session Management
|
||||
|
||||
Session management in Crawl4AI allows you to maintain state across multiple requests and handle complex multi-page crawling tasks, particularly useful for dynamic websites.
|
||||
|
||||
## Basic Session Usage
|
||||
|
||||
Use `session_id` to maintain state between requests:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
session_id = "my_session"
|
||||
|
||||
# First request
|
||||
result1 = await crawler.arun(
|
||||
url="https://example.com/page1",
|
||||
session_id=session_id
|
||||
)
|
||||
|
||||
# Subsequent request using same session
|
||||
result2 = await crawler.arun(
|
||||
url="https://example.com/page2",
|
||||
session_id=session_id
|
||||
)
|
||||
|
||||
# Clean up when done
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
```
|
||||
|
||||
## Dynamic Content with Sessions
|
||||
|
||||
Here's a real-world example of crawling GitHub commits across multiple pages:
|
||||
|
||||
```python
|
||||
async def crawl_dynamic_content():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
url = "https://github.com/microsoft/TypeScript/commits/main"
|
||||
session_id = "typescript_commits_session"
|
||||
all_commits = []
|
||||
|
||||
# Define navigation JavaScript
|
||||
js_next_page = """
|
||||
const button = document.querySelector('a[data-testid="pagination-next-button"]');
|
||||
if (button) button.click();
|
||||
"""
|
||||
|
||||
# Define wait condition
|
||||
wait_for = """() => {
|
||||
const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
|
||||
if (commits.length === 0) return false;
|
||||
const firstCommit = commits[0].textContent.trim();
|
||||
return firstCommit !== window.firstCommit;
|
||||
}"""
|
||||
|
||||
# Define extraction schema
|
||||
schema = {
|
||||
"name": "Commit Extractor",
|
||||
"baseSelector": "li.Box-sc-g0xbh4-0",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h4.markdown-title",
|
||||
"type": "text",
|
||||
"transform": "strip",
|
||||
},
|
||||
],
|
||||
}
|
||||
extraction_strategy = JsonCssExtractionStrategy(schema)
|
||||
|
||||
# Crawl multiple pages
|
||||
for page in range(3):
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
session_id=session_id,
|
||||
extraction_strategy=extraction_strategy,
|
||||
js_code=js_next_page if page > 0 else None,
|
||||
wait_for=wait_for if page > 0 else None,
|
||||
js_only=page > 0,
|
||||
bypass_cache=True
|
||||
)
|
||||
|
||||
if result.success:
|
||||
commits = json.loads(result.extracted_content)
|
||||
all_commits.extend(commits)
|
||||
print(f"Page {page + 1}: Found {len(commits)} commits")
|
||||
|
||||
# Clean up session
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
return all_commits
|
||||
```
|
||||
|
||||
## Session Best Practices
|
||||
|
||||
1. **Session Naming**:
|
||||
```python
|
||||
# Use descriptive session IDs
|
||||
session_id = "login_flow_session"
|
||||
session_id = "product_catalog_session"
|
||||
```
|
||||
|
||||
2. **Resource Management**:
|
||||
```python
|
||||
try:
|
||||
# Your crawling code
|
||||
pass
|
||||
finally:
|
||||
# Always clean up sessions
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
```
|
||||
|
||||
3. **State Management**:
|
||||
```python
|
||||
# First page: login
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/login",
|
||||
session_id=session_id,
|
||||
js_code="document.querySelector('form').submit();"
|
||||
)
|
||||
|
||||
# Second page: verify login success
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/dashboard",
|
||||
session_id=session_id,
|
||||
wait_for="css:.user-profile" # Wait for authenticated content
|
||||
)
|
||||
```
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
1. **Authentication Flows**
|
||||
2. **Pagination Handling**
|
||||
3. **Form Submissions**
|
||||
4. **Multi-step Processes**
|
||||
5. **Dynamic Content Navigation**
|
||||
226
docs/md_v2/api/arun.md
Normal file
226
docs/md_v2/api/arun.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# Complete Parameter Guide for arun()
|
||||
|
||||
The following parameters can be passed to the `arun()` method. They are organized by their primary usage context and functionality.
|
||||
|
||||
## Core Parameters
|
||||
|
||||
```python
|
||||
await crawler.arun(
|
||||
url="https://example.com", # Required: URL to crawl
|
||||
verbose=True, # Enable detailed logging
|
||||
bypass_cache=False, # Skip cache for this request
|
||||
warmup=True # Whether to run warmup check
|
||||
)
|
||||
```
|
||||
|
||||
## Content Processing Parameters
|
||||
|
||||
### Text Processing
|
||||
```python
|
||||
await crawler.arun(
|
||||
word_count_threshold=10, # Minimum words per content block
|
||||
image_description_min_word_threshold=5, # Minimum words for image descriptions
|
||||
only_text=False, # Extract only text content
|
||||
excluded_tags=['form', 'nav'], # HTML tags to exclude
|
||||
keep_data_attributes=False, # Preserve data-* attributes
|
||||
)
|
||||
```
|
||||
|
||||
### Content Selection
|
||||
```python
|
||||
await crawler.arun(
|
||||
css_selector=".main-content", # CSS selector for content extraction
|
||||
remove_forms=True, # Remove all form elements
|
||||
remove_overlay_elements=True, # Remove popups/modals/overlays
|
||||
)
|
||||
```
|
||||
|
||||
### Link Handling
|
||||
```python
|
||||
await crawler.arun(
|
||||
exclude_external_links=True, # Remove external links
|
||||
exclude_social_media_links=True, # Remove social media links
|
||||
exclude_external_images=True, # Remove external images
|
||||
exclude_domains=["ads.example.com"], # Specific domains to exclude
|
||||
social_media_domains=[ # Additional social media domains
|
||||
"facebook.com",
|
||||
"twitter.com",
|
||||
"instagram.com"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
## Browser Control Parameters
|
||||
|
||||
### Basic Browser Settings
|
||||
```python
|
||||
await crawler.arun(
|
||||
headless=True, # Run browser in headless mode
|
||||
browser_type="chromium", # Browser engine: "chromium", "firefox", "webkit"
|
||||
page_timeout=60000, # Page load timeout in milliseconds
|
||||
user_agent="custom-agent", # Custom user agent
|
||||
)
|
||||
```
|
||||
|
||||
### Navigation and Waiting
|
||||
```python
|
||||
await crawler.arun(
|
||||
wait_for="css:.dynamic-content", # Wait for element/condition
|
||||
delay_before_return_html=2.0, # Wait before returning HTML (seconds)
|
||||
)
|
||||
```
|
||||
|
||||
### JavaScript Execution
|
||||
```python
|
||||
await crawler.arun(
|
||||
js_code=[ # JavaScript to execute (string or list)
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more').click();"
|
||||
],
|
||||
js_only=False, # Only execute JavaScript without reloading page
|
||||
)
|
||||
```
|
||||
|
||||
### Anti-Bot Features
|
||||
```python
|
||||
await crawler.arun(
|
||||
magic=True, # Enable all anti-detection features
|
||||
simulate_user=True, # Simulate human behavior
|
||||
override_navigator=True # Override navigator properties
|
||||
)
|
||||
```
|
||||
|
||||
### Session Management
|
||||
```python
|
||||
await crawler.arun(
|
||||
session_id="my_session", # Session identifier for persistent browsing
|
||||
)
|
||||
```
|
||||
|
||||
### Screenshot Options
|
||||
```python
|
||||
await crawler.arun(
|
||||
screenshot=True, # Take page screenshot
|
||||
screenshot_wait_for=2.0, # Wait before screenshot (seconds)
|
||||
)
|
||||
```
|
||||
|
||||
### Proxy Configuration
|
||||
```python
|
||||
await crawler.arun(
|
||||
proxy="http://proxy.example.com:8080", # Simple proxy URL
|
||||
proxy_config={ # Advanced proxy settings
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
## Content Extraction Parameters
|
||||
|
||||
### Extraction Strategy
|
||||
```python
|
||||
await crawler.arun(
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="ollama/llama2",
|
||||
schema=MySchema.schema(),
|
||||
instruction="Extract specific data"
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### Chunking Strategy
|
||||
```python
|
||||
await crawler.arun(
|
||||
chunking_strategy=RegexChunking(
|
||||
patterns=[r'\n\n', r'\.\s+']
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### HTML to Text Options
|
||||
```python
|
||||
await crawler.arun(
|
||||
html2text={
|
||||
"ignore_links": False,
|
||||
"ignore_images": False,
|
||||
"escape_dot": False,
|
||||
"body_width": 0,
|
||||
"protect_links": True,
|
||||
"unicode_snob": True
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
## Debug Options
|
||||
```python
|
||||
await crawler.arun(
|
||||
log_console=True, # Log browser console messages
|
||||
)
|
||||
```
|
||||
|
||||
## Parameter Interactions and Notes
|
||||
|
||||
1. **Magic Mode Combinations**
|
||||
```python
|
||||
# Full anti-detection setup
|
||||
await crawler.arun(
|
||||
magic=True,
|
||||
headless=False,
|
||||
simulate_user=True,
|
||||
override_navigator=True
|
||||
)
|
||||
```
|
||||
|
||||
2. **Dynamic Content Handling**
|
||||
```python
|
||||
# Handle lazy-loaded content
|
||||
await crawler.arun(
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="css:.lazy-content",
|
||||
delay_before_return_html=2.0
|
||||
)
|
||||
```
|
||||
|
||||
3. **Content Extraction Pipeline**
|
||||
```python
|
||||
# Complete extraction setup
|
||||
await crawler.arun(
|
||||
css_selector=".main-content",
|
||||
word_count_threshold=20,
|
||||
extraction_strategy=my_strategy,
|
||||
chunking_strategy=my_chunking,
|
||||
process_iframes=True,
|
||||
remove_overlay_elements=True
|
||||
)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Performance Optimization**
|
||||
```python
|
||||
await crawler.arun(
|
||||
bypass_cache=False, # Use cache when possible
|
||||
word_count_threshold=10, # Filter out noise
|
||||
process_iframes=False # Skip iframes if not needed
|
||||
)
|
||||
```
|
||||
|
||||
2. **Reliable Scraping**
|
||||
```python
|
||||
await crawler.arun(
|
||||
magic=True, # Enable anti-detection
|
||||
delay_before_return_html=1.0, # Wait for dynamic content
|
||||
page_timeout=60000 # Longer timeout for slow pages
|
||||
)
|
||||
```
|
||||
|
||||
3. **Clean Content**
|
||||
```python
|
||||
await crawler.arun(
|
||||
remove_overlay_elements=True, # Remove popups
|
||||
excluded_tags=['nav', 'aside'],# Remove unnecessary elements
|
||||
keep_data_attributes=False # Remove data attributes
|
||||
)
|
||||
```
|
||||
320
docs/md_v2/api/async-webcrawler.md
Normal file
320
docs/md_v2/api/async-webcrawler.md
Normal file
@@ -0,0 +1,320 @@
|
||||
# AsyncWebCrawler
|
||||
|
||||
The `AsyncWebCrawler` class is the main interface for web crawling operations. It provides asynchronous web crawling capabilities with extensive configuration options.
|
||||
|
||||
## Constructor
|
||||
|
||||
```python
|
||||
AsyncWebCrawler(
|
||||
# Browser Settings
|
||||
browser_type: str = "chromium", # Options: "chromium", "firefox", "webkit"
|
||||
headless: bool = True, # Run browser in headless mode
|
||||
verbose: bool = False, # Enable verbose logging
|
||||
|
||||
# Cache Settings
|
||||
always_by_pass_cache: bool = False, # Always bypass cache
|
||||
base_directory: str = str(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())), # Base directory for cache
|
||||
|
||||
# Network Settings
|
||||
proxy: str = None, # Simple proxy URL
|
||||
proxy_config: Dict = None, # Advanced proxy configuration
|
||||
|
||||
# Browser Behavior
|
||||
sleep_on_close: bool = False, # Wait before closing browser
|
||||
|
||||
# Custom Settings
|
||||
user_agent: str = None, # Custom user agent
|
||||
headers: Dict[str, str] = {}, # Custom HTTP headers
|
||||
js_code: Union[str, List[str]] = None, # Default JavaScript to execute
|
||||
)
|
||||
```
|
||||
|
||||
### Parameters in Detail
|
||||
|
||||
#### Browser Settings
|
||||
|
||||
- **browser_type** (str, optional)
|
||||
- Default: `"chromium"`
|
||||
- Options: `"chromium"`, `"firefox"`, `"webkit"`
|
||||
- Controls which browser engine to use
|
||||
```python
|
||||
# Example: Using Firefox
|
||||
crawler = AsyncWebCrawler(browser_type="firefox")
|
||||
```
|
||||
|
||||
- **headless** (bool, optional)
|
||||
- Default: `True`
|
||||
- When `True`, browser runs without GUI
|
||||
- Set to `False` for debugging
|
||||
```python
|
||||
# Visible browser for debugging
|
||||
crawler = AsyncWebCrawler(headless=False)
|
||||
```
|
||||
|
||||
- **verbose** (bool, optional)
|
||||
- Default: `False`
|
||||
- Enables detailed logging
|
||||
```python
|
||||
# Enable detailed logging
|
||||
crawler = AsyncWebCrawler(verbose=True)
|
||||
```
|
||||
|
||||
#### Cache Settings
|
||||
|
||||
- **always_by_pass_cache** (bool, optional)
|
||||
- Default: `False`
|
||||
- When `True`, always fetches fresh content
|
||||
```python
|
||||
# Always fetch fresh content
|
||||
crawler = AsyncWebCrawler(always_by_pass_cache=True)
|
||||
```
|
||||
|
||||
- **base_directory** (str, optional)
|
||||
- Default: User's home directory
|
||||
- Base path for cache storage
|
||||
```python
|
||||
# Custom cache directory
|
||||
crawler = AsyncWebCrawler(base_directory="/path/to/cache")
|
||||
```
|
||||
|
||||
#### Network Settings
|
||||
|
||||
- **proxy** (str, optional)
|
||||
- Simple proxy URL
|
||||
```python
|
||||
# Using simple proxy
|
||||
crawler = AsyncWebCrawler(proxy="http://proxy.example.com:8080")
|
||||
```
|
||||
|
||||
- **proxy_config** (Dict, optional)
|
||||
- Advanced proxy configuration with authentication
|
||||
```python
|
||||
# Advanced proxy with auth
|
||||
crawler = AsyncWebCrawler(proxy_config={
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
})
|
||||
```
|
||||
|
||||
#### Browser Behavior
|
||||
|
||||
- **sleep_on_close** (bool, optional)
|
||||
- Default: `False`
|
||||
- Adds delay before closing browser
|
||||
```python
|
||||
# Wait before closing
|
||||
crawler = AsyncWebCrawler(sleep_on_close=True)
|
||||
```
|
||||
|
||||
#### Custom Settings
|
||||
|
||||
- **user_agent** (str, optional)
|
||||
- Custom user agent string
|
||||
```python
|
||||
# Custom user agent
|
||||
crawler = AsyncWebCrawler(
|
||||
user_agent="Mozilla/5.0 (Custom Agent) Chrome/90.0"
|
||||
)
|
||||
```
|
||||
|
||||
- **headers** (Dict[str, str], optional)
|
||||
- Custom HTTP headers
|
||||
```python
|
||||
# Custom headers
|
||||
crawler = AsyncWebCrawler(
|
||||
headers={
|
||||
"Accept-Language": "en-US",
|
||||
"Custom-Header": "Value"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
- **js_code** (Union[str, List[str]], optional)
|
||||
- Default JavaScript to execute on each page
|
||||
```python
|
||||
# Default JavaScript
|
||||
crawler = AsyncWebCrawler(
|
||||
js_code=[
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more').click();"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
## Methods
|
||||
|
||||
### arun()
|
||||
|
||||
The primary method for crawling web pages.
|
||||
|
||||
```python
|
||||
async def arun(
|
||||
# Required
|
||||
url: str, # URL to crawl
|
||||
|
||||
# Content Selection
|
||||
css_selector: str = None, # CSS selector for content
|
||||
word_count_threshold: int = 10, # Minimum words per block
|
||||
|
||||
# Cache Control
|
||||
bypass_cache: bool = False, # Bypass cache for this request
|
||||
|
||||
# Session Management
|
||||
session_id: str = None, # Session identifier
|
||||
|
||||
# Screenshot Options
|
||||
screenshot: bool = False, # Take screenshot
|
||||
screenshot_wait_for: float = None, # Wait before screenshot
|
||||
|
||||
# Content Processing
|
||||
process_iframes: bool = False, # Process iframe content
|
||||
remove_overlay_elements: bool = False, # Remove popups/modals
|
||||
|
||||
# Anti-Bot Settings
|
||||
simulate_user: bool = False, # Simulate human behavior
|
||||
override_navigator: bool = False, # Override navigator properties
|
||||
magic: bool = False, # Enable all anti-detection
|
||||
|
||||
# Content Filtering
|
||||
excluded_tags: List[str] = None, # HTML tags to exclude
|
||||
exclude_external_links: bool = False, # Remove external links
|
||||
exclude_social_media_links: bool = False, # Remove social media links
|
||||
|
||||
# JavaScript Handling
|
||||
js_code: Union[str, List[str]] = None, # JavaScript to execute
|
||||
wait_for: str = None, # Wait condition
|
||||
|
||||
# Page Loading
|
||||
page_timeout: int = 60000, # Page load timeout (ms)
|
||||
delay_before_return_html: float = None, # Wait before return
|
||||
|
||||
# Extraction
|
||||
extraction_strategy: ExtractionStrategy = None # Extraction strategy
|
||||
) -> CrawlResult:
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
|
||||
#### Basic Crawling
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
#### Advanced Crawling
|
||||
```python
|
||||
async with AsyncWebCrawler(
|
||||
browser_type="firefox",
|
||||
verbose=True,
|
||||
headers={"Custom-Header": "Value"}
|
||||
) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
css_selector=".main-content",
|
||||
word_count_threshold=20,
|
||||
process_iframes=True,
|
||||
magic=True,
|
||||
wait_for="css:.dynamic-content",
|
||||
screenshot=True
|
||||
)
|
||||
```
|
||||
|
||||
#### Session Management
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# First request
|
||||
result1 = await crawler.arun(
|
||||
url="https://example.com/login",
|
||||
session_id="my_session"
|
||||
)
|
||||
|
||||
# Subsequent request using same session
|
||||
result2 = await crawler.arun(
|
||||
url="https://example.com/protected",
|
||||
session_id="my_session"
|
||||
)
|
||||
```
|
||||
|
||||
## Context Manager
|
||||
|
||||
AsyncWebCrawler implements the async context manager protocol:
|
||||
|
||||
```python
|
||||
async def __aenter__(self) -> 'AsyncWebCrawler':
|
||||
# Initialize browser and resources
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *args):
|
||||
# Cleanup resources
|
||||
pass
|
||||
```
|
||||
|
||||
Always use AsyncWebCrawler with async context manager:
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Your crawling code here
|
||||
pass
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Resource Management**
|
||||
```python
|
||||
# Always use context manager
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Crawler will be properly cleaned up
|
||||
pass
|
||||
```
|
||||
|
||||
2. **Error Handling**
|
||||
```python
|
||||
try:
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
if not result.success:
|
||||
print(f"Crawl failed: {result.error_message}")
|
||||
except Exception as e:
|
||||
print(f"Error: {str(e)}")
|
||||
```
|
||||
|
||||
3. **Performance Optimization**
|
||||
```python
|
||||
# Enable caching for better performance
|
||||
crawler = AsyncWebCrawler(
|
||||
always_by_pass_cache=False,
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
4. **Anti-Detection**
|
||||
```python
|
||||
# Maximum stealth
|
||||
crawler = AsyncWebCrawler(
|
||||
headless=True,
|
||||
user_agent="Mozilla/5.0...",
|
||||
headers={"Accept-Language": "en-US"}
|
||||
)
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
magic=True,
|
||||
simulate_user=True
|
||||
)
|
||||
```
|
||||
|
||||
## Note on Browser Types
|
||||
|
||||
Each browser type has its characteristics:
|
||||
|
||||
- **chromium**: Best overall compatibility
|
||||
- **firefox**: Good for specific use cases
|
||||
- **webkit**: Lighter weight, good for basic crawling
|
||||
|
||||
Choose based on your specific needs:
|
||||
```python
|
||||
# High compatibility
|
||||
crawler = AsyncWebCrawler(browser_type="chromium")
|
||||
|
||||
# Memory efficient
|
||||
crawler = AsyncWebCrawler(browser_type="webkit")
|
||||
```
|
||||
301
docs/md_v2/api/crawl-result.md
Normal file
301
docs/md_v2/api/crawl-result.md
Normal file
@@ -0,0 +1,301 @@
|
||||
# CrawlResult
|
||||
|
||||
The `CrawlResult` class represents the result of a web crawling operation. It provides access to various forms of extracted content and metadata from the crawled webpage.
|
||||
|
||||
## Class Definition
|
||||
|
||||
```python
|
||||
class CrawlResult(BaseModel):
|
||||
"""Result of a web crawling operation."""
|
||||
|
||||
# Basic Information
|
||||
url: str # Crawled URL
|
||||
success: bool # Whether crawl succeeded
|
||||
status_code: Optional[int] = None # HTTP status code
|
||||
error_message: Optional[str] = None # Error message if failed
|
||||
|
||||
# Content
|
||||
html: str # Raw HTML content
|
||||
cleaned_html: Optional[str] = None # Cleaned HTML
|
||||
fit_html: Optional[str] = None # Most relevant HTML content
|
||||
markdown: Optional[str] = None # HTML converted to markdown
|
||||
fit_markdown: Optional[str] = None # Most relevant markdown content
|
||||
|
||||
# Extracted Data
|
||||
extracted_content: Optional[str] = None # Content from extraction strategy
|
||||
media: Dict[str, List[Dict]] = {} # Extracted media information
|
||||
links: Dict[str, List[Dict]] = {} # Extracted links
|
||||
metadata: Optional[dict] = None # Page metadata
|
||||
|
||||
# Additional Data
|
||||
screenshot: Optional[str] = None # Base64 encoded screenshot
|
||||
session_id: Optional[str] = None # Session identifier
|
||||
response_headers: Optional[dict] = None # HTTP response headers
|
||||
```
|
||||
|
||||
## Properties and Their Data Structures
|
||||
|
||||
### Basic Information
|
||||
|
||||
```python
|
||||
# Access basic information
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
print(result.url) # "https://example.com"
|
||||
print(result.success) # True/False
|
||||
print(result.status_code) # 200, 404, etc.
|
||||
print(result.error_message) # Error details if failed
|
||||
```
|
||||
|
||||
### Content Properties
|
||||
|
||||
#### HTML Content
|
||||
```python
|
||||
# Raw HTML
|
||||
html_content = result.html
|
||||
|
||||
# Cleaned HTML (removed ads, popups, etc.)
|
||||
clean_content = result.cleaned_html
|
||||
|
||||
# Most relevant HTML content
|
||||
main_content = result.fit_html
|
||||
```
|
||||
|
||||
#### Markdown Content
|
||||
```python
|
||||
# Full markdown version
|
||||
markdown_content = result.markdown
|
||||
|
||||
# Most relevant markdown content
|
||||
main_content = result.fit_markdown
|
||||
```
|
||||
|
||||
### Media Content
|
||||
|
||||
The media dictionary contains organized media elements:
|
||||
|
||||
```python
|
||||
# Structure
|
||||
media = {
|
||||
"images": [
|
||||
{
|
||||
"src": str, # Image URL
|
||||
"alt": str, # Alt text
|
||||
"desc": str, # Contextual description
|
||||
"score": float, # Relevance score (0-10)
|
||||
"type": str, # "image"
|
||||
"width": int, # Image width (if available)
|
||||
"height": int, # Image height (if available)
|
||||
"context": str, # Surrounding text
|
||||
"lazy": bool # Whether image was lazy-loaded
|
||||
}
|
||||
],
|
||||
"videos": [
|
||||
{
|
||||
"src": str, # Video URL
|
||||
"type": str, # "video"
|
||||
"title": str, # Video title
|
||||
"poster": str, # Thumbnail URL
|
||||
"duration": str, # Video duration
|
||||
"description": str # Video description
|
||||
}
|
||||
],
|
||||
"audios": [
|
||||
{
|
||||
"src": str, # Audio URL
|
||||
"type": str, # "audio"
|
||||
"title": str, # Audio title
|
||||
"duration": str, # Audio duration
|
||||
"description": str # Audio description
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Example usage
|
||||
for image in result.media["images"]:
|
||||
if image["score"] > 5: # High-relevance images
|
||||
print(f"High-quality image: {image['src']}")
|
||||
print(f"Context: {image['context']}")
|
||||
```
|
||||
|
||||
### Link Analysis
|
||||
|
||||
The links dictionary organizes discovered links:
|
||||
|
||||
```python
|
||||
# Structure
|
||||
links = {
|
||||
"internal": [
|
||||
{
|
||||
"href": str, # URL
|
||||
"text": str, # Link text
|
||||
"title": str, # Title attribute
|
||||
"type": str, # Link type (nav, content, etc.)
|
||||
"context": str, # Surrounding text
|
||||
"score": float # Relevance score
|
||||
}
|
||||
],
|
||||
"external": [
|
||||
{
|
||||
"href": str, # External URL
|
||||
"text": str, # Link text
|
||||
"title": str, # Title attribute
|
||||
"domain": str, # Domain name
|
||||
"type": str, # Link type
|
||||
"context": str # Surrounding text
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Example usage
|
||||
for link in result.links["internal"]:
|
||||
print(f"Internal link: {link['href']}")
|
||||
print(f"Context: {link['context']}")
|
||||
```
|
||||
|
||||
### Metadata
|
||||
|
||||
The metadata dictionary contains page information:
|
||||
|
||||
```python
|
||||
# Structure
|
||||
metadata = {
|
||||
"title": str, # Page title
|
||||
"description": str, # Meta description
|
||||
"keywords": List[str], # Meta keywords
|
||||
"author": str, # Author information
|
||||
"published_date": str, # Publication date
|
||||
"modified_date": str, # Last modified date
|
||||
"language": str, # Page language
|
||||
"canonical_url": str, # Canonical URL
|
||||
"og_data": Dict, # Open Graph data
|
||||
"twitter_data": Dict # Twitter card data
|
||||
}
|
||||
|
||||
# Example usage
|
||||
if result.metadata:
|
||||
print(f"Title: {result.metadata['title']}")
|
||||
print(f"Author: {result.metadata.get('author', 'Unknown')}")
|
||||
```
|
||||
|
||||
### Extracted Content
|
||||
|
||||
Content from extraction strategies:
|
||||
|
||||
```python
|
||||
# For LLM or CSS extraction strategies
|
||||
if result.extracted_content:
|
||||
structured_data = json.loads(result.extracted_content)
|
||||
print(structured_data)
|
||||
```
|
||||
|
||||
### Screenshot
|
||||
|
||||
Base64 encoded screenshot:
|
||||
|
||||
```python
|
||||
# Save screenshot if available
|
||||
if result.screenshot:
|
||||
import base64
|
||||
|
||||
# Decode and save
|
||||
with open("screenshot.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Content Access
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
if result.success:
|
||||
# Get clean content
|
||||
print(result.fit_markdown)
|
||||
|
||||
# Process images
|
||||
for image in result.media["images"]:
|
||||
if image["score"] > 7:
|
||||
print(f"High-quality image: {image['src']}")
|
||||
```
|
||||
|
||||
### Complete Data Processing
|
||||
```python
|
||||
async def process_webpage(url: str) -> Dict:
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url=url)
|
||||
|
||||
if not result.success:
|
||||
raise Exception(f"Crawl failed: {result.error_message}")
|
||||
|
||||
return {
|
||||
"content": result.fit_markdown,
|
||||
"images": [
|
||||
img for img in result.media["images"]
|
||||
if img["score"] > 5
|
||||
],
|
||||
"internal_links": [
|
||||
link["href"] for link in result.links["internal"]
|
||||
],
|
||||
"metadata": result.metadata,
|
||||
"status": result.status_code
|
||||
}
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
```python
|
||||
async def safe_crawl(url: str) -> Dict:
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
try:
|
||||
result = await crawler.arun(url=url)
|
||||
|
||||
if not result.success:
|
||||
return {
|
||||
"success": False,
|
||||
"error": result.error_message,
|
||||
"status": result.status_code
|
||||
}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"content": result.fit_markdown,
|
||||
"status": result.status_code
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": str(e),
|
||||
"status": None
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always Check Success**
|
||||
```python
|
||||
if not result.success:
|
||||
print(f"Error: {result.error_message}")
|
||||
return
|
||||
```
|
||||
|
||||
2. **Use fit_markdown for Articles**
|
||||
```python
|
||||
# Better for article content
|
||||
content = result.fit_markdown if result.fit_markdown else result.markdown
|
||||
```
|
||||
|
||||
3. **Filter Media by Score**
|
||||
```python
|
||||
relevant_images = [
|
||||
img for img in result.media["images"]
|
||||
if img["score"] > 5
|
||||
]
|
||||
```
|
||||
|
||||
4. **Handle Missing Data**
|
||||
```python
|
||||
metadata = result.metadata or {}
|
||||
title = metadata.get('title', 'Unknown Title')
|
||||
```
|
||||
35
docs/md_v2/api/parameters.md
Normal file
35
docs/md_v2/api/parameters.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# Parameter Reference Table
|
||||
|
||||
| File Name | Parameter Name | Code Usage | Strategy/Class | Description |
|
||||
|-----------|---------------|------------|----------------|-------------|
|
||||
| async_crawler_strategy.py | user_agent | `kwargs.get("user_agent")` | AsyncPlaywrightCrawlerStrategy | User agent string for browser identification |
|
||||
| async_crawler_strategy.py | proxy | `kwargs.get("proxy")` | AsyncPlaywrightCrawlerStrategy | Proxy server configuration for network requests |
|
||||
| async_crawler_strategy.py | proxy_config | `kwargs.get("proxy_config")` | AsyncPlaywrightCrawlerStrategy | Detailed proxy configuration including auth |
|
||||
| async_crawler_strategy.py | headless | `kwargs.get("headless", True)` | AsyncPlaywrightCrawlerStrategy | Whether to run browser in headless mode |
|
||||
| async_crawler_strategy.py | browser_type | `kwargs.get("browser_type", "chromium")` | AsyncPlaywrightCrawlerStrategy | Type of browser to use (chromium/firefox/webkit) |
|
||||
| async_crawler_strategy.py | headers | `kwargs.get("headers", {})` | AsyncPlaywrightCrawlerStrategy | Custom HTTP headers for requests |
|
||||
| async_crawler_strategy.py | verbose | `kwargs.get("verbose", False)` | AsyncPlaywrightCrawlerStrategy | Enable detailed logging output |
|
||||
| async_crawler_strategy.py | sleep_on_close | `kwargs.get("sleep_on_close", False)` | AsyncPlaywrightCrawlerStrategy | Add delay before closing browser |
|
||||
| async_crawler_strategy.py | use_managed_browser | `kwargs.get("use_managed_browser", False)` | AsyncPlaywrightCrawlerStrategy | Use managed browser instance |
|
||||
| async_crawler_strategy.py | user_data_dir | `kwargs.get("user_data_dir", None)` | AsyncPlaywrightCrawlerStrategy | Custom directory for browser profile data |
|
||||
| async_crawler_strategy.py | session_id | `kwargs.get("session_id")` | AsyncPlaywrightCrawlerStrategy | Unique identifier for browser session |
|
||||
| async_crawler_strategy.py | override_navigator | `kwargs.get("override_navigator", False)` | AsyncPlaywrightCrawlerStrategy | Override browser navigator properties |
|
||||
| async_crawler_strategy.py | simulate_user | `kwargs.get("simulate_user", False)` | AsyncPlaywrightCrawlerStrategy | Simulate human-like behavior |
|
||||
| async_crawler_strategy.py | magic | `kwargs.get("magic", False)` | AsyncPlaywrightCrawlerStrategy | Enable advanced anti-detection features |
|
||||
| async_crawler_strategy.py | log_console | `kwargs.get("log_console", False)` | AsyncPlaywrightCrawlerStrategy | Log browser console messages |
|
||||
| async_crawler_strategy.py | js_only | `kwargs.get("js_only", False)` | AsyncPlaywrightCrawlerStrategy | Only execute JavaScript without page load |
|
||||
| async_crawler_strategy.py | page_timeout | `kwargs.get("page_timeout", 60000)` | AsyncPlaywrightCrawlerStrategy | Timeout for page load in milliseconds |
|
||||
| async_crawler_strategy.py | ignore_body_visibility | `kwargs.get("ignore_body_visibility", True)` | AsyncPlaywrightCrawlerStrategy | Process page even if body is hidden |
|
||||
| async_crawler_strategy.py | js_code | `kwargs.get("js_code", kwargs.get("js", self.js_code))` | AsyncPlaywrightCrawlerStrategy | Custom JavaScript code to execute |
|
||||
| async_crawler_strategy.py | wait_for | `kwargs.get("wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait for specific element/condition |
|
||||
| async_crawler_strategy.py | process_iframes | `kwargs.get("process_iframes", False)` | AsyncPlaywrightCrawlerStrategy | Extract content from iframes |
|
||||
| async_crawler_strategy.py | delay_before_return_html | `kwargs.get("delay_before_return_html")` | AsyncPlaywrightCrawlerStrategy | Additional delay before returning HTML |
|
||||
| async_crawler_strategy.py | remove_overlay_elements | `kwargs.get("remove_overlay_elements", False)` | AsyncPlaywrightCrawlerStrategy | Remove pop-ups and overlay elements |
|
||||
| async_crawler_strategy.py | screenshot | `kwargs.get("screenshot")` | AsyncPlaywrightCrawlerStrategy | Take page screenshot |
|
||||
| async_crawler_strategy.py | screenshot_wait_for | `kwargs.get("screenshot_wait_for")` | AsyncPlaywrightCrawlerStrategy | Wait before taking screenshot |
|
||||
| async_crawler_strategy.py | semaphore_count | `kwargs.get("semaphore_count", 5)` | AsyncPlaywrightCrawlerStrategy | Concurrent request limit |
|
||||
| async_webcrawler.py | verbose | `kwargs.get("verbose", False)` | AsyncWebCrawler | Enable detailed logging |
|
||||
| async_webcrawler.py | warmup | `kwargs.get("warmup", True)` | AsyncWebCrawler | Initialize crawler with warmup request |
|
||||
| async_webcrawler.py | session_id | `kwargs.get("session_id", None)` | AsyncWebCrawler | Session identifier for browser reuse |
|
||||
| async_webcrawler.py | only_text | `kwargs.get("only_text", False)` | AsyncWebCrawler | Extract only text content |
|
||||
| async_webcrawler.py | bypass_cache | `kwargs.get("bypass_cache", False)` | AsyncWebCrawler | Skip cache and force fresh crawl |
|
||||
255
docs/md_v2/api/strategies.md
Normal file
255
docs/md_v2/api/strategies.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# Extraction & Chunking Strategies API
|
||||
|
||||
This documentation covers the API reference for extraction and chunking strategies in Crawl4AI.
|
||||
|
||||
## Extraction Strategies
|
||||
|
||||
All extraction strategies inherit from the base `ExtractionStrategy` class and implement two key methods:
|
||||
- `extract(url: str, html: str) -> List[Dict[str, Any]]`
|
||||
- `run(url: str, sections: List[str]) -> List[Dict[str, Any]]`
|
||||
|
||||
### LLMExtractionStrategy
|
||||
|
||||
Used for extracting structured data using Language Models.
|
||||
|
||||
```python
|
||||
LLMExtractionStrategy(
|
||||
# Required Parameters
|
||||
provider: str = DEFAULT_PROVIDER, # LLM provider (e.g., "ollama/llama2")
|
||||
api_token: Optional[str] = None, # API token
|
||||
|
||||
# Extraction Configuration
|
||||
instruction: str = None, # Custom extraction instruction
|
||||
schema: Dict = None, # Pydantic model schema for structured data
|
||||
extraction_type: str = "block", # "block" or "schema"
|
||||
|
||||
# Chunking Parameters
|
||||
chunk_token_threshold: int = 4000, # Maximum tokens per chunk
|
||||
overlap_rate: float = 0.1, # Overlap between chunks
|
||||
word_token_rate: float = 0.75, # Word to token conversion rate
|
||||
apply_chunking: bool = True, # Enable/disable chunking
|
||||
|
||||
# API Configuration
|
||||
base_url: str = None, # Base URL for API
|
||||
extra_args: Dict = {}, # Additional provider arguments
|
||||
verbose: bool = False # Enable verbose logging
|
||||
)
|
||||
```
|
||||
|
||||
### CosineStrategy
|
||||
|
||||
Used for content similarity-based extraction and clustering.
|
||||
|
||||
```python
|
||||
CosineStrategy(
|
||||
# Content Filtering
|
||||
semantic_filter: str = None, # Topic/keyword filter
|
||||
word_count_threshold: int = 10, # Minimum words per cluster
|
||||
sim_threshold: float = 0.3, # Similarity threshold
|
||||
|
||||
# Clustering Parameters
|
||||
max_dist: float = 0.2, # Maximum cluster distance
|
||||
linkage_method: str = 'ward', # Clustering method
|
||||
top_k: int = 3, # Top clusters to return
|
||||
|
||||
# Model Configuration
|
||||
model_name: str = 'sentence-transformers/all-MiniLM-L6-v2', # Embedding model
|
||||
|
||||
verbose: bool = False # Enable verbose logging
|
||||
)
|
||||
```
|
||||
|
||||
### JsonCssExtractionStrategy
|
||||
|
||||
Used for CSS selector-based structured data extraction.
|
||||
|
||||
```python
|
||||
JsonCssExtractionStrategy(
|
||||
schema: Dict[str, Any], # Extraction schema
|
||||
verbose: bool = False # Enable verbose logging
|
||||
)
|
||||
|
||||
# Schema Structure
|
||||
schema = {
|
||||
"name": str, # Schema name
|
||||
"baseSelector": str, # Base CSS selector
|
||||
"fields": [ # List of fields to extract
|
||||
{
|
||||
"name": str, # Field name
|
||||
"selector": str, # CSS selector
|
||||
"type": str, # Field type: "text", "attribute", "html", "regex"
|
||||
"attribute": str, # For type="attribute"
|
||||
"pattern": str, # For type="regex"
|
||||
"transform": str, # Optional: "lowercase", "uppercase", "strip"
|
||||
"default": Any # Default value if extraction fails
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Chunking Strategies
|
||||
|
||||
All chunking strategies inherit from `ChunkingStrategy` and implement the `chunk(text: str) -> list` method.
|
||||
|
||||
### RegexChunking
|
||||
|
||||
Splits text based on regex patterns.
|
||||
|
||||
```python
|
||||
RegexChunking(
|
||||
patterns: List[str] = None # Regex patterns for splitting
|
||||
# Default: [r'\n\n']
|
||||
)
|
||||
```
|
||||
|
||||
### SlidingWindowChunking
|
||||
|
||||
Creates overlapping chunks with a sliding window approach.
|
||||
|
||||
```python
|
||||
SlidingWindowChunking(
|
||||
window_size: int = 100, # Window size in words
|
||||
step: int = 50 # Step size between windows
|
||||
)
|
||||
```
|
||||
|
||||
### OverlappingWindowChunking
|
||||
|
||||
Creates chunks with specified overlap.
|
||||
|
||||
```python
|
||||
OverlappingWindowChunking(
|
||||
window_size: int = 1000, # Chunk size in words
|
||||
overlap: int = 100 # Overlap size in words
|
||||
)
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### LLM Extraction
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
# Define schema
|
||||
class Article(BaseModel):
|
||||
title: str
|
||||
content: str
|
||||
author: str
|
||||
|
||||
# Create strategy
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider="ollama/llama2",
|
||||
schema=Article.schema(),
|
||||
instruction="Extract article details"
|
||||
)
|
||||
|
||||
# Use with crawler
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/article",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
|
||||
# Access extracted data
|
||||
data = json.loads(result.extracted_content)
|
||||
```
|
||||
|
||||
### CSS Extraction
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
# Define schema
|
||||
schema = {
|
||||
"name": "Product List",
|
||||
"baseSelector": ".product-card",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h2.title",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": ".price",
|
||||
"type": "text",
|
||||
"transform": "strip"
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"selector": "img",
|
||||
"type": "attribute",
|
||||
"attribute": "src"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Create and use strategy
|
||||
strategy = JsonCssExtractionStrategy(schema)
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/products",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
```
|
||||
|
||||
### Content Chunking
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import OverlappingWindowChunking
|
||||
|
||||
# Create chunking strategy
|
||||
chunker = OverlappingWindowChunking(
|
||||
window_size=500, # 500 words per chunk
|
||||
overlap=50 # 50 words overlap
|
||||
)
|
||||
|
||||
# Use with extraction strategy
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider="ollama/llama2",
|
||||
chunking_strategy=chunker
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/long-article",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Choose the Right Strategy**
|
||||
- Use `LLMExtractionStrategy` for complex, unstructured content
|
||||
- Use `JsonCssExtractionStrategy` for well-structured HTML
|
||||
- Use `CosineStrategy` for content similarity and clustering
|
||||
|
||||
2. **Optimize Chunking**
|
||||
```python
|
||||
# For long documents
|
||||
strategy = LLMExtractionStrategy(
|
||||
chunk_token_threshold=2000, # Smaller chunks
|
||||
overlap_rate=0.1 # 10% overlap
|
||||
)
|
||||
```
|
||||
|
||||
3. **Handle Errors**
|
||||
```python
|
||||
try:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
if result.success:
|
||||
content = json.loads(result.extracted_content)
|
||||
except Exception as e:
|
||||
print(f"Extraction failed: {e}")
|
||||
```
|
||||
|
||||
4. **Monitor Performance**
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
verbose=True, # Enable logging
|
||||
word_count_threshold=20, # Filter short content
|
||||
top_k=5 # Limit results
|
||||
)
|
||||
```
|
||||
BIN
docs/md_v2/assets/docs.zip
Normal file
BIN
docs/md_v2/assets/docs.zip
Normal file
Binary file not shown.
@@ -150,4 +150,11 @@ strong,
|
||||
.tab-content pre {
|
||||
margin: 0;
|
||||
max-height: 300px; overflow: auto; border:none;
|
||||
}
|
||||
|
||||
ol li::before {
|
||||
content: counters(item, ".") ". ";
|
||||
counter-increment: item;
|
||||
/* float: left; */
|
||||
/* padding-right: 5px; */
|
||||
}
|
||||
208
docs/md_v2/basic/browser-config.md
Normal file
208
docs/md_v2/basic/browser-config.md
Normal file
@@ -0,0 +1,208 @@
|
||||
# Browser Configuration
|
||||
|
||||
Crawl4AI supports multiple browser engines and offers extensive configuration options for browser behavior.
|
||||
|
||||
## Browser Types
|
||||
|
||||
Choose from three browser engines:
|
||||
|
||||
```python
|
||||
# Chromium (default)
|
||||
async with AsyncWebCrawler(browser_type="chromium") as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Firefox
|
||||
async with AsyncWebCrawler(browser_type="firefox") as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# WebKit
|
||||
async with AsyncWebCrawler(browser_type="webkit") as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Basic Configuration
|
||||
|
||||
Common browser settings:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler(
|
||||
headless=True, # Run in headless mode (no GUI)
|
||||
verbose=True, # Enable detailed logging
|
||||
sleep_on_close=False # No delay when closing browser
|
||||
) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Identity Management
|
||||
|
||||
Control how your crawler appears to websites:
|
||||
|
||||
```python
|
||||
# Custom user agent
|
||||
async with AsyncWebCrawler(
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Custom headers
|
||||
headers = {
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Cache-Control": "no-cache"
|
||||
}
|
||||
async with AsyncWebCrawler(headers=headers) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Screenshot Capabilities
|
||||
|
||||
Capture page screenshots with enhanced error handling:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
screenshot=True, # Enable screenshot
|
||||
screenshot_wait_for=2.0 # Wait 2 seconds before capture
|
||||
)
|
||||
|
||||
if result.screenshot: # Base64 encoded image
|
||||
import base64
|
||||
with open("screenshot.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
```
|
||||
|
||||
## Timeouts and Waiting
|
||||
|
||||
Control page loading behavior:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
page_timeout=60000, # Page load timeout (ms)
|
||||
delay_before_return_html=2.0, # Wait before content capture
|
||||
wait_for="css:.dynamic-content" # Wait for specific element
|
||||
)
|
||||
```
|
||||
|
||||
## JavaScript Execution
|
||||
|
||||
Execute custom JavaScript before crawling:
|
||||
|
||||
```python
|
||||
# Single JavaScript command
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);"
|
||||
)
|
||||
|
||||
# Multiple commands
|
||||
js_commands = [
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more').click();"
|
||||
]
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=js_commands
|
||||
)
|
||||
```
|
||||
|
||||
## Proxy Configuration
|
||||
|
||||
Use proxies for enhanced access:
|
||||
|
||||
```python
|
||||
# Simple proxy
|
||||
async with AsyncWebCrawler(
|
||||
proxy="http://proxy.example.com:8080"
|
||||
) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Proxy with authentication
|
||||
proxy_config = {
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
}
|
||||
async with AsyncWebCrawler(proxy_config=proxy_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Anti-Detection Features
|
||||
|
||||
Enable stealth features to avoid bot detection:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
simulate_user=True, # Simulate human behavior
|
||||
override_navigator=True, # Mask automation signals
|
||||
magic=True # Enable all anti-detection features
|
||||
)
|
||||
```
|
||||
|
||||
## Handling Dynamic Content
|
||||
|
||||
Configure browser to handle dynamic content:
|
||||
|
||||
```python
|
||||
# Wait for dynamic content
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
wait_for="js:() => document.querySelector('.content').children.length > 10",
|
||||
process_iframes=True # Process iframe content
|
||||
)
|
||||
|
||||
# Handle lazy-loaded images
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
delay_before_return_html=2.0 # Wait for images to load
|
||||
)
|
||||
```
|
||||
|
||||
## Comprehensive Example
|
||||
|
||||
Here's how to combine various browser configurations:
|
||||
|
||||
```python
|
||||
async def crawl_with_advanced_config(url: str):
|
||||
async with AsyncWebCrawler(
|
||||
# Browser setup
|
||||
browser_type="chromium",
|
||||
headless=True,
|
||||
verbose=True,
|
||||
|
||||
# Identity
|
||||
user_agent="Custom User Agent",
|
||||
headers={"Accept-Language": "en-US"},
|
||||
|
||||
# Proxy setup
|
||||
proxy="http://proxy.example.com:8080"
|
||||
) as crawler:
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
# Content handling
|
||||
process_iframes=True,
|
||||
screenshot=True,
|
||||
|
||||
# Timing
|
||||
page_timeout=60000,
|
||||
delay_before_return_html=2.0,
|
||||
|
||||
# Anti-detection
|
||||
magic=True,
|
||||
simulate_user=True,
|
||||
|
||||
# Dynamic content
|
||||
js_code=[
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more')?.click();"
|
||||
],
|
||||
wait_for="css:.dynamic-content"
|
||||
)
|
||||
|
||||
return {
|
||||
"content": result.markdown,
|
||||
"screenshot": result.screenshot,
|
||||
"success": result.success
|
||||
}
|
||||
```
|
||||
199
docs/md_v2/basic/content-selection.md
Normal file
199
docs/md_v2/basic/content-selection.md
Normal file
@@ -0,0 +1,199 @@
|
||||
# Content Selection
|
||||
|
||||
Crawl4AI provides multiple ways to select and filter specific content from webpages. Learn how to precisely target the content you need.
|
||||
|
||||
## CSS Selectors
|
||||
|
||||
The simplest way to extract specific content:
|
||||
|
||||
```python
|
||||
# Extract specific content using CSS selector
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
css_selector=".main-article" # Target main article content
|
||||
)
|
||||
|
||||
# Multiple selectors
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
css_selector="article h1, article .content" # Target heading and content
|
||||
)
|
||||
```
|
||||
|
||||
## Content Filtering
|
||||
|
||||
Control what content is included or excluded:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
# Content thresholds
|
||||
word_count_threshold=10, # Minimum words per block
|
||||
|
||||
# Tag exclusions
|
||||
excluded_tags=['form', 'header', 'footer', 'nav'],
|
||||
|
||||
# Link filtering
|
||||
exclude_external_links=True, # Remove external links
|
||||
exclude_social_media_links=True, # Remove social media links
|
||||
|
||||
# Media filtering
|
||||
exclude_external_images=True # Remove external images
|
||||
)
|
||||
```
|
||||
|
||||
## Iframe Content
|
||||
|
||||
Process content inside iframes:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
process_iframes=True, # Extract iframe content
|
||||
remove_overlay_elements=True # Remove popups/modals that might block iframes
|
||||
)
|
||||
```
|
||||
|
||||
## Structured Content Selection
|
||||
|
||||
### Using LLMs for Smart Selection
|
||||
|
||||
Use LLMs to intelligently extract specific types of content:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
class ArticleContent(BaseModel):
|
||||
title: str
|
||||
main_points: List[str]
|
||||
conclusion: str
|
||||
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider="ollama/nemotron", # Works with any supported LLM
|
||||
schema=ArticleContent.schema(),
|
||||
instruction="Extract the main article title, key points, and conclusion"
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
article = json.loads(result.extracted_content)
|
||||
```
|
||||
|
||||
### Pattern-Based Selection
|
||||
|
||||
For repeated content patterns (like product listings, news feeds):
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
schema = {
|
||||
"name": "News Articles",
|
||||
"baseSelector": "article.news-item", # Repeated element
|
||||
"fields": [
|
||||
{"name": "headline", "selector": "h2", "type": "text"},
|
||||
{"name": "summary", "selector": ".summary", "type": "text"},
|
||||
{"name": "category", "selector": ".category", "type": "text"},
|
||||
{
|
||||
"name": "metadata",
|
||||
"type": "nested",
|
||||
"fields": [
|
||||
{"name": "author", "selector": ".author", "type": "text"},
|
||||
{"name": "date", "selector": ".date", "type": "text"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
strategy = JsonCssExtractionStrategy(schema)
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
articles = json.loads(result.extracted_content)
|
||||
```
|
||||
|
||||
## Domain-Based Filtering
|
||||
|
||||
Control content based on domains:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
exclude_domains=["ads.com", "tracker.com"],
|
||||
exclude_social_media_domains=["facebook.com", "twitter.com"], # Custom social media domains to exclude
|
||||
exclude_social_media_links=True
|
||||
)
|
||||
```
|
||||
|
||||
## Media Selection
|
||||
|
||||
Select specific types of media:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Access different media types
|
||||
images = result.media["images"] # List of image details
|
||||
videos = result.media["videos"] # List of video details
|
||||
audios = result.media["audios"] # List of audio details
|
||||
|
||||
# Image with metadata
|
||||
for image in images:
|
||||
print(f"URL: {image['src']}")
|
||||
print(f"Alt text: {image['alt']}")
|
||||
print(f"Description: {image['desc']}")
|
||||
print(f"Relevance score: {image['score']}")
|
||||
```
|
||||
|
||||
## Comprehensive Example
|
||||
|
||||
Here's how to combine different selection methods:
|
||||
|
||||
```python
|
||||
async def extract_article_content(url: str):
|
||||
# Define structured extraction
|
||||
article_schema = {
|
||||
"name": "Article",
|
||||
"baseSelector": "article.main",
|
||||
"fields": [
|
||||
{"name": "title", "selector": "h1", "type": "text"},
|
||||
{"name": "content", "selector": ".content", "type": "text"}
|
||||
]
|
||||
}
|
||||
|
||||
# Define LLM extraction
|
||||
class ArticleAnalysis(BaseModel):
|
||||
key_points: List[str]
|
||||
sentiment: str
|
||||
category: str
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Get structured content
|
||||
pattern_result = await crawler.arun(
|
||||
url=url,
|
||||
extraction_strategy=JsonCssExtractionStrategy(article_schema),
|
||||
word_count_threshold=10,
|
||||
excluded_tags=['nav', 'footer'],
|
||||
exclude_external_links=True
|
||||
)
|
||||
|
||||
# Get semantic analysis
|
||||
analysis_result = await crawler.arun(
|
||||
url=url,
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="ollama/nemotron",
|
||||
schema=ArticleAnalysis.schema(),
|
||||
instruction="Analyze the article content"
|
||||
)
|
||||
)
|
||||
|
||||
# Combine results
|
||||
return {
|
||||
"article": json.loads(pattern_result.extracted_content),
|
||||
"analysis": json.loads(analysis_result.extracted_content),
|
||||
"media": pattern_result.media
|
||||
}
|
||||
```
|
||||
459
docs/md_v2/basic/docker-deploymeny.md
Normal file
459
docs/md_v2/basic/docker-deploymeny.md
Normal file
@@ -0,0 +1,459 @@
|
||||
# Docker Deployment
|
||||
|
||||
Crawl4AI provides official Docker images for easy deployment and scalability. This guide covers installation, configuration, and usage of Crawl4AI in Docker environments.
|
||||
|
||||
## Quick Start 🚀
|
||||
|
||||
Pull and run the basic version:
|
||||
|
||||
```bash
|
||||
docker pull unclecode/crawl4ai:basic
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic
|
||||
```
|
||||
|
||||
Test the deployment:
|
||||
```python
|
||||
import requests
|
||||
|
||||
# Test health endpoint
|
||||
health = requests.get("http://localhost:11235/health")
|
||||
print("Health check:", health.json())
|
||||
|
||||
# Test basic crawl
|
||||
response = requests.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 10
|
||||
}
|
||||
)
|
||||
task_id = response.json()["task_id"]
|
||||
print("Task ID:", task_id)
|
||||
```
|
||||
|
||||
## Available Images 🏷️
|
||||
|
||||
- `unclecode/crawl4ai:basic` - Basic web crawling capabilities
|
||||
- `unclecode/crawl4ai:all` - Full installation with all features
|
||||
- `unclecode/crawl4ai:gpu` - GPU-enabled version for ML features
|
||||
|
||||
## Configuration Options 🔧
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
docker run -p 11235:11235 \
|
||||
-e MAX_CONCURRENT_TASKS=5 \
|
||||
-e OPENAI_API_KEY=your_key \
|
||||
unclecode/crawl4ai:all
|
||||
```
|
||||
|
||||
### Volume Mounting
|
||||
|
||||
Mount a directory for persistent data:
|
||||
```bash
|
||||
docker run -p 11235:11235 \
|
||||
-v $(pwd)/data:/app/data \
|
||||
unclecode/crawl4ai:all
|
||||
```
|
||||
|
||||
### Resource Limits
|
||||
|
||||
Control container resources:
|
||||
```bash
|
||||
docker run -p 11235:11235 \
|
||||
--memory=4g \
|
||||
--cpus=2 \
|
||||
unclecode/crawl4ai:all
|
||||
```
|
||||
|
||||
## Usage Examples 📝
|
||||
|
||||
### Basic Crawling
|
||||
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 10
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/crawl", json=request)
|
||||
task_id = response.json()["task_id"]
|
||||
|
||||
# Get results
|
||||
result = requests.get(f"http://localhost:11235/task/{task_id}")
|
||||
```
|
||||
|
||||
### Structured Data Extraction
|
||||
|
||||
```python
|
||||
schema = {
|
||||
"name": "Crypto Prices",
|
||||
"baseSelector": ".cds-tableRow-t45thuk",
|
||||
"fields": [
|
||||
{
|
||||
"name": "crypto",
|
||||
"selector": "td:nth-child(1) h2",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": "td:nth-child(2)",
|
||||
"type": "text",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
request = {
|
||||
"urls": "https://www.coinbase.com/explore",
|
||||
"extraction_config": {
|
||||
"type": "json_css",
|
||||
"params": {"schema": schema}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Dynamic Content Handling
|
||||
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"js_code": [
|
||||
"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
|
||||
],
|
||||
"wait_for": "article.tease-card:nth-child(10)"
|
||||
}
|
||||
```
|
||||
|
||||
### AI-Powered Extraction (Full Version)
|
||||
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"extraction_config": {
|
||||
"type": "cosine",
|
||||
"params": {
|
||||
"semantic_filter": "business finance economy",
|
||||
"word_count_threshold": 10,
|
||||
"max_dist": 0.2,
|
||||
"top_k": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Platform-Specific Instructions 💻
|
||||
|
||||
### macOS
|
||||
```bash
|
||||
docker pull unclecode/crawl4ai:basic
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic
|
||||
```
|
||||
|
||||
### Ubuntu
|
||||
```bash
|
||||
# Basic version
|
||||
docker pull unclecode/crawl4ai:basic
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic
|
||||
|
||||
# With GPU support
|
||||
docker pull unclecode/crawl4ai:gpu
|
||||
docker run --gpus all -p 11235:11235 unclecode/crawl4ai:gpu
|
||||
```
|
||||
|
||||
### Windows (PowerShell)
|
||||
```powershell
|
||||
docker pull unclecode/crawl4ai:basic
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic
|
||||
```
|
||||
|
||||
## Testing 🧪
|
||||
|
||||
Save this as `test_docker.py`:
|
||||
|
||||
```python
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import sys
|
||||
|
||||
class Crawl4AiTester:
|
||||
def __init__(self, base_url: str = "http://localhost:11235"):
|
||||
self.base_url = base_url
|
||||
|
||||
def submit_and_wait(self, request_data: dict, timeout: int = 300) -> dict:
|
||||
# Submit crawl job
|
||||
response = requests.post(f"{self.base_url}/crawl", json=request_data)
|
||||
task_id = response.json()["task_id"]
|
||||
print(f"Task ID: {task_id}")
|
||||
|
||||
# Poll for result
|
||||
start_time = time.time()
|
||||
while True:
|
||||
if time.time() - start_time > timeout:
|
||||
raise TimeoutError(f"Task {task_id} timeout")
|
||||
|
||||
result = requests.get(f"{self.base_url}/task/{task_id}")
|
||||
status = result.json()
|
||||
|
||||
if status["status"] == "completed":
|
||||
return status
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
def test_deployment():
|
||||
tester = Crawl4AiTester()
|
||||
|
||||
# Test basic crawl
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"priority": 10
|
||||
}
|
||||
|
||||
result = tester.submit_and_wait(request)
|
||||
print("Basic crawl successful!")
|
||||
print(f"Content length: {len(result['result']['markdown'])}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_deployment()
|
||||
```
|
||||
|
||||
## Advanced Configuration ⚙️
|
||||
|
||||
### Crawler Parameters
|
||||
|
||||
The `crawler_params` field allows you to configure the browser instance and crawling behavior. Here are key parameters you can use:
|
||||
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://example.com",
|
||||
"crawler_params": {
|
||||
# Browser Configuration
|
||||
"headless": True, # Run in headless mode
|
||||
"browser_type": "chromium", # chromium/firefox/webkit
|
||||
"user_agent": "custom-agent", # Custom user agent
|
||||
"proxy": "http://proxy:8080", # Proxy configuration
|
||||
|
||||
# Performance & Behavior
|
||||
"page_timeout": 30000, # Page load timeout (ms)
|
||||
"verbose": True, # Enable detailed logging
|
||||
"semaphore_count": 5, # Concurrent request limit
|
||||
|
||||
# Anti-Detection Features
|
||||
"simulate_user": True, # Simulate human behavior
|
||||
"magic": True, # Advanced anti-detection
|
||||
"override_navigator": True, # Override navigator properties
|
||||
|
||||
# Session Management
|
||||
"user_data_dir": "./browser-data", # Browser profile location
|
||||
"use_managed_browser": True, # Use persistent browser
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Extra Parameters
|
||||
|
||||
The `extra` field allows passing additional parameters directly to the crawler's `arun` function:
|
||||
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://example.com",
|
||||
"extra": {
|
||||
"word_count_threshold": 10, # Min words per block
|
||||
"only_text": True, # Extract only text
|
||||
"bypass_cache": True, # Force fresh crawl
|
||||
"process_iframes": True, # Include iframe content
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Complete Examples
|
||||
|
||||
1. **Advanced News Crawling**
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://www.nbcnews.com/business",
|
||||
"crawler_params": {
|
||||
"headless": True,
|
||||
"page_timeout": 30000,
|
||||
"remove_overlay_elements": True # Remove popups
|
||||
},
|
||||
"extra": {
|
||||
"word_count_threshold": 50, # Longer content blocks
|
||||
"bypass_cache": True # Fresh content
|
||||
},
|
||||
"css_selector": ".article-body"
|
||||
}
|
||||
```
|
||||
|
||||
2. **Anti-Detection Configuration**
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://example.com",
|
||||
"crawler_params": {
|
||||
"simulate_user": True,
|
||||
"magic": True,
|
||||
"override_navigator": True,
|
||||
"user_agent": "Mozilla/5.0 ...",
|
||||
"headers": {
|
||||
"Accept-Language": "en-US,en;q=0.9"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **LLM Extraction with Custom Parameters**
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://openai.com/pricing",
|
||||
"extraction_config": {
|
||||
"type": "llm",
|
||||
"params": {
|
||||
"provider": "openai/gpt-4",
|
||||
"schema": pricing_schema
|
||||
}
|
||||
},
|
||||
"crawler_params": {
|
||||
"verbose": True,
|
||||
"page_timeout": 60000
|
||||
},
|
||||
"extra": {
|
||||
"word_count_threshold": 1,
|
||||
"only_text": True
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. **Session-Based Dynamic Content**
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://example.com",
|
||||
"crawler_params": {
|
||||
"session_id": "dynamic_session",
|
||||
"headless": False,
|
||||
"page_timeout": 60000
|
||||
},
|
||||
"js_code": ["window.scrollTo(0, document.body.scrollHeight);"],
|
||||
"wait_for": "js:() => document.querySelectorAll('.item').length > 10",
|
||||
"extra": {
|
||||
"delay_before_return_html": 2.0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
5. **Screenshot with Custom Timing**
|
||||
```python
|
||||
request = {
|
||||
"urls": "https://example.com",
|
||||
"screenshot": True,
|
||||
"crawler_params": {
|
||||
"headless": True,
|
||||
"screenshot_wait_for": ".main-content"
|
||||
},
|
||||
"extra": {
|
||||
"delay_before_return_html": 3.0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Parameter Reference Table
|
||||
|
||||
| Category | Parameter | Type | Description |
|
||||
|----------|-----------|------|-------------|
|
||||
| Browser | headless | bool | Run browser in headless mode |
|
||||
| Browser | browser_type | str | Browser engine selection |
|
||||
| Browser | user_agent | str | Custom user agent string |
|
||||
| Network | proxy | str | Proxy server URL |
|
||||
| Network | headers | dict | Custom HTTP headers |
|
||||
| Timing | page_timeout | int | Page load timeout (ms) |
|
||||
| Timing | delay_before_return_html | float | Wait before capture |
|
||||
| Anti-Detection | simulate_user | bool | Human behavior simulation |
|
||||
| Anti-Detection | magic | bool | Advanced protection |
|
||||
| Session | session_id | str | Browser session ID |
|
||||
| Session | user_data_dir | str | Profile directory |
|
||||
| Content | word_count_threshold | int | Minimum words per block |
|
||||
| Content | only_text | bool | Text-only extraction |
|
||||
| Content | process_iframes | bool | Include iframe content |
|
||||
| Debug | verbose | bool | Detailed logging |
|
||||
| Debug | log_console | bool | Browser console logs |
|
||||
|
||||
## Troubleshooting 🔍
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Connection Refused**
|
||||
```
|
||||
Error: Connection refused at localhost:11235
|
||||
```
|
||||
Solution: Ensure the container is running and ports are properly mapped.
|
||||
|
||||
2. **Resource Limits**
|
||||
```
|
||||
Error: No available slots
|
||||
```
|
||||
Solution: Increase MAX_CONCURRENT_TASKS or container resources.
|
||||
|
||||
3. **GPU Access**
|
||||
```
|
||||
Error: GPU not found
|
||||
```
|
||||
Solution: Ensure proper NVIDIA drivers and use `--gpus all` flag.
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Access container for debugging:
|
||||
```bash
|
||||
docker run -it --entrypoint /bin/bash unclecode/crawl4ai:all
|
||||
```
|
||||
|
||||
View container logs:
|
||||
```bash
|
||||
docker logs [container_id]
|
||||
```
|
||||
|
||||
## Best Practices 🌟
|
||||
|
||||
1. **Resource Management**
|
||||
- Set appropriate memory and CPU limits
|
||||
- Monitor resource usage via health endpoint
|
||||
- Use basic version for simple crawling tasks
|
||||
|
||||
2. **Scaling**
|
||||
- Use multiple containers for high load
|
||||
- Implement proper load balancing
|
||||
- Monitor performance metrics
|
||||
|
||||
3. **Security**
|
||||
- Use environment variables for sensitive data
|
||||
- Implement proper network isolation
|
||||
- Regular security updates
|
||||
|
||||
## API Reference 📚
|
||||
|
||||
### Health Check
|
||||
```http
|
||||
GET /health
|
||||
```
|
||||
|
||||
### Submit Crawl Task
|
||||
```http
|
||||
POST /crawl
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"urls": "string or array",
|
||||
"extraction_config": {
|
||||
"type": "basic|llm|cosine|json_css",
|
||||
"params": {}
|
||||
},
|
||||
"priority": 1-10,
|
||||
"ttl": 3600
|
||||
}
|
||||
```
|
||||
|
||||
### Get Task Status
|
||||
```http
|
||||
GET /task/{task_id}
|
||||
```
|
||||
|
||||
For more details, visit the [official documentation](https://crawl4ai.com/mkdocs/).
|
||||
195
docs/md_v2/basic/output-formats.md
Normal file
195
docs/md_v2/basic/output-formats.md
Normal file
@@ -0,0 +1,195 @@
|
||||
# Output Formats
|
||||
|
||||
Crawl4AI provides multiple output formats to suit different needs, from raw HTML to structured data using LLM or pattern-based extraction.
|
||||
|
||||
## Basic Formats
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Access different formats
|
||||
raw_html = result.html # Original HTML
|
||||
clean_html = result.cleaned_html # Sanitized HTML
|
||||
markdown = result.markdown # Standard markdown
|
||||
fit_md = result.fit_markdown # Most relevant content in markdown
|
||||
```
|
||||
|
||||
## Raw HTML
|
||||
|
||||
Original, unmodified HTML from the webpage. Useful when you need to:
|
||||
- Preserve the exact page structure
|
||||
- Process HTML with your own tools
|
||||
- Debug page issues
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
print(result.html) # Complete HTML including headers, scripts, etc.
|
||||
```
|
||||
|
||||
## Cleaned HTML
|
||||
|
||||
Sanitized HTML with unnecessary elements removed. Automatically:
|
||||
- Removes scripts and styles
|
||||
- Cleans up formatting
|
||||
- Preserves semantic structure
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
excluded_tags=['form', 'header', 'footer'], # Additional tags to remove
|
||||
keep_data_attributes=False # Remove data-* attributes
|
||||
)
|
||||
print(result.cleaned_html)
|
||||
```
|
||||
|
||||
## Standard Markdown
|
||||
|
||||
HTML converted to clean markdown format. Great for:
|
||||
- Content analysis
|
||||
- Documentation
|
||||
- Readability
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
include_links_on_markdown=True # Include links in markdown
|
||||
)
|
||||
print(result.markdown)
|
||||
```
|
||||
|
||||
## Fit Markdown
|
||||
|
||||
Most relevant content extracted and converted to markdown. Ideal for:
|
||||
- Article extraction
|
||||
- Main content focus
|
||||
- Removing boilerplate
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
print(result.fit_markdown) # Only the main content
|
||||
```
|
||||
|
||||
## Structured Data Extraction
|
||||
|
||||
Crawl4AI offers two powerful approaches for structured data extraction:
|
||||
|
||||
### 1. LLM-Based Extraction
|
||||
|
||||
Use any LLM (OpenAI, HuggingFace, Ollama, etc.) to extract structured data with high accuracy:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
class KnowledgeGraph(BaseModel):
|
||||
entities: List[dict]
|
||||
relationships: List[dict]
|
||||
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider="ollama/nemotron", # or "huggingface/...", "ollama/..."
|
||||
api_token="your-token", # not needed for Ollama
|
||||
schema=KnowledgeGraph.schema(),
|
||||
instruction="Extract entities and relationships from the content"
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
knowledge_graph = json.loads(result.extracted_content)
|
||||
```
|
||||
|
||||
### 2. Pattern-Based Extraction
|
||||
|
||||
For pages with repetitive patterns (e.g., product listings, article feeds), use JsonCssExtractionStrategy:
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
schema = {
|
||||
"name": "Product Listing",
|
||||
"baseSelector": ".product-card", # Repeated element
|
||||
"fields": [
|
||||
{"name": "title", "selector": "h2", "type": "text"},
|
||||
{"name": "price", "selector": ".price", "type": "text"},
|
||||
{"name": "description", "selector": ".desc", "type": "text"}
|
||||
]
|
||||
}
|
||||
|
||||
strategy = JsonCssExtractionStrategy(schema)
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
products = json.loads(result.extracted_content)
|
||||
```
|
||||
|
||||
## Content Customization
|
||||
|
||||
### HTML to Text Options
|
||||
|
||||
Configure markdown conversion:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
html2text={
|
||||
"escape_dot": False,
|
||||
"body_width": 0,
|
||||
"protect_links": True,
|
||||
"unicode_snob": True
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### Content Filters
|
||||
|
||||
Control what content is included:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
word_count_threshold=10, # Minimum words per block
|
||||
exclude_external_links=True, # Remove external links
|
||||
exclude_external_images=True, # Remove external images
|
||||
excluded_tags=['form', 'nav'] # Remove specific HTML tags
|
||||
)
|
||||
```
|
||||
|
||||
## Comprehensive Example
|
||||
|
||||
Here's how to use multiple output formats together:
|
||||
|
||||
```python
|
||||
async def crawl_content(url: str):
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Extract main content with fit markdown
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
word_count_threshold=10,
|
||||
exclude_external_links=True
|
||||
)
|
||||
|
||||
# Get structured data using LLM
|
||||
llm_result = await crawler.arun(
|
||||
url=url,
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="ollama/nemotron",
|
||||
schema=YourSchema.schema(),
|
||||
instruction="Extract key information"
|
||||
)
|
||||
)
|
||||
|
||||
# Get repeated patterns (if any)
|
||||
pattern_result = await crawler.arun(
|
||||
url=url,
|
||||
extraction_strategy=JsonCssExtractionStrategy(your_schema)
|
||||
)
|
||||
|
||||
return {
|
||||
"main_content": result.fit_markdown,
|
||||
"structured_data": json.loads(llm_result.extracted_content),
|
||||
"pattern_data": json.loads(pattern_result.extracted_content),
|
||||
"media": result.media
|
||||
}
|
||||
```
|
||||
207
docs/md_v2/basic/page-interaction.md
Normal file
207
docs/md_v2/basic/page-interaction.md
Normal file
@@ -0,0 +1,207 @@
|
||||
# Page Interaction
|
||||
|
||||
Crawl4AI provides powerful features for interacting with dynamic webpages, handling JavaScript execution, and managing page events.
|
||||
|
||||
## JavaScript Execution
|
||||
|
||||
### Basic Execution
|
||||
|
||||
```python
|
||||
# Single JavaScript command
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);"
|
||||
)
|
||||
|
||||
# Multiple commands
|
||||
js_commands = [
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more').click();",
|
||||
"document.querySelector('#consent-button').click();"
|
||||
]
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=js_commands
|
||||
)
|
||||
```
|
||||
|
||||
## Wait Conditions
|
||||
|
||||
### CSS-Based Waiting
|
||||
|
||||
Wait for elements to appear:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
wait_for="css:.dynamic-content" # Wait for element with class 'dynamic-content'
|
||||
)
|
||||
```
|
||||
|
||||
### JavaScript-Based Waiting
|
||||
|
||||
Wait for custom conditions:
|
||||
|
||||
```python
|
||||
# Wait for number of elements
|
||||
wait_condition = """() => {
|
||||
return document.querySelectorAll('.item').length > 10;
|
||||
}"""
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
wait_for=f"js:{wait_condition}"
|
||||
)
|
||||
|
||||
# Wait for dynamic content to load
|
||||
wait_for_content = """() => {
|
||||
const content = document.querySelector('.content');
|
||||
return content && content.innerText.length > 100;
|
||||
}"""
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
wait_for=f"js:{wait_for_content}"
|
||||
)
|
||||
```
|
||||
|
||||
## Handling Dynamic Content
|
||||
|
||||
### Load More Content
|
||||
|
||||
Handle infinite scroll or load more buttons:
|
||||
|
||||
```python
|
||||
# Scroll and wait pattern
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=[
|
||||
# Scroll to bottom
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
# Click load more if exists
|
||||
"const loadMore = document.querySelector('.load-more'); if(loadMore) loadMore.click();"
|
||||
],
|
||||
# Wait for new content
|
||||
wait_for="js:() => document.querySelectorAll('.item').length > previousCount"
|
||||
)
|
||||
```
|
||||
|
||||
### Form Interaction
|
||||
|
||||
Handle forms and inputs:
|
||||
|
||||
```python
|
||||
js_form_interaction = """
|
||||
// Fill form fields
|
||||
document.querySelector('#search').value = 'search term';
|
||||
// Submit form
|
||||
document.querySelector('form').submit();
|
||||
"""
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=js_form_interaction,
|
||||
wait_for="css:.results" # Wait for results to load
|
||||
)
|
||||
```
|
||||
|
||||
## Timing Control
|
||||
|
||||
### Delays and Timeouts
|
||||
|
||||
Control timing of interactions:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
page_timeout=60000, # Page load timeout (ms)
|
||||
delay_before_return_html=2.0, # Wait before capturing content
|
||||
)
|
||||
```
|
||||
|
||||
## Complex Interactions Example
|
||||
|
||||
Here's an example of handling a dynamic page with multiple interactions:
|
||||
|
||||
```python
|
||||
async def crawl_dynamic_content():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Initial page load
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
# Handle cookie consent
|
||||
js_code="document.querySelector('.cookie-accept')?.click();",
|
||||
wait_for="css:.main-content"
|
||||
)
|
||||
|
||||
# Load more content
|
||||
session_id = "dynamic_session" # Keep session for multiple interactions
|
||||
|
||||
for page in range(3): # Load 3 pages of content
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
session_id=session_id,
|
||||
js_code=[
|
||||
# Scroll to bottom
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
# Store current item count
|
||||
"window.previousCount = document.querySelectorAll('.item').length;",
|
||||
# Click load more
|
||||
"document.querySelector('.load-more')?.click();"
|
||||
],
|
||||
# Wait for new items
|
||||
wait_for="""() => {
|
||||
const currentCount = document.querySelectorAll('.item').length;
|
||||
return currentCount > window.previousCount;
|
||||
}""",
|
||||
# Only execute JS without reloading page
|
||||
js_only=True if page > 0 else False
|
||||
)
|
||||
|
||||
# Process content after each load
|
||||
print(f"Page {page + 1} items:", len(result.cleaned_html))
|
||||
|
||||
# Clean up session
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
```
|
||||
|
||||
## Using with Extraction Strategies
|
||||
|
||||
Combine page interaction with structured extraction:
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy
|
||||
|
||||
# Pattern-based extraction after interaction
|
||||
schema = {
|
||||
"name": "Dynamic Items",
|
||||
"baseSelector": ".item",
|
||||
"fields": [
|
||||
{"name": "title", "selector": "h2", "type": "text"},
|
||||
{"name": "description", "selector": ".desc", "type": "text"}
|
||||
]
|
||||
}
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="css:.item:nth-child(10)", # Wait for 10 items
|
||||
extraction_strategy=JsonCssExtractionStrategy(schema)
|
||||
)
|
||||
|
||||
# Or use LLM to analyze dynamic content
|
||||
class ContentAnalysis(BaseModel):
|
||||
topics: List[str]
|
||||
summary: str
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="document.querySelector('.show-more').click();",
|
||||
wait_for="css:.full-content",
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="ollama/nemotron",
|
||||
schema=ContentAnalysis.schema(),
|
||||
instruction="Analyze the full content"
|
||||
)
|
||||
)
|
||||
```
|
||||
297
docs/md_v2/basic/quickstart.md
Normal file
297
docs/md_v2/basic/quickstart.md
Normal file
@@ -0,0 +1,297 @@
|
||||
# Quick Start Guide 🚀
|
||||
|
||||
Welcome to the Crawl4AI Quickstart Guide! In this tutorial, we'll walk you through the basic usage of Crawl4AI with a friendly and humorous tone. We'll cover everything from basic usage to advanced features like chunking and extraction strategies, all with the power of asynchronous programming. Let's dive in! 🌟
|
||||
|
||||
## Getting Started 🛠️
|
||||
|
||||
First, let's import the necessary modules and create an instance of `AsyncWebCrawler`. We'll use an async context manager, which handles the setup and teardown of the crawler for us.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# We'll add our crawling code here
|
||||
pass
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simply provide a URL and let Crawl4AI do the magic!
|
||||
|
||||
```python
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.nbcnews.com/business")
|
||||
print(f"Basic crawl result: {result.markdown[:500]}") # Print first 500 characters
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Taking Screenshots 📸
|
||||
|
||||
Capture screenshots of web pages easily:
|
||||
|
||||
```python
|
||||
async def capture_and_save_screenshot(url: str, output_path: str):
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
screenshot=True,
|
||||
bypass_cache=True
|
||||
)
|
||||
|
||||
if result.success and result.screenshot:
|
||||
import base64
|
||||
screenshot_data = base64.b64decode(result.screenshot)
|
||||
with open(output_path, 'wb') as f:
|
||||
f.write(screenshot_data)
|
||||
print(f"Screenshot saved successfully to {output_path}")
|
||||
else:
|
||||
print("Failed to capture screenshot")
|
||||
```
|
||||
|
||||
### Browser Selection 🌐
|
||||
|
||||
Crawl4AI supports multiple browser engines. Here's how to use different browsers:
|
||||
|
||||
```python
|
||||
# Use Firefox
|
||||
async with AsyncWebCrawler(browser_type="firefox", verbose=True, headless=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
|
||||
|
||||
# Use WebKit
|
||||
async with AsyncWebCrawler(browser_type="webkit", verbose=True, headless=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
|
||||
|
||||
# Use Chromium (default)
|
||||
async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.example.com", bypass_cache=True)
|
||||
```
|
||||
|
||||
### User Simulation 🎭
|
||||
|
||||
Simulate real user behavior to avoid detection:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler(verbose=True, headless=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="YOUR-URL-HERE",
|
||||
bypass_cache=True,
|
||||
simulate_user=True, # Causes random mouse movements and clicks
|
||||
override_navigator=True # Makes the browser appear more like a real user
|
||||
)
|
||||
```
|
||||
|
||||
### Understanding Parameters 🧠
|
||||
|
||||
By default, Crawl4AI caches the results of your crawls. This means that subsequent crawls of the same URL will be much faster! Let's see this in action.
|
||||
|
||||
```python
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# First crawl (caches the result)
|
||||
result1 = await crawler.arun(url="https://www.nbcnews.com/business")
|
||||
print(f"First crawl result: {result1.markdown[:100]}...")
|
||||
|
||||
# Force to crawl again
|
||||
result2 = await crawler.arun(url="https://www.nbcnews.com/business", bypass_cache=True)
|
||||
print(f"Second crawl result: {result2.markdown[:100]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Adding a Chunking Strategy 🧩
|
||||
|
||||
Let's add a chunking strategy: `RegexChunking`! This strategy splits the text based on a given regex pattern.
|
||||
|
||||
```python
|
||||
from crawl4ai.chunking_strategy import RegexChunking
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
chunking_strategy=RegexChunking(patterns=["\n\n"])
|
||||
)
|
||||
print(f"RegexChunking result: {result.extracted_content[:200]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Using LLMExtractionStrategy with Different Providers 🤖
|
||||
|
||||
Crawl4AI supports multiple LLM providers for extraction:
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
class OpenAIModelFee(BaseModel):
|
||||
model_name: str = Field(..., description="Name of the OpenAI model.")
|
||||
input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
|
||||
output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
|
||||
|
||||
# OpenAI
|
||||
await extract_structured_data_using_llm("openai/gpt-4o", os.getenv("OPENAI_API_KEY"))
|
||||
|
||||
# Hugging Face
|
||||
await extract_structured_data_using_llm(
|
||||
"huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
os.getenv("HUGGINGFACE_API_KEY")
|
||||
)
|
||||
|
||||
# Ollama
|
||||
await extract_structured_data_using_llm("ollama/llama3.2")
|
||||
|
||||
# With custom headers
|
||||
custom_headers = {
|
||||
"Authorization": "Bearer your-custom-token",
|
||||
"X-Custom-Header": "Some-Value"
|
||||
}
|
||||
await extract_structured_data_using_llm(extra_headers=custom_headers)
|
||||
```
|
||||
|
||||
### Knowledge Graph Generation 🕸️
|
||||
|
||||
Generate knowledge graphs from web content:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from typing import List
|
||||
|
||||
class Entity(BaseModel):
|
||||
name: str
|
||||
description: str
|
||||
|
||||
class Relationship(BaseModel):
|
||||
entity1: Entity
|
||||
entity2: Entity
|
||||
description: str
|
||||
relation_type: str
|
||||
|
||||
class KnowledgeGraph(BaseModel):
|
||||
entities: List[Entity]
|
||||
relationships: List[Relationship]
|
||||
|
||||
extraction_strategy = LLMExtractionStrategy(
|
||||
provider='openai/gpt-4o-mini',
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
schema=KnowledgeGraph.model_json_schema(),
|
||||
extraction_type="schema",
|
||||
instruction="Extract entities and relationships from the given text."
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://paulgraham.com/love.html",
|
||||
bypass_cache=True,
|
||||
extraction_strategy=extraction_strategy
|
||||
)
|
||||
```
|
||||
|
||||
### Advanced Session-Based Crawling with Dynamic Content 🔄
|
||||
|
||||
For modern web applications with dynamic content loading, here's how to handle pagination and content updates:
|
||||
|
||||
```python
|
||||
async def crawl_dynamic_content():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
url = "https://github.com/microsoft/TypeScript/commits/main"
|
||||
session_id = "typescript_commits_session"
|
||||
|
||||
js_next_page = """
|
||||
const button = document.querySelector('a[data-testid="pagination-next-button"]');
|
||||
if (button) button.click();
|
||||
"""
|
||||
|
||||
wait_for = """() => {
|
||||
const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');
|
||||
if (commits.length === 0) return false;
|
||||
const firstCommit = commits[0].textContent.trim();
|
||||
return firstCommit !== window.firstCommit;
|
||||
}"""
|
||||
|
||||
schema = {
|
||||
"name": "Commit Extractor",
|
||||
"baseSelector": "li.Box-sc-g0xbh4-0",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h4.markdown-title",
|
||||
"type": "text",
|
||||
"transform": "strip",
|
||||
},
|
||||
],
|
||||
}
|
||||
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
|
||||
|
||||
for page in range(3): # Crawl 3 pages
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
session_id=session_id,
|
||||
css_selector="li.Box-sc-g0xbh4-0",
|
||||
extraction_strategy=extraction_strategy,
|
||||
js_code=js_next_page if page > 0 else None,
|
||||
wait_for=wait_for if page > 0 else None,
|
||||
js_only=page > 0,
|
||||
bypass_cache=True,
|
||||
headless=False,
|
||||
)
|
||||
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
```
|
||||
|
||||
### Handling Overlays and Fitting Content 📏
|
||||
|
||||
Remove overlay elements and fit content appropriately:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler(headless=False) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="your-url-here",
|
||||
bypass_cache=True,
|
||||
word_count_threshold=10,
|
||||
remove_overlay_elements=True,
|
||||
screenshot=True
|
||||
)
|
||||
```
|
||||
|
||||
## Performance Comparison 🏎️
|
||||
|
||||
Crawl4AI offers impressive performance compared to other solutions:
|
||||
|
||||
```python
|
||||
# Firecrawl comparison
|
||||
from firecrawl import FirecrawlApp
|
||||
app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
|
||||
start = time.time()
|
||||
scrape_status = app.scrape_url(
|
||||
'https://www.nbcnews.com/business',
|
||||
params={'formats': ['markdown', 'html']}
|
||||
)
|
||||
end = time.time()
|
||||
|
||||
# Crawl4AI comparison
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
start = time.time()
|
||||
result = await crawler.arun(
|
||||
url="https://www.nbcnews.com/business",
|
||||
word_count_threshold=0,
|
||||
bypass_cache=True,
|
||||
verbose=False,
|
||||
)
|
||||
end = time.time()
|
||||
```
|
||||
|
||||
Note: Performance comparisons should be conducted in environments with stable and fast internet connections for accurate results.
|
||||
|
||||
## Congratulations! 🎉
|
||||
|
||||
You've made it through the updated Crawl4AI Quickstart Guide! Now you're equipped with even more powerful features to crawl the web asynchronously like a pro! 🕸️
|
||||
|
||||
Happy crawling! 🚀
|
||||
120
docs/md_v2/basic/simple-crawling.md
Normal file
120
docs/md_v2/basic/simple-crawling.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Simple Crawling
|
||||
|
||||
This guide covers the basics of web crawling with Crawl4AI. You'll learn how to set up a crawler, make your first request, and understand the response.
|
||||
|
||||
## Basic Usage
|
||||
|
||||
Here's the simplest way to crawl a webpage:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
print(result.markdown) # Print clean markdown content
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Understanding the Response
|
||||
|
||||
The `arun()` method returns a `CrawlResult` object with several useful properties. Here's a quick overview (see [CrawlResult](../api/crawl-result.md) for complete details):
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
# Different content formats
|
||||
print(result.html) # Raw HTML
|
||||
print(result.cleaned_html) # Cleaned HTML
|
||||
print(result.markdown) # Markdown version
|
||||
print(result.fit_markdown) # Most relevant content in markdown
|
||||
|
||||
# Check success status
|
||||
print(result.success) # True if crawl succeeded
|
||||
print(result.status_code) # HTTP status code (e.g., 200, 404)
|
||||
|
||||
# Access extracted media and links
|
||||
print(result.media) # Dictionary of found media (images, videos, audio)
|
||||
print(result.links) # Dictionary of internal and external links
|
||||
```
|
||||
|
||||
## Adding Basic Options
|
||||
|
||||
Customize your crawl with these common options:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
word_count_threshold=10, # Minimum words per content block
|
||||
exclude_external_links=True, # Remove external links
|
||||
remove_overlay_elements=True, # Remove popups/modals
|
||||
process_iframes=True # Process iframe content
|
||||
)
|
||||
```
|
||||
|
||||
## Handling Errors
|
||||
|
||||
Always check if the crawl was successful:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
if not result.success:
|
||||
print(f"Crawl failed: {result.error_message}")
|
||||
print(f"Status code: {result.status_code}")
|
||||
```
|
||||
|
||||
## Logging and Debugging
|
||||
|
||||
Enable verbose mode for detailed logging:
|
||||
|
||||
```python
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
Here's a more comprehensive example showing common usage patterns:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
# Content filtering
|
||||
word_count_threshold=10,
|
||||
excluded_tags=['form', 'header'],
|
||||
exclude_external_links=True,
|
||||
|
||||
# Content processing
|
||||
process_iframes=True,
|
||||
remove_overlay_elements=True,
|
||||
|
||||
# Cache control
|
||||
bypass_cache=False # Use cache if available
|
||||
)
|
||||
|
||||
if result.success:
|
||||
# Print clean content
|
||||
print("Content:", result.markdown[:500]) # First 500 chars
|
||||
|
||||
# Process images
|
||||
for image in result.media["images"]:
|
||||
print(f"Found image: {image['src']}")
|
||||
|
||||
# Process links
|
||||
for link in result.links["internal"]:
|
||||
print(f"Internal link: {link['href']}")
|
||||
|
||||
else:
|
||||
print(f"Crawl failed: {result.error_message}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
222
docs/md_v2/extraction/cosine.md
Normal file
222
docs/md_v2/extraction/cosine.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Cosine Strategy
|
||||
|
||||
The Cosine Strategy in Crawl4AI uses similarity-based clustering to identify and extract relevant content sections from web pages. This strategy is particularly useful when you need to find and extract content based on semantic similarity rather than structural patterns.
|
||||
|
||||
## How It Works
|
||||
|
||||
The Cosine Strategy:
|
||||
1. Breaks down page content into meaningful chunks
|
||||
2. Converts text into vector representations
|
||||
3. Calculates similarity between chunks
|
||||
4. Clusters similar content together
|
||||
5. Ranks and filters content based on relevance
|
||||
|
||||
## Basic Usage
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="product reviews", # Target content type
|
||||
word_count_threshold=10, # Minimum words per cluster
|
||||
sim_threshold=0.3 # Similarity threshold
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/reviews",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
|
||||
content = result.extracted_content
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Core Parameters
|
||||
|
||||
```python
|
||||
CosineStrategy(
|
||||
# Content Filtering
|
||||
semantic_filter: str = None, # Keywords/topic for content filtering
|
||||
word_count_threshold: int = 10, # Minimum words per cluster
|
||||
sim_threshold: float = 0.3, # Similarity threshold (0.0 to 1.0)
|
||||
|
||||
# Clustering Parameters
|
||||
max_dist: float = 0.2, # Maximum distance for clustering
|
||||
linkage_method: str = 'ward', # Clustering linkage method
|
||||
top_k: int = 3, # Number of top categories to extract
|
||||
|
||||
# Model Configuration
|
||||
model_name: str = 'sentence-transformers/all-MiniLM-L6-v2', # Embedding model
|
||||
|
||||
verbose: bool = False # Enable logging
|
||||
)
|
||||
```
|
||||
|
||||
### Parameter Details
|
||||
|
||||
1. **semantic_filter**
|
||||
- Sets the target topic or content type
|
||||
- Use keywords relevant to your desired content
|
||||
- Example: "technical specifications", "user reviews", "pricing information"
|
||||
|
||||
2. **sim_threshold**
|
||||
- Controls how similar content must be to be grouped together
|
||||
- Higher values (e.g., 0.8) mean stricter matching
|
||||
- Lower values (e.g., 0.3) allow more variation
|
||||
```python
|
||||
# Strict matching
|
||||
strategy = CosineStrategy(sim_threshold=0.8)
|
||||
|
||||
# Loose matching
|
||||
strategy = CosineStrategy(sim_threshold=0.3)
|
||||
```
|
||||
|
||||
3. **word_count_threshold**
|
||||
- Filters out short content blocks
|
||||
- Helps eliminate noise and irrelevant content
|
||||
```python
|
||||
# Only consider substantial paragraphs
|
||||
strategy = CosineStrategy(word_count_threshold=50)
|
||||
```
|
||||
|
||||
4. **top_k**
|
||||
- Number of top content clusters to return
|
||||
- Higher values return more diverse content
|
||||
```python
|
||||
# Get top 5 most relevant content clusters
|
||||
strategy = CosineStrategy(top_k=5)
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Article Content Extraction
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="main article content",
|
||||
word_count_threshold=100, # Longer blocks for articles
|
||||
top_k=1 # Usually want single main content
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/blog/post",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Product Review Analysis
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="customer reviews and ratings",
|
||||
word_count_threshold=20, # Reviews can be shorter
|
||||
top_k=10, # Get multiple reviews
|
||||
sim_threshold=0.4 # Allow variety in review content
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Technical Documentation
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="technical specifications documentation",
|
||||
word_count_threshold=30,
|
||||
sim_threshold=0.6, # Stricter matching for technical content
|
||||
max_dist=0.3 # Allow related technical sections
|
||||
)
|
||||
```
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Custom Clustering
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
linkage_method='complete', # Alternative clustering method
|
||||
max_dist=0.4, # Larger clusters
|
||||
model_name='sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2' # Multilingual support
|
||||
)
|
||||
```
|
||||
|
||||
### Content Filtering Pipeline
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="pricing plans features",
|
||||
word_count_threshold=15,
|
||||
sim_threshold=0.5,
|
||||
top_k=3
|
||||
)
|
||||
|
||||
async def extract_pricing_features(url: str):
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
|
||||
if result.success:
|
||||
content = json.loads(result.extracted_content)
|
||||
return {
|
||||
'pricing_features': content,
|
||||
'clusters': len(content),
|
||||
'similarity_scores': [item['score'] for item in content]
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Adjust Thresholds Iteratively**
|
||||
- Start with default values
|
||||
- Adjust based on results
|
||||
- Monitor clustering quality
|
||||
|
||||
2. **Choose Appropriate Word Count Thresholds**
|
||||
- Higher for articles (100+)
|
||||
- Lower for reviews/comments (20+)
|
||||
- Medium for product descriptions (50+)
|
||||
|
||||
3. **Optimize Performance**
|
||||
```python
|
||||
strategy = CosineStrategy(
|
||||
word_count_threshold=10, # Filter early
|
||||
top_k=5, # Limit results
|
||||
verbose=True # Monitor performance
|
||||
)
|
||||
```
|
||||
|
||||
4. **Handle Different Content Types**
|
||||
```python
|
||||
# For mixed content pages
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="product features",
|
||||
sim_threshold=0.4, # More flexible matching
|
||||
max_dist=0.3, # Larger clusters
|
||||
top_k=3 # Multiple relevant sections
|
||||
)
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
try:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
|
||||
if result.success:
|
||||
content = json.loads(result.extracted_content)
|
||||
if not content:
|
||||
print("No relevant content found")
|
||||
else:
|
||||
print(f"Extraction failed: {result.error_message}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error during extraction: {str(e)}")
|
||||
```
|
||||
|
||||
The Cosine Strategy is particularly effective when:
|
||||
- Content structure is inconsistent
|
||||
- You need semantic understanding
|
||||
- You want to find similar content blocks
|
||||
- Structure-based extraction (CSS/XPath) isn't reliable
|
||||
|
||||
It works well with other strategies and can be used as a pre-processing step for LLM-based extraction.
|
||||
@@ -139,4 +139,4 @@ This advanced example demonstrates how to:
|
||||
|
||||
By mastering the `JsonCssExtractionStrategy`, you can efficiently extract structured data from a wide variety of web pages, making it a valuable tool in your web scraping toolkit.
|
||||
|
||||
For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](../full_details/advanced_jsoncss_extraction.md).
|
||||
For more details on schema definitions and advanced extraction strategies, check out the[Advanced JsonCssExtraction](./css-advanced.md).
|
||||
@@ -27,7 +27,7 @@ async def extract_openai_fees():
|
||||
url=url,
|
||||
word_count_threshold=1,
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o",
|
||||
provider="openai/gpt-4o", # Or use ollama like provider="ollama/nemotron"
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
schema=OpenAIModelFee.model_json_schema(),
|
||||
extraction_type="schema",
|
||||
197
docs/md_v2/extraction/overview.md
Normal file
197
docs/md_v2/extraction/overview.md
Normal file
@@ -0,0 +1,197 @@
|
||||
# Extraction Strategies Overview
|
||||
|
||||
Crawl4AI provides powerful extraction strategies to help you get structured data from web pages. Each strategy is designed for specific use cases and offers different approaches to data extraction.
|
||||
|
||||
## Available Strategies
|
||||
|
||||
### [LLM-Based Extraction](llm.md)
|
||||
|
||||
`LLMExtractionStrategy` uses Language Models to extract structured data from web content. This approach is highly flexible and can understand content semantically.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
class Product(BaseModel):
|
||||
name: str
|
||||
price: float
|
||||
description: str
|
||||
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider="ollama/llama2",
|
||||
schema=Product.schema(),
|
||||
instruction="Extract product details from the page"
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/product",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
```
|
||||
|
||||
**Best for:**
|
||||
- Complex data structures
|
||||
- Content requiring interpretation
|
||||
- Flexible content formats
|
||||
- Natural language processing
|
||||
|
||||
### [CSS-Based Extraction](css.md)
|
||||
|
||||
`JsonCssExtractionStrategy` extracts data using CSS selectors. This is fast, reliable, and perfect for consistently structured pages.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
schema = {
|
||||
"name": "Product Listing",
|
||||
"baseSelector": ".product-card",
|
||||
"fields": [
|
||||
{"name": "title", "selector": "h2", "type": "text"},
|
||||
{"name": "price", "selector": ".price", "type": "text"},
|
||||
{"name": "image", "selector": "img", "type": "attribute", "attribute": "src"}
|
||||
]
|
||||
}
|
||||
|
||||
strategy = JsonCssExtractionStrategy(schema)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/products",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
```
|
||||
|
||||
**Best for:**
|
||||
- E-commerce product listings
|
||||
- News article collections
|
||||
- Structured content pages
|
||||
- High-performance needs
|
||||
|
||||
### [Cosine Strategy](cosine.md)
|
||||
|
||||
`CosineStrategy` uses similarity-based clustering to identify and extract relevant content sections.
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import CosineStrategy
|
||||
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="product reviews", # Content focus
|
||||
word_count_threshold=10, # Minimum words per cluster
|
||||
sim_threshold=0.3, # Similarity threshold
|
||||
max_dist=0.2, # Maximum cluster distance
|
||||
top_k=3 # Number of top clusters to extract
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/reviews",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
```
|
||||
|
||||
**Best for:**
|
||||
- Content similarity analysis
|
||||
- Topic clustering
|
||||
- Relevant content extraction
|
||||
- Pattern recognition in text
|
||||
|
||||
## Strategy Selection Guide
|
||||
|
||||
Choose your strategy based on these factors:
|
||||
|
||||
1. **Content Structure**
|
||||
- Well-structured HTML → Use CSS Strategy
|
||||
- Natural language text → Use LLM Strategy
|
||||
- Mixed/Complex content → Use Cosine Strategy
|
||||
|
||||
2. **Performance Requirements**
|
||||
- Fastest: CSS Strategy
|
||||
- Moderate: Cosine Strategy
|
||||
- Variable: LLM Strategy (depends on provider)
|
||||
|
||||
3. **Accuracy Needs**
|
||||
- Highest structure accuracy: CSS Strategy
|
||||
- Best semantic understanding: LLM Strategy
|
||||
- Best content relevance: Cosine Strategy
|
||||
|
||||
## Combining Strategies
|
||||
|
||||
You can combine strategies for more powerful extraction:
|
||||
|
||||
```python
|
||||
# First use CSS strategy for initial structure
|
||||
css_result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=css_strategy
|
||||
)
|
||||
|
||||
# Then use LLM for semantic analysis
|
||||
llm_result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=llm_strategy
|
||||
)
|
||||
```
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
1. **E-commerce Scraping**
|
||||
```python
|
||||
# CSS Strategy for product listings
|
||||
schema = {
|
||||
"name": "Products",
|
||||
"baseSelector": ".product",
|
||||
"fields": [
|
||||
{"name": "name", "selector": ".title", "type": "text"},
|
||||
{"name": "price", "selector": ".price", "type": "text"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
2. **News Article Extraction**
|
||||
```python
|
||||
# LLM Strategy for article content
|
||||
class Article(BaseModel):
|
||||
title: str
|
||||
content: str
|
||||
author: str
|
||||
date: str
|
||||
|
||||
strategy = LLMExtractionStrategy(
|
||||
provider="ollama/llama2",
|
||||
schema=Article.schema()
|
||||
)
|
||||
```
|
||||
|
||||
3. **Content Analysis**
|
||||
```python
|
||||
# Cosine Strategy for topic analysis
|
||||
strategy = CosineStrategy(
|
||||
semantic_filter="technology trends",
|
||||
top_k=5
|
||||
)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Choose the Right Strategy**
|
||||
- Start with CSS for structured data
|
||||
- Use LLM for complex interpretation
|
||||
- Try Cosine for content relevance
|
||||
|
||||
2. **Optimize Performance**
|
||||
- Cache LLM results
|
||||
- Keep CSS selectors specific
|
||||
- Tune similarity thresholds
|
||||
|
||||
3. **Handle Errors**
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
extraction_strategy=strategy
|
||||
)
|
||||
|
||||
if not result.success:
|
||||
print(f"Extraction failed: {result.error_message}")
|
||||
else:
|
||||
data = json.loads(result.extracted_content)
|
||||
```
|
||||
|
||||
Each strategy has its strengths and optimal use cases. Explore the detailed documentation for each strategy to learn more about their specific features and configurations.
|
||||
113
docs/md_v2/index.md
Normal file
113
docs/md_v2/index.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Crawl4AI
|
||||
|
||||
Welcome to the official documentation for Crawl4AI! 🕷️🤖 Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages. This documentation will guide you through the features, usage, and customization of Crawl4AI.
|
||||
|
||||
## Introduction
|
||||
|
||||
Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution with full asynchronous support.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
# Create an instance of AsyncWebCrawler
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Run the crawler on a URL
|
||||
result = await crawler.arun(url="https://www.nbcnews.com/business")
|
||||
|
||||
# Print the extracted content
|
||||
print(result.markdown)
|
||||
|
||||
# Run the async main function
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Key Features ✨
|
||||
|
||||
- 🆓 Completely free and open-source
|
||||
- 🚀 Blazing fast performance, outperforming many paid services
|
||||
- 🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
|
||||
- 📄 Fit markdown generation for extracting main article content.
|
||||
- 🌐 Multi-browser support (Chromium, Firefox, WebKit)
|
||||
- 🌍 Supports crawling multiple URLs simultaneously
|
||||
- 🎨 Extracts and returns all media tags (Images, Audio, and Video)
|
||||
- 🔗 Extracts all external and internal links
|
||||
- 📚 Extracts metadata from the page
|
||||
- 🔄 Custom hooks for authentication, headers, and page modifications
|
||||
- 🕵️ User-agent customization
|
||||
- 🖼️ Takes screenshots of pages with enhanced error handling
|
||||
- 📜 Executes multiple custom JavaScripts before crawling
|
||||
- 📊 Generates structured output without LLM using JsonCssExtractionStrategy
|
||||
- 📚 Various chunking strategies: topic-based, regex, sentence, and more
|
||||
- 🧠 Advanced extraction strategies: cosine clustering, LLM, and more
|
||||
- 🎯 CSS selector support for precise data extraction
|
||||
- 📝 Passes instructions/keywords to refine extraction
|
||||
- 🔒 Proxy support with authentication for enhanced access
|
||||
- 🔄 Session management for complex multi-page crawling
|
||||
- 🌐 Asynchronous architecture for improved performance
|
||||
- 🖼️ Improved image processing with lazy-loading detection
|
||||
- 🕰️ Enhanced handling of delayed content loading
|
||||
- 🔑 Custom headers support for LLM interactions
|
||||
- 🖼️ iframe content extraction for comprehensive analysis
|
||||
- ⏱️ Flexible timeout and delayed content retrieval options
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
Our documentation is organized into several sections:
|
||||
|
||||
### Basic Usage
|
||||
- [Installation](basic/installation.md)
|
||||
- [Quick Start](basic/quickstart.md)
|
||||
- [Simple Crawling](basic/simple-crawling.md)
|
||||
- [Browser Configuration](basic/browser-config.md)
|
||||
- [Content Selection](basic/content-selection.md)
|
||||
- [Output Formats](basic/output-formats.md)
|
||||
- [Page Interaction](basic/page-interaction.md)
|
||||
|
||||
### Advanced Features
|
||||
- [Magic Mode](advanced/magic-mode.md)
|
||||
- [Session Management](advanced/session-management.md)
|
||||
- [Hooks & Authentication](advanced/hooks-auth.md)
|
||||
- [Proxy & Security](advanced/proxy-security.md)
|
||||
- [Content Processing](advanced/content-processing.md)
|
||||
|
||||
### Extraction & Processing
|
||||
- [Extraction Strategies Overview](extraction/overview.md)
|
||||
- [LLM Integration](extraction/llm.md)
|
||||
- [CSS-Based Extraction](extraction/css.md)
|
||||
- [Cosine Strategy](extraction/cosine.md)
|
||||
- [Chunking Strategies](extraction/chunking.md)
|
||||
|
||||
### API Reference
|
||||
- [AsyncWebCrawler](api/async-webcrawler.md)
|
||||
- [CrawlResult](api/crawl-result.md)
|
||||
- [Extraction Strategies](api/strategies.md)
|
||||
- [arun() Method Parameters](api/arun.md)
|
||||
|
||||
### Examples
|
||||
- Coming soon!
|
||||
|
||||
## Getting Started
|
||||
|
||||
1. Install Crawl4AI:
|
||||
```bash
|
||||
pip install crawl4ai
|
||||
```
|
||||
|
||||
2. Check out our [Quick Start Guide](basic/quickstart.md) to begin crawling web pages.
|
||||
|
||||
3. Explore our [examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples) to see Crawl4AI in action.
|
||||
|
||||
## Support
|
||||
|
||||
For questions, suggestions, or issues:
|
||||
- GitHub Issues: [Report a Bug](https://github.com/unclecode/crawl4ai/issues)
|
||||
- Twitter: [@unclecode](https://twitter.com/unclecode)
|
||||
- Website: [crawl4ai.com](https://crawl4ai.com)
|
||||
|
||||
Happy Crawling! 🕸️🚀
|
||||
@@ -0,0 +1,51 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 1: Introduction to Crawl4AI and Basic Installation
|
||||
|
||||
### Quick Intro
|
||||
Walk through installation from PyPI, setup, and verification. Show how to install with options like `torch` or `transformer` for advanced capabilities.
|
||||
|
||||
Here's a condensed outline of the **Installation and Setup** video content:
|
||||
|
||||
---
|
||||
|
||||
1) **Introduction to Crawl4AI**: Briefly explain that Crawl4AI is a powerful tool for web scraping, data extraction, and content processing, with customizable options for various needs.
|
||||
|
||||
2) **Installation Overview**:
|
||||
|
||||
- **Basic Install**: Run `pip install crawl4ai` and `playwright install` (to set up browser dependencies).
|
||||
|
||||
- **Optional Advanced Installs**:
|
||||
- `pip install crawl4ai[torch]` - Adds PyTorch for clustering.
|
||||
- `pip install crawl4ai[transformer]` - Adds support for LLM-based extraction.
|
||||
- `pip install crawl4ai[all]` - Installs all features for complete functionality.
|
||||
|
||||
3) **Verifying the Installation**:
|
||||
|
||||
- Walk through a simple test script to confirm the setup:
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.example.com")
|
||||
print(result.markdown[:500]) # Show first 500 characters
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
- Explain that this script initializes the crawler and runs it on a test URL, displaying part of the extracted content to verify functionality.
|
||||
|
||||
4) **Important Tips**:
|
||||
|
||||
- **Run** `playwright install` **after installation** to set up dependencies.
|
||||
- **For full performance** on text-related tasks, run `crawl4ai-download-models` after installing with `[torch]`, `[transformer]`, or `[all]` options.
|
||||
- If you encounter issues, refer to the documentation or GitHub issues.
|
||||
|
||||
5) **Wrap Up**:
|
||||
|
||||
- Introduce the next topic in the series, which will cover Crawl4AI's browser configuration options (like choosing between `chromium`, `firefox`, and `webkit`).
|
||||
|
||||
---
|
||||
|
||||
This structure provides a concise, effective guide to get viewers up and running with Crawl4AI in minutes.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 2: Overview of Advanced Features
|
||||
|
||||
### Quick Intro
|
||||
A general overview of advanced features like hooks, CSS selectors, and JSON CSS extraction.
|
||||
|
||||
Here's a condensed outline for an **Overview of Advanced Features** video covering Crawl4AI's powerful customization and extraction options:
|
||||
|
||||
---
|
||||
|
||||
### **Overview of Advanced Features**
|
||||
|
||||
1) **Introduction to Advanced Features**:
|
||||
|
||||
- Briefly introduce Crawl4AI’s advanced tools, which let users go beyond basic crawling to customize and fine-tune their scraping workflows.
|
||||
|
||||
2) **Taking Screenshots**:
|
||||
|
||||
- Explain the screenshot capability for capturing page state and verifying content.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://www.example.com", screenshot=True)
|
||||
```
|
||||
- Mention that screenshots are saved as a base64 string in `result`, allowing easy decoding and saving.
|
||||
|
||||
3) **Media and Link Extraction**:
|
||||
|
||||
- Demonstrate how to pull all media (images, videos) and links (internal and external) from a page for deeper analysis or content gathering.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://www.example.com")
|
||||
print("Media:", result.media)
|
||||
print("Links:", result.links)
|
||||
```
|
||||
|
||||
4) **Custom User Agent**:
|
||||
|
||||
- Show how to set a custom user agent to disguise the crawler or simulate specific devices/browsers.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://www.example.com", user_agent="Mozilla/5.0 (compatible; MyCrawler/1.0)")
|
||||
```
|
||||
|
||||
5) **Custom Hooks for Enhanced Control**:
|
||||
|
||||
- Briefly cover how to use hooks, which allow custom actions like setting headers or handling login during the crawl.
|
||||
- **Example**: Setting a custom header with `before_get_url` hook.
|
||||
```python
|
||||
async def before_get_url(page):
|
||||
await page.set_extra_http_headers({"X-Test-Header": "test"})
|
||||
```
|
||||
|
||||
6) **CSS Selectors for Targeted Extraction**:
|
||||
|
||||
- Explain the use of CSS selectors to extract specific elements, ideal for structured data like articles or product details.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://www.example.com", css_selector="h2")
|
||||
print("H2 Tags:", result.extracted_content)
|
||||
```
|
||||
|
||||
7) **Crawling Inside Iframes**:
|
||||
|
||||
- Mention how enabling `process_iframes=True` allows extracting content within iframes, useful for sites with embedded content or ads.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://www.example.com", process_iframes=True)
|
||||
```
|
||||
|
||||
8) **Wrap-Up**:
|
||||
|
||||
- Summarize these advanced features and how they allow users to customize every part of their web scraping experience.
|
||||
- Tease upcoming videos where each feature will be explored in detail.
|
||||
|
||||
---
|
||||
|
||||
This covers each advanced feature with a brief example, providing a useful overview to prepare viewers for the more in-depth videos.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 3: Browser Configurations & Headless Crawling
|
||||
|
||||
### Quick Intro
|
||||
Explain browser options (`chromium`, `firefox`, `webkit`) and settings for headless mode, caching, and verbose logging.
|
||||
|
||||
Here’s a streamlined outline for the **Browser Configurations & Headless Crawling** video:
|
||||
|
||||
---
|
||||
|
||||
### **Browser Configurations & Headless Crawling**
|
||||
|
||||
1) **Overview of Browser Options**:
|
||||
|
||||
- Crawl4AI supports three browser engines:
|
||||
- **Chromium** (default) - Highly compatible.
|
||||
- **Firefox** - Great for specialized use cases.
|
||||
- **Webkit** - Lightweight, ideal for basic needs.
|
||||
- **Example**:
|
||||
```python
|
||||
# Using Chromium (default)
|
||||
crawler = AsyncWebCrawler(browser_type="chromium")
|
||||
|
||||
# Using Firefox
|
||||
crawler = AsyncWebCrawler(browser_type="firefox")
|
||||
|
||||
# Using WebKit
|
||||
crawler = AsyncWebCrawler(browser_type="webkit")
|
||||
```
|
||||
|
||||
2) **Headless Mode**:
|
||||
|
||||
- Headless mode runs the browser without a visible GUI, making it faster and less resource-intensive.
|
||||
- To enable or disable:
|
||||
```python
|
||||
# Headless mode (default is True)
|
||||
crawler = AsyncWebCrawler(headless=True)
|
||||
|
||||
# Disable headless mode for debugging
|
||||
crawler = AsyncWebCrawler(headless=False)
|
||||
```
|
||||
|
||||
3) **Verbose Logging**:
|
||||
- Use `verbose=True` to get detailed logs for each action, useful for debugging:
|
||||
```python
|
||||
crawler = AsyncWebCrawler(verbose=True)
|
||||
```
|
||||
|
||||
4) **Running a Basic Crawl with Configuration**:
|
||||
- Example of a simple crawl with custom browser settings:
|
||||
```python
|
||||
async with AsyncWebCrawler(browser_type="firefox", headless=True, verbose=True) as crawler:
|
||||
result = await crawler.arun(url="https://www.example.com")
|
||||
print(result.markdown[:500]) # Show first 500 characters
|
||||
```
|
||||
- This example uses Firefox in headless mode with logging enabled, demonstrating the flexibility of Crawl4AI’s setup.
|
||||
|
||||
5) **Recap & Next Steps**:
|
||||
- Recap the power of selecting different browsers and running headless mode for speed and efficiency.
|
||||
- Tease the next video: **Proxy & Security Settings** for navigating blocked or restricted content and protecting IP identity.
|
||||
|
||||
---
|
||||
|
||||
This breakdown covers browser configuration essentials in Crawl4AI, providing users with practical steps to optimize their scraping setup.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 4: Advanced Proxy and Security Settings
|
||||
|
||||
### Quick Intro
|
||||
Showcase proxy configurations (HTTP, SOCKS5, authenticated proxies). Demo: Use rotating proxies and set custom headers to avoid IP blocking and enhance security.
|
||||
|
||||
Here’s a focused outline for the **Proxy and Security Settings** video:
|
||||
|
||||
---
|
||||
|
||||
### **Proxy & Security Settings**
|
||||
|
||||
1) **Why Use Proxies in Web Crawling**:
|
||||
|
||||
- Proxies are essential for bypassing IP-based restrictions, improving anonymity, and managing rate limits.
|
||||
- Crawl4AI supports simple proxies, authenticated proxies, and proxy rotation for robust web scraping.
|
||||
|
||||
2) **Basic Proxy Setup**:
|
||||
|
||||
- **Using a Simple Proxy**:
|
||||
```python
|
||||
# HTTP proxy
|
||||
crawler = AsyncWebCrawler(proxy="http://proxy.example.com:8080")
|
||||
|
||||
# SOCKS proxy
|
||||
crawler = AsyncWebCrawler(proxy="socks5://proxy.example.com:1080")
|
||||
```
|
||||
|
||||
3) **Authenticated Proxies**:
|
||||
|
||||
- Use `proxy_config` for proxies requiring a username and password:
|
||||
```python
|
||||
proxy_config = {
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
}
|
||||
crawler = AsyncWebCrawler(proxy_config=proxy_config)
|
||||
```
|
||||
|
||||
4) **Rotating Proxies**:
|
||||
|
||||
- Rotating proxies helps avoid IP bans by switching IP addresses for each request:
|
||||
```python
|
||||
async def get_next_proxy():
|
||||
# Define proxy rotation logic here
|
||||
return {"server": "http://next.proxy.com:8080"}
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
for url in urls:
|
||||
proxy = await get_next_proxy()
|
||||
crawler.update_proxy(proxy)
|
||||
result = await crawler.arun(url=url)
|
||||
```
|
||||
- This setup periodically switches the proxy for enhanced security and access.
|
||||
|
||||
5) **Custom Headers for Additional Security**:
|
||||
|
||||
- Set custom headers to mask the crawler’s identity and avoid detection:
|
||||
```python
|
||||
headers = {
|
||||
"X-Forwarded-For": "203.0.113.195",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Cache-Control": "no-cache",
|
||||
"Pragma": "no-cache"
|
||||
}
|
||||
crawler = AsyncWebCrawler(headers=headers)
|
||||
```
|
||||
|
||||
6) **Combining Proxies with Magic Mode for Anti-Bot Protection**:
|
||||
|
||||
- For sites with aggressive bot detection, combine `proxy` settings with `magic=True`:
|
||||
```python
|
||||
async with AsyncWebCrawler(proxy="http://proxy.example.com:8080", headers={"Accept-Language": "en-US"}) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
magic=True # Enables anti-detection features
|
||||
)
|
||||
```
|
||||
- **Magic Mode** automatically enables user simulation, random timing, and browser property masking.
|
||||
|
||||
7) **Wrap Up & Next Steps**:
|
||||
|
||||
- Summarize the importance of proxies and anti-detection in accessing restricted content and avoiding bans.
|
||||
- Tease the next video: **JavaScript Execution and Handling Dynamic Content** for working with interactive and dynamically loaded pages.
|
||||
|
||||
---
|
||||
|
||||
This outline provides a practical guide to setting up proxies and security configurations, empowering users to navigate restricted sites while staying undetected.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 5: JavaScript Execution and Dynamic Content Handling
|
||||
|
||||
### Quick Intro
|
||||
Explain JavaScript code injection with examples (e.g., simulating scrolling, clicking ‘load more’). Demo: Extract content from a page that uses dynamic loading with lazy-loaded images.
|
||||
|
||||
Here’s a focused outline for the **JavaScript Execution and Dynamic Content Handling** video:
|
||||
|
||||
---
|
||||
|
||||
### **JavaScript Execution & Dynamic Content Handling**
|
||||
|
||||
1) **Why JavaScript Execution Matters**:
|
||||
|
||||
- Many modern websites load content dynamically via JavaScript, requiring special handling to access all elements.
|
||||
- Crawl4AI can execute JavaScript on pages, enabling it to interact with elements like “load more” buttons, infinite scrolls, and content that appears only after certain actions.
|
||||
|
||||
2) **Basic JavaScript Execution**:
|
||||
|
||||
- Use `js_code` to execute JavaScript commands on a page:
|
||||
```python
|
||||
# Scroll to bottom of the page
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);"
|
||||
)
|
||||
```
|
||||
- This command scrolls to the bottom, triggering any lazy-loaded or dynamically added content.
|
||||
|
||||
3) **Multiple Commands & Simulating Clicks**:
|
||||
|
||||
- Combine multiple JavaScript commands to interact with elements like “load more” buttons:
|
||||
```python
|
||||
js_commands = [
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more').click();"
|
||||
]
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=js_commands
|
||||
)
|
||||
```
|
||||
- This script scrolls down and then clicks the “load more” button, useful for loading additional content blocks.
|
||||
|
||||
4) **Waiting for Dynamic Content**:
|
||||
|
||||
- Use `wait_for` to ensure the page loads specific elements before proceeding:
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="css:.dynamic-content" # Wait for elements with class `.dynamic-content`
|
||||
)
|
||||
```
|
||||
- This example waits until elements with `.dynamic-content` are loaded, helping to capture content that appears after JavaScript actions.
|
||||
|
||||
5) **Handling Complex Dynamic Content (e.g., Infinite Scroll)**:
|
||||
|
||||
- Combine JavaScript execution with conditional waiting to handle infinite scrolls or paginated content:
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=[
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"const loadMore = document.querySelector('.load-more'); if (loadMore) loadMore.click();"
|
||||
],
|
||||
wait_for="js:() => document.querySelectorAll('.item').length > 10" # Wait until 10 items are loaded
|
||||
)
|
||||
```
|
||||
- This example scrolls and clicks "load more" repeatedly, waiting each time for a specified number of items to load.
|
||||
|
||||
6) **Complete Example: Dynamic Content Handling with Extraction**:
|
||||
|
||||
- Full example demonstrating a dynamic load and content extraction in one process:
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code=[
|
||||
"window.scrollTo(0, document.body.scrollHeight);",
|
||||
"document.querySelector('.load-more').click();"
|
||||
],
|
||||
wait_for="css:.main-content",
|
||||
css_selector=".main-content"
|
||||
)
|
||||
print(result.markdown[:500]) # Output the main content extracted
|
||||
```
|
||||
|
||||
7) **Wrap Up & Next Steps**:
|
||||
|
||||
- Recap how JavaScript execution allows access to dynamic content, enabling powerful interactions.
|
||||
- Tease the next video: **Content Cleaning and Fit Markdown** to show how Crawl4AI can extract only the most relevant content from complex pages.
|
||||
|
||||
---
|
||||
|
||||
This outline explains how to handle dynamic content and JavaScript-based interactions effectively, enabling users to scrape and interact with complex, modern websites.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 6: Magic Mode and Anti-Bot Protection
|
||||
|
||||
### Quick Intro
|
||||
Highlight `Magic Mode` and anti-bot features like user simulation, navigator overrides, and timing randomization. Demo: Access a site with anti-bot protection and show how `Magic Mode` seamlessly handles it.
|
||||
|
||||
Here’s a concise outline for the **Magic Mode and Anti-Bot Protection** video:
|
||||
|
||||
---
|
||||
|
||||
### **Magic Mode & Anti-Bot Protection**
|
||||
|
||||
1) **Why Anti-Bot Protection is Important**:
|
||||
|
||||
- Many websites use bot detection mechanisms to block automated scraping. Crawl4AI’s anti-detection features help avoid IP bans, CAPTCHAs, and access restrictions.
|
||||
- **Magic Mode** is a one-step solution to enable a range of anti-bot features without complex configuration.
|
||||
|
||||
2) **Enabling Magic Mode**:
|
||||
|
||||
- Simply set `magic=True` to activate Crawl4AI’s full anti-bot suite:
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
magic=True # Enables all anti-detection features
|
||||
)
|
||||
```
|
||||
- This enables a blend of stealth techniques, including masking automation signals, randomizing timings, and simulating real user behavior.
|
||||
|
||||
3) **What Magic Mode Does Behind the Scenes**:
|
||||
|
||||
- **User Simulation**: Mimics human actions like mouse movements and scrolling.
|
||||
- **Navigator Overrides**: Hides signals that indicate an automated browser.
|
||||
- **Timing Randomization**: Adds random delays to simulate natural interaction patterns.
|
||||
- **Cookie Handling**: Accepts and manages cookies dynamically to avoid triggers from cookie pop-ups.
|
||||
|
||||
4) **Manual Anti-Bot Options (If Not Using Magic Mode)**:
|
||||
|
||||
- For granular control, you can configure individual settings without Magic Mode:
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
simulate_user=True, # Enables human-like behavior
|
||||
override_navigator=True # Hides automation fingerprints
|
||||
)
|
||||
```
|
||||
- **Use Cases**: This approach allows more specific adjustments when certain anti-bot features are needed but others are not.
|
||||
|
||||
5) **Combining Proxies with Magic Mode**:
|
||||
|
||||
- To avoid rate limits or IP blocks, combine Magic Mode with a proxy:
|
||||
```python
|
||||
async with AsyncWebCrawler(
|
||||
proxy="http://proxy.example.com:8080",
|
||||
headers={"Accept-Language": "en-US"}
|
||||
) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
magic=True # Full anti-detection
|
||||
)
|
||||
```
|
||||
- This setup maximizes stealth by pairing anti-bot detection with IP obfuscation.
|
||||
|
||||
6) **Example of Anti-Bot Protection in Action**:
|
||||
|
||||
- Full example with Magic Mode and proxies to scrape a protected page:
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/protected-content",
|
||||
magic=True,
|
||||
proxy="http://proxy.example.com:8080",
|
||||
wait_for="css:.content-loaded" # Wait for the main content to load
|
||||
)
|
||||
print(result.markdown[:500]) # Display first 500 characters of the content
|
||||
```
|
||||
- This example ensures seamless access to protected content by combining anti-detection and waiting for full content load.
|
||||
|
||||
7) **Wrap Up & Next Steps**:
|
||||
|
||||
- Recap the power of Magic Mode and anti-bot features for handling restricted websites.
|
||||
- Tease the next video: **Content Cleaning and Fit Markdown** to show how to extract clean and focused content from a page.
|
||||
|
||||
---
|
||||
|
||||
This outline shows users how to easily avoid bot detection and access restricted content, demonstrating both the power and simplicity of Magic Mode in Crawl4AI.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 7: Content Cleaning and Fit Markdown
|
||||
|
||||
### Quick Intro
|
||||
Explain content cleaning options, including `fit_markdown` to keep only the most relevant content. Demo: Extract and compare regular vs. fit markdown from a news site or blog.
|
||||
|
||||
Here’s a streamlined outline for the **Content Cleaning and Fit Markdown** video:
|
||||
|
||||
---
|
||||
|
||||
### **Content Cleaning & Fit Markdown**
|
||||
|
||||
1) **Overview of Content Cleaning in Crawl4AI**:
|
||||
|
||||
- Explain that web pages often include extra elements like ads, navigation bars, footers, and popups.
|
||||
- Crawl4AI’s content cleaning features help extract only the main content, reducing noise and enhancing readability.
|
||||
|
||||
2) **Basic Content Cleaning Options**:
|
||||
|
||||
- **Removing Unwanted Elements**: Exclude specific HTML tags, like forms or navigation bars:
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
word_count_threshold=10, # Filter out blocks with fewer than 10 words
|
||||
excluded_tags=['form', 'nav'], # Exclude specific tags
|
||||
remove_overlay_elements=True # Remove popups and modals
|
||||
)
|
||||
```
|
||||
- This example extracts content while excluding forms, navigation, and modal overlays, ensuring clean results.
|
||||
|
||||
3) **Fit Markdown for Main Content Extraction**:
|
||||
|
||||
- **What is Fit Markdown**: Uses advanced analysis to identify the most relevant content (ideal for articles, blogs, and documentation).
|
||||
- **How it Works**: Analyzes content density, removes boilerplate elements, and maintains formatting for a clear output.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
main_content = result.fit_markdown # Extracted main content
|
||||
print(main_content[:500]) # Display first 500 characters
|
||||
```
|
||||
- Fit Markdown is especially helpful for long-form content like news articles or blog posts.
|
||||
|
||||
4) **Comparing Fit Markdown with Regular Markdown**:
|
||||
|
||||
- **Fit Markdown** returns the primary content without extraneous elements.
|
||||
- **Regular Markdown** includes all extracted text in markdown format.
|
||||
- Example to show the difference:
|
||||
```python
|
||||
all_content = result.markdown # Full markdown
|
||||
main_content = result.fit_markdown # Only the main content
|
||||
|
||||
print(f"All Content Length: {len(all_content)}")
|
||||
print(f"Main Content Length: {len(main_content)}")
|
||||
```
|
||||
- This comparison shows the effectiveness of Fit Markdown in focusing on essential content.
|
||||
|
||||
5) **Media and Metadata Handling with Content Cleaning**:
|
||||
|
||||
- **Media Extraction**: Crawl4AI captures images and videos with metadata like alt text, descriptions, and relevance scores:
|
||||
```python
|
||||
for image in result.media["images"]:
|
||||
print(f"Source: {image['src']}, Alt Text: {image['alt']}, Relevance Score: {image['score']}")
|
||||
```
|
||||
- **Use Case**: Useful for saving only relevant images or videos from an article or content-heavy page.
|
||||
|
||||
6) **Example of Clean Content Extraction in Action**:
|
||||
|
||||
- Full example extracting cleaned content and Fit Markdown:
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
word_count_threshold=10,
|
||||
excluded_tags=['nav', 'footer'],
|
||||
remove_overlay_elements=True
|
||||
)
|
||||
print(result.fit_markdown[:500]) # Show main content
|
||||
```
|
||||
- This example demonstrates content cleaning with settings for filtering noise and focusing on the core text.
|
||||
|
||||
7) **Wrap Up & Next Steps**:
|
||||
|
||||
- Summarize the power of Crawl4AI’s content cleaning features and Fit Markdown for capturing clean, relevant content.
|
||||
- Tease the next video: **Link Analysis and Smart Filtering** to focus on analyzing and filtering links within crawled pages.
|
||||
|
||||
---
|
||||
|
||||
This outline covers Crawl4AI’s content cleaning features and the unique benefits of Fit Markdown, showing users how to retrieve focused, high-quality content from web pages.
|
||||
@@ -0,0 +1,116 @@
|
||||
# Crawl4AI
|
||||
|
||||
## Episode 8: Media Handling: Images, Videos, and Audio
|
||||
|
||||
### Quick Intro
|
||||
Showcase Crawl4AI’s media extraction capabilities, including lazy-loaded media and metadata. Demo: Crawl a multimedia page, extract images, and show metadata (alt text, context, relevance score).
|
||||
|
||||
Here’s a clear and focused outline for the **Media Handling: Images, Videos, and Audio** video:
|
||||
|
||||
---
|
||||
|
||||
### **Media Handling: Images, Videos, and Audio**
|
||||
|
||||
1) **Overview of Media Extraction in Crawl4AI**:
|
||||
|
||||
- Crawl4AI can detect and extract different types of media (images, videos, and audio) along with useful metadata.
|
||||
- This functionality is essential for gathering visual content from multimedia-heavy pages like e-commerce sites, news articles, and social media feeds.
|
||||
|
||||
2) **Image Extraction and Metadata**:
|
||||
|
||||
- Crawl4AI captures images with detailed metadata, including:
|
||||
- **Source URL**: The direct URL to the image.
|
||||
- **Alt Text**: Image description if available.
|
||||
- **Relevance Score**: A score (0–10) indicating how relevant the image is to the main content.
|
||||
- **Context**: Text surrounding the image on the page.
|
||||
- **Example**:
|
||||
```python
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
|
||||
for image in result.media["images"]:
|
||||
print(f"Source: {image['src']}")
|
||||
print(f"Alt Text: {image['alt']}")
|
||||
print(f"Relevance Score: {image['score']}")
|
||||
print(f"Context: {image['context']}")
|
||||
```
|
||||
- This example shows how to access each image’s metadata, making it easy to filter for the most relevant visuals.
|
||||
|
||||
3) **Handling Lazy-Loaded Images**:
|
||||
|
||||
- Crawl4AI automatically supports lazy-loaded images, which are commonly used to optimize webpage loading.
|
||||
- **Example with Wait for Lazy-Loaded Content**:
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
wait_for="css:img[data-src]", # Wait for lazy-loaded images
|
||||
delay_before_return_html=2.0 # Allow extra time for images to load
|
||||
)
|
||||
```
|
||||
- This setup waits for lazy-loaded images to appear, ensuring they are fully captured.
|
||||
|
||||
4) **Video Extraction and Metadata**:
|
||||
|
||||
- Crawl4AI captures video elements, including:
|
||||
- **Source URL**: The video’s direct URL.
|
||||
- **Type**: Format of the video (e.g., MP4).
|
||||
- **Thumbnail**: A poster or thumbnail image if available.
|
||||
- **Duration**: Video length, if metadata is provided.
|
||||
- **Example**:
|
||||
```python
|
||||
for video in result.media["videos"]:
|
||||
print(f"Video Source: {video['src']}")
|
||||
print(f"Type: {video['type']}")
|
||||
print(f"Thumbnail: {video.get('poster')}")
|
||||
print(f"Duration: {video.get('duration')}")
|
||||
```
|
||||
- This allows users to gather video content and relevant details for further processing or analysis.
|
||||
|
||||
5) **Audio Extraction and Metadata**:
|
||||
|
||||
- Audio elements can also be extracted, with metadata like:
|
||||
- **Source URL**: The audio file’s direct URL.
|
||||
- **Type**: Format of the audio file (e.g., MP3).
|
||||
- **Duration**: Length of the audio, if available.
|
||||
- **Example**:
|
||||
```python
|
||||
for audio in result.media["audios"]:
|
||||
print(f"Audio Source: {audio['src']}")
|
||||
print(f"Type: {audio['type']}")
|
||||
print(f"Duration: {audio.get('duration')}")
|
||||
```
|
||||
- Useful for sites with podcasts, sound bites, or other audio content.
|
||||
|
||||
6) **Filtering Media by Relevance**:
|
||||
|
||||
- Use metadata like relevance score to filter only the most useful media content:
|
||||
```python
|
||||
relevant_images = [img for img in result.media["images"] if img['score'] > 5]
|
||||
```
|
||||
- This is especially helpful for content-heavy pages where you only want media directly related to the main content.
|
||||
|
||||
7) **Example: Full Media Extraction with Content Filtering**:
|
||||
|
||||
- Full example extracting images, videos, and audio along with filtering by relevance:
|
||||
```python
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
word_count_threshold=10, # Filter content blocks for relevance
|
||||
exclude_external_images=True # Only keep internal images
|
||||
)
|
||||
|
||||
# Display media summaries
|
||||
print(f"Relevant Images: {len(relevant_images)}")
|
||||
print(f"Videos: {len(result.media['videos'])}")
|
||||
print(f"Audio Clips: {len(result.media['audios'])}")
|
||||
```
|
||||
- This example shows how to capture and filter various media types, focusing on what’s most relevant.
|
||||
|
||||
8) **Wrap Up & Next Steps**:
|
||||
|
||||
- Recap the comprehensive media extraction capabilities, emphasizing how metadata helps users focus on relevant content.
|
||||
- Tease the next video: **Link Analysis and Smart Filtering** to explore how Crawl4AI handles internal, external, and social media links for more focused data gathering.
|
||||
|
||||
---
|
||||
|
||||
This outline provides users with a complete guide to handling images, videos, and audio in Crawl4AI, using metadata to enhance relevance and precision in multimedia extraction.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user