Update documents, upload new version of quickstart.
This commit is contained in:
UncleCode
235
docs/notebooks/Crawl4AI_v0.3.72_Release_Announcement.ipynb
Normal file
235
docs/notebooks/Crawl4AI_v0.3.72_Release_Announcement.ipynb
Normal file
@@ -0,0 +1,235 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 🚀 Crawl4AI v0.3.72 Release Announcement\n",
|
||||
"\n",
|
||||
"Welcome to the new release of **Crawl4AI v0.3.72**! This notebook highlights the latest features and demonstrates how they work in real-time. Follow along to see each feature in action!\n",
|
||||
"\n",
|
||||
"### What’s New?\n",
|
||||
"- ✨ `Fit Markdown`: Extracts only the main content from articles and blogs\n",
|
||||
"- 🛡️ **Magic Mode**: Comprehensive anti-bot detection bypass\n",
|
||||
"- 🌐 **Multi-browser support**: Switch between Chromium, Firefox, WebKit\n",
|
||||
"- 🔍 **Knowledge Graph Extraction**: Generate structured graphs of entities & relationships from any URL\n",
|
||||
"- 🤖 **Crawl4AI GPT Assistant**: Chat directly with our AI assistant for help, code generation, and faster learning (available [here](https://tinyurl.com/your-gpt-assistant-link))\n",
|
||||
"\n",
|
||||
"---\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 📥 Setup\n",
|
||||
"To start, we'll install `Crawl4AI` along with Playwright and `nest_asyncio` to ensure compatibility with Colab’s asynchronous environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install Crawl4AI and dependencies\n",
|
||||
"!pip install crawl4ai\n",
|
||||
"!playwright install\n",
|
||||
"!pip install nest_asyncio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Import nest_asyncio and apply it to allow asyncio in Colab\n",
|
||||
"import nest_asyncio\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"print('Setup complete!')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## ✨ Feature 1: `Fit Markdown`\n",
|
||||
"Extracts only the main content from articles and blog pages, removing sidebars, ads, and other distractions.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import asyncio\n",
|
||||
"from crawl4ai import AsyncWebCrawler\n",
|
||||
"\n",
|
||||
"async def fit_markdown_demo():\n",
|
||||
" async with AsyncWebCrawler() as crawler:\n",
|
||||
" result = await crawler.arun(url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\")\n",
|
||||
" print(result.fit_markdown) # Shows main content in Markdown format\n",
|
||||
"\n",
|
||||
"# Run the demo\n",
|
||||
"await fit_markdown_demo()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 🛡️ Feature 2: Magic Mode\n",
|
||||
"Magic Mode bypasses anti-bot detection to make crawling more reliable on protected websites.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"async def magic_mode_demo():\n",
|
||||
" async with AsyncWebCrawler() as crawler: # Enables anti-bot detection bypass\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://www.reuters.com/markets/us/global-markets-view-usa-pix-2024-08-29/\",\n",
|
||||
" magic=True # Enables magic mode\n",
|
||||
" )\n",
|
||||
" print(result.markdown) # Shows the full content in Markdown format\n",
|
||||
"\n",
|
||||
"# Run the demo\n",
|
||||
"await magic_mode_demo()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 🌐 Feature 3: Multi-Browser Support\n",
|
||||
"Crawl4AI now supports Chromium, Firefox, and WebKit. Here’s how to specify Firefox for a crawl.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"async def multi_browser_demo():\n",
|
||||
" async with AsyncWebCrawler(browser_type=\"firefox\") as crawler: # Using Firefox instead of default Chromium\n",
|
||||
" result = await crawler.arun(url=\"https://crawl4i.com\")\n",
|
||||
" print(result.markdown) # Shows content extracted using Firefox\n",
|
||||
"\n",
|
||||
"# Run the demo\n",
|
||||
"await multi_browser_demo()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## ✨ Put them all together\n",
|
||||
"\n",
|
||||
"Let's combine all the features to extract the main content from a blog post, bypass anti-bot detection, and generate a knowledge graph from the extracted content."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from crawl4ai.extraction_strategy import LLMExtractionStrategy\n",
|
||||
"from pydantic import BaseModel\n",
|
||||
"import json, os\n",
|
||||
"from typing import List\n",
|
||||
"\n",
|
||||
"# Define classes for the knowledge graph structure\n",
|
||||
"class Landmark(BaseModel):\n",
|
||||
" name: str\n",
|
||||
" description: str\n",
|
||||
" activities: list[str] # E.g., visiting, sightseeing, relaxing\n",
|
||||
"\n",
|
||||
"class City(BaseModel):\n",
|
||||
" name: str\n",
|
||||
" description: str\n",
|
||||
" landmarks: list[Landmark]\n",
|
||||
" cultural_highlights: list[str] # E.g., food, music, traditional crafts\n",
|
||||
"\n",
|
||||
"class TravelKnowledgeGraph(BaseModel):\n",
|
||||
" cities: list[City] # Central Mexican cities to visit\n",
|
||||
"\n",
|
||||
"async def combined_demo():\n",
|
||||
" # Define the knowledge graph extraction strategy\n",
|
||||
" strategy = LLMExtractionStrategy(\n",
|
||||
" # provider=\"ollama/nemotron\",\n",
|
||||
" provider='openai/gpt-4o-mini', # Or any other provider, including Ollama and open source models\n",
|
||||
" pi_token=os.getenv('OPENAI_API_KEY'), # In case of Ollama just pass \"no-token\"\n",
|
||||
" schema=TravelKnowledgeGraph.schema(),\n",
|
||||
" instruction=(\n",
|
||||
" \"Extract cities, landmarks, and cultural highlights for places to visit in Central Mexico. \"\n",
|
||||
" \"For each city, list main landmarks with descriptions and activities, as well as cultural highlights.\"\n",
|
||||
" )\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Set up the AsyncWebCrawler with multi-browser support, Magic Mode, and Fit Markdown\n",
|
||||
" async with AsyncWebCrawler(browser_type=\"firefox\") as crawler:\n",
|
||||
" result = await crawler.arun(\n",
|
||||
" url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\",\n",
|
||||
" extraction_strategy=strategy,\n",
|
||||
" bypass_cache=True,\n",
|
||||
" magic=True\n",
|
||||
" )\n",
|
||||
" \n",
|
||||
" # Display main article content in Fit Markdown format\n",
|
||||
" print(\"Extracted Main Content:\\n\", result.fit_markdown)\n",
|
||||
" \n",
|
||||
" # Display extracted knowledge graph of cities, landmarks, and cultural highlights\n",
|
||||
" if result.extracted_content:\n",
|
||||
" travel_graph = json.loads(result.extracted_content)\n",
|
||||
" print(\"\\nExtracted Knowledge Graph:\\n\", json.dumps(travel_graph, indent=2))\n",
|
||||
"\n",
|
||||
"# Run the combined demo\n",
|
||||
"await combined_demo()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 🤖 Crawl4AI GPT Assistant\n",
|
||||
"Chat with the Crawl4AI GPT Assistant for code generation, support, and learning Crawl4AI faster. Try it out [here](https://tinyurl.com/crawl4ai-gpt)!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.9"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Reference in New Issue
Block a user