c6fc5c0518
This commit introduces significant updates to the LinkedIn data discovery documentation by adding two new Jupyter notebooks that provide detailed insights into data discovery processes. The previous workshop notebook has been removed to streamline the content and avoid redundancy. Additionally, the URL seeder documentation has been expanded with a new tutorial and several enhancements to existing scripts, improving usability and clarity. The changes include: - Added and for comprehensive LinkedIn data discovery. - Removed to eliminate outdated content. - Updated to reflect new data visualization requirements. - Introduced and to facilitate easier access to URL seeding techniques. - Enhanced existing Python scripts and markdown files in the URL seeder section for better documentation and examples. These changes aim to improve the overall documentation quality and user experience for developers working with LinkedIn data and URL seeding techniques.
1172 lines
50 KiB
Plaintext
1172 lines
50 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 🔬 Building an AI Research Assistant with Crawl4AI: Smart URL Discovery\n",
|
|
"\n",
|
|
"## Welcome to the Research Pipeline Workshop!\n",
|
|
"\n",
|
|
"In this tutorial, we'll build an **AI-powered research assistant** that intelligently discovers, filters, and analyzes web content. Instead of blindly crawling hundreds of pages, we'll use Crawl4AI's URL Seeder to:\n",
|
|
"\n",
|
|
"- 🔍 **Discover all available URLs** without crawling them first\n",
|
|
"- 🎯 **Score and rank** them by relevance using AI\n",
|
|
"- 🕷️ **Crawl only the most relevant** content\n",
|
|
"- 🤖 **Generate research insights** with proper citations\n",
|
|
"\n",
|
|
"By the end, you'll have a complete research pipeline that can analyze any topic across multiple websites efficiently.\n",
|
|
"\n",
|
|
"## What You'll Build\n",
|
|
"\n",
|
|
"A **smart research assistant** that:\n",
|
|
"1. Takes any research query (e.g., \"Premier League transfer news\")\n",
|
|
"2. Discovers relevant articles from news sites\n",
|
|
"3. Ranks them by relevance using BM25 scoring\n",
|
|
"4. Crawls only the top-ranked articles\n",
|
|
"5. Synthesizes findings into a comprehensive report\n",
|
|
"\n",
|
|
"## Prerequisites\n",
|
|
"\n",
|
|
"- Python 3.8+ environment\n",
|
|
"- Basic understanding of async Python\n",
|
|
"- API keys for LLM (Gemini or OpenAI recommended)\n",
|
|
"\n",
|
|
"## Pipeline Overview\n",
|
|
"\n",
|
|
"```\n",
|
|
"User Query → Query Enhancement → URL Discovery → Relevance Scoring → Smart Crawling → AI Synthesis → Research Report\n",
|
|
"```\n",
|
|
"\n",
|
|
"Each step builds on the previous one, creating an efficient research system that saves time and resources.\n",
|
|
"\n",
|
|
"Let's begin! 🚀\n",
|
|
"\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 0: Environment Setup and Dependencies\n",
|
|
"\n",
|
|
"First, we'll set up our environment with all necessary libraries. We need Crawl4AI for intelligent web crawling, LiteLLM for AI integration, and Rich for beautiful terminal output. This foundation ensures our research assistant has all the tools it needs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%capture\n",
|
|
"!git clone -b next https://github.com/unclecode/crawl4ai.git\n",
|
|
"!cp -r /content/crawl4ai/docs/apps/linkdin/{snippets,schemas} /content/\n",
|
|
"%cd crawl4ai\n",
|
|
"!uv pip install -e .\n",
|
|
"!crawl4ai-setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!crawl4ai-doctor"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import asyncio\n",
|
|
"import json\n",
|
|
"import os\n",
|
|
"from typing import List, Dict, Optional, Tuple\n",
|
|
"from dataclasses import dataclass, asdict\n",
|
|
"from datetime import datetime\n",
|
|
"from pathlib import Path\n",
|
|
"\n",
|
|
"# Rich for beautiful console output\n",
|
|
"from rich.console import Console\n",
|
|
"from rich.panel import Panel\n",
|
|
"from rich.table import Table\n",
|
|
"from rich.progress import Progress, SpinnerColumn, TextColumn\n",
|
|
"\n",
|
|
"# Crawl4AI imports for intelligent crawling\n",
|
|
"from crawl4ai import (\n",
|
|
" AsyncWebCrawler, \n",
|
|
" BrowserConfig, \n",
|
|
" CrawlerRunConfig,\n",
|
|
" AsyncUrlSeeder, \n",
|
|
" SeedingConfig,\n",
|
|
" AsyncLogger\n",
|
|
")\n",
|
|
"from crawl4ai.content_filter_strategy import PruningContentFilter\n",
|
|
"from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n",
|
|
"\n",
|
|
"# LiteLLM for AI capabilities\n",
|
|
"import litellm\n",
|
|
"\n",
|
|
"# Initialize Rich console for pretty output\n",
|
|
"console = Console()\n",
|
|
"\n",
|
|
"print(\"✅ Environment ready! All dependencies loaded successfully.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 1: Configuration and Data Classes\n",
|
|
"\n",
|
|
"Here we define our research pipeline configuration. These dataclasses act as our control center, allowing us to fine-tune every aspect of the research process. Think of them as the settings panel for your research assistant - from discovery limits to AI model choices."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"@dataclass\n",
|
|
"class ResearchConfig:\n",
|
|
" \"\"\"Configuration for the research pipeline\n",
|
|
" \n",
|
|
" This class controls every aspect of our research assistant:\n",
|
|
" - How many URLs to discover and crawl\n",
|
|
" - Which scoring methods to use\n",
|
|
" - Whether to use AI enhancement\n",
|
|
" - Output preferences\n",
|
|
" \"\"\"\n",
|
|
" # Core settings\n",
|
|
" domain: str = \"www.bbc.com/sport\"\n",
|
|
" max_urls_discovery: int = 100 # Cast a wide net initially\n",
|
|
" max_urls_to_crawl: int = 10 # But only crawl the best\n",
|
|
" top_k_urls: int = 10 # Focus on top results\n",
|
|
" \n",
|
|
" # Scoring and filtering\n",
|
|
" score_threshold: float = 0.3 # Minimum relevance score\n",
|
|
" scoring_method: str = \"bm25\" # BM25 is great for relevance\n",
|
|
" \n",
|
|
" # AI and processing\n",
|
|
" use_llm_enhancement: bool = True # Enhance queries with AI\n",
|
|
" llm_model: str = \"gemini/gemini-1.5-flash\" # Fast and capable\n",
|
|
" \n",
|
|
" # URL discovery options\n",
|
|
" extract_head_metadata: bool = True # Get titles, descriptions\n",
|
|
" live_check: bool = False # Verify URLs are accessible\n",
|
|
" force_refresh: bool = False # Bypass cache\n",
|
|
" \n",
|
|
" # Crawler settings\n",
|
|
" max_concurrent_crawls: int = 5 # Parallel crawling\n",
|
|
" timeout: int = 30000 # 30 second timeout\n",
|
|
" headless: bool = True # No browser window\n",
|
|
" \n",
|
|
" # Output settings\n",
|
|
" output_dir: Path = Path(\"research_results\")\n",
|
|
" verbose: bool = True\n",
|
|
"\n",
|
|
"@dataclass\n",
|
|
"class ResearchQuery:\n",
|
|
" \"\"\"Container for research query and metadata\"\"\"\n",
|
|
" original_query: str\n",
|
|
" enhanced_query: Optional[str] = None\n",
|
|
" search_patterns: List[str] = None\n",
|
|
" timestamp: str = None\n",
|
|
"\n",
|
|
"@dataclass\n",
|
|
"class ResearchResult:\n",
|
|
" \"\"\"Container for research results\"\"\"\n",
|
|
" query: ResearchQuery\n",
|
|
" discovered_urls: List[Dict]\n",
|
|
" crawled_content: List[Dict]\n",
|
|
" synthesis: str\n",
|
|
" citations: List[Dict]\n",
|
|
" metadata: Dict\n",
|
|
"\n",
|
|
"# Create default configuration\n",
|
|
"config = ResearchConfig()\n",
|
|
"console.print(Panel(\n",
|
|
" f\"[bold cyan]Research Configuration[/bold cyan]\\n\\n\"\n",
|
|
" f\"🌐 Domain: {config.domain}\\n\"\n",
|
|
" f\"🔍 Max Discovery: {config.max_urls_discovery} URLs\\n\"\n",
|
|
" f\"🕷️ Max Crawl: {config.max_urls_to_crawl} pages\\n\"\n",
|
|
" f\"🤖 AI Model: {config.llm_model}\",\n",
|
|
" title=\"⚙️ Settings\"\n",
|
|
"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 2: Query Enhancement with AI\n",
|
|
"\n",
|
|
"Not all search queries are created equal. Here we use AI to transform simple queries into comprehensive search strategies. The LLM analyzes your query, extracts key concepts, and generates related terms - turning \"football news\" into a rich set of search patterns."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def enhance_query_with_llm(query: str, config: ResearchConfig) -> ResearchQuery:\n",
|
|
" \"\"\"\n",
|
|
" Transform simple queries into comprehensive search strategies\n",
|
|
" \n",
|
|
" Why enhance queries?\n",
|
|
" - Users often use simple terms (\"football news\")\n",
|
|
" - But relevant content might use varied terminology\n",
|
|
" - AI helps capture all relevant variations\n",
|
|
" \"\"\"\n",
|
|
" console.print(f\"\\n[cyan]🤖 Enhancing query: '{query}'...[/cyan]\")\n",
|
|
" \n",
|
|
" try:\n",
|
|
" # Ask AI to analyze and expand the query\n",
|
|
" response = await litellm.acompletion(\n",
|
|
" model=config.llm_model,\n",
|
|
" messages=[{\n",
|
|
" \"role\": \"user\", \n",
|
|
" \"content\": f\"\"\"Given this research query: \"{query}\"\n",
|
|
" \n",
|
|
" Extract:\n",
|
|
" 1. Key terms and concepts (as a list)\n",
|
|
" 2. Related search terms\n",
|
|
" 3. A more specific/enhanced version of the query\n",
|
|
" \n",
|
|
" Return as JSON:\n",
|
|
" {{\n",
|
|
" \"key_terms\": [\"term1\", \"term2\"],\n",
|
|
" \"related_terms\": [\"related1\", \"related2\"],\n",
|
|
" \"enhanced_query\": \"enhanced version of query\"\n",
|
|
" }}\"\"\"\n",
|
|
" }],\n",
|
|
" temperature=0.3, # Low temperature for consistency\n",
|
|
" response_format={\"type\": \"json_object\"}\n",
|
|
" )\n",
|
|
" \n",
|
|
" data = json.loads(response.choices[0].message.content)\n",
|
|
" \n",
|
|
" # Create search patterns from extracted terms\n",
|
|
" # These patterns help the URL seeder find relevant pages\n",
|
|
" all_terms = data[\"key_terms\"] + data[\"related_terms\"]\n",
|
|
" patterns = [f\"*{term.lower()}*\" for term in all_terms]\n",
|
|
" \n",
|
|
" result = ResearchQuery(\n",
|
|
" original_query=query,\n",
|
|
" enhanced_query=data[\"enhanced_query\"],\n",
|
|
" search_patterns=patterns[:10], # Limit to 10 patterns\n",
|
|
" timestamp=datetime.now().isoformat()\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Show the enhancement\n",
|
|
" console.print(Panel(\n",
|
|
" f\"[green]✅ Enhanced Query:[/green] {result.enhanced_query}\\n\"\n",
|
|
" f\"[dim]Key terms: {', '.join(data['key_terms'])}[/dim]\",\n",
|
|
" title=\"🔍 Query Enhancement\"\n",
|
|
" ))\n",
|
|
" \n",
|
|
" return result\n",
|
|
" \n",
|
|
" except Exception as e:\n",
|
|
" console.print(f\"[yellow]⚠️ Enhancement failed, using original query: {e}[/yellow]\")\n",
|
|
" # Fallback to simple tokenization\n",
|
|
" words = query.lower().split()\n",
|
|
" patterns = [f\"*{word}*\" for word in words if len(word) > 2]\n",
|
|
" \n",
|
|
" return ResearchQuery(\n",
|
|
" original_query=query,\n",
|
|
" enhanced_query=query,\n",
|
|
" search_patterns=patterns,\n",
|
|
" timestamp=datetime.now().isoformat()\n",
|
|
" )\n",
|
|
"\n",
|
|
"# Example usage\n",
|
|
"test_query = \"Premier League transfer news\"\n",
|
|
"enhanced = await enhance_query_with_llm(test_query, config)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 3: Smart URL Discovery with AsyncUrlSeeder\n",
|
|
"\n",
|
|
"This is where the magic begins! Instead of crawling pages to find links, AsyncUrlSeeder discovers URLs from sitemaps and Common Crawl data. It's like having a map of the entire website before you start exploring. We'll discover hundreds of URLs in seconds, complete with metadata."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def discover_urls(\n",
|
|
" domain: str, \n",
|
|
" query: ResearchQuery, \n",
|
|
" config: ResearchConfig\n",
|
|
") -> List[Dict]:\n",
|
|
" \"\"\"\n",
|
|
" Discover and rank URLs without crawling them\n",
|
|
" \n",
|
|
" The URL Seeder is incredibly powerful because it:\n",
|
|
" 1. Gets URLs from sitemaps (official site maps)\n",
|
|
" 2. Gets URLs from Common Crawl (web-scale data)\n",
|
|
" 3. Extracts metadata without full page loads\n",
|
|
" 4. Scores relevance using BM25 algorithm\n",
|
|
" \n",
|
|
" This means we know which pages are worth crawling\n",
|
|
" BEFORE we spend time crawling them!\n",
|
|
" \"\"\"\n",
|
|
" console.print(f\"\\n[cyan]🔍 Discovering URLs from {domain}...[/cyan]\")\n",
|
|
" \n",
|
|
" # Use context manager for automatic cleanup\n",
|
|
" async with AsyncUrlSeeder(logger=AsyncLogger(verbose=config.verbose)) as seeder:\n",
|
|
" # Configure the discovery process\n",
|
|
" seeding_config = SeedingConfig(\n",
|
|
" # Data sources\n",
|
|
" source=\"sitemap+cc\", # Use both sitemap AND Common Crawl\n",
|
|
" \n",
|
|
" # Metadata extraction\n",
|
|
" extract_head=config.extract_head_metadata, # Get titles, descriptions\n",
|
|
" \n",
|
|
" # Relevance scoring\n",
|
|
" query=query.enhanced_query or query.original_query,\n",
|
|
" scoring_method=config.scoring_method, # BM25 scoring\n",
|
|
" score_threshold=config.score_threshold, # Minimum score\n",
|
|
" \n",
|
|
" # Limits and performance\n",
|
|
" max_urls=config.max_urls_discovery,\n",
|
|
" live_check=config.live_check, # Verify URLs work\n",
|
|
" force=config.force_refresh, # Bypass cache if needed\n",
|
|
" \n",
|
|
" # Performance tuning\n",
|
|
" concurrency=20, # Parallel workers\n",
|
|
" )\n",
|
|
" \n",
|
|
" try:\n",
|
|
" # Discover URLs - this is FAST!\n",
|
|
" urls = await seeder.urls(domain, seeding_config)\n",
|
|
" \n",
|
|
" # Results are already sorted by relevance\n",
|
|
" # thanks to BM25 scoring\n",
|
|
" top_urls = urls[:config.top_k_urls]\n",
|
|
" \n",
|
|
" # Show discovery results\n",
|
|
" console.print(f\"[green]✅ Discovered {len(urls)} URLs, selected top {len(top_urls)}[/green]\")\n",
|
|
" \n",
|
|
" # Display a sample of what we found\n",
|
|
" if top_urls:\n",
|
|
" table = Table(title=\"🎯 Top Discovered URLs\")\n",
|
|
" table.add_column(\"Score\", style=\"cyan\")\n",
|
|
" table.add_column(\"Title\", style=\"green\")\n",
|
|
" table.add_column(\"URL\", style=\"dim\")\n",
|
|
" \n",
|
|
" for url in top_urls[:5]:\n",
|
|
" score = f\"{url.get('relevance_score', 0):.3f}\"\n",
|
|
" title = \"N/A\"\n",
|
|
" if url.get('head_data') and url['head_data'].get('title'):\n",
|
|
" title = url['head_data']['title'][:50] + \"...\"\n",
|
|
" url_str = url['url'][:60] + \"...\"\n",
|
|
" \n",
|
|
" table.add_row(score, title, url_str)\n",
|
|
" \n",
|
|
" console.print(table)\n",
|
|
" \n",
|
|
" return top_urls\n",
|
|
" \n",
|
|
" except Exception as e:\n",
|
|
" console.print(f\"[red]❌ URL discovery failed: {e}[/red]\")\n",
|
|
" return []\n",
|
|
"\n",
|
|
"# Example discovery\n",
|
|
"discovered = await discover_urls(config.domain, enhanced, config)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 4: Intelligent Content Crawling\n",
|
|
"\n",
|
|
"Now we crawl only the most relevant URLs. This is where our smart filtering pays off - instead of crawling hundreds of pages, we focus on the top 10-20 most relevant ones. We use content filtering to extract only the meaningful text, removing ads and navigation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def crawl_selected_urls(\n",
|
|
" urls: List[Dict], \n",
|
|
" query: ResearchQuery, \n",
|
|
" config: ResearchConfig\n",
|
|
") -> List[Dict]:\n",
|
|
" \"\"\"\n",
|
|
" Crawl only the most relevant URLs with smart content filtering\n",
|
|
" \n",
|
|
" Key optimizations:\n",
|
|
" 1. We already know these URLs are relevant (from scoring)\n",
|
|
" 2. We crawl them in parallel for speed\n",
|
|
" 3. We extract only meaningful content (no ads/nav)\n",
|
|
" 4. We generate clean markdown for analysis\n",
|
|
" \"\"\"\n",
|
|
" # Extract URLs from discovery results\n",
|
|
" url_list = [u['url'] for u in urls if 'url' in u][:config.max_urls_to_crawl]\n",
|
|
" \n",
|
|
" if not url_list:\n",
|
|
" console.print(\"[red]❌ No URLs to crawl[/red]\")\n",
|
|
" return []\n",
|
|
" \n",
|
|
" console.print(f\"\\n[cyan]🕷️ Crawling {len(url_list)} URLs...[/cyan]\")\n",
|
|
" \n",
|
|
" # Configure intelligent content extraction\n",
|
|
" # This removes ads, navigation, and other noise\n",
|
|
" md_generator = DefaultMarkdownGenerator(\n",
|
|
" content_filter=PruningContentFilter(\n",
|
|
" threshold=0.48, # Content relevance threshold\n",
|
|
" threshold_type=\"dynamic\", # Adapts to page structure\n",
|
|
" min_word_threshold=10 # Ignore tiny text blocks\n",
|
|
" ),\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Configure the crawler\n",
|
|
" crawler_config = CrawlerRunConfig(\n",
|
|
" markdown_generator=md_generator,\n",
|
|
" exclude_external_links=True, # Focus on content, not links\n",
|
|
" excluded_tags=['nav', 'header', 'footer', 'aside'], # Skip UI elements\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Create crawler with browser config\n",
|
|
" async with AsyncWebCrawler(\n",
|
|
" config=BrowserConfig(\n",
|
|
" headless=config.headless,\n",
|
|
" verbose=config.verbose\n",
|
|
" )\n",
|
|
" ) as crawler:\n",
|
|
" # Crawl URLs in parallel for speed\n",
|
|
" # arun_many handles concurrency automatically\n",
|
|
" results = await crawler.arun_many(\n",
|
|
" url_list,\n",
|
|
" config=crawler_config,\n",
|
|
" max_concurrent=config.max_concurrent_crawls\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Process successful results\n",
|
|
" crawled_content = []\n",
|
|
" for url, result in zip(url_list, results):\n",
|
|
" if result.success:\n",
|
|
" # Extract the content we need\n",
|
|
" content_data = {\n",
|
|
" 'url': url,\n",
|
|
" 'title': result.metadata.get('title', 'No title'),\n",
|
|
" 'markdown': result.markdown.fit_markdown or result.markdown.raw_markdown,\n",
|
|
" 'metadata': result.metadata\n",
|
|
" }\n",
|
|
" crawled_content.append(content_data)\n",
|
|
" console.print(f\" [green]✓[/green] Crawled: {url[:60]}...\")\n",
|
|
" else:\n",
|
|
" console.print(f\" [red]✗[/red] Failed: {url[:50]}... - {result.error}\")\n",
|
|
" \n",
|
|
" console.print(f\"[green]✅ Successfully crawled {len(crawled_content)} pages[/green]\")\n",
|
|
" return crawled_content\n",
|
|
"\n",
|
|
"# Example crawling\n",
|
|
"crawled = await crawl_selected_urls(discovered[:5], enhanced, config)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 5: AI-Powered Research Synthesis\n",
|
|
"\n",
|
|
"This is where we transform raw content into insights. The AI analyzes all crawled articles, identifies key themes, and generates a comprehensive synthesis with proper citations. It's like having a research assistant read everything and write you a summary."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def generate_research_synthesis(\n",
|
|
" query: ResearchQuery,\n",
|
|
" crawled_content: List[Dict],\n",
|
|
" config: ResearchConfig\n",
|
|
") -> Tuple[str, List[Dict]]:\n",
|
|
" \"\"\"\n",
|
|
" Use AI to synthesize findings from multiple sources\n",
|
|
" \n",
|
|
" The synthesis process:\n",
|
|
" 1. Sends all content to the LLM\n",
|
|
" 2. Asks for key findings and analysis\n",
|
|
" 3. Ensures proper citation of sources\n",
|
|
" 4. Generates actionable insights\n",
|
|
" \"\"\"\n",
|
|
" if not crawled_content:\n",
|
|
" return \"No content available for synthesis.\", []\n",
|
|
" \n",
|
|
" console.print(\"\\n[cyan]🤖 Generating research synthesis...[/cyan]\")\n",
|
|
" \n",
|
|
" # Prepare content for the AI\n",
|
|
" # We include source info for proper citations\n",
|
|
" content_sections = []\n",
|
|
" for i, content in enumerate(crawled_content, 1):\n",
|
|
" section = f\"\"\"\n",
|
|
"SOURCE {i}:\n",
|
|
"Title: {content['title']}\n",
|
|
"URL: {content['url']}\n",
|
|
"Content Preview:\n",
|
|
"{content['markdown'][:1500]}...\n",
|
|
"\"\"\"\n",
|
|
" content_sections.append(section)\n",
|
|
" \n",
|
|
" combined_content = \"\\n---\\n\".join(content_sections)\n",
|
|
" \n",
|
|
" try:\n",
|
|
" # Generate comprehensive synthesis\n",
|
|
" response = await litellm.acompletion(\n",
|
|
" model=config.llm_model,\n",
|
|
" messages=[{\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": f\"\"\"Research Query: \"{query.original_query}\"\n",
|
|
"\n",
|
|
"Based on the following sources, provide a comprehensive research synthesis.\n",
|
|
"\n",
|
|
"{combined_content}\n",
|
|
"\n",
|
|
"Please provide:\n",
|
|
"1. An executive summary (2-3 sentences)\n",
|
|
"2. Key findings (3-5 bullet points)\n",
|
|
"3. Detailed analysis (2-3 paragraphs)\n",
|
|
"4. Future implications or trends\n",
|
|
"\n",
|
|
"Format your response with clear sections and cite sources using [Source N] notation.\n",
|
|
"Keep the total response under 800 words.\"\"\"\n",
|
|
" }],\n",
|
|
" temperature=0.7 # Some creativity for synthesis\n",
|
|
" )\n",
|
|
" \n",
|
|
" synthesis = response.choices[0].message.content\n",
|
|
" \n",
|
|
" # Extract citations from the synthesis\n",
|
|
" citations = []\n",
|
|
" for i, content in enumerate(crawled_content, 1):\n",
|
|
" # Check if this source was cited\n",
|
|
" if f\"[Source {i}]\" in synthesis or f\"Source {i}\" in synthesis:\n",
|
|
" citations.append({\n",
|
|
" 'source_id': i,\n",
|
|
" 'title': content['title'],\n",
|
|
" 'url': content['url']\n",
|
|
" })\n",
|
|
" \n",
|
|
" return synthesis, citations\n",
|
|
" \n",
|
|
" except Exception as e:\n",
|
|
" console.print(f\"[red]❌ Synthesis generation failed: {e}[/red]\")\n",
|
|
" # Fallback to simple summary\n",
|
|
" summary = f\"Research on '{query.original_query}' found {len(crawled_content)} relevant articles:\\n\\n\"\n",
|
|
" for content in crawled_content[:3]:\n",
|
|
" summary += f\"- {content['title']}\\n {content['url']}\\n\\n\"\n",
|
|
" return summary, []\n",
|
|
"\n",
|
|
"# Example synthesis\n",
|
|
"synthesis, citations = await generate_research_synthesis(enhanced, crawled, config)\n",
|
|
"console.print(Panel(synthesis[:500] + \"...\", title=\"📝 Research Synthesis Preview\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 6: Complete Research Pipeline\n",
|
|
"\n",
|
|
"Now let's put it all together! This orchestrator function manages the entire research pipeline from query to final report. It coordinates all the components we've built, handling errors gracefully and providing progress updates."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def research_pipeline(\n",
|
|
" query: str,\n",
|
|
" config: ResearchConfig = None\n",
|
|
") -> ResearchResult:\n",
|
|
" \"\"\"\n",
|
|
" Main research pipeline orchestrator\n",
|
|
" \n",
|
|
" This brings together all components:\n",
|
|
" 1. Query enhancement (AI-powered)\n",
|
|
" 2. URL discovery (AsyncUrlSeeder)\n",
|
|
" 3. Smart crawling (AsyncWebCrawler)\n",
|
|
" 4. AI synthesis (LiteLLM)\n",
|
|
" \n",
|
|
" Returns a complete research result\n",
|
|
" \"\"\"\n",
|
|
" if config is None:\n",
|
|
" config = ResearchConfig()\n",
|
|
" \n",
|
|
" start_time = datetime.now()\n",
|
|
" \n",
|
|
" # Display pipeline header\n",
|
|
" console.print(Panel(\n",
|
|
" f\"[bold cyan]Research Pipeline[/bold cyan]\\n\\n\"\n",
|
|
" f\"[dim]Query:[/dim] {query}\\n\"\n",
|
|
" f\"[dim]Domain:[/dim] {config.domain}\",\n",
|
|
" title=\"🚀 Starting Research\",\n",
|
|
" border_style=\"cyan\"\n",
|
|
" ))\n",
|
|
" \n",
|
|
" # Step 1: Enhance query\n",
|
|
" console.print(f\"\\n[bold cyan]📝 Step 1: Query Processing[/bold cyan]\")\n",
|
|
" if config.use_llm_enhancement:\n",
|
|
" research_query = await enhance_query_with_llm(query, config)\n",
|
|
" else:\n",
|
|
" # Simple fallback without AI\n",
|
|
" research_query = ResearchQuery(\n",
|
|
" original_query=query,\n",
|
|
" enhanced_query=query,\n",
|
|
" search_patterns=[f\"*{word}*\" for word in query.lower().split()],\n",
|
|
" timestamp=datetime.now().isoformat()\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Step 2: Discover URLs\n",
|
|
" console.print(f\"\\n[bold cyan]🔍 Step 2: URL Discovery[/bold cyan]\")\n",
|
|
" discovered_urls = await discover_urls(\n",
|
|
" domain=config.domain,\n",
|
|
" query=research_query,\n",
|
|
" config=config\n",
|
|
" )\n",
|
|
" \n",
|
|
" if not discovered_urls:\n",
|
|
" # No URLs found - return empty result\n",
|
|
" return ResearchResult(\n",
|
|
" query=research_query,\n",
|
|
" discovered_urls=[],\n",
|
|
" crawled_content=[],\n",
|
|
" synthesis=\"No relevant URLs found for the given query.\",\n",
|
|
" citations=[],\n",
|
|
" metadata={'duration': str(datetime.now() - start_time)}\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Step 3: Crawl selected URLs\n",
|
|
" console.print(f\"\\n[bold cyan]🕷️ Step 3: Content Crawling[/bold cyan]\")\n",
|
|
" crawled_content = await crawl_selected_urls(\n",
|
|
" urls=discovered_urls,\n",
|
|
" query=research_query,\n",
|
|
" config=config\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Step 4: Generate synthesis\n",
|
|
" console.print(f\"\\n[bold cyan]🤖 Step 4: Synthesis Generation[/bold cyan]\")\n",
|
|
" synthesis, citations = await generate_research_synthesis(\n",
|
|
" query=research_query,\n",
|
|
" crawled_content=crawled_content,\n",
|
|
" config=config\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Create final result\n",
|
|
" result = ResearchResult(\n",
|
|
" query=research_query,\n",
|
|
" discovered_urls=discovered_urls,\n",
|
|
" crawled_content=crawled_content,\n",
|
|
" synthesis=synthesis,\n",
|
|
" citations=citations,\n",
|
|
" metadata={\n",
|
|
" 'duration': str(datetime.now() - start_time),\n",
|
|
" 'domain': config.domain,\n",
|
|
" 'timestamp': datetime.now().isoformat(),\n",
|
|
" 'total_discovered': len(discovered_urls),\n",
|
|
" 'total_crawled': len(crawled_content),\n",
|
|
" 'total_cited': len(citations)\n",
|
|
" }\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Display summary\n",
|
|
" duration = datetime.now() - start_time\n",
|
|
" console.print(Panel(\n",
|
|
" f\"[bold green]✅ Research completed in {duration}[/bold green]\\n\\n\"\n",
|
|
" f\"📊 Discovered: {len(discovered_urls)} URLs\\n\"\n",
|
|
" f\"🕷️ Crawled: {len(crawled_content)} pages\\n\"\n",
|
|
" f\"📚 Citations: {len(citations)} sources\",\n",
|
|
" title=\"🎉 Pipeline Complete\",\n",
|
|
" border_style=\"green\"\n",
|
|
" ))\n",
|
|
" \n",
|
|
" return result\n",
|
|
"\n",
|
|
"# Example: Run complete pipeline\n",
|
|
"result = await research_pipeline(\"Champions League latest results\", config)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 7: Beautiful Output Formatting\n",
|
|
"\n",
|
|
"A good research report needs clear presentation. Here we format our results into a professional report with executive summary, key findings, and proper citations. This makes the research actionable and easy to share."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def format_research_output(result: ResearchResult) -> None:\n",
|
|
" \"\"\"\n",
|
|
" Create a beautifully formatted research report\n",
|
|
" \n",
|
|
" Good formatting makes insights actionable:\n",
|
|
" - Clear structure with sections\n",
|
|
" - Highlighted key findings\n",
|
|
" - Proper source attribution\n",
|
|
" - Easy to scan and understand\n",
|
|
" \"\"\"\n",
|
|
" # Header\n",
|
|
" console.print(\"\\n\" + \"=\" * 60)\n",
|
|
" console.print(\"[bold cyan]🔬 RESEARCH REPORT[/bold cyan]\")\n",
|
|
" console.print(\"=\" * 60)\n",
|
|
" \n",
|
|
" # Query information\n",
|
|
" console.print(f\"\\n[bold]Query:[/bold] {result.query.original_query}\")\n",
|
|
" if result.query.enhanced_query != result.query.original_query:\n",
|
|
" console.print(f\"[dim]Enhanced: {result.query.enhanced_query}[/dim]\")\n",
|
|
" \n",
|
|
" # Statistics\n",
|
|
" stats_table = Table(show_header=False, box=None)\n",
|
|
" stats_table.add_column(style=\"cyan\")\n",
|
|
" stats_table.add_column()\n",
|
|
" \n",
|
|
" stats_table.add_row(\"📊 URLs Discovered\", str(result.metadata['total_discovered']))\n",
|
|
" stats_table.add_row(\"🕷️ Pages Crawled\", str(result.metadata['total_crawled']))\n",
|
|
" stats_table.add_row(\"📚 Sources Cited\", str(result.metadata['total_cited']))\n",
|
|
" stats_table.add_row(\"⏱️ Processing Time\", result.metadata['duration'])\n",
|
|
" \n",
|
|
" console.print(\"\\n[bold]Statistics:[/bold]\")\n",
|
|
" console.print(stats_table)\n",
|
|
" \n",
|
|
" # Synthesis\n",
|
|
" console.print(\"\\n[bold]📝 SYNTHESIS[/bold]\")\n",
|
|
" console.print(\"-\" * 60)\n",
|
|
" console.print(result.synthesis)\n",
|
|
" \n",
|
|
" # Citations\n",
|
|
" if result.citations:\n",
|
|
" console.print(\"\\n[bold]📚 SOURCES[/bold]\")\n",
|
|
" console.print(\"-\" * 60)\n",
|
|
" for citation in result.citations:\n",
|
|
" console.print(f\"\\n[{citation['source_id']}] [cyan]{citation['title']}[/cyan]\")\n",
|
|
" console.print(f\" [dim]{citation['url']}[/dim]\")\n",
|
|
" \n",
|
|
" # Top discovered URLs\n",
|
|
" console.print(\"\\n[bold]🔍 TOP DISCOVERED URLS[/bold]\")\n",
|
|
" console.print(\"-\" * 60)\n",
|
|
" \n",
|
|
" urls_table = Table()\n",
|
|
" urls_table.add_column(\"Score\", style=\"cyan\")\n",
|
|
" urls_table.add_column(\"Title\")\n",
|
|
" urls_table.add_column(\"URL\", style=\"dim\")\n",
|
|
" \n",
|
|
" for url_data in result.discovered_urls[:5]:\n",
|
|
" score = f\"{url_data.get('relevance_score', 0):.3f}\"\n",
|
|
" title = \"N/A\"\n",
|
|
" if url_data.get('head_data') and url_data['head_data'].get('title'):\n",
|
|
" title = url_data['head_data']['title'][:40] + \"...\"\n",
|
|
" url = url_data['url'][:50] + \"...\"\n",
|
|
" \n",
|
|
" urls_table.add_row(score, title, url)\n",
|
|
" \n",
|
|
" console.print(urls_table)\n",
|
|
"\n",
|
|
"# Display the formatted report\n",
|
|
"format_research_output(result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 8: Save Research Results\n",
|
|
"\n",
|
|
"Finally, let's save our research for future reference. We'll create both JSON (for data analysis) and Markdown (for reading) formats. This ensures your research is preserved and shareable."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def save_research_results(\n",
|
|
" result: ResearchResult, \n",
|
|
" config: ResearchConfig\n",
|
|
") -> Tuple[Path, Path]:\n",
|
|
" \"\"\"\n",
|
|
" Save research results in multiple formats\n",
|
|
" \n",
|
|
" Why save in multiple formats?\n",
|
|
" - JSON: Perfect for further analysis or automation\n",
|
|
" - Markdown: Human-readable, great for sharing\n",
|
|
" \"\"\"\n",
|
|
" # Create output directory\n",
|
|
" config.output_dir.mkdir(parents=True, exist_ok=True)\n",
|
|
" \n",
|
|
" # Generate filename based on query and timestamp\n",
|
|
" timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
|
|
" query_slug = result.query.original_query[:30].replace(\" \", \"_\").replace(\"/\", \"_\")\n",
|
|
" base_filename = f\"{timestamp}_{query_slug}\"\n",
|
|
" \n",
|
|
" # Save JSON\n",
|
|
" json_path = config.output_dir / f\"{base_filename}.json\"\n",
|
|
" with open(json_path, 'w') as f:\n",
|
|
" json.dump(asdict(result), f, indent=2, default=str)\n",
|
|
" \n",
|
|
" # Create markdown report\n",
|
|
" md_content = [\n",
|
|
" f\"# Research Report: {result.query.original_query}\",\n",
|
|
" f\"\\n**Generated on:** {result.metadata.get('timestamp', 'N/A')}\",\n",
|
|
" f\"\\n**Domain:** {result.metadata.get('domain', 'N/A')}\",\n",
|
|
" f\"\\n**Processing time:** {result.metadata.get('duration', 'N/A')}\",\n",
|
|
" \"\\n---\\n\",\n",
|
|
" \"## Query Information\",\n",
|
|
" f\"- **Original Query:** {result.query.original_query}\",\n",
|
|
" f\"- **Enhanced Query:** {result.query.enhanced_query or 'N/A'}\",\n",
|
|
" \"\\n## Statistics\",\n",
|
|
" f\"- **URLs Discovered:** {result.metadata['total_discovered']}\",\n",
|
|
" f\"- **Pages Crawled:** {result.metadata['total_crawled']}\",\n",
|
|
" f\"- **Sources Cited:** {result.metadata['total_cited']}\",\n",
|
|
" \"\\n## Research Synthesis\\n\",\n",
|
|
" result.synthesis,\n",
|
|
" \"\\n## Sources\\n\"\n",
|
|
" ]\n",
|
|
" \n",
|
|
" # Add citations\n",
|
|
" for citation in result.citations:\n",
|
|
" md_content.extend([\n",
|
|
" f\"### [{citation['source_id']}] {citation['title']}\",\n",
|
|
" f\"- **URL:** [{citation['url']}]({citation['url']})\",\n",
|
|
" \"\"\n",
|
|
" ])\n",
|
|
" \n",
|
|
" # Add discovered URLs\n",
|
|
" md_content.extend([\n",
|
|
" \"\\n## Discovered URLs (Top 10)\\n\",\n",
|
|
" \"| Score | Title | URL |\",\n",
|
|
" \"|-------|-------|-----|\"\n",
|
|
" ])\n",
|
|
" \n",
|
|
" for url_data in result.discovered_urls[:10]:\n",
|
|
" score = url_data.get('relevance_score', 0)\n",
|
|
" title = 'N/A'\n",
|
|
" if url_data.get('head_data') and url_data['head_data'].get('title'):\n",
|
|
" title = url_data['head_data']['title'][:50] + '...'\n",
|
|
" url = url_data['url'][:60] + '...'\n",
|
|
" md_content.append(f\"| {score:.3f} | {title} | {url} |\")\n",
|
|
" \n",
|
|
" # Save markdown\n",
|
|
" md_path = config.output_dir / f\"{base_filename}.md\"\n",
|
|
" with open(md_path, 'w') as f:\n",
|
|
" f.write('\\n'.join(md_content))\n",
|
|
" \n",
|
|
" console.print(f\"\\n[green]💾 Results saved:[/green]\")\n",
|
|
" console.print(f\" JSON: {json_path}\")\n",
|
|
" console.print(f\" Markdown: {md_path}\")\n",
|
|
" \n",
|
|
" return json_path, md_path\n",
|
|
"\n",
|
|
"# Save our results\n",
|
|
"json_path, md_path = await save_research_results(result, config)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 🎯 Putting It All Together: Interactive Research Assistant\n",
|
|
"\n",
|
|
"Now let's create an interactive version where you can research any topic! This brings together everything we've learned into a user-friendly tool."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def interactive_research_assistant():\n",
|
|
" \"\"\"\n",
|
|
" Interactive research assistant with example queries\n",
|
|
" \n",
|
|
" This demonstrates how to build a user-friendly interface\n",
|
|
" for your research pipeline.\n",
|
|
" \"\"\"\n",
|
|
" # Welcome message\n",
|
|
" console.print(Panel.fit(\n",
|
|
" \"[bold cyan]🔬 AI Research Assistant[/bold cyan]\\n\\n\"\n",
|
|
" \"Powered by Crawl4AI's intelligent URL discovery\\n\"\n",
|
|
" \"[dim]• Discover without crawling\\n\"\n",
|
|
" \"• Score by relevance\\n\"\n",
|
|
" \"• Crawl only what matters\\n\"\n",
|
|
" \"• Generate AI insights[/dim]\",\n",
|
|
" title=\"Welcome\",\n",
|
|
" border_style=\"cyan\"\n",
|
|
" ))\n",
|
|
" \n",
|
|
" # Example queries\n",
|
|
" examples = [\n",
|
|
" \"Premier League transfer news and rumors\",\n",
|
|
" \"Champions League match results and analysis\",\n",
|
|
" \"Tennis grand slam tournament updates\",\n",
|
|
" \"Formula 1 race results and standings\",\n",
|
|
" \"NBA playoff predictions and analysis\"\n",
|
|
" ]\n",
|
|
" \n",
|
|
" # Display examples\n",
|
|
" console.print(\"\\n[bold]📋 Example queries:[/bold]\")\n",
|
|
" for i, example in enumerate(examples, 1):\n",
|
|
" console.print(f\" {i}. {example}\")\n",
|
|
" \n",
|
|
" # Get user input\n",
|
|
" console.print(\"\\n[bold]Enter a number (1-5) or type your own query:[/bold]\")\n",
|
|
" user_input = input(\"🔍 > \").strip()\n",
|
|
" \n",
|
|
" # Determine query\n",
|
|
" if user_input.isdigit() and 1 <= int(user_input) <= len(examples):\n",
|
|
" query = examples[int(user_input) - 1]\n",
|
|
" else:\n",
|
|
" query = user_input if user_input else examples[0]\n",
|
|
" \n",
|
|
" console.print(f\"\\n[cyan]Selected query: {query}[/cyan]\")\n",
|
|
" \n",
|
|
" # Configuration options\n",
|
|
" console.print(\"\\n[bold]Choose configuration:[/bold]\")\n",
|
|
" console.print(\" 1. Quick (5 URLs, fast)\")\n",
|
|
" console.print(\" 2. Standard (10 URLs, balanced)\")\n",
|
|
" console.print(\" 3. Comprehensive (20 URLs, thorough)\")\n",
|
|
" \n",
|
|
" config_choice = input(\"⚙️ > \").strip()\n",
|
|
" \n",
|
|
" # Create configuration\n",
|
|
" if config_choice == \"1\":\n",
|
|
" config = ResearchConfig(max_urls_to_crawl=5, top_k_urls=5)\n",
|
|
" elif config_choice == \"3\":\n",
|
|
" config = ResearchConfig(max_urls_to_crawl=20, top_k_urls=20)\n",
|
|
" else:\n",
|
|
" config = ResearchConfig() # Standard\n",
|
|
" \n",
|
|
" # Run research\n",
|
|
" result = await research_pipeline(query, config)\n",
|
|
" \n",
|
|
" # Display results\n",
|
|
" format_research_output(result)\n",
|
|
" \n",
|
|
" # Save results\n",
|
|
" save_choice = input(\"\\n💾 Save results? (y/n): \").strip().lower()\n",
|
|
" if save_choice == 'y':\n",
|
|
" await save_research_results(result, config)\n",
|
|
"\n",
|
|
"# Run the interactive assistant\n",
|
|
"await interactive_research_assistant()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 🚀 Advanced Tips and Best Practices\n",
|
|
"\n",
|
|
"### 1. Domain-Specific Research\n",
|
|
"\n",
|
|
"Customize the pipeline for specific domains:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Research across multiple sports sites\n",
|
|
"async def multi_domain_research(query: str):\n",
|
|
" \"\"\"Research across multiple sports websites\"\"\"\n",
|
|
" \n",
|
|
" domains = [\n",
|
|
" \"www.bbc.com/sport\",\n",
|
|
" \"www.espn.com\",\n",
|
|
" \"www.skysports.com\"\n",
|
|
" ]\n",
|
|
" \n",
|
|
" all_results = []\n",
|
|
" \n",
|
|
" for domain in domains:\n",
|
|
" config = ResearchConfig(\n",
|
|
" domain=domain,\n",
|
|
" max_urls_to_crawl=5 # 5 per domain\n",
|
|
" )\n",
|
|
" \n",
|
|
" console.print(f\"\\n[cyan]Researching {domain}...[/cyan]\")\n",
|
|
" result = await research_pipeline(query, config)\n",
|
|
" all_results.append(result)\n",
|
|
" \n",
|
|
" # Combine insights from all domains\n",
|
|
" console.print(\"\\n[bold green]✅ Multi-domain research complete![/bold green]\")\n",
|
|
" return all_results\n",
|
|
"\n",
|
|
"# Example usage\n",
|
|
"# results = await multi_domain_research(\"World Cup 2024\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 2. Performance Optimization\n",
|
|
"\n",
|
|
"Tips for faster research:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Optimized configuration for speed\n",
|
|
"speed_config = ResearchConfig(\n",
|
|
" # Reduce discovery scope\n",
|
|
" max_urls_discovery=50, # Don't discover too many\n",
|
|
" \n",
|
|
" # Skip live checking (trust the sitemap)\n",
|
|
" live_check=False,\n",
|
|
" \n",
|
|
" # Increase parallelism\n",
|
|
" max_concurrent_crawls=10,\n",
|
|
" \n",
|
|
" # Skip AI enhancement for simple queries\n",
|
|
" use_llm_enhancement=False,\n",
|
|
" \n",
|
|
" # Use faster model\n",
|
|
" llm_model=\"gemini/gemini-1.5-flash\"\n",
|
|
")\n",
|
|
"\n",
|
|
"console.print(Panel(\n",
|
|
" \"[green]⚡ Speed Optimizations:[/green]\\n\\n\"\n",
|
|
" \"• Reduced discovery scope\\n\"\n",
|
|
" \"• Disabled live URL checking\\n\"\n",
|
|
" \"• Increased parallelism\\n\"\n",
|
|
" \"• Using faster AI model\",\n",
|
|
" title=\"Performance Tips\"\n",
|
|
"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 3. Caching Strategy\n",
|
|
"\n",
|
|
"The URL Seeder automatically caches results for efficiency:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Cache demonstration\n",
|
|
"console.print(\"[bold]🗄️ Understanding Caching:[/bold]\\n\")\n",
|
|
"\n",
|
|
"console.print(\"1. [cyan]First run:[/cyan] Fetches fresh data\")\n",
|
|
"console.print(\" - Discovers URLs from sitemap/Common Crawl\")\n",
|
|
"console.print(\" - Extracts metadata\")\n",
|
|
"console.print(\" - Caches results for 7 days\")\n",
|
|
"\n",
|
|
"console.print(\"\\n2. [cyan]Subsequent runs:[/cyan] Uses cache (instant!)\")\n",
|
|
"console.print(\" - No network requests needed\")\n",
|
|
"console.print(\" - Same query returns cached results\")\n",
|
|
"\n",
|
|
"console.print(\"\\n3. [cyan]Force refresh:[/cyan] Bypass cache when needed\")\n",
|
|
"console.print(\" - Set `force_refresh=True` in config\")\n",
|
|
"console.print(\" - Useful for breaking news or updates\")\n",
|
|
"\n",
|
|
"# Example with cache control\n",
|
|
"cache_config = ResearchConfig(\n",
|
|
" force_refresh=True # Always get fresh data\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 🎓 Summary & Next Steps\n",
|
|
"\n",
|
|
"### What You've Learned\n",
|
|
"\n",
|
|
"You've built a complete AI research assistant that:\n",
|
|
"\n",
|
|
"✅ **Discovers URLs intelligently** - No blind crawling \n",
|
|
"✅ **Scores by relevance** - Focus on what matters \n",
|
|
"✅ **Crawls efficiently** - Parallel processing \n",
|
|
"✅ **Generates insights** - AI-powered synthesis \n",
|
|
"✅ **Saves results** - JSON and Markdown formats \n",
|
|
"\n",
|
|
"### Key Advantages\n",
|
|
"\n",
|
|
"1. **Efficiency**: Discover 1000s of URLs in seconds, crawl only the best\n",
|
|
"2. **Intelligence**: BM25 scoring ensures relevance\n",
|
|
"3. **Scalability**: Works across multiple domains\n",
|
|
"4. **Flexibility**: Configurable for any use case\n",
|
|
"\n",
|
|
"### Next Steps\n",
|
|
"\n",
|
|
"1. **Customize for your domain**: Adapt the pipeline for your specific needs\n",
|
|
"2. **Add persistence**: Store results in a database\n",
|
|
"3. **Build an API**: Turn this into a web service\n",
|
|
"4. **Schedule updates**: Monitor topics over time\n",
|
|
"5. **Enhance with more AI**: Add summarization, sentiment analysis, etc.\n",
|
|
"\n",
|
|
"### Resources\n",
|
|
"\n",
|
|
"- 🐙 **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
|
|
"- 📚 **Documentation**: [crawl4ai.com/docs](https://crawl4ai.com/docs)\n",
|
|
"- 💬 **Discord**: [Join our community](https://discord.gg/crawl4ai)\n",
|
|
"\n",
|
|
"Thank you for learning with Crawl4AI! 🙏\n",
|
|
"\n",
|
|
"Happy researching! 🚀🔬"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"collapsed_sections": [],
|
|
"name": "Crawl4AI_URL_Seeder_Tutorial.ipynb",
|
|
"provenance": [],
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"name": "python"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|