Delete test.txt

feat(config): Adding a configurable way of setting the cache directory for constrained environments

README.md

@@ -1,4 +1,4 @@
-# 🔥🕷️ Crawl4AI: LLM Friendly Web Crawler & Scrapper
+# 🔥🕷️ Crawl4AI: LLM Friendly Web Crawler & Scraper
 
 <a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 
@@ -19,14 +19,13 @@ Use the [Crawl4AI GPT Assistant](https://tinyurl.com/crawl4ai-gpt) as your AI-po
 - 💡 Get tailored support and examples
 - 📘 Learn Crawl4AI faster with step-by-step guidance
 
-## New in 0.3.72 ✨
+## New in 0.3.73 ✨
 
-- 📄 Fit markdown generation for extracting main article content.
-- 🪄 Magic mode for comprehensive anti-bot detection bypass.
-- 🌐 Enhanced multi-browser support with seamless switching (Chromium, Firefox, WebKit)
-- 📚 New chunking strategies(Sliding window, Overlapping window, Flexible size control)
-- 💾 Improved caching system for better performance
-- ⚡ Optimized batch processing with automatic rate limiting
+- 🐳 Docker Ready: Full API server with seamless deployment & scaling
+- 🎯 Browser Takeover: Use your own browser with cookies & history intact (CDP support)
+- 📝 Mockdown+: Enhanced tag preservation & content extraction
+- ⚡️ Parallel Power: Supercharged multi-URL crawling performance
+- 🌟 And many more exciting updates...
 
 ## Try it Now!
 
@@ -433,6 +432,30 @@ You can find the full comparison code in our repository at `docs/examples/crawl4
 
 For detailed documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://crawl4ai.com/mkdocs/).
 
+## Crawl4AI Roadmap 🗺️
+
+For detailed information on our development plans and upcoming features, check out our [Roadmap](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md).
+
+### Advanced Crawling Systems 🔧
+- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
+- [ ] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
+- [ ] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
+- [ ] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
+
+### Specialized Features 🛠️
+- [ ] 4. Automated Schema Generator: Convert natural language to extraction schemas
+- [ ] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
+- [ ] 6. Web Embedding Index: Semantic search infrastructure for crawled content
+
+### Development Tools 🔨
+- [ ] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
+- [ ] 8. Performance Monitor: Real-time insights into crawler operations
+- [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers
+
+### Community & Growth 🌱
+- [ ] 10. Sponsorship Program: Structured support system with tiered benefits
+- [ ] 11. Educational Content: "How to Crawl" video series and interactive tutorials
+
 ## Contributing 🤝
 
 We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTING.md) for more information.
@@ -481,4 +504,4 @@ For a detailed exploration of our vision, challenges, and solutions, please see
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)
+[![Star History Chart](https://api.star-history.com/svg?repos=unclecode/crawl4ai&type=Date)](https://star-history.com/#unclecode/crawl4ai&Date)

ROADMAP.md

@@ -0,0 +1,503 @@
+# Crawl4AI Strategic Roadmap
+
+```mermaid
+%%{init: {'themeVariables': { 'fontSize': '14px'}}}%%
+graph TD
+    subgraph A1[Advanced Crawling Systems 🔧]
+        A["`
+        • Graph Crawler ✓
+        • Question-Based Crawler
+        • Knowledge-Optimal Crawler
+        • Agentic Crawler
+        `"]
+    end
+
+    subgraph A2[Specialized Features 🛠️]
+        B["`
+        • Automated Schema Generator
+        • Domain-Specific Scrapers
+        • 
+        • 
+        `"]
+    end
+
+    subgraph A3[Development Tools 🔨]
+        C["`
+        • Interactive Playground
+        • Performance Monitor
+        • Cloud Integration
+        • 
+        `"]
+    end
+
+    subgraph A4[Community & Growth 🌱]
+        D["`
+        • Sponsorship Program
+        • Educational Content
+        • 
+        • 
+        `"]
+    end
+
+    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px
+    classDef section fill:#f0f0f0,stroke:#333,stroke-width:4px,rx:10
+    class A1,A2,A3,A4 section
+
+    %% Layout hints
+    A1 --> A2[" "]
+    A3 --> A4[" "]
+    linkStyle 0,1 stroke:none
+```
+
+Crawl4AI is evolving to provide more intelligent, efficient, and versatile web crawling capabilities. This roadmap outlines the key developments and features planned for the project, organized into strategic sections that build upon our current foundation.
+
+## 1. Advanced Crawling Systems 🔧
+
+This section introduces three powerful crawling systems that extend Crawl4AI's capabilities from basic web crawling to intelligent, purpose-driven data extraction.
+
+### 1.1 Question-Based Crawler
+The Question-Based Crawler enhances our core engine by enabling automatic discovery and extraction of relevant web content based on natural language questions.
+
+Key Features:
+- SerpiAPI integration for intelligent web search
+- Relevancy scoring for search results
+- Automatic URL discovery and prioritization
+- Cross-source validation
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.discovery import QuestionBasedDiscovery
+
+async with AsyncWebCrawler() as crawler:
+    discovery = QuestionBasedDiscovery(crawler)
+    results = await discovery.arun(
+        question="What are the system requirements for major cloud providers' GPU instances?",
+        max_urls=5,
+        relevance_threshold=0.7
+    )
+    
+    for result in results:
+        print(f"Source: {result.url} (Relevance: {result.relevance_score})")
+        print(f"Content: {result.markdown}\n")
+```
+
+### 1.2 Knowledge-Optimal Crawler
+An intelligent crawling system that solves the optimization problem of minimizing data extraction while maximizing knowledge acquisition for specific objectives.
+
+Key Features:
+- Smart content prioritization
+- Minimal data extraction for maximum knowledge
+- Probabilistic relevance assessment
+- Objective-driven crawling paths
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.optimization import KnowledgeOptimizer
+
+async with AsyncWebCrawler() as crawler:
+    optimizer = KnowledgeOptimizer(
+        objective="Understand GPU instance pricing and limitations across cloud providers",
+        required_knowledge=[
+            "pricing structure",
+            "GPU specifications",
+            "usage limits",
+            "availability zones"
+        ],
+        confidence_threshold=0.85
+    )
+    
+    result = await crawler.arun(
+        urls=[
+            "https://aws.amazon.com/ec2/pricing/",
+            "https://cloud.google.com/gpu",
+            "https://azure.microsoft.com/pricing/"
+        ],
+        optimizer=optimizer,
+        optimization_mode="minimal_extraction"
+    )
+    
+    print(f"Knowledge Coverage: {result.knowledge_coverage}")
+    print(f"Data Efficiency: {result.efficiency_ratio}")
+    print(f"Extracted Content: {result.optimal_content}")
+```
+
+### 1.3 Agentic Crawler
+An autonomous system capable of understanding complex goals and automatically planning and executing multi-step crawling operations.
+
+Key Features:
+- Autonomous goal interpretation
+- Dynamic step planning
+- Interactive navigation capabilities
+- Visual recognition and interaction
+- Automatic error recovery
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.agents import CrawlerAgent
+
+async with AsyncWebCrawler() as crawler:
+    agent = CrawlerAgent(crawler)
+    
+    # Automatic planning and execution
+    result = await agent.arun(
+        goal="Find research papers about quantum computing published in 2023 with more than 50 citations",
+        auto_retry=True
+    )
+    print("Generated Plan:", result.executed_steps)
+    print("Extracted Data:", result.data)
+    
+    # Using custom steps with automatic execution
+    result = await agent.arun(
+        goal="Extract conference deadlines from ML conferences",
+        custom_plan=[
+            "Navigate to conference page",
+            "Find important dates section",
+            "Extract submission deadlines",
+            "Verify dates are for 2024"
+        ]
+    )
+    
+    # Monitoring execution
+    print("Step Completion:", result.step_status)
+    print("Execution Time:", result.execution_time)
+    print("Success Rate:", result.success_rate)
+```
+
+# Section 2: Specialized Features 🛠️
+
+This section introduces specialized tools and features that enhance Crawl4AI's capabilities for specific use cases and data extraction needs.
+
+### 2.1 Automated Schema Generator
+A system that automatically generates JsonCssExtractionStrategy schemas from natural language descriptions, making structured data extraction accessible to all users.
+
+Key Features:
+- Natural language schema generation
+- Automatic pattern detection
+- Predefined schema templates
+- Chrome extension for visual schema building
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.schema import SchemaGenerator
+
+# Generate schema from natural language description
+generator = SchemaGenerator()
+schema = await generator.generate(
+    url="https://news-website.com",
+    description="For each news article on the page, I need the headline, publication date, and main image"
+)
+
+# Use generated schema with crawler
+async with AsyncWebCrawler() as crawler:
+    result = await crawler.arun(
+        url="https://news-website.com",
+        extraction_strategy=schema
+    )
+
+# Example of generated schema:
+"""
+{
+    "name": "News Article Extractor",
+    "baseSelector": "article.news-item",
+    "fields": [
+        {
+            "name": "headline",
+            "selector": "h2.article-title",
+            "type": "text"
+        },
+        {
+            "name": "date",
+            "selector": "span.publish-date",
+            "type": "text"
+        },
+        {
+            "name": "image",
+            "selector": "img.article-image",
+            "type": "attribute",
+            "attribute": "src"
+        }
+    ]
+}
+"""
+```
+
+### 2.2 Domain Specific Scrapers
+Specialized extraction strategies optimized for common website types and platforms, providing consistent and reliable data extraction without additional configuration.
+
+Key Features:
+- Pre-configured extractors for popular platforms
+- Academic site specialization (arXiv, NCBI)
+- E-commerce standardization
+- Documentation site handling
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.extractors import AcademicExtractor, EcommerceExtractor
+
+async with AsyncWebCrawler() as crawler:
+    # Academic paper extraction
+    papers = await crawler.arun(
+        url="https://arxiv.org/list/cs.AI/recent",
+        extractor="academic",  # Built-in extractor type
+        site_type="arxiv",     # Specific site optimization
+        extract_fields=[
+            "title", 
+            "authors", 
+            "abstract", 
+            "citations"
+        ]
+    )
+    
+    # E-commerce product data
+    products = await crawler.arun(
+        url="https://store.example.com/products",
+        extractor="ecommerce",
+        extract_fields=[
+            "name",
+            "price",
+            "availability",
+            "reviews"
+        ]
+    )
+```
+
+### 2.3 Web Embedding Index
+Creates and maintains a semantic search infrastructure for crawled content, enabling efficient retrieval and querying of web content through vector embeddings.
+
+Key Features:
+- Automatic embedding generation
+- Intelligent content chunking
+- Efficient vector storage and indexing
+- Semantic search capabilities
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.indexing import WebIndex
+
+# Initialize and build index
+index = WebIndex(model="efficient-mini")
+
+async with AsyncWebCrawler() as crawler:
+    # Crawl and index content
+    await index.build(
+        urls=["https://docs.example.com"],
+        crawler=crawler,
+        options={
+            "chunk_method": "semantic",
+            "update_policy": "incremental",
+            "embedding_batch_size": 100
+        }
+    )
+
+    # Search through indexed content
+    results = await index.search(
+        query="How to implement OAuth authentication?",
+        filters={
+            "content_type": "technical",
+            "recency": "6months"
+        },
+        top_k=5
+    )
+
+    # Get similar content
+    similar = await index.find_similar(
+        url="https://docs.example.com/auth/oauth",
+        threshold=0.85
+    )
+```
+
+Each of these specialized features builds upon Crawl4AI's core functionality while providing targeted solutions for specific use cases. They can be used independently or combined for more complex data extraction and processing needs.
+
+# Section 3: Development Tools 🔧
+
+This section covers tools designed to enhance the development experience, monitoring, and deployment of Crawl4AI applications.
+
+### 3.1 Crawl4AI Playground 🎮
+
+The Crawl4AI Playground is an interactive web-based development environment that simplifies web scraping experimentation, development, and deployment. With its intuitive interface and AI-powered assistance, users can quickly prototype, test, and deploy web scraping solutions.
+
+#### Key Features 🌟
+
+##### Visual Strategy Builder
+- Interactive point-and-click interface for building extraction strategies
+- Real-time preview of selected elements
+- Side-by-side comparison of different extraction approaches
+- Visual validation of CSS selectors and XPath queries
+
+##### AI Assistant Integration
+- Strategy recommendations based on target website analysis
+- Parameter optimization suggestions
+- Best practices guidance for specific use cases
+- Automated error detection and resolution
+- Performance optimization tips
+
+##### Real-Time Testing & Validation
+- Live preview of extraction results
+- Side-by-side comparison of multiple strategies
+- Performance metrics visualization
+- Automatic validation of extracted data
+- Error detection and debugging tools
+
+##### Project Management
+- Save and organize multiple scraping projects
+- Version control for configurations
+- Export/import project settings
+- Share configurations with team members
+- Project templates for common use cases
+
+##### Deployment Pipeline
+- One-click deployment to various environments
+- Docker container generation
+- Cloud deployment templates (AWS, GCP, Azure)
+- Scaling configuration management
+- Monitoring setup automation
+
+
+### 3.2 Performance Monitoring System
+A comprehensive monitoring solution providing real-time insights into crawler operations, resource usage, and system health through both CLI and GUI interfaces.
+
+Key Features:
+- Real-time resource tracking
+- Active crawl monitoring
+- Performance statistics
+- Customizable alerting system
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.monitor import CrawlMonitor
+
+# Initialize monitoring
+monitor = CrawlMonitor()
+
+# Start monitoring with CLI interface
+await monitor.start(
+    mode="cli",  # or "gui"
+    refresh_rate="1s",
+    metrics={
+        "resources": ["cpu", "memory", "network"],
+        "crawls": ["active", "queued", "completed"],
+        "performance": ["success_rate", "response_times"]
+    }
+)
+
+# Example CLI output:
+"""
+Crawl4AI Monitor (Live) - Press Q to exit
+────────────────────────────────────────
+System Usage:
+ ├─ CPU: ███████░░░ 70%
+ └─ Memory: ████░░░░░ 2.1GB/8GB
+
+Active Crawls:
+ID    URL                   Status    Progress
+001   docs.example.com     🟢 Active   75%
+002   api.service.com      🟡 Queue    -
+
+Metrics (Last 5min):
+ ├─ Success Rate: 98%
+ ├─ Avg Response: 0.6s
+ └─ Pages/sec: 8.5
+"""
+```
+
+### 3.3 Cloud Integration
+Streamlined deployment tools for setting up Crawl4AI in various cloud environments, with support for scaling and monitoring.
+
+Key Features:
+- One-click deployment solutions
+- Auto-scaling configuration
+- Load balancing setup
+- Cloud-specific optimizations
+- Monitoring integration
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.deploy import CloudDeployer
+
+# Initialize deployer
+deployer = CloudDeployer()
+
+# Deploy crawler service
+deployment = await deployer.deploy(
+    service_name="crawler-cluster",
+    platform="aws",  # or "gcp", "azure"
+    config={
+        "instance_type": "compute-optimized",
+        "auto_scaling": {
+            "min_instances": 2,
+            "max_instances": 10,
+            "scale_based_on": "cpu_usage"
+        },
+        "region": "us-east-1",
+        "monitoring": True
+    }
+)
+
+# Get deployment status and endpoints
+print(f"Service Status: {deployment.status}")
+print(f"API Endpoint: {deployment.endpoint}")
+print(f"Monitor URL: {deployment.monitor_url}")
+```
+
+These development tools work together to provide a comprehensive environment for developing, testing, monitoring, and deploying Crawl4AI applications. The Playground helps users experiment and generate optimal configurations, the Performance Monitor ensures smooth operation, and the Cloud Integration tools simplify deployment and scaling.
+
+# Section 4: Community & Growth 🌱
+
+This section outlines initiatives designed to build and support the Crawl4AI community, provide educational resources, and ensure sustainable project growth.
+
+### 4.1 Sponsorship Program
+A structured program to support ongoing development and maintenance of Crawl4AI while providing valuable benefits to sponsors.
+
+Key Features:
+- Multiple sponsorship tiers
+- Sponsor recognition system
+- Priority support for sponsors
+- Early access to new features
+- Custom feature development opportunities
+
+Program Structure (not yet finalized):
+```
+Sponsorship Tiers:
+
+🥉 Bronze Supporter
+- GitHub Sponsor badge
+- Priority issue response
+- Community Discord role
+
+🥈 Silver Supporter
+- All Bronze benefits
+- Technical support channel
+- Vote on roadmap priorities
+- Early access to beta features
+
+🥇 Gold Supporter
+- All Silver benefits
+- Custom feature requests
+- Direct developer access
+- Private support sessions
+
+💎 Diamond Partner
+- All Gold benefits
+- Custom development
+- On-demand consulting
+- Integration support
+```
+
+### 4.2 "How to Crawl" Video Series
+A comprehensive educational resource teaching users how to effectively use Crawl4AI for various web scraping and data extraction scenarios.
+
+Key Features:
+- Step-by-step tutorials
+- Real-world use cases
+- Best practices
+- Integration guides
+- Advanced feature deep-dives
+
+These community initiatives are designed to:
+- Provide comprehensive learning resources
+- Foster a supportive user community
+- Ensure sustainable project development
+- Share knowledge and best practices
+- Create opportunities for collaboration
+
+The combination of structured support through sponsorship, educational content through video series, and interactive learning through the playground creates a robust ecosystem for both new and experienced users of Crawl4AI.

crawl4ai/async_crawler_strategy.py

@@ -187,6 +187,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         self.use_managed_browser = kwargs.get("use_managed_browser", False)
         self.user_data_dir = kwargs.get("user_data_dir", None)
         self.managed_browser = None
+        self.default_context = None
         self.hooks = {
             'on_browser_created': None,
             'on_user_agent_updated': None,
@@ -217,6 +218,25 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
                 )
                 cdp_url = await self.managed_browser.start()
                 self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)
+                
+                # Get the default context that maintains the user profile
+                contexts = self.browser.contexts
+                if contexts:
+                    self.default_context = contexts[0]
+                else:
+                    # If no default context exists, create one
+                    self.default_context = await self.browser.new_context(
+                        viewport={"width": 1920, "height": 1080}
+                    )
+                
+                # Set up the default context
+                if self.default_context:
+                    await self.default_context.set_extra_http_headers(self.headers)
+                    
+                    if self.user_agent:
+                        await self.default_context.set_extra_http_headers({
+                            "User-Agent": self.user_agent
+                        })
             else:
                 browser_args = {
                     "headless": self.headless,
@@ -254,12 +274,20 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
     async def close(self):
         if self.sleep_on_close:
             await asyncio.sleep(0.5)
+            
+        # Close all active sessions
+        session_ids = list(self.sessions.keys())
+        for session_id in session_ids:
+            await self.kill_session(session_id)
+            
         if self.browser:
             await self.browser.close()
             self.browser = None
+            
         if self.managed_browser:
             await self.managed_browser.cleanup()
             self.managed_browser = None
+            
         if self.playwright:
             await self.playwright.stop()
             self.playwright = None
@@ -293,7 +321,8 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         if session_id in self.sessions:
             context, page, _ = self.sessions[session_id]
             await page.close()
-            await context.close()
+            if not self.use_managed_browser:
+                await context.close()
             del self.sessions[session_id]
 
     def _cleanup_expired_sessions(self):
@@ -415,61 +444,75 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
         
         self._cleanup_expired_sessions()
         session_id = kwargs.get("session_id")
-        if session_id:
-            context, page, _ = self.sessions.get(session_id, (None, None, None))
-            if not context:
+        
+        # Handle page creation differently for managed browser
+        if self.use_managed_browser:
+            if session_id:
+                # Reuse existing session if available
+                context, page, _ = self.sessions.get(session_id, (None, None, None))
+                if not page:
+                    # Create new page in default context if session doesn't exist
+                    page = await self.default_context.new_page()
+                    self.sessions[session_id] = (self.default_context, page, time.time())
+            else:
+                # Create new page in default context for non-session requests
+                page = await self.default_context.new_page()
+        else:
+            if session_id:
+                context, page, _ = self.sessions.get(session_id, (None, None, None))
+                if not context:
+                    context = await self.browser.new_context(
+                        user_agent=self.user_agent,
+                        viewport={"width": 1920, "height": 1080},
+                        proxy={"server": self.proxy} if self.proxy else None,
+                        accept_downloads=True,
+                        java_script_enabled=True
+                    )
+                    await context.add_cookies([{"name": "cookiesEnabled", "value": "true", "url": url}])
+                    await context.set_extra_http_headers(self.headers)
+                    page = await context.new_page()
+                    self.sessions[session_id] = (context, page, time.time())
+            else:
                 context = await self.browser.new_context(
                     user_agent=self.user_agent,
                     viewport={"width": 1920, "height": 1080},
-                    proxy={"server": self.proxy} if self.proxy else None,
-                    accept_downloads=True,
-                    java_script_enabled=True
+                    proxy={"server": self.proxy} if self.proxy else None
                 )
-                await context.add_cookies([{"name": "cookiesEnabled", "value": "true", "url": url}])
                 await context.set_extra_http_headers(self.headers)
+                
+                if kwargs.get("override_navigator", False) or kwargs.get("simulate_user", False) or kwargs.get("magic", False):
+                    # Inject scripts to override navigator properties
+                    await context.add_init_script("""
+                        // Pass the Permissions Test.
+                        const originalQuery = window.navigator.permissions.query;
+                        window.navigator.permissions.query = (parameters) => (
+                            parameters.name === 'notifications' ?
+                                Promise.resolve({ state: Notification.permission }) :
+                                originalQuery(parameters)
+                        );
+                        Object.defineProperty(navigator, 'webdriver', {
+                            get: () => undefined
+                        });
+                        window.navigator.chrome = {
+                            runtime: {},
+                            // Add other properties if necessary
+                        };
+                        Object.defineProperty(navigator, 'plugins', {
+                            get: () => [1, 2, 3, 4, 5],
+                        });
+                        Object.defineProperty(navigator, 'languages', {
+                            get: () => ['en-US', 'en'],
+                        });
+                        Object.defineProperty(document, 'hidden', {
+                            get: () => false
+                        });
+                        Object.defineProperty(document, 'visibilityState', {
+                            get: () => 'visible'
+                        });
+                    """)
+                
                 page = await context.new_page()
-                self.sessions[session_id] = (context, page, time.time())
-        else:
-            context = await self.browser.new_context(
-                user_agent=self.user_agent,
-                viewport={"width": 1920, "height": 1080},
-                proxy={"server": self.proxy} if self.proxy else None
-            )
-            await context.set_extra_http_headers(self.headers)
-            
-            if kwargs.get("override_navigator", False) or kwargs.get("simulate_user", False) or kwargs.get("magic", False):
-                # Inject scripts to override navigator properties
-                await context.add_init_script("""
-                    // Pass the Permissions Test.
-                    const originalQuery = window.navigator.permissions.query;
-                    window.navigator.permissions.query = (parameters) => (
-                        parameters.name === 'notifications' ?
-                            Promise.resolve({ state: Notification.permission }) :
-                            originalQuery(parameters)
-                    );
-                    Object.defineProperty(navigator, 'webdriver', {
-                        get: () => undefined
-                    });
-                    window.navigator.chrome = {
-                        runtime: {},
-                        // Add other properties if necessary
-                    };
-                    Object.defineProperty(navigator, 'plugins', {
-                        get: () => [1, 2, 3, 4, 5],
-                    });
-                    Object.defineProperty(navigator, 'languages', {
-                        get: () => ['en-US', 'en'],
-                    });
-                    Object.defineProperty(document, 'hidden', {
-                        get: () => false
-                    });
-                    Object.defineProperty(document, 'visibilityState', {
-                        get: () => 'visible'
-                    });
-                """)
-            
-            page = await context.new_page()
-            # await stealth_async(page) #, stealth_config)
+                # await stealth_async(page) #, stealth_config)
 
         # Add console message and error logging
         if kwargs.get("log_console", False):
@@ -482,7 +525,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
 
             if self.use_cached_html:
                 cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
+                    os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
                 )
                 if os.path.exists(cache_file_path):
                     html = ""
@@ -682,7 +725,7 @@ class AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy):
 
             if self.use_cached_html:
                 cache_file_path = os.path.join(
-                    Path.home(), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
+                    os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", hashlib.md5(url.encode()).hexdigest()
                 )
                 with open(cache_file_path, "w", encoding="utf-8") as f:
                     f.write(html)

crawl4ai/async_database.py

@@ -10,7 +10,7 @@ import logging
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 
-DB_PATH = os.path.join(Path.home(), ".crawl4ai")
+DB_PATH = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
 os.makedirs(DB_PATH, exist_ok=True)
 DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")
 

crawl4ai/async_webcrawler.py

@@ -23,14 +23,14 @@ class AsyncWebCrawler:
         self,
         crawler_strategy: Optional[AsyncCrawlerStrategy] = None,
         always_by_pass_cache: bool = False,
-        base_directory: str = str(Path.home()),
+        base_directory: str = str(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())),
         **kwargs,
     ):
         self.crawler_strategy = crawler_strategy or AsyncPlaywrightCrawlerStrategy(
             **kwargs
         )
         self.always_by_pass_cache = always_by_pass_cache
-        # self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        # self.crawl4ai_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
         self.crawl4ai_folder = os.path.join(base_directory, ".crawl4ai")
         os.makedirs(self.crawl4ai_folder, exist_ok=True)
         os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)

crawl4ai/content_scrapping_strategy.py

@@ -93,8 +93,8 @@ class CustomHTML2Text(HTML2Text):
             else:
                 self.o('\n```')
                 self.inside_pre = False
-        elif tag in ["h1", "h2", "h3", "h4", "h5", "h6"]:
-            pass
+        # elif tag in ["h1", "h2", "h3", "h4", "h5", "h6"]:
+        #     pass
         else:
             super().handle_tag(tag, attrs, start)
 

crawl4ai/crawler_strategy.py

@@ -132,7 +132,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
 
         # chromedriver_autoinstaller.install()
         # import chromedriver_autoinstaller
-        # crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        # crawl4ai_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
         # driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=self.options)
         # chromedriver_path = chromedriver_autoinstaller.install()
         # chromedriver_path = chromedriver_autoinstaller.utils.download_chromedriver()
@@ -205,7 +205,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
         url_hash = hashlib.md5(url.encode()).hexdigest()
         
         if self.use_cached_html:
-            cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", url_hash)
+            cache_file_path = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", url_hash)
             if os.path.exists(cache_file_path):
                 with open(cache_file_path, "r") as f:
                     return sanitize_input_encode(f.read())
@@ -275,7 +275,7 @@ class LocalSeleniumCrawlerStrategy(CrawlerStrategy):
             self.driver = self.execute_hook('before_return_html', self.driver, html)
             
             # Store in cache
-            cache_file_path = os.path.join(Path.home(), ".crawl4ai", "cache", url_hash)
+            cache_file_path = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai", "cache", url_hash)
             with open(cache_file_path, "w", encoding="utf-8") as f:
                 f.write(html)
                 

crawl4ai/database.py

@@ -3,7 +3,7 @@ from pathlib import Path
 import sqlite3
 from typing import Optional, Tuple
 
-DB_PATH = os.path.join(Path.home(), ".crawl4ai")
+DB_PATH = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
 os.makedirs(DB_PATH, exist_ok=True)
 DB_PATH = os.path.join(DB_PATH, "crawl4ai.db")
 

crawl4ai/model_loader.py

@@ -56,7 +56,7 @@ def set_model_device(model):
 
 @lru_cache()
 def get_home_folder():
-    home_folder = os.path.join(Path.home(), ".crawl4ai")
+    home_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
     os.makedirs(home_folder, exist_ok=True)
     os.makedirs(f"{home_folder}/cache", exist_ok=True)
     os.makedirs(f"{home_folder}/models", exist_ok=True)

crawl4ai/train.py

@@ -1,146 +0,0 @@
-import spacy
-from spacy.training import Example
-import random
-import nltk
-from nltk.corpus import reuters
-import torch
-
-def save_spacy_model_as_torch(nlp, model_dir="models/reuters"):
-    # Extract the TextCategorizer component
-    textcat = nlp.get_pipe("textcat_multilabel")
-
-    # Convert the weights to a PyTorch state dictionary
-    state_dict = {name: torch.tensor(param.data) for name, param in textcat.model.named_parameters()}
-
-    # Save the state dictionary
-    torch.save(state_dict, f"{model_dir}/model_weights.pth")
-
-    # Extract and save the vocabulary
-    vocab = extract_vocab(nlp)
-    with open(f"{model_dir}/vocab.txt", "w") as vocab_file:
-        for word, idx in vocab.items():
-            vocab_file.write(f"{word}\t{idx}\n")
-    
-    print(f"Model weights and vocabulary saved to: {model_dir}")
-
-def extract_vocab(nlp):
-    # Extract vocabulary from the SpaCy model
-    vocab = {word: i for i, word in enumerate(nlp.vocab.strings)}
-    return vocab
-
-nlp = spacy.load("models/reuters")
-save_spacy_model_as_torch(nlp, model_dir="models")
-
-def train_and_save_reuters_model(model_dir="models/reuters"):
-    # Ensure the Reuters corpus is downloaded
-    nltk.download('reuters')
-    nltk.download('punkt')
-    if not reuters.fileids():
-        print("Reuters corpus not found.")
-        return
-
-    # Load a blank English spaCy model
-    nlp = spacy.blank("en")
-
-    # Create a TextCategorizer with the ensemble model for multi-label classification
-    textcat = nlp.add_pipe("textcat_multilabel")
-
-    # Add labels to text classifier
-    for label in reuters.categories():
-        textcat.add_label(label)
-
-    # Prepare training data
-    train_examples = []
-    for fileid in reuters.fileids():
-        categories = reuters.categories(fileid)
-        text = reuters.raw(fileid)
-        cats = {label: label in categories for label in reuters.categories()}
-        # Prepare spacy Example objects
-        doc = nlp.make_doc(text)
-        example = Example.from_dict(doc, {'cats': cats})
-        train_examples.append(example)
-
-    # Initialize the text categorizer with the example objects
-    nlp.initialize(lambda: train_examples)
-
-    # Train the model
-    random.seed(1)
-    spacy.util.fix_random_seed(1)
-    for i in range(5):  # Adjust iterations for better accuracy
-        random.shuffle(train_examples)
-        losses = {}
-        # Create batches of data
-        batches = spacy.util.minibatch(train_examples, size=8)
-        for batch in batches:
-            nlp.update(batch, drop=0.2, losses=losses)
-        print(f"Losses at iteration {i}: {losses}")
-
-    # Save the trained model
-    nlp.to_disk(model_dir)
-    print(f"Model saved to: {model_dir}")
-
-def train_model(model_dir, additional_epochs=0):
-    # Load the model if it exists, otherwise start with a blank model
-    try:
-        nlp = spacy.load(model_dir)
-        print("Model loaded from disk.")
-    except IOError:
-        print("No existing model found. Starting with a new model.")
-        nlp = spacy.blank("en")
-        textcat = nlp.add_pipe("textcat_multilabel")
-        for label in reuters.categories():
-            textcat.add_label(label)
-
-    # Prepare training data
-    train_examples = []
-    for fileid in reuters.fileids():
-        categories = reuters.categories(fileid)
-        text = reuters.raw(fileid)
-        cats = {label: label in categories for label in reuters.categories()}
-        doc = nlp.make_doc(text)
-        example = Example.from_dict(doc, {'cats': cats})
-        train_examples.append(example)
-
-    # Initialize the model if it was newly created
-    if 'textcat_multilabel' not in nlp.pipe_names:
-        nlp.initialize(lambda: train_examples)
-    else:
-        print("Continuing training with existing model.")
-
-    # Train the model
-    random.seed(1)
-    spacy.util.fix_random_seed(1)
-    num_epochs = 5 + additional_epochs
-    for i in range(num_epochs):
-        random.shuffle(train_examples)
-        losses = {}
-        batches = spacy.util.minibatch(train_examples, size=8)
-        for batch in batches:
-            nlp.update(batch, drop=0.2, losses=losses)
-        print(f"Losses at iteration {i}: {losses}")
-
-    # Save the trained model
-    nlp.to_disk(model_dir)
-    print(f"Model saved to: {model_dir}")
-
-def load_model_and_predict(model_dir, text, tok_k = 3):
-    # Load the trained model from the specified directory
-    nlp = spacy.load(model_dir)
-    
-    # Process the text with the loaded model
-    doc = nlp(text)
-    
-    # gee top 3 categories
-    top_categories = sorted(doc.cats.items(), key=lambda x: x[1], reverse=True)[:tok_k]
-    print(f"Top {tok_k} categories:")
-    
-    return top_categories    
-
-if __name__ == "__main__":
-    train_and_save_reuters_model()
-    train_model("models/reuters", additional_epochs=5)
-    model_directory = "reuters_model_10"
-    print(reuters.categories())
-    example_text = "Apple Inc. is reportedly buying a startup for $1 billion"
-    r =load_model_and_predict(model_directory, example_text)
-    print(r)

crawl4ai/utils.py

@@ -60,7 +60,7 @@ def get_system_memory():
         raise OSError("Unsupported operating system")
 
 def get_home_folder():
-    home_folder = os.path.join(Path.home(), ".crawl4ai")
+    home_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())), ".crawl4ai")
     os.makedirs(home_folder, exist_ok=True)
     os.makedirs(f"{home_folder}/cache", exist_ok=True)
     os.makedirs(f"{home_folder}/models", exist_ok=True)
@@ -706,9 +706,12 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
     body = flatten_nested_elements(body)
     base64_pattern = re.compile(r'data:image/[^;]+;base64,([^"]+)')
     for img in imgs:
-        src = img.get('src', '')
-        if base64_pattern.match(src):
-            img['src'] = base64_pattern.sub('', src)
+        try:
+            src = img.get('src', '')
+            if base64_pattern.match(src):
+                img['src'] = base64_pattern.sub('', src)
+        except:
+            pass        
 
     cleaned_html = str(body).replace('\n\n', '\n').replace('  ', ' ')
     cleaned_html = sanitize_html(cleaned_html)

crawl4ai/web_crawler.back.py

@@ -1,357 +0,0 @@
-import os, time
-os.environ["TOKENIZERS_PARALLELISM"] = "false"
-from pathlib import Path
-
-from .models import UrlModel, CrawlResult
-from .database import init_db, get_cached_url, cache_url, DB_PATH, flush_db
-from .utils import *
-from .chunking_strategy import *
-from .extraction_strategy import *
-from .crawler_strategy import *
-from typing import List
-from concurrent.futures import ThreadPoolExecutor
-from .config import *
-
-
-class WebCrawler:
-    def __init__(
-        self,
-        # db_path: str = None,
-        crawler_strategy: CrawlerStrategy = None,
-        always_by_pass_cache: bool = False,
-        verbose: bool = False,
-    ):
-        # self.db_path = db_path
-        self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose)
-        self.always_by_pass_cache = always_by_pass_cache
-
-        # Create the .crawl4ai folder in the user's home directory if it doesn't exist
-        self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
-        os.makedirs(self.crawl4ai_folder, exist_ok=True)
-        os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
-
-        # If db_path is not provided, use the default path
-        # if not db_path:
-            # self.db_path = f"{self.crawl4ai_folder}/crawl4ai.db"
-        
-        # flush_db()
-        init_db()
-        
-        self.ready = False
-        
-    def warmup(self):
-        print("[LOG] 🌤️  Warming up the WebCrawler")
-        result = self.run(
-            url='https://crawl4ai.uccode.io/',
-            word_count_threshold=5,
-            extraction_strategy= NoExtractionStrategy(),
-            bypass_cache=False,
-            verbose = False
-        )
-        self.ready = True
-        print("[LOG] 🌞 WebCrawler is ready to crawl")
-        
-    def fetch_page(
-        self,
-        url_model: UrlModel,
-        provider: str = DEFAULT_PROVIDER,
-        api_token: str = None,
-        extract_blocks_flag: bool = True,
-        word_count_threshold=MIN_WORD_THRESHOLD,
-        css_selector: str = None,
-        screenshot: bool = False,
-        use_cached_html: bool = False,
-        extraction_strategy: ExtractionStrategy = None,
-        chunking_strategy: ChunkingStrategy = RegexChunking(),
-        **kwargs,
-    ) -> CrawlResult:
-        return self.run(
-            url_model.url,
-            word_count_threshold,
-            extraction_strategy or NoExtractionStrategy(),
-            chunking_strategy,
-            bypass_cache=url_model.forced,
-            css_selector=css_selector,
-            screenshot=screenshot,
-            **kwargs,
-        )
-        pass
-
-    def run_old(
-        self,
-        url: str,
-        word_count_threshold=MIN_WORD_THRESHOLD,
-        extraction_strategy: ExtractionStrategy = None,
-        chunking_strategy: ChunkingStrategy = RegexChunking(),
-        bypass_cache: bool = False,
-        css_selector: str = None,
-        screenshot: bool = False,
-        user_agent: str = None,
-        verbose=True,
-        **kwargs,
-    ) -> CrawlResult:
-        if user_agent:
-            self.crawler_strategy.update_user_agent(user_agent)
-        extraction_strategy = extraction_strategy or NoExtractionStrategy()
-        extraction_strategy.verbose = verbose
-        # Check if extraction strategy is an instance of ExtractionStrategy if not raise an error
-        if not isinstance(extraction_strategy, ExtractionStrategy):
-            raise ValueError("Unsupported extraction strategy")
-        if not isinstance(chunking_strategy, ChunkingStrategy):
-            raise ValueError("Unsupported chunking strategy")
-        
-        # make sure word_count_threshold is not lesser than MIN_WORD_THRESHOLD
-        if word_count_threshold < MIN_WORD_THRESHOLD:
-            word_count_threshold = MIN_WORD_THRESHOLD
-
-        # Check cache first
-        if not bypass_cache and not self.always_by_pass_cache:
-            cached = get_cached_url(url)
-            if cached:
-                return CrawlResult(
-                    **{
-                        "url": cached[0],
-                        "html": cached[1],
-                        "cleaned_html": cached[2],
-                        "markdown": cached[3],
-                        "extracted_content": cached[4],
-                        "success": cached[5],
-                        "media": json.loads(cached[6] or "{}"),
-                        "links": json.loads(cached[7] or "{}"),
-                        "metadata": json.loads(cached[8] or "{}"), # "metadata": "{}
-                        "screenshot": cached[9],
-                        "error_message": "",
-                    }
-                )
-
-        # Initialize WebDriver for crawling
-        t = time.time()
-        if kwargs.get("js", None):
-            self.crawler_strategy.js_code = kwargs.get("js")
-        html = self.crawler_strategy.crawl(url)
-        base64_image = None
-        if screenshot:
-            base64_image = self.crawler_strategy.take_screenshot()
-        success = True
-        error_message = ""
-        # Extract content from HTML
-        try:
-            result = get_content_of_website(url, html, word_count_threshold, css_selector=css_selector)
-            metadata = extract_metadata(html)
-            if result is None:
-                raise ValueError(f"Failed to extract content from the website: {url}")
-        except InvalidCSSSelectorError as e:
-            raise ValueError(str(e))
-        
-        cleaned_html = result.get("cleaned_html", "")
-        markdown = result.get("markdown", "")
-        media = result.get("media", [])
-        links = result.get("links", [])
-
-        # Print a profession LOG style message, show time taken and say crawling is done
-        if verbose:
-            print(
-                f"[LOG] 🚀 Crawling done for {url}, success: {success}, time taken: {time.time() - t} seconds"
-            )
-
-        extracted_content = []
-        if verbose:
-            print(f"[LOG] 🔥 Extracting semantic blocks for {url}, Strategy: {extraction_strategy.name}")
-        t = time.time()
-        # Split markdown into sections
-        sections = chunking_strategy.chunk(markdown)
-        # sections = merge_chunks_based_on_token_threshold(sections, CHUNK_TOKEN_THRESHOLD)
-
-        extracted_content = extraction_strategy.run(
-            url, sections,
-        )
-        extracted_content = json.dumps(extracted_content)
-
-        if verbose:
-            print(
-                f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t} seconds."
-            )
-
-        # Cache the result
-        cleaned_html = beautify_html(cleaned_html)
-        cache_url(
-            url,
-            html,
-            cleaned_html,
-            markdown,
-            extracted_content,
-            success,
-            json.dumps(media),
-            json.dumps(links),
-            json.dumps(metadata),
-            screenshot=base64_image,
-        )
-
-        return CrawlResult(
-            url=url,
-            html=html,
-            cleaned_html=cleaned_html,
-            markdown=markdown,
-            media=media,
-            links=links,
-            metadata=metadata,
-            screenshot=base64_image,
-            extracted_content=extracted_content,
-            success=success,
-            error_message=error_message,
-        )
-
-    def fetch_pages(
-        self,
-        url_models: List[UrlModel],
-        provider: str = DEFAULT_PROVIDER,
-        api_token: str = None,
-        extract_blocks_flag: bool = True,
-        word_count_threshold=MIN_WORD_THRESHOLD,
-        use_cached_html: bool = False,
-        css_selector: str = None,
-        screenshot: bool = False,
-        extraction_strategy: ExtractionStrategy = None,
-        chunking_strategy: ChunkingStrategy = RegexChunking(),
-        **kwargs,
-    ) -> List[CrawlResult]:
-        extraction_strategy = extraction_strategy or NoExtractionStrategy()
-        def fetch_page_wrapper(url_model, *args, **kwargs):
-            return self.fetch_page(url_model, *args, **kwargs)
-
-        with ThreadPoolExecutor() as executor:
-            results = list(
-                executor.map(
-                    fetch_page_wrapper,
-                    url_models,
-                    [provider] * len(url_models),
-                    [api_token] * len(url_models),
-                    [extract_blocks_flag] * len(url_models),
-                    [word_count_threshold] * len(url_models),
-                    [css_selector] * len(url_models),
-                    [screenshot] * len(url_models),
-                    [use_cached_html] * len(url_models),
-                    [extraction_strategy] * len(url_models),
-                    [chunking_strategy] * len(url_models),
-                    *[kwargs] * len(url_models),
-                )
-            )
-
-        return results
-
-    def run(
-            self,
-            url: str,
-            word_count_threshold=MIN_WORD_THRESHOLD,
-            extraction_strategy: ExtractionStrategy = None,
-            chunking_strategy: ChunkingStrategy = RegexChunking(),
-            bypass_cache: bool = False,
-            css_selector: str = None,
-            screenshot: bool = False,
-            user_agent: str = None,
-            verbose=True,
-            **kwargs,
-        ) -> CrawlResult:
-            extraction_strategy = extraction_strategy or NoExtractionStrategy()
-            extraction_strategy.verbose = verbose
-            if not isinstance(extraction_strategy, ExtractionStrategy):
-                raise ValueError("Unsupported extraction strategy")
-            if not isinstance(chunking_strategy, ChunkingStrategy):
-                raise ValueError("Unsupported chunking strategy")
-            
-            if word_count_threshold < MIN_WORD_THRESHOLD:
-                word_count_threshold = MIN_WORD_THRESHOLD
-
-            # Check cache first
-            cached = None
-            extracted_content = None
-            if not bypass_cache and not self.always_by_pass_cache:
-                cached = get_cached_url(url)
-            
-            if cached:
-                html = cached[1]
-                extracted_content = cached[2]
-                if screenshot:
-                    screenshot = cached[9]
-            
-            else:
-                if user_agent:
-                    self.crawler_strategy.update_user_agent(user_agent)
-                html = self.crawler_strategy.crawl(url)
-                if screenshot:
-                    screenshot = self.crawler_strategy.take_screenshot()
-            
-            return self.process_html(url, html, extracted_content, word_count_threshold, extraction_strategy, chunking_strategy, css_selector, screenshot, verbose, bool(cached), **kwargs)
-
-    def process_html(
-            self,
-            url: str,
-            html: str,
-            extracted_content: str,
-            word_count_threshold: int,
-            extraction_strategy: ExtractionStrategy,
-            chunking_strategy: ChunkingStrategy,
-            css_selector: str,
-            screenshot: bool,
-            verbose: bool,
-            is_cached: bool,
-            **kwargs,
-        ) -> CrawlResult:
-            t = time.time()
-            # Extract content from HTML
-            try:
-                result = get_content_of_website(url, html, word_count_threshold, css_selector=css_selector)
-                metadata = extract_metadata(html)
-                if result is None:
-                    raise ValueError(f"Failed to extract content from the website: {url}")
-            except InvalidCSSSelectorError as e:
-                raise ValueError(str(e))
-            
-            cleaned_html = result.get("cleaned_html", "")
-            markdown = result.get("markdown", "")
-            media = result.get("media", [])
-            links = result.get("links", [])
-
-            if verbose:
-                print(f"[LOG] 🚀 Crawling done for {url}, success: True, time taken: {time.time() - t} seconds")
-                        
-            if extracted_content is None:
-                if verbose:
-                    print(f"[LOG] 🔥 Extracting semantic blocks for {url}, Strategy: {extraction_strategy.name}")
-
-                sections = chunking_strategy.chunk(markdown)
-                extracted_content = extraction_strategy.run(url, sections)
-                extracted_content = json.dumps(extracted_content)
-
-                if verbose:
-                    print(f"[LOG] 🚀 Extraction done for {url}, time taken: {time.time() - t} seconds.")
-                
-            screenshot = None if not screenshot else screenshot
-            
-            if not is_cached:
-                cache_url(
-                    url,
-                    html,
-                    cleaned_html,
-                    markdown,
-                    extracted_content,
-                    True,
-                    json.dumps(media),
-                    json.dumps(links),
-                    json.dumps(metadata),
-                    screenshot=screenshot,
-                )                
-
-            return CrawlResult(
-                url=url,
-                html=html,
-                cleaned_html=cleaned_html,
-                markdown=markdown,
-                media=media,
-                links=links,
-                metadata=metadata,
-                screenshot=screenshot,
-                extracted_content=extracted_content,
-                success=True,
-                error_message="",
-            )

crawl4ai/web_crawler.py

@@ -20,7 +20,7 @@ class WebCrawler:
     def __init__(self, crawler_strategy: CrawlerStrategy = None, always_by_pass_cache: bool = False, verbose: bool = False):
         self.crawler_strategy = crawler_strategy or LocalSeleniumCrawlerStrategy(verbose=verbose)
         self.always_by_pass_cache = always_by_pass_cache
-        self.crawl4ai_folder = os.path.join(Path.home(), ".crawl4ai")
+        self.crawl4ai_folder = os.path.join(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai")
         os.makedirs(self.crawl4ai_folder, exist_ok=True)
         os.makedirs(f"{self.crawl4ai_folder}/cache", exist_ok=True)
         init_db()

docs/examples/docker_example.py

@@ -0,0 +1,300 @@
+import requests
+import json
+import time
+import sys
+import base64
+import os
+from typing import Dict, Any
+
+class Crawl4AiTester:
+    def __init__(self, base_url: str = "http://localhost:11235"):
+        self.base_url = base_url
+        
+    def submit_and_wait(self, request_data: Dict[str, Any], timeout: int = 300) -> Dict[str, Any]:
+        # Submit crawl job
+        response = requests.post(f"{self.base_url}/crawl", json=request_data)
+        task_id = response.json()["task_id"]
+        print(f"Task ID: {task_id}")
+        
+        # Poll for result
+        start_time = time.time()
+        while True:
+            if time.time() - start_time > timeout:
+                raise TimeoutError(f"Task {task_id} did not complete within {timeout} seconds")
+                
+            result = requests.get(f"{self.base_url}/task/{task_id}")
+            status = result.json()
+            
+            if status["status"] == "failed":
+                print("Task failed:", status.get("error"))
+                raise Exception(f"Task failed: {status.get('error')}")
+                
+            if status["status"] == "completed":
+                return status
+                
+            time.sleep(2)
+
+def test_docker_deployment(version="basic"):
+    tester = Crawl4AiTester()
+    print(f"Testing Crawl4AI Docker {version} version")
+    
+    # Health check with timeout and retry
+    max_retries = 5
+    for i in range(max_retries):
+        try:
+            health = requests.get(f"{tester.base_url}/health", timeout=10)
+            print("Health check:", health.json())
+            break
+        except requests.exceptions.RequestException as e:
+            if i == max_retries - 1:
+                print(f"Failed to connect after {max_retries} attempts")
+                sys.exit(1)
+            print(f"Waiting for service to start (attempt {i+1}/{max_retries})...")
+            time.sleep(5)
+    
+    # Test cases based on version
+    test_basic_crawl(tester)
+    
+    # if version in ["full", "transformer"]:
+    #     test_cosine_extraction(tester)
+
+    # test_js_execution(tester)
+    # test_css_selector(tester)
+    # test_structured_extraction(tester)
+    # test_llm_extraction(tester)
+    # test_llm_with_ollama(tester)
+    # test_screenshot(tester)
+    
+
+def test_basic_crawl(tester: Crawl4AiTester):
+    print("\n=== Testing Basic Crawl ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 10
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"Basic crawl result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+    assert len(result["result"]["markdown"]) > 0
+
+def test_js_execution(tester: Crawl4AiTester):
+    print("\n=== Testing JS Execution ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "js_code": [
+            "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
+        ],
+        "wait_for": "article.tease-card:nth-child(10)",
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"JS execution result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_css_selector(tester: Crawl4AiTester):
+    print("\n=== Testing CSS Selector ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 7,
+        "css_selector": ".wide-tease-item__description",
+        "crawler_params": {
+            "headless": True
+        },
+        "extra": {"word_count_threshold": 10}
+        
+    }
+    
+    result = tester.submit_and_wait(request)
+    print(f"CSS selector result length: {len(result['result']['markdown'])}")
+    assert result["result"]["success"]
+
+def test_structured_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Structured Extraction ===")
+    schema = {
+        "name": "Coinbase Crypto Prices",
+        "baseSelector": ".cds-tableRow-t45thuk",
+        "fields": [
+            {
+                "name": "crypto",
+                "selector": "td:nth-child(1) h2",
+                "type": "text",
+            },
+            {
+                "name": "symbol",
+                "selector": "td:nth-child(1) p",
+                "type": "text",
+            },
+            {
+                "name": "price",
+                "selector": "td:nth-child(2)",
+                "type": "text",
+            }
+        ],
+    }
+    
+    request = {
+        "urls": "https://www.coinbase.com/explore",
+        "priority": 9,
+        "extraction_config": {
+            "type": "json_css",
+            "params": {
+                "schema": schema
+            }
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    extracted = json.loads(result["result"]["extracted_content"])
+    print(f"Extracted {len(extracted)} items")
+    print("Sample item:", json.dumps(extracted[0], indent=2))
+    assert result["result"]["success"]
+    assert len(extracted) > 0
+
+def test_llm_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing LLM Extraction ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "model_name": {
+                "type": "string",
+                "description": "Name of the OpenAI model."
+            },
+            "input_fee": {
+                "type": "string",
+                "description": "Fee for input token for the OpenAI model."
+            },
+            "output_fee": {
+                "type": "string",
+                "description": "Fee for output token for the OpenAI model."
+            }
+        },
+        "required": ["model_name", "input_fee", "output_fee"]
+    }
+    
+    request = {
+        "urls": "https://openai.com/api/pricing",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "openai/gpt-4o-mini",
+                "api_token": os.getenv("OPENAI_API_KEY"),
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": """From the crawled content, extract all mentioned model names along with their fees for input and output tokens."""
+            }
+        },
+        "crawler_params": {"word_count_threshold": 1}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} model pricing entries")
+        print("Sample entry:", json.dumps(extracted[0], indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"LLM extraction test failed (might be due to missing API key): {str(e)}")
+
+def test_llm_with_ollama(tester: Crawl4AiTester):
+    print("\n=== Testing LLM with Ollama ===")
+    schema = {
+        "type": "object",
+        "properties": {
+            "article_title": {
+                "type": "string",
+                "description": "The main title of the news article"
+            },
+            "summary": {
+                "type": "string",
+                "description": "A brief summary of the article content"
+            },
+            "main_topics": {
+                "type": "array",
+                "items": {"type": "string"},
+                "description": "Main topics or themes discussed in the article"
+            }
+        }
+    }
+    
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "llm",
+            "params": {
+                "provider": "ollama/llama2",
+                "schema": schema,
+                "extraction_type": "schema",
+                "instruction": "Extract the main article information including title, summary, and main topics."
+            }
+        },
+        "extra": {"word_count_threshold": 1},
+        "crawler_params": {"verbose": True}
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print("Extracted content:", json.dumps(extracted, indent=2))
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Ollama extraction test failed: {str(e)}")
+
+def test_cosine_extraction(tester: Crawl4AiTester):
+    print("\n=== Testing Cosine Extraction ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 8,
+        "extraction_config": {
+            "type": "cosine",
+            "params": {
+                "semantic_filter": "business finance economy",
+                "word_count_threshold": 10,
+                "max_dist": 0.2,
+                "top_k": 3
+            }
+        }
+    }
+    
+    try:
+        result = tester.submit_and_wait(request)
+        extracted = json.loads(result["result"]["extracted_content"])
+        print(f"Extracted {len(extracted)} text clusters")
+        print("First cluster tags:", extracted[0]["tags"])
+        assert result["result"]["success"]
+    except Exception as e:
+        print(f"Cosine extraction test failed: {str(e)}")
+
+def test_screenshot(tester: Crawl4AiTester):
+    print("\n=== Testing Screenshot ===")
+    request = {
+        "urls": "https://www.nbcnews.com/business",
+        "priority": 5,
+        "screenshot": True,
+        "crawler_params": {
+            "headless": True
+        }
+    }
+    
+    result = tester.submit_and_wait(request)
+    print("Screenshot captured:", bool(result["result"]["screenshot"]))
+    
+    if result["result"]["screenshot"]:
+        # Save screenshot
+        screenshot_data = base64.b64decode(result["result"]["screenshot"])
+        with open("test_screenshot.jpg", "wb") as f:
+            f.write(screenshot_data)
+        print("Screenshot saved as test_screenshot.jpg")
+    
+    assert result["result"]["success"]
+
+if __name__ == "__main__":
+    version = sys.argv[1] if len(sys.argv) > 1 else "basic"
+    # version = "full"
+    test_docker_deployment(version)

docs/md_v2/api/async-webcrawler.md

@@ -13,7 +13,7 @@ AsyncWebCrawler(
     
     # Cache Settings
     always_by_pass_cache: bool = False,     # Always bypass cache
-    base_directory: str = str(Path.home()), # Base directory for cache
+    base_directory: str = str(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())), # Base directory for cache
     
     # Network Settings
     proxy: str = None,                      # Simple proxy URL

docs/md_v2/tutorial/episode_08_Media_Handling_Images_Videos_and_Audio.md

@@ -113,4 +113,4 @@ Here’s a clear and focused outline for the **Media Handling: Images, Videos, a
 
 ---
 
-This outline provides users with a complete guide to handling images, videos, and audio in Crawl4AI, using metadata to enhance relevance and precision in multimedia extraction.
+This outline provides users with a complete guide to handling images, videos, and audio in Crawl4AI, using metadata to enhance relevance and precision in multimedia extraction.

docs/md_v2/tutorial/episode_11_1_Extraction_Strategies_JSON_CSS.md

@@ -183,4 +183,4 @@ Here’s a detailed outline for the **JSON-CSS Extraction Strategy** video, cove
 
 ---
 
-This outline covers each JSON-CSS Extraction option in Crawl4AI, with practical examples and schema configurations, making it a thorough guide for users.
+This outline covers each JSON-CSS Extraction option in Crawl4AI, with practical examples and schema configurations, making it a thorough guide for users.

docs/md_v2/tutorial/episode_11_2_Extraction_Strategies_LLM.md

@@ -150,4 +150,4 @@ Here’s a comprehensive outline for the **LLM Extraction Strategy** video, cove
 
 ---
 
-This outline explains LLM Extraction in Crawl4AI, with examples showing how to extract structured data using custom schemas and instructions. It demonstrates flexibility with multiple providers, ensuring practical application for different use cases.
+This outline explains LLM Extraction in Crawl4AI, with examples showing how to extract structured data using custom schemas and instructions. It demonstrates flexibility with multiple providers, ensuring practical application for different use cases.

docs/md_v2/tutorial/episode_11_3_Extraction_Strategies_Cosine.md

@@ -133,4 +133,4 @@ Here’s a structured outline for the **Cosine Similarity Strategy** video, cove
 
 ---
 
-This outline covers Cosine Similarity Strategy’s speed and effectiveness, providing examples that showcase its potential for clustering various content types efficiently.
+This outline covers Cosine Similarity Strategy’s speed and effectiveness, providing examples that showcase its potential for clustering various content types efficiently.

main.py

@@ -1,6 +1,16 @@
-import asyncio
+import asyncio, os
 from fastapi import FastAPI, HTTPException, BackgroundTasks, Request
 from fastapi.responses import JSONResponse
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.responses import HTMLResponse, JSONResponse
+from fastapi.staticfiles import StaticFiles
+from fastapi.middleware.cors import CORSMiddleware  
+from fastapi.templating import Jinja2Templates
+from fastapi.exceptions import RequestValidationError
+from starlette.middleware.base import BaseHTTPMiddleware
+from starlette.responses import FileResponse
+from fastapi.responses import RedirectResponse
+
 from pydantic import BaseModel, HttpUrl, Field
 from typing import Optional, List, Dict, Any, Union
 import psutil
@@ -20,6 +30,8 @@ from crawl4ai.extraction_strategy import (
     JsonCssExtractionStrategy,
 )
 
+__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
+
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 
@@ -50,6 +62,7 @@ class CrawlRequest(BaseModel):
     css_selector: Optional[str] = None
     screenshot: bool = False
     magic: bool = False
+    extra: Optional[Dict[str, Any]] = {}
 
 @dataclass
 class TaskInfo:
@@ -239,7 +252,7 @@ class CrawlerService:
         while True:
             try:
                 available_slots = await self.resource_monitor.get_available_slots()
-                if available_slots <= 0:
+                if False and available_slots <= 0:
                     await asyncio.sleep(1)
                     continue
 
@@ -295,6 +308,23 @@ class CrawlerService:
                 await asyncio.sleep(1)
 
 app = FastAPI(title="Crawl4AI API")
+
+# CORS configuration
+origins = ["*"]  # Allow all origins
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=origins,  # List of origins that are allowed to make requests
+    allow_credentials=True,
+    allow_methods=["*"],  # Allows all methods
+    allow_headers=["*"],  # Allows all headers
+)
+
+# Mount the pages directory as a static directory
+app.mount("/pages", StaticFiles(directory=__location__ + "/pages"), name="pages")
+app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
+site_templates = Jinja2Templates(directory=__location__ + "/site")
+templates = Jinja2Templates(directory=__location__ + "/pages")
+
 crawler_service = CrawlerService()
 
 @app.on_event("startup")
@@ -305,6 +335,11 @@ async def startup_event():
 async def shutdown_event():
     await crawler_service.stop()
 
+@app.get("/")
+def read_root():
+    return RedirectResponse(url="/mkdocs")
+
+
 @app.post("/crawl")
 async def crawl(request: CrawlRequest) -> Dict[str, str]:
     task_id = await crawler_service.submit_task(request)

main_v0.py

@@ -1,254 +0,0 @@
-import os
-import importlib
-import asyncio
-from functools import lru_cache
-import logging
-logging.basicConfig(level=logging.DEBUG)
-
-from fastapi import FastAPI, HTTPException, Request
-from fastapi.responses import HTMLResponse, JSONResponse
-from fastapi.staticfiles import StaticFiles
-from fastapi.middleware.cors import CORSMiddleware  
-from fastapi.templating import Jinja2Templates
-from fastapi.exceptions import RequestValidationError
-from starlette.middleware.base import BaseHTTPMiddleware
-from starlette.responses import FileResponse
-from fastapi.responses import RedirectResponse
-
-from pydantic import BaseModel, HttpUrl
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import List, Optional
-
-from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.database import get_total_count, clear_db
-
-import time
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
-
-# load .env file
-from dotenv import load_dotenv
-load_dotenv()
-
-# Configuration
-__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
-MAX_CONCURRENT_REQUESTS = 10  # Adjust this to change the maximum concurrent requests
-current_requests = 0
-lock = asyncio.Lock()
-
-app = FastAPI()
-
-# Initialize rate limiter
-def rate_limit_key_func(request: Request):
-    access_token = request.headers.get("access-token")
-    if access_token == os.environ.get('ACCESS_TOKEN'):
-        return None
-    return get_remote_address(request)
-
-limiter = Limiter(key_func=rate_limit_key_func)
-app.state.limiter = limiter
-
-# Dictionary to store last request times for each client
-last_request_times = {}
-last_rate_limit = {}
-
-
-def get_rate_limit():
-    limit = os.environ.get('ACCESS_PER_MIN', "5")
-    return f"{limit}/minute"
-
-# Custom rate limit exceeded handler
-async def custom_rate_limit_exceeded_handler(request: Request, exc: RateLimitExceeded) -> JSONResponse:
-    if request.client.host not in last_rate_limit or time.time() - last_rate_limit[request.client.host] > 60:
-        last_rate_limit[request.client.host] = time.time()
-    retry_after = 60 - (time.time() - last_rate_limit[request.client.host])
-    reset_at = time.time() + retry_after
-    return JSONResponse(
-        status_code=429,
-        content={
-            "detail": "Rate limit exceeded",
-            "limit": str(exc.limit.limit),
-            "retry_after": retry_after,
-            'reset_at': reset_at,
-            "message": f"You have exceeded the rate limit of {exc.limit.limit}."
-        }
-    )
-    
-app.add_exception_handler(RateLimitExceeded, custom_rate_limit_exceeded_handler)
-
-
-# Middleware for token-based bypass and per-request limit
-class RateLimitMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        SPAN = int(os.environ.get('ACCESS_TIME_SPAN', 10))
-        access_token = request.headers.get("access-token")
-        if access_token == os.environ.get('ACCESS_TOKEN'):
-            return await call_next(request)
-        
-        path = request.url.path
-        if path in ["/crawl", "/old"]:
-            client_ip = request.client.host
-            current_time = time.time()
-            
-            # Check time since last request
-            if client_ip in last_request_times:
-                time_since_last_request = current_time - last_request_times[client_ip]
-                if time_since_last_request < SPAN:
-                    return JSONResponse(
-                        status_code=429,
-                        content={
-                            "detail": "Too many requests",
-                            "message": "Rate limit exceeded. Please wait 10 seconds between requests.",
-                            "retry_after": max(0, SPAN - time_since_last_request),
-                            "reset_at": current_time + max(0, SPAN - time_since_last_request),
-                        }
-                    )
-            
-            last_request_times[client_ip] = current_time
-
-        return await call_next(request)
-
-app.add_middleware(RateLimitMiddleware)
-
-# CORS configuration
-origins = ["*"]  # Allow all origins
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=origins,  # List of origins that are allowed to make requests
-    allow_credentials=True,
-    allow_methods=["*"],  # Allows all methods
-    allow_headers=["*"],  # Allows all headers
-)
-
-# Mount the pages directory as a static directory
-app.mount("/pages", StaticFiles(directory=__location__ + "/pages"), name="pages")
-app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
-site_templates = Jinja2Templates(directory=__location__ + "/site")
-templates = Jinja2Templates(directory=__location__ + "/pages")
-
-@lru_cache()
-def get_crawler():
-    # Initialize and return a WebCrawler instance
-    crawler = WebCrawler(verbose = True)
-    crawler.warmup()
-    return crawler
-
-class CrawlRequest(BaseModel):
-    urls: List[str]
-    include_raw_html: Optional[bool] = False
-    bypass_cache: bool = False
-    extract_blocks: bool = True
-    word_count_threshold: Optional[int] = 5
-    extraction_strategy: Optional[str] = "NoExtractionStrategy"
-    extraction_strategy_args: Optional[dict] = {}
-    chunking_strategy: Optional[str] = "RegexChunking"
-    chunking_strategy_args: Optional[dict] = {}
-    css_selector: Optional[str] = None
-    screenshot: Optional[bool] = False
-    user_agent: Optional[str] = None
-    verbose: Optional[bool] = True
-
-@app.get("/")
-def read_root():
-    return RedirectResponse(url="/mkdocs")
-
-@app.get("/old", response_class=HTMLResponse)
-@limiter.limit(get_rate_limit())
-async def read_index(request: Request):
-    partials_dir = os.path.join(__location__, "pages", "partial")
-    partials = {}
-
-    for filename in os.listdir(partials_dir):
-        if filename.endswith(".html"):
-            with open(os.path.join(partials_dir, filename), "r", encoding="utf8") as file:
-                partials[filename[:-5]] = file.read()
-
-    return templates.TemplateResponse("index.html", {"request": request, **partials})
-
-@app.get("/total-count")
-async def get_total_url_count():
-    count = get_total_count()
-    return JSONResponse(content={"count": count})
-
-@app.get("/clear-db")
-async def clear_database():
-    # clear_db()
-    return JSONResponse(content={"message": "Database cleared."})
-
-def import_strategy(module_name: str, class_name: str, *args, **kwargs):
-    try:
-        module = importlib.import_module(module_name)
-        strategy_class = getattr(module, class_name)
-        return strategy_class(*args, **kwargs)
-    except ImportError:
-        print("ImportError: Module not found.")
-        raise HTTPException(status_code=400, detail=f"Module {module_name} not found.")
-    except AttributeError:
-        print("AttributeError: Class not found.")
-        raise HTTPException(status_code=400, detail=f"Class {class_name} not found in {module_name}.")
-
-@app.post("/crawl")
-@limiter.limit(get_rate_limit())
-async def crawl_urls(crawl_request: CrawlRequest, request: Request):
-    logging.debug(f"[LOG] Crawl request for URL: {crawl_request.urls}")
-    global current_requests
-    async with lock:
-        if current_requests >= MAX_CONCURRENT_REQUESTS:
-            raise HTTPException(status_code=429, detail="Too many requests - please try again later.")
-        current_requests += 1
-
-    try:
-        logging.debug("[LOG] Loading extraction and chunking strategies...")
-        crawl_request.extraction_strategy_args['verbose'] = True
-        crawl_request.chunking_strategy_args['verbose'] = True
-        
-        extraction_strategy = import_strategy("crawl4ai.extraction_strategy", crawl_request.extraction_strategy, **crawl_request.extraction_strategy_args)
-        chunking_strategy = import_strategy("crawl4ai.chunking_strategy", crawl_request.chunking_strategy, **crawl_request.chunking_strategy_args)
-
-        # Use ThreadPoolExecutor to run the synchronous WebCrawler in async manner
-        logging.debug("[LOG] Running the WebCrawler...")
-        with ThreadPoolExecutor() as executor:
-            loop = asyncio.get_event_loop()
-            futures = [
-                loop.run_in_executor(
-                    executor, 
-                    get_crawler().run,
-                    str(url),
-                    crawl_request.word_count_threshold,
-                    extraction_strategy,
-                    chunking_strategy,
-                    crawl_request.bypass_cache,
-                    crawl_request.css_selector,
-                    crawl_request.screenshot,
-                    crawl_request.user_agent,
-                    crawl_request.verbose
-                )
-                for url in crawl_request.urls
-            ]
-            results = await asyncio.gather(*futures)
-
-        # if include_raw_html is False, remove the raw HTML content from the results
-        if not crawl_request.include_raw_html:
-            for result in results:
-                result.html = None
-
-        return {"results": [result.model_dump() for result in results]}
-    finally:
-        async with lock:
-            current_requests -= 1
-            
-@app.get("/strategies/extraction", response_class=JSONResponse)
-async def get_extraction_strategies():
-    with open(f"{__location__}/docs/extraction_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
-
-@app.get("/strategies/chunking", response_class=JSONResponse)
-async def get_chunking_strategies():
-    with open(f"{__location__}/docs/chunking_strategies.json", "r") as file:
-        return JSONResponse(content=file.read())
-
-
-if __name__ == "__main__":
-    import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8888)

requirements-dev.txt

@@ -2,4 +2,4 @@
 pytest
 pytest-asyncio
 selenium
-setuptools
+setuptools

requirements.txt

@@ -8,4 +8,4 @@ playwright>=1.47,<1.48
 python-dotenv~=1.0
 requests~=2.26
 beautifulsoup4~=4.12
-playwright_stealth~=1.0
+playwright_stealth~=1.0

setup.py

@@ -8,7 +8,7 @@ import sys
 
 # Create the .crawl4ai folder in the user's home directory if it doesn't exist
 # If the folder already exists, remove the cache folder
-crawl4ai_folder = Path.home() / ".crawl4ai"
+crawl4ai_folder = os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()) / ".crawl4ai"
 cache_folder = crawl4ai_folder / "cache"
 
 if cache_folder.exists():

tests/test_docker.py

@@ -7,7 +7,7 @@ import os
 from typing import Dict, Any
 
 class Crawl4AiTester:
-    def __init__(self, base_url: str = "http://localhost:8000"):
+    def __init__(self, base_url: str = "http://localhost:11235"):
         self.base_url = base_url
         
     def submit_and_wait(self, request_data: Dict[str, Any], timeout: int = 300) -> Dict[str, Any]:
@@ -54,8 +54,9 @@ def test_docker_deployment(version="basic"):
     
     # Test cases based on version
     test_basic_crawl(tester)
-    if version in ["full", "transformer"]:
-        test_cosine_extraction(tester)
+    
+    # if version in ["full", "transformer"]:
+    #     test_cosine_extraction(tester)
 
     # test_js_execution(tester)
     # test_css_selector(tester)