ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Release/v0.7.6 (#1556)

Browse Source 
* fix(docker-api): migrate to modern datetime library API

Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>

* Fix examples in README.md

* feat(docker): add user-provided hooks support to Docker API

Implements comprehensive hooks functionality allowing users to provide custom Python
functions as strings that execute at specific points in the crawling pipeline.

Key Features:
- Support for all 8 crawl4ai hook points:
  • on_browser_created: Initialize browser settings
  • on_page_context_created: Configure page context
  • before_goto: Pre-navigation setup
  • after_goto: Post-navigation processing
  • on_user_agent_updated: User agent modification handling
  • on_execution_started: Crawl execution initialization
  • before_retrieve_html: Pre-extraction processing
  • before_return_html: Final HTML processing

Implementation Details:
- Created UserHookManager for validation, compilation, and safe execution
- Added IsolatedHookWrapper for error isolation and timeout protection
- AST-based validation ensures code structure correctness
- Sandboxed execution with restricted builtins for security
- Configurable timeout (1-120 seconds) prevents infinite loops
- Comprehensive error handling ensures hooks don't crash main process
- Execution tracking with detailed statistics and logging

API Changes:
- Added HookConfig schema with code and timeout fields
- Extended CrawlRequest with optional hooks parameter
- Added /hooks/info endpoint for hook discovery
- Updated /crawl and /crawl/stream endpoints to support hooks

Safety Features:
- Malformed hooks return clear validation errors
- Hook errors are isolated and reported without stopping crawl
- Execution statistics track success/failure/timeout rates
- All hook results are JSON-serializable

Testing:
- Comprehensive test suite covering all 8 hooks
- Error handling and timeout scenarios validated
- Authentication, performance, and content extraction examples
- 100% success rate in production testing

Documentation:
- Added extensive hooks section to docker-deployment.md
- Security warnings about user-provided code risks
- Real-world examples using httpbin.org, GitHub, BBC
- Best practices and troubleshooting guide

ref #1377

* fix(deep-crawl): BestFirst priority inversion; remove pre-scoring truncation. ref #1253

  Use negative scores in PQ to visit high-score URLs first and drop link cap prior to scoring; add test for ordering.

* docs: Update URL seeding examples to use proper async context managers
- Wrap all AsyncUrlSeeder usage with async context managers
- Update URL seeding adventure example to use "sitemap+cc" source, focus on course posts, and add stream=True parameter to fix runtime error

* fix(crawler): Removed the incorrect reference in browser_config variable #1310

* docs: update Docker instructions to use the latest release tag

* fix(docker): Fix LLM API key handling for multi-provider support

Previously, the system incorrectly used OPENAI_API_KEY for all LLM providers
due to a hardcoded api_key_env fallback in config.yml. This caused authentication
errors when using non-OpenAI providers like Gemini.

Changes:
- Remove api_key_env from config.yml to let litellm handle provider-specific env vars
- Simplify get_llm_api_key() to return None, allowing litellm to auto-detect keys
- Update validate_llm_provider() to trust litellm's built-in key detection
- Update documentation to reflect the new automatic key handling

The fix leverages litellm's existing capability to automatically find the correct
environment variable for each provider (OPENAI_API_KEY, GEMINI_API_TOKEN, etc.)
without manual configuration.

ref #1291

* docs: update adaptive crawler docs and cache defaults; remove deprecated examples (#1330)
- Replace BaseStrategy with CrawlStrategy in custom strategy examples (DomainSpecificStrategy, HybridStrategy)
- Remove “Custom Link Scoring” and “Caching Strategy” sections no longer aligned with current library
- Revise memory pruning example to use adaptive.get_relevant_content and index-based retention of top 500 docs
- Correct Quickstart note: default cache mode is CacheMode.BYPASS; instruct enabling with CacheMode.ENABLED

* fix(utils): Improve URL normalization by avoiding quote/unquote to preserve '+' signs. ref #1332

* feat: Add comprehensive website to API example with frontend

This commit adds a complete, web scraping API example that demonstrates how to get structured data from any website and use it like an API using the crawl4ai library with a minimalist frontend interface.

Core Functionality
- AI-powered web scraping with plain English queries
- Dual scraping approaches: Schema-based (faster) and LLM-based (flexible)
- Intelligent schema caching for improved performance
- Custom LLM model support with API key management
- Automatic duplicate request prevention

Modern Frontend Interface
- Minimalist black-and-white design inspired by modern web apps
- Responsive layout with smooth animations and transitions
- Three main pages: Scrape Data, Models Management, API Request History
- Real-time results display with JSON formatting
- Copy-to-clipboard functionality for extracted data
- Toast notifications for user feedback
- Auto-scroll to results when scraping starts

Model Management System
- Web-based model configuration interface
- Support for any LLM provider (OpenAI, Gemini, Anthropic, etc.)
- Simplified configuration requiring only provider and API token
- Add, list, and delete model configurations
- Secure storage of API keys in local JSON files

API Request History
- Automatic saving of all API requests and responses
- Display of request history with URL, query, and cURL commands
- Duplicate prevention (same URL + query combinations)
- Request deletion functionality
- Clean, simplified display focusing on essential information

Technical Implementation

Backend (FastAPI)
- RESTful API with comprehensive endpoints
- Pydantic models for request/response validation
- Async web scraping with crawl4ai library
- Error handling with detailed error messages
- File-based storage for models and request history

Frontend (Vanilla JS/CSS/HTML)
- No framework dependencies - pure HTML, CSS, JavaScript
- Modern CSS Grid and Flexbox layouts
- Custom dropdown styling with SVG arrows
- Responsive design for mobile and desktop
- Smooth scrolling and animations

Core Library Integration
- WebScraperAgent class for orchestration
- ModelConfig class for LLM configuration management
- Schema generation and caching system
- LLM extraction strategy support
- Browser configuration with headless mode

* fix(dependencies): add cssselect to project dependencies

Fixes bug reported in issue #1405
[Bug]: Excluded selector (excluded_selector) doesn't work

This commit reintroduces the cssselect library which was removed by PR (https://github.com/unclecode/crawl4ai/pull/1368) and merged via (437395e490).

Integration tested against 0.7.4 Docker container. Reintroducing cssselector package eliminated errors seen in logs and excluded_selector functionality was restored.

Refs: #1405

* fix(docker): resolve filter serialization and JSON encoding errors in deep crawl strategy (ref #1419)

  - Fix URLPatternFilter serialization by preventing private __slots__ from being serialized as constructor params
  - Add public attributes to URLPatternFilter to store original constructor parameters for proper serialization
  - Handle property descriptors in CrawlResult.model_dump() to prevent JSON serialization errors
  - Ensure filter chains work correctly with Docker client and REST API

  The issue occurred because:
  1. Private implementation details (_simple_suffixes, etc.) were being serialized and passed as constructor arguments during deserialization
  2. Property descriptors were being included in the serialized output, causing "Object of type property is not JSON serializable" errors

  Changes:
  - async_configs.py: Comment out __slots__ serialization logic (lines 100-109)
  - filters.py: Add patterns, use_glob, reverse to URLPatternFilter __slots__ and store as public attributes
  - models.py: Convert property descriptors to strings in model_dump() instead of including them directly

* fix(logger): ensure logger is a Logger instance in crawling strategies. ref #1437

* feat(docker): Add temperature and base_url parameters for LLM configuration. ref #1035

  Implement hierarchical configuration for LLM parameters with support for:
  - Temperature control (0.0-2.0) to adjust response creativity
  - Custom base_url for proxy servers and alternative endpoints
  - 4-tier priority: request params > provider env > global env > defaults

  Add helper functions in utils.py, update API schemas and handlers,
  support environment variables (LLM_TEMPERATURE, OPENAI_TEMPERATURE, etc.),
  and provide comprehensive documentation with examples.

* feat(docker): improve docker error handling
- Return comprehensive error messages along with status codes for api internal errors.
- Fix fit_html property serialization issue in both /crawl and /crawl/stream endpoints
- Add sanitization to ensure fit_html is always JSON-serializable (string or None)
- Add comprehensive error handling test suite.

* #1375 : refactor(proxy) Deprecate 'proxy' parameter in BrowserConfig and enhance proxy string parsing

- Updated ProxyConfig.from_string to support multiple proxy formats, including URLs with credentials.
- Deprecated the 'proxy' parameter in BrowserConfig, replacing it with 'proxy_config' for better flexibility.
- Added warnings for deprecated usage and clarified behavior when both parameters are provided.
- Updated documentation and tests to reflect changes in proxy configuration handling.

* Remove deprecated test for 'proxy' parameter in BrowserConfig and update .gitignore to include test_scripts directory.

* feat: add preserve_https_for_internal_links flag to maintain HTTPS during crawling. Ref #1410

Added a new `preserve_https_for_internal_links` configuration flag that preserves the original HTTPS scheme for same-domain links even when the server redirects to HTTP.

* feat: update documentation for preserve_https_for_internal_links. ref #1410

* fix: drop Python 3.9 support and require Python >=3.10.
The library no longer supports Python 3.9 and so it was important to drop all references to python 3.9.
Following changes have been made:
- pyproject.toml: set requires-python to ">=3.10"; remove 3.9 classifier
- setup.py: set python_requires to ">=3.10"; remove 3.9 classifier
- docs: update Python version mentions
  - deploy/docker/c4ai-doc-context.md: options -> 3.10, 3.11, 3.12, 3.13

* issue #1329 refactor(crawler): move unwanted properties to CrawlerRunConfig class

* fix(auth): fixed Docker JWT authentication. ref #1442

* remove: delete unused yoyo snapshot subproject

* fix: raise error on last attempt failure in perform_completion_with_backoff. ref #989

* Commit without API

* fix: update option labels in request builder for clarity

* fix: allow custom LLM providers for adaptive crawler embedding config. ref: #1291

  - Change embedding_llm_config from Dict to Union[LLMConfig, Dict] for type safety
  - Add backward-compatible conversion property _embedding_llm_config_dict
  - Replace all hardcoded OpenAI embedding configs with configurable options
  - Fix LLMConfig object attribute access in query expansion logic
  - Add comprehensive example demonstrating multiple provider configurations
  - Update documentation with both LLMConfig object and dictionary usage patterns

  Users can now specify any LLM provider for query expansion in embedding strategy:
  - New: embedding_llm_config=LLMConfig(provider='anthropic/claude-3', api_token='key')
  - Old: embedding_llm_config={'provider': 'openai/gpt-4', 'api_token': 'key'} (still works)

* refactor(BrowserConfig): change deprecation warning for 'proxy' parameter to UserWarning

* feat(StealthAdapter): fix stealth features for Playwright integration. ref #1481

* #1505 fix(api): update config handling to only set base config if not provided by user

* fix(docker-deployment): replace console.log with print for metadata extraction

* Release v0.7.5: The Update

- Updated version to 0.7.5
- Added comprehensive demo and release notes
- Updated documentation

* refactor(release): remove memory management section for cleaner documentation. ref #1443

* feat(docs): add brand book and page copy functionality

- Add comprehensive brand book with color system, typography, components
- Add page copy dropdown with markdown copy/view functionality
- Update mkdocs.yml with new assets and branding navigation
- Use terminal-style ASCII icons and condensed menu design

* Update gitignore add local scripts folder

* fix: remove this import as it causes python to treat "json" as a variable in the except block

* fix: always return a list, even if we catch an exception

* feat(marketplace): Add Crawl4AI marketplace with secure configuration

- Implement marketplace frontend and admin dashboard
- Add FastAPI backend with environment-based configuration
- Use .env file for secrets management
- Include data generation scripts
- Add proper CORS configuration
- Remove hardcoded password from admin login
- Update gitignore for security

* fix(marketplace): Update URLs to use /marketplace path and relative API endpoints

- Change API_BASE to relative '/api' for production
- Move marketplace to /marketplace instead of /marketplace/frontend
- Update MkDocs navigation
- Fix logo path in marketplace index

* fix(docs): hide copy menu on non-markdown pages

* feat(marketplace): add sponsor logo uploads

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* feat(docs): add chatgpt quick link to page actions

* fix(marketplace): align admin api with backend endpoints

* fix(marketplace): isolate api under marketplace prefix

* fix(marketplace): resolve app detail page routing and styling issues

- Fixed JavaScript errors from missing HTML elements (install-code, usage-code, integration-code)
- Added missing CSS classes for tabs, overview layout, sidebar, and integration content
- Fixed tab navigation to display horizontally in single line
- Added proper padding to tab content sections (removed from container, added to content)
- Fixed tab selector from .nav-tab to .tab-btn to match HTML structure
- Added sidebar styling with stats grid and metadata display
- Improved responsive design with mobile-friendly tab scrolling
- Fixed code block positioning for copy buttons
- Removed margin from first headings to prevent extra spacing
- Added null checks for DOM elements in JavaScript to prevent errors

These changes resolve the routing issue where clicking on apps caused page redirects,
and fix the broken layout where CSS was not properly applied to the app detail page.

* fix(marketplace): prevent hero image overflow and secondary card stretching

- Fixed hero image to 200px height with min/max constraints
- Added object-fit: cover to hero-image img elements
- Changed secondary-featured align-items from stretch to flex-start
- Fixed secondary-card height to 118px (no flex: 1 stretching)
- Updated responsive grid layouts for wider screens
- Added flex: 1 to hero-content for better content distribution

These changes ensure a rigid, predictable layout that prevents:
1. Large images from pushing text content down
2. Single secondary cards from stretching to fill entire height

* feat: Add hooks utility for function-based hooks with Docker client integration. ref #1377

   Add hooks_to_string() utility function that converts Python function objects
   to string representations for the Docker API, enabling developers to write hooks
   as regular Python functions instead of strings.

   Core Changes:
   - New hooks_to_string() utility in crawl4ai/utils.py using inspect.getsource()
   - Docker client now accepts both function objects and strings for hooks
   - Automatic detection and conversion in Crawl4aiDockerClient._prepare_request()
   - New hooks and hooks_timeout parameters in client.crawl() method

   Documentation:
   - Docker client examples with function-based hooks (docs/examples/docker_client_hooks_example.py)
   - Updated main Docker deployment guide with comprehensive hooks section
   - Added unit tests for hooks utility (tests/docker/test_hooks_utility.py)

* feat: Add hooks utility for function-based hooks with Docker client integration. ref #1377

   Add hooks_to_string() utility function that converts Python function objects
   to string representations for the Docker API, enabling developers to write hooks
   as regular Python functions instead of strings.

   Core Changes:
   - New hooks_to_string() utility in crawl4ai/utils.py using inspect.getsource()
   - Docker client now accepts both function objects and strings for hooks
   - Automatic detection and conversion in Crawl4aiDockerClient._prepare_request()
   - New hooks and hooks_timeout parameters in client.crawl() method

   Documentation:
   - Docker client examples with function-based hooks (docs/examples/docker_client_hooks_example.py)
   - Updated main Docker deployment guide with comprehensive hooks section
   - Added unit tests for hooks utility (tests/docker/test_hooks_utility.py)

* fix(docs): clarify Docker Hooks System with function-based API in README

* docs: Add demonstration files for v0.7.5 release, showcasing the new Docker Hooks System and all other features.

* docs: Update 0.7.5 video walkthrough

* docs: add complete SDK reference documentation

Add comprehensive single-page SDK reference combining:
- Installation & Setup
- Quick Start
- Core API (AsyncWebCrawler, arun, arun_many, CrawlResult)
- Configuration (BrowserConfig, CrawlerConfig, Parameters)
- Crawling Patterns
- Content Processing (Markdown, Fit Markdown, Selection, Interaction, Link & Media)
- Extraction Strategies (LLM and No-LLM)
- Advanced Features (Session Management, Hooks & Auth)

Generated using scripts/generate_sdk_docs.py in ultra-dense mode
optimized for AI assistant consumption.

Stats: 23K words, 185 code blocks, 220KB

* feat: add AI assistant skill package for Crawl4AI

- Create comprehensive skill package for AI coding assistants
- Include complete SDK reference (23K words, v0.7.4)
- Add three extraction scripts (basic, batch, pipeline)
- Implement version tracking in skill and scripts
- Add prominent download section on homepage
- Place skill in docs/assets for web distribution

The skill enables AI assistants like Claude, Cursor, and Windsurf
to effectively use Crawl4AI with optimized workflows for markdown
generation and data extraction.

* fix: remove non-existent wiki link and clarify skill usage instructions

* fix: update Crawl4AI skill with corrected parameters and examples

- Fixed CrawlerConfig → CrawlerRunConfig throughout
- Fixed parameter names (timeout → page_timeout, store_html removed)
- Fixed schema format (selector → baseSelector)
- Corrected proxy configuration (in BrowserConfig, not CrawlerRunConfig)
- Fixed fit_markdown usage with content filters
- Added comprehensive references to docs/examples/ directory
- Created safe packaging script to avoid root directory pollution
- All scripts tested and verified working

* fix: thoroughly verify and fix all Crawl4AI skill examples

- Cross-checked every section against actual docs
- Fixed BM25ContentFilter parameters (user_query, bm25_threshold)
- Removed incorrect wait_for selector from basic example
- Added comprehensive test suite (4 test files)
- All examples now tested and verified working
- Tests validate: basic crawling, markdown generation, data extraction, advanced patterns
- Package size: 76.6 KB (includes tests for future validation)

* feat(ci): split release pipeline and add Docker caching

- Split release.yml into PyPI/GitHub release and Docker workflows
- Add GitHub Actions cache for Docker builds (10-15x faster rebuilds)
- Implement dual-trigger for docker-release.yml (auto + manual)
- Add comprehensive workflow documentation in .github/workflows/docs/
- Backup original workflow as release.yml.backup

* feat: add webhook notifications for crawl job completion

Implements webhook support for the crawl job API to eliminate polling requirements.

Changes:
- Added WebhookConfig and WebhookPayload schemas to schemas.py
- Created webhook.py with WebhookDeliveryService class
- Integrated webhook notifications in api.py handle_crawl_job
- Updated job.py CrawlJobPayload to accept webhook_config
- Added webhook configuration section to config.yml
- Included comprehensive usage examples in WEBHOOK_EXAMPLES.md

Features:
- Webhook notifications on job completion (success/failure)
- Configurable data inclusion in webhook payload
- Custom webhook headers support
- Global default webhook URL configuration
- Exponential backoff retry logic (5 attempts: 1s, 2s, 4s, 8s, 16s)
- 30-second timeout per webhook call

Usage:
POST /crawl/job with optional webhook_config:
- webhook_url: URL to receive notifications
- webhook_data_in_payload: include full results (default: false)
- webhook_headers: custom headers for authentication

Generated with Claude Code https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add webhook documentation to Docker README

Added comprehensive webhook section to README.md including:
- Overview of asynchronous job queue with webhooks
- Benefits and use cases
- Quick start examples
- Webhook authentication
- Global webhook configuration
- Job status polling alternative

Updated table of contents and summary to include webhook feature.
Maintains consistent tone and style with rest of README.

Generated with Claude Code https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add webhook example for Docker deployment

Added docker_webhook_example.py demonstrating:
- Submitting crawl jobs with webhook configuration
- Flask-based webhook receiver implementation
- Three usage patterns:
  1. Webhook notification only (fetch data separately)
  2. Webhook with full data in payload
  3. Traditional polling approach for comparison

Includes comprehensive comments explaining:
- Webhook payload structure
- Authentication headers setup
- Error handling
- Production deployment tips

Example is fully functional and ready to run with Flask installed.

Generated with Claude Code https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>

* test: add webhook implementation validation tests

Added comprehensive test suite to validate webhook implementation:
- Module import verification
- WebhookDeliveryService initialization
- Pydantic model validation (WebhookConfig)
- Payload construction logic
- Exponential backoff calculation
- API integration checks

All tests pass (6/6), confirming implementation is correct.

Generated with Claude Code https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>

* test: add comprehensive webhook feature test script

Added end-to-end test script that automates webhook feature testing:

Script Features (test_webhook_feature.sh):
- Automatic branch switching and dependency installation
- Redis and server startup/shutdown management
- Webhook receiver implementation
- Integration test for webhook notifications
- Comprehensive cleanup and error handling
- Returns to original branch after completion

Test Flow:
1. Fetch and checkout webhook feature branch
2. Activate venv and install dependencies
3. Start Redis and Crawl4AI server
4. Submit crawl job with webhook config
5. Verify webhook delivery and payload
6. Clean up all processes and return to original branch

Documentation:
- WEBHOOK_TEST_README.md with usage instructions
- Troubleshooting guide
- Exit codes and safety features

Usage: ./tests/test_webhook_feature.sh

Generated with Claude Code https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: properly serialize Pydantic HttpUrl in webhook config

Use model_dump(mode='json') instead of deprecated dict() method to ensure
Pydantic special types (HttpUrl, UUID, etc.) are properly serialized to
JSON-compatible native Python types.

This fixes webhook delivery failures caused by HttpUrl objects remaining
as Pydantic types in the webhook_config dict, which caused JSON
serialization errors and httpx request failures.

Also update mcp requirement to >=1.18.0 for compatibility.

* feat: add webhook support for /llm/job endpoint

Add comprehensive webhook notification support for the /llm/job endpoint,
following the same pattern as the existing /crawl/job implementation.

Changes:
- Add webhook_config field to LlmJobPayload model (job.py)
- Implement webhook notifications in process_llm_extraction() with 4
  notification points: success, provider validation failure, extraction
  failure, and general exceptions (api.py)
- Store webhook_config in Redis task data for job tracking
- Initialize WebhookDeliveryService with exponential backoff retry logic
Documentation:
- Add Example 6 to WEBHOOK_EXAMPLES.md showing LLM extraction with webhooks
- Update Flask webhook handler to support both crawl and llm_extraction tasks
- Add TypeScript client examples for LLM jobs
- Add comprehensive examples to docker_webhook_example.py with schema support
- Clarify data structure differences between webhook and API responses

Testing:
- Add test_llm_webhook_feature.py with 7 validation tests (all passing)
- Verify pattern consistency with /crawl/job implementation
- Add implementation guide (WEBHOOK_LLM_JOB_IMPLEMENTATION.md)

* fix: remove duplicate comma in webhook_config parameter

* fix: update Crawl4AI Docker container port from 11234 to 11235

* Release v0.7.6: The 0.7.6 Update

- Updated version to 0.7.6
- Added comprehensive demo and release notes
- Updated all documentation
- Update the veriosn in Dockerfile to 0.7.6

---------

Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
Co-authored-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
Co-authored-by: Nezar Ali <abu5sohaib@gmail.com>
Co-authored-by: Soham Kukreti <kukretisoham@gmail.com>
Co-authored-by: James T. Wood <jamesthomaswood@gmail.com>
Co-authored-by: AHMET YILMAZ <tawfik@kidocode.com>
Co-authored-by: nafeqq-1306 <nafiquee@yahoo.com>
Co-authored-by: unclecode <unclecode@kidocode.com>
Co-authored-by: Martin Sjöborg <martin.sjoborg@quartr.se>
Co-authored-by: Martin Sjöborg <martin@sjoborg.org>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Nasrin
2025-10-22 20:41:06 +08:00
committed by GitHub
parent fdbcddbf1a
commit 7cac008c10
127 changed files with 33817 additions and 514 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
docs/blog/release-v0.7.4.md
View File

@@ -10,7 +10,6 @@ Today I'm releasing Crawl4AI v0.7.4—the Intelligent Table Extraction & Perform
- **🚀 LLMTableExtraction**: Revolutionary table extraction with intelligent chunking for massive tables
- **⚡ Enhanced Concurrency**: True concurrency improvements for fast-completing tasks in batch operations
- **🧹 Memory Management Refactor**: Streamlined memory utilities and better resource management
- **🔧 Browser Manager Fixes**: Resolved race conditions in concurrent page creation
- **⌨️ Cross-Platform Browser Profiler**: Improved keyboard handling and quit mechanisms
- **🔗 Advanced URL Processing**: Better handling of raw URLs and base tag link resolution
@@ -158,40 +157,6 @@ async with AsyncWebCrawler() as crawler:
- **Monitoring Systems**: Faster health checks and status page monitoring
- **Data Aggregation**: Improved performance for real-time data collection
## 🧹 Memory Management Refactor: Cleaner Architecture
**The Problem:** Memory utilities were scattered and difficult to maintain, with potential import conflicts and unclear organization.
**My Solution:** I consolidated all memory-related utilities into the main `utils.py` module, creating a cleaner, more maintainable architecture.
### Improved Memory Handling
```python
# All memory utilities now consolidated
from crawl4ai.utils import get_true_memory_usage_percent, MemoryMonitor
# Enhanced memory monitoring
monitor = MemoryMonitor()
monitor.start_monitoring()
async with AsyncWebCrawler() as crawler:
# Memory-efficient batch processing
results = await crawler.arun_many(large_url_list)
# Get accurate memory metrics
memory_usage = get_true_memory_usage_percent()
memory_report = monitor.get_report()
print(f"Memory efficiency: {memory_report['efficiency']:.1f}%")
print(f"Peak usage: {memory_report['peak_mb']:.1f} MB")
```
**Expected Real-World Impact:**
- **Production Stability**: More reliable memory tracking and management
- **Code Maintainability**: Cleaner architecture for easier debugging
- **Import Clarity**: Resolved potential conflicts and import issues
- **Developer Experience**: Simpler API for memory monitoring
## 🔧 Critical Stability Fixes
### Browser Manager Race Condition Resolution

318
docs/blog/release-v0.7.5.md Normal file
View File

@@ -0,0 +1,318 @@
# 🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update
*September 29, 2025 • 8 min read*
---
Today I'm releasing Crawl4AI v0.7.5—focused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.
## 🎯 What's New at a Glance
- **Docker Hooks System**: Custom Python functions at key pipeline points with function-based API
- **Function-Based Hooks**: New `hooks_to_string()` utility with Docker client auto-conversion
- **Enhanced LLM Integration**: Custom providers with temperature control
- **HTTPS Preservation**: Secure internal link handling
- **Bug Fixes**: Resolved multiple community-reported issues
- **Improved Docker Error Handling**: Better debugging and reliability
## 🔧 Docker Hooks System: Pipeline Customization
Every scraping project needs custom logic—authentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.
### Real Example: Authentication & Performance
```python
import requests
# Real working hooks for httpbin.org
hooks_config = {
"on_page_context_created": """
async def hook(page, context, **kwargs):
print("Hook: Setting up page context")
# Block images to speed up crawling
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
print("Hook: Images blocked")
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
print("Hook: Before retrieving HTML")
# Scroll to bottom to load lazy content
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
print("Hook: Scrolled to bottom")
return page
""",
"before_goto": """
async def hook(page, context, url, **kwargs):
print(f"Hook: About to navigate to {url}")
# Add custom headers
await page.set_extra_http_headers({
'X-Test-Header': 'crawl4ai-hooks-test'
})
return page
"""
}
# Test with Docker API
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {
"code": hooks_config,
"timeout": 30
}
}
response = requests.post("http://localhost:11235/crawl", json=payload)
result = response.json()
if result.get('success'):
print("✅ Hooks executed successfully!")
print(f"Content length: {len(result.get('markdown', ''))} characters")
```
**Available Hook Points:**
- `on_browser_created`: Browser setup
- `on_page_context_created`: Page context configuration
- `before_goto`: Pre-navigation setup
- `after_goto`: Post-navigation processing
- `on_user_agent_updated`: User agent changes
- `on_execution_started`: Crawl initialization
- `before_retrieve_html`: Pre-extraction processing
- `before_return_html`: Final HTML processing
### Function-Based Hooks API
Writing hooks as strings works, but lacks IDE support and type checking. v0.7.5 introduces a function-based approach with automatic conversion!
**Option 1: Using the `hooks_to_string()` Utility**
```python
from crawl4ai import hooks_to_string
import requests
# Define hooks as regular Python functions (with full IDE support!)
async def on_page_context_created(page, context, **kwargs):
"""Block images to speed up crawling"""
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
await page.set_viewport_size({"width": 1920, "height": 1080})
return page
async def before_goto(page, context, url, **kwargs):
"""Add custom headers"""
await page.set_extra_http_headers({
'X-Crawl4AI': 'v0.7.5',
'X-Custom-Header': 'my-value'
})
return page
# Convert functions to strings
hooks_code = hooks_to_string({
"on_page_context_created": on_page_context_created,
"before_goto": before_goto
})
# Use with REST API
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {"code": hooks_code, "timeout": 30}
}
response = requests.post("http://localhost:11235/crawl", json=payload)
```
**Option 2: Docker Client with Automatic Conversion (Recommended!)**
```python
from crawl4ai.docker_client import Crawl4aiDockerClient
# Define hooks as functions (same as above)
async def on_page_context_created(page, context, **kwargs):
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
return page
async def before_retrieve_html(page, context, **kwargs):
# Scroll to load lazy content
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
return page
# Use Docker client - conversion happens automatically!
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
results = await client.crawl(
urls=["https://httpbin.org/html"],
hooks={
"on_page_context_created": on_page_context_created,
"before_retrieve_html": before_retrieve_html
},
hooks_timeout=30
)
if results and results.success:
print(f"✅ Hooks executed! HTML length: {len(results.html)}")
```
**Benefits of Function-Based Hooks:**
- ✅ Full IDE support (autocomplete, syntax highlighting)
- ✅ Type checking and linting
- ✅ Easier to test and debug
- ✅ Reusable across projects
- ✅ Automatic conversion in Docker client
- ✅ No breaking changes - string hooks still work!
## 🤖 Enhanced LLM Integration
Enhanced LLM integration with custom providers, temperature control, and base URL configuration.
### Multi-Provider Support
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
# Test with different providers
async def test_llm_providers():
# OpenAI with custom temperature
openai_strategy = LLMExtractionStrategy(
provider="gemini/gemini-2.5-flash-lite",
api_token="your-api-token",
temperature=0.7, # New in v0.7.5
instruction="Summarize this page in one sentence"
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://example.com",
config=CrawlerRunConfig(extraction_strategy=openai_strategy)
)
if result.success:
print("✅ LLM extraction completed")
print(result.extracted_content)
# Docker API with enhanced LLM config
llm_payload = {
"url": "https://example.com",
"f": "llm",
"q": "Summarize this page in one sentence.",
"provider": "gemini/gemini-2.5-flash-lite",
"temperature": 0.7
}
response = requests.post("http://localhost:11235/md", json=llm_payload)
```
**New Features:**
- Custom `temperature` parameter for creativity control
- `base_url` for custom API endpoints
- Multi-provider environment variable support
- Docker API integration
## 🔒 HTTPS Preservation
**The Problem:** Modern web apps require HTTPS everywhere. When crawlers downgrade internal links from HTTPS to HTTP, authentication breaks and security warnings appear.
**Solution:** HTTPS preservation maintains secure protocols throughout crawling.
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy
async def test_https_preservation():
# Enable HTTPS preservation
url_filter = URLPatternFilter(
patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
)
config = CrawlerRunConfig(
exclude_external_links=True,
preserve_https_for_internal_links=True, # New in v0.7.5
deep_crawl_strategy=BFSDeepCrawlStrategy(
max_depth=2,
max_pages=5,
filter_chain=FilterChain([url_filter])
)
)
async with AsyncWebCrawler() as crawler:
async for result in await crawler.arun(
url="https://quotes.toscrape.com",
config=config
):
# All internal links maintain HTTPS
internal_links = [link['href'] for link in result.links['internal']]
https_links = [link for link in internal_links if link.startswith('https://')]
print(f"HTTPS links preserved: {len(https_links)}/{len(internal_links)}")
for link in https_links[:3]:
print(f" → {link}")
```
## 🛠️ Bug Fixes and Improvements
### Major Fixes
- **URL Processing**: Fixed '+' sign preservation in query parameters (#1332)
- **Proxy Configuration**: Enhanced proxy string parsing (old `proxy` parameter deprecated)
- **Docker Error Handling**: Comprehensive error messages with status codes
- **Memory Management**: Fixed leaks in long-running sessions
- **JWT Authentication**: Fixed Docker JWT validation issues (#1442)
- **Playwright Stealth**: Fixed stealth features for Playwright integration (#1481)
- **API Configuration**: Fixed config handling to prevent overriding user-provided settings (#1505)
- **Docker Filter Serialization**: Resolved JSON encoding errors in deep crawl strategy (#1419)
- **LLM Provider Support**: Fixed custom LLM provider integration for adaptive crawler (#1291)
- **Performance Issues**: Resolved backoff strategy failures and timeout handling (#989)
### Community-Reported Issues Fixed
This release addresses multiple issues reported by the community through GitHub issues and Discord discussions:
- Fixed browser configuration reference errors
- Resolved dependency conflicts with cssselect
- Improved error messaging for failed authentications
- Enhanced compatibility with various proxy configurations
- Fixed edge cases in URL normalization
### Configuration Updates
```python
# Old proxy config (deprecated)
# browser_config = BrowserConfig(proxy="http://proxy:8080")
# New enhanced proxy config
browser_config = BrowserConfig(
proxy_config={
"server": "http://proxy:8080",
"username": "optional-user",
"password": "optional-pass"
}
)
```
## 🔄 Breaking Changes
1. **Python 3.10+ Required**: Upgrade from Python 3.9
2. **Proxy Parameter Deprecated**: Use new `proxy_config` structure
3. **New Dependency**: Added `cssselect` for better CSS handling
## 🚀 Get Started
```bash
# Install latest version
pip install crawl4ai==0.7.5
# Docker deployment
docker pull unclecode/crawl4ai:latest
docker run -p 11235:11235 unclecode/crawl4ai:latest
```
**Try the Demo:**
```bash
# Run working examples
python docs/releases_review/demo_v0.7.5.py
```
**Resources:**
- 📖 Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
- 🐙 GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
- 💬 Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
- 🐦 Twitter: [@unclecode](https://x.com/unclecode)
Happy crawling! 🕷️

314
docs/blog/release-v0.7.6.md Normal file
View File

@@ -0,0 +1,314 @@
# Crawl4AI v0.7.6 Release Notes
*Release Date: October 22, 2025*
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
## 🎯 What's New
### Webhook Support for Docker Job Queue API
The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!
**Key Capabilities:**
- ✅ **Universal Webhook Support**: Both `/crawl/job` and `/llm/job` endpoints now support webhooks
- ✅ **Flexible Delivery Modes**: Choose notification-only or include full data in the webhook payload
- ✅ **Reliable Delivery**: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
- ✅ **Custom Authentication**: Add custom headers for webhook authentication
- ✅ **Global Configuration**: Set default webhook URL in `config.yml` for all jobs
- ✅ **Task Type Identification**: Distinguish between `crawl` and `llm_extraction` tasks
### How It Works
Instead of constantly checking job status:
**OLD WAY (Polling):**
```python
# Submit job
response = requests.post("http://localhost:11235/crawl/job", json=payload)
task_id = response.json()['task_id']
# Poll until complete
while True:
status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
if status.json()['status'] == 'completed':
break
time.sleep(5) # Wait and try again
```
**NEW WAY (Webhooks):**
```python
# Submit job with webhook
payload = {
"urls": ["https://example.com"],
"webhook_config": {
"webhook_url": "https://myapp.com/webhook",
"webhook_data_in_payload": True
}
}
response = requests.post("http://localhost:11235/crawl/job", json=payload)
# Done! Webhook will notify you when complete
# Your webhook handler receives the results automatically
```
### Crawl Job Webhooks
```bash
curl -X POST http://localhost:11235/crawl/job \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"browser_config": {"headless": true},
"crawler_config": {"cache_mode": "bypass"},
"webhook_config": {
"webhook_url": "https://myapp.com/webhooks/crawl-complete",
"webhook_data_in_payload": false,
"webhook_headers": {
"X-Webhook-Secret": "your-secret-token"
}
}
}'
```
### LLM Extraction Job Webhooks (NEW!)
```bash
curl -X POST http://localhost:11235/llm/job \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/article",
"q": "Extract the article title, author, and publication date",
"schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
"provider": "openai/gpt-4o-mini",
"webhook_config": {
"webhook_url": "https://myapp.com/webhooks/llm-complete",
"webhook_data_in_payload": true
}
}'
```
### Webhook Payload Structure
**Success (with data):**
```json
{
"task_id": "llm_1698765432",
"task_type": "llm_extraction",
"status": "completed",
"timestamp": "2025-10-22T10:30:00.000000+00:00",
"urls": ["https://example.com/article"],
"data": {
"extracted_content": {
"title": "Understanding Web Scraping",
"author": "John Doe",
"date": "2025-10-22"
}
}
}
```
**Failure:**
```json
{
"task_id": "crawl_abc123",
"task_type": "crawl",
"status": "failed",
"timestamp": "2025-10-22T10:30:00.000000+00:00",
"urls": ["https://example.com"],
"error": "Connection timeout after 30s"
}
```
### Simple Webhook Handler Example
```python
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route('/webhook', methods=['POST'])
def handle_webhook():
payload = request.json
task_id = payload['task_id']
task_type = payload['task_type']
status = payload['status']
if status == 'completed':
if 'data' in payload:
# Process data directly
data = payload['data']
else:
# Fetch from API
endpoint = 'crawl' if task_type == 'crawl' else 'llm'
response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
data = response.json()
# Your business logic here
print(f"Job {task_id} completed!")
elif status == 'failed':
error = payload.get('error', 'Unknown error')
print(f"Job {task_id} failed: {error}")
return jsonify({"status": "received"}), 200
app.run(port=8080)
```
## 📊 Performance Improvements
- **Reduced Server Load**: Eliminates constant polling requests
- **Lower Latency**: Instant notification vs. polling interval delay
- **Better Resource Usage**: Frees up client connections while jobs run in background
- **Scalable Architecture**: Handles high-volume crawling workflows efficiently
## 🐛 Bug Fixes
- Fixed webhook configuration serialization for Pydantic HttpUrl fields
- Improved error handling in webhook delivery service
- Enhanced Redis task storage for webhook config persistence
## 🌍 Expected Real-World Impact
### For Web Scraping Workflows
- **Reduced Costs**: Less API calls = lower bandwidth and server costs
- **Better UX**: Instant notifications improve user experience
- **Scalability**: Handle 100s of concurrent jobs without polling overhead
### For LLM Extraction Pipelines
- **Async Processing**: Submit LLM extraction jobs and move on
- **Batch Processing**: Queue multiple extractions, get notified as they complete
- **Integration**: Easy integration with workflow automation tools (Zapier, n8n, etc.)
### For Microservices
- **Event-Driven**: Perfect for event-driven microservice architectures
- **Decoupling**: Decouple job submission from result processing
- **Reliability**: Automatic retries ensure webhooks are delivered
## 🔄 Breaking Changes
**None!** This release is fully backward compatible.
- Webhook configuration is optional
- Existing code continues to work without modification
- Polling is still supported for jobs without webhook config
## 📚 Documentation
### New Documentation
- **[WEBHOOK_EXAMPLES.md](../deploy/docker/WEBHOOK_EXAMPLES.md)** - Comprehensive webhook usage guide
- **[docker_webhook_example.py](../docs/examples/docker_webhook_example.py)** - Working code examples
### Updated Documentation
- **[Docker README](../deploy/docker/README.md)** - Added webhook sections
- API documentation with webhook examples
## 🛠️ Migration Guide
No migration needed! Webhooks are opt-in:
1. **To use webhooks**: Add `webhook_config` to your job payload
2. **To keep polling**: Continue using your existing code
### Quick Start
```python
# Just add webhook_config to your existing payload
payload = {
# Your existing configuration
"urls": ["https://example.com"],
"browser_config": {...},
"crawler_config": {...},
# NEW: Add webhook configuration
"webhook_config": {
"webhook_url": "https://myapp.com/webhook",
"webhook_data_in_payload": True
}
}
```
## 🔧 Configuration
### Global Webhook Configuration (config.yml)
```yaml
webhooks:
enabled: true
default_url: "https://myapp.com/webhooks/default" # Optional
data_in_payload: false
retry:
max_attempts: 5
initial_delay_ms: 1000
max_delay_ms: 32000
timeout_ms: 30000
headers:
User-Agent: "Crawl4AI-Webhook/1.0"
```
## 🚀 Upgrade Instructions
### Docker
```bash
# Pull the latest image
docker pull unclecode/crawl4ai:0.7.6
# Or use latest tag
docker pull unclecode/crawl4ai:latest
# Run with webhook support
docker run -d \
-p 11235:11235 \
--env-file .llm.env \
--name crawl4ai \
unclecode/crawl4ai:0.7.6
```
### Python Package
```bash
pip install --upgrade crawl4ai
```
## 💡 Pro Tips
1. **Use notification-only mode** for large results - fetch data separately to avoid large webhook payloads
2. **Set custom headers** for webhook authentication and request tracking
3. **Configure global default webhook** for consistent handling across all jobs
4. **Implement idempotent webhook handlers** - same webhook may be delivered multiple times on retry
5. **Use structured schemas** with LLM extraction for predictable webhook data
## 🎬 Demo
Try the release demo:
```bash
python docs/releases_review/demo_v0.7.6.py
```
This comprehensive demo showcases:
- Crawl job webhooks (notification-only and with data)
- LLM extraction webhooks (with JSON schema support)
- Custom headers for authentication
- Webhook retry mechanism
- Real-time webhook receiver
## 🙏 Acknowledgments
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
## 📞 Support
- **Documentation**: https://docs.crawl4ai.com
- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
- **Discord**: https://discord.gg/crawl4ai
---
**Happy crawling with webhooks!** 🕷️🪝
*- unclecode*

154
docs/examples/adaptive_crawling/llm_config_example.py Normal file
View File

@@ -0,0 +1,154 @@
import asyncio
import os
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig, LLMConfig
async def test_configuration(name: str, config: AdaptiveConfig, url: str, query: str):
"""Test a specific configuration"""
print(f"\n{'='*60}")
print(f"Configuration: {name}")
print(f"{'='*60}")
async with AsyncWebCrawler(verbose=False) as crawler:
adaptive = AdaptiveCrawler(crawler, config)
result = await adaptive.digest(start_url=url, query=query)
print("\n" + "="*50)
print("CRAWL STATISTICS")
print("="*50)
adaptive.print_stats(detailed=False)
# Get the most relevant content found
print("\n" + "="*50)
print("MOST RELEVANT PAGES")
print("="*50)
relevant_pages = adaptive.get_relevant_content(top_k=5)
for i, page in enumerate(relevant_pages, 1):
print(f"\n{i}. {page['url']}")
print(f" Relevance Score: {page['score']:.2%}")
# Show a snippet of the content
content = page['content'] or ""
if content:
snippet = content[:200].replace('\n', ' ')
if len(content) > 200:
snippet += "..."
print(f" Preview: {snippet}")
print(f"\n{'='*50}")
print(f"Pages crawled: {len(result.crawled_urls)}")
print(f"Final confidence: {adaptive.confidence:.1%}")
print(f"Stopped reason: {result.metrics.get('stopped_reason', 'max_pages')}")
if result.metrics.get('is_irrelevant', False):
print("⚠️ Query detected as irrelevant!")
return result
async def llm_embedding():
"""Demonstrate various embedding configurations"""
print("EMBEDDING STRATEGY CONFIGURATION EXAMPLES")
print("=" * 60)
# Base URL and query for testing
test_url = "https://docs.python.org/3/library/asyncio.html"
openai_llm_config = LLMConfig(
provider='openai/text-embedding-3-small',
api_token=os.getenv('OPENAI_API_KEY'),
temperature=0.7,
max_tokens=2000
)
config_openai = AdaptiveConfig(
strategy="embedding",
max_pages=10,
# Use OpenAI embeddings
embedding_llm_config=openai_llm_config,
# embedding_llm_config={
# 'provider': 'openai/text-embedding-3-small',
# 'api_token': os.getenv('OPENAI_API_KEY')
# },
# OpenAI embeddings are high quality, can be stricter
embedding_k_exp=4.0,
n_query_variations=12
)
await test_configuration(
"OpenAI Embeddings",
config_openai,
test_url,
# "event-driven architecture patterns"
"async await context managers coroutines"
)
return
async def basic_adaptive_crawling():
"""Basic adaptive crawling example"""
# Initialize the crawler
async with AsyncWebCrawler(verbose=True) as crawler:
# Create an adaptive crawler with default settings (statistical strategy)
adaptive = AdaptiveCrawler(crawler)
# Note: You can also use embedding strategy for semantic understanding:
# from crawl4ai import AdaptiveConfig
# config = AdaptiveConfig(strategy="embedding")
# adaptive = AdaptiveCrawler(crawler, config)
# Start adaptive crawling
print("Starting adaptive crawl for Python async programming information...")
result = await adaptive.digest(
start_url="https://docs.python.org/3/library/asyncio.html",
query="async await context managers coroutines"
)
# Display crawl statistics
print("\n" + "="*50)
print("CRAWL STATISTICS")
print("="*50)
adaptive.print_stats(detailed=False)
# Get the most relevant content found
print("\n" + "="*50)
print("MOST RELEVANT PAGES")
print("="*50)
relevant_pages = adaptive.get_relevant_content(top_k=5)
for i, page in enumerate(relevant_pages, 1):
print(f"\n{i}. {page['url']}")
print(f" Relevance Score: {page['score']:.2%}")
# Show a snippet of the content
content = page['content'] or ""
if content:
snippet = content[:200].replace('\n', ' ')
if len(content) > 200:
snippet += "..."
print(f" Preview: {snippet}")
# Show final confidence
print(f"\n{'='*50}")
print(f"Final Confidence: {adaptive.confidence:.2%}")
print(f"Total Pages Crawled: {len(result.crawled_urls)}")
print(f"Knowledge Base Size: {len(adaptive.state.knowledge_base)} documents")
if adaptive.confidence >= 0.8:
print("✓ High confidence - can answer detailed questions about async Python")
elif adaptive.confidence >= 0.6:
print("~ Moderate confidence - can answer basic questions")
else:
print("✗ Low confidence - need more information")
if __name__ == "__main__":
asyncio.run(llm_embedding())
# asyncio.run(basic_adaptive_crawling())

522
docs/examples/docker_client_hooks_example.py Normal file
View File

@@ -0,0 +1,522 @@
#!/usr/bin/env python3
"""
Comprehensive hooks examples using Docker Client with function objects.
This approach is recommended because:
- Write hooks as regular Python functions
- Full IDE support (autocomplete, type checking)
- Automatic conversion to API format
- Reusable and testable code
- Clean, readable syntax
"""
import asyncio
from crawl4ai import Crawl4aiDockerClient
# API_BASE_URL = "http://localhost:11235"
API_BASE_URL = "http://localhost:11234"
# ============================================================================
# Hook Function Definitions
# ============================================================================
# --- All Hooks Demo ---
async def browser_created_hook(browser, **kwargs):
"""Called after browser is created"""
print("[HOOK] Browser created and ready")
return browser
async def page_context_hook(page, context, **kwargs):
"""Setup page environment"""
print("[HOOK] Setting up page environment")
# Set viewport
await page.set_viewport_size({"width": 1920, "height": 1080})
# Add cookies
await context.add_cookies([{
"name": "test_session",
"value": "abc123xyz",
"domain": ".httpbin.org",
"path": "/"
}])
# Block resources
await context.route("**/*.{png,jpg,jpeg,gif}", lambda route: route.abort())
await context.route("**/analytics/*", lambda route: route.abort())
print("[HOOK] Environment configured")
return page
async def user_agent_hook(page, context, user_agent, **kwargs):
"""Called when user agent is updated"""
print(f"[HOOK] User agent: {user_agent[:50]}...")
return page
async def before_goto_hook(page, context, url, **kwargs):
"""Called before navigating to URL"""
print(f"[HOOK] Navigating to: {url}")
await page.set_extra_http_headers({
"X-Custom-Header": "crawl4ai-test",
"Accept-Language": "en-US"
})
return page
async def after_goto_hook(page, context, url, response, **kwargs):
"""Called after page loads"""
print(f"[HOOK] Page loaded: {url}")
await page.wait_for_timeout(1000)
try:
await page.wait_for_selector("body", timeout=2000)
print("[HOOK] Body element ready")
except:
print("[HOOK] Timeout, continuing")
return page
async def execution_started_hook(page, context, **kwargs):
"""Called when custom JS execution starts"""
print("[HOOK] JS execution started")
await page.evaluate("console.log('[HOOK] Custom JS');")
return page
async def before_retrieve_hook(page, context, **kwargs):
"""Called before retrieving HTML"""
print("[HOOK] Preparing HTML retrieval")
# Scroll for lazy content
await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
await page.wait_for_timeout(500)
await page.evaluate("window.scrollTo(0, 0);")
print("[HOOK] Scrolling complete")
return page
async def before_return_hook(page, context, html, **kwargs):
"""Called before returning HTML"""
print(f"[HOOK] HTML ready: {len(html)} chars")
metrics = await page.evaluate('''() => ({
images: document.images.length,
links: document.links.length,
scripts: document.scripts.length
})''')
print(f"[HOOK] Metrics - Images: {metrics['images']}, Links: {metrics['links']}")
return page
# --- Authentication Hooks ---
async def auth_context_hook(page, context, **kwargs):
"""Setup authentication context"""
print("[HOOK] Setting up authentication")
# Add auth cookies
await context.add_cookies([{
"name": "auth_token",
"value": "fake_jwt_token",
"domain": ".httpbin.org",
"path": "/",
"httpOnly": True
}])
# Set localStorage
await page.evaluate('''
localStorage.setItem('user_id', '12345');
localStorage.setItem('auth_time', new Date().toISOString());
''')
print("[HOOK] Auth context ready")
return page
async def auth_headers_hook(page, context, url, **kwargs):
"""Add authentication headers"""
print(f"[HOOK] Adding auth headers for {url}")
import base64
credentials = base64.b64encode(b"user:passwd").decode('ascii')
await page.set_extra_http_headers({
'Authorization': f'Basic {credentials}',
'X-API-Key': 'test-key-123'
})
return page
# --- Performance Optimization Hooks ---
async def performance_hook(page, context, **kwargs):
"""Optimize page for performance"""
print("[HOOK] Optimizing for performance")
# Block resource-heavy content
await context.route("**/*.{png,jpg,jpeg,gif,webp,svg}", lambda r: r.abort())
await context.route("**/*.{woff,woff2,ttf}", lambda r: r.abort())
await context.route("**/*.{mp4,webm,ogg}", lambda r: r.abort())
await context.route("**/googletagmanager.com/*", lambda r: r.abort())
await context.route("**/google-analytics.com/*", lambda r: r.abort())
await context.route("**/facebook.com/*", lambda r: r.abort())
# Disable animations
await page.add_style_tag(content='''
*, *::before, *::after {
animation-duration: 0s !important;
transition-duration: 0s !important;
}
''')
print("[HOOK] Optimizations applied")
return page
async def cleanup_hook(page, context, **kwargs):
"""Clean page before extraction"""
print("[HOOK] Cleaning page")
await page.evaluate('''() => {
const selectors = [
'.ad', '.ads', '.advertisement',
'.popup', '.modal', '.overlay',
'.cookie-banner', '.newsletter'
];
selectors.forEach(sel => {
document.querySelectorAll(sel).forEach(el => el.remove());
});
document.querySelectorAll('script, style').forEach(el => el.remove());
}''')
print("[HOOK] Page cleaned")
return page
# --- Content Extraction Hooks ---
async def wait_dynamic_content_hook(page, context, url, response, **kwargs):
"""Wait for dynamic content to load"""
print(f"[HOOK] Waiting for dynamic content on {url}")
await page.wait_for_timeout(2000)
# Click "Load More" if exists
try:
load_more = await page.query_selector('[class*="load-more"], button:has-text("Load More")')
if load_more:
await load_more.click()
await page.wait_for_timeout(1000)
print("[HOOK] Clicked 'Load More'")
except:
pass
return page
async def extract_metadata_hook(page, context, **kwargs):
"""Extract page metadata"""
print("[HOOK] Extracting metadata")
metadata = await page.evaluate('''() => {
const getMeta = (name) => {
const el = document.querySelector(`meta[name="${name}"], meta[property="${name}"]`);
return el ? el.getAttribute('content') : null;
};
return {
title: document.title,
description: getMeta('description'),
author: getMeta('author'),
keywords: getMeta('keywords'),
};
}''')
print(f"[HOOK] Metadata: {metadata}")
# Infinite scroll
for i in range(3):
await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
await page.wait_for_timeout(1000)
print(f"[HOOK] Scroll {i+1}/3")
return page
# --- Multi-URL Hooks ---
async def url_specific_hook(page, context, url, **kwargs):
"""Apply URL-specific logic"""
print(f"[HOOK] Processing URL: {url}")
# URL-specific headers
if 'html' in url:
await page.set_extra_http_headers({"X-Type": "HTML"})
elif 'json' in url:
await page.set_extra_http_headers({"X-Type": "JSON"})
return page
async def track_progress_hook(page, context, url, response, **kwargs):
"""Track crawl progress"""
status = response.status if response else 'unknown'
print(f"[HOOK] Loaded {url} - Status: {status}")
return page
# ============================================================================
# Test Functions
# ============================================================================
async def test_all_hooks_comprehensive():
"""Test all 8 hook types"""
print("=" * 70)
print("Test 1: All Hooks Comprehensive Demo (Docker Client)")
print("=" * 70)
async with Crawl4aiDockerClient(base_url=API_BASE_URL, verbose=False) as client:
print("\nCrawling with all 8 hooks...")
# Define hooks with function objects
hooks = {
"on_browser_created": browser_created_hook,
"on_page_context_created": page_context_hook,
"on_user_agent_updated": user_agent_hook,
"before_goto": before_goto_hook,
"after_goto": after_goto_hook,
"on_execution_started": execution_started_hook,
"before_retrieve_html": before_retrieve_hook,
"before_return_html": before_return_hook
}
result = await client.crawl(
["https://httpbin.org/html"],
hooks=hooks,
hooks_timeout=30
)
print("\n✅ Success!")
print(f" URL: {result.url}")
print(f" Success: {result.success}")
print(f" HTML: {len(result.html)} chars")
async def test_authentication_workflow():
"""Test authentication with hooks"""
print("\n" + "=" * 70)
print("Test 2: Authentication Workflow (Docker Client)")
print("=" * 70)
async with Crawl4aiDockerClient(base_url=API_BASE_URL, verbose=False) as client:
print("\nTesting authentication...")
hooks = {
"on_page_context_created": auth_context_hook,
"before_goto": auth_headers_hook
}
result = await client.crawl(
["https://httpbin.org/basic-auth/user/passwd"],
hooks=hooks,
hooks_timeout=15
)
print("\n✅ Authentication completed")
if result.success:
if '"authenticated"' in result.html and 'true' in result.html:
print(" ✅ Basic auth successful!")
else:
print(" ⚠️ Auth status unclear")
else:
print(f" ❌ Failed: {result.error_message}")
async def test_performance_optimization():
"""Test performance optimization"""
print("\n" + "=" * 70)
print("Test 3: Performance Optimization (Docker Client)")
print("=" * 70)
async with Crawl4aiDockerClient(base_url=API_BASE_URL, verbose=False) as client:
print("\nTesting performance hooks...")
hooks = {
"on_page_context_created": performance_hook,
"before_retrieve_html": cleanup_hook
}
result = await client.crawl(
["https://httpbin.org/html"],
hooks=hooks,
hooks_timeout=10
)
print("\n✅ Optimization completed")
print(f" HTML size: {len(result.html):,} chars")
print(" Resources blocked, ads removed")
async def test_content_extraction():
"""Test content extraction"""
print("\n" + "=" * 70)
print("Test 4: Content Extraction (Docker Client)")
print("=" * 70)
async with Crawl4aiDockerClient(base_url=API_BASE_URL, verbose=False) as client:
print("\nTesting extraction hooks...")
hooks = {
"after_goto": wait_dynamic_content_hook,
"before_retrieve_html": extract_metadata_hook
}
result = await client.crawl(
["https://www.kidocode.com/"],
hooks=hooks,
hooks_timeout=20
)
print("\n✅ Extraction completed")
print(f" URL: {result.url}")
print(f" Success: {result.success}")
print(f" Metadata: {result.metadata}")
async def test_multi_url_crawl():
"""Test hooks with multiple URLs"""
print("\n" + "=" * 70)
print("Test 5: Multi-URL Crawl (Docker Client)")
print("=" * 70)
async with Crawl4aiDockerClient(base_url=API_BASE_URL, verbose=False) as client:
print("\nCrawling multiple URLs...")
hooks = {
"before_goto": url_specific_hook,
"after_goto": track_progress_hook
}
results = await client.crawl(
[
"https://httpbin.org/html",
"https://httpbin.org/json",
"https://httpbin.org/xml"
],
hooks=hooks,
hooks_timeout=15
)
print("\n✅ Multi-URL crawl completed")
print(f"\n Crawled {len(results)} URLs:")
for i, result in enumerate(results, 1):
status = "✅" if result.success else "❌"
print(f" {status} {i}. {result.url}")
async def test_reusable_hook_library():
"""Test using reusable hook library"""
print("\n" + "=" * 70)
print("Test 6: Reusable Hook Library (Docker Client)")
print("=" * 70)
# Create a library of reusable hooks
class HookLibrary:
@staticmethod
async def block_images(page, context, **kwargs):
"""Block all images"""
await context.route("**/*.{png,jpg,jpeg,gif}", lambda r: r.abort())
print("[LIBRARY] Images blocked")
return page
@staticmethod
async def block_analytics(page, context, **kwargs):
"""Block analytics"""
await context.route("**/analytics/*", lambda r: r.abort())
await context.route("**/google-analytics.com/*", lambda r: r.abort())
print("[LIBRARY] Analytics blocked")
return page
@staticmethod
async def scroll_infinite(page, context, **kwargs):
"""Handle infinite scroll"""
for i in range(5):
prev = await page.evaluate("document.body.scrollHeight")
await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
await page.wait_for_timeout(1000)
curr = await page.evaluate("document.body.scrollHeight")
if curr == prev:
break
print("[LIBRARY] Infinite scroll complete")
return page
async with Crawl4aiDockerClient(base_url=API_BASE_URL, verbose=False) as client:
print("\nUsing hook library...")
hooks = {
"on_page_context_created": HookLibrary.block_images,
"before_retrieve_html": HookLibrary.scroll_infinite
}
result = await client.crawl(
["https://www.kidocode.com/"],
hooks=hooks,
hooks_timeout=20
)
print("\n✅ Library hooks completed")
print(f" Success: {result.success}")
# ============================================================================
# Main
# ============================================================================
async def main():
"""Run all Docker client hook examples"""
print("🔧 Crawl4AI Docker Client - Hooks Examples (Function-Based)")
print("Using Python function objects with automatic conversion")
print("=" * 70)
tests = [
("All Hooks Demo", test_all_hooks_comprehensive),
("Authentication", test_authentication_workflow),
("Performance", test_performance_optimization),
("Extraction", test_content_extraction),
("Multi-URL", test_multi_url_crawl),
("Hook Library", test_reusable_hook_library)
]
for i, (name, test_func) in enumerate(tests, 1):
try:
await test_func()
print(f"\n✅ Test {i}/{len(tests)}: {name} completed\n")
except Exception as e:
print(f"\n❌ Test {i}/{len(tests)}: {name} failed: {e}\n")
import traceback
traceback.print_exc()
print("=" * 70)
print("🎉 All Docker client hook examples completed!")
print("\n💡 Key Benefits of Function-Based Hooks:")
print(" • Write as regular Python functions")
print(" • Full IDE support (autocomplete, types)")
print(" • Automatic conversion to API format")
print(" • Reusable across projects")
print(" • Clean, readable code")
print(" • Easy to test and debug")
print("=" * 70)
if __name__ == "__main__":
asyncio.run(main())

513
docs/examples/docker_hooks_examples.py Normal file
View File

@@ -0,0 +1,513 @@
#!/usr/bin/env python3
"""
Comprehensive test demonstrating all hook types from hooks_example.py
adapted for the Docker API with real URLs
"""
import requests
import json
import time
from typing import Dict, Any
# API_BASE_URL = "http://localhost:11234"
API_BASE_URL = "http://localhost:11235"
def test_all_hooks_demo():
"""Demonstrate all 8 hook types with practical examples"""
print("=" * 70)
print("Testing: All Hooks Comprehensive Demo")
print("=" * 70)
hooks_code = {
"on_browser_created": """
async def hook(browser, **kwargs):
# Hook called after browser is created
print("[HOOK] on_browser_created - Browser is ready!")
# Browser-level configurations would go here
return browser
""",
"on_page_context_created": """
async def hook(page, context, **kwargs):
# Hook called after a new page and context are created
print("[HOOK] on_page_context_created - New page created!")
# Set viewport size for consistent rendering
await page.set_viewport_size({"width": 1920, "height": 1080})
# Add cookies for the session (using httpbin.org domain)
await context.add_cookies([
{
"name": "test_session",
"value": "abc123xyz",
"domain": ".httpbin.org",
"path": "/",
"httpOnly": True,
"secure": True
}
])
# Block ads and tracking scripts to speed up crawling
await context.route("**/*.{png,jpg,jpeg,gif,webp,svg}", lambda route: route.abort())
await context.route("**/analytics/*", lambda route: route.abort())
await context.route("**/ads/*", lambda route: route.abort())
print("[HOOK] Viewport set, cookies added, and ads blocked")
return page
""",
"on_user_agent_updated": """
async def hook(page, context, user_agent, **kwargs):
# Hook called when user agent is updated
print(f"[HOOK] on_user_agent_updated - User agent: {user_agent[:50]}...")
return page
""",
"before_goto": """
async def hook(page, context, url, **kwargs):
# Hook called before navigating to each URL
print(f"[HOOK] before_goto - About to visit: {url}")
# Add custom headers for the request
await page.set_extra_http_headers({
"X-Custom-Header": "crawl4ai-test",
"Accept-Language": "en-US,en;q=0.9",
"DNT": "1"
})
return page
""",
"after_goto": """
async def hook(page, context, url, response, **kwargs):
# Hook called after navigating to each URL
print(f"[HOOK] after_goto - Successfully loaded: {url}")
# Wait a moment for dynamic content to load
await page.wait_for_timeout(1000)
# Check if specific elements exist (with error handling)
try:
# For httpbin.org, wait for body element
await page.wait_for_selector("body", timeout=2000)
print("[HOOK] Body element found and loaded")
except:
print("[HOOK] Timeout waiting for body, continuing anyway")
return page
""",
"on_execution_started": """
async def hook(page, context, **kwargs):
# Hook called after custom JavaScript execution
print("[HOOK] on_execution_started - Custom JS executed!")
# You could inject additional JavaScript here if needed
await page.evaluate("console.log('[INJECTED] Hook JS running');")
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
# Hook called before retrieving the HTML content
print("[HOOK] before_retrieve_html - Preparing to get HTML")
# Scroll to bottom to trigger lazy loading
await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
await page.wait_for_timeout(500)
# Scroll back to top
await page.evaluate("window.scrollTo(0, 0);")
await page.wait_for_timeout(500)
# One more scroll to middle for good measure
await page.evaluate("window.scrollTo(0, document.body.scrollHeight / 2);")
print("[HOOK] Scrolling completed for lazy-loaded content")
return page
""",
"before_return_html": """
async def hook(page, context, html, **kwargs):
# Hook called before returning the HTML content
print(f"[HOOK] before_return_html - HTML length: {len(html)} characters")
# Log some page metrics
metrics = await page.evaluate('''() => {
return {
images: document.images.length,
links: document.links.length,
scripts: document.scripts.length
}
}''')
print(f"[HOOK] Page metrics - Images: {metrics['images']}, Links: {metrics['links']}, Scripts: {metrics['scripts']}")
return page
"""
}
# Create request payload
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {
"code": hooks_code,
"timeout": 30
},
"crawler_config": {
"js_code": "window.scrollTo(0, document.body.scrollHeight);",
"wait_for": "body",
"cache_mode": "bypass"
}
}
print("\nSending request with all 8 hooks...")
start_time = time.time()
response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
elapsed_time = time.time() - start_time
print(f"Request completed in {elapsed_time:.2f} seconds")
if response.status_code == 200:
data = response.json()
print("\n✅ Request successful!")
# Check hooks execution
if 'hooks' in data:
hooks_info = data['hooks']
print("\n📊 Hooks Execution Summary:")
print(f" Status: {hooks_info['status']['status']}")
print(f" Attached hooks: {len(hooks_info['status']['attached_hooks'])}")
for hook_name in hooks_info['status']['attached_hooks']:
print(f" ✓ {hook_name}")
if 'summary' in hooks_info:
summary = hooks_info['summary']
print(f"\n📈 Execution Statistics:")
print(f" Total executions: {summary['total_executions']}")
print(f" Successful: {summary['successful']}")
print(f" Failed: {summary['failed']}")
print(f" Timed out: {summary['timed_out']}")
print(f" Success rate: {summary['success_rate']:.1f}%")
if hooks_info.get('execution_log'):
print(f"\n📝 Execution Log:")
for log_entry in hooks_info['execution_log']:
status_icon = "✅" if log_entry['status'] == 'success' else "❌"
exec_time = log_entry.get('execution_time', 0)
print(f" {status_icon} {log_entry['hook_point']}: {exec_time:.3f}s")
# Check crawl results
if 'results' in data and len(data['results']) > 0:
print(f"\n📄 Crawl Results:")
for result in data['results']:
print(f" URL: {result['url']}")
print(f" Success: {result.get('success', False)}")
if result.get('html'):
print(f" HTML length: {len(result['html'])} characters")
else:
print(f"❌ Error: {response.status_code}")
try:
error_data = response.json()
print(f"Error details: {json.dumps(error_data, indent=2)}")
except:
print(f"Error text: {response.text[:500]}")
def test_authentication_flow():
"""Test a complete authentication flow with multiple hooks"""
print("\n" + "=" * 70)
print("Testing: Authentication Flow with Multiple Hooks")
print("=" * 70)
hooks_code = {
"on_page_context_created": """
async def hook(page, context, **kwargs):
print("[HOOK] Setting up authentication context")
# Add authentication cookies
await context.add_cookies([
{
"name": "auth_token",
"value": "fake_jwt_token_here",
"domain": ".httpbin.org",
"path": "/",
"httpOnly": True,
"secure": True
}
])
# Set localStorage items (for SPA authentication)
await page.evaluate('''
localStorage.setItem('user_id', '12345');
localStorage.setItem('auth_time', new Date().toISOString());
''')
return page
""",
"before_goto": """
async def hook(page, context, url, **kwargs):
print(f"[HOOK] Adding auth headers for {url}")
# Add Authorization header
import base64
credentials = base64.b64encode(b"user:passwd").decode('ascii')
await page.set_extra_http_headers({
'Authorization': f'Basic {credentials}',
'X-API-Key': 'test-api-key-123'
})
return page
"""
}
payload = {
"urls": [
"https://httpbin.org/basic-auth/user/passwd"
],
"hooks": {
"code": hooks_code,
"timeout": 15
}
}
print("\nTesting authentication with httpbin endpoints...")
response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
if response.status_code == 200:
data = response.json()
print("✅ Authentication test completed")
if 'results' in data:
for i, result in enumerate(data['results']):
print(f"\n URL {i+1}: {result['url']}")
if result.get('success'):
# Check for authentication success indicators
html_content = result.get('html', '')
if '"authenticated"' in html_content and 'true' in html_content:
print(" ✅ Authentication successful! Basic auth worked.")
else:
print(" ⚠️ Page loaded but auth status unclear")
else:
print(f" ❌ Failed: {result.get('error_message', 'Unknown error')}")
else:
print(f"❌ Error: {response.status_code}")
def test_performance_optimization_hooks():
"""Test hooks for performance optimization"""
print("\n" + "=" * 70)
print("Testing: Performance Optimization Hooks")
print("=" * 70)
hooks_code = {
"on_page_context_created": """
async def hook(page, context, **kwargs):
print("[HOOK] Optimizing page for performance")
# Block resource-heavy content
await context.route("**/*.{png,jpg,jpeg,gif,webp,svg,ico}", lambda route: route.abort())
await context.route("**/*.{woff,woff2,ttf,otf}", lambda route: route.abort())
await context.route("**/*.{mp4,webm,ogg,mp3,wav}", lambda route: route.abort())
await context.route("**/googletagmanager.com/*", lambda route: route.abort())
await context.route("**/google-analytics.com/*", lambda route: route.abort())
await context.route("**/doubleclick.net/*", lambda route: route.abort())
await context.route("**/facebook.com/*", lambda route: route.abort())
# Disable animations and transitions
await page.add_style_tag(content='''
*, *::before, *::after {
animation-duration: 0s !important;
animation-delay: 0s !important;
transition-duration: 0s !important;
transition-delay: 0s !important;
}
''')
print("[HOOK] Performance optimizations applied")
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
print("[HOOK] Removing unnecessary elements before extraction")
# Remove ads, popups, and other unnecessary elements
await page.evaluate('''() => {
// Remove common ad containers
const adSelectors = [
'.ad', '.ads', '.advertisement', '[id*="ad-"]', '[class*="ad-"]',
'.popup', '.modal', '.overlay', '.cookie-banner', '.newsletter-signup'
];
adSelectors.forEach(selector => {
document.querySelectorAll(selector).forEach(el => el.remove());
});
// Remove script tags to clean up HTML
document.querySelectorAll('script').forEach(el => el.remove());
// Remove style tags we don't need
document.querySelectorAll('style').forEach(el => el.remove());
}''')
return page
"""
}
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {
"code": hooks_code,
"timeout": 10
}
}
print("\nTesting performance optimization hooks...")
start_time = time.time()
response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
elapsed_time = time.time() - start_time
print(f"Request completed in {elapsed_time:.2f} seconds")
if response.status_code == 200:
data = response.json()
print("✅ Performance optimization test completed")
if 'results' in data and len(data['results']) > 0:
result = data['results'][0]
if result.get('html'):
print(f" HTML size: {len(result['html'])} characters")
print(" Resources blocked, ads removed, animations disabled")
else:
print(f"❌ Error: {response.status_code}")
def test_content_extraction_hooks():
"""Test hooks for intelligent content extraction"""
print("\n" + "=" * 70)
print("Testing: Content Extraction Hooks")
print("=" * 70)
hooks_code = {
"after_goto": """
async def hook(page, context, url, response, **kwargs):
print(f"[HOOK] Waiting for dynamic content on {url}")
# Wait for any lazy-loaded content
await page.wait_for_timeout(2000)
# Trigger any "Load More" buttons
try:
load_more = await page.query_selector('[class*="load-more"], [class*="show-more"], button:has-text("Load More")')
if load_more:
await load_more.click()
await page.wait_for_timeout(1000)
print("[HOOK] Clicked 'Load More' button")
except:
pass
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
print("[HOOK] Extracting structured data")
# Extract metadata
metadata = await page.evaluate('''() => {
const getMeta = (name) => {
const element = document.querySelector(`meta[name="${name}"], meta[property="${name}"]`);
return element ? element.getAttribute('content') : null;
};
return {
title: document.title,
description: getMeta('description') || getMeta('og:description'),
author: getMeta('author'),
keywords: getMeta('keywords'),
ogTitle: getMeta('og:title'),
ogImage: getMeta('og:image'),
canonical: document.querySelector('link[rel="canonical"]')?.href,
jsonLd: Array.from(document.querySelectorAll('script[type="application/ld+json"]'))
.map(el => el.textContent).filter(Boolean)
};
}''')
print(f"[HOOK] Extracted metadata: {json.dumps(metadata, indent=2)}")
# Infinite scroll handling
for i in range(3):
await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
await page.wait_for_timeout(1000)
print(f"[HOOK] Scroll iteration {i+1}/3")
return page
"""
}
payload = {
"urls": ["https://httpbin.org/html", "https://httpbin.org/json"],
"hooks": {
"code": hooks_code,
"timeout": 20
}
}
print("\nTesting content extraction hooks...")
response = requests.post(f"{API_BASE_URL}/crawl", json=payload)
if response.status_code == 200:
data = response.json()
print("✅ Content extraction test completed")
if 'hooks' in data and 'summary' in data['hooks']:
summary = data['hooks']['summary']
print(f" Hooks executed: {summary['successful']}/{summary['total_executions']}")
if 'results' in data:
for result in data['results']:
print(f"\n URL: {result['url']}")
print(f" Success: {result.get('success', False)}")
else:
print(f"❌ Error: {response.status_code}")
def main():
"""Run comprehensive hook tests"""
print("🔧 Crawl4AI Docker API - Comprehensive Hooks Testing")
print("Based on docs/examples/hooks_example.py")
print("=" * 70)
tests = [
("All Hooks Demo", test_all_hooks_demo),
("Authentication Flow", test_authentication_flow),
("Performance Optimization", test_performance_optimization_hooks),
("Content Extraction", test_content_extraction_hooks),
]
for i, (name, test_func) in enumerate(tests, 1):
print(f"\n📌 Test {i}/{len(tests)}: {name}")
try:
test_func()
print(f"✅ {name} completed")
except Exception as e:
print(f"❌ {name} failed: {e}")
import traceback
traceback.print_exc()
print("\n" + "=" * 70)
print("🎉 All comprehensive hook tests completed!")
print("=" * 70)
if __name__ == "__main__":
main()

461
docs/examples/docker_webhook_example.py Normal file
View File

@@ -0,0 +1,461 @@
"""
Docker Webhook Example for Crawl4AI
This example demonstrates how to use webhooks with the Crawl4AI job queue API.
Instead of polling for results, webhooks notify your application when jobs complete.
Supports both:
- /crawl/job - Raw crawling with markdown extraction
- /llm/job - LLM-powered content extraction
Prerequisites:
1. Crawl4AI Docker container running on localhost:11235
2. Flask installed: pip install flask requests
3. LLM API key configured in .llm.env (for LLM extraction examples)
Usage:
1. Run this script: python docker_webhook_example.py
2. The webhook server will start on http://localhost:8080
3. Jobs will be submitted and webhooks will be received automatically
"""
import requests
import json
import time
from flask import Flask, request, jsonify
from threading import Thread
# Configuration
CRAWL4AI_BASE_URL = "http://localhost:11235"
WEBHOOK_BASE_URL = "http://localhost:8080" # Your webhook receiver URL
# Initialize Flask app for webhook receiver
app = Flask(__name__)
# Store received webhook data for demonstration
received_webhooks = []
@app.route('/webhooks/crawl-complete', methods=['POST'])
def handle_crawl_webhook():
"""
Webhook handler that receives notifications when crawl jobs complete.
Payload structure:
{
"task_id": "crawl_abc123",
"task_type": "crawl",
"status": "completed" or "failed",
"timestamp": "2025-10-21T10:30:00.000000+00:00",
"urls": ["https://example.com"],
"error": "error message" (only if failed),
"data": {...} (only if webhook_data_in_payload=True)
}
"""
payload = request.json
print(f"\n{'='*60}")
print(f"📬 Webhook received for task: {payload['task_id']}")
print(f" Status: {payload['status']}")
print(f" Timestamp: {payload['timestamp']}")
print(f" URLs: {payload['urls']}")
if payload['status'] == 'completed':
# If data is in payload, process it directly
if 'data' in payload:
print(f" ✅ Data included in webhook")
data = payload['data']
# Process the crawl results here
for result in data.get('results', []):
print(f" - Crawled: {result.get('url')}")
print(f" - Markdown length: {len(result.get('markdown', ''))}")
else:
# Fetch results from API if not included
print(f" 📥 Fetching results from API...")
task_id = payload['task_id']
result_response = requests.get(f"{CRAWL4AI_BASE_URL}/crawl/job/{task_id}")
if result_response.ok:
data = result_response.json()
print(f" ✅ Results fetched successfully")
# Process the crawl results here
for result in data['result'].get('results', []):
print(f" - Crawled: {result.get('url')}")
print(f" - Markdown length: {len(result.get('markdown', ''))}")
elif payload['status'] == 'failed':
print(f" ❌ Job failed: {payload.get('error', 'Unknown error')}")
print(f"{'='*60}\n")
# Store webhook for demonstration
received_webhooks.append(payload)
# Return 200 OK to acknowledge receipt
return jsonify({"status": "received"}), 200
@app.route('/webhooks/llm-complete', methods=['POST'])
def handle_llm_webhook():
"""
Webhook handler that receives notifications when LLM extraction jobs complete.
Payload structure:
{
"task_id": "llm_1698765432_12345",
"task_type": "llm_extraction",
"status": "completed" or "failed",
"timestamp": "2025-10-21T10:30:00.000000+00:00",
"urls": ["https://example.com/article"],
"error": "error message" (only if failed),
"data": {"extracted_content": {...}} (only if webhook_data_in_payload=True)
}
"""
payload = request.json
print(f"\n{'='*60}")
print(f"🤖 LLM Webhook received for task: {payload['task_id']}")
print(f" Task Type: {payload['task_type']}")
print(f" Status: {payload['status']}")
print(f" Timestamp: {payload['timestamp']}")
print(f" URL: {payload['urls'][0]}")
if payload['status'] == 'completed':
# If data is in payload, process it directly
if 'data' in payload:
print(f" ✅ Data included in webhook")
data = payload['data']
# Webhook wraps extracted content in 'extracted_content' field
extracted = data.get('extracted_content', {})
print(f" - Extracted content:")
print(f" {json.dumps(extracted, indent=8)}")
else:
# Fetch results from API if not included
print(f" 📥 Fetching results from API...")
task_id = payload['task_id']
result_response = requests.get(f"{CRAWL4AI_BASE_URL}/llm/job/{task_id}")
if result_response.ok:
data = result_response.json()
print(f" ✅ Results fetched successfully")
# API returns unwrapped content in 'result' field
extracted = data['result']
print(f" - Extracted content:")
print(f" {json.dumps(extracted, indent=8)}")
elif payload['status'] == 'failed':
print(f" ❌ Job failed: {payload.get('error', 'Unknown error')}")
print(f"{'='*60}\n")
# Store webhook for demonstration
received_webhooks.append(payload)
# Return 200 OK to acknowledge receipt
return jsonify({"status": "received"}), 200
def start_webhook_server():
"""Start the Flask webhook server in a separate thread"""
app.run(host='0.0.0.0', port=8080, debug=False, use_reloader=False)
def submit_crawl_job_with_webhook(urls, webhook_url, include_data=False):
"""
Submit a crawl job with webhook notification.
Args:
urls: List of URLs to crawl
webhook_url: URL to receive webhook notifications
include_data: Whether to include full results in webhook payload
Returns:
task_id: The job's task identifier
"""
payload = {
"urls": urls,
"browser_config": {"headless": True},
"crawler_config": {"cache_mode": "bypass"},
"webhook_config": {
"webhook_url": webhook_url,
"webhook_data_in_payload": include_data,
# Optional: Add custom headers for authentication
# "webhook_headers": {
# "X-Webhook-Secret": "your-secret-token"
# }
}
}
print(f"\n🚀 Submitting crawl job...")
print(f" URLs: {urls}")
print(f" Webhook: {webhook_url}")
print(f" Include data: {include_data}")
response = requests.post(
f"{CRAWL4AI_BASE_URL}/crawl/job",
json=payload,
headers={"Content-Type": "application/json"}
)
if response.ok:
data = response.json()
task_id = data['task_id']
print(f" ✅ Job submitted successfully")
print(f" Task ID: {task_id}")
return task_id
else:
print(f" ❌ Failed to submit job: {response.text}")
return None
def submit_llm_job_with_webhook(url, query, webhook_url, include_data=False, schema=None, provider=None):
"""
Submit an LLM extraction job with webhook notification.
Args:
url: URL to extract content from
query: Instruction for the LLM (e.g., "Extract article title and author")
webhook_url: URL to receive webhook notifications
include_data: Whether to include full results in webhook payload
schema: Optional JSON schema for structured extraction
provider: Optional LLM provider (e.g., "openai/gpt-4o-mini")
Returns:
task_id: The job's task identifier
"""
payload = {
"url": url,
"q": query,
"cache": False,
"webhook_config": {
"webhook_url": webhook_url,
"webhook_data_in_payload": include_data,
# Optional: Add custom headers for authentication
# "webhook_headers": {
# "X-Webhook-Secret": "your-secret-token"
# }
}
}
if schema:
payload["schema"] = schema
if provider:
payload["provider"] = provider
print(f"\n🤖 Submitting LLM extraction job...")
print(f" URL: {url}")
print(f" Query: {query}")
print(f" Webhook: {webhook_url}")
print(f" Include data: {include_data}")
if provider:
print(f" Provider: {provider}")
response = requests.post(
f"{CRAWL4AI_BASE_URL}/llm/job",
json=payload,
headers={"Content-Type": "application/json"}
)
if response.ok:
data = response.json()
task_id = data['task_id']
print(f" ✅ Job submitted successfully")
print(f" Task ID: {task_id}")
return task_id
else:
print(f" ❌ Failed to submit job: {response.text}")
return None
def submit_job_without_webhook(urls):
"""
Submit a job without webhook (traditional polling approach).
Args:
urls: List of URLs to crawl
Returns:
task_id: The job's task identifier
"""
payload = {
"urls": urls,
"browser_config": {"headless": True},
"crawler_config": {"cache_mode": "bypass"}
}
print(f"\n🚀 Submitting crawl job (without webhook)...")
print(f" URLs: {urls}")
response = requests.post(
f"{CRAWL4AI_BASE_URL}/crawl/job",
json=payload
)
if response.ok:
data = response.json()
task_id = data['task_id']
print(f" ✅ Job submitted successfully")
print(f" Task ID: {task_id}")
return task_id
else:
print(f" ❌ Failed to submit job: {response.text}")
return None
def poll_job_status(task_id, timeout=60):
"""
Poll for job status (used when webhook is not configured).
Args:
task_id: The job's task identifier
timeout: Maximum time to wait in seconds
"""
print(f"\n⏳ Polling for job status...")
start_time = time.time()
while time.time() - start_time < timeout:
response = requests.get(f"{CRAWL4AI_BASE_URL}/crawl/job/{task_id}")
if response.ok:
data = response.json()
status = data.get('status', 'unknown')
if status == 'completed':
print(f" ✅ Job completed!")
return data
elif status == 'failed':
print(f" ❌ Job failed: {data.get('error', 'Unknown error')}")
return data
else:
print(f" ⏳ Status: {status}, waiting...")
time.sleep(2)
else:
print(f" ❌ Failed to get status: {response.text}")
return None
print(f" ⏰ Timeout reached")
return None
def main():
"""Run the webhook demonstration"""
# Check if Crawl4AI is running
try:
health = requests.get(f"{CRAWL4AI_BASE_URL}/health", timeout=5)
print(f"✅ Crawl4AI is running: {health.json()}")
except:
print(f"❌ Cannot connect to Crawl4AI at {CRAWL4AI_BASE_URL}")
print(" Please make sure Docker container is running:")
print(" docker run -d -p 11235:11235 --name crawl4ai unclecode/crawl4ai:latest")
return
# Start webhook server in background thread
print(f"\n🌐 Starting webhook server at {WEBHOOK_BASE_URL}...")
webhook_thread = Thread(target=start_webhook_server, daemon=True)
webhook_thread.start()
time.sleep(2) # Give server time to start
# Example 1: Job with webhook (notification only, fetch data separately)
print(f"\n{'='*60}")
print("Example 1: Webhook Notification Only")
print(f"{'='*60}")
task_id_1 = submit_crawl_job_with_webhook(
urls=["https://example.com"],
webhook_url=f"{WEBHOOK_BASE_URL}/webhooks/crawl-complete",
include_data=False
)
# Example 2: Job with webhook (data included in payload)
time.sleep(5) # Wait a bit between requests
print(f"\n{'='*60}")
print("Example 2: Webhook with Full Data")
print(f"{'='*60}")
task_id_2 = submit_crawl_job_with_webhook(
urls=["https://www.python.org"],
webhook_url=f"{WEBHOOK_BASE_URL}/webhooks/crawl-complete",
include_data=True
)
# Example 3: LLM extraction with webhook (notification only)
time.sleep(5) # Wait a bit between requests
print(f"\n{'='*60}")
print("Example 3: LLM Extraction with Webhook (Notification Only)")
print(f"{'='*60}")
task_id_3 = submit_llm_job_with_webhook(
url="https://www.example.com",
query="Extract the main heading and description from this page.",
webhook_url=f"{WEBHOOK_BASE_URL}/webhooks/llm-complete",
include_data=False,
provider="openai/gpt-4o-mini"
)
# Example 4: LLM extraction with webhook (data included + schema)
time.sleep(5) # Wait a bit between requests
print(f"\n{'='*60}")
print("Example 4: LLM Extraction with Schema and Full Data")
print(f"{'='*60}")
# Define a schema for structured extraction
schema = json.dumps({
"type": "object",
"properties": {
"title": {"type": "string", "description": "Page title"},
"description": {"type": "string", "description": "Page description"}
},
"required": ["title"]
})
task_id_4 = submit_llm_job_with_webhook(
url="https://www.python.org",
query="Extract the title and description of this website",
webhook_url=f"{WEBHOOK_BASE_URL}/webhooks/llm-complete",
include_data=True,
schema=schema,
provider="openai/gpt-4o-mini"
)
# Example 5: Traditional polling (no webhook)
time.sleep(5) # Wait a bit between requests
print(f"\n{'='*60}")
print("Example 5: Traditional Polling (No Webhook)")
print(f"{'='*60}")
task_id_5 = submit_job_without_webhook(
urls=["https://github.com"]
)
if task_id_5:
result = poll_job_status(task_id_5)
if result and result.get('status') == 'completed':
print(f" ✅ Results retrieved via polling")
# Wait for webhooks to arrive
print(f"\n⏳ Waiting for webhooks to be received...")
time.sleep(30) # Give jobs time to complete and webhooks to arrive (longer for LLM)
# Summary
print(f"\n{'='*60}")
print("Summary")
print(f"{'='*60}")
print(f"Total webhooks received: {len(received_webhooks)}")
crawl_webhooks = [w for w in received_webhooks if w['task_type'] == 'crawl']
llm_webhooks = [w for w in received_webhooks if w['task_type'] == 'llm_extraction']
print(f"\n📊 Breakdown:")
print(f" - Crawl webhooks: {len(crawl_webhooks)}")
print(f" - LLM extraction webhooks: {len(llm_webhooks)}")
print(f"\n📋 Details:")
for i, webhook in enumerate(received_webhooks, 1):
task_type = webhook['task_type']
icon = "🕷️" if task_type == "crawl" else "🤖"
print(f"{i}. {icon} Task {webhook['task_id']}: {webhook['status']} ({task_type})")
print(f"\n✅ Demo completed!")
print(f"\n💡 Pro tips:")
print(f" - In production, your webhook URL should be publicly accessible")
print(f" (e.g., https://myapp.com/webhooks) or use ngrok for testing")
print(f" - Both /crawl/job and /llm/job support the same webhook configuration")
print(f" - Use webhook_data_in_payload=true to get results directly in the webhook")
print(f" - LLM jobs may take longer, adjust timeouts accordingly")
if __name__ == "__main__":
main()

221
docs/examples/website-to-api/.gitignore vendored Normal file
View File

@@ -0,0 +1,221 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[codz]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py.cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
#uv.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
#poetry.toml
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
# pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
# https://pdm-project.org/en/latest/usage/project/#working-with-version-control
#pdm.lock
#pdm.toml
.pdm-python
.pdm-build/
# pixi
# Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
#pixi.lock
# Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
# in the .venv directory. It is recommended not to include this directory in version control.
.pixi
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# Redis
*.rdb
*.aof
*.pid
# RabbitMQ
mnesia/
rabbitmq/
rabbitmq-data/
# ActiveMQ
activemq-data/
# SageMath parsed files
*.sage.py
# Environments
.env
.envrc
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# Abstra
# Abstra is an AI-powered process automation framework.
# Ignore directories containing user credentials, local state, and settings.
# Learn more at https://abstra.io/docs
.abstra/
# Visual Studio Code
# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
# and can be added to the global gitignore or merged into this file. However, if you prefer,
# you could uncomment the following to ignore the entire vscode folder
# .vscode/
# Ruff stuff:
.ruff_cache/
# PyPI configuration file
.pypirc
# Marimo
marimo/_static/
marimo/_lsp/
__marimo__/
# Streamlit
.streamlit/secrets.toml
#directories
models
schemas
saved_requests

252
docs/examples/website-to-api/README.md Normal file
View File

@@ -0,0 +1,252 @@
# Web Scraper API with Custom Model Support
A powerful web scraping API that converts any website into structured data using AI. Features a beautiful minimalist frontend interface and support for custom LLM models!
## Features
- **AI-Powered Scraping**: Provide a URL and plain English query to extract structured data
- **Beautiful Frontend**: Modern minimalist black-and-white interface with smooth UX
- **Custom Model Support**: Use any LLM provider (OpenAI, Gemini, Anthropic, etc.) with your own API keys
- **Model Management**: Save, list, and manage multiple model configurations via web interface
- **Dual Scraping Approaches**: Choose between Schema-based (faster) or LLM-based (more flexible) extraction
- **API Request History**: Automatic saving and display of all API requests with cURL commands
- **Schema Caching**: Intelligent caching of generated schemas for faster subsequent requests
- **Duplicate Prevention**: Avoids saving duplicate requests (same URL + query)
- **RESTful API**: Easy-to-use HTTP endpoints for all operations
## Quick Start
### 1. Install Dependencies
```bash
pip install -r requirements.txt
```
### 2. Start the API Server
```bash
python app.py
```
The server will start on `http://localhost:8000` with a beautiful web interface!
### 3. Using the Web Interface
Once the server is running, open your browser and go to `http://localhost:8000` to access the modern web interface!
#### Pages:
- **Scrape Data**: Enter URLs and queries to extract structured data
- **Models**: Manage your AI model configurations (add, list, delete)
- **API Requests**: View history of all scraping requests with cURL commands
#### Features:
- **Minimalist Design**: Clean black-and-white theme inspired by modern web apps
- **Real-time Results**: See extracted data in formatted JSON
- **Copy to Clipboard**: Easy copying of results
- **Toast Notifications**: User-friendly feedback
- **Dual Scraping Modes**: Choose between Schema-based and LLM-based approaches
## Model Management
### Adding Models via Web Interface
1. Go to the **Models** page
2. Enter your model details:
- **Provider**: LLM provider (e.g., `gemini/gemini-2.5-flash`, `openai/gpt-4o`)
- **API Token**: Your API key for the provider
3. Click "Add Model"
### API Usage for Model Management
#### Save a Model Configuration
```bash
curl -X POST "http://localhost:8000/models" \
-H "Content-Type: application/json" \
-d '{
"provider": "gemini/gemini-2.5-flash",
"api_token": "your-api-key-here"
}'
```
#### List Saved Models
```bash
curl -X GET "http://localhost:8000/models"
```
#### Delete a Model Configuration
```bash
curl -X DELETE "http://localhost:8000/models/my-gemini"
```
## Scraping Approaches
### 1. Schema-based Scraping (Faster)
- Generates CSS selectors for targeted extraction
- Caches schemas for repeated requests
- Faster execution for structured websites
### 2. LLM-based Scraping (More Flexible)
- Direct LLM extraction without schema generation
- More flexible for complex or dynamic content
- Better for unstructured data extraction
## Supported LLM Providers
The API supports any LLM provider that crawl4ai supports, including:
- **Google Gemini**: `gemini/gemini-2.5-flash`, `gemini/gemini-pro`
- **OpenAI**: `openai/gpt-4`, `openai/gpt-3.5-turbo`
- **Anthropic**: `anthropic/claude-3-opus`, `anthropic/claude-3-sonnet`
- **And more...**
## API Endpoints
### Core Endpoints
- `POST /scrape` - Schema-based scraping
- `POST /scrape-with-llm` - LLM-based scraping
- `GET /schemas` - List cached schemas
- `POST /clear-cache` - Clear schema cache
- `GET /health` - Health check
### Model Management Endpoints
- `GET /models` - List saved model configurations
- `POST /models` - Save a new model configuration
- `DELETE /models/{model_name}` - Delete a model configuration
### API Request History
- `GET /saved-requests` - List all saved API requests
- `DELETE /saved-requests/{request_id}` - Delete a saved request
## Request/Response Examples
### Scrape Request
```json
{
"url": "https://example.com",
"query": "Extract the product name, price, and description",
"model_name": "my-custom-model"
}
```
### Scrape Response
```json
{
"success": true,
"url": "https://example.com",
"query": "Extract the product name, price, and description",
"extracted_data": {
"product_name": "Example Product",
"price": "$99.99",
"description": "This is an example product description"
},
"schema_used": { ... },
"timestamp": "2024-01-01T12:00:00Z"
}
```
### Model Configuration Request
```json
{
"provider": "gemini/gemini-2.5-flash",
"api_token": "your-api-key-here"
}
```
## Testing
Run the test script to verify the model management functionality:
```bash
python test_models.py
```
## File Structure
```
parse_example/
├── api_server.py # FastAPI server with all endpoints
├── web_scraper_lib.py # Core scraping library
├── test_models.py # Test script for model management
├── requirements.txt # Dependencies
├── static/ # Frontend files
│ ├── index.html # Main HTML interface
│ ├── styles.css # CSS styles (minimalist theme)
│ └── script.js # JavaScript functionality
├── schemas/ # Cached schemas
├── models/ # Saved model configurations
├── saved_requests/ # API request history
└── README.md # This file
```
## Advanced Usage
### Using the Library Directly
```python
from web_scraper_lib import WebScraperAgent
# Initialize agent
agent = WebScraperAgent()
# Save a model configuration
agent.save_model_config(
model_name="my-model",
provider="openai/gpt-4",
api_token="your-api-key"
)
# Schema-based scraping
result = await agent.scrape_data(
url="https://example.com",
query="Extract product information",
model_name="my-model"
)
# LLM-based scraping
result = await agent.scrape_data_with_llm(
url="https://example.com",
query="Extract product information",
model_name="my-model"
)
```
### Schema Caching
The system automatically caches generated schemas based on URL and query combinations:
- **First request**: Generates schema using AI
- **Subsequent requests**: Uses cached schema for faster extraction
### API Request History
All API requests are automatically saved with:
- Request details (URL, query, model used)
- Response data
- Timestamp
- cURL command for re-execution
### Duplicate Prevention
The system prevents saving duplicate requests:
- Same URL + query combinations are not saved multiple times
- Returns existing request ID for duplicates
- Keeps the API request history clean
## Error Handling
The API provides detailed error messages for common issues:
- Invalid URLs
- Missing model configurations
- API key errors
- Network timeouts
- Parsing errors

363
docs/examples/website-to-api/api_server.py Normal file
View File

@@ -0,0 +1,363 @@
from fastapi import FastAPI, HTTPException
from fastapi.staticfiles import StaticFiles
from fastapi.responses import FileResponse
from pydantic import BaseModel, HttpUrl
from typing import Dict, Any, Optional, Union, List
import uvicorn
import asyncio
import os
import json
from datetime import datetime
from web_scraper_lib import WebScraperAgent, scrape_website
app = FastAPI(
title="Web Scraper API",
description="Convert any website into a structured data API. Provide a URL and tell AI what data you need in plain English.",
version="1.0.0"
)
# Mount static files
if os.path.exists("static"):
app.mount("/static", StaticFiles(directory="static"), name="static")
# Mount assets directory
if os.path.exists("assets"):
app.mount("/assets", StaticFiles(directory="assets"), name="assets")
# Initialize the scraper agent
scraper_agent = WebScraperAgent()
# Create directory for saved API requests
os.makedirs("saved_requests", exist_ok=True)
class ScrapeRequest(BaseModel):
url: HttpUrl
query: str
model_name: Optional[str] = None
class ModelConfigRequest(BaseModel):
model_name: str
provider: str
api_token: str
class ScrapeResponse(BaseModel):
success: bool
url: str
query: str
extracted_data: Union[Dict[str, Any], list]
schema_used: Optional[Dict[str, Any]] = None
timestamp: Optional[str] = None
error: Optional[str] = None
class SavedApiRequest(BaseModel):
id: str
endpoint: str
method: str
headers: Dict[str, str]
body: Dict[str, Any]
timestamp: str
response: Optional[Dict[str, Any]] = None
def save_api_request(endpoint: str, method: str, headers: Dict[str, str], body: Dict[str, Any], response: Optional[Dict[str, Any]] = None) -> str:
"""Save an API request to a JSON file."""
# Check for duplicate requests (same URL and query)
if endpoint in ["/scrape", "/scrape-with-llm"] and "url" in body and "query" in body:
existing_requests = get_saved_requests()
for existing_request in existing_requests:
if (existing_request.endpoint == endpoint and
existing_request.body.get("url") == body["url"] and
existing_request.body.get("query") == body["query"]):
print(f"Duplicate request found for URL: {body['url']} and query: {body['query']}")
return existing_request.id # Return existing request ID instead of creating new one
request_id = datetime.now().strftime("%Y%m%d_%H%M%S_%f")[:-3]
saved_request = SavedApiRequest(
id=request_id,
endpoint=endpoint,
method=method,
headers=headers,
body=body,
timestamp=datetime.now().isoformat(),
response=response
)
file_path = os.path.join("saved_requests", f"{request_id}.json")
with open(file_path, "w") as f:
json.dump(saved_request.dict(), f, indent=2)
return request_id
def get_saved_requests() -> List[SavedApiRequest]:
"""Get all saved API requests."""
requests = []
if os.path.exists("saved_requests"):
for filename in os.listdir("saved_requests"):
if filename.endswith('.json'):
file_path = os.path.join("saved_requests", filename)
try:
with open(file_path, "r") as f:
data = json.load(f)
requests.append(SavedApiRequest(**data))
except Exception as e:
print(f"Error loading saved request {filename}: {e}")
# Sort by timestamp (newest first)
requests.sort(key=lambda x: x.timestamp, reverse=True)
return requests
@app.get("/")
async def root():
"""Serve the frontend interface."""
if os.path.exists("static/index.html"):
return FileResponse("static/index.html")
else:
return {
"message": "Web Scraper API",
"description": "Convert any website into structured data with AI",
"endpoints": {
"/scrape": "POST - Scrape data from a website",
"/schemas": "GET - List cached schemas",
"/clear-cache": "POST - Clear schema cache",
"/models": "GET - List saved model configurations",
"/models": "POST - Save a new model configuration",
"/models/{model_name}": "DELETE - Delete a model configuration",
"/saved-requests": "GET - List saved API requests"
}
}
@app.post("/scrape", response_model=ScrapeResponse)
async def scrape_website_endpoint(request: ScrapeRequest):
"""
Scrape structured data from any website.
This endpoint:
1. Takes a URL and plain English query
2. Generates a custom scraper using AI
3. Returns structured data
"""
try:
# Save the API request
headers = {"Content-Type": "application/json"}
body = {
"url": str(request.url),
"query": request.query,
"model_name": request.model_name
}
result = await scraper_agent.scrape_data(
url=str(request.url),
query=request.query,
model_name=request.model_name
)
response_data = ScrapeResponse(
success=True,
url=result["url"],
query=result["query"],
extracted_data=result["extracted_data"],
schema_used=result["schema_used"],
timestamp=result["timestamp"]
)
# Save the request with response
save_api_request(
endpoint="/scrape",
method="POST",
headers=headers,
body=body,
response=response_data.dict()
)
return response_data
except Exception as e:
# Save the failed request
headers = {"Content-Type": "application/json"}
body = {
"url": str(request.url),
"query": request.query,
"model_name": request.model_name
}
save_api_request(
endpoint="/scrape",
method="POST",
headers=headers,
body=body,
response={"error": str(e)}
)
raise HTTPException(status_code=500, detail=f"Scraping failed: {str(e)}")
@app.post("/scrape-with-llm", response_model=ScrapeResponse)
async def scrape_website_endpoint_with_llm(request: ScrapeRequest):
"""
Scrape structured data from any website using a custom LLM model.
"""
try:
# Save the API request
headers = {"Content-Type": "application/json"}
body = {
"url": str(request.url),
"query": request.query,
"model_name": request.model_name
}
result = await scraper_agent.scrape_data_with_llm(
url=str(request.url),
query=request.query,
model_name=request.model_name
)
response_data = ScrapeResponse(
success=True,
url=result["url"],
query=result["query"],
extracted_data=result["extracted_data"],
timestamp=result["timestamp"]
)
# Save the request with response
save_api_request(
endpoint="/scrape-with-llm",
method="POST",
headers=headers,
body=body,
response=response_data.dict()
)
return response_data
except Exception as e:
# Save the failed request
headers = {"Content-Type": "application/json"}
body = {
"url": str(request.url),
"query": request.query,
"model_name": request.model_name
}
save_api_request(
endpoint="/scrape-with-llm",
method="POST",
headers=headers,
body=body,
response={"error": str(e)}
)
raise HTTPException(status_code=500, detail=f"Scraping failed: {str(e)}")
@app.get("/saved-requests")
async def list_saved_requests():
"""List all saved API requests."""
try:
requests = get_saved_requests()
return {
"success": True,
"requests": [req.dict() for req in requests],
"count": len(requests)
}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to list saved requests: {str(e)}")
@app.delete("/saved-requests/{request_id}")
async def delete_saved_request(request_id: str):
"""Delete a saved API request."""
try:
file_path = os.path.join("saved_requests", f"{request_id}.json")
if os.path.exists(file_path):
os.remove(file_path)
return {
"success": True,
"message": f"Saved request '{request_id}' deleted successfully"
}
else:
raise HTTPException(status_code=404, detail=f"Saved request '{request_id}' not found")
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to delete saved request: {str(e)}")
@app.get("/schemas")
async def list_cached_schemas():
"""List all cached schemas."""
try:
schemas = await scraper_agent.get_cached_schemas()
return {
"success": True,
"cached_schemas": schemas,
"count": len(schemas)
}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to list schemas: {str(e)}")
@app.post("/clear-cache")
async def clear_schema_cache():
"""Clear all cached schemas."""
try:
scraper_agent.clear_cache()
return {
"success": True,
"message": "Schema cache cleared successfully"
}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to clear cache: {str(e)}")
@app.get("/models")
async def list_models():
"""List all saved model configurations."""
try:
models = scraper_agent.list_saved_models()
return {
"success": True,
"models": models,
"count": len(models)
}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to list models: {str(e)}")
@app.post("/models")
async def save_model_config(request: ModelConfigRequest):
"""Save a new model configuration."""
try:
success = scraper_agent.save_model_config(
model_name=request.model_name,
provider=request.provider,
api_token=request.api_token
)
if success:
return {
"success": True,
"message": f"Model configuration '{request.model_name}' saved successfully"
}
else:
raise HTTPException(status_code=500, detail="Failed to save model configuration")
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to save model: {str(e)}")
@app.delete("/models/{model_name}")
async def delete_model_config(model_name: str):
"""Delete a model configuration."""
try:
success = scraper_agent.delete_model_config(model_name)
if success:
return {
"success": True,
"message": f"Model configuration '{model_name}' deleted successfully"
}
else:
raise HTTPException(status_code=404, detail=f"Model configuration '{model_name}' not found")
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to delete model: {str(e)}")
@app.get("/health")
async def health_check():
"""Health check endpoint."""
return {"status": "healthy", "service": "web-scraper-api"}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)

49
docs/examples/website-to-api/app.py Normal file
View File

@@ -0,0 +1,49 @@
#!/usr/bin/env python3
"""
Startup script for the Web Scraper API with frontend interface.
"""
import os
import sys
import uvicorn
from pathlib import Path
def main():
# Check if static directory exists
static_dir = Path("static")
if not static_dir.exists():
print("❌ Static directory not found!")
print("Please make sure the 'static' directory exists with the frontend files.")
sys.exit(1)
# Check if required frontend files exist
required_files = ["index.html", "styles.css", "script.js"]
missing_files = []
for file in required_files:
if not (static_dir / file).exists():
missing_files.append(file)
if missing_files:
print(f"❌ Missing frontend files: {', '.join(missing_files)}")
print("Please make sure all frontend files are present in the static directory.")
sys.exit(1)
print("🚀 Starting Web Scraper API with Frontend Interface")
print("=" * 50)
print("📁 Static files found and ready to serve")
print("🌐 Frontend will be available at: http://localhost:8000")
print("🔌 API endpoints available at: http://localhost:8000/docs")
print("=" * 50)
# Start the server
uvicorn.run(
"api_server:app",
host="0.0.0.0",
port=8000,
reload=True,
log_level="info"
)
if __name__ == "__main__":
main()

BIN
docs/examples/website-to-api/assets/crawl4ai_logo.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.8 KiB

5
docs/examples/website-to-api/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
crawl4ai
fastapi
uvicorn
pydantic
litellm

201
docs/examples/website-to-api/static/index.html Normal file
View File

@@ -0,0 +1,201 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Web2API Example</title>
<link rel="stylesheet" href="/static/styles.css">
<link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css" rel="stylesheet">
</head>
<body>
<!-- Header -->
<header class="header">
<div class="header-content">
<div class="logo">
<img src="/assets/crawl4ai_logo.jpg" alt="Crawl4AI Logo" class="logo-image">
<span>Web2API Example</span>
</div>
<nav class="nav-links">
<a href="#" class="nav-link active" data-page="scrape">Scrape</a>
<a href="#" class="nav-link" data-page="models">Models</a>
<a href="#" class="nav-link" data-page="requests">API Requests</a>
</nav>
</div>
</header>
<!-- Main Content -->
<main class="main-content">
<!-- Scrape Page -->
<div id="scrape-page" class="page active">
<div class="hero-section">
<h1 class="hero-title">Turn Any Website Into An API</h1>
<p class="hero-subtitle">This example shows how to turn any website into an API using Crawl4AI.</p>
</div>
<!-- Workflow Demonstration -->
<div class="workflow-demo">
<div class="workflow-step">
<h3 class="step-title">1. Your Request</h3>
<div class="request-box">
<div class="input-group">
<label>URL:</label>
<input type="url" id="url" name="url" placeholder="https://example-bookstore.com/new-releases" required>
</div>
<div class="input-group">
<label>QUERY:</label>
<textarea id="query" name="query" placeholder="Extract all the book titles, their authors, and the biography of the author" required></textarea>
</div>
<div class="form-options">
<div class="option-group">
<label for="scraping-approach">Approach:</label>
<select id="scraping-approach" name="scraping_approach">
<option value="llm">LLM-based (More Flexible)</option>
<option value="schema">Schema-based (Uses LLM once!)</option>
</select>
</div>
<div class="option-group">
<label for="model-select">Model:</label>
<select id="model-select" name="model_name" required>
<option value="">Select a Model</option>
</select>
</div>
</div>
<button type="submit" id="extract-btn" class="extract-btn">
<i class="fas fa-magic"></i>
Extract Data
</button>
</div>
</div>
<div class="workflow-arrow">→</div>
<div class="workflow-step">
<h3 class="step-title">2. Your Instant API & Data</h3>
<div class="response-container">
<div class="api-request-box">
<label>API Request (cURL):</label>
<pre id="curl-example">curl -X POST http://localhost:8000/scrape -H "Content-Type: application/json" -d '{"url": "...", "query": "..."}'
# Or for LLM-based approach:
curl -X POST http://localhost:8000/scrape-with-llm -H "Content-Type: application/json" -d '{"url": "...", "query": "..."}'</pre>
</div>
<div class="json-response-box">
<label>JSON Response:</label>
<pre id="json-output">{
"success": true,
"extracted_data": [
{
"title": "Example Book",
"author": "John Doe",
"description": "A great book..."
}
]
}</pre>
</div>
</div>
</div>
</div>
<!-- Results Section -->
<div id="results-section" class="results-section" style="display: none;">
<div class="results-header">
<h2>Extracted Data</h2>
<button id="copy-json" class="copy-btn">
<i class="fas fa-copy"></i>
Copy JSON
</button>
</div>
<div class="results-content">
<div class="result-info">
<div class="info-item">
<span class="label">URL:</span>
<span id="result-url" class="value"></span>
</div>
<div class="info-item">
<span class="label">Query:</span>
<span id="result-query" class="value"></span>
</div>
<div class="info-item">
<span class="label">Model Used:</span>
<span id="result-model" class="value"></span>
</div>
</div>
<div class="json-display">
<pre id="actual-json-output"></pre>
</div>
</div>
</div>
<!-- Loading State -->
<div id="loading" class="loading" style="display: none;">
<div class="spinner"></div>
<p>AI is analyzing the website and extracting data...</p>
</div>
</div>
<!-- Models Page -->
<div id="models-page" class="page">
<div class="models-header">
<h1>Model Configuration</h1>
<p>Configure and manage your AI model configurations</p>
</div>
<div class="models-container">
<!-- Add New Model Form -->
<div class="model-form-section">
<h3>Add New Model</h3>
<form id="model-form" class="model-form">
<div class="form-row">
<div class="input-group">
<label for="model-name">Model Name:</label>
<input type="text" id="model-name" name="model_name" placeholder="my-gemini" required>
</div>
<div class="input-group">
<label for="provider">Provider:</label>
<input type="text" id="provider" name="provider" placeholder="gemini/gemini-2.5-flash" required>
</div>
</div>
<div class="input-group">
<label for="api-token">API Token:</label>
<input type="password" id="api-token" name="api_token" placeholder="Enter your API token" required>
</div>
<button type="submit" class="save-btn">
<i class="fas fa-save"></i>
Save Model
</button>
</form>
</div>
<!-- Saved Models List -->
<div class="saved-models-section">
<h3>Saved Models</h3>
<div id="models-list" class="models-list">
<!-- Models will be loaded here -->
</div>
</div>
</div>
</div>
<!-- API Requests Page -->
<div id="requests-page" class="page">
<div class="requests-header">
<h1>Saved API Requests</h1>
<p>View and manage your previous API requests</p>
</div>
<div class="requests-container">
<div class="requests-list" id="requests-list">
<!-- Saved requests will be loaded here -->
</div>
</div>
</div>
</main>
<!-- Toast Notifications -->
<div id="toast-container" class="toast-container"></div>
<script src="/static/script.js"></script>
</body>
</html>

401
docs/examples/website-to-api/static/script.js Normal file
View File

@@ -0,0 +1,401 @@
// API Configuration
const API_BASE_URL = 'http://localhost:8000';
// DOM Elements
const navLinks = document.querySelectorAll('.nav-link');
const pages = document.querySelectorAll('.page');
const scrapeForm = document.getElementById('scrape-form');
const modelForm = document.getElementById('model-form');
const modelSelect = document.getElementById('model-select');
const modelsList = document.getElementById('models-list');
const resultsSection = document.getElementById('results-section');
const loadingSection = document.getElementById('loading');
const copyJsonBtn = document.getElementById('copy-json');
// Navigation
navLinks.forEach(link => {
link.addEventListener('click', (e) => {
e.preventDefault();
const targetPage = link.dataset.page;
// Update active nav link
navLinks.forEach(l => l.classList.remove('active'));
link.classList.add('active');
// Show target page
pages.forEach(page => page.classList.remove('active'));
document.getElementById(`${targetPage}-page`).classList.add('active');
// Load data for the page
if (targetPage === 'models') {
loadModels();
} else if (targetPage === 'requests') {
loadSavedRequests();
}
});
});
// Scrape Form Handler
document.getElementById('extract-btn').addEventListener('click', async (e) => {
e.preventDefault();
// Scroll to results section immediately when button is clicked
document.getElementById('results-section').scrollIntoView({
behavior: 'smooth',
block: 'start'
});
const url = document.getElementById('url').value;
const query = document.getElementById('query').value;
const headless = true; // Always use headless mode
const model_name = document.getElementById('model-select').value || null;
const scraping_approach = document.getElementById('scraping-approach').value;
if (!url || !query) {
showToast('Please fill in both URL and query fields', 'error');
return;
}
if (!model_name) {
showToast('Please select a model from the dropdown or add one from the Models page', 'error');
return;
}
const data = {
url: url,
query: query,
headless: headless,
model_name: model_name
};
// Show loading state
showLoading(true);
hideResults();
try {
// Choose endpoint based on scraping approach
const endpoint = scraping_approach === 'llm' ? '/scrape-with-llm' : '/scrape';
const response = await fetch(`${API_BASE_URL}${endpoint}`, {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(data)
});
const result = await response.json();
if (response.ok) {
displayResults(result);
showToast(`Data extracted successfully using ${scraping_approach === 'llm' ? 'LLM-based' : 'Schema-based'} approach!`, 'success');
} else {
throw new Error(result.detail || 'Failed to extract data');
}
} catch (error) {
console.error('Scraping error:', error);
showToast(`Error: ${error.message}`, 'error');
} finally {
showLoading(false);
}
});
// Model Form Handler
modelForm.addEventListener('submit', async (e) => {
e.preventDefault();
const formData = new FormData(modelForm);
const data = {
model_name: formData.get('model_name'),
provider: formData.get('provider'),
api_token: formData.get('api_token')
};
try {
const response = await fetch(`${API_BASE_URL}/models`, {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(data)
});
const result = await response.json();
if (response.ok) {
showToast('Model saved successfully!', 'success');
modelForm.reset();
loadModels();
loadModelSelect();
} else {
throw new Error(result.detail || 'Failed to save model');
}
} catch (error) {
console.error('Model save error:', error);
showToast(`Error: ${error.message}`, 'error');
}
});
// Copy JSON Button
copyJsonBtn.addEventListener('click', () => {
const actualJsonOutput = document.getElementById('actual-json-output');
const textToCopy = actualJsonOutput.textContent;
navigator.clipboard.writeText(textToCopy).then(() => {
showToast('JSON copied to clipboard!', 'success');
}).catch(() => {
showToast('Failed to copy JSON', 'error');
});
});
// Load Models
async function loadModels() {
try {
const response = await fetch(`${API_BASE_URL}/models`);
const result = await response.json();
if (response.ok) {
displayModels(result.models);
} else {
throw new Error(result.detail || 'Failed to load models');
}
} catch (error) {
console.error('Load models error:', error);
showToast(`Error: ${error.message}`, 'error');
}
}
// Display Models
function displayModels(models) {
if (models.length === 0) {
modelsList.innerHTML = '<p style="text-align: center; color: #7f8c8d; padding: 2rem;">No models saved yet. Add your first model above!</p>';
return;
}
modelsList.innerHTML = models.map(model => `
<div class="model-card">
<div class="model-info">
<div class="model-name">${model}</div>
<div class="model-provider">Model Configuration</div>
</div>
<div class="model-actions">
<button class="btn btn-danger" onclick="deleteModel('${model}')">
<i class="fas fa-trash"></i>
Delete
</button>
</div>
</div>
`).join('');
}
// Delete Model
async function deleteModel(modelName) {
if (!confirm(`Are you sure you want to delete the model "${modelName}"?`)) {
return;
}
try {
const response = await fetch(`${API_BASE_URL}/models/${modelName}`, {
method: 'DELETE'
});
const result = await response.json();
if (response.ok) {
showToast('Model deleted successfully!', 'success');
loadModels();
loadModelSelect();
} else {
throw new Error(result.detail || 'Failed to delete model');
}
} catch (error) {
console.error('Delete model error:', error);
showToast(`Error: ${error.message}`, 'error');
}
}
// Load Model Select Options
async function loadModelSelect() {
try {
const response = await fetch(`${API_BASE_URL}/models`);
const result = await response.json();
if (response.ok) {
// Clear existing options
modelSelect.innerHTML = '<option value="">Select a Model</option>';
// Add model options
result.models.forEach(model => {
const option = document.createElement('option');
option.value = model;
option.textContent = model;
modelSelect.appendChild(option);
});
}
} catch (error) {
console.error('Load model select error:', error);
}
}
// Display Results
function displayResults(result) {
// Update result info
document.getElementById('result-url').textContent = result.url;
document.getElementById('result-query').textContent = result.query;
document.getElementById('result-model').textContent = result.model_name || 'Default Model';
// Display JSON in the actual results section
const actualJsonOutput = document.getElementById('actual-json-output');
actualJsonOutput.textContent = JSON.stringify(result.extracted_data, null, 2);
// Don't update the sample JSON in the workflow demo - keep it as example
// Update the cURL example based on the approach used
const scraping_approach = document.getElementById('scraping-approach').value;
const endpoint = scraping_approach === 'llm' ? '/scrape-with-llm' : '/scrape';
const curlExample = document.getElementById('curl-example');
curlExample.textContent = `curl -X POST http://localhost:8000${endpoint} -H "Content-Type: application/json" -d '{"url": "${result.url}", "query": "${result.query}"}'`;
// Show results section
resultsSection.style.display = 'block';
resultsSection.scrollIntoView({ behavior: 'smooth' });
}
// Show/Hide Loading
function showLoading(show) {
loadingSection.style.display = show ? 'block' : 'none';
}
// Hide Results
function hideResults() {
resultsSection.style.display = 'none';
}
// Toast Notifications
function showToast(message, type = 'info') {
const toastContainer = document.getElementById('toast-container');
const toast = document.createElement('div');
toast.className = `toast ${type}`;
const icon = type === 'success' ? 'fas fa-check-circle' :
type === 'error' ? 'fas fa-exclamation-circle' :
'fas fa-info-circle';
toast.innerHTML = `
<i class="${icon}"></i>
<span>${message}</span>
`;
toastContainer.appendChild(toast);
// Auto remove after 5 seconds
setTimeout(() => {
toast.remove();
}, 5000);
}
// Load Saved Requests
async function loadSavedRequests() {
try {
const response = await fetch(`${API_BASE_URL}/saved-requests`);
const result = await response.json();
if (response.ok) {
displaySavedRequests(result.requests);
} else {
throw new Error(result.detail || 'Failed to load saved requests');
}
} catch (error) {
console.error('Load saved requests error:', error);
showToast(`Error: ${error.message}`, 'error');
}
}
// Display Saved Requests
function displaySavedRequests(requests) {
const requestsList = document.getElementById('requests-list');
if (requests.length === 0) {
requestsList.innerHTML = '<p style="text-align: center; color: #CCCCCC; padding: 2rem;">No saved API requests yet. Make your first request from the Scrape page!</p>';
return;
}
requestsList.innerHTML = requests.map(request => {
const url = request.body.url;
const query = request.body.query;
const model = request.body.model_name || 'Default Model';
const endpoint = request.endpoint;
// Create curl command
const curlCommand = `curl -X POST http://localhost:8000${endpoint} \\
-H "Content-Type: application/json" \\
-d '{
"url": "${url}",
"query": "${query}",
"model_name": "${model}"
}'`;
return `
<div class="request-card">
<div class="request-header">
<div class="request-info">
<div class="request-url">${url}</div>
<div class="request-query">${query}</div>
</div>
<div class="request-actions">
<button class="btn-danger" onclick="deleteSavedRequest('${request.id}')">
<i class="fas fa-trash"></i>
Delete
</button>
</div>
</div>
<div class="request-curl">
<h4>cURL Command:</h4>
<pre>${curlCommand}</pre>
</div>
</div>
`;
}).join('');
}
// Delete Saved Request
async function deleteSavedRequest(requestId) {
if (!confirm('Are you sure you want to delete this saved request?')) {
return;
}
try {
const response = await fetch(`${API_BASE_URL}/saved-requests/${requestId}`, {
method: 'DELETE'
});
const result = await response.json();
if (response.ok) {
showToast('Saved request deleted successfully!', 'success');
loadSavedRequests();
} else {
throw new Error(result.detail || 'Failed to delete saved request');
}
} catch (error) {
console.error('Delete saved request error:', error);
showToast(`Error: ${error.message}`, 'error');
}
}
// Initialize
document.addEventListener('DOMContentLoaded', () => {
loadModelSelect();
// Check if API is available
fetch(`${API_BASE_URL}/health`)
.then(response => {
if (!response.ok) {
showToast('Warning: API server might not be running', 'error');
}
})
.catch(() => {
showToast('Warning: Cannot connect to API server. Make sure it\'s running on localhost:8000', 'error');
});
});

765
docs/examples/website-to-api/static/styles.css Normal file
View File

@@ -0,0 +1,765 @@
/* Reset and Base Styles */
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: #000000;
color: #FFFFFF;
line-height: 1.6;
font-size: 16px;
}
/* Header */
.header {
border-bottom: 1px solid #333;
padding: 1rem 0;
background: #000000;
position: sticky;
top: 0;
z-index: 100;
}
.header-content {
max-width: 1200px;
margin: 0 auto;
padding: 0 2rem;
display: flex;
justify-content: space-between;
align-items: center;
}
.logo {
display: flex;
align-items: center;
gap: 0.5rem;
font-size: 1.5rem;
font-weight: 600;
color: #FFFFFF;
}
.logo-image {
width: 40px;
height: 40px;
border-radius: 4px;
object-fit: contain;
}
.nav-links {
display: flex;
gap: 2rem;
}
.nav-link {
color: #CCCCCC;
text-decoration: none;
font-weight: 500;
transition: color 0.2s ease;
}
.nav-link:hover,
.nav-link.active {
color: #FFFFFF;
}
/* Main Content */
.main-content {
max-width: 1200px;
margin: 0 auto;
padding: 2rem;
}
.page {
display: none;
}
.page.active {
display: block;
}
/* Hero Section */
.hero-section {
text-align: center;
margin-bottom: 4rem;
padding: 2rem 0;
}
.hero-title {
font-size: 3rem;
font-weight: 700;
color: #FFFFFF;
margin-bottom: 1rem;
line-height: 1.2;
}
.hero-subtitle {
font-size: 1.25rem;
color: #CCCCCC;
max-width: 600px;
margin: 0 auto;
}
/* Workflow Demo */
.workflow-demo {
display: grid;
grid-template-columns: 1fr auto 1fr;
gap: 2rem;
align-items: start;
margin-bottom: 4rem;
}
.workflow-step {
display: flex;
flex-direction: column;
gap: 1rem;
}
.step-title {
font-size: 1.25rem;
font-weight: 600;
color: #FFFFFF;
text-align: center;
margin-bottom: 1rem;
}
.workflow-arrow {
font-size: 2rem;
font-weight: 700;
color: #09b5a5;
display: flex;
align-items: center;
justify-content: center;
margin-top: 20rem;
}
/* Request Box */
.request-box {
border: 2px solid #333;
border-radius: 8px;
padding: 2rem;
background: #111111;
}
.input-group {
margin-bottom: 1.5rem;
}
.input-group label {
display: block;
font-family: 'Courier New', monospace;
font-weight: 600;
color: #FFFFFF;
margin-bottom: 0.5rem;
font-size: 0.9rem;
}
.input-group input,
.input-group textarea,
.input-group select {
width: 100%;
padding: 0.75rem;
border: 1px solid #333;
border-radius: 4px;
font-family: 'Courier New', monospace;
font-size: 0.9rem;
background: #1A1A1A;
color: #FFFFFF;
transition: border-color 0.2s ease;
}
.input-group input:focus,
.input-group textarea:focus,
.input-group select:focus {
outline: none;
border-color: #09b5a5;
}
.input-group textarea {
min-height: 80px;
resize: vertical;
}
.form-options {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 1rem;
margin-bottom: 1.5rem;
}
.option-group {
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.option-group label {
font-family: 'Courier New', monospace;
font-weight: 600;
color: #FFFFFF;
font-size: 0.9rem;
}
.option-group input[type="checkbox"] {
width: auto;
margin-right: 0.5rem;
}
.extract-btn {
width: 100%;
padding: 1rem;
background: #09b5a5;
color: #000000;
border: none;
border-radius: 4px;
font-size: 1rem;
font-weight: 600;
cursor: pointer;
transition: background-color 0.2s ease;
display: flex;
align-items: center;
justify-content: center;
gap: 0.5rem;
}
.extract-btn:hover {
background: #09b5a5;
}
/* Dropdown specific styling */
select,
.input-group select,
.option-group select {
cursor: pointer !important;
appearance: none !important;
-webkit-appearance: none !important;
-moz-appearance: none !important;
-ms-appearance: none !important;
background-image: url("data:image/svg+xml;charset=UTF-8,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='%23FFFFFF' stroke-width='2' stroke-linecap='round' stroke-linejoin='round'%3e%3cpolyline points='6,9 12,15 18,9'%3e%3c/polyline%3e%3c/svg%3e") !important;
background-repeat: no-repeat !important;
background-position: right 0.75rem center !important;
background-size: 1rem !important;
padding-right: 2.5rem !important;
border: 1px solid #333 !important;
border-radius: 4px !important;
font-family: 'Courier New', monospace !important;
font-size: 0.9rem !important;
background-color: #1A1A1A !important;
color: #FFFFFF !important;
}
select:hover,
.input-group select:hover,
.option-group select:hover {
border-color: #09b5a5 !important;
}
select:focus,
.input-group select:focus,
.option-group select:focus {
outline: none !important;
border-color: #09b5a5 !important;
}
select option,
.input-group select option,
.option-group select option {
background: #1A1A1A !important;
color: #FFFFFF !important;
padding: 0.5rem !important;
}
/* Response Container */
.response-container {
display: flex;
flex-direction: column;
gap: 1rem;
}
.api-request-box,
.json-response-box {
border: 2px solid #333;
border-radius: 8px;
padding: 1.5rem;
background: #111111;
}
.api-request-box label,
.json-response-box label {
display: block;
font-family: 'Courier New', monospace;
font-weight: 600;
color: #FFFFFF;
margin-bottom: 0.5rem;
font-size: 0.9rem;
}
.api-request-box pre,
.json-response-box pre {
font-family: 'Courier New', monospace;
font-size: 0.85rem;
line-height: 1.5;
color: #FFFFFF;
background: #1A1A1A;
padding: 1rem;
border-radius: 4px;
overflow-x: auto;
white-space: pre-wrap;
word-break: break-all;
}
/* Results Section */
.results-section {
border: 2px solid #333;
border-radius: 8px;
overflow: hidden;
margin-top: 2rem;
background: #111111;
}
.results-header {
background: #1A1A1A;
color: #FFFFFF;
padding: 1rem 1.5rem;
display: flex;
justify-content: space-between;
align-items: center;
border-bottom: 1px solid #333;
}
.results-header h2 {
font-size: 1.25rem;
font-weight: 600;
color: #FFFFFF;
}
.copy-btn {
background: #09b5a5;
color: #000000;
border: none;
padding: 0.5rem 1rem;
border-radius: 4px;
font-size: 0.9rem;
font-weight: 600;
cursor: pointer;
display: flex;
align-items: center;
gap: 0.5rem;
transition: background-color 0.2s ease;
}
.copy-btn:hover {
background: #09b5a5;
}
.results-content {
padding: 1.5rem;
}
.result-info {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
margin-bottom: 1.5rem;
padding: 1rem;
background: #1A1A1A;
border-radius: 4px;
border: 1px solid #333;
}
.info-item {
display: flex;
flex-direction: column;
gap: 0.25rem;
}
.info-item .label {
font-weight: 600;
color: #FFFFFF;
font-size: 0.9rem;
}
.info-item .value {
color: #CCCCCC;
word-break: break-all;
}
.json-display {
background: #1A1A1A;
border-radius: 4px;
overflow: hidden;
border: 1px solid #333;
}
.json-display pre {
color: #FFFFFF;
padding: 1.5rem;
margin: 0;
overflow-x: auto;
font-family: 'Courier New', monospace;
font-size: 0.9rem;
line-height: 1.5;
}
/* Loading State */
.loading {
text-align: center;
padding: 3rem;
}
.spinner {
width: 40px;
height: 40px;
border: 3px solid #333;
border-top: 3px solid #09b5a5;
border-radius: 50%;
animation: spin 1s linear infinite;
margin: 0 auto 1rem;
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
/* Models Page */
.models-header {
text-align: center;
margin-bottom: 3rem;
}
.models-header h1 {
font-size: 2.5rem;
font-weight: 700;
color: #FFFFFF;
margin-bottom: 1rem;
}
.models-header p {
font-size: 1.1rem;
color: #CCCCCC;
}
/* API Requests Page */
.requests-header {
text-align: center;
margin-bottom: 3rem;
}
.requests-header h1 {
font-size: 2.5rem;
font-weight: 700;
color: #FFFFFF;
margin-bottom: 1rem;
}
.requests-header p {
font-size: 1.1rem;
color: #CCCCCC;
}
.requests-container {
max-width: 1200px;
margin: 0 auto;
}
.requests-list {
display: grid;
gap: 1.5rem;
}
.request-card {
border: 2px solid #333;
border-radius: 8px;
padding: 1.5rem;
background: #111111;
transition: border-color 0.2s ease;
}
.request-card:hover {
border-color: #09b5a5;
}
.request-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
padding-bottom: 1rem;
border-bottom: 1px solid #333;
}
.request-info {
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.request-url {
font-family: 'Courier New', monospace;
font-weight: 600;
color: #09b5a5;
font-size: 1.1rem;
word-break: break-all;
}
.request-query {
color: #CCCCCC;
font-size: 0.9rem;
margin-top: 0.5rem;
word-break: break-all;
}
.request-actions {
display: flex;
gap: 0.5rem;
}
.request-curl {
background: #1A1A1A;
border: 1px solid #333;
border-radius: 4px;
padding: 1rem;
margin-top: 1rem;
}
.request-curl h4 {
color: #FFFFFF;
font-size: 0.9rem;
font-weight: 600;
margin-bottom: 0.5rem;
font-family: 'Courier New', monospace;
}
.request-curl pre {
color: #CCCCCC;
font-size: 0.8rem;
line-height: 1.4;
overflow-x: auto;
white-space: pre-wrap;
word-break: break-all;
background: #111111;
padding: 0.75rem;
border-radius: 4px;
border: 1px solid #333;
}
.models-container {
max-width: 800px;
margin: 0 auto;
}
.model-form-section {
border: 2px solid #333;
border-radius: 8px;
padding: 2rem;
margin-bottom: 2rem;
background: #111111;
}
.model-form-section h3 {
font-size: 1.25rem;
font-weight: 600;
color: #FFFFFF;
margin-bottom: 1.5rem;
}
.model-form {
display: flex;
flex-direction: column;
gap: 1.5rem;
}
.form-row {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 1rem;
}
.save-btn {
padding: 1rem;
background: #09b5a5;
color: #000000;
border: none;
border-radius: 4px;
font-size: 1rem;
font-weight: 600;
cursor: pointer;
transition: background-color 0.2s ease;
display: flex;
align-items: center;
justify-content: center;
gap: 0.5rem;
}
.save-btn:hover {
background: #09b5a5;
}
.saved-models-section h3 {
font-size: 1.25rem;
font-weight: 600;
color: #FFFFFF;
margin-bottom: 1.5rem;
}
.models-list {
display: grid;
gap: 1rem;
}
.model-card {
border: 2px solid #333;
border-radius: 8px;
padding: 1.5rem;
display: flex;
justify-content: space-between;
align-items: center;
transition: border-color 0.2s ease;
background: #111111;
}
.model-card:hover {
border-color: #09b5a5;
}
.model-info {
flex: 1;
}
.model-name {
font-weight: 600;
color: #FFFFFF;
font-size: 1.1rem;
margin-bottom: 0.5rem;
}
.model-provider {
color: #CCCCCC;
font-size: 0.9rem;
}
.model-actions {
display: flex;
gap: 0.5rem;
}
.btn-danger {
background: #FF4444;
color: #FFFFFF;
border: none;
padding: 0.5rem 1rem;
border-radius: 4px;
font-size: 0.9rem;
font-weight: 600;
cursor: pointer;
transition: background-color 0.2s ease;
display: flex;
align-items: center;
gap: 0.5rem;
}
.btn-danger:hover {
background: #CC3333;
}
/* Toast Notifications */
.toast-container {
position: fixed;
top: 20px;
right: 20px;
z-index: 1000;
}
.toast {
background: #111111;
border: 2px solid #333;
border-radius: 4px;
padding: 1rem 1.5rem;
margin-bottom: 0.5rem;
display: flex;
align-items: center;
gap: 0.5rem;
animation: slideIn 0.3s ease;
max-width: 400px;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
color: #FFFFFF;
}
.toast.success {
border-color: #09b5a5;
background: #0A1A1A;
}
.toast.error {
border-color: #FF4444;
background: #1A0A0A;
}
.toast.info {
border-color: #09b5a5;
background: #0A1A1A;
}
@keyframes slideIn {
from {
transform: translateX(100%);
opacity: 0;
}
to {
transform: translateX(0);
opacity: 1;
}
}
/* Responsive Design */
@media (max-width: 768px) {
.header-content {
padding: 0 1rem;
}
.main-content {
padding: 1rem;
}
.hero-title {
font-size: 2rem;
}
.workflow-demo {
grid-template-columns: 1fr;
gap: 1rem;
}
.workflow-arrow {
transform: rotate(90deg);
margin: 1rem 0;
}
.form-options {
grid-template-columns: 1fr;
}
.form-row {
grid-template-columns: 1fr;
}
.result-info {
grid-template-columns: 1fr;
}
.model-card {
flex-direction: column;
gap: 1rem;
text-align: center;
}
.model-actions {
width: 100%;
justify-content: center;
}
}

28
docs/examples/website-to-api/test_api.py Normal file
View File

@@ -0,0 +1,28 @@
import asyncio
from web_scraper_lib import scrape_website
import os
async def test_library():
"""Test the mini library directly."""
print("=== Testing Mini Library ===")
# Test 1: Scrape with a custom model
url = "https://marketplace.mainstreet.co.in/collections/adidas-yeezy/products/adidas-yeezy-boost-350-v2-yecheil-non-reflective"
query = "Extract the following data: Product name, Product price, Product description, Product size. DO NOT EXTRACT ANYTHING ELSE."
if os.path.exists("models"):
model_name = os.listdir("models")[0].split(".")[0]
else:
raise Exception("No models found in models directory")
print(f"Scraping: {url}")
print(f"Query: {query}")
try:
result = await scrape_website(url, query, model_name)
print("✅ Library test successful!")
print(f"Extracted data: {result['extracted_data']}")
except Exception as e:
print(f"❌ Library test failed: {e}")
if __name__ == "__main__":
asyncio.run(test_library())

67
docs/examples/website-to-api/test_models.py Normal file
View File

@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""
Test script for the new model management functionality.
This script demonstrates how to save and use custom model configurations.
"""
import asyncio
import requests
import json
# API base URL
BASE_URL = "http://localhost:8000"
def test_model_management():
"""Test the model management endpoints."""
print("=== Testing Model Management ===")
# 1. List current models
print("\n1. Listing current models:")
response = requests.get(f"{BASE_URL}/models")
print(f"Status: {response.status_code}")
print(f"Response: {json.dumps(response.json(), indent=2)}")
# 2. Save another model configuration (OpenAI example)
print("\n2. Saving OpenAI model configuration:")
openai_config = {
"model_name": "my-openai",
"provider": "openai",
"api_token": "your-openai-api-key-here"
}
response = requests.post(f"{BASE_URL}/models", json=openai_config)
print(f"Status: {response.status_code}")
print(f"Response: {json.dumps(response.json(), indent=2)}")
# 3. List models again to see the new ones
print("\n3. Listing models after adding new ones:")
response = requests.get(f"{BASE_URL}/models")
print(f"Status: {response.status_code}")
print(f"Response: {json.dumps(response.json(), indent=2)}")
# 4. Delete a model configuration
print("\n4. Deleting a model configuration:")
response = requests.delete(f"{BASE_URL}/models/my-openai")
print(f"Status: {response.status_code}")
print(f"Response: {json.dumps(response.json(), indent=2)}")
# 5. Final list of models
print("\n5. Final list of models:")
response = requests.get(f"{BASE_URL}/models")
print(f"Status: {response.status_code}")
print(f"Response: {json.dumps(response.json(), indent=2)}")
if __name__ == "__main__":
print("Model Management Test Script")
print("Make sure the API server is running on http://localhost:8000")
print("=" * 50)
try:
test_model_management()
except requests.exceptions.ConnectionError:
print("Error: Could not connect to the API server.")
print("Make sure the server is running with: python api_server.py")
except Exception as e:
print(f"Error: {e}")

397
docs/examples/website-to-api/web_scraper_lib.py Normal file
View File

@@ -0,0 +1,397 @@
from crawl4ai import (
AsyncWebCrawler,
BrowserConfig,
CacheMode,
CrawlerRunConfig,
LLMConfig,
JsonCssExtractionStrategy,
LLMExtractionStrategy
)
import os
import json
import hashlib
from typing import Dict, Any, Optional, List
from litellm import completion
class ModelConfig:
"""Configuration for LLM models."""
def __init__(self, provider: str, api_token: str):
self.provider = provider
self.api_token = api_token
def to_dict(self) -> Dict[str, Any]:
return {
"provider": self.provider,
"api_token": self.api_token
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'ModelConfig':
return cls(
provider=data["provider"],
api_token=data["api_token"]
)
class WebScraperAgent:
"""
A mini library that converts any website into a structured data API.
Features:
1. Provide a URL and tell AI what data you need in plain English
2. Generate: Agent reverse-engineers the site and deploys custom scraper
3. Integrate: Use private API endpoint to get structured data
4. Support for custom LLM models and API keys
"""
def __init__(self, schemas_dir: str = "schemas", models_dir: str = "models"):
self.schemas_dir = schemas_dir
self.models_dir = models_dir
os.makedirs(self.schemas_dir, exist_ok=True)
os.makedirs(self.models_dir, exist_ok=True)
def _generate_schema_key(self, url: str, query: str) -> str:
"""Generate a unique key for schema caching based on URL and query."""
content = f"{url}:{query}"
return hashlib.md5(content.encode()).hexdigest()
def save_model_config(self, model_name: str, provider: str, api_token: str) -> bool:
"""
Save a model configuration for later use.
Args:
model_name: User-friendly name for the model
provider: LLM provider (e.g., 'gemini', 'openai', 'anthropic')
api_token: API token for the provider
Returns:
True if saved successfully
"""
try:
model_config = ModelConfig(provider, api_token)
config_path = os.path.join(self.models_dir, f"{model_name}.json")
with open(config_path, "w") as f:
json.dump(model_config.to_dict(), f, indent=2)
print(f"Model configuration saved: {model_name}")
return True
except Exception as e:
print(f"Failed to save model configuration: {e}")
return False
def load_model_config(self, model_name: str) -> Optional[ModelConfig]:
"""
Load a saved model configuration.
Args:
model_name: Name of the saved model configuration
Returns:
ModelConfig object or None if not found
"""
try:
config_path = os.path.join(self.models_dir, f"{model_name}.json")
if not os.path.exists(config_path):
return None
with open(config_path, "r") as f:
data = json.load(f)
return ModelConfig.from_dict(data)
except Exception as e:
print(f"Failed to load model configuration: {e}")
return None
def list_saved_models(self) -> List[str]:
"""List all saved model configurations."""
models = []
for filename in os.listdir(self.models_dir):
if filename.endswith('.json'):
models.append(filename[:-5]) # Remove .json extension
return models
def delete_model_config(self, model_name: str) -> bool:
"""
Delete a saved model configuration.
Args:
model_name: Name of the model configuration to delete
Returns:
True if deleted successfully
"""
try:
config_path = os.path.join(self.models_dir, f"{model_name}.json")
if os.path.exists(config_path):
os.remove(config_path)
print(f"Model configuration deleted: {model_name}")
return True
return False
except Exception as e:
print(f"Failed to delete model configuration: {e}")
return False
async def _load_or_generate_schema(self, url: str, query: str, session_id: str = "schema_generator", model_name: Optional[str] = None) -> Dict[str, Any]:
"""
Loads schema from cache if exists, otherwise generates using AI.
This is the "Generate" step - our agent reverse-engineers the site.
Args:
url: URL to scrape
query: Query for data extraction
session_id: Session identifier
model_name: Name of saved model configuration to use
"""
schema_key = self._generate_schema_key(url, query)
schema_path = os.path.join(self.schemas_dir, f"{schema_key}.json")
if os.path.exists(schema_path):
print(f"Schema found in cache for {url}")
with open(schema_path, "r") as f:
return json.load(f)
print(f"Generating new schema for {url}")
print(f"Query: {query}")
query += """
IMPORTANT:
GENERATE THE SCHEMA WITH ONLY THE FIELDS MENTIONED IN THE QUERY. MAKE SURE THE NUMBER OF FIELDS IN THE SCHEME MATCH THE NUMBER OF FIELDS IN THE QUERY.
"""
# Step 1: Fetch the page HTML
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
result = await crawler.arun(
url=url,
config=CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
session_id=session_id,
simulate_user=True,
remove_overlay_elements=True,
delay_before_return_html=5,
)
)
html = result.fit_html
# Step 2: Generate schema using AI with custom model if specified
print("AI is analyzing the page structure...")
# Use custom model configuration if provided
if model_name:
model_config = self.load_model_config(model_name)
if model_config:
llm_config = LLMConfig(
provider=model_config.provider,
api_token=model_config.api_token
)
print(f"Using custom model: {model_name}")
else:
raise ValueError(f"Model configuration '{model_name}' not found. Please add it from the Models page.")
else:
# Require a model to be specified
raise ValueError("No model specified. Please select a model from the dropdown or add one from the Models page.")
schema = JsonCssExtractionStrategy.generate_schema(
html=html,
llm_config=llm_config,
query=query
)
# Step 3: Cache the generated schema
print(f"Schema generated and cached: {json.dumps(schema, indent=2)}")
with open(schema_path, "w") as f:
json.dump(schema, f, indent=2)
return schema
def _generate_llm_schema(self, query: str, llm_config: LLMConfig) -> Dict[str, Any]:
"""
Generate a schema for a given query using a custom LLM model.
Args:
query: Plain English description of what data to extract
model_config: Model configuration to use
"""
# ask the model to generate a schema for the given query in the form of a json.
prompt = f"""
IDENTIFY THE FIELDS FOR EXTRACTION MENTIONED IN THE QUERY and GENERATE A JSON SCHEMA FOR THE FIELDS.
eg.
{{
"name": "str",
"age": "str",
"email": "str",
"product_name": "str",
"product_price": "str",
"product_description": "str",
"product_image": "str",
"product_url": "str",
"product_rating": "str",
"product_reviews": "str",
}}
Here is the query:
{query}
IMPORTANT:
THE RESULT SHOULD BE A JSON OBJECT.
MAKE SURE THE NUMBER OF FIELDS IN THE RESULT MATCH THE NUMBER OF FIELDS IN THE QUERY.
THE RESULT SHOULD BE A JSON OBJECT.
"""
response = completion(
model=llm_config.provider,
messages=[{"role": "user", "content": prompt}],
api_key=llm_config.api_token,
result_type="json"
)
return response.json()["choices"][0]["message"]["content"]
async def scrape_data_with_llm(self, url: str, query: str, model_name: Optional[str] = None) -> Dict[str, Any]:
"""
Scrape structured data from any website using a custom LLM model.
Args:
url: The website URL to scrape
query: Plain English description of what data to extract
model_name: Name of saved model configuration to use
"""
if model_name:
model_config = self.load_model_config(model_name)
if model_config:
llm_config = LLMConfig(
provider=model_config.provider,
api_token=model_config.api_token
)
print(f"Using custom model: {model_name}")
else:
raise ValueError(f"Model configuration '{model_name}' not found. Please add it from the Models page.")
else:
# Require a model to be specified
raise ValueError("No model specified. Please select a model from the dropdown or add one from the Models page.")
query += """\n
IMPORTANT:
THE RESULT SHOULD BE A JSON OBJECT WITH THE ONLY THE FIELDS MENTIONED IN THE QUERY.
MAKE SURE THE NUMBER OF FIELDS IN THE RESULT MATCH THE NUMBER OF FIELDS IN THE QUERY.
THE RESULT SHOULD BE A JSON OBJECT.
"""
schema = self._generate_llm_schema(query, llm_config)
print(f"Schema: {schema}")
llm_extraction_strategy = LLMExtractionStrategy(
llm_config=llm_config,
instruction=query,
result_type="json",
schema=schema
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url=url,
config=CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
simulate_user=True,
extraction_strategy=llm_extraction_strategy,
)
)
extracted_data = result.extracted_content
if isinstance(extracted_data, str):
try:
extracted_data = json.loads(extracted_data)
except json.JSONDecodeError:
# If it's not valid JSON, keep it as string
pass
return {
"url": url,
"query": query,
"extracted_data": extracted_data,
"timestamp": result.timestamp if hasattr(result, 'timestamp') else None
}
async def scrape_data(self, url: str, query: str, model_name: Optional[str] = None) -> Dict[str, Any]:
"""
Main method to scrape structured data from any website.
Args:
url: The website URL to scrape
query: Plain English description of what data to extract
model_name: Name of saved model configuration to use
Returns:
Structured data extracted from the website
"""
# Step 1: Generate or load schema (reverse-engineer the site)
schema = await self._load_or_generate_schema(url=url, query=query, model_name=model_name)
# Step 2: Deploy custom high-speed scraper
print(f"Deploying custom scraper for {url}")
browser_config = BrowserConfig(headless=True)
async with AsyncWebCrawler(config=browser_config) as crawler:
run_config = CrawlerRunConfig(
extraction_strategy=JsonCssExtractionStrategy(schema=schema),
)
result = await crawler.arun(url=url, config=run_config)
# Step 3: Return structured data
# Parse extracted_content if it's a JSON string
extracted_data = result.extracted_content
if isinstance(extracted_data, str):
try:
extracted_data = json.loads(extracted_data)
except json.JSONDecodeError:
# If it's not valid JSON, keep it as string
pass
return {
"url": url,
"query": query,
"extracted_data": extracted_data,
"schema_used": schema,
"timestamp": result.timestamp if hasattr(result, 'timestamp') else None
}
async def get_cached_schemas(self) -> Dict[str, str]:
"""Get list of cached schemas."""
schemas = {}
for filename in os.listdir(self.schemas_dir):
if filename.endswith('.json'):
schema_key = filename[:-5] # Remove .json extension
schemas[schema_key] = filename
return schemas
def clear_cache(self):
"""Clear all cached schemas."""
import shutil
if os.path.exists(self.schemas_dir):
shutil.rmtree(self.schemas_dir)
os.makedirs(self.schemas_dir, exist_ok=True)
print("Schema cache cleared")
# Convenience function for simple usage
async def scrape_website(url: str, query: str, model_name: Optional[str] = None) -> Dict[str, Any]:
"""
Simple function to scrape any website with plain English instructions.
Args:
url: Website URL
query: Plain English description of what data to extract
model_name: Name of saved model configuration to use
Returns:
Extracted structured data
"""
agent = WebScraperAgent()
return await agent.scrape_data(url, query, model_name)
async def scrape_website_with_llm(url: str, query: str, model_name: Optional[str] = None):
"""
Scrape structured data from any website using a custom LLM model.
Args:
url: The website URL to scrape
query: Plain English description of what data to extract
model_name: Name of saved model configuration to use
"""
agent = WebScraperAgent()
return await agent.scrape_data_with_llm(url, query, model_name)

50
docs/md_v2/advanced/adaptive-strategies.md
View File

@@ -126,30 +126,6 @@ Factors:
- URL depth (fewer slashes = higher authority)
- Clean URL structure
### Custom Link Scoring
```python
class CustomLinkScorer:
def score(self, link: Link, query: str, state: CrawlState) -> float:
# Prioritize specific URL patterns
if "/api/reference/" in link.href:
return 2.0 # Double the score
# Deprioritize certain sections
if "/archive/" in link.href:
return 0.1 # Reduce score by 90%
# Default scoring
return 1.0
# Use with adaptive crawler
adaptive = AdaptiveCrawler(
crawler,
config=config,
link_scorer=CustomLinkScorer()
)
```
## Domain-Specific Configurations
### Technical Documentation
@@ -230,8 +206,12 @@ config = AdaptiveConfig(
# Periodically clean state
if len(state.knowledge_base) > 1000:
# Keep only most relevant
state.knowledge_base = get_top_relevant(state.knowledge_base, 500)
# Keep only the top 500 most relevant docs
top_content = adaptive.get_relevant_content(top_k=500)
keep_indices = {d["index"] for d in top_content}
state.knowledge_base = [
doc for i, doc in enumerate(state.knowledge_base) if i in keep_indices
]
```
### Parallel Processing
@@ -252,18 +232,6 @@ tasks = [
results = await asyncio.gather(*tasks)
```
### Caching Strategy
```python
# Enable caching for repeated crawls
async with AsyncWebCrawler(
config=BrowserConfig(
cache_mode=CacheMode.ENABLED
)
) as crawler:
adaptive = AdaptiveCrawler(crawler, config)
```
## Debugging & Analysis
### Enable Verbose Logging
@@ -322,9 +290,9 @@ with open("crawl_analysis.json", "w") as f:
### Implementing a Custom Strategy
```python
from crawl4ai.adaptive_crawler import BaseStrategy
from crawl4ai.adaptive_crawler import CrawlStrategy
class DomainSpecificStrategy(BaseStrategy):
class DomainSpecificStrategy(CrawlStrategy):
def calculate_coverage(self, state: CrawlState) -> float:
# Custom coverage calculation
# e.g., weight certain terms more heavily
@@ -351,7 +319,7 @@ adaptive = AdaptiveCrawler(
### Combining Strategies
```python
class HybridStrategy(BaseStrategy):
class HybridStrategy(CrawlStrategy):
def __init__(self):
self.strategies = [
TechnicalDocStrategy(),

12
docs/md_v2/advanced/proxy-security.md
View File

@@ -7,13 +7,13 @@ Simple proxy configuration with `BrowserConfig`:
```python
from crawl4ai.async_configs import BrowserConfig
# Using proxy URL
browser_config = BrowserConfig(proxy="http://proxy.example.com:8080")
# Using HTTP proxy
browser_config = BrowserConfig(proxy_config={"server": "http://proxy.example.com:8080"})
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com")
# Using SOCKS proxy
browser_config = BrowserConfig(proxy="socks5://proxy.example.com:1080")
browser_config = BrowserConfig(proxy_config={"server": "socks5://proxy.example.com:1080"})
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com")
```
@@ -25,7 +25,11 @@ Use an authenticated proxy with `BrowserConfig`:
```python
from crawl4ai.async_configs import BrowserConfig
browser_config = BrowserConfig(proxy="http://[username]:[password]@[host]:[port]")
browser_config = BrowserConfig(proxy_config={
"server": "http://[host]:[port]",
"username": "[username]",
"password": "[password]",
})
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com")
```

3
docs/md_v2/api/parameters.md
View File

@@ -23,7 +23,7 @@ browser_cfg = BrowserConfig(
| **`headless`** | `bool` (default: `True`) | Headless means no visible UI. `False` is handy for debugging. |
| **`viewport_width`** | `int` (default: `1080`) | Initial page width (in px). Useful for testing responsive layouts. |
| **`viewport_height`** | `int` (default: `600`) | Initial page height (in px). |
| **`proxy`** | `str` (default: `None`) | Single-proxy URL if you want all traffic to go through it, e.g. `"http://user:pass@proxy:8080"`. |
| **`proxy`** | `str` (deprecated) | Deprecated. Use `proxy_config` instead. If set, it will be auto-converted internally. |
| **`proxy_config`** | `dict` (default: `None`) | For advanced or multi-proxy needs, specify details like `{"server": "...", "username": "...", ...}`. |
| **`use_persistent_context`** | `bool` (default: `False`) | If `True`, uses a **persistent** browser context (keep cookies, sessions across runs). Also sets `use_managed_browser=True`. |
| **`user_data_dir`** | `str or None` (default: `None`) | Directory to store user data (profiles, cookies). Must be set if you want permanent sessions. |
@@ -155,6 +155,7 @@ If your page is a single-page app with repeated JS updates, set `js_only=True` i
| **`exclude_external_links`** | `bool` (False) | Removes all links pointing outside the current domain. |
| **`exclude_social_media_links`** | `bool` (False) | Strips links specifically to social sites (like Facebook or Twitter). |
| **`exclude_domains`** | `list` ([]) | Provide a custom list of domains to exclude (like `["ads.com", "trackers.io"]`). |
| **`preserve_https_for_internal_links`** | `bool` (False) | If `True`, preserves HTTPS scheme for internal links even when the server redirects to HTTP. Useful for security-conscious crawling. |
Use these for link-level content filtering (often to keep crawls “internal” or to remove spammy domains).

BIN
docs/md_v2/assets/crawl4ai-skill.zip Normal file
View File

Binary file not shown.

BIN
docs/md_v2/assets/images/logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.6 KiB

376
docs/md_v2/assets/page_actions.css Normal file
View File

@@ -0,0 +1,376 @@
/* ==== File: assets/page_actions.css ==== */
/* Page Actions Dropdown - Terminal Style */
/* Wrapper - positioned in content area */
.page-actions-wrapper {
position: absolute;
top: 1.3rem;
right: 1rem;
z-index: 1000;
}
/* Floating Action Button */
.page-actions-button {
position: relative;
display: inline-flex;
align-items: center;
gap: 0.5rem;
background: #3f3f44;
border: 1px solid #50ffff;
color: #e8e9ed;
padding: 0.75rem 1rem;
border-radius: 6px;
font-family: 'Dank Mono', Monaco, monospace;
font-size: 0.875rem;
cursor: pointer;
transition: all 0.2s ease;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
}
.page-actions-button:hover {
background: #50ffff;
color: #070708;
transform: translateY(-2px);
box-shadow: 0 6px 16px rgba(80, 255, 255, 0.3);
}
.page-actions-button::before {
content: '▤';
font-size: 1.2rem;
line-height: 1;
}
.page-actions-button::after {
content: '▼';
font-size: 0.6rem;
transition: transform 0.2s ease;
}
.page-actions-button.active::after {
transform: rotate(180deg);
}
/* Dropdown Menu */
.page-actions-dropdown {
position: absolute;
top: 3.5rem;
right: 0;
z-index: 1001;
background: #1a1a1a;
border: 1px solid #3f3f44;
border-radius: 8px;
min-width: 280px;
opacity: 0;
visibility: hidden;
transform: translateY(-10px);
transition: all 0.2s ease;
box-shadow: 0 8px 24px rgba(0, 0, 0, 0.5);
overflow: hidden;
}
.page-actions-dropdown.active {
opacity: 1;
visibility: visible;
transform: translateY(0);
}
.page-actions-dropdown::before {
content: '';
position: absolute;
top: -8px;
right: 1.5rem;
width: 0;
height: 0;
border-left: 8px solid transparent;
border-right: 8px solid transparent;
border-bottom: 8px solid #3f3f44;
}
/* Menu Header */
.page-actions-header {
background: #3f3f44;
padding: 0.5rem 0.75rem;
border-bottom: 1px solid #50ffff;
font-family: 'Dank Mono', Monaco, monospace;
font-size: 0.7rem;
color: #a3abba;
text-transform: uppercase;
letter-spacing: 0.05em;
}
.page-actions-header::before {
content: '┌─';
margin-right: 0.5rem;
color: #50ffff;
}
/* Menu Items */
.page-actions-menu {
list-style: none;
margin: 0;
padding: 0.25rem 0;
}
.page-action-item {
display: block;
padding: 0;
}
ul>li.page-action-item::after{
content: '';
}
.page-action-link {
display: flex;
align-items: center;
gap: 0.5rem;
padding: 0.5rem 0.75rem;
color: #e8e9ed;
text-decoration: none !important;
font-family: 'Dank Mono', Monaco, monospace;
font-size: 0.8rem;
transition: all 0.15s ease;
cursor: pointer;
border-left: 3px solid transparent;
}
.page-action-link:hover:not(.disabled) {
background: #3f3f44;
border-left-color: #50ffff;
color: #50ffff;
text-decoration: none;
}
.page-action-link.disabled {
opacity: 0.5;
cursor: not-allowed;
}
.page-action-link.disabled:hover {
background: transparent;
color: #e8e9ed;
text-decoration: none;
}
/* Icons using ASCII/Terminal characters */
.page-action-icon {
font-size: 1rem;
width: 1.5rem;
text-align: center;
font-weight: bold;
color: #50ffff;
}
.page-action-link:hover:not(.disabled) .page-action-icon {
color: #50ffff;
}
.page-action-link.disabled .page-action-icon {
color: #666;
}
/* Specific icons */
.icon-copy::before {
content: '⎘'; /* Copy/duplicate symbol */
}
.icon-view::before {
content: '⎙'; /* Document symbol */
}
.icon-ai::before {
content: '⚡'; /* Lightning/AI symbol */
}
/* Action Text */
.page-action-text {
flex: 1;
}
.page-action-label {
display: block;
font-weight: 600;
margin-bottom: 0.05rem;
line-height: 1.3;
}
.page-action-description {
display: block;
font-size: 0.7rem;
color: #a3abba;
line-height: 1.2;
}
/* Badge */
/* External link indicator */
.page-action-external::after {
content: '→';
margin-left: 0.25rem;
font-size: 0.75rem;
}
/* Divider */
.page-actions-divider {
height: 1px;
background: #3f3f44;
margin: 0.25rem 0;
}
/* Success/Copy feedback */
.page-action-copied {
background: #50ff50 !important;
color: #070708 !important;
border-left-color: #50ff50 !important;
}
.page-action-copied .page-action-icon {
color: #070708 !important;
}
.page-action-copied .page-action-icon::before {
content: '✓';
}
/* Mobile Responsive */
@media (max-width: 768px) {
.page-actions-wrapper {
top: 0.5rem;
right: 0.5rem;
}
.page-actions-button {
padding: 0.6rem 0.8rem;
font-size: 0.8rem;
}
.page-actions-dropdown {
min-width: 260px;
max-width: calc(100vw - 2rem);
right: -0.5rem;
}
.page-action-link {
padding: 0.6rem 0.8rem;
font-size: 0.8rem;
}
.page-action-description {
font-size: 0.7rem;
}
}
/* Animation for tooltip/notification */
@keyframes slideInFromTop {
from {
transform: translateY(-20px);
opacity: 0;
}
to {
transform: translateY(0);
opacity: 1;
}
}
.page-actions-notification {
position: fixed;
top: calc(var(--header-height) + 0.5rem);
right: 50%;
transform: translateX(50%);
z-index: 1100;
background: #50ff50;
color: #070708;
padding: 0.75rem 1.5rem;
border-radius: 6px;
font-family: 'Dank Mono', Monaco, monospace;
font-size: 0.875rem;
font-weight: 600;
box-shadow: 0 4px 12px rgba(80, 255, 80, 0.4);
animation: slideInFromTop 0.3s ease;
pointer-events: none;
}
.page-actions-notification::before {
content: '✓ ';
margin-right: 0.5rem;
}
/* Hide on print */
@media print {
.page-actions-button,
.page-actions-dropdown {
display: none !important;
}
}
/* Overlay for mobile */
.page-actions-overlay {
display: none;
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.5);
z-index: 998;
opacity: 0;
transition: opacity 0.2s ease;
}
.page-actions-overlay.active {
display: block;
opacity: 1;
}
@media (max-width: 768px) {
.page-actions-overlay {
display: block;
}
}
/* Keyboard focus styles */
.page-action-link:focus {
outline: 2px solid #50ffff;
outline-offset: -2px;
}
.page-actions-button:focus {
outline: 2px solid #50ffff;
outline-offset: 2px;
}
/* Loading state */
.page-action-link.loading {
pointer-events: none;
opacity: 0.7;
}
.page-action-link.loading .page-action-icon::before {
content: '⟳';
animation: spin 1s linear infinite;
}
@keyframes spin {
from { transform: rotate(0deg); }
to { transform: rotate(360deg); }
}
/* Terminal-style border effect on hover */
.page-actions-dropdown:hover {
border-color: #50ffff;
}
/* Footer info */
.page-actions-footer {
background: #070708;
padding: 0.4rem 0.75rem;
border-top: 1px solid #3f3f44;
font-size: 0.65rem;
color: #666;
text-align: center;
font-family: 'Dank Mono', Monaco, monospace;
}
.page-actions-footer::before {
content: '└─';
margin-right: 0.5rem;
color: #3f3f44;
}

427
docs/md_v2/assets/page_actions.js Normal file
View File

@@ -0,0 +1,427 @@
// ==== File: assets/page_actions.js ====
// Page Actions - Copy/View Markdown functionality
document.addEventListener('DOMContentLoaded', () => {
// Configuration
const config = {
githubRepo: 'unclecode/crawl4ai',
githubBranch: 'main',
docsPath: 'docs/md_v2',
excludePaths: ['/apps/c4a-script/', '/apps/llmtxt/', '/apps/crawl4ai-assistant/', '/core/ask-ai/'], // Don't show on app pages
};
let cachedMarkdown = null;
let cachedMarkdownPath = null;
// Check if we should show the button on this page
function shouldShowButton() {
const currentPath = window.location.pathname;
// Don't show on homepage
if (currentPath === '/' || currentPath === '/index.html') {
return false;
}
// Don't show on 404 pages
if (document.title && document.title.toLowerCase().includes('404')) {
return false;
}
// Require mkdocs main content container
const mainContent = document.getElementById('terminal-mkdocs-main-content');
if (!mainContent) {
return false;
}
// Don't show on excluded paths (apps)
for (const excludePath of config.excludePaths) {
if (currentPath.includes(excludePath)) {
return false;
}
}
// Only show on documentation pages
return true;
}
if (!shouldShowButton()) {
return;
}
// Get current page markdown path
function getCurrentMarkdownPath() {
let path = window.location.pathname;
// Remove leading/trailing slashes
path = path.replace(/^\/|\/$/g, '');
// Remove .html extension if present
path = path.replace(/\.html$/, '');
// Handle root/index
if (!path || path === 'index') {
return 'index.md';
}
// Add .md extension
return `${path}.md`;
}
async function loadMarkdownContent() {
const mdPath = getCurrentMarkdownPath();
if (!mdPath) {
throw new Error('Invalid markdown path');
}
const rawUrl = getGithubRawUrl();
const response = await fetch(rawUrl);
if (!response.ok) {
throw new Error(`Failed to fetch markdown: ${response.status}`);
}
const markdown = await response.text();
cachedMarkdown = markdown;
cachedMarkdownPath = mdPath;
return markdown;
}
async function ensureMarkdownCached() {
const mdPath = getCurrentMarkdownPath();
if (!mdPath) {
return false;
}
if (cachedMarkdown && cachedMarkdownPath === mdPath) {
return true;
}
try {
await loadMarkdownContent();
return true;
} catch (error) {
console.warn('Page Actions: Markdown not available for this page.', error);
cachedMarkdown = null;
cachedMarkdownPath = null;
return false;
}
}
async function getMarkdownContent() {
const available = await ensureMarkdownCached();
if (!available) {
throw new Error('Markdown not available for this page.');
}
return cachedMarkdown;
}
// Get GitHub raw URL for current page
function getGithubRawUrl() {
const mdPath = getCurrentMarkdownPath();
return `https://raw.githubusercontent.com/${config.githubRepo}/${config.githubBranch}/${config.docsPath}/${mdPath}`;
}
// Get GitHub file URL for current page (for viewing)
function getGithubFileUrl() {
const mdPath = getCurrentMarkdownPath();
return `https://github.com/${config.githubRepo}/blob/${config.githubBranch}/${config.docsPath}/${mdPath}`;
}
// Create the UI
function createPageActionsUI() {
// Find the main content area
const mainContent = document.getElementById('terminal-mkdocs-main-content');
if (!mainContent) {
console.warn('Page Actions: Could not find #terminal-mkdocs-main-content');
return null;
}
// Create button
const button = document.createElement('button');
button.className = 'page-actions-button';
button.setAttribute('aria-label', 'Page copy');
button.setAttribute('aria-expanded', 'false');
button.innerHTML = '<span>Page Copy</span>';
// Create overlay for mobile
const overlay = document.createElement('div');
overlay.className = 'page-actions-overlay';
// Create dropdown
const dropdown = document.createElement('div');
dropdown.className = 'page-actions-dropdown';
dropdown.setAttribute('role', 'menu');
dropdown.innerHTML = `
<div class="page-actions-header">Page Copy</div>
<ul class="page-actions-menu">
<li class="page-action-item">
<a href="#" class="page-action-link" id="action-copy-markdown" role="menuitem">
<span class="page-action-icon icon-copy"></span>
<span class="page-action-text">
<span class="page-action-label">Copy as Markdown</span>
<span class="page-action-description">Copy page for LLMs</span>
</span>
</a>
</li>
<li class="page-action-item">
<a href="#" class="page-action-link page-action-external" id="action-view-markdown" target="_blank" role="menuitem">
<span class="page-action-icon icon-view"></span>
<span class="page-action-text">
<span class="page-action-label">View as Markdown</span>
<span class="page-action-description">Open raw source</span>
</span>
</a>
</li>
<div class="page-actions-divider"></div>
<li class="page-action-item">
<a href="#" class="page-action-link page-action-external" id="action-open-chatgpt" role="menuitem">
<span class="page-action-icon icon-ai"></span>
<span class="page-action-text">
<span class="page-action-label">Open in ChatGPT</span>
<span class="page-action-description">Ask questions about this page</span>
</span>
</a>
</li>
</ul>
<div class="page-actions-footer">ESC to close</div>
`;
// Create a wrapper for button and dropdown
const wrapper = document.createElement('div');
wrapper.className = 'page-actions-wrapper';
wrapper.appendChild(button);
wrapper.appendChild(dropdown);
// Inject into main content area
mainContent.appendChild(wrapper);
// Append overlay to body
document.body.appendChild(overlay);
return { button, dropdown, overlay, wrapper };
}
// Toggle dropdown
function toggleDropdown(button, dropdown, overlay) {
const isActive = dropdown.classList.contains('active');
if (isActive) {
closeDropdown(button, dropdown, overlay);
} else {
openDropdown(button, dropdown, overlay);
}
}
function openDropdown(button, dropdown, overlay) {
dropdown.classList.add('active');
// Don't activate overlay - not needed
button.classList.add('active');
button.setAttribute('aria-expanded', 'true');
}
function closeDropdown(button, dropdown, overlay) {
dropdown.classList.remove('active');
// Don't deactivate overlay - not needed
button.classList.remove('active');
button.setAttribute('aria-expanded', 'false');
}
// Show notification
function showNotification(message, duration = 2000) {
const notification = document.createElement('div');
notification.className = 'page-actions-notification';
notification.textContent = message;
document.body.appendChild(notification);
setTimeout(() => {
notification.remove();
}, duration);
}
// Copy markdown to clipboard
async function copyMarkdownToClipboard(link) {
// Add loading state
link.classList.add('loading');
try {
const markdown = await getMarkdownContent();
// Copy to clipboard
await navigator.clipboard.writeText(markdown);
// Visual feedback
link.classList.remove('loading');
link.classList.add('page-action-copied');
showNotification('Markdown copied to clipboard!');
// Reset after delay
setTimeout(() => {
link.classList.remove('page-action-copied');
}, 2000);
} catch (error) {
console.error('Error copying markdown:', error);
link.classList.remove('loading');
showNotification('Error: Could not copy markdown');
}
}
// View markdown in new tab
function viewMarkdown() {
const githubUrl = getGithubFileUrl();
window.open(githubUrl, '_blank', 'noopener,noreferrer');
}
function getCurrentPageUrl() {
const { href } = window.location;
return href.split('#')[0];
}
function openChatGPT() {
const pageUrl = getCurrentPageUrl();
const prompt = encodeURIComponent(`Read ${pageUrl} so I can ask questions about it.`);
const chatUrl = `https://chatgpt.com/?hint=search&prompt=${prompt}`;
window.open(chatUrl, '_blank', 'noopener,noreferrer');
}
(async () => {
if (!shouldShowButton()) {
return;
}
const markdownAvailable = await ensureMarkdownCached();
if (!markdownAvailable) {
return;
}
const ui = createPageActionsUI();
if (!ui) {
return;
}
const { button, dropdown, overlay } = ui;
// Event listeners
button.addEventListener('click', (e) => {
e.stopPropagation();
toggleDropdown(button, dropdown, overlay);
});
overlay.addEventListener('click', () => {
closeDropdown(button, dropdown, overlay);
});
// Copy markdown action
document.getElementById('action-copy-markdown').addEventListener('click', async (e) => {
e.preventDefault();
e.stopPropagation();
await copyMarkdownToClipboard(e.currentTarget);
});
// View markdown action
document.getElementById('action-view-markdown').addEventListener('click', (e) => {
e.preventDefault();
e.stopPropagation();
viewMarkdown();
closeDropdown(button, dropdown, overlay);
});
// Open in ChatGPT action
document.getElementById('action-open-chatgpt').addEventListener('click', (e) => {
e.preventDefault();
e.stopPropagation();
openChatGPT();
closeDropdown(button, dropdown, overlay);
});
// Close on ESC key
document.addEventListener('keydown', (e) => {
if (e.key === 'Escape' && dropdown.classList.contains('active')) {
closeDropdown(button, dropdown, overlay);
}
});
// Close when clicking outside
document.addEventListener('click', (e) => {
if (!dropdown.contains(e.target) && !button.contains(e.target)) {
closeDropdown(button, dropdown, overlay);
}
});
// Prevent dropdown from closing when clicking inside
dropdown.addEventListener('click', (e) => {
// Only stop propagation if not clicking on a link
if (!e.target.closest('.page-action-link')) {
e.stopPropagation();
}
});
// Close dropdown on link click (except for copy which handles itself)
dropdown.querySelectorAll('.page-action-link:not(#action-copy-markdown)').forEach(link => {
link.addEventListener('click', () => {
if (!link.classList.contains('disabled')) {
setTimeout(() => {
closeDropdown(button, dropdown, overlay);
}, 100);
}
});
});
// Handle window resize
let resizeTimer;
window.addEventListener('resize', () => {
clearTimeout(resizeTimer);
resizeTimer = setTimeout(() => {
// Close dropdown on resize to prevent positioning issues
if (dropdown.classList.contains('active')) {
closeDropdown(button, dropdown, overlay);
}
}, 250);
});
// Accessibility: Focus management
button.addEventListener('keydown', (e) => {
if (e.key === 'Enter' || e.key === ' ') {
e.preventDefault();
toggleDropdown(button, dropdown, overlay);
// Focus first menu item when opening
if (dropdown.classList.contains('active')) {
const firstLink = dropdown.querySelector('.page-action-link:not(.disabled)');
if (firstLink) {
setTimeout(() => firstLink.focus(), 100);
}
}
}
});
// Arrow key navigation within menu
dropdown.addEventListener('keydown', (e) => {
if (!dropdown.classList.contains('active')) return;
const links = Array.from(dropdown.querySelectorAll('.page-action-link:not(.disabled)'));
const currentIndex = links.indexOf(document.activeElement);
if (e.key === 'ArrowDown') {
e.preventDefault();
const nextIndex = (currentIndex + 1) % links.length;
links[nextIndex].focus();
} else if (e.key === 'ArrowUp') {
e.preventDefault();
const prevIndex = (currentIndex - 1 + links.length) % links.length;
links[prevIndex].focus();
} else if (e.key === 'Home') {
e.preventDefault();
links[0].focus();
} else if (e.key === 'End') {
e.preventDefault();
links[links.length - 1].focus();
}
});
console.log('Page Actions initialized for:', getCurrentMarkdownPath());
})();
});

42
docs/md_v2/blog/index.md
View File

@@ -20,17 +20,43 @@ Ever wondered why your AI coding assistant struggles with your library despite c
## Latest Release
### [Crawl4AI v0.7.6 – The Webhook Infrastructure Update](../blog/release-v0.7.6.md)
*October 22, 2025*
Crawl4AI v0.7.6 introduces comprehensive webhook support for the Docker job queue API, bringing real-time notifications to both crawling and LLM extraction workflows. No more polling!
Key highlights:
- **🪝 Complete Webhook Support**: Real-time notifications for both `/crawl/job` and `/llm/job` endpoints
- **🔄 Reliable Delivery**: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
- **🔐 Custom Authentication**: Add custom headers for webhook authentication
- **📊 Flexible Delivery**: Choose notification-only or include full data in payload
- **⚙️ Global Configuration**: Set default webhook URL in config.yml for all jobs
- **🎯 Zero Breaking Changes**: Fully backward compatible, webhooks are opt-in
[Read full release notes →](../blog/release-v0.7.6.md)
## Recent Releases
### [Crawl4AI v0.7.5 – The Docker Hooks & Security Update](../blog/release-v0.7.5.md)
*September 29, 2025*
Crawl4AI v0.7.5 introduces the powerful Docker Hooks System for complete pipeline customization, enhanced LLM integration with custom providers, HTTPS preservation for modern web security, and resolves multiple community-reported issues.
Key highlights:
- **🔧 Docker Hooks System**: Custom Python functions at 8 key pipeline points for unprecedented customization
- **🤖 Enhanced LLM Integration**: Custom providers with temperature control and base_url configuration
- **🔒 HTTPS Preservation**: Secure internal link handling for modern web applications
- **🐍 Python 3.10+ Support**: Modern language features and enhanced performance
- **🛠️ Bug Fixes**: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration
[Read full release notes →](../blog/release-v0.7.5.md)
## Recent Releases
### [Crawl4AI v0.7.4 – The Intelligent Table Extraction & Performance Update](../blog/release-v0.7.4.md)
*August 17, 2025*
Crawl4AI v0.7.4 introduces revolutionary LLM-powered table extraction with intelligent chunking, performance improvements for concurrent crawling, enhanced browser management, and critical stability fixes that make Crawl4AI more robust for production workloads.
Key highlights:
- **🚀 LLMTableExtraction**: Revolutionary table extraction with intelligent chunking for massive tables
- **⚡ Dispatcher Bug Fix**: Fixed sequential processing issue in arun_many for fast-completing tasks
- **🧹 Memory Management Refactor**: Streamlined memory utilities and better resource management
- **🔧 Browser Manager Fixes**: Resolved race conditions in concurrent page creation
- **🔗 Advanced URL Processing**: Better handling of raw URLs and base tag link resolution
Revolutionary LLM-powered table extraction with intelligent chunking, performance improvements for concurrent crawling, enhanced browser management, and critical stability fixes.
[Read full release notes →](../blog/release-v0.7.4.md)

314
docs/md_v2/blog/releases/0.7.6.md Normal file
View File

@@ -0,0 +1,314 @@
# Crawl4AI v0.7.6 Release Notes
*Release Date: October 22, 2025*
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
## 🎯 What's New
### Webhook Support for Docker Job Queue API
The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!
**Key Capabilities:**
- ✅ **Universal Webhook Support**: Both `/crawl/job` and `/llm/job` endpoints now support webhooks
- ✅ **Flexible Delivery Modes**: Choose notification-only or include full data in the webhook payload
- ✅ **Reliable Delivery**: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
- ✅ **Custom Authentication**: Add custom headers for webhook authentication
- ✅ **Global Configuration**: Set default webhook URL in `config.yml` for all jobs
- ✅ **Task Type Identification**: Distinguish between `crawl` and `llm_extraction` tasks
### How It Works
Instead of constantly checking job status:
**OLD WAY (Polling):**
```python
# Submit job
response = requests.post("http://localhost:11235/crawl/job", json=payload)
task_id = response.json()['task_id']
# Poll until complete
while True:
status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
if status.json()['status'] == 'completed':
break
time.sleep(5) # Wait and try again
```
**NEW WAY (Webhooks):**
```python
# Submit job with webhook
payload = {
"urls": ["https://example.com"],
"webhook_config": {
"webhook_url": "https://myapp.com/webhook",
"webhook_data_in_payload": True
}
}
response = requests.post("http://localhost:11235/crawl/job", json=payload)
# Done! Webhook will notify you when complete
# Your webhook handler receives the results automatically
```
### Crawl Job Webhooks
```bash
curl -X POST http://localhost:11235/crawl/job \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"browser_config": {"headless": true},
"crawler_config": {"cache_mode": "bypass"},
"webhook_config": {
"webhook_url": "https://myapp.com/webhooks/crawl-complete",
"webhook_data_in_payload": false,
"webhook_headers": {
"X-Webhook-Secret": "your-secret-token"
}
}
}'
```
### LLM Extraction Job Webhooks (NEW!)
```bash
curl -X POST http://localhost:11235/llm/job \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/article",
"q": "Extract the article title, author, and publication date",
"schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
"provider": "openai/gpt-4o-mini",
"webhook_config": {
"webhook_url": "https://myapp.com/webhooks/llm-complete",
"webhook_data_in_payload": true
}
}'
```
### Webhook Payload Structure
**Success (with data):**
```json
{
"task_id": "llm_1698765432",
"task_type": "llm_extraction",
"status": "completed",
"timestamp": "2025-10-22T10:30:00.000000+00:00",
"urls": ["https://example.com/article"],
"data": {
"extracted_content": {
"title": "Understanding Web Scraping",
"author": "John Doe",
"date": "2025-10-22"
}
}
}
```
**Failure:**
```json
{
"task_id": "crawl_abc123",
"task_type": "crawl",
"status": "failed",
"timestamp": "2025-10-22T10:30:00.000000+00:00",
"urls": ["https://example.com"],
"error": "Connection timeout after 30s"
}
```
### Simple Webhook Handler Example
```python
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route('/webhook', methods=['POST'])
def handle_webhook():
payload = request.json
task_id = payload['task_id']
task_type = payload['task_type']
status = payload['status']
if status == 'completed':
if 'data' in payload:
# Process data directly
data = payload['data']
else:
# Fetch from API
endpoint = 'crawl' if task_type == 'crawl' else 'llm'
response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
data = response.json()
# Your business logic here
print(f"Job {task_id} completed!")
elif status == 'failed':
error = payload.get('error', 'Unknown error')
print(f"Job {task_id} failed: {error}")
return jsonify({"status": "received"}), 200
app.run(port=8080)
```
## 📊 Performance Improvements
- **Reduced Server Load**: Eliminates constant polling requests
- **Lower Latency**: Instant notification vs. polling interval delay
- **Better Resource Usage**: Frees up client connections while jobs run in background
- **Scalable Architecture**: Handles high-volume crawling workflows efficiently
## 🐛 Bug Fixes
- Fixed webhook configuration serialization for Pydantic HttpUrl fields
- Improved error handling in webhook delivery service
- Enhanced Redis task storage for webhook config persistence
## 🌍 Expected Real-World Impact
### For Web Scraping Workflows
- **Reduced Costs**: Less API calls = lower bandwidth and server costs
- **Better UX**: Instant notifications improve user experience
- **Scalability**: Handle 100s of concurrent jobs without polling overhead
### For LLM Extraction Pipelines
- **Async Processing**: Submit LLM extraction jobs and move on
- **Batch Processing**: Queue multiple extractions, get notified as they complete
- **Integration**: Easy integration with workflow automation tools (Zapier, n8n, etc.)
### For Microservices
- **Event-Driven**: Perfect for event-driven microservice architectures
- **Decoupling**: Decouple job submission from result processing
- **Reliability**: Automatic retries ensure webhooks are delivered
## 🔄 Breaking Changes
**None!** This release is fully backward compatible.
- Webhook configuration is optional
- Existing code continues to work without modification
- Polling is still supported for jobs without webhook config
## 📚 Documentation
### New Documentation
- **[WEBHOOK_EXAMPLES.md](../deploy/docker/WEBHOOK_EXAMPLES.md)** - Comprehensive webhook usage guide
- **[docker_webhook_example.py](../docs/examples/docker_webhook_example.py)** - Working code examples
### Updated Documentation
- **[Docker README](../deploy/docker/README.md)** - Added webhook sections
- API documentation with webhook examples
## 🛠️ Migration Guide
No migration needed! Webhooks are opt-in:
1. **To use webhooks**: Add `webhook_config` to your job payload
2. **To keep polling**: Continue using your existing code
### Quick Start
```python
# Just add webhook_config to your existing payload
payload = {
# Your existing configuration
"urls": ["https://example.com"],
"browser_config": {...},
"crawler_config": {...},
# NEW: Add webhook configuration
"webhook_config": {
"webhook_url": "https://myapp.com/webhook",
"webhook_data_in_payload": True
}
}
```
## 🔧 Configuration
### Global Webhook Configuration (config.yml)
```yaml
webhooks:
enabled: true
default_url: "https://myapp.com/webhooks/default" # Optional
data_in_payload: false
retry:
max_attempts: 5
initial_delay_ms: 1000
max_delay_ms: 32000
timeout_ms: 30000
headers:
User-Agent: "Crawl4AI-Webhook/1.0"
```
## 🚀 Upgrade Instructions
### Docker
```bash
# Pull the latest image
docker pull unclecode/crawl4ai:0.7.6
# Or use latest tag
docker pull unclecode/crawl4ai:latest
# Run with webhook support
docker run -d \
-p 11235:11235 \
--env-file .llm.env \
--name crawl4ai \
unclecode/crawl4ai:0.7.6
```
### Python Package
```bash
pip install --upgrade crawl4ai
```
## 💡 Pro Tips
1. **Use notification-only mode** for large results - fetch data separately to avoid large webhook payloads
2. **Set custom headers** for webhook authentication and request tracking
3. **Configure global default webhook** for consistent handling across all jobs
4. **Implement idempotent webhook handlers** - same webhook may be delivered multiple times on retry
5. **Use structured schemas** with LLM extraction for predictable webhook data
## 🎬 Demo
Try the release demo:
```bash
python docs/releases_review/demo_v0.7.6.py
```
This comprehensive demo showcases:
- Crawl job webhooks (notification-only and with data)
- LLM extraction webhooks (with JSON schema support)
- Custom headers for authentication
- Webhook retry mechanism
- Real-time webhook receiver
## 🙏 Acknowledgments
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
## 📞 Support
- **Documentation**: https://docs.crawl4ai.com
- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
- **Discord**: https://discord.gg/crawl4ai
---
**Happy crawling with webhooks!** 🕷️🪝
*- unclecode*

318
docs/md_v2/blog/releases/v0.7.5.md Normal file
View File

@@ -0,0 +1,318 @@
# 🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update
*September 29, 2025 • 8 min read*
---
Today I'm releasing Crawl4AI v0.7.5—focused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.
## 🎯 What's New at a Glance
- **Docker Hooks System**: Custom Python functions at key pipeline points with function-based API
- **Function-Based Hooks**: New `hooks_to_string()` utility with Docker client auto-conversion
- **Enhanced LLM Integration**: Custom providers with temperature control
- **HTTPS Preservation**: Secure internal link handling
- **Bug Fixes**: Resolved multiple community-reported issues
- **Improved Docker Error Handling**: Better debugging and reliability
## 🔧 Docker Hooks System: Pipeline Customization
Every scraping project needs custom logic—authentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.
### Real Example: Authentication & Performance
```python
import requests
# Real working hooks for httpbin.org
hooks_config = {
"on_page_context_created": """
async def hook(page, context, **kwargs):
print("Hook: Setting up page context")
# Block images to speed up crawling
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
print("Hook: Images blocked")
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
print("Hook: Before retrieving HTML")
# Scroll to bottom to load lazy content
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
print("Hook: Scrolled to bottom")
return page
""",
"before_goto": """
async def hook(page, context, url, **kwargs):
print(f"Hook: About to navigate to {url}")
# Add custom headers
await page.set_extra_http_headers({
'X-Test-Header': 'crawl4ai-hooks-test'
})
return page
"""
}
# Test with Docker API
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {
"code": hooks_config,
"timeout": 30
}
}
response = requests.post("http://localhost:11235/crawl", json=payload)
result = response.json()
if result.get('success'):
print("✅ Hooks executed successfully!")
print(f"Content length: {len(result.get('markdown', ''))} characters")
```
**Available Hook Points:**
- `on_browser_created`: Browser setup
- `on_page_context_created`: Page context configuration
- `before_goto`: Pre-navigation setup
- `after_goto`: Post-navigation processing
- `on_user_agent_updated`: User agent changes
- `on_execution_started`: Crawl initialization
- `before_retrieve_html`: Pre-extraction processing
- `before_return_html`: Final HTML processing
### Function-Based Hooks API
Writing hooks as strings works, but lacks IDE support and type checking. v0.7.5 introduces a function-based approach with automatic conversion!
**Option 1: Using the `hooks_to_string()` Utility**
```python
from crawl4ai import hooks_to_string
import requests
# Define hooks as regular Python functions (with full IDE support!)
async def on_page_context_created(page, context, **kwargs):
"""Block images to speed up crawling"""
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
await page.set_viewport_size({"width": 1920, "height": 1080})
return page
async def before_goto(page, context, url, **kwargs):
"""Add custom headers"""
await page.set_extra_http_headers({
'X-Crawl4AI': 'v0.7.5',
'X-Custom-Header': 'my-value'
})
return page
# Convert functions to strings
hooks_code = hooks_to_string({
"on_page_context_created": on_page_context_created,
"before_goto": before_goto
})
# Use with REST API
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {"code": hooks_code, "timeout": 30}
}
response = requests.post("http://localhost:11235/crawl", json=payload)
```
**Option 2: Docker Client with Automatic Conversion (Recommended!)**
```python
from crawl4ai.docker_client import Crawl4aiDockerClient
# Define hooks as functions (same as above)
async def on_page_context_created(page, context, **kwargs):
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
return page
async def before_retrieve_html(page, context, **kwargs):
# Scroll to load lazy content
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
return page
# Use Docker client - conversion happens automatically!
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
results = await client.crawl(
urls=["https://httpbin.org/html"],
hooks={
"on_page_context_created": on_page_context_created,
"before_retrieve_html": before_retrieve_html
},
hooks_timeout=30
)
if results and results.success:
print(f"✅ Hooks executed! HTML length: {len(results.html)}")
```
**Benefits of Function-Based Hooks:**
- ✅ Full IDE support (autocomplete, syntax highlighting)
- ✅ Type checking and linting
- ✅ Easier to test and debug
- ✅ Reusable across projects
- ✅ Automatic conversion in Docker client
- ✅ No breaking changes - string hooks still work!
## 🤖 Enhanced LLM Integration
Enhanced LLM integration with custom providers, temperature control, and base URL configuration.
### Multi-Provider Support
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
# Test with different providers
async def test_llm_providers():
# OpenAI with custom temperature
openai_strategy = LLMExtractionStrategy(
provider="gemini/gemini-2.5-flash-lite",
api_token="your-api-token",
temperature=0.7, # New in v0.7.5
instruction="Summarize this page in one sentence"
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://example.com",
config=CrawlerRunConfig(extraction_strategy=openai_strategy)
)
if result.success:
print("✅ LLM extraction completed")
print(result.extracted_content)
# Docker API with enhanced LLM config
llm_payload = {
"url": "https://example.com",
"f": "llm",
"q": "Summarize this page in one sentence.",
"provider": "gemini/gemini-2.5-flash-lite",
"temperature": 0.7
}
response = requests.post("http://localhost:11235/md", json=llm_payload)
```
**New Features:**
- Custom `temperature` parameter for creativity control
- `base_url` for custom API endpoints
- Multi-provider environment variable support
- Docker API integration
## 🔒 HTTPS Preservation
**The Problem:** Modern web apps require HTTPS everywhere. When crawlers downgrade internal links from HTTPS to HTTP, authentication breaks and security warnings appear.
**Solution:** HTTPS preservation maintains secure protocols throughout crawling.
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy
async def test_https_preservation():
# Enable HTTPS preservation
url_filter = URLPatternFilter(
patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
)
config = CrawlerRunConfig(
exclude_external_links=True,
preserve_https_for_internal_links=True, # New in v0.7.5
deep_crawl_strategy=BFSDeepCrawlStrategy(
max_depth=2,
max_pages=5,
filter_chain=FilterChain([url_filter])
)
)
async with AsyncWebCrawler() as crawler:
async for result in await crawler.arun(
url="https://quotes.toscrape.com",
config=config
):
# All internal links maintain HTTPS
internal_links = [link['href'] for link in result.links['internal']]
https_links = [link for link in internal_links if link.startswith('https://')]
print(f"HTTPS links preserved: {len(https_links)}/{len(internal_links)}")
for link in https_links[:3]:
print(f" → {link}")
```
## 🛠️ Bug Fixes and Improvements
### Major Fixes
- **URL Processing**: Fixed '+' sign preservation in query parameters (#1332)
- **Proxy Configuration**: Enhanced proxy string parsing (old `proxy` parameter deprecated)
- **Docker Error Handling**: Comprehensive error messages with status codes
- **Memory Management**: Fixed leaks in long-running sessions
- **JWT Authentication**: Fixed Docker JWT validation issues (#1442)
- **Playwright Stealth**: Fixed stealth features for Playwright integration (#1481)
- **API Configuration**: Fixed config handling to prevent overriding user-provided settings (#1505)
- **Docker Filter Serialization**: Resolved JSON encoding errors in deep crawl strategy (#1419)
- **LLM Provider Support**: Fixed custom LLM provider integration for adaptive crawler (#1291)
- **Performance Issues**: Resolved backoff strategy failures and timeout handling (#989)
### Community-Reported Issues Fixed
This release addresses multiple issues reported by the community through GitHub issues and Discord discussions:
- Fixed browser configuration reference errors
- Resolved dependency conflicts with cssselect
- Improved error messaging for failed authentications
- Enhanced compatibility with various proxy configurations
- Fixed edge cases in URL normalization
### Configuration Updates
```python
# Old proxy config (deprecated)
# browser_config = BrowserConfig(proxy="http://proxy:8080")
# New enhanced proxy config
browser_config = BrowserConfig(
proxy_config={
"server": "http://proxy:8080",
"username": "optional-user",
"password": "optional-pass"
}
)
```
## 🔄 Breaking Changes
1. **Python 3.10+ Required**: Upgrade from Python 3.9
2. **Proxy Parameter Deprecated**: Use new `proxy_config` structure
3. **New Dependency**: Added `cssselect` for better CSS handling
## 🚀 Get Started
```bash
# Install latest version
pip install crawl4ai==0.7.5
# Docker deployment
docker pull unclecode/crawl4ai:latest
docker run -p 11235:11235 unclecode/crawl4ai:latest
```
**Try the Demo:**
```bash
# Run working examples
python docs/releases_review/demo_v0.7.5.py
```
**Resources:**
- 📖 Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
- 🐙 GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
- 💬 Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
- 🐦 Twitter: [@unclecode](https://x.com/unclecode)
Happy crawling! 🕷️

1371
docs/md_v2/branding/index.md Normal file
View File

File diff suppressed because it is too large Load Diff

5196
docs/md_v2/complete-sdk-reference.md Normal file
View File

File diff suppressed because it is too large Load Diff

14
docs/md_v2/core/adaptive-crawling.md
View File

@@ -108,7 +108,19 @@ config = AdaptiveConfig(
embedding_min_confidence_threshold=0.1 # Stop if completely irrelevant
)
# With custom embedding provider (e.g., OpenAI)
# With custom LLM provider for query expansion (recommended)
from crawl4ai import LLMConfig
config = AdaptiveConfig(
strategy="embedding",
embedding_llm_config=LLMConfig(
provider='openai/text-embedding-3-small',
api_token='your-api-key',
temperature=0.7
)
)
# Alternative: Dictionary format (backward compatible)
config = AdaptiveConfig(
strategy="embedding",
embedding_llm_config={

11
docs/md_v2/core/deep-crawling.md
View File

@@ -472,6 +472,17 @@ Note that for BestFirstCrawlingStrategy, score_threshold is not needed since pag
5.**Balance breadth vs. depth.** Choose your strategy wisely - BFS for comprehensive coverage, DFS for deep exploration, BestFirst for focused relevance-based crawling.
6.**Preserve HTTPS for security.** If crawling HTTPS sites that redirect to HTTP, use `preserve_https_for_internal_links=True` to maintain secure connections:
```python
config = CrawlerRunConfig(
deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2),
preserve_https_for_internal_links=True # Keep HTTPS even if server redirects to HTTP
)
```
This is especially useful for security-conscious crawling or when dealing with sites that support both protocols.
---
## 10. Summary & Next Steps

957
docs/md_v2/core/docker-deployment.md
View File

File diff suppressed because it is too large Load Diff

2
docs/md_v2/core/quickstart.md
View File

@@ -79,7 +79,7 @@ if __name__ == "__main__":
asyncio.run(main())
```
> IMPORTANT: By default cache mode is set to `CacheMode.ENABLED`. So to have fresh content, you need to set it to `CacheMode.BYPASS`
> IMPORTANT: By default cache mode is set to `CacheMode.BYPASS` to have fresh content. Set `CacheMode.ENABLED` to enable caching.
We’ll explore more advanced config in later tutorials (like enabling proxies, PDF output, multi-tab sessions, etc.). For now, just note how you pass these objects to manage crawling.

64
docs/md_v2/core/url-seeding.md
View File

@@ -102,16 +102,16 @@ async def smart_blog_crawler():
# Step 2: Configure discovery - let's find all blog posts
config = SeedingConfig(
source="sitemap", # Use the website's sitemap
pattern="*/blog/*.html", # Only blog posts
source="sitemap+cc", # Use the website's sitemap+cc
pattern="*/courses/*", # Only courses related posts
extract_head=True, # Get page metadata
max_urls=100 # Limit for this example
)
# Step 3: Discover URLs from the Python blog
print("🔍 Discovering blog posts...")
print("🔍 Discovering course posts...")
urls = await seeder.urls("realpython.com", config)
print(f"✅ Found {len(urls)} blog posts")
print(f"✅ Found {len(urls)} course posts")
# Step 4: Filter for Python tutorials (using metadata!)
tutorials = [
@@ -134,7 +134,8 @@ async def smart_blog_crawler():
async with AsyncWebCrawler() as crawler:
config = CrawlerRunConfig(
only_text=True,
word_count_threshold=300 # Only substantial articles
word_count_threshold=300, # Only substantial articles
stream=True
)
# Extract URLs and crawl them
@@ -155,7 +156,7 @@ asyncio.run(smart_blog_crawler())
**What just happened?**
1. We discovered all blog URLs from the sitemap
1. We discovered all blog URLs from the sitemap+cc
2. We filtered using metadata (no crawling needed!)
3. We crawled only the relevant tutorials
4. We saved tons of time and bandwidth
@@ -282,8 +283,8 @@ config = SeedingConfig(
live_check=True, # Verify each URL is accessible
concurrency=20 # Check 20 URLs in parallel
)
urls = await seeder.urls("example.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("example.com", config)
# Now you can filter by status
live_urls = [u for u in urls if u["status"] == "valid"]
@@ -311,8 +312,8 @@ This is where URL seeding gets really powerful. Instead of crawling entire pages
config = SeedingConfig(
extract_head=True # Extract metadata from <head> section
)
urls = await seeder.urls("example.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("example.com", config)
# Now each URL has rich metadata
for url in urls[:3]:
@@ -387,8 +388,8 @@ config = SeedingConfig(
scoring_method="bm25",
score_threshold=0.3
)
urls = await seeder.urls("example.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("example.com", config)
# URLs are scored based on:
# 1. Domain parts matching (e.g., 'python' in python.example.com)
@@ -429,8 +430,8 @@ config = SeedingConfig(
extract_head=True,
live_check=True
)
urls = await seeder.urls("blog.example.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("blog.example.com", config)
# Analyze the results
for url in urls[:5]:
@@ -488,8 +489,8 @@ config = SeedingConfig(
scoring_method="bm25", # Use BM25 algorithm
score_threshold=0.3 # Minimum relevance score
)
urls = await seeder.urls("realpython.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("realpython.com", config)
# Results are automatically sorted by relevance!
for url in urls[:5]:
@@ -511,8 +512,8 @@ config = SeedingConfig(
score_threshold=0.5,
max_urls=20
)
urls = await seeder.urls("docs.example.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("docs.example.com", config)
# The highest scoring URLs will be API docs!
```
@@ -529,8 +530,8 @@ config = SeedingConfig(
score_threshold=0.4,
pattern="*/product/*" # Combine with pattern matching
)
urls = await seeder.urls("shop.example.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("shop.example.com", config)
# Filter further by price (from metadata)
affordable = [
@@ -550,8 +551,8 @@ config = SeedingConfig(
scoring_method="bm25",
score_threshold=0.35
)
urls = await seeder.urls("technews.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("technews.com", config)
# Filter by date
from datetime import datetime, timedelta
@@ -591,8 +592,8 @@ for query in queries:
score_threshold=0.4,
max_urls=10 # Top 10 per topic
)
urls = await seeder.urls("learning-platform.com", config)
async with AsyncUrlSeeder() as seeder:
urls = await seeder.urls("learning-platform.com", config)
all_tutorials.extend(urls)
# Remove duplicates while preserving order
@@ -625,7 +626,8 @@ config = SeedingConfig(
)
# Returns a dictionary: {domain: [urls]}
results = await seeder.many_urls(domains, config)
async with AsyncUrlSeeder() as seeder:
results = await seeder.many_urls(domains, config)
# Process results
for domain, urls in results.items():
@@ -654,8 +656,8 @@ config = SeedingConfig(
pattern="*/blog/*",
max_urls=100
)
results = await seeder.many_urls(competitors, config)
async with AsyncUrlSeeder() as seeder:
results = await seeder.many_urls(competitors, config)
# Analyze content types
for domain, urls in results.items():
@@ -690,8 +692,8 @@ config = SeedingConfig(
score_threshold=0.3,
max_urls=20 # Per site
)
results = await seeder.many_urls(educational_sites, config)
async with AsyncUrlSeeder() as seeder:
results = await seeder.many_urls(educational_sites, config)
# Find the best beginner tutorials
all_tutorials = []
@@ -731,8 +733,8 @@ config = SeedingConfig(
score_threshold=0.5, # High threshold for relevance
max_urls=10
)
results = await seeder.many_urls(news_sites, config)
async with AsyncUrlSeeder() as seeder:
results = await seeder.many_urls(news_sites, config)
# Collect all mentions
mentions = []

21
docs/md_v2/index.md
View File

@@ -59,6 +59,27 @@ Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant
> **Note**: If you're looking for the old documentation, you can access it [here](https://old.docs.crawl4ai.com).
## 🆕 AI Assistant Skill Now Available!
<div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); padding: 20px; border-radius: 10px; margin: 20px 0; box-shadow: 0 4px 6px rgba(0,0,0,0.1);">
<h3 style="color: white; margin: 0 0 10px 0;">🤖 Crawl4AI Skill for Claude & AI Assistants</h3>
<p style="color: white; margin: 10px 0;">Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:</p>
<ul style="color: white; margin: 10px 0;">
<li>📚 Complete SDK reference (23K+ words)</li>
<li>🚀 Ready-to-use extraction scripts</li>
<li>⚡ Schema generation for efficient scraping</li>
<li>🔧 Version 0.7.4 compatible</li>
</ul>
<div style="text-align: center; margin-top: 15px;">
<a href="assets/crawl4ai-skill.zip" download style="background: white; color: #667eea; padding: 12px 30px; border-radius: 5px; text-decoration: none; font-weight: bold; display: inline-block; transition: transform 0.2s;">
📦 Download Skill Package
</a>
</div>
<p style="color: white; margin: 15px 0 0 0; font-size: 0.9em; text-align: center;">
Works with Claude, Cursor, Windsurf, and other AI coding assistants. Import the .zip file into your AI assistant's skill/knowledge system.
</p>
</div>
## 🎯 New: Adaptive Web Crawling
Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your query.

66
docs/md_v2/marketplace/README.md Normal file
View File

@@ -0,0 +1,66 @@
# Crawl4AI Marketplace
A terminal-themed marketplace for tools, integrations, and resources related to Crawl4AI.
## Setup
### Backend
1. Install dependencies:
```bash
cd backend
pip install -r requirements.txt
```
2. Generate dummy data:
```bash
python dummy_data.py
```
3. Run the server:
```bash
python server.py
```
The API will be available at http://localhost:8100
### Frontend
1. Open `frontend/index.html` in your browser
2. Or serve via MkDocs as part of the documentation site
## Database Schema
The marketplace uses SQLite with automatic migration from `schema.yaml`. Tables include:
- **apps**: Tools and integrations
- **articles**: Reviews, tutorials, and news
- **categories**: App categories
- **sponsors**: Sponsored content
## API Endpoints
- `GET /api/apps` - List apps with filters
- `GET /api/articles` - List articles
- `GET /api/categories` - Get all categories
- `GET /api/sponsors` - Get active sponsors
- `GET /api/search?q=query` - Search across content
- `GET /api/stats` - Marketplace statistics
## Features
- **Smart caching**: LocalStorage with TTL (1 hour)
- **Terminal theme**: Consistent with Crawl4AI branding
- **Responsive design**: Works on all devices
- **Fast search**: Debounced with 300ms delay
- **CORS protected**: Only crawl4ai.com and localhost
## Admin Panel
Coming soon - for now, edit the database directly or modify `dummy_data.py`
## Deployment
For production deployment on EC2:
1. Update `API_BASE` in `marketplace.js` to production URL
2. Run FastAPI with proper production settings (use gunicorn/uvicorn)
3. Set up nginx proxy if needed

759
docs/md_v2/marketplace/admin/admin.css Normal file
View File

@@ -0,0 +1,759 @@
/* Admin Dashboard - C4AI Terminal Style */
/* Utility Classes */
.hidden {
display: none !important;
}
/* Brand Colors */
:root {
--c4ai-cyan: #50ffff;
--c4ai-green: #50ff50;
--c4ai-yellow: #ffff50;
--c4ai-pink: #ff50ff;
--c4ai-blue: #5050ff;
}
.admin-container {
min-height: 100vh;
background: var(--bg-dark);
}
/* Login Screen */
.login-screen {
min-height: 100vh;
display: flex;
align-items: center;
justify-content: center;
background: linear-gradient(135deg, #070708 0%, #1a1a2e 100%);
}
.login-box {
background: var(--bg-secondary);
border: 2px solid var(--primary-cyan);
padding: 3rem;
width: 400px;
box-shadow: 0 0 40px rgba(80, 255, 255, 0.2);
text-align: center;
}
.login-logo {
height: 60px;
margin-bottom: 2rem;
filter: brightness(1.2);
}
.login-box h1 {
color: var(--primary-cyan);
font-size: 1.5rem;
margin-bottom: 2rem;
}
#login-form input {
width: 100%;
padding: 0.75rem;
background: var(--bg-dark);
border: 1px solid var(--border-color);
color: var(--text-primary);
font-family: inherit;
margin-bottom: 1rem;
}
#login-form input:focus {
outline: none;
border-color: var(--primary-cyan);
}
#login-form button {
width: 100%;
padding: 0.75rem;
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
border: none;
color: var(--bg-dark);
font-weight: 600;
cursor: pointer;
transition: all 0.2s;
}
#login-form button:hover {
box-shadow: 0 4px 15px rgba(80, 255, 255, 0.3);
transform: translateY(-2px);
}
.error-msg {
color: var(--error);
font-size: 0.875rem;
margin-top: 1rem;
}
/* Admin Dashboard */
.admin-dashboard.hidden {
display: none;
}
.admin-header {
background: var(--bg-secondary);
border-bottom: 2px solid var(--primary-cyan);
padding: 1rem 0;
}
.header-content {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem;
display: flex;
justify-content: space-between;
align-items: center;
}
.header-left {
display: flex;
align-items: center;
gap: 1rem;
}
.header-logo {
height: 35px;
}
.admin-header h1 {
font-size: 1.25rem;
color: var(--primary-cyan);
}
.header-right {
display: flex;
align-items: center;
gap: 2rem;
}
.admin-user {
color: var(--text-secondary);
}
.logout-btn {
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--error);
color: var(--error);
cursor: pointer;
transition: all 0.2s;
}
.logout-btn:hover {
background: rgba(255, 60, 116, 0.1);
}
/* Layout */
.admin-layout {
display: flex;
max-width: 1800px;
margin: 0 auto;
min-height: calc(100vh - 60px);
}
/* Sidebar */
.admin-sidebar {
width: 250px;
background: var(--bg-secondary);
border-right: 1px solid var(--border-color);
display: flex;
flex-direction: column;
justify-content: space-between;
}
.sidebar-nav {
padding: 1rem 0;
}
.nav-btn {
width: 100%;
padding: 1rem 1.5rem;
background: transparent;
border: none;
border-left: 3px solid transparent;
color: var(--text-secondary);
text-align: left;
cursor: pointer;
transition: all 0.2s;
display: flex;
align-items: center;
gap: 0.75rem;
}
.nav-btn:hover {
background: rgba(80, 255, 255, 0.05);
color: var(--primary-cyan);
}
.nav-btn.active {
border-left-color: var(--primary-cyan);
background: rgba(80, 255, 255, 0.1);
color: var(--primary-cyan);
}
.nav-icon {
font-size: 1.25rem;
margin-right: 0.25rem;
display: inline-block;
width: 1.5rem;
text-align: center;
}
.nav-btn[data-section="stats"] .nav-icon {
color: var(--c4ai-cyan);
}
.nav-btn[data-section="apps"] .nav-icon {
color: var(--c4ai-green);
}
.nav-btn[data-section="articles"] .nav-icon {
color: var(--c4ai-yellow);
}
.nav-btn[data-section="categories"] .nav-icon {
color: var(--c4ai-pink);
}
.nav-btn[data-section="sponsors"] .nav-icon {
color: var(--c4ai-blue);
}
.sidebar-actions {
padding: 1rem;
border-top: 1px solid var(--border-color);
}
.action-btn {
width: 100%;
padding: 0.75rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
color: var(--text-secondary);
cursor: pointer;
margin-bottom: 0.5rem;
transition: all 0.2s;
}
.action-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
/* Main Content */
.admin-main {
flex: 1;
padding: 2rem;
overflow-y: auto;
}
.content-section {
display: none;
}
.content-section.active {
display: block;
}
/* Stats Grid */
.stats-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1.5rem;
margin-bottom: 3rem;
}
.stat-card {
background: linear-gradient(135deg, rgba(80, 255, 255, 0.03), rgba(243, 128, 245, 0.02));
border: 1px solid rgba(80, 255, 255, 0.3);
padding: 1.5rem;
display: flex;
gap: 1.5rem;
}
.stat-icon {
font-size: 2rem;
width: 3rem;
height: 3rem;
display: flex;
align-items: center;
justify-content: center;
border: 2px solid;
border-radius: 4px;
}
.stat-card:nth-child(1) .stat-icon {
color: var(--c4ai-cyan);
border-color: var(--c4ai-cyan);
}
.stat-card:nth-child(2) .stat-icon {
color: var(--c4ai-green);
border-color: var(--c4ai-green);
}
.stat-card:nth-child(3) .stat-icon {
color: var(--c4ai-yellow);
border-color: var(--c4ai-yellow);
}
.stat-card:nth-child(4) .stat-icon {
color: var(--c4ai-pink);
border-color: var(--c4ai-pink);
}
.stat-number {
font-size: 2rem;
color: var(--primary-cyan);
font-weight: 600;
}
.stat-label {
color: var(--text-secondary);
}
.stat-detail {
font-size: 0.875rem;
color: var(--text-tertiary);
margin-top: 0.5rem;
}
/* Quick Actions */
.quick-actions {
display: flex;
gap: 1rem;
}
.quick-btn {
padding: 0.75rem 1.5rem;
background: transparent;
border: 1px solid var(--primary-cyan);
color: var(--primary-cyan);
cursor: pointer;
transition: all 0.2s;
}
.quick-btn:hover {
background: rgba(80, 255, 255, 0.1);
transform: translateY(-2px);
}
/* Section Headers */
.section-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 2rem;
}
.section-header h2 {
font-size: 1.5rem;
color: var(--text-primary);
}
.header-actions {
display: flex;
gap: 1rem;
}
.search-input {
padding: 0.5rem 1rem;
background: var(--bg-dark);
border: 1px solid var(--border-color);
color: var(--text-primary);
width: 250px;
}
.search-input:focus {
outline: none;
border-color: var(--primary-cyan);
}
.filter-select {
padding: 0.5rem;
background: var(--bg-dark);
border: 1px solid var(--border-color);
color: var(--text-primary);
}
.add-btn {
padding: 0.5rem 1rem;
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
border: none;
color: var(--bg-dark);
font-weight: 600;
cursor: pointer;
transition: all 0.2s;
}
.add-btn:hover {
box-shadow: 0 4px 15px rgba(80, 255, 255, 0.3);
transform: translateY(-2px);
}
/* Data Tables */
.data-table {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
overflow-x: auto;
}
.data-table table {
width: 100%;
border-collapse: collapse;
}
.data-table th {
background: var(--bg-tertiary);
padding: 1rem;
text-align: left;
color: var(--primary-cyan);
font-weight: 600;
border-bottom: 2px solid var(--border-color);
position: sticky;
top: 0;
z-index: 10;
}
.data-table td {
padding: 1rem;
border-bottom: 1px solid var(--border-color);
}
.data-table tr:hover {
background: rgba(80, 255, 255, 0.03);
}
/* Table Actions */
.table-actions {
display: flex;
gap: 0.5rem;
}
.table-logo {
width: 48px;
height: 48px;
object-fit: contain;
border-radius: 6px;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
padding: 4px;
}
.btn-edit, .btn-delete, .btn-duplicate {
padding: 0.25rem 0.5rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
cursor: pointer;
font-size: 0.875rem;
}
.btn-edit:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
.btn-delete:hover {
border-color: var(--error);
color: var(--error);
}
.btn-duplicate:hover {
border-color: var(--accent-pink);
color: var(--accent-pink);
}
/* Badges in Tables */
.badge {
padding: 0.25rem 0.5rem;
font-size: 0.75rem;
text-transform: uppercase;
}
.badge.featured {
background: var(--primary-cyan);
color: var(--bg-dark);
}
.badge.sponsored {
background: var(--warning);
color: var(--bg-dark);
}
.badge.active {
background: var(--success);
color: var(--bg-dark);
}
/* Modal Enhancements */
.modal-content.large {
max-width: 1000px;
width: 90%;
max-height: 90vh;
}
.modal-header {
display: flex;
justify-content: space-between;
align-items: center;
padding: 1.5rem;
border-bottom: 1px solid var(--border-color);
}
.modal-body {
padding: 1.5rem;
overflow-y: auto;
max-height: calc(90vh - 140px);
}
.modal-footer {
display: flex;
justify-content: flex-end;
gap: 1rem;
padding: 1rem 1.5rem;
border-top: 1px solid var(--border-color);
}
.btn-cancel, .btn-save {
padding: 0.5rem 1.5rem;
cursor: pointer;
transition: all 0.2s;
}
.btn-cancel {
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
}
.btn-cancel:hover {
border-color: var(--error);
color: var(--error);
}
.btn-save {
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
border: none;
color: var(--bg-dark);
font-weight: 600;
}
.btn-save:hover {
box-shadow: 0 4px 15px rgba(80, 255, 255, 0.3);
}
/* Form Styles */
.form-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 1.5rem;
}
.form-group {
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.form-group label {
color: var(--text-secondary);
font-size: 0.875rem;
}
.form-group input,
.form-group select,
.form-group textarea {
padding: 0.5rem;
background: var(--bg-dark);
border: 1px solid var(--border-color);
color: var(--text-primary);
font-family: inherit;
}
.form-group input:focus,
.form-group select:focus,
.form-group textarea:focus {
outline: none;
border-color: var(--primary-cyan);
}
.form-group.full-width {
grid-column: 1 / -1;
}
.checkbox-group {
display: flex;
gap: 2rem;
}
.checkbox-label {
display: flex;
align-items: center;
gap: 0.5rem;
cursor: pointer;
}
.sponsor-form {
grid-template-columns: 200px repeat(2, minmax(220px, 1fr));
align-items: flex-start;
grid-auto-flow: dense;
}
.sponsor-logo-group {
grid-row: span 3;
display: flex;
flex-direction: column;
gap: 0.75rem;
}
.span-two {
grid-column: span 2;
}
.logo-upload {
position: relative;
width: 180px;
}
.image-preview {
width: 180px;
height: 180px;
border: 1px dashed var(--border-color);
border-radius: 12px;
display: flex;
align-items: center;
justify-content: center;
background: var(--bg-tertiary);
overflow: hidden;
}
.image-preview.empty {
color: var(--text-secondary);
font-size: 0.75rem;
text-align: center;
padding: 0.75rem;
}
.image-preview img {
max-width: 100%;
max-height: 100%;
object-fit: contain;
}
.upload-btn {
position: absolute;
left: 50%;
bottom: 12px;
transform: translateX(-50%);
padding: 0.35rem 1rem;
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
border: none;
border-radius: 999px;
color: var(--bg-dark);
font-size: 0.75rem;
font-weight: 600;
cursor: pointer;
box-shadow: 0 6px 18px rgba(80, 255, 255, 0.25);
}
.upload-btn:hover {
box-shadow: 0 8px 22px rgba(80, 255, 255, 0.35);
}
.logo-upload input[type="file"] {
display: none;
}
.upload-hint {
font-size: 0.75rem;
color: var(--text-secondary);
margin: 0;
}
@media (max-width: 960px) {
.sponsor-form {
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
}
.sponsor-logo-group {
grid-column: 1 / -1;
grid-row: auto;
flex-direction: row;
align-items: center;
gap: 1.5rem;
}
.logo-upload {
width: 160px;
}
.span-two {
grid-column: 1 / -1;
}
}
/* Rich Text Editor */
.editor-toolbar {
display: flex;
gap: 0.5rem;
padding: 0.5rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
border-bottom: none;
}
.editor-btn {
padding: 0.25rem 0.5rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
cursor: pointer;
}
.editor-btn:hover {
background: rgba(80, 255, 255, 0.1);
border-color: var(--primary-cyan);
}
.editor-content {
min-height: 300px;
padding: 1rem;
background: var(--bg-dark);
border: 1px solid var(--border-color);
font-family: 'Dank Mono', Monaco, monospace;
}
/* Responsive */
@media (max-width: 1024px) {
.admin-layout {
flex-direction: column;
}
.admin-sidebar {
width: 100%;
border-right: none;
border-bottom: 1px solid var(--border-color);
}
.sidebar-nav {
display: flex;
overflow-x: auto;
padding: 0;
}
.nav-btn {
border-left: none;
border-bottom: 3px solid transparent;
white-space: nowrap;
}
.nav-btn.active {
border-bottom-color: var(--primary-cyan);
}
.sidebar-actions {
display: none;
}
}

920
docs/md_v2/marketplace/admin/admin.js Normal file
View File

@@ -0,0 +1,920 @@
// Admin Dashboard - Smart & Powerful
const { API_BASE, API_ORIGIN } = (() => {
const cleanOrigin = (value) => value ? value.replace(/\/$/, '') : '';
const params = new URLSearchParams(window.location.search);
const overrideParam = cleanOrigin(params.get('api_origin'));
let storedOverride = '';
try {
storedOverride = cleanOrigin(localStorage.getItem('marketplace_api_origin'));
} catch (error) {
storedOverride = '';
}
let origin = overrideParam || storedOverride;
if (overrideParam && overrideParam !== storedOverride) {
try {
localStorage.setItem('marketplace_api_origin', overrideParam);
} catch (error) {
// ignore storage errors (private mode, etc.)
}
}
const { protocol, hostname, port } = window.location;
const isLocalHost = ['localhost', '127.0.0.1', '0.0.0.0'].includes(hostname);
if (!origin && isLocalHost && port !== '8100') {
origin = `${protocol}//127.0.0.1:8100`;
}
if (origin) {
const normalized = cleanOrigin(origin);
return { API_BASE: `${normalized}/marketplace/api`, API_ORIGIN: normalized };
}
return { API_BASE: '/marketplace/api', API_ORIGIN: '' };
})();
const resolveAssetUrl = (path) => {
if (!path) return '';
if (/^https?:\/\//i.test(path)) return path;
if (path.startsWith('/') && API_ORIGIN) {
return `${API_ORIGIN}${path}`;
}
return path;
};
class AdminDashboard {
constructor() {
this.token = localStorage.getItem('admin_token');
this.currentSection = 'stats';
this.data = {
apps: [],
articles: [],
categories: [],
sponsors: []
};
this.editingItem = null;
this.init();
}
async init() {
// Check auth
if (!this.token) {
this.showLogin();
return;
}
// Try to load stats to verify token
try {
await this.loadStats();
this.showDashboard();
this.setupEventListeners();
await this.loadAllData();
} catch (error) {
if (error.status === 401) {
this.showLogin();
}
}
}
showLogin() {
document.getElementById('login-screen').classList.remove('hidden');
document.getElementById('admin-dashboard').classList.add('hidden');
// Set up login button click handler
const loginBtn = document.getElementById('login-btn');
if (loginBtn) {
loginBtn.onclick = async () => {
const password = document.getElementById('password').value;
await this.login(password);
};
}
}
async login(password) {
try {
const response = await fetch(`${API_BASE}/admin/login`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ password })
});
if (!response.ok) throw new Error('Invalid password');
const data = await response.json();
this.token = data.token;
localStorage.setItem('admin_token', this.token);
document.getElementById('login-screen').classList.add('hidden');
this.showDashboard();
this.setupEventListeners();
await this.loadAllData();
} catch (error) {
document.getElementById('login-error').textContent = 'Invalid password';
document.getElementById('password').value = '';
}
}
showDashboard() {
document.getElementById('login-screen').classList.add('hidden');
document.getElementById('admin-dashboard').classList.remove('hidden');
}
setupEventListeners() {
// Navigation
document.querySelectorAll('.nav-btn').forEach(btn => {
btn.onclick = () => this.switchSection(btn.dataset.section);
});
// Logout
document.getElementById('logout-btn').onclick = () => this.logout();
// Export/Backup
document.getElementById('export-btn').onclick = () => this.exportData();
document.getElementById('backup-btn').onclick = () => this.backupDatabase();
// Search
['apps', 'articles'].forEach(type => {
const searchInput = document.getElementById(`${type}-search`);
if (searchInput) {
searchInput.oninput = (e) => this.filterTable(type, e.target.value);
}
});
// Category filter
const categoryFilter = document.getElementById('apps-filter');
if (categoryFilter) {
categoryFilter.onchange = (e) => this.filterByCategory(e.target.value);
}
// Save button in modal
document.getElementById('save-btn').onclick = () => this.saveItem();
}
async loadAllData() {
try {
await this.loadStats();
} catch (e) {
console.error('Failed to load stats:', e);
}
try {
await this.loadApps();
} catch (e) {
console.error('Failed to load apps:', e);
}
try {
await this.loadArticles();
} catch (e) {
console.error('Failed to load articles:', e);
}
try {
await this.loadCategories();
} catch (e) {
console.error('Failed to load categories:', e);
}
try {
await this.loadSponsors();
} catch (e) {
console.error('Failed to load sponsors:', e);
}
this.populateCategoryFilter();
}
async apiCall(endpoint, options = {}) {
const isFormData = options.body instanceof FormData;
const headers = {
'Authorization': `Bearer ${this.token}`,
...options.headers
};
if (!isFormData && !headers['Content-Type']) {
headers['Content-Type'] = 'application/json';
}
const response = await fetch(`${API_BASE}${endpoint}`, {
...options,
headers
});
if (response.status === 401) {
this.logout();
throw { status: 401 };
}
if (!response.ok) throw new Error(`API Error: ${response.status}`);
return response.json();
}
async loadStats() {
const stats = await this.apiCall(`/admin/stats?_=${Date.now()}`, {
cache: 'no-store'
});
document.getElementById('stat-apps').textContent = stats.apps.total;
document.getElementById('stat-featured').textContent = stats.apps.featured;
document.getElementById('stat-sponsored').textContent = stats.apps.sponsored;
document.getElementById('stat-articles').textContent = stats.articles;
document.getElementById('stat-sponsors').textContent = stats.sponsors.active;
document.getElementById('stat-views').textContent = this.formatNumber(stats.total_views);
}
async loadApps() {
this.data.apps = await this.apiCall(`/apps?limit=100&_=${Date.now()}`, {
cache: 'no-store'
});
this.renderAppsTable(this.data.apps);
}
async loadArticles() {
this.data.articles = await this.apiCall(`/articles?limit=100&_=${Date.now()}`, {
cache: 'no-store'
});
this.renderArticlesTable(this.data.articles);
}
async loadCategories() {
const cacheBuster = Date.now();
this.data.categories = await this.apiCall(`/categories?_=${cacheBuster}`, {
cache: 'no-store'
});
this.renderCategoriesTable(this.data.categories);
}
async loadSponsors() {
const cacheBuster = Date.now();
this.data.sponsors = await this.apiCall(`/sponsors?limit=100&_=${cacheBuster}`, {
cache: 'no-store'
});
this.renderSponsorsTable(this.data.sponsors);
}
renderAppsTable(apps) {
const table = document.getElementById('apps-table');
table.innerHTML = `
<table>
<thead>
<tr>
<th>ID</th>
<th>Name</th>
<th>Category</th>
<th>Type</th>
<th>Rating</th>
<th>Downloads</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
${apps.map(app => `
<tr>
<td>${app.id}</td>
<td>${app.name}</td>
<td>${app.category}</td>
<td>${app.type}</td>
<td>◆ ${app.rating}/5</td>
<td>${this.formatNumber(app.downloads)}</td>
<td>
${app.featured ? '<span class="badge featured">Featured</span>' : ''}
${app.sponsored ? '<span class="badge sponsored">Sponsored</span>' : ''}
</td>
<td>
<div class="table-actions">
<button class="btn-edit" onclick="admin.editItem('apps', ${app.id})">Edit</button>
<button class="btn-duplicate" onclick="admin.duplicateItem('apps', ${app.id})">Duplicate</button>
<button class="btn-delete" onclick="admin.deleteItem('apps', ${app.id})">Delete</button>
</div>
</td>
</tr>
`).join('')}
</tbody>
</table>
`;
}
renderArticlesTable(articles) {
const table = document.getElementById('articles-table');
table.innerHTML = `
<table>
<thead>
<tr>
<th>ID</th>
<th>Title</th>
<th>Category</th>
<th>Author</th>
<th>Published</th>
<th>Views</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
${articles.map(article => `
<tr>
<td>${article.id}</td>
<td>${article.title}</td>
<td>${article.category}</td>
<td>${article.author}</td>
<td>${new Date(article.published_date).toLocaleDateString()}</td>
<td>${this.formatNumber(article.views)}</td>
<td>
<div class="table-actions">
<button class="btn-edit" onclick="admin.editItem('articles', ${article.id})">Edit</button>
<button class="btn-duplicate" onclick="admin.duplicateItem('articles', ${article.id})">Duplicate</button>
<button class="btn-delete" onclick="admin.deleteItem('articles', ${article.id})">Delete</button>
</div>
</td>
</tr>
`).join('')}
</tbody>
</table>
`;
}
renderCategoriesTable(categories) {
const table = document.getElementById('categories-table');
table.innerHTML = `
<table>
<thead>
<tr>
<th>Order</th>
<th>Icon</th>
<th>Name</th>
<th>Description</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
${categories.map(cat => `
<tr>
<td>${cat.order_index}</td>
<td>${cat.icon}</td>
<td>${cat.name}</td>
<td>${cat.description}</td>
<td>
<div class="table-actions">
<button class="btn-edit" onclick="admin.editItem('categories', ${cat.id})">Edit</button>
<button class="btn-delete" onclick="admin.deleteCategory(${cat.id})">Delete</button>
</div>
</td>
</tr>
`).join('')}
</tbody>
</table>
`;
}
renderSponsorsTable(sponsors) {
const table = document.getElementById('sponsors-table');
table.innerHTML = `
<table>
<thead>
<tr>
<th>ID</th>
<th>Logo</th>
<th>Company</th>
<th>Tier</th>
<th>Start</th>
<th>End</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
${sponsors.map(sponsor => `
<tr>
<td>${sponsor.id}</td>
<td>${sponsor.logo_url ? `<img class="table-logo" src="${resolveAssetUrl(sponsor.logo_url)}" alt="${sponsor.company_name} logo">` : '-'}</td>
<td>${sponsor.company_name}</td>
<td>${sponsor.tier}</td>
<td>${new Date(sponsor.start_date).toLocaleDateString()}</td>
<td>${new Date(sponsor.end_date).toLocaleDateString()}</td>
<td>${sponsor.active ? '<span class="badge active">Active</span>' : 'Inactive'}</td>
<td>
<div class="table-actions">
<button class="btn-edit" onclick="admin.editItem('sponsors', ${sponsor.id})">Edit</button>
<button class="btn-delete" onclick="admin.deleteItem('sponsors', ${sponsor.id})">Delete</button>
</div>
</td>
</tr>
`).join('')}
</tbody>
</table>
`;
}
showAddForm(type) {
this.editingItem = null;
this.showModal(type, null);
}
async editItem(type, id) {
const item = this.data[type].find(i => i.id === id);
if (item) {
this.editingItem = item;
this.showModal(type, item);
}
}
async duplicateItem(type, id) {
const item = this.data[type].find(i => i.id === id);
if (item) {
const newItem = { ...item };
delete newItem.id;
newItem.name = `${newItem.name || newItem.title} (Copy)`;
if (newItem.slug) newItem.slug = `${newItem.slug}-copy-${Date.now()}`;
this.editingItem = null;
this.showModal(type, newItem);
}
}
showModal(type, item) {
const modal = document.getElementById('form-modal');
const title = document.getElementById('modal-title');
const body = document.getElementById('modal-body');
title.textContent = item ? `Edit ${type.slice(0, -1)}` : `Add New ${type.slice(0, -1)}`;
if (type === 'apps') {
body.innerHTML = this.getAppForm(item);
} else if (type === 'articles') {
body.innerHTML = this.getArticleForm(item);
} else if (type === 'categories') {
body.innerHTML = this.getCategoryForm(item);
} else if (type === 'sponsors') {
body.innerHTML = this.getSponsorForm(item);
}
modal.classList.remove('hidden');
modal.dataset.type = type;
if (type === 'sponsors') {
this.setupLogoUploadHandlers();
}
}
getAppForm(app) {
return `
<div class="form-grid">
<div class="form-group">
<label>Name *</label>
<input type="text" id="form-name" value="${app?.name || ''}" required>
</div>
<div class="form-group">
<label>Slug</label>
<input type="text" id="form-slug" value="${app?.slug || ''}" placeholder="auto-generated">
</div>
<div class="form-group">
<label>Category</label>
<select id="form-category">
${this.data.categories.map(cat =>
`<option value="${cat.name}" ${app?.category === cat.name ? 'selected' : ''}>${cat.name}</option>`
).join('')}
</select>
</div>
<div class="form-group">
<label>Type</label>
<select id="form-type">
<option value="Open Source" ${app?.type === 'Open Source' ? 'selected' : ''}>Open Source</option>
<option value="Paid" ${app?.type === 'Paid' ? 'selected' : ''}>Paid</option>
<option value="Freemium" ${app?.type === 'Freemium' ? 'selected' : ''}>Freemium</option>
</select>
</div>
<div class="form-group">
<label>Rating</label>
<input type="number" id="form-rating" value="${app?.rating || 4.5}" min="0" max="5" step="0.1">
</div>
<div class="form-group">
<label>Downloads</label>
<input type="number" id="form-downloads" value="${app?.downloads || 0}">
</div>
<div class="form-group full-width">
<label>Description</label>
<textarea id="form-description" rows="3">${app?.description || ''}</textarea>
</div>
<div class="form-group full-width">
<label>Image URL</label>
<input type="text" id="form-image" value="${app?.image || ''}" placeholder="https://...">
</div>
<div class="form-group">
<label>Website URL</label>
<input type="text" id="form-website" value="${app?.website_url || ''}">
</div>
<div class="form-group">
<label>GitHub URL</label>
<input type="text" id="form-github" value="${app?.github_url || ''}">
</div>
<div class="form-group">
<label>Pricing</label>
<input type="text" id="form-pricing" value="${app?.pricing || 'Free'}">
</div>
<div class="form-group">
<label>Contact Email</label>
<input type="email" id="form-email" value="${app?.contact_email || ''}">
</div>
<div class="form-group full-width checkbox-group">
<label class="checkbox-label">
<input type="checkbox" id="form-featured" ${app?.featured ? 'checked' : ''}>
Featured
</label>
<label class="checkbox-label">
<input type="checkbox" id="form-sponsored" ${app?.sponsored ? 'checked' : ''}>
Sponsored
</label>
</div>
<div class="form-group full-width">
<label>Integration Guide</label>
<textarea id="form-integration" rows="10">${app?.integration_guide || ''}</textarea>
</div>
</div>
`;
}
getArticleForm(article) {
return `
<div class="form-grid">
<div class="form-group full-width">
<label>Title *</label>
<input type="text" id="form-title" value="${article?.title || ''}" required>
</div>
<div class="form-group">
<label>Author</label>
<input type="text" id="form-author" value="${article?.author || 'Crawl4AI Team'}">
</div>
<div class="form-group">
<label>Category</label>
<select id="form-category">
<option value="News" ${article?.category === 'News' ? 'selected' : ''}>News</option>
<option value="Tutorial" ${article?.category === 'Tutorial' ? 'selected' : ''}>Tutorial</option>
<option value="Review" ${article?.category === 'Review' ? 'selected' : ''}>Review</option>
<option value="Comparison" ${article?.category === 'Comparison' ? 'selected' : ''}>Comparison</option>
</select>
</div>
<div class="form-group full-width">
<label>Featured Image URL</label>
<input type="text" id="form-image" value="${article?.featured_image || ''}">
</div>
<div class="form-group full-width">
<label>Content</label>
<textarea id="form-content" rows="20">${article?.content || ''}</textarea>
</div>
</div>
`;
}
getCategoryForm(category) {
return `
<div class="form-grid">
<div class="form-group">
<label>Name *</label>
<input type="text" id="form-name" value="${category?.name || ''}" required>
</div>
<div class="form-group">
<label>Icon</label>
<input type="text" id="form-icon" value="${category?.icon || '📁'}" maxlength="2">
</div>
<div class="form-group">
<label>Order</label>
<input type="number" id="form-order" value="${category?.order_index || 0}">
</div>
<div class="form-group full-width">
<label>Description</label>
<textarea id="form-description" rows="3">${category?.description || ''}</textarea>
</div>
</div>
`;
}
getSponsorForm(sponsor) {
const existingFile = sponsor?.logo_url ? sponsor.logo_url.split('/').pop().split('?')[0] : '';
return `
<div class="form-grid sponsor-form">
<div class="form-group sponsor-logo-group">
<label>Logo</label>
<input type="hidden" id="form-logo-url" value="${sponsor?.logo_url || ''}">
<div class="logo-upload">
<div class="image-preview ${sponsor?.logo_url ? '' : 'empty'}" id="form-logo-preview">
${sponsor?.logo_url ? `<img src="${resolveAssetUrl(sponsor.logo_url)}" alt="Logo preview">` : '<span>No logo uploaded</span>'}
</div>
<button type="button" class="upload-btn" id="form-logo-button">Upload Logo</button>
<input type="file" id="form-logo-file" accept="image/png,image/jpeg,image/webp,image/svg+xml" hidden>
</div>
<p class="upload-hint" id="form-logo-filename">${existingFile ? `Current: ${existingFile}` : 'No file selected'}</p>
</div>
<div class="form-group span-two">
<label>Company Name *</label>
<input type="text" id="form-name" value="${sponsor?.company_name || ''}" required>
</div>
<div class="form-group">
<label>Tier</label>
<select id="form-tier">
<option value="Bronze" ${sponsor?.tier === 'Bronze' ? 'selected' : ''}>Bronze</option>
<option value="Silver" ${sponsor?.tier === 'Silver' ? 'selected' : ''}>Silver</option>
<option value="Gold" ${sponsor?.tier === 'Gold' ? 'selected' : ''}>Gold</option>
</select>
</div>
<div class="form-group">
<label>Landing URL</label>
<input type="text" id="form-landing" value="${sponsor?.landing_url || ''}">
</div>
<div class="form-group">
<label>Banner URL</label>
<input type="text" id="form-banner" value="${sponsor?.banner_url || ''}">
</div>
<div class="form-group">
<label>Start Date</label>
<input type="date" id="form-start" value="${sponsor?.start_date?.split('T')[0] || ''}">
</div>
<div class="form-group">
<label>End Date</label>
<input type="date" id="form-end" value="${sponsor?.end_date?.split('T')[0] || ''}">
</div>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" id="form-active" ${sponsor?.active ? 'checked' : ''}>
Active
</label>
</div>
</div>
`;
}
async saveItem() {
const modal = document.getElementById('form-modal');
const type = modal.dataset.type;
try {
if (type === 'sponsors') {
const fileInput = document.getElementById('form-logo-file');
if (fileInput && fileInput.files && fileInput.files[0]) {
const formData = new FormData();
formData.append('file', fileInput.files[0]);
formData.append('folder', 'sponsors');
const uploadResponse = await this.apiCall('/admin/upload-image', {
method: 'POST',
body: formData
});
if (!uploadResponse.url) {
throw new Error('Image upload failed');
}
document.getElementById('form-logo-url').value = uploadResponse.url;
}
}
const data = this.collectFormData(type);
if (this.editingItem) {
await this.apiCall(`/admin/${type}/${this.editingItem.id}`, {
method: 'PUT',
body: JSON.stringify(data)
});
} else {
await this.apiCall(`/admin/${type}`, {
method: 'POST',
body: JSON.stringify(data)
});
}
this.closeModal();
await this[`load${type.charAt(0).toUpperCase() + type.slice(1)}`]();
await this.loadStats();
} catch (error) {
alert('Error saving item: ' + error.message);
}
}
collectFormData(type) {
const data = {};
if (type === 'apps') {
data.name = document.getElementById('form-name').value;
data.slug = document.getElementById('form-slug').value || this.generateSlug(data.name);
data.description = document.getElementById('form-description').value;
data.category = document.getElementById('form-category').value;
data.type = document.getElementById('form-type').value;
const rating = parseFloat(document.getElementById('form-rating').value);
const downloads = parseInt(document.getElementById('form-downloads').value, 10);
data.rating = Number.isFinite(rating) ? rating : 0;
data.downloads = Number.isFinite(downloads) ? downloads : 0;
data.image = document.getElementById('form-image').value;
data.website_url = document.getElementById('form-website').value;
data.github_url = document.getElementById('form-github').value;
data.pricing = document.getElementById('form-pricing').value;
data.contact_email = document.getElementById('form-email').value;
data.featured = document.getElementById('form-featured').checked ? 1 : 0;
data.sponsored = document.getElementById('form-sponsored').checked ? 1 : 0;
data.integration_guide = document.getElementById('form-integration').value;
} else if (type === 'articles') {
data.title = document.getElementById('form-title').value;
data.slug = this.generateSlug(data.title);
data.author = document.getElementById('form-author').value;
data.category = document.getElementById('form-category').value;
data.featured_image = document.getElementById('form-image').value;
data.content = document.getElementById('form-content').value;
} else if (type === 'categories') {
data.name = document.getElementById('form-name').value;
data.slug = this.generateSlug(data.name);
data.icon = document.getElementById('form-icon').value;
data.description = document.getElementById('form-description').value;
const orderIndex = parseInt(document.getElementById('form-order').value, 10);
data.order_index = Number.isFinite(orderIndex) ? orderIndex : 0;
} else if (type === 'sponsors') {
data.company_name = document.getElementById('form-name').value;
data.logo_url = document.getElementById('form-logo-url').value;
data.tier = document.getElementById('form-tier').value;
data.landing_url = document.getElementById('form-landing').value;
data.banner_url = document.getElementById('form-banner').value;
data.start_date = document.getElementById('form-start').value;
data.end_date = document.getElementById('form-end').value;
data.active = document.getElementById('form-active').checked ? 1 : 0;
}
return data;
}
setupLogoUploadHandlers() {
const fileInput = document.getElementById('form-logo-file');
const preview = document.getElementById('form-logo-preview');
const logoUrlInput = document.getElementById('form-logo-url');
const trigger = document.getElementById('form-logo-button');
const fileNameEl = document.getElementById('form-logo-filename');
if (!fileInput || !preview || !logoUrlInput) return;
const setFileName = (text) => {
if (fileNameEl) {
fileNameEl.textContent = text;
}
};
const setEmptyState = () => {
preview.innerHTML = '<span>No logo uploaded</span>';
preview.classList.add('empty');
setFileName('No file selected');
};
const setExistingState = () => {
if (logoUrlInput.value) {
const existingFile = logoUrlInput.value.split('/').pop().split('?')[0];
preview.innerHTML = `<img src="${resolveAssetUrl(logoUrlInput.value)}" alt="Logo preview">`;
preview.classList.remove('empty');
setFileName(existingFile ? `Current: ${existingFile}` : 'Current logo');
} else {
setEmptyState();
}
};
setExistingState();
if (trigger) {
trigger.onclick = () => fileInput.click();
}
fileInput.addEventListener('change', (event) => {
const file = event.target.files && event.target.files[0];
if (!file) {
setExistingState();
return;
}
setFileName(file.name);
const reader = new FileReader();
reader.onload = () => {
preview.innerHTML = `<img src="${reader.result}" alt="Logo preview">`;
preview.classList.remove('empty');
};
reader.readAsDataURL(file);
});
}
async deleteItem(type, id) {
if (!confirm(`Are you sure you want to delete this ${type.slice(0, -1)}?`)) return;
try {
await this.apiCall(`/admin/${type}/${id}`, { method: 'DELETE' });
await this[`load${type.charAt(0).toUpperCase() + type.slice(1)}`]();
await this.loadStats();
} catch (error) {
alert('Error deleting item: ' + error.message);
}
}
async deleteCategory(id) {
const hasApps = this.data.apps.some(app =>
app.category === this.data.categories.find(c => c.id === id)?.name
);
if (hasApps) {
alert('Cannot delete category with existing apps');
return;
}
await this.deleteItem('categories', id);
}
closeModal() {
document.getElementById('form-modal').classList.add('hidden');
this.editingItem = null;
}
switchSection(section) {
// Update navigation
document.querySelectorAll('.nav-btn').forEach(btn => {
btn.classList.toggle('active', btn.dataset.section === section);
});
// Show section
document.querySelectorAll('.content-section').forEach(sec => {
sec.classList.remove('active');
});
document.getElementById(`${section}-section`).classList.add('active');
this.currentSection = section;
}
filterTable(type, query) {
const items = this.data[type].filter(item => {
const searchText = Object.values(item).join(' ').toLowerCase();
return searchText.includes(query.toLowerCase());
});
if (type === 'apps') {
this.renderAppsTable(items);
} else if (type === 'articles') {
this.renderArticlesTable(items);
}
}
filterByCategory(category) {
const apps = category
? this.data.apps.filter(app => app.category === category)
: this.data.apps;
this.renderAppsTable(apps);
}
populateCategoryFilter() {
const filter = document.getElementById('apps-filter');
if (!filter) return;
filter.innerHTML = '<option value="">All Categories</option>';
this.data.categories.forEach(cat => {
filter.innerHTML += `<option value="${cat.name}">${cat.name}</option>`;
});
}
async exportData() {
const data = {
apps: this.data.apps,
articles: this.data.articles,
categories: this.data.categories,
sponsors: this.data.sponsors,
exported: new Date().toISOString()
};
const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = `marketplace-export-${Date.now()}.json`;
a.click();
}
async backupDatabase() {
// In production, this would download the SQLite file
alert('Database backup would be implemented on the server side');
}
generateSlug(text) {
return text.toLowerCase()
.replace(/[^\w\s-]/g, '')
.replace(/\s+/g, '-')
.replace(/-+/g, '-')
.trim();
}
formatNumber(num) {
if (num >= 1000000) return (num / 1000000).toFixed(1) + 'M';
if (num >= 1000) return (num / 1000).toFixed(1) + 'K';
return num.toString();
}
logout() {
localStorage.removeItem('admin_token');
this.token = null;
this.showLogin();
}
}
// Initialize
const admin = new AdminDashboard();

215
docs/md_v2/marketplace/admin/index.html Normal file
View File

@@ -0,0 +1,215 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Admin Dashboard - Crawl4AI Marketplace</title>
<link rel="stylesheet" href="../frontend/marketplace.css?v=1759329000">
<link rel="stylesheet" href="admin.css?v=1759329000">
</head>
<body>
<div class="admin-container">
<!-- Login Screen -->
<div id="login-screen" class="login-screen">
<div class="login-box">
<img src="../../assets/images/logo.png" alt="Crawl4AI" class="login-logo">
<h1>[ Admin Access ]</h1>
<div id="login-form">
<input type="password" id="password" placeholder="Enter admin password" autofocus onkeypress="if(event.key==='Enter'){document.getElementById('login-btn').click()}">
<button type="button" id="login-btn">→ Login</button>
</div>
<div id="login-error" class="error-msg"></div>
</div>
</div>
<!-- Admin Dashboard -->
<div id="admin-dashboard" class="admin-dashboard hidden">
<!-- Header -->
<header class="admin-header">
<div class="header-content">
<div class="header-left">
<img src="../../assets/images/logo.png" alt="Crawl4AI" class="header-logo">
<h1>[ Admin Dashboard ]</h1>
</div>
<div class="header-right">
<span class="admin-user">Administrator</span>
<button id="logout-btn" class="logout-btn">↗ Logout</button>
</div>
</div>
</header>
<!-- Main Layout -->
<div class="admin-layout">
<!-- Sidebar -->
<aside class="admin-sidebar">
<nav class="sidebar-nav">
<button class="nav-btn active" data-section="stats">
<span class="nav-icon">▓</span> Dashboard
</button>
<button class="nav-btn" data-section="apps">
<span class="nav-icon">◆</span> Apps
</button>
<button class="nav-btn" data-section="articles">
<span class="nav-icon">■</span> Articles
</button>
<button class="nav-btn" data-section="categories">
<span class="nav-icon">□</span> Categories
</button>
<button class="nav-btn" data-section="sponsors">
<span class="nav-icon">◆</span> Sponsors
</button>
</nav>
<div class="sidebar-actions">
<button id="export-btn" class="action-btn">
<span>↓</span> Export Data
</button>
<button id="backup-btn" class="action-btn">
<span>▪</span> Backup DB
</button>
</div>
</aside>
<!-- Main Content -->
<main class="admin-main">
<!-- Stats Section -->
<section id="stats-section" class="content-section active">
<h2>Dashboard Overview</h2>
<div class="stats-grid">
<div class="stat-card">
<div class="stat-icon">◆</div>
<div class="stat-info">
<div class="stat-number" id="stat-apps">--</div>
<div class="stat-label">Total Apps</div>
<div class="stat-detail">
<span id="stat-featured">--</span> featured,
<span id="stat-sponsored">--</span> sponsored
</div>
</div>
</div>
<div class="stat-card">
<div class="stat-icon">■</div>
<div class="stat-info">
<div class="stat-number" id="stat-articles">--</div>
<div class="stat-label">Articles</div>
</div>
</div>
<div class="stat-card">
<div class="stat-icon">◆</div>
<div class="stat-info">
<div class="stat-number" id="stat-sponsors">--</div>
<div class="stat-label">Active Sponsors</div>
</div>
</div>
<div class="stat-card">
<div class="stat-icon">●</div>
<div class="stat-info">
<div class="stat-number" id="stat-views">--</div>
<div class="stat-label">Total Views</div>
</div>
</div>
</div>
<h3>Quick Actions</h3>
<div class="quick-actions">
<button class="quick-btn" onclick="admin.showAddForm('apps')">
<span>→</span> Add New App
</button>
<button class="quick-btn" onclick="admin.showAddForm('articles')">
<span>→</span> Write Article
</button>
<button class="quick-btn" onclick="admin.showAddForm('sponsors')">
<span>→</span> Add Sponsor
</button>
</div>
</section>
<!-- Apps Section -->
<section id="apps-section" class="content-section">
<div class="section-header">
<h2>Apps Management</h2>
<div class="header-actions">
<input type="text" id="apps-search" class="search-input" placeholder="Search apps...">
<select id="apps-filter" class="filter-select">
<option value="">All Categories</option>
</select>
<button class="add-btn" onclick="admin.showAddForm('apps')">
<span>→</span> Add App
</button>
</div>
</div>
<div class="data-table" id="apps-table">
<!-- Apps table will be populated here -->
</div>
</section>
<!-- Articles Section -->
<section id="articles-section" class="content-section">
<div class="section-header">
<h2>Articles Management</h2>
<div class="header-actions">
<input type="text" id="articles-search" class="search-input" placeholder="Search articles...">
<button class="add-btn" onclick="admin.showAddForm('articles')">
<span>→</span> Add Article
</button>
</div>
</div>
<div class="data-table" id="articles-table">
<!-- Articles table will be populated here -->
</div>
</section>
<!-- Categories Section -->
<section id="categories-section" class="content-section">
<div class="section-header">
<h2>Categories Management</h2>
<div class="header-actions">
<button class="add-btn" onclick="admin.showAddForm('categories')">
<span>→</span> Add Category
</button>
</div>
</div>
<div class="data-table" id="categories-table">
<!-- Categories table will be populated here -->
</div>
</section>
<!-- Sponsors Section -->
<section id="sponsors-section" class="content-section">
<div class="section-header">
<h2>Sponsors Management</h2>
<div class="header-actions">
<button class="add-btn" onclick="admin.showAddForm('sponsors')">
<span>→</span> Add Sponsor
</button>
</div>
</div>
<div class="data-table" id="sponsors-table">
<!-- Sponsors table will be populated here -->
</div>
</section>
</main>
</div>
</div>
<!-- Modal for Add/Edit Forms -->
<div id="form-modal" class="modal hidden">
<div class="modal-content large">
<div class="modal-header">
<h2 id="modal-title">Add/Edit</h2>
<button class="modal-close" onclick="admin.closeModal()">✕</button>
</div>
<div class="modal-body" id="modal-body">
<!-- Dynamic form content -->
</div>
<div class="modal-footer">
<button class="btn-cancel" onclick="admin.closeModal()">Cancel</button>
<button class="btn-save" id="save-btn">Save</button>
</div>
</div>
</div>
</div>
<script src="admin.js?v=1759335000"></script>
</body>
</html>

658
docs/md_v2/marketplace/app-detail.css Normal file
View File

@@ -0,0 +1,658 @@
/* App Detail Page Styles */
.app-detail-container {
min-height: 100vh;
background: var(--bg-dark);
}
/* Back Button */
.header-nav {
display: flex;
align-items: center;
}
.back-btn {
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--primary-cyan);
text-decoration: none;
transition: all 0.2s;
font-size: 0.875rem;
}
.back-btn:hover {
border-color: var(--primary-cyan);
background: rgba(80, 255, 255, 0.1);
}
/* App Hero Section */
.app-hero {
max-width: 1800px;
margin: 2rem auto;
padding: 0 2rem;
}
.app-hero-content {
display: grid;
grid-template-columns: 1fr 2fr;
gap: 3rem;
background: linear-gradient(135deg, #1a1a2e, #0f0f1e);
border: 2px solid var(--primary-cyan);
padding: 2rem;
box-shadow: 0 0 30px rgba(80, 255, 255, 0.15),
inset 0 0 20px rgba(80, 255, 255, 0.05);
}
.app-hero-image {
width: 100%;
height: 300px;
background: linear-gradient(135deg, rgba(80, 255, 255, 0.1), rgba(243, 128, 245, 0.05));
background-size: cover;
background-position: center;
border: 1px solid var(--border-color);
display: flex;
align-items: center;
justify-content: center;
font-size: 4rem;
color: var(--primary-cyan);
}
.app-badges {
display: flex;
gap: 0.5rem;
margin-bottom: 1rem;
}
.app-badge {
padding: 0.3rem 0.6rem;
background: var(--bg-tertiary);
color: var(--text-secondary);
font-size: 0.75rem;
text-transform: uppercase;
font-weight: 600;
}
.app-badge.featured {
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
color: var(--bg-dark);
box-shadow: 0 2px 10px rgba(80, 255, 255, 0.3);
}
.app-badge.sponsored {
background: linear-gradient(135deg, var(--warning), #ff8c00);
color: var(--bg-dark);
box-shadow: 0 2px 10px rgba(245, 158, 11, 0.3);
}
.app-hero-info h1 {
font-size: 2.5rem;
color: var(--primary-cyan);
margin: 0.5rem 0;
text-shadow: 0 0 20px rgba(80, 255, 255, 0.5);
}
.app-tagline {
font-size: 1.1rem;
color: var(--text-secondary);
margin-bottom: 2rem;
}
/* Stats */
.app-stats {
display: flex;
gap: 2rem;
margin: 2rem 0;
padding: 1rem 0;
border-top: 1px solid var(--border-color);
border-bottom: 1px solid var(--border-color);
}
.stat {
display: flex;
flex-direction: column;
gap: 0.25rem;
}
.stat-value {
font-size: 1.5rem;
color: var(--primary-cyan);
font-weight: 600;
}
.stat-label {
font-size: 0.875rem;
color: var(--text-tertiary);
}
/* Action Buttons */
.app-actions {
display: flex;
gap: 1rem;
margin: 2rem 0;
}
.action-btn {
padding: 0.75rem 1.5rem;
border: 1px solid var(--border-color);
background: transparent;
color: var(--text-primary);
text-decoration: none;
display: inline-flex;
align-items: center;
gap: 0.5rem;
transition: all 0.2s;
cursor: pointer;
font-family: inherit;
font-size: 0.9rem;
}
.action-btn.primary {
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
color: var(--bg-dark);
border-color: var(--primary-cyan);
font-weight: 600;
}
.action-btn.primary:hover {
box-shadow: 0 4px 15px rgba(80, 255, 255, 0.3);
transform: translateY(-2px);
}
.action-btn.secondary {
border-color: var(--accent-pink);
color: var(--accent-pink);
}
.action-btn.secondary:hover {
background: rgba(243, 128, 245, 0.1);
box-shadow: 0 4px 15px rgba(243, 128, 245, 0.2);
}
.action-btn.ghost {
border-color: var(--border-color);
color: var(--text-secondary);
}
.action-btn.ghost:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
/* Pricing */
.pricing-info {
display: flex;
align-items: center;
gap: 1rem;
font-size: 1.1rem;
}
.pricing-label {
color: var(--text-tertiary);
}
.pricing-value {
color: var(--warning);
font-weight: 600;
}
/* Navigation Tabs */
.tabs {
display: flex;
flex-direction: row;
gap: 0;
border-bottom: 2px solid var(--border-color);
margin-bottom: 0;
background: var(--bg-tertiary);
}
.tab-btn {
padding: 1rem 2rem;
background: transparent;
border: none;
border-bottom: 3px solid transparent;
color: var(--text-secondary);
cursor: pointer;
transition: all 0.2s;
font-family: inherit;
font-size: 0.95rem;
margin-bottom: -2px;
white-space: nowrap;
font-weight: 500;
}
.tab-btn:hover {
color: var(--primary-cyan);
background: rgba(80, 255, 255, 0.05);
}
.tab-btn.active {
color: var(--primary-cyan);
border-bottom-color: var(--primary-cyan);
background: var(--bg-secondary);
}
.app-nav {
max-width: 1800px;
margin: 2rem auto 0;
padding: 0 2rem;
display: flex;
gap: 1rem;
border-bottom: 2px solid var(--border-color);
}
.nav-tab {
padding: 1rem 1.5rem;
background: transparent;
border: none;
border-bottom: 2px solid transparent;
color: var(--text-secondary);
cursor: pointer;
transition: all 0.2s;
font-family: inherit;
font-size: 0.9rem;
margin-bottom: -2px;
}
.nav-tab:hover {
color: var(--primary-cyan);
}
.nav-tab.active {
color: var(--primary-cyan);
border-bottom-color: var(--primary-cyan);
}
/* Main Content Wrapper */
.app-main {
max-width: 1800px;
margin: 2rem auto;
padding: 0 2rem;
}
/* Content Sections */
.app-content {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
padding: 0;
}
.tab-content {
display: none;
padding: 2rem;
}
.tab-content.active {
display: block;
}
/* Overview Layout */
.overview-columns {
display: grid;
grid-template-columns: 2fr 1fr;
gap: 2rem;
}
.overview-main h2, .overview-main h3 {
color: var(--primary-cyan);
margin-top: 2rem;
margin-bottom: 1rem;
}
.overview-main h2:first-child {
margin-top: 0;
}
.overview-main h2 {
font-size: 1.8rem;
border-bottom: 2px solid var(--border-color);
padding-bottom: 0.5rem;
}
.overview-main h3 {
font-size: 1.3rem;
}
.features-list {
list-style: none;
padding: 0;
}
.features-list li {
padding: 0.5rem 0;
padding-left: 1.5rem;
position: relative;
color: var(--text-secondary);
}
.features-list li:before {
content: "▸";
position: absolute;
left: 0;
color: var(--primary-cyan);
}
.use-cases p {
color: var(--text-secondary);
line-height: 1.6;
}
/* Sidebar */
.sidebar {
display: flex;
flex-direction: column;
gap: 1rem;
}
.sidebar-card {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
padding: 1.5rem;
}
.sidebar-card h3 {
font-size: 1.1rem;
color: var(--primary-cyan);
margin: 0 0 1rem 0;
border-bottom: 1px solid var(--border-color);
padding-bottom: 0.5rem;
}
.stats-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 1rem;
}
.stats-grid > div {
text-align: center;
}
.metadata {
margin: 0;
}
.metadata div {
display: flex;
justify-content: space-between;
padding: 0.75rem 0;
border-bottom: 1px solid var(--border-color);
}
.metadata dt {
color: var(--text-tertiary);
font-weight: normal;
}
.metadata dd {
color: var(--text-primary);
margin: 0;
font-weight: 600;
}
.sidebar-card p {
color: var(--text-secondary);
margin: 0;
}
/* Integration Content */
.integration-content {
max-width: 100%;
}
.integration-content h2 {
font-size: 1.8rem;
color: var(--primary-cyan);
margin: 0 0 2rem 0;
padding-bottom: 0.5rem;
border-bottom: 2px solid var(--border-color);
}
.integration-content h3 {
font-size: 1.3rem;
color: var(--text-primary);
margin: 2rem 0 1rem;
}
.docs-content {
max-width: 100%;
}
.docs-content h2 {
font-size: 1.8rem;
color: var(--primary-cyan);
margin: 0 0 1.5rem 0;
padding-bottom: 0.5rem;
border-bottom: 2px solid var(--border-color);
}
.docs-content h3 {
font-size: 1.3rem;
color: var(--text-primary);
margin: 2rem 0 1rem;
}
.docs-content h4 {
font-size: 1.1rem;
color: var(--accent-pink);
margin: 1.5rem 0 0.5rem;
}
.docs-content p {
color: var(--text-secondary);
line-height: 1.6;
margin-bottom: 1rem;
}
.docs-content code {
background: var(--bg-tertiary);
padding: 0.2rem 0.4rem;
color: var(--primary-cyan);
font-family: 'Dank Mono', Monaco, monospace;
font-size: 0.9em;
}
/* Code Blocks */
.code-block {
background: var(--bg-dark);
border: 1px solid var(--border-color);
margin: 1rem 0;
overflow: hidden;
position: relative;
}
.code-header {
display: flex;
justify-content: space-between;
align-items: center;
padding: 0.5rem 1rem;
background: var(--bg-tertiary);
border-bottom: 1px solid var(--border-color);
}
.code-lang {
color: var(--primary-cyan);
font-size: 0.875rem;
text-transform: uppercase;
}
.copy-btn {
position: absolute;
top: 0.5rem;
right: 0.5rem;
padding: 0.4rem 0.8rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
color: var(--text-secondary);
cursor: pointer;
font-size: 0.75rem;
transition: all 0.2s;
z-index: 10;
}
.copy-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
background: var(--bg-secondary);
}
.code-block pre {
margin: 0;
padding: 1rem;
overflow-x: auto;
}
.code-block code {
background: transparent;
padding: 0;
color: var(--text-secondary);
font-size: 0.875rem;
line-height: 1.5;
}
/* Feature Grid */
.feature-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
margin: 2rem 0;
}
.feature-card {
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
padding: 1.5rem;
transition: all 0.2s;
}
.feature-card:hover {
border-color: var(--primary-cyan);
background: rgba(80, 255, 255, 0.05);
}
.feature-card h4 {
margin-top: 0;
}
/* Info Box */
.info-box {
background: linear-gradient(135deg, rgba(80, 255, 255, 0.05), rgba(243, 128, 245, 0.03));
border: 1px solid var(--primary-cyan);
border-left: 4px solid var(--primary-cyan);
padding: 1.5rem;
margin: 2rem 0;
}
.info-box h4 {
margin-top: 0;
color: var(--primary-cyan);
}
/* Support Grid */
.support-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
margin: 2rem 0;
}
.support-card {
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
padding: 1.5rem;
text-align: center;
}
.support-card h3 {
color: var(--primary-cyan);
margin-bottom: 0.5rem;
}
/* Related Apps */
.related-apps {
max-width: 1800px;
margin: 4rem auto;
padding: 0 2rem;
}
.related-apps h2 {
font-size: 1.5rem;
color: var(--text-primary);
margin-bottom: 1.5rem;
}
.related-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
gap: 1rem;
}
.related-app-card {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
padding: 1rem;
cursor: pointer;
transition: all 0.2s;
}
.related-app-card:hover {
border-color: var(--primary-cyan);
transform: translateY(-2px);
}
/* Responsive */
@media (max-width: 1024px) {
.app-hero-content {
grid-template-columns: 1fr;
}
.app-stats {
justify-content: space-around;
}
.overview-columns {
grid-template-columns: 1fr;
}
}
@media (max-width: 768px) {
.app-hero-info h1 {
font-size: 2rem;
}
.app-actions {
flex-direction: column;
}
.tabs {
overflow-x: auto;
-webkit-overflow-scrolling: touch;
}
.tab-btn {
padding: 0.75rem 1.5rem;
font-size: 0.875rem;
}
.app-nav {
overflow-x: auto;
gap: 0;
}
.nav-tab {
white-space: nowrap;
}
.feature-grid,
.support-grid {
grid-template-columns: 1fr;
}
.tab-content {
padding: 1rem;
}
.app-main {
padding: 0 1rem;
}
}

209
docs/md_v2/marketplace/app-detail.html Normal file
View File

@@ -0,0 +1,209 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>App Details - Crawl4AI Marketplace</title>
<link rel="stylesheet" href="marketplace.css">
<link rel="stylesheet" href="app-detail.css">
</head>
<body>
<div class="app-detail-container">
<!-- Header -->
<header class="marketplace-header">
<div class="header-content">
<div class="header-left">
<div class="logo-title">
<img src="../assets/images/logo.png" alt="Crawl4AI" class="header-logo">
<h1>
<span class="ascii-border">[</span>
Marketplace
<span class="ascii-border">]</span>
</h1>
</div>
</div>
<div class="header-nav">
<a href="index.html" class="back-btn">← Back to Marketplace</a>
</div>
</div>
</header>
<!-- App Hero Section -->
<section class="app-hero">
<div class="app-hero-content">
<div class="app-hero-image" id="app-image">
<!-- Dynamic image -->
</div>
<div class="app-hero-info">
<div class="app-badges">
<span class="app-badge" id="app-type">Open Source</span>
<span class="app-badge featured" id="app-featured" style="display:none">FEATURED</span>
<span class="app-badge sponsored" id="app-sponsored" style="display:none">SPONSORED</span>
</div>
<h1 id="app-name">App Name</h1>
<p id="app-description" class="app-tagline">App description goes here</p>
<div class="app-stats">
<div class="stat">
<span class="stat-value" id="app-rating">★★★★★</span>
<span class="stat-label">Rating</span>
</div>
<div class="stat">
<span class="stat-value" id="app-downloads">0</span>
<span class="stat-label">Downloads</span>
</div>
<div class="stat">
<span class="stat-value" id="app-category">Category</span>
<span class="stat-label">Category</span>
</div>
</div>
<div class="app-actions">
<a href="#" id="app-website" class="action-btn primary" target="_blank">Visit Website</a>
<a href="#" id="app-github" class="action-btn" target="_blank">View GitHub</a>
<a href="#" id="app-demo" class="action-btn" target="_blank" style="display:none">Live Demo</a>
</div>
</div>
</div>
</section>
<!-- App Details Section -->
<main class="app-main">
<div class="app-content">
<div class="tabs">
<button class="tab-btn active" data-tab="overview">Overview</button>
<button class="tab-btn" data-tab="integration">Integration</button>
<button class="tab-btn" data-tab="docs">Documentation</button>
<button class="tab-btn" data-tab="support">Support</button>
</div>
<section id="overview-tab" class="tab-content active">
<div class="overview-columns">
<div class="overview-main">
<h2>Overview</h2>
<div id="app-overview">Overview content goes here.</div>
<h3>Key Features</h3>
<ul id="app-features" class="features-list">
<li>Feature 1</li>
<li>Feature 2</li>
<li>Feature 3</li>
</ul>
<h3>Use Cases</h3>
<div id="app-use-cases" class="use-cases">
<p>Describe how this app can help your workflow.</p>
</div>
</div>
<aside class="sidebar">
<div class="sidebar-card">
<h3>Download Stats</h3>
<div class="stats-grid">
<div>
<span class="stat-value" id="sidebar-downloads">0</span>
<span class="stat-label">Downloads</span>
</div>
<div>
<span class="stat-value" id="sidebar-rating">0.0</span>
<span class="stat-label">Rating</span>
</div>
</div>
</div>
<div class="sidebar-card">
<h3>App Metadata</h3>
<dl class="metadata">
<div>
<dt>Category</dt>
<dd id="sidebar-category">-</dd>
</div>
<div>
<dt>Type</dt>
<dd id="sidebar-type">-</dd>
</div>
<div>
<dt>Status</dt>
<dd id="sidebar-status">Active</dd>
</div>
<div>
<dt>Pricing</dt>
<dd id="sidebar-pricing">-</dd>
</div>
</dl>
</div>
<div class="sidebar-card">
<h3>Contact</h3>
<p id="sidebar-contact">contact@example.com</p>
</div>
</aside>
</div>
</section>
<section id="integration-tab" class="tab-content">
<div class="integration-content">
<h2>Integration Guide</h2>
<h3>Installation</h3>
<div class="code-block">
<pre><code id="install-code"># Installation instructions will appear here</code></pre>
</div>
<h3>Basic Usage</h3>
<div class="code-block">
<pre><code id="usage-code"># Usage example will appear here</code></pre>
</div>
<h3>Complete Integration Example</h3>
<div class="code-block">
<button class="copy-btn" id="copy-integration">Copy</button>
<pre><code id="integration-code"># Complete integration guide will appear here</code></pre>
</div>
</div>
</section>
<section id="docs-tab" class="tab-content">
<div class="docs-content">
<h2>Documentation</h2>
<div id="app-docs" class="doc-sections">
<p>Documentation coming soon.</p>
</div>
</div>
</section>
<section id="support-tab" class="tab-content">
<div class="docs-content">
<h2>Support</h2>
<div class="support-grid">
<div class="support-card">
<h3>📧 Contact</h3>
<p id="app-contact">contact@example.com</p>
</div>
<div class="support-card">
<h3>🐛 Report Issues</h3>
<p>Found a bug? Report it on GitHub Issues.</p>
</div>
<div class="support-card">
<h3>💬 Community</h3>
<p>Join our Discord for help and discussions.</p>
</div>
</div>
</div>
</section>
</div>
</main>
<!-- Related Apps -->
<section class="related-apps">
<h2>Related Apps</h2>
<div id="related-apps-grid" class="related-grid">
<!-- Dynamic related apps -->
</div>
</section>
</div>
<script src="app-detail.js"></script>
</body>
</html>

348
docs/md_v2/marketplace/app-detail.js Normal file
View File

@@ -0,0 +1,348 @@
// App Detail Page JavaScript
const { API_BASE, API_ORIGIN } = (() => {
const { hostname, port, protocol } = window.location;
const isLocalHost = ['localhost', '127.0.0.1', '0.0.0.0'].includes(hostname);
if (isLocalHost && port && port !== '8100') {
const origin = `${protocol}//127.0.0.1:8100`;
return { API_BASE: `${origin}/marketplace/api`, API_ORIGIN: origin };
}
return { API_BASE: '/marketplace/api', API_ORIGIN: '' };
})();
class AppDetailPage {
constructor() {
this.appSlug = this.getAppSlugFromURL();
this.appData = null;
this.init();
}
getAppSlugFromURL() {
const params = new URLSearchParams(window.location.search);
return params.get('app') || '';
}
async init() {
if (!this.appSlug) {
window.location.href = 'index.html';
return;
}
await this.loadAppDetails();
this.setupEventListeners();
await this.loadRelatedApps();
}
async loadAppDetails() {
try {
const response = await fetch(`${API_BASE}/apps/${this.appSlug}`);
if (!response.ok) throw new Error('App not found');
this.appData = await response.json();
this.renderAppDetails();
} catch (error) {
console.error('Error loading app details:', error);
// Fallback to loading all apps and finding the right one
try {
const response = await fetch(`${API_BASE}/apps`);
const apps = await response.json();
this.appData = apps.find(app => app.slug === this.appSlug || app.name.toLowerCase().replace(/\s+/g, '-') === this.appSlug);
if (this.appData) {
this.renderAppDetails();
} else {
window.location.href = 'index.html';
}
} catch (err) {
console.error('Error loading apps:', err);
window.location.href = 'index.html';
}
}
}
renderAppDetails() {
if (!this.appData) return;
// Update title
document.title = `${this.appData.name} - Crawl4AI Marketplace`;
// Hero image
const appImage = document.getElementById('app-image');
if (this.appData.image) {
appImage.style.backgroundImage = `url('${this.appData.image}')`;
appImage.innerHTML = '';
} else {
appImage.innerHTML = `[${this.appData.category || 'APP'}]`;
}
// Basic info
document.getElementById('app-name').textContent = this.appData.name;
document.getElementById('app-description').textContent = this.appData.description;
document.getElementById('app-type').textContent = this.appData.type || 'Open Source';
document.getElementById('app-category').textContent = this.appData.category;
// Badges
if (this.appData.featured) {
document.getElementById('app-featured').style.display = 'inline-block';
}
if (this.appData.sponsored) {
document.getElementById('app-sponsored').style.display = 'inline-block';
}
// Stats
const rating = this.appData.rating || 0;
const stars = '★'.repeat(Math.floor(rating)) + '☆'.repeat(5 - Math.floor(rating));
document.getElementById('app-rating').textContent = stars + ` ${rating}/5`;
document.getElementById('app-downloads').textContent = this.formatNumber(this.appData.downloads || 0);
// Action buttons
const websiteBtn = document.getElementById('app-website');
const githubBtn = document.getElementById('app-github');
if (this.appData.website_url) {
websiteBtn.href = this.appData.website_url;
} else {
websiteBtn.style.display = 'none';
}
if (this.appData.github_url) {
githubBtn.href = this.appData.github_url;
} else {
githubBtn.style.display = 'none';
}
// Contact
document.getElementById('app-contact').textContent = this.appData.contact_email || 'Not available';
// Sidebar info
document.getElementById('sidebar-downloads').textContent = this.formatNumber(this.appData.downloads || 0);
document.getElementById('sidebar-rating').textContent = (this.appData.rating || 0).toFixed(1);
document.getElementById('sidebar-category').textContent = this.appData.category || '-';
document.getElementById('sidebar-type').textContent = this.appData.type || '-';
document.getElementById('sidebar-status').textContent = this.appData.status || 'Active';
document.getElementById('sidebar-pricing').textContent = this.appData.pricing || 'Free';
document.getElementById('sidebar-contact').textContent = this.appData.contact_email || 'contact@example.com';
// Integration guide
this.renderIntegrationGuide();
}
renderIntegrationGuide() {
// Installation code
const installCode = document.getElementById('install-code');
if (installCode) {
if (this.appData.type === 'Open Source' && this.appData.github_url) {
installCode.textContent = `# Clone from GitHub
git clone ${this.appData.github_url}
# Install dependencies
pip install -r requirements.txt`;
} else if (this.appData.name.toLowerCase().includes('api')) {
installCode.textContent = `# Install via pip
pip install ${this.appData.slug}
# Or install from source
pip install git+${this.appData.github_url || 'https://github.com/example/repo'}`;
}
}
// Usage code - customize based on category
const usageCode = document.getElementById('usage-code');
if (usageCode) {
if (this.appData.category === 'Browser Automation') {
usageCode.textContent = `from crawl4ai import AsyncWebCrawler
from ${this.appData.slug.replace(/-/g, '_')} import ${this.appData.name.replace(/\s+/g, '')}
async def main():
# Initialize ${this.appData.name}
automation = ${this.appData.name.replace(/\s+/g, '')}()
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
browser_config=automation.config,
wait_for="css:body"
)
print(result.markdown)`;
} else if (this.appData.category === 'Proxy Services') {
usageCode.textContent = `from crawl4ai import AsyncWebCrawler
import ${this.appData.slug.replace(/-/g, '_')}
# Configure proxy
proxy_config = {
"server": "${this.appData.website_url || 'https://proxy.example.com'}",
"username": "your_username",
"password": "your_password"
}
async with AsyncWebCrawler(proxy=proxy_config) as crawler:
result = await crawler.arun(
url="https://example.com",
bypass_cache=True
)
print(result.status_code)`;
} else if (this.appData.category === 'LLM Integration') {
usageCode.textContent = `from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
# Configure LLM extraction
strategy = LLMExtractionStrategy(
provider="${this.appData.name.toLowerCase().includes('gpt') ? 'openai' : 'anthropic'}",
api_key="your-api-key",
model="${this.appData.name.toLowerCase().includes('gpt') ? 'gpt-4' : 'claude-3'}",
instruction="Extract structured data"
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
extraction_strategy=strategy
)
print(result.extracted_content)`;
}
}
// Integration example
const integrationCode = document.getElementById('integration-code');
if (integrationCode) {
integrationCode.textContent = this.appData.integration_guide ||
`# Complete ${this.appData.name} Integration Example
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
import json
async def crawl_with_${this.appData.slug.replace(/-/g, '_')}():
"""
Complete example showing how to use ${this.appData.name}
with Crawl4AI for production web scraping
"""
# Define extraction schema
schema = {
"name": "ProductList",
"baseSelector": "div.product",
"fields": [
{"name": "title", "selector": "h2", "type": "text"},
{"name": "price", "selector": ".price", "type": "text"},
{"name": "image", "selector": "img", "type": "attribute", "attribute": "src"},
{"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
]
}
# Initialize crawler with ${this.appData.name}
async with AsyncWebCrawler(
browser_type="chromium",
headless=True,
verbose=True
) as crawler:
# Crawl with extraction
result = await crawler.arun(
url="https://example.com/products",
extraction_strategy=JsonCssExtractionStrategy(schema),
cache_mode="bypass",
wait_for="css:.product",
screenshot=True
)
# Process results
if result.success:
products = json.loads(result.extracted_content)
print(f"Found {len(products)} products")
for product in products[:5]:
print(f"- {product['title']}: {product['price']}")
return products
# Run the crawler
if __name__ == "__main__":
import asyncio
asyncio.run(crawl_with_${this.appData.slug.replace(/-/g, '_')}())`;
}
}
formatNumber(num) {
if (num >= 1000000) {
return (num / 1000000).toFixed(1) + 'M';
} else if (num >= 1000) {
return (num / 1000).toFixed(1) + 'K';
}
return num.toString();
}
setupEventListeners() {
// Tab switching
const tabs = document.querySelectorAll('.tab-btn');
tabs.forEach(tab => {
tab.addEventListener('click', () => {
// Update active tab
tabs.forEach(t => t.classList.remove('active'));
tab.classList.add('active');
// Show corresponding content
const tabName = tab.dataset.tab;
document.querySelectorAll('.tab-content').forEach(content => {
content.classList.remove('active');
});
document.getElementById(`${tabName}-tab`).classList.add('active');
});
});
// Copy integration code
document.getElementById('copy-integration').addEventListener('click', () => {
const code = document.getElementById('integration-code').textContent;
navigator.clipboard.writeText(code).then(() => {
const btn = document.getElementById('copy-integration');
const originalText = btn.innerHTML;
btn.innerHTML = '<span>✓</span> Copied!';
setTimeout(() => {
btn.innerHTML = originalText;
}, 2000);
});
});
// Copy code buttons
document.querySelectorAll('.copy-btn').forEach(btn => {
btn.addEventListener('click', (e) => {
const codeBlock = e.target.closest('.code-block');
const code = codeBlock.querySelector('code').textContent;
navigator.clipboard.writeText(code).then(() => {
btn.textContent = 'Copied!';
setTimeout(() => {
btn.textContent = 'Copy';
}, 2000);
});
});
});
}
async loadRelatedApps() {
try {
const response = await fetch(`${API_BASE}/apps?category=${encodeURIComponent(this.appData.category)}&limit=4`);
const apps = await response.json();
const relatedApps = apps.filter(app => app.slug !== this.appSlug).slice(0, 3);
const grid = document.getElementById('related-apps-grid');
grid.innerHTML = relatedApps.map(app => `
<div class="related-app-card" onclick="window.location.href='app-detail.html?app=${app.slug || app.name.toLowerCase().replace(/\s+/g, '-')}'">
<h4>${app.name}</h4>
<p>${app.description.substring(0, 100)}...</p>
<div style="display: flex; justify-content: space-between; margin-top: 0.5rem; font-size: 0.75rem;">
<span style="color: var(--primary-cyan)">${app.type}</span>
<span style="color: var(--warning)">★ ${app.rating}/5</span>
</div>
</div>
`).join('');
} catch (error) {
console.error('Error loading related apps:', error);
}
}
}
// Initialize when DOM is loaded
document.addEventListener('DOMContentLoaded', () => {
new AppDetailPage();
});

14
docs/md_v2/marketplace/backend/.env.example Normal file
View File

@@ -0,0 +1,14 @@
# Marketplace Configuration
# Copy this to .env and update with your values
# Admin password (required)
MARKETPLACE_ADMIN_PASSWORD=change_this_password
# JWT secret key (required) - generate with: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
MARKETPLACE_JWT_SECRET=change_this_to_a_secure_random_key
# Database path (optional, defaults to ./marketplace.db)
MARKETPLACE_DB_PATH=./marketplace.db
# Token expiry in hours (optional, defaults to 4)
MARKETPLACE_TOKEN_EXPIRY=4

59
docs/md_v2/marketplace/backend/config.py Normal file
View File

@@ -0,0 +1,59 @@
"""
Marketplace Configuration - Loads from .env file
"""
import os
import sys
import hashlib
from pathlib import Path
from dotenv import load_dotenv
# Load .env file
env_path = Path(__file__).parent / '.env'
if not env_path.exists():
print("\n❌ ERROR: No .env file found!")
print("Please copy .env.example to .env and update with your values:")
print(f" cp {Path(__file__).parent}/.env.example {Path(__file__).parent}/.env")
print("\nThen edit .env with your secure values.")
sys.exit(1)
load_dotenv(env_path)
# Required environment variables
required_vars = ['MARKETPLACE_ADMIN_PASSWORD', 'MARKETPLACE_JWT_SECRET']
missing_vars = [var for var in required_vars if not os.getenv(var)]
if missing_vars:
print(f"\n❌ ERROR: Missing required environment variables: {', '.join(missing_vars)}")
print("Please check your .env file and ensure all required variables are set.")
sys.exit(1)
class Config:
"""Configuration loaded from environment variables"""
# Admin authentication - hashed from password in .env
ADMIN_PASSWORD_HASH = hashlib.sha256(
os.getenv('MARKETPLACE_ADMIN_PASSWORD').encode()
).hexdigest()
# JWT secret for token generation
JWT_SECRET_KEY = os.getenv('MARKETPLACE_JWT_SECRET')
# Database path
DATABASE_PATH = os.getenv('MARKETPLACE_DB_PATH', './marketplace.db')
# Token expiry in hours
TOKEN_EXPIRY_HOURS = int(os.getenv('MARKETPLACE_TOKEN_EXPIRY', '4'))
# CORS origins - hardcoded as they don't contain secrets
ALLOWED_ORIGINS = [
"http://localhost:8000",
"http://localhost:8080",
"http://localhost:8100",
"http://127.0.0.1:8000",
"http://127.0.0.1:8080",
"http://127.0.0.1:8100",
"https://crawl4ai.com",
"https://www.crawl4ai.com",
"https://docs.crawl4ai.com",
"https://market.crawl4ai.com"
]

117
docs/md_v2/marketplace/backend/database.py Normal file
View File

@@ -0,0 +1,117 @@
import sqlite3
import yaml
import json
from pathlib import Path
from typing import Dict, List, Any
class DatabaseManager:
def __init__(self, db_path=None, schema_path='schema.yaml'):
self.schema = self._load_schema(schema_path)
# Use provided path or fallback to schema default
self.db_path = db_path or self.schema['database']['name']
self.conn = None
self._init_database()
def _load_schema(self, path: str) -> Dict:
with open(path, 'r') as f:
return yaml.safe_load(f)
def _init_database(self):
"""Auto-create/migrate database from schema"""
self.conn = sqlite3.connect(self.db_path, check_same_thread=False)
self.conn.row_factory = sqlite3.Row
for table_name, table_def in self.schema['tables'].items():
self._create_or_update_table(table_name, table_def['columns'])
def _create_or_update_table(self, table_name: str, columns: Dict):
cursor = self.conn.cursor()
# Check if table exists
cursor.execute(f"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table_name,))
table_exists = cursor.fetchone() is not None
if not table_exists:
# Create table
col_defs = []
for col_name, col_spec in columns.items():
col_def = f"{col_name} {col_spec['type']}"
if col_spec.get('primary'):
col_def += " PRIMARY KEY"
if col_spec.get('autoincrement'):
col_def += " AUTOINCREMENT"
if col_spec.get('unique'):
col_def += " UNIQUE"
if col_spec.get('required'):
col_def += " NOT NULL"
if 'default' in col_spec:
default = col_spec['default']
if default == 'CURRENT_TIMESTAMP':
col_def += f" DEFAULT {default}"
elif isinstance(default, str):
col_def += f" DEFAULT '{default}'"
else:
col_def += f" DEFAULT {default}"
col_defs.append(col_def)
create_sql = f"CREATE TABLE {table_name} ({', '.join(col_defs)})"
cursor.execute(create_sql)
else:
# Check for new columns and add them
cursor.execute(f"PRAGMA table_info({table_name})")
existing_columns = {row[1] for row in cursor.fetchall()}
for col_name, col_spec in columns.items():
if col_name not in existing_columns:
col_def = f"{col_spec['type']}"
if 'default' in col_spec:
default = col_spec['default']
if default == 'CURRENT_TIMESTAMP':
col_def += f" DEFAULT {default}"
elif isinstance(default, str):
col_def += f" DEFAULT '{default}'"
else:
col_def += f" DEFAULT {default}"
cursor.execute(f"ALTER TABLE {table_name} ADD COLUMN {col_name} {col_def}")
self.conn.commit()
def get_all(self, table: str, limit: int = 100, offset: int = 0, where: str = None) -> List[Dict]:
cursor = self.conn.cursor()
query = f"SELECT * FROM {table}"
if where:
query += f" WHERE {where}"
query += f" LIMIT {limit} OFFSET {offset}"
cursor.execute(query)
rows = cursor.fetchall()
return [dict(row) for row in rows]
def search(self, query: str, tables: List[str] = None) -> Dict[str, List[Dict]]:
if not tables:
tables = list(self.schema['tables'].keys())
results = {}
cursor = self.conn.cursor()
for table in tables:
# Search in text columns
columns = self.schema['tables'][table]['columns']
text_cols = [col for col, spec in columns.items()
if spec['type'] == 'TEXT' and col != 'id']
if text_cols:
where_clause = ' OR '.join([f"{col} LIKE ?" for col in text_cols])
params = [f'%{query}%'] * len(text_cols)
cursor.execute(f"SELECT * FROM {table} WHERE {where_clause} LIMIT 10", params)
rows = cursor.fetchall()
if rows:
results[table] = [dict(row) for row in rows]
return results
def close(self):
if self.conn:
self.conn.close()

267
docs/md_v2/marketplace/backend/dummy_data.py Normal file
View File

@@ -0,0 +1,267 @@
import sqlite3
import json
import random
from datetime import datetime, timedelta
from database import DatabaseManager
def generate_slug(text):
return text.lower().replace(' ', '-').replace('&', 'and')
def generate_dummy_data():
db = DatabaseManager()
conn = db.conn
cursor = conn.cursor()
# Clear existing data
for table in ['apps', 'articles', 'categories', 'sponsors']:
cursor.execute(f"DELETE FROM {table}")
# Categories
categories = [
("Browser Automation", "⚙", "Tools for browser automation and control"),
("Proxy Services", "🔒", "Proxy providers and rotation services"),
("LLM Integration", "🤖", "AI/LLM tools and integrations"),
("Data Processing", "📊", "Data extraction and processing tools"),
("Cloud Infrastructure", "☁", "Cloud browser and computing services"),
("Developer Tools", "🛠", "Development and testing utilities")
]
for i, (name, icon, desc) in enumerate(categories):
cursor.execute("""
INSERT INTO categories (name, slug, icon, description, order_index)
VALUES (?, ?, ?, ?, ?)
""", (name, generate_slug(name), icon, desc, i))
# Apps with real Unsplash images
apps_data = [
# Browser Automation
("Playwright Cloud", "Browser Automation", "Paid", True, True,
"Scalable browser automation in the cloud with Playwright", "https://playwright.cloud",
None, "$99/month starter", 4.8, 12500,
"https://images.unsplash.com/photo-1633356122544-f134324a6cee?w=800&h=400&fit=crop"),
("Selenium Grid Hub", "Browser Automation", "Freemium", False, False,
"Distributed Selenium grid for parallel testing", "https://seleniumhub.io",
"https://github.com/seleniumhub/grid", "Free - $299/month", 4.2, 8400,
"https://images.unsplash.com/photo-1555066931-4365d14bab8c?w=800&h=400&fit=crop"),
("Puppeteer Extra", "Browser Automation", "Open Source", True, False,
"Enhanced Puppeteer with stealth plugins and more", "https://puppeteer-extra.dev",
"https://github.com/berstend/puppeteer-extra", "Free", 4.6, 15200,
"https://images.unsplash.com/photo-1461749280684-dccba630e2f6?w=800&h=400&fit=crop"),
# Proxy Services
("BrightData", "Proxy Services", "Paid", True, True,
"Premium proxy network with 72M+ IPs worldwide", "https://brightdata.com",
None, "Starting $500/month", 4.7, 9800,
"https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=800&h=400&fit=crop"),
("SmartProxy", "Proxy Services", "Paid", False, True,
"Residential and datacenter proxies with rotation", "https://smartproxy.com",
None, "Starting $75/month", 4.3, 7600,
"https://images.unsplash.com/photo-1544197150-b99a580bb7a8?w=800&h=400&fit=crop"),
("ProxyMesh", "Proxy Services", "Freemium", False, False,
"Rotating proxy servers with sticky sessions", "https://proxymesh.com",
None, "$10-$50/month", 4.0, 4200,
"https://images.unsplash.com/photo-1451187580459-43490279c0fa?w=800&h=400&fit=crop"),
# LLM Integration
("LangChain Crawl", "LLM Integration", "Open Source", True, False,
"LangChain integration for Crawl4AI workflows", "https://langchain-crawl.dev",
"https://github.com/langchain/crawl", "Free", 4.5, 18900,
"https://images.unsplash.com/photo-1677442136019-21780ecad995?w=800&h=400&fit=crop"),
("GPT Scraper", "LLM Integration", "Freemium", False, False,
"Extract structured data using GPT models", "https://gptscraper.ai",
None, "Free - $99/month", 4.1, 5600,
"https://images.unsplash.com/photo-1655720828018-edd2daec9349?w=800&h=400&fit=crop"),
("Claude Extract", "LLM Integration", "Paid", True, True,
"Professional extraction using Claude AI", "https://claude-extract.com",
None, "$199/month", 4.9, 3200,
"https://images.unsplash.com/photo-1686191128892-3b09ad503b4f?w=800&h=400&fit=crop"),
# Data Processing
("DataMiner Pro", "Data Processing", "Paid", False, False,
"Advanced data extraction and transformation", "https://dataminer.pro",
None, "$149/month", 4.2, 6700,
"https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=800&h=400&fit=crop"),
("ScraperAPI", "Data Processing", "Freemium", True, True,
"Simple API for web scraping with proxy rotation", "https://scraperapi.com",
None, "Free - $299/month", 4.6, 22300,
"https://images.unsplash.com/photo-1460925895917-afdab827c52f?w=800&h=400&fit=crop"),
("Apify", "Data Processing", "Freemium", False, False,
"Web scraping and automation platform", "https://apify.com",
None, "$49-$499/month", 4.4, 14500,
"https://images.unsplash.com/photo-1504639725590-34d0984388bd?w=800&h=400&fit=crop"),
# Cloud Infrastructure
("BrowserCloud", "Cloud Infrastructure", "Paid", True, True,
"Managed headless browsers in the cloud", "https://browsercloud.io",
None, "$199/month", 4.5, 8900,
"https://images.unsplash.com/photo-1667372393119-3d4c48d07fc9?w=800&h=400&fit=crop"),
("LambdaTest", "Cloud Infrastructure", "Freemium", False, False,
"Cross-browser testing on cloud", "https://lambdatest.com",
None, "Free - $99/month", 4.1, 11200,
"https://images.unsplash.com/photo-1451187580459-43490279c0fa?w=800&h=400&fit=crop"),
("Browserless", "Cloud Infrastructure", "Freemium", True, False,
"Headless browser automation API", "https://browserless.io",
None, "$50-$500/month", 4.7, 19800,
"https://images.unsplash.com/photo-1639762681485-074b7f938ba0?w=800&h=400&fit=crop"),
# Developer Tools
("Crawl4AI VSCode", "Developer Tools", "Open Source", True, False,
"VSCode extension for Crawl4AI development", "https://marketplace.visualstudio.com",
"https://github.com/crawl4ai/vscode", "Free", 4.8, 34500,
"https://images.unsplash.com/photo-1629654297299-c8506221ca97?w=800&h=400&fit=crop"),
("Postman Collection", "Developer Tools", "Open Source", False, False,
"Postman collection for Crawl4AI API testing", "https://postman.com/crawl4ai",
"https://github.com/crawl4ai/postman", "Free", 4.3, 7800,
"https://images.unsplash.com/photo-1599507593499-a3f7d7d97667?w=800&h=400&fit=crop"),
("Debug Toolkit", "Developer Tools", "Open Source", False, False,
"Debugging tools for crawler development", "https://debug.crawl4ai.com",
"https://github.com/crawl4ai/debug", "Free", 4.0, 4300,
"https://images.unsplash.com/photo-1515879218367-8466d910aaa4?w=800&h=400&fit=crop"),
]
for name, category, type_, featured, sponsored, desc, url, github, pricing, rating, downloads, image in apps_data:
screenshots = json.dumps([
f"https://images.unsplash.com/photo-{random.randint(1500000000000, 1700000000000)}-{random.randint(1000000000000, 9999999999999)}?w=800&h=600&fit=crop",
f"https://images.unsplash.com/photo-{random.randint(1500000000000, 1700000000000)}-{random.randint(1000000000000, 9999999999999)}?w=800&h=600&fit=crop"
])
cursor.execute("""
INSERT INTO apps (name, slug, description, category, type, featured, sponsored,
website_url, github_url, pricing, rating, downloads, image, screenshots, logo_url,
integration_guide, contact_email, views)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (name, generate_slug(name), desc, category, type_, featured, sponsored,
url, github, pricing, rating, downloads, image, screenshots,
f"https://ui-avatars.com/api/?name={name}&background=50ffff&color=070708&size=128",
f"# {name} Integration\n\n```python\nfrom crawl4ai import AsyncWebCrawler\n# Integration code coming soon...\n```",
f"contact@{generate_slug(name)}.com",
random.randint(100, 5000)))
# Articles with real images
articles_data = [
("Browser Automation Showdown: Playwright vs Puppeteer vs Selenium",
"Review", "John Doe", ["Playwright Cloud", "Puppeteer Extra"],
["browser-automation", "comparison", "2024"],
"https://images.unsplash.com/photo-1587620962725-abab7fe55159?w=1200&h=630&fit=crop"),
("Top 5 Proxy Services for Web Scraping in 2024",
"Comparison", "Jane Smith", ["BrightData", "SmartProxy", "ProxyMesh"],
["proxy", "web-scraping", "guide"],
"https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=1200&h=630&fit=crop"),
("Integrating LLMs with Crawl4AI: A Complete Guide",
"Tutorial", "Crawl4AI Team", ["LangChain Crawl", "GPT Scraper", "Claude Extract"],
["llm", "integration", "tutorial"],
"https://images.unsplash.com/photo-1677442136019-21780ecad995?w=1200&h=630&fit=crop"),
("Building Scalable Crawlers with Cloud Infrastructure",
"Tutorial", "Mike Johnson", ["BrowserCloud", "Browserless"],
["cloud", "scalability", "architecture"],
"https://images.unsplash.com/photo-1667372393119-3d4c48d07fc9?w=1200&h=630&fit=crop"),
("What's New in Crawl4AI Marketplace",
"News", "Crawl4AI Team", [],
["marketplace", "announcement", "news"],
"https://images.unsplash.com/photo-1556075798-4825dfaaf498?w=1200&h=630&fit=crop"),
("Cost Analysis: Self-Hosted vs Cloud Browser Solutions",
"Comparison", "Sarah Chen", ["BrowserCloud", "LambdaTest", "Browserless"],
["cost", "cloud", "comparison"],
"https://images.unsplash.com/photo-1554224155-8d04cb21cd6c?w=1200&h=630&fit=crop"),
("Getting Started with Browser Automation",
"Tutorial", "Crawl4AI Team", ["Playwright Cloud", "Selenium Grid Hub"],
["beginner", "tutorial", "automation"],
"https://images.unsplash.com/photo-1498050108023-c5249f4df085?w=1200&h=630&fit=crop"),
("The Future of Web Scraping: AI-Powered Extraction",
"News", "Dr. Alan Turing", ["Claude Extract", "GPT Scraper"],
["ai", "future", "trends"],
"https://images.unsplash.com/photo-1593720213428-28a5b9e94613?w=1200&h=630&fit=crop")
]
for title, category, author, related_apps, tags, image in articles_data:
# Get app IDs for related apps
related_ids = []
for app_name in related_apps:
cursor.execute("SELECT id FROM apps WHERE name = ?", (app_name,))
result = cursor.fetchone()
if result:
related_ids.append(result[0])
content = f"""# {title}
By {author} | {datetime.now().strftime('%B %d, %Y')}
## Introduction
This is a comprehensive article about {title.lower()}. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
## Key Points
- Important point about the topic
- Another crucial insight
- Technical details and specifications
- Performance comparisons
## Conclusion
In summary, this article explored various aspects of the topic. Stay tuned for more updates!
"""
cursor.execute("""
INSERT INTO articles (title, slug, content, author, category, related_apps,
featured_image, tags, views)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (title, generate_slug(title), content, author, category,
json.dumps(related_ids), image, json.dumps(tags),
random.randint(200, 10000)))
# Sponsors
sponsors_data = [
("BrightData", "Gold", "https://brightdata.com",
"https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=728&h=90&fit=crop"),
("ScraperAPI", "Gold", "https://scraperapi.com",
"https://images.unsplash.com/photo-1460925895917-afdab827c52f?w=728&h=90&fit=crop"),
("BrowserCloud", "Silver", "https://browsercloud.io",
"https://images.unsplash.com/photo-1667372393119-3d4c48d07fc9?w=728&h=90&fit=crop"),
("Claude Extract", "Silver", "https://claude-extract.com",
"https://images.unsplash.com/photo-1686191128892-3b09ad503b4f?w=728&h=90&fit=crop"),
("SmartProxy", "Bronze", "https://smartproxy.com",
"https://images.unsplash.com/photo-1544197150-b99a580bb7a8?w=728&h=90&fit=crop")
]
for company, tier, landing_url, banner in sponsors_data:
start_date = datetime.now() - timedelta(days=random.randint(1, 30))
end_date = datetime.now() + timedelta(days=random.randint(30, 180))
cursor.execute("""
INSERT INTO sponsors (company_name, logo_url, tier, banner_url,
landing_url, active, start_date, end_date)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", (company,
f"https://ui-avatars.com/api/?name={company}&background=09b5a5&color=fff&size=200",
tier, banner, landing_url, 1,
start_date.isoformat(), end_date.isoformat()))
conn.commit()
print("✓ Dummy data generated successfully!")
print(f" - {len(categories)} categories")
print(f" - {len(apps_data)} apps")
print(f" - {len(articles_data)} articles")
print(f" - {len(sponsors_data)} sponsors")
if __name__ == "__main__":
generate_dummy_data()

5
docs/md_v2/marketplace/backend/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
fastapi
uvicorn
pyyaml
python-multipart
python-dotenv

75
docs/md_v2/marketplace/backend/schema.yaml Normal file
View File

@@ -0,0 +1,75 @@
database:
name: marketplace.db
tables:
apps:
columns:
id: {type: INTEGER, primary: true, autoincrement: true}
name: {type: TEXT, required: true}
slug: {type: TEXT, unique: true}
description: {type: TEXT}
long_description: {type: TEXT}
logo_url: {type: TEXT}
image: {type: TEXT}
screenshots: {type: JSON, default: '[]'}
category: {type: TEXT}
type: {type: TEXT, default: 'Open Source'}
status: {type: TEXT, default: 'Active'}
website_url: {type: TEXT}
github_url: {type: TEXT}
demo_url: {type: TEXT}
video_url: {type: TEXT}
documentation_url: {type: TEXT}
support_url: {type: TEXT}
discord_url: {type: TEXT}
pricing: {type: TEXT}
rating: {type: REAL, default: 0.0}
downloads: {type: INTEGER, default: 0}
featured: {type: BOOLEAN, default: 0}
sponsored: {type: BOOLEAN, default: 0}
integration_guide: {type: TEXT}
documentation: {type: TEXT}
examples: {type: TEXT}
installation_command: {type: TEXT}
requirements: {type: TEXT}
changelog: {type: TEXT}
tags: {type: JSON, default: '[]'}
added_date: {type: DATETIME, default: CURRENT_TIMESTAMP}
updated_date: {type: DATETIME, default: CURRENT_TIMESTAMP}
contact_email: {type: TEXT}
views: {type: INTEGER, default: 0}
articles:
columns:
id: {type: INTEGER, primary: true, autoincrement: true}
title: {type: TEXT, required: true}
slug: {type: TEXT, unique: true}
content: {type: TEXT}
author: {type: TEXT, default: 'Crawl4AI Team'}
category: {type: TEXT}
related_apps: {type: JSON, default: '[]'}
featured_image: {type: TEXT}
published_date: {type: DATETIME, default: CURRENT_TIMESTAMP}
tags: {type: JSON, default: '[]'}
views: {type: INTEGER, default: 0}
categories:
columns:
id: {type: INTEGER, primary: true, autoincrement: true}
name: {type: TEXT, unique: true}
slug: {type: TEXT, unique: true}
icon: {type: TEXT}
description: {type: TEXT}
order_index: {type: INTEGER, default: 0}
sponsors:
columns:
id: {type: INTEGER, primary: true, autoincrement: true}
company_name: {type: TEXT, required: true}
logo_url: {type: TEXT}
tier: {type: TEXT, default: 'Bronze'}
banner_url: {type: TEXT}
landing_url: {type: TEXT}
active: {type: BOOLEAN, default: 1}
start_date: {type: DATETIME}
end_date: {type: DATETIME}

493
docs/md_v2/marketplace/backend/server.py Normal file
View File

@@ -0,0 +1,493 @@
from fastapi import FastAPI, HTTPException, Query, Depends, Body, UploadFile, File, Form, APIRouter
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from fastapi.staticfiles import StaticFiles
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from typing import Optional, Dict, Any
import json
import hashlib
import secrets
import re
from pathlib import Path
from database import DatabaseManager
from datetime import datetime, timedelta
# Import configuration (will exit if .env not found or invalid)
from config import Config
app = FastAPI(title="Crawl4AI Marketplace API")
router = APIRouter(prefix="/marketplace/api")
# Security setup
security = HTTPBearer()
tokens = {} # In production, use Redis or database for token storage
# CORS configuration
app.add_middleware(
CORSMiddleware,
allow_origins=Config.ALLOWED_ORIGINS,
allow_credentials=True,
allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
allow_headers=["*"],
max_age=3600
)
# Initialize database with configurable path
db = DatabaseManager(Config.DATABASE_PATH)
BASE_DIR = Path(__file__).parent
UPLOAD_ROOT = BASE_DIR / "uploads"
UPLOAD_ROOT.mkdir(parents=True, exist_ok=True)
app.mount("/uploads", StaticFiles(directory=UPLOAD_ROOT), name="uploads")
ALLOWED_IMAGE_TYPES = {
"image/png": ".png",
"image/jpeg": ".jpg",
"image/webp": ".webp",
"image/svg+xml": ".svg"
}
ALLOWED_UPLOAD_FOLDERS = {"sponsors"}
MAX_UPLOAD_SIZE = 2 * 1024 * 1024 # 2 MB
def json_response(data, cache_time=3600):
"""Helper to return JSON with cache headers"""
return JSONResponse(
content=data,
headers={
"Cache-Control": f"public, max-age={cache_time}",
"X-Content-Type-Options": "nosniff"
}
)
def to_int(value, default=0):
"""Coerce incoming values to integers, falling back to default."""
if value is None:
return default
if isinstance(value, bool):
return int(value)
if isinstance(value, (int, float)):
return int(value)
if isinstance(value, str):
stripped = value.strip()
if not stripped:
return default
match = re.match(r"^-?\d+", stripped)
if match:
try:
return int(match.group())
except ValueError:
return default
return default
# ============= PUBLIC ENDPOINTS =============
@router.get("/apps")
async def get_apps(
category: Optional[str] = None,
type: Optional[str] = None,
featured: Optional[bool] = None,
sponsored: Optional[bool] = None,
limit: int = Query(default=20, le=10000),
offset: int = Query(default=0)
):
"""Get apps with optional filters"""
where_clauses = []
if category:
where_clauses.append(f"category = '{category}'")
if type:
where_clauses.append(f"type = '{type}'")
if featured is not None:
where_clauses.append(f"featured = {1 if featured else 0}")
if sponsored is not None:
where_clauses.append(f"sponsored = {1 if sponsored else 0}")
where = " AND ".join(where_clauses) if where_clauses else None
apps = db.get_all('apps', limit=limit, offset=offset, where=where)
# Parse JSON fields
for app in apps:
if app.get('screenshots'):
app['screenshots'] = json.loads(app['screenshots'])
return json_response(apps)
@router.get("/apps/{slug}")
async def get_app(slug: str):
"""Get single app by slug"""
apps = db.get_all('apps', where=f"slug = '{slug}'", limit=1)
if not apps:
raise HTTPException(status_code=404, detail="App not found")
app = apps[0]
if app.get('screenshots'):
app['screenshots'] = json.loads(app['screenshots'])
return json_response(app)
@router.get("/articles")
async def get_articles(
category: Optional[str] = None,
limit: int = Query(default=20, le=10000),
offset: int = Query(default=0)
):
"""Get articles with optional category filter"""
where = f"category = '{category}'" if category else None
articles = db.get_all('articles', limit=limit, offset=offset, where=where)
# Parse JSON fields
for article in articles:
if article.get('related_apps'):
article['related_apps'] = json.loads(article['related_apps'])
if article.get('tags'):
article['tags'] = json.loads(article['tags'])
return json_response(articles)
@router.get("/articles/{slug}")
async def get_article(slug: str):
"""Get single article by slug"""
articles = db.get_all('articles', where=f"slug = '{slug}'", limit=1)
if not articles:
raise HTTPException(status_code=404, detail="Article not found")
article = articles[0]
if article.get('related_apps'):
article['related_apps'] = json.loads(article['related_apps'])
if article.get('tags'):
article['tags'] = json.loads(article['tags'])
return json_response(article)
@router.get("/categories")
async def get_categories():
"""Get all categories ordered by index"""
categories = db.get_all('categories', limit=50)
for category in categories:
category['order_index'] = to_int(category.get('order_index'), 0)
categories.sort(key=lambda x: x.get('order_index', 0))
return json_response(categories, cache_time=7200)
@router.get("/sponsors")
async def get_sponsors(active: Optional[bool] = True):
"""Get sponsors, default active only"""
where = f"active = {1 if active else 0}" if active is not None else None
sponsors = db.get_all('sponsors', where=where, limit=20)
# Filter by date if active
if active:
now = datetime.now().isoformat()
sponsors = [s for s in sponsors
if (not s.get('start_date') or s['start_date'] <= now) and
(not s.get('end_date') or s['end_date'] >= now)]
return json_response(sponsors)
@router.get("/search")
async def search(q: str = Query(min_length=2)):
"""Search across apps and articles"""
if len(q) < 2:
return json_response({})
results = db.search(q, tables=['apps', 'articles'])
# Parse JSON fields in results
for table, items in results.items():
for item in items:
if table == 'apps' and item.get('screenshots'):
item['screenshots'] = json.loads(item['screenshots'])
elif table == 'articles':
if item.get('related_apps'):
item['related_apps'] = json.loads(item['related_apps'])
if item.get('tags'):
item['tags'] = json.loads(item['tags'])
return json_response(results, cache_time=1800)
@router.get("/stats")
async def get_stats():
"""Get marketplace statistics"""
stats = {
"total_apps": len(db.get_all('apps', limit=10000)),
"total_articles": len(db.get_all('articles', limit=10000)),
"total_categories": len(db.get_all('categories', limit=1000)),
"active_sponsors": len(db.get_all('sponsors', where="active = 1", limit=1000))
}
return json_response(stats, cache_time=1800)
# ============= ADMIN AUTHENTICATION =============
def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Verify admin authentication token"""
token = credentials.credentials
if token not in tokens or tokens[token] < datetime.now():
raise HTTPException(status_code=401, detail="Invalid or expired token")
return token
@router.post("/admin/upload-image", dependencies=[Depends(verify_token)])
async def upload_image(file: UploadFile = File(...), folder: str = Form("sponsors")):
"""Upload image files for admin assets"""
folder = (folder or "").strip().lower()
if folder not in ALLOWED_UPLOAD_FOLDERS:
raise HTTPException(status_code=400, detail="Invalid upload folder")
if file.content_type not in ALLOWED_IMAGE_TYPES:
raise HTTPException(status_code=400, detail="Unsupported file type")
contents = await file.read()
if len(contents) > MAX_UPLOAD_SIZE:
raise HTTPException(status_code=400, detail="File too large (max 2MB)")
extension = ALLOWED_IMAGE_TYPES[file.content_type]
filename = f"{datetime.now().strftime('%Y%m%d%H%M%S')}_{secrets.token_hex(8)}{extension}"
target_dir = UPLOAD_ROOT / folder
target_dir.mkdir(parents=True, exist_ok=True)
target_path = target_dir / filename
target_path.write_bytes(contents)
return {"url": f"/uploads/{folder}/{filename}"}
@router.post("/admin/login")
async def admin_login(password: str = Body(..., embed=True)):
"""Admin login with password"""
provided_hash = hashlib.sha256(password.encode()).hexdigest()
if provided_hash != Config.ADMIN_PASSWORD_HASH:
# Log failed attempt in production
print(f"Failed login attempt at {datetime.now()}")
raise HTTPException(status_code=401, detail="Invalid password")
# Generate secure token
token = secrets.token_urlsafe(32)
tokens[token] = datetime.now() + timedelta(hours=Config.TOKEN_EXPIRY_HOURS)
return {
"token": token,
"expires_in": Config.TOKEN_EXPIRY_HOURS * 3600
}
# ============= ADMIN ENDPOINTS =============
@router.get("/admin/stats", dependencies=[Depends(verify_token)])
async def get_admin_stats():
"""Get detailed admin statistics"""
stats = {
"apps": {
"total": len(db.get_all('apps', limit=10000)),
"featured": len(db.get_all('apps', where="featured = 1", limit=10000)),
"sponsored": len(db.get_all('apps', where="sponsored = 1", limit=10000))
},
"articles": len(db.get_all('articles', limit=10000)),
"categories": len(db.get_all('categories', limit=1000)),
"sponsors": {
"active": len(db.get_all('sponsors', where="active = 1", limit=1000)),
"total": len(db.get_all('sponsors', limit=10000))
},
"total_views": sum(app.get('views', 0) for app in db.get_all('apps', limit=10000))
}
return stats
# Apps CRUD
@router.post("/admin/apps", dependencies=[Depends(verify_token)])
async def create_app(app_data: Dict[str, Any]):
"""Create new app"""
try:
# Handle JSON fields
for field in ['screenshots', 'tags']:
if field in app_data and isinstance(app_data[field], list):
app_data[field] = json.dumps(app_data[field])
cursor = db.conn.cursor()
columns = ', '.join(app_data.keys())
placeholders = ', '.join(['?' for _ in app_data])
cursor.execute(f"INSERT INTO apps ({columns}) VALUES ({placeholders})",
list(app_data.values()))
db.conn.commit()
return {"id": cursor.lastrowid, "message": "App created"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.put("/admin/apps/{app_id}", dependencies=[Depends(verify_token)])
async def update_app(app_id: int, app_data: Dict[str, Any]):
"""Update app"""
try:
# Handle JSON fields
for field in ['screenshots', 'tags']:
if field in app_data and isinstance(app_data[field], list):
app_data[field] = json.dumps(app_data[field])
set_clause = ', '.join([f"{k} = ?" for k in app_data.keys()])
cursor = db.conn.cursor()
cursor.execute(f"UPDATE apps SET {set_clause} WHERE id = ?",
list(app_data.values()) + [app_id])
db.conn.commit()
return {"message": "App updated"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/admin/apps/{app_id}", dependencies=[Depends(verify_token)])
async def delete_app(app_id: int):
"""Delete app"""
cursor = db.conn.cursor()
cursor.execute("DELETE FROM apps WHERE id = ?", (app_id,))
db.conn.commit()
return {"message": "App deleted"}
# Articles CRUD
@router.post("/admin/articles", dependencies=[Depends(verify_token)])
async def create_article(article_data: Dict[str, Any]):
"""Create new article"""
try:
for field in ['related_apps', 'tags']:
if field in article_data and isinstance(article_data[field], list):
article_data[field] = json.dumps(article_data[field])
cursor = db.conn.cursor()
columns = ', '.join(article_data.keys())
placeholders = ', '.join(['?' for _ in article_data])
cursor.execute(f"INSERT INTO articles ({columns}) VALUES ({placeholders})",
list(article_data.values()))
db.conn.commit()
return {"id": cursor.lastrowid, "message": "Article created"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.put("/admin/articles/{article_id}", dependencies=[Depends(verify_token)])
async def update_article(article_id: int, article_data: Dict[str, Any]):
"""Update article"""
try:
for field in ['related_apps', 'tags']:
if field in article_data and isinstance(article_data[field], list):
article_data[field] = json.dumps(article_data[field])
set_clause = ', '.join([f"{k} = ?" for k in article_data.keys()])
cursor = db.conn.cursor()
cursor.execute(f"UPDATE articles SET {set_clause} WHERE id = ?",
list(article_data.values()) + [article_id])
db.conn.commit()
return {"message": "Article updated"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/admin/articles/{article_id}", dependencies=[Depends(verify_token)])
async def delete_article(article_id: int):
"""Delete article"""
cursor = db.conn.cursor()
cursor.execute("DELETE FROM articles WHERE id = ?", (article_id,))
db.conn.commit()
return {"message": "Article deleted"}
# Categories CRUD
@router.post("/admin/categories", dependencies=[Depends(verify_token)])
async def create_category(category_data: Dict[str, Any]):
"""Create new category"""
try:
category_data = dict(category_data)
category_data['order_index'] = to_int(category_data.get('order_index'), 0)
cursor = db.conn.cursor()
columns = ', '.join(category_data.keys())
placeholders = ', '.join(['?' for _ in category_data])
cursor.execute(f"INSERT INTO categories ({columns}) VALUES ({placeholders})",
list(category_data.values()))
db.conn.commit()
return {"id": cursor.lastrowid, "message": "Category created"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.put("/admin/categories/{cat_id}", dependencies=[Depends(verify_token)])
async def update_category(cat_id: int, category_data: Dict[str, Any]):
"""Update category"""
try:
category_data = dict(category_data)
if 'order_index' in category_data:
category_data['order_index'] = to_int(category_data.get('order_index'), 0)
set_clause = ', '.join([f"{k} = ?" for k in category_data.keys()])
cursor = db.conn.cursor()
cursor.execute(f"UPDATE categories SET {set_clause} WHERE id = ?",
list(category_data.values()) + [cat_id])
db.conn.commit()
return {"message": "Category updated"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/admin/categories/{cat_id}", dependencies=[Depends(verify_token)])
async def delete_category(cat_id: int):
"""Delete category"""
try:
cursor = db.conn.cursor()
cursor.execute("DELETE FROM categories WHERE id = ?", (cat_id,))
db.conn.commit()
return {"message": "Category deleted"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
# Sponsors CRUD
@router.post("/admin/sponsors", dependencies=[Depends(verify_token)])
async def create_sponsor(sponsor_data: Dict[str, Any]):
"""Create new sponsor"""
try:
cursor = db.conn.cursor()
columns = ', '.join(sponsor_data.keys())
placeholders = ', '.join(['?' for _ in sponsor_data])
cursor.execute(f"INSERT INTO sponsors ({columns}) VALUES ({placeholders})",
list(sponsor_data.values()))
db.conn.commit()
return {"id": cursor.lastrowid, "message": "Sponsor created"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.put("/admin/sponsors/{sponsor_id}", dependencies=[Depends(verify_token)])
async def update_sponsor(sponsor_id: int, sponsor_data: Dict[str, Any]):
"""Update sponsor"""
try:
set_clause = ', '.join([f"{k} = ?" for k in sponsor_data.keys()])
cursor = db.conn.cursor()
cursor.execute(f"UPDATE sponsors SET {set_clause} WHERE id = ?",
list(sponsor_data.values()) + [sponsor_id])
db.conn.commit()
return {"message": "Sponsor updated"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/admin/sponsors/{sponsor_id}", dependencies=[Depends(verify_token)])
async def delete_sponsor(sponsor_id: int):
"""Delete sponsor"""
try:
cursor = db.conn.cursor()
cursor.execute("DELETE FROM sponsors WHERE id = ?", (sponsor_id,))
db.conn.commit()
return {"message": "Sponsor deleted"}
except Exception as e:
raise HTTPException(status_code=400, detail=str(e))
app.include_router(router)
@app.get("/")
async def root():
"""API info"""
return {
"name": "Crawl4AI Marketplace API",
"version": "1.0.0",
"endpoints": [
"/marketplace/api/apps",
"/marketplace/api/articles",
"/marketplace/api/categories",
"/marketplace/api/sponsors",
"/marketplace/api/search?q=query",
"/marketplace/api/stats"
]
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="127.0.0.1", port=8100)

2
docs/md_v2/marketplace/backend/uploads/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
*
!.gitignore

462
docs/md_v2/marketplace/frontend/app-detail.css Normal file
View File

@@ -0,0 +1,462 @@
/* App Detail Page Styles */
.app-detail-container {
min-height: 100vh;
background: var(--bg-dark);
}
/* Back Button */
.header-nav {
display: flex;
align-items: center;
}
.back-btn {
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--primary-cyan);
text-decoration: none;
transition: all 0.2s;
font-size: 0.875rem;
}
.back-btn:hover {
border-color: var(--primary-cyan);
background: rgba(80, 255, 255, 0.1);
}
/* App Hero Section */
.app-hero {
max-width: 1800px;
margin: 2rem auto;
padding: 0 2rem;
}
.app-hero-content {
display: grid;
grid-template-columns: 1fr 2fr;
gap: 3rem;
background: linear-gradient(135deg, #1a1a2e, #0f0f1e);
border: 2px solid var(--primary-cyan);
padding: 2rem;
box-shadow: 0 0 30px rgba(80, 255, 255, 0.15),
inset 0 0 20px rgba(80, 255, 255, 0.05);
}
.app-hero-image {
width: 100%;
height: 300px;
background: linear-gradient(135deg, rgba(80, 255, 255, 0.1), rgba(243, 128, 245, 0.05));
background-size: cover;
background-position: center;
border: 1px solid var(--border-color);
display: flex;
align-items: center;
justify-content: center;
font-size: 4rem;
color: var(--primary-cyan);
}
.app-badges {
display: flex;
gap: 0.5rem;
margin-bottom: 1rem;
}
.app-badge {
padding: 0.3rem 0.6rem;
background: var(--bg-tertiary);
color: var(--text-secondary);
font-size: 0.75rem;
text-transform: uppercase;
font-weight: 600;
}
.app-badge.featured {
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
color: var(--bg-dark);
box-shadow: 0 2px 10px rgba(80, 255, 255, 0.3);
}
.app-badge.sponsored {
background: linear-gradient(135deg, var(--warning), #ff8c00);
color: var(--bg-dark);
box-shadow: 0 2px 10px rgba(245, 158, 11, 0.3);
}
.app-hero-info h1 {
font-size: 2.5rem;
color: var(--primary-cyan);
margin: 0.5rem 0;
text-shadow: 0 0 20px rgba(80, 255, 255, 0.5);
}
.app-tagline {
font-size: 1.1rem;
color: var(--text-secondary);
margin-bottom: 2rem;
}
/* Stats */
.app-stats {
display: flex;
gap: 2rem;
margin: 2rem 0;
padding: 1rem 0;
border-top: 1px solid var(--border-color);
border-bottom: 1px solid var(--border-color);
}
.stat {
display: flex;
flex-direction: column;
gap: 0.25rem;
}
.stat-value {
font-size: 1.5rem;
color: var(--primary-cyan);
font-weight: 600;
}
.stat-label {
font-size: 0.875rem;
color: var(--text-tertiary);
}
/* Action Buttons */
.app-actions {
display: flex;
gap: 1rem;
margin: 2rem 0;
}
.action-btn {
padding: 0.75rem 1.5rem;
border: 1px solid var(--border-color);
background: transparent;
color: var(--text-primary);
text-decoration: none;
display: inline-flex;
align-items: center;
gap: 0.5rem;
transition: all 0.2s;
cursor: pointer;
font-family: inherit;
font-size: 0.9rem;
}
.action-btn.primary {
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
color: var(--bg-dark);
border-color: var(--primary-cyan);
font-weight: 600;
}
.action-btn.primary:hover {
box-shadow: 0 4px 15px rgba(80, 255, 255, 0.3);
transform: translateY(-2px);
}
.action-btn.secondary {
border-color: var(--accent-pink);
color: var(--accent-pink);
}
.action-btn.secondary:hover {
background: rgba(243, 128, 245, 0.1);
box-shadow: 0 4px 15px rgba(243, 128, 245, 0.2);
}
.action-btn.ghost {
border-color: var(--border-color);
color: var(--text-secondary);
}
.action-btn.ghost:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
/* Pricing */
.pricing-info {
display: flex;
align-items: center;
gap: 1rem;
font-size: 1.1rem;
}
.pricing-label {
color: var(--text-tertiary);
}
.pricing-value {
color: var(--warning);
font-weight: 600;
}
/* Navigation Tabs */
.app-nav {
max-width: 1800px;
margin: 2rem auto 0;
padding: 0 2rem;
display: flex;
gap: 1rem;
border-bottom: 2px solid var(--border-color);
}
.nav-tab {
padding: 1rem 1.5rem;
background: transparent;
border: none;
border-bottom: 2px solid transparent;
color: var(--text-secondary);
cursor: pointer;
transition: all 0.2s;
font-family: inherit;
font-size: 0.9rem;
margin-bottom: -2px;
}
.nav-tab:hover {
color: var(--primary-cyan);
}
.nav-tab.active {
color: var(--primary-cyan);
border-bottom-color: var(--primary-cyan);
}
/* Content Sections */
.app-content {
max-width: 1800px;
margin: 2rem auto;
padding: 0 2rem;
}
.tab-content {
display: none;
}
.tab-content.active {
display: block;
}
.docs-content {
max-width: 1200px;
padding: 2rem;
background: var(--bg-secondary);
border: 1px solid var(--border-color);
}
.docs-content h2 {
font-size: 1.8rem;
color: var(--primary-cyan);
margin-bottom: 1rem;
padding-bottom: 0.5rem;
border-bottom: 1px solid var(--border-color);
}
.docs-content h3 {
font-size: 1.3rem;
color: var(--text-primary);
margin: 2rem 0 1rem;
}
.docs-content h4 {
font-size: 1.1rem;
color: var(--accent-pink);
margin: 1.5rem 0 0.5rem;
}
.docs-content p {
color: var(--text-secondary);
line-height: 1.6;
margin-bottom: 1rem;
}
.docs-content code {
background: var(--bg-tertiary);
padding: 0.2rem 0.4rem;
color: var(--primary-cyan);
font-family: 'Dank Mono', Monaco, monospace;
font-size: 0.9em;
}
/* Code Blocks */
.code-block {
background: var(--bg-dark);
border: 1px solid var(--border-color);
margin: 1rem 0;
overflow: hidden;
}
.code-header {
display: flex;
justify-content: space-between;
align-items: center;
padding: 0.5rem 1rem;
background: var(--bg-tertiary);
border-bottom: 1px solid var(--border-color);
}
.code-lang {
color: var(--primary-cyan);
font-size: 0.875rem;
text-transform: uppercase;
}
.copy-btn {
padding: 0.25rem 0.5rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
cursor: pointer;
font-size: 0.75rem;
transition: all 0.2s;
}
.copy-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
.code-block pre {
margin: 0;
padding: 1rem;
overflow-x: auto;
}
.code-block code {
background: transparent;
padding: 0;
color: var(--text-secondary);
font-size: 0.875rem;
line-height: 1.5;
}
/* Feature Grid */
.feature-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
margin: 2rem 0;
}
.feature-card {
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
padding: 1.5rem;
transition: all 0.2s;
}
.feature-card:hover {
border-color: var(--primary-cyan);
background: rgba(80, 255, 255, 0.05);
}
.feature-card h4 {
margin-top: 0;
}
/* Info Box */
.info-box {
background: linear-gradient(135deg, rgba(80, 255, 255, 0.05), rgba(243, 128, 245, 0.03));
border: 1px solid var(--primary-cyan);
border-left: 4px solid var(--primary-cyan);
padding: 1.5rem;
margin: 2rem 0;
}
.info-box h4 {
margin-top: 0;
color: var(--primary-cyan);
}
/* Support Grid */
.support-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
margin: 2rem 0;
}
.support-card {
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
padding: 1.5rem;
text-align: center;
}
.support-card h3 {
color: var(--primary-cyan);
margin-bottom: 0.5rem;
}
/* Related Apps */
.related-apps {
max-width: 1800px;
margin: 4rem auto;
padding: 0 2rem;
}
.related-apps h2 {
font-size: 1.5rem;
color: var(--text-primary);
margin-bottom: 1.5rem;
}
.related-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
gap: 1rem;
}
.related-app-card {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
padding: 1rem;
cursor: pointer;
transition: all 0.2s;
}
.related-app-card:hover {
border-color: var(--primary-cyan);
transform: translateY(-2px);
}
/* Responsive */
@media (max-width: 1024px) {
.app-hero-content {
grid-template-columns: 1fr;
}
.app-stats {
justify-content: space-around;
}
}
@media (max-width: 768px) {
.app-hero-info h1 {
font-size: 2rem;
}
.app-actions {
flex-direction: column;
}
.app-nav {
overflow-x: auto;
gap: 0;
}
.nav-tab {
white-space: nowrap;
}
.feature-grid,
.support-grid {
grid-template-columns: 1fr;
}
}

234
docs/md_v2/marketplace/frontend/app-detail.html Normal file
View File

@@ -0,0 +1,234 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>App Details - Crawl4AI Marketplace</title>
<link rel="stylesheet" href="marketplace.css">
<link rel="stylesheet" href="app-detail.css">
</head>
<body>
<div class="app-detail-container">
<!-- Header -->
<header class="marketplace-header">
<div class="header-content">
<div class="header-left">
<div class="logo-title">
<img src="../../assets/images/logo.png" alt="Crawl4AI" class="header-logo">
<h1>
<span class="ascii-border">[</span>
Marketplace
<span class="ascii-border">]</span>
</h1>
</div>
</div>
<div class="header-nav">
<a href="index.html" class="back-btn">← Back to Marketplace</a>
</div>
</div>
</header>
<!-- App Hero Section -->
<section class="app-hero">
<div class="app-hero-content">
<div class="app-hero-image" id="app-image">
<!-- Dynamic image -->
</div>
<div class="app-hero-info">
<div class="app-badges">
<span class="app-badge" id="app-type">Open Source</span>
<span class="app-badge featured" id="app-featured" style="display:none">FEATURED</span>
<span class="app-badge sponsored" id="app-sponsored" style="display:none">SPONSORED</span>
</div>
<h1 id="app-name">App Name</h1>
<p id="app-description" class="app-tagline">App description goes here</p>
<div class="app-stats">
<div class="stat">
<span class="stat-value" id="app-rating">★★★★★</span>
<span class="stat-label">Rating</span>
</div>
<div class="stat">
<span class="stat-value" id="app-downloads">0</span>
<span class="stat-label">Downloads</span>
</div>
<div class="stat">
<span class="stat-value" id="app-category">Category</span>
<span class="stat-label">Category</span>
</div>
</div>
<div class="app-actions">
<a href="#" id="app-website" class="action-btn primary" target="_blank">
<span>→</span> Visit Website
</a>
<a href="#" id="app-github" class="action-btn secondary" target="_blank">
<span>⚡</span> View on GitHub
</a>
<button id="copy-integration" class="action-btn ghost">
<span>📋</span> Copy Integration
</button>
</div>
<div class="pricing-info">
<span class="pricing-label">Pricing:</span>
<span id="app-pricing" class="pricing-value">Free</span>
</div>
</div>
</div>
</section>
<!-- Navigation Tabs -->
<nav class="app-nav">
<button class="nav-tab active" data-tab="integration">Integration Guide</button>
<button class="nav-tab" data-tab="docs">Documentation</button>
<button class="nav-tab" data-tab="examples">Examples</button>
<button class="nav-tab" data-tab="support">Support</button>
</nav>
<!-- Content Sections -->
<main class="app-content">
<!-- Integration Guide Tab -->
<section id="integration-tab" class="tab-content active">
<div class="docs-content">
<h2>Quick Start</h2>
<p>Get started with this integration in just a few steps.</p>
<h3>Installation</h3>
<div class="code-block">
<div class="code-header">
<span class="code-lang">bash</span>
<button class="copy-btn">Copy</button>
</div>
<pre><code id="install-code">pip install crawl4ai</code></pre>
</div>
<h3>Basic Usage</h3>
<div class="code-block">
<div class="code-header">
<span class="code-lang">python</span>
<button class="copy-btn">Copy</button>
</div>
<pre><code id="usage-code">from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
# Your configuration here
)
print(result.markdown)
if __name__ == "__main__":
import asyncio
asyncio.run(main())</code></pre>
</div>
<h3>Advanced Configuration</h3>
<p>Customize the crawler with these advanced options:</p>
<div class="feature-grid">
<div class="feature-card">
<h4>🚀 Performance</h4>
<p>Optimize crawling speed with parallel processing and caching strategies.</p>
</div>
<div class="feature-card">
<h4>🔒 Authentication</h4>
<p>Handle login forms, cookies, and session management automatically.</p>
</div>
<div class="feature-card">
<h4>🎯 Extraction</h4>
<p>Use CSS selectors, XPath, or AI-powered content extraction.</p>
</div>
<div class="feature-card">
<h4>🔄 Proxy Support</h4>
<p>Rotate proxies and bypass rate limiting with built-in proxy management.</p>
</div>
</div>
<h3>Integration Example</h3>
<div class="code-block">
<div class="code-header">
<span class="code-lang">python</span>
<button class="copy-btn">Copy</button>
</div>
<pre><code id="integration-code">from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
async def extract_with_llm():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
extraction_strategy=LLMExtractionStrategy(
provider="openai",
api_key="your-api-key",
instruction="Extract product information"
),
bypass_cache=True
)
return result.extracted_content
# Run the extraction
data = await extract_with_llm()
print(data)</code></pre>
</div>
<div class="info-box">
<h4>💡 Pro Tip</h4>
<p>Use the <code>bypass_cache=True</code> parameter when you need fresh data, or set <code>cache_mode="write"</code> to update the cache with new content.</p>
</div>
</div>
</section>
<!-- Documentation Tab -->
<section id="docs-tab" class="tab-content">
<div class="docs-content">
<h2>Documentation</h2>
<p>Complete documentation and API reference.</p>
<!-- Dynamic content loaded here -->
</div>
</section>
<!-- Examples Tab -->
<section id="examples-tab" class="tab-content">
<div class="docs-content">
<h2>Examples</h2>
<p>Real-world examples and use cases.</p>
<!-- Dynamic content loaded here -->
</div>
</section>
<!-- Support Tab -->
<section id="support-tab" class="tab-content">
<div class="docs-content">
<h2>Support</h2>
<div class="support-grid">
<div class="support-card">
<h3>📧 Contact</h3>
<p id="app-contact">contact@example.com</p>
</div>
<div class="support-card">
<h3>🐛 Report Issues</h3>
<p>Found a bug? Report it on GitHub Issues.</p>
</div>
<div class="support-card">
<h3>💬 Community</h3>
<p>Join our Discord for help and discussions.</p>
</div>
</div>
</div>
</section>
</main>
<!-- Related Apps -->
<section class="related-apps">
<h2>Related Apps</h2>
<div id="related-apps-grid" class="related-grid">
<!-- Dynamic related apps -->
</div>
</section>
</div>
<script src="app-detail.js"></script>
</body>
</html>

334
docs/md_v2/marketplace/frontend/app-detail.js Normal file
View File

@@ -0,0 +1,334 @@
// App Detail Page JavaScript
const { API_BASE, API_ORIGIN } = (() => {
const { hostname, port, protocol } = window.location;
const isLocalHost = ['localhost', '127.0.0.1', '0.0.0.0'].includes(hostname);
if (isLocalHost && port && port !== '8100') {
const origin = `${protocol}//127.0.0.1:8100`;
return { API_BASE: `${origin}/marketplace/api`, API_ORIGIN: origin };
}
return { API_BASE: '/marketplace/api', API_ORIGIN: '' };
})();
class AppDetailPage {
constructor() {
this.appSlug = this.getAppSlugFromURL();
this.appData = null;
this.init();
}
getAppSlugFromURL() {
const params = new URLSearchParams(window.location.search);
return params.get('app') || '';
}
async init() {
if (!this.appSlug) {
window.location.href = 'index.html';
return;
}
await this.loadAppDetails();
this.setupEventListeners();
await this.loadRelatedApps();
}
async loadAppDetails() {
try {
const response = await fetch(`${API_BASE}/apps/${this.appSlug}`);
if (!response.ok) throw new Error('App not found');
this.appData = await response.json();
this.renderAppDetails();
} catch (error) {
console.error('Error loading app details:', error);
// Fallback to loading all apps and finding the right one
try {
const response = await fetch(`${API_BASE}/apps`);
const apps = await response.json();
this.appData = apps.find(app => app.slug === this.appSlug || app.name.toLowerCase().replace(/\s+/g, '-') === this.appSlug);
if (this.appData) {
this.renderAppDetails();
} else {
window.location.href = 'index.html';
}
} catch (err) {
console.error('Error loading apps:', err);
window.location.href = 'index.html';
}
}
}
renderAppDetails() {
if (!this.appData) return;
// Update title
document.title = `${this.appData.name} - Crawl4AI Marketplace`;
// Hero image
const appImage = document.getElementById('app-image');
if (this.appData.image) {
appImage.style.backgroundImage = `url('${this.appData.image}')`;
appImage.innerHTML = '';
} else {
appImage.innerHTML = `[${this.appData.category || 'APP'}]`;
}
// Basic info
document.getElementById('app-name').textContent = this.appData.name;
document.getElementById('app-description').textContent = this.appData.description;
document.getElementById('app-type').textContent = this.appData.type || 'Open Source';
document.getElementById('app-category').textContent = this.appData.category;
document.getElementById('app-pricing').textContent = this.appData.pricing || 'Free';
// Badges
if (this.appData.featured) {
document.getElementById('app-featured').style.display = 'inline-block';
}
if (this.appData.sponsored) {
document.getElementById('app-sponsored').style.display = 'inline-block';
}
// Stats
const rating = this.appData.rating || 0;
const stars = '★'.repeat(Math.floor(rating)) + '☆'.repeat(5 - Math.floor(rating));
document.getElementById('app-rating').textContent = stars + ` ${rating}/5`;
document.getElementById('app-downloads').textContent = this.formatNumber(this.appData.downloads || 0);
// Action buttons
const websiteBtn = document.getElementById('app-website');
const githubBtn = document.getElementById('app-github');
if (this.appData.website_url) {
websiteBtn.href = this.appData.website_url;
} else {
websiteBtn.style.display = 'none';
}
if (this.appData.github_url) {
githubBtn.href = this.appData.github_url;
} else {
githubBtn.style.display = 'none';
}
// Contact
document.getElementById('app-contact').textContent = this.appData.contact_email || 'Not available';
// Integration guide
this.renderIntegrationGuide();
}
renderIntegrationGuide() {
// Installation code
const installCode = document.getElementById('install-code');
if (this.appData.type === 'Open Source' && this.appData.github_url) {
installCode.textContent = `# Clone from GitHub
git clone ${this.appData.github_url}
# Install dependencies
pip install -r requirements.txt`;
} else if (this.appData.name.toLowerCase().includes('api')) {
installCode.textContent = `# Install via pip
pip install ${this.appData.slug}
# Or install from source
pip install git+${this.appData.github_url || 'https://github.com/example/repo'}`;
}
// Usage code - customize based on category
const usageCode = document.getElementById('usage-code');
if (this.appData.category === 'Browser Automation') {
usageCode.textContent = `from crawl4ai import AsyncWebCrawler
from ${this.appData.slug.replace(/-/g, '_')} import ${this.appData.name.replace(/\s+/g, '')}
async def main():
# Initialize ${this.appData.name}
automation = ${this.appData.name.replace(/\s+/g, '')}()
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
browser_config=automation.config,
wait_for="css:body"
)
print(result.markdown)`;
} else if (this.appData.category === 'Proxy Services') {
usageCode.textContent = `from crawl4ai import AsyncWebCrawler
import ${this.appData.slug.replace(/-/g, '_')}
# Configure proxy
proxy_config = {
"server": "${this.appData.website_url || 'https://proxy.example.com'}",
"username": "your_username",
"password": "your_password"
}
async with AsyncWebCrawler(proxy=proxy_config) as crawler:
result = await crawler.arun(
url="https://example.com",
bypass_cache=True
)
print(result.status_code)`;
} else if (this.appData.category === 'LLM Integration') {
usageCode.textContent = `from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
# Configure LLM extraction
strategy = LLMExtractionStrategy(
provider="${this.appData.name.toLowerCase().includes('gpt') ? 'openai' : 'anthropic'}",
api_key="your-api-key",
model="${this.appData.name.toLowerCase().includes('gpt') ? 'gpt-4' : 'claude-3'}",
instruction="Extract structured data"
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
extraction_strategy=strategy
)
print(result.extracted_content)`;
}
// Integration example
const integrationCode = document.getElementById('integration-code');
integrationCode.textContent = this.appData.integration_guide ||
`# Complete ${this.appData.name} Integration Example
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
import json
async def crawl_with_${this.appData.slug.replace(/-/g, '_')}():
"""
Complete example showing how to use ${this.appData.name}
with Crawl4AI for production web scraping
"""
# Define extraction schema
schema = {
"name": "ProductList",
"baseSelector": "div.product",
"fields": [
{"name": "title", "selector": "h2", "type": "text"},
{"name": "price", "selector": ".price", "type": "text"},
{"name": "image", "selector": "img", "type": "attribute", "attribute": "src"},
{"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
]
}
# Initialize crawler with ${this.appData.name}
async with AsyncWebCrawler(
browser_type="chromium",
headless=True,
verbose=True
) as crawler:
# Crawl with extraction
result = await crawler.arun(
url="https://example.com/products",
extraction_strategy=JsonCssExtractionStrategy(schema),
cache_mode="bypass",
wait_for="css:.product",
screenshot=True
)
# Process results
if result.success:
products = json.loads(result.extracted_content)
print(f"Found {len(products)} products")
for product in products[:5]:
print(f"- {product['title']}: {product['price']}")
return products
# Run the crawler
if __name__ == "__main__":
import asyncio
asyncio.run(crawl_with_${this.appData.slug.replace(/-/g, '_')}())`;
}
formatNumber(num) {
if (num >= 1000000) {
return (num / 1000000).toFixed(1) + 'M';
} else if (num >= 1000) {
return (num / 1000).toFixed(1) + 'K';
}
return num.toString();
}
setupEventListeners() {
// Tab switching
const tabs = document.querySelectorAll('.nav-tab');
tabs.forEach(tab => {
tab.addEventListener('click', () => {
// Update active tab
tabs.forEach(t => t.classList.remove('active'));
tab.classList.add('active');
// Show corresponding content
const tabName = tab.dataset.tab;
document.querySelectorAll('.tab-content').forEach(content => {
content.classList.remove('active');
});
document.getElementById(`${tabName}-tab`).classList.add('active');
});
});
// Copy integration code
document.getElementById('copy-integration').addEventListener('click', () => {
const code = document.getElementById('integration-code').textContent;
navigator.clipboard.writeText(code).then(() => {
const btn = document.getElementById('copy-integration');
const originalText = btn.innerHTML;
btn.innerHTML = '<span>✓</span> Copied!';
setTimeout(() => {
btn.innerHTML = originalText;
}, 2000);
});
});
// Copy code buttons
document.querySelectorAll('.copy-btn').forEach(btn => {
btn.addEventListener('click', (e) => {
const codeBlock = e.target.closest('.code-block');
const code = codeBlock.querySelector('code').textContent;
navigator.clipboard.writeText(code).then(() => {
btn.textContent = 'Copied!';
setTimeout(() => {
btn.textContent = 'Copy';
}, 2000);
});
});
});
}
async loadRelatedApps() {
try {
const response = await fetch(`${API_BASE}/apps?category=${encodeURIComponent(this.appData.category)}&limit=4`);
const apps = await response.json();
const relatedApps = apps.filter(app => app.slug !== this.appSlug).slice(0, 3);
const grid = document.getElementById('related-apps-grid');
grid.innerHTML = relatedApps.map(app => `
<div class="related-app-card" onclick="window.location.href='app-detail.html?app=${app.slug || app.name.toLowerCase().replace(/\s+/g, '-')}'">
<h4>${app.name}</h4>
<p>${app.description.substring(0, 100)}...</p>
<div style="display: flex; justify-content: space-between; margin-top: 0.5rem; font-size: 0.75rem;">
<span style="color: var(--primary-cyan)">${app.type}</span>
<span style="color: var(--warning)">★ ${app.rating}/5</span>
</div>
</div>
`).join('');
} catch (error) {
console.error('Error loading related apps:', error);
}
}
}
// Initialize when DOM is loaded
document.addEventListener('DOMContentLoaded', () => {
new AppDetailPage();
});

147
docs/md_v2/marketplace/frontend/index.html Normal file
View File

@@ -0,0 +1,147 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Marketplace - Crawl4AI</title>
<link rel="stylesheet" href="marketplace.css">
</head>
<body>
<div class="marketplace-container">
<!-- Header -->
<header class="marketplace-header">
<div class="header-content">
<div class="header-left">
<div class="logo-title">
<img src="../../assets/images/logo.png" alt="Crawl4AI" class="header-logo">
<h1>
<span class="ascii-border">[</span>
Marketplace
<span class="ascii-border">]</span>
</h1>
</div>
<p class="tagline">Tools, Integrations & Resources for Web Crawling</p>
</div>
<div class="header-stats" id="stats">
<span class="stat-item">Apps: <span id="total-apps">--</span></span>
<span class="stat-item">Articles: <span id="total-articles">--</span></span>
<span class="stat-item">Downloads: <span id="total-downloads">--</span></span>
</div>
</div>
</header>
<!-- Search and Category Bar -->
<div class="search-filter-bar">
<div class="search-box">
<span class="search-icon">></span>
<input type="text" id="search-input" placeholder="Search apps, articles, tools..." />
<kbd>/</kbd>
</div>
<div class="category-filter" id="category-filter">
<button class="filter-btn active" data-category="all">All</button>
<!-- Categories will be loaded here -->
</div>
</div>
<!-- Magazine Grid Layout -->
<main class="magazine-layout">
<!-- Hero Featured Section -->
<section class="hero-featured">
<div id="featured-hero" class="featured-hero-card">
<!-- Large featured card with big image -->
</div>
</section>
<!-- Secondary Featured -->
<section class="secondary-featured">
<div id="featured-secondary" class="featured-secondary-cards">
<!-- 2-3 medium featured cards with images -->
</div>
</section>
<!-- Sponsored Section -->
<section class="sponsored-section">
<div class="section-label">SPONSORED</div>
<div id="sponsored-content" class="sponsored-cards">
<!-- Sponsored content cards -->
</div>
</section>
<!-- Main Content Grid -->
<section class="main-content">
<!-- Apps Column -->
<div class="apps-column">
<div class="column-header">
<h2><span class="ascii-icon">></span> Latest Apps</h2>
<select id="type-filter" class="mini-filter">
<option value="">All</option>
<option value="Open Source">Open Source</option>
<option value="Paid">Paid</option>
</select>
</div>
<div id="apps-grid" class="apps-compact-grid">
<!-- Compact app cards -->
</div>
</div>
<!-- Articles Column -->
<div class="articles-column">
<div class="column-header">
<h2><span class="ascii-icon">></span> Latest Articles</h2>
</div>
<div id="articles-list" class="articles-compact-list">
<!-- Article items -->
</div>
</div>
<!-- Trending/Tools Column -->
<div class="trending-column">
<div class="column-header">
<h2><span class="ascii-icon">#</span> Trending</h2>
</div>
<div id="trending-list" class="trending-items">
<!-- Trending items -->
</div>
<div class="submit-box">
<h3><span class="ascii-icon">+</span> Submit Your Tool</h3>
<p>Share your integration</p>
<a href="mailto:marketplace@crawl4ai.com" class="submit-btn">Submit →</a>
</div>
</div>
</section>
<!-- More Apps Grid -->
<section class="more-apps">
<div class="section-header">
<h2><span class="ascii-icon">></span> More Apps</h2>
<button id="load-more" class="load-more-btn">Load More ↓</button>
</div>
<div id="more-apps-grid" class="more-apps-grid">
<!-- Additional app cards -->
</div>
</section>
</main>
<!-- Footer -->
<footer class="marketplace-footer">
<div class="footer-content">
<div class="footer-section">
<h3>About Marketplace</h3>
<p>Discover tools and integrations built by the Crawl4AI community.</p>
</div>
<div class="footer-section">
<h3>Become a Sponsor</h3>
<p>Reach developers building with Crawl4AI</p>
<a href="mailto:sponsors@crawl4ai.com" class="sponsor-btn">Learn More →</a>
</div>
</div>
<div class="footer-bottom">
<p>[ Crawl4AI Marketplace · Updated <span id="last-update">--</span> ]</p>
</div>
</footer>
</div>
<script src="marketplace.js"></script>
</body>
</html>

957
docs/md_v2/marketplace/frontend/marketplace.css Normal file
View File

@@ -0,0 +1,957 @@
/* Marketplace CSS - Magazine Style Terminal Theme */
@import url('../../assets/styles.css');
:root {
--primary-cyan: #50ffff;
--primary-teal: #09b5a5;
--accent-pink: #f380f5;
--bg-dark: #070708;
--bg-secondary: #1a1a1a;
--bg-tertiary: #3f3f44;
--text-primary: #e8e9ed;
--text-secondary: #d5cec0;
--text-tertiary: #a3abba;
--border-color: #3f3f44;
--success: #50ff50;
--error: #ff3c74;
--warning: #f59e0b;
}
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Dank Mono', Monaco, monospace;
background: var(--bg-dark);
color: var(--text-primary);
line-height: 1.6;
}
/* Global link styles */
a {
color: var(--primary-cyan);
text-decoration: none;
transition: color 0.2s;
}
a:hover {
color: var(--accent-pink);
}
.marketplace-container {
min-height: 100vh;
}
/* Header */
.marketplace-header {
background: var(--bg-secondary);
border-bottom: 1px solid var(--border-color);
padding: 1.5rem 0;
}
.header-content {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem;
display: flex;
justify-content: space-between;
align-items: center;
}
.logo-title {
display: flex;
align-items: center;
gap: 1rem;
}
.header-logo {
height: 40px;
width: auto;
filter: brightness(1.2);
}
.marketplace-header h1 {
font-size: 1.5rem;
color: var(--primary-cyan);
margin: 0;
}
.ascii-border {
color: var(--border-color);
}
.tagline {
font-size: 0.875rem;
color: var(--text-tertiary);
margin-top: 0.25rem;
}
.header-stats {
display: flex;
gap: 2rem;
}
.stat-item {
font-size: 0.875rem;
color: var(--text-secondary);
}
.stat-item span {
color: var(--primary-cyan);
font-weight: 600;
}
/* Search and Filter Bar */
.search-filter-bar {
max-width: 1800px;
margin: 1.5rem auto;
padding: 0 2rem;
display: flex;
gap: 1rem;
align-items: center;
}
.search-box {
flex: 1;
max-width: 500px;
display: flex;
align-items: center;
background: var(--bg-secondary);
border: 1px solid var(--border-color);
padding: 0.75rem 1rem;
transition: border-color 0.2s;
}
.search-box:focus-within {
border-color: var(--primary-cyan);
}
.search-icon {
color: var(--text-tertiary);
margin-right: 1rem;
}
#search-input {
flex: 1;
background: transparent;
border: none;
color: var(--text-primary);
font-family: inherit;
font-size: 0.9rem;
outline: none;
}
.search-box kbd {
font-size: 0.75rem;
padding: 0.2rem 0.5rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
color: var(--text-tertiary);
}
.category-filter {
display: flex;
gap: 0.5rem;
flex-wrap: wrap;
}
.filter-btn {
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
padding: 0.5rem 1rem;
font-family: inherit;
font-size: 0.875rem;
cursor: pointer;
transition: all 0.2s;
}
.filter-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
.filter-btn.active {
background: var(--primary-cyan);
color: var(--bg-dark);
border-color: var(--primary-cyan);
}
/* Magazine Layout */
.magazine-layout {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem 4rem;
display: grid;
grid-template-columns: 1fr;
gap: 2rem;
}
/* Hero Featured Section */
.hero-featured {
grid-column: 1 / -1;
position: relative;
}
.hero-featured::before {
content: '';
position: absolute;
top: -20px;
left: -20px;
right: -20px;
bottom: -20px;
background: radial-gradient(ellipse at center, rgba(80, 255, 255, 0.05), transparent 70%);
pointer-events: none;
z-index: -1;
}
.featured-hero-card {
background: linear-gradient(135deg, #1a1a2e, #0f0f1e);
border: 2px solid var(--primary-cyan);
box-shadow: 0 0 30px rgba(80, 255, 255, 0.15),
inset 0 0 20px rgba(80, 255, 255, 0.05);
height: 380px;
position: relative;
overflow: hidden;
cursor: pointer;
transition: all 0.3s ease;
display: flex;
flex-direction: column;
}
.featured-hero-card:hover {
border-color: var(--accent-pink);
box-shadow: 0 0 40px rgba(243, 128, 245, 0.2),
inset 0 0 30px rgba(243, 128, 245, 0.05);
transform: translateY(-2px);
}
.hero-image {
width: 100%;
height: 240px;
background: linear-gradient(135deg, rgba(80, 255, 255, 0.1), rgba(243, 128, 245, 0.05));
background-size: cover;
background-position: center;
display: flex;
align-items: center;
justify-content: center;
font-size: 3rem;
color: var(--primary-cyan);
flex-shrink: 0;
position: relative;
filter: brightness(1.1) contrast(1.1);
}
.hero-image::after {
content: '';
position: absolute;
bottom: 0;
left: 0;
right: 0;
height: 60%;
background: linear-gradient(to top, rgba(10, 10, 20, 0.95), transparent);
}
.hero-content {
padding: 1.5rem;
}
.hero-badge {
display: inline-block;
padding: 0.3rem 0.6rem;
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
color: var(--bg-dark);
font-size: 0.7rem;
text-transform: uppercase;
margin-bottom: 0.5rem;
font-weight: 600;
box-shadow: 0 2px 10px rgba(80, 255, 255, 0.3);
}
.hero-title {
font-size: 1.6rem;
color: var(--primary-cyan);
margin: 0.5rem 0;
text-shadow: 0 0 20px rgba(80, 255, 255, 0.5);
}
.hero-description {
color: var(--text-secondary);
line-height: 1.5;
}
.hero-meta {
display: flex;
gap: 1.5rem;
margin-top: 1rem;
font-size: 0.875rem;
}
.hero-meta span {
color: var(--text-tertiary);
}
.hero-meta span:first-child {
color: var(--warning);
}
/* Secondary Featured */
.secondary-featured {
grid-column: 1 / -1;
height: 380px;
display: flex;
align-items: stretch;
}
.featured-secondary-cards {
width: 100%;
display: flex;
flex-direction: column;
gap: 0.75rem;
justify-content: space-between;
}
.secondary-card {
background: linear-gradient(135deg, rgba(80, 255, 255, 0.03), rgba(243, 128, 245, 0.02));
border: 1px solid rgba(80, 255, 255, 0.3);
cursor: pointer;
transition: all 0.3s ease;
display: flex;
overflow: hidden;
height: calc((380px - 1.5rem) / 3);
flex: 1;
box-shadow: 0 2px 10px rgba(0, 0, 0, 0.3);
}
.secondary-card:hover {
border-color: var(--accent-pink);
background: linear-gradient(135deg, rgba(243, 128, 245, 0.05), rgba(80, 255, 255, 0.03));
box-shadow: 0 4px 15px rgba(243, 128, 245, 0.2);
transform: translateX(-3px);
}
.secondary-image {
width: 120px;
background: linear-gradient(135deg, var(--bg-tertiary), var(--bg-secondary));
background-size: cover;
background-position: center;
display: flex;
align-items: center;
justify-content: center;
font-size: 1.5rem;
color: var(--primary-cyan);
flex-shrink: 0;
}
.secondary-content {
flex: 1;
padding: 1rem;
display: flex;
flex-direction: column;
justify-content: space-between;
}
.secondary-title {
font-size: 1rem;
color: var(--text-primary);
margin-bottom: 0.25rem;
}
.secondary-desc {
font-size: 0.75rem;
color: var(--text-secondary);
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
.secondary-meta {
font-size: 0.75rem;
color: var(--text-tertiary);
}
.secondary-meta span:last-child {
color: var(--warning);
}
/* Sponsored Section */
.sponsored-section {
grid-column: 1 / -1;
background: var(--bg-secondary);
border: 1px solid var(--warning);
padding: 1rem;
position: relative;
}
.section-label {
position: absolute;
top: -0.5rem;
left: 1rem;
background: var(--bg-secondary);
padding: 0 0.5rem;
color: var(--warning);
font-size: 0.65rem;
letter-spacing: 0.1em;
}
.sponsored-cards {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
}
.sponsor-card {
padding: 1rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
}
.sponsor-card h4 {
color: var(--accent-pink);
margin-bottom: 0.5rem;
}
.sponsor-card p {
color: var(--text-secondary);
font-size: 0.85rem;
margin-bottom: 0.75rem;
}
.sponsor-card a {
color: var(--primary-cyan);
text-decoration: none;
font-size: 0.85rem;
}
.sponsor-card a:hover {
color: var(--accent-pink);
}
/* Main Content Grid */
.main-content {
grid-column: 1 / -1;
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 2rem;
}
/* Column Headers */
.column-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
border-bottom: 1px solid var(--border-color);
padding-bottom: 0.5rem;
}
.column-header h2 {
font-size: 1.1rem;
color: var(--text-primary);
}
.mini-filter {
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
color: var(--text-primary);
padding: 0.25rem 0.5rem;
font-family: inherit;
font-size: 0.75rem;
}
.ascii-icon {
color: var(--primary-cyan);
}
/* Apps Column */
.apps-compact-grid {
display: flex;
flex-direction: column;
gap: 0.75rem;
}
.app-compact {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
border-left: 3px solid var(--border-color);
padding: 0.75rem;
cursor: pointer;
transition: all 0.2s;
}
.app-compact:hover {
border-color: var(--primary-cyan);
border-left-color: var(--accent-pink);
transform: translateX(2px);
}
.app-compact-header {
display: flex;
justify-content: space-between;
font-size: 0.75rem;
color: var(--text-tertiary);
margin-bottom: 0.25rem;
}
.app-compact-header span:first-child {
color: var(--primary-cyan);
}
.app-compact-header span:last-child {
color: var(--warning);
}
.app-compact-title {
font-size: 0.9rem;
color: var(--text-primary);
margin-bottom: 0.25rem;
}
.app-compact-desc {
font-size: 0.75rem;
color: var(--text-secondary);
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
/* Articles Column */
.articles-compact-list {
display: flex;
flex-direction: column;
gap: 1rem;
}
.article-compact {
border-left: 2px solid var(--border-color);
padding-left: 1rem;
cursor: pointer;
transition: all 0.2s;
}
.article-compact:hover {
border-left-color: var(--primary-cyan);
}
.article-meta {
font-size: 0.7rem;
color: var(--text-tertiary);
margin-bottom: 0.25rem;
}
.article-meta span:first-child {
color: var(--accent-pink);
}
.article-title {
font-size: 0.9rem;
color: var(--text-primary);
margin-bottom: 0.25rem;
}
.article-author {
font-size: 0.75rem;
color: var(--text-secondary);
}
/* Trending Column */
.trending-items {
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.trending-item {
display: flex;
align-items: center;
gap: 0.75rem;
padding: 0.5rem;
background: var(--bg-secondary);
border: 1px solid var(--border-color);
cursor: pointer;
transition: all 0.2s;
}
.trending-item:hover {
border-color: var(--primary-cyan);
}
.trending-rank {
font-size: 1.2rem;
color: var(--primary-cyan);
width: 2rem;
text-align: center;
}
.trending-info {
flex: 1;
}
.trending-name {
font-size: 0.85rem;
color: var(--text-primary);
}
.trending-stats {
font-size: 0.7rem;
color: var(--text-tertiary);
}
/* Submit Box */
.submit-box {
margin-top: 1.5rem;
background: var(--bg-secondary);
border: 1px solid var(--primary-cyan);
padding: 1rem;
text-align: center;
}
.submit-box h3 {
font-size: 1rem;
color: var(--primary-cyan);
margin-bottom: 0.5rem;
}
.submit-box p {
font-size: 0.8rem;
color: var(--text-secondary);
margin-bottom: 0.75rem;
}
.submit-btn {
display: inline-block;
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--primary-cyan);
color: var(--primary-cyan);
text-decoration: none;
transition: all 0.2s;
}
.submit-btn:hover {
background: var(--primary-cyan);
color: var(--bg-dark);
}
/* More Apps Section */
.more-apps {
grid-column: 1 / -1;
margin-top: 2rem;
}
.section-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
}
.more-apps-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
gap: 1rem;
}
.load-more-btn {
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
padding: 0.5rem 1.5rem;
font-family: inherit;
cursor: pointer;
transition: all 0.2s;
}
.load-more-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
/* Footer */
.marketplace-footer {
background: var(--bg-secondary);
border-top: 1px solid var(--border-color);
margin-top: 4rem;
padding: 2rem 0;
}
.footer-content {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem;
display: grid;
grid-template-columns: 1fr 1fr;
gap: 2rem;
}
.footer-section h3 {
font-size: 1rem;
margin-bottom: 0.5rem;
color: var(--primary-cyan);
}
.footer-section p {
font-size: 0.875rem;
color: var(--text-secondary);
margin-bottom: 1rem;
}
.sponsor-btn {
display: inline-block;
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--primary-cyan);
color: var(--primary-cyan);
text-decoration: none;
transition: all 0.2s;
}
.sponsor-btn:hover {
background: var(--primary-cyan);
color: var(--bg-dark);
}
.footer-bottom {
max-width: 1800px;
margin: 2rem auto 0;
padding: 1rem 2rem 0;
border-top: 1px solid var(--border-color);
font-size: 0.75rem;
color: var(--text-tertiary);
}
/* Modal */
.modal {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.8);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
}
.modal.hidden {
display: none;
}
.modal-content {
background: var(--bg-secondary);
border: 1px solid var(--primary-cyan);
max-width: 800px;
width: 90%;
max-height: 80vh;
overflow-y: auto;
position: relative;
}
.modal-close {
position: absolute;
top: 1rem;
right: 1rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-primary);
padding: 0.25rem 0.5rem;
cursor: pointer;
font-size: 1.2rem;
}
.modal-close:hover {
border-color: var(--error);
color: var(--error);
}
.app-detail {
padding: 2rem;
}
.app-detail h2 {
font-size: 1.5rem;
margin-bottom: 1rem;
color: var(--primary-cyan);
}
/* Loading */
.loading {
text-align: center;
padding: 2rem;
color: var(--text-tertiary);
}
.no-results {
text-align: center;
padding: 2rem;
color: var(--text-tertiary);
}
/* Responsive - Tablet */
@media (min-width: 768px) {
.magazine-layout {
grid-template-columns: repeat(2, 1fr);
}
.hero-featured {
grid-column: 1 / -1;
}
.secondary-featured {
grid-column: 1 / -1;
}
.sponsored-section {
grid-column: 1 / -1;
}
.main-content {
grid-column: 1 / -1;
grid-template-columns: repeat(2, 1fr);
}
}
/* Responsive - Desktop */
@media (min-width: 1024px) {
.magazine-layout {
grid-template-columns: repeat(3, 1fr);
}
.hero-featured {
grid-column: 1 / 3;
grid-row: 1;
}
.secondary-featured {
grid-column: 3 / 4;
grid-row: 1;
}
.featured-secondary-cards {
flex-direction: column;
}
.sponsored-section {
grid-column: 1 / -1;
}
.main-content {
grid-column: 1 / -1;
grid-template-columns: repeat(3, 1fr);
}
}
/* Responsive - Wide Desktop */
@media (min-width: 1400px) {
.magazine-layout {
grid-template-columns: repeat(4, 1fr);
}
.hero-featured {
grid-column: 1 / 3;
}
.secondary-featured {
grid-column: 3 / 5;
grid-row: 1;
}
.featured-secondary-cards {
grid-template-columns: repeat(2, 1fr);
}
.main-content {
grid-template-columns: repeat(4, 1fr);
}
.apps-column {
grid-column: span 2;
}
.more-apps-grid {
grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
}
}
/* Responsive - Ultra Wide Desktop (for coders with wide monitors) */
@media (min-width: 1800px) {
.magazine-layout {
grid-template-columns: repeat(5, 1fr);
}
.hero-featured {
grid-column: 1 / 3;
}
.secondary-featured {
grid-column: 3 / 6;
}
.featured-secondary-cards {
grid-template-columns: repeat(3, 1fr);
}
.sponsored-section {
grid-column: 1 / -1;
}
.sponsored-cards {
grid-template-columns: repeat(5, 1fr);
}
.main-content {
grid-template-columns: repeat(5, 1fr);
}
.apps-column {
grid-column: span 2;
}
.articles-column {
grid-column: span 2;
}
.more-apps-grid {
grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
}
}
/* Responsive - Mobile */
@media (max-width: 767px) {
.header-content {
flex-direction: column;
gap: 1rem;
}
.search-filter-bar {
flex-direction: column;
align-items: stretch;
}
.search-box {
max-width: none;
}
.magazine-layout {
padding: 0 1rem 2rem;
}
.footer-content {
grid-template-columns: 1fr;
}
.secondary-card {
flex-direction: column;
}
.secondary-image {
width: 100%;
height: 150px;
}
}

395
docs/md_v2/marketplace/frontend/marketplace.js Normal file
View File

@@ -0,0 +1,395 @@
// Marketplace JS - Magazine Layout
const API_BASE = '/marketplace/api';
const CACHE_TTL = 3600000; // 1 hour in ms
class MarketplaceCache {
constructor() {
this.prefix = 'c4ai_market_';
}
get(key) {
const item = localStorage.getItem(this.prefix + key);
if (!item) return null;
const data = JSON.parse(item);
if (Date.now() > data.expires) {
localStorage.removeItem(this.prefix + key);
return null;
}
return data.value;
}
set(key, value, ttl = CACHE_TTL) {
const data = {
value: value,
expires: Date.now() + ttl
};
localStorage.setItem(this.prefix + key, JSON.stringify(data));
}
clear() {
Object.keys(localStorage)
.filter(k => k.startsWith(this.prefix))
.forEach(k => localStorage.removeItem(k));
}
}
class MarketplaceAPI {
constructor() {
this.cache = new MarketplaceCache();
this.searchTimeout = null;
}
async fetch(endpoint, useCache = true) {
const cacheKey = endpoint.replace(/[^\w]/g, '_');
if (useCache) {
const cached = this.cache.get(cacheKey);
if (cached) return cached;
}
try {
const response = await fetch(`${API_BASE}${endpoint}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const data = await response.json();
this.cache.set(cacheKey, data);
return data;
} catch (error) {
console.error('API Error:', error);
return null;
}
}
async getStats() {
return this.fetch('/stats');
}
async getCategories() {
return this.fetch('/categories');
}
async getApps(params = {}) {
const query = new URLSearchParams(params).toString();
return this.fetch(`/apps${query ? '?' + query : ''}`);
}
async getArticles(params = {}) {
const query = new URLSearchParams(params).toString();
return this.fetch(`/articles${query ? '?' + query : ''}`);
}
async getSponsors() {
return this.fetch('/sponsors');
}
async search(query) {
if (query.length < 2) return {};
return this.fetch(`/search?q=${encodeURIComponent(query)}`, false);
}
}
class MarketplaceUI {
constructor() {
this.api = new MarketplaceAPI();
this.currentCategory = 'all';
this.currentType = '';
this.searchTimeout = null;
this.loadedApps = 10;
this.init();
}
async init() {
await this.loadStats();
await this.loadCategories();
await this.loadFeaturedContent();
await this.loadSponsors();
await this.loadMainContent();
this.setupEventListeners();
}
async loadStats() {
const stats = await this.api.getStats();
if (stats) {
document.getElementById('total-apps').textContent = stats.total_apps || '0';
document.getElementById('total-articles').textContent = stats.total_articles || '0';
document.getElementById('total-downloads').textContent = stats.total_downloads || '0';
document.getElementById('last-update').textContent = new Date().toLocaleDateString();
}
}
async loadCategories() {
const categories = await this.api.getCategories();
if (!categories) return;
const filter = document.getElementById('category-filter');
categories.forEach(cat => {
const btn = document.createElement('button');
btn.className = 'filter-btn';
btn.dataset.category = cat.slug;
btn.textContent = cat.name;
btn.onclick = () => this.filterByCategory(cat.slug);
filter.appendChild(btn);
});
}
async loadFeaturedContent() {
// Load hero featured
const featured = await this.api.getApps({ featured: true, limit: 4 });
if (!featured || !featured.length) return;
// Hero card (first featured)
const hero = featured[0];
const heroCard = document.getElementById('featured-hero');
if (hero) {
const imageUrl = hero.image || '';
heroCard.innerHTML = `
<div class="hero-image" ${imageUrl ? `style="background-image: url('${imageUrl}')"` : ''}>
${!imageUrl ? `[${hero.category || 'APP'}]` : ''}
</div>
<div class="hero-content">
<span class="hero-badge">${hero.type || 'PAID'}</span>
<h2 class="hero-title">${hero.name}</h2>
<p class="hero-description">${hero.description}</p>
<div class="hero-meta">
<span>★ ${hero.rating || 0}/5</span>
<span>${hero.downloads || 0} downloads</span>
</div>
</div>
`;
heroCard.onclick = () => this.showAppDetail(hero);
}
// Secondary featured cards
const secondary = document.getElementById('featured-secondary');
secondary.innerHTML = '';
if (featured.length > 1) {
featured.slice(1, 4).forEach(app => {
const card = document.createElement('div');
card.className = 'secondary-card';
const imageUrl = app.image || '';
card.innerHTML = `
<div class="secondary-image" ${imageUrl ? `style="background-image: url('${imageUrl}')"` : ''}>
${!imageUrl ? `[${app.category || 'APP'}]` : ''}
</div>
<div class="secondary-content">
<h3 class="secondary-title">${app.name}</h3>
<p class="secondary-desc">${(app.description || '').substring(0, 100)}...</p>
<div class="secondary-meta">
<span>${app.type || 'Open Source'}</span> · <span>★ ${app.rating || 0}/5</span>
</div>
</div>
`;
card.onclick = () => this.showAppDetail(app);
secondary.appendChild(card);
});
}
}
async loadSponsors() {
const sponsors = await this.api.getSponsors();
if (!sponsors || !sponsors.length) {
// Show placeholder if no sponsors
const container = document.getElementById('sponsored-content');
container.innerHTML = `
<div class="sponsor-card">
<h4>Become a Sponsor</h4>
<p>Reach thousands of developers using Crawl4AI</p>
<a href="mailto:sponsors@crawl4ai.com">Contact Us →</a>
</div>
`;
return;
}
const container = document.getElementById('sponsored-content');
container.innerHTML = sponsors.slice(0, 5).map(sponsor => `
<div class="sponsor-card">
<h4>${sponsor.company_name}</h4>
<p>${sponsor.tier} Sponsor - Premium Solutions</p>
<a href="${sponsor.landing_url}" target="_blank">Learn More →</a>
</div>
`).join('');
}
async loadMainContent() {
// Load apps column
const apps = await this.api.getApps({ limit: 8 });
if (apps && apps.length) {
const appsGrid = document.getElementById('apps-grid');
appsGrid.innerHTML = apps.map(app => `
<div class="app-compact" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="app-compact-header">
<span>${app.category}</span>
<span>★ ${app.rating}/5</span>
</div>
<div class="app-compact-title">${app.name}</div>
<div class="app-compact-desc">${app.description}</div>
</div>
`).join('');
}
// Load articles column
const articles = await this.api.getArticles({ limit: 6 });
if (articles && articles.length) {
const articlesList = document.getElementById('articles-list');
articlesList.innerHTML = articles.map(article => `
<div class="article-compact" onclick="marketplace.showArticle('${article.id}')">
<div class="article-meta">
<span>${article.category}</span> · <span>${new Date(article.published_at).toLocaleDateString()}</span>
</div>
<div class="article-title">${article.title}</div>
<div class="article-author">by ${article.author}</div>
</div>
`).join('');
}
// Load trending
if (apps && apps.length) {
const trending = apps.slice(0, 5);
const trendingList = document.getElementById('trending-list');
trendingList.innerHTML = trending.map((app, i) => `
<div class="trending-item" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="trending-rank">${i + 1}</div>
<div class="trending-info">
<div class="trending-name">${app.name}</div>
<div class="trending-stats">${app.downloads} downloads</div>
</div>
</div>
`).join('');
}
// Load more apps grid
const moreApps = await this.api.getApps({ offset: 8, limit: 12 });
if (moreApps && moreApps.length) {
const moreGrid = document.getElementById('more-apps-grid');
moreGrid.innerHTML = moreApps.map(app => `
<div class="app-compact" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="app-compact-header">
<span>${app.category}</span>
<span>${app.type}</span>
</div>
<div class="app-compact-title">${app.name}</div>
</div>
`).join('');
}
}
setupEventListeners() {
// Search
const searchInput = document.getElementById('search-input');
searchInput.addEventListener('input', (e) => {
clearTimeout(this.searchTimeout);
this.searchTimeout = setTimeout(() => this.search(e.target.value), 300);
});
// Keyboard shortcut
document.addEventListener('keydown', (e) => {
if (e.key === '/' && !searchInput.contains(document.activeElement)) {
e.preventDefault();
searchInput.focus();
}
if (e.key === 'Escape' && searchInput.contains(document.activeElement)) {
searchInput.blur();
searchInput.value = '';
}
});
// Type filter
const typeFilter = document.getElementById('type-filter');
typeFilter.addEventListener('change', (e) => {
this.currentType = e.target.value;
this.loadMainContent();
});
// Load more
const loadMore = document.getElementById('load-more');
loadMore.addEventListener('click', () => this.loadMoreApps());
}
async filterByCategory(category) {
// Update active state
document.querySelectorAll('.filter-btn').forEach(btn => {
btn.classList.toggle('active', btn.dataset.category === category);
});
this.currentCategory = category;
await this.loadMainContent();
}
async search(query) {
if (!query) {
await this.loadMainContent();
return;
}
const results = await this.api.search(query);
if (!results) return;
// Update apps grid with search results
if (results.apps && results.apps.length) {
const appsGrid = document.getElementById('apps-grid');
appsGrid.innerHTML = results.apps.map(app => `
<div class="app-compact" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="app-compact-header">
<span>${app.category}</span>
<span>★ ${app.rating}/5</span>
</div>
<div class="app-compact-title">${app.name}</div>
<div class="app-compact-desc">${app.description}</div>
</div>
`).join('');
}
// Update articles with search results
if (results.articles && results.articles.length) {
const articlesList = document.getElementById('articles-list');
articlesList.innerHTML = results.articles.map(article => `
<div class="article-compact" onclick="marketplace.showArticle('${article.id}')">
<div class="article-meta">
<span>${article.category}</span> · <span>${new Date(article.published_at).toLocaleDateString()}</span>
</div>
<div class="article-title">${article.title}</div>
<div class="article-author">by ${article.author}</div>
</div>
`).join('');
}
}
async loadMoreApps() {
this.loadedApps += 12;
const moreApps = await this.api.getApps({ offset: this.loadedApps, limit: 12 });
if (moreApps && moreApps.length) {
const moreGrid = document.getElementById('more-apps-grid');
moreApps.forEach(app => {
const card = document.createElement('div');
card.className = 'app-compact';
card.innerHTML = `
<div class="app-compact-header">
<span>${app.category}</span>
<span>${app.type}</span>
</div>
<div class="app-compact-title">${app.name}</div>
`;
card.onclick = () => this.showAppDetail(app);
moreGrid.appendChild(card);
});
}
}
showAppDetail(app) {
// Navigate to detail page instead of showing modal
const slug = app.slug || app.name.toLowerCase().replace(/\s+/g, '-');
window.location.href = `app-detail.html?app=${slug}`;
}
showArticle(articleId) {
// Could create article detail page similarly
console.log('Show article:', articleId);
}
}
// Initialize marketplace
let marketplace;
document.addEventListener('DOMContentLoaded', () => {
marketplace = new MarketplaceUI();
});

147
docs/md_v2/marketplace/index.html Normal file
View File

@@ -0,0 +1,147 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Marketplace - Crawl4AI</title>
<link rel="stylesheet" href="marketplace.css">
</head>
<body>
<div class="marketplace-container">
<!-- Header -->
<header class="marketplace-header">
<div class="header-content">
<div class="header-left">
<div class="logo-title">
<img src="../assets/images/logo.png" alt="Crawl4AI" class="header-logo">
<h1>
<span class="ascii-border">[</span>
Marketplace
<span class="ascii-border">]</span>
</h1>
</div>
<p class="tagline">Tools, Integrations & Resources for Web Crawling</p>
</div>
<div class="header-stats" id="stats">
<span class="stat-item">Apps: <span id="total-apps">--</span></span>
<span class="stat-item">Articles: <span id="total-articles">--</span></span>
<span class="stat-item">Downloads: <span id="total-downloads">--</span></span>
</div>
</div>
</header>
<!-- Search and Category Bar -->
<div class="search-filter-bar">
<div class="search-box">
<span class="search-icon">></span>
<input type="text" id="search-input" placeholder="Search apps, articles, tools..." />
<kbd>/</kbd>
</div>
<div class="category-filter" id="category-filter">
<button class="filter-btn active" data-category="all">All</button>
<!-- Categories will be loaded here -->
</div>
</div>
<!-- Magazine Grid Layout -->
<main class="magazine-layout">
<!-- Hero Featured Section -->
<section class="hero-featured">
<div id="featured-hero" class="featured-hero-card">
<!-- Large featured card with big image -->
</div>
</section>
<!-- Secondary Featured -->
<section class="secondary-featured">
<div id="featured-secondary" class="featured-secondary-cards">
<!-- 2-3 medium featured cards with images -->
</div>
</section>
<!-- Sponsored Section -->
<section class="sponsored-section">
<div class="section-label">SPONSORED</div>
<div id="sponsored-content" class="sponsored-cards">
<!-- Sponsored content cards -->
</div>
</section>
<!-- Main Content Grid -->
<section class="main-content">
<!-- Apps Column -->
<div class="apps-column">
<div class="column-header">
<h2><span class="ascii-icon">></span> Latest Apps</h2>
<select id="type-filter" class="mini-filter">
<option value="">All</option>
<option value="Open Source">Open Source</option>
<option value="Paid">Paid</option>
</select>
</div>
<div id="apps-grid" class="apps-compact-grid">
<!-- Compact app cards -->
</div>
</div>
<!-- Articles Column -->
<div class="articles-column">
<div class="column-header">
<h2><span class="ascii-icon">></span> Latest Articles</h2>
</div>
<div id="articles-list" class="articles-compact-list">
<!-- Article items -->
</div>
</div>
<!-- Trending/Tools Column -->
<div class="trending-column">
<div class="column-header">
<h2><span class="ascii-icon">#</span> Trending</h2>
</div>
<div id="trending-list" class="trending-items">
<!-- Trending items -->
</div>
<div class="submit-box">
<h3><span class="ascii-icon">+</span> Submit Your Tool</h3>
<p>Share your integration</p>
<a href="mailto:marketplace@crawl4ai.com" class="submit-btn">Submit →</a>
</div>
</div>
</section>
<!-- More Apps Grid -->
<section class="more-apps">
<div class="section-header">
<h2><span class="ascii-icon">></span> More Apps</h2>
<button id="load-more" class="load-more-btn">Load More ↓</button>
</div>
<div id="more-apps-grid" class="more-apps-grid">
<!-- Additional app cards -->
</div>
</section>
</main>
<!-- Footer -->
<footer class="marketplace-footer">
<div class="footer-content">
<div class="footer-section">
<h3>About Marketplace</h3>
<p>Discover tools and integrations built by the Crawl4AI community.</p>
</div>
<div class="footer-section">
<h3>Become a Sponsor</h3>
<p>Reach developers building with Crawl4AI</p>
<a href="mailto:sponsors@crawl4ai.com" class="sponsor-btn">Learn More →</a>
</div>
</div>
<div class="footer-bottom">
<p>[ Crawl4AI Marketplace · Updated <span id="last-update">--</span> ]</p>
</div>
</footer>
</div>
<script src="marketplace.js"></script>
</body>
</html>

994
docs/md_v2/marketplace/marketplace.css Normal file
View File

@@ -0,0 +1,994 @@
/* Marketplace CSS - Magazine Style Terminal Theme */
@import url('../../assets/styles.css');
:root {
--primary-cyan: #50ffff;
--primary-teal: #09b5a5;
--accent-pink: #f380f5;
--bg-dark: #070708;
--bg-secondary: #1a1a1a;
--bg-tertiary: #3f3f44;
--text-primary: #e8e9ed;
--text-secondary: #d5cec0;
--text-tertiary: #a3abba;
--border-color: #3f3f44;
--success: #50ff50;
--error: #ff3c74;
--warning: #f59e0b;
}
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Dank Mono', Monaco, monospace;
background: var(--bg-dark);
color: var(--text-primary);
line-height: 1.6;
}
/* Global link styles */
a {
color: var(--primary-cyan);
text-decoration: none;
transition: color 0.2s;
}
a:hover {
color: var(--accent-pink);
}
.marketplace-container {
min-height: 100vh;
}
/* Header */
.marketplace-header {
background: var(--bg-secondary);
border-bottom: 1px solid var(--border-color);
padding: 1.5rem 0;
}
.header-content {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem;
display: flex;
justify-content: space-between;
align-items: center;
}
.logo-title {
display: flex;
align-items: center;
gap: 1rem;
}
.header-logo {
height: 40px;
width: auto;
filter: brightness(1.2);
}
.marketplace-header h1 {
font-size: 1.5rem;
color: var(--primary-cyan);
margin: 0;
}
.ascii-border {
color: var(--border-color);
}
.tagline {
font-size: 0.875rem;
color: var(--text-tertiary);
margin-top: 0.25rem;
}
.header-stats {
display: flex;
gap: 2rem;
}
.stat-item {
font-size: 0.875rem;
color: var(--text-secondary);
}
.stat-item span {
color: var(--primary-cyan);
font-weight: 600;
}
/* Search and Filter Bar */
.search-filter-bar {
max-width: 1800px;
margin: 1.5rem auto;
padding: 0 2rem;
display: flex;
gap: 1rem;
align-items: center;
}
.search-box {
flex: 1;
max-width: 500px;
display: flex;
align-items: center;
background: var(--bg-secondary);
border: 1px solid var(--border-color);
padding: 0.75rem 1rem;
transition: border-color 0.2s;
}
.search-box:focus-within {
border-color: var(--primary-cyan);
}
.search-icon {
color: var(--text-tertiary);
margin-right: 1rem;
}
#search-input {
flex: 1;
background: transparent;
border: none;
color: var(--text-primary);
font-family: inherit;
font-size: 0.9rem;
outline: none;
}
.search-box kbd {
font-size: 0.75rem;
padding: 0.2rem 0.5rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
color: var(--text-tertiary);
}
.category-filter {
display: flex;
gap: 0.5rem;
flex-wrap: wrap;
}
.filter-btn {
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
padding: 0.5rem 1rem;
font-family: inherit;
font-size: 0.875rem;
cursor: pointer;
transition: all 0.2s;
}
.filter-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
.filter-btn.active {
background: var(--primary-cyan);
color: var(--bg-dark);
border-color: var(--primary-cyan);
}
/* Magazine Layout */
.magazine-layout {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem 4rem;
display: grid;
grid-template-columns: 1fr;
gap: 2rem;
}
/* Hero Featured Section */
.hero-featured {
grid-column: 1 / -1;
position: relative;
}
.hero-featured::before {
content: '';
position: absolute;
top: -20px;
left: -20px;
right: -20px;
bottom: -20px;
background: radial-gradient(ellipse at center, rgba(80, 255, 255, 0.05), transparent 70%);
pointer-events: none;
z-index: -1;
}
.featured-hero-card {
background: linear-gradient(135deg, #1a1a2e, #0f0f1e);
border: 2px solid var(--primary-cyan);
box-shadow: 0 0 30px rgba(80, 255, 255, 0.15),
inset 0 0 20px rgba(80, 255, 255, 0.05);
height: 380px;
position: relative;
overflow: hidden;
cursor: pointer;
transition: all 0.3s ease;
display: flex;
flex-direction: column;
}
.featured-hero-card:hover {
border-color: var(--accent-pink);
box-shadow: 0 0 40px rgba(243, 128, 245, 0.2),
inset 0 0 30px rgba(243, 128, 245, 0.05);
transform: translateY(-2px);
}
.hero-image {
width: 100%;
height: 200px;
min-height: 200px;
max-height: 200px;
background: linear-gradient(135deg, rgba(80, 255, 255, 0.1), rgba(243, 128, 245, 0.05));
background-size: cover;
background-position: center;
display: flex;
align-items: center;
justify-content: center;
font-size: 3rem;
color: var(--primary-cyan);
flex-shrink: 0;
position: relative;
filter: brightness(1.1) contrast(1.1);
overflow: hidden;
}
.hero-image img {
width: 100%;
height: 100%;
object-fit: cover;
object-position: center;
}
.hero-image::after {
content: '';
position: absolute;
bottom: 0;
left: 0;
right: 0;
height: 60%;
background: linear-gradient(to top, rgba(10, 10, 20, 0.95), transparent);
}
.hero-content {
padding: 1.5rem;
flex: 1;
display: flex;
flex-direction: column;
justify-content: space-between;
}
.hero-badge {
display: inline-block;
padding: 0.3rem 0.6rem;
background: linear-gradient(135deg, var(--primary-cyan), var(--primary-teal));
color: var(--bg-dark);
font-size: 0.7rem;
text-transform: uppercase;
margin-bottom: 0.5rem;
font-weight: 600;
box-shadow: 0 2px 10px rgba(80, 255, 255, 0.3);
}
.hero-title {
font-size: 1.6rem;
color: var(--primary-cyan);
margin: 0.5rem 0;
text-shadow: 0 0 20px rgba(80, 255, 255, 0.5);
}
.hero-description {
color: var(--text-secondary);
line-height: 1.5;
}
.hero-meta {
display: flex;
gap: 1.5rem;
margin-top: 1rem;
font-size: 0.875rem;
}
.hero-meta span {
color: var(--text-tertiary);
}
.hero-meta span:first-child {
color: var(--warning);
}
/* Secondary Featured */
.secondary-featured {
grid-column: 1 / -1;
min-height: 380px;
display: flex;
align-items: flex-start;
}
.featured-secondary-cards {
width: 100%;
display: flex;
flex-direction: column;
gap: 0.75rem;
align-items: stretch;
}
.secondary-card {
background: linear-gradient(135deg, rgba(80, 255, 255, 0.03), rgba(243, 128, 245, 0.02));
border: 1px solid rgba(80, 255, 255, 0.3);
cursor: pointer;
transition: all 0.3s ease;
display: flex;
overflow: hidden;
height: 118px;
min-height: 118px;
max-height: 118px;
flex-shrink: 0;
box-shadow: 0 2px 10px rgba(0, 0, 0, 0.3);
}
.secondary-card:hover {
border-color: var(--accent-pink);
background: linear-gradient(135deg, rgba(243, 128, 245, 0.05), rgba(80, 255, 255, 0.03));
box-shadow: 0 4px 15px rgba(243, 128, 245, 0.2);
transform: translateX(-3px);
}
.secondary-image {
width: 120px;
background: linear-gradient(135deg, var(--bg-tertiary), var(--bg-secondary));
background-size: cover;
background-position: center;
display: flex;
align-items: center;
justify-content: center;
font-size: 1.5rem;
color: var(--primary-cyan);
flex-shrink: 0;
}
.secondary-content {
flex: 1;
padding: 1rem;
display: flex;
flex-direction: column;
justify-content: space-between;
}
.secondary-title {
font-size: 1rem;
color: var(--text-primary);
margin-bottom: 0.25rem;
}
.secondary-desc {
font-size: 0.75rem;
color: var(--text-secondary);
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
.secondary-meta {
font-size: 0.75rem;
color: var(--text-tertiary);
}
.secondary-meta span:last-child {
color: var(--warning);
}
/* Sponsored Section */
.sponsored-section {
grid-column: 1 / -1;
background: var(--bg-secondary);
border: 1px solid var(--warning);
padding: 1rem;
position: relative;
}
.section-label {
position: absolute;
top: -0.5rem;
left: 1rem;
background: var(--bg-secondary);
padding: 0 0.5rem;
color: var(--warning);
font-size: 0.65rem;
letter-spacing: 0.1em;
}
.sponsored-cards {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
}
.sponsor-card {
padding: 1rem;
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
}
.sponsor-logo {
display: flex;
align-items: center;
justify-content: center;
height: 60px;
margin-bottom: 0.75rem;
}
.sponsor-logo img {
max-height: 60px;
max-width: 100%;
width: auto;
object-fit: contain;
}
.sponsor-card h4 {
color: var(--accent-pink);
margin-bottom: 0.5rem;
}
.sponsor-card p {
color: var(--text-secondary);
font-size: 0.85rem;
margin-bottom: 0.75rem;
}
.sponsor-card a {
color: var(--primary-cyan);
text-decoration: none;
font-size: 0.85rem;
}
.sponsor-card a:hover {
color: var(--accent-pink);
}
/* Main Content Grid */
.main-content {
grid-column: 1 / -1;
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 2rem;
}
/* Column Headers */
.column-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
border-bottom: 1px solid var(--border-color);
padding-bottom: 0.5rem;
}
.column-header h2 {
font-size: 1.1rem;
color: var(--text-primary);
}
.mini-filter {
background: var(--bg-tertiary);
border: 1px solid var(--border-color);
color: var(--text-primary);
padding: 0.25rem 0.5rem;
font-family: inherit;
font-size: 0.75rem;
}
.ascii-icon {
color: var(--primary-cyan);
}
/* Apps Column */
.apps-compact-grid {
display: flex;
flex-direction: column;
gap: 0.75rem;
}
.app-compact {
background: var(--bg-secondary);
border: 1px solid var(--border-color);
border-left: 3px solid var(--border-color);
padding: 0.75rem;
cursor: pointer;
transition: all 0.2s;
}
.app-compact:hover {
border-color: var(--primary-cyan);
border-left-color: var(--accent-pink);
transform: translateX(2px);
}
.app-compact-header {
display: flex;
justify-content: space-between;
font-size: 0.75rem;
color: var(--text-tertiary);
margin-bottom: 0.25rem;
}
.app-compact-header span:first-child {
color: var(--primary-cyan);
}
.app-compact-header span:last-child {
color: var(--warning);
}
.app-compact-title {
font-size: 0.9rem;
color: var(--text-primary);
margin-bottom: 0.25rem;
}
.app-compact-desc {
font-size: 0.75rem;
color: var(--text-secondary);
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
/* Articles Column */
.articles-compact-list {
display: flex;
flex-direction: column;
gap: 1rem;
}
.article-compact {
border-left: 2px solid var(--border-color);
padding-left: 1rem;
cursor: pointer;
transition: all 0.2s;
}
.article-compact:hover {
border-left-color: var(--primary-cyan);
}
.article-meta {
font-size: 0.7rem;
color: var(--text-tertiary);
margin-bottom: 0.25rem;
}
.article-meta span:first-child {
color: var(--accent-pink);
}
.article-title {
font-size: 0.9rem;
color: var(--text-primary);
margin-bottom: 0.25rem;
}
.article-author {
font-size: 0.75rem;
color: var(--text-secondary);
}
/* Trending Column */
.trending-items {
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.trending-item {
display: flex;
align-items: center;
gap: 0.75rem;
padding: 0.5rem;
background: var(--bg-secondary);
border: 1px solid var(--border-color);
cursor: pointer;
transition: all 0.2s;
}
.trending-item:hover {
border-color: var(--primary-cyan);
}
.trending-rank {
font-size: 1.2rem;
color: var(--primary-cyan);
width: 2rem;
text-align: center;
}
.trending-info {
flex: 1;
}
.trending-name {
font-size: 0.85rem;
color: var(--text-primary);
}
.trending-stats {
font-size: 0.7rem;
color: var(--text-tertiary);
}
/* Submit Box */
.submit-box {
margin-top: 1.5rem;
background: var(--bg-secondary);
border: 1px solid var(--primary-cyan);
padding: 1rem;
text-align: center;
}
.submit-box h3 {
font-size: 1rem;
color: var(--primary-cyan);
margin-bottom: 0.5rem;
}
.submit-box p {
font-size: 0.8rem;
color: var(--text-secondary);
margin-bottom: 0.75rem;
}
.submit-btn {
display: inline-block;
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--primary-cyan);
color: var(--primary-cyan);
text-decoration: none;
transition: all 0.2s;
}
.submit-btn:hover {
background: var(--primary-cyan);
color: var(--bg-dark);
}
/* More Apps Section */
.more-apps {
grid-column: 1 / -1;
margin-top: 2rem;
}
.section-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 1rem;
}
.more-apps-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
gap: 1rem;
}
.load-more-btn {
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-secondary);
padding: 0.5rem 1.5rem;
font-family: inherit;
cursor: pointer;
transition: all 0.2s;
}
.load-more-btn:hover {
border-color: var(--primary-cyan);
color: var(--primary-cyan);
}
/* Footer */
.marketplace-footer {
background: var(--bg-secondary);
border-top: 1px solid var(--border-color);
margin-top: 4rem;
padding: 2rem 0;
}
.footer-content {
max-width: 1800px;
margin: 0 auto;
padding: 0 2rem;
display: grid;
grid-template-columns: 1fr 1fr;
gap: 2rem;
}
.footer-section h3 {
font-size: 1rem;
margin-bottom: 0.5rem;
color: var(--primary-cyan);
}
.footer-section p {
font-size: 0.875rem;
color: var(--text-secondary);
margin-bottom: 1rem;
}
.sponsor-btn {
display: inline-block;
padding: 0.5rem 1rem;
background: transparent;
border: 1px solid var(--primary-cyan);
color: var(--primary-cyan);
text-decoration: none;
transition: all 0.2s;
}
.sponsor-btn:hover {
background: var(--primary-cyan);
color: var(--bg-dark);
}
.footer-bottom {
max-width: 1800px;
margin: 2rem auto 0;
padding: 1rem 2rem 0;
border-top: 1px solid var(--border-color);
font-size: 0.75rem;
color: var(--text-tertiary);
}
/* Modal */
.modal {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background: rgba(0, 0, 0, 0.8);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
}
.modal.hidden {
display: none;
}
.modal-content {
background: var(--bg-secondary);
border: 1px solid var(--primary-cyan);
max-width: 800px;
width: 90%;
max-height: 80vh;
overflow-y: auto;
position: relative;
}
.modal-close {
position: absolute;
top: 1rem;
right: 1rem;
background: transparent;
border: 1px solid var(--border-color);
color: var(--text-primary);
padding: 0.25rem 0.5rem;
cursor: pointer;
font-size: 1.2rem;
}
.modal-close:hover {
border-color: var(--error);
color: var(--error);
}
.app-detail {
padding: 2rem;
}
.app-detail h2 {
font-size: 1.5rem;
margin-bottom: 1rem;
color: var(--primary-cyan);
}
/* Loading */
.loading {
text-align: center;
padding: 2rem;
color: var(--text-tertiary);
}
.no-results {
text-align: center;
padding: 2rem;
color: var(--text-tertiary);
}
/* Responsive - Tablet */
@media (min-width: 768px) {
.magazine-layout {
grid-template-columns: repeat(2, 1fr);
}
.hero-featured {
grid-column: 1 / -1;
}
.secondary-featured {
grid-column: 1 / -1;
}
.sponsored-section {
grid-column: 1 / -1;
}
.main-content {
grid-column: 1 / -1;
grid-template-columns: repeat(2, 1fr);
}
}
/* Responsive - Desktop */
@media (min-width: 1024px) {
.magazine-layout {
grid-template-columns: repeat(3, 1fr);
}
.hero-featured {
grid-column: 1 / 3;
grid-row: 1;
}
.secondary-featured {
grid-column: 3 / 4;
grid-row: 1;
}
.featured-secondary-cards {
flex-direction: column;
}
.sponsored-section {
grid-column: 1 / -1;
}
.main-content {
grid-column: 1 / -1;
grid-template-columns: repeat(3, 1fr);
}
}
/* Responsive - Wide Desktop */
@media (min-width: 1400px) {
.magazine-layout {
grid-template-columns: repeat(4, 1fr);
}
.hero-featured {
grid-column: 1 / 3;
}
.secondary-featured {
grid-column: 3 / 5;
grid-row: 1;
min-height: auto;
}
.featured-secondary-cards {
display: grid;
grid-template-columns: repeat(2, 1fr);
flex-direction: unset;
}
.main-content {
grid-template-columns: repeat(4, 1fr);
}
.apps-column {
grid-column: span 2;
}
.more-apps-grid {
grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
}
}
/* Responsive - Ultra Wide Desktop (for coders with wide monitors) */
@media (min-width: 1800px) {
.magazine-layout {
grid-template-columns: repeat(5, 1fr);
}
.hero-featured {
grid-column: 1 / 3;
}
.secondary-featured {
grid-column: 3 / 6;
min-height: auto;
}
.featured-secondary-cards {
display: grid;
grid-template-columns: repeat(3, 1fr);
flex-direction: unset;
}
.sponsored-section {
grid-column: 1 / -1;
}
.sponsored-cards {
grid-template-columns: repeat(5, 1fr);
}
.main-content {
grid-template-columns: repeat(5, 1fr);
}
.apps-column {
grid-column: span 2;
}
.articles-column {
grid-column: span 2;
}
.more-apps-grid {
grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
}
}
/* Responsive - Mobile */
@media (max-width: 767px) {
.header-content {
flex-direction: column;
gap: 1rem;
}
.search-filter-bar {
flex-direction: column;
align-items: stretch;
}
.search-box {
max-width: none;
}
.magazine-layout {
padding: 0 1rem 2rem;
}
.footer-content {
grid-template-columns: 1fr;
}
.secondary-card {
flex-direction: column;
}
.secondary-image {
width: 100%;
height: 150px;
}
}

412
docs/md_v2/marketplace/marketplace.js Normal file
View File

@@ -0,0 +1,412 @@
// Marketplace JS - Magazine Layout
const { API_BASE, API_ORIGIN } = (() => {
const { hostname, port } = window.location;
if ((hostname === 'localhost' || hostname === '127.0.0.1') && port === '8000') {
const origin = 'http://127.0.0.1:8100';
return { API_BASE: `${origin}/marketplace/api`, API_ORIGIN: origin };
}
return { API_BASE: '/marketplace/api', API_ORIGIN: '' };
})();
const resolveAssetUrl = (path) => {
if (!path) return '';
if (/^https?:\/\//i.test(path)) return path;
if (path.startsWith('/') && API_ORIGIN) {
return `${API_ORIGIN}${path}`;
}
return path;
};
const CACHE_TTL = 3600000; // 1 hour in ms
class MarketplaceCache {
constructor() {
this.prefix = 'c4ai_market_';
}
get(key) {
const item = localStorage.getItem(this.prefix + key);
if (!item) return null;
const data = JSON.parse(item);
if (Date.now() > data.expires) {
localStorage.removeItem(this.prefix + key);
return null;
}
return data.value;
}
set(key, value, ttl = CACHE_TTL) {
const data = {
value: value,
expires: Date.now() + ttl
};
localStorage.setItem(this.prefix + key, JSON.stringify(data));
}
clear() {
Object.keys(localStorage)
.filter(k => k.startsWith(this.prefix))
.forEach(k => localStorage.removeItem(k));
}
}
class MarketplaceAPI {
constructor() {
this.cache = new MarketplaceCache();
this.searchTimeout = null;
}
async fetch(endpoint, useCache = true) {
const cacheKey = endpoint.replace(/[^\w]/g, '_');
if (useCache) {
const cached = this.cache.get(cacheKey);
if (cached) return cached;
}
try {
const response = await fetch(`${API_BASE}${endpoint}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const data = await response.json();
this.cache.set(cacheKey, data);
return data;
} catch (error) {
console.error('API Error:', error);
return null;
}
}
async getStats() {
return this.fetch('/stats');
}
async getCategories() {
return this.fetch('/categories');
}
async getApps(params = {}) {
const query = new URLSearchParams(params).toString();
return this.fetch(`/apps${query ? '?' + query : ''}`);
}
async getArticles(params = {}) {
const query = new URLSearchParams(params).toString();
return this.fetch(`/articles${query ? '?' + query : ''}`);
}
async getSponsors() {
return this.fetch('/sponsors');
}
async search(query) {
if (query.length < 2) return {};
return this.fetch(`/search?q=${encodeURIComponent(query)}`, false);
}
}
class MarketplaceUI {
constructor() {
this.api = new MarketplaceAPI();
this.currentCategory = 'all';
this.currentType = '';
this.searchTimeout = null;
this.loadedApps = 10;
this.init();
}
async init() {
await this.loadStats();
await this.loadCategories();
await this.loadFeaturedContent();
await this.loadSponsors();
await this.loadMainContent();
this.setupEventListeners();
}
async loadStats() {
const stats = await this.api.getStats();
if (stats) {
document.getElementById('total-apps').textContent = stats.total_apps || '0';
document.getElementById('total-articles').textContent = stats.total_articles || '0';
document.getElementById('total-downloads').textContent = stats.total_downloads || '0';
document.getElementById('last-update').textContent = new Date().toLocaleDateString();
}
}
async loadCategories() {
const categories = await this.api.getCategories();
if (!categories) return;
const filter = document.getElementById('category-filter');
categories.forEach(cat => {
const btn = document.createElement('button');
btn.className = 'filter-btn';
btn.dataset.category = cat.slug;
btn.textContent = cat.name;
btn.onclick = () => this.filterByCategory(cat.slug);
filter.appendChild(btn);
});
}
async loadFeaturedContent() {
// Load hero featured
const featured = await this.api.getApps({ featured: true, limit: 4 });
if (!featured || !featured.length) return;
// Hero card (first featured)
const hero = featured[0];
const heroCard = document.getElementById('featured-hero');
if (hero) {
const imageUrl = hero.image || '';
heroCard.innerHTML = `
<div class="hero-image" ${imageUrl ? `style="background-image: url('${imageUrl}')"` : ''}>
${!imageUrl ? `[${hero.category || 'APP'}]` : ''}
</div>
<div class="hero-content">
<span class="hero-badge">${hero.type || 'PAID'}</span>
<h2 class="hero-title">${hero.name}</h2>
<p class="hero-description">${hero.description}</p>
<div class="hero-meta">
<span>★ ${hero.rating || 0}/5</span>
<span>${hero.downloads || 0} downloads</span>
</div>
</div>
`;
heroCard.onclick = () => this.showAppDetail(hero);
}
// Secondary featured cards
const secondary = document.getElementById('featured-secondary');
secondary.innerHTML = '';
if (featured.length > 1) {
featured.slice(1, 4).forEach(app => {
const card = document.createElement('div');
card.className = 'secondary-card';
const imageUrl = app.image || '';
card.innerHTML = `
<div class="secondary-image" ${imageUrl ? `style="background-image: url('${imageUrl}')"` : ''}>
${!imageUrl ? `[${app.category || 'APP'}]` : ''}
</div>
<div class="secondary-content">
<h3 class="secondary-title">${app.name}</h3>
<p class="secondary-desc">${(app.description || '').substring(0, 100)}...</p>
<div class="secondary-meta">
<span>${app.type || 'Open Source'}</span> · <span>★ ${app.rating || 0}/5</span>
</div>
</div>
`;
card.onclick = () => this.showAppDetail(app);
secondary.appendChild(card);
});
}
}
async loadSponsors() {
const sponsors = await this.api.getSponsors();
if (!sponsors || !sponsors.length) {
// Show placeholder if no sponsors
const container = document.getElementById('sponsored-content');
container.innerHTML = `
<div class="sponsor-card">
<h4>Become a Sponsor</h4>
<p>Reach thousands of developers using Crawl4AI</p>
<a href="mailto:sponsors@crawl4ai.com">Contact Us →</a>
</div>
`;
return;
}
const container = document.getElementById('sponsored-content');
container.innerHTML = sponsors.slice(0, 5).map(sponsor => `
<div class="sponsor-card">
${sponsor.logo_url ? `<div class="sponsor-logo"><img src="${resolveAssetUrl(sponsor.logo_url)}" alt="${sponsor.company_name} logo"></div>` : ''}
<h4>${sponsor.company_name}</h4>
<p>${sponsor.tier} Sponsor - Premium Solutions</p>
<a href="${sponsor.landing_url}" target="_blank">Learn More →</a>
</div>
`).join('');
}
async loadMainContent() {
// Load apps column
const apps = await this.api.getApps({ limit: 8 });
if (apps && apps.length) {
const appsGrid = document.getElementById('apps-grid');
appsGrid.innerHTML = apps.map(app => `
<div class="app-compact" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="app-compact-header">
<span>${app.category}</span>
<span>★ ${app.rating}/5</span>
</div>
<div class="app-compact-title">${app.name}</div>
<div class="app-compact-desc">${app.description}</div>
</div>
`).join('');
}
// Load articles column
const articles = await this.api.getArticles({ limit: 6 });
if (articles && articles.length) {
const articlesList = document.getElementById('articles-list');
articlesList.innerHTML = articles.map(article => `
<div class="article-compact" onclick="marketplace.showArticle('${article.id}')">
<div class="article-meta">
<span>${article.category}</span> · <span>${new Date(article.published_at).toLocaleDateString()}</span>
</div>
<div class="article-title">${article.title}</div>
<div class="article-author">by ${article.author}</div>
</div>
`).join('');
}
// Load trending
if (apps && apps.length) {
const trending = apps.slice(0, 5);
const trendingList = document.getElementById('trending-list');
trendingList.innerHTML = trending.map((app, i) => `
<div class="trending-item" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="trending-rank">${i + 1}</div>
<div class="trending-info">
<div class="trending-name">${app.name}</div>
<div class="trending-stats">${app.downloads} downloads</div>
</div>
</div>
`).join('');
}
// Load more apps grid
const moreApps = await this.api.getApps({ offset: 8, limit: 12 });
if (moreApps && moreApps.length) {
const moreGrid = document.getElementById('more-apps-grid');
moreGrid.innerHTML = moreApps.map(app => `
<div class="app-compact" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="app-compact-header">
<span>${app.category}</span>
<span>${app.type}</span>
</div>
<div class="app-compact-title">${app.name}</div>
</div>
`).join('');
}
}
setupEventListeners() {
// Search
const searchInput = document.getElementById('search-input');
searchInput.addEventListener('input', (e) => {
clearTimeout(this.searchTimeout);
this.searchTimeout = setTimeout(() => this.search(e.target.value), 300);
});
// Keyboard shortcut
document.addEventListener('keydown', (e) => {
if (e.key === '/' && !searchInput.contains(document.activeElement)) {
e.preventDefault();
searchInput.focus();
}
if (e.key === 'Escape' && searchInput.contains(document.activeElement)) {
searchInput.blur();
searchInput.value = '';
}
});
// Type filter
const typeFilter = document.getElementById('type-filter');
typeFilter.addEventListener('change', (e) => {
this.currentType = e.target.value;
this.loadMainContent();
});
// Load more
const loadMore = document.getElementById('load-more');
loadMore.addEventListener('click', () => this.loadMoreApps());
}
async filterByCategory(category) {
// Update active state
document.querySelectorAll('.filter-btn').forEach(btn => {
btn.classList.toggle('active', btn.dataset.category === category);
});
this.currentCategory = category;
await this.loadMainContent();
}
async search(query) {
if (!query) {
await this.loadMainContent();
return;
}
const results = await this.api.search(query);
if (!results) return;
// Update apps grid with search results
if (results.apps && results.apps.length) {
const appsGrid = document.getElementById('apps-grid');
appsGrid.innerHTML = results.apps.map(app => `
<div class="app-compact" onclick="marketplace.showAppDetail(${JSON.stringify(app).replace(/"/g, '&quot;')})">
<div class="app-compact-header">
<span>${app.category}</span>
<span>★ ${app.rating}/5</span>
</div>
<div class="app-compact-title">${app.name}</div>
<div class="app-compact-desc">${app.description}</div>
</div>
`).join('');
}
// Update articles with search results
if (results.articles && results.articles.length) {
const articlesList = document.getElementById('articles-list');
articlesList.innerHTML = results.articles.map(article => `
<div class="article-compact" onclick="marketplace.showArticle('${article.id}')">
<div class="article-meta">
<span>${article.category}</span> · <span>${new Date(article.published_at).toLocaleDateString()}</span>
</div>
<div class="article-title">${article.title}</div>
<div class="article-author">by ${article.author}</div>
</div>
`).join('');
}
}
async loadMoreApps() {
this.loadedApps += 12;
const moreApps = await this.api.getApps({ offset: this.loadedApps, limit: 12 });
if (moreApps && moreApps.length) {
const moreGrid = document.getElementById('more-apps-grid');
moreApps.forEach(app => {
const card = document.createElement('div');
card.className = 'app-compact';
card.innerHTML = `
<div class="app-compact-header">
<span>${app.category}</span>
<span>${app.type}</span>
</div>
<div class="app-compact-title">${app.name}</div>
`;
card.onclick = () => this.showAppDetail(app);
moreGrid.appendChild(card);
});
}
}
showAppDetail(app) {
// Navigate to detail page instead of showing modal
const slug = app.slug || app.name.toLowerCase().replace(/\s+/g, '-');
window.location.href = `app-detail.html?app=${slug}`;
}
showArticle(articleId) {
// Could create article detail page similarly
console.log('Show article:', articleId);
}
}
// Initialize marketplace
let marketplace;
document.addEventListener('DOMContentLoaded', () => {
marketplace = new MarketplaceUI();
});

338
docs/releases_review/demo_v0.7.5.py Normal file
View File

@@ -0,0 +1,338 @@
"""
🚀 Crawl4AI v0.7.5 Release Demo - Working Examples
==================================================
This demo showcases key features introduced in v0.7.5 with real, executable examples.
Featured Demos:
1. ✅ Docker Hooks System - Real API calls with custom hooks (string & function-based)
2. ✅ Enhanced LLM Integration - Working LLM configurations
3. ✅ HTTPS Preservation - Live crawling with HTTPS maintenance
Requirements:
- crawl4ai v0.7.5 installed
- Docker running with crawl4ai image (optional for Docker demos)
- Valid API keys for LLM demos (optional)
"""
import asyncio
import requests
import time
import sys
from crawl4ai import (AsyncWebCrawler, CrawlerRunConfig, BrowserConfig,
CacheMode, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy,
hooks_to_string)
from crawl4ai.docker_client import Crawl4aiDockerClient
def print_section(title: str, description: str = ""):
"""Print a section header"""
print(f"\n{'=' * 60}")
print(f"{title}")
if description:
print(f"{description}")
print(f"{'=' * 60}\n")
async def demo_1_docker_hooks_system():
"""Demo 1: Docker Hooks System - Real API calls with custom hooks"""
print_section(
"Demo 1: Docker Hooks System",
"Testing both string-based and function-based hooks (NEW in v0.7.5!)"
)
# Check Docker service availability
def check_docker_service():
try:
response = requests.get("http://localhost:11235/", timeout=3)
return response.status_code == 200
except:
return False
print("Checking Docker service...")
docker_running = check_docker_service()
if not docker_running:
print("⚠️ Docker service not running on localhost:11235")
print("To test Docker hooks:")
print("1. Run: docker run -p 11235:11235 unclecode/crawl4ai:latest")
print("2. Wait for service to start")
print("3. Re-run this demo\n")
return
print("✓ Docker service detected!")
# ============================================================================
# PART 1: Traditional String-Based Hooks (Works with REST API)
# ============================================================================
print("\n" + "─" * 60)
print("Part 1: String-Based Hooks (REST API)")
print("─" * 60)
hooks_config_string = {
"on_page_context_created": """
async def hook(page, context, **kwargs):
print("[String Hook] Setting up page context")
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
print("[String Hook] Before retrieving HTML")
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
return page
"""
}
payload = {
"urls": ["https://httpbin.org/html"],
"hooks": {
"code": hooks_config_string,
"timeout": 30
}
}
print("🔧 Using string-based hooks for REST API...")
try:
start_time = time.time()
response = requests.post("http://localhost:11235/crawl", json=payload, timeout=60)
execution_time = time.time() - start_time
if response.status_code == 200:
result = response.json()
print(f"✅ String-based hooks executed in {execution_time:.2f}s")
if result.get('results') and result['results'][0].get('success'):
html_length = len(result['results'][0].get('html', ''))
print(f" 📄 HTML length: {html_length} characters")
else:
print(f"❌ Request failed: {response.status_code}")
except Exception as e:
print(f"❌ Error: {str(e)}")
# ============================================================================
# PART 2: NEW Function-Based Hooks with Docker Client (v0.7.5)
# ============================================================================
print("\n" + "─" * 60)
print("Part 2: Function-Based Hooks with Docker Client (✨ NEW!)")
print("─" * 60)
# Define hooks as regular Python functions
async def on_page_context_created_func(page, context, **kwargs):
"""Block images to speed up crawling"""
print("[Function Hook] Setting up page context")
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
await page.set_viewport_size({"width": 1920, "height": 1080})
return page
async def before_goto_func(page, context, url, **kwargs):
"""Add custom headers before navigation"""
print(f"[Function Hook] About to navigate to {url}")
await page.set_extra_http_headers({
'X-Crawl4AI': 'v0.7.5-function-hooks',
'X-Test-Header': 'demo'
})
return page
async def before_retrieve_html_func(page, context, **kwargs):
"""Scroll to load lazy content"""
print("[Function Hook] Scrolling page for lazy-loaded content")
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(500)
await page.evaluate("window.scrollTo(0, 0)")
return page
# Use the hooks_to_string utility (can be used standalone)
print("\n📦 Converting functions to strings with hooks_to_string()...")
hooks_as_strings = hooks_to_string({
"on_page_context_created": on_page_context_created_func,
"before_goto": before_goto_func,
"before_retrieve_html": before_retrieve_html_func
})
print(f" ✓ Converted {len(hooks_as_strings)} hooks to string format")
# OR use Docker Client which does conversion automatically!
print("\n🐳 Using Docker Client with automatic conversion...")
try:
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
# Pass function objects directly - conversion happens automatically!
results = await client.crawl(
urls=["https://httpbin.org/html"],
hooks={
"on_page_context_created": on_page_context_created_func,
"before_goto": before_goto_func,
"before_retrieve_html": before_retrieve_html_func
},
hooks_timeout=30
)
if results and results.success:
print(f"✅ Function-based hooks executed successfully!")
print(f" 📄 HTML length: {len(results.html)} characters")
print(f" 🎯 URL: {results.url}")
else:
print("⚠️ Crawl completed but may have warnings")
except Exception as e:
print(f"❌ Docker client error: {str(e)}")
# Show the benefits
print("\n" + "=" * 60)
print("✨ Benefits of Function-Based Hooks:")
print("=" * 60)
print("✓ Full IDE support (autocomplete, syntax highlighting)")
print("✓ Type checking and linting")
print("✓ Easier to test and debug")
print("✓ Reusable across projects")
print("✓ Automatic conversion in Docker client")
print("=" * 60)
async def demo_2_enhanced_llm_integration():
"""Demo 2: Enhanced LLM Integration - Working LLM configurations"""
print_section(
"Demo 2: Enhanced LLM Integration",
"Testing custom LLM providers and configurations"
)
print("🤖 Testing Enhanced LLM Integration Features")
provider = "gemini/gemini-2.5-flash-lite"
payload = {
"url": "https://example.com",
"f": "llm",
"q": "Summarize this page in one sentence.",
"provider": provider, # Explicitly set provider
"temperature": 0.7
}
try:
response = requests.post(
"http://localhost:11235/md",
json=payload,
timeout=60
)
if response.status_code == 200:
result = response.json()
print(f"✓ Request successful with provider: {provider}")
print(f" - Response keys: {list(result.keys())}")
print(f" - Content length: {len(result.get('markdown', ''))} characters")
print(f" - Note: Actual LLM call may fail without valid API key")
else:
print(f"❌ Request failed: {response.status_code}")
print(f" - Response: {response.text[:500]}")
except Exception as e:
print(f"[red]Error: {e}[/]")
async def demo_3_https_preservation():
"""Demo 3: HTTPS Preservation - Live crawling with HTTPS maintenance"""
print_section(
"Demo 3: HTTPS Preservation",
"Testing HTTPS preservation for internal links"
)
print("🔒 Testing HTTPS Preservation Feature")
# Test with HTTPS preservation enabled
print("\nTest 1: HTTPS Preservation ENABLED")
url_filter = URLPatternFilter(
patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
)
config = CrawlerRunConfig(
exclude_external_links=True,
stream=True,
verbose=False,
preserve_https_for_internal_links=True,
deep_crawl_strategy=BFSDeepCrawlStrategy(
max_depth=2,
max_pages=5,
filter_chain=FilterChain([url_filter])
)
)
test_url = "https://quotes.toscrape.com"
print(f"🎯 Testing URL: {test_url}")
async with AsyncWebCrawler() as crawler:
async for result in await crawler.arun(url=test_url, config=config):
print("✓ HTTPS Preservation Test Completed")
internal_links = [i['href'] for i in result.links['internal']]
for link in internal_links:
print(f" → {link}")
async def main():
"""Run all demos"""
print("\n" + "=" * 60)
print("🚀 Crawl4AI v0.7.5 Working Demo")
print("=" * 60)
# Check system requirements
print("🔍 System Requirements Check:")
print(f" - Python version: {sys.version.split()[0]} {'✓' if sys.version_info >= (3, 10) else '❌ (3.10+ required)'}")
try:
import requests
print(f" - Requests library: ✓")
except ImportError:
print(f" - Requests library: ❌")
print()
demos = [
("Docker Hooks System", demo_1_docker_hooks_system),
("Enhanced LLM Integration", demo_2_enhanced_llm_integration),
("HTTPS Preservation", demo_3_https_preservation),
]
for i, (name, demo_func) in enumerate(demos, 1):
try:
print(f"\n📍 Starting Demo {i}/{len(demos)}: {name}")
await demo_func()
if i < len(demos):
print(f"\n✨ Demo {i} complete! Press Enter for next demo...")
input()
except KeyboardInterrupt:
print(f"\n⏹️ Demo interrupted by user")
break
except Exception as e:
print(f"❌ Demo {i} error: {str(e)}")
print("Continuing to next demo...")
continue
print("\n" + "=" * 60)
print("🎉 Demo Complete!")
print("=" * 60)
print("You've experienced the power of Crawl4AI v0.7.5!")
print("")
print("Key Features Demonstrated:")
print("🔧 Docker Hooks - String-based & function-based (NEW!)")
print(" • hooks_to_string() utility for function conversion")
print(" • Docker client with automatic conversion")
print(" • Full IDE support and type checking")
print("🤖 Enhanced LLM - Better AI integration")
print("🔒 HTTPS Preservation - Secure link handling")
print("")
print("Ready to build something amazing? 🚀")
print("")
print("📖 Docs: https://docs.crawl4ai.com/")
print("🐙 GitHub: https://github.com/unclecode/crawl4ai")
print("=" * 60)
if __name__ == "__main__":
print("🚀 Crawl4AI v0.7.5 Live Demo Starting...")
print("Press Ctrl+C anytime to exit\n")
try:
asyncio.run(main())
except KeyboardInterrupt:
print("\n👋 Demo stopped by user. Thanks for trying Crawl4AI v0.7.5!")
except Exception as e:
print(f"\n❌ Demo error: {str(e)}")
print("Make sure you have the required dependencies installed.")

359
docs/releases_review/demo_v0.7.6.py Normal file
View File

@@ -0,0 +1,359 @@
#!/usr/bin/env python3
"""
Crawl4AI v0.7.6 Release Demo
============================
This demo showcases the major feature in v0.7.6:
**Webhook Support for Docker Job Queue API**
Features Demonstrated:
1. Asynchronous job processing with webhook notifications
2. Webhook support for /crawl/job endpoint
3. Webhook support for /llm/job endpoint
4. Notification-only vs data-in-payload modes
5. Custom webhook headers for authentication
6. Structured extraction with JSON schemas
7. Exponential backoff retry for reliable delivery
Prerequisites:
- Crawl4AI Docker container running on localhost:11235
- Flask installed: pip install flask requests
- LLM API key configured (for LLM examples)
Usage:
python docs/releases_review/demo_v0.7.6.py
"""
import requests
import json
import time
from flask import Flask, request, jsonify
from threading import Thread
# Configuration
CRAWL4AI_BASE_URL = "http://localhost:11235"
WEBHOOK_BASE_URL = "http://localhost:8080"
# Flask app for webhook receiver
app = Flask(__name__)
received_webhooks = []
@app.route('/webhook', methods=['POST'])
def webhook_handler():
"""Universal webhook handler for both crawl and LLM extraction jobs."""
payload = request.json
task_id = payload['task_id']
task_type = payload['task_type']
status = payload['status']
print(f"\n{'='*70}")
print(f"📬 Webhook Received!")
print(f" Task ID: {task_id}")
print(f" Task Type: {task_type}")
print(f" Status: {status}")
print(f" Timestamp: {payload['timestamp']}")
if status == 'completed':
if 'data' in payload:
print(f" ✅ Data included in webhook")
if task_type == 'crawl':
results = payload['data'].get('results', [])
print(f" 📊 Crawled {len(results)} URL(s)")
elif task_type == 'llm_extraction':
extracted = payload['data'].get('extracted_content', {})
print(f" 🤖 Extracted: {json.dumps(extracted, indent=6)}")
else:
print(f" 📥 Notification only (fetch data separately)")
elif status == 'failed':
print(f" ❌ Error: {payload.get('error', 'Unknown')}")
print(f"{'='*70}\n")
received_webhooks.append(payload)
return jsonify({"status": "received"}), 200
def start_webhook_server():
"""Start Flask webhook server in background."""
app.run(host='0.0.0.0', port=8080, debug=False, use_reloader=False)
def demo_1_crawl_webhook_notification_only():
"""Demo 1: Crawl job with webhook notification (data fetched separately)."""
print("\n" + "="*70)
print("DEMO 1: Crawl Job - Webhook Notification Only")
print("="*70)
print("Submitting crawl job with webhook notification...")
payload = {
"urls": ["https://example.com"],
"browser_config": {"headless": True},
"crawler_config": {"cache_mode": "bypass"},
"webhook_config": {
"webhook_url": f"{WEBHOOK_BASE_URL}/webhook",
"webhook_data_in_payload": False,
"webhook_headers": {
"X-Demo": "v0.7.6",
"X-Type": "crawl"
}
}
}
response = requests.post(f"{CRAWL4AI_BASE_URL}/crawl/job", json=payload)
if response.ok:
task_id = response.json()['task_id']
print(f"✅ Job submitted: {task_id}")
print("⏳ Webhook will notify when complete...")
return task_id
else:
print(f"❌ Failed: {response.text}")
return None
def demo_2_crawl_webhook_with_data():
"""Demo 2: Crawl job with full data in webhook payload."""
print("\n" + "="*70)
print("DEMO 2: Crawl Job - Webhook with Full Data")
print("="*70)
print("Submitting crawl job with data included in webhook...")
payload = {
"urls": ["https://www.python.org"],
"browser_config": {"headless": True},
"crawler_config": {"cache_mode": "bypass"},
"webhook_config": {
"webhook_url": f"{WEBHOOK_BASE_URL}/webhook",
"webhook_data_in_payload": True,
"webhook_headers": {
"X-Demo": "v0.7.6",
"X-Type": "crawl-with-data"
}
}
}
response = requests.post(f"{CRAWL4AI_BASE_URL}/crawl/job", json=payload)
if response.ok:
task_id = response.json()['task_id']
print(f"✅ Job submitted: {task_id}")
print("⏳ Webhook will include full results...")
return task_id
else:
print(f"❌ Failed: {response.text}")
return None
def demo_3_llm_webhook_notification_only():
"""Demo 3: LLM extraction with webhook notification (NEW in v0.7.6!)."""
print("\n" + "="*70)
print("DEMO 3: LLM Extraction - Webhook Notification Only (NEW!)")
print("="*70)
print("Submitting LLM extraction job with webhook notification...")
payload = {
"url": "https://www.example.com",
"q": "Extract the main heading and description from this page",
"provider": "openai/gpt-4o-mini",
"cache": False,
"webhook_config": {
"webhook_url": f"{WEBHOOK_BASE_URL}/webhook",
"webhook_data_in_payload": False,
"webhook_headers": {
"X-Demo": "v0.7.6",
"X-Type": "llm"
}
}
}
response = requests.post(f"{CRAWL4AI_BASE_URL}/llm/job", json=payload)
if response.ok:
task_id = response.json()['task_id']
print(f"✅ Job submitted: {task_id}")
print("⏳ Webhook will notify when LLM extraction completes...")
return task_id
else:
print(f"❌ Failed: {response.text}")
return None
def demo_4_llm_webhook_with_schema():
"""Demo 4: LLM extraction with JSON schema and data in webhook (NEW in v0.7.6!)."""
print("\n" + "="*70)
print("DEMO 4: LLM Extraction - Schema + Full Data in Webhook (NEW!)")
print("="*70)
print("Submitting LLM extraction with JSON schema...")
schema = {
"type": "object",
"properties": {
"title": {"type": "string", "description": "Page title"},
"description": {"type": "string", "description": "Page description"},
"main_topics": {
"type": "array",
"items": {"type": "string"},
"description": "Main topics covered"
}
},
"required": ["title"]
}
payload = {
"url": "https://www.python.org",
"q": "Extract the title, description, and main topics from this website",
"schema": json.dumps(schema),
"provider": "openai/gpt-4o-mini",
"cache": False,
"webhook_config": {
"webhook_url": f"{WEBHOOK_BASE_URL}/webhook",
"webhook_data_in_payload": True,
"webhook_headers": {
"X-Demo": "v0.7.6",
"X-Type": "llm-with-schema"
}
}
}
response = requests.post(f"{CRAWL4AI_BASE_URL}/llm/job", json=payload)
if response.ok:
task_id = response.json()['task_id']
print(f"✅ Job submitted: {task_id}")
print("⏳ Webhook will include structured extraction results...")
return task_id
else:
print(f"❌ Failed: {response.text}")
return None
def demo_5_global_webhook_config():
"""Demo 5: Using global webhook configuration from config.yml."""
print("\n" + "="*70)
print("DEMO 5: Global Webhook Configuration")
print("="*70)
print("💡 You can configure a default webhook URL in config.yml:")
print("""
webhooks:
enabled: true
default_url: "https://myapp.com/webhooks/default"
data_in_payload: false
retry:
max_attempts: 5
initial_delay_ms: 1000
max_delay_ms: 32000
timeout_ms: 30000
""")
print("Then submit jobs WITHOUT webhook_config - they'll use the default!")
print("This is useful for consistent webhook handling across all jobs.")
def demo_6_webhook_retry_logic():
"""Demo 6: Webhook retry mechanism with exponential backoff."""
print("\n" + "="*70)
print("DEMO 6: Webhook Retry Logic")
print("="*70)
print("🔄 Webhook delivery uses exponential backoff retry:")
print(" • Max attempts: 5")
print(" • Delays: 1s → 2s → 4s → 8s → 16s")
print(" • Timeout: 30s per attempt")
print(" • Retries on: 5xx errors, network errors, timeouts")
print(" • No retry on: 4xx client errors")
print("\nThis ensures reliable webhook delivery even with temporary failures!")
def print_summary():
"""Print demo summary and results."""
print("\n" + "="*70)
print("📊 DEMO SUMMARY")
print("="*70)
print(f"Total webhooks received: {len(received_webhooks)}")
crawl_webhooks = [w for w in received_webhooks if w['task_type'] == 'crawl']
llm_webhooks = [w for w in received_webhooks if w['task_type'] == 'llm_extraction']
print(f"\nBreakdown:")
print(f" 🕷️ Crawl jobs: {len(crawl_webhooks)}")
print(f" 🤖 LLM extraction jobs: {len(llm_webhooks)}")
print(f"\nDetails:")
for i, webhook in enumerate(received_webhooks, 1):
icon = "🕷️" if webhook['task_type'] == 'crawl' else "🤖"
print(f" {i}. {icon} {webhook['task_id']}: {webhook['status']}")
print("\n" + "="*70)
print("✨ v0.7.6 KEY FEATURES DEMONSTRATED:")
print("="*70)
print("✅ Webhook support for /crawl/job")
print("✅ Webhook support for /llm/job (NEW!)")
print("✅ Notification-only mode (fetch data separately)")
print("✅ Data-in-payload mode (get full results in webhook)")
print("✅ Custom headers for authentication")
print("✅ JSON schema for structured LLM extraction")
print("✅ Exponential backoff retry for reliable delivery")
print("✅ Global webhook configuration support")
print("✅ Universal webhook handler for both job types")
print("\n💡 Benefits:")
print(" • No more polling - get instant notifications")
print(" • Better resource utilization")
print(" • Reliable delivery with automatic retries")
print(" • Consistent API across crawl and LLM jobs")
print(" • Production-ready webhook infrastructure")
def main():
"""Run all demos."""
print("\n" + "="*70)
print("🚀 Crawl4AI v0.7.6 Release Demo")
print("="*70)
print("Feature: Webhook Support for Docker Job Queue API")
print("="*70)
# Check if server is running
try:
health = requests.get(f"{CRAWL4AI_BASE_URL}/health", timeout=5)
print(f"✅ Crawl4AI server is running")
except:
print(f"❌ Cannot connect to Crawl4AI at {CRAWL4AI_BASE_URL}")
print("Please start Docker container:")
print(" docker run -d -p 11235:11235 --env-file .llm.env unclecode/crawl4ai:0.7.6")
return
# Start webhook server
print(f"\n🌐 Starting webhook server at {WEBHOOK_BASE_URL}...")
webhook_thread = Thread(target=start_webhook_server, daemon=True)
webhook_thread.start()
time.sleep(2)
# Run demos
demo_1_crawl_webhook_notification_only()
time.sleep(5)
demo_2_crawl_webhook_with_data()
time.sleep(5)
demo_3_llm_webhook_notification_only()
time.sleep(5)
demo_4_llm_webhook_with_schema()
time.sleep(5)
demo_5_global_webhook_config()
demo_6_webhook_retry_logic()
# Wait for webhooks
print("\n⏳ Waiting for all webhooks to arrive...")
time.sleep(30)
# Print summary
print_summary()
print("\n" + "="*70)
print("✅ Demo completed!")
print("="*70)
print("\n📚 Documentation:")
print(" • deploy/docker/WEBHOOK_EXAMPLES.md")
print(" • docs/examples/docker_webhook_example.py")
print("\n🔗 Upgrade:")
print(" docker pull unclecode/crawl4ai:0.7.6")
if __name__ == "__main__":
main()

655
docs/releases_review/v0.7.5_docker_hooks_demo.py Normal file
View File

@@ -0,0 +1,655 @@
#!/usr/bin/env python3
"""
🚀 Crawl4AI v0.7.5 - Docker Hooks System Complete Demonstration
================================================================
This file demonstrates the NEW Docker Hooks System introduced in v0.7.5.
The Docker Hooks System is a completely NEW feature that provides pipeline
customization through user-provided Python functions. It offers three approaches:
1. String-based hooks for REST API
2. hooks_to_string() utility to convert functions
3. Docker Client with automatic conversion (most convenient)
All three approaches are part of this NEW v0.7.5 feature!
Perfect for video recording and demonstration purposes.
Requirements:
- Docker container running: docker run -p 11235:11235 unclecode/crawl4ai:latest
- crawl4ai v0.7.5 installed: pip install crawl4ai==0.7.5
"""
import asyncio
import requests
import json
import time
from typing import Dict, Any
# Import Crawl4AI components
from crawl4ai import hooks_to_string
from crawl4ai.docker_client import Crawl4aiDockerClient
# Configuration
DOCKER_URL = "http://localhost:11235"
# DOCKER_URL = "http://localhost:11234"
TEST_URLS = [
# "https://httpbin.org/html",
"https://www.kidocode.com",
"https://quotes.toscrape.com",
]
def print_section(title: str, description: str = ""):
"""Print a formatted section header"""
print("\n" + "=" * 70)
print(f" {title}")
if description:
print(f" {description}")
print("=" * 70 + "\n")
def check_docker_service() -> bool:
"""Check if Docker service is running"""
try:
response = requests.get(f"{DOCKER_URL}/health", timeout=3)
return response.status_code == 200
except:
return False
# ============================================================================
# REUSABLE HOOK LIBRARY (NEW in v0.7.5)
# ============================================================================
async def performance_optimization_hook(page, context, **kwargs):
"""
Performance Hook: Block unnecessary resources to speed up crawling
"""
print(" [Hook] 🚀 Optimizing performance - blocking images and ads...")
# Block images
await context.route(
"**/*.{png,jpg,jpeg,gif,webp,svg,ico}",
lambda route: route.abort()
)
# Block ads and analytics
await context.route("**/analytics/*", lambda route: route.abort())
await context.route("**/ads/*", lambda route: route.abort())
await context.route("**/google-analytics.com/*", lambda route: route.abort())
print(" [Hook] ✓ Performance optimization applied")
return page
async def viewport_setup_hook(page, context, **kwargs):
"""
Viewport Hook: Set consistent viewport size for rendering
"""
print(" [Hook] 🖥️ Setting viewport to 1920x1080...")
await page.set_viewport_size({"width": 1920, "height": 1080})
print(" [Hook] ✓ Viewport configured")
return page
async def authentication_headers_hook(page, context, url, **kwargs):
"""
Headers Hook: Add custom authentication and tracking headers
"""
print(f" [Hook] 🔐 Adding custom headers for {url[:50]}...")
await page.set_extra_http_headers({
'X-Crawl4AI-Version': '0.7.5',
'X-Custom-Hook': 'function-based-demo',
'Accept-Language': 'en-US,en;q=0.9',
'User-Agent': 'Crawl4AI/0.7.5 (Educational Demo)'
})
print(" [Hook] ✓ Custom headers added")
return page
async def lazy_loading_handler_hook(page, context, **kwargs):
"""
Content Hook: Handle lazy-loaded content by scrolling
"""
print(" [Hook] 📜 Scrolling to load lazy content...")
# Scroll to bottom
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
# Scroll to middle
await page.evaluate("window.scrollTo(0, document.body.scrollHeight / 2)")
await page.wait_for_timeout(500)
# Scroll back to top
await page.evaluate("window.scrollTo(0, 0)")
await page.wait_for_timeout(500)
print(" [Hook] ✓ Lazy content loaded")
return page
async def page_analytics_hook(page, context, **kwargs):
"""
Analytics Hook: Log page metrics before extraction
"""
print(" [Hook] 📊 Collecting page analytics...")
metrics = await page.evaluate('''
() => ({
title: document.title,
images: document.images.length,
links: document.links.length,
scripts: document.scripts.length,
headings: document.querySelectorAll('h1, h2, h3').length,
paragraphs: document.querySelectorAll('p').length
})
''')
print(f" [Hook] 📈 Page: {metrics['title'][:50]}...")
print(f" Links: {metrics['links']}, Images: {metrics['images']}, "
f"Headings: {metrics['headings']}, Paragraphs: {metrics['paragraphs']}")
return page
# ============================================================================
# DEMO 1: String-Based Hooks (NEW Docker Hooks System)
# ============================================================================
def demo_1_string_based_hooks():
"""
Demonstrate string-based hooks with REST API (part of NEW Docker Hooks System)
"""
print_section(
"DEMO 1: String-Based Hooks (REST API)",
"Part of the NEW Docker Hooks System - hooks as strings"
)
# Define hooks as strings
hooks_config = {
"on_page_context_created": """
async def hook(page, context, **kwargs):
print(" [String Hook] Setting up page context...")
# Block images for performance
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
await page.set_viewport_size({"width": 1920, "height": 1080})
return page
""",
"before_goto": """
async def hook(page, context, url, **kwargs):
print(f" [String Hook] Navigating to {url[:50]}...")
await page.set_extra_http_headers({
'X-Crawl4AI': 'string-based-hooks',
'X-Demo': 'v0.7.5'
})
return page
""",
"before_retrieve_html": """
async def hook(page, context, **kwargs):
print(" [String Hook] Scrolling page...")
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
return page
"""
}
# Prepare request payload
payload = {
"urls": [TEST_URLS[0]],
"hooks": {
"code": hooks_config,
"timeout": 30
},
"crawler_config": {
"cache_mode": "bypass"
}
}
print(f"🎯 Target URL: {TEST_URLS[0]}")
print(f"🔧 Configured {len(hooks_config)} string-based hooks")
print(f"📡 Sending request to Docker API...\n")
try:
start_time = time.time()
response = requests.post(f"{DOCKER_URL}/crawl", json=payload, timeout=60)
execution_time = time.time() - start_time
if response.status_code == 200:
result = response.json()
print(f"\n✅ Request successful! (took {execution_time:.2f}s)")
# Display results
if result.get('results') and result['results'][0].get('success'):
crawl_result = result['results'][0]
html_length = len(crawl_result.get('html', ''))
markdown_length = len(crawl_result.get('markdown', ''))
print(f"\n📊 Results:")
print(f" • HTML length: {html_length:,} characters")
print(f" • Markdown length: {markdown_length:,} characters")
print(f" • URL: {crawl_result.get('url')}")
# Check hooks execution
if 'hooks' in result:
hooks_info = result['hooks']
print(f"\n🎣 Hooks Execution:")
print(f" • Status: {hooks_info['status']['status']}")
print(f" • Attached hooks: {len(hooks_info['status']['attached_hooks'])}")
if 'summary' in hooks_info:
summary = hooks_info['summary']
print(f" • Total executions: {summary['total_executions']}")
print(f" • Successful: {summary['successful']}")
print(f" • Success rate: {summary['success_rate']:.1f}%")
else:
print(f"⚠️ Crawl completed but no results")
else:
print(f"❌ Request failed with status {response.status_code}")
print(f" Error: {response.text[:200]}")
except requests.exceptions.Timeout:
print("⏰ Request timed out after 60 seconds")
except Exception as e:
print(f"❌ Error: {str(e)}")
print("\n" + "─" * 70)
print("✓ String-based hooks demo complete\n")
# ============================================================================
# DEMO 2: Function-Based Hooks with hooks_to_string() Utility
# ============================================================================
def demo_2_hooks_to_string_utility():
"""
Demonstrate the new hooks_to_string() utility for converting functions
"""
print_section(
"DEMO 2: hooks_to_string() Utility (NEW! ✨)",
"Convert Python functions to strings for REST API"
)
print("📦 Creating hook functions...")
print(" • performance_optimization_hook")
print(" • viewport_setup_hook")
print(" • authentication_headers_hook")
print(" • lazy_loading_handler_hook")
# Convert function objects to strings using the NEW utility
print("\n🔄 Converting functions to strings with hooks_to_string()...")
hooks_dict = {
"on_page_context_created": performance_optimization_hook,
"before_goto": authentication_headers_hook,
"before_retrieve_html": lazy_loading_handler_hook,
}
hooks_as_strings = hooks_to_string(hooks_dict)
print(f"✅ Successfully converted {len(hooks_as_strings)} functions to strings")
# Show a preview
print("\n📝 Sample converted hook (first 250 characters):")
print("─" * 70)
sample_hook = list(hooks_as_strings.values())[0]
print(sample_hook[:250] + "...")
print("─" * 70)
# Use the converted hooks with REST API
print("\n📡 Using converted hooks with REST API...")
payload = {
"urls": [TEST_URLS[0]],
"hooks": {
"code": hooks_as_strings,
"timeout": 30
}
}
try:
start_time = time.time()
response = requests.post(f"{DOCKER_URL}/crawl", json=payload, timeout=60)
execution_time = time.time() - start_time
if response.status_code == 200:
result = response.json()
print(f"\n✅ Request successful! (took {execution_time:.2f}s)")
if result.get('results') and result['results'][0].get('success'):
crawl_result = result['results'][0]
print(f" • HTML length: {len(crawl_result.get('html', '')):,} characters")
print(f" • Hooks executed successfully!")
else:
print(f"❌ Request failed: {response.status_code}")
except Exception as e:
print(f"❌ Error: {str(e)}")
print("\n💡 Benefits of hooks_to_string():")
print(" ✓ Write hooks as regular Python functions")
print(" ✓ Full IDE support (autocomplete, syntax highlighting)")
print(" ✓ Type checking and linting")
print(" ✓ Easy to test and debug")
print(" ✓ Reusable across projects")
print(" ✓ Works with any REST API client")
print("\n" + "─" * 70)
print("✓ hooks_to_string() utility demo complete\n")
# ============================================================================
# DEMO 3: Docker Client with Automatic Conversion (RECOMMENDED! 🌟)
# ============================================================================
async def demo_3_docker_client_auto_conversion():
"""
Demonstrate Docker Client with automatic hook conversion (RECOMMENDED)
"""
print_section(
"DEMO 3: Docker Client with Auto-Conversion (RECOMMENDED! 🌟)",
"Pass function objects directly - conversion happens automatically!"
)
print("🐳 Initializing Crawl4AI Docker Client...")
client = Crawl4aiDockerClient(base_url=DOCKER_URL)
print("✅ Client ready!\n")
# Use our reusable hook library - just pass the function objects!
print("📚 Using reusable hook library:")
print(" • performance_optimization_hook")
print(" • viewport_setup_hook")
print(" • authentication_headers_hook")
print(" • lazy_loading_handler_hook")
print(" • page_analytics_hook")
print("\n🎯 Target URL: " + TEST_URLS[1])
print("🚀 Starting crawl with automatic hook conversion...\n")
try:
start_time = time.time()
# Pass function objects directly - NO manual conversion needed! ✨
results = await client.crawl(
urls=[TEST_URLS[0]],
hooks={
"on_page_context_created": performance_optimization_hook,
"before_goto": authentication_headers_hook,
"before_retrieve_html": lazy_loading_handler_hook,
"before_return_html": page_analytics_hook,
},
hooks_timeout=30
)
execution_time = time.time() - start_time
print(f"\n✅ Crawl completed! (took {execution_time:.2f}s)\n")
# Display results
if results and results.success:
result = results
print(f"📊 Results:")
print(f" • URL: {result.url}")
print(f" • Success: {result.success}")
print(f" • HTML length: {len(result.html):,} characters")
print(f" • Markdown length: {len(result.markdown):,} characters")
# Show metadata
if result.metadata:
print(f"\n📋 Metadata:")
print(f" • Title: {result.metadata.get('title', 'N/A')}")
print(f" • Description: {result.metadata.get('description', 'N/A')}")
# Show links
if result.links:
internal_count = len(result.links.get('internal', []))
external_count = len(result.links.get('external', []))
print(f"\n🔗 Links Found:")
print(f" • Internal: {internal_count}")
print(f" • External: {external_count}")
else:
print(f"⚠️ Crawl completed but no successful results")
if results:
print(f" Error: {results.error_message}")
except Exception as e:
print(f"❌ Error: {str(e)}")
import traceback
traceback.print_exc()
print("\n🌟 Why Docker Client is RECOMMENDED:")
print(" ✓ Automatic function-to-string conversion")
print(" ✓ No manual hooks_to_string() calls needed")
print(" ✓ Cleaner, more Pythonic code")
print(" ✓ Full type hints and IDE support")
print(" ✓ Built-in error handling")
print(" ✓ Async/await support")
print("\n" + "─" * 70)
print("✓ Docker Client auto-conversion demo complete\n")
# ============================================================================
# DEMO 4: Advanced Use Case - Complete Hook Pipeline
# ============================================================================
async def demo_4_complete_hook_pipeline():
"""
Demonstrate a complete hook pipeline using all 8 hook points
"""
print_section(
"DEMO 4: Complete Hook Pipeline",
"Using all 8 available hook points for comprehensive control"
)
# Define all 8 hooks
async def on_browser_created_hook(browser, **kwargs):
"""Hook 1: Called after browser is created"""
print(" [Pipeline] 1/8 Browser created")
return browser
async def on_page_context_created_hook(page, context, **kwargs):
"""Hook 2: Called after page context is created"""
print(" [Pipeline] 2/8 Page context created - setting up...")
await page.set_viewport_size({"width": 1920, "height": 1080})
return page
async def on_user_agent_updated_hook(page, context, user_agent, **kwargs):
"""Hook 3: Called when user agent is updated"""
print(f" [Pipeline] 3/8 User agent updated: {user_agent[:50]}...")
return page
async def before_goto_hook(page, context, url, **kwargs):
"""Hook 4: Called before navigating to URL"""
print(f" [Pipeline] 4/8 Before navigation to: {url[:60]}...")
return page
async def after_goto_hook(page, context, url, response, **kwargs):
"""Hook 5: Called after navigation completes"""
print(f" [Pipeline] 5/8 After navigation - Status: {response.status if response else 'N/A'}")
await page.wait_for_timeout(1000)
return page
async def on_execution_started_hook(page, context, **kwargs):
"""Hook 6: Called when JavaScript execution starts"""
print(" [Pipeline] 6/8 JavaScript execution started")
return page
async def before_retrieve_html_hook(page, context, **kwargs):
"""Hook 7: Called before retrieving HTML"""
print(" [Pipeline] 7/8 Before HTML retrieval - scrolling...")
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
return page
async def before_return_html_hook(page, context, html, **kwargs):
"""Hook 8: Called before returning HTML"""
print(f" [Pipeline] 8/8 Before return - HTML length: {len(html):,} chars")
return page
print("🎯 Target URL: " + TEST_URLS[0])
print("🔧 Configured ALL 8 hook points for complete pipeline control\n")
client = Crawl4aiDockerClient(base_url=DOCKER_URL)
try:
print("🚀 Starting complete pipeline crawl...\n")
start_time = time.time()
results = await client.crawl(
urls=[TEST_URLS[0]],
hooks={
"on_browser_created": on_browser_created_hook,
"on_page_context_created": on_page_context_created_hook,
"on_user_agent_updated": on_user_agent_updated_hook,
"before_goto": before_goto_hook,
"after_goto": after_goto_hook,
"on_execution_started": on_execution_started_hook,
"before_retrieve_html": before_retrieve_html_hook,
"before_return_html": before_return_html_hook,
},
hooks_timeout=45
)
execution_time = time.time() - start_time
if results and results.success:
print(f"\n✅ Complete pipeline executed successfully! (took {execution_time:.2f}s)")
print(f" • All 8 hooks executed in sequence")
print(f" • HTML length: {len(results.html):,} characters")
else:
print(f"⚠️ Pipeline completed with warnings")
except Exception as e:
print(f"❌ Error: {str(e)}")
print("\n📚 Available Hook Points:")
print(" 1. on_browser_created - Browser initialization")
print(" 2. on_page_context_created - Page context setup")
print(" 3. on_user_agent_updated - User agent configuration")
print(" 4. before_goto - Pre-navigation setup")
print(" 5. after_goto - Post-navigation processing")
print(" 6. on_execution_started - JavaScript execution start")
print(" 7. before_retrieve_html - Pre-extraction processing")
print(" 8. before_return_html - Final HTML processing")
print("\n" + "─" * 70)
print("✓ Complete hook pipeline demo complete\n")
# ============================================================================
# MAIN EXECUTION
# ============================================================================
async def main():
"""
Run all demonstrations
"""
print("\n" + "=" * 70)
print(" 🚀 Crawl4AI v0.7.5 - Docker Hooks Complete Demonstration")
print("=" * 70)
# Check Docker service
print("\n🔍 Checking Docker service status...")
if not check_docker_service():
print("❌ Docker service is not running!")
print("\n📋 To start the Docker service:")
print(" docker run -p 11235:11235 unclecode/crawl4ai:latest")
print("\nPlease start the service and run this demo again.")
return
print("✅ Docker service is running!\n")
# Run all demos
demos = [
("String-Based Hooks (REST API)", demo_1_string_based_hooks, False),
("hooks_to_string() Utility", demo_2_hooks_to_string_utility, False),
("Docker Client Auto-Conversion", demo_3_docker_client_auto_conversion, True),
# ("Complete Hook Pipeline", demo_4_complete_hook_pipeline, True),
]
for i, (name, demo_func, is_async) in enumerate(demos, 1):
print(f"\n{'🔷' * 35}")
print(f"Starting Demo {i}/{len(demos)}: {name}")
print(f"{'🔷' * 35}\n")
try:
if is_async:
await demo_func()
else:
demo_func()
print(f"✅ Demo {i} completed successfully!")
# Pause between demos (except the last one)
if i < len(demos):
print("\n⏸️ Press Enter to continue to next demo...")
# input()
except KeyboardInterrupt:
print(f"\n⏹️ Demo interrupted by user")
break
except Exception as e:
print(f"\n❌ Demo {i} failed: {str(e)}")
import traceback
traceback.print_exc()
print("\nContinuing to next demo...\n")
continue
# Final summary
print("\n" + "=" * 70)
print(" 🎉 All Demonstrations Complete!")
print("=" * 70)
print("\n📊 Summary of v0.7.5 Docker Hooks System:")
print("\n🆕 COMPLETELY NEW FEATURE in v0.7.5:")
print(" The Docker Hooks System lets you customize the crawling pipeline")
print(" with user-provided Python functions at 8 strategic points.")
print("\n✨ Three Ways to Use Docker Hooks (All NEW!):")
print(" 1. String-based - Write hooks as strings for REST API")
print(" 2. hooks_to_string() - Convert Python functions to strings")
print(" 3. Docker Client - Automatic conversion (RECOMMENDED)")
print("\n💡 Key Benefits:")
print(" ✓ Full IDE support (autocomplete, syntax highlighting)")
print(" ✓ Type checking and linting")
print(" ✓ Easy to test and debug")
print(" ✓ Reusable across projects")
print(" ✓ Complete pipeline control")
print("\n🎯 8 Hook Points Available:")
print(" • on_browser_created, on_page_context_created")
print(" • on_user_agent_updated, before_goto, after_goto")
print(" • on_execution_started, before_retrieve_html, before_return_html")
print("\n📚 Resources:")
print(" • Docs: https://docs.crawl4ai.com")
print(" • GitHub: https://github.com/unclecode/crawl4ai")
print(" • Discord: https://discord.gg/jP8KfhDhyN")
print("\n" + "=" * 70)
print(" Happy Crawling with v0.7.5! 🕷️")
print("=" * 70 + "\n")
if __name__ == "__main__":
print("\n🎬 Starting Crawl4AI v0.7.5 Docker Hooks Demonstration...")
print("Press Ctrl+C anytime to exit\n")
try:
asyncio.run(main())
except KeyboardInterrupt:
print("\n\n👋 Demo stopped by user. Thanks for exploring Crawl4AI v0.7.5!")
except Exception as e:
print(f"\n\n❌ Demo error: {str(e)}")
import traceback
traceback.print_exc()

1516
docs/releases_review/v0.7.5_video_walkthrough.ipynb Normal file
View File

File diff suppressed because it is too large Load Diff