91a5fea11f
Implements a full-featured CLI for Crawl4AI with the following capabilities: - Basic and advanced web crawling - Configuration management via YAML/JSON files - Multiple extraction strategies (CSS, XPath, LLM) - Content filtering and optimization - Interactive Q&A capabilities - Various output formats - Comprehensive documentation and examples Also includes: - Home directory setup for configuration and cache - Environment variable support for API tokens - Test suite for CLI functionality
84 lines
2.8 KiB
YAML
84 lines
2.8 KiB
YAML
site_name: Crawl4AI Documentation (v0.4.3b2)
|
|
site_description: 🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper
|
|
site_url: https://docs.crawl4ai.com
|
|
repo_url: https://github.com/unclecode/crawl4ai
|
|
repo_name: unclecode/crawl4ai
|
|
docs_dir: docs/md_v2
|
|
|
|
nav:
|
|
- Home: 'index.md'
|
|
- Setup & Installation:
|
|
- "Installation": "core/installation.md"
|
|
- "Docker Deployment": "core/docker-deploymeny.md"
|
|
- "Quick Start": "core/quickstart.md"
|
|
- "Blog & Changelog":
|
|
- "Blog Home": "blog/index.md"
|
|
- "Changelog": "https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md"
|
|
- Core:
|
|
- "Command Line Interface": "core/cli.md"
|
|
- "Simple Crawling": "core/simple-crawling.md"
|
|
- "Crawler Result": "core/crawler-result.md"
|
|
- "Browser & Crawler Config": "core/browser-crawler-config.md"
|
|
- "Markdown Generation": "core/markdown-generation.md"
|
|
- "Fit Markdown": "core/fit-markdown.md"
|
|
- "Page Interaction": "core/page-interaction.md"
|
|
- "Content Selection": "core/content-selection.md"
|
|
- "Cache Modes": "core/cache-modes.md"
|
|
- "Local Files & Raw HTML": "core/local-files.md"
|
|
- "Link & Media": "core/link-media.md"
|
|
- Advanced:
|
|
- "Overview": "advanced/advanced-features.md"
|
|
- "File Downloading": "advanced/file-downloading.md"
|
|
- "Lazy Loading": "advanced/lazy-loading.md"
|
|
- "Hooks & Auth": "advanced/hooks-auth.md"
|
|
- "Proxy & Security": "advanced/proxy-security.md"
|
|
- "Session Management": "advanced/session-management.md"
|
|
- "Multi-URL Crawling": "advanced/multi-url-crawling.md"
|
|
- "Crawl Dispatcher": "advanced/crawl-dispatcher.md"
|
|
- "Identity Based Crawling": "advanced/identity-based-crawling.md"
|
|
- "SSL Certificate": "advanced/ssl-certificate.md"
|
|
- Extraction:
|
|
- "LLM-Free Strategies": "extraction/no-llm-strategies.md"
|
|
- "LLM Strategies": "extraction/llm-strategies.md"
|
|
- "Clustering Strategies": "extraction/clustring-strategies.md"
|
|
- "Chunking": "extraction/chunking.md"
|
|
- API Reference:
|
|
- "AsyncWebCrawler": "api/async-webcrawler.md"
|
|
- "arun()": "api/arun.md"
|
|
- "arun_many()": "api/arun_many.md"
|
|
- "Browser & Crawler Config": "api/parameters.md"
|
|
- "CrawlResult": "api/crawl-result.md"
|
|
- "Strategies": "api/strategies.md"
|
|
|
|
theme:
|
|
name: 'terminal'
|
|
palette: 'dark'
|
|
icon:
|
|
repo: fontawesome/brands/github
|
|
|
|
plugins:
|
|
- search
|
|
|
|
markdown_extensions:
|
|
- pymdownx.highlight:
|
|
anchor_linenums: true
|
|
- pymdownx.inlinehilite
|
|
- pymdownx.snippets
|
|
- pymdownx.superfences
|
|
- admonition
|
|
- pymdownx.details
|
|
- attr_list
|
|
- tables
|
|
|
|
extra:
|
|
version: !ENV [CRAWL4AI_VERSION, 'development']
|
|
|
|
extra_css:
|
|
- assets/styles.css
|
|
- assets/highlight.css
|
|
- assets/dmvendor.css
|
|
|
|
extra_javascript:
|
|
- assets/highlight.min.js
|
|
- assets/highlight_init.js
|
|
- https://buttons.github.io/buttons.js |