crawl4ai/docs/llm.txt/2_configuration.md

# Core Configurations

## BrowserConfig
`BrowserConfig` centralizes all parameters required to set up and manage a browser instance and its context. This configuration ensures consistent and documented browser behavior for the crawler. Below is a detailed explanation of each parameter and its optimal use cases.

### Parameters and Use Cases

#### `browser_type`
- **Description**: Specifies the type of browser to launch.
  - Supported values: `"chromium"`, `"firefox"`, `"webkit"`
  - Default: `"chromium"`
- **Use Case**:
  - Use `"chromium"` for general-purpose crawling with modern web standards.
  - Use `"firefox"` when testing against Firefox-specific behavior.
  - Use `"webkit"` for testing Safari-like environments.

#### `headless`
- **Description**: Determines whether the browser runs in headless mode (no GUI).
  - Default: `True`
- **Use Case**:
  - Enable for faster, automated operations without UI overhead.
  - Disable (`False`) when debugging or inspecting browser behavior visually.

#### `use_managed_browser`
- **Description**: Enables advanced manipulation via a managed browser approach.
  - Default: `False`
- **Use Case**:
  - Use when fine-grained control is needed over browser sessions, such as debugging network requests or reusing sessions.

#### `debugging_port`
- **Description**: Port for remote debugging.
  - Default: 9222
- **Use Case**:
  - Use for debugging browser sessions with DevTools or external tools.

#### `use_persistent_context`
- **Description**: Uses a persistent browser context (e.g., saved profiles).
  - Automatically enables `use_managed_browser`.
  - Default: `False`
- **Use Case**:
  - Persistent login sessions for authenticated crawling.
  - Retaining cookies or local storage across multiple runs.

#### `user_data_dir`
- **Description**: Path to a directory for storing persistent browser data.
  - Default: `None`
- **Use Case**:
  - Specify a directory to save browser profiles for multi-run crawls or debugging.

#### `chrome_channel`
- **Description**: Specifies the Chrome channel to launch (e.g., `"chrome"`, `"msedge"`).
  - Applies only when `browser_type` is `"chromium"`.
  - Default: `"chrome"`
- **Use Case**:
  - Use `"msedge"` for compatibility testing with Edge browsers.

#### `proxy` and `proxy_config`
- **Description**:
  - `proxy`: Proxy server URL for the browser.
  - `proxy_config`: Detailed proxy configuration.
  - Default: `None`
- **Use Case**:
  - Set `proxy` for single-proxy setups.
  - Use `proxy_config` for advanced configurations, such as authenticated proxies or regional routing.

#### `viewport_width` and `viewport_height`
- **Description**: Sets the default browser viewport dimensions.
  - Default: `1080` (width), `600` (height)
- **Use Case**:
  - Adjust for crawling responsive layouts or specific device emulations.

#### `accept_downloads` and `downloads_path`
- **Description**:
  - `accept_downloads`: Allows file downloads.
  - `downloads_path`: Directory for storing downloads.
  - Default: `False`, `None`
- **Use Case**:
  - Use when downloading and analyzing files like PDFs or spreadsheets.

#### `storage_state`
- **Description**: Specifies cookies and local storage state.
  - Default: `None`
- **Use Case**:
  - Provide state data for authenticated or preconfigured sessions.

#### `ignore_https_errors`
- **Description**: Ignores HTTPS certificate errors.
  - Default: `True`
- **Use Case**:
  - Enable for crawling sites with invalid certificates (testing environments).

#### `java_script_enabled`
- **Description**: Toggles JavaScript execution in pages.
  - Default: `True`
- **Use Case**:
  - Disable for simpler, faster crawls where JavaScript is unnecessary.

#### `cookies`
- **Description**: List of cookies to add to the browser context.
  - Default: `[]`
- **Use Case**:
  - Use for authenticated or preconfigured crawling scenarios.

#### `headers`
- **Description**: Extra HTTP headers applied to all requests.
  - Default: `{}`
- **Use Case**:
  - Customize headers for API-like crawling or bypassing bot detections.

#### `user_agent` and `user_agent_mode`
- **Description**:
  - `user_agent`: Custom User-Agent string.
  - `user_agent_mode`: Mode for generating User-Agent (e.g., `"random"`).
  - Default: Standard Chromium-based User-Agent.
- **Use Case**:
  - Set static User-Agent for consistent identification.
  - Use `"random"` mode to reduce bot detection likelihood.

#### `text_mode`
- **Description**: Disables images and other rich content for faster load times.
  - Default: `False`
- **Use Case**:
  - Enable for text-only extraction tasks where speed is prioritized.

#### `light_mode`
- **Description**: Disables background features for performance gains.
  - Default: `False`
- **Use Case**:
  - Enable for high-performance crawls on resource-constrained environments.

#### `extra_args`
- **Description**: Additional command-line arguments for browser execution.
  - Default: `[]`
- **Use Case**:
  - Use for advanced browser configurations like WebRTC or GPU tuning.

#### `verbose`
- **Description**: Enable verbose logging of browser operations.
  - Default: `True`
- **Use Case**:
  - Enable for detailed logging during development and debugging.
  - Disable in production for better performance.

#### `sleep_on_close`
- **Description**: Adds a delay before closing the browser.
  - Default: `False`
- **Use Case**:
  - Enable when you need to ensure all browser operations are complete before closing.

## CrawlerRunConfig
The `CrawlerRunConfig` class centralizes parameters for controlling crawl operations. This configuration covers content extraction, page interactions, caching, and runtime behaviors. Below is an exhaustive breakdown of parameters and their best-use scenarios.

### Parameters and Use Cases

#### Content Processing Parameters

##### `word_count_threshold`
- **Description**: Minimum word count threshold for processing content.
  - Default: `200`
- **Use Case**:
  - Set a higher threshold for content-heavy pages to skip lightweight or irrelevant content.

##### `extraction_strategy`
- **Description**: Strategy for extracting structured data from crawled pages.
  - Default: `None` (uses `NoExtractionStrategy` by default).
- **Use Case**:
  - Use for schema-driven extraction when working with well-defined data models like JSON.

##### `chunking_strategy`
- **Description**: Strategy to chunk content before extraction.
  - Default: `RegexChunking()`.
- **Use Case**:
  - Use NLP-based chunking for semantic extractions or regex for predictable text blocks.

##### `markdown_generator`
- **Description**: Strategy for generating Markdown output.
  - Default: `None`.
- **Use Case**:
  - Use custom Markdown strategies for AI-ready outputs like RAG pipelines.

##### `content_filter`
- **Description**: Optional filter to prune irrelevant content.
  - Default: `None`.
- **Use Case**:
  - Use relevance-based filters for focused crawls, e.g., keyword-specific searches.

##### `only_text`
- **Description**: Extracts text-only content where applicable.
  - Default: `False`.
- **Use Case**:
  - Enable for extracting clean text without HTML tags or rich content.

##### `css_selector`
- **Description**: CSS selector to extract a specific portion of the page.
  - Default: `None`.
- **Use Case**:
  - Use when targeting specific page elements, like articles or headlines.

##### `excluded_tags`
- **Description**: List of HTML tags to exclude from processing.
  - Default: `None`.
- **Use Case**:
  - Remove elements like `<script>` or `<style>` during text extraction.

##### `keep_data_attributes`
- **Description**: Retain `data-*` attributes in the HTML.
  - Default: `False`.
- **Use Case**:
  - Enable for extracting custom attributes in HTML structures.

##### `remove_forms`
- **Description**: Removes all `<form>` elements from the page.
  - Default: `False`.
- **Use Case**:
  - Use when forms are irrelevant and clutter the extracted content.

##### `prettiify`
- **Description**: Beautifies the HTML output.
  - Default: `False`.
- **Use Case**:
  - Enable for generating readable HTML outputs.

---

#### Caching Parameters

##### `cache_mode`
- **Description**: Controls how caching is handled.
  - Default: `CacheMode.ENABLED`.
- **Use Case**:
  - Use `WRITE_ONLY` mode for crawls where fresh content is critical.

##### `session_id`
- **Description**: Specifies a session ID to persist browser context.
  - Default: `None`.
- **Use Case**:
  - Use for maintaining login states or multi-page workflows.

##### `bypass_cache`, `disable_cache`, `no_cache_read`, `no_cache_write`
- **Description**: Legacy parameters for cache handling.
  - Default: `False`.
- **Use Case**:
  - These options provide finer control when overriding default caching behaviors.

---

#### Page Navigation and Timing Parameters

##### `wait_until`
- **Description**: Defines the navigation wait condition (e.g., `"domcontentloaded"`).
  - Default: `"domcontentloaded"`.
- **Use Case**:
  - Adjust to `"networkidle"` for pages with heavy JavaScript rendering.

##### `page_timeout`
- **Description**: Timeout in milliseconds for page operations.
  - Default: `60000` (60 seconds).
- **Use Case**:
  - Increase for slow-loading pages or complex sites.

##### `wait_for`
- **Description**: CSS selector or JS condition to wait for before extraction.
  - Default: `None`.
- **Use Case**:
  - Use for dynamic content that requires specific elements to load.

##### `wait_for_images`
- **Description**: Waits for images to load before content extraction.
  - Default: `True`.
- **Use Case**:
  - Disable for faster crawls when image data isn’t required.

##### `delay_before_return_html`
- **Description**: Delay in seconds before retrieving HTML.
  - Default: `0.1`.
- **Use Case**:
  - Use for ensuring final DOM updates are captured.

##### `mean_delay` and `max_range`
- **Description**: Configures base and random delays between requests.
  - Default: `0.1` (mean), `0.3` (max).
- **Use Case**:
  - Increase for stealthy crawls to avoid bot detection.

##### `semaphore_count`
- **Description**: Number of concurrent operations allowed.
  - Default: `5`.
- **Use Case**:
  - Adjust based on system resources and network limitations.

---

#### Page Interaction Parameters

##### `js_code`
- **Description**: JavaScript code or snippets to execute on the page.
  - Default: `None`.
- **Use Case**:
  - Use for custom interactions like clicking tabs or dynamically loading content.

##### `js_only`
- **Description**: Indicates subsequent calls rely only on JS updates.
  - Default: `False`.
- **Use Case**:
  - Enable for single-page applications (SPAs) with dynamic content.

##### `scan_full_page`
- **Description**: Simulates scrolling to load all content.
  - Default: `False`.
- **Use Case**:
  - Use for infinite-scroll pages or loading all dynamic elements.

##### `adjust_viewport_to_content`
- **Description**: Adjusts viewport to match content dimensions.
  - Default: `False`.
- **Use Case**:
  - Enable for capturing content-heavy pages fully.

---

#### Media Handling Parameters

##### `screenshot`
- **Description**: Captures a screenshot after crawling.
  - Default: `False`.
- **Use Case**:
  - Enable for visual debugging or reporting purposes.

##### `pdf`
- **Description**: Generates a PDF of the page.
  - Default: `False`.
- **Use Case**:
  - Use for archiving or sharing rendered page outputs.

##### `image_description_min_word_threshold` and `image_score_threshold`
- **Description**: Controls thresholds for image description extraction and processing.
  - Default: `50` (words), `3` (score).
- **Use Case**:
  - Adjust for higher relevance or descriptive quality of image metadata.

---

#### Debugging and Logging Parameters

##### `verbose`
- **Description**: Enables detailed logging.
  - Default: `True`.
- **Use Case**:
  - Use for troubleshooting or analyzing crawler behavior.

##### `log_console`
- **Description**: Logs browser console messages.
  - Default: `False`.
- **Use Case**:
  - Enable when debugging JavaScript errors on pages.

##### `parser_type`
- **Description**: Type of parser to use for HTML parsing.
  - Default: `"lxml"`
- **Use Case**:
  - Use when specific HTML parsing requirements are needed.
  - `"lxml"` provides good performance and standards compliance.

##### `prettiify`
- **Description**: Apply `fast_format_html` to produce prettified HTML output.
  - Default: `False`
- **Use Case**:
  - Enable for better readability of extracted HTML content.
  - Useful during development and debugging.

##### `fetch_ssl_certificate`
- **Description**: Fetch and store SSL certificate information during crawling.
  - Default: `False`
- **Use Case**:
  - Enable when SSL certificate analysis is required.
  - Useful for security audits and certificate validation.

##### `url`
- **Description**: Target URL for the crawl operation.
  - Default: `None`
- **Use Case**:
  - Set when initializing a crawler for a specific URL.
  - Can be overridden during actual crawl operations.

##### `log_console`
- **Description**: Log browser console messages during crawling.
  - Default: `False`
- **Use Case**:
  - Enable to capture JavaScript console output.
  - Useful for debugging JavaScript-heavy pages.