# Crawl4ai - AI Friendly Documentation (aka LLM.TXT)

This document provides AI-friendly documentation for the Crawl4ai library. It contains three types of content:

**Memory (Facts)**: Similar to traditional LLM.txt files, this section contains factual information about the library - what it is, what it does, its components, APIs, and capabilities. This is the reference knowledge that an AI needs to understand the library.

**Reasoning (Instructions)**: This section instructs AI models on how to use the factual knowledge, think about problems in the way the library authors intended, and provide solutions that align with the library's design philosophy. It guides the AI's problem-solving approach when working with Crawl4ai.

**Examples**: Practical code examples demonstrating how to use the library's features in real-world scenarios. These examples help AI models understand the practical application of the concepts.

## Content (Memory)

Okay, I have read the objective and instructions. I will now generate the detailed Markdown outline for a "Foundational Memory" document for the `core` component of `crawl4ai`, using the provided `code_analysis_output.md` (the concatenated code snippets from the previous prompt) as the primary source of truth for API details.

```markdown
# Detailed Outline for crawl4ai - core Component

**Target Document Type:** memory
**Target Output Filename Suggestion:** `llm_memory_core.md`
**Library Version Context:** 0.6.3
**Outline Generation Date:** 2025-05-24
---

## 1. Introduction to Core Components
    * 1.1. Purpose: Provides the foundational classes, configurations, and data models for web crawling and scraping operations within the `crawl4ai` library.
    * 1.2. Key Functionalities:
        *   Orchestration of asynchronous web crawling (`AsyncWebCrawler`).
        *   Configuration of browser behavior and specific crawl runs (`BrowserConfig`, `CrawlerRunConfig`).
        *   Standardized data structures for crawl results and associated data (`CrawlResult`, `Media`, `Links`, etc.).
        *   Strategies for fetching web content (`AsyncPlaywrightCrawlerStrategy`, `AsyncHTTPCrawlerStrategy`).
        *   Management of browser instances and sessions (`BrowserManager`, `ManagedBrowser`).
        *   Asynchronous logging (`AsyncLogger`).
    * 1.3. Relationship with other `crawl4ai` components:
        *   The `core` component serves as the foundation upon which specialized strategies (e.g., PDF processing, Markdown generation, content extraction, chunking, content filtering) are built and integrated.

## 2. Main Class: `AsyncWebCrawler`
    * 2.1. Purpose: The primary class for orchestrating asynchronous web crawling operations. It manages browser instances (via a `BrowserManager`), applies crawling strategies, and processes HTML content to produce structured results.
    * 2.2. Initialization (`__init__`)
        * 2.2.1. Signature:
            ```python
            def __init__(
                self,
                crawler_strategy: Optional[AsyncCrawlerStrategy] = None,
                config: Optional[BrowserConfig] = None,
                base_directory: str = str(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())),
                thread_safe: bool = False,
                logger: Optional[AsyncLoggerBase] = None,
                **kwargs,
            ):
            ```
        * 2.2.2. Parameters:
            * `crawler_strategy (Optional[AsyncCrawlerStrategy])`: The strategy to use for fetching web content. If `None`, defaults to `AsyncPlaywrightCrawlerStrategy` initialized with `config` and `logger`.
            * `config (Optional[BrowserConfig])`: Configuration object for browser settings. If `None`, a default `BrowserConfig()` is created.
            * `base_directory (str)`: The base directory for storing crawl4ai related files, such as cache and logs. Defaults to `os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())`.
            * `thread_safe (bool)`: If `True`, uses an `asyncio.Lock` for thread-safe operations, particularly relevant for `arun_many`. Default: `False`.
            * `logger (Optional[AsyncLoggerBase])`: An instance of a logger. If `None`, a default `AsyncLogger` is initialized using `base_directory` and `config.verbose`.
            * `**kwargs`: Additional keyword arguments, primarily for backward compatibility, passed to the `AsyncPlaywrightCrawlerStrategy` if `crawler_strategy` is not provided.
    * 2.3. Key Public Attributes/Properties:
        * `browser_config (BrowserConfig)`: Read-only. The browser configuration object used by the crawler.
        * `crawler_strategy (AsyncCrawlerStrategy)`: Read-only. The active crawling strategy instance.
        * `logger (AsyncLoggerBase)`: Read-only. The logger instance used by the crawler.
        * `ready (bool)`: Read-only. `True` if the crawler has been started and is ready to perform crawl operations, `False` otherwise.
    * 2.4. Lifecycle Methods:
        * 2.4.1. `async start() -> AsyncWebCrawler`:
            * Purpose: Asynchronously initializes the crawler strategy (e.g., launches the browser). This must be called before `arun` or `arun_many` if the crawler is not used as an asynchronous context manager.
            * Returns: The `AsyncWebCrawler` instance (`self`).
        * 2.4.2. `async close() -> None`:
            * Purpose: Asynchronously closes the crawler strategy and cleans up resources (e.g., closes the browser). This should be called if `start()` was used explicitly.
        * 2.4.3. `async __aenter__() -> AsyncWebCrawler`:
            * Purpose: Entry point for asynchronous context management. Calls `self.start()`.
            * Returns: The `AsyncWebCrawler` instance (`self`).
        * 2.4.4. `async __aexit__(exc_type, exc_val, exc_tb) -> None`:
            * Purpose: Exit point for asynchronous context management. Calls `self.close()`.
    * 2.5. Primary Crawl Methods:
        * 2.5.1. `async arun(url: str, config: Optional[CrawlerRunConfig] = None, **kwargs) -> RunManyReturn`:
            * Purpose: Performs a single crawl operation for the given URL or raw HTML content.
            * Parameters:
                * `url (str)`: The URL to crawl (e.g., "http://example.com", "file:///path/to/file.html") or raw HTML content prefixed with "raw:" (e.g., "raw:<html>...</html>").
                * `config (Optional[CrawlerRunConfig])`: Configuration for this specific crawl run. If `None`, a default `CrawlerRunConfig()` is used.
                * `**kwargs`: Additional parameters passed to the underlying `aprocess_html` method, can be used to override settings in `config`.
            * Returns: `RunManyReturn` (which resolves to `CrawlResultContainer` containing a single `CrawlResult`).
        * 2.5.2. `async arun_many(urls: List[str], config: Optional[CrawlerRunConfig] = None, dispatcher: Optional[BaseDispatcher] = None, **kwargs) -> RunManyReturn`:
            * Purpose: Crawls multiple URLs concurrently using a specified or default dispatcher strategy.
            * Parameters:
                * `urls (List[str])`: A list of URLs to crawl.
                * `config (Optional[CrawlerRunConfig])`: Configuration applied to all crawl runs in this batch.
                * `dispatcher (Optional[BaseDispatcher])`: The dispatcher strategy to manage concurrent crawls. Defaults to `MemoryAdaptiveDispatcher`.
                * `**kwargs`: Additional parameters passed to the underlying `arun` method for each URL.
            * Returns: `RunManyReturn`. If `config.stream` is `True`, returns an `AsyncGenerator[CrawlResult, None]`. Otherwise, returns a `CrawlResultContainer` (list-like) of `CrawlResult` objects.
    * 2.6. Internal Processing Method (User-Facing Effects):
        * 2.6.1. `async aprocess_html(url: str, html: str, extracted_content: Optional[str], config: CrawlerRunConfig, screenshot_data: Optional[str], pdf_data: Optional[bytes], verbose: bool, **kwargs) -> CrawlResult`:
            * Purpose: Processes the fetched HTML content. This method is called internally by `arun` after content is fetched (either from a live crawl or cache). It applies scraping strategies, content filtering, and Markdown generation based on the `config`.
            * Parameters:
                * `url (str)`: The URL of the content being processed.
                * `html (str)`: The raw HTML content.
                * `extracted_content (Optional[str])`: Pre-extracted content from a previous step or cache.
                * `config (CrawlerRunConfig)`: Configuration for this processing run.
                * `screenshot_data (Optional[str])`: Base64 encoded screenshot data, if available.
                * `pdf_data (Optional[bytes])`: PDF data, if available.
                * `verbose (bool)`: Verbosity setting for logging during processing.
                * `**kwargs`: Additional parameters, including `is_raw_html` and `redirected_url`.
            * Returns: A `CrawlResult` object containing the processed data.

## 3. Core Configuration Objects

    * 3.1. Class `BrowserConfig` (from `crawl4ai.async_configs`)
        * 3.1.1. Purpose: Configures the browser instance launched by Playwright, including its type, mode, display settings, proxy, user agent, and persistent storage options.
        * 3.1.2. Initialization (`__init__`)
            * Signature:
            ```python
            def __init__(
                self,
                browser_type: str = "chromium",
                headless: bool = True,
                browser_mode: str = "dedicated",
                use_managed_browser: bool = False,
                cdp_url: Optional[str] = None,
                use_persistent_context: bool = False,
                user_data_dir: Optional[str] = None,
                channel: Optional[str] = "chromium", # Note: 'channel' from code, outline had 'chrome_channel'
                proxy: Optional[str] = None, # Note: 'proxy' from code, outline had 'proxy_config' for this level
                proxy_config: Optional[Union[ProxyConfig, dict, None]] = None,
                viewport_width: int = 1080,
                viewport_height: int = 600,
                viewport: Optional[dict] = None,
                accept_downloads: bool = False,
                downloads_path: Optional[str] = None,
                storage_state: Optional[Union[str, dict, None]] = None,
                ignore_https_errors: bool = True,
                java_script_enabled: bool = True,
                sleep_on_close: bool = False,
                verbose: bool = True,
                cookies: Optional[list] = None,
                headers: Optional[dict] = None,
                user_agent: str = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36",
                user_agent_mode: str = "",
                user_agent_generator_config: Optional[dict] = None, # Note: 'user_agent_generator_config' from code
                text_mode: bool = False,
                light_mode: bool = False,
                extra_args: Optional[list] = None,
                debugging_port: int = 9222,
                host: str = "localhost",
            ):
            ```
            * Key Parameters:
                * `browser_type (str)`: Type of browser to launch ("chromium", "firefox", "webkit"). Default: "chromium".
                * `headless (bool)`: Whether to run the browser in headless mode. Default: `True`.
                * `browser_mode (str)`: How the browser should be initialized ("builtin", "dedicated", "cdp", "docker"). Default: "dedicated".
                * `use_managed_browser (bool)`: Whether to launch the browser using a managed approach (e.g., via CDP). Default: `False`.
                * `cdp_url (Optional[str])`: URL for Chrome DevTools Protocol endpoint. Default: `None`.
                * `use_persistent_context (bool)`: Use a persistent browser context (profile). Default: `False`.
                * `user_data_dir (Optional[str])`: Path to user data directory for persistent sessions. Default: `None`.
                * `channel (Optional[str])`: Browser channel (e.g., "chromium", "chrome", "msedge"). Default: "chromium".
                * `proxy (Optional[str])`: Simple proxy server URL string.
                * `proxy_config (Optional[Union[ProxyConfig, dict, None]])`: Detailed proxy configuration object or dictionary. Takes precedence over `proxy`.
                * `viewport_width (int)`: Default viewport width. Default: `1080`.
                * `viewport_height (int)`: Default viewport height. Default: `600`.
                * `viewport (Optional[dict])`: Dictionary to set viewport dimensions, overrides `viewport_width` and `viewport_height` if set (e.g., `{"width": 1920, "height": 1080}`). Default: `None`.
                * `accept_downloads (bool)`: Whether to allow file downloads. Default: `False`.
                * `downloads_path (Optional[str])`: Directory to store downloaded files. Default: `None`.
                * `storage_state (Optional[Union[str, dict, None]])`: Path to a file or a dictionary containing browser storage state (cookies, localStorage). Default: `None`.
                * `ignore_https_errors (bool)`: Ignore HTTPS certificate errors. Default: `True`.
                * `java_script_enabled (bool)`: Enable JavaScript execution. Default: `True`.
                * `user_agent (str)`: Custom User-Agent string. Default: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36".
                * `user_agent_mode (str)`: Mode for generating User-Agent (e.g., "random"). Default: `""` (uses provided `user_agent`).
                * `user_agent_generator_config (Optional[dict])`: Configuration for User-Agent generation if `user_agent_mode` is active. Default: `{}`.
                * `text_mode (bool)`: If `True`, disables images and rich content for faster loading. Default: `False`.
                * `light_mode (bool)`: Disables certain background features for performance. Default: `False`.
                * `extra_args (Optional[list])`: Additional command-line arguments for the browser. Default: `None` (resolves to `[]`).
                * `debugging_port (int)`: Port for browser debugging protocol. Default: `9222`.
                * `host (str)`: Host for browser debugging protocol. Default: "localhost".
        * 3.1.3. Key Public Methods:
            * `clone(**kwargs) -> BrowserConfig`: Creates a new `BrowserConfig` instance as a copy of the current one, with specified keyword arguments overriding existing values.
            * `to_dict() -> dict`: Returns a dictionary representation of the configuration object's attributes.
            * `dump() -> dict`: Serializes the configuration object to a JSON-serializable dictionary, including nested objects.
            * `static load(data: dict) -> BrowserConfig`: Deserializes a `BrowserConfig` instance from a dictionary (previously created by `dump`).
            * `static from_kwargs(kwargs: dict) -> BrowserConfig`: Creates a `BrowserConfig` instance directly from a dictionary of keyword arguments.

    * 3.2. Class `CrawlerRunConfig` (from `crawl4ai.async_configs`)
        * 3.2.1. Purpose: Specifies settings for an individual crawl operation initiated by `arun()` or `arun_many()`. These settings can override or augment the global `BrowserConfig`.
        * 3.2.2. Initialization (`__init__`)
            * Signature:
            ```python
            def __init__(
                self,
                # Content Processing Parameters
                word_count_threshold: int = MIN_WORD_THRESHOLD,
                extraction_strategy: Optional[ExtractionStrategy] = None,
                chunking_strategy: ChunkingStrategy = RegexChunking(),
                markdown_generator: MarkdownGenerationStrategy = DefaultMarkdownGenerator(),
                only_text: bool = False,
                css_selector: Optional[str] = None,
                target_elements: Optional[List[str]] = None,
                excluded_tags: Optional[list] = None,
                excluded_selector: Optional[str] = None,
                keep_data_attributes: bool = False,
                keep_attrs: Optional[list] = None,
                remove_forms: bool = False,
                prettify: bool = False,
                parser_type: str = "lxml",
                scraping_strategy: ContentScrapingStrategy = None, # Will default to WebScrapingStrategy
                proxy_config: Optional[Union[ProxyConfig, dict, None]] = None,
                proxy_rotation_strategy: Optional[ProxyRotationStrategy] = None,
                # Browser Location and Identity Parameters
                locale: Optional[str] = None,
                timezone_id: Optional[str] = None,
                geolocation: Optional[GeolocationConfig] = None,
                # SSL Parameters
                fetch_ssl_certificate: bool = False,
                # Caching Parameters
                cache_mode: CacheMode = CacheMode.BYPASS,
                session_id: Optional[str] = None,
                bypass_cache: bool = False, # Legacy
                disable_cache: bool = False, # Legacy
                no_cache_read: bool = False, # Legacy
                no_cache_write: bool = False, # Legacy
                shared_data: Optional[dict] = None,
                # Page Navigation and Timing Parameters
                wait_until: str = "domcontentloaded",
                page_timeout: int = PAGE_TIMEOUT,
                wait_for: Optional[str] = None,
                wait_for_timeout: Optional[int] = None,
                wait_for_images: bool = False,
                delay_before_return_html: float = 0.1,
                mean_delay: float = 0.1,
                max_range: float = 0.3,
                semaphore_count: int = 5,
                # Page Interaction Parameters
                js_code: Optional[Union[str, List[str]]] = None,
                js_only: bool = False,
                ignore_body_visibility: bool = True,
                scan_full_page: bool = False,
                scroll_delay: float = 0.2,
                process_iframes: bool = False,
                remove_overlay_elements: bool = False,
                simulate_user: bool = False,
                override_navigator: bool = False,
                magic: bool = False,
                adjust_viewport_to_content: bool = False,
                # Media Handling Parameters
                screenshot: bool = False,
                screenshot_wait_for: Optional[float] = None,
                screenshot_height_threshold: int = SCREENSHOT_HEIGHT_THRESHOLD,
                pdf: bool = False,
                capture_mhtml: bool = False,
                image_description_min_word_threshold: int = IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD,
                image_score_threshold: int = IMAGE_SCORE_THRESHOLD,
                table_score_threshold: int = 7,
                exclude_external_images: bool = False,
                exclude_all_images: bool = False,
                # Link and Domain Handling Parameters
                exclude_social_media_domains: Optional[list] = None, # Note: 'exclude_social_media_domains' from code
                exclude_external_links: bool = False,
                exclude_social_media_links: bool = False,
                exclude_domains: Optional[list] = None,
                exclude_internal_links: bool = False,
                # Debugging and Logging Parameters
                verbose: bool = True,
                log_console: bool = False,
                # Network and Console Capturing Parameters
                capture_network_requests: bool = False,
                capture_console_messages: bool = False,
                # Connection Parameters (for HTTPCrawlerStrategy)
                method: str = "GET",
                stream: bool = False,
                url: Optional[str] = None,
                # Robots.txt Handling
                check_robots_txt: bool = False,
                # User Agent Parameters
                user_agent: Optional[str] = None,
                user_agent_mode: Optional[str] = None,
                user_agent_generator_config: Optional[dict] = None, # Note: 'user_agent_generator_config' from code
                # Deep Crawl Parameters
                deep_crawl_strategy: Optional[DeepCrawlStrategy] = None,
                # Experimental Parameters
                experimental: Optional[Dict[str, Any]] = None,
            ):
            ```
            * Key Parameters:
                * `word_count_threshold (int)`: Minimum word count for a content block to be considered. Default: `MIN_WORD_THRESHOLD` (200).
                * `extraction_strategy (Optional[ExtractionStrategy])`: Strategy for structured data extraction (e.g., `LLMExtractionStrategy`, `JsonCssExtractionStrategy`). Default: `None` (falls back to `NoExtractionStrategy`).
                * `chunking_strategy (ChunkingStrategy)`: Strategy for splitting content into chunks before extraction. Default: `RegexChunking()`.
                * `markdown_generator (MarkdownGenerationStrategy)`: Strategy for converting HTML to Markdown. Default: `DefaultMarkdownGenerator()`.
                * `cache_mode (CacheMode)`: Caching behavior for this run. Default: `CacheMode.BYPASS`.
                * `session_id (Optional[str])`: ID for session persistence (reusing browser tabs/contexts). Default: `None`.
                * `js_code (Optional[Union[str, List[str]]])`: JavaScript code snippets to execute on the page. Default: `None`.
                * `wait_for (Optional[str])`: CSS selector or JS condition (prefixed with "js:") to wait for before proceeding. Default: `None`.
                * `page_timeout (int)`: Timeout for page operations (e.g., navigation) in milliseconds. Default: `PAGE_TIMEOUT` (60000ms).
                * `screenshot (bool)`: If `True`, capture a screenshot of the page. Default: `False`.
                * `pdf (bool)`: If `True`, generate a PDF of the page. Default: `False`.
                * `capture_mhtml (bool)`: If `True`, capture an MHTML snapshot of the page. Default: `False`.
                * `exclude_external_links (bool)`: If `True`, exclude external links from results. Default: `False`.
                * `stream (bool)`: If `True` (used with `arun_many`), results are yielded as an `AsyncGenerator`. Default: `False`.
                * `check_robots_txt (bool)`: If `True`, crawler will check and respect `robots.txt` rules. Default: `False`.
                * `user_agent (Optional[str])`: Override the browser's User-Agent for this specific run.
        * 3.2.3. Key Public Methods:
            * `clone(**kwargs) -> CrawlerRunConfig`: Creates a new `CrawlerRunConfig` instance as a copy of the current one, with specified keyword arguments overriding existing values.
            * `to_dict() -> dict`: Returns a dictionary representation of the configuration object's attributes.
            * `dump() -> dict`: Serializes the configuration object to a JSON-serializable dictionary, including nested objects.
            * `static load(data: dict) -> CrawlerRunConfig`: Deserializes a `CrawlerRunConfig` instance from a dictionary (previously created by `dump`).
            * `static from_kwargs(kwargs: dict) -> CrawlerRunConfig`: Creates a `CrawlerRunConfig` instance directly from a dictionary of keyword arguments.

    * 3.3. Supporting Configuration Objects (from `crawl4ai.async_configs`)
        * 3.3.1. Class `GeolocationConfig`
            * Purpose: Defines geolocation (latitude, longitude, accuracy) to be emulated by the browser.
            * Initialization (`__init__`):
                ```python
                def __init__(
                    self,
                    latitude: float,
                    longitude: float,
                    accuracy: Optional[float] = 0.0
                ):
                ```
            * Parameters:
                * `latitude (float)`: Latitude coordinate (e.g., 37.7749).
                * `longitude (float)`: Longitude coordinate (e.g., -122.4194).
                * `accuracy (Optional[float])`: Accuracy in meters. Default: `0.0`.
            * Methods:
                * `static from_dict(geo_dict: Dict) -> GeolocationConfig`: Creates an instance from a dictionary.
                * `to_dict() -> Dict`: Converts the instance to a dictionary.
                * `clone(**kwargs) -> GeolocationConfig`: Creates a copy with updated values.
        * 3.3.2. Class `ProxyConfig`
            * Purpose: Defines the settings for a single proxy server, including server address, authentication credentials, and optional IP.
            * Initialization (`__init__`):
                ```python
                def __init__(
                    self,
                    server: str,
                    username: Optional[str] = None,
                    password: Optional[str] = None,
                    ip: Optional[str] = None,
                ):
                ```
            * Parameters:
                * `server (str)`: Proxy server URL (e.g., "http://127.0.0.1:8080", "socks5://user:pass@host:port").
                * `username (Optional[str])`: Username for proxy authentication.
                * `password (Optional[str])`: Password for proxy authentication.
                * `ip (Optional[str])`: Optional IP address associated with the proxy for verification.
            * Methods:
                * `static from_string(proxy_str: str) -> ProxyConfig`: Creates an instance from a string (e.g., "ip:port:username:password" or "ip:port").
                * `static from_dict(proxy_dict: Dict) -> ProxyConfig`: Creates an instance from a dictionary.
                * `static from_env(env_var: str = "PROXIES") -> List[ProxyConfig]`: Loads a list of proxies from a comma-separated environment variable.
                * `to_dict() -> Dict`: Converts the instance to a dictionary.
                * `clone(**kwargs) -> ProxyConfig`: Creates a copy with updated values.
        * 3.3.3. Class `HTTPCrawlerConfig`
            * Purpose: Configuration for the `AsyncHTTPCrawlerStrategy`, specifying HTTP method, headers, data/JSON payload, and redirect/SSL verification behavior.
            * Initialization (`__init__`):
                ```python
                def __init__(
                    self,
                    method: str = "GET",
                    headers: Optional[Dict[str, str]] = None,
                    data: Optional[Dict[str, Any]] = None,
                    json: Optional[Dict[str, Any]] = None,
                    follow_redirects: bool = True,
                    verify_ssl: bool = True,
                ):
                ```
            * Parameters:
                * `method (str)`: HTTP method (e.g., "GET", "POST"). Default: "GET".
                * `headers (Optional[Dict[str, str]])`: Dictionary of HTTP request headers. Default: `None`.
                * `data (Optional[Dict[str, Any]])`: Dictionary of form data to send in the request body. Default: `None`.
                * `json (Optional[Dict[str, Any]])`: JSON data to send in the request body. Default: `None`.
                * `follow_redirects (bool)`: Whether to automatically follow HTTP redirects. Default: `True`.
                * `verify_ssl (bool)`: Whether to verify SSL certificates. Default: `True`.
            * Methods:
                * `static from_kwargs(kwargs: dict) -> HTTPCrawlerConfig`: Creates an instance from keyword arguments.
                * `to_dict() -> dict`: Converts config to a dictionary.
                * `clone(**kwargs) -> HTTPCrawlerConfig`: Creates a copy with updated values.
                * `dump() -> dict`: Serializes the config to a dictionary.
                * `static load(data: dict) -> HTTPCrawlerConfig`: Deserializes from a dictionary.
        * 3.3.4. Class `LLMConfig`
            * Purpose: Configures settings for interacting with Large Language Models, including provider choice, API credentials, and generation parameters.
            * Initialization (`__init__`):
                ```python
                def __init__(
                    self,
                    provider: str = DEFAULT_PROVIDER,
                    api_token: Optional[str] = None,
                    base_url: Optional[str] = None,
                    temperature: Optional[float] = None,
                    max_tokens: Optional[int] = None,
                    top_p: Optional[float] = None,
                    frequency_penalty: Optional[float] = None,
                    presence_penalty: Optional[float] = None,
                    stop: Optional[List[str]] = None,
                    n: Optional[int] = None,
                ):
                ```
            * Key Parameters:
                * `provider (str)`: Name of the LLM provider (e.g., "openai/gpt-4o", "ollama/llama3.3", "groq/llama3-8b-8192"). Default: `DEFAULT_PROVIDER` (from `crawl4ai.config`).
                * `api_token (Optional[str])`: API token for the LLM provider. If prefixed with "env:", it reads from the specified environment variable (e.g., "env:OPENAI_API_KEY"). If not provided, it attempts to load from default environment variables based on the provider.
                * `base_url (Optional[str])`: Custom base URL for the LLM API endpoint.
                * `temperature (Optional[float])`: Sampling temperature for generation.
                * `max_tokens (Optional[int])`: Maximum number of tokens to generate.
                * `top_p (Optional[float])`: Nucleus sampling parameter.
                * `frequency_penalty (Optional[float])`: Penalty for token frequency.
                * `presence_penalty (Optional[float])`: Penalty for token presence.
                * `stop (Optional[List[str]])`: List of stop sequences for generation.
                * `n (Optional[int])`: Number of completions to generate.
            * Methods:
                * `static from_kwargs(kwargs: dict) -> LLMConfig`: Creates an instance from keyword arguments.
                * `to_dict() -> dict`: Converts config to a dictionary.
                * `clone(**kwargs) -> LLMConfig`: Creates a copy with updated values.

## 4. Core Data Models (Results & Payloads from `crawl4ai.models`)

    * 4.1. Class `CrawlResult(BaseModel)`
        * Purpose: A Pydantic model representing the comprehensive result of a single crawl and processing operation.
        * Key Fields:
            * `url (str)`: The final URL that was crawled (after any redirects).
            * `html (str)`: The raw HTML content fetched from the URL.
            * `success (bool)`: `True` if the crawl operation (fetching and initial processing) was successful, `False` otherwise.
            * `cleaned_html (Optional[str])`: HTML content after sanitization and removal of unwanted tags/attributes as per configuration. Default: `None`.
            * `_markdown (Optional[MarkdownGenerationResult])`: (Private Attribute) Holds the `MarkdownGenerationResult` object if Markdown generation was performed. Use the `markdown` property to access. Default: `None`.
            * `markdown (Optional[Union[str, MarkdownGenerationResult]])`: (Property) Provides access to Markdown content. Behaves as a string (raw markdown) by default but allows access to `MarkdownGenerationResult` attributes (e.g., `result.markdown.fit_markdown`).
            * `extracted_content (Optional[str])`: JSON string representation of structured data extracted by an `ExtractionStrategy`. Default: `None`.
            * `media (Media)`: An object containing lists of `MediaItem` for images, videos, audio, and extracted tables. Default: `Media()`.
            * `links (Links)`: An object containing lists of `Link` for internal and external hyperlinks found on the page. Default: `Links()`.
            * `downloaded_files (Optional[List[str]])`: A list of file paths if any files were downloaded during the crawl. Default: `None`.
            * `js_execution_result (Optional[Dict[str, Any]])`: The result of any JavaScript code executed on the page. Default: `None`.
            * `screenshot (Optional[str])`: Base64 encoded string of the page screenshot, if `screenshot=True` was set. Default: `None`.
            * `pdf (Optional[bytes])`: Raw bytes of the PDF generated from the page, if `pdf=True` was set. Default: `None`.
            * `mhtml (Optional[str])`: MHTML snapshot of the page, if `capture_mhtml=True` was set. Default: `None`.
            * `metadata (Optional[dict])`: Dictionary of metadata extracted from the page (e.g., title, description, OpenGraph tags, Twitter card data). Default: `None`.
            * `error_message (Optional[str])`: A message describing the error if `success` is `False`. Default: `None`.
            * `session_id (Optional[str])`: The session ID used for this crawl, if applicable. Default: `None`.
            * `response_headers (Optional[dict])`: HTTP response headers from the server. Default: `None`.
            * `status_code (Optional[int])`: HTTP status code of the response. Default: `None`.
            * `ssl_certificate (Optional[SSLCertificate])`: Information about the SSL certificate if `fetch_ssl_certificate=True`. Default: `None`.
            * `dispatch_result (Optional[DispatchResult])`: Metadata about the task execution from the dispatcher (e.g., timings, memory usage). Default: `None`.
            * `redirected_url (Optional[str])`: The original URL if the request was redirected. Default: `None`.
            * `network_requests (Optional[List[Dict[str, Any]]])`: List of captured network requests if `capture_network_requests=True`. Default: `None`.
            * `console_messages (Optional[List[Dict[str, Any]]])`: List of captured browser console messages if `capture_console_messages=True`. Default: `None`.
        * Methods:
            * `model_dump(*args, **kwargs)`: Serializes the `CrawlResult` model to a dictionary, ensuring the `_markdown` private attribute is correctly handled and included as "markdown" in the output if present.

    * 4.2. Class `MarkdownGenerationResult(BaseModel)`
        * Purpose: A Pydantic model that holds various forms of Markdown generated from HTML content.
        * Fields:
            * `raw_markdown (str)`: The basic, direct conversion of HTML to Markdown.
            * `markdown_with_citations (str)`: Markdown content with inline citations (e.g., [^1^]) and a references section.
            * `references_markdown (str)`: The Markdown content for the "References" section, listing all cited links.
            * `fit_markdown (Optional[str])`: Markdown generated specifically from content deemed "relevant" by a content filter (like `PruningContentFilter` or `LLMContentFilter`), if such a filter was applied. Default: `None`.
            * `fit_html (Optional[str])`: The filtered HTML content that was used to generate `fit_markdown`. Default: `None`.
        * Methods:
            * `__str__(self) -> str`: Returns `self.raw_markdown` when the object is cast to a string.

    * 4.3. Class `ScrapingResult(BaseModel)`
        * Purpose: A Pydantic model representing a standardized output from content scraping strategies.
        * Fields:
            * `cleaned_html (str)`: The primary sanitized and processed HTML content.
            * `success (bool)`: Indicates if the scraping operation was successful.
            * `media (Media)`: A `Media` object containing extracted images, videos, audio, and tables.
            * `links (Links)`: A `Links` object containing extracted internal and external links.
            * `metadata (Dict[str, Any])`: A dictionary of metadata extracted from the page (e.g., title, description).

    * 4.4. Class `MediaItem(BaseModel)`
        * Purpose: A Pydantic model representing a generic media item like an image, video, or audio file.
        * Fields:
            * `src (Optional[str])`: The source URL of the media item. Default: `""`.
            * `data (Optional[str])`: Base64 encoded data for inline media. Default: `""`.
            * `alt (Optional[str])`: Alternative text for the media item (e.g., image alt text). Default: `""`.
            * `desc (Optional[str])`: A description or surrounding text related to the media item. Default: `""`.
            * `score (Optional[int])`: A relevance or importance score, if calculated by a strategy. Default: `0`.
            * `type (str)`: The type of media (e.g., "image", "video", "audio"). Default: "image".
            * `group_id (Optional[int])`: An identifier to group related media variants (e.g., different resolutions of the same image from a srcset). Default: `0`.
            * `format (Optional[str])`: The detected file format (e.g., "jpeg", "png", "mp4"). Default: `None`.
            * `width (Optional[int])`: The width of the media item in pixels, if available. Default: `None`.

    * 4.5. Class `Link(BaseModel)`
        * Purpose: A Pydantic model representing an extracted hyperlink.
        * Fields:
            * `href (Optional[str])`: The URL (href attribute) of the link. Default: `""`.
            * `text (Optional[str])`: The anchor text of the link. Default: `""`.
            * `title (Optional[str])`: The title attribute of the link, if present. Default: `""`.
            * `base_domain (Optional[str])`: The base domain extracted from the `href`. Default: `""`.

    * 4.6. Class `Media(BaseModel)`
        * Purpose: A Pydantic model that acts as a container for lists of different types of media items found on a page.
        * Fields:
            * `images (List[MediaItem])`: A list of `MediaItem` objects representing images. Default: `[]`.
            * `videos (List[MediaItem])`: A list of `MediaItem` objects representing videos. Default: `[]`.
            * `audios (List[MediaItem])`: A list of `MediaItem` objects representing audio files. Default: `[]`.
            * `tables (List[Dict])`: A list of dictionaries, where each dictionary represents an extracted HTML table with keys like "headers", "rows", "caption", "summary". Default: `[]`.

    * 4.7. Class `Links(BaseModel)`
        * Purpose: A Pydantic model that acts as a container for lists of internal and external links.
        * Fields:
            * `internal (List[Link])`: A list of `Link` objects considered internal to the crawled site. Default: `[]`.
            * `external (List[Link])`: A list of `Link` objects pointing to external sites. Default: `[]`.

    * 4.8. Class `AsyncCrawlResponse(BaseModel)`
        * Purpose: A Pydantic model representing the raw response from a crawler strategy's `crawl` method. This data is then processed further to create a `CrawlResult`.
        * Fields:
            * `html (str)`: The raw HTML content of the page.
            * `response_headers (Dict[str, str])`: A dictionary of HTTP response headers.
            * `js_execution_result (Optional[Dict[str, Any]])`: The result from any JavaScript code executed on the page. Default: `None`.
            * `status_code (int)`: The HTTP status code of the response.
            * `screenshot (Optional[str])`: Base64 encoded screenshot data, if captured. Default: `None`.
            * `pdf_data (Optional[bytes])`: Raw PDF data, if captured. Default: `None`.
            * `mhtml_data (Optional[str])`: MHTML snapshot data, if captured. Default: `None`.
            * `downloaded_files (Optional[List[str]])`: A list of local file paths for any files downloaded during the crawl. Default: `None`.
            * `ssl_certificate (Optional[SSLCertificate])`: SSL certificate information for the site. Default: `None`.
            * `redirected_url (Optional[str])`: The original URL requested if the final URL is a result of redirection. Default: `None`.
            * `network_requests (Optional[List[Dict[str, Any]]])`: Captured network requests if enabled. Default: `None`.
            * `console_messages (Optional[List[Dict[str, Any]]])`: Captured console messages if enabled. Default: `None`.

    * 4.9. Class `TokenUsage(BaseModel)`
        * Purpose: A Pydantic model to track token usage statistics for interactions with Large Language Models.
        * Fields:
            * `completion_tokens (int)`: Number of tokens used for the LLM's completion/response. Default: `0`.
            * `prompt_tokens (int)`: Number of tokens used for the input prompt to the LLM. Default: `0`.
            * `total_tokens (int)`: Total number of tokens used (prompt + completion). Default: `0`.
            * `completion_tokens_details (Optional[dict])`: Provider-specific detailed breakdown of completion tokens. Default: `None`.
            * `prompt_tokens_details (Optional[dict])`: Provider-specific detailed breakdown of prompt tokens. Default: `None`.

    * 4.10. Class `SSLCertificate(dict)` (from `crawl4ai.ssl_certificate`)
        * Purpose: Represents an SSL certificate's information, behaving like a dictionary for direct JSON serialization and easy access to its fields.
        * Key Fields (accessed as dictionary keys):
            * `subject (dict)`: Dictionary of subject fields (e.g., `{"CN": "example.com", "O": "Example Inc."}`).
            * `issuer (dict)`: Dictionary of issuer fields.
            * `version (int)`: Certificate version number.
            * `serial_number (str)`: Certificate serial number (hexadecimal string).
            * `not_before (str)`: Validity start date and time (ASN.1/UTC format string, e.g., "YYYYMMDDHHMMSSZ").
            * `not_after (str)`: Validity end date and time (ASN.1/UTC format string).
            * `fingerprint (str)`: SHA-256 fingerprint of the certificate (lowercase hex string).
            * `signature_algorithm (str)`: The algorithm used to sign the certificate (e.g., "sha256WithRSAEncryption").
            * `raw_cert (str)`: Base64 encoded string of the raw DER-encoded certificate.
            * `extensions (List[dict])`: A list of dictionaries, each representing a certificate extension with "name" and "value" keys.
        * Static Methods:
            * `from_url(url: str, timeout: int = 10) -> Optional[SSLCertificate]`: Fetches the SSL certificate from the given URL and returns an `SSLCertificate` instance, or `None` on failure.
        * Instance Methods:
            * `to_json(filepath: Optional[str] = None) -> Optional[str]`: Exports the certificate information as a JSON string. If `filepath` is provided, writes to the file and returns `None`.
            * `to_pem(filepath: Optional[str] = None) -> Optional[str]`: Exports the certificate in PEM format as a string. If `filepath` is provided, writes to the file and returns `None`.
            * `to_der(filepath: Optional[str] = None) -> Optional[bytes]`: Exports the raw certificate in DER format as bytes. If `filepath` is provided, writes to the file and returns `None`.
        * Example:
            ```python
            # Assuming 'cert' is an SSLCertificate instance
            # print(cert["subject"]["CN"])
            # cert.to_pem("my_cert.pem")
            ```

    * 4.11. Class `DispatchResult(BaseModel)`
        * Purpose: Contains metadata about a task's execution when processed by a dispatcher (e.g., in `arun_many`).
        * Fields:
            * `task_id (str)`: A unique identifier for the dispatched task.
            * `memory_usage (float)`: Memory usage (in MB) recorded during the task's execution.
            * `peak_memory (float)`: Peak memory usage (in MB) recorded during the task's execution.
            * `start_time (Union[datetime, float])`: The start time of the task (can be a `datetime` object or a Unix timestamp float).
            * `end_time (Union[datetime, float])`: The end time of the task.
            * `error_message (str)`: Any error message if the task failed during dispatch or execution. Default: `""`.

    * 4.12. `CrawlResultContainer(Generic[CrawlResultT])`
        * Purpose: A generic container for `CrawlResult` objects, primarily used as the return type for `arun_many` when `stream=False`. It behaves like a list, allowing iteration, indexing, and length checking.
        * Methods:
            * `__iter__(self)`: Allows iteration over the contained `CrawlResult` objects.
            * `__getitem__(self, index)`: Allows accessing `CrawlResult` objects by index.
            * `__len__(self)`: Returns the number of `CrawlResult` objects contained.
            * `__repr__(self)`: Provides a string representation of the container.
        * Attribute:
            * `_results (List[CrawlResultT])`: The internal list holding the `CrawlResult` objects.

    * 4.13. `RunManyReturn` (Type Alias from `crawl4ai.models`)
        * Purpose: A type alias defining the possible return types for the `arun_many` method of `AsyncWebCrawler`.
        * Definition: `Union[CrawlResultContainer[CrawlResult], AsyncGenerator[CrawlResult, None]]`
            * This means `arun_many` will return a `CrawlResultContainer` (a list-like object of all `CrawlResult` instances) if `CrawlerRunConfig.stream` is `False` (the default).
            * It will return an `AsyncGenerator` yielding individual `CrawlResult` instances if `CrawlerRunConfig.stream` is `True`.

## 5. Core Crawler Strategies (from `crawl4ai.async_crawler_strategy`)

    * 5.1. Abstract Base Class `AsyncCrawlerStrategy(ABC)`
        * Purpose: Defines the common interface that all asynchronous crawler strategies must implement. This allows `AsyncWebCrawler` to use different fetching mechanisms (e.g., Playwright, HTTP requests) interchangeably.
        * Initialization (`__init__`):
            ```python
            def __init__(self, browser_config: BrowserConfig, logger: AsyncLoggerBase):
            ```
            * Parameters:
                * `browser_config (BrowserConfig)`: The browser configuration to be used by the strategy.
                * `logger (AsyncLoggerBase)`: The logger instance for logging strategy-specific events.
        * Key Abstract Methods (must be implemented by concrete subclasses):
            * `async crawl(self, url: str, config: CrawlerRunConfig) -> AsyncCrawlResponse`:
                * Purpose: Fetches the content from the given URL according to the `config`.
                * Returns: An `AsyncCrawlResponse` object containing the raw fetched data.
            * `async __aenter__(self)`:
                * Purpose: Asynchronous context manager entry, typically for initializing resources (e.g., launching a browser).
            * `async __aexit__(self, exc_type, exc_val, exc_tb)`:
                * Purpose: Asynchronous context manager exit, for cleaning up resources.
        * Key Concrete Methods (available to all strategies):
            * `set_custom_headers(self, headers: dict) -> None`:
                * Purpose: Sets custom HTTP headers to be used by the strategy for subsequent requests.
            * `update_user_agent(self, user_agent: str) -> None`:
                * Purpose: Updates the User-Agent string used by the strategy.
            * `set_hook(self, hook_name: str, callback: Callable) -> None`:
                * Purpose: Registers a callback function for a specific hook point in the crawling lifecycle.
            * `async_run_hook(self, hook_name: str, *args, **kwargs) -> Any`:
                * Purpose: Executes a registered hook with the given arguments.
            * `async_get_default_context(self) -> BrowserContext`:
                * Purpose: Retrieves the default browser context (Playwright specific, might raise `NotImplementedError` in non-Playwright strategies).
            * `async_create_new_page(self, context: BrowserContext) -> Page`:
                * Purpose: Creates a new page within a given browser context (Playwright specific).
            * `async_get_page(self, url: str, config: CrawlerRunConfig, session_id: Optional[str]) -> Tuple[Page, BrowserContext]`:
                * Purpose: Gets an existing page/context for a session or creates a new one (Playwright specific, managed by `BrowserManager`).
            * `async_close_page(self, page: Page, session_id: Optional[str]) -> None`:
                * Purpose: Closes a page, potentially keeping the associated context/session alive (Playwright specific).
            * `async_kill_session(self, session_id: str) -> None`:
                * Purpose: Kills (closes) a specific browser session, including its page and context (Playwright specific).

    * 5.2. Class `AsyncPlaywrightCrawlerStrategy(AsyncCrawlerStrategy)`
        * Purpose: The default crawler strategy, using Playwright to control a web browser for fetching and interacting with web pages. It supports complex JavaScript execution and provides hooks for various stages of the crawl.
        * Initialization (`__init__`):
            ```python
            def __init__(
                self,
                browser_config: Optional[BrowserConfig] = None,
                logger: Optional[AsyncLoggerBase] = None,
                browser_manager: Optional[BrowserManager] = None
            ):
            ```
            * Parameters:
                * `browser_config (Optional[BrowserConfig])`: Browser configuration. Defaults to a new `BrowserConfig()` if not provided.
                * `logger (Optional[AsyncLoggerBase])`: Logger instance. Defaults to a new `AsyncLogger()`.
                * `browser_manager (Optional[BrowserManager])`: An instance of `BrowserManager` to manage browser lifecycles and contexts. If `None`, a new `BrowserManager` is created internally.
        * Key Overridden/Implemented Methods:
            * `async crawl(self, url: str, config: CrawlerRunConfig) -> AsyncCrawlResponse`:
                * Purpose: Implements the crawling logic using Playwright. It navigates to the URL, executes JavaScript if specified, waits for conditions, captures screenshots/PDFs if requested, and returns the page content and other metadata.
            * `async aprocess_html(self, url: str, html: str, config: CrawlerRunConfig, **kwargs) -> CrawlResult`:
                * Purpose: (Note: While `AsyncWebCrawler` calls this, the default implementation is in `AsyncPlaywrightCrawlerStrategy` for convenience, acting as a bridge to the scraping strategy.) Processes the fetched HTML to produce a `CrawlResult`. This involves using the `scraping_strategy` from the `config` (defaults to `WebScrapingStrategy`) to clean HTML, extract media/links, and then uses the `markdown_generator` to produce Markdown.
        * Specific Public Methods:
            * `async_create_new_context(self, config: Optional[CrawlerRunConfig] = None) -> BrowserContext`:
                * Purpose: Creates a new Playwright `BrowserContext` based on the global `BrowserConfig` and optional overrides from `CrawlerRunConfig`.
            * `async_setup_context_default(self, context: BrowserContext, config: Optional[CrawlerRunConfig] = None) -> None`:
                * Purpose: Applies default settings to a `BrowserContext`, such as viewport size, user agent, custom headers, locale, timezone, and geolocation, based on `BrowserConfig` and `CrawlerRunConfig`.
            * `async_setup_context_hooks(self, context: BrowserContext, config: CrawlerRunConfig) -> None`:
                * Purpose: Sets up event listeners on the context for capturing network requests and console messages if `config.capture_network_requests` or `config.capture_console_messages` is `True`.
            * `async_handle_storage_state(self, context: BrowserContext, config: CrawlerRunConfig) -> None`:
                * Purpose: Loads cookies and localStorage from a `storage_state` file or dictionary (specified in `BrowserConfig` or `CrawlerRunConfig`) into the given `BrowserContext`.
        * Hooks (Callable via `set_hook(hook_name, callback)` and executed by `async_run_hook`):
            * `on_browser_created`: Called after the Playwright browser instance is launched or connected. Callback receives `(browser, **kwargs)`.
            * `on_page_context_created`: Called after a new Playwright `BrowserContext` and `Page` are created. Callback receives `(page, context, **kwargs)`.
            * `before_goto`: Called just before `page.goto(url)` is executed. Callback receives `(page, context, url, **kwargs)`.
            * `after_goto`: Called after `page.goto(url)` completes successfully. Callback receives `(page, context, url, response, **kwargs)`.
            * `on_user_agent_updated`: Called when the User-Agent string is updated for a context. Callback receives `(page, context, user_agent, **kwargs)`.
            * `on_execution_started`: Called when `js_code` execution begins on a page. Callback receives `(page, context, **kwargs)`.
            * `before_retrieve_html`: Called just before the final HTML content is retrieved from the page. Callback receives `(page, context, **kwargs)`.
            * `before_return_html`: Called just before the `AsyncCrawlResponse` is returned by the `crawl()` method of the strategy. Callback receives `(page, context, html_content, **kwargs)`.

    * 5.3. Class `AsyncHTTPCrawlerStrategy(AsyncCrawlerStrategy)`
        * Purpose: A lightweight crawler strategy that uses direct HTTP requests (via `httpx`) instead of a full browser. Suitable for static sites or when JavaScript execution is not needed.
        * Initialization (`__init__`):
            ```python
            def __init__(self, http_config: Optional[HTTPCrawlerConfig] = None, logger: Optional[AsyncLoggerBase] = None):
            ```
            * Parameters:
                * `http_config (Optional[HTTPCrawlerConfig])`: Configuration for HTTP requests (method, headers, data, etc.). Defaults to a new `HTTPCrawlerConfig()`.
                * `logger (Optional[AsyncLoggerBase])`: Logger instance. Defaults to a new `AsyncLogger()`.
        * Key Overridden/Implemented Methods:
            * `async crawl(self, url: str, http_config: Optional[HTTPCrawlerConfig] = None, **kwargs) -> AsyncCrawlResponse`:
                * Purpose: Fetches content from the URL using an HTTP GET or POST request via `httpx`. Does not execute JavaScript. Returns an `AsyncCrawlResponse` with HTML, status code, and headers. Screenshot, PDF, and MHTML capabilities are not available with this strategy.

## 6. Browser Management (from `crawl4ai.browser_manager`)

    * 6.1. Class `BrowserManager`
        * Purpose: Manages the lifecycle of Playwright browser instances and their contexts. It handles launching/connecting to browsers, creating new contexts with specific configurations, managing sessions for page reuse, and cleaning up resources.
        * Initialization (`__init__`):
            ```python
            def __init__(self, browser_config: BrowserConfig, logger: Optional[AsyncLoggerBase] = None):
            ```
            * Parameters:
                * `browser_config (BrowserConfig)`: The global browser configuration settings.
                * `logger (Optional[AsyncLoggerBase])`: Logger instance for browser management events.
        * Key Methods:
            * `async start() -> None`: Initializes the Playwright instance and launches or connects to the browser based on `browser_config` (e.g., launches a new browser instance or connects to an existing CDP endpoint via `ManagedBrowser`).
            * `async create_browser_context(self, crawlerRunConfig: Optional[CrawlerRunConfig] = None) -> playwright.async_api.BrowserContext`: Creates a new browser context. If `crawlerRunConfig` is provided, its settings (e.g., locale, viewport, proxy) can override the global `BrowserConfig`.
            * `async setup_context(self, context: playwright.async_api.BrowserContext, crawlerRunConfig: Optional[CrawlerRunConfig] = None, is_default: bool = False) -> None`: Applies various settings to a given browser context, including headers, cookies, viewport, geolocation, permissions, and storage state, based on `BrowserConfig` and `CrawlerRunConfig`.
            * `async get_page(self, crawlerRunConfig: CrawlerRunConfig) -> Tuple[playwright.async_api.Page, playwright.async_api.BrowserContext]`: Retrieves an existing page and context for a given `session_id` (if present in `crawlerRunConfig` and the session is active) or creates a new page and context. Manages context reuse based on a signature derived from `CrawlerRunConfig` to ensure contexts with different core settings (like proxy, locale) are isolated.
            * `async kill_session(self, session_id: str) -> None`: Closes the page and browser context associated with the given `session_id`, effectively ending that session.
            * `async close() -> None`: Closes all managed browser contexts and the main browser instance.

    * 6.2. Class `ManagedBrowser`
        * Purpose: Manages the lifecycle of a single, potentially persistent, browser process. It's used when `BrowserConfig.use_managed_browser` is `True` or `BrowserConfig.use_persistent_context` is `True`. It handles launching the browser with a specific user data directory and connecting via CDP.
        * Initialization (`__init__`):
            ```python
            def __init__(
                self,
                browser_type: str = "chromium",
                user_data_dir: Optional[str] = None,
                headless: bool = False,
                logger=None,
                host: str = "localhost",
                debugging_port: int = 9222,
                cdp_url: Optional[str] = None, # Added as per code_analysis
                browser_config: Optional[BrowserConfig] = None # Added as per code_analysis
            ):
            ```
            * Parameters:
                * `browser_type (str)`: "chromium", "firefox", or "webkit". Default: "chromium".
                * `user_data_dir (Optional[str])`: Path to the user data directory for the browser profile. If `None`, a temporary directory might be created.
                * `headless (bool)`: Whether to launch the browser in headless mode. Default: `False` (typically for managed/persistent scenarios).
                * `logger`: Logger instance.
                * `host (str)`: Host for the debugging port. Default: "localhost".
                * `debugging_port (int)`: Port for the Chrome DevTools Protocol. Default: `9222`.
                * `cdp_url (Optional[str])`: If provided, attempts to connect to an existing browser at this CDP URL instead of launching a new one.
                * `browser_config (Optional[BrowserConfig])`: The `BrowserConfig` object providing overall browser settings.
        * Key Methods:
            * `async start() -> str`: Starts the browser process (if not connecting to an existing `cdp_url`). If a new browser is launched, it uses the specified `user_data_dir` and `debugging_port`.
            * Returns: The CDP endpoint URL (e.g., "http://localhost:9222").
            * `async cleanup() -> None`: Terminates the browser process (if launched by this instance) and removes any temporary user data directory created by it.
        * Static Methods:
            * `async create_profile(cls, browser_config: Optional[BrowserConfig] = None, profile_name: Optional[str] = None, logger=None) -> str`:
                * Purpose: Launches a browser instance with a new or existing user profile, allowing interactive setup (e.g., manual login, cookie acceptance). The browser remains open until the user closes it.
                * Parameters:
                    * `browser_config (Optional[BrowserConfig])`: Optional browser configuration to use.
                    * `profile_name (Optional[str])`: Name for the profile. If `None`, a default name is used.
                    * `logger`: Logger instance.
                * Returns: The path to the created/used user data directory, which can then be passed to `BrowserConfig.user_data_dir`.
            * `list_profiles(cls) -> List[str]`:
                * Purpose: Lists the names of all browser profiles stored in the default Crawl4AI profiles directory (`~/.crawl4ai/profiles`).
                * Returns: A list of profile name strings.
            * `delete_profile(cls, profile_name_or_path: str) -> bool`:
                * Purpose: Deletes a browser profile either by its name (if in the default directory) or by its full path.
                * Returns: `True` if deletion was successful, `False` otherwise.

    * 6.3. Function `clone_runtime_state(src: BrowserContext, dst: BrowserContext, crawlerRunConfig: Optional[CrawlerRunConfig] = None, browserConfig: Optional[BrowserConfig] = None) -> None`
        * Purpose: Asynchronously copies runtime state (cookies, localStorage, session storage) from a source `BrowserContext` to a destination `BrowserContext`. Can also apply headers and geolocation from `CrawlerRunConfig` or `BrowserConfig` to the destination context.
        * Parameters:
            * `src (BrowserContext)`: The source browser context.
            * `dst (BrowserContext)`: The destination browser context.
            * `crawlerRunConfig (Optional[CrawlerRunConfig])`: Optional run configuration to apply to `dst`.
            * `browserConfig (Optional[BrowserConfig])`: Optional browser configuration to apply to `dst`.

## 7. Proxy Rotation Strategies (from `crawl4ai.proxy_strategy`)

    * 7.1. Abstract Base Class `ProxyRotationStrategy(ABC)`
        * Purpose: Defines the interface for strategies that provide a sequence of proxy configurations, enabling proxy rotation.
        * Abstract Methods:
            * `async get_next_proxy(self) -> Optional[ProxyConfig]`:
                * Purpose: Asynchronously retrieves the next `ProxyConfig` from the strategy.
                * Returns: A `ProxyConfig` object or `None` if no more proxies are available or an error occurs.
            * `add_proxies(self, proxies: List[ProxyConfig]) -> None`:
                * Purpose: Adds a list of `ProxyConfig` objects to the strategy's pool of proxies.

    * 7.2. Class `RoundRobinProxyStrategy(ProxyRotationStrategy)`
        * Purpose: A simple proxy rotation strategy that cycles through a list of provided proxies in a round-robin fashion.
        * Initialization (`__init__`):
            ```python
            def __init__(self, proxies: Optional[List[ProxyConfig]] = None):
            ```
            * Parameters:
                * `proxies (Optional[List[ProxyConfig]])`: An initial list of `ProxyConfig` objects. If `None`, the list is empty and proxies must be added via `add_proxies`.
        * Methods:
            * `add_proxies(self, proxies: List[ProxyConfig]) -> None`: Adds new `ProxyConfig` objects to the internal list of proxies and reinitializes the cycle.
            * `async get_next_proxy(self) -> Optional[ProxyConfig]`: Returns the next `ProxyConfig` from the list, cycling back to the beginning when the end is reached. Returns `None` if the list is empty.

## 8. Logging (from `crawl4ai.async_logger`)

    * 8.1. Abstract Base Class `AsyncLoggerBase(ABC)`
        * Purpose: Defines the basic interface for an asynchronous logger. Concrete implementations should provide methods for logging messages at different levels.
    * 8.2. Class `AsyncLogger(AsyncLoggerBase)`
        * Purpose: The default asynchronous logger for `crawl4ai`. It provides structured logging to both the console and optionally to a file, with customizable icons, colors, and verbosity levels.
        * Initialization (`__init__`):
            ```python
            def __init__(
                self,
                log_file: Optional[str] = None,
                verbose: bool = True,
                tag_width: int = 15, # outline had 10, code has 15
                icons: Optional[Dict[str, str]] = None,
                colors: Optional[Dict[LogLevel, LogColor]] = None, # Corrected type annotation
                log_level: LogLevel = LogLevel.INFO # Assuming LogLevel.INFO is a typical default
            ):
            ```
            * Parameters:
                * `log_file (Optional[str])`: Path to a file where logs should be written. If `None`, logs only to console.
                * `verbose (bool)`: If `True`, enables more detailed logging (DEBUG level). Default: `True`.
                * `tag_width (int)`: Width for the tag part of the log message. Default: `15`.
                * `icons (Optional[Dict[str, str]])`: Custom icons for different log tags.
                * `colors (Optional[Dict[LogLevel, LogColor]])`: Custom colors for different log levels.
                * `log_level (LogLevel)`: Minimum log level to output.
        * Key Methods (for logging):
            * `info(self, message: str, tag: Optional[str] = None, **params) -> None`: Logs an informational message.
            * `warning(self, message: str, tag: Optional[str] = None, **params) -> None`: Logs a warning message.
            * `error(self, message: str, tag: Optional[str] = None, **params) -> None`: Logs an error message.
            * `debug(self, message: str, tag: Optional[str] = None, **params) -> None`: Logs a debug message (only if `verbose=True` or `log_level` is DEBUG).
            * `url_status(self, url: str, success: bool, timing: float, tag: str = "FETCH", **params) -> None`: Logs the status of a URL fetch operation, including success/failure and timing.
            * `error_status(self, url: str, error: str, tag: str = "ERROR", **params) -> None`: Logs an error encountered for a specific URL.

## 9. Core Utility Functions (from `crawl4ai.async_configs`)
    * 9.1. `to_serializable_dict(obj: Any, ignore_default_value: bool = False) -> Dict`
        * Purpose: Recursively converts a Python object (often a Pydantic model or a dataclass instance used for configuration) into a dictionary that is safe for JSON serialization. It handles nested objects, enums, and basic types.
        * Parameters:
            * `obj (Any)`: The object to be serialized.
            * `ignore_default_value (bool)`: If `True`, fields whose current value is the same as their default value (if applicable, e.g., for Pydantic models) might be omitted from the resulting dictionary. Default: `False`.
        * Returns: `Dict` - A JSON-serializable dictionary representation of the object.
    * 9.2. `from_serializable_dict(data: Any) -> Any`
        * Purpose: Recursively reconstructs Python objects from a dictionary representation (typically one created by `to_serializable_dict`). It attempts to instantiate classes based on a "type" key in the dictionary if present.
        * Parameters:
            * `data (Any)`: The dictionary (or basic type) to be deserialized.
        * Returns: `Any` - The reconstructed Python object or the original data if no special deserialization rule applies.
    * 9.3. `is_empty_value(value: Any) -> bool`
        * Purpose: Checks if a given value is considered "empty" (e.g., `None`, an empty string, an empty list, an empty dictionary).
        * Returns: `bool` - `True` if the value is empty, `False` otherwise.

## 10. Enumerations (Key Enums used in Core)
    * 10.1. `CacheMode` (from `crawl4ai.cache_context`, defined in `crawl4ai.async_configs` as per provided code)
        * Purpose: Defines the caching behavior for crawl operations.
        * Members:
            * `ENABLE`: (Value: "enable") Normal caching behavior; read from cache if available, write to cache after fetching.
            * `DISABLE`: (Value: "disable") No caching at all; always fetch fresh content and do not write to cache.
            * `READ_ONLY`: (Value: "read_only") Only read from the cache; do not write new or updated content to the cache.
            * `WRITE_ONLY`: (Value: "write_only") Only write to the cache after fetching; do not read from the cache.
            * `BYPASS`: (Value: "bypass") Skip the cache entirely for this specific operation; fetch fresh content and do not write to cache. This is often the default for individual `CrawlerRunConfig` instances.
    * 10.2. `DisplayMode` (from `crawl4ai.models`, used by `CrawlerMonitor`)
        * Purpose: Defines the display mode for the `CrawlerMonitor`.
        * Members:
            * `DETAILED`: Shows detailed information for each task.
            * `AGGREGATED`: Shows summary statistics and overall progress.
    * 10.3. `CrawlStatus` (from `crawl4ai.models`, used by `CrawlStats`)
        * Purpose: Represents the status of a crawl task.
        * Members:
            * `QUEUED`: Task is waiting to be processed.
            * `IN_PROGRESS`: Task is currently being processed.
            * `COMPLETED`: Task finished successfully.
            * `FAILED`: Task failed.

## 11. Versioning
    * 11.1. Accessing Library Version:
        * The current version of the `crawl4ai` library can be accessed programmatically via the `__version__` attribute of the top-level `crawl4ai` package.
        * Example:
            ```python
            from crawl4ai import __version__ as crawl4ai_version
            print(f"Crawl4AI Version: {crawl4ai_version}")
            # Expected output based on provided code: Crawl4AI Version: 0.6.3
            ```

## 12. Basic Usage Examples

    * 12.1. Minimal Crawl:
        ```python
        import asyncio
        from crawl4ai import AsyncWebCrawler

        async def main():
            async with AsyncWebCrawler() as crawler:
                result = await crawler.arun(url="http://example.com")
                if result.success:
                    print("Markdown (first 300 chars):")
                    print(result.markdown.raw_markdown[:300]) # Accessing raw_markdown
                else:
                    print(f"Error: {result.error_message}")

        if __name__ == "__main__":
            asyncio.run(main())
        ```

    * 12.2. Crawl with Basic Configuration:
        ```python
        import asyncio
        from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

        async def main():
            browser_cfg = BrowserConfig(headless=True, browser_type="firefox")
            run_cfg = CrawlerRunConfig(
                cache_mode=CacheMode.BYPASS,
                word_count_threshold=50
            )
            async with AsyncWebCrawler(config=browser_cfg) as crawler:
                result = await crawler.arun(url="http://example.com", config=run_cfg)
                if result.success:
                    print(f"Status Code: {result.status_code}")
                    print(f"Cleaned HTML length: {len(result.cleaned_html)}")
                else:
                    print(f"Error: {result.error_message}")
        
        if __name__ == "__main__":
            asyncio.run(main())
        ```

    * 12.3. Accessing Links and Images from Result:
        ```python
        import asyncio
        from crawl4ai import AsyncWebCrawler

        async def main():
            async with AsyncWebCrawler() as crawler:
                result = await crawler.arun(url="http://example.com")
                if result.success:
                    print(f"Found {len(result.links.internal)} internal links.")
                    if result.links.internal:
                        print(f"First internal link: {result.links.internal[0].href}")
                    
                    print(f"Found {len(result.media.images)} images.")
                    if result.media.images:
                        print(f"First image src: {result.media.images[0].src}")
                else:
                    print(f"Error: {result.error_message}")

        if __name__ == "__main__":
            asyncio.run(main())
        ```
```

## Content (Reasoning)

Okay, I have read and understood the objective and the detailed outline provided for the `core` component. I will now generate the "Reasoning & Problem-Solving Framework" document in Markdown format, strictly adhering to the outline and incorporating the principles for a "reasoning" document.

I will use the code snippets and documentation details you provided in the previous turn as the basis for this document.

```markdown
# Detailed Outline for crawl4ai - core Component

**Target Document Type:** reasoning
**Target Output Filename Suggestion:** `llm_reasoning_core.md`
**Library Version Context:** 0.6.3
**Outline Generation Date:** 2024-05-24
---

# I. Introduction to the Crawl4ai Core

The `core` component of Crawl4ai is the engine that powers all web crawling and scraping activities. Understanding its architecture and how its main pieces fit together is crucial for leveraging the full potential of the library. This section provides a high-level overview to set the stage.

*   A. **Purpose of the Core Component:**
    *   **Explaining why the `core` is central to `crawl4ai`'s functionality:**
        The `core` component serves as the central nervous system of Crawl4ai. It's responsible for orchestrating the entire lifecycle of a web crawl, from launching and managing browser instances to fetching web content, processing it through various strategies (scraping, markdown generation, data extraction), and finally delivering structured results. Without the `core`, other specialized components like PDF processors or advanced extraction strategies would lack the fundamental web interaction capabilities.
    *   **Core problems the `core` component aims to solve for users:**
        *   **Simplified Browser Automation:** Abstracts away the complexities of browser automation libraries like Playwright, providing a cleaner and more focused API for web crawling.
        *   **Flexible Configuration:** Offers granular control over how browsers are launched (`BrowserConfig`) and how individual crawl operations are executed (`CrawlerRunConfig`), allowing users to tailor crawls to specific site behaviors and data needs.
        *   **Unified Result Processing:** Provides a consistent `CrawlResult` object, regardless of whether the content was fetched via a full browser or a simple HTTP request, making it easier to build downstream processing pipelines.
        *   **Efficient Resource Management:** Includes mechanisms for managing browser contexts, sessions, and caching to optimize performance and resource utilization, especially for larger crawls.
        *   **Extensibility:** Designed with a strategy pattern, allowing users to plug in custom behaviors for crawling, scraping, and content processing.

*   B. **Key Abstractions and Design Philosophy:**
    *   **Brief overview of the main classes (`AsyncWebCrawler`, `BrowserConfig`, `CrawlerRunConfig`) and their roles:**
        *   `AsyncWebCrawler`: This is your primary interaction point. You instantiate it, (optionally) configure it with a `BrowserConfig`, and then use its methods (`arun`, `arun_many`) to perform crawls, passing in `CrawlerRunConfig` objects for per-crawl specifics.
        *   `BrowserConfig`: Defines *how the browser itself is set up*. This is typically a one-time configuration for a `AsyncWebCrawler` instance, covering aspects like which browser engine to use, whether to run headless, proxy settings, user agent strings, and persistent user profiles. Think of it as setting up your physical web browser application.
        *   `CrawlerRunConfig`: Defines *how a specific URL or set of URLs should be crawled and processed*. This is passed to `arun` or `arun_many` and can override some browser-level settings. It controls aspects like caching, JavaScript execution for that run, content selectors, media capture (screenshots/PDFs), and extraction strategies. Think of it as the instructions you give your browser for a particular browsing session on a specific site.
    *   **How the separation of configurations (browser vs. run-specific) aids flexibility:**
        This separation is a key design choice. It allows you to:
        1.  **Reuse Browser Setups:** Configure a browser once (e.g., with specific proxies or a logged-in profile via `BrowserConfig`) and then use that same setup for multiple different crawl tasks, each with its own `CrawlerRunConfig` (e.g., one task extracts text, another captures screenshots, another looks for specific data structures).
        2.  **Targeted Overrides:** For a specific `arun` call, you might need to temporarily use a different user-agent or a different proxy without altering the global browser setup. `CrawlerRunConfig` allows these granular overrides.
        3.  **Clarity:** Keeps concerns separate. Browser setup is distinct from the instructions for a particular crawling job.

*   C. **Common Workflows Involving the Core:**
    *   **Single Page Static Crawl:** Initialize `AsyncWebCrawler`, call `arun()` with a URL and a basic `CrawlerRunConfig` (often just defaults or `CacheMode.BYPASS`).
    *   **Single Page Dynamic Crawl:** Similar to static, but `CrawlerRunConfig` might include `js_code` to interact with the page (e.g., click buttons, scroll) and `wait_for` conditions to ensure dynamic content loads.
    *   **Multi-Page Batch Crawl:** Use `arun_many()` with a list of URLs. You might use a shared `CrawlerRunConfig` if processing is similar for all URLs, or provide custom logic to generate different `CrawlerRunConfig` objects per URL if needed.
    *   **Persistent Session Crawl:** Use `BrowserConfig` with `user_data_dir` for profile persistence or `storage_state`. Then, use `CrawlerRunConfig` with a consistent `session_id` across multiple `arun()` calls to maintain login state or navigate multi-step processes.
    *   **API/JSON Fetching:** Use `AsyncHTTPCrawlerStrategy` with `AsyncWebCrawler` and an appropriate `HTTPCrawlerConfig` (within `CrawlerRunConfig`) for lightweight, non-browser data retrieval.

# II. Mastering `AsyncWebCrawler`: The Heart of Crawling

The `AsyncWebCrawler` class is the cornerstone of Crawl4ai. It orchestrates browser interactions, manages configurations, and processes web content. Understanding its nuances will empower you to build sophisticated and efficient web crawlers.

*   A. **Understanding `AsyncWebCrawler`'s Role:**
    *   **Why `AsyncWebCrawler` is the primary entry point for most crawling tasks:**
        `AsyncWebCrawler` provides a high-level, user-friendly API that abstracts away the complexities of direct browser automation (like Playwright or Selenium). It integrates browser launching, context management, page navigation, content retrieval, and initial processing into a cohesive workflow. Whether you're fetching a single page or thousands, `AsyncWebCrawler` is designed to be your go-to tool.
    *   **Its responsibilities in managing browser instances and executing crawl operations:**
        *   **Browser Lifecycle:** Manages the launching and closing of browser instances based on the provided `BrowserConfig`.
        *   **Context and Page Management:** Handles the creation and isolation of browser contexts and pages, which is crucial for maintaining separate states (cookies, local storage) if needed, especially with `session_id` usage.
        *   **Strategy Execution:** Delegates the actual "crawling" (fetching content from a URL) to a configurable `AsyncCrawlerStrategy` (defaulting to `AsyncPlaywrightCrawlerStrategy`).
        *   **Configuration Orchestration:** Applies both `BrowserConfig` (global browser settings) and `CrawlerRunConfig` (per-crawl settings) to each operation.
        *   **Result Aggregation:** Takes the raw response from the crawler strategy and processes it through scraping and markdown generation (via a `ContentScrapingStrategy` and `MarkdownGenerationStrategy`) to produce the final `CrawlResult`.

*   B. **Initialization and Lifecycle Management:**
    *   1.  **Best Practices for Initializing `AsyncWebCrawler`:**
        *   **When to pass a `BrowserConfig` vs. relying on defaults:**
            *   **Rely on defaults:** For quick, simple crawls where standard browser behavior is sufficient (e.g., fetching a public, static webpage).
            *   **Pass a `BrowserConfig`:**
                *   When you need to specify a browser engine other than Chromium (e.g., Firefox, WebKit).
                *   To run in headed (non-headless) mode for debugging.
                *   To configure proxies.
                *   To set custom user agents or other HTTP headers globally.
                *   To use persistent browser profiles (`user_data_dir`) or load existing browser state (`storage_state`).
                *   For advanced launch arguments.
        *   **Considerations for `logger` and `thread_safe` parameters:**
            *   `logger`: Pass a custom `AsyncLogger` instance if you have a centralized logging setup or need specific log formatting/destinations. If `None`, a default logger is created. For reasoning documents, verbose logging is often helpful.
            *   `thread_safe`: The default `False` is generally fine for most `asyncio`-based applications. Set to `True` only if you are interacting with the same `AsyncWebCrawler` instance from multiple OS-level threads, which is a less common pattern in asyncio. It introduces a lock around critical sections.

    *   2.  **Understanding the Crawler Lifecycle (`start`, `close`, and Context Management):**
        *   **When and why to use explicit `crawler.start()` and `crawler.close()`:**
            *   **Scenarios:**
                *   **Long-running applications:** Where the crawler needs to be available for an extended period to process requests on demand.
                *   **Reusing crawler instances:** If you intend to make multiple, separate `arun()` or `arun_many()` calls with the same underlying browser instance and configuration over time.
                *   **Pre-warming/Setup:** If you need to perform setup tasks (like logging into a site) once and then run multiple crawls using that established session.
            *   **Benefits:**
                *   **Resource Control:** You explicitly manage when browser resources are allocated and released.
                *   **Performance:** Avoids the overhead of launching a new browser for every crawl operation if the instance is reused.
            *   **Code Example: Illustrating explicit start/close for a persistent crawler**
                ```python
                import asyncio
                from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

                async def long_running_crawler_task():
                    # Configure browser once
                    browser_cfg = BrowserConfig(headless=True, user_agent="MyPersistentCrawler/1.0")
                    crawler = AsyncWebCrawler(config=browser_cfg)

                    print("Starting crawler...")
                    await crawler.start() # Explicitly start the browser

                    try:
                        # First crawl
                        run_cfg1 = CrawlerRunConfig(url="https://example.com/page1")
                        result1 = await crawler.arun(config=run_cfg1)
                        if result1.success:
                            print(f"Page 1 ({result1.status_code}): {result1.markdown.raw_markdown[:100]}...")

                        # ... some time later, or another task ...

                        # Second crawl with the same browser instance
                        run_cfg2 = CrawlerRunConfig(url="https://example.com/page2")
                        result2 = await crawler.arun(config=run_cfg2)
                        if result2.success:
                            print(f"Page 2 ({result2.status_code}): {result2.markdown.raw_markdown[:100]}...")

                    finally:
                        print("Closing crawler...")
                        await crawler.close() # Explicitly close the browser and release resources

                if __name__ == "__main__":
                    asyncio.run(long_running_crawler_task())
                ```
        *   **Leveraging `async with AsyncWebCrawler(...)` (Context Manager):**
            *   **Benefits:**
                *   **Automatic Resource Cleanup:** Ensures `crawler.start()` is called at the beginning and `crawler.close()` is called at the end, even if errors occur within the block. This is the most common and recommended way for most use cases.
                *   **Cleaner Code:** Reduces boilerplate for resource management.
            *   **When it's most appropriate:**
                For most scripts or functions where the crawler's lifetime is confined to that specific block of code.
            *   **Code Example: Illustrating typical context manager usage**
                ```python
                import asyncio
                from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

                async def simple_crawl_with_context_manager():
                    browser_cfg = BrowserConfig(headless=True)
                    run_cfg = CrawlerRunConfig(url="https://example.com")

                    async with AsyncWebCrawler(config=browser_cfg) as crawler:
                        # crawler.start() is implicitly called
                        result = await crawler.arun(config=run_cfg)
                        if result.success:
                            print(f"Content from {result.url}: {result.markdown.raw_markdown[:100]}...")
                        else:
                            print(f"Failed: {result.error_message}")
                    # crawler.close() is implicitly called here, even if an exception occurred

                if __name__ == "__main__":
                    asyncio.run(simple_crawl_with_context_manager())
                ```
        *   **Potential pitfalls:**
            *   Forgetting to call `await crawler.close()` when using explicit `await crawler.start()`. This can lead to dangling browser processes and resource leaks. The context manager (`async with`) prevents this.

*   C. **Executing Crawl Operations:**
    *   1.  **Single URL Crawling with `arun()`:**
        *   **Common use cases for `arun()`:**
            *   Fetching and processing a single webpage.
            *   Performing a specific interaction on one page (e.g., filling a form, then extracting results).
            *   Testing configurations on a sample URL.
        *   **Essential `CrawlerRunConfig` parameters for `arun()`:**
            *   `url` (str): The URL to crawl. This is the most fundamental parameter.
            *   `js_code` (Optional[str | List[str]]): JavaScript to execute after the page loads.
            *   `wait_for` (Optional[str]): A CSS selector or JS condition to wait for before proceeding.
            *   See Section IV for a full dive into `CrawlerRunConfig`.
        *   **Workflow: Fetching and processing a single page:**
            1.  Initialize `AsyncWebCrawler` (potentially with a `BrowserConfig`).
            2.  Create a `CrawlerRunConfig` instance, setting the `url` and any other desired options.
            3.  Call `await crawler.arun(config=your_run_config)`.
            4.  Process the returned `CrawlResult` object.
        *   **Code Example: Basic `arun()` usage with a simple `CrawlerRunConfig`**
            ```python
            import asyncio
            from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

            async def crawl_single_page():
                # Basic CrawlerRunConfig for a specific URL
                run_config = CrawlerRunConfig(
                    url="https://quotes.toscrape.com/",
                    cache_mode=CacheMode.BYPASS # Ensure fresh fetch for this example
                )

                async with AsyncWebCrawler() as crawler:
                    result = await crawler.arun(config=run_config)
                    if result.success:
                        print(f"Successfully crawled: {result.url}")
                        print(f"Markdown (first 200 chars): {result.markdown.raw_markdown[:200]}...")
                    else:
                        print(f"Failed to crawl {result.url}: {result.error_message}")

            if __name__ == "__main__":
                asyncio.run(crawl_single_page())
            ```
        *   **Troubleshooting common `arun()` issues:**
            *   **Timeouts:** The page might be taking too long to load, or a `wait_for` condition isn't met. Adjust `page_timeout` or `wait_for_timeout` in `CrawlerRunConfig`.
            *   **JavaScript Errors:** If `js_code` execution fails, check the browser console (run with `headless=False` in `BrowserConfig`) or enable `log_console=True` in `CrawlerRunConfig` to see console messages in Crawl4ai logs.
            *   **Content Not Found:** The `css_selector` or `target_elements` might be incorrect, or the content might load after the crawler has finished processing. Use `wait_for` or `delay_before_retrieve_html`.

    *   2.  **Batch Crawling with `arun_many()`:**
        *   **When to prefer `arun_many()` over looping `arun()`:**
            `arun_many()` is generally preferred when you have a list of URLs to process because it leverages an internal dispatcher (like `MemoryAdaptiveDispatcher` by default) to manage concurrency, rate limiting, and resource usage more effectively than a simple Python loop of `await crawler.arun()`. This leads to better performance and stability for bulk operations.
        *   **How `arun_many()` handles concurrency and dispatching:**
            It uses a dispatcher strategy. The default `MemoryAdaptiveDispatcher` attempts to run multiple crawl tasks concurrently, adapting the level of concurrency based on available system memory. You can also provide custom dispatchers for more fine-grained control.
        *   **Using a shared `CrawlerRunConfig` vs. per-URL configurations:**
            *   **Shared `CrawlerRunConfig`:** If all URLs require similar processing (e.g., same extraction schema, same JS interactions), you can pass a single `CrawlerRunConfig` to `arun_many()`. The `url` property within this config will be ignored as URLs are taken from the input list.
            *   **Per-URL Configurations:** `arun_many()` is not designed to take a list of `CrawlerRunConfig` objects directly. If you need vastly different configurations per URL, you might iterate and call `arun()` or, for more sophisticated needs, look into customizing the dispatcher or using higher-level orchestration tools. A common pattern is to use a base `CrawlerRunConfig` and then clone/modify it slightly for each URL within a loop if minor variations are needed, though this would be outside the direct `arun_many` call. For `arun_many`, the primary variation is the URL itself.
        *   **Strategies for rate limiting and error handling within `arun_many()` (via Dispatcher configuration):**
            The dispatcher, particularly its `RateLimiter`, handles rate limiting (delays between requests, exponential backoff on 429/503 errors) and retries. If a URL fails after retries, `arun_many` will still proceed with other URLs and the failed result will indicate the error.
        *   **Code Example: Using `arun_many()` with a list of URLs and a shared `CrawlerRunConfig`**
            ```python
            import asyncio
            from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

            async def crawl_multiple_pages():
                urls_to_crawl = [
                    "http://quotes.toscrape.com/page/1/",
                    "http://quotes.toscrape.com/page/2/",
                    "http://quotes.toscrape.com/tag/humor/", # This might be a different structure
                ]

                # Shared config for all URLs in this batch
                shared_run_config = CrawlerRunConfig(
                    cache_mode=CacheMode.BYPASS,
                    word_count_threshold=50 # Only process pages with meaningful content
                )

                async with AsyncWebCrawler() as crawler:
                    # results will be a list of CrawlResult objects, in the same order as urls_to_crawl
                    results_container = await crawler.arun_many(urls=urls_to_crawl, config=shared_run_config)
                    
                    for result in results_container: # Iterate through CrawlResultContainer
                        if result.success:
                            print(f"Processed: {result.url}, Markdown Length: {len(result.markdown.raw_markdown)}")
                        else:
                            print(f"Failed: {result.url}, Error: {result.error_message}")

            if __name__ == "__main__":
                asyncio.run(crawl_multiple_pages())
            ```
        *   **Code Example: Using `arun_many()` with a custom dispatcher for advanced control**
            *(Note: Custom dispatchers are more advanced; the default `MemoryAdaptiveDispatcher` is usually sufficient.)*
            ```python
            import asyncio
            from crawl4ai import (
                AsyncWebCrawler, CrawlerRunConfig, CacheMode,
                MemoryAdaptiveDispatcher, RateLimiter, CrawlerMonitor, DisplayMode
            )

            async def crawl_with_custom_dispatcher():
                urls = [f"http://quotes.toscrape.com/page/{i}/" for i in range(1, 4)]
                
                # Custom rate limiter: 0.5-1.0s base delay, max 10s, 2 retries
                rate_limiter = RateLimiter(base_delay=(0.5, 1.0), max_delay=10.0, max_retries=2)
                
                # Custom monitor
                monitor = CrawlerMonitor(display_mode=DisplayMode.DETAILED)

                # Custom dispatcher with more conservative settings
                custom_dispatcher = MemoryAdaptiveDispatcher(
                    memory_threshold_percent=60.0, # Pause if memory > 60%
                    check_interval=2.0,           # Check memory every 2 seconds
                    max_session_permit=5,         # Max 5 concurrent crawls
                    rate_limiter=rate_limiter,
                    monitor=monitor
                )

                run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

                async with AsyncWebCrawler() as crawler:
                    results_container = await crawler.arun_many(
                        urls=urls,
                        config=run_config,
                        dispatcher=custom_dispatcher
                    )
                    for result in results_container:
                        if result.success:
                            print(f"URL: {result.url} - Success")
                        else:
                            print(f"URL: {result.url} - Failed: {result.error_message}")
                            
            if __name__ == "__main__":
                asyncio.run(crawl_with_custom_dispatcher())
            ```
*   D. **Understanding the Internal Processing Flow (`aprocess_html`):**
    *   **High-level explanation of what happens after HTML is fetched:**
        Once the `AsyncCrawlerStrategy` (e.g., `AsyncPlaywrightCrawlerStrategy`) successfully fetches the HTML content for a URL, `AsyncWebCrawler`'s `aprocess_html` method (or a similar internal handler) takes over. This method orchestrates several subsequent steps:
        1.  **Scraping Strategy Application:** The raw HTML is passed to a `ContentScrapingStrategy` (defaulting to `WebScrapingStrategy` or `LXMLExtractionStrategy` based on configuration). This strategy is responsible for:
            *   Cleaning the HTML (removing scripts, styles, comments).
            *   Applying `css_selector` or `target_elements` to scope the content if specified.
            *   Extracting links (`<a>` tags).
            *   Extracting media information (`<img>`, `<video>`, `<audio>` tags, and data tables).
            *   Extracting page metadata (title, meta description, OpenGraph tags, etc.).
            The output of this step is a `ScrapingResult` object containing `cleaned_html`, `media`, `links`, and `metadata`.
        2.  **Content Filtering (Optional):** If a `content_filter` (like `PruningContentFilter` or `LLMContentFilter`) is configured in the `MarkdownGenerationStrategy` (which is part of `CrawlerRunConfig`), the `cleaned_html` from the `ScrapingResult` is passed through this filter. This step aims to further refine the HTML to only include the most relevant content blocks. The output is "fit HTML".
        3.  **Markdown Generation:** The (potentially filtered) HTML is then passed to a `MarkdownGenerationStrategy` (defaulting to `DefaultMarkdownGenerator`). This strategy converts the HTML into Markdown. If content filtering occurred, it might generate both a `raw_markdown` (from the original cleaned HTML) and a `fit_markdown` (from the filtered HTML). It also handles citation generation for links. The output is a `MarkdownGenerationResult`.
        4.  **Structured Data Extraction (Optional):** If an `extraction_strategy` is provided in `CrawlerRunConfig`, it's applied *after* Markdown generation. The input to the extraction strategy depends on its `input_format` (e.g., "markdown", "html", "fit_markdown"). This strategy extracts structured data (like JSON) based on its specific logic (e.g., LLM prompting, CSS selectors via `JsonCssExtractionStrategy`).
        5.  **Result Aggregation:** Finally, all these pieces (`CrawlResult.html` (original), `cleaned_html`, `markdown` (the `MarkdownGenerationResult` object), `extracted_content`, `media`, `links`, `metadata`, etc.) are assembled into the final `CrawlResult` object.
    *   **How this internal flow influences the choice and impact of `CrawlerRunConfig` parameters:**
        *   `css_selector`: Applied early by the `ScrapingStrategy`. If set, *only* the HTML within this selector is processed for everything downstream (Markdown, extraction, etc.). This can significantly speed up processing and improve relevance.
        *   `target_elements`: Also applied by `ScrapingStrategy`, but its effect is primarily on what content is considered for *Markdown generation* and *structured data extraction* if the extraction strategy uses HTML as input. Links and media are still typically extracted from the whole (scoped by `css_selector` if present) page.
        *   `extraction_strategy`: This is one of the last steps. Its effectiveness depends on the quality and format of its input (e.g., clean Markdown or specific HTML structure).
        *   `markdown_generator`: Affects how `cleaned_html` (or `fit_html`) is converted to Markdown. The `content_filter` within it can drastically change the `fit_markdown`.
    *   **Decision Guide: When to rely on default processing vs. providing custom strategies:**
        *   **Default Processing:** Sufficient for many common use cases where you need the main content of a page converted to clean Markdown, along with standard metadata, links, and images.
        *   **Custom `ContentScrapingStrategy`:** Consider if you need highly specialized HTML cleaning, link/media extraction logic that differs significantly from the defaults, or if you're dealing with non-standard HTML structures.
        *   **Custom `MarkdownGenerationStrategy`:** If you need a different HTML-to-Markdown conversion engine, different citation styles, or very specific pre/post-processing of the Markdown.
        *   **Custom `ContentFilter` (within `MarkdownGenerationStrategy`):** If the default pruning/fitting logic isn't capturing the desired content accurately and you need a more sophisticated way (e.g., LLM-based, custom heuristics) to identify relevant sections.
        *   **Custom `ExtractionStrategy`:** Essential if you need to extract structured data beyond what `JsoupCssExtractionStrategy` can offer, or if you want to use LLMs for extraction based on natural language prompts or a defined schema.

# III. Configuring the Browser: `BrowserConfig` Deep Dive

`BrowserConfig` is your toolkit for defining the environment in which your web crawls will run. It dictates everything from the browser engine and its appearance (headless/headed) to network settings like proxies and crucial identity aspects like user agents and persistent storage. A well-configured `BrowserConfig` is often the key to successful and reliable crawling, especially on modern, dynamic websites.

*   A. **Purpose and Importance of `BrowserConfig`:**
    *   **Explaining how `BrowserConfig` defines the foundational browser environment:**
        Think of `BrowserConfig` as setting up your web browser application *before* you even type a URL. It's the global configuration for the `AsyncWebCrawler` instance. Changes here affect all crawl operations performed by that instance, unless specifically overridden by a `CrawlerRunConfig`.
    *   **Why getting this right is crucial for successful and stealthy crawling:**
        *   **Site Compatibility:** Some sites render or behave differently based on the browser engine or viewport size.
        *   **Stealth:** Websites employ various techniques to detect and block automated crawlers. A realistic `BrowserConfig` (e.g., common user agent, appropriate headers, possibly proxies) can significantly reduce the chances of detection.
        *   **Resource Management:** Settings like `headless` mode or `browser_mode` can impact how many resources your crawler consumes.
        *   **Session Persistence:** For sites requiring logins or multi-step interactions, `use_persistent_context` and `user_data_dir` are essential for maintaining state.

*   B. **Key `BrowserConfig` Decision Points and Workflows:**
    *   1.  **Choosing a `browser_type` (`chromium`, `firefox`, `webkit`):**
        *   **Trade-offs:**
            *   `chromium`: Most widely used, excellent DevTools support, generally good performance and compatibility. Often the default and a good starting point.
            *   `firefox`: Strong alternative, good privacy features, sometimes handles certain sites differently.
            *   `webkit`: Engine behind Safari. Useful for testing Safari-specific rendering or behavior.
        *   **When one might be preferred:**
            *   Start with `chromium`.
            *   If you encounter issues specific to Chromium-based browsers or need to emulate Firefox/Safari users, switch accordingly.
            *   Some sites might have better compatibility or less aggressive bot detection for one engine over others.
    *   2.  **Headless vs. Headed Mode (`headless`):**
        *   **Rationale for headless (`headless=True`, default):**
            *   No visible GUI, runs in the background.
            *   Essential for server environments or automated scripts.
            *   Generally consumes fewer resources than headed mode.
        *   **When to use headed mode (`headless=False`):**
            *   **Debugging:** Visually inspect what the crawler is doing, observe page rendering, and identify issues with selectors or interactions.
            *   **Complex Interactions:** For sites with very complex JavaScript, CAPTCHAs (though Crawl4ai doesn't solve these directly), or interactions that are hard to automate blindly.
            *   **Initial Setup:** When first developing a script for a new site, running in headed mode can be invaluable.
        *   **Impact on resource consumption and stealth:**
            *   Headed mode consumes more CPU and memory.
            *   Some rudimentary bot detection systems might flag headless browsers, though modern headless modes are much harder to detect than older versions.
    *   3.  **Browser Launch Modes (`browser_mode`):**
        *   **Understanding `"dedicated"` (default):**
            *   **Pros:** Each `AsyncWebCrawler` instance (or more accurately, its `BrowserManager`) launches and manages its own independent browser process. This provides strong isolation.
            *   **Cons:** Can be resource-intensive if you have many `AsyncWebCrawler` instances, as each spawns a new browser.
        *   **Understanding `"builtin"` (CDP based):**
            *   The `builtin` mode is designed to use a browser instance that Crawl4ai itself manages in the background, potentially shared across different `AsyncWebCrawler` instances if configured carefully (though this is an advanced use case). It connects via the Chrome DevTools Protocol (CDP).
            *   **When to use:** When you want Crawl4ai to manage the browser lifecycle but need to connect to it via CDP for more direct control or to share a browser instance.
            *   **Setup:** `use_managed_browser` is implicitly `True`. `cdp_url` is typically set by the internal `ManagedBrowser`.
        *   **Understanding `"cdp"` (external CDP):**
            *   **Scenarios:** You have an existing browser instance already running (e.g., a Chrome browser launched with `--remote-debugging-port=9222`) and you want Crawl4ai to connect to and control that instance.
            *   **Setup:** You must provide the `cdp_url` (e.g., `"ws://localhost:9222/devtools/browser/..."`).
        *   **Understanding `"docker"`:**
            *   **Benefits:** Runs the browser within a Docker container, providing excellent isolation, reproducibility, and easier dependency management, especially in CI/CD environments or when deploying to different systems. The core library handles launching a pre-configured Docker container (like `browserless/chrome`) and connecting to it.
            *   Refer to [Docker Deployment Guide](../../../basic/docker-deployment.md) for details.
        *   **Decision Guide: Selecting the appropriate `browser_mode`:**
            *   `"dedicated"`: Good default for simplicity and isolation when running a moderate number of crawlers.
            *   `"cdp"`: Use if you need to attach to an externally managed browser.
            *   `"docker"`: Excellent for reproducible environments, CI/CD, and avoiding local browser/driver issues.
            *   `"builtin"`: More of an internal mechanism, usually not directly set by users unless for advanced shared browser scenarios.
    *   4.  **User Agent Management (`user_agent`, `user_agent_mode`):**
        *   **Importance of a realistic User-Agent:** Many sites use the User-Agent string for basic bot detection or to serve different content. A missing or suspicious UA can lead to blocks or incorrect page versions.
        *   **Strategies for User-Agent rotation using `user_agent_mode="random"`:**
            Crawl4ai uses the `fake_useragent` library (via `ValidUAGenerator`) to pick from a list of common, valid user agents. This helps in making requests appear as if they are coming from different browsers/devices.
        *   **When to provide a specific `user_agent`:**
            If you need to emulate a very specific browser or device, or if a site requires a particular User-Agent to function correctly.
        *   **How `user_agent_generator_config` can be used for fine-tuning:**
            You can pass a dictionary to `user_agent_generator_config` to control aspects of the `ValidUAGenerator`, such as specifying browser types (`browsers=['chrome', 'edge']`), OS types, or min/max popularity.
        *   **Code Example: Setting a custom User-Agent**
            ```python
            from crawl4ai import BrowserConfig

            # To emulate a specific mobile browser
            custom_ua_config = BrowserConfig(
                user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
            )
            # Now use this custom_ua_config when initializing AsyncWebCrawler
            ```
        *   **Code Example: Using random User-Agent generation for Chrome and Edge on Windows**
            ```python
            from crawl4ai import BrowserConfig

            random_ua_config = BrowserConfig(
                user_agent_mode="random",
                user_agent_generator_config={
                    "os_names": ["windows"],
                    "browser_names": ["chrome", "edge"]
                }
            )
            # This will generate UAs like Chrome on Windows or Edge on Windows
            ```
    *   5.  **Proxy Configuration (`proxy_config` via `ProxyConfig` object):**
        *   **When and why to use proxies:**
            *   **IP Rotation:** To avoid IP-based blocking when making many requests to the same site.
            *   **Geo-targeting:** To access content as if you are browsing from a specific geographical location.
            *   **Anonymity/Privacy:** To mask your actual IP address.
        *   **Workflow: Setting up a single proxy:**
            1.  Create a `ProxyConfig` instance with `server`, and optionally `username` and `password`.
            2.  Assign this `ProxyConfig` instance to `BrowserConfig.proxy_config`.
        *   **Workflow: Integrating with proxy rotation strategies:**
            This is typically handled at the `CrawlerRunConfig` level for more dynamic rotation, but `BrowserConfig` can set a default proxy. For rotation, you'd use `CrawlerRunConfig(proxy_rotation_strategy=YourStrategy())`. See Section VII.A for details on rotation.
        *   **Code Example: Configuring a single HTTP/SOCKS proxy with authentication**
            ```python
            from crawl4ai import BrowserConfig, ProxyConfig

            # HTTP Proxy with authentication
            http_proxy_cfg = ProxyConfig(
                server="http://proxy.example.com:8080",
                username="proxy_user",
                password="proxy_password"
            )
            browser_with_http_proxy = BrowserConfig(proxy_config=http_proxy_cfg)

            # SOCKS5 Proxy (authentication handled in server string if supported by proxy)
            socks_proxy_cfg = ProxyConfig(
                server="socks5://another-proxy.example.com:1080"
            )
            browser_with_socks_proxy = BrowserConfig(proxy_config=socks_proxy_cfg)
            ```
        *   **Troubleshooting common proxy connection issues:**
            *   Incorrect server address or port.
            *   Wrong username/password.
            *   Proxy server is down or not reachable.
            *   Firewall blocking connection to the proxy.
            *   The website itself might be blocking the proxy's IP.
    *   6.  **Persistent Context and User Data (`use_persistent_context`, `user_data_dir`):**
        *   **Understanding the benefits of persistent contexts:**
            When `use_persistent_context=True` and a `user_data_dir` is specified, Playwright creates a persistent browser profile in that directory. This means cookies, localStorage, sessionStorage, and other browser data are saved between sessions. This is invaluable for:
            *   **Login Persistence:** Log in to a site once, and subsequent crawls using the same `user_data_dir` will likely remain logged in.
            *   **Maintaining Site Preferences:** If a site stores user preferences (e.g., dark mode, language) in local storage or cookies.
            *   **Reducing Repeated Setup:** Avoid re-doing consent banners or initial setup steps on every crawl.
        *   **Workflow: Creating and reusing a browser profile for login persistence:**
            1.  **First Run (Profile Creation & Login):**
                *   Set `headless=False` initially to manually perform the login.
                *   Specify a `user_data_dir` (e.g., `./my_browser_profile`).
                *   Set `use_persistent_context=True`.
                *   Run the crawler, navigate to the login page, and log in manually.
                *   Close the browser. The session data is now saved in `user_data_dir`.
            2.  **Subsequent Runs (Reusing Profile):**
                *   Use the *same* `user_data_dir`.
                *   Ensure `use_persistent_context=True`.
                *   You can now set `headless=True`.
                *   The crawler should start with the saved session, already logged in.
        *   **Best practices for managing `user_data_dir`:**
            *   Choose a unique, descriptive path for each distinct profile/session you want to maintain.
            *   Be aware of the disk space used, as profiles can grow.
            *   Ensure your script has write permissions to the specified directory.
        *   **Code Example: Setting up and using a persistent context for a login workflow**
            ```python
            import asyncio
            import os
            from pathlib import Path
            from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

            PROFILE_DIR = Path("./my_persistent_profile")

            async def initial_login_setup():
                # Create profile dir if it doesn't exist
                PROFILE_DIR.mkdir(parents=True, exist_ok=True)
                
                login_browser_cfg = BrowserConfig(
                    headless=False, # Run headed to manually log in
                    user_data_dir=str(PROFILE_DIR),
                    use_persistent_context=True
                )
                # For this setup run, we don't need a complex CrawlerRunConfig
                login_run_cfg = CrawlerRunConfig(url="https://example.com/login") # Replace with actual login URL

                print(f"Please log in manually in the browser window using profile: {PROFILE_DIR}")
                print("Close the browser window once logged in to save the session.")
                
                # We use a longer timeout to allow for manual login
                # And we don't need to process the result here, just establish the session
                async with AsyncWebCrawler(config=login_browser_cfg) as crawler:
                    # The page_timeout in BrowserConfig will apply here if not overridden in CrawlerRunConfig
                    # A simple goto is enough; the user handles the login in the browser
                    await crawler.arun(config=CrawlerRunConfig(url=login_run_cfg.url, page_timeout=300000)) # 5 min timeout

                print("Login session should be saved.")

            async def crawl_protected_page_with_session():
                if not PROFILE_DIR.exists() or not any(PROFILE_DIR.iterdir()):
                     print(f"Profile directory {PROFILE_DIR} is empty or does not exist. Run initial_login_setup() first.")
                     return

                persistent_browser_cfg = BrowserConfig(
                    headless=True, # Can now run headless
                    user_data_dir=str(PROFILE_DIR),
                    use_persistent_context=True
                )
                # Target a page that requires login
                protected_run_cfg = CrawlerRunConfig(url="https://example.com/dashboard") 

                async with AsyncWebCrawler(config=persistent_browser_cfg) as crawler:
                    result = await crawler.arun(config=protected_run_cfg)
                    if result.success and "Welcome User" in result.html: # Check for logged-in content
                        print(f"Successfully accessed protected page: {result.url}")
                        print(f"Content snippet: {result.markdown.raw_markdown[:200]}...")
                    elif result.status_code == 403 or "login" in result.url.lower():
                         print(f"Failed to access protected page. Still on login page or got 403. URL: {result.url}")
                    else:
                        print(f"Failed to crawl protected page: {result.error_message}, URL: {result.url}")
            
            async def main():
                # Run this once to log in and create the profile
                # await initial_login_setup() 
                
                # Then run this to use the saved session
                await crawl_protected_page_with_session()

            if __name__ == "__main__":
                # IMPORTANT: You'd typically run initial_login_setup() once manually,
                # then comment it out and run crawl_protected_page_with_session() for subsequent crawls.
                asyncio.run(main())
            ```
    *   7.  **Viewport Configuration (`viewport_width`, `viewport_height`, `viewport`):**
        *   **How viewport size can affect page rendering and element visibility:**
            Websites often use responsive design, meaning their layout and the visibility of certain elements change based on the viewport (browser window) size. If your target elements are only visible on larger screens (or mobile views), setting an appropriate viewport is crucial.
        *   **Strategies for choosing appropriate viewport dimensions:**
            *   Inspect the target website in a regular browser, resize the window, and see how content changes.
            *   Common desktop viewports: `1920x1080`, `1366x768`, `1280x720`.
            *   Common mobile viewports (emulated): `375x667` (iPhone 6/7/8), `414x896` (iPhone XR/11).
            *   If `viewport` dict is provided, it overrides `viewport_width` and `viewport_height`. E.g., `viewport={"width": 1920, "height": 1080}`.
    *   8.  **Advanced Browser Arguments (`extra_args`):**
        *   **When to use `extra_args`:**
            For passing command-line arguments directly to the browser executable. This is useful for enabling/disabling experimental features, or for very specific browser configurations not exposed by Playwright's high-level API.
        *   **Commonly used arguments and their effects (Chromium examples):**
            *   `--disable-gpu`: Can sometimes help in headless environments or reduce resource usage.
            *   `--no-sandbox`: Often required in Docker/CI environments, but use with caution as it reduces security. (Crawl4ai's `ManagedBrowser` often adds this automatically in headless Linux).
            *   `--lang=fr-FR`: Set browser UI language.
            *   `--disable-blink-features=AutomationControlled`: Attempt to make the browser appear less like an automated tool.
        *   **Caution:** Incorrect or incompatible arguments can prevent the browser from launching or cause instability. Refer to the browser's documentation (e.g., [Chromium Command Line Switches](https://peter.sh/experiments/chromium-command-line-switches/)) for a full list.
    *   9.  **Storage State Management (`storage_state`):**
        *   **What is storage state and why is it useful?**
            Storage state is a JSON object (or path to a JSON file) that captures cookies, `localStorage`, and `sessionStorage` from a browser context. It's a powerful way to persist and reuse login sessions or application states without relying on full user profiles (`user_data_dir`). It's generally more lightweight than managing entire profile directories.
        *   **Workflow: Capturing storage state:**
            1.  Launch a browser (often headed for manual interaction).
            2.  Perform the necessary actions (e.g., log in, accept cookies, set preferences).
            3.  Use Playwright's `context.storage_state(path="my_storage_state.json")` to save the state. Crawl4ai doesn't have a direct command to save state *during* a crawl, so this is typically done in a separate setup script.
                ```python
                # Example: Separate script to save storage state
                import asyncio
                from playwright.async_api import async_playwright

                async def save_state():
                    async with async_playwright() as p:
                        browser = await p.chromium.launch(headless=False)
                        context = await browser.new_context()
                        page = await context.new_page()
                        await page.goto("https://example.com/login") # Replace
                        # ... user manually logs in ...
                        input("Press Enter after logging in to save state...")
                        await context.storage_state(path="auth_state.json")
                        await browser.close()
                        print("Storage state saved to auth_state.json")

                # asyncio.run(save_state()) # Run this once
                ```
        *   **Workflow: Reusing storage state in Crawl4ai:**
            Pass the path to the saved JSON file (or the loaded JSON object itself) to `BrowserConfig(storage_state="auth_state.json")`.
        *   **Code Example: Illustrating how to load storage state in Crawl4ai**
            ```python
            import asyncio
            from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

            async def crawl_with_saved_state():
                # Assumes "auth_state.json" was created by a previous login session
                browser_cfg = BrowserConfig(
                    headless=True,
                    storage_state="auth_state.json" # Load cookies, localStorage, etc.
                )
                # Target a page that requires the saved session
                run_cfg = CrawlerRunConfig(url="https://example.com/dashboard")

                async with AsyncWebCrawler(config=browser_cfg) as crawler:
                    result = await crawler.arun(config=run_cfg)
                    if result.success and "Welcome User" in result.html: # Or other logged-in indicator
                        print("Successfully accessed dashboard using saved state.")
                    else:
                        print(f"Failed. Status: {result.status_code}, URL: {result.url}")
            
            if __name__ == "__main__":
                asyncio.run(crawl_with_saved_state())
            ```
*   C. **Best Practices for `BrowserConfig`:**
    *   **Start Simple:** Begin with minimal configuration (e.g., just `headless`) and add options as needed to address specific site requirements or improve stealth.
    *   **Prioritize Realistic Emulation:** For stealth, aim to make your crawler's browser fingerprint (User-Agent, viewport, headers, language settings) appear as close to a real user's browser as possible.
    *   **Manage User Data:** If using `user_data_dir` or `storage_state`, have a clear strategy for creating, updating, and cleaning up these profile/state files. Version control or separate directories for different tasks/sites can be helpful.
    *   **Test Proxy Configurations:** Thoroughly test proxies to ensure they are working and not blocked by target sites.
*   D. **Troubleshooting `BrowserConfig` Issues:**
    *   **Browser Not Launching:**
        *   Ensure Playwright and its browser drivers are correctly installed (`playwright install` or `crawl4ai-setup`).
        *   Check for conflicting `extra_args`.
        *   Permissions issues if `user_data_dir` is in a restricted location.
    *   **Proxy Connection Errors:**
        *   Verify proxy server address, port, username, and password.
        *   Test proxy connectivity outside of Crawl4ai.
        *   The proxy itself might be down or the target site might be blocking it.
    *   **User-Agent Related Detection:**
        *   If blocked, try different, common User-Agents or use `user_agent_mode="random"`.
        *   Ensure client hints (`sec-ch-ua`) sent by the browser (often managed by Playwright automatically based on UA) are consistent.
    *   **Persistent Context Not Working:**
        *   Double-check `user_data_dir` path and permissions.
        *   Ensure `use_persistent_context=True`.
        *   Some sites have aggressive session invalidation mechanisms.
        *   `storage_state` might be more reliable than `user_data_dir` for some sites as it's a more focused snapshot.

# IV. Customizing Each Crawl: `CrawlerRunConfig` In-Depth

While `BrowserConfig` sets the stage for the browser environment, `CrawlerRunConfig` provides the script for each act. It allows you to fine-tune how Crawl4ai interacts with and processes individual URLs or batches of URLs. This granular control is essential for adapting to diverse website structures and extracting the specific information you need.

*   A. **Purpose and Flexibility of `CrawlerRunConfig`:**
    *   **Explaining how `CrawlerRunConfig` allows fine-tuning behavior for each `arun()` or for individual URLs in `arun_many()`:**
        `CrawlerRunConfig` is passed directly to `crawler.arun()` or as a base configuration to `crawler.arun_many()`. This means you can have one `AsyncWebCrawler` instance (with its `BrowserConfig`) and then execute various types of crawls with different objectives by simply changing the `CrawlerRunConfig` for each call.
    *   **How it can override or augment settings from `BrowserConfig`:**
        Certain parameters in `CrawlerRunConfig` (like `user_agent` or `proxy_config`) can override those set in the `BrowserConfig` for that specific run. This is useful for targeted adjustments without reconfiguring the entire browser. For example, you might use a general pool of proxies in `BrowserConfig` but need a specific geo-targeted proxy for one particular `arun()` call.

*   B. **Key `CrawlerRunConfig` Decision Points and Workflows:**
    *   1.  **Caching Strategies (`cache_mode`):**
        *   **Understanding `CacheMode` options:**
            *   `CacheMode.BYPASS` (Default for `arun_many`, was `False` for `bypass_cache` in `arun`): Fetches fresh content every time. Use when up-to-date data is critical.
            *   `CacheMode.ENABLED`: Reads from cache if available and valid; writes to cache if not. This is the default for `arun()`. Good for development to speed up iterations.
            *   `CacheMode.READ_ONLY`: Only reads from the cache; never writes. Useful if you have a pre-populated cache and don't want to modify it.
            *   `CacheMode.WRITE_ONLY`: Always fetches fresh content and writes it to the cache, but never reads from the cache for serving the request. Useful for populating/updating a cache without using stale data.
            *   `CacheMode.DISABLED`: No caching involved at all. Similar to `BYPASS` but might have subtle differences in how the cache check is skipped entirely. `BYPASS` is generally clearer for "always fetch fresh".
        *   **Decision Guide: Choosing the right cache mode:**
            *   **Development/Testing:** `CacheMode.ENABLED` can save significant time by reusing previously fetched content. Use `CacheMode.BYPASS` when you explicitly need to test fresh fetching.
            *   **Production (Data Freshness Critical):** `CacheMode.BYPASS` or `CacheMode.WRITE_ONLY` (if you still want to update your cache).
            *   **Production (Speed/Cost Important, Some Staleness OK):** `CacheMode.ENABLED` with appropriate cache TTL settings (not directly in `CrawlerRunConfig`, but a general system consideration).
            *   **Using a Static Cache:** `CacheMode.READ_ONLY` if you have a pre-built dataset you want the crawler to use.
        *   **Impact on crawl speed and data freshness:** Caching improves speed for repeated requests to the same URL but can serve stale data if not managed. `BYPASS` ensures freshness but is slower for repeated crawls.
    *   2.  **Content Selection and Filtering (`css_selector`, `target_elements`, `excluded_tags`, `excluded_selector`):**
        These parameters help you narrow down the HTML content that Crawl4ai processes, leading to cleaner Markdown, more accurate extractions, and faster processing.
        *   **`css_selector` (string):**
            *   **Workflow:** If you only care about a specific section of a webpage (e.g., the main article content, a product description block), provide a CSS selector that uniquely identifies this container.
            *   **Impact:** Crawl4ai will take the HTML *within* this selected element and process only that for Markdown generation, link/media extraction *within that scope*, and structured data extraction.
            *   **When this is more efficient:** When the desired content is well-contained and you want to discard irrelevant surrounding HTML (headers, footers, sidebars, ads) *before* any further processing, saving computational resources.
        *   **`target_elements` (List[str]):**
            *   **Workflow:** If you want to generate Markdown or extract structured data from *multiple specific sections* of a page, provide a list of CSS selectors.
            *   **Impact:** Unlike `css_selector` which creates a single "scoped" document, `target_elements` tells the `MarkdownGenerationStrategy` and some `ExtractionStrategy` instances to focus their efforts only on the content within these elements. However, link and media extraction might still consider the whole page (or the scope defined by `css_selector` if also present).
            *   **Comparison with `css_selector`:** `css_selector` creates a new, smaller effective document for all downstream processing. `target_elements` guides specific downstream processes (like Markdown or schema-based extraction) to look only within those targets, but the "document" for other purposes (like general link extraction) might still be broader.
        *   **`excluded_tags` (List[str]) and `excluded_selector` (string):**
            *   **Strategies for removing unwanted content:** These are applied *after* `css_selector` (if used) but generally *before* `target_elements` are specifically focused on for Markdown/extraction.
            *   `excluded_tags`: Provide a list of HTML tag names (e.g., `["nav", "footer", "script", "style"]`) to remove entirely.
            *   `excluded_selector`: Provide a CSS selector for more complex elements to remove (e.g., `".ads-sidebar", "#cookie-banner"`).
            *   **Impact:** Cleaner input for Markdown generation and extraction, leading to more relevant and concise results.
        *   **Code Example: Using `css_selector` to focus on an article body**
            ```python
            # Assuming an article page where main content is in <article class="main-story">
            run_config_article = CrawlerRunConfig(
                url="https://example.com/news/my-article",
                css_selector="article.main-story" 
            )
            # result.cleaned_html and result.markdown will only contain content from within that article tag.
            ```
        *   **Code Example: Using `target_elements` to extract multiple sections for structured data**
            ```python
            # Assuming a product page with separate sections for description and reviews
            run_config_product = CrawlerRunConfig(
                url="https://example.com/product/123",
                target_elements=["#product-description", ".customer-reviews-section"],
                # Extraction strategy would then know to look within these.
            )
            ```
        *   **Code Example: Using `excluded_tags` to remove headers and footers**
            ```python
            run_config_clean = CrawlerRunConfig(
                url="https://example.com/some-page",
                excluded_tags=["header", "footer", "nav", "aside"]
            )
            ```
    *   3.  **JavaScript Execution and Interaction (`js_code`, `js_only`, `wait_for`, `wait_for_timeout`, `scan_full_page`, `scroll_delay`):**
        These parameters are vital for handling dynamic websites where content is loaded or modified by JavaScript.
        *   **Workflow: Executing simple JS snippets with `js_code` (str or List[str]):**
            *   Use `js_code` to run JavaScript after the initial page load. This can be for clicking buttons, expanding sections, scrolling, or any other client-side interaction needed to reveal content.
            *   If providing a list, scripts are executed sequentially.
        *   **Understanding `js_only` (bool):**
            *   Set to `True` for subsequent interactions on a page *within the same session* where you don't need a full page reload/navigation, but only want to execute new `js_code` and re-evaluate the page state.
            *   Requires `session_id` to be set to maintain the page context.
            *   Example: Clicking pagination buttons on a Single Page Application (SPA).
        *   **`wait_for` (Optional[str]):**
            *   **Strategies for effective waiting:**
                *   **CSS Selector:** `wait_for="css:.my-dynamic-content"` - Waits until an element matching the selector appears.
                *   **JavaScript Condition:** `wait_for="js:() => window.myAppDataLoaded === true"` - Waits until the JS expression evaluates to true.
                *   **Network Idle:** `wait_for="networkidle"` (Playwright specific event) - Waits until network activity subsides.
                *   **Timeout (number):** `wait_for=5000` - Simply waits for a fixed number of milliseconds.
            *   **Impact of `wait_for_timeout` (int, milliseconds):**
                If the `wait_for` condition isn't met within this timeout, the crawl proceeds or may fail depending on other settings. Defaults to `page_timeout`.
        *   **`scan_full_page` (bool) and `scroll_delay` (float, seconds):**
            *   **Handling lazy-loaded content and infinite scroll:**
                Set `scan_full_page=True` to make Crawl4ai attempt to scroll through the entire page, triggering lazy-loaded images or infinite scroll content.
            *   **Best practices for `scroll_delay`:**
                The `scroll_delay` is the pause between each scroll step. Adjust this based on how quickly the target site loads new content upon scrolling. Too short might miss content; too long will slow down the crawl. Start with `0.2` to `0.5` and adjust.
        *   **Code Example: Clicking a "Load More" button and waiting for new content**
            ```python
            # Assumes a button with id="load-more-btn" and new items get class ".new-item"
            run_config_load_more = CrawlerRunConfig(
                url="https://example.com/feed",
                js_code="document.getElementById('load-more-btn').click();",
                wait_for="css:.new-item" # Wait for at least one new item to appear
            )
            ```
        *   **Code Example: Scrolling to the bottom of a page to trigger lazy loading**
            ```python
            run_config_scroll = CrawlerRunConfig(
                url="https://example.com/gallery",
                scan_full_page=True,
                scroll_delay=0.75 # Give a bit more time for images to load
            )
            ```
    *   4.  **Media Capture (`screenshot`, `pdf`, `capture_mhtml`):**
        *   **When to capture screenshots (`screenshot=True`):**
            Useful for debugging (to see what the crawler "saw"), for visual verification of page state, or for archiving a visual snapshot. The result is a base64-encoded PNG string in `result.screenshot`.
        *   **When to generate PDFs (`pdf=True`):**
            Ideal for archiving web content in a stable, printable format. The result is raw PDF bytes in `result.pdf_data`.
        *   **Understanding MHTML (`capture_mhtml=True`):**
            MHTML (MIME HTML) saves a complete webpage (HTML, CSS, images, etc.) into a single `.mhtml` file. This is excellent for perfect archival as it can be opened in browsers like Chrome/Edge and displays the page as it was. Result is in `result.mhtml_data`.
        *   **Configuring related parameters like `screenshot_wait_for`:**
            If you need to wait for a specific element or condition *before* taking the screenshot (e.g., an animation to complete), you can use `screenshot_wait_for` with similar syntax to `wait_for`. (Note: check if `screenshot_wait_for` is a direct param or if you should use a general `wait_for` before setting `screenshot=True`). Typically, `wait_for` applies before the final HTML/media capture.
    *   5.  **Link and Domain Handling (`exclude_external_links`, `exclude_social_media_links`, `exclude_domains`):**
        *   **Strategies for controlling the scope of link extraction:**
            These flags help you refine the `result.links` collection.
            *   `exclude_external_links=True`: Only keeps links pointing to the same base domain.
            *   `exclude_social_media_links=True`: Removes links to common social media platforms (Facebook, Twitter, LinkedIn, etc.). It uses a predefined list which can be augmented via `exclude_social_media_domains`.
            *   `exclude_domains=["ads.example.com", "tracker.net"]`: Provide a list of specific domains whose links should be entirely ignored.
        *   **When and why to exclude certain types of links:**
            *   To focus on internal site structure.
            *   To avoid crawling irrelevant third-party sites.
            *   To clean up link data for analysis.
    *   6.  **Identity and Proxy Overrides (`user_agent`, `proxy_config` in `CrawlerRunConfig`):**
        *   **Scenarios where overriding `BrowserConfig` settings per-run is useful:**
            *   **A/B Testing:** Test how a site responds to different User-Agents.
            *   **Geo-Targeted Proxies:** Use a general proxy pool in `BrowserConfig` but switch to a specific country's proxy for a particular URL via `CrawlerRunConfig(proxy_config=...)`.
            *   **Site-Specific UAs:** If one site requires a very specific User-Agent not suitable for others.
    *   7.  **Session Management (`session_id`):**
        *   **Workflow: Maintaining a consistent browser state across multiple `arun()` calls:**
            1.  Choose a unique string for `session_id`.
            2.  On the first `arun()` call for this session, Crawl4ai creates a new browser page/tab associated with this ID.
            3.  On subsequent `arun()` calls with the *same* `session_id`, Crawl4ai reuses that existing page/tab, preserving its cookies, `localStorage`, and current URL (unless a new URL is provided in the `CrawlerRunConfig` for navigation).
            4.  Use `js_only=True` for these subsequent calls if you're just running JS on the *current* page of that session rather than navigating to a new URL.
            5.  When done with the session, explicitly kill it using `await crawler.browser_manager.kill_session(session_id)`.
        *   **How `session_id` interacts with `BrowserConfig.use_persistent_context`:**
            They are complementary. `use_persistent_context` (with `user_data_dir`) saves state *between crawler restarts*. `session_id` maintains state *within a single `AsyncWebCrawler` instance's lifetime* across multiple `arun` calls. You can use both: `use_persistent_context` to load an initial logged-in state, and then `session_id` to continue interacting with that state across several `arun` operations.
        *   **Best practices for managing session lifecycles:**
            *   Always kill sessions when they are no longer needed to free up browser resources.
            *   Use descriptive `session_id`s if managing multiple concurrent logical sessions.
        *   **Code Example: A multi-step form submission using the same `session_id`**
            ```python
            import asyncio
            from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

            async def multi_step_form():
                session_id = "form_submission_1"
                base_url = "https://example.com/multi-step-form" # Replace

                async with AsyncWebCrawler() as crawler:
                    # Step 1: Load first page of the form
                    config_step1 = CrawlerRunConfig(url=f"{base_url}?step=1", session_id=session_id)
                    result_step1 = await crawler.arun(config=config_step1)
                    print(f"Step 1 loaded: {result_step1.url}")

                    # Step 2: Fill first part, submit (JS might handle this, or a POST via strategy)
                    # For JS-driven submit on the same page:
                    js_fill_step1 = """
                        document.getElementById('field1').value = 'Value 1';
                        document.getElementById('nextButtonStep1').click();
                    """
                    config_step2_action = CrawlerRunConfig(
                        session_id=session_id, 
                        js_code=js_fill_step1,
                        js_only=True, # We are on the same page context
                        wait_for="css:#formStep2Indicator" # Wait for step 2 to appear
                    )
                    result_step2_load = await crawler.arun(config=config_step2_action) 
                    print(f"Step 2 loaded after JS action. Current URL: {result_step2_load.url}")
                    
                    # ... continue for other steps ...

                    # Finally, kill the session
                    await crawler.browser_manager.kill_session(session_id)
                    print(f"Session {session_id} killed.")

            if __name__ == "__main__":
                asyncio.run(multi_step_form())
            ```
    *   8.  **Robots.txt Handling (`check_robots_txt`):**
        *   **Understanding the importance of respecting `robots.txt`:**
            `robots.txt` is a standard used by websites to communicate with web crawlers about which parts of their site should not be accessed. Respecting these rules is crucial for ethical and legal crawling.
        *   **How Crawl4ai handles it:**
            If `check_robots_txt=True` (default is `False` in `CrawlerRunConfig` for flexibility, but often good practice to enable), Crawl4ai will:
            1.  Attempt to fetch `/robots.txt` for the domain.
            2.  Parse the rules relevant to its User-Agent (or a generic one if specific rules are absent).
            3.  If the target URL is disallowed, the crawl will be skipped, and the `CrawlResult` will indicate this (e.g., status code 403, error message).
            *   **When you might override it (with caution):**
                Disabling `check_robots_txt` (`False`) should only be done if you have explicit permission to crawl restricted areas or if you are certain `robots.txt` is misconfigured and blocking legitimate access. Always prioritize ethical crawling.

*   C. **Combining `CrawlerRunConfig` with `BrowserConfig`:**
    *   **Understanding precedence:** Settings in `CrawlerRunConfig` generally take precedence over those in `BrowserConfig` for the duration of that specific `arun()` call. For example, if `BrowserConfig` defines a default User-Agent, but `CrawlerRunConfig` also specifies a `user_agent`, the latter will be used for that run.
    *   **Examples of synergistic configurations:**
        *   `BrowserConfig` with a `user_data_dir` to load a logged-in profile.
        *   `CrawlerRunConfig` with `js_code` to navigate within the logged-in application and `screenshot=True` to capture the state of a specific dashboard.
        *   `BrowserConfig` with a default set of `extra_args` for browser hardening.
        *   `CrawlerRunConfig` with a specific `css_selector` to target only the main content of articles from various sites, using the same hardened browser setup.

*   D. **Best Practices for `CrawlerRunConfig`:**
    *   **Be Specific:** Tailor your `CrawlerRunConfig` to the specific requirements of the URL(s) you are crawling and the data you intend to extract.
    *   **Iterate and Test:** Start with a minimal `CrawlerRunConfig`. Test it. Add complexity (JS interactions, wait conditions, extraction strategies) step-by-step, testing at each stage.
    *   **Use `wait_for` Judiciously:** Effective `wait_for` conditions are key for dynamic content. Avoid overly long fixed delays; prefer waiting for specific elements or JS flags.
    *   **Manage `session_id` State:** If using `session_id`, ensure you have a clear understanding of the session's lifecycle and kill sessions when they are no longer needed.
    *   **Scope Content Wisely:** Use `css_selector` or `target_elements` to reduce processing overhead and improve the relevance of extracted data.

*   E. **Troubleshooting `CrawlerRunConfig` Issues:**
    *   **Content Not Being Extracted as Expected:**
        *   Verify your `css_selector` or `target_elements` are correct and unique using browser developer tools.
        *   Content might be loaded dynamically *after* your `js_code` or `wait_for` completes. Add more specific waits or delays.
        *   The structure of the page might be different from what your selectors expect.
    *   **Timeouts or Hangs During JS Execution or Waiting:**
        *   `js_code` might have an error or an infinite loop. Test it in the browser console.
        *   `wait_for` condition might never be met. Check the selector/JS expression and increase `wait_for_timeout` if necessary, or make the condition more robust.
        *   The page itself might be unresponsive or very slow.
    *   **Incorrect Media Capture:**
        *   Screenshots/PDFs might be taken too early. Use `wait_for` effectively before enabling capture.
        *   Content might be outside the configured viewport.
    *   **`CacheMode` Not Behaving as Expected:**
        *   Ensure you understand the differences between `ENABLED`, `BYPASS`, `READ_ONLY`, and `WRITE_ONLY`.
        *   Cache keys are based on the URL. If URL parameters change, it's a new cache entry.

# V. Interpreting Crawl Outputs: `CrawlResult` and Data Models

After a crawl operation completes, `AsyncWebCrawler.arun()` (or `arun_many()`) returns a `CrawlResult` object (or a `CrawlResultContainer` which wraps a list of `CrawlResult` objects). This object is a rich container holding all the data and metadata collected during the crawl.

*   A. **Understanding the `CrawlResult` Object:**
    *   **Navigating the key attributes:**
        *   `url` (str): The final URL crawled, after any redirects.
        *   `html` (str): The raw HTML content of the page as fetched.
        *   `cleaned_html` (Optional[str]): HTML after basic cleaning (scripts, styles removed) and potential scoping by `css_selector`. This is often the input for Markdown generation.
        *   `markdown` (Optional[MarkdownGenerationResult]): An object containing various Markdown outputs. See B.
        *   `extracted_content` (Optional[str]): JSON string (or other text) if an `ExtractionStrategy` was used and successfully extracted data.
        *   `media` (Dict[str, List[Dict]]): A dictionary containing lists of media items, keyed by type (e.g., "images", "videos", "audios", "tables").
        *   `links` (Dict[str, List[Dict]]): A dictionary containing lists of links, keyed by "internal" and "external".
        *   `metadata` (Optional[dict]): Page metadata (title, meta description, OpenGraph tags, etc.).
        *   `status_code` (Optional[int]): The HTTP status code of the final response.
        *   `success` (bool): `True` if the crawl and initial processing were successful.
        *   `error_message` (Optional[str]): Description of an error if `success` is `False`.
        *   `session_id` (Optional[str]): The session ID used for this crawl, if any.
        *   `response_headers` (Optional[dict]): HTTP response headers.
        *   `ssl_certificate` (Optional[SSLCertificate]): SSL certificate information if requested.
        *   `mhtml_data` (Optional[str]): MHTML snapshot if `capture_mhtml=True`.
        *   `screenshot` (Optional[str]): Base64 encoded screenshot if `screenshot=True`.
        *   `pdf_data` (Optional[bytes]): Raw PDF bytes if `pdf=True`.
        *   `downloaded_files` (Optional[List[str]]): List of paths to downloaded files if downloads occurred.
        *   `js_execution_result` (Optional[Dict[str, Any]]): Result of the last executed JS snippet if it returned a value.
    *   **How different `CrawlerRunConfig` settings populate these fields:**
        *   `css_selector`, `target_elements`, `excluded_tags`: Affect `cleaned_html` and subsequently `markdown`.
        *   `extraction_strategy`: Populates `extracted_content`.
        *   `screenshot`, `pdf`, `capture_mhtml`: Populate their respective fields.
        *   `fetch_ssl_certificate`: Populates `ssl_certificate`.

*   B. **Working with `MarkdownGenerationResult`:**
    The `result.markdown` attribute (if Markdown generation occurred) is an instance of `MarkdownGenerationResult`.
    *   `raw_markdown` (str): Markdown generated from `cleaned_html` *before* any `ContentFilter` (like `PruningContentFilter`) is applied.
    *   `markdown_with_citations` (str): `raw_markdown` but with inline links converted to citation style (e.g., `[text](1)`) if `citations=True` in the generator.
    *   `references_markdown` (str): The list of numbered references corresponding to the citations.
    *   `fit_markdown` (Optional[str]): Markdown generated from HTML that has been processed by a `ContentFilter`. This usually represents the most "relevant" content. If no filter was used, this might be empty or same as `raw_markdown`.
    *   `fit_html` (Optional[str]): The HTML content *after* a `ContentFilter` has been applied, which was then used to generate `fit_markdown`.
    *   **When to use `fit_markdown`:** This is often your desired output if you've configured a content filter to isolate the core article or relevant sections, as it's the most distilled version.
    *   **Code Example:**
        ```python
        if result.success and result.markdown:
            print("--- Raw Markdown (Snippet) ---")
            print(result.markdown.raw_markdown[:300])
            
            if result.markdown.fit_markdown:
                print("\n--- Fit Markdown (More Relevant Content - Snippet) ---")
                print(result.markdown.fit_markdown[:300])
            
            if result.markdown.references_markdown:
                print("\n--- References ---")
                print(result.markdown.references_markdown)
        ```

*   C. **Leveraging `ScrapingResult` (from `ContentScrapingStrategy`):**
    The `ScrapingResult` is an intermediate object produced by the `ContentScrapingStrategy` (e.g., `WebScrapingStrategy`). Its fields are then used to populate the main `CrawlResult`.
    *   `cleaned_html`: This becomes `CrawlResult.cleaned_html`.
    *   `media`: This dictionary populates `CrawlResult.media`.
    *   `links`: This dictionary populates `CrawlResult.links`.
    *   `metadata`: This populates `CrawlResult.metadata`.
    Understanding this helps if you're creating a custom scraping strategy, as you'll need to return a `ScrapingResult` object.

*   D. **Understanding `MediaItem` and `Link` Structures:**
    *   **Common fields in `MediaItem` (for images, videos, audio):**
        *   `src` (str): The source URL of the media.
        *   `alt` (Optional[str]): Alt text (primarily for images).
        *   `type` (str): "image", "video", or "audio".
        *   `score` (Optional[int]): Heuristic score indicating relevance (if scoring is applied).
        *   `desc` (Optional[str]): Text from nearby elements, potentially describing the image.
        *   `group_id` (Optional[int]): If part of a group of related images (e.g., variants from `srcset`).
        *   `format` (Optional[str]): Detected format (e.g., "jpeg", "png").
        *   `width` (Optional[int]): Detected width.
        *   `data` (Optional[str]): Base64 encoded data if it's a data URI.
    *   **Common fields in `Link`:**
        *   `href` (str): The target URL of the link.
        *   `text` (Optional[str]): The anchor text of the link.
        *   `title` (Optional[str]): The `title` attribute of the link.
        *   `base_domain` (Optional[str]): The base domain of the `href`.
    *   **Practical examples of iterating and processing these collections:**
        ```python
        if result.success:
            # Example: Filter high-score images or JPEGs
            high_quality_images = [
                img for img in result.media.get("images", [])
                if (img.get("score", 0) > 3 and img.get("format") == "jpeg") or \
                   (img.get("score", 0) > 3 and img.get("src", "").lower().endswith(".jpg"))
            ]
            print(f"\nFound {len(high_quality_images)} high-quality JPEGs:")
            for img in high_quality_images[:2]: # Show first 2
                print(f"  - Src: {img.get('src')}, Alt: {img.get('alt', 'N/A')}")

            # Example: Extract all PDF links
            pdf_links = [
                link.get("href") for link in result.links.get("internal", []) + result.links.get("external", [])
                if link.get("href", "").lower().endswith(".pdf")
            ]
            if pdf_links:
                print(f"\nFound {len(pdf_links)} PDF links:")
                for pdf_url in pdf_links[:2]:
                    print(f"  - {pdf_url}")
        ```

*   E. **Making Sense of `AsyncCrawlResponse` (from `AsyncCrawlerStrategy`):**
    This object is returned by the `AsyncCrawlerStrategy.crawl()` method and is more low-level than `CrawlResult`. It typically contains:
    *   `html` (str): The raw HTML.
    *   `response_headers` (Dict): HTTP headers from the server's response.
    *   `status_code` (int): HTTP status code.
    *   `screenshot` (Optional[str]): Base64 screenshot data.
    *   `pdf_data` (Optional[bytes]): PDF data.
    *   `mhtml_data` (Optional[str]): MHTML data.
    *   And potentially other strategy-specific fields.
    The `AsyncWebCrawler` uses the information from `AsyncCrawlResponse` to construct the more comprehensive `CrawlResult`. You'd typically interact with `AsyncCrawlResponse` only if you are developing a custom `AsyncCrawlerStrategy`.

*   F. **Tracking Costs with `TokenUsage` (for LLM-based strategies):**
    *   When using strategies that involve Large Language Models (e.g., `LLMExtractionStrategy`, `LLMContentFilter`), the `TokenUsage` object provides a breakdown of token consumption.
    *   **Where to find it:**
        *   For `LLMExtractionStrategy`: The `run()` method of the strategy itself often returns a list of results, and each result or an aggregated summary might contain `TokenUsage`. The exact location can depend on the specific LLM provider integration within the strategy. It's often part of the metadata returned by the LLM API call.
        *   For `LLMContentFilter`: Similarly, the filter might expose token usage details.
        *   It's not directly a field in `CrawlResult` unless a specific extraction strategy adds it to `extracted_content` or `metadata`. You'd typically access it from the strategy's direct output or by inspecting logs if the strategy logs token usage.
    *   **Importance:** Crucial for monitoring and managing costs associated with LLM API calls. `TokenUsage` usually includes `prompt_tokens`, `completion_tokens`, and `total_tokens`.

*   G. **Inspecting `SSLCertificate` Information:**
    *   **How to access:** If `CrawlerRunConfig(fetch_ssl_certificate=True)` is set, the `CrawlResult.ssl_certificate` attribute will be an instance of `SSLCertificate`.
    *   **Use cases:**
        *   **Security Audits:** Programmatically check certificate issuers, validity periods, or signature algorithms.
        *   **Verifying Certificate Validity:** Ensure the certificate is not expired and is issued by a trusted authority.
        *   **Data Collection:** Gather data on SSL/TLS deployment across a set of websites.
    *   The `SSLCertificate` object (as seen in `crawl4ai/ssl_certificate.py`) provides properties like `.issuer`, `.subject`, `.valid_from`, `.valid_until`, `.fingerprint`, and methods like `.to_pem()`, `.to_der()`, `.to_json()`.
        ```python
        if result.success and result.ssl_certificate:
            cert = result.ssl_certificate
            print(f"\nSSL Certificate for {result.url}:")
            print(f"  Issuer CN: {cert.issuer.get('CN', 'N/A')}")
            print(f"  Subject CN: {cert.subject.get('CN', 'N/A')}")
            print(f"  Valid until: {cert.valid_until}") # Already a decoded string
            # cert.to_pem(path="certificate.pem") # Example of saving
        ```

# VI. Core Crawler Strategies: `AsyncPlaywrightCrawlerStrategy` and `AsyncHTTPCrawlerStrategy`

Crawl4ai's power comes from its flexible strategy pattern. At the core of fetching web content are the "Crawler Strategies." You primarily interact with these through `AsyncWebCrawler`, but understanding their differences and capabilities helps in choosing the right tool for the job and customizing behavior.

*   A. **Choosing the Right Strategy:**
    The `AsyncWebCrawler` is initialized with a crawler strategy. If none is provided, `AsyncPlaywrightCrawlerStrategy` is the default.
    *   **`AsyncPlaywrightCrawlerStrategy` (Default):**
        *   **Pros:**
            *   **Full JavaScript Rendering:** Executes JavaScript on the page, just like a real browser. Essential for modern SPAs (Single Page Applications) and sites that load content dynamically.
            *   **Complex Interactions:** Can handle clicks, form submissions, scrolling, and other user-like interactions via `js_code` or hooks.
            *   **Robust:** Benefits from Playwright's mature browser automation capabilities.
            *   **Media Capture:** Natively supports screenshots, PDFs, and MHTML.
        *   **Cons:**
            *   **Higher Resource Usage:** Launching and managing a full browser instance is more memory and CPU intensive.
            *   **Slower than HTTP:** Browser rendering and JS execution take time.
        *   **When it's the best choice:**
            *   Most modern websites that rely heavily on JavaScript.
            *   When you need to interact with the page (e.g., click buttons, fill forms) to access content.
            *   When you need to capture screenshots or PDFs accurately reflecting the rendered page.
    *   **`AsyncHTTPCrawlerStrategy`:**
        *   **Pros:**
            *   **Lightweight & Very Fast:** Makes direct HTTP requests using libraries like `requests` or `httpx`. Skips browser rendering entirely.
            *   **Low Resource Usage:** Minimal memory and CPU footprint.
            *   Good for high-volume, simple fetches.
        *   **Cons:**
            *   **No JavaScript Execution:** Only retrieves the initial HTML response from the server. Dynamic content loaded by client-side JS will be missing.
            *   **Limited Interaction:** Cannot perform clicks, scrolls, or form submissions in the browser context.
            *   Screenshots/PDFs are not applicable as there's no rendering.
        *   **When it's a suitable alternative:**
            *   Fetching data from APIs that return JSON or XML.
            *   Crawling purely static HTML websites where all content is present in the initial server response.
            *   Parsing sitemaps or `robots.txt` files.
            *   When speed and resource efficiency are paramount, and JS rendering is not required.
    *   **Decision Guide: Playwright vs. HTTP:**
        1.  **Does the target content require JavaScript to load/render?**
            *   **Yes:** Use `AsyncPlaywrightCrawlerStrategy`.
            *   **No:** `AsyncHTTPCrawlerStrategy` is a good candidate.
        2.  **Do you need to interact with the page (clicks, forms, scrolls)?**
            *   **Yes:** Use `AsyncPlaywrightCrawlerStrategy`.
            *   **No:** `AsyncHTTPCrawlerStrategy` might be sufficient.
        3.  **Are you fetching from an API endpoint that returns structured data (JSON/XML)?**
            *   **Yes:** `AsyncHTTPCrawlerStrategy` is ideal.
        4.  **Is performance (requests/second) and low resource use a top priority for simple HTML pages?**
            *   **Yes:** `AsyncHTTPCrawlerStrategy`.
        5.  **Do you need screenshots or PDFs of the rendered page?**
            *   **Yes:** `AsyncPlaywrightCrawlerStrategy`.
        
        *If unsure, start with `AsyncPlaywrightCrawlerStrategy` as it's more versatile, then consider `AsyncHTTPCrawlerStrategy` if you find JS rendering is unnecessary for your target.*

*   B. **Deep Dive into `AsyncPlaywrightCrawlerStrategy` Hooks:**
    Hooks are a powerful feature of `AsyncPlaywrightCrawlerStrategy` that allow you to inject custom asynchronous code at various stages of the crawling lifecycle for a specific page context.
    *   1.  **Purpose and Power of Hooks:**
        *   **Customization:** Tailor browser behavior beyond what standard configuration options offer.
        *   **Interaction:** Perform complex page interactions that are difficult to express with simple `js_code`.
        *   **State Management:** Set up or clean up page/context state.
        *   **Conditional Logic:** Implement dynamic logic based on page content or state during the crawl.
    *   2.  **Available Hooks and Their Triggers:**
        *(Refer to `async_crawler_strategy.py` for exact signatures; parameters often include `page`, `context`, `url`, `response`, `config`, `shared_data` etc.)*
        *   `on_browser_created(browser, **kwargs)`: Called once after the Playwright `Browser` object is created but before any `BrowserContext` is made. Useful for global browser setup, but limited as there's no page yet.
        *   `on_page_context_created(page: Page, context: BrowserContext, **kwargs)`: Called right after a new `BrowserContext` and its initial `Page` are created. This is the **most common and recommended hook for setup tasks** like authentication, setting cookies, or configuring routes, as you have both `page` and `context` available.
        *   `before_goto(page: Page, context: BrowserContext, url: str, **kwargs)`: Called just before `page.goto(url)` is executed. Useful for setting request interception or last-minute header modifications specific to this navigation.
        *   `after_goto(page: Page, context: BrowserContext, url: str, response, **kwargs)`: Called after `page.goto(url)` completes successfully (i.e., navigation didn't fail). `response` is the Playwright `Response` object. Good for initial checks post-navigation.
        *   `on_user_agent_updated(page: Page, context: BrowserContext, user_agent: str, **kwargs)`: Triggered if the user agent is changed dynamically during the crawl lifecycle (less common for standard crawls).
        *   `on_execution_started(page: Page, context: BrowserContext, **kwargs)`: Called just before any `js_code` specified in `CrawlerRunConfig` is executed on the page.
        *   `before_retrieve_html(page: Page, context: BrowserContext, **kwargs)`: Called right before the final HTML content of the page is snapshot. This is your last chance to perform interactions that might alter the DOM before it's captured.
        *   `before_return_html(page: Page, context: BrowserContext, html: str, **kwargs)`: Called just before the `AsyncCrawlResponse` is constructed. You have access to the `html` string that's about to be returned. Useful for final HTML modifications if absolutely necessary, though generally, it's better to do this in a `ContentScrapingStrategy`.
    *   3.  **Common Workflows Using Hooks:**
        *   **Workflow: Implementing a login sequence using `on_page_context_created`:**
            This is the ideal hook for logins as it runs once per new context (or session).
            *   **Code Example:**
                ```python
                import asyncio
                from playwright.async_api import Page, BrowserContext
                from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, AsyncPlaywrightCrawlerStrategy

                async def login_hook(page: Page, context: BrowserContext, shared_data: dict, **kwargs):
                    print("[HOOK login_hook] Navigating to login page...")
                    await page.goto(shared_data["login_url"])
                    await page.fill("input[name='username']", shared_data["username"])
                    await page.fill("input[name='password']", shared_data["password"])
                    await page.click("button[type='submit']")
                    # Wait for a post-login element to ensure success
                    await page.wait_for_selector("#dashboard-welcome", timeout=10000) 
                    print("[HOOK login_hook] Login successful!")
                    # Cookies are now set in the context and will be used for subsequent requests

                async def main():
                    # Create a BrowserConfig that will use our hook
                    strategy = AsyncPlaywrightCrawlerStrategy()
                    strategy.set_hook("on_page_context_created", login_hook)
                    
                    # Use a persistent context to potentially save the login cookies across crawler restarts
                    # PROFILE_DIR_HOOK = Path("./hook_login_profile")
                    # PROFILE_DIR_HOOK.mkdir(parents=True, exist_ok=True)

                    browser_cfg = BrowserConfig(
                        headless=False, # Easier to debug login
                        # user_data_dir=str(PROFILE_DIR_HOOK), # Optional: for persistence
                        # use_persistent_context=True      # Optional: for persistence
                    )
                    
                    # Pass login credentials and URL via shared_data
                    run_cfg = CrawlerRunConfig(
                        url="https://example.com/protected-page", # Target page after login
                        shared_data={
                            "login_url": "https://example.com/login", # Replace
                            "username": "testuser",
                            "password": "testpassword"
                        }
                    )

                    async with AsyncWebCrawler(config=browser_cfg, crawler_strategy=strategy) as crawler:
                        result = await crawler.arun(config=run_cfg)
                        if result.success:
                            print(f"Crawled protected page content: {result.markdown.raw_markdown[:200]}...")
                        else:
                            print(f"Failed: {result.error_message}")
                
                if __name__ == "__main__":
                    asyncio.run(main())
                ```
        *   **Workflow: Setting custom cookies or headers for all requests in a context via `on_page_context_created`:**
            ```python
            async def set_custom_headers_hook(page: Page, context: BrowserContext, **kwargs):
                await context.set_extra_http_headers({"X-Custom-Auth": "mysecrettoken"})
                print("[HOOK set_custom_headers_hook] Custom headers set.")
            
            # strategy.set_hook("on_page_context_created", set_custom_headers_hook)
            ```
        *   **Workflow: Blocking specific resource types (e.g., fonts, images) using `page.route()` in `on_page_context_created`:**
            This can speed up page loads if you don't need these resources.
            ```python
            async def block_resources_hook(page: Page, context: BrowserContext, **kwargs):
                await context.route("**/*.{png,jpg,jpeg,woff,woff2}", lambda route: route.abort())
                print("[HOOK block_resources_hook] Aborting image and font requests.")

            # strategy.set_hook("on_page_context_created", block_resources_hook)
            ```
        *   **Workflow: Performing complex page interactions before HTML retrieval using `before_retrieve_html`:**
            Useful if content is revealed only after a sequence of actions not easily done with simple `js_code`.
            ```python
            async def reveal_content_hook(page: Page, context: BrowserContext, **kwargs):
                print("[HOOK reveal_content_hook] Clicking multiple expanders...")
                expanders = await page.query_selector_all(".accordion-header")
                for expander in expanders:
                    await expander.click()
                    await page.wait_for_timeout(100) # Small delay for content to render
            
            # strategy.set_hook("before_retrieve_html", reveal_content_hook)
            ```
    *   4.  **Best Practices for Writing Hook Functions:**
        *   **Keep them Focused:** Each hook should perform a specific, well-defined task.
        *   **Handle Errors Gracefully:** Use `try-except` blocks within hooks to catch potential errors (e.g., element not found, navigation failure) and prevent them from crashing the entire crawl. Log errors appropriately.
        *   **Async/Await:** Hooks are `async` functions. Ensure you `await` any Playwright calls or other asynchronous operations within them.
        *   **Idempotency (where applicable):** If a hook might be called multiple times for the same context (less common but possible in complex scenarios), design it to be idempotent if its actions are stateful (e.g., don't try to set the same route handler multiple times without removing the old one).
    *   5.  **Passing Shared Data to Hooks (`CrawlerRunConfig.shared_data`):**
        *   The `shared_data` dictionary in `CrawlerRunConfig` is passed as keyword arguments to your hook functions.
        *   This is the recommended way to pass dynamic information (like credentials, API keys, or state from previous steps) into your hooks without hardcoding them or relying on global variables.
        *   **Code Example:** (Shown in the login hook example above where `shared_data` passes `login_url`, `username`, `password`).

*   C. **Using `AsyncHTTPCrawlerStrategy`:**
    When you don't need JavaScript rendering, `AsyncHTTPCrawlerStrategy` offers a much faster and lighter alternative.
    *   **Simpler Configuration:** Many `BrowserConfig` options (like `headless`, `viewport`) are irrelevant. The core HTTP request details are controlled via `HTTPCrawlerConfig` passed within `CrawlerRunConfig`.
    *   **When to use `HTTPCrawlerConfig` for `method`, `headers`, `data`, `json`:**
        *   `method`: "GET" (default), "POST", "PUT", etc.
        *   `headers`: Custom HTTP headers for the request.
        *   `data`: For form-encoded POST data (`application/x-www-form-urlencoded`).
        *   `json`: For JSON POST data (`application/json`).
    *   **Code Example: Fetching JSON from an API using `AsyncHTTPCrawlerStrategy` and `HTTPCrawlerConfig`**
        ```python
        import asyncio
        import json
        from crawl4ai import (
            AsyncWebCrawler, CrawlerRunConfig, AsyncHTTPCrawlerStrategy, HTTPCrawlerConfig
        )

        async def fetch_api_data():
            # HTTPCrawlerConfig for a POST request with JSON payload
            http_cfg = HTTPCrawlerConfig(
                method="POST",
                headers={"Authorization": "Bearer myapitoken"},
                json_data={"query": "fetch_user_data", "user_id": 123}
            )

            # CrawlerRunConfig specific to this API call
            run_cfg = CrawlerRunConfig(
                url="https://api.example.com/data",
                http_crawler_config=http_cfg # Pass the HTTP-specific config
            )
            
            # Initialize crawler with AsyncHTTPCrawlerStrategy
            http_strategy = AsyncHTTPCrawlerStrategy()
            async with AsyncWebCrawler(crawler_strategy=http_strategy) as crawler:
                result = await crawler.arun(config=run_cfg)
                if result.success and result.status_code == 200:
                    try:
                        api_data = json.loads(result.html) # The 'html' field contains the raw response body
                        print("API Data:", api_data)
                    except json.JSONDecodeError:
                        print("Failed to parse JSON response:", result.html)
                else:
                    print(f"API call failed. Status: {result.status_code}, Error: {result.error_message}")
                    print(f"Response body: {result.html}")


        if __name__ == "__main__":
            asyncio.run(fetch_api_data())
        ```

# VII. Advanced Core Concepts and Workflows

Beyond basic crawling, Crawl4ai's core offers mechanisms for more sophisticated scenarios, including robust proxy management, fine-grained browser control, and configuration persistence.

*   A. **Proxy Management and Rotation:**
    While single proxies can be set in `BrowserConfig` or `CrawlerRunConfig`, robust applications often require rotating through a list of proxies to avoid IP bans and access geo-restricted content.
    *   1.  **Using `ProxyConfig` for Single Proxies:**
        (Recap from `BrowserConfig` and `CrawlerRunConfig` sections for completeness)
        -   **Purpose:** To route a specific `AsyncWebCrawler` instance or a particular `arun()` call through a designated proxy server.
        -   **How:**
            ```python
            from crawl4ai import BrowserConfig, CrawlerRunConfig, ProxyConfig
            
            # For all crawls by a crawler instance
            browser_cfg_proxy = BrowserConfig(
                proxy_config=ProxyConfig(server="http://user:pass@proxy1.example.com:8000")
            )
            # crawler = AsyncWebCrawler(config=browser_cfg_proxy)

            # For a single arun() call
            run_cfg_proxy = CrawlerRunConfig(
                url="https://example.com",
                proxy_config=ProxyConfig(server="http://user:pass@proxy2.example.com:8001")
            )
            # result = await crawler.arun(config=run_cfg_proxy)
            ```
        -   **Why:** Useful for testing with a specific proxy or when you have a stable, dedicated proxy for certain tasks.
    *   2.  **Implementing Proxy Rotation with `ProxyRotationStrategy`:**
        *   **Understanding the need for proxy rotation:**
            Websites often track and block IPs that make too many requests. Rotating proxies distributes your requests across multiple IPs, making your crawler appear less like a bot and reducing the likelihood of blocks.
        *   **How `RoundRobinProxyStrategy` works:**
            This is the primary built-in strategy. You provide it with a list of `ProxyConfig` objects. For each new crawl operation (or more granularly, depending on how the strategy is integrated), it cycles through this list, selecting the next proxy in a round-robin fashion.
        *   **Workflow: Setting up `RoundRobinProxyStrategy`:**
            1.  Define a list of `ProxyConfig` objects, each representing one proxy server.
            2.  Instantiate `RoundRobinProxyStrategy` with this list.
            3.  Assign this strategy instance to `CrawlerRunConfig.proxy_rotation_strategy`.
        *   **Code Example: Using `RoundRobinProxyStrategy` in `CrawlerRunConfig`**
            (Note: `ProxyRotationStrategy` is typically used with `arun_many` or in custom loops with `arun` where the `CrawlerRunConfig` is updated before each call).
            ```python
            import asyncio
            from crawl4ai import (
                AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, 
                RoundRobinProxyStrategy, BrowserConfig
            )

            async def crawl_with_rotating_proxies():
                proxies = [
                    ProxyConfig(server="http://user1:pass1@proxy1.example.com:8000", ip="1.1.1.1"),
                    ProxyConfig(server="http://user2:pass2@proxy2.example.com:8001", ip="2.2.2.2"),
                    ProxyConfig(server="http://user3:pass3@proxy3.example.com:8002", ip="3.3.3.3"),
                ]
                
                proxy_strategy = RoundRobinProxyStrategy(proxies=proxies)

                # This run_config will now use the rotation strategy.
                # The proxy_config within it will be set dynamically by the strategy.
                run_cfg = CrawlerRunConfig(
                    url="https://api.ipify.org?format=json", # A site to check current IP
                    proxy_rotation_strategy=proxy_strategy
                )
                
                # Browser config remains simple or can have its own default proxy (which would be overridden)
                browser_cfg = BrowserConfig(headless=True)

                async with AsyncWebCrawler(config=browser_cfg) as crawler:
                    for i in range(5): # Make 5 requests to see rotation
                        print(f"\nRequest {i+1}:")
                        # The strategy will pick the next proxy for each arun() call
                        # when proxy_rotation_strategy is set on the CrawlerRunConfig.
                        # The proxy_config field in run_cfg is effectively managed by the strategy.
                        
                        # It's more typical to see proxy_rotation_strategy used with arun_many,
                        # where the dispatcher would call get_next_proxy for each task.
                        # For a loop with arun(), you'd typically call get_next_proxy manually:
                        
                        next_proxy = await proxy_strategy.get_next_proxy()
                        current_run_cfg = run_cfg.clone(proxy_config=next_proxy) # Clone and set specific proxy

                        result = await crawler.arun(config=current_run_cfg)
                        if result.success:
                            print(f"  Crawled {result.url} via proxy {next_proxy.server if next_proxy else 'None'}")
                            print(f"  Response: {result.html[:100]}")
                        else:
                            print(f"  Failed to crawl {result.url}: {result.error_message}")
                        await asyncio.sleep(0.5) # Small delay between requests

            if __name__ == "__main__":
                asyncio.run(crawl_with_rotating_proxies())
            ```
            *Important Note:* The `ProxyRotationStrategy` itself usually doesn't automatically apply the proxy to each `arun` call if you're looping. It's designed to be queried (e.g., `await strategy.get_next_proxy()`) and then that specific `ProxyConfig` is passed to the `CrawlerRunConfig` for *that particular* `arun` call. `AsyncDispatcher` in `arun_many` handles this querying internally.

        *   **Considerations for managing proxy lists and health:**
            *   Real-world proxy lists can be large and proxies can become unavailable.
            *   You might need external logic to check proxy health and update the list passed to `RoundRobinProxyStrategy` periodically or implement a more advanced strategy that handles proxy failures (e.g., by temporarily removing a failing proxy from rotation).

*   B. **Browser and Session Management (`BrowserManager`, `ManagedBrowser`):**
    These classes are mostly for internal use by `AsyncPlaywrightCrawlerStrategy` but understanding their concepts can be beneficial.
    *   1.  **Understanding `BrowserManager`'s Role:**
        *   The `BrowserManager` (an internal component of `AsyncPlaywrightCrawlerStrategy`) is responsible for the lifecycle of Playwright `BrowserContext` objects.
        *   It handles creating new contexts based on the `BrowserConfig` and the specifics of a `CrawlerRunConfig` (like `session_id` or per-run proxy/UA overrides).
        *   **Context Signatures:** It generates a "signature" for each requested context configuration. If a subsequent request has the same signature (e.g., same `session_id`, same identity overrides), it can reuse an existing, compatible `BrowserContext` instead of creating a new one, which is efficient.
        *   It also manages the closing of contexts when a session is killed or the crawler is shut down.
    *   2.  **Leveraging `ManagedBrowser` (typically internal but concepts are useful):**
        *   `ManagedBrowser` is used when `BrowserConfig.use_managed_browser` is `True` (e.g., for `"builtin"` or `"docker"` modes, or when `use_persistent_context=True`).
        *   It's responsible for launching and managing the actual browser *process* (e.g., Chrome executable).
        *   **Persistent Profiles:** When `user_data_dir` is set, `ManagedBrowser` launches the browser with that profile directory, enabling persistence.
        *   **CDP Connection:** If `cdp_url` is provided in `BrowserConfig` (for `browser_mode="cdp"`), `ManagedBrowser` is bypassed, and Playwright connects directly to the existing browser endpoint. If `ManagedBrowser` launches the browser, it makes the CDP endpoint available (e.g., `ws://localhost:9222/...`).
    *   3.  **Explicit Session Termination with `crawler.browser_manager.kill_session(session_id)`:**
        *   **When and why:** If you've used a `session_id` in `CrawlerRunConfig` to maintain a specific page/tab across multiple `arun` calls, you should explicitly kill that session when it's no longer needed.
        *   **Impact:** This closes the associated browser page and potentially the `BrowserContext` (if no other pages are using it and it's not the default shared context), freeing up browser resources. If you don't kill sessions, they might linger until the entire `AsyncWebCrawler` is closed.
        *   Code Example:
            ```python
            # ... (inside an async with AsyncWebCrawler as crawler:)
            # await crawler.arun(config=CrawlerRunConfig(url="...", session_id="my_task_session"))
            # ...
            # await crawler.arun(config=CrawlerRunConfig(url="...", session_id="my_task_session", js_only=True, ...))
            # ...
            # Done with this logical task
            # await crawler.browser_manager.kill_session("my_task_session")
            ```

*   C. **Configuration Serialization and Deserialization (`to_serializable_dict`, `from_serializable_dict`):**
    `BrowserConfig`, `CrawlerRunConfig`, and their nested config objects (like `ProxyConfig`, `GeolocationConfig`) provide methods for easy serialization and deserialization. This is useful for:
    *   Storing complex configurations in files (e.g., JSON, YAML).
    *   Sharing configurations between different parts of an application or different scripts.
    *   Dynamically loading configurations at runtime.
    *   **Workflow: Saving and Loading Configurations:**
        1.  Create your config object (e.g., `my_run_config = CrawlerRunConfig(...)`).
        2.  Get a serializable dictionary: `serializable_dict = my_run_config.to_dict()` (or use `my_run_config.dump()` which leverages `to_serializable_dict` from `async_configs.py`).
        3.  Save this dictionary to a file (e.g., using `json.dump()`).
        4.  To load, read the dictionary from the file.
        5.  Recreate the config object: `loaded_run_config = CrawlerRunConfig.from_kwargs(loaded_dict)` (or use `CrawlerRunConfig.load(loaded_dict)`).
    *   **Code Example: Saving and loading a `CrawlerRunConfig` object**
        ```python
        import json
        import asyncio
        from crawl4ai import CrawlerRunConfig, CacheMode, GeolocationConfig, ProxyConfig
        from crawl4ai.async_configs import to_serializable_dict # For direct use if needed
        
        # Create a complex CrawlerRunConfig
        original_config = CrawlerRunConfig(
            url="https://example.com",
            cache_mode=CacheMode.ENABLED,
            js_code="console.log('hello');",
            screenshot=True,
            geolocation=GeolocationConfig(latitude=34.05, longitude=-118.24),
            proxy_config=ProxyConfig(server="http://myproxy.com:3128")
        )

        # 1. Serialize to a dictionary (using the .dump() method which calls to_serializable_dict internally)
        # config_dict = original_config.dump() 
        # Or, if using the utility function directly on an object not having .dump()
        config_dict = to_serializable_dict(original_config, ignore_default_value=True)


        # 2. Save to JSON file
        with open("my_crawler_run_config.json", "w") as f:
            json.dump(config_dict, f, indent=2)
        print("Config saved to my_crawler_run_config.json")

        # 3. Load from JSON file
        with open("my_crawler_run_config.json", "r") as f:
            loaded_dict = json.load(f)
        
        # 4. Recreate CrawlerRunConfig object (using .load() static method)
        # loaded_config = CrawlerRunConfig.load(loaded_dict)
        # Or using from_kwargs if .load() is not available or for more general objects
        from crawl4ai.async_configs import from_serializable_dict
        loaded_config_generic = from_serializable_dict(loaded_dict) # This will give back the CrawlerRunConfig instance

        # Verify (loaded_config_generic will be an instance of CrawlerRunConfig)
        assert isinstance(loaded_config_generic, CrawlerRunConfig)
        assert loaded_config_generic.url == original_config.url
        assert loaded_config_generic.cache_mode == original_config.cache_mode
        print(f"Loaded config successfully. Screenshot setting: {loaded_config_generic.screenshot}")
        print(f"Loaded proxy server: {loaded_config_generic.proxy_config.server if loaded_config_generic.proxy_config else 'None'}")

        # Example of using it
        # async with AsyncWebCrawler() as crawler:
        #     result = await crawler.arun(config=loaded_config_generic)
        #     # ...
        ```
    *   **Benefits:** This structured approach is more robust than pickling and ensures that only intended configuration parameters are serialized, making it easier to manage versions and share configurations.

*   D. **Logging and Debugging Core Operations:**
    *   **Leveraging `AsyncLogger`:**
        Crawl4ai uses an `AsyncLogger` instance (by default, or you can pass your own). This logger provides tagged and colored output, making it easier to follow the flow of operations.
        *   Configure verbosity via `BrowserConfig(verbose=True)` or `CrawlerRunConfig(verbose=True)`. The `CrawlerRunConfig` setting usually takes precedence for a specific run.
    *   **Interpreting verbose logs for troubleshooting:**
        Verbose logs will show:
        *   [INIT] tags for crawler and strategy initialization.
        *   [BROWSER] tags for browser launching, context creation, page events.
        *   [FETCH] tags for URL fetching attempts, cache hits/misses, status codes.
        *   [SCRAPE] tags for scraping strategy execution.
        *   [MARKDOWN] tags for markdown generation steps.
        *   [EXTRACTION] tags if an extraction strategy is active.
        *   [JS] tags for JavaScript execution.
        *   [HOOK] tags when custom hooks are executed.
        *   [ERROR] tags for any exceptions or failures.
        *   [CACHE] tags for cache operations.
    *   **Key log messages to watch for:**
        *   Browser launch arguments.
        *   CDP connection messages (if using managed/CDP modes).
        *   Cache hits/misses: `Cache HIT for <URL>` or `Cache MISS for <URL>`.
        *   Navigation events: `Navigating to <URL>...`, `Navigation to <URL> successful`.
        *   JavaScript execution: `Executing JS...`, `JS execution result...`.
        *   Hook execution: `Executing hook <hook_name>...`.
        *   Any messages tagged with [ERROR] or [WARNING].

# VIII. Integrating Core with Specialized Strategies

The `core` components of Crawl4ai (like `AsyncWebCrawler`, `BrowserConfig`, `CrawlerRunConfig`) provide the foundation upon which more specialized strategies for PDF processing, Markdown generation, data extraction, chunking, and content filtering are built. Understanding how core configurations influence these specialized strategies is key to using them effectively.

*   A. **How Core Configurations Influence Specialized Crawlers/Processors:**
    *   **`PDFCrawlerStrategy` & `PDFContentScrapingStrategy`:**
        *   The `PDFCrawlerStrategy` itself often uses an underlying `AsyncHTTPCrawlerStrategy` to download the PDF file because PDFs are typically static assets not requiring browser rendering for the download itself.
        *   Thus, `BrowserConfig` settings like `headless` or `js_code` are usually *not directly relevant* to the PDF fetching part.
        *   However, `HTTPCrawlerConfig` (passed via `CrawlerRunConfig.http_crawler_config`) for setting custom headers (e.g., authentication) for the PDF download request can be important.
        *   The `PDFContentScrapingStrategy` (used by `PDFCrawlerStrategy` and can be used standalone with `AsyncWebCrawler` if you point it to a PDF URL with the right Content-Type) then processes the downloaded PDF bytes. Core configs like `word_count_threshold` or `css_selector` are *not applicable* here as it's not HTML. PDF-specific parameters for this strategy (like `save_images_locally`, `extract_images`) are passed during its initialization.
    *   **`MarkdownGenerationStrategy` (e.g., `DefaultMarkdownGenerator`):**
        *   This strategy consumes HTML, typically `CrawlResult.cleaned_html` or `CrawlResult.fit_html` (if a `content_filter` was used).
        *   Therefore, `CrawlerRunConfig` parameters that affect `cleaned_html` (like `css_selector`, `excluded_tags`, `target_elements`) directly influence the input to the Markdown generator. A well-scoped `cleaned_html` leads to more relevant Markdown.
        *   The `content_filter` *within* the `MarkdownGenerationStrategy` (e.g., `PruningContentFilter`) further refines the HTML before final Markdown conversion, producing `fit_markdown`.
    *   **`ExtractionStrategy` (e.g., `LLMExtractionStrategy`, `JsonCSSExtractionStrategy`, `RegexExtractionStrategy`):**
        *   This is typically set via `CrawlerRunConfig.extraction_strategy`.
        *   The `input_format` property of the chosen `ExtractionStrategy` determines what content it receives from `CrawlResult`. Common options are:
            *   `"markdown"`: Uses `CrawlResult.markdown.raw_markdown` (or `fit_markdown` if available and preferred by the strategy).
            *   `"html"`: Uses `CrawlResult.cleaned_html`.
            *   `"fit_markdown"`: Specifically uses `CrawlResult.markdown.fit_markdown`.
            *   `"fit_html"`: Specifically uses `CrawlResult.markdown.fit_html`.
        *   Core configurations that shape these inputs (e.g., `css_selector`, `target_elements`, `markdown_generator`'s `content_filter`) are thus critical for the success of the extraction.
        *   For `LLMExtractionStrategy`, the `LLMConfig` (set when initializing the strategy or passed via `CrawlerRunConfig.llm_config`) is paramount, controlling the LLM provider, model, API key, and prompting parameters.
        *   `CrawlerRunConfig.target_elements` can be particularly useful with extraction. If your extraction schema is designed to pull data from specific sections, `target_elements` ensures only those sections are considered by strategies that respect it (like an LLM strategy prompted to process focused HTML snippets).
    *   **`ChunkingStrategy` (e.g., `RegexChunking`):**
        *   Chunking strategies typically operate on plain text, often derived from `CrawlResult.markdown.raw_markdown` or `CrawlResult.markdown.fit_markdown`.
        *   Core configurations affecting the quality and scope of the source Markdown will, therefore, impact the chunks produced.
    *   **`ContentFilteringStrategy` (e.g., `PruningContentFilter`, `LLMContentFilter`):**
        *   These are usually part of a `MarkdownGenerationStrategy` configuration.
        *   They take `cleaned_html` (output of `ContentScrapingStrategy`) as input and produce a "fit" (more relevant) HTML, which is then converted to `fit_markdown`.
        *   Core settings like `css_selector` apply *before* these filters, pre-scoping the HTML they receive.

*   B. **Pointers to Dedicated Documentation for Specialized Strategies:**
    *   **PDF Processing:** For details on `PDFCrawlerStrategy` and `PDFContentScrapingStrategy`, including image extraction and text processing from PDFs, refer to:
        *   [PDF Processing Guide](../../processors/pdf-processing.md) *(Assuming this link structure)*
    *   **Markdown Generation:** To understand `DefaultMarkdownGenerator`, custom HTML-to-Markdown options, and content filtering within Markdown generation, see:
        *   [Markdown Generation Guide](../../strategies/markdown-generation.md)
    *   **Data Extraction:**
        *   For LLM-based extraction (`LLMExtractionStrategy`), schema definition, and prompting: [LLM Extraction Guide](../../strategies/llm-extraction.md)
        *   For CSS/XPath-based structured data extraction (`JsonCssExtractionStrategy`, `JsonXpathExtractionStrategy`): [Selector-Based Extraction Guide](../../strategies/selector-extraction.md)
        *   For Regex-based extraction (`RegexExtractionStrategy`): [Regex Extraction Guide](../../strategies/regex-extraction.md)
    *   **Content Chunking:** For details on `RegexChunking` and other chunking approaches for LLM context preparation:
        *   [Content Chunking Guide](../../strategies/chunking.md)
    *   **Content Filtering:** For `PruningContentFilter`, `LLMContentFilter`, and creating custom filters:
        *   [Content Filtering Guide](../../strategies/content-filtering.md)

# IX. Troubleshooting Common Core Issues

Even with robust configurations, crawling can sometimes hit snags. Here are common issues related to the core components and how to approach them:

*   A. **Browser Launch Failures:**
    *   **Symptom:** `AsyncWebCrawler.start()` or the `async with` block fails immediately, often with errors related to Playwright or browser executables.
    *   **Causes & Solutions:**
        *   **Missing Playwright Drivers:** Playwright needs browser-specific drivers.
            *   **Fix:** Run `playwright install` or `crawl4ai-setup` to install them. If in a restricted environment, you might need to specify driver download paths manually using Playwright environment variables.
        *   **Permissions Issues for `user_data_dir`:** If you've specified a `user_data_dir` in `BrowserConfig`, the crawler process needs write permissions to that directory.
            *   **Fix:** Ensure permissions are correct or choose a writable directory.
        *   **Conflicting Browser Processes:** An old or hung browser process might be using the same debugging port or profile directory.
            *   **Fix:** Manually kill lingering browser processes. `ManagedBrowser` attempts some cleanup, but it's not always foolproof.
        *   **Incompatible `extra_args`:** Custom browser arguments might be incorrect or conflict.
            *   **Fix:** Remove or verify `extra_args` one by one.
        *   **Unsupported OS/Architecture:** Ensure your Playwright version supports your OS and CPU architecture.

*   B. **Navigation Timeouts or Errors (`page.goto()` failures):**
    *   **Symptom:** The crawl hangs or fails during the page navigation step, often with `TimeoutError` or network-related errors in logs.
    *   **Causes & Solutions:**
        *   **Incorrect `wait_for` Conditions:** If `CrawlerRunConfig.wait_for` specifies a condition that's never met, the page will time out.
            *   **Fix:** Verify your CSS selector or JS condition. Use browser dev tools to test selectors. Increase `wait_for_timeout` if the condition legitimately takes longer to meet.
        *   **Network Issues or Site Blocking:** The target site might be down, slow, or actively blocking your IP/User-Agent.
            *   **Fix:** Check site accessibility manually. Try different User-Agents or proxies (`BrowserConfig` or `CrawlerRunConfig`). Implement robust rate limiting via a custom dispatcher if hitting site limits.
        *   **`page_timeout` Too Short:** The global `CrawlerRunConfig.page_timeout` (default 60s) might be too short for very slow-loading pages.
            *   **Fix:** Increase `page_timeout` for problematic sites.
        *   **SSL/TLS Errors:** The site might have an invalid SSL certificate.
            *   **Fix:** Set `BrowserConfig(ignore_https_errors=True)` if you trust the site and want to proceed (use with caution). Check `result.ssl_certificate` if `fetch_ssl_certificate=True` for details.

*   C. **Content Not Appearing as Expected:**
    *   **Symptom:** `result.markdown` is empty or missing key content; `result.extracted_content` is empty or incorrect.
    *   **Causes & Solutions:**
        *   **Lazy Loading:** Content is loaded via JavaScript as the user scrolls.
            *   **Fix:** Set `CrawlerRunConfig(scan_full_page=True, scroll_delay=0.5)` (adjust delay as needed). For more complex infinite scroll, you might need custom `js_code` to trigger loads and appropriate `wait_for` conditions.
        *   **JavaScript Errors on the Page:** Errors in the site's own JS can prevent content from rendering.
            *   **Fix:** Run with `BrowserConfig(headless=False)` to inspect the browser console. Enable `CrawlerRunConfig(log_console=True)` to capture console messages in Crawl4ai's logs.
        *   **Incorrect CSS Selectors or `target_elements`:** Your selectors might be wrong or too broad/narrow.
            *   **Fix:** Use browser developer tools to test and refine your CSS selectors.
        *   **Content Loaded After Crawler Finishes:** The crawler might be capturing HTML before all dynamic content has finished loading.
            *   **Fix:** Use more robust `wait_for` conditions (e.g., wait for a specific "data loaded" flag set by page JS, or wait for a specific network request to complete using Playwright's advanced routing/waiting features via hooks). Increase `delay_before_retrieve_html` as a simpler, less precise fix.
        *   **AJAX Content:** Content is loaded via AJAX calls. `wait_for="networkidle"` or waiting for specific XHR responses (via hooks) can help.

*   D. **Proxy-Related Problems:**
    *   **Symptom:** Connection errors, timeouts, or getting content from your own IP instead of the proxy.
    *   **Causes & Solutions:**
        *   **Invalid Proxy Credentials/Format:** Ensure `ProxyConfig.server` (e.g., `http://user:pass@host:port`) and auth details are correct.
        *   **Proxy Server Unresponsive:** The proxy server itself might be down or overloaded. Test it independently.
        *   **Website Blocking Proxy IP:** The target site may have blacklisted the proxy's IP. Rotate proxies.
        *   **Proxy Type Mismatch:** Ensure the proxy protocol (HTTP, HTTPS, SOCKS5) matches what the server expects.
        *   **Firewall Issues:** Local or network firewalls might be blocking outbound connections to the proxy port.

*   E. **Session State Not Persisting (`session_id`, `user_data_dir`, `storage_state`):**
    *   **Symptom:** Logins are lost between `arun()` calls despite using `session_id`, or site preferences aren't remembered when using `user_data_dir` or `storage_state`.
    *   **Causes & Solutions:**
        *   **Incorrect `session_id` Usage:** Ensure you're passing the *exact same* `session_id` string to subsequent `arun()` calls that need to share the state.
        *   **`js_only=False` on Subsequent Calls:** If you navigate to a new URL (even the same one) in a subsequent `arun` call for an existing session, it's a full navigation. If you only want to run JS on the *current page* of that session, set `js_only=True`.
        *   **`use_persistent_context=False` with `user_data_dir`:** Both must be set correctly in `BrowserConfig` for profile persistence to work.
        *   **Incorrect `user_data_dir` Path or Permissions:** Verify the path and ensure write access.
        *   **Invalid `storage_state` File/Format:** Ensure the JSON file is valid and was correctly saved by Playwright.
        *   **Site-Specific Session Handling:** Some sites have very aggressive session timeout mechanisms, use multiple cookies, or employ JavaScript-based session checks that might invalidate sessions even if cookies are present. Debugging these often requires running headed and observing network requests and storage.
        *   **Context Replaced by `BrowserManager`:** If you drastically change other context-affecting parameters in `CrawlerRunConfig` (like per-run proxy or UA) even with the same `session_id`, the `BrowserManager` might decide it needs a new `BrowserContext` due to a different "signature," potentially losing the state of the old one associated with that `session_id`. Keep context-affecting overrides consistent for a given session.

# X. Conclusion and Best Practices Summary

You've now explored the foundational `core` components of Crawl4ai, from launching browsers with `BrowserConfig` to executing fine-tuned crawl operations with `CrawlerRunConfig`, and interpreting the rich `CrawlResult`. The core is designed for flexibility and power, enabling you to tackle a wide array of web data extraction challenges.

**Recap of Key Core Concepts:**

*   **`AsyncWebCrawler`:** Your main interaction point, orchestrating the crawl.
*   **`BrowserConfig`:** Defines the global browser environment (engine, headless, proxy, UA). Set it up once for a crawler instance.
*   **`CrawlerRunConfig`:** Defines per-crawl behavior (URL, caching, JS, content selection, extraction). Pass it to `arun()` or `arun_many()`.
*   **Strategies:** Crawl4ai uses a strategy pattern for crawling (`AsyncPlaywrightCrawlerStrategy`, `AsyncHTTPCrawlerStrategy`), scraping (`WebScrapingStrategy`), Markdown generation (`DefaultMarkdownGenerator`), and data extraction.
*   **Hooks (in `AsyncPlaywrightCrawlerStrategy`):** Allow custom code injection at key lifecycle points for advanced control and interactions.
*   **`CrawlResult`:** The comprehensive output object containing HTML variants, Markdown, media, links, metadata, and more.

**Best Practices Summary:**

1.  **Plan Your Configuration:**
    *   Distinguish between global browser setup (`BrowserConfig`) and per-crawl instructions (`CrawlerRunConfig`).
    *   Start with simpler configurations and add complexity (like JS execution or specific selectors) iteratively.
2.  **Leverage Caching Wisely:**
    *   Use `CacheMode.ENABLED` during development to speed up iterations.
    *   Use `CacheMode.BYPASS` or `CacheMode.WRITE_ONLY` for production runs where fresh data is paramount.
3.  **Be Specific with Content Selection:**
    *   Use `css_selector` to scope processing to relevant page areas early.
    *   Employ `target_elements`, `excluded_tags`, and `excluded_selector` to refine the content fed into Markdown and extraction processes.
4.  **Handle Dynamic Content Effectively:**
    *   Use `js_code` for interactions.
    *   Rely on robust `wait_for` conditions (CSS selectors or JS expressions) rather than fixed delays.
    *   Utilize `scan_full_page` for lazy-loading and infinite scroll, adjusting `scroll_delay` as needed.
5.  **Manage Sessions and State:**
    *   For multi-step processes or login persistence within a crawler's lifetime, use `session_id` and `js_only=True` for subsequent on-page JS executions.
    *   For persistence across crawler restarts, use `BrowserConfig(use_persistent_context=True, user_data_dir="...")` or `BrowserConfig(storage_state="...")`.
    *   Always `kill_session()` when a `session_id`-based task is complete.
6.  **Prioritize Realistic Browser Emulation for Stealth:**
    *   Use common, valid User-Agents (consider `user_agent_mode="random"`).
    *   Set appropriate viewports.
    *   Use proxies, especially for larger crawls or geo-specific content.
7.  **Use Hooks for Complex Interactions:**
    *   For tasks like login sequences or intricate page manipulations, hooks (especially `on_page_context_created` and `before_retrieve_html`) offer more control than simple `js_code`.
    *   Pass dynamic data to hooks via `CrawlerRunConfig.shared_data`.
8.  **Choose the Right Crawler Strategy:**
    *   Default to `AsyncPlaywrightCrawlerStrategy` for most modern web pages.
    *   Opt for `AsyncHTTPCrawlerStrategy` for APIs, static HTML, or when JS rendering is not needed, to gain significant speed and reduce resource usage.
9.  **Log and Debug Systematically:**
    *   Enable `verbose=True` in `BrowserConfig` or `CrawlerRunConfig`.
    *   Run `headless=False` during development to visually inspect browser behavior.
    *   Check `result.error_message` and `result.status_code` for failures.
10. **Respect `robots.txt`:**
    *   Enable `check_robots_txt=True` in `CrawlerRunConfig` for ethical crawling, unless you have specific reasons and permissions to do otherwise.

By mastering these core components and applying these best practices, you can build powerful, efficient, and resilient web crawlers with Crawl4ai, capable of extracting valuable data from even the most challenging websites. Happy crawling!
```

## Content (Examples)

```markdown
# Examples Outline for crawl4ai - core Component

**Target Document Type:** Examples Collection
**Target Output Filename Suggestion:** `llm_examples_core.md`
**Library Version Context:** 0.6.3
**Outline Generation Date:** 2024-05-24 10:00:00
---

This document provides a collection of runnable code examples for the `core` component of the `crawl4ai` library. Each example is designed to showcase a specific feature or configuration.

## 1. Basic `AsyncWebCrawler` Usage

### 1.1. Example: Simplest crawl of a single URL with default `BrowserConfig` and `CrawlerRunConfig`.

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def simplest_crawl():
    # Uses default BrowserConfig and CrawlerRunConfig
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        if result.success:
            print("Crawl successful!")
            print(f"Markdown (first 300 chars):\n{result.markdown.raw_markdown[:300]}...")
        else:
            print(f"Crawl failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(simplest_crawl())
```

---
### 1.2. Example: Using `AsyncWebCrawler` as an asynchronous context manager (`async with`).

This is the recommended way to manage the crawler's lifecycle.

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def context_manager_crawl():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        if result.success:
            print("Crawl successful using context manager!")
            print(f"Page title from metadata: {result.metadata.get('title')}")
        else:
            print(f"Crawl failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(context_manager_crawl())
```

---
### 1.3. Example: Explicitly starting and closing the `AsyncWebCrawler` using `start()` and `close()`.

Useful for scenarios where the crawler's lifecycle needs more manual control.

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def explicit_lifecycle_crawl():
    crawler = AsyncWebCrawler()
    await crawler.start()  # Explicitly start the crawler and browser
    try:
        result = await crawler.arun(url="https://example.com")
        if result.success:
            print("Crawl successful with explicit start/close!")
            print(f"Cleaned HTML (first 300 chars):\n{result.cleaned_html[:300]}...")
        else:
            print(f"Crawl failed: {result.error_message}")
    finally:
        await crawler.close()  # Ensure the crawler is closed

if __name__ == "__main__":
    asyncio.run(explicit_lifecycle_crawl())
```

---
### 1.4. Example: Handling a failed crawl (e.g., non-existent URL, network error) and checking `CrawlResult.success` and `CrawlResult.error_message`.

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def failed_crawl_handling():
    async with AsyncWebCrawler() as crawler:
        # Using a deliberately non-existent URL
        result = await crawler.arun(url="https://thissitedoesnotexist.crawl4ai")
        if not result.success:
            print(f"Crawl failed as expected for URL: {result.url}")
            print(f"Status Code: {result.status_code}")
            print(f"Error Message: {result.error_message}")
        else:
            print("Crawl unexpectedly succeeded!")

if __name__ == "__main__":
    asyncio.run(failed_crawl_handling())
```

---
### 1.5. Example: Processing raw HTML content directly using `crawler.aprocess_html()`.

This is useful if you already have HTML content and want to use Crawl4ai's processing capabilities.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def process_raw_html_directly():
    raw_html_content = """
    <html>
        <head><title>My Test Page</title></head>
        <body>
            <h1>Welcome!</h1>
            <p>This is a paragraph with a <a href="https://example.com">link</a>.</p>
            <script>console.log("This should be removed");</script>
        </body>
    </html>
    """
    # No need for BrowserConfig as we are not navigating
    async with AsyncWebCrawler() as crawler:
        # Use CrawlerRunConfig if you need specific processing options
        config = CrawlerRunConfig()
        result = await crawler.aprocess_html(
            url="raw://my_virtual_page", # Provide a conceptual URL
            html=raw_html_content,
            config=config
        )
        if result.success:
            print("Raw HTML processed successfully!")
            print(f"Markdown:\n{result.markdown.raw_markdown}")
            print(f"Cleaned HTML:\n{result.cleaned_html}")
        else:
            print(f"HTML processing failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(process_raw_html_directly())
```

---
### 1.6. Example: Crawling a local HTML file using the `file:///` prefix.

First, create a dummy HTML file named `local_test.html` in the same directory as your script.

```python
# local_test.html
# <!DOCTYPE html>
# <html>
# <head>
#     <title>Local Test File</title>
# </head>
# <body>
#     <h1>Hello from a local file!</h1>
#     <p>This content is loaded from the local filesystem.</p>
# </body>
# </html>
```

```python
import asyncio
import os
from pathlib import Path
from crawl4ai import AsyncWebCrawler

async def crawl_local_file():
    # Create a dummy local HTML file for the example
    script_dir = Path(__file__).parent
    local_file_path = script_dir / "local_test_for_crawl.html"
    with open(local_file_path, "w", encoding="utf-8") as f:
        f.write("<!DOCTYPE html><html><head><title>Local Test</title></head><body><h1>Local Content</h1></body></html>")

    file_url = f"file:///{local_file_path.resolve()}"
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url=file_url)
        if result.success:
            print(f"Successfully crawled local file: {file_url}")
            print(f"Markdown (first 100 chars): {result.markdown.raw_markdown[:100]}...")
        else:
            print(f"Failed to crawl local file: {result.error_message}")
    
    # Clean up the dummy file
    if os.path.exists(local_file_path):
        os.remove(local_file_path)

if __name__ == "__main__":
    asyncio.run(crawl_local_file())
```

---
### 1.7. Example: Accessing basic fields from `CrawlResult` (e.g., `url`, `html`, `markdown.raw_markdown`, `status_code`, `response_headers`).

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def access_crawl_result_fields():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        if result.success:
            print(f"URL Crawled: {result.url}")
            print(f"Status Code: {result.status_code}")
            
            print("\n--- Response Headers (sample) ---")
            if result.response_headers:
                for key, value in list(result.response_headers.items())[:3]: # Print first 3 headers
                    print(f"{key}: {value}")
            
            print(f"\n--- Raw HTML (first 100 chars) ---\n{result.html[:100]}...")
            print(f"\n--- Cleaned HTML (first 100 chars) ---\n{result.cleaned_html[:100]}...")
            
            if result.markdown:
                 print(f"\n--- Raw Markdown (first 100 chars) ---\n{result.markdown.raw_markdown[:100]}...")
            
            print(f"\n--- Metadata (sample) ---")
            if result.metadata:
                 for key, value in list(result.metadata.items())[:3]: # Print first 3 metadata items
                    print(f"{key}: {value}")
        else:
            print(f"Crawl failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(access_crawl_result_fields())
```

---
## 2. Configuring the Browser (`BrowserConfig`)

### 2.1. Example: Initializing `AsyncWebCrawler` with a custom `BrowserConfig` object.

This example sets the browser to run in non-headless mode and uses Firefox.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig

async def custom_browser_config_init():
    # Configure browser to be Firefox and visible
    browser_config = BrowserConfig(
        browser_type="firefox",
        headless=False  # Set to True to run without UI
    )
    
    # Pass the custom config to the crawler
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(url="https://example.com")
        if result.success:
            print(f"Crawl successful with custom BrowserConfig (Firefox, visible)!")
            print(f"Page title: {result.metadata.get('title')}")
        else:
            print(f"Crawl failed: {result.error_message}")

if __name__ == "__main__":
    # This example might open a visible browser window.
    # Ensure Firefox is installed if you run this.
    # asyncio.run(custom_browser_config_init()) 
    print("Skipping custom_browser_config_init example in automated run to avoid GUI interaction.")
```

---
### 2.2. Browser Type and Headless Mode

#### 2.2.1. Example: Using Chromium browser (default).

This shows the default behavior if no `browser_type` is specified.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig

async def chromium_default_crawl():
    # Chromium is the default, but we can explicitly set it
    browser_config = BrowserConfig(browser_type="chromium", headless=True)
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(url="https://example.com")
        if result.success:
            print("Crawl successful with Chromium (default)!")
        else:
            print(f"Crawl failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(chromium_default_crawl())
```

---
#### 2.2.2. Example: Using Firefox browser (`browser_type="firefox"`).

Ensure Firefox is installed on your system for this example to run.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig

async def firefox_crawl():
    browser_config = BrowserConfig(browser_type="firefox", headless=True)
    try:
        async with AsyncWebCrawler(config=browser_config) as crawler:
            result = await crawler.arun(url="https://example.com")
            if result.success:
                print("Crawl successful with Firefox!")
            else:
                print(f"Crawl failed with Firefox: {result.error_message}")
    except Exception as e:
        print(f"Error running Firefox example: {e}. Ensure Firefox is installed and Playwright browsers are set up (`crawl4ai-setup`).")


if __name__ == "__main__":
    # asyncio.run(firefox_crawl())
    print("Skipping Firefox example in automated run. Uncomment to run if Firefox is installed.")
```

---
#### 2.2.3. Example: Using WebKit browser (`browser_type="webkit"`).

Ensure WebKit (Safari's engine) is installed via Playwright.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig

async def webkit_crawl():
    browser_config = BrowserConfig(browser_type="webkit", headless=True)
    try:
        async with AsyncWebCrawler(config=browser_config) as crawler:
            result = await crawler.arun(url="https://example.com")
            if result.success:
                print("Crawl successful with WebKit!")
            else:
                print(f"Crawl failed with WebKit: {result.error_message}")
    except Exception as e:
         print(f"Error running WebKit example: {e}. Ensure WebKit is installed and Playwright browsers are set up (`crawl4ai-setup`).")


if __name__ == "__main__":
    # asyncio.run(webkit_crawl())
    print("Skipping WebKit example in automated run. Uncomment to run if WebKit is installed.")
```

---
#### 2.2.4. Example: Running the browser in non-headless mode (`headless=False`) for visual debugging.

This will open a visible browser window.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig

async def non_headless_crawl():
    browser_config = BrowserConfig(headless=False)  # Browser window will be visible
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(url="https
```