crawl4ai/docs/llm.txt/2_configuration.q.md

Hypothetical Questions

BrowserConfig:

1. Browser Types and Headless Mode

   • "How do I choose between chromium, firefox, or webkit for browser_type?"
   • "What are the benefits of running the browser in headless=True mode versus a visible UI?"

2. Managed Browser and Persistent Context

   • "When should I enable use_managed_browser for advanced session control?"
   • "How do I use use_persistent_context and user_data_dir to maintain login sessions and persistent storage?"

3. Debugging and Remote Access

   • "How do I use the debugging_port to remotely inspect the browser with DevTools?"

4. Proxy and Network Configurations

   • "How can I configure a proxy or proxy_config for region-specific crawling or authentication?"

5. Viewports and Layout Testing

   • "How do I adjust viewport_width and viewport_height for responsive layout testing?"

6. Downloads and Storage States

   • "What steps do I need to take to enable accept_downloads and specify a downloads_path?"
   • "How can I use storage_state to preload cookies or session data?"

7. HTTPS and JavaScript Settings

   • "What happens if I set ignore_https_errors=True on sites with invalid SSL certificates?"
   • "When should I disable java_script_enabled to improve speed and stability?"

8. Cookies, Headers, and User Agents

   • "How do I add custom cookies or headers to every browser request?"
   • "How can I set a custom user_agent or use a user_agent_mode like random to avoid detection?"

9. Performance Tuning

   • "What is the difference between text_mode, light_mode, and adding extra_args for performance tuning?"

---

CrawlerRunConfig:

10. Content Extraction and Filtering

    • "How does the word_count_threshold affect which pages or sections get processed?"
    • "What extraction_strategy should I use for structured data extraction and how does chunking_strategy help organize the content?"
    • "How do I apply a css_selector or excluded_tags to refine my extracted content?"

11. Markdown and Text-Only Modes

    • "Can I generate Markdown output directly and what markdown_generator should I use?"
    • "When should I set only_text=True to strip out non-textual content?"

12. Caching and Session Handling

    • "How does cache_mode=ENABLED improve performance, and when should I consider WRITE_ONLY or disabling the cache?"
    • "What is the role of session_id in maintaining state across requests?"

13. Page Loading and Timing

    • "How do wait_until, page_timeout, and wait_for elements help control page load timing before extraction?"
    • "When should I disable wait_for_images to speed up the crawl?"

14. Delays and Concurrency

    • "How do mean_delay and max_range randomize request intervals to avoid detection?"
    • "What is semaphore_count and how does it manage concurrency for multiple crawling tasks?"

15. JavaScript Execution and Dynamic Content

    • "How can I inject custom js_code to load additional data or simulate user interactions?"
    • "When should I use scan_full_page or adjust_viewport_to_content to handle infinite scrolling?"

16. Screenshots, PDFs, and Media

    • "How do I enable screenshot or pdf generation to capture page states?"
    • "What are image_description_min_word_threshold and image_score_threshold for, and how do they enhance image-related extraction?"

17. Logging and Debugging

    • "How do verbose and log_console help me troubleshoot issues with crawling or page scripts?"

---

Topics Discussed in the File

• BrowserConfig Essentials:

  • Browser types (chromium, firefox, webkit)
  • Headless vs. non-headless mode
  • Persistent context and managed browser sessions
  • Proxy configurations and network settings
  • Viewport dimensions and responsive testing
  • Download handling and storage states
  • HTTPS errors and JavaScript enablement
  • Cookies, headers, and user agents
  • Performance tuning via text_mode, light_mode, and extra_args

• CrawlerRunConfig Core Settings:

  • Content extraction parameters (word_count_threshold, extraction_strategy, chunking_strategy)
  • Markdown generation and text-only extraction
  • Content filtering (css_selector, excluded_tags)
  • Caching strategies and cache_mode options
  • Page load conditions (wait_until, wait_for) and timeouts (page_timeout)
  • Delays, concurrency, and scaling (mean_delay, max_range, semaphore_count)
  • JavaScript injections (js_code) and handling dynamic/infinite scroll content
  • Screenshots, PDFs, and image thresholds for enhanced outputs
  • Logging and debugging modes (verbose, log_console)