ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2060c7e96575263132fb832fb5b37ce9c4b9a3c0
crawl4ai/docs/examples
History
unclecode 8576331d4e Add Shadow DOM flattening and reorder js_code execution pipeline 
- Add `flatten_shadow_dom` option to CrawlerRunConfig that serializes
  shadow DOM content into the light DOM before HTML capture. Uses a
  recursive serializer that resolves <slot> projections and strips
  only shadow-scoped <style> tags. Also injects an init script to
  force-open closed shadow roots via attachShadow patching.

- Move `js_code` execution to after `wait_for` + `delay_before_return_html`
  so user scripts run on the fully-hydrated page. Add `js_code_before_wait`
  for the less common case of triggering loading before waiting.

- Add JS snippet (flatten_shadow_dom.js), integration test, example,
  and documentation across all relevant doc files.
2026-02-18 06:43:00 +00:00
..
adaptive_crawling
Release/v0.7.6 (#1556)
2025-10-22 20:41:06 +08:00
assets
feat: Add virtual scroll support for modern web scraping
2025-06-29 20:41:37 +08:00
c4a_script
fix(docs): standardize C4A-Script tutorial, add CLI identity-based crawling, and add sponsorship CTA
2025-10-03 22:00:46 +05:30
capsolver_captcha_solver
Add CapSolver Captcha Solver
2025-11-06 15:37:31 +08:00
cli
feat(cli): add command line interface with comprehensive features
2025-02-10 16:58:52 +08:00
cloud_browser
Sponsors/new (#1637)
2025-11-24 13:29:33 +01:00
docker
feat(docker-api): add job-based polling endpoints for crawl and LLM tasks
2025-05-01 21:24:52 +08:00
markdown
feat(markdown): add content source selection for markdown generation
2025-04-17 20:13:53 +08:00
nst_proxy
Rename folder name for NSTProxy integration examples for crawl4ai
2025-11-13 09:36:39 +01:00
undetectability
feat: add stealth mode and enhance undetected browser support
2025-07-17 16:59:10 +08:00
url_seeder
docs(tutorial_url_seeder): refine summary and next steps, enhance agentic design patterns section
2025-06-05 16:20:58 +08:00
website-to-api
Release/v0.7.6 (#1556)
2025-10-22 20:41:06 +08:00
amazon_product_extraction_direct_url.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
amazon_product_extraction_using_hooks.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
amazon_product_extraction_using_use_javascript.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
arun_vs_arun_many.py
feat(crawler): optimize single URL handling and add performance comparison
2025-03-13 22:15:15 +08:00
async_webcrawler_multiple_urls_example.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
browser_optimization_example.py
Release prep (#749)
2025-02-28 19:53:35 +08:00
builtin_browser_example.py
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
chainlit.md
Add research assistant example using Chainlit
2024-06-04 22:43:09 +08:00
crawlai_vs_firecrawl.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
crawler_monitor_example.py
feat(monitor): add real-time crawler monitoring system with memory management
2025-03-12 19:05:24 +08:00
crypto_analysis_example.py
feat(models): add dedicated tables field to CrawlResult
2025-04-24 18:36:25 +08:00
deep_crawl_cancellation.py
Fix deep crawl cancellation example to use DFS for precise control
2026-01-22 06:25:34 +00:00
deep_crawl_crash_recovery.py
Add examples for deep crawl crash recovery and prefetch mode in documentation
2026-01-14 12:58:44 +01:00
deepcrawl_example.py
fix(docs): correct section numbering in deepcrawl_example.py tutorial
2025-03-04 20:57:33 +08:00
demo_multi_config_clean.py
fix: Correct URL matcher fallback behavior and improve memory monitoring
2025-08-03 16:50:54 +08:00
dfs_crawl_demo.py
#1510 : Add DFS deep crawler demonstration script and enhance DFS strategy with seen URL tracking
2025-11-12 17:44:43 +08:00
dispatcher_example.py
feat(content): add target_elements parameter for selective content extraction
2025-03-10 18:54:51 +08:00
docker_client_hooks_example.py
Release/v0.7.6 (#1556)
2025-10-22 20:41:06 +08:00
docker_config_obj.py
refactor(config): enhance serialization and config handling
2025-02-19 17:23:25 +08:00
docker_example.py
docs: remove CRAWL4AI_API_TOKEN references and use correct endpoints in Docker example scripts (#1015)
2025-08-09 19:37:22 +05:30
docker_hooks_examples.py
Update Docker hooks and Webhook documents (#1557)
2025-10-22 22:34:19 +08:00
docker_python_rest_api.py
feat(crawler): optimize single URL handling and add performance comparison
2025-03-13 22:15:15 +08:00
docker_python_sdk.py
refactor(config): enhance serialization and config handling
2025-02-19 17:23:25 +08:00
docker_webhook_example.py
Release/v0.7.6 (#1556)
2025-10-22 20:41:06 +08:00
extraction_strategies_examples.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
full_page_screenshot_and_pdf_export.md
Add contributor and docs for force_viewport_screenshot feature
2026-02-01 01:10:20 +00:00
hello_world_undetected.py
feat: add stealth mode and enhance undetected browser support
2025-07-17 16:59:10 +08:00
hello_world.py
feat(crawler): add session management and view-source support
2025-05-08 17:13:35 +08:00
hooks_example.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
identity_based_browsing.py
feat(profiles): add CLI command for crawling with browser profiles
2025-03-02 21:33:33 +08:00
language_support_example.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
link_head_extraction_example.py
feat: cleanup unused code and enhance documentation for v0.7.1
2025-07-17 11:35:16 +02:00
llm_extraction_openai_pricing.py
refactor: Update LLM extraction example with the updated structure
2025-06-12 12:23:03 +02:00
llm_markdown_generator.py
feat(browser): add standalone CDP browser launch and lxml extraction strategy
2025-03-07 20:55:56 +08:00
llm_table_extraction_example.py
refactor: Update LLMTableExtraction examples and tests
2025-08-15 18:47:31 +08:00
network_console_capture_example.py
Fixed capturing console messages for case the url is the local file. Update docker configuration (work in progress)
2025-04-10 23:22:38 +08:00
prefetch_two_phase_crawl.py
Add examples for deep crawl crash recovery and prefetch mode in documentation
2026-01-14 12:58:44 +01:00
proxy_rotation_demo.py
feat(proxy): add proxy rotation strategy
2025-02-09 18:49:10 +08:00
quickstart_examples_set_1.py
refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling
2025-04-15 22:27:18 +08:00
quickstart_examples_set_2.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
quickstart.ipynb
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
quickstart.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
README_BUILTIN_BROWSER.md
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
regex_extraction_quickstart.py
feat(extraction): add RegexExtractionStrategy for pattern-based extraction
2025-05-02 21:15:24 +08:00
research_assistant.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
rest_call.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
sample_ecommerce.html
Push async version last changes for merge to main branch
2024-09-24 20:52:08 +08:00
scraping_strategies_performance.py
Squashed commit of the following:
2025-08-04 19:02:01 +08:00
serp_api_project_11_feb.py
Release prep (#749)
2025-02-28 19:53:35 +08:00
session_id_example.py
feat(crawler): add session management and view-source support
2025-05-08 17:13:35 +08:00
shadow_dom_crawling.py
Add Shadow DOM flattening and reorder js_code execution pipeline
2026-02-18 06:43:00 +00:00
simple_anti_bot_examples.py
docs: add simple anti-bot examples
2025-07-17 17:05:35 +08:00
ssl_example.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
stealth_mode_example.py
feat: add stealth mode and enhance undetected browser support
2025-07-17 16:59:10 +08:00
stealth_mode_quick_start.py
feat: add stealth mode and enhance undetected browser support
2025-07-17 16:59:10 +08:00
stealth_test_simple.py
feat: add stealth mode and enhance undetected browser support
2025-07-17 16:59:10 +08:00
storage_state_tutorial.md
Implement new async crawler features and stability updates
2024-12-10 17:55:29 +08:00
summarize_page.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
table_extraction_example.py
feat: 🚀 Introduce revolutionary LLMTableExtraction with intelligent chunking for massive tables
2025-08-14 18:21:24 +08:00
tutorial_dynamic_clicks.md
Add PDF & screenshot functionality, new tutorial
2024-12-10 20:10:39 +08:00
tutorial_v0.5.py
Update all documentation to import extraction strategies directly from crawl4ai.
2025-06-10 18:08:27 +08:00
undetected_simple_demo.py
feat: add stealth mode and enhance undetected browser support
2025-07-17 16:59:10 +08:00
use_geo_location.py
feat(browser): add geolocation, locale and timezone support
2025-04-21 23:20:59 +08:00
virtual_scroll_example.py
feat: Add virtual scroll support for modern web scraping
2025-06-29 20:41:37 +08:00