crawl4ai

html2text

Implement new async crawler features and stability updates

2024-12-10 17:55:29 +08:00

js_snippet

Implement new async crawler features and stability updates

2024-12-10 17:55:29 +08:00

__init__.py

Enhance AsyncWebCrawler and related configurations

2024-12-12 19:35:09 +08:00

__version__.py

Update version file

2025-01-01 19:42:35 +08:00

async_configs.py

refactor(crawler): optimize response handling and default settings

2025-01-01 19:39:02 +08:00

async_crawler_strategy.py

Commit Message:

2024-12-29 20:05:18 +08:00

async_database.py

Commit Message:

2024-12-29 20:05:18 +08:00

async_logger.py

Enhance crawler features and improve documentation

2024-12-16 18:52:51 +08:00

async_webcrawler.py

refactor(crawler): optimize response handling and default settings

2025-01-01 19:39:02 +08:00

cache_context.py

Commit Message:

2024-12-29 20:05:18 +08:00

chunking_strategy.py

Commit Message:

2024-12-29 20:05:18 +08:00

cli.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

config.py

Enhance crawler features and improve documentation

2024-12-16 18:52:51 +08:00

content_filter_strategy.py

Update 0.4.24 walkthrough

2024-12-31 21:01:46 +08:00

content_scraping_strategy.py

Commit Message:

2024-12-29 20:05:18 +08:00

crawler_strategy.py

fix: crawler strategy exception handling and fixes (#271 )

2024-11-20 20:30:25 +08:00

database.py

feat(config): Adding a configurable way of setting the cache directory for constrained environments

2024-11-12 14:52:51 -07:00

docs_manager.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

extraction_strategy.bak.py

Commit Message:

2024-12-29 20:05:18 +08:00

extraction_strategy.py

Commit Message:

2024-12-29 20:05:18 +08:00

install.py

refactor(install): use chromium as default browser

2025-01-01 17:19:54 +08:00

llmtxt.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

markdown_generation_strategy.py

refactor(crawler): optimize response handling and default settings

2025-01-01 19:39:02 +08:00

migrations.py

Enhance installation and migration processes

2024-11-29 18:48:44 +08:00

model_loader.py

feat(config): Adding a configurable way of setting the cache directory for constrained environments

2024-11-12 14:52:51 -07:00

models.py

Commit Message:

2024-12-26 15:17:07 +08:00

prompts.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

ssl_certificate.py

Commit Message:

2024-12-29 20:05:18 +08:00

user_agent_generator.py

Commit Message:

2024-12-29 20:05:18 +08:00

utils.py

Commit Message:

2024-12-29 20:05:18 +08:00

utils.scraping.py

Refactored web scraping components

2024-12-05 22:33:47 +08:00

version_manager.py

feat(docker): add Docker Compose configurations for local and hub deployment; enhance GPU support checks in Dockerfile

2024-11-18 21:00:06 +08:00

web_crawler.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00