crawl4ai

Enhance Markdown generation and external content control

2024-10-20 18:56:58 +08:00

__init__.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00

__version__.py

feat: update version to 0.3.741 and enhance content filtering with heuristic strategy. Fixing the issue that when the past HTML to BM25 content filter does not have any HTML elements.

2024-11-23 19:45:41 +08:00

async_crawler_strategy.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00

async_database.py

Merge remote-tracking branch 'origin/main' into 0.3.74

2024-11-18 21:15:04 +08:00

async_logger.py

feat(docker): add Docker Compose configurations for local and hub deployment; enhance GPU support checks in Dockerfile

2024-11-18 21:00:06 +08:00

async_webcrawler.py

Merge branch 'main' of https://github.com/unclecode/crawl4ai

2024-11-22 19:56:56 +08:00

cache_context.py

feat(cache): introduce CacheMode and CacheContext for enhanced caching behavior

2024-11-17 15:30:56 +08:00

chunking_strategy.py

[v0.3.71] Enhance chunking strategies and improve overall performance

2024-10-19 18:36:59 +08:00

config.py

feat(docs): update README for version 0.3.74 with new features and improvements

2024-11-17 21:09:26 +08:00

content_filter_strategy.py

feat: update version to 0.3.741 and enhance content filtering with heuristic strategy. Fixing the issue that when the past HTML to BM25 content filter does not have any HTML elements.

2024-11-23 19:45:41 +08:00

content_scraping_strategy.py

feat: update version to 0.3.741 and enhance content filtering with heuristic strategy. Fixing the issue that when the past HTML to BM25 content filter does not have any HTML elements.

2024-11-23 19:45:41 +08:00

crawler_strategy.py

fix: crawler strategy exception handling and fixes (#271 )

2024-11-20 20:30:25 +08:00

database.py

feat(config): Adding a configurable way of setting the cache directory for constrained environments

2024-11-12 14:52:51 -07:00

extraction_strategy.py

feat: enhance crawler with overlay removal and improved screenshot capabilities

2024-10-24 20:22:47 +08:00

markdown_generation_strategy.py

feat: enhance Markdown generation to include fit_html attribute

2024-11-22 18:47:17 +08:00

migrations.py

New async database manager and migration support

2024-11-16 14:54:41 +08:00

model_loader.py

feat(config): Adding a configurable way of setting the cache directory for constrained environments

2024-11-12 14:52:51 -07:00

models.py

feat: enhance Markdown generation to include fit_html attribute

2024-11-22 18:47:17 +08:00

prompts.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

tools.py

feat: enhance image processing capabilities

2024-11-22 16:00:17 +08:00

utils.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00

version_manager.py

feat(docker): add Docker Compose configurations for local and hub deployment; enhance GPU support checks in Dockerfile

2024-11-18 21:00:06 +08:00

web_crawler.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00