UncleCode dbb751c8f0 In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00

..

sample_wikipedia.html

perf(crawler): major performance improvements & raw HTML support

2024-11-13 19:40:40 +08:00

test_async_doanloader.py

- In this commit, the library is updated to process file downloads. Users can now specify a download folder and trigger the download process via JavaScript or other means, with all files being saved. The list of downloaded files will also be added to the crowd result object.

2024-11-14 22:50:59 +08:00

test_basic_crawling.py

Enhance AsyncWebCrawler with smart waiting and screenshot capabilities

2024-10-02 17:34:56 +08:00

test_caching.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_chunking_and_extraction_strategies.py

Update .gitignore to ignore temporary and test directories

2024-09-26 15:09:49 +08:00

test_content_extraction.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_content_filter.py

test(content_filter): add comprehensive tests for BM25ContentFilter functionality

2024-11-15 18:11:11 +08:00

test_content_scraper_strategy.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00

test_crawler_strategy.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_database_operations.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_edge_cases.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_error_handling.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_markdown_genertor.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00

test_parameters_and_options.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_performance.py

Add Async Version, JsonCss Extrator

2024-09-03 01:27:00 +08:00

test_screenshot.py

Enhance AsyncWebCrawler with smart waiting and screenshot capabilities

2024-10-02 17:34:56 +08:00