UncleCode ae376f15fb docs(extraction): add clarifying comments for CSS selector behavior ...

Add explanatory comments to JsonCssExtractionStrategy._get_elements() method to clarify that it returns all matching elements using select() instead of select_one(). This helps developers understand the method's behavior and its difference from single element selection.

Removed trailing whitespace at end of file.

2025-01-05 19:39:15 +08:00

..

html2text

Implement new async crawler features and stability updates

2024-12-10 17:55:29 +08:00

js_snippet

Implement new async crawler features and stability updates

2024-12-10 17:55:29 +08:00

__init__.py

Enhance AsyncWebCrawler and related configurations

2024-12-12 19:35:09 +08:00

__version__.py

fix(extraction): JsonCss selector and crawler improvements

2025-01-05 19:26:46 +08:00

async_configs.py

refactor(crawler): optimize response handling and default settings

2025-01-01 19:39:02 +08:00

async_crawler_strategy.py

fix(extraction): JsonCss selector and crawler improvements

2025-01-05 19:26:46 +08:00

async_database.py

Commit Message:

2024-12-29 20:05:18 +08:00

async_logger.py

Enhance crawler features and improve documentation

2024-12-16 18:52:51 +08:00

async_webcrawler.py

refactor(crawler): optimize response handling and default settings

2025-01-01 19:39:02 +08:00

cache_context.py

Commit Message:

2024-12-29 20:05:18 +08:00

chunking_strategy.py

Commit Message:

2024-12-29 20:05:18 +08:00

cli.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

config.py

Enhance crawler features and improve documentation

2024-12-16 18:52:51 +08:00

content_filter_strategy.py

Update 0.4.24 walkthrough

2024-12-31 21:01:46 +08:00

content_scraping_strategy.py

fix(extraction): JsonCss selector and crawler improvements

2025-01-05 19:26:46 +08:00

crawler_strategy.py

fix: crawler strategy exception handling and fixes (#271)

2024-11-20 20:30:25 +08:00

database.py

feat(config): Adding a configurable way of setting the cache directory for constrained environments

2024-11-12 14:52:51 -07:00

docs_manager.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

extraction_strategy.bak.py

Commit Message:

2024-12-29 20:05:18 +08:00

extraction_strategy.py

docs(extraction): add clarifying comments for CSS selector behavior

2025-01-05 19:39:15 +08:00

install.py

refactor(install): use chromium as default browser

2025-01-01 17:19:54 +08:00

llmtxt.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

markdown_generation_strategy.py

refactor(crawler): optimize response handling and default settings

2025-01-01 19:39:02 +08:00

migrations.py

Enhance installation and migration processes

2024-11-29 18:48:44 +08:00

model_loader.py

feat(config): Adding a configurable way of setting the cache directory for constrained environments

2024-11-12 14:52:51 -07:00

models.py

Commit Message:

2024-12-26 15:17:07 +08:00

prompts.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

ssl_certificate.py

Commit Message:

2024-12-29 20:05:18 +08:00

user_agent_generator.py

Commit Message:

2024-12-29 20:05:18 +08:00

utils.py

fix(extraction): JsonCss selector and crawler improvements

2025-01-05 19:26:46 +08:00

utils.scraping.py

Refactored web scraping components

2024-12-05 22:33:47 +08:00

version_manager.py

feat(docker): add Docker Compose configurations for local and hub deployment; enhance GPU support checks in Dockerfile

2024-11-18 21:00:06 +08:00

web_crawler.py

In this commit, we introduce the new concept of MakrdownGenerationStrategy, which allows us to expand our future strategies to generate better markdown. Right now, we generate raw markdown as we were doing before. We have a new algorithm for fitting markdown based on BM25, and now we add the ability to refine markdown into a citation form. Our links will be extracted and replaced by a citation reference number, and then we will have reference sections at the very end; we add all the links with the descriptions. This format is more suitable for large language models. In case we don't need to pass links, we can reduce the size of the markdown significantly and also attach the list of references as a separate file to a large language model. This commit contains changes for this direction.

2024-11-21 18:21:43 +08:00