Aravind Karnam ce7fce4b16 1. Moved to asyncio.wait instead of gather so that results can be yeilded just as they are ready, rather than in batches ...

2. Moved the visted.add(url), to before the task is put in queue rather than after the crawl is completed. This makes sure that  duplicate crawls doesn't happen when same URL is found at different depth and that get's queued too because the crawl is not yet completed and visted set is not updated.
3. Named the yield_results attribute to stream instead. Since that seems to be popularly used in all other AI libraries for intermediate results.

2024-10-17 12:25:17 +05:30

..

models/onnx

- Add ONNX embedding model for CPU devices, Update the similarithy threshold, improve the embedding speed.

2024-05-19 22:30:10 +08:00

scraper

1. Moved to asyncio.wait instead of gather so that results can be yeilded just as they are ready, rather than in batches

2024-10-17 12:25:17 +05:30

__init__.py

feat(v0.3.6): Add screenshot capture, delayed content, and custom timeouts

2024-10-12 13:42:42 +08:00

async_crawler_strategy.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

async_database.py

feat(v0.3.6): Add screenshot capture, delayed content, and custom timeouts

2024-10-12 13:42:42 +08:00

async_webcrawler.py

feat: Enhance crawling control and LLM extraction flexibility

2024-10-12 14:48:22 +08:00

chunking_strategy.py

chore: Fix typo in chunking_strategy.py and crawler_strategy.py

2024-07-19 17:40:31 +08:00

config.py

refactor: Update image description minimum word threshold in get_content_of_website_optimized

2024-08-02 14:53:11 +08:00

content_scrapping_strategy.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

crawler_strategy.py

Push async version last changes for merge to main branch

2024-09-24 20:52:08 +08:00

database.py

## [v0.2.74] - 2024-07-08

2024-07-08 16:33:25 +08:00

extraction_strategy.py

feat: Enhance crawling control and LLM extraction flexibility

2024-10-12 14:48:22 +08:00

model_loader.py

Push async version last changes for merge to main branch

2024-09-24 20:52:08 +08:00

models.py

Enhance AsyncWebCrawler with smart waiting and screenshot capabilities

2024-10-02 17:34:56 +08:00

prompts.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

train.py

Remove dependency on Spacy model.

2024-05-17 15:08:03 +08:00

utils.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00

web_crawler.back.py

vital: Right now, only raw html is retrived from datbase, therefore, css selector and other filter will be executed every time.

2024-06-08 18:37:40 +08:00

web_crawler.py

feat: Enhance crawler flexibility and LLM extraction capabilities

2024-10-14 21:03:28 +08:00