Aravind Karnam 513d008de5 feat: Merge reviews from unclecode for scorers and filters & Remove the robots.txt compliance from scraper since that will be now handled by crawler

2025-01-27 11:54:10 +05:30

..

html2text

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

js_snippet

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

scraper

feat: Merge reviews from unclecode for scorers and filters & Remove the robots.txt compliance from scraper since that will be now handled by crawler

2025-01-27 11:54:10 +05:30

__init__.py

feat(content-filter): add LLMContentFilter for intelligent markdown generation

2025-01-18 19:31:07 +08:00

__version__.py

refactor(docs): reorganize documentation structure and update styles

2025-01-07 20:49:50 +08:00

async_configs.py

feat(config): add streaming support and config cloning

2025-01-19 17:51:47 +08:00

async_crawler_strategy.py

feat(crawler): add URL redirection tracking

2025-01-19 19:53:38 +08:00

async_database.py

refactor(browser): improve browser path management

2025-01-17 22:14:37 +08:00

async_dispatcher_.py

feat(dispatcher): add streaming support for URL processing

2025-01-19 14:03:34 +08:00

async_dispatcher.py

feat(dispatcher): add streaming support for URL processing

2025-01-19 14:03:34 +08:00

async_logger.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

async_webcrawler.py

feat(crawler): add URL redirection tracking

2025-01-19 19:53:38 +08:00

cache_context.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

chunking_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

cli.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

config.py

Refactor: Removed all scheduling logic from scraper. From now scraper expects arun_many to handle all scheduling. Scraper will only do traversal, validations, compliance checks, URL filtering and scoring etc. Reformatted some of the scraper files with Black code formatter

2025-01-21 17:49:51 +05:30

content_filter_strategy.py

feat(content-filter): add LLMContentFilter for intelligent markdown generation

2025-01-18 19:31:07 +08:00

content_scraping_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

crawler_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

database.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

docs_manager.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

extraction_strategy.py

feat(extraction): add LLM-powered schema generation utility

2025-01-20 17:28:00 +08:00

install.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

llmtxt.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

markdown_generation_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

migrations.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

model_loader.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

models.py

feat(crawler): add URL redirection tracking

2025-01-19 19:53:38 +08:00

prompts.py

feat(extraction): add LLM-powered schema generation utility

2025-01-20 17:28:00 +08:00

ssl_certificate.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

user_agent_generator.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

utils.py

refactor(browser): improve browser path management

2025-01-17 22:14:37 +08:00

utils.scraping.py

Refactored web scraping components

2024-12-05 22:33:47 +08:00

version_manager.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

web_crawler.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00