Aravind Karnam 8a7d29ce85 updated some comments and removed content type checking functionality from core as it's implemented as a filter

2024-10-16 15:59:37 +05:30

..

filters

Created scaffolding for Scraper as per the plan. Implemented the ascrape method in bfs_scraper_strategy

2024-09-09 13:13:34 +05:30

scorers

Created scaffolding for Scraper as per the plan. Implemented the ascrape method in bfs_scraper_strategy

2024-09-09 13:13:34 +05:30

__init__.py

Parallel processing with retry on failure with exponential backoff - Simplified URL validation and normalisation - respecting Robots.txt

2024-09-19 12:34:12 +05:30

async_web_scraper.py

1. Introduced a bool flag to ascrape method to switch between sequential and concurrent processing

2024-10-03 11:17:11 +05:30

bfs_scraper_strategy.py

updated some comments and removed content type checking functionality from core as it's implemented as a filter

2024-10-16 15:59:37 +05:30

models.py

Parallel processing with retry on failure with exponential backoff - Simplified URL validation and normalisation - respecting Robots.txt

2024-09-19 12:34:12 +05:30

scraper_strategy.py

Parallel processing with retry on failure with exponential backoff - Simplified URL validation and normalisation - respecting Robots.txt

2024-09-19 12:34:12 +05:30