ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f89113377aa2e7ac40023976e63cb2d1d9a93255
crawl4ai/crawl4ai
History
Aravind Karnam f89113377a fix: Move adding of visited urls to the 'visited' set, when queueing the URLs instead of after dequeuing, this is to prevent duplicate crawls. https://github.com/unclecode/crawl4ai/issues/843
2025-03-21 13:44:57 +05:30
..
components
feat(monitor): add real-time crawler monitoring system with memory management
2025-03-12 19:05:24 +08:00
crawlers
feat(schema): improve HTML preprocessing for schema generation
2025-03-12 22:40:46 +08:00
deep_crawling
fix: Move adding of visited urls to the 'visited' set, when queueing the URLs instead of after dequeuing, this is to prevent duplicate crawls. https://github.com/unclecode/crawl4ai/issues/843
2025-03-21 13:44:57 +05:30
html2text
2025 feb alpha 1 (#685)
2025-02-19 14:13:17 +08:00
js_snippet
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
legacy
refactor(core): reorganize project structure and remove legacy code
2025-01-30 19:35:06 +08:00
processors/pdf
Release prep (#749)
2025-02-28 19:53:35 +08:00
__init__.py
feat(monitor): add real-time crawler monitoring system with memory management
2025-03-12 19:05:24 +08:00
__version__.py
feat(deep-crawling): improve URL normalization and domain filtering
2025-03-06 22:45:57 +08:00
async_configs.py
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
async_crawler_strategy.py
feat(crawler): add experimental parameters support and optimize browser handling
2025-03-14 14:39:24 +08:00
async_database.py
refactor: clean up imports and improve JSON schema generation instructions
2025-03-18 18:53:34 +08:00
async_dispatcher.py
feat(monitor): add real-time crawler monitoring system with memory management
2025-03-12 19:05:24 +08:00
async_logger.py
fix: improve logged url readability by decoding encoded urls
2025-03-21 13:40:23 +05:30
async_webcrawler.py
fix: streamline url status logging via single entrypoint i.e. logger.url_status
2025-03-20 18:59:15 +05:30
browser_manager.py
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
browser_profiler.py
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
cache_context.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
chunking_strategy.py
refactor(core): reorganize project structure and remove legacy code
2025-01-30 19:35:06 +08:00
cli.py
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
config.py
feat(schema): improve HTML preprocessing for schema generation
2025-03-12 22:40:46 +08:00
content_filter_strategy.py
refactor(llm): rename LlmConfig to LLMConfig for consistency
2025-03-05 14:17:04 +08:00
content_scraping_strategy.py
chore: Raise an exception with clear messaging when body tag is missing in the fetched html. The message should warn users to add appropriate wait_for condition to wait until body tag is loaded into DOM.
2025-03-18 15:26:44 +05:30
docker_client.py
feat(docker): add JWT authentication and improve server architecture
2025-02-18 22:07:13 +08:00
extraction_strategy.py
refactor: clean up imports and improve JSON schema generation instructions
2025-03-18 18:53:34 +08:00
hub.py
feat(logger): add abstract logger base class and file logger implementation
2025-02-23 21:23:41 +08:00
install.py
feat(browser): add builtin browser management system
2025-03-20 12:13:59 +08:00
markdown_generation_strategy.py
fix(auth): make JWT authentication optional with fallback
2025-03-05 17:14:42 +08:00
migrations.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
model_loader.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
models.py
refactor: clean up imports and improve JSON schema generation instructions
2025-03-18 18:53:34 +08:00
prompts.py
2025 feb alpha 1 (#685)
2025-02-19 14:13:17 +08:00
proxy_strategy.py
refactor(proxy): consolidate proxy configuration handling
2025-03-07 23:14:11 +08:00
ssl_certificate.py
Apply Ruff Corrections
2025-01-13 19:19:58 +08:00
types.py
feat(schema): improve HTML preprocessing for schema generation
2025-03-12 22:40:46 +08:00
user_agent_generator.py
feat(browser): add BrowserProfiler class for identity-based browsing
2025-03-02 20:32:29 +08:00
utils.py
fix: remove trailing slash when the path is empty. This is causing dupicate crawls
2025-03-21 13:41:31 +05:30