UncleCode 4d7f91b378 refactor(user-agent): improve user agent generation system ...

Redesign user agent generation to be more modular and reliable:
- Add abstract base class UAGen for user agent generation
- Implement ValidUAGenerator using fake-useragent library
- Add OnlineUAGenerator for fetching real-world user agents
- Update browser configurations to use new UA generation system
- Improve client hints generation

This change makes the user agent system more maintainable and provides better real-world user agent coverage.

2025-01-25 21:16:39 +08:00

..

html2text

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

js_snippet

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

__init__.py

refactor(core): improve type hints and remove unused file

2025-01-23 18:53:22 +08:00

__version__.py

refactor(core): improve type hints and remove unused file

2025-01-23 18:53:22 +08:00

async_configs.py

refactor(user-agent): improve user agent generation system

2025-01-25 21:16:39 +08:00

async_crawler_strategy.py

refactor(user-agent): improve user agent generation system

2025-01-25 21:16:39 +08:00

async_database.py

refactor(browser): improve browser path management

2025-01-17 22:14:37 +08:00

async_dispatcher_.py

feat(dispatcher): add streaming support for URL processing

2025-01-19 14:03:34 +08:00

async_dispatcher.py

feat(dispatcher): add streaming support for URL processing

2025-01-19 14:03:34 +08:00

async_logger.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

async_webcrawler.py

refactor(models): rename final_url to redirected_url for consistency

2025-01-22 17:14:24 +08:00

cache_context.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

chunking_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

cli.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

config.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

content_filter_strategy.py

feat(content-filter): add LLMContentFilter for intelligent markdown generation

2025-01-18 19:31:07 +08:00

content_scraping_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

crawler_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

database.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

docs_manager.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

extraction_strategy.py

feat(extraction): add LLM-powered schema generation utility

2025-01-20 17:28:00 +08:00

install.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

llmtxt.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

markdown_generation_strategy.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

migrations.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

model_loader.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

models.py

refactor(models): rename final_url to redirected_url for consistency

2025-01-22 17:14:24 +08:00

prompts.py

feat(extraction): add LLM-powered schema generation utility

2025-01-20 17:28:00 +08:00

ssl_certificate.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

user_agent_generator.py

refactor(user-agent): improve user agent generation system

2025-01-25 21:16:39 +08:00

utils.py

feat(robots): add robots.txt compliance support

2025-01-21 17:54:13 +08:00

version_manager.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

web_crawler.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00