ntohidi 2b3b728dcd fix(metadata): improve title extraction with fallbacks for edge cases. REF #995 ...

Some pages include a <title> tag in HTML but lxml fails to parse it due to unusual structure.
Added fallback logic using .find() and OpenGraph/Twitter meta tags to ensure reliable title extraction.

2025-05-28 10:17:50 +02:00

..

components

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

crawlers

feat(schema): improve HTML preprocessing for schema generation

2025-03-12 22:40:46 +08:00

deep_crawling

fix: link normalisation in BestFirstStrategy

2025-03-21 17:34:13 +05:30

html2text

2025 feb alpha 1 (#685)

2025-02-19 14:13:17 +08:00

js_snippet

fix: Add a nominal wait time for remove overlay elements since it's already controllable through delay_before_return_html

2025-04-14 12:39:05 +05:30

legacy

refactor(core): reorganize project structure and remove legacy code

2025-01-30 19:35:06 +08:00

processors/pdf

fix(pdf): add timeout to PDF downloads to prevent hanging (#1141)

2025-05-23 16:05:44 +08:00

__init__.py

Refactor content scraping strategies: comment out WebScrapingStrategy references and update to use LXMLWebScrapingStrategy across multiple files. Bring WebScrapingStrategy methods to LXMLWebScrapingStrategy

2025-05-27 17:32:45 +02:00

__version__.py

Set version to 0.6.3

2025-05-12 21:20:10 +08:00

async_configs.py

Refactor content scraping strategies: comment out WebScrapingStrategy references and update to use LXMLWebScrapingStrategy across multiple files. Bring WebScrapingStrategy methods to LXMLWebScrapingStrategy

2025-05-27 17:32:45 +02:00

async_crawler_strategy.py

feat(browser): improve browser session management and profile handling

2025-05-21 20:23:17 +08:00

async_database.py

WIP: logger migriate to rich

2025-04-11 00:44:43 +08:00

async_dispatcher.py

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

async_logger.py

Merge branch 'pr-971' into merge-pr971

2025-05-01 18:57:28 +08:00

async_webcrawler.py

feat(extraction): add RegexExtractionStrategy for pattern-based extraction

2025-05-02 21:15:24 +08:00

browser_manager.py

Fix temperature typo and enhance LinkedIn extraction with Colab support

2025-05-25 16:47:12 +08:00

browser_profiler.py

feat(browser): improve browser session management and profile handling

2025-05-21 20:23:17 +08:00

cache_context.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

chunking_strategy.py

refactor(core): reorganize project structure and remove legacy code

2025-01-30 19:35:06 +08:00

cli.py

feat(cli): add output file option and integrate LXML web scraping strategy

2025-03-25 21:38:24 +08:00

config.py

refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling

2025-04-15 22:27:18 +08:00

content_filter_strategy.py

refactor(logger): Apply the Enumeration for color

2025-05-01 17:04:44 +08:00

content_scraping_strategy.py

fix(metadata): improve title extraction with fallbacks for edge cases. REF #995

2025-05-28 10:17:50 +02:00

docker_client.py

feat(docker): add JWT authentication and improve server architecture

2025-02-18 22:07:13 +08:00

extraction_strategy.py

Fix temperature typo and enhance LinkedIn extraction with Colab support

2025-05-25 16:47:12 +08:00

hub.py

feat(logger): add abstract logger base class and file logger implementation

2025-02-23 21:23:41 +08:00

install.py

feat(docs): enhance documentation UI with ToC and GitHub stats

2025-04-14 20:46:32 +08:00

markdown_generation_strategy.py

feat(markdown): add content source selection for markdown generation

2025-04-17 20:13:53 +08:00

migrations.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

model_loader.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

models.py

feat(extraction): add RegexExtractionStrategy for pattern-based extraction

2025-05-02 21:15:24 +08:00

prompts.py

feat(cli): add quick JSON extraction and global config management

2025-03-25 20:30:25 +08:00

proxy_strategy.py

refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling

2025-04-15 22:27:18 +08:00

ssl_certificate.py

refactor(proxy): move ProxyConfig to async_configs and improve LLM token handling

2025-04-15 22:27:18 +08:00

types.py

Refactor content scraping strategies: comment out WebScrapingStrategy references and update to use LXMLWebScrapingStrategy across multiple files. Bring WebScrapingStrategy methods to LXMLWebScrapingStrategy

2025-05-27 17:32:45 +02:00

user_agent_generator.py

feat(browser): add BrowserProfiler class for identity-based browsing

2025-03-02 20:32:29 +08:00

utils.py

fix(metadata): improve title extraction with fallbacks for edge cases. REF #995

2025-05-28 10:17:50 +02:00