UncleCode 18e8227dfb feat(crawler): add console message capture functionality ...

Add ability to capture browser console messages during crawling:
- Implement _capture_console_messages method to collect console logs
- Update crawl method to support console message capture
- Modify browser_manager page creation to accept full CrawlerRunConfig
- Fix request failure text formatting

This enhancement allows debugging and monitoring of JavaScript console output during crawling operations.

2025-04-10 23:26:09 +08:00

..

components

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

crawlers

feat(schema): improve HTML preprocessing for schema generation

2025-03-12 22:40:46 +08:00

deep_crawling

feat(filters): add reverse option to URLPatternFilter

2025-03-08 18:54:41 +08:00

html2text

2025 feb alpha 1 (#685)

2025-02-19 14:13:17 +08:00

js_snippet

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

legacy

refactor(core): reorganize project structure and remove legacy code

2025-01-30 19:35:06 +08:00

processors/pdf

Release prep (#749)

2025-02-28 19:53:35 +08:00

__init__.py

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

__version__.py

chore(version): bump version to 0.5.0.post8 and update post-installation setup

2025-03-25 21:56:49 +08:00

async_configs.py

feat(crawler): add network request and console message capturing

2025-04-10 16:03:48 +08:00

async_crawler_strategy.py

feat(crawler): add console message capture functionality

2025-04-10 23:26:09 +08:00

async_database.py

refactor: clean up imports and improve JSON schema generation instructions

2025-03-18 18:53:34 +08:00

async_dispatcher.py

feat(monitor): add real-time crawler monitoring system with memory management

2025-03-12 19:05:24 +08:00

async_logger.py

feat(config): set DefaultMarkdownGenerator as the default markdown generator in CrawlerRunConfig

2025-04-03 20:34:19 +08:00

async_webcrawler.py

feat(crawler): add network request and console message capturing

2025-04-10 16:03:48 +08:00

browser_manager.py

fix(browser): add null check for crawlerRunConfig.url

2025-04-10 23:25:07 +08:00

browser_profiler.py

feat(browser): implement modular browser management system

2025-03-21 22:50:00 +08:00

cache_context.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

chunking_strategy.py

refactor(core): reorganize project structure and remove legacy code

2025-01-30 19:35:06 +08:00

cli.py

feat(cli): add output file option and integrate LXML web scraping strategy

2025-03-25 21:38:24 +08:00

config.py

feat(cli): add quick JSON extraction and global config management

2025-03-25 20:30:25 +08:00

content_filter_strategy.py

refactor(llm): rename LlmConfig to LLMConfig for consistency

2025-03-05 14:17:04 +08:00

content_scraping_strategy.py

feat(crawler): add MHTML capture functionality

2025-04-09 15:39:04 +08:00

docker_client.py

feat(docker): add JWT authentication and improve server architecture

2025-02-18 22:07:13 +08:00

extraction_strategy.py

feat(cli): add quick JSON extraction and global config management

2025-03-25 20:30:25 +08:00

hub.py

feat(logger): add abstract logger base class and file logger implementation

2025-02-23 21:23:41 +08:00

install.py

chore(version): bump version to 0.5.0.post8 and update post-installation setup

2025-03-25 21:56:49 +08:00

markdown_generation_strategy.py

fix(auth): make JWT authentication optional with fallback

2025-03-05 17:14:42 +08:00

migrations.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

model_loader.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

models.py

feat(crawler): add network request and console message capturing

2025-04-10 16:03:48 +08:00

prompts.py

feat(cli): add quick JSON extraction and global config management

2025-03-25 20:30:25 +08:00

proxy_strategy.py

refactor(proxy): consolidate proxy configuration handling

2025-03-07 23:14:11 +08:00

ssl_certificate.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

types.py

feat(schema): improve HTML preprocessing for schema generation

2025-03-12 22:40:46 +08:00

user_agent_generator.py

feat(browser): add BrowserProfiler class for identity-based browsing

2025-03-02 20:32:29 +08:00

utils.py

feat(cli): add quick JSON extraction and global config management

2025-03-25 20:30:25 +08:00