BREAKING CHANGE: Table extraction now uses Strategy Design Pattern
This epic commit introduces a game-changing approach to table extraction in Crawl4AI:
✨ NEW FEATURES:
- LLMTableExtraction: AI-powered extraction for complex HTML tables with rowspan/colspan
- Smart Chunking: Automatically splits massive tables into optimal chunks at row boundaries
- Parallel Processing: Processes multiple chunks simultaneously for blazing-fast extraction
- Intelligent Merging: Seamlessly combines chunk results into complete tables
- Header Preservation: Each chunk maintains context with original headers
- Auto-retry Logic: Built-in resilience with configurable retry attempts
🏗️ ARCHITECTURE:
- Strategy Design Pattern for pluggable table extraction strategies
- ThreadPoolExecutor for concurrent chunk processing
- Token-based chunking with configurable thresholds
- Handles tables without headers gracefully
⚡ PERFORMANCE:
- Process 1000+ row tables without timeout
- Parallel processing with up to 5 concurrent chunks
- Smart token estimation prevents LLM context overflow
- Optimized for providers like Groq for massive tables
🔧 CONFIGURATION:
- enable_chunking: Auto-handle large tables (default: True)
- chunk_token_threshold: When to split (default: 3000 tokens)
- min_rows_per_chunk: Meaningful chunk sizes (default: 10)
- max_parallel_chunks: Concurrent processing (default: 5)
📚 BACKWARD COMPATIBILITY:
- Existing code continues to work unchanged
- DefaultTableExtraction remains the default strategy
- Progressive enhancement approach
This is the future of web table extraction - handling everything from simple tables to massive, complex data grids with merged cells and nested structures. The chunking is completely transparent to users while providing unprecedented scalability.
- fix handling of special keys in Windows msvcrt implementation
- Guard against UnicodeDecodeError from multi-byte key sequences
- Filter out non-printable characters and control sequences
- Add error handling to prevent coroutine crashes
- Add unit test to verify keyboard input handling
Key changes:
- Safe UTF-8 decoding with try/except for special keys
- Skip non-printable and multi-byte character sequences
- Add broad exception handling in keyboard listener
Test runs on Windows only due to msvcrt dependency.
This commit introduces platform-specific handling for the 'q' key press to quit the browser profiler, ensuring compatibility with both Windows and Unix-like systems. It also adds a check to see if the browser process has already exited, terminating the input listener if so.
- Implemented `msvcrt` for Windows to capture keyboard input without requiring a newline.
- Retained `termios`, `tty`, and `select` for Unix-like systems.
- Added a check for browser process termination to gracefully exit the input listener.
- Updated logger messages to use colored output for better user experience.
BREAKING CHANGE: Table extraction now uses Strategy Design Pattern
This epic commit introduces a game-changing approach to table extraction in Crawl4AI:
✨ NEW FEATURES:
- LLMTableExtraction: AI-powered extraction for complex HTML tables with rowspan/colspan
- Smart Chunking: Automatically splits massive tables into optimal chunks at row boundaries
- Parallel Processing: Processes multiple chunks simultaneously for blazing-fast extraction
- Intelligent Merging: Seamlessly combines chunk results into complete tables
- Header Preservation: Each chunk maintains context with original headers
- Auto-retry Logic: Built-in resilience with configurable retry attempts
🏗️ ARCHITECTURE:
- Strategy Design Pattern for pluggable table extraction strategies
- ThreadPoolExecutor for concurrent chunk processing
- Token-based chunking with configurable thresholds
- Handles tables without headers gracefully
⚡ PERFORMANCE:
- Process 1000+ row tables without timeout
- Parallel processing with up to 5 concurrent chunks
- Smart token estimation prevents LLM context overflow
- Optimized for providers like Groq for massive tables
🔧 CONFIGURATION:
- enable_chunking: Auto-handle large tables (default: True)
- chunk_token_threshold: When to split (default: 3000 tokens)
- min_rows_per_chunk: Meaningful chunk sizes (default: 10)
- max_parallel_chunks: Concurrent processing (default: 5)
📚 BACKWARD COMPATIBILITY:
- Existing code continues to work unchanged
- DefaultTableExtraction remains the default strategy
- Progressive enhancement approach
This is the future of web table extraction - handling everything from simple tables to massive, complex data grids with merged cells and nested structures. The chunking is completely transparent to users while providing unprecedented scalability.
- Wrap all AsyncUrlSeeder usage with async context managers
- Update URL seeding adventure example to use "sitemap+cc" source, focus on course posts, and add stream=True parameter to fix runtime error
The MemoryAdaptiveDispatcher was processing tasks sequentially despite
max_session_permit > 1 due to fetching only one task per event loop iteration.
This particularly affected raw:// URLs which complete in microseconds.
Changes:
- Replace single task fetch with greedy slot filling using get_nowait()
- Fill all available slots (up to max_session_permit) immediately
- Break on empty queue instead of waiting with timeout
This ensures proper parallelization for all task types, especially
ultra-fast operations like raw HTML processing.
- Add raw HTML URL validation alongside http/https checks
- Fix URL preprocessing logic to handle raw: and raw:// prefixes
- Update error message and add comprehensive test cases
- Remove deprecated API token authentication from all Docker examples
- Fix async job endpoints: /crawl -> /crawl/job for submission, /task/{id} -> /crawl/job/{id} for polling
- Fix sync endpoint: /crawl_sync -> /crawl (synchronous)
- Remove non-existent /crawl_direct endpoint
- Update request format to use new structure with browser_config and crawler_config
- Fix response handling for both async and sync calls
- Update extraction strategy format to use proper nested structure
- Add Ollama connectivity check before running tests
- Update test schemas and selectors for current website structures
This makes the Docker examples work out-of-the-box with the current API structure.
- Introduced a new section in SPONSORS.md to recognize the first 50 sponsors as Founding Sponsors.
- Updated README-first.md to include comprehensive project details, features, installation instructions, and advanced usage examples.
- Highlighted the recent version 0.7.0 release with new features and improvements.
- Added a sponsorship program with tiered benefits and a mission statement to promote data democratization.
- Add FUNDING.yml to enable sponsor button
- Add sponsor section to README with tier overview
- Create SPONSORS.md for sponsor recognition
- Set up 4 tiers: Believer, Builder, Growing Team, Data Infrastructure Partner
- Extract base href from <head><base> tag using XPath in _process_element method
- Use base URL as the primary URL for link normalization when present
- Add error handling with logging for malformed or problematic base tags
- Maintain backward compatibility when no base tag is present
- Add test to verify the functionality of the base tag extraction.
- Update Table-to-DataFrame Extraction example in README.md
- Replace old method of accessing tables via result.media directly with result.tables in the documentation
- Remove tables section from links & media page.
- Add tables section to crawler result page.
- Support LLM_PROVIDER env var to override default provider (openai/gpt-4o-mini)
- Add optional 'provider' parameter to API endpoints for per-request overrides
- Implement provider validation to ensure API keys exist
- Update documentation and examples with new configuration options
Closes the need to hardcode providers in config.yml
commit 2def6524cdacb69c72760bf55a41089257c0bb07
Author: ntohidi <nasrin@kidocode.com>
Date: Mon Aug 4 18:59:10 2025 +0800
refactor: consolidate WebScrapingStrategy to use LXML implementation only
BREAKING CHANGE: None - full backward compatibility maintained
This commit simplifies the content scraping architecture by removing the
redundant BeautifulSoup-based WebScrapingStrategy implementation and making
it an alias for LXMLWebScrapingStrategy.
Changes:
- Remove ~1000 lines of BeautifulSoup-based WebScrapingStrategy code
- Make WebScrapingStrategy an alias for LXMLWebScrapingStrategy
- Update LXMLWebScrapingStrategy to inherit directly from ContentScrapingStrategy
- Add required methods (scrap, ascrap, process_element, _log) to LXMLWebScrapingStrategy
- Maintain 100% backward compatibility - existing code continues to work
Code changes:
- crawl4ai/content_scraping_strategy.py: Remove WebScrapingStrategy class, add alias
- crawl4ai/async_configs.py: Remove WebScrapingStrategy from imports
- crawl4ai/__init__.py: Update imports to show alias relationship
- crawl4ai/types.py: Update type definitions
- crawl4ai/legacy/web_crawler.py: Update import to use alias
- tests/async/test_content_scraper_strategy.py: Update to use LXMLWebScrapingStrategy
- docs/examples/scraping_strategies_performance.py: Update to use single strategy
Documentation updates:
- docs/md_v2/core/content-selection.md: Update scraping modes section
- docs/md_v2/migration/webscraping-strategy-migration.md: Add migration guide
- CHANGELOG.md: Document the refactoring under [Unreleased]
Benefits:
- 10-20x faster HTML parsing for large documents
- Reduced memory usage and simplified codebase
- Consistent parsing behavior
- No migration required for existing users
All existing code using WebScrapingStrategy continues to work without
modification, while benefiting from LXML's superior performance.
