d11a83c232
• Refactored `crawler_strategy.py` to handle exceptions and improve error messages • Improved `get_content_of_website_optimized` function in `utils.py` for better performance • Updated `utils.py` with latest changes • Migrated to `ChromeDriverManager` for resolving Chrome driver download issues
1.5 KiB
1.5 KiB
Changelog
[0.2.71] 2024-06-26
• Refactored crawler_strategy.py to handle exceptions and improve error messages
• Improved get_content_of_website_optimized function in utils.py for better performance
• Updated utils.py with latest changes
• Migrated to ChromeDriverManager for resolving Chrome driver download issues
[0.2.71] - 2024-06-25
Fixed
- Speed up twice the extraction function.
[0.2.6] - 2024-06-22
Fixed
- Fix issue #19: Update Dockerfile to ensure compatibility across multiple platforms.
[0.2.5] - 2024-06-18
Added
- Added five important hooks to the crawler:
- on_driver_created: Called when the driver is ready for initializations.
- before_get_url: Called right before Selenium fetches the URL.
- after_get_url: Called after Selenium fetches the URL.
- before_return_html: Called when the data is parsed and ready.
- on_user_agent_updated: Called when the user changes the user_agent, causing the driver to reinitialize.
- Added an example in
quickstart.pyin the example folder under the docs. - Enhancement issue #24: Replaced inline HTML tags (e.g., DEL, INS, SUB, ABBR) with textual format for better context handling in LLM.
- Maintaining the semantic context of inline tags (e.g., abbreviation, DEL, INS) for improved LLM-friendliness.
- Updated Dockerfile to ensure compatibility across multiple platforms (Hopefully!).
[0.2.4] - 2024-06-17
Fixed
- Fix issue #22: Use MD5 hash for caching HTML files to handle long URLs