Redesign user agent generation to be more modular and reliable:
- Add abstract base class UAGen for user agent generation
- Implement ValidUAGenerator using fake-useragent library
- Add OnlineUAGenerator for fetching real-world user agents
- Update browser configurations to use new UA generation system
- Improve client hints generation
This change makes the user agent system more maintainable and provides better real-world user agent coverage.
- Added examples for Amazon product data extraction methods
- Updated configuration options and enhance documentation
- Minor refactoring for improved performance and readability
- Cleaned up version control settings.
- Enhanced error handling in async crawler.
- Added flexible options in Markdown generation.
- Updated user agent settings for improved reliability.
- Reflected changes in documentation and examples.
- Added a new UserAgentGenerator class for generating random User-Agents.
- Integrated User-Agent generation in AsyncPlaywrightCrawlerStrategy for randomization.
- Enhanced HTTP headers with generated Client Hints.
