6a728cbe5b
- Add playwright-stealth integration with enable_stealth parameter in BrowserConfig - Merge undetected browser strategy into main async_crawler_strategy.py using adapter pattern - Add browser adapters (BrowserAdapter, PlaywrightAdapter, UndetectedAdapter) for flexible browser switching - Update install.py to install both playwright and patchright browsers automatically - Add comprehensive documentation for anti-bot features (stealth mode + undetected browser) - Create examples demonstrating stealth mode usage and comparison tests - Update pyproject.toml and requirements.txt with patchright>=1.49.0 and other dependencies - Remove duplicate/unused dependencies (alphashape, cssselect, pyperclip, shapely, selenium) - Add dependency checker tool in tests/check_dependencies.py Breaking changes: None - all existing functionality preserved 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
34 lines
656 B
Plaintext
34 lines
656 B
Plaintext
# Note: These requirements are also specified in pyproject.toml
|
|
# This file is kept for development environment setup and compatibility
|
|
aiofiles>=24.1.0
|
|
aiohttp>=3.11.11
|
|
aiosqlite~=0.20
|
|
anyio>=4.0.0
|
|
lxml~=5.3
|
|
litellm>=1.53.1
|
|
numpy>=1.26.0,<3
|
|
pillow>=10.4
|
|
playwright>=1.49.0
|
|
patchright>=1.49.0
|
|
python-dotenv~=1.0
|
|
requests~=2.26
|
|
beautifulsoup4~=4.12
|
|
tf-playwright-stealth>=1.1.0
|
|
xxhash~=3.4
|
|
rank-bm25~=0.2
|
|
colorama~=0.4
|
|
snowballstemmer~=2.2
|
|
pydantic>=2.10
|
|
pyOpenSSL>=24.3.0
|
|
psutil>=6.1.1
|
|
PyYAML>=6.0
|
|
nltk>=3.9.1
|
|
rich>=13.9.4
|
|
chardet>=5.2.0
|
|
brotli>=1.1.0
|
|
httpx[http2]>=0.27.2
|
|
sentence-transformers>=2.2.0
|
|
|
|
fake-useragent>=2.2.0
|
|
pdf2image>=1.17.0
|
|
PyPDF2>=3.0.1 |