refactor(core): reorganize project structure and remove legacy code

Major reorganization of the project structure:
- Moved legacy synchronous crawler code to legacy folder
- Removed deprecated CLI and docs manager
- Consolidated version manager into utils.py
- Added CrawlerHub to __init__.py exports
- Fixed type hints in async_webcrawler.py
- Fixed minor bugs in chunking and crawler strategies

BREAKING CHANGE: Removed synchronous WebCrawler, CLI, and docs management functionality. Users should migrate to AsyncWebCrawler.
This commit is contained in:
UncleCode
2025-01-30 19:35:06 +08:00
parent 31938fb922
commit f81712eb91
23 changed files with 425 additions and 4 deletions

View File

@@ -0,0 +1,17 @@
# example_usageexample_usageexample_usage# example_usage.py
import asyncio
from crawl4ai.crawlers import get_crawler
async def main():
# Get the registered crawler
example_crawler = get_crawler("example_site.content")
# Crawl example.com
result = await example_crawler(url="https://example.com")
print(result)
if __name__ == "__main__":
asyncio.run(main())