ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

New async database manager and migration support

Browse Source 
- Introduced AsyncDatabaseManager for async DB management.
  - Added migration feature to transition to file-based storage.
  - Enhanced web crawler with improved caching logic.
  - Updated requirements and setup for async processing.
This commit is contained in:
UncleCode
2024-11-16 14:54:41 +08:00
parent ae7ebc0bd8
commit d0014c6793
8 changed files with 685 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
crawl4ai/config.py
View File

@@ -52,4 +52,6 @@ SOCIAL_MEDIA_DOMAINS = [
# If image is in the first half of the total images extracted from the page
IMAGE_SCORE_THRESHOLD = 2
MAX_METRICS_HISTORY = 1000
MAX_METRICS_HISTORY = 1000
NEED_MIGRATION = True