ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(models): make model fields optional with default values

Browse Source 
Make fields in MediaItem and Link models optional with default values to prevent validation errors when data is incomplete. Also expose BaseDispatcher in __init__ and fix markdown field handling in database manager.

BREAKING CHANGE: MediaItem and Link model fields are now optional with default values which may affect existing code expecting required fields.
This commit is contained in:
UncleCode
2025-01-15 22:58:14 +08:00
parent 20c027b79c
commit 9d694da939
6 changed files with 681 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
crawl4ai/async_database.py
View File

@@ -14,8 +14,8 @@ from .async_logger import AsyncLogger
from .utils import get_error_context, create_box_message
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# logging.basicConfig(level=logging.INFO)
# logger = logging.getLogger(__name__)
base_directory = DB_PATH = os.path.join(
os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()), ".crawl4ai"
@@ -333,7 +333,11 @@ class AsyncDatabaseManager:
json.loads(row_dict[field]) if row_dict[field] else {}
)
except json.JSONDecodeError:
row_dict[field] = {}
# Very UGLY, never mention it to me please
if field == "markdown" and isinstance(row_dict[field], str):
row_dict[field] = row_dict[field]
else:
row_dict[field] = {}
if isinstance(row_dict["markdown"], Dict):
row_dict["markdown_v2"] = row_dict["markdown"]