build: modernize package configuration with pyproject.toml

- Add pyproject.toml for PEP 517 build system support
- Configure dependencies, scripts, and metadata in pyproject.toml
- Set Python requirement to >=3.9 and add support up to 3.13
- Keep setup.py for backwards compatibility
- Move package dependencies and entry points to pyproject.toml

setup.py

@@ -3,6 +3,8 @@ import os
 from pathlib import Path
 import shutil
 
+# Note: Most configuration is now in pyproject.toml
+# This setup.py is kept for backwards compatibility
 
 # Create the .crawl4ai folder in the user's home directory if it doesn't exist
 # If the folder already exists, remove the cache folder