Profiling/monitoring :Add interactive monitoring dashboard and integration tests for monitoring endpoints

- Implemented an interactive monitoring dashboard in `demo_monitoring_dashboard.py` for real-time statistics, profiling session management, and system resource monitoring.
- Created a quick test script `test_monitoring_quick.py` to verify the functionality of monitoring endpoints.
- Developed comprehensive integration tests in `test_monitoring_endpoints.py` covering health checks, statistics, profiling sessions, and real-time streaming.
- Added error handling and user-friendly output for better usability in the dashboard.

deploy/docker/server.py

@@ -87,7 +87,7 @@ from prometheus_fastapi_instrumentator import Instrumentator
 from pydantic import BaseModel, Field
 from rank_bm25 import BM25Okapi
 from redis import asyncio as aioredis
-from routers import adaptive, dispatchers, scripts
+from routers import adaptive, dispatchers, scripts, monitoring
 from schemas import (
     CrawlRequest,
     CrawlRequestWithHooks,
@@ -297,6 +297,7 @@ app.include_router(init_job_router(redis, config, token_dep))
 app.include_router(adaptive.router)
 app.include_router(dispatchers.router)
 app.include_router(scripts.router)
+app.include_router(monitoring.router)
 
 
 # ──────────────────────── Endpoints ──────────────────────────