feat(monitor): add real-time monitoring dashboard with Redis persistence
Complete observability solution for production deployments with terminal-style UI.
**Backend Implementation:**
- `monitor.py`: Stats manager tracking requests, browsers, errors, timeline data
- `monitor_routes.py`: REST API endpoints for all monitor functionality
- GET /monitor/health - System health snapshot
- GET /monitor/requests - Active & completed requests
- GET /monitor/browsers - Browser pool details
- GET /monitor/endpoints/stats - Aggregated endpoint analytics
- GET /monitor/timeline - Time-series data (memory, requests, browsers)
- GET /monitor/logs/{janitor,errors} - Event logs
- POST /monitor/actions/{cleanup,kill_browser,restart_browser} - Control actions
- POST /monitor/stats/reset - Reset counters
- Redis persistence for endpoint stats (survives restart)
- Timeline tracking (5min window, 5s resolution, 60 data points)
**Frontend Dashboard** (`/dashboard`):
- **System Health Bar**: CPU%, Memory%, Network I/O, Uptime
- **Pool Status**: Live counts (permanent/hot/cold browsers + memory)
- **Live Activity Tabs**:
- Requests: Active (realtime) + recent completed (last 100)
- Browsers: Detailed table with actions (kill/restart)
- Janitor: Cleanup event log with timestamps
- Errors: Recent errors with stack traces
- **Endpoint Analytics**: Count, avg latency, success%, pool hit%
- **Resource Timeline**: SVG charts (memory/requests/browsers) with terminal aesthetics
- **Control Actions**: Force cleanup, restart permanent, reset stats
- **Auto-refresh**: 5s polling (toggleable)
**Integration:**
- Janitor events tracked (close_cold, close_hot, promote)
- Crawler pool promotion events logged
- Timeline updater background task (5s interval)
- Lifespan hooks for monitor initialization
**UI Design:**
- Terminal vibe matching Crawl4AI theme
- Dark background, cyan/pink accents, monospace font
- Neon glow effects on charts
- Responsive layout, hover interactions
- Cross-navigation: Playground ↔ Monitor
**Key Features:**
- Zero-config: Works out of the box with existing Redis
- Real-time visibility into pool efficiency
- Manual browser management (kill/restart)
- Historical data persistence
- DevOps-friendly UX
Routes:
- API: `/monitor/*` (backend endpoints)
- UI: `/dashboard` (static HTML)
This commit is contained in:
@@ -16,6 +16,7 @@ from fastapi import Request, Depends
|
||||
from fastapi.responses import FileResponse
|
||||
import base64
|
||||
import re
|
||||
import logging
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
from api import (
|
||||
handle_markdown_request, handle_llm_qa,
|
||||
@@ -112,15 +113,40 @@ AsyncWebCrawler.arun = capped_arun
|
||||
@asynccontextmanager
|
||||
async def lifespan(_: FastAPI):
|
||||
from crawler_pool import init_permanent
|
||||
from monitor import MonitorStats
|
||||
import monitor as monitor_module
|
||||
|
||||
# Initialize monitor
|
||||
monitor_module.monitor_stats = MonitorStats(redis)
|
||||
await monitor_module.monitor_stats.load_from_redis()
|
||||
|
||||
# Initialize browser pool
|
||||
await init_permanent(BrowserConfig(
|
||||
extra_args=config["crawler"]["browser"].get("extra_args", []),
|
||||
**config["crawler"]["browser"].get("kwargs", {}),
|
||||
))
|
||||
|
||||
# Start background tasks
|
||||
app.state.janitor = asyncio.create_task(janitor())
|
||||
app.state.timeline_updater = asyncio.create_task(_timeline_updater())
|
||||
|
||||
yield
|
||||
|
||||
# Cleanup
|
||||
app.state.janitor.cancel()
|
||||
app.state.timeline_updater.cancel()
|
||||
await close_all()
|
||||
|
||||
async def _timeline_updater():
|
||||
"""Update timeline data every 5 seconds."""
|
||||
from monitor import get_monitor
|
||||
while True:
|
||||
await asyncio.sleep(5)
|
||||
try:
|
||||
await get_monitor().update_timeline()
|
||||
except Exception as e:
|
||||
logger.warning(f"Timeline update error: {e}")
|
||||
|
||||
# ───────────────────── FastAPI instance ──────────────────────
|
||||
app = FastAPI(
|
||||
title=config["app"]["title"],
|
||||
@@ -138,6 +164,16 @@ app.mount(
|
||||
name="play",
|
||||
)
|
||||
|
||||
# ── static monitor dashboard ────────────────────────────────
|
||||
MONITOR_DIR = pathlib.Path(__file__).parent / "static" / "monitor"
|
||||
if not MONITOR_DIR.exists():
|
||||
raise RuntimeError(f"Monitor assets not found at {MONITOR_DIR}")
|
||||
app.mount(
|
||||
"/dashboard",
|
||||
StaticFiles(directory=MONITOR_DIR, html=True),
|
||||
name="monitor_ui",
|
||||
)
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def root():
|
||||
@@ -221,6 +257,12 @@ def _safe_eval_config(expr: str) -> dict:
|
||||
# ── job router ──────────────────────────────────────────────
|
||||
app.include_router(init_job_router(redis, config, token_dep))
|
||||
|
||||
# ── monitor router ──────────────────────────────────────────
|
||||
from monitor_routes import router as monitor_router
|
||||
app.include_router(monitor_router)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ──────────────────────── Endpoints ──────────────────────────
|
||||
@app.post("/token")
|
||||
async def get_token(req: TokenRequest):
|
||||
|
||||
Reference in New Issue
Block a user