crawl4ai/deploy/installer/README.md

Crawl4AI Node Manager (cnode) - Installation & Distribution

This directory contains the standalone cnode package and installation scripts for managing Crawl4AI Docker server instances.

Overview

cnode is a fast, lightweight CLI tool for managing Crawl4AI Docker servers. It provides:

• One-command deployment with automatic scaling
• Single container for development (N=1)
• Docker Swarm for production with built-in load balancing (N>1)
• Docker Compose + Nginx as fallback (N>1)

Directory Structure

deploy/installer/
├── README.md                 # This file
├── cnode_pkg/                # Standalone Python package
│   ├── __init__.py           # Package marker
│   ├── cli.py                # CLI interface (commands)
│   ├── server_manager.py     # Docker orchestration logic
│   └── requirements.txt      # Python dependencies
├── install-cnode.sh          # Local installation script
├── install-cnode.sh                 # Remote installation script (for users)
└── releases/                 # Release artifacts for distribution

Installation

For Users (Remote Installation)

Users can install cnode directly from the web:

# Install from GitHub/website
curl -sSL https://crawl4ai.com/install-cnode.sh | bash

# Or with wget
wget -qO- https://crawl4ai.com/install-cnode.sh | bash

For Local Testing

Test the installation locally:

cd deploy/installer
./install-cnode.sh

Package Contents

cnode_pkg/ - Python Package

This is a self-contained Python package with:

• cli.py: Click-based CLI with all commands (start, stop, status, scale, logs, cleanup, restart)
• server_manager.py: Core Docker orchestration logic
• requirements.txt: Dependencies (click, rich, anyio, pyyaml)
• __init__.py: Package initialization

Installation Script

install-cnode.sh does the following:

1. Checks for Python 3.8+ and pip
2. Checks for Docker (warns if not found)
3. Installs Python dependencies
4. Copies cnode_pkg/ to /usr/local/lib/cnode/
5. Creates wrapper script at /usr/local/bin/cnode
6. Verifies installation

Wrapper Script

Created at /usr/local/bin/cnode:

#!/usr/bin/env bash
set -e

# Find Python
if command -v python3 &> /dev/null; then
    PYTHON_CMD="python3"
elif command -v python &> /dev/null; then
    PYTHON_CMD="python"
else
    echo "Error: Python 3.8+ required" >&2
    exit 1
fi

# Run cnode
export PYTHONPATH="/usr/local/lib/cnode:$PYTHONPATH"
exec $PYTHON_CMD -m cnode_pkg.cli "$@"

Performance

Blazing Fast Startup:

• ~0.1 seconds to launch
• 49x faster than compiled binary alternatives
• Minimal overhead, maximum responsiveness

Requirements

User Requirements

• Python 3.8 or higher
• pip (Python package manager)
• Docker (for running servers)

Dependencies (Auto-installed)

• click >= 8.0.0 (CLI framework)
• rich >= 13.0.0 (Terminal formatting)
• anyio >= 3.0.0 (Async I/O)
• pyyaml >= 6.0.0 (YAML parsing)

Usage

After installation:

# Quick start
cnode start                    # Single container on port 11235
cnode start --replicas 5       # 5-replica cluster
cnode status                   # Check server status
cnode logs -f                  # Follow logs
cnode scale 10                 # Scale to 10 replicas
cnode stop                     # Stop server

# Get help
cnode --help
cnode start --help

Development Workflow

Making Changes

1. Edit source code in deploy/docker/:

   vim deploy/docker/cnode_cli.py
   vim deploy/docker/server_manager.py
   

2. Update package by copying to installer:

   # Copy CLI
   cp deploy/docker/cnode_cli.py deploy/installer/cnode_pkg/cli.py
   
   # Fix imports (deploy.docker → cnode_pkg)
   sed -i 's/from deploy\.docker\./from cnode_pkg./g' deploy/installer/cnode_pkg/cli.py
   
   # Copy server manager
   cp deploy/docker/server_manager.py deploy/installer/cnode_pkg/server_manager.py
   

3. Test locally:

   cd deploy/installer
   ./install-cnode.sh
   cnode --help
   

4. Commit both:

   git add deploy/docker/cnode_cli.py
   git add deploy/installer/cnode_pkg/cli.py
   git commit -m "Update cnode: [description]"
   

Creating a Release

1. Tag the release:

   git tag -a v1.0.0 -m "Release v1.0.0"
   git push origin v1.0.0
   

2. Package for distribution:

   cd deploy/installer
   tar -czf releases/cnode-v1.0.0.tar.gz cnode_pkg/ install-cnode.sh
   

3. Create GitHub release:

   gh release create v1.0.0 \
     releases/cnode-v1.0.0.tar.gz \
     --title "cnode v1.0.0" \
     --notes "Release notes here"
   

4. Update deployment script (if needed):

   • Update install-cnode.sh with new version/URL
   • Upload to hosting (e.g., https://crawl4ai.com/install-cnode.sh)

Deployment

Remote Installation Script

The install-cnode.sh script is meant to be hosted at a public URL for user installation:

# Upload to your server
scp install-cnode.sh user@crawl4ai.com:/var/www/html/install-cnode.sh

# Or use GitHub raw URL
https://raw.githubusercontent.com/unclecode/crawl4ai/main/deploy/installer/install-cnode.sh

Users can then install with:

curl -sSL https://crawl4ai.com/install-cnode.sh | bash

Backward Compatibility

The main Crawl4AI CLI (crwl) includes a redirect for backward compatibility:

# These work identically:
crwl server start --replicas 3
cnode start --replicas 3

# All subcommands redirect:
crwl server status  → cnode status
crwl server stop    → cnode stop
crwl server scale 5 → cnode scale 5
crwl server logs -f → cnode logs -f

This ensures existing scripts continue working while users migrate to cnode.

Uninstallation

To remove cnode:

# Remove command
sudo rm /usr/local/bin/cnode

# Remove package
sudo rm -rf /usr/local/lib/cnode

# (Optional) Uninstall dependencies
pip uninstall click rich anyio pyyaml

Troubleshooting

Python Not Found

# Install Python 3.8+
# macOS: brew install python3
# Ubuntu: sudo apt install python3 python3-pip
# RHEL/CentOS: sudo yum install python3 python3-pip

Permission Denied

# Run installer with sudo
sudo ./install-cnode.sh

# Or change install location
INSTALL_DIR=$HOME/.local/bin ./install-cnode.sh

Command Not Found After Install

# Add to PATH in ~/.bashrc or ~/.zshrc
export PATH="/usr/local/bin:$PATH"

# Reload shell
source ~/.bashrc  # or source ~/.zshrc

Dependencies Install Failed

# Install manually
pip install --user click rich anyio pyyaml

# Or with break-system-packages (if needed)
pip install --user --break-system-packages click rich anyio pyyaml

Docker Not Running

# macOS: Start Docker Desktop
# Linux: sudo systemctl start docker

# Check Docker
docker --version
docker ps

Architecture

Component Flow

User runs: cnode start
         ↓
/usr/local/bin/cnode (wrapper script)
         ↓
Finds python3 executable
         ↓
Sets PYTHONPATH=/usr/local/lib/cnode
         ↓
python3 -m cnode_pkg.cli start
         ↓
cli.py → start_cmd()
         ↓
server_manager.py → ServerManager.start()
         ↓
Docker orchestration (single/swarm/compose)
         ↓
Server running!

Why Python Wrapper vs Binary?

We chose a Python wrapper over compiled binaries (PyInstaller) because:

Metric	Python Wrapper	PyInstaller Binary
Startup time	0.1s	4.7s
Size	~50KB wrapper	8.8MB
Updates	Easy (just copy files)	Rebuild required
Dependencies	Python 3.8+	None
Platform	Any with Python	OS-specific builds

Since users running Crawl4AI already have Python, the wrapper is the clear winner.

Support

For issues or questions:

• GitHub Issues: https://github.com/unclecode/crawl4ai/issues
• Documentation: https://docs.crawl4ai.com
• Discord: https://discord.gg/crawl4ai

Version History

• v1.0.0: Initial release with Python wrapper approach
  • Fast startup (~0.1s)
  • Supports single container, Docker Swarm, and Compose modes
  • Auto-scaling and load balancing
  • Real-time monitoring and logs