NETBOX_INTEGRATION.md

# NetBox Integration Guide

Overgrowth now uses NetBox/Nautobot as the authoritative Source of Truth for network designs. This replaces YAML files with a proper IPAM/DCIM system used by enterprises worldwide.

## Why NetBox?

- **Industry Standard**: Used by Netflix, DigitalOcean, Dropbox, and thousands of organizations
- **Rich Data Model**: Devices, racks, cables, VLANs, IPs, circuits, power, and more
- **API-First**: RESTful API for automation
- **Extensible**: Custom fields, webhooks, plugins
- **Multi-Vendor**: Cisco, Juniper, Arista, HPE, Dell - all supported
- **Open Source**: Free and actively maintained

## Quick Start (Local Development)

### 1. Start NetBox with Docker Compose

```bash
cd overgrowth

# Copy environment template
cp netbox.env.example netbox.env

# Start NetBox stack
docker-compose -f docker-compose-netbox.yml up -d

# Wait for services to start (30-60 seconds)
docker-compose -f docker-compose-netbox.yml logs -f netbox
```

### 2. Access NetBox

Open http://localhost:8000 in your browser

- **Username**: admin
- **Password**: admin (from netbox.env)
- **API Token**: 0123456789abcdef0123456789abcdef01234567

### 3. Configure Overgrowth

```bash
# Set environment variables
export NETBOX_URL="http://localhost:8000"
export NETBOX_TOKEN="0123456789abcdef0123456789abcdef01234567"

# Test connection
python test_netbox.py
```

### 4. Run Pipeline with NetBox

```bash
# Pipeline will automatically sync to NetBox
python app.py

# Or test from CLI
python -c "
from agent.pipeline_engine import OvergrowthPipeline, NetworkIntent
p = OvergrowthPipeline(use_netbox=True)
intent = NetworkIntent(
    description='Coffee shop network',
    business_requirements=['Guest WiFi', 'POS systems'],
    constraints=['Budget under $5000']
)
model = p.stage2_generate_sot(intent)
"
```

## Production Deployment

### Option 1: Self-Hosted NetBox

Follow official docs: https://docs.netbox.dev/en/stable/installation/

Requirements:
- PostgreSQL 12+
- Redis 6.2+
- Python 3.8+
- 2GB RAM minimum

### Option 2: Nautobot Cloud

Enterprise-supported hosted NetBox alternative:
- https://www.networktocode.com/nautobot/
- Free tier available
- Fully compatible with NetBox API

Configuration:
```bash
export NAUTOBOT_URL="https://yourinstance.nautobot.cloud"
export NAUTOBOT_TOKEN="your_api_token_here"
```

### Option 3: NetBox Cloud

Official hosted NetBox service:
- https://netboxlabs.com/netbox-cloud/
- 30-day free trial
- Managed infrastructure

## NetBox Configuration

### Initial Setup

1. **Create API Token** (in NetBox UI)
   - User menu → API Tokens → Add
   - Copy token to environment variable

2. **Enable Webhooks** (optional)
   - Admin → Webhooks → Add
   - URL: `http://your-overgrowth-server/webhook/netbox`
   - Events: `dcim.device`, `ipam.vlan`, `ipam.prefix`

3. **Import Manufacturers**
   ```bash
   # NetBox has built-in device library
   # Or import custom manufacturers via API
   ```

### Recommended Plugins

Add to `netbox.env`:
```
PLUGINS=['netbox_topology_views', 'netbox_acls', 'netbox_bgp']
```

Install plugins:
```bash
docker-compose -f docker-compose-netbox.yml exec netbox pip install \
  netbox-topology-views \
  netbox-acls \
  netbox-bgp
```

## Architecture

```
┌─────────────────────┐
│   Overgrowth UI     │
│   (Gradio App)      │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│  Pipeline Engine    │
│  - Consultation     │
│  - LLM Design       │
│  - BOM Generation   │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐      ┌──────────────────┐
│  NetBox Client      │◄────►│ NetBox/Nautobot  │
│  - Sites            │      │  - PostgreSQL    │
│  - Devices          │      │  - Redis         │
│  - VLANs/IPs        │      │  - REST API      │
│  - Sync Logic       │      └──────────────────┘
└─────────────────────┘
           │
           ▼
┌─────────────────────┐
│  GNS3 / Real Gear   │
│  (Deployment)       │
└─────────────────────┘
```

## Data Flow

1. **Consultation** → User provides requirements via natural language
2. **LLM Design** → Claude generates VLANs, subnets, device roles
3. **NetBox Sync** → Design written to NetBox via API
4. **YAML Backup** → Local YAML file created for version control
5. **BOM Generation** → Read from NetBox to create shopping list
6. **GNS3 Deploy** → Read from NetBox to generate configs
7. **Validation** → Compare actual network state vs NetBox

## API Usage Examples

### Python SDK (pynetbox)

```python
from agent.netbox_client import NetBoxClient

# Initialize client
nb = NetBoxClient()

# Create a site
site = nb.create_site(
    name="Office HQ",
    description="Main office location"
)

# Create VLANs
vlan = nb.create_vlan(
    vid=10,
    name="Management",
    site="Office HQ",
    description="Network management VLAN"
)

# Create IP prefix
prefix = nb.create_prefix(
    prefix="10.0.10.0/24",
    description="Management subnet",
    site="Office HQ",
    vlan=10
)

# Sync entire network model
network = {
    "name": "retail-store",
    "vlans": [...],
    "subnets": [...],
    "devices": [...]
}
summary = nb.sync_network_model(network)
print(f"Created {summary['devices']} devices, {summary['vlans']} VLANs")
```

### Direct REST API

```bash
# Get all devices
curl -H "Authorization: Token 0123456789abcdef0123456789abcdef01234567" \
     http://localhost:8000/api/dcim/devices/

# Create a VLAN
curl -X POST \
     -H "Authorization: Token 0123456789abcdef0123456789abcdef01234567" \
     -H "Content-Type: application/json" \
     -d '{"vid": 20, "name": "Users"}' \
     http://localhost:8000/api/ipam/vlans/
```

## Troubleshooting

### NetBox not connecting

```bash
# Check NetBox is running
docker-compose -f docker-compose-netbox.yml ps

# View logs
docker-compose -f docker-compose-netbox.yml logs netbox

# Restart services
docker-compose -f docker-compose-netbox.yml restart
```

### Import errors

```bash
# Install pynetbox
pip install pynetbox>=7.0.0

# Verify installation
python -c "import pynetbox; print(pynetbox.__version__)"
```

### API token issues

```bash
# Regenerate token in NetBox UI
# Update environment variable
export NETBOX_TOKEN="new_token_here"

# Test connection
python test_netbox.py
```

## Fallback Mode

If NetBox is unavailable, Overgrowth automatically falls back to YAML files:

```python
pipeline = OvergrowthPipeline(use_netbox=True)
# If NetBox connection fails, pipeline.use_netbox becomes False
# All operations continue using local YAML files
```

## Migration from YAML

To migrate existing YAML network models to NetBox:

```python
from agent.pipeline_engine import OvergrowthPipeline, NetworkModel

# Load existing YAML
pipeline = OvergrowthPipeline(use_netbox=True)
yaml_content = open("infra/network_model.yaml").read()
model = NetworkModel.from_yaml(yaml_content)

# Sync to NetBox
pipeline.netbox.sync_network_model(model.to_dict())
```

## Next Steps

- [ ] Configure NetBox webhooks for real-time updates
- [ ] Set up Batfish for pre-deployment validation (Stage 0)
- [ ] Integrate SuzieQ for drift detection (Stage 7b)
- [ ] Add GitOps workflow for NetBox changes
- [ ] Enable digital twin simulation (Stage 6b)

## Resources

- NetBox Docs: https://docs.netbox.dev/
- Nautobot Docs: https://docs.nautobot.com/
- pynetbox SDK: https://pynetbox.readthedocs.io/
- NetBox Community: https://netbox.dev/community/
- Overgrowth Issues: https://huggingface.co/spaces/MCP-1st-Birthday/overgrowth/discussions