SUZIEQ_INTEGRATION.md

# SuzieQ Integration - Multi-Vendor Drift Detection

## Overview

SuzieQ is an open-source network observability framework that provides multi-vendor state collection, topology discovery, and historical analysis. Overgrowth integrates SuzieQ for **continuous drift detection** - comparing actual network state against intended state (NetBox SoT) and automatically generating remediation plans.

### What is Configuration Drift?

Configuration drift occurs when the actual network state diverges from the intended state defined in your source of truth (SoT). Common causes:
- Manual changes made directly on devices
- Failed automation runs leaving partial configs
- Hardware failures requiring emergency workarounds
- Shadow IT adding unauthorized VLANs/subnets
- Config erosion over time

### Why SuzieQ?

| Feature | SuzieQ | Traditional Monitoring |
|---------|--------|----------------------|
| **Multi-vendor** | ✅ Arista, Cisco, Juniper, Cumulus, etc. | ❌ Vendor-specific |
| **Agentless** | ✅ SSH-based collection | ❌ Requires agents |
| **Historical data** | ✅ Parquet files for time-travel | ❌ Limited retention |
| **Topology discovery** | ✅ LLDP/CDP-based | ❌ Manual mapping |
| **Open source** | ✅ Apache 2.0 | ❌ Commercial |

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Overgrowth Pipeline                       │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │   NetBox    │→ │  SuzieQ      │→ │ Drift Detection  │   │
│  │   (SoT)     │  │  Collector   │  │  & Remediation   │   │
│  └─────────────┘  └──────────────┘  └──────────────────┘   │
└─────────────────────────────────────────────────────────────┘
         ↓                  ↓                      ↓
    Intended State    Actual State         Drift Analysis
    
┌────────────────────────────────────────────────────────────┐
│  Stage 7: Observability - Collect actual network state     │
│  Stage 7b: Drift Detection - Compare actual vs intended    │
│  Stage 8: Validation - Auto-remediate approved changes     │
└────────────────────────────────────────────────────────────┘
```

## What SuzieQ Detects

### 1. Configuration Mismatches
- Device hostname changes
- Management IP changes
- Unexpected device roles (leaf acting as spine)

### 2. VLAN Drift
- **Missing VLANs**: Intended in SoT but not on device
- **Extra VLANs**: Present on device but not in SoT
- VLAN name mismatches

### 3. IP Address Conflicts
- Duplicate IPs across devices
- IP mismatches vs NetBox IPAM
- Gateway conflicts

### 4. Interface State Drift
- Interfaces expected UP but actually DOWN
- Interfaces expected DOWN but actually UP
- Description mismatches

### 5. Routing Issues
- BGP neighbor states
- OSPF adjacency problems
- Route count anomalies

## Usage Examples

### Basic Drift Detection

```python
from agent.pipeline_engine import OvergrowthPipeline
from agent.network_model import NetworkModel

# Create pipeline with SuzieQ enabled
pipeline = OvergrowthPipeline()

# Your network model (from NetBox or YAML)
model = NetworkModel(...)

# Stage 7: Collect actual state
obs_results = pipeline.stage7_observability(model)
print(f"Collected state from {obs_results['collection']['devices_polled']} devices")

# Stage 7b: Detect drift
drift_results = pipeline.stage7b_drift_detection(model)

if drift_results['drift_detected']:
    print(f"⚠️  Drift detected! Score: {drift_results['drift_score']:.2f}")
    print(f"Issues found:")
    print(f"  - Config mismatches: {drift_results['summary']['config_mismatches']}")
    print(f"  - Missing VLANs: {drift_results['summary']['missing_vlans']}")
    print(f"  - Interface issues: {drift_results['summary']['interfaces_down']}")
else:
    print("✓ No drift - network matches SoT")
```

### Auto-Remediation

```python
# Stage 8: Validate and auto-remediate
validation = pipeline.stage8_validation(model)

compliance = validation['compliance_report']
print(f"Compliance Status: {compliance['status']}")
print(f"Drift Score: {compliance['drift_score']:.2f}")

if 'remediation' in validation:
    print(f"Applied {validation['remediation']['applied']} automatic fixes")
    print(f"Skipped {validation['remediation']['skipped']} (require manual approval)")
```

### Direct SuzieQ Client Usage

```python
from agent.suzieq_client import SuzieQClient

# Initialize client
suzieq = SuzieQClient(use_suzieq=True)

# Collect state from devices
devices = [
    {'name': 'leaf-01', 'ip': '10.0.0.11', 'username': 'admin', 'password': 'admin'},
    {'name': 'spine-01', 'ip': '10.0.0.1', 'username': 'admin', 'password': 'admin'}
]

collection = suzieq.collect_network_state(devices)
print(f"Collected from {collection['devices_polled']} devices")

# Get topology
topology = suzieq.get_topology()
print(f"Discovered {len(topology['nodes'])} nodes")
print(f"Found {len(topology['edges'])} LLDP/CDP connections")

# Get VLAN summary
vlans = suzieq.get_vlan_summary()
for device, vlan_list in vlans.items():
    print(f"{device}: {vlan_list}")

# Detect drift
intended_state = {
    'devices': [...],
    'vlans': [...],
    'subnets': [...]
}

drift = suzieq.detect_drift(intended_state)

if drift.has_drift:
    print(f"Drift Score: {drift.drift_score:.2f}")
    print(f"Missing VLANs: {len(drift.missing_vlans)}")
    print(f"Extra VLANs: {len(drift.extra_vlans)}")
    
    # Generate remediation plan
    plan = suzieq.generate_remediation_plan(drift)
    
    for action in plan:
        status = "AUTO-FIX" if action['auto_fix'] else "MANUAL"
        print(f"[{status}] {action['action']} on {action['device']}")
        print(f"  Commands: {action['commands']}")
    
    # Apply auto-approved fixes
    results = suzieq.apply_remediation(plan, auto_approve=True)
    print(f"Applied: {results['applied']}, Skipped: {results['skipped']}")
```

## Remediation Safety

### Auto-Fix vs Manual Approval

SuzieQ classifies remediation actions by safety:

| Action | Auto-Fix | Reason |
|--------|----------|--------|
| **Add missing VLAN** | ✅ Yes | Safe - doesn't disrupt traffic |
| **Remove extra VLAN** | ❌ No | Dangerous - could break connectivity |
| **Enable interface** | ❌ No | Dangerous - interface may be down intentionally |
| **Fix IP mismatch** | ✅ Yes | Safe - corrects IPAM drift |
| **Update descriptions** | ✅ Yes | Safe - cosmetic change |

### Approval Workflow

```python
# Get remediation plan
plan = suzieq.generate_remediation_plan(drift)

# Filter by auto-fix status
auto_fixes = [a for a in plan if a['auto_fix']]
manual_review = [a for a in plan if not a['auto_fix']]

print(f"Auto-fix ready: {len(auto_fixes)}")
print(f"Require approval: {len(manual_review)}")

# Apply only auto-approved
suzieq.apply_remediation(plan, auto_approve=True)

# For manual items, integrate with ticketing system
for action in manual_review:
    # Create Jira ticket, ServiceNow change request, etc.
    create_change_request(
        title=f"Fix {action['action']} on {action['device']}",
        commands=action['commands'],
        reason=action['reason']
    )
```

## Installation

### Option 1: Mock Mode (Default)

No installation required! Overgrowth includes mock SuzieQ for testing:

```python
suzieq = SuzieQClient(use_suzieq=True)
# Automatically uses mock mode if suzieq not installed
```

Mock mode simulates:
- State collection from devices
- Topology discovery
- Drift detection with heuristic rules
- Remediation plan generation

### Option 2: Real SuzieQ

Install SuzieQ for production use:

```bash
# Install SuzieQ
pip install suzieq

# Verify installation
suzieq-cli --help

# Create SuzieQ directory
mkdir -p ~/.suzieq/parquet
```

Configure SuzieQ inventory (`~/.suzieq/inventory.yml`):

```yaml
sources:
  - name: overgrowth
    hosts:
      - url: ssh://admin@10.0.0.11
        devtype: eos
        
      - url: ssh://admin@10.0.0.1
        devtype: eos
```

Start SuzieQ poller:

```bash
suzieq-poller -I ~/.suzieq/inventory.yml -d ~/.suzieq/parquet
```

## Configuration

### SuzieQ Client Options

```python
from pathlib import Path

# Custom data directory
suzieq = SuzieQClient(
    suzieq_dir=Path("/opt/suzieq/data"),
    use_suzieq=True
)

# Collect with custom namespace
suzieq.collect_network_state(
    devices=[...],
    namespace="production"  # vs "staging", "lab", etc.
)

# Query specific namespace
topology = suzieq.get_topology(namespace="production")
```

### Drift Tolerance

Adjust drift score threshold in `stage8_validation()`:

```python
# Default: 20% drift allowed
results['validation_passed'] = drift_score < 0.2

# Stricter: 10% drift
results['validation_passed'] = drift_score < 0.1

# Looser: 30% drift
results['validation_passed'] = drift_score < 0.3
```

Drift score calculation:
```
drift_score = total_drift_items / (devices_checked * expected_resources)

Examples:
- 0.0 = Perfect match
- 0.15 = Minor drift (2-3 VLANs missing)
- 0.5 = Moderate drift (half of config missing)
- 1.0 = Complete drift (nothing matches)
```

## Integration with Pipeline

### Stage 7: Observability

Collects actual network state via SuzieQ:
- Device inventory
- Interface states
- VLAN configurations
- IP addressing
- Routing protocol status
- Topology via LLDP/CDP

```python
obs_result = pipeline.stage7_observability(model)
# Returns: collection stats, topology, VLAN summary
```

### Stage 7b: Drift Detection

Compares actual vs intended (NetBox SoT):
- Config mismatches
- Missing/extra VLANs
- IP conflicts
- Interface state drift
- Routing issues

```python
drift_result = pipeline.stage7b_drift_detection(model)
# Returns: drift score, detailed findings, remediation plan
```

### Stage 8: Validation & Remediation

Validates network compliance and auto-remediates:
- Generates compliance report
- Applies auto-approved fixes
- Queues manual approval items
- Re-checks drift after remediation

```python
val_result = pipeline.stage8_validation(model)
# Returns: validation status, compliance report, remediation results
```

## Drift Detection Examples

### Example 1: Missing VLAN

**Intended (NetBox):**
```yaml
vlans:
  - id: 10
    name: Users
  - id: 20
    name: Servers
  - id: 99
    name: Management
```

**Actual (Device):**
```
show vlan brief
VLAN Name                             Status    Ports
---- -------------------------------- --------- ------
1    default                          active    
10   Users                            active    Et1-10
99   Management                       active    Et48
```

**Drift Detected:**
```json
{
  "missing_vlans": [{
    "device": "leaf-01",
    "vlan_id": 20,
    "vlan_name": "Servers",
    "severity": "ERROR"
  }]
}
```

**Remediation:**
```
! Auto-fix: Add missing VLAN
vlan 20
  name Servers
exit
```

### Example 2: Extra VLAN (Shadow IT)

**Intended:** VLANs 10, 20, 99  
**Actual:** VLANs 10, 20, 99, **666** (unauthorized)

**Drift Detected:**
```json
{
  "extra_vlans": [{
    "device": "leaf-01",
    "vlan_id": 666,
    "severity": "WARNING"
  }]
}
```

**Remediation:**
```
! Manual approval required - could disrupt traffic
no vlan 666
```

### Example 3: Interface Down

**Intended:** All uplinks should be UP  
**Actual:** Ethernet48 is DOWN

**Drift Detected:**
```json
{
  "interface_down": [{
    "device": "leaf-01",
    "interface": "Ethernet48",
    "expected_state": "up",
    "actual_state": "down",
    "severity": "WARNING"
  }]
}
```

**Remediation:**
```
! Manual approval - verify interface should be up
interface Ethernet48
  no shutdown
exit
```

## Troubleshooting

### Mock Mode vs Real Mode

Check if SuzieQ is installed:

```python
suzieq = SuzieQClient(use_suzieq=True)
print(f"Mock mode: {suzieq.mock_mode}")

# Expected output:
# WARNING: suzieq not installed - using mock mode
# Mock mode: True
```

### SuzieQ Not Collecting Data

1. **Check SSH connectivity:**
```bash
ssh admin@10.0.0.11
```

2. **Verify inventory:**
```bash
cat ~/.suzieq/inventory.yml
```

3. **Check poller logs:**
```bash
tail -f ~/.suzieq/suzieq-poller.log
```

4. **Test with CLI:**
```bash
suzieq-cli
device show
```

### Drift Detection Returns Empty

**Cause:** SuzieQ hasn't collected data yet

**Solution:** Run initial collection
```bash
# Start poller for 1 minute
suzieq-poller -I ~/.suzieq/inventory.yml -d ~/.suzieq/parquet --run-once
```

### Auto-Fix Not Working

**Cause:** `auto_approve=False` (default)

**Solution:**
```python
# Enable auto-approval
results = suzieq.apply_remediation(plan, auto_approve=True)

# Or apply manually via Netmiko
for action in plan:
    if action['auto_fix']:
        device = ConnectHandler(
            device_type='cisco_ios',
            host=action['device'],
            username='admin',
            password='admin'
        )
        device.send_config_set(action['commands'])
```

## Performance

### Collection Frequency

SuzieQ poller intervals:
- **Lab:** Every 1 minute (rapid testing)
- **Staging:** Every 5 minutes (drift detection)
- **Production:** Every 15 minutes (capacity planning)

### Data Retention

SuzieQ stores data in Parquet files:
```bash
# Check storage usage
du -sh ~/.suzieq/parquet

# Cleanup old data (>30 days)
find ~/.suzieq/parquet -mtime +30 -delete
```

### Drift Detection Performance

| Network Size | Devices | Drift Check Time |
|--------------|---------|------------------|
| Small        | 1-10    | < 1 second       |
| Medium       | 10-100  | 1-5 seconds      |
| Large        | 100-500 | 5-15 seconds     |
| Enterprise   | 500+    | 15-60 seconds    |

## Best Practices

### 1. Use Namespaces

Separate environments:
```python
# Production namespace
suzieq.collect_network_state(devices, namespace="production")

# Staging namespace
suzieq.collect_network_state(devices, namespace="staging")
```

### 2. Schedule Regular Drift Checks

```python
# Cron job: Check drift every hour
#!/bin/bash
cd /opt/overgrowth
source venv/bin/activate
python -c "
from agent.pipeline_engine import OvergrowthPipeline
pipeline = OvergrowthPipeline()
model = NetworkModel.from_yaml('network.yaml')
drift = pipeline.stage7b_drift_detection(model)
if drift['drift_detected']:
    print(f'ALERT: Drift score {drift[\"drift_score\"]:.2f}')
    # Send alert to Slack/PagerDuty
"
```

### 3. Auto-Fix Low-Risk Changes

```python
# Safe changes: Add VLANs, update descriptions
auto_fix_actions = ['add_vlan', 'update_description', 'fix_ip_mismatch']

# Apply only safe actions
safe_plan = [a for a in plan if a['action'] in auto_fix_actions]
suzieq.apply_remediation(safe_plan, auto_approve=True)

# Manual review for everything else
manual_plan = [a for a in plan if a['action'] not in auto_fix_actions]
notify_team(manual_plan)
```

### 4. Track Drift Over Time

```python
from datetime import datetime

# Log drift history
drift_log = {
    'timestamp': datetime.now().isoformat(),
    'drift_score': drift.drift_score,
    'devices_checked': drift.devices_checked,
    'issues': {
        'config_mismatches': len(drift.config_mismatches),
        'missing_vlans': len(drift.missing_vlans),
        'extra_vlans': len(drift.extra_vlans)
    }
}

# Store in database or CSV
append_to_history(drift_log)

# Alert if drift increasing
if drift_score > previous_score * 1.5:
    alert("Drift increasing rapidly!")
```

## Future Enhancements

### Planned Features

- **StackStorm Integration:** Event-driven auto-remediation when drift detected
- **RAG-based Learning:** Learn from past drift incidents to prevent recurrence
- **Change Correlation:** Link drift events to recent changes (Git, tickets)
- **Predictive Drift:** ML model to predict drift before it happens
- **Multi-Region Sync:** Ensure consistency across global deployments

### Community Contributions

See `CONTRIBUTING.md` for how to add:
- New drift detection rules
- Additional remediation actions
- Custom compliance policies
- Integration with other observability tools

## References

- **SuzieQ Documentation:** https://suzieq.readthedocs.io/
- **SuzieQ GitHub:** https://github.com/netenglabs/suzieq
- **Overgrowth Repo:** https://huggingface.co/spaces/MCP-1st-Birthday/overgrowth
- **NetBox Integration:** See `NETBOX_INTEGRATION.md`
- **Batfish Digital Twin:** See `BATFISH_INTEGRATION.md`

## Support

Questions? Issues? Contributions?
- Open an issue on HuggingFace Spaces
- Join our Discord: [link]
- Email: overgrowth@example.com