PIPELINE_ENHANCEMENT.md

# Overgrowth Pipeline Enhancement Plan

## What's Missing (Your Feedback Summary)

1. **❌ No Interactive Consultation** - Should ask follow-up questions
2. **❌ Source of Truth Incomplete** - No subnets, VLANs, real design
3. **❌ BOM Pricing Wrong** - Shows $0 for everything
4. **❌ Setup Guide Generic** - Missing firmware updates, real steps
5. **❌ No Progress Visibility** - Can't see what AI agents are doing
6. **❌ No Network Simulation Link** - Should show GNS3 topology

## Implementation Plan

### Phase 1: LLM Integration (PRIORITY)

**Files Created:**
- `agent/llm_client.py` - Unified LLM client (OpenAI/Anthropic/OpenRouter)
- `agent/consultation.py` - Interactive multi-turn consultation

**Setup Required:**

```bash
# Add to .env file:
OPENROUTER_API_KEY=sk-or-v1-xxxxx  # From your hackathon credits
# OR
OPENAI_API_KEY=sk-xxxxx
# OR
ANTHROPIC_API_KEY=sk-ant-xxxxx
```

**Testing:**
```python
from agent.consultation import NetworkConsultant

consultant = NetworkConsultant()
is_complete, output, intent = consultant.start_consultation(
    "We're a coffee shop chain with 3 locations..."
)
print(output)  # Will show follow-up questions

# User answers questions
is_complete, output, intent = consultant.continue_consultation(
    "Budget is $50k, need it in 3 months, prefer Ubiquiti gear"
)
```

### Phase 2: Smart Network Design

**Enhance `stage2_generate_sot()` in pipeline_engine.py:**

```python
def stage2_generate_sot(self, intent: NetworkIntent) -> NetworkModel:
    """Use LLM to design actual network architecture"""
    
    prompt = f"""
    Design a production-ready network for:
    {intent.description}
    
    Requirements:
    - Budget: {intent.budget}
    - Locations: {intent.locations}
    - Compliance: {intent.compliance_requirements}
    
    Generate:
    1. VLAN scheme (management, data, voice, guest, security cameras, POS)
    2. IP subnetting plan (RFC1918 private addressing)
    3. Device list (switches, APs, routers, firewalls)
    4. Routing protocol (static, OSPF, BGP)
    5. Security policies
    
    Return as structured JSON.
    """
    
    # Call LLM to generate real design
    design = llm.chat([LLMMessage(role="user", content=prompt)])
    
    # Parse into NetworkModel
    return self._parse_network_design(design)
```

**Example Output:**
```yaml
vlans:
  - id: 10
    name: Management
    subnet: 10.0.10.0/24
  - id: 20
    name: Guest_WiFi
    subnet: 10.0.20.0/24
  - id: 30
    name: POS_Systems
    subnet: 10.0.30.0/24
  - id: 40
    name: Security_Cameras
    subnet: 10.0.40.0/24

devices:
  - name: HQ-Core-SW01
    role: core
    model: Ubiquiti USW-Enterprise-48-PoE
    mgmt_ip: 10.0.10.10
    interfaces:
      - name: eth0/1
        vlan: 10
        mode: access
```

### Phase 3: Real BOM Pricing

**Create `agent/hardware_pricing.py`:**

```python
# Hardware database with real prices
HARDWARE_DB = {
    "Ubiquiti USW-Enterprise-48-PoE": {
        "price": 1799.00,
        "category": "switch",
        "vendor": "Ubiquiti"
    },
    "Ubiquiti U6-Enterprise": {
        "price": 379.00,
        "category": "access_point"
    },
    # ... more devices
}

def calculate_bom_cost(devices: List[Device]) -> float:
    total = 0
    for device in devices:
        if device.model in HARDWARE_DB:
            total += HARDWARE_DB[device.model]["price"]
    return total
```

### Phase 4: Streaming Progress Updates

**Modify `app.py` to use Gradio streaming:**

```python
def run_pipeline_streaming(user_input):
    """Stream progress updates to UI"""
    pipeline = OvergrowthPipeline()
    
    # Stage 1: Consultation
    yield "🤝 Stage 1: Starting consultation...\n"
    consultant = NetworkConsultant()
    is_complete, output, intent = consultant.start_consultation(user_input)
    
    if not is_complete:
        yield f"❓ **Follow-up questions:**\n{output}\n\n"
        # Wait for user response (need UI update for this)
        return
    
    yield f"✅ Stage 1 Complete\n{output}\n\n"
    
    # Stage 2: Generate SoT
    yield "📋 Stage 2: Designing network architecture...\n"
    model = pipeline.stage2_generate_sot(intent)
    yield f"✅ Stage 2 Complete - {len(model.devices)} devices, {len(model.vlans)} VLANs\n\n"
    
    # Stage 3: Diagrams
    yield "📊 Stage 3: Generating topology diagrams...\n"
    diagrams = pipeline.stage3_generate_diagrams(model)
    yield f"✅ Stage 3 Complete\n\n"
    
    # Continue with other stages...
```

**Update Gradio interface:**
```python
run_pipeline_btn.click(
    fn=run_pipeline_streaming,
    inputs=[pipeline_input],
    outputs=[pipeline_status],  # Single streaming output
    show_progress=True
)
```

### Phase 5: GNS3 Simulation Integration

**Add to pipeline results:**

```python
def stage6_autonomous_deploy(self, model: NetworkModel) -> Dict:
    """Deploy to GNS3 lab"""
    from agent.local_mcp import call_tool
    
    # Build topology in GNS3
    result = call_tool("create_project", {
        "name": model.name,
        "auto_start": True
    })
    
    project_id = result['project_id']
    
    # Add devices
    for device in model.devices:
        call_tool("add_node", {
            "project_id": project_id,
            "name": device.name,
            "node_type": device.role,
            "x": ...,  # Calculate layout
            "y": ...
        })
    
    # Return simulation URL
    return {
        "success": True,
        "gns3_url": f"http://lab.grahampaasch.com:3080/#/projects/{project_id}",
        "topology_link": f"View live simulation: {gns3_url}"
    }
```

**Display in UI:**
```markdown
## 🌐 Live Network Simulation

Your network is being built in GNS3:
- **Project:** {model.name}
- **Devices:** {len(model.devices)} nodes
- **Status:** Deploying...

[View in GNS3](http://lab.grahampaasch.com:3080/#/projects/{project_id})
```

### Phase 6: Setup Guide with Real Steps

**Enhance `stage5_setup_guide()`:**

```python
def generate_setup_guide(self, model: NetworkModel) -> SetupGuide:
    """Generate detailed deployment guide"""
    
    phases = [
        {
            "name": "Pre-Deployment Validation",
            "duration": "1 hour",
            "steps": [
                "Verify all equipment received matches BOM",
                "Check firmware versions - minimum required:",
                *[f"  - {d.model}: firmware v{get_min_firmware(d.model)}" 
                  for d in model.devices],
                "Unbox and inventory all equipment",
                "Download latest firmware if upgrades needed"
            ]
        },
        {
            "name": "Firmware Updates",
            "duration": "2-4 hours",
            "steps": [
                "Backup factory configs",
                "Update devices one at a time",
                *[f"Update {d.name} to {get_latest_firmware(d.model)}" 
                  for d in model.devices],
                "Verify boot-up and basic connectivity",
                "Document firmware versions"
            ]
        },
        # ... more realistic phases
    ]
    
    return SetupGuide(
        network_name=model.name,
        phases=phases,
        # ... other details
    )
```

## Next Steps

1. **Add LLM API key to .env**
2. **Test consultation flow**
3. **Enhance pipeline stages with LLM calls**
4. **Add hardware pricing database**
5. **Implement streaming UI updates**
6. **Wire up GNS3 deployment**

## Cost Estimate

Using OpenRouter with Claude 3.5 Sonnet:
- Consultation: ~$0.02 per session
- Network Design: ~$0.05 per design
- Total per pipeline run: ~$0.10

Your hackathon credits should cover hundreds of runs.