Capstone: AIOps Team

mkdir -p aiops/skills/{log-analysis,diagnostics,remediation}

cat > aiops/subagents.yaml << 'EOF'
sre_log_analyst:
  description: >
    ALWAYS use this first to analyze logs when investigating incidents.
    Expert at log parsing, pattern detection, and identifying error sequences.
    Use when you need to understand what happened from log data.
  model: anthropic:claude-haiku-4-5-20251001
  system_prompt: |
    You are an SRE log analysis specialist. Your job is to examine
    logs and extract meaningful patterns, error sequences, and anomalies.

    ## Your Tools
    - fetch_logs(service, minutes=30) - Retrieve recent logs from a service
    - query_metrics(service, metric) - Query monitoring metrics

    ## Your Process
    1. Fetch logs for the affected service
    2. Identify error patterns and sequences
    3. Note timestamps and correlations
    4. Look for anomalies (OOM kills, connection pool exhaustion, etc.)
    5. Summarize findings clearly for the diagnostician

    ## Output
    Write a structured summary with:
    - Timeline of events
    - Key error patterns
    - Suspicious metrics or anomalies
    - Recommendations for further investigation
  tools:
    - fetch_logs
    - query_metrics
  skills:
    - ./aiops/skills/log-analysis/

sre_diagnostician:
  description: >
    Use this to perform root cause analysis after log analysis is complete.
    Expert at connecting symptoms to underlying causes using diagnostic
    procedures and failure mode knowledge. Use when you have findings
    and need to determine the root cause.
  model: anthropic:claude-sonnet-4-6
  system_prompt: |
    You are an SRE diagnostician. You analyze findings from log analysis
    and apply diagnostic procedures to determine root causes.

    ## Your Tools
    - query_metrics(service, metric) - Query monitoring metrics for validation

    ## Your Process
    1. Review findings from the log analyst
    2. Apply diagnostic decision trees from your skill
    3. Query metrics to validate hypotheses
    4. Identify the most likely root cause
    5. Assess confidence level and recommend remediation approach

    ## Output
    Write a diagnostic report with:
    - Root cause analysis (what failed and why)
    - Supporting evidence from logs and metrics
    - Confidence level (high/medium/low)
    - Recommended remediation action
  tools:
    - query_metrics
  skills:
    - ./aiops/skills/diagnostics/

sre_remediator:
  description: >
    Use this to execute remediation after root cause is identified.
    Expert at applying fixes, rollbacks, and scaling actions with
    appropriate risk controls. ALWAYS requires human approval for
    execution via the interrupt mechanism.
  model: anthropic:claude-sonnet-4-6
  system_prompt: |
    You are an SRE remediation specialist. You apply fixes based on
    diagnostic findings, following runbooks and risk procedures.

    ## Your Tools
    - execute_remediation(action, target) - Execute a remediation action
      (REQUIRES HUMAN APPROVAL via interrupt)

    ## Your Process
    1. Review the root cause diagnosis
    2. Consult runbooks from your skill for the specific failure mode
    3. Propose a remediation plan with risk assessment
    4. Execute using execute_remediation (will pause for approval)
    5. Report results and recommend follow-up actions

    ## Risk Levels
    - Low: service restart, scale up resources
    - Medium: rollback deployment, connection pool reset
    - High: database failover, emergency maintenance

    ## Output
    Write a remediation report with:
    - Proposed action and risk level
    - Expected impact and recovery time
    - Execution results
    - Follow-up recommendations
  tools:
    - execute_remediation
  skills:
    - ./aiops/skills/remediation/
EOF

cat > aiops/skills/log-analysis/SKILL.md << 'EOF'
---
name: log-analysis
description: Expert procedures for analyzing application and infrastructure logs to identify error patterns, anomalies, and failure sequences.
---

# Log Analysis Skill

## Error Pattern Detection

When analyzing logs, look for these common patterns:

### Connection Pool Exhaustion
- `Connection pool exhausted`
- `max connections reached`
- `Timeout waiting for connection`
- Often followed by cascading failures

### Out of Memory (OOM)
- `OOM Kill`
- `container memory limit exceeded`
- `java.lang.OutOfMemoryError`
- `GC overhead limit exceeded`
- Usually preceded by gradual memory growth

### Database Issues
- `deadlock detected`
- `too many connections`
- `connection refused`
- `query timeout`
- Check for long-running queries

### Network Problems
- `connection reset by peer`
- `connection timed out`
- `no route to host`
- `DNS resolution failed`

## Anomaly Detection

Watch for deviations from normal patterns:

1. **Frequency anomalies**: Error rates spiking from <1% to >5%
2. **Timing anomalies**: Response times jumping 10x or more
3. **Volume anomalies**: Traffic patterns changing dramatically
4. **Sequence anomalies**: Errors in unexpected order

## Correlation Analysis

Look for correlations between events:

- Error spikes correlated with deployments
- Resource exhaustion preceding service failures
- Cascading failures (A fails, then B, then C)
- Time-based patterns (daily cycles, weekly patterns)

## Output Format

Structure your findings:
1. **Timeline**: Chronological sequence of key events
2. **Patterns**: Identified error patterns with examples
3. **Anomalies**: Deviations from baseline behavior
4. **Metrics**: Supporting data (error rates, response times, resource usage)
5. **Hypothesis**: Initial theory about what went wrong
EOF

cat > aiops/skills/diagnostics/SKILL.md << 'EOF'
---
name: diagnostics
description: Diagnostic decision trees and failure mode knowledge for determining root causes of service incidents.
---

# Diagnostic Skill

## Common Failure Modes

### Out of Memory (OOM)
**Symptoms**:
- OOM kill messages in logs
- Container restarts
- Memory metrics at/near limit

**Diagnostic Steps**:
1. Check memory trend before failure (gradual vs sudden)
2. Review application memory profile
3. Check for memory leaks (heap dumps if available)
4. Verify memory limits are appropriate

**Common Causes**:
- Memory leak in application code
- Undersized container limits
- Traffic spike exceeding capacity
- Memory-intensive operations (large queries, caching)

### Connection Pool Exhaustion
**Symptoms**:
- "Connection pool exhausted" errors
- Timeouts waiting for connections
- Service degradation under load

**Diagnostic Steps**:
1. Check current vs max connection settings
2. Review connection lifecycle (are connections released?)
3. Check for long-running queries holding connections
4. Verify downstream service health

**Common Causes**:
- Connection leaks (not closing properly)
- Pool sized too small for load
- Downstream service slowness
- Database performance issues

### Disk Space Exhaustion
**Symptoms**:
- "No space left on device"
- Write failures
- Application crashes

**Diagnostic Steps**:
1. Check disk usage metrics
2. Identify largest files/directories
3. Review log rotation policies
4. Check for unexpected data growth

**Common Causes**:
- Log rotation not configured
- Temporary file buildup
- Database/cache growth
- Failed cleanup jobs

### DNS/Network Issues
**Symptoms**:
- "Name resolution failed"
- "Connection timed out"
- Intermittent connectivity

**Diagnostic Steps**:
1. Check DNS resolution for affected services
2. Verify network connectivity paths
3. Review firewall/security group rules
4. Check for network congestion

**Common Causes**:
- DNS server issues
- Network partition
- Firewall rule changes
- Service mesh configuration

### Certificate Expiry
**Symptoms**:
- "Certificate has expired"
- TLS handshake failures
- Sudden service unavailability

**Diagnostic Steps**:
1. Check certificate expiration dates
2. Verify certificate chain validity
3. Review renewal automation
4. Check for certificate mismatch

**Common Causes**:
- Expired certificates
- Failed auto-renewal
- Certificate configuration errors

## Diagnostic Decision Tree

```
START: Service degradation or failure
  ↓
[Check recent changes]
  → Recent deployment? → Likely: Code regression or config change
  → Infrastructure change? → Likely: Resource/network issue
  → No changes? → Continue to symptoms
  ↓
[Examine error patterns]
  → OOM/memory errors? → Run OOM diagnostics
  → Connection errors? → Run connection pool diagnostics
  → Disk errors? → Run disk space diagnostics
  → Network/timeout? → Run network diagnostics
  → Certificate errors? → Run certificate diagnostics
  ↓
[Validate hypothesis]
  → Query metrics to confirm
  → Check correlating events
  → Assess confidence (high/medium/low)
  ↓
[Recommend remediation]
```

## Confidence Assessment

Rate your diagnostic confidence:

- **High**: Clear evidence, well-known failure mode, metrics confirm
- **Medium**: Evidence present but some ambiguity, metrics partially confirm
- **Low**: Multiple possible causes, limited evidence, recommend deeper investigation
EOF

cat > aiops/skills/remediation/SKILL.md << 'EOF'
---
name: remediation
description: Runbooks and procedures for safely remediating common service failures with appropriate risk controls.
---

# Remediation Skill

## Risk Classification

All remediation actions are classified by risk:

- **Low**: Minimal impact, fast rollback, safe to automate
- **Medium**: Some impact, requires approval, rollback available
- **High**: Significant impact, requires senior approval, complex rollback

## Runbooks

### Service Restart (Low Risk)
**When to use**: Process crashes, hung services, minor memory leaks

**Procedure**:
1. Execute: `execute_remediation("restart", "service-name")`
2. Monitor: Service comes back healthy within 30s
3. Verify: Health checks passing, traffic resuming

**Expected downtime**: 10-30 seconds

**Rollback**: N/A (restart is the rollback)

### Scale Up Resources (Low Risk)
**When to use**: Resource exhaustion (CPU, memory, connections)

**Procedure**:
1. Execute: `execute_remediation("scale_up", "service-name")`
2. Monitor: New instances starting, load distributing
3. Verify: Metrics returning to normal

**Expected impact**: None (adds capacity)

**Rollback**: Scale down after incident resolved

### Rollback Deployment (Medium Risk)
**When to use**: Recent deployment causing errors, regression detected

**Procedure**:
1. Identify previous stable version
2. Execute: `execute_remediation("rollback_deployment", "service-name")`
3. Monitor: Rollback progress, service health
4. Verify: Error rate dropping, functionality restored

**Expected downtime**: 30-90 seconds during rollback

**Rollback**: Re-deploy if rollback causes issues (rare)

### Connection Pool Reset (Medium Risk)
**When to use**: Connection pool exhaustion, leaked connections

**Procedure**:
1. Execute: `execute_remediation("reset_connection_pool", "service-name")`
2. Monitor: Pool draining and re-initializing
3. Verify: Connections available, errors cleared

**Expected impact**: Brief connection errors during reset

**Rollback**: Service restart if reset fails

### Increase Connection Pool Size (Low Risk)
**When to use**: Pool legitimately too small for load

**Procedure**:
1. Execute: `execute_remediation("increase_pool_size", "service-name")`
2. Monitor: Configuration update, pool expansion
3. Verify: Connections available, no exhaustion

**Expected impact**: None (increases capacity)

**Rollback**: Revert configuration change

### Clear Disk Space (Medium Risk)
**When to use**: Disk space exhaustion

**Procedure**:
1. Identify safe-to-delete files (old logs, temp files)
2. Execute: `execute_remediation("clear_disk_space", "service-name")`
3. Monitor: Disk usage decreasing
4. Verify: Service writing successfully

**Expected impact**: Minimal, may lose old logs

**Rollback**: N/A (files deleted)

## Remediation Plan Template

When proposing remediation, structure it:

1. **Root Cause**: Brief summary from diagnostics
2. **Proposed Action**: Specific remediation from runbook
3. **Risk Level**: Low/Medium/High
4. **Expected Impact**: Downtime, data loss, user impact
5. **Recovery Time**: How long until service restored
6. **Rollback Plan**: How to undo if it doesn't work
7. **Approval Required**: Yes (always for production)

## Follow-up Actions

After remediation, always recommend:

1. **Monitoring**: What to watch for recurrence
2. **Post-mortem**: Schedule incident review
3. **Prevention**: Long-term fixes (increase limits, fix leaks, etc.)
4. **Documentation**: Update runbooks with learnings
EOF

cat > aiops/AGENTS.md << 'EOF'
# AIOps Team Operational Context

## Environment

**Production Environment**:
- 3 application servers (payment-api, inventory-api, user-api)
- 1 PostgreSQL database (primary + read replica)
- 1 Redis cache cluster
- Kubernetes orchestration
- Prometheus + Grafana monitoring

**Resource Limits**:
- Application containers: 512Mi memory, 1 CPU
- Database: 4Gi memory, 2 CPU
- Redis: 2Gi memory, 1 CPU

**Connection Pools**:
- Default max connections: 100 per service
- Database max connections: 500 total
- Typical load: 20-30 connections per service

## Known Failure Modes

1. **Connection Pool Exhaustion** (seen 3 times in last month)
   - Usually during traffic spikes
   - Often combined with slow database queries
   - Resolution: increase pool size or optimize queries

2. **Memory Leaks** (payment-api specifically)
   - Gradual memory growth over 3-4 days
   - Requires weekly restarts as workaround
   - Fix in progress: PR #1247

3. **Database Deadlocks** (rare but recurring)
   - Complex transaction interactions
   - Usually during bulk operations
   - Retry logic handles most cases

## Escalation Procedures

**Low Risk Actions** (no escalation required):
- Service restart
- Scale up resources
- Clear disk space

**Medium Risk Actions** (notify on-call lead):
- Rollback deployment
- Connection pool reset
- Configuration changes

**High Risk Actions** (require senior approval):
- Database failover
- Emergency maintenance
- Data restoration

## Recent Incidents

**2026-03-28**: payment-api OOM due to traffic spike during flash sale
- Resolution: scaled up from 3 to 6 instances
- Prevention: auto-scaling rules updated

**2026-03-25**: inventory-api connection pool exhaustion
- Resolution: increased pool size from 100 to 150
- Prevention: monitoring alert added

**2026-03-20**: user-api deployment rollback (authentication regression)
- Resolution: rolled back to v2.4.1
- Prevention: added auth integration tests
EOF

cd aiops
uv run agent.py

cat >> aiops/subagents.yaml << 'EOF'

sre_communicator:
  description: >
    Use this to draft stakeholder communications about incidents.
    Expert at translating technical details into business-friendly
    updates. Use after remediation to notify customers or leadership.
  model: anthropic:claude-sonnet-4-6
  system_prompt: |
    You are an SRE communications specialist. You draft clear,
    honest, and professional incident updates for stakeholders.

    ## Your Process
    1. Review incident details and remediation
    2. Draft customer-facing status update (non-technical)
    3. Draft internal incident summary (technical)
    4. Recommend communication timing and channels

    ## Tone
    - Transparent about issues
    - Clear about impact and resolution
    - Professional and calm
    - Avoid jargon in external communications
EOF

mkdir -p aiops/skills/escalation
cat > aiops/skills/escalation/SKILL.md << 'EOF'
---
name: escalation
description: Procedures for escalating incidents based on severity, impact, and complexity.
---

# Escalation Skill

## Escalation Triggers

**Immediate Escalation**:
- Customer data at risk
- Security breach suspected
- Multiple services down
- Revenue-impacting outage >15 minutes

**Standard Escalation**:
- Root cause unclear after 30 minutes
- Remediation requires high-risk actions
- Cross-team coordination needed
- Incident affecting >10% of users

## Escalation Paths

1. **On-call SRE Lead**: First escalation for technical decisions
2. **Engineering Manager**: For deployment/rollback approvals
3. **VP Engineering**: For customer-impacting decisions
4. **Incident Commander**: For multi-team coordination

## Communication Template

When escalating, provide:
- Incident summary (1-2 sentences)
- Current impact (users affected, services down)
- Actions taken so far
- Reason for escalation
- Recommended next steps
EOF