ETB/ETB-API/sla_oncall/Documentations/SLA_ONCALL_API.md

# SLA & On-Call Management API Documentation

## Overview

The SLA & On-Call Management module provides comprehensive Service Level Agreement (SLA) tracking, escalation policies, and on-call rotation management for enterprise incident management systems.

## Features

### Dynamic SLAs
- **Incident Type-Based SLAs**: Different SLA targets based on incident category, severity, and priority
- **Business Hours Support**: SLA calculations that respect business hours and timezones
- **Multiple SLA Types**: Response time, resolution time, acknowledgment time, and first response time
- **Automatic SLA Instance Creation**: SLAs are automatically created when incidents are reported

### Escalation Policies
- **Multi-Level Escalation**: Configurable escalation steps with different actions and timing
- **Condition-Based Triggering**: Escalations triggered by SLA breaches, thresholds, or custom conditions
- **Multi-Channel Notifications**: Email, SMS, Slack, Teams, and webhook notifications
- **Integration with On-Call**: Automatic escalation to current on-call personnel

### On-Call Rotation Management
- **Flexible Scheduling**: Weekly, daily, monthly, and custom rotation schedules
- **External System Integration**: Built-in support for PagerDuty and OpsGenie
- **Handoff Management**: Structured handoff processes with notes and tracking
- **Performance Metrics**: Track incident handling and response times

### Business Hours Management
- **Timezone Support**: Multiple timezone configurations
- **Holiday Calendar**: Holiday and special day handling
- **Day Overrides**: Custom hours for specific dates
- **Weekend Configuration**: Separate weekend business hours

## API Endpoints

### Business Hours Management

#### GET /api/sla-oncall/api/v1/business-hours/
List all business hours configurations.

**Query Parameters:**
- `is_active`: Filter by active status
- `is_default`: Filter by default status
- `timezone`: Filter by timezone
- `search`: Search by name or description

**Response:**
```json
{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": "uuid",
            "name": "Standard Business Hours",
            "description": "Standard 9-5 business hours",
            "timezone": "UTC",
            "weekday_start": "09:00:00",
            "weekday_end": "17:00:00",
            "weekend_start": "10:00:00",
            "weekend_end": "16:00:00",
            "day_overrides": {},
            "holiday_calendar": [],
            "is_active": true,
            "is_default": true,
            "created_at": "2024-01-01T00:00:00Z",
            "updated_at": "2024-01-01T00:00:00Z"
        }
    ]
}
```

#### POST /api/sla-oncall/api/v1/business-hours/
Create a new business hours configuration.

**Request Body:**
```json
{
    "name": "Custom Business Hours",
    "description": "Custom business hours for special team",
    "timezone": "America/New_York",
    "weekday_start": "08:00:00",
    "weekday_end": "18:00:00",
    "weekend_start": "10:00:00",
    "weekend_end": "16:00:00",
    "holiday_calendar": ["2024-12-25", "2024-01-01"],
    "is_active": true
}
```

#### POST /api/sla-oncall/api/v1/business-hours/{id}/test_business_hours/
Test if a given time is within business hours.

**Request Body:**
```json
{
    "test_time": "2024-01-08T14:30:00Z"
}
```

**Response:**
```json
{
    "is_business_hours": true,
    "test_time": "2024-01-08T14:30:00Z"
}
```

### SLA Definitions

#### GET /api/sla-oncall/api/v1/sla-definitions/
List all SLA definitions.

**Query Parameters:**
- `sla_type`: Filter by SLA type (RESPONSE_TIME, RESOLUTION_TIME, etc.)
- `is_active`: Filter by active status
- `business_hours_only`: Filter by business hours requirement

**Response:**
```json
{
    "count": 3,
    "results": [
        {
            "id": "uuid",
            "name": "Critical Incident Response",
            "description": "SLA for critical incidents",
            "sla_type": "RESPONSE_TIME",
            "incident_categories": ["SYSTEM", "NETWORK"],
            "incident_severities": ["CRITICAL", "EMERGENCY"],
            "incident_priorities": ["P1"],
            "target_duration_minutes": 15,
            "business_hours_only": false,
            "business_hours": null,
            "business_hours_name": null,
            "escalation_enabled": true,
            "escalation_threshold_percent": 75.0,
            "is_active": true,
            "is_default": false,
            "created_at": "2024-01-01T00:00:00Z"
        }
    ]
}
```

#### POST /api/sla-oncall/api/v1/sla-definitions/
Create a new SLA definition.

**Request Body:**
```json
{
    "name": "High Priority Response",
    "description": "SLA for high priority incidents",
    "sla_type": "RESPONSE_TIME",
    "incident_severities": ["HIGH"],
    "incident_priorities": ["P2"],
    "target_duration_minutes": 30,
    "business_hours_only": false,
    "escalation_enabled": true,
    "escalation_threshold_percent": 80.0,
    "is_active": true
}
```

#### POST /api/sla-oncall/api/v1/sla-definitions/{id}/test_applicability/
Test if SLA definition applies to a given incident.

**Request Body:**
```json
{
    "category": "SYSTEM",
    "severity": "HIGH",
    "priority": "P2"
}
```

**Response:**
```json
{
    "applies": true,
    "incident_data": {
        "category": "SYSTEM",
        "severity": "HIGH",
        "priority": "P2"
    }
}
```

### On-Call Rotations

#### GET /api/sla-oncall/api/v1/oncall-rotations/
List all on-call rotations.

**Query Parameters:**
- `rotation_type`: Filter by rotation type (WEEKLY, DAILY, etc.)
- `status`: Filter by status (ACTIVE, PAUSED, INACTIVE)
- `external_system`: Filter by external system integration

**Response:**
```json
{
    "count": 1,
    "results": [
        {
            "id": "uuid",
            "name": "Primary On-Call Rotation",
            "description": "Primary rotation for incident response",
            "rotation_type": "WEEKLY",
            "status": "ACTIVE",
            "team_name": "Incident Response Team",
            "team_description": "Primary team responsible for incidents",
            "schedule_config": {
                "rotation_length_days": 7,
                "handoff_time": "09:00"
            },
            "timezone": "UTC",
            "external_system": "INTERNAL",
            "external_system_id": null,
            "integration_config": {},
            "current_oncall": {
                "user_id": "uuid",
                "username": "john.doe",
                "start_time": "2024-01-08T09:00:00Z",
                "end_time": "2024-01-15T09:00:00Z"
            },
            "created_at": "2024-01-01T00:00:00Z"
        }
    ]
}
```

#### GET /api/sla-oncall/api/v1/oncall-rotations/{id}/current_oncall/
Get the current on-call person for a rotation.

**Response:**
```json
{
    "id": "uuid",
    "rotation": "uuid",
    "rotation_name": "Primary On-Call Rotation",
    "user": "uuid",
    "user_name": "john.doe",
    "user_email": "john.doe@company.com",
    "start_time": "2024-01-08T09:00:00Z",
    "end_time": "2024-01-15T09:00:00Z",
    "status": "ACTIVE",
    "incidents_handled": 5,
    "response_time_avg": "00:15:30"
}
```

#### GET /api/sla-oncall/api/v1/oncall-rotations/{id}/upcoming_assignments/
Get upcoming on-call assignments.

**Query Parameters:**
- `days`: Number of days ahead to look (default: 30)

### SLA Instances

#### GET /api/sla-oncall/api/v1/sla-instances/
List all SLA instances.

**Query Parameters:**
- `status`: Filter by status (ACTIVE, MET, BREACHED, CANCELLED)
- `escalation_triggered`: Filter by escalation status
- `sla_definition`: Filter by SLA definition

**Response:**
```json
{
    "count": 10,
    "results": [
        {
            "id": "uuid",
            "sla_definition": "uuid",
            "sla_definition_name": "Critical Incident Response",
            "incident": "uuid",
            "incident_title": "Database Connection Failure",
            "status": "ACTIVE",
            "target_time": "2024-01-08T15:15:00Z",
            "started_at": "2024-01-08T15:00:00Z",
            "met_at": null,
            "breached_at": null,
            "escalation_policy": "uuid",
            "escalation_triggered": false,
            "escalation_triggered_at": null,
            "escalation_level": 0,
            "response_time": null,
            "resolution_time": null,
            "is_breached": false,
            "time_remaining": "00:12:30",
            "breach_time": "00:00:00",
            "created_at": "2024-01-08T15:00:00Z"
        }
    ]
}
```

#### GET /api/sla-oncall/api/v1/sla-instances/breached/
Get all breached SLA instances.

#### GET /api/sla-oncall/api/v1/sla-instances/at_risk/
Get SLA instances at risk of breaching (within 15 minutes).

#### POST /api/sla-oncall/api/v1/sla-instances/{id}/mark_met/
Mark an SLA instance as met.

**Response:**
```json
{
    "message": "SLA marked as met"
}
```

#### POST /api/sla-oncall/api/v1/sla-instances/{id}/mark_breached/
Mark an SLA instance as breached.

### On-Call Assignments

#### GET /api/sla-oncall/api/v1/oncall-assignments/
List all on-call assignments.

**Query Parameters:**
- `rotation`: Filter by rotation
- `user`: Filter by user
- `status`: Filter by status (SCHEDULED, ACTIVE, COMPLETED, CANCELLED)

#### POST /api/sla-oncall/api/v1/oncall-assignments/
Create a new on-call assignment.

**Request Body:**
```json
{
    "rotation": "uuid",
    "user": "uuid",
    "start_time": "2024-01-15T09:00:00Z",
    "end_time": "2024-01-22T09:00:00Z",
    "handoff_notes": "All systems stable, no pending incidents"
}
```

#### POST /api/sla-oncall/api/v1/oncall-assignments/{id}/handoff/
Perform on-call handoff.

**Request Body:**
```json
{
    "handoff_notes": "Handing off to next person. 3 active incidents."
}
```

#### POST /api/sla-oncall/api/v1/oncall-assignments/{id}/activate/
Activate a scheduled assignment.

#### POST /api/sla-oncall/api/v1/oncall-assignments/{id}/complete/
Complete an active assignment.

### Escalation Policies

#### GET /api/sla-oncall/api/v1/escalation-policies/
List all escalation policies.

**Query Parameters:**
- `escalation_type`: Filter by escalation type
- `trigger_condition`: Filter by trigger condition
- `is_active`: Filter by active status

#### POST /api/sla-oncall/api/v1/escalation-policies/
Create a new escalation policy.

**Request Body:**
```json
{
    "name": "Critical Escalation",
    "description": "Escalation for critical incidents",
    "escalation_type": "TIME_BASED",
    "trigger_condition": "SLA_THRESHOLD",
    "incident_severities": ["CRITICAL", "EMERGENCY"],
    "trigger_delay_minutes": 0,
    "escalation_steps": [
        {
            "level": 1,
            "delay_minutes": 5,
            "actions": ["notify_oncall", "notify_manager"],
            "channels": ["email", "sms"]
        },
        {
            "level": 2,
            "delay_minutes": 15,
            "actions": ["notify_director", "page_oncall"],
            "channels": ["email", "sms", "phone"]
        }
    ],
    "notification_channels": ["email", "sms", "phone"],
    "is_active": true
}
```

### Escalation Instances

#### GET /api/sla-oncall/api/v1/escalation-instances/
List all escalation instances.

**Query Parameters:**
- `status`: Filter by status (PENDING, TRIGGERED, ACKNOWLEDGED, RESOLVED, CANCELLED)
- `escalation_level`: Filter by escalation level
- `escalation_policy`: Filter by escalation policy

#### POST /api/sla-oncall/api/v1/escalation-instances/{id}/acknowledge/
Acknowledge an escalation.

#### POST /api/sla-oncall/api/v1/escalation-instances/{id}/resolve/
Resolve an escalation.

### Notification Templates

#### GET /api/sla-oncall/api/v1/notification-templates/
List all notification templates.

**Query Parameters:**
- `template_type`: Filter by template type (ESCALATION, ONCALL_HANDOFF, etc.)
- `channel_type`: Filter by channel type (EMAIL, SMS, SLACK, etc.)
- `is_active`: Filter by active status

#### POST /api/sla-oncall/api/v1/notification-templates/
Create a new notification template.

**Request Body:**
```json
{
    "name": "Email Escalation Alert",
    "template_type": "ESCALATION",
    "channel_type": "EMAIL",
    "subject_template": "URGENT: Incident #{incident_id} Escalated",
    "body_template": "Incident #{incident_id} has been escalated to Level {escalation_level}. Please respond immediately.",
    "variables": ["incident_id", "incident_title", "escalation_level"],
    "is_active": true,
    "is_default": true
}
```

## Setup and Configuration

### Initial Setup

Run the setup command to create default configurations:

```bash
python manage.py setup_sla_oncall
```

This command creates:
- Default business hours configurations
- Standard SLA definitions for different incident types
- Default escalation policies
- Notification templates
- Sample on-call rotation (if users exist)

### Configuration Examples

#### Business Hours for Different Teams

```python
# 24/7 Operations
business_hours = BusinessHours.objects.create(
    name='24/7 Operations',
    description='Always business hours',
    timezone='UTC',
    weekday_start=time(0, 0),
    weekday_end=time(23, 59),
    weekend_start=time(0, 0),
    weekend_end=time(23, 59),
)

# EMEA Business Hours
business_hours = BusinessHours.objects.create(
    name='EMEA Business Hours',
    description='EMEA timezone business hours',
    timezone='Europe/London',
    weekday_start=time(9, 0),
    weekday_end=time(17, 0),
    weekend_start=time(10, 0),
    weekend_end=time(16, 0),
    holiday_calendar=['2024-12-25', '2024-01-01', '2024-04-19'],
)
```

#### SLA Definitions

```python
# Critical incidents - 15 minute response
critical_sla = SLADefinition.objects.create(
    name='Critical Incident Response',
    description='SLA for critical and emergency incidents',
    sla_type='RESPONSE_TIME',
    incident_severities=['CRITICAL', 'EMERGENCY'],
    incident_priorities=['P1'],
    target_duration_minutes=15,
    business_hours_only=False,
    escalation_enabled=True,
    escalation_threshold_percent=75.0,
)

# Medium incidents - 2 hour response during business hours
medium_sla = SLADefinition.objects.create(
    name='Medium Priority Response',
    description='SLA for medium priority incidents',
    sla_type='RESPONSE_TIME',
    incident_severities=['MEDIUM'],
    incident_priorities=['P3'],
    target_duration_minutes=120,
    business_hours_only=True,
    business_hours=business_hours,
    escalation_enabled=True,
    escalation_threshold_percent=85.0,
)
```

#### Escalation Policies

```python
# Critical escalation policy
escalation_policy = EscalationPolicy.objects.create(
    name='Critical Incident Escalation',
    description='Escalation for critical incidents',
    escalation_type='TIME_BASED',
    trigger_condition='SLA_THRESHOLD',
    incident_severities=['CRITICAL', 'EMERGENCY'],
    trigger_delay_minutes=0,
    escalation_steps=[
        {
            'level': 1,
            'delay_minutes': 5,
            'actions': ['notify_oncall', 'notify_manager'],
            'channels': ['email', 'sms']
        },
        {
            'level': 2,
            'delay_minutes': 15,
            'actions': ['notify_director', 'page_oncall'],
            'channels': ['email', 'sms', 'phone']
        },
        {
            'level': 3,
            'delay_minutes': 30,
            'actions': ['notify_executive', 'escalate_to_vendor'],
            'channels': ['email', 'phone', 'webhook']
        }
    ],
    notification_channels=['email', 'sms', 'phone'],
)
```

#### On-Call Rotations

```python
# Weekly rotation
rotation = OnCallRotation.objects.create(
    name='Primary On-Call Rotation',
    description='Primary rotation for incident response',
    rotation_type='WEEKLY',
    team_name='Incident Response Team',
    schedule_config={
        'rotation_length_days': 7,
        'handoff_time': '09:00',
        'timezone': 'UTC'
    },
    timezone='UTC',
)

# Create assignments
assignment = OnCallAssignment.objects.create(
    rotation=rotation,
    user=user,
    start_time=timezone.now(),
    end_time=timezone.now() + timedelta(days=7),
    status='ACTIVE'
)
```

## Integration with Other Modules

### Incident Intelligence Integration

The SLA module automatically creates SLA instances when incidents are created:

```python
# When an incident is created, applicable SLA definitions are found
# and SLA instances are automatically created
incident = Incident.objects.create(
    title='Database Connection Failure',
    description='Unable to connect to primary database',
    severity='CRITICAL',
    category='DATABASE',
    reporter=user,
)

# This automatically triggers SLA instance creation via signals
```

### Automation Orchestration Integration

SLA breaches can trigger automation workflows:

```python
# In automation_orchestration models, you can reference SLA instances
class RunbookExecution(models.Model):
    # ... existing fields ...
    sla_instance = models.ForeignKey(
        'sla_oncall.SLAInstance',
        on_delete=models.SET_NULL,
        null=True,
        blank=True,
        related_name='runbook_executions'
    )
```

### Security Integration

On-call assignments respect security clearances:

```python
# Users with appropriate clearance levels can be assigned to sensitive incidents
if user.clearance_level.level >= incident.get_required_clearance_level():
    # User can be assigned to this incident
    assignment = OnCallAssignment.objects.create(...)
```

## Monitoring and Alerting

### SLA Breach Monitoring

Monitor SLA instances for breaches:

```python
# Get all breached SLAs
breached_slas = SLAInstance.objects.filter(status='BREACHED')

# Get SLAs at risk (within 15 minutes of breach)
warning_time = timezone.now() + timedelta(minutes=15)
at_risk_slas = SLAInstance.objects.filter(
    status='ACTIVE',
    target_time__lte=warning_time
)
```

### Escalation Monitoring

Monitor active escalations:

```python
# Get all active escalations
active_escalations = EscalationInstance.objects.filter(
    status__in=['PENDING', 'TRIGGERED']
)

# Get escalations by level
level_2_escalations = EscalationInstance.objects.filter(
    escalation_level=2,
    status='TRIGGERED'
)
```

### Performance Metrics

Track on-call performance:

```python
# Get current on-call assignments
current_assignments = OnCallAssignment.objects.filter(
    status='ACTIVE',
    start_time__lte=timezone.now(),
    end_time__gte=timezone.now()
)

# Calculate average response time
avg_response_time = current_assignments.aggregate(
    avg_response=Avg('response_time_avg')
)['avg_response']
```

## Best Practices

### SLA Definition Best Practices

1. **Start Simple**: Begin with basic SLAs and add complexity as needed
2. **Business Hours Consideration**: Use business hours for non-critical incidents
3. **Escalation Thresholds**: Set escalation thresholds at 75-85% of SLA time
4. **Regular Review**: Review and adjust SLAs based on performance data

### On-Call Management Best Practices

1. **Clear Handoffs**: Use structured handoff processes with notes
2. **Rotation Length**: Keep rotations between 1-2 weeks for optimal coverage
3. **Backup Coverage**: Always have backup on-call personnel
4. **Training**: Ensure on-call personnel are properly trained

### Escalation Best Practices

1. **Progressive Escalation**: Use multiple levels with increasing urgency
2. **Clear Actions**: Define specific actions for each escalation level
3. **Multiple Channels**: Use multiple notification channels for critical escalations
4. **Documentation**: Document all escalation actions and outcomes

## Troubleshooting

### Common Issues

1. **SLA Not Created**: Check if SLA definition criteria match incident attributes
2. **Escalation Not Triggered**: Verify escalation policy is active and criteria match
3. **On-Call Not Found**: Ensure active assignments exist for the rotation
4. **Business Hours Issues**: Verify timezone configuration and business hours setup

### Debugging Commands

```bash
# Check SLA instances for a specific incident
python manage.py shell
>>> incident = Incident.objects.get(id='incident-id')
>>> sla_instances = incident.sla_instances.all()
>>> for sla in sla_instances:
...     print(f"SLA: {sla.sla_definition.name}, Status: {sla.status}")

# Check current on-call for a rotation
>>> rotation = OnCallRotation.objects.get(id='rotation-id')
>>> current = rotation.get_current_oncall()
>>> print(f"Current on-call: {current.user.username if current else 'None'}")

# Check business hours
>>> business_hours = BusinessHours.objects.get(id='business-hours-id')
>>> now = timezone.now()
>>> print(f"Is business hours: {business_hours.is_business_hours(now)}")
```