Updates

ETB-API/incident_intelligence/Documentations/INCIDENT_INTELLIGENCE_API.md

@@ -0,0 +1,363 @@
+# Incident Intelligence API Documentation
+
+## Overview
+
+The Incident Intelligence module provides AI-driven capabilities for incident management, including:
+
+- **AI-driven incident classification** using NLP to categorize incidents from free text
+- **Automated severity suggestion** based on impact analysis
+- **Correlation engine** for linking related incidents and problem detection
+- **Duplication detection** for merging incidents that describe the same outage
+
+## Features
+
+### 1. AI-Driven Incident Classification
+
+Automatically classifies incidents into categories and subcategories based on their content:
+
+- **Categories**: Infrastructure, Application, Security, User Experience, Data, Integration
+- **Subcategories**: Specific types within each category (e.g., API_ISSUE, DATABASE_ISSUE)
+- **Confidence Scoring**: AI confidence level for each classification
+- **Keyword Extraction**: Identifies relevant keywords from incident text
+- **Sentiment Analysis**: Analyzes the sentiment of incident descriptions
+- **Urgency Detection**: Identifies urgency indicators in the text
+
+### 2. Automated Severity Suggestion
+
+Suggests incident severity based on multiple factors:
+
+- **User Impact Analysis**: Number of affected users and impact level
+- **Business Impact Assessment**: Revenue and operational impact
+- **Technical Impact Evaluation**: System and infrastructure impact
+- **Text Analysis**: Severity indicators in incident descriptions
+- **Confidence Scoring**: AI confidence in severity suggestions
+
+### 3. Correlation Engine
+
+Links related incidents and detects patterns:
+
+- **Correlation Types**: Same Service, Same Component, Temporal, Pattern Match, Dependency, Cascade
+- **Problem Detection**: Identifies when correlations suggest larger problems
+- **Time-based Analysis**: Considers temporal proximity of incidents
+- **Service Similarity**: Analyzes shared services and components
+- **Pattern Recognition**: Detects recurring issues and trends
+
+### 4. Duplication Detection
+
+Identifies and manages duplicate incidents:
+
+- **Duplication Types**: Exact, Near Duplicate, Similar, Potential Duplicate
+- **Similarity Analysis**: Text, temporal, and service similarity scoring
+- **Merge Recommendations**: Suggests actions (Merge, Link, Review, No Action)
+- **Confidence Scoring**: AI confidence in duplication detection
+- **Shared Elements**: Identifies common elements between incidents
+
+## API Endpoints
+
+### Incidents
+
+#### Create Incident
+```http
+POST /api/incidents/incidents/
+Content-Type: application/json
+
+{
+    "title": "Database Connection Timeout",
+    "description": "Users are experiencing timeouts when trying to access the database",
+    "free_text": "Database is down, can't connect, getting timeout errors",
+    "affected_users": 150,
+    "business_impact": "Critical business operations are affected",
+    "reporter": 1
+}
+```
+
+#### Get Incident Analysis
+```http
+GET /api/incidents/incidents/{id}/analysis/
+```
+
+Returns comprehensive AI analysis including:
+- Classification results
+- Severity suggestions
+- Correlations with other incidents
+- Potential duplicates
+- Associated patterns
+
+#### Trigger AI Analysis
+```http
+POST /api/incidents/incidents/{id}/analyze/
+```
+
+Manually triggers AI analysis for a specific incident.
+
+#### Get Incident Statistics
+```http
+GET /api/incidents/incidents/stats/
+```
+
+Returns statistics including:
+- Total incidents by status and severity
+- Average resolution time
+- AI processing statistics
+- Duplicate and correlation counts
+
+### Correlations
+
+#### Get Correlations
+```http
+GET /api/incidents/correlations/
+```
+
+#### Get Problem Indicators
+```http
+GET /api/incidents/correlations/problem_indicators/
+```
+
+Returns correlations that indicate larger problems.
+
+### Duplications
+
+#### Get Duplications
+```http
+GET /api/incidents/duplications/
+```
+
+#### Approve Merge
+```http
+POST /api/incidents/duplications/{id}/approve_merge/
+```
+
+#### Reject Merge
+```http
+POST /api/incidents/duplications/{id}/reject_merge/
+```
+
+### Patterns
+
+#### Get Patterns
+```http
+GET /api/incidents/patterns/
+```
+
+#### Get Active Patterns
+```http
+GET /api/incidents/patterns/active_patterns/
+```
+
+#### Resolve Pattern
+```http
+POST /api/incidents/patterns/{id}/resolve_pattern/
+```
+
+## Data Models
+
+### Incident
+- **id**: UUID primary key
+- **title**: Incident title
+- **description**: Detailed description
+- **free_text**: Original free text from user
+- **category**: AI-classified category
+- **subcategory**: AI-classified subcategory
+- **severity**: Current severity level
+- **suggested_severity**: AI-suggested severity
+- **status**: Current status (Open, In Progress, Resolved, Closed)
+- **assigned_to**: Assigned user
+- **reporter**: User who reported the incident
+- **affected_users**: Number of affected users
+- **business_impact**: Business impact description
+- **ai_processed**: Whether AI analysis has been completed
+- **is_duplicate**: Whether this is a duplicate incident
+
+### IncidentClassification
+- **incident**: Related incident
+- **predicted_category**: AI-predicted category
+- **predicted_subcategory**: AI-predicted subcategory
+- **confidence_score**: AI confidence (0.0-1.0)
+- **alternative_categories**: Alternative predictions
+- **extracted_keywords**: Keywords extracted from text
+- **sentiment_score**: Sentiment analysis score (-1 to 1)
+- **urgency_indicators**: Detected urgency indicators
+
+### SeveritySuggestion
+- **incident**: Related incident
+- **suggested_severity**: AI-suggested severity
+- **confidence_score**: AI confidence (0.0-1.0)
+- **user_impact_score**: User impact score (0.0-1.0)
+- **business_impact_score**: Business impact score (0.0-1.0)
+- **technical_impact_score**: Technical impact score (0.0-1.0)
+- **reasoning**: AI explanation for suggestion
+- **impact_factors**: Factors that influenced the severity
+
+### IncidentCorrelation
+- **primary_incident**: Primary incident in correlation
+- **related_incident**: Related incident
+- **correlation_type**: Type of correlation
+- **confidence_score**: Correlation confidence (0.0-1.0)
+- **correlation_strength**: Strength of correlation
+- **shared_keywords**: Keywords shared between incidents
+- **time_difference**: Time difference between incidents
+- **similarity_score**: Overall similarity score
+- **is_problem_indicator**: Whether this suggests a larger problem
+
+### DuplicationDetection
+- **incident_a**: First incident in pair
+- **incident_b**: Second incident in pair
+- **duplication_type**: Type of duplication
+- **similarity_score**: Overall similarity score
+- **confidence_score**: Duplication confidence (0.0-1.0)
+- **text_similarity**: Text similarity score
+- **temporal_proximity**: Temporal proximity score
+- **service_similarity**: Service similarity score
+- **recommended_action**: Recommended action (Merge, Link, Review, No Action)
+- **status**: Current status (Detected, Reviewed, Merged, Rejected)
+
+### IncidentPattern
+- **name**: Pattern name
+- **pattern_type**: Type of pattern (Recurring, Seasonal, Trend, Anomaly)
+- **description**: Pattern description
+- **frequency**: How often the pattern occurs
+- **affected_services**: Services affected by the pattern
+- **common_keywords**: Common keywords in pattern incidents
+- **incidents**: Related incidents
+- **confidence_score**: Pattern confidence (0.0-1.0)
+- **is_active**: Whether the pattern is active
+- **is_resolved**: Whether the pattern is resolved
+
+## AI Components
+
+### IncidentClassifier
+- **Categories**: Predefined categories with keywords
+- **Keyword Extraction**: Extracts relevant keywords from text
+- **Sentiment Analysis**: Analyzes sentiment of incident text
+- **Urgency Detection**: Identifies urgency indicators
+- **Confidence Scoring**: Provides confidence scores for classifications
+
+### SeverityAnalyzer
+- **Impact Analysis**: Analyzes user, business, and technical impact
+- **Severity Indicators**: Identifies severity keywords in text
+- **Weighted Scoring**: Combines multiple factors for severity suggestion
+- **Reasoning Generation**: Provides explanations for severity suggestions
+
+### IncidentCorrelationEngine
+- **Similarity Analysis**: Calculates various similarity metrics
+- **Temporal Analysis**: Considers time-based correlations
+- **Service Analysis**: Analyzes service and component similarities
+- **Problem Detection**: Identifies patterns that suggest larger problems
+- **Cluster Detection**: Groups related incidents into clusters
+
+### DuplicationDetector
+- **Text Similarity**: Multiple text similarity algorithms
+- **Temporal Proximity**: Time-based duplication detection
+- **Service Similarity**: Service and component similarity
+- **Metadata Similarity**: Similarity based on incident metadata
+- **Merge Recommendations**: Suggests appropriate actions
+
+## Background Processing
+
+The module uses Celery for background processing of AI analysis:
+
+### Tasks
+- **process_incident_ai**: Processes a single incident with AI analysis
+- **batch_process_incidents_ai**: Processes multiple incidents
+- **find_correlations**: Finds correlations for an incident
+- **find_duplicates**: Finds duplicates for an incident
+- **detect_all_duplicates**: Batch duplicate detection
+- **correlate_all_incidents**: Batch correlation analysis
+- **merge_duplicate_incidents**: Merges duplicate incidents
+
+### Processing Logs
+All AI processing activities are logged in the `AIProcessingLog` model for audit and debugging purposes.
+
+## Setup and Configuration
+
+### 1. Install Dependencies
+```bash
+pip install -r requirements.txt
+```
+
+### 2. Run Migrations
+```bash
+python manage.py makemigrations incident_intelligence
+python manage.py migrate
+```
+
+### 3. Create Sample Data
+```bash
+python manage.py setup_incident_intelligence --create-sample-data --create-patterns
+```
+
+### 4. Run AI Analysis
+```bash
+python manage.py setup_incident_intelligence --run-ai-analysis
+```
+
+### 5. Start Celery Worker
+```bash
+celery -A core worker -l info
+```
+
+## Usage Examples
+
+### Creating an Incident with AI Analysis
+```python
+from incident_intelligence.models import Incident
+from incident_intelligence.tasks import process_incident_ai
+
+# Create incident
+incident = Incident.objects.create(
+    title="API Response Slow",
+    description="The user service API is responding slowly",
+    free_text="API is slow, taking forever to respond",
+    affected_users=50,
+    business_impact="User experience is degraded"
+)
+
+# Trigger AI analysis
+process_incident_ai.delay(incident.id)
+```
+
+### Finding Correlations
+```python
+from incident_intelligence.ai.correlation import IncidentCorrelationEngine
+
+engine = IncidentCorrelationEngine()
+correlations = engine.find_related_incidents(incident_data, all_incidents)
+```
+
+### Detecting Duplicates
+```python
+from incident_intelligence.ai.duplication import DuplicationDetector
+
+detector = DuplicationDetector()
+duplicates = detector.find_duplicate_candidates(incident_data, all_incidents)
+```
+
+## Performance Considerations
+
+- **Batch Processing**: Use batch operations for large datasets
+- **Caching**: Consider caching frequently accessed data
+- **Indexing**: Database indexes are configured for optimal query performance
+- **Background Tasks**: AI processing runs asynchronously to avoid blocking requests
+- **Rate Limiting**: Consider implementing rate limiting for API endpoints
+
+## Security Considerations
+
+- **Authentication**: All endpoints require authentication
+- **Authorization**: Users can only access incidents they have permission to view
+- **Data Privacy**: Sensitive information is handled according to data classification levels
+- **Audit Logging**: All AI processing activities are logged for audit purposes
+
+## Monitoring and Maintenance
+
+- **Processing Logs**: Monitor AI processing logs for errors and performance
+- **Model Performance**: Track AI model accuracy and update as needed
+- **Database Maintenance**: Regular cleanup of old processing logs and resolved incidents
+- **Health Checks**: Monitor Celery workers and Redis for background processing health
+
+## Future Enhancements
+
+- **Machine Learning Models**: Integration with more sophisticated ML models
+- **Real-time Processing**: Real-time incident analysis and correlation
+- **Advanced NLP**: More sophisticated natural language processing
+- **Predictive Analytics**: Predictive incident analysis and prevention
+- **Integration APIs**: APIs for integrating with external incident management systems

ETB-API/incident_intelligence/admin.py

@@ -0,0 +1,355 @@
+"""
+Admin configuration for incident intelligence
+"""
+from django.contrib import admin
+from django.utils.html import format_html
+from django.urls import reverse
+from django.utils.safestring import mark_safe
+from .models import (
+    Incident, IncidentClassification, SeveritySuggestion, IncidentCorrelation,
+    DuplicationDetection, IncidentPattern, AIProcessingLog
+)
+
+
+@admin.register(Incident)
+class IncidentAdmin(admin.ModelAdmin):
+    """Admin interface for Incident model"""
+    
+    list_display = [
+        'title', 'severity', 'status', 'category', 'assigned_to', 'reporter',
+        'created_at', 'ai_processed', 'is_duplicate', 'ai_analysis_link'
+    ]
+    list_filter = [
+        'severity', 'status', 'category', 'ai_processed', 'is_duplicate',
+        'created_at', 'assigned_to', 'reporter'
+    ]
+    search_fields = ['title', 'description', 'free_text']
+    readonly_fields = [
+        'id', 'created_at', 'updated_at', 'ai_processed', 'ai_processing_error',
+        'last_ai_analysis', 'classification_confidence', 'severity_confidence',
+        'duplicate_confidence'
+    ]
+    fieldsets = (
+        ('Basic Information', {
+            'fields': ('id', 'title', 'description', 'free_text')
+        }),
+        ('Classification', {
+            'fields': ('category', 'subcategory', 'classification_confidence')
+        }),
+        ('Severity & Priority', {
+            'fields': ('severity', 'suggested_severity', 'severity_confidence', 'priority')
+        }),
+        ('Status & Assignment', {
+            'fields': ('status', 'assigned_to', 'reporter')
+        }),
+        ('Impact & Business Context', {
+            'fields': ('affected_users', 'business_impact', 'estimated_downtime')
+        }),
+        ('AI Processing', {
+            'fields': ('ai_processed', 'ai_processing_error', 'last_ai_analysis'),
+            'classes': ('collapse',)
+        }),
+        ('Duplication Detection', {
+            'fields': ('is_duplicate', 'original_incident', 'duplicate_confidence'),
+            'classes': ('collapse',)
+        }),
+        ('Timestamps', {
+            'fields': ('created_at', 'updated_at', 'resolved_at'),
+            'classes': ('collapse',)
+        })
+    )
+    
+    def ai_analysis_link(self, obj):
+        """Link to AI analysis details"""
+        if obj.ai_processed:
+            url = reverse('admin:incident_intelligence_incident_analysis', args=[obj.pk])
+            return format_html('<a href="{}">View Analysis</a>', url)
+        return "Not Processed"
+    ai_analysis_link.short_description = "AI Analysis"
+    
+    actions = ['trigger_ai_analysis', 'mark_as_duplicate', 'mark_as_resolved']
+    
+    def trigger_ai_analysis(self, request, queryset):
+        """Trigger AI analysis for selected incidents"""
+        from .tasks import batch_process_incidents_ai
+        incident_ids = [str(incident.id) for incident in queryset]
+        batch_process_incidents_ai.delay(incident_ids)
+        self.message_user(request, f"AI analysis triggered for {len(incident_ids)} incidents")
+    trigger_ai_analysis.short_description = "Trigger AI Analysis"
+    
+    def mark_as_duplicate(self, request, queryset):
+        """Mark selected incidents as duplicates"""
+        count = queryset.update(is_duplicate=True)
+        self.message_user(request, f"Marked {count} incidents as duplicates")
+    mark_as_duplicate.short_description = "Mark as Duplicates"
+    
+    def mark_as_resolved(self, request, queryset):
+        """Mark selected incidents as resolved"""
+        count = queryset.update(status='RESOLVED')
+        self.message_user(request, f"Marked {count} incidents as resolved")
+    mark_as_resolved.short_description = "Mark as Resolved"
+
+
+@admin.register(IncidentClassification)
+class IncidentClassificationAdmin(admin.ModelAdmin):
+    """Admin interface for IncidentClassification model"""
+    
+    list_display = [
+        'incident', 'predicted_category', 'predicted_subcategory',
+        'confidence_score', 'model_version', 'created_at'
+    ]
+    list_filter = ['predicted_category', 'predicted_subcategory', 'model_version', 'created_at']
+    search_fields = ['incident__title', 'predicted_category', 'predicted_subcategory']
+    readonly_fields = ['created_at']
+    
+    fieldsets = (
+        ('Classification Results', {
+            'fields': ('incident', 'predicted_category', 'predicted_subcategory', 'confidence_score')
+        }),
+        ('Alternative Classifications', {
+            'fields': ('alternative_categories',),
+            'classes': ('collapse',)
+        }),
+        ('NLP Analysis', {
+            'fields': ('extracted_keywords', 'sentiment_score', 'urgency_indicators'),
+            'classes': ('collapse',)
+        }),
+        ('Processing Metadata', {
+            'fields': ('model_version', 'processing_time', 'created_at'),
+            'classes': ('collapse',)
+        })
+    )
+
+
+@admin.register(SeveritySuggestion)
+class SeveritySuggestionAdmin(admin.ModelAdmin):
+    """Admin interface for SeveritySuggestion model"""
+    
+    list_display = [
+        'incident', 'suggested_severity', 'confidence_score',
+        'user_impact_score', 'business_impact_score', 'technical_impact_score',
+        'created_at'
+    ]
+    list_filter = ['suggested_severity', 'model_version', 'created_at']
+    search_fields = ['incident__title', 'reasoning']
+    readonly_fields = ['created_at']
+    
+    fieldsets = (
+        ('Severity Prediction', {
+            'fields': ('incident', 'suggested_severity', 'confidence_score')
+        }),
+        ('Impact Analysis', {
+            'fields': ('user_impact_score', 'business_impact_score', 'technical_impact_score')
+        }),
+        ('Reasoning', {
+            'fields': ('reasoning', 'impact_factors'),
+            'classes': ('collapse',)
+        }),
+        ('Processing Metadata', {
+            'fields': ('model_version', 'processing_time', 'created_at'),
+            'classes': ('collapse',)
+        })
+    )
+
+
+@admin.register(IncidentCorrelation)
+class IncidentCorrelationAdmin(admin.ModelAdmin):
+    """Admin interface for IncidentCorrelation model"""
+    
+    list_display = [
+        'primary_incident', 'related_incident', 'correlation_type',
+        'correlation_strength', 'confidence_score', 'is_problem_indicator',
+        'created_at'
+    ]
+    list_filter = [
+        'correlation_type', 'correlation_strength', 'is_problem_indicator',
+        'model_version', 'created_at'
+    ]
+    search_fields = ['primary_incident__title', 'related_incident__title']
+    readonly_fields = ['created_at']
+    
+    fieldsets = (
+        ('Correlation Details', {
+            'fields': ('primary_incident', 'related_incident', 'correlation_type', 'correlation_strength')
+        }),
+        ('Analysis Results', {
+            'fields': ('confidence_score', 'similarity_score', 'time_difference')
+        }),
+        ('Shared Elements', {
+            'fields': ('shared_keywords',),
+            'classes': ('collapse',)
+        }),
+        ('Problem Detection', {
+            'fields': ('is_problem_indicator', 'problem_description'),
+            'classes': ('collapse',)
+        }),
+        ('Metadata', {
+            'fields': ('model_version', 'created_at'),
+            'classes': ('collapse',)
+        })
+    )
+
+
+@admin.register(DuplicationDetection)
+class DuplicationDetectionAdmin(admin.ModelAdmin):
+    """Admin interface for DuplicationDetection model"""
+    
+    list_display = [
+        'incident_a', 'incident_b', 'duplication_type', 'confidence_score',
+        'recommended_action', 'status', 'created_at'
+    ]
+    list_filter = [
+        'duplication_type', 'recommended_action', 'status', 'model_version', 'created_at'
+    ]
+    search_fields = ['incident_a__title', 'incident_b__title']
+    readonly_fields = ['created_at', 'reviewed_at']
+    
+    fieldsets = (
+        ('Incident Pair', {
+            'fields': ('incident_a', 'incident_b')
+        }),
+        ('Duplication Analysis', {
+            'fields': ('duplication_type', 'confidence_score', 'similarity_score')
+        }),
+        ('Similarity Breakdown', {
+            'fields': ('text_similarity', 'temporal_proximity', 'service_similarity'),
+            'classes': ('collapse',)
+        }),
+        ('Recommendation', {
+            'fields': ('recommended_action', 'merge_confidence', 'reasoning')
+        }),
+        ('Shared Elements', {
+            'fields': ('shared_elements',),
+            'classes': ('collapse',)
+        }),
+        ('Status & Review', {
+            'fields': ('status', 'reviewed_at', 'reviewed_by'),
+            'classes': ('collapse',)
+        }),
+        ('Metadata', {
+            'fields': ('model_version', 'created_at'),
+            'classes': ('collapse',)
+        })
+    )
+    
+    actions = ['approve_merge', 'reject_merge']
+    
+    def approve_merge(self, request, queryset):
+        """Approve merging of selected duplications"""
+        from .tasks import merge_duplicate_incidents
+        for duplication in queryset.filter(status='DETECTED'):
+            duplication.status = 'REVIEWED'
+            duplication.reviewed_by = request.user
+            duplication.save()
+            merge_duplicate_incidents.delay(duplication.id)
+        self.message_user(request, f"Approved {queryset.count()} duplications for merging")
+    approve_merge.short_description = "Approve Merge"
+    
+    def reject_merge(self, request, queryset):
+        """Reject merging of selected duplications"""
+        count = queryset.filter(status='DETECTED').update(
+            status='REJECTED',
+            reviewed_by=request.user
+        )
+        self.message_user(request, f"Rejected {count} duplications")
+    reject_merge.short_description = "Reject Merge"
+
+
+@admin.register(IncidentPattern)
+class IncidentPatternAdmin(admin.ModelAdmin):
+    """Admin interface for IncidentPattern model"""
+    
+    list_display = [
+        'name', 'pattern_type', 'incident_count', 'confidence_score',
+        'is_active', 'is_resolved', 'created_at'
+    ]
+    list_filter = ['pattern_type', 'is_active', 'is_resolved', 'model_version', 'created_at']
+    search_fields = ['name', 'description']
+    readonly_fields = ['incident_count', 'created_at', 'updated_at']
+    filter_horizontal = ['incidents']
+    
+    fieldsets = (
+        ('Pattern Information', {
+            'fields': ('name', 'pattern_type', 'description')
+        }),
+        ('Pattern Characteristics', {
+            'fields': ('frequency', 'affected_services', 'common_keywords')
+        }),
+        ('Related Incidents', {
+            'fields': ('incidents', 'incident_count')
+        }),
+        ('Analysis Results', {
+            'fields': ('confidence_score', 'last_occurrence', 'next_predicted_occurrence')
+        }),
+        ('Status', {
+            'fields': ('is_active', 'is_resolved')
+        }),
+        ('Metadata', {
+            'fields': ('model_version', 'created_at', 'updated_at'),
+            'classes': ('collapse',)
+        })
+    )
+    
+    actions = ['resolve_pattern', 'activate_pattern']
+    
+    def resolve_pattern(self, request, queryset):
+        """Mark selected patterns as resolved"""
+        count = queryset.update(is_resolved=True, is_active=False)
+        self.message_user(request, f"Resolved {count} patterns")
+    resolve_pattern.short_description = "Resolve Patterns"
+    
+    def activate_pattern(self, request, queryset):
+        """Activate selected patterns"""
+        count = queryset.update(is_active=True, is_resolved=False)
+        self.message_user(request, f"Activated {count} patterns")
+    activate_pattern.short_description = "Activate Patterns"
+
+
+@admin.register(AIProcessingLog)
+class AIProcessingLogAdmin(admin.ModelAdmin):
+    """Admin interface for AIProcessingLog model"""
+    
+    list_display = [
+        'processing_type', 'incident', 'status', 'processing_time',
+        'confidence_score', 'started_at', 'completed_at'
+    ]
+    list_filter = [
+        'processing_type', 'status', 'model_version', 'started_at'
+    ]
+    search_fields = ['incident__title', 'error_message']
+    readonly_fields = ['started_at', 'completed_at']
+    
+    fieldsets = (
+        ('Processing Details', {
+            'fields': ('processing_type', 'status', 'incident', 'related_incidents')
+        }),
+        ('Input/Output Data', {
+            'fields': ('input_data', 'output_data'),
+            'classes': ('collapse',)
+        }),
+        ('Error Information', {
+            'fields': ('error_message',),
+            'classes': ('collapse',)
+        }),
+        ('Performance Metrics', {
+            'fields': ('processing_time', 'confidence_score', 'model_version')
+        }),
+        ('Timestamps', {
+            'fields': ('started_at', 'completed_at'),
+            'classes': ('collapse',)
+        })
+    )
+    
+    def has_add_permission(self, request):
+        """Disable adding new processing logs"""
+        return False
+    
+    def has_change_permission(self, request, obj=None):
+        """Disable editing processing logs"""
+        return False
+
+
+# Custom admin site configuration
+admin.site.site_header = "ETB Incident Intelligence Admin"
+admin.site.site_title = "Incident Intelligence"
+admin.site.index_title = "Incident Intelligence Administration"

ETB-API/incident_intelligence/ai/__init__.py

@@ -0,0 +1 @@
+# AI components for incident intelligence

ETB-API/incident_intelligence/ai/classification.py

@@ -0,0 +1,471 @@
+"""
+AI-driven incident classification using NLP techniques
+"""
+import re
+import time
+from typing import Dict, List, Tuple, Optional
+from dataclasses import dataclass
+from django.conf import settings
+
+
+@dataclass
+class ClassificationResult:
+    """Result of incident classification"""
+    category: str
+    subcategory: str
+    confidence: float
+    alternative_categories: List[Dict[str, float]]
+    keywords: List[str]
+    sentiment_score: float
+    urgency_indicators: List[str]
+
+
+class IncidentClassifier:
+    """
+    AI-driven incident classifier using rule-based and ML techniques
+    """
+    
+    def __init__(self):
+        self.model_version = "v1.0"
+        
+        # Predefined categories and their keywords
+        self.categories = {
+            'INFRASTRUCTURE': {
+                'keywords': ['server', 'database', 'network', 'storage', 'disk', 'memory', 'cpu', 'load', 'bandwidth', 'connection', 'timeout', 'latency'],
+                'subcategories': {
+                    'SERVER_ISSUE': ['server', 'host', 'machine', 'instance', 'vm', 'container'],
+                    'DATABASE_ISSUE': ['database', 'db', 'sql', 'query', 'connection', 'timeout', 'deadlock'],
+                    'NETWORK_ISSUE': ['network', 'connectivity', 'dns', 'firewall', 'routing', 'packet', 'bandwidth'],
+                    'STORAGE_ISSUE': ['storage', 'disk', 'volume', 'space', 'capacity', 'i/o', 'read', 'write'],
+                }
+            },
+            'APPLICATION': {
+                'keywords': ['application', 'app', 'service', 'api', 'endpoint', 'response', 'error', 'exception', 'crash', 'bug'],
+                'subcategories': {
+                    'API_ISSUE': ['api', 'endpoint', 'response', 'status', 'code', 'timeout', 'rate', 'limit'],
+                    'SERVICE_ISSUE': ['service', 'microservice', 'dependency', 'circuit', 'breaker', 'fallback'],
+                    'PERFORMANCE_ISSUE': ['performance', 'slow', 'latency', 'response', 'time', 'throughput', 'bottleneck'],
+                    'FUNCTIONALITY_ISSUE': ['bug', 'feature', 'functionality', 'behavior', 'unexpected', 'incorrect'],
+                }
+            },
+            'SECURITY': {
+                'keywords': ['security', 'authentication', 'authorization', 'access', 'permission', 'breach', 'attack', 'vulnerability', 'malware'],
+                'subcategories': {
+                    'AUTH_ISSUE': ['authentication', 'login', 'password', 'token', 'session', 'credential'],
+                    'ACCESS_ISSUE': ['authorization', 'permission', 'access', 'denied', 'forbidden', 'unauthorized'],
+                    'THREAT_ISSUE': ['attack', 'breach', 'malware', 'virus', 'intrusion', 'suspicious', 'anomaly'],
+                    'VULNERABILITY': ['vulnerability', 'exploit', 'patch', 'update', 'security', 'fix'],
+                }
+            },
+            'USER_EXPERIENCE': {
+                'keywords': ['user', 'interface', 'ui', 'ux', 'experience', 'usability', 'navigation', 'button', 'form', 'page'],
+                'subcategories': {
+                    'UI_ISSUE': ['interface', 'ui', 'button', 'form', 'page', 'layout', 'display', 'rendering'],
+                    'NAVIGATION_ISSUE': ['navigation', 'menu', 'link', 'redirect', 'routing', 'page', 'not', 'found'],
+                    'USABILITY_ISSUE': ['usability', 'experience', 'confusing', 'difficult', 'unclear', 'intuitive'],
+                    'MOBILE_ISSUE': ['mobile', 'app', 'responsive', 'device', 'screen', 'touch', 'gesture'],
+                }
+            },
+            'DATA': {
+                'keywords': ['data', 'file', 'import', 'export', 'sync', 'backup', 'recovery', 'corruption', 'missing', 'duplicate'],
+                'subcategories': {
+                    'DATA_CORRUPTION': ['corruption', 'corrupted', 'invalid', 'malformed', 'broken', 'damaged'],
+                    'DATA_LOSS': ['missing', 'lost', 'deleted', 'removed', 'disappeared', 'not', 'found'],
+                    'SYNC_ISSUE': ['sync', 'synchronization', 'conflict', 'merge', 'update', 'latest'],
+                    'BACKUP_ISSUE': ['backup', 'restore', 'recovery', 'archive', 'retention', 'storage'],
+                }
+            },
+            'INTEGRATION': {
+                'keywords': ['integration', 'third-party', 'external', 'webhook', 'api', 'connection', 'sync', 'import', 'export'],
+                'subcategories': {
+                    'THIRD_PARTY_ISSUE': ['third-party', 'external', 'vendor', 'partner', 'service', 'provider'],
+                    'WEBHOOK_ISSUE': ['webhook', 'callback', 'notification', 'event', 'trigger', 'delivery'],
+                    'API_INTEGRATION': ['api', 'integration', 'endpoint', 'connection', 'authentication', 'response'],
+                    'DATA_INTEGRATION': ['import', 'export', 'migration', 'transformation', 'mapping', 'format'],
+                }
+            }
+        }
+        
+        # Urgency indicators
+        self.urgency_indicators = {
+            'CRITICAL': ['down', 'outage', 'critical', 'emergency', 'urgent', 'immediate', 'severe', 'complete', 'total'],
+            'HIGH': ['major', 'significant', 'important', 'priority', 'escalate', 'escalated', 'blocking'],
+            'MEDIUM': ['moderate', 'some', 'partial', 'intermittent', 'occasional', 'sometimes'],
+            'LOW': ['minor', 'small', 'cosmetic', 'enhancement', 'improvement', 'suggestion']
+        }
+        
+        # Sentiment analysis keywords
+        self.sentiment_keywords = {
+            'positive': ['working', 'fixed', 'resolved', 'good', 'excellent', 'improved', 'better', 'success'],
+            'negative': ['broken', 'failed', 'error', 'issue', 'problem', 'bug', 'crash', 'down', 'slow', 'terrible', 'awful'],
+            'neutral': ['report', 'incident', 'ticket', 'request', 'update', 'status', 'information']
+        }
+
+    def classify_incident(self, title: str, description: str, free_text: str = "") -> ClassificationResult:
+        """
+        Classify an incident based on its text content
+        """
+        start_time = time.time()
+        
+        # Combine all text for analysis
+        combined_text = f"{title} {description} {free_text}".lower()
+        
+        # Extract keywords
+        keywords = self._extract_keywords(combined_text)
+        
+        # Analyze sentiment
+        sentiment_score = self._analyze_sentiment(combined_text)
+        
+        # Detect urgency indicators
+        urgency_indicators = self._detect_urgency_indicators(combined_text)
+        
+        # Classify category and subcategory
+        category, subcategory, confidence, alternatives = self._classify_category(combined_text, keywords)
+        
+        processing_time = time.time() - start_time
+        
+        return ClassificationResult(
+            category=category,
+            subcategory=subcategory,
+            confidence=confidence,
+            alternative_categories=alternatives,
+            keywords=keywords,
+            sentiment_score=sentiment_score,
+            urgency_indicators=urgency_indicators
+        )
+
+    def _extract_keywords(self, text: str) -> List[str]:
+        """Extract relevant keywords from text"""
+        # Simple keyword extraction - in production, use more sophisticated NLP
+        words = re.findall(r'\b[a-zA-Z]{3,}\b', text)
+        
+        # Filter out common stop words
+        stop_words = {'the', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for', 'of', 'with', 'by', 'is', 'are', 'was', 'were', 'be', 'been', 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could', 'should', 'may', 'might', 'can', 'this', 'that', 'these', 'those', 'a', 'an'}
+        
+        keywords = [word for word in words if word not in stop_words]
+        
+        # Count frequency and return top keywords
+        from collections import Counter
+        keyword_counts = Counter(keywords)
+        return [word for word, count in keyword_counts.most_common(10)]
+
+    def _analyze_sentiment(self, text: str) -> float:
+        """Analyze sentiment of the text (-1 to 1)"""
+        positive_count = sum(1 for word in self.sentiment_keywords['positive'] if word in text)
+        negative_count = sum(1 for word in self.sentiment_keywords['negative'] if word in text)
+        
+        total_sentiment_words = positive_count + negative_count
+        if total_sentiment_words == 0:
+            return 0.0
+        
+        return (positive_count - negative_count) / total_sentiment_words
+
+    def _detect_urgency_indicators(self, text: str) -> List[str]:
+        """Detect urgency indicators in the text"""
+        detected_indicators = []
+        
+        for urgency_level, indicators in self.urgency_indicators.items():
+            for indicator in indicators:
+                if indicator in text:
+                    detected_indicators.append(f"{urgency_level}: {indicator}")
+        
+        return detected_indicators
+
+    def _classify_category(self, text: str, keywords: List[str]) -> Tuple[str, str, float, List[Dict[str, float]]]:
+        """Classify the incident category and subcategory"""
+        category_scores = {}
+        subcategory_scores = {}
+        
+        # Score each category based on keyword matches
+        for category, data in self.categories.items():
+            score = 0
+            category_keywords = data['keywords']
+            
+            # Count keyword matches
+            for keyword in category_keywords:
+                if keyword in text:
+                    score += 1
+                # Also check for partial matches in keywords list
+                for extracted_keyword in keywords:
+                    if keyword in extracted_keyword or extracted_keyword in keyword:
+                        score += 0.5
+            
+            category_scores[category] = score
+            
+            # Score subcategories
+            for subcategory, subcategory_keywords in data['subcategories'].items():
+                subcategory_score = 0
+                for keyword in subcategory_keywords:
+                    if keyword in text:
+                        subcategory_score += 1
+                    for extracted_keyword in keywords:
+                        if keyword in extracted_keyword or extracted_keyword in keyword:
+                            subcategory_score += 0.5
+                
+                subcategory_scores[subcategory] = subcategory_score
+        
+        # Find best category
+        if not category_scores or max(category_scores.values()) == 0:
+            best_category = 'GENERAL'
+            best_subcategory = 'UNKNOWN'
+            confidence = 0.1
+        else:
+            best_category = max(category_scores, key=category_scores.get)
+            max_score = max(category_scores.values())
+            confidence = min(max_score / 10.0, 1.0)  # Normalize to 0-1
+            
+            # Find best subcategory within the category
+            if best_category in self.categories:
+                category_subcategories = self.categories[best_category]['subcategories']
+                subcategory_scores_filtered = {k: v for k, v in subcategory_scores.items() if k in category_subcategories}
+                
+                if subcategory_scores_filtered and max(subcategory_scores_filtered.values()) > 0:
+                    best_subcategory = max(subcategory_scores_filtered, key=subcategory_scores_filtered.get)
+                else:
+                    best_subcategory = 'GENERAL'
+            else:
+                best_subcategory = 'GENERAL'
+        
+        # Create alternative categories
+        alternatives = []
+        sorted_categories = sorted(category_scores.items(), key=lambda x: x[1], reverse=True)
+        for category, score in sorted_categories[:3]:
+            if category != best_category and score > 0:
+                alternatives.append({
+                    'category': category,
+                    'confidence': min(score / 10.0, 1.0)
+                })
+        
+        return best_category, best_subcategory, confidence, alternatives
+
+
+class SeverityAnalyzer:
+    """
+    AI-driven severity analyzer based on impact assessment
+    """
+    
+    def __init__(self):
+        self.model_version = "v1.0"
+        
+        # Severity indicators
+        self.severity_indicators = {
+            'EMERGENCY': {
+                'keywords': ['down', 'outage', 'critical', 'emergency', 'complete', 'total', 'all', 'entire', 'system'],
+                'impact_multiplier': 2.0,
+                'user_impact_threshold': 0.8,
+                'business_impact_threshold': 0.9
+            },
+            'CRITICAL': {
+                'keywords': ['major', 'significant', 'severe', 'blocking', 'cannot', 'unable', 'failed', 'broken'],
+                'impact_multiplier': 1.5,
+                'user_impact_threshold': 0.6,
+                'business_impact_threshold': 0.7
+            },
+            'HIGH': {
+                'keywords': ['important', 'priority', 'escalate', 'escalated', 'urgent', 'immediate', 'soon'],
+                'impact_multiplier': 1.2,
+                'user_impact_threshold': 0.4,
+                'business_impact_threshold': 0.5
+            },
+            'MEDIUM': {
+                'keywords': ['moderate', 'some', 'partial', 'intermittent', 'occasional', 'sometimes', 'minor'],
+                'impact_multiplier': 1.0,
+                'user_impact_threshold': 0.2,
+                'business_impact_threshold': 0.3
+            },
+            'LOW': {
+                'keywords': ['small', 'cosmetic', 'enhancement', 'improvement', 'suggestion', 'nice', 'to', 'have'],
+                'impact_multiplier': 0.5,
+                'user_impact_threshold': 0.1,
+                'business_impact_threshold': 0.1
+            }
+        }
+
+    def analyze_severity(self, incident_data: Dict) -> Dict:
+        """
+        Analyze incident severity based on various factors
+        """
+        start_time = time.time()
+        
+        title = incident_data.get('title', '').lower()
+        description = incident_data.get('description', '').lower()
+        free_text = incident_data.get('free_text', '').lower()
+        affected_users = incident_data.get('affected_users', 0)
+        business_impact = incident_data.get('business_impact', '').lower()
+        
+        combined_text = f"{title} {description} {free_text} {business_impact}"
+        
+        # Calculate impact scores
+        user_impact_score = self._calculate_user_impact(affected_users, combined_text)
+        business_impact_score = self._calculate_business_impact(business_impact, combined_text)
+        technical_impact_score = self._calculate_technical_impact(combined_text)
+        
+        # Determine severity based on impact scores and keywords
+        suggested_severity, confidence, reasoning, impact_factors = self._determine_severity(
+            combined_text, user_impact_score, business_impact_score, technical_impact_score
+        )
+        
+        processing_time = time.time() - start_time
+        
+        return {
+            'suggested_severity': suggested_severity,
+            'confidence_score': confidence,
+            'user_impact_score': user_impact_score,
+            'business_impact_score': business_impact_score,
+            'technical_impact_score': technical_impact_score,
+            'reasoning': reasoning,
+            'impact_factors': impact_factors,
+            'processing_time': processing_time
+        }
+
+    def _calculate_user_impact(self, affected_users: int, text: str) -> float:
+        """Calculate user impact score (0-1)"""
+        # Base score from affected users count
+        if affected_users == 0:
+            # Try to extract from text
+            user_indicators = ['all users', 'everyone', 'entire user base', 'all customers']
+            if any(indicator in text for indicator in user_indicators):
+                base_score = 0.9
+            else:
+                base_score = 0.1
+        elif affected_users < 10:
+            base_score = 0.2
+        elif affected_users < 100:
+            base_score = 0.4
+        elif affected_users < 1000:
+            base_score = 0.6
+        elif affected_users < 10000:
+            base_score = 0.8
+        else:
+            base_score = 1.0
+        
+        # Adjust based on text indicators
+        if 'all' in text or 'everyone' in text:
+            base_score = min(base_score + 0.2, 1.0)
+        elif 'some' in text or 'few' in text:
+            base_score = max(base_score - 0.1, 0.0)
+        
+        return base_score
+
+    def _calculate_business_impact(self, business_impact: str, text: str) -> float:
+        """Calculate business impact score (0-1)"""
+        if not business_impact:
+            # Try to infer from text
+            high_impact_indicators = ['revenue', 'sales', 'customer', 'business', 'critical', 'essential', 'production']
+            if any(indicator in text for indicator in high_impact_indicators):
+                return 0.6
+            return 0.3
+        
+        # Analyze business impact text
+        high_impact_keywords = ['revenue', 'sales', 'customer', 'business', 'critical', 'essential', 'production', 'outage', 'down']
+        medium_impact_keywords = ['service', 'feature', 'functionality', 'performance', 'slow']
+        low_impact_keywords = ['cosmetic', 'minor', 'enhancement', 'improvement']
+        
+        score = 0.3  # Base score
+        
+        for keyword in high_impact_keywords:
+            if keyword in business_impact:
+                score += 0.1
+        
+        for keyword in medium_impact_keywords:
+            if keyword in business_impact:
+                score += 0.05
+        
+        for keyword in low_impact_keywords:
+            if keyword in business_impact:
+                score -= 0.05
+        
+        return min(max(score, 0.0), 1.0)
+
+    def _calculate_technical_impact(self, text: str) -> float:
+        """Calculate technical impact score (0-1)"""
+        technical_indicators = {
+            'high': ['down', 'outage', 'crash', 'failed', 'broken', 'unavailable', 'error', 'exception'],
+            'medium': ['slow', 'performance', 'latency', 'timeout', 'intermittent', 'partial'],
+            'low': ['cosmetic', 'display', 'ui', 'minor', 'enhancement']
+        }
+        
+        score = 0.3  # Base score
+        
+        for level, keywords in technical_indicators.items():
+            for keyword in keywords:
+                if keyword in text:
+                    if level == 'high':
+                        score += 0.15
+                    elif level == 'medium':
+                        score += 0.08
+                    elif level == 'low':
+                        score -= 0.05
+        
+        return min(max(score, 0.0), 1.0)
+
+    def _determine_severity(self, text: str, user_impact: float, business_impact: float, technical_impact: float) -> Tuple[str, float, str, List[str]]:
+        """Determine severity based on impact scores and text analysis"""
+        impact_factors = []
+        
+        # Calculate weighted impact score
+        weighted_score = (user_impact * 0.4 + business_impact * 0.4 + technical_impact * 0.2)
+        
+        # Check for severity indicators in text
+        severity_scores = {}
+        for severity, data in self.severity_indicators.items():
+            score = 0
+            for keyword in data['keywords']:
+                if keyword in text:
+                    score += 1
+            
+            # Apply impact multiplier
+            score *= data['impact_multiplier']
+            severity_scores[severity] = score
+        
+        # Find best severity match
+        if severity_scores and max(severity_scores.values()) > 0:
+            best_severity = max(severity_scores, key=severity_scores.get)
+            text_confidence = min(max(severity_scores.values()) / 5.0, 1.0)
+        else:
+            # Fallback to impact-based severity
+            if weighted_score >= 0.8:
+                best_severity = 'CRITICAL'
+            elif weighted_score >= 0.6:
+                best_severity = 'HIGH'
+            elif weighted_score >= 0.4:
+                best_severity = 'MEDIUM'
+            else:
+                best_severity = 'LOW'
+            text_confidence = 0.5
+        
+        # Combine text and impact confidence
+        confidence = (text_confidence + (1.0 - abs(weighted_score - self._severity_to_score(best_severity)))) / 2.0
+        
+        # Generate reasoning
+        reasoning_parts = []
+        if user_impact > 0.6:
+            reasoning_parts.append(f"High user impact ({user_impact:.1%})")
+            impact_factors.append(f"User Impact: {user_impact:.1%}")
+        
+        if business_impact > 0.6:
+            reasoning_parts.append(f"Significant business impact ({business_impact:.1%})")
+            impact_factors.append(f"Business Impact: {business_impact:.1%}")
+        
+        if technical_impact > 0.6:
+            reasoning_parts.append(f"Major technical impact ({technical_impact:.1%})")
+            impact_factors.append(f"Technical Impact: {technical_impact:.1%}")
+        
+        if severity_scores and max(severity_scores.values()) > 0:
+            reasoning_parts.append("Severity indicators detected in incident description")
+            impact_factors.append("Text Analysis: Severity keywords found")
+        
+        reasoning = "; ".join(reasoning_parts) if reasoning_parts else "Based on overall impact assessment"
+        
+        return best_severity, confidence, reasoning, impact_factors
+
+    def _severity_to_score(self, severity: str) -> float:
+        """Convert severity level to numeric score"""
+        severity_scores = {
+            'LOW': 0.2,
+            'MEDIUM': 0.4,
+            'HIGH': 0.6,
+            'CRITICAL': 0.8,
+            'EMERGENCY': 1.0
+        }
+        return severity_scores.get(severity, 0.4)

ETB-API/incident_intelligence/ai/correlation.py

@@ -0,0 +1,481 @@
+"""
+Correlation engine for linking related incidents and problem detection
+"""
+import time
+from typing import Dict, List, Tuple, Optional
+from dataclasses import dataclass
+from datetime import datetime, timedelta
+from django.utils import timezone
+from .classification import IncidentClassifier
+
+
+@dataclass
+class CorrelationResult:
+    """Result of incident correlation analysis"""
+    correlation_type: str
+    confidence_score: float
+    correlation_strength: str
+    shared_keywords: List[str]
+    time_difference: timedelta
+    similarity_score: float
+    is_problem_indicator: bool
+    problem_description: Optional[str]
+
+
+class IncidentCorrelationEngine:
+    """
+    AI-driven correlation engine for linking related incidents
+    """
+    
+    def __init__(self):
+        self.model_version = "v1.0"
+        self.classifier = IncidentClassifier()
+        
+        # Correlation thresholds
+        self.correlation_thresholds = {
+            'VERY_STRONG': 0.9,
+            'STRONG': 0.7,
+            'MODERATE': 0.5,
+            'WEAK': 0.3
+        }
+        
+        # Problem detection patterns
+        self.problem_patterns = {
+            'CASCADE_FAILURE': {
+                'keywords': ['cascade', 'chain', 'reaction', 'domino', 'ripple', 'effect'],
+                'time_window': timedelta(hours=2),
+                'min_incidents': 3
+            },
+            'RECURRING_ISSUE': {
+                'keywords': ['same', 'again', 'recurring', 'repeated', 'similar', 'identical'],
+                'time_window': timedelta(days=7),
+                'min_incidents': 2
+            },
+            'SERVICE_DEPENDENCY': {
+                'keywords': ['dependency', 'dependent', 'downstream', 'upstream', 'service', 'api'],
+                'time_window': timedelta(hours=1),
+                'min_incidents': 2
+            },
+            'INFRASTRUCTURE_PATTERN': {
+                'keywords': ['server', 'database', 'network', 'storage', 'infrastructure'],
+                'time_window': timedelta(hours=4),
+                'min_incidents': 3
+            }
+        }
+
+    def correlate_incidents(self, incident_a: Dict, incident_b: Dict) -> Optional[CorrelationResult]:
+        """
+        Correlate two incidents and determine if they are related
+        """
+        # Calculate various similarity metrics
+        text_similarity = self._calculate_text_similarity(incident_a, incident_b)
+        temporal_similarity = self._calculate_temporal_similarity(incident_a, incident_b)
+        service_similarity = self._calculate_service_similarity(incident_a, incident_b)
+        category_similarity = self._calculate_category_similarity(incident_a, incident_b)
+        
+        # Calculate overall similarity score
+        overall_similarity = (
+            text_similarity * 0.4 +
+            temporal_similarity * 0.2 +
+            service_similarity * 0.2 +
+            category_similarity * 0.2
+        )
+        
+        # Determine if incidents are correlated
+        if overall_similarity < 0.3:
+            return None
+        
+        # Determine correlation type
+        correlation_type = self._determine_correlation_type(
+            incident_a, incident_b, text_similarity, temporal_similarity, service_similarity
+        )
+        
+        # Calculate confidence score
+        confidence_score = self._calculate_confidence_score(
+            overall_similarity, correlation_type, incident_a, incident_b
+        )
+        
+        # Determine correlation strength
+        correlation_strength = self._determine_correlation_strength(confidence_score)
+        
+        # Extract shared keywords
+        shared_keywords = self._extract_shared_keywords(incident_a, incident_b)
+        
+        # Calculate time difference
+        time_diff = self._calculate_time_difference(incident_a, incident_b)
+        
+        # Check for problem indicators
+        is_problem_indicator, problem_description = self._detect_problem_patterns(
+            incident_a, incident_b, correlation_type, confidence_score
+        )
+        
+        return CorrelationResult(
+            correlation_type=correlation_type,
+            confidence_score=confidence_score,
+            correlation_strength=correlation_strength,
+            shared_keywords=shared_keywords,
+            time_difference=time_diff,
+            similarity_score=overall_similarity,
+            is_problem_indicator=is_problem_indicator,
+            problem_description=problem_description
+        )
+
+    def _calculate_text_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate text similarity between two incidents"""
+        # Combine text fields
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')} {incident_a.get('free_text', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')} {incident_b.get('free_text', '')}".lower()
+        
+        # Extract keywords
+        keywords_a = set(self.classifier._extract_keywords(text_a))
+        keywords_b = set(self.classifier._extract_keywords(text_b))
+        
+        if not keywords_a or not keywords_b:
+            return 0.0
+        
+        # Calculate Jaccard similarity
+        intersection = len(keywords_a.intersection(keywords_b))
+        union = len(keywords_a.union(keywords_b))
+        
+        jaccard_similarity = intersection / union if union > 0 else 0.0
+        
+        # Also check for exact phrase matches
+        phrase_similarity = self._calculate_phrase_similarity(text_a, text_b)
+        
+        # Combine similarities
+        return (jaccard_similarity * 0.7 + phrase_similarity * 0.3)
+
+    def _calculate_phrase_similarity(self, text_a: str, text_b: str) -> float:
+        """Calculate similarity based on common phrases"""
+        # Extract 2-3 word phrases
+        phrases_a = set()
+        phrases_b = set()
+        
+        words_a = text_a.split()
+        words_b = text_b.split()
+        
+        # Extract 2-word phrases
+        for i in range(len(words_a) - 1):
+            phrases_a.add(f"{words_a[i]} {words_a[i+1]}")
+        
+        for i in range(len(words_b) - 1):
+            phrases_b.add(f"{words_b[i]} {words_b[i+1]}")
+        
+        # Extract 3-word phrases
+        for i in range(len(words_a) - 2):
+            phrases_a.add(f"{words_a[i]} {words_a[i+1]} {words_a[i+2]}")
+        
+        for i in range(len(words_b) - 2):
+            phrases_b.add(f"{words_b[i]} {words_b[i+1]} {words_b[i+2]}")
+        
+        if not phrases_a or not phrases_b:
+            return 0.0
+        
+        intersection = len(phrases_a.intersection(phrases_b))
+        union = len(phrases_a.union(phrases_b))
+        
+        return intersection / union if union > 0 else 0.0
+
+    def _calculate_temporal_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate temporal similarity between incidents"""
+        created_a = incident_a.get('created_at')
+        created_b = incident_b.get('created_at')
+        
+        if not created_a or not created_b:
+            return 0.0
+        
+        # Convert to datetime if needed
+        if isinstance(created_a, str):
+            created_a = datetime.fromisoformat(created_a.replace('Z', '+00:00'))
+        if isinstance(created_b, str):
+            created_b = datetime.fromisoformat(created_b.replace('Z', '+00:00'))
+        
+        time_diff = abs((created_a - created_b).total_seconds())
+        
+        # Calculate similarity based on time difference
+        # Incidents within 1 hour: high similarity
+        # Incidents within 24 hours: medium similarity
+        # Incidents within 7 days: low similarity
+        if time_diff <= 3600:  # 1 hour
+            return 1.0
+        elif time_diff <= 86400:  # 24 hours
+            return 0.7
+        elif time_diff <= 604800:  # 7 days
+            return 0.3
+        else:
+            return 0.0
+
+    def _calculate_service_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate service/component similarity"""
+        # Extract service/component information from text
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')}".lower()
+        
+        # Common service/component keywords
+        service_keywords = [
+            'api', 'service', 'database', 'server', 'application', 'website', 'mobile',
+            'frontend', 'backend', 'microservice', 'gateway', 'load balancer', 'cache',
+            'queue', 'message', 'notification', 'email', 'sms', 'payment', 'auth'
+        ]
+        
+        services_a = set()
+        services_b = set()
+        
+        for keyword in service_keywords:
+            if keyword in text_a:
+                services_a.add(keyword)
+            if keyword in text_b:
+                services_b.add(keyword)
+        
+        if not services_a or not services_b:
+            return 0.0
+        
+        intersection = len(services_a.intersection(services_b))
+        union = len(services_a.union(services_b))
+        
+        return intersection / union if union > 0 else 0.0
+
+    def _calculate_category_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate category similarity"""
+        category_a = incident_a.get('category', '')
+        category_b = incident_b.get('category', '')
+        
+        if not category_a or not category_b:
+            return 0.0
+        
+        if category_a == category_b:
+            return 1.0
+        
+        # Check for related categories
+        related_categories = {
+            'INFRASTRUCTURE': ['APPLICATION', 'SECURITY'],
+            'APPLICATION': ['INFRASTRUCTURE', 'USER_EXPERIENCE'],
+            'SECURITY': ['INFRASTRUCTURE', 'APPLICATION'],
+            'USER_EXPERIENCE': ['APPLICATION', 'DATA'],
+            'DATA': ['USER_EXPERIENCE', 'INTEGRATION'],
+            'INTEGRATION': ['DATA', 'APPLICATION']
+        }
+        
+        if category_b in related_categories.get(category_a, []):
+            return 0.5
+        
+        return 0.0
+
+    def _determine_correlation_type(self, incident_a: Dict, incident_b: Dict, 
+                                  text_similarity: float, temporal_similarity: float, 
+                                  service_similarity: float) -> str:
+        """Determine the type of correlation between incidents"""
+        
+        # Same service correlation
+        if service_similarity > 0.7:
+            return 'SAME_SERVICE'
+        
+        # Same component correlation
+        if text_similarity > 0.6 and service_similarity > 0.4:
+            return 'SAME_COMPONENT'
+        
+        # Temporal correlation
+        if temporal_similarity > 0.7 and text_similarity > 0.3:
+            return 'TEMPORAL'
+        
+        # Pattern match
+        if text_similarity > 0.5:
+            return 'PATTERN'
+        
+        # Dependency correlation
+        if service_similarity > 0.4 and temporal_similarity > 0.5:
+            return 'DEPENDENCY'
+        
+        # Cascade effect
+        if temporal_similarity > 0.8 and text_similarity > 0.4:
+            return 'CASCADE'
+        
+        return 'PATTERN'  # Default
+
+    def _calculate_confidence_score(self, overall_similarity: float, correlation_type: str, 
+                                  incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate confidence score for the correlation"""
+        base_confidence = overall_similarity
+        
+        # Adjust based on correlation type
+        type_adjustments = {
+            'SAME_SERVICE': 0.1,
+            'SAME_COMPONENT': 0.15,
+            'TEMPORAL': 0.05,
+            'PATTERN': 0.0,
+            'DEPENDENCY': 0.1,
+            'CASCADE': 0.2
+        }
+        
+        base_confidence += type_adjustments.get(correlation_type, 0.0)
+        
+        # Adjust based on incident characteristics
+        if incident_a.get('severity') == incident_b.get('severity'):
+            base_confidence += 0.05
+        
+        if incident_a.get('status') == incident_b.get('status'):
+            base_confidence += 0.03
+        
+        return min(base_confidence, 1.0)
+
+    def _determine_correlation_strength(self, confidence_score: float) -> str:
+        """Determine correlation strength based on confidence score"""
+        if confidence_score >= self.correlation_thresholds['VERY_STRONG']:
+            return 'VERY_STRONG'
+        elif confidence_score >= self.correlation_thresholds['STRONG']:
+            return 'STRONG'
+        elif confidence_score >= self.correlation_thresholds['MODERATE']:
+            return 'MODERATE'
+        else:
+            return 'WEAK'
+
+    def _extract_shared_keywords(self, incident_a: Dict, incident_b: Dict) -> List[str]:
+        """Extract keywords shared between incidents"""
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')}".lower()
+        
+        keywords_a = set(self.classifier._extract_keywords(text_a))
+        keywords_b = set(self.classifier._extract_keywords(text_b))
+        
+        shared = list(keywords_a.intersection(keywords_b))
+        return shared[:10]  # Return top 10 shared keywords
+
+    def _calculate_time_difference(self, incident_a: Dict, incident_b: Dict) -> timedelta:
+        """Calculate time difference between incidents"""
+        created_a = incident_a.get('created_at')
+        created_b = incident_b.get('created_at')
+        
+        if not created_a or not created_b:
+            return timedelta(0)
+        
+        # Convert to datetime if needed
+        if isinstance(created_a, str):
+            created_a = datetime.fromisoformat(created_a.replace('Z', '+00:00'))
+        if isinstance(created_b, str):
+            created_b = datetime.fromisoformat(created_b.replace('Z', '+00:00'))
+        
+        return abs(created_a - created_b)
+
+    def _detect_problem_patterns(self, incident_a: Dict, incident_b: Dict, 
+                               correlation_type: str, confidence_score: float) -> Tuple[bool, Optional[str]]:
+        """Detect if correlation indicates a larger problem"""
+        
+        # High confidence correlations are more likely to indicate problems
+        if confidence_score < 0.6:
+            return False, None
+        
+        # Check for specific problem patterns
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')}".lower()
+        combined_text = f"{text_a} {text_b}"
+        
+        for pattern_name, pattern_data in self.problem_patterns.items():
+            # Check for pattern keywords
+            keyword_matches = sum(1 for keyword in pattern_data['keywords'] if keyword in combined_text)
+            
+            if keyword_matches >= 2:  # At least 2 keywords match
+                return True, f"Potential {pattern_name.replace('_', ' ').lower()} detected"
+        
+        # Check for cascade effects
+        if correlation_type == 'CASCADE' and confidence_score > 0.7:
+            return True, "Potential cascade failure detected"
+        
+        # Check for recurring issues
+        if correlation_type == 'SAME_SERVICE' and confidence_score > 0.8:
+            return True, "Potential recurring service issue detected"
+        
+        return False, None
+
+    def find_related_incidents(self, target_incident: Dict, all_incidents: List[Dict], 
+                             limit: int = 10) -> List[Tuple[Dict, CorrelationResult]]:
+        """Find incidents related to a target incident"""
+        correlations = []
+        
+        for incident in all_incidents:
+            if incident['id'] == target_incident['id']:
+                continue
+            
+            correlation = self.correlate_incidents(target_incident, incident)
+            if correlation:
+                correlations.append((incident, correlation))
+        
+        # Sort by confidence score and return top results
+        correlations.sort(key=lambda x: x[1].confidence_score, reverse=True)
+        return correlations[:limit]
+
+    def detect_problem_clusters(self, incidents: List[Dict], 
+                              min_incidents: int = 3, 
+                              time_window: timedelta = timedelta(hours=24)) -> List[Dict]:
+        """Detect clusters of related incidents that might indicate larger problems"""
+        clusters = []
+        processed_incidents = set()
+        
+        for incident in incidents:
+            if incident['id'] in processed_incidents:
+                continue
+            
+            # Find related incidents within time window
+            related_incidents = []
+            incident_time = incident.get('created_at')
+            
+            if isinstance(incident_time, str):
+                incident_time = datetime.fromisoformat(incident_time.replace('Z', '+00:00'))
+            
+            for other_incident in incidents:
+                if other_incident['id'] == incident['id'] or other_incident['id'] in processed_incidents:
+                    continue
+                
+                other_time = other_incident.get('created_at')
+                if isinstance(other_time, str):
+                    other_time = datetime.fromisoformat(other_time.replace('Z', '+00:00'))
+                
+                # Check if within time window
+                if abs((incident_time - other_time).total_seconds()) <= time_window.total_seconds():
+                    correlation = self.correlate_incidents(incident, other_incident)
+                    if correlation and correlation.confidence_score > 0.5:
+                        related_incidents.append((other_incident, correlation))
+            
+            # If we found enough related incidents, create a cluster
+            if len(related_incidents) >= min_incidents - 1:  # -1 because we include the original incident
+                cluster = {
+                    'incidents': [incident] + [inc[0] for inc in related_incidents],
+                    'correlations': [inc[1] for inc in related_incidents],
+                    'problem_type': self._classify_problem_type(incident, related_incidents),
+                    'confidence': sum(inc[1].confidence_score for inc in related_incidents) / len(related_incidents),
+                    'time_span': self._calculate_cluster_time_span([incident] + [inc[0] for inc in related_incidents])
+                }
+                clusters.append(cluster)
+                
+                # Mark incidents as processed
+                processed_incidents.add(incident['id'])
+                for related_incident, _ in related_incidents:
+                    processed_incidents.add(related_incident['id'])
+        
+        return clusters
+
+    def _classify_problem_type(self, incident: Dict, related_incidents: List[Tuple[Dict, CorrelationResult]]) -> str:
+        """Classify the type of problem based on incident cluster"""
+        correlation_types = [corr.correlation_type for _, corr in related_incidents]
+        
+        if 'CASCADE' in correlation_types:
+            return 'CASCADE_FAILURE'
+        elif 'SAME_SERVICE' in correlation_types:
+            return 'SERVICE_OUTAGE'
+        elif 'TEMPORAL' in correlation_types:
+            return 'RECURRING_ISSUE'
+        else:
+            return 'PATTERN_BASED_PROBLEM'
+
+    def _calculate_cluster_time_span(self, incidents: List[Dict]) -> timedelta:
+        """Calculate the time span of a cluster of incidents"""
+        times = []
+        for incident in incidents:
+            created_at = incident.get('created_at')
+            if isinstance(created_at, str):
+                created_at = datetime.fromisoformat(created_at.replace('Z', '+00:00'))
+            times.append(created_at)
+        
+        if len(times) < 2:
+            return timedelta(0)
+        
+        return max(times) - min(times)

ETB-API/incident_intelligence/ai/duplication.py

@@ -0,0 +1,516 @@
+"""
+Duplication detection engine for identifying and merging duplicate incidents
+"""
+import time
+from typing import Dict, List, Tuple, Optional
+from dataclasses import dataclass
+from datetime import datetime, timedelta
+from .classification import IncidentClassifier
+
+
+@dataclass
+class DuplicationResult:
+    """Result of duplication detection analysis"""
+    duplication_type: str
+    similarity_score: float
+    confidence_score: float
+    text_similarity: float
+    temporal_proximity: float
+    service_similarity: float
+    recommended_action: str
+    merge_confidence: float
+    reasoning: str
+    shared_elements: List[str]
+
+
+class DuplicationDetector:
+    """
+    AI-driven duplication detector for identifying duplicate incidents
+    """
+    
+    def __init__(self):
+        self.model_version = "v1.0"
+        self.classifier = IncidentClassifier()
+        
+        # Duplication thresholds
+        self.duplication_thresholds = {
+            'EXACT': 0.95,
+            'NEAR_DUPLICATE': 0.85,
+            'SIMILAR': 0.70,
+            'POTENTIAL_DUPLICATE': 0.50
+        }
+        
+        # Action thresholds
+        self.action_thresholds = {
+            'MERGE': 0.90,
+            'LINK': 0.75,
+            'REVIEW': 0.60,
+            'NO_ACTION': 0.0
+        }
+        
+        # Time windows for temporal proximity
+        self.time_windows = {
+            'EXACT': timedelta(minutes=30),
+            'NEAR_DUPLICATE': timedelta(hours=2),
+            'SIMILAR': timedelta(hours=24),
+            'POTENTIAL_DUPLICATE': timedelta(days=7)
+        }
+
+    def detect_duplication(self, incident_a: Dict, incident_b: Dict) -> Optional[DuplicationResult]:
+        """
+        Detect if two incidents are duplicates
+        """
+        # Calculate various similarity metrics
+        text_similarity = self._calculate_text_similarity(incident_a, incident_b)
+        temporal_proximity = self._calculate_temporal_proximity(incident_a, incident_b)
+        service_similarity = self._calculate_service_similarity(incident_a, incident_b)
+        metadata_similarity = self._calculate_metadata_similarity(incident_a, incident_b)
+        
+        # Calculate overall similarity score
+        overall_similarity = (
+            text_similarity * 0.5 +
+            temporal_proximity * 0.2 +
+            service_similarity * 0.2 +
+            metadata_similarity * 0.1
+        )
+        
+        # Determine duplication type
+        duplication_type = self._determine_duplication_type(overall_similarity, text_similarity, temporal_proximity)
+        
+        if duplication_type == 'NO_DUPLICATE':
+            return None
+        
+        # Calculate confidence score
+        confidence_score = self._calculate_confidence_score(
+            overall_similarity, text_similarity, temporal_proximity, service_similarity
+        )
+        
+        # Determine recommended action
+        recommended_action = self._determine_recommended_action(confidence_score, duplication_type)
+        
+        # Calculate merge confidence
+        merge_confidence = self._calculate_merge_confidence(
+            confidence_score, duplication_type, incident_a, incident_b
+        )
+        
+        # Generate reasoning
+        reasoning = self._generate_reasoning(
+            duplication_type, text_similarity, temporal_proximity, service_similarity
+        )
+        
+        # Extract shared elements
+        shared_elements = self._extract_shared_elements(incident_a, incident_b)
+        
+        return DuplicationResult(
+            duplication_type=duplication_type,
+            similarity_score=overall_similarity,
+            confidence_score=confidence_score,
+            text_similarity=text_similarity,
+            temporal_proximity=temporal_proximity,
+            service_similarity=service_similarity,
+            recommended_action=recommended_action,
+            merge_confidence=merge_confidence,
+            reasoning=reasoning,
+            shared_elements=shared_elements
+        )
+
+    def _calculate_text_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate text similarity between incidents"""
+        # Combine all text fields
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')} {incident_a.get('free_text', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')} {incident_b.get('free_text', '')}".lower()
+        
+        # Calculate multiple similarity metrics
+        jaccard_similarity = self._calculate_jaccard_similarity(text_a, text_b)
+        cosine_similarity = self._calculate_cosine_similarity(text_a, text_b)
+        phrase_similarity = self._calculate_phrase_similarity(text_a, text_b)
+        semantic_similarity = self._calculate_semantic_similarity(text_a, text_b)
+        
+        # Weighted combination
+        return (
+            jaccard_similarity * 0.3 +
+            cosine_similarity * 0.3 +
+            phrase_similarity * 0.2 +
+            semantic_similarity * 0.2
+        )
+
+    def _calculate_jaccard_similarity(self, text_a: str, text_b: str) -> float:
+        """Calculate Jaccard similarity based on word sets"""
+        words_a = set(text_a.split())
+        words_b = set(text_b.split())
+        
+        if not words_a or not words_b:
+            return 0.0
+        
+        intersection = len(words_a.intersection(words_b))
+        union = len(words_a.union(words_b))
+        
+        return intersection / union if union > 0 else 0.0
+
+    def _calculate_cosine_similarity(self, text_a: str, text_b: str) -> float:
+        """Calculate cosine similarity based on word frequency"""
+        from collections import Counter
+        
+        words_a = Counter(text_a.split())
+        words_b = Counter(text_b.split())
+        
+        # Get all unique words
+        all_words = set(words_a.keys()) | set(words_b.keys())
+        
+        if not all_words:
+            return 0.0
+        
+        # Create vectors
+        vector_a = [words_a.get(word, 0) for word in all_words]
+        vector_b = [words_b.get(word, 0) for word in all_words]
+        
+        # Calculate cosine similarity
+        dot_product = sum(a * b for a, b in zip(vector_a, vector_b))
+        magnitude_a = sum(a * a for a in vector_a) ** 0.5
+        magnitude_b = sum(b * b for b in vector_b) ** 0.5
+        
+        if magnitude_a == 0 or magnitude_b == 0:
+            return 0.0
+        
+        return dot_product / (magnitude_a * magnitude_b)
+
+    def _calculate_phrase_similarity(self, text_a: str, text_b: str) -> float:
+        """Calculate similarity based on common phrases"""
+        # Extract 2-3 word phrases
+        phrases_a = set()
+        phrases_b = set()
+        
+        words_a = text_a.split()
+        words_b = text_b.split()
+        
+        # Extract 2-word phrases
+        for i in range(len(words_a) - 1):
+            phrases_a.add(f"{words_a[i]} {words_a[i+1]}")
+        
+        for i in range(len(words_b) - 1):
+            phrases_b.add(f"{words_b[i]} {words_b[i+1]}")
+        
+        # Extract 3-word phrases
+        for i in range(len(words_a) - 2):
+            phrases_a.add(f"{words_a[i]} {words_a[i+1]} {words_a[i+2]}")
+        
+        for i in range(len(words_b) - 2):
+            phrases_b.add(f"{words_b[i]} {words_b[i+1]} {words_b[i+2]}")
+        
+        if not phrases_a or not phrases_b:
+            return 0.0
+        
+        intersection = len(phrases_a.intersection(phrases_b))
+        union = len(phrases_a.union(phrases_b))
+        
+        return intersection / union if union > 0 else 0.0
+
+    def _calculate_semantic_similarity(self, text_a: str, text_b: str) -> float:
+        """Calculate semantic similarity using keyword analysis"""
+        # Extract keywords using the classifier
+        keywords_a = set(self.classifier._extract_keywords(text_a))
+        keywords_b = set(self.classifier._extract_keywords(text_b))
+        
+        if not keywords_a or not keywords_b:
+            return 0.0
+        
+        # Calculate semantic similarity based on keyword overlap
+        intersection = len(keywords_a.intersection(keywords_b))
+        union = len(keywords_a.union(keywords_b))
+        
+        base_similarity = intersection / union if union > 0 else 0.0
+        
+        # Boost similarity for technical terms
+        technical_terms = {
+            'error', 'exception', 'timeout', 'connection', 'database', 'server',
+            'api', 'service', 'application', 'network', 'storage', 'memory',
+            'cpu', 'disk', 'bandwidth', 'latency', 'performance', 'crash'
+        }
+        
+        technical_intersection = len(keywords_a.intersection(keywords_b).intersection(technical_terms))
+        if technical_intersection > 0:
+            base_similarity += 0.1 * technical_intersection
+        
+        return min(base_similarity, 1.0)
+
+    def _calculate_temporal_proximity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate temporal proximity between incidents"""
+        created_a = incident_a.get('created_at')
+        created_b = incident_b.get('created_at')
+        
+        if not created_a or not created_b:
+            return 0.0
+        
+        # Convert to datetime if needed
+        if isinstance(created_a, str):
+            created_a = datetime.fromisoformat(created_a.replace('Z', '+00:00'))
+        if isinstance(created_b, str):
+            created_b = datetime.fromisoformat(created_b.replace('Z', '+00:00'))
+        
+        time_diff = abs((created_a - created_b).total_seconds())
+        
+        # Calculate proximity score based on time difference
+        if time_diff <= 300:  # 5 minutes
+            return 1.0
+        elif time_diff <= 1800:  # 30 minutes
+            return 0.9
+        elif time_diff <= 3600:  # 1 hour
+            return 0.7
+        elif time_diff <= 7200:  # 2 hours
+            return 0.5
+        elif time_diff <= 86400:  # 24 hours
+            return 0.3
+        elif time_diff <= 604800:  # 7 days
+            return 0.1
+        else:
+            return 0.0
+
+    def _calculate_service_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate service/component similarity"""
+        # Extract service information from text
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')}".lower()
+        
+        # Service/component keywords
+        service_keywords = [
+            'api', 'service', 'database', 'server', 'application', 'website', 'mobile',
+            'frontend', 'backend', 'microservice', 'gateway', 'load balancer', 'cache',
+            'queue', 'message', 'notification', 'email', 'sms', 'payment', 'auth',
+            'user service', 'order service', 'payment service', 'notification service'
+        ]
+        
+        services_a = set()
+        services_b = set()
+        
+        for keyword in service_keywords:
+            if keyword in text_a:
+                services_a.add(keyword)
+            if keyword in text_b:
+                services_b.add(keyword)
+        
+        if not services_a or not services_b:
+            return 0.0
+        
+        intersection = len(services_a.intersection(services_b))
+        union = len(services_a.union(services_b))
+        
+        return intersection / union if union > 0 else 0.0
+
+    def _calculate_metadata_similarity(self, incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate similarity based on metadata fields"""
+        similarity_score = 0.0
+        total_fields = 0
+        
+        # Compare severity
+        if incident_a.get('severity') == incident_b.get('severity'):
+            similarity_score += 1.0
+        total_fields += 1
+        
+        # Compare status
+        if incident_a.get('status') == incident_b.get('status'):
+            similarity_score += 1.0
+        total_fields += 1
+        
+        # Compare category
+        if incident_a.get('category') == incident_b.get('category'):
+            similarity_score += 1.0
+        total_fields += 1
+        
+        # Compare assigned user
+        if incident_a.get('assigned_to') == incident_b.get('assigned_to'):
+            similarity_score += 1.0
+        total_fields += 1
+        
+        # Compare reporter
+        if incident_a.get('reporter') == incident_b.get('reporter'):
+            similarity_score += 1.0
+        total_fields += 1
+        
+        return similarity_score / total_fields if total_fields > 0 else 0.0
+
+    def _determine_duplication_type(self, overall_similarity: float, text_similarity: float, 
+                                  temporal_proximity: float) -> str:
+        """Determine the type of duplication"""
+        if overall_similarity >= self.duplication_thresholds['EXACT']:
+            return 'EXACT'
+        elif overall_similarity >= self.duplication_thresholds['NEAR_DUPLICATE']:
+            return 'NEAR_DUPLICATE'
+        elif overall_similarity >= self.duplication_thresholds['SIMILAR']:
+            return 'SIMILAR'
+        elif overall_similarity >= self.duplication_thresholds['POTENTIAL_DUPLICATE']:
+            return 'POTENTIAL_DUPLICATE'
+        else:
+            return 'NO_DUPLICATE'
+
+    def _calculate_confidence_score(self, overall_similarity: float, text_similarity: float,
+                                  temporal_proximity: float, service_similarity: float) -> float:
+        """Calculate confidence score for duplication detection"""
+        base_confidence = overall_similarity
+        
+        # Boost confidence for high text similarity
+        if text_similarity > 0.8:
+            base_confidence += 0.1
+        
+        # Boost confidence for high temporal proximity
+        if temporal_proximity > 0.8:
+            base_confidence += 0.1
+        
+        # Boost confidence for high service similarity
+        if service_similarity > 0.8:
+            base_confidence += 0.05
+        
+        return min(base_confidence, 1.0)
+
+    def _determine_recommended_action(self, confidence_score: float, duplication_type: str) -> str:
+        """Determine recommended action based on confidence and duplication type"""
+        if confidence_score >= self.action_thresholds['MERGE']:
+            return 'MERGE'
+        elif confidence_score >= self.action_thresholds['LINK']:
+            return 'LINK'
+        elif confidence_score >= self.action_thresholds['REVIEW']:
+            return 'REVIEW'
+        else:
+            return 'NO_ACTION'
+
+    def _calculate_merge_confidence(self, confidence_score: float, duplication_type: str,
+                                  incident_a: Dict, incident_b: Dict) -> float:
+        """Calculate confidence for merging incidents"""
+        merge_confidence = confidence_score
+        
+        # Adjust based on duplication type
+        type_adjustments = {
+            'EXACT': 0.1,
+            'NEAR_DUPLICATE': 0.05,
+            'SIMILAR': 0.0,
+            'POTENTIAL_DUPLICATE': -0.1
+        }
+        
+        merge_confidence += type_adjustments.get(duplication_type, 0.0)
+        
+        # Adjust based on incident status
+        if incident_a.get('status') == incident_b.get('status'):
+            merge_confidence += 0.05
+        
+        # Adjust based on severity
+        if incident_a.get('severity') == incident_b.get('severity'):
+            merge_confidence += 0.03
+        
+        return min(max(merge_confidence, 0.0), 1.0)
+
+    def _generate_reasoning(self, duplication_type: str, text_similarity: float,
+                          temporal_proximity: float, service_similarity: float) -> str:
+        """Generate human-readable reasoning for duplication detection"""
+        reasoning_parts = []
+        
+        if text_similarity > 0.8:
+            reasoning_parts.append(f"Very high text similarity ({text_similarity:.1%})")
+        elif text_similarity > 0.6:
+            reasoning_parts.append(f"High text similarity ({text_similarity:.1%})")
+        elif text_similarity > 0.4:
+            reasoning_parts.append(f"Moderate text similarity ({text_similarity:.1%})")
+        
+        if temporal_proximity > 0.8:
+            reasoning_parts.append(f"Very close temporal proximity ({temporal_proximity:.1%})")
+        elif temporal_proximity > 0.6:
+            reasoning_parts.append(f"Close temporal proximity ({temporal_proximity:.1%})")
+        
+        if service_similarity > 0.8:
+            reasoning_parts.append(f"Very high service similarity ({service_similarity:.1%})")
+        elif service_similarity > 0.6:
+            reasoning_parts.append(f"High service similarity ({service_similarity:.1%})")
+        
+        if duplication_type == 'EXACT':
+            reasoning_parts.append("Incidents appear to be exact duplicates")
+        elif duplication_type == 'NEAR_DUPLICATE':
+            reasoning_parts.append("Incidents appear to be near duplicates")
+        elif duplication_type == 'SIMILAR':
+            reasoning_parts.append("Incidents appear to be similar")
+        elif duplication_type == 'POTENTIAL_DUPLICATE':
+            reasoning_parts.append("Incidents may be duplicates")
+        
+        return "; ".join(reasoning_parts) if reasoning_parts else "Based on overall similarity analysis"
+
+    def _extract_shared_elements(self, incident_a: Dict, incident_b: Dict) -> List[str]:
+        """Extract elements shared between incidents"""
+        shared_elements = []
+        
+        # Shared keywords
+        text_a = f"{incident_a.get('title', '')} {incident_a.get('description', '')}".lower()
+        text_b = f"{incident_b.get('title', '')} {incident_b.get('description', '')}".lower()
+        
+        keywords_a = set(self.classifier._extract_keywords(text_a))
+        keywords_b = set(self.classifier._extract_keywords(text_b))
+        shared_keywords = keywords_a.intersection(keywords_b)
+        
+        if shared_keywords:
+            shared_elements.append(f"Keywords: {', '.join(list(shared_keywords)[:5])}")
+        
+        # Shared services
+        service_keywords = [
+            'api', 'service', 'database', 'server', 'application', 'website', 'mobile'
+        ]
+        
+        services_a = set()
+        services_b = set()
+        
+        for keyword in service_keywords:
+            if keyword in text_a:
+                services_a.add(keyword)
+            if keyword in text_b:
+                services_b.add(keyword)
+        
+        shared_services = services_a.intersection(services_b)
+        if shared_services:
+            shared_elements.append(f"Services: {', '.join(shared_services)}")
+        
+        # Shared metadata
+        if incident_a.get('severity') == incident_b.get('severity'):
+            shared_elements.append(f"Severity: {incident_a.get('severity')}")
+        
+        if incident_a.get('category') == incident_b.get('category'):
+            shared_elements.append(f"Category: {incident_a.get('category')}")
+        
+        if incident_a.get('status') == incident_b.get('status'):
+            shared_elements.append(f"Status: {incident_a.get('status')}")
+        
+        return shared_elements
+
+    def find_duplicate_candidates(self, target_incident: Dict, all_incidents: List[Dict],
+                                limit: int = 10) -> List[Tuple[Dict, DuplicationResult]]:
+        """Find incidents that might be duplicates of the target incident"""
+        candidates = []
+        
+        for incident in all_incidents:
+            if incident['id'] == target_incident['id']:
+                continue
+            
+            duplication = self.detect_duplication(target_incident, incident)
+            if duplication:
+                candidates.append((incident, duplication))
+        
+        # Sort by confidence score and return top results
+        candidates.sort(key=lambda x: x[1].confidence_score, reverse=True)
+        return candidates[:limit]
+
+    def batch_detect_duplicates(self, incidents: List[Dict]) -> List[Tuple[Dict, Dict, DuplicationResult]]:
+        """Batch detect duplicates in a list of incidents"""
+        duplicates = []
+        processed_pairs = set()
+        
+        for i, incident_a in enumerate(incidents):
+            for j, incident_b in enumerate(incidents[i+1:], i+1):
+                # Create a unique pair identifier
+                pair_id = tuple(sorted([incident_a['id'], incident_b['id']]))
+                
+                if pair_id in processed_pairs:
+                    continue
+                
+                processed_pairs.add(pair_id)
+                
+                duplication = self.detect_duplication(incident_a, incident_b)
+                if duplication:
+                    duplicates.append((incident_a, incident_b, duplication))
+        
+        # Sort by confidence score
+        duplicates.sort(key=lambda x: x[2].confidence_score, reverse=True)
+        return duplicates

ETB-API/incident_intelligence/apps.py

@@ -0,0 +1,11 @@
+from django.apps import AppConfig
+
+
+class IncidentIntelligenceConfig(AppConfig):
+    default_auto_field = 'django.db.models.BigAutoField'
+    name = 'incident_intelligence'
+    verbose_name = 'Incident Intelligence'
+    
+    def ready(self):
+        """Initialize the app when Django starts"""
+        import incident_intelligence.signals

ETB-API/incident_intelligence/management/__init__.py

@@ -0,0 +1 @@
+# Management commands for incident intelligence

ETB-API/incident_intelligence/management/commands/__init__.py

@@ -0,0 +1 @@
+# Management commands

ETB-API/incident_intelligence/management/commands/setup_incident_intelligence.py

@@ -0,0 +1,213 @@
+"""
+Management command to set up incident intelligence module
+"""
+from django.core.management.base import BaseCommand
+from django.db import transaction
+from django.contrib.auth import get_user_model
+from incident_intelligence.models import Incident, IncidentPattern
+
+User = get_user_model()
+
+
+class Command(BaseCommand):
+    help = 'Set up the incident intelligence module with sample data and configurations'
+    
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--create-sample-data',
+            action='store_true',
+            help='Create sample incidents for testing',
+        )
+        parser.add_argument(
+            '--create-patterns',
+            action='store_true',
+            help='Create sample patterns',
+        )
+        parser.add_argument(
+            '--run-ai-analysis',
+            action='store_true',
+            help='Run AI analysis on existing incidents',
+        )
+    
+    def handle(self, *args, **options):
+        self.stdout.write(
+            self.style.SUCCESS('Setting up Incident Intelligence module...')
+        )
+        
+        if options['create_sample_data']:
+            self.create_sample_data()
+        
+        if options['create_patterns']:
+            self.create_sample_patterns()
+        
+        if options['run_ai_analysis']:
+            self.run_ai_analysis()
+        
+        self.stdout.write(
+            self.style.SUCCESS('Incident Intelligence module setup completed!')
+        )
+    
+    def create_sample_data(self):
+        """Create sample incidents for testing"""
+        self.stdout.write('Creating sample incidents...')
+        
+        sample_incidents = [
+            {
+                'title': 'Database Connection Timeout',
+                'description': 'Users are experiencing timeouts when trying to access the database. The issue started around 2 PM and affects all users.',
+                'free_text': 'Database is down, can\'t connect, getting timeout errors',
+                'severity': 'HIGH',
+                'affected_users': 150,
+                'business_impact': 'Critical business operations are affected. Users cannot access their data.',
+            },
+            {
+                'title': 'API Response Slow',
+                'description': 'The user service API is responding slowly, causing delays in user authentication and profile updates.',
+                'free_text': 'API is slow, taking forever to respond, users complaining',
+                'severity': 'MEDIUM',
+                'affected_users': 50,
+                'business_impact': 'User experience is degraded but core functionality still works.',
+            },
+            {
+                'title': 'Payment Gateway Error',
+                'description': 'Payment processing is failing with 500 errors. Customers cannot complete purchases.',
+                'free_text': 'Payment not working, getting errors, customers can\'t buy',
+                'severity': 'CRITICAL',
+                'affected_users': 200,
+                'business_impact': 'Revenue is directly impacted. Customers cannot make purchases.',
+            },
+            {
+                'title': 'Email Service Down',
+                'description': 'Email notifications are not being sent. Users are not receiving order confirmations and password reset emails.',
+                'free_text': 'Emails not sending, notifications broken, users not getting emails',
+                'severity': 'MEDIUM',
+                'affected_users': 75,
+                'business_impact': 'Communication with customers is disrupted.',
+            },
+            {
+                'title': 'Mobile App Crash',
+                'description': 'The mobile application is crashing on iOS devices when users try to view their order history.',
+                'free_text': 'App crashing on iPhone, can\'t see orders, keeps closing',
+                'severity': 'HIGH',
+                'affected_users': 100,
+                'business_impact': 'Mobile users cannot access their order information.',
+            },
+            {
+                'title': 'Database Connection Timeout',
+                'description': 'Users are experiencing timeouts when trying to access the database. The issue started around 3 PM and affects all users.',
+                'free_text': 'Database is down, can\'t connect, getting timeout errors',
+                'severity': 'HIGH',
+                'affected_users': 150,
+                'business_impact': 'Critical business operations are affected. Users cannot access their data.',
+            },
+            {
+                'title': 'Load Balancer Issue',
+                'description': 'The load balancer is not distributing traffic evenly, causing some servers to be overloaded.',
+                'free_text': 'Load balancer not working properly, servers overloaded',
+                'severity': 'HIGH',
+                'affected_users': 300,
+                'business_impact': 'System performance is degraded across multiple services.',
+            },
+            {
+                'title': 'Cache Miss Rate High',
+                'description': 'Redis cache is experiencing high miss rates, causing increased database load.',
+                'free_text': 'Cache not working, database overloaded, slow responses',
+                'severity': 'MEDIUM',
+                'affected_users': 0,
+                'business_impact': 'System performance is degraded but not directly visible to users.',
+            }
+        ]
+        
+        with transaction.atomic():
+            for incident_data in sample_incidents:
+                incident, created = Incident.objects.get_or_create(
+                    title=incident_data['title'],
+                    defaults=incident_data
+                )
+                if created:
+                    self.stdout.write(f'  Created incident: {incident.title}')
+                else:
+                    self.stdout.write(f'  Incident already exists: {incident.title}')
+        
+        self.stdout.write(
+            self.style.SUCCESS(f'Created {len(sample_incidents)} sample incidents')
+        )
+    
+    def create_sample_patterns(self):
+        """Create sample patterns"""
+        self.stdout.write('Creating sample patterns...')
+        
+        sample_patterns = [
+            {
+                'name': 'Database Connectivity Issues',
+                'pattern_type': 'RECURRING',
+                'description': 'Recurring database connection problems affecting multiple services',
+                'frequency': 'Weekly',
+                'affected_services': ['user-service', 'order-service', 'payment-service'],
+                'common_keywords': ['database', 'connection', 'timeout', 'error'],
+                'confidence_score': 0.85,
+                'is_active': True,
+                'is_resolved': False
+            },
+            {
+                'name': 'API Performance Degradation',
+                'pattern_type': 'TREND',
+                'description': 'Gradual degradation in API response times across services',
+                'frequency': 'Daily',
+                'affected_services': ['api-gateway', 'user-service', 'order-service'],
+                'common_keywords': ['slow', 'performance', 'latency', 'timeout'],
+                'confidence_score': 0.75,
+                'is_active': True,
+                'is_resolved': False
+            },
+            {
+                'name': 'Mobile App Crashes',
+                'pattern_type': 'RECURRING',
+                'description': 'Frequent crashes in mobile applications, particularly on iOS',
+                'frequency': 'Bi-weekly',
+                'affected_services': ['mobile-app', 'ios-app'],
+                'common_keywords': ['crash', 'mobile', 'ios', 'app'],
+                'confidence_score': 0.90,
+                'is_active': True,
+                'is_resolved': False
+            }
+        ]
+        
+        with transaction.atomic():
+            for pattern_data in sample_patterns:
+                pattern, created = IncidentPattern.objects.get_or_create(
+                    name=pattern_data['name'],
+                    defaults=pattern_data
+                )
+                if created:
+                    self.stdout.write(f'  Created pattern: {pattern.name}')
+                else:
+                    self.stdout.write(f'  Pattern already exists: {pattern.name}')
+        
+        self.stdout.write(
+            self.style.SUCCESS(f'Created {len(sample_patterns)} sample patterns')
+        )
+    
+    def run_ai_analysis(self):
+        """Run AI analysis on existing incidents"""
+        self.stdout.write('Running AI analysis on existing incidents...')
+        
+        try:
+            from incident_intelligence.tasks import batch_process_incidents_ai
+            
+            # Get incidents that haven't been processed
+            unprocessed_incidents = Incident.objects.filter(ai_processed=False)
+            incident_ids = [str(incident.id) for incident in unprocessed_incidents]
+            
+            if incident_ids:
+                batch_process_incidents_ai.delay(incident_ids)
+                self.stdout.write(
+                    self.style.SUCCESS(f'Queued {len(incident_ids)} incidents for AI analysis')
+                )
+            else:
+                self.stdout.write('No unprocessed incidents found')
+        
+        except Exception as e:
+            self.stdout.write(
+                self.style.ERROR(f'Failed to run AI analysis: {e}')
+            )

ETB-API/incident_intelligence/management/commands/setup_security_integration.py

@@ -0,0 +1,190 @@
+"""
+Management command to set up security integration for incident intelligence
+"""
+from django.core.management.base import BaseCommand
+from django.db import transaction
+from security.models import DataClassification, Role, Permission
+from django.contrib.contenttypes.models import ContentType
+from incident_intelligence.models import Incident
+
+
+class Command(BaseCommand):
+    help = 'Set up security integration for incident intelligence module'
+    
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--create-permissions',
+            action='store_true',
+            help='Create incident intelligence permissions',
+        )
+        parser.add_argument(
+            '--create-roles',
+            action='store_true',
+            help='Create incident intelligence roles',
+        )
+        parser.add_argument(
+            '--assign-classifications',
+            action='store_true',
+            help='Assign data classifications to existing incidents',
+        )
+    
+    def handle(self, *args, **options):
+        self.stdout.write(
+            self.style.SUCCESS('Setting up security integration for Incident Intelligence...')
+        )
+        
+        if options['create_permissions']:
+            self.create_permissions()
+        
+        if options['create_roles']:
+            self.create_roles()
+        
+        if options['assign_classifications']:
+            self.assign_classifications()
+        
+        self.stdout.write(
+            self.style.SUCCESS('Security integration setup completed!')
+        )
+    
+    def create_permissions(self):
+        """Create incident intelligence permissions"""
+        self.stdout.write('Creating incident intelligence permissions...')
+        
+        # Get content type for Incident model
+        incident_content_type = ContentType.objects.get_for_model(Incident)
+        
+        permissions_data = [
+            ('view_incident', 'Can view incident'),
+            ('add_incident', 'Can add incident'),
+            ('change_incident', 'Can change incident'),
+            ('delete_incident', 'Can delete incident'),
+            ('analyze_incident', 'Can analyze incident'),
+            ('view_incidentcorrelation', 'Can view incident correlation'),
+            ('view_duplicationdetection', 'Can view duplication detection'),
+            ('view_incidentpattern', 'Can view incident pattern'),
+            ('approve_merge', 'Can approve incident merge'),
+            ('reject_merge', 'Can reject incident merge'),
+            ('resolve_pattern', 'Can resolve incident pattern'),
+        ]
+        
+        with transaction.atomic():
+            for codename, name in permissions_data:
+                permission, created = Permission.objects.get_or_create(
+                    codename=codename,
+                    content_type=incident_content_type,
+                    defaults={'name': name}
+                )
+                if created:
+                    self.stdout.write(f'  Created permission: {permission.name}')
+                else:
+                    self.stdout.write(f'  Permission already exists: {permission.name}')
+        
+        self.stdout.write(
+            self.style.SUCCESS('Created incident intelligence permissions')
+        )
+    
+    def create_roles(self):
+        """Create incident intelligence roles"""
+        self.stdout.write('Creating incident intelligence roles...')
+        
+        # Get permissions
+        incident_content_type = ContentType.objects.get_for_model(Incident)
+        permissions = Permission.objects.filter(content_type=incident_content_type)
+        
+        # Get data classifications
+        public_classification = DataClassification.objects.filter(name='PUBLIC').first()
+        internal_classification = DataClassification.objects.filter(name='INTERNAL').first()
+        confidential_classification = DataClassification.objects.filter(name='CONFIDENTIAL').first()
+        
+        roles_data = [
+            {
+                'name': 'Incident Viewer',
+                'description': 'Can view incidents and basic analysis results',
+                'permissions': ['view_incident', 'view_incidentcorrelation', 'view_duplicationdetection', 'view_incidentpattern'],
+                'classifications': [public_classification, internal_classification] if public_classification and internal_classification else []
+            },
+            {
+                'name': 'Incident Analyst',
+                'description': 'Can view and analyze incidents, trigger AI analysis',
+                'permissions': ['view_incident', 'add_incident', 'change_incident', 'analyze_incident', 'view_incidentcorrelation', 'view_duplicationdetection', 'view_incidentpattern'],
+                'classifications': [public_classification, internal_classification, confidential_classification] if all([public_classification, internal_classification, confidential_classification]) else []
+            },
+            {
+                'name': 'Incident Manager',
+                'description': 'Can manage incidents, approve merges, resolve patterns',
+                'permissions': ['view_incident', 'add_incident', 'change_incident', 'delete_incident', 'analyze_incident', 'view_incidentcorrelation', 'view_duplicationdetection', 'view_incidentpattern', 'approve_merge', 'reject_merge', 'resolve_pattern'],
+                'classifications': [public_classification, internal_classification, confidential_classification] if all([public_classification, internal_classification, confidential_classification]) else []
+            },
+            {
+                'name': 'Incident Administrator',
+                'description': 'Full access to all incident intelligence features',
+                'permissions': [p.codename for p in permissions],
+                'classifications': [public_classification, internal_classification, confidential_classification] if all([public_classification, internal_classification, confidential_classification]) else []
+            }
+        ]
+        
+        with transaction.atomic():
+            for role_data in roles_data:
+                role, created = Role.objects.get_or_create(
+                    name=role_data['name'],
+                    defaults={
+                        'description': role_data['description'],
+                        'is_active': True
+                    }
+                )
+                
+                if created:
+                    # Assign permissions
+                    role_permissions = Permission.objects.filter(
+                        codename__in=role_data['permissions'],
+                        content_type=incident_content_type
+                    )
+                    role.permissions.set(role_permissions)
+                    
+                    # Assign data classifications
+                    if role_data['classifications']:
+                        role.data_classification_access.set(role_data['classifications'])
+                    
+                    self.stdout.write(f'  Created role: {role.name}')
+                else:
+                    self.stdout.write(f'  Role already exists: {role.name}')
+        
+        self.stdout.write(
+            self.style.SUCCESS('Created incident intelligence roles')
+        )
+    
+    def assign_classifications(self):
+        """Assign data classifications to existing incidents"""
+        self.stdout.write('Assigning data classifications to existing incidents...')
+        
+        # Get data classifications
+        public_classification = DataClassification.objects.filter(name='PUBLIC').first()
+        internal_classification = DataClassification.objects.filter(name='INTERNAL').first()
+        confidential_classification = DataClassification.objects.filter(name='CONFIDENTIAL').first()
+        
+        if not public_classification:
+            self.stdout.write(
+                self.style.WARNING('No data classifications found. Please create them first.')
+            )
+            return
+        
+        with transaction.atomic():
+            # Assign classifications based on incident severity
+            incidents = Incident.objects.filter(data_classification__isnull=True)
+            
+            for incident in incidents:
+                if incident.severity in ['CRITICAL', 'EMERGENCY']:
+                    incident.data_classification = confidential_classification or internal_classification or public_classification
+                    incident.security_clearance_required = True
+                    incident.is_sensitive = True
+                elif incident.severity == 'HIGH':
+                    incident.data_classification = internal_classification or public_classification
+                    incident.is_sensitive = True
+                else:
+                    incident.data_classification = public_classification
+                
+                incident.save()
+            
+            self.stdout.write(
+                self.style.SUCCESS(f'Assigned classifications to {incidents.count()} incidents')
+            )

ETB-API/incident_intelligence/migrations/0001_initial.py

@@ -0,0 +1,230 @@
+# Generated by Django 5.2.6 on 2025-09-18 15:08
+
+import django.core.validators
+import django.db.models.deletion
+import uuid
+from django.conf import settings
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Incident',
+            fields=[
+                ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
+                ('title', models.CharField(max_length=200)),
+                ('description', models.TextField()),
+                ('free_text', models.TextField(help_text='Original free text description from user')),
+                ('category', models.CharField(blank=True, max_length=100, null=True)),
+                ('subcategory', models.CharField(blank=True, max_length=100, null=True)),
+                ('classification_confidence', models.FloatField(blank=True, help_text='AI confidence score for classification (0.0-1.0)', null=True, validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('severity', models.CharField(choices=[('LOW', 'Low'), ('MEDIUM', 'Medium'), ('HIGH', 'High'), ('CRITICAL', 'Critical'), ('EMERGENCY', 'Emergency')], default='MEDIUM', max_length=20)),
+                ('suggested_severity', models.CharField(blank=True, choices=[('LOW', 'Low'), ('MEDIUM', 'Medium'), ('HIGH', 'High'), ('CRITICAL', 'Critical'), ('EMERGENCY', 'Emergency')], max_length=20, null=True)),
+                ('severity_confidence', models.FloatField(blank=True, help_text='AI confidence score for severity suggestion (0.0-1.0)', null=True, validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('priority', models.CharField(choices=[('P1', 'P1 - Critical'), ('P2', 'P2 - High'), ('P3', 'P3 - Medium'), ('P4', 'P4 - Low')], default='P3', max_length=10)),
+                ('status', models.CharField(choices=[('OPEN', 'Open'), ('IN_PROGRESS', 'In Progress'), ('RESOLVED', 'Resolved'), ('CLOSED', 'Closed'), ('CANCELLED', 'Cancelled')], default='OPEN', max_length=20)),
+                ('created_at', models.DateTimeField(auto_now_add=True)),
+                ('updated_at', models.DateTimeField(auto_now=True)),
+                ('resolved_at', models.DateTimeField(blank=True, null=True)),
+                ('affected_users', models.PositiveIntegerField(default=0)),
+                ('business_impact', models.TextField(blank=True, null=True)),
+                ('estimated_downtime', models.DurationField(blank=True, null=True)),
+                ('ai_processed', models.BooleanField(default=False)),
+                ('ai_processing_error', models.TextField(blank=True, null=True)),
+                ('last_ai_analysis', models.DateTimeField(blank=True, null=True)),
+                ('is_duplicate', models.BooleanField(default=False)),
+                ('duplicate_confidence', models.FloatField(blank=True, help_text='AI confidence score for duplication detection (0.0-1.0)', null=True, validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('assigned_to', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to=settings.AUTH_USER_MODEL)),
+                ('original_incident', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='duplicates', to='incident_intelligence.incident')),
+                ('reporter', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='reported_incidents', to=settings.AUTH_USER_MODEL)),
+            ],
+            options={
+                'ordering': ['-created_at'],
+            },
+        ),
+        migrations.CreateModel(
+            name='DuplicationDetection',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('duplication_type', models.CharField(choices=[('EXACT', 'Exact Duplicate'), ('NEAR_DUPLICATE', 'Near Duplicate'), ('SIMILAR', 'Similar Incident'), ('POTENTIAL_DUPLICATE', 'Potential Duplicate')], max_length=20)),
+                ('similarity_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('confidence_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('text_similarity', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('temporal_proximity', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('service_similarity', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('recommended_action', models.CharField(choices=[('MERGE', 'Merge Incidents'), ('LINK', 'Link Incidents'), ('REVIEW', 'Manual Review'), ('NO_ACTION', 'No Action')], max_length=20)),
+                ('merge_confidence', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('reasoning', models.TextField(help_text='AI explanation for duplication detection')),
+                ('shared_elements', models.JSONField(default=list, help_text='Elements shared between incidents')),
+                ('status', models.CharField(choices=[('DETECTED', 'Detected'), ('REVIEWED', 'Reviewed'), ('MERGED', 'Merged'), ('REJECTED', 'Rejected')], default='DETECTED', max_length=20)),
+                ('created_at', models.DateTimeField(auto_now_add=True)),
+                ('reviewed_at', models.DateTimeField(blank=True, null=True)),
+                ('model_version', models.CharField(default='v1.0', max_length=50)),
+                ('reviewed_by', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to=settings.AUTH_USER_MODEL)),
+                ('incident_a', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='duplication_as_a', to='incident_intelligence.incident')),
+                ('incident_b', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='duplication_as_b', to='incident_intelligence.incident')),
+            ],
+        ),
+        migrations.CreateModel(
+            name='AIProcessingLog',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('processing_type', models.CharField(choices=[('CLASSIFICATION', 'Classification'), ('SEVERITY_ANALYSIS', 'Severity Analysis'), ('CORRELATION', 'Correlation Analysis'), ('DUPLICATION_DETECTION', 'Duplication Detection'), ('PATTERN_DETECTION', 'Pattern Detection')], max_length=30)),
+                ('status', models.CharField(choices=[('PENDING', 'Pending'), ('PROCESSING', 'Processing'), ('COMPLETED', 'Completed'), ('FAILED', 'Failed'), ('SKIPPED', 'Skipped')], default='PENDING', max_length=20)),
+                ('related_incidents', models.JSONField(default=list, help_text='List of related incident IDs')),
+                ('input_data', models.JSONField(help_text='Input data for processing')),
+                ('output_data', models.JSONField(blank=True, help_text='Output data from processing', null=True)),
+                ('error_message', models.TextField(blank=True, null=True)),
+                ('processing_time', models.FloatField(blank=True, help_text='Processing time in seconds', null=True)),
+                ('model_version', models.CharField(default='v1.0', max_length=50)),
+                ('confidence_score', models.FloatField(blank=True, null=True, validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('started_at', models.DateTimeField(auto_now_add=True)),
+                ('completed_at', models.DateTimeField(blank=True, null=True)),
+                ('incident', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='incident_intelligence.incident')),
+            ],
+            options={
+                'ordering': ['-started_at'],
+            },
+        ),
+        migrations.CreateModel(
+            name='IncidentClassification',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('predicted_category', models.CharField(max_length=100)),
+                ('predicted_subcategory', models.CharField(max_length=100)),
+                ('confidence_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('alternative_categories', models.JSONField(default=list, help_text='List of alternative category predictions')),
+                ('extracted_keywords', models.JSONField(default=list, help_text='Keywords extracted from incident text')),
+                ('sentiment_score', models.FloatField(blank=True, help_text='Sentiment analysis score (-1 to 1)', null=True)),
+                ('urgency_indicators', models.JSONField(default=list, help_text='Detected urgency indicators')),
+                ('model_version', models.CharField(default='v1.0', max_length=50)),
+                ('processing_time', models.FloatField(help_text='Time taken for classification in seconds')),
+                ('created_at', models.DateTimeField(auto_now_add=True)),
+                ('incident', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='ai_classification', to='incident_intelligence.incident')),
+            ],
+        ),
+        migrations.CreateModel(
+            name='IncidentCorrelation',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('correlation_type', models.CharField(choices=[('SAME_SERVICE', 'Same Service'), ('SAME_COMPONENT', 'Same Component'), ('TEMPORAL', 'Temporal Correlation'), ('PATTERN', 'Pattern Match'), ('DEPENDENCY', 'Dependency Related'), ('CASCADE', 'Cascade Effect')], max_length=20)),
+                ('confidence_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('correlation_strength', models.CharField(choices=[('WEAK', 'Weak'), ('MODERATE', 'Moderate'), ('STRONG', 'Strong'), ('VERY_STRONG', 'Very Strong')], max_length=20)),
+                ('shared_keywords', models.JSONField(default=list, help_text='Keywords shared between incidents')),
+                ('time_difference', models.DurationField(help_text='Time difference between incidents')),
+                ('similarity_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('is_problem_indicator', models.BooleanField(default=False, help_text='Indicates if this correlation suggests a larger problem')),
+                ('problem_description', models.TextField(blank=True, null=True)),
+                ('created_at', models.DateTimeField(auto_now_add=True)),
+                ('model_version', models.CharField(default='v1.0', max_length=50)),
+                ('primary_incident', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='correlations_as_primary', to='incident_intelligence.incident')),
+                ('related_incident', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='correlations_as_related', to='incident_intelligence.incident')),
+            ],
+        ),
+        migrations.CreateModel(
+            name='IncidentPattern',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('name', models.CharField(max_length=200)),
+                ('pattern_type', models.CharField(choices=[('RECURRING', 'Recurring Issue'), ('SEASONAL', 'Seasonal Pattern'), ('TREND', 'Trend Analysis'), ('ANOMALY', 'Anomaly Detection')], max_length=20)),
+                ('description', models.TextField()),
+                ('frequency', models.CharField(help_text='How often this pattern occurs', max_length=50)),
+                ('affected_services', models.JSONField(default=list, help_text='Services affected by this pattern')),
+                ('common_keywords', models.JSONField(default=list, help_text='Common keywords in incidents with this pattern')),
+                ('incident_count', models.PositiveIntegerField(default=0)),
+                ('confidence_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('last_occurrence', models.DateTimeField(blank=True, null=True)),
+                ('next_predicted_occurrence', models.DateTimeField(blank=True, null=True)),
+                ('is_active', models.BooleanField(default=True)),
+                ('is_resolved', models.BooleanField(default=False)),
+                ('created_at', models.DateTimeField(auto_now_add=True)),
+                ('updated_at', models.DateTimeField(auto_now=True)),
+                ('model_version', models.CharField(default='v1.0', max_length=50)),
+                ('incidents', models.ManyToManyField(related_name='patterns', to='incident_intelligence.incident')),
+            ],
+            options={
+                'ordering': ['-confidence_score', '-incident_count'],
+            },
+        ),
+        migrations.CreateModel(
+            name='SeveritySuggestion',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('suggested_severity', models.CharField(choices=[('LOW', 'Low'), ('MEDIUM', 'Medium'), ('HIGH', 'High'), ('CRITICAL', 'Critical'), ('EMERGENCY', 'Emergency')], max_length=20)),
+                ('confidence_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('user_impact_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('business_impact_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('technical_impact_score', models.FloatField(validators=[django.core.validators.MinValueValidator(0.0), django.core.validators.MaxValueValidator(1.0)])),
+                ('reasoning', models.TextField(help_text='AI explanation for severity suggestion')),
+                ('impact_factors', models.JSONField(default=list, help_text='List of factors that influenced the severity')),
+                ('model_version', models.CharField(default='v1.0', max_length=50)),
+                ('processing_time', models.FloatField(help_text='Time taken for severity analysis in seconds')),
+                ('created_at', models.DateTimeField(auto_now_add=True)),
+                ('incident', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='severity_suggestion', to='incident_intelligence.incident')),
+            ],
+        ),
+        migrations.AddIndex(
+            model_name='incident',
+            index=models.Index(fields=['status', 'severity'], name='incident_in_status_69cd1f_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='incident',
+            index=models.Index(fields=['category', 'subcategory'], name='incident_in_categor_b1397a_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='incident',
+            index=models.Index(fields=['created_at'], name='incident_in_created_95a890_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='incident',
+            index=models.Index(fields=['assigned_to'], name='incident_in_assigne_a00121_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='duplicationdetection',
+            index=models.Index(fields=['duplication_type', 'confidence_score'], name='incident_in_duplica_170e8c_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='duplicationdetection',
+            index=models.Index(fields=['status'], name='incident_in_status_d1db68_idx'),
+        ),
+        migrations.AlterUniqueTogether(
+            name='duplicationdetection',
+            unique_together={('incident_a', 'incident_b')},
+        ),
+        migrations.AddIndex(
+            model_name='aiprocessinglog',
+            index=models.Index(fields=['processing_type', 'status'], name='incident_in_process_3b7238_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='aiprocessinglog',
+            index=models.Index(fields=['incident', 'processing_type'], name='incident_in_inciden_ef07e9_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='incidentcorrelation',
+            index=models.Index(fields=['correlation_type', 'confidence_score'], name='incident_in_correla_a6263a_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='incidentcorrelation',
+            index=models.Index(fields=['is_problem_indicator'], name='incident_in_is_prob_746ecd_idx'),
+        ),
+        migrations.AlterUniqueTogether(
+            name='incidentcorrelation',
+            unique_together={('primary_incident', 'related_incident')},
+        ),
+        migrations.AddIndex(
+            model_name='incidentpattern',
+            index=models.Index(fields=['pattern_type', 'is_active'], name='incident_in_pattern_336ec0_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='incidentpattern',
+            index=models.Index(fields=['confidence_score'], name='incident_in_confide_d39236_idx'),
+        ),
+    ]

ETB-API/incident_intelligence/migrations/0002_incident_data_classification_incident_is_sensitive_and_more.py

@@ -0,0 +1,30 @@
+# Generated by Django 5.2.6 on 2025-09-18 15:16
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('incident_intelligence', '0001_initial'),
+        ('security', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='incident',
+            name='data_classification',
+            field=models.ForeignKey(blank=True, help_text='Data classification level for this incident', null=True, on_delete=django.db.models.deletion.SET_NULL, to='security.dataclassification'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='is_sensitive',
+            field=models.BooleanField(default=False, help_text='Whether this incident contains sensitive information'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='security_clearance_required',
+            field=models.BooleanField(default=False, help_text='Whether this incident requires special security clearance'),
+        ),
+    ]

ETB-API/incident_intelligence/migrations/0003_incident_auto_remediation_attempted_and_more.py

@@ -0,0 +1,33 @@
+# Generated by Django 5.2.6 on 2025-09-18 15:29
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('incident_intelligence', '0002_incident_data_classification_incident_is_sensitive_and_more'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='incident',
+            name='auto_remediation_attempted',
+            field=models.BooleanField(default=False, help_text='Whether auto-remediation has been attempted'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='automation_enabled',
+            field=models.BooleanField(default=True, help_text='Whether automation can be triggered for this incident'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='maintenance_window_override',
+            field=models.BooleanField(default=False, help_text='Whether this incident should override maintenance window suppressions'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='runbook_suggested',
+            field=models.BooleanField(default=False, help_text='Whether a runbook has been suggested for this incident'),
+        ),
+    ]

ETB-API/incident_intelligence/migrations/0004_incident_oncall_assignment_incident_sla_override_and_more.py

@@ -0,0 +1,30 @@
+# Generated by Django 5.2.6 on 2025-09-18 15:51
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('incident_intelligence', '0003_incident_auto_remediation_attempted_and_more'),
+        ('sla_oncall', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='incident',
+            name='oncall_assignment',
+            field=models.ForeignKey(blank=True, help_text='On-call assignment responsible for this incident', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='assigned_incidents', to='sla_oncall.oncallassignment'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='sla_override',
+            field=models.BooleanField(default=False, help_text='Whether this incident overrides normal SLA calculations'),
+        ),
+        migrations.AddField(
+            model_name='incident',
+            name='sla_override_reason',
+            field=models.TextField(blank=True, help_text='Reason for SLA override', null=True),
+        ),
+    ]

ETB-API/incident_intelligence/models.py

@@ -0,0 +1,484 @@
+from django.db import models
+from django.contrib.auth import get_user_model
+from django.core.validators import MinValueValidator, MaxValueValidator
+from django.utils import timezone
+import uuid
+
+User = get_user_model()
+
+
+class Incident(models.Model):
+    """Main incident model for tracking and managing incidents"""
+    
+    SEVERITY_CHOICES = [
+        ('LOW', 'Low'),
+        ('MEDIUM', 'Medium'),
+        ('HIGH', 'High'),
+        ('CRITICAL', 'Critical'),
+        ('EMERGENCY', 'Emergency'),
+    ]
+    
+    STATUS_CHOICES = [
+        ('OPEN', 'Open'),
+        ('IN_PROGRESS', 'In Progress'),
+        ('RESOLVED', 'Resolved'),
+        ('CLOSED', 'Closed'),
+        ('CANCELLED', 'Cancelled'),
+    ]
+    
+    PRIORITY_CHOICES = [
+        ('P1', 'P1 - Critical'),
+        ('P2', 'P2 - High'),
+        ('P3', 'P3 - Medium'),
+        ('P4', 'P4 - Low'),
+    ]
+    
+    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
+    title = models.CharField(max_length=200)
+    description = models.TextField()
+    free_text = models.TextField(help_text="Original free text description from user")
+    
+    # Classification fields
+    category = models.CharField(max_length=100, blank=True, null=True)
+    subcategory = models.CharField(max_length=100, blank=True, null=True)
+    classification_confidence = models.FloatField(
+        validators=[MinValueValidator(0.0), MaxValueValidator(1.0)],
+        null=True, blank=True,
+        help_text="AI confidence score for classification (0.0-1.0)"
+    )
+    
+    # Severity and Priority
+    severity = models.CharField(max_length=20, choices=SEVERITY_CHOICES, default='MEDIUM')
+    suggested_severity = models.CharField(max_length=20, choices=SEVERITY_CHOICES, blank=True, null=True)
+    severity_confidence = models.FloatField(
+        validators=[MinValueValidator(0.0), MaxValueValidator(1.0)],
+        null=True, blank=True,
+        help_text="AI confidence score for severity suggestion (0.0-1.0)"
+    )
+    priority = models.CharField(max_length=10, choices=PRIORITY_CHOICES, default='P3')
+    
+    # Status and Assignment
+    status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='OPEN')
+    assigned_to = models.ForeignKey(User, on_delete=models.SET_NULL, null=True, blank=True)
+    reporter = models.ForeignKey(User, on_delete=models.SET_NULL, null=True, blank=True, related_name='reported_incidents')
+    
+    # Timestamps
+    created_at = models.DateTimeField(auto_now_add=True)
+    updated_at = models.DateTimeField(auto_now=True)
+    resolved_at = models.DateTimeField(null=True, blank=True)
+    
+    # Impact and Business Context
+    affected_users = models.PositiveIntegerField(default=0)
+    business_impact = models.TextField(blank=True, null=True)
+    estimated_downtime = models.DurationField(null=True, blank=True)
+    
+    # AI Processing Flags
+    ai_processed = models.BooleanField(default=False)
+    ai_processing_error = models.TextField(blank=True, null=True)
+    last_ai_analysis = models.DateTimeField(null=True, blank=True)
+    
+    # Automation Integration
+    automation_enabled = models.BooleanField(
+        default=True,
+        help_text="Whether automation can be triggered for this incident"
+    )
+    runbook_suggested = models.BooleanField(
+        default=False,
+        help_text="Whether a runbook has been suggested for this incident"
+    )
+    auto_remediation_attempted = models.BooleanField(
+        default=False,
+        help_text="Whether auto-remediation has been attempted"
+    )
+    maintenance_window_override = models.BooleanField(
+        default=False,
+        help_text="Whether this incident should override maintenance window suppressions"
+    )
+    
+    # SLA Integration
+    sla_override = models.BooleanField(
+        default=False,
+        help_text="Whether this incident overrides normal SLA calculations"
+    )
+    sla_override_reason = models.TextField(
+        blank=True,
+        null=True,
+        help_text="Reason for SLA override"
+    )
+    oncall_assignment = models.ForeignKey(
+        'sla_oncall.OnCallAssignment',
+        on_delete=models.SET_NULL,
+        null=True,
+        blank=True,
+        related_name='assigned_incidents',
+        help_text="On-call assignment responsible for this incident"
+    )
+    
+    # Duplication Detection
+    is_duplicate = models.BooleanField(default=False)
+    original_incident = models.ForeignKey('self', on_delete=models.SET_NULL, null=True, blank=True, related_name='duplicates')
+    duplicate_confidence = models.FloatField(
+        validators=[MinValueValidator(0.0), MaxValueValidator(1.0)],
+        null=True, blank=True,
+        help_text="AI confidence score for duplication detection (0.0-1.0)"
+    )
+    
+    # Security Integration
+    data_classification = models.ForeignKey(
+        'security.DataClassification',
+        on_delete=models.SET_NULL,
+        null=True,
+        blank=True,
+        help_text="Data classification level for this incident"
+    )
+    security_clearance_required = models.BooleanField(
+        default=False,
+        help_text="Whether this incident requires special security clearance"
+    )
+    is_sensitive = models.BooleanField(
+        default=False,
+        help_text="Whether this incident contains sensitive information"
+    )
+    
+    class Meta:
+        ordering = ['-created_at']
+        indexes = [
+            models.Index(fields=['status', 'severity']),
+            models.Index(fields=['category', 'subcategory']),
+            models.Index(fields=['created_at']),
+            models.Index(fields=['assigned_to']),
+        ]
+    
+    def __str__(self):
+        return f"{self.title} ({self.severity})"
+    
+    @property
+    def is_resolved(self):
+        return self.status in ['RESOLVED', 'CLOSED']
+    
+    @property
+    def resolution_time(self):
+        if self.resolved_at and self.created_at:
+            return self.resolved_at - self.created_at
+        return None
+    
+    def has_user_access(self, user):
+        """Check if user has access to this incident based on security clearance"""
+        if not self.data_classification:
+            return True  # No classification means accessible to all
+        
+        return user.has_data_access(self.data_classification.level)
+    
+    def get_required_clearance_level(self):
+        """Get the required clearance level for this incident"""
+        if self.data_classification:
+            return self.data_classification.level
+        return 1  # Default to PUBLIC level
+    
+    def is_accessible_by_user(self, user):
+        """Check if user can access this incident"""
+        # Check basic access
+        if not self.has_user_access(user):
+            return False
+        
+        # Check if incident is sensitive and user has appropriate clearance
+        if self.is_sensitive and not user.clearance_level:
+            return False
+        
+        # Check security clearance requirement
+        if self.security_clearance_required:
+            if not user.clearance_level or user.clearance_level.level < self.get_required_clearance_level():
+                return False
+        
+        return True
+    
+    def should_suppress_for_maintenance(self):
+        """Check if this incident should be suppressed due to active maintenance windows"""
+        if self.maintenance_window_override:
+            return False
+        
+        from automation_orchestration.models import MaintenanceWindow
+        from django.utils import timezone
+        
+        now = timezone.now()
+        active_maintenance = MaintenanceWindow.objects.filter(
+            start_time__lte=now,
+            end_time__gte=now,
+            status='ACTIVE',
+            suppress_incident_creation=True
+        )
+        
+        # Check if any active maintenance window affects this incident
+        for maintenance in active_maintenance:
+            # Check if incident category matches affected services/components
+            if (self.category in maintenance.affected_services or 
+                self.category in maintenance.affected_components):
+                return True
+        
+        return False
+
+
+class IncidentClassification(models.Model):
+    """AI-driven incident classification results"""
+    
+    incident = models.OneToOneField(Incident, on_delete=models.CASCADE, related_name='ai_classification')
+    
+    # Classification results
+    predicted_category = models.CharField(max_length=100)
+    predicted_subcategory = models.CharField(max_length=100)
+    confidence_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Alternative classifications
+    alternative_categories = models.JSONField(default=list, help_text="List of alternative category predictions")
+    
+    # NLP Analysis
+    extracted_keywords = models.JSONField(default=list, help_text="Keywords extracted from incident text")
+    sentiment_score = models.FloatField(null=True, blank=True, help_text="Sentiment analysis score (-1 to 1)")
+    urgency_indicators = models.JSONField(default=list, help_text="Detected urgency indicators")
+    
+    # Processing metadata
+    model_version = models.CharField(max_length=50, default='v1.0')
+    processing_time = models.FloatField(help_text="Time taken for classification in seconds")
+    created_at = models.DateTimeField(auto_now_add=True)
+    
+    def __str__(self):
+        return f"Classification for {self.incident.title}: {self.predicted_category}"
+
+
+class SeveritySuggestion(models.Model):
+    """AI-driven severity suggestions based on impact analysis"""
+    
+    incident = models.OneToOneField(Incident, on_delete=models.CASCADE, related_name='severity_suggestion')
+    
+    # Severity prediction
+    suggested_severity = models.CharField(max_length=20, choices=Incident.SEVERITY_CHOICES)
+    confidence_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Impact analysis factors
+    user_impact_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    business_impact_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    technical_impact_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Reasoning
+    reasoning = models.TextField(help_text="AI explanation for severity suggestion")
+    impact_factors = models.JSONField(default=list, help_text="List of factors that influenced the severity")
+    
+    # Processing metadata
+    model_version = models.CharField(max_length=50, default='v1.0')
+    processing_time = models.FloatField(help_text="Time taken for severity analysis in seconds")
+    created_at = models.DateTimeField(auto_now_add=True)
+    
+    def __str__(self):
+        return f"Severity suggestion for {self.incident.title}: {self.suggested_severity}"
+
+
+class IncidentCorrelation(models.Model):
+    """Correlation engine for linking related incidents"""
+    
+    CORRELATION_TYPE_CHOICES = [
+        ('SAME_SERVICE', 'Same Service'),
+        ('SAME_COMPONENT', 'Same Component'),
+        ('TEMPORAL', 'Temporal Correlation'),
+        ('PATTERN', 'Pattern Match'),
+        ('DEPENDENCY', 'Dependency Related'),
+        ('CASCADE', 'Cascade Effect'),
+    ]
+    
+    # Related incidents
+    primary_incident = models.ForeignKey(Incident, on_delete=models.CASCADE, related_name='correlations_as_primary')
+    related_incident = models.ForeignKey(Incident, on_delete=models.CASCADE, related_name='correlations_as_related')
+    
+    # Correlation details
+    correlation_type = models.CharField(max_length=20, choices=CORRELATION_TYPE_CHOICES)
+    confidence_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    correlation_strength = models.CharField(max_length=20, choices=[
+        ('WEAK', 'Weak'),
+        ('MODERATE', 'Moderate'),
+        ('STRONG', 'Strong'),
+        ('VERY_STRONG', 'Very Strong'),
+    ])
+    
+    # Analysis details
+    shared_keywords = models.JSONField(default=list, help_text="Keywords shared between incidents")
+    time_difference = models.DurationField(help_text="Time difference between incidents")
+    similarity_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Problem detection
+    is_problem_indicator = models.BooleanField(default=False, help_text="Indicates if this correlation suggests a larger problem")
+    problem_description = models.TextField(blank=True, null=True)
+    
+    # Metadata
+    created_at = models.DateTimeField(auto_now_add=True)
+    model_version = models.CharField(max_length=50, default='v1.0')
+    
+    class Meta:
+        unique_together = ['primary_incident', 'related_incident']
+        indexes = [
+            models.Index(fields=['correlation_type', 'confidence_score']),
+            models.Index(fields=['is_problem_indicator']),
+        ]
+    
+    def __str__(self):
+        return f"Correlation: {self.primary_incident.title} <-> {self.related_incident.title}"
+
+
+class DuplicationDetection(models.Model):
+    """Duplication detection results for incident merging"""
+    
+    DUPLICATION_TYPE_CHOICES = [
+        ('EXACT', 'Exact Duplicate'),
+        ('NEAR_DUPLICATE', 'Near Duplicate'),
+        ('SIMILAR', 'Similar Incident'),
+        ('POTENTIAL_DUPLICATE', 'Potential Duplicate'),
+    ]
+    
+    # Incident pair
+    incident_a = models.ForeignKey(Incident, on_delete=models.CASCADE, related_name='duplication_as_a')
+    incident_b = models.ForeignKey(Incident, on_delete=models.CASCADE, related_name='duplication_as_b')
+    
+    # Duplication analysis
+    duplication_type = models.CharField(max_length=20, choices=DUPLICATION_TYPE_CHOICES)
+    similarity_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    confidence_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Analysis details
+    text_similarity = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    temporal_proximity = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    service_similarity = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Merge recommendation
+    recommended_action = models.CharField(max_length=20, choices=[
+        ('MERGE', 'Merge Incidents'),
+        ('LINK', 'Link Incidents'),
+        ('REVIEW', 'Manual Review'),
+        ('NO_ACTION', 'No Action'),
+    ])
+    merge_confidence = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Reasoning
+    reasoning = models.TextField(help_text="AI explanation for duplication detection")
+    shared_elements = models.JSONField(default=list, help_text="Elements shared between incidents")
+    
+    # Status
+    status = models.CharField(max_length=20, choices=[
+        ('DETECTED', 'Detected'),
+        ('REVIEWED', 'Reviewed'),
+        ('MERGED', 'Merged'),
+        ('REJECTED', 'Rejected'),
+    ], default='DETECTED')
+    
+    # Metadata
+    created_at = models.DateTimeField(auto_now_add=True)
+    reviewed_at = models.DateTimeField(null=True, blank=True)
+    reviewed_by = models.ForeignKey(User, on_delete=models.SET_NULL, null=True, blank=True)
+    model_version = models.CharField(max_length=50, default='v1.0')
+    
+    class Meta:
+        unique_together = ['incident_a', 'incident_b']
+        indexes = [
+            models.Index(fields=['duplication_type', 'confidence_score']),
+            models.Index(fields=['status']),
+        ]
+    
+    def __str__(self):
+        return f"Duplication: {self.incident_a.title} <-> {self.incident_b.title}"
+
+
+class IncidentPattern(models.Model):
+    """Pattern detection for identifying recurring issues"""
+    
+    PATTERN_TYPE_CHOICES = [
+        ('RECURRING', 'Recurring Issue'),
+        ('SEASONAL', 'Seasonal Pattern'),
+        ('TREND', 'Trend Analysis'),
+        ('ANOMALY', 'Anomaly Detection'),
+    ]
+    
+    name = models.CharField(max_length=200)
+    pattern_type = models.CharField(max_length=20, choices=PATTERN_TYPE_CHOICES)
+    description = models.TextField()
+    
+    # Pattern characteristics
+    frequency = models.CharField(max_length=50, help_text="How often this pattern occurs")
+    affected_services = models.JSONField(default=list, help_text="Services affected by this pattern")
+    common_keywords = models.JSONField(default=list, help_text="Common keywords in incidents with this pattern")
+    
+    # Related incidents
+    incidents = models.ManyToManyField(Incident, related_name='patterns')
+    incident_count = models.PositiveIntegerField(default=0)
+    
+    # Pattern analysis
+    confidence_score = models.FloatField(validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    last_occurrence = models.DateTimeField(null=True, blank=True)
+    next_predicted_occurrence = models.DateTimeField(null=True, blank=True)
+    
+    # Status
+    is_active = models.BooleanField(default=True)
+    is_resolved = models.BooleanField(default=False)
+    
+    # Metadata
+    created_at = models.DateTimeField(auto_now_add=True)
+    updated_at = models.DateTimeField(auto_now=True)
+    model_version = models.CharField(max_length=50, default='v1.0')
+    
+    class Meta:
+        ordering = ['-confidence_score', '-incident_count']
+        indexes = [
+            models.Index(fields=['pattern_type', 'is_active']),
+            models.Index(fields=['confidence_score']),
+        ]
+    
+    def __str__(self):
+        return f"Pattern: {self.name} ({self.pattern_type})"
+
+
+class AIProcessingLog(models.Model):
+    """Log of AI processing activities for audit and debugging"""
+    
+    PROCESSING_TYPE_CHOICES = [
+        ('CLASSIFICATION', 'Classification'),
+        ('SEVERITY_ANALYSIS', 'Severity Analysis'),
+        ('CORRELATION', 'Correlation Analysis'),
+        ('DUPLICATION_DETECTION', 'Duplication Detection'),
+        ('PATTERN_DETECTION', 'Pattern Detection'),
+    ]
+    
+    STATUS_CHOICES = [
+        ('PENDING', 'Pending'),
+        ('PROCESSING', 'Processing'),
+        ('COMPLETED', 'Completed'),
+        ('FAILED', 'Failed'),
+        ('SKIPPED', 'Skipped'),
+    ]
+    
+    # Processing details
+    processing_type = models.CharField(max_length=30, choices=PROCESSING_TYPE_CHOICES)
+    status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')
+    
+    # Related objects
+    incident = models.ForeignKey(Incident, on_delete=models.CASCADE, null=True, blank=True)
+    related_incidents = models.JSONField(default=list, help_text="List of related incident IDs")
+    
+    # Processing results
+    input_data = models.JSONField(help_text="Input data for processing")
+    output_data = models.JSONField(null=True, blank=True, help_text="Output data from processing")
+    error_message = models.TextField(blank=True, null=True)
+    
+    # Performance metrics
+    processing_time = models.FloatField(null=True, blank=True, help_text="Processing time in seconds")
+    model_version = models.CharField(max_length=50, default='v1.0')
+    confidence_score = models.FloatField(null=True, blank=True, validators=[MinValueValidator(0.0), MaxValueValidator(1.0)])
+    
+    # Timestamps
+    started_at = models.DateTimeField(auto_now_add=True)
+    completed_at = models.DateTimeField(null=True, blank=True)
+    
+    class Meta:
+        ordering = ['-started_at']
+        indexes = [
+            models.Index(fields=['processing_type', 'status']),
+            models.Index(fields=['incident', 'processing_type']),
+        ]
+    
+    def __str__(self):
+        return f"{self.processing_type} - {self.status} - {self.incident.title if self.incident else 'N/A'}"

ETB-API/incident_intelligence/security.py

@@ -0,0 +1,206 @@
+"""
+Security integration for incident intelligence
+"""
+from django.core.exceptions import PermissionDenied
+from django.http import JsonResponse
+from django.utils.deprecation import MiddlewareMixin
+from django.contrib.auth import get_user_model
+from django.db import models
+from security.models import AuditLog
+import logging
+
+User = get_user_model()
+logger = logging.getLogger(__name__)
+
+
+class IncidentSecurityMiddleware(MiddlewareMixin):
+    """
+    Middleware to enforce security policies for incident intelligence
+    """
+    
+    def process_request(self, request):
+        """Process request and check security permissions"""
+        # Skip security checks for non-incident endpoints
+        if not request.path.startswith('/api/incidents/'):
+            return None
+        
+        # Skip for admin and authentication endpoints
+        if any(path in request.path for path in ['/admin/', '/auth/', '/login/', '/logout/']):
+            return None
+        
+        # Check if user is authenticated
+        if not request.user.is_authenticated:
+            return JsonResponse({
+                'error': 'Authentication required',
+                'detail': 'You must be logged in to access incident data'
+            }, status=401)
+        
+        # Log access attempt
+        self._log_access_attempt(request)
+        
+        return None
+    
+    def process_response(self, request, response):
+        """Process response and log security events"""
+        # Log security-relevant responses
+        if request.path.startswith('/api/incidents/'):
+            self._log_security_event(request, response)
+        
+        return response
+    
+    def _log_access_attempt(self, request):
+        """Log incident access attempt"""
+        try:
+            AuditLog.objects.create(
+                user=request.user,
+                action_type='DATA_ACCESS',
+                resource_type='incident_intelligence',
+                resource_id=request.path,
+                ip_address=self._get_client_ip(request),
+                user_agent=request.META.get('HTTP_USER_AGENT', ''),
+                details={
+                    'method': request.method,
+                    'path': request.path,
+                    'query_params': dict(request.GET),
+                },
+                severity='LOW'
+            )
+        except Exception as e:
+            logger.error(f"Failed to log access attempt: {e}")
+    
+    def _log_security_event(self, request, response):
+        """Log security events"""
+        try:
+            # Determine severity based on response status
+            if response.status_code >= 400:
+                severity = 'HIGH' if response.status_code >= 500 else 'MEDIUM'
+            else:
+                severity = 'LOW'
+            
+            # Log the event
+            AuditLog.objects.create(
+                user=request.user,
+                action_type='DATA_ACCESS',
+                resource_type='incident_intelligence',
+                resource_id=request.path,
+                ip_address=self._get_client_ip(request),
+                user_agent=request.META.get('HTTP_USER_AGENT', ''),
+                details={
+                    'method': request.method,
+                    'path': request.path,
+                    'status_code': response.status_code,
+                    'response_size': len(response.content) if hasattr(response, 'content') else 0,
+                },
+                severity=severity
+            )
+        except Exception as e:
+            logger.error(f"Failed to log security event: {e}")
+    
+    def _get_client_ip(self, request):
+        """Get client IP address"""
+        x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
+        if x_forwarded_for:
+            ip = x_forwarded_for.split(',')[0]
+        else:
+            ip = request.META.get('REMOTE_ADDR')
+        return ip
+
+
+class IncidentAccessControl:
+    """
+    Access control utilities for incident intelligence
+    """
+    
+    @staticmethod
+    def check_incident_access(user, incident, action='read'):
+        """Check if user has access to perform action on incident"""
+        # Check basic incident access
+        if not incident.is_accessible_by_user(user):
+            raise PermissionDenied("Insufficient clearance level for this incident")
+        
+        # Check role-based permissions
+        if not IncidentAccessControl._check_role_permissions(user, action):
+            raise PermissionDenied(f"User does not have permission to {action} incidents")
+        
+        # Check data classification access
+        if not IncidentAccessControl._check_data_classification_access(user, incident):
+            raise PermissionDenied("Insufficient data classification access")
+        
+        return True
+    
+    @staticmethod
+    def _check_role_permissions(user, action):
+        """Check if user has required role permissions"""
+        # Get user's effective permissions
+        permissions = user.get_effective_permissions()
+        
+        # Define required permissions for each action
+        required_permissions = {
+            'read': ['incident_intelligence.view_incident'],
+            'create': ['incident_intelligence.add_incident'],
+            'update': ['incident_intelligence.change_incident'],
+            'delete': ['incident_intelligence.delete_incident'],
+            'analyze': ['incident_intelligence.analyze_incident'],
+        }
+        
+        required = required_permissions.get(action, [])
+        return all(perm in [p.codename for p in permissions] for perm in required)
+    
+    @staticmethod
+    def _check_data_classification_access(user, incident):
+        """Check if user has access to incident's data classification"""
+        if not incident.data_classification:
+            return True  # No classification means accessible to all
+        
+        return user.has_data_access(incident.data_classification.level)
+    
+    @staticmethod
+    def filter_incidents_by_access(user, queryset):
+        """Filter incidents based on user's access level"""
+        # Get user's clearance level
+        user_clearance = user.clearance_level.level if user.clearance_level else 1
+        
+        # Filter incidents based on data classification
+        accessible_incidents = queryset.filter(
+            models.Q(data_classification__isnull=True) |  # No classification
+            models.Q(data_classification__level__lte=user_clearance)  # User has sufficient clearance
+        )
+        
+        # Filter out sensitive incidents if user doesn't have clearance
+        if not user.clearance_level or user.clearance_level.level < 3:  # Less than CONFIDENTIAL
+            accessible_incidents = accessible_incidents.filter(is_sensitive=False)
+        
+        return accessible_incidents
+
+
+def log_incident_operation(user, incident, operation, details=None):
+    """Log incident operations for audit trail"""
+    try:
+        AuditLog.objects.create(
+            user=user,
+            action_type='DATA_MODIFIED' if operation in ['create', 'update', 'delete'] else 'DATA_ACCESS',
+            resource_type='incident',
+            resource_id=str(incident.id),
+            details={
+                'operation': operation,
+                'incident_title': incident.title,
+                'incident_severity': incident.severity,
+                'incident_category': incident.category,
+                'data_classification': incident.data_classification.name if incident.data_classification else None,
+                **(details or {})
+            },
+            severity='HIGH' if incident.severity in ['CRITICAL', 'EMERGENCY'] else 'MEDIUM'
+        )
+    except Exception as e:
+        logger.error(f"Failed to log incident operation: {e}")
+
+
+def get_user_accessible_incidents(user):
+    """Get incidents accessible by user based on security policies"""
+    from .models import Incident
+    
+    # Start with all incidents
+    queryset = Incident.objects.all()
+    
+    # Apply access control filters
+    return IncidentAccessControl.filter_incidents_by_access(user, queryset)

ETB-API/incident_intelligence/serializers/__init__.py

@@ -0,0 +1 @@
+# Serializers for incident intelligence

ETB-API/incident_intelligence/serializers/incident.py

@@ -0,0 +1,271 @@
+"""
+Serializers for incident intelligence models
+"""
+from rest_framework import serializers
+from django.contrib.auth import get_user_model
+from ..models import (
+    Incident, IncidentClassification, SeveritySuggestion, IncidentCorrelation,
+    DuplicationDetection, IncidentPattern, AIProcessingLog
+)
+
+User = get_user_model()
+
+
+class IncidentSerializer(serializers.ModelSerializer):
+    """Serializer for Incident model"""
+    
+    reporter_name = serializers.CharField(source='reporter.get_full_name', read_only=True)
+    assigned_to_name = serializers.CharField(source='assigned_to.get_full_name', read_only=True)
+    resolution_time_display = serializers.SerializerMethodField()
+    is_resolved = serializers.BooleanField(read_only=True)
+    
+    class Meta:
+        model = Incident
+        fields = [
+            'id', 'title', 'description', 'free_text', 'category', 'subcategory',
+            'classification_confidence', 'severity', 'suggested_severity',
+            'severity_confidence', 'priority', 'status', 'assigned_to', 'assigned_to_name',
+            'reporter', 'reporter_name', 'created_at', 'updated_at', 'resolved_at',
+            'affected_users', 'business_impact', 'estimated_downtime', 'ai_processed',
+            'ai_processing_error', 'last_ai_analysis', 'is_duplicate', 'original_incident',
+            'duplicate_confidence', 'resolution_time_display', 'is_resolved',
+            'data_classification', 'security_clearance_required', 'is_sensitive'
+        ]
+        read_only_fields = ['id', 'created_at', 'updated_at', 'ai_processed', 'ai_processing_error']
+    
+    def get_resolution_time_display(self, obj):
+        """Get human-readable resolution time"""
+        if obj.resolution_time:
+            total_seconds = obj.resolution_time.total_seconds()
+            hours = int(total_seconds // 3600)
+            minutes = int((total_seconds % 3600) // 60)
+            if hours > 0:
+                return f"{hours}h {minutes}m"
+            else:
+                return f"{minutes}m"
+        return None
+
+
+class IncidentCreateSerializer(serializers.ModelSerializer):
+    """Serializer for creating new incidents"""
+    
+    class Meta:
+        model = Incident
+        fields = [
+            'title', 'description', 'free_text', 'affected_users', 'business_impact',
+            'estimated_downtime', 'reporter'
+        ]
+    
+    def create(self, validated_data):
+        """Create a new incident and trigger AI processing"""
+        incident = super().create(validated_data)
+        
+        # Trigger AI processing asynchronously
+        from ..tasks import process_incident_ai
+        process_incident_ai.delay(incident.id)
+        
+        return incident
+
+
+class IncidentUpdateSerializer(serializers.ModelSerializer):
+    """Serializer for updating incidents"""
+    
+    class Meta:
+        model = Incident
+        fields = [
+            'title', 'description', 'category', 'subcategory', 'severity', 'priority',
+            'status', 'assigned_to', 'affected_users', 'business_impact', 'estimated_downtime'
+        ]
+    
+    def update(self, instance, validated_data):
+        """Update incident and trigger re-analysis if needed"""
+        # Check if fields that affect AI analysis have changed
+        ai_relevant_fields = ['title', 'description', 'category', 'subcategory', 'severity']
+        needs_reanalysis = any(field in validated_data for field in ai_relevant_fields)
+        
+        incident = super().update(instance, validated_data)
+        
+        # Trigger re-analysis if needed
+        if needs_reanalysis:
+            from ..tasks import process_incident_ai
+            process_incident_ai.delay(incident.id)
+        
+        return incident
+
+
+class IncidentClassificationSerializer(serializers.ModelSerializer):
+    """Serializer for IncidentClassification model"""
+    
+    class Meta:
+        model = IncidentClassification
+        fields = [
+            'id', 'incident', 'predicted_category', 'predicted_subcategory',
+            'confidence_score', 'alternative_categories', 'extracted_keywords',
+            'sentiment_score', 'urgency_indicators', 'model_version',
+            'processing_time', 'created_at'
+        ]
+        read_only_fields = ['id', 'created_at']
+
+
+class SeveritySuggestionSerializer(serializers.ModelSerializer):
+    """Serializer for SeveritySuggestion model"""
+    
+    class Meta:
+        model = SeveritySuggestion
+        fields = [
+            'id', 'incident', 'suggested_severity', 'confidence_score',
+            'user_impact_score', 'business_impact_score', 'technical_impact_score',
+            'reasoning', 'impact_factors', 'model_version', 'processing_time', 'created_at'
+        ]
+        read_only_fields = ['id', 'created_at']
+
+
+class IncidentCorrelationSerializer(serializers.ModelSerializer):
+    """Serializer for IncidentCorrelation model"""
+    
+    primary_incident_title = serializers.CharField(source='primary_incident.title', read_only=True)
+    related_incident_title = serializers.CharField(source='related_incident.title', read_only=True)
+    
+    class Meta:
+        model = IncidentCorrelation
+        fields = [
+            'id', 'primary_incident', 'primary_incident_title', 'related_incident',
+            'related_incident_title', 'correlation_type', 'confidence_score',
+            'correlation_strength', 'shared_keywords', 'time_difference',
+            'similarity_score', 'is_problem_indicator', 'problem_description',
+            'model_version', 'created_at'
+        ]
+        read_only_fields = ['id', 'created_at']
+
+
+class DuplicationDetectionSerializer(serializers.ModelSerializer):
+    """Serializer for DuplicationDetection model"""
+    
+    incident_a_title = serializers.CharField(source='incident_a.title', read_only=True)
+    incident_b_title = serializers.CharField(source='incident_b.title', read_only=True)
+    reviewed_by_name = serializers.CharField(source='reviewed_by.get_full_name', read_only=True)
+    
+    class Meta:
+        model = DuplicationDetection
+        fields = [
+            'id', 'incident_a', 'incident_a_title', 'incident_b', 'incident_b_title',
+            'duplication_type', 'similarity_score', 'confidence_score',
+            'text_similarity', 'temporal_proximity', 'service_similarity',
+            'recommended_action', 'merge_confidence', 'reasoning', 'shared_elements',
+            'status', 'created_at', 'reviewed_at', 'reviewed_by', 'reviewed_by_name',
+            'model_version'
+        ]
+        read_only_fields = ['id', 'created_at', 'reviewed_at']
+
+
+class IncidentPatternSerializer(serializers.ModelSerializer):
+    """Serializer for IncidentPattern model"""
+    
+    incident_count = serializers.IntegerField(read_only=True)
+    
+    class Meta:
+        model = IncidentPattern
+        fields = [
+            'id', 'name', 'pattern_type', 'description', 'frequency',
+            'affected_services', 'common_keywords', 'incidents', 'incident_count',
+            'confidence_score', 'last_occurrence', 'next_predicted_occurrence',
+            'is_active', 'is_resolved', 'created_at', 'updated_at', 'model_version'
+        ]
+        read_only_fields = ['id', 'created_at', 'updated_at', 'incident_count']
+
+
+class AIProcessingLogSerializer(serializers.ModelSerializer):
+    """Serializer for AIProcessingLog model"""
+    
+    incident_title = serializers.CharField(source='incident.title', read_only=True)
+    
+    class Meta:
+        model = AIProcessingLog
+        fields = [
+            'id', 'processing_type', 'status', 'incident', 'incident_title',
+            'related_incidents', 'input_data', 'output_data', 'error_message',
+            'processing_time', 'model_version', 'confidence_score',
+            'started_at', 'completed_at'
+        ]
+        read_only_fields = ['id', 'started_at', 'completed_at']
+
+
+class IncidentAnalysisSerializer(serializers.Serializer):
+    """Serializer for incident analysis results"""
+    
+    incident = IncidentSerializer(read_only=True)
+    classification = IncidentClassificationSerializer(read_only=True, allow_null=True)
+    severity_suggestion = SeveritySuggestionSerializer(read_only=True, allow_null=True)
+    correlations = IncidentCorrelationSerializer(many=True, read_only=True)
+    duplications = DuplicationDetectionSerializer(many=True, read_only=True)
+    patterns = IncidentPatternSerializer(many=True, read_only=True)
+    
+    def to_representation(self, instance):
+        """Custom representation to include related data"""
+        data = super().to_representation(instance)
+        
+        # Add classification data
+        try:
+            data['classification'] = IncidentClassificationSerializer(instance.ai_classification).data
+        except IncidentClassification.DoesNotExist:
+            data['classification'] = None
+        
+        # Add severity suggestion data
+        try:
+            data['severity_suggestion'] = SeveritySuggestionSerializer(instance.severity_suggestion).data
+        except SeveritySuggestion.DoesNotExist:
+            data['severity_suggestion'] = None
+        
+        # Add correlations
+        data['correlations'] = IncidentCorrelationSerializer(
+            instance.correlations_as_primary.all()[:10], many=True
+        ).data
+        
+        # Add duplications
+        data['duplications'] = DuplicationDetectionSerializer(
+            instance.duplication_as_a.all()[:10], many=True
+        ).data
+        
+        # Add patterns
+        data['patterns'] = IncidentPatternSerializer(
+            instance.patterns.all()[:5], many=True
+        ).data
+        
+        return data
+
+
+class IncidentSearchSerializer(serializers.Serializer):
+    """Serializer for incident search parameters"""
+    
+    query = serializers.CharField(required=False, help_text="Search query")
+    category = serializers.CharField(required=False, help_text="Filter by category")
+    severity = serializers.ChoiceField(choices=Incident.SEVERITY_CHOICES, required=False)
+    status = serializers.ChoiceField(choices=Incident.STATUS_CHOICES, required=False)
+    assigned_to = serializers.IntegerField(required=False, help_text="Filter by assigned user ID")
+    reporter = serializers.IntegerField(required=False, help_text="Filter by reporter user ID")
+    date_from = serializers.DateTimeField(required=False, help_text="Filter incidents from date")
+    date_to = serializers.DateTimeField(required=False, help_text="Filter incidents to date")
+    has_ai_analysis = serializers.BooleanField(required=False, help_text="Filter by AI analysis status")
+    is_duplicate = serializers.BooleanField(required=False, help_text="Filter by duplication status")
+    page = serializers.IntegerField(default=1, min_value=1)
+    page_size = serializers.IntegerField(default=20, min_value=1, max_value=100)
+
+
+class IncidentStatsSerializer(serializers.Serializer):
+    """Serializer for incident statistics"""
+    
+    total_incidents = serializers.IntegerField()
+    open_incidents = serializers.IntegerField()
+    resolved_incidents = serializers.IntegerField()
+    critical_incidents = serializers.IntegerField()
+    high_incidents = serializers.IntegerField()
+    medium_incidents = serializers.IntegerField()
+    low_incidents = serializers.IntegerField()
+    average_resolution_time = serializers.DurationField()
+    incidents_by_category = serializers.DictField()
+    incidents_by_severity = serializers.DictField()
+    incidents_by_status = serializers.DictField()
+    ai_processed_count = serializers.IntegerField()
+    duplicate_count = serializers.IntegerField()
+    correlation_count = serializers.IntegerField()
+    pattern_count = serializers.IntegerField()

ETB-API/incident_intelligence/signals.py

@@ -0,0 +1,61 @@
+"""
+Django signals for incident intelligence
+"""
+from django.db.models.signals import post_save, pre_save
+from django.dispatch import receiver
+from django.utils import timezone
+import logging
+
+from .models import Incident
+
+logger = logging.getLogger(__name__)
+
+
+@receiver(post_save, sender=Incident)
+def incident_post_save(sender, instance, created, **kwargs):
+    """
+    Handle incident post-save events
+    """
+    if created:
+        logger.info(f"New incident created: {instance.id}")
+        
+        # Trigger AI processing for new incidents
+        try:
+            from .tasks import process_incident_ai
+            process_incident_ai.delay(instance.id)
+        except Exception as e:
+            logger.error(f"Failed to trigger AI processing for incident {instance.id}: {e}")
+    
+    # Update resolution time if status changed to resolved
+    if not created and instance.status in ['RESOLVED', 'CLOSED'] and not instance.resolved_at:
+        instance.resolved_at = timezone.now()
+        instance.save(update_fields=['resolved_at'])
+
+
+@receiver(pre_save, sender=Incident)
+def incident_pre_save(sender, instance, **kwargs):
+    """
+    Handle incident pre-save events
+    """
+    # Auto-assign priority based on severity
+    if instance.severity and not instance.priority:
+        severity_priority_map = {
+            'EMERGENCY': 'P1',
+            'CRITICAL': 'P1',
+            'HIGH': 'P2',
+            'MEDIUM': 'P3',
+            'LOW': 'P4'
+        }
+        instance.priority = severity_priority_map.get(instance.severity, 'P3')
+    
+    # Update AI processing flag if key fields changed
+    if instance.pk:
+        try:
+            old_instance = Incident.objects.get(pk=instance.pk)
+            ai_relevant_fields = ['title', 'description', 'free_text', 'category', 'subcategory', 'severity']
+            
+            if any(getattr(old_instance, field) != getattr(instance, field) for field in ai_relevant_fields):
+                instance.ai_processed = False
+                instance.ai_processing_error = None
+        except Incident.DoesNotExist:
+            pass

ETB-API/incident_intelligence/tasks.py

@@ -0,0 +1,677 @@
+"""
+Celery tasks for incident intelligence processing
+"""
+from celery import shared_task
+from django.utils import timezone
+from django.db import transaction
+import logging
+
+from .models import (
+    Incident, IncidentClassification, SeveritySuggestion, IncidentCorrelation,
+    DuplicationDetection, IncidentPattern, AIProcessingLog
+)
+from .ai.classification import IncidentClassifier, SeverityAnalyzer
+from .ai.correlation import IncidentCorrelationEngine
+from .ai.duplication import DuplicationDetector
+
+logger = logging.getLogger(__name__)
+
+
+@shared_task(bind=True, max_retries=3)
+def process_incident_ai(self, incident_id):
+    """
+    Process a single incident with AI analysis
+    """
+    try:
+        incident = Incident.objects.get(id=incident_id)
+        
+        # Create processing log
+        log = AIProcessingLog.objects.create(
+            processing_type='CLASSIFICATION',
+            status='PROCESSING',
+            incident=incident,
+            input_data={
+                'incident_id': str(incident_id),
+                'title': incident.title,
+                'description': incident.description,
+                'free_text': incident.free_text
+            }
+        )
+        
+        # Initialize AI components
+        classifier = IncidentClassifier()
+        severity_analyzer = SeverityAnalyzer()
+        
+        # Perform classification
+        classification_result = classifier.classify_incident(
+            incident.title, incident.description, incident.free_text
+        )
+        
+        # Save classification results
+        with transaction.atomic():
+            # Update incident with classification
+            incident.category = classification_result.category
+            incident.subcategory = classification_result.subcategory
+            incident.classification_confidence = classification_result.confidence
+            
+            # Create detailed classification record
+            IncidentClassification.objects.update_or_create(
+                incident=incident,
+                defaults={
+                    'predicted_category': classification_result.category,
+                    'predicted_subcategory': classification_result.subcategory,
+                    'confidence_score': classification_result.confidence,
+                    'alternative_categories': [
+                        {'category': alt['category'], 'confidence': alt['confidence']}
+                        for alt in classification_result.alternative_categories
+                    ],
+                    'extracted_keywords': classification_result.keywords,
+                    'sentiment_score': classification_result.sentiment_score,
+                    'urgency_indicators': classification_result.urgency_indicators,
+                    'processing_time': classification_result.processing_time,
+                    'model_version': classifier.model_version
+                }
+            )
+            
+            # Perform severity analysis
+            severity_result = severity_analyzer.analyze_severity({
+                'title': incident.title,
+                'description': incident.description,
+                'free_text': incident.free_text,
+                'affected_users': incident.affected_users,
+                'business_impact': incident.business_impact
+            })
+            
+            # Update incident with severity suggestion
+            incident.suggested_severity = severity_result['suggested_severity']
+            incident.severity_confidence = severity_result['confidence_score']
+            
+            # Create detailed severity suggestion record
+            SeveritySuggestion.objects.update_or_create(
+                incident=incident,
+                defaults={
+                    'suggested_severity': severity_result['suggested_severity'],
+                    'confidence_score': severity_result['confidence_score'],
+                    'user_impact_score': severity_result['user_impact_score'],
+                    'business_impact_score': severity_result['business_impact_score'],
+                    'technical_impact_score': severity_result['technical_impact_score'],
+                    'reasoning': severity_result['reasoning'],
+                    'impact_factors': severity_result['impact_factors'],
+                    'processing_time': severity_result['processing_time'],
+                    'model_version': severity_analyzer.model_version
+                }
+            )
+            
+            # Mark as processed
+            incident.ai_processed = True
+            incident.last_ai_analysis = timezone.now()
+            incident.save()
+        
+        # Update processing log
+        log.status = 'COMPLETED'
+        log.completed_at = timezone.now()
+        log.output_data = {
+            'classification': {
+                'category': classification_result.category,
+                'subcategory': classification_result.subcategory,
+                'confidence': classification_result.confidence
+            },
+            'severity': {
+                'suggested_severity': severity_result['suggested_severity'],
+                'confidence': severity_result['confidence_score']
+            }
+        }
+        log.processing_time = (log.completed_at - log.started_at).total_seconds()
+        log.save()
+        
+        logger.info(f"Successfully processed incident {incident_id} with AI analysis")
+        
+        # Trigger correlation and duplication detection
+        find_correlations.delay(incident_id)
+        find_duplicates.delay(incident_id)
+        
+        return f"Successfully processed incident {incident_id}"
+    
+    except Incident.DoesNotExist:
+        logger.error(f"Incident {incident_id} not found")
+        return f"Incident {incident_id} not found"
+    
+    except Exception as e:
+        logger.error(f"Error processing incident {incident_id}: {str(e)}")
+        
+        # Update processing log with error
+        try:
+            log = AIProcessingLog.objects.filter(
+                incident_id=incident_id,
+                processing_type='CLASSIFICATION',
+                status='PROCESSING'
+            ).first()
+            
+            if log:
+                log.status = 'FAILED'
+                log.error_message = str(e)
+                log.completed_at = timezone.now()
+                log.save()
+        except:
+            pass
+        
+        # Update incident with error
+        try:
+            incident = Incident.objects.get(id=incident_id)
+            incident.ai_processing_error = str(e)
+            incident.save()
+        except:
+            pass
+        
+        # Retry if not max retries reached
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        
+        return f"Failed to process incident {incident_id}: {str(e)}"
+
+
+@shared_task(bind=True, max_retries=3)
+def batch_process_incidents_ai(self, incident_ids):
+    """
+    Process multiple incidents with AI analysis
+    """
+    try:
+        results = []
+        for incident_id in incident_ids:
+            try:
+                result = process_incident_ai.delay(incident_id)
+                results.append({'incident_id': incident_id, 'task_id': result.id})
+            except Exception as e:
+                logger.error(f"Failed to queue incident {incident_id}: {str(e)}")
+                results.append({'incident_id': incident_id, 'error': str(e)})
+        
+        return f"Queued {len(incident_ids)} incidents for processing"
+    
+    except Exception as e:
+        logger.error(f"Error in batch processing: {str(e)}")
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        return f"Failed to batch process incidents: {str(e)}"
+
+
+@shared_task(bind=True, max_retries=3)
+def find_correlations(self, incident_id):
+    """
+    Find correlations for a specific incident
+    """
+    try:
+        incident = Incident.objects.get(id=incident_id)
+        
+        # Create processing log
+        log = AIProcessingLog.objects.create(
+            processing_type='CORRELATION',
+            status='PROCESSING',
+            incident=incident,
+            input_data={'incident_id': str(incident_id)}
+        )
+        
+        # Get recent incidents for correlation analysis
+        recent_incidents = Incident.objects.filter(
+            created_at__gte=timezone.now() - timezone.timedelta(days=30)
+        ).exclude(id=incident_id)
+        
+        # Initialize correlation engine
+        correlation_engine = IncidentCorrelationEngine()
+        
+        # Find correlations
+        correlations = correlation_engine.find_related_incidents(
+            {
+                'id': str(incident.id),
+                'title': incident.title,
+                'description': incident.description,
+                'free_text': incident.free_text,
+                'category': incident.category,
+                'severity': incident.severity,
+                'status': incident.status,
+                'created_at': incident.created_at
+            },
+            [
+                {
+                    'id': str(inc.id),
+                    'title': inc.title,
+                    'description': inc.description,
+                    'free_text': inc.free_text,
+                    'category': inc.category,
+                    'severity': inc.severity,
+                    'status': inc.status,
+                    'created_at': inc.created_at
+                }
+                for inc in recent_incidents
+            ]
+        )
+        
+        # Save correlations
+        correlation_count = 0
+        with transaction.atomic():
+            for related_incident, correlation_result in correlations:
+                if correlation_result.confidence_score > 0.5:  # Only save high-confidence correlations
+                    IncidentCorrelation.objects.update_or_create(
+                        primary_incident=incident,
+                        related_incident_id=related_incident['id'],
+                        defaults={
+                            'correlation_type': correlation_result.correlation_type,
+                            'confidence_score': correlation_result.confidence_score,
+                            'correlation_strength': correlation_result.correlation_strength,
+                            'shared_keywords': correlation_result.shared_keywords,
+                            'time_difference': correlation_result.time_difference,
+                            'similarity_score': correlation_result.similarity_score,
+                            'is_problem_indicator': correlation_result.is_problem_indicator,
+                            'problem_description': correlation_result.problem_description,
+                            'model_version': correlation_engine.model_version
+                        }
+                    )
+                    correlation_count += 1
+        
+        # Update processing log
+        log.status = 'COMPLETED'
+        log.completed_at = timezone.now()
+        log.output_data = {
+            'correlations_found': correlation_count,
+            'total_analyzed': len(recent_incidents)
+        }
+        log.processing_time = (log.completed_at - log.started_at).total_seconds()
+        log.save()
+        
+        logger.info(f"Found {correlation_count} correlations for incident {incident_id}")
+        return f"Found {correlation_count} correlations for incident {incident_id}"
+    
+    except Incident.DoesNotExist:
+        logger.error(f"Incident {incident_id} not found")
+        return f"Incident {incident_id} not found"
+    
+    except Exception as e:
+        logger.error(f"Error finding correlations for incident {incident_id}: {str(e)}")
+        
+        # Update processing log with error
+        try:
+            log = AIProcessingLog.objects.filter(
+                incident_id=incident_id,
+                processing_type='CORRELATION',
+                status='PROCESSING'
+            ).first()
+            
+            if log:
+                log.status = 'FAILED'
+                log.error_message = str(e)
+                log.completed_at = timezone.now()
+                log.save()
+        except:
+            pass
+        
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        
+        return f"Failed to find correlations for incident {incident_id}: {str(e)}"
+
+
+@shared_task(bind=True, max_retries=3)
+def find_duplicates(self, incident_id):
+    """
+    Find duplicates for a specific incident
+    """
+    try:
+        incident = Incident.objects.get(id=incident_id)
+        
+        # Create processing log
+        log = AIProcessingLog.objects.create(
+            processing_type='DUPLICATION_DETECTION',
+            status='PROCESSING',
+            incident=incident,
+            input_data={'incident_id': str(incident_id)}
+        )
+        
+        # Get recent incidents for duplication detection
+        recent_incidents = Incident.objects.filter(
+            created_at__gte=timezone.now() - timezone.timedelta(days=7)
+        ).exclude(id=incident_id)
+        
+        # Initialize duplication detector
+        duplication_detector = DuplicationDetector()
+        
+        # Find duplicates
+        duplicates = duplication_detector.find_duplicate_candidates(
+            {
+                'id': str(incident.id),
+                'title': incident.title,
+                'description': incident.description,
+                'free_text': incident.free_text,
+                'category': incident.category,
+                'severity': incident.severity,
+                'status': incident.status,
+                'created_at': incident.created_at,
+                'assigned_to': incident.assigned_to_id,
+                'reporter': incident.reporter_id
+            },
+            [
+                {
+                    'id': str(inc.id),
+                    'title': inc.title,
+                    'description': inc.description,
+                    'free_text': inc.free_text,
+                    'category': inc.category,
+                    'severity': inc.severity,
+                    'status': inc.status,
+                    'created_at': inc.created_at,
+                    'assigned_to': inc.assigned_to_id,
+                    'reporter': inc.reporter_id
+                }
+                for inc in recent_incidents
+            ]
+        )
+        
+        # Save duplications
+        duplication_count = 0
+        with transaction.atomic():
+            for duplicate_incident, duplication_result in duplicates:
+                if duplication_result.confidence_score > 0.6:  # Only save high-confidence duplications
+                    DuplicationDetection.objects.update_or_create(
+                        incident_a=incident,
+                        incident_b_id=duplicate_incident['id'],
+                        defaults={
+                            'duplication_type': duplication_result.duplication_type,
+                            'similarity_score': duplication_result.similarity_score,
+                            'confidence_score': duplication_result.confidence_score,
+                            'text_similarity': duplication_result.text_similarity,
+                            'temporal_proximity': duplication_result.temporal_proximity,
+                            'service_similarity': duplication_result.service_similarity,
+                            'recommended_action': duplication_result.recommended_action,
+                            'merge_confidence': duplication_result.merge_confidence,
+                            'reasoning': duplication_result.reasoning,
+                            'shared_elements': duplication_result.shared_elements,
+                            'model_version': duplication_detector.model_version
+                        }
+                    )
+                    duplication_count += 1
+        
+        # Update processing log
+        log.status = 'COMPLETED'
+        log.completed_at = timezone.now()
+        log.output_data = {
+            'duplicates_found': duplication_count,
+            'total_analyzed': len(recent_incidents)
+        }
+        log.processing_time = (log.completed_at - log.started_at).total_seconds()
+        log.save()
+        
+        logger.info(f"Found {duplication_count} duplicates for incident {incident_id}")
+        return f"Found {duplication_count} duplicates for incident {incident_id}"
+    
+    except Incident.DoesNotExist:
+        logger.error(f"Incident {incident_id} not found")
+        return f"Incident {incident_id} not found"
+    
+    except Exception as e:
+        logger.error(f"Error finding duplicates for incident {incident_id}: {str(e)}")
+        
+        # Update processing log with error
+        try:
+            log = AIProcessingLog.objects.filter(
+                incident_id=incident_id,
+                processing_type='DUPLICATION_DETECTION',
+                status='PROCESSING'
+            ).first()
+            
+            if log:
+                log.status = 'FAILED'
+                log.error_message = str(e)
+                log.completed_at = timezone.now()
+                log.save()
+        except:
+            pass
+        
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        
+        return f"Failed to find duplicates for incident {incident_id}: {str(e)}"
+
+
+@shared_task(bind=True, max_retries=3)
+def detect_all_duplicates(self):
+    """
+    Detect duplicates across all incidents
+    """
+    try:
+        # Get all incidents
+        incidents = Incident.objects.all()
+        
+        # Initialize duplication detector
+        duplication_detector = DuplicationDetector()
+        
+        # Convert to list of dictionaries
+        incident_data = [
+            {
+                'id': str(inc.id),
+                'title': inc.title,
+                'description': inc.description,
+                'free_text': inc.free_text,
+                'category': inc.category,
+                'severity': inc.severity,
+                'status': inc.status,
+                'created_at': inc.created_at,
+                'assigned_to': inc.assigned_to_id,
+                'reporter': inc.reporter_id
+            }
+            for inc in incidents
+        ]
+        
+        # Batch detect duplicates
+        duplicates = duplication_detector.batch_detect_duplicates(incident_data)
+        
+        # Save duplications
+        duplication_count = 0
+        with transaction.atomic():
+            for incident_a_data, incident_b_data, duplication_result in duplicates:
+                if duplication_result.confidence_score > 0.6:  # Only save high-confidence duplications
+                    DuplicationDetection.objects.update_or_create(
+                        incident_a_id=incident_a_data['id'],
+                        incident_b_id=incident_b_data['id'],
+                        defaults={
+                            'duplication_type': duplication_result.duplication_type,
+                            'similarity_score': duplication_result.similarity_score,
+                            'confidence_score': duplication_result.confidence_score,
+                            'text_similarity': duplication_result.text_similarity,
+                            'temporal_proximity': duplication_result.temporal_proximity,
+                            'service_similarity': duplication_result.service_similarity,
+                            'recommended_action': duplication_result.recommended_action,
+                            'merge_confidence': duplication_result.merge_confidence,
+                            'reasoning': duplication_result.reasoning,
+                            'shared_elements': duplication_result.shared_elements,
+                            'model_version': duplication_detector.model_version
+                        }
+                    )
+                    duplication_count += 1
+        
+        logger.info(f"Detected {duplication_count} duplicate pairs across all incidents")
+        return f"Detected {duplication_count} duplicate pairs across all incidents"
+    
+    except Exception as e:
+        logger.error(f"Error in batch duplicate detection: {str(e)}")
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        return f"Failed to detect duplicates: {str(e)}"
+
+
+@shared_task(bind=True, max_retries=3)
+def correlate_all_incidents(self):
+    """
+    Run correlation analysis on all incidents
+    """
+    try:
+        # Get all incidents
+        incidents = Incident.objects.all()
+        
+        # Initialize correlation engine
+        correlation_engine = IncidentCorrelationEngine()
+        
+        # Convert to list of dictionaries
+        incident_data = [
+            {
+                'id': str(inc.id),
+                'title': inc.title,
+                'description': inc.description,
+                'free_text': inc.free_text,
+                'category': inc.category,
+                'severity': inc.severity,
+                'status': inc.status,
+                'created_at': inc.created_at
+            }
+            for inc in incidents
+        ]
+        
+        # Detect problem clusters
+        clusters = correlation_engine.detect_problem_clusters(incident_data)
+        
+        # Save correlations and patterns
+        correlation_count = 0
+        pattern_count = 0
+        
+        with transaction.atomic():
+            for cluster in clusters:
+                # Create pattern if significant
+                if len(cluster['incidents']) >= 3 and cluster['confidence'] > 0.7:
+                    pattern, created = IncidentPattern.objects.update_or_create(
+                        name=f"Pattern {cluster['problem_type']}",
+                        pattern_type=cluster['problem_type'],
+                        defaults={
+                            'description': f"Detected pattern of {cluster['problem_type']} affecting {len(cluster['incidents'])} incidents",
+                            'frequency': f"Occurred {len(cluster['incidents'])} times",
+                            'affected_services': list(set([
+                                inc.get('category', 'Unknown') for inc in cluster['incidents']
+                            ])),
+                            'common_keywords': list(set([
+                                keyword for corr in cluster['correlations']
+                                for keyword in corr.shared_keywords
+                            ]))[:10],
+                            'confidence_score': cluster['confidence'],
+                            'last_occurrence': max([
+                                inc['created_at'] for inc in cluster['incidents']
+                            ]),
+                            'model_version': correlation_engine.model_version
+                        }
+                    )
+                    
+                    if created:
+                        pattern_count += 1
+                    
+                    # Add incidents to pattern
+                    for incident_data in cluster['incidents']:
+                        try:
+                            incident = Incident.objects.get(id=incident_data['id'])
+                            pattern.incidents.add(incident)
+                        except Incident.DoesNotExist:
+                            continue
+                    
+                    pattern.incident_count = pattern.incidents.count()
+                    pattern.save()
+                
+                # Save correlations
+                for incident_a_data, incident_b_data, correlation_result in zip(
+                    cluster['incidents'][:-1], cluster['incidents'][1:], cluster['correlations']
+                ):
+                    if correlation_result.confidence_score > 0.5:
+                        try:
+                            incident_a = Incident.objects.get(id=incident_a_data['id'])
+                            incident_b = Incident.objects.get(id=incident_b_data['id'])
+                            
+                            IncidentCorrelation.objects.update_or_create(
+                                primary_incident=incident_a,
+                                related_incident=incident_b,
+                                defaults={
+                                    'correlation_type': correlation_result.correlation_type,
+                                    'confidence_score': correlation_result.confidence_score,
+                                    'correlation_strength': correlation_result.correlation_strength,
+                                    'shared_keywords': correlation_result.shared_keywords,
+                                    'time_difference': correlation_result.time_difference,
+                                    'similarity_score': correlation_result.similarity_score,
+                                    'is_problem_indicator': correlation_result.is_problem_indicator,
+                                    'problem_description': correlation_result.problem_description,
+                                    'model_version': correlation_engine.model_version
+                                }
+                            )
+                            correlation_count += 1
+                        except Incident.DoesNotExist:
+                            continue
+        
+        logger.info(f"Created {correlation_count} correlations and {pattern_count} patterns")
+        return f"Created {correlation_count} correlations and {pattern_count} patterns"
+    
+    except Exception as e:
+        logger.error(f"Error in correlation analysis: {str(e)}")
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        return f"Failed to correlate incidents: {str(e)}"
+
+
+@shared_task(bind=True, max_retries=3)
+def merge_duplicate_incidents(self, duplication_id):
+    """
+    Merge duplicate incidents
+    """
+    try:
+        duplication = DuplicationDetection.objects.get(id=duplication_id)
+        
+        if duplication.status != 'REVIEWED':
+            return f"Duplication {duplication_id} is not approved for merging"
+        
+        incident_a = duplication.incident_a
+        incident_b = duplication.incident_b
+        
+        with transaction.atomic():
+            # Merge incident data (keep the older incident as primary)
+            if incident_a.created_at <= incident_b.created_at:
+                primary_incident = incident_a
+                secondary_incident = incident_b
+            else:
+                primary_incident = incident_b
+                secondary_incident = incident_a
+            
+            # Update primary incident with information from secondary
+            if not primary_incident.description and secondary_incident.description:
+                primary_incident.description = secondary_incident.description
+            
+            if not primary_incident.free_text and secondary_incident.free_text:
+                primary_incident.free_text = secondary_incident.free_text
+            
+            if not primary_incident.business_impact and secondary_incident.business_impact:
+                primary_incident.business_impact = secondary_incident.business_impact
+            
+            # Update affected users (take maximum)
+            primary_incident.affected_users = max(
+                primary_incident.affected_users, secondary_incident.affected_users
+            )
+            
+            # Update severity (take higher severity)
+            severity_order = {'LOW': 1, 'MEDIUM': 2, 'HIGH': 3, 'CRITICAL': 4, 'EMERGENCY': 5}
+            if severity_order.get(secondary_incident.severity, 0) > severity_order.get(primary_incident.severity, 0):
+                primary_incident.severity = secondary_incident.severity
+            
+            primary_incident.save()
+            
+            # Mark secondary incident as duplicate
+            secondary_incident.is_duplicate = True
+            secondary_incident.original_incident = primary_incident
+            secondary_incident.duplicate_confidence = duplication.confidence_score
+            secondary_incident.status = 'CLOSED'
+            secondary_incident.save()
+            
+            # Update duplication status
+            duplication.status = 'MERGED'
+            duplication.save()
+        
+        logger.info(f"Successfully merged incidents {incident_a.id} and {incident_b.id}")
+        return f"Successfully merged incidents {incident_a.id} and {incident_b.id}"
+    
+    except DuplicationDetection.DoesNotExist:
+        logger.error(f"Duplication {duplication_id} not found")
+        return f"Duplication {duplication_id} not found"
+    
+    except Exception as e:
+        logger.error(f"Error merging incidents for duplication {duplication_id}: {str(e)}")
+        if self.request.retries < self.max_retries:
+            raise self.retry(countdown=60 * (2 ** self.request.retries))
+        return f"Failed to merge incidents: {str(e)}"

ETB-API/incident_intelligence/tests.py

@@ -0,0 +1,3 @@
+from django.test import TestCase
+
+# Create your tests here.

ETB-API/incident_intelligence/urls.py

@@ -0,0 +1,21 @@
+"""
+URL configuration for incident intelligence
+"""
+from django.urls import path, include
+from rest_framework.routers import DefaultRouter
+from .views.incident import (
+    IncidentViewSet, IncidentCorrelationViewSet, 
+    DuplicationDetectionViewSet, IncidentPatternViewSet
+)
+
+router = DefaultRouter()
+router.register(r'incidents', IncidentViewSet, basename='incident')
+router.register(r'correlations', IncidentCorrelationViewSet, basename='correlation')
+router.register(r'duplications', DuplicationDetectionViewSet, basename='duplication')
+router.register(r'patterns', IncidentPatternViewSet, basename='pattern')
+
+app_name = 'incidents'
+
+urlpatterns = [
+    path('', include(router.urls)),
+]

ETB-API/incident_intelligence/views.py

@@ -0,0 +1,3 @@
+from django.shortcuts import render
+
+# Create your views here.

ETB-API/incident_intelligence/views/__init__.py

@@ -0,0 +1 @@
+# Views for incident intelligence

ETB-API/incident_intelligence/views/incident.py

@@ -0,0 +1,457 @@
+"""
+Views for incident intelligence API endpoints
+"""
+from rest_framework import viewsets, status, filters
+from rest_framework.decorators import action
+from rest_framework.response import Response
+from rest_framework.permissions import IsAuthenticated
+from django_filters.rest_framework import DjangoFilterBackend
+from django.db.models import Q, Count, Avg, DurationField, F
+from django.db.models.functions import Extract
+from django.utils import timezone
+from datetime import timedelta
+import logging
+
+from ..models import (
+    Incident, IncidentClassification, SeveritySuggestion, IncidentCorrelation,
+    DuplicationDetection, IncidentPattern, AIProcessingLog
+)
+from ..serializers.incident import (
+    IncidentSerializer, IncidentCreateSerializer, IncidentUpdateSerializer,
+    IncidentAnalysisSerializer, IncidentSearchSerializer, IncidentStatsSerializer,
+    IncidentCorrelationSerializer, DuplicationDetectionSerializer, IncidentPatternSerializer
+)
+from ..ai.classification import IncidentClassifier, SeverityAnalyzer
+from ..ai.correlation import IncidentCorrelationEngine
+from ..ai.duplication import DuplicationDetector
+from ..security import IncidentAccessControl, log_incident_operation, get_user_accessible_incidents
+
+logger = logging.getLogger(__name__)
+
+
+class IncidentViewSet(viewsets.ModelViewSet):
+    """
+    ViewSet for managing incidents with AI intelligence
+    """
+    queryset = Incident.objects.all()
+    permission_classes = [IsAuthenticated]
+    filter_backends = [DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter]
+    filterset_fields = ['status', 'severity', 'category', 'assigned_to', 'reporter', 'ai_processed', 'is_duplicate']
+    search_fields = ['title', 'description', 'free_text']
+    ordering_fields = ['created_at', 'updated_at', 'severity', 'status']
+    ordering = ['-created_at']
+    
+    def get_serializer_class(self):
+        """Return appropriate serializer based on action"""
+        if self.action == 'create':
+            return IncidentCreateSerializer
+        elif self.action in ['update', 'partial_update']:
+            return IncidentUpdateSerializer
+        elif self.action == 'analysis':
+            return IncidentAnalysisSerializer
+        else:
+            return IncidentSerializer
+    
+    def get_queryset(self):
+        """Filter queryset based on user permissions and search parameters"""
+        # Start with user-accessible incidents
+        queryset = get_user_accessible_incidents(self.request.user)
+        
+        # Apply search filters
+        search_serializer = IncidentSearchSerializer(data=self.request.query_params)
+        if search_serializer.is_valid():
+            params = search_serializer.validated_data
+            
+            if params.get('query'):
+                queryset = queryset.filter(
+                    Q(title__icontains=params['query']) |
+                    Q(description__icontains=params['query']) |
+                    Q(free_text__icontains=params['query'])
+                )
+            
+            if params.get('category'):
+                queryset = queryset.filter(category=params['category'])
+            
+            if params.get('severity'):
+                queryset = queryset.filter(severity=params['severity'])
+            
+            if params.get('status'):
+                queryset = queryset.filter(status=params['status'])
+            
+            if params.get('assigned_to'):
+                queryset = queryset.filter(assigned_to_id=params['assigned_to'])
+            
+            if params.get('reporter'):
+                queryset = queryset.filter(reporter_id=params['reporter'])
+            
+            if params.get('date_from'):
+                queryset = queryset.filter(created_at__gte=params['date_from'])
+            
+            if params.get('date_to'):
+                queryset = queryset.filter(created_at__lte=params['date_to'])
+            
+            if params.get('has_ai_analysis') is not None:
+                queryset = queryset.filter(ai_processed=params['has_ai_analysis'])
+            
+            if params.get('is_duplicate') is not None:
+                queryset = queryset.filter(is_duplicate=params['is_duplicate'])
+        
+        return queryset
+    
+    def perform_create(self, serializer):
+        """Create incident and trigger AI processing"""
+        # Check create permissions
+        IncidentAccessControl.check_incident_access(self.request.user, None, 'create')
+        
+        incident = serializer.save()
+        
+        # Log the creation
+        log_incident_operation(self.request.user, incident, 'create')
+        
+        # Trigger AI processing
+        try:
+            from ..tasks import process_incident_ai
+            process_incident_ai.delay(incident.id)
+        except Exception as e:
+            logger.error(f"Failed to trigger AI processing for incident {incident.id}: {e}")
+    
+    @action(detail=True, methods=['post'])
+    def analyze(self, request, pk=None):
+        """Trigger AI analysis for a specific incident"""
+        incident = self.get_object()
+        
+        # Check analyze permissions
+        IncidentAccessControl.check_incident_access(request.user, incident, 'analyze')
+        
+        try:
+            from ..tasks import process_incident_ai
+            process_incident_ai.delay(incident.id)
+            
+            # Log the analysis trigger
+            log_incident_operation(request.user, incident, 'analyze')
+            
+            return Response({
+                'message': 'AI analysis triggered successfully',
+                'incident_id': str(incident.id)
+            }, status=status.HTTP_202_ACCEPTED)
+        
+        except Exception as e:
+            logger.error(f"Failed to trigger AI analysis for incident {incident.id}: {e}")
+            return Response({
+                'error': 'Failed to trigger AI analysis',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
+    
+    @action(detail=True, methods=['get'])
+    def analysis(self, request, pk=None):
+        """Get comprehensive AI analysis for an incident"""
+        incident = self.get_object()
+        serializer = self.get_serializer(incident)
+        return Response(serializer.data)
+    
+    @action(detail=True, methods=['get'])
+    def correlations(self, request, pk=None):
+        """Get correlations for a specific incident"""
+        incident = self.get_object()
+        
+        # Get correlations where this incident is primary
+        primary_correlations = incident.correlations_as_primary.all()
+        
+        # Get correlations where this incident is related
+        related_correlations = incident.correlations_as_related.all()
+        
+        # Combine and serialize
+        all_correlations = list(primary_correlations) + list(related_correlations)
+        serializer = IncidentCorrelationSerializer(all_correlations, many=True)
+        
+        return Response(serializer.data)
+    
+    @action(detail=True, methods=['get'])
+    def duplicates(self, request, pk=None):
+        """Get potential duplicates for a specific incident"""
+        incident = self.get_object()
+        
+        # Get duplications where this incident is incident_a
+        duplications_a = incident.duplication_as_a.all()
+        
+        # Get duplications where this incident is incident_b
+        duplications_b = incident.duplication_as_b.all()
+        
+        # Combine and serialize
+        all_duplications = list(duplications_a) + list(duplications_b)
+        serializer = DuplicationDetectionSerializer(all_duplications, many=True)
+        
+        return Response(serializer.data)
+    
+    @action(detail=True, methods=['get'])
+    def patterns(self, request, pk=None):
+        """Get patterns associated with a specific incident"""
+        incident = self.get_object()
+        patterns = incident.patterns.all()
+        serializer = IncidentPatternSerializer(patterns, many=True)
+        return Response(serializer.data)
+    
+    @action(detail=False, methods=['get'])
+    def stats(self, request):
+        """Get incident statistics"""
+        queryset = self.get_queryset()
+        
+        # Basic counts
+        total_incidents = queryset.count()
+        open_incidents = queryset.filter(status__in=['OPEN', 'IN_PROGRESS']).count()
+        resolved_incidents = queryset.filter(status__in=['RESOLVED', 'CLOSED']).count()
+        
+        # Severity counts
+        critical_incidents = queryset.filter(severity='CRITICAL').count()
+        high_incidents = queryset.filter(severity='HIGH').count()
+        medium_incidents = queryset.filter(severity='MEDIUM').count()
+        low_incidents = queryset.filter(severity='LOW').count()
+        
+        # Average resolution time
+        resolved_with_time = queryset.filter(
+            status__in=['RESOLVED', 'CLOSED'],
+            resolved_at__isnull=False
+        ).annotate(
+            resolution_time=F('resolved_at') - F('created_at')
+        )
+        
+        avg_resolution_time = resolved_with_time.aggregate(
+            avg_time=Avg('resolution_time')
+        )['avg_time']
+        
+        # Category distribution
+        incidents_by_category = dict(
+            queryset.values('category').annotate(count=Count('id')).values_list('category', 'count')
+        )
+        
+        # Severity distribution
+        incidents_by_severity = dict(
+            queryset.values('severity').annotate(count=Count('id')).values_list('severity', 'count')
+        )
+        
+        # Status distribution
+        incidents_by_status = dict(
+            queryset.values('status').annotate(count=Count('id')).values_list('status', 'count')
+        )
+        
+        # AI processing stats
+        ai_processed_count = queryset.filter(ai_processed=True).count()
+        duplicate_count = queryset.filter(is_duplicate=True).count()
+        correlation_count = IncidentCorrelation.objects.count()
+        pattern_count = IncidentPattern.objects.count()
+        
+        stats_data = {
+            'total_incidents': total_incidents,
+            'open_incidents': open_incidents,
+            'resolved_incidents': resolved_incidents,
+            'critical_incidents': critical_incidents,
+            'high_incidents': high_incidents,
+            'medium_incidents': medium_incidents,
+            'low_incidents': low_incidents,
+            'average_resolution_time': avg_resolution_time,
+            'incidents_by_category': incidents_by_category,
+            'incidents_by_severity': incidents_by_severity,
+            'incidents_by_status': incidents_by_status,
+            'ai_processed_count': ai_processed_count,
+            'duplicate_count': duplicate_count,
+            'correlation_count': correlation_count,
+            'pattern_count': pattern_count
+        }
+        
+        serializer = IncidentStatsSerializer(stats_data)
+        return Response(serializer.data)
+    
+    @action(detail=False, methods=['post'])
+    def batch_analyze(self, request):
+        """Trigger AI analysis for multiple incidents"""
+        incident_ids = request.data.get('incident_ids', [])
+        
+        if not incident_ids:
+            return Response({
+                'error': 'No incident IDs provided'
+            }, status=status.HTTP_400_BAD_REQUEST)
+        
+        try:
+            from ..tasks import batch_process_incidents_ai
+            batch_process_incidents_ai.delay(incident_ids)
+            
+            return Response({
+                'message': f'AI analysis triggered for {len(incident_ids)} incidents',
+                'incident_ids': incident_ids
+            }, status=status.HTTP_202_ACCEPTED)
+        
+        except Exception as e:
+            logger.error(f"Failed to trigger batch AI analysis: {e}")
+            return Response({
+                'error': 'Failed to trigger batch AI analysis',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
+    
+    @action(detail=False, methods=['post'])
+    def detect_duplicates(self, request):
+        """Detect duplicates across all incidents"""
+        try:
+            from ..tasks import detect_all_duplicates
+            detect_all_duplicates.delay()
+            
+            return Response({
+                'message': 'Duplicate detection process started'
+            }, status=status.HTTP_202_ACCEPTED)
+        
+        except Exception as e:
+            logger.error(f"Failed to trigger duplicate detection: {e}")
+            return Response({
+                'error': 'Failed to trigger duplicate detection',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
+    
+    @action(detail=False, methods=['post'])
+    def correlate_incidents(self, request):
+        """Run correlation analysis on all incidents"""
+        try:
+            from ..tasks import correlate_all_incidents
+            correlate_all_incidents.delay()
+            
+            return Response({
+                'message': 'Correlation analysis process started'
+            }, status=status.HTTP_202_ACCEPTED)
+        
+        except Exception as e:
+            logger.error(f"Failed to trigger correlation analysis: {e}")
+            return Response({
+                'error': 'Failed to trigger correlation analysis',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
+
+
+class IncidentCorrelationViewSet(viewsets.ReadOnlyModelViewSet):
+    """
+    ViewSet for viewing incident correlations
+    """
+    queryset = IncidentCorrelation.objects.all()
+    serializer_class = IncidentCorrelationSerializer
+    permission_classes = [IsAuthenticated]
+    filter_backends = [DjangoFilterBackend, filters.OrderingFilter]
+    filterset_fields = ['correlation_type', 'correlation_strength', 'is_problem_indicator']
+    ordering_fields = ['confidence_score', 'created_at']
+    ordering = ['-confidence_score']
+    
+    @action(detail=False, methods=['get'])
+    def problem_indicators(self, request):
+        """Get correlations that indicate larger problems"""
+        queryset = self.get_queryset().filter(is_problem_indicator=True)
+        serializer = self.get_serializer(queryset, many=True)
+        return Response(serializer.data)
+
+
+class DuplicationDetectionViewSet(viewsets.ReadOnlyModelViewSet):
+    """
+    ViewSet for viewing duplication detection results
+    """
+    queryset = DuplicationDetection.objects.all()
+    serializer_class = DuplicationDetectionSerializer
+    permission_classes = [IsAuthenticated]
+    filter_backends = [DjangoFilterBackend, filters.OrderingFilter]
+    filterset_fields = ['duplication_type', 'status', 'recommended_action']
+    ordering_fields = ['confidence_score', 'similarity_score', 'created_at']
+    ordering = ['-confidence_score']
+    
+    @action(detail=True, methods=['post'])
+    def approve_merge(self, request, pk=None):
+        """Approve merging of duplicate incidents"""
+        duplication = self.get_object()
+        
+        if duplication.status != 'DETECTED':
+            return Response({
+                'error': 'Only detected duplications can be approved for merging'
+            }, status=status.HTTP_400_BAD_REQUEST)
+        
+        try:
+            # Mark as reviewed and approved
+            duplication.status = 'REVIEWED'
+            duplication.reviewed_by = request.user
+            duplication.reviewed_at = timezone.now()
+            duplication.save()
+            
+            # Trigger merge process
+            from ..tasks import merge_duplicate_incidents
+            merge_duplicate_incidents.delay(duplication.id)
+            
+            return Response({
+                'message': 'Merge approved and queued for processing'
+            }, status=status.HTTP_200_OK)
+        
+        except Exception as e:
+            logger.error(f"Failed to approve merge for duplication {duplication.id}: {e}")
+            return Response({
+                'error': 'Failed to approve merge',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
+    
+    @action(detail=True, methods=['post'])
+    def reject_merge(self, request, pk=None):
+        """Reject merging of duplicate incidents"""
+        duplication = self.get_object()
+        
+        if duplication.status != 'DETECTED':
+            return Response({
+                'error': 'Only detected duplications can be rejected'
+            }, status=status.HTTP_400_BAD_REQUEST)
+        
+        try:
+            duplication.status = 'REJECTED'
+            duplication.reviewed_by = request.user
+            duplication.reviewed_at = timezone.now()
+            duplication.save()
+            
+            return Response({
+                'message': 'Merge rejected'
+            }, status=status.HTTP_200_OK)
+        
+        except Exception as e:
+            logger.error(f"Failed to reject merge for duplication {duplication.id}: {e}")
+            return Response({
+                'error': 'Failed to reject merge',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
+
+
+class IncidentPatternViewSet(viewsets.ReadOnlyModelViewSet):
+    """
+    ViewSet for viewing incident patterns
+    """
+    queryset = IncidentPattern.objects.all()
+    serializer_class = IncidentPatternSerializer
+    permission_classes = [IsAuthenticated]
+    filter_backends = [DjangoFilterBackend, filters.OrderingFilter]
+    filterset_fields = ['pattern_type', 'is_active', 'is_resolved']
+    ordering_fields = ['confidence_score', 'incident_count', 'created_at']
+    ordering = ['-confidence_score']
+    
+    @action(detail=False, methods=['get'])
+    def active_patterns(self, request):
+        """Get active patterns that need attention"""
+        queryset = self.get_queryset().filter(is_active=True, is_resolved=False)
+        serializer = self.get_serializer(queryset, many=True)
+        return Response(serializer.data)
+    
+    @action(detail=True, methods=['post'])
+    def resolve_pattern(self, request, pk=None):
+        """Mark a pattern as resolved"""
+        pattern = self.get_object()
+        
+        try:
+            pattern.is_resolved = True
+            pattern.is_active = False
+            pattern.save()
+            
+            return Response({
+                'message': 'Pattern marked as resolved'
+            }, status=status.HTTP_200_OK)
+        
+        except Exception as e:
+            logger.error(f"Failed to resolve pattern {pattern.id}: {e}")
+            return Response({
+                'error': 'Failed to resolve pattern',
+                'details': str(e)
+            }, status=status.HTTP_500_INTERNAL_SERVER_ERROR)