ETB/ETB-API/ENTERPRISE_DEPLOYMENT_GUIDE.md

# ETB-API Enterprise Deployment Guide

## 🚀 **Enterprise-Grade Deployment for Production**

This comprehensive guide provides step-by-step instructions for deploying the ETB-API platform in an enterprise environment with high availability, security, and scalability.

## 📋 **Table of Contents**

1. [Prerequisites](#prerequisites)
2. [Infrastructure Setup](#infrastructure-setup)
3. [Database Configuration](#database-configuration)
4. [Application Deployment](#application-deployment)
5. [Security Configuration](#security-configuration)
6. [Monitoring & Observability](#monitoring--observability)
7. [Backup & Recovery](#backup--recovery)
8. [High Availability](#high-availability)
9. [Performance Optimization](#performance-optimization)
10. [Maintenance & Operations](#maintenance--operations)

## 🔧 **Prerequisites**

### System Requirements

- **Operating System**: Ubuntu 20.04 LTS or CentOS 8+
- **CPU**: 8+ cores (16+ recommended for production)
- **RAM**: 32GB+ (64GB+ recommended for production)
- **Storage**: 500GB+ SSD (1TB+ recommended for production)
- **Network**: 1Gbps+ bandwidth

### Software Requirements

- **Python**: 3.9+
- **PostgreSQL**: 13+
- **Redis**: 6.2+
- **Nginx**: 1.18+
- **Docker**: 20.10+ (optional)
- **Kubernetes**: 1.21+ (optional)

### Dependencies

```bash
# Install system packages
sudo apt-get update
sudo apt-get install -y python3.9 python3.9-dev python3-pip
sudo apt-get install -y postgresql-13 postgresql-client-13
sudo apt-get install -y redis-server nginx
sudo apt-get install -y git curl wget unzip

# Install Python dependencies
pip3 install -r requirements.txt
```

## 🏗️ **Infrastructure Setup**

### 1. Database Cluster Setup

#### PostgreSQL Primary-Replica Configuration

```bash
# Primary Database Server
sudo -u postgres psql
CREATE DATABASE etb_incident_management;
CREATE USER etb_user WITH PASSWORD 'secure_password';
GRANT ALL PRIVILEGES ON DATABASE etb_incident_management TO etb_user;

# Configure replication
sudo nano /etc/postgresql/13/main/postgresql.conf
```

```conf
# postgresql.conf
listen_addresses = '*'
wal_level = replica
max_wal_senders = 3
max_replication_slots = 3
hot_standby = on
```

```bash
# Configure authentication
sudo nano /etc/postgresql/13/main/pg_hba.conf
```

```conf
# pg_hba.conf
host replication replicator 10.0.0.0/8 md5
host all all 10.0.0.0/8 md5
```

#### Redis Cluster Setup

```bash
# Redis Master
sudo nano /etc/redis/redis.conf
```

```conf
# redis.conf
bind 0.0.0.0
port 6379
requirepass secure_redis_password
maxmemory 2gb
maxmemory-policy allkeys-lru
save 900 1
save 300 10
save 60 10000
```

### 2. Load Balancer Configuration

#### Nginx Load Balancer

```bash
sudo nano /etc/nginx/sites-available/etb-api
```

```nginx
upstream etb_api_backend {
    least_conn;
    server 10.0.1.10:8000 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8000 max_fails=3 fail_timeout=30s;
}

upstream etb_api_websocket {
    ip_hash;
    server 10.0.1.10:8001;
    server 10.0.1.11:8001;
    server 10.0.1.12:8001;
}

server {
    listen 80;
    server_name api.yourcompany.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.yourcompany.com;
    
    # SSL Configuration
    ssl_certificate /etc/ssl/certs/etb-api.crt;
    ssl_certificate_key /etc/ssl/private/etb-api.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # Security Headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options DENY;
    add_header X-XSS-Protection "1; mode=block";
    add_header Content-Security-Policy "default-src 'self'";
    
    # Rate Limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    limit_req zone=api burst=20 nodelay;
    
    # API Routes
    location /api/ {
        proxy_pass http://etb_api_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 30s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;
    }
    
    # WebSocket Routes
    location /ws/ {
        proxy_pass http://etb_api_websocket;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    
    # Static Files
    location /static/ {
        alias /var/www/etb-api/static/;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
    
    # Media Files
    location /media/ {
        alias /var/www/etb-api/media/;
        expires 1y;
        add_header Cache-Control "public";
    }
    
    # Health Checks
    location /health/ {
        proxy_pass http://etb_api_backend;
        access_log off;
    }
}
```

## 🗄️ **Database Configuration**

### 1. Database Optimization

```sql
-- PostgreSQL Performance Tuning
ALTER SYSTEM SET shared_buffers = '4GB';
ALTER SYSTEM SET effective_cache_size = '12GB';
ALTER SYSTEM SET maintenance_work_mem = '1GB';
ALTER SYSTEM SET checkpoint_completion_target = 0.9;
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET default_statistics_target = 100;
ALTER SYSTEM SET random_page_cost = 1.1;
ALTER SYSTEM SET effective_io_concurrency = 200;

-- Restart PostgreSQL
sudo systemctl restart postgresql
```

### 2. Database Indexes

```sql
-- Create performance indexes
CREATE INDEX CONCURRENTLY idx_incident_status_priority ON incident_intelligence_incident(status, priority);
CREATE INDEX CONCURRENTLY idx_incident_created_at ON incident_intelligence_incident(created_at);
CREATE INDEX CONCURRENTLY idx_sla_instance_status ON sla_oncall_slainstance(status);
CREATE INDEX CONCURRENTLY idx_security_event_timestamp ON security_securityevent(timestamp);
CREATE INDEX CONCURRENTLY idx_monitoring_metric_timestamp ON monitoring_metric(timestamp);

-- Create partial indexes for active records
CREATE INDEX CONCURRENTLY idx_incident_active ON incident_intelligence_incident(id) WHERE status = 'active';
CREATE INDEX CONCURRENTLY idx_sla_active ON sla_oncall_slainstance(id) WHERE status = 'active';
```

## 🚀 **Application Deployment**

### 1. Environment Configuration

```bash
# Create environment file
sudo nano /etc/etb-api/.env
```

```env
# Database Configuration
DB_NAME=etb_incident_management
DB_USER=etb_user
DB_PASSWORD=secure_password
DB_HOST=10.0.1.5
DB_PORT=5432

# Redis Configuration
REDIS_URL=redis://:secure_redis_password@10.0.1.6:6379/0
CELERY_BROKER_URL=redis://:secure_redis_password@10.0.1.6:6379/0
CELERY_RESULT_BACKEND=redis://:secure_redis_password@10.0.1.6:6379/0

# Security
SECRET_KEY=your-super-secret-key-here
DEBUG=False
ALLOWED_HOSTS=api.yourcompany.com,10.0.1.10,10.0.1.11,10.0.1.12

# Email Configuration
EMAIL_HOST=smtp.yourcompany.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_HOST_USER=noreply@yourcompany.com
EMAIL_HOST_PASSWORD=email_password
DEFAULT_FROM_EMAIL=noreply@yourcompany.com

# Monitoring
PROMETHEUS_ENABLED=True
GRAFANA_ENABLED=True
ELASTICSEARCH_URL=http://10.0.1.8:9200

# Backup
BACKUP_ENABLED=True
BACKUP_RETENTION_DAYS=30
AWS_S3_BACKUP_BUCKET=etb-api-backups
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_REGION=us-east-1

# Security
SIEM_WEBHOOK_URL=https://siem.yourcompany.com/webhook
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
ALERT_WEBHOOK_URL=https://alerts.yourcompany.com/webhook
```

### 2. Application Deployment Script

```bash
#!/bin/bash
# deploy.sh - Enterprise deployment script

set -e

# Configuration
APP_NAME="etb-api"
APP_USER="etb"
APP_DIR="/var/www/etb-api"
VENV_DIR="/var/www/etb-api/venv"
LOG_DIR="/var/log/etb-api"
BACKUP_DIR="/backups/etb-api"

# Create application user
sudo useradd -r -s /bin/false $APP_USER || true

# Create directories
sudo mkdir -p $APP_DIR $LOG_DIR $BACKUP_DIR
sudo chown -R $APP_USER:$APP_USER $APP_DIR $LOG_DIR $BACKUP_DIR

# Clone repository
cd $APP_DIR
sudo -u $APP_USER git clone https://github.com/yourcompany/etb-api.git .

# Create virtual environment
sudo -u $APP_USER python3 -m venv $VENV_DIR
sudo -u $APP_USER $VENV_DIR/bin/pip install --upgrade pip
sudo -u $APP_USER $VENV_DIR/bin/pip install -r requirements.txt

# Set up environment
sudo -u $APP_USER cp .env.example .env
sudo -u $APP_USER nano .env  # Configure environment variables

# Run migrations
sudo -u $APP_USER $VENV_DIR/bin/python manage.py migrate

# Collect static files
sudo -u $APP_USER $VENV_DIR/bin/python manage.py collectstatic --noinput

# Create superuser
sudo -u $APP_USER $VENV_DIR/bin/python manage.py createsuperuser

# Set up systemd services
sudo cp deployment/systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable etb-api etb-celery etb-celery-beat
sudo systemctl start etb-api etb-celery etb-celery-beat

# Set up log rotation
sudo cp deployment/logrotate/etb-api /etc/logrotate.d/
sudo chmod 644 /etc/logrotate.d/etb-api

# Set up monitoring
sudo -u $APP_USER $VENV_DIR/bin/python manage.py setup_monitoring

echo "Deployment completed successfully!"
```

### 3. Systemd Services

#### ETB-API Service

```ini
# /etc/systemd/system/etb-api.service
[Unit]
Description=ETB-API Django Application
After=network.target postgresql.service redis.service
Requires=postgresql.service redis.service

[Service]
Type=exec
User=etb
Group=etb
WorkingDirectory=/var/www/etb-api
Environment=PATH=/var/www/etb-api/venv/bin
EnvironmentFile=/etc/etb-api/.env
ExecStart=/var/www/etb-api/venv/bin/gunicorn --bind 0.0.0.0:8000 --workers 4 --worker-class gevent --worker-connections 1000 --max-requests 1000 --max-requests-jitter 100 --timeout 30 --keep-alive 2 --preload core.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=etb-api

[Install]
WantedBy=multi-user.target
```

#### Celery Worker Service

```ini
# /etc/systemd/system/etb-celery.service
[Unit]
Description=ETB-API Celery Worker
After=network.target redis.service
Requires=redis.service

[Service]
Type=exec
User=etb
Group=etb
WorkingDirectory=/var/www/etb-api
Environment=PATH=/var/www/etb-api/venv/bin
EnvironmentFile=/etc/etb-api/.env
ExecStart=/var/www/etb-api/venv/bin/celery -A core worker -l info --concurrency=4 --max-tasks-per-child=1000
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=etb-celery

[Install]
WantedBy=multi-user.target
```

#### Celery Beat Service

```ini
# /etc/systemd/system/etb-celery-beat.service
[Unit]
Description=ETB-API Celery Beat Scheduler
After=network.target redis.service
Requires=redis.service

[Service]
Type=exec
User=etb
Group=etb
WorkingDirectory=/var/www/etb-api
Environment=PATH=/var/www/etb-api/venv/bin
EnvironmentFile=/etc/etb-api/.env
ExecStart=/var/www/etb-api/venv/bin/celery -A core beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=etb-celery-beat

[Install]
WantedBy=multi-user.target
```

## 🔒 **Security Configuration**

### 1. Firewall Configuration

```bash
# UFW Firewall Rules
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.0.0.0/8 to any port 5432  # PostgreSQL
sudo ufw allow from 10.0.0.0/8 to any port 6379  # Redis
sudo ufw enable
```

### 2. SSL/TLS Configuration

```bash
# Generate SSL certificate
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -keyout /etc/ssl/private/etb-api.key \
    -out /etc/ssl/certs/etb-api.crt \
    -subj "/C=US/ST=State/L=City/O=Organization/CN=api.yourcompany.com"

# Set permissions
sudo chmod 600 /etc/ssl/private/etb-api.key
sudo chmod 644 /etc/ssl/certs/etb-api.crt
```

### 3. Security Hardening

```bash
# Disable unnecessary services
sudo systemctl disable apache2
sudo systemctl disable mysql

# Configure fail2ban
sudo apt-get install fail2ban
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

# Add custom jail for ETB-API
sudo nano /etc/fail2ban/jail.d/etb-api.conf
```

```ini
[etb-api]
enabled = true
port = 80,443
filter = etb-api
logpath = /var/log/etb-api/application.log
maxretry = 5
bantime = 3600
findtime = 600
```

## 📊 **Monitoring & Observability**

### 1. Prometheus Configuration

```yaml
# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "etb-api-rules.yml"

scrape_configs:
  - job_name: 'etb-api'
    static_configs:
      - targets: ['10.0.1.10:8000', '10.0.1.11:8000', '10.0.1.12:8000']
    metrics_path: '/api/monitoring/metrics/'
    scrape_interval: 30s

  - job_name: 'postgresql'
    static_configs:
      - targets: ['10.0.1.5:9187']

  - job_name: 'redis'
    static_configs:
      - targets: ['10.0.1.6:9121']

  - job_name: 'nginx'
    static_configs:
      - targets: ['10.0.1.7:9113']
```

### 2. Grafana Dashboard

```json
{
  "dashboard": {
    "title": "ETB-API Enterprise Dashboard",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{endpoint}}"
          }
        ]
      },
      {
        "title": "Response Time",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "95th percentile"
          }
        ]
      },
      {
        "title": "System Resources",
        "type": "graph",
        "targets": [
          {
            "expr": "system_cpu_usage_percent",
            "legendFormat": "CPU Usage"
          },
          {
            "expr": "system_memory_usage_percent",
            "legendFormat": "Memory Usage"
          }
        ]
      }
    ]
  }
}
```

### 3. Log Aggregation

```yaml
# /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/etb-api/*.log
  fields:
    service: etb-api
  fields_under_root: true

output.elasticsearch:
  hosts: ["10.0.1.8:9200"]
  index: "etb-api-%{+yyyy.MM.dd}"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
```

## 💾 **Backup & Recovery**

### 1. Automated Backup Script

```bash
#!/bin/bash
# backup.sh - Automated backup script

set -e

BACKUP_DIR="/backups/etb-api"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_NAME="etb-api-backup-$DATE"

# Create backup directory
mkdir -p $BACKUP_DIR/$BACKUP_NAME

# Database backup
pg_dump -h 10.0.1.5 -U etb_user etb_incident_management > $BACKUP_DIR/$BACKUP_NAME/database.sql

# Application backup
tar -czf $BACKUP_DIR/$BACKUP_NAME/application.tar.gz -C /var/www etb-api

# Configuration backup
tar -czf $BACKUP_DIR/$BACKUP_NAME/config.tar.gz /etc/etb-api /etc/nginx/sites-available/etb-api

# Create backup manifest
cat > $BACKUP_DIR/$BACKUP_NAME/manifest.json << EOF
{
  "backup_name": "$BACKUP_NAME",
  "created_at": "$(date -Iseconds)",
  "components": {
    "database": "database.sql",
    "application": "application.tar.gz",
    "configuration": "config.tar.gz"
  },
  "size": "$(du -sh $BACKUP_DIR/$BACKUP_NAME | cut -f1)"
}
EOF

# Compress backup
tar -czf $BACKUP_DIR/$BACKUP_NAME.tar.gz -C $BACKUP_DIR $BACKUP_NAME
rm -rf $BACKUP_DIR/$BACKUP_NAME

# Upload to S3
aws s3 cp $BACKUP_DIR/$BACKUP_NAME.tar.gz s3://etb-api-backups/

# Cleanup old backups
find $BACKUP_DIR -name "*.tar.gz" -mtime +30 -delete

echo "Backup completed: $BACKUP_NAME"
```

### 2. Recovery Script

```bash
#!/bin/bash
# restore.sh - Recovery script

set -e

BACKUP_NAME=$1
BACKUP_DIR="/backups/etb-api"

if [ -z "$BACKUP_NAME" ]; then
    echo "Usage: $0 <backup_name>"
    exit 1
fi

# Download from S3
aws s3 cp s3://etb-api-backups/$BACKUP_NAME.tar.gz $BACKUP_DIR/

# Extract backup
tar -xzf $BACKUP_DIR/$BACKUP_NAME.tar.gz -C $BACKUP_DIR

# Stop services
sudo systemctl stop etb-api etb-celery etb-celery-beat

# Restore database
psql -h 10.0.1.5 -U etb_user etb_incident_management < $BACKUP_DIR/$BACKUP_NAME/database.sql

# Restore application
tar -xzf $BACKUP_DIR/$BACKUP_NAME/application.tar.gz -C /var/www

# Restore configuration
tar -xzf $BACKUP_DIR/$BACKUP_NAME/config.tar.gz -C /

# Start services
sudo systemctl start etb-api etb-celery etb-celery-beat

echo "Recovery completed: $BACKUP_NAME"
```

## 🔄 **High Availability**

### 1. Load Balancer Health Checks

```nginx
# Health check configuration
upstream etb_api_backend {
    least_conn;
    server 10.0.1.10:8000 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8000 max_fails=3 fail_timeout=30s;
}

# Health check endpoint
location /health/ {
    proxy_pass http://etb_api_backend;
    proxy_connect_timeout 5s;
    proxy_send_timeout 5s;
    proxy_read_timeout 5s;
    access_log off;
}
```

### 2. Database Failover

```bash
# PostgreSQL failover script
#!/bin/bash
# failover.sh - Database failover script

PRIMARY_HOST="10.0.1.5"
STANDBY_HOST="10.0.1.6"

# Check primary health
if ! pg_isready -h $PRIMARY_HOST -p 5432; then
    echo "Primary database is down, initiating failover..."
    
    # Promote standby
    ssh $STANDBY_HOST "sudo -u postgres pg_ctl promote -D /var/lib/postgresql/13/main"
    
    # Update application configuration
    sed -i "s/DB_HOST=$PRIMARY_HOST/DB_HOST=$STANDBY_HOST/" /etc/etb-api/.env
    
    # Restart application
    sudo systemctl restart etb-api
    
    echo "Failover completed to $STANDBY_HOST"
fi
```

## ⚡ **Performance Optimization**

### 1. Application Optimization

```python
# settings.py optimizations
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://10.0.1.6:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'CONNECTION_POOL_KWARGS': {
                'max_connections': 50,
                'retry_on_timeout': True,
            },
            'COMPRESSOR': 'django_redis.compressors.zlib.ZlibCompressor',
        }
    }
}

# Database connection pooling
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'etb_incident_management',
        'USER': 'etb_user',
        'PASSWORD': 'secure_password',
        'HOST': '10.0.1.5',
        'PORT': '5432',
        'CONN_MAX_AGE': 600,
        'CONN_HEALTH_CHECKS': True,
        'OPTIONS': {
            'MAX_CONNS': 20,
            'MIN_CONNS': 5,
        }
    }
}
```

### 2. Nginx Optimization

```nginx
# nginx.conf optimizations
worker_processes auto;
worker_cpu_affinity auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    
    # Buffer sizes
    client_body_buffer_size 128k;
    client_max_body_size 10m;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 4k;
    
    # Timeouts
    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    send_timeout 10;
}
```

## 🔧 **Maintenance & Operations**

### 1. Monitoring Scripts

```bash
#!/bin/bash
# health_check.sh - Comprehensive health check

# Check application health
curl -f http://localhost:8000/health/ || exit 1

# Check database connectivity
pg_isready -h 10.0.1.5 -p 5432 || exit 1

# Check Redis connectivity
redis-cli -h 10.0.1.6 -p 6379 ping || exit 1

# Check disk space
df -h | awk '$5 > 90 {print $0; exit 1}'

# Check memory usage
free | awk 'NR==2{printf "%.2f%%\n", $3*100/$2}' | awk '{if($1 > 90) exit 1}'

echo "All health checks passed"
```

### 2. Log Rotation

```bash
# /etc/logrotate.d/etb-api
/var/log/etb-api/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 etb etb
    postrotate
        systemctl reload etb-api
    endscript
}
```

### 3. Update Script

```bash
#!/bin/bash
# update.sh - Application update script

set -e

# Backup current version
./backup.sh

# Pull latest changes
cd /var/www/etb-api
sudo -u etb git pull origin main

# Update dependencies
sudo -u etb /var/www/etb-api/venv/bin/pip install -r requirements.txt

# Run migrations
sudo -u etb /var/www/etb-api/venv/bin/python manage.py migrate

# Collect static files
sudo -u etb /var/www/etb-api/venv/bin/python manage.py collectstatic --noinput

# Restart services
sudo systemctl restart etb-api etb-celery etb-celery-beat

echo "Update completed successfully"
```

## 📈 **Scaling Guidelines**

### 1. Horizontal Scaling

- **Application Servers**: Add more instances behind load balancer
- **Database**: Implement read replicas for read-heavy workloads
- **Cache**: Use Redis Cluster for distributed caching
- **Storage**: Implement distributed file storage (S3, Ceph)

### 2. Vertical Scaling

- **CPU**: Increase cores for compute-intensive operations
- **Memory**: Add RAM for caching and in-memory operations
- **Storage**: Use SSD for better I/O performance
- **Network**: Upgrade bandwidth for high-traffic scenarios

## 🚨 **Troubleshooting**

### Common Issues

1. **High Memory Usage**
   ```bash
   # Check memory usage
   free -h
   ps aux --sort=-%mem | head
   
   # Restart services if needed
   sudo systemctl restart etb-api
   ```

2. **Database Connection Issues**
   ```bash
   # Check database status
   sudo systemctl status postgresql
   pg_isready -h 10.0.1.5 -p 5432
   
   # Check connection pool
   psql -h 10.0.1.5 -U etb_user -c "SELECT * FROM pg_stat_activity;"
   ```

3. **Cache Issues**
   ```bash
   # Check Redis status
   sudo systemctl status redis
   redis-cli -h 10.0.1.6 -p 6379 ping
   
   # Clear cache if needed
   redis-cli -h 10.0.1.6 -p 6379 FLUSHALL
   ```

## 📞 **Support & Maintenance**

### Regular Tasks

- **Daily**: Monitor system health and alerts
- **Weekly**: Review performance metrics and logs
- **Monthly**: Update dependencies and security patches
- **Quarterly**: Review and optimize performance

### Emergency Procedures

1. **Service Outage**: Check health endpoints and restart services
2. **Database Issues**: Check connectivity and failover if needed
3. **Security Incident**: Review logs and implement containment
4. **Performance Degradation**: Analyze metrics and scale resources

This comprehensive deployment guide provides enterprise-grade setup for the ETB-API platform. Adjust configurations based on your specific requirements and infrastructure.