Compare commits
9 Commits
78bcfce43a
...
e5da6cfdad
| Author | SHA1 | Date | |
|---|---|---|---|
 Mateusz Suski
|
e5da6cfdad | ||
|
Mateusz Suski
|
a67f7e33e0 | ||
|
Mateusz Suski
|
ff6a81a3d5 | ||
|
Mateusz Suski
|
796712ef2e | ||
|
Mateusz Suski
|
21d8d7e666 | ||
|
Mateusz Suski
|
ad12b8581d | ||
|
Mateusz Suski
|
8783892241 | ||
|
Mateusz Suski
|
b0537b4bff | ||
|
Mateusz Suski
|
7757020014 |
@@ -0,0 +1,30 @@
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: ubuntu
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install deps
|
||||
run: |
|
||||
apt-get update
|
||||
apt-get install -y python3 python3-pip ansible docker.io
|
||||
|
||||
- name: Python syntax check
|
||||
run: |
|
||||
find migration-validation-framework -name "*.py" -exec python3 -m py_compile {} \;
|
||||
|
||||
- name: Ansible syntax check
|
||||
run: |
|
||||
ansible-playbook -i enterprise-infra-simulator/inventory/hosts.ini \
|
||||
--syntax-check enterprise-infra-simulator/playbooks/*.yml
|
||||
|
||||
- name: Docker compose validation
|
||||
run: |
|
||||
docker compose -f observability-stack/docker-compose.yml config
|
||||
@@ -0,0 +1,5 @@
|
||||
**pycache**/
|
||||
*.pyc
|
||||
*.log
|
||||
.env
|
||||
.DS_Store
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
# AI Context File - Portfolio Expansion Guide
|
||||
|
||||
## Portfolio Overview
|
||||
This is a comprehensive enterprise Linux infrastructure portfolio demonstrating advanced engineering skills across three main domains:
|
||||
1. **Enterprise Infrastructure Simulator** - Ansible-based container infrastructure automation
|
||||
2. **Migration Validation Framework** - Python CLI for system migration validation
|
||||
3. **Observability Stack** - ELK + Grafana monitoring platform
|
||||
|
||||
## Current Architecture
|
||||
|
||||
### Enterprise Infrastructure Simulator
|
||||
**Technology Stack**: Ansible, Docker Compose, Bash
|
||||
**Key Components**:
|
||||
- Container-based Linux node simulation
|
||||
- Ansible playbooks for provisioning, patching, hardening, decommissioning
|
||||
- Operational scripts for scaling and failure simulation
|
||||
- Multi-group inventory with realistic enterprise structure
|
||||
|
||||
**Expansion Opportunities**:
|
||||
- Add Kubernetes support for container orchestration
|
||||
- Implement multi-cloud deployment (AWS, Azure, GCP)
|
||||
- Add Terraform integration for infrastructure provisioning
|
||||
- Create custom Ansible modules for enterprise-specific tasks
|
||||
- Implement backup and disaster recovery procedures
|
||||
|
||||
### Migration Validation Framework
|
||||
**Technology Stack**: Python 3.8+, HTML/CSS/JavaScript
|
||||
**Key Components**:
|
||||
- CLI application with snapshot/compare/report commands
|
||||
- Modular collectors (mounts, services, disk usage)
|
||||
- Intelligent comparison engine with drift detection
|
||||
- Interactive HTML reporting with Bootstrap styling
|
||||
|
||||
**Expansion Opportunities**:
|
||||
- Add database migration validation (MySQL, PostgreSQL, MongoDB)
|
||||
- Implement cloud migration support (AWS, Azure)
|
||||
- Add performance benchmarking capabilities
|
||||
- Create REST API for integration with CI/CD pipelines
|
||||
- Implement machine learning for change prediction
|
||||
- Add compliance validation (PCI-DSS, HIPAA, GDPR)
|
||||
|
||||
### Observability Stack
|
||||
**Technology Stack**: ELK Stack, Grafana, Docker Compose
|
||||
**Key Components**:
|
||||
- Elasticsearch, Logstash, Kibana, Grafana
|
||||
- Filebeat for log collection
|
||||
- Comprehensive alerting rules
|
||||
- Incident simulation framework
|
||||
- Sample logs for testing
|
||||
|
||||
**Expansion Opportunities**:
|
||||
- Add Prometheus and Grafana for metrics collection
|
||||
- Implement distributed tracing (Jaeger, Zipkin)
|
||||
- Add anomaly detection with machine learning
|
||||
- Create custom dashboards for each project
|
||||
- Implement log aggregation from cloud services
|
||||
- Add synthetic monitoring and uptime checks
|
||||
|
||||
## Technical Standards & Conventions
|
||||
|
||||
### Code Quality
|
||||
- Python: Type hints, comprehensive error handling, logging
|
||||
- Ansible: Modern syntax (true/false booleans), modular structure
|
||||
- Docker: Multi-stage builds, security best practices
|
||||
- Documentation: Comprehensive READMEs, inline comments
|
||||
|
||||
### Naming Conventions
|
||||
- Projects: kebab-case (enterprise-infra-simulator)
|
||||
- Files: snake_case for Python, kebab-case for YAML
|
||||
- Variables: snake_case, descriptive names
|
||||
- Services: realistic enterprise naming (no "foo", "bar")
|
||||
|
||||
### Security Standards
|
||||
- CIS benchmarks for Linux hardening
|
||||
- Secure defaults in all configurations
|
||||
- Input validation and sanitization
|
||||
- Least privilege principles
|
||||
|
||||
## Future Development Roadmap
|
||||
|
||||
### Phase 1: Infrastructure Enhancement
|
||||
- [ ] Add Kubernetes manifests for container orchestration
|
||||
- [ ] Implement Helm charts for service deployment
|
||||
- [ ] Add Terraform modules for cloud infrastructure
|
||||
- [ ] Create Ansible Tower/AWX integration
|
||||
|
||||
### Phase 2: Application Expansion
|
||||
- [ ] Extend migration framework with database support
|
||||
- [ ] Add REST API to validation framework
|
||||
- [ ] Implement OAuth2 authentication
|
||||
- [ ] Create web-based dashboard for validation results
|
||||
|
||||
### Phase 3: Monitoring & Observability
|
||||
- [ ] Add Prometheus metrics collection
|
||||
- [ ] Implement distributed tracing
|
||||
- [ ] Create ML-based anomaly detection
|
||||
- [ ] Add synthetic monitoring capabilities
|
||||
|
||||
### Phase 4: Enterprise Integration
|
||||
- [ ] Jira/ServiceNow integration for incident management
|
||||
- [ ] Slack/Microsoft Teams notifications
|
||||
- [ ] LDAP/Active Directory authentication
|
||||
- [ ] Audit logging and compliance reporting
|
||||
|
||||
### Phase 5: Cloud & Multi-Platform
|
||||
- [ ] AWS ECS/EKS deployment support
|
||||
- [ ] Azure AKS deployment support
|
||||
- [ ] GCP GKE deployment support
|
||||
- [ ] Multi-cloud failover capabilities
|
||||
|
||||
## Development Guidelines
|
||||
|
||||
### Code Style
|
||||
- Follow PEP 8 for Python code
|
||||
- Use ansible-lint for playbook validation
|
||||
- Implement comprehensive error handling
|
||||
- Add logging at appropriate levels
|
||||
- Write unit tests for critical functions
|
||||
|
||||
### Documentation Standards
|
||||
- Update README.md for each new feature
|
||||
- Maintain CHANGELOG.md with detailed entries
|
||||
- Document API endpoints and CLI commands
|
||||
- Include setup and troubleshooting guides
|
||||
- Add architecture diagrams for complex features
|
||||
|
||||
### Testing Strategy
|
||||
- Unit tests for Python modules
|
||||
- Integration tests for Ansible playbooks
|
||||
- End-to-end tests for complete workflows
|
||||
- Performance testing for critical paths
|
||||
- Security testing and vulnerability scanning
|
||||
|
||||
## Project Dependencies & Requirements
|
||||
|
||||
### System Requirements
|
||||
- Docker Engine 20.10+
|
||||
- Docker Compose 2.0+
|
||||
- Python 3.8+
|
||||
- Ansible 2.10+
|
||||
- Git 2.25+
|
||||
|
||||
### External Services
|
||||
- Gitea for CI/CD (optional)
|
||||
- SMTP server for notifications (optional)
|
||||
- LDAP server for authentication (optional)
|
||||
|
||||
## Risk Assessment & Mitigation
|
||||
|
||||
### Technical Risks
|
||||
- **Dependency Updates**: Regular security updates and compatibility testing
|
||||
- **Performance**: Monitoring and optimization of resource usage
|
||||
- **Security**: Regular vulnerability scanning and patching
|
||||
- **Scalability**: Load testing and capacity planning
|
||||
|
||||
### Operational Risks
|
||||
- **Documentation**: Keep runbooks current with system changes
|
||||
- **Monitoring**: Comprehensive alerting for all critical components
|
||||
- **Backup**: Regular backups of configurations and data
|
||||
- **Disaster Recovery**: Tested recovery procedures
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Technical Metrics
|
||||
- Code coverage > 80%
|
||||
- Performance benchmarks met
|
||||
- Security scan clean
|
||||
- Zero critical vulnerabilities
|
||||
|
||||
### Operational Metrics
|
||||
- Successful deployments
|
||||
- Incident response < 15 minutes
|
||||
- System uptime > 99.9%
|
||||
- User satisfaction scores
|
||||
|
||||
## Communication & Collaboration
|
||||
|
||||
### Internal Communication
|
||||
- Regular architecture reviews
|
||||
- Code review requirements
|
||||
- Documentation standards
|
||||
- Knowledge sharing sessions
|
||||
|
||||
### External Communication
|
||||
- Clear project documentation
|
||||
- API documentation
|
||||
- User guides and tutorials
|
||||
- Support and troubleshooting guides
|
||||
|
||||
---
|
||||
|
||||
*This context file serves as a comprehensive guide for future portfolio expansion and maintenance. Update this file as new features are added or architectural decisions are made.*
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
# Portfolio Changelog
|
||||
|
||||
## [1.0.0] - 2026-04-29 - Initial Enterprise Portfolio Release
|
||||
|
||||
### Added
|
||||
#### Enterprise Infrastructure Simulator
|
||||
- **Container-based Linux node simulation** with Docker Compose
|
||||
- **Comprehensive Ansible automation suite**:
|
||||
- `provision.yml`: Node provisioning with security hardening, package installation, and service configuration
|
||||
- `patch.yml`: Automated patching with rollback capabilities and notification system
|
||||
- `hardening.yml`: Security hardening following CIS benchmarks (firewall, SSH, user management)
|
||||
- `decommission.yml`: Graceful node decommissioning with cleanup and notification
|
||||
- **Operational scripts**:
|
||||
- `simulate_scaling.sh`: Infrastructure scaling simulation
|
||||
- `simulate_failure.sh`: Failure injection for testing resilience
|
||||
- **Realistic scenarios**:
|
||||
- `scaling_event.yml`: Automated scaling event playbook
|
||||
- **Production Makefile** with targets: `up`, `patch`, `harden`, `destroy`
|
||||
- **Multi-group Ansible inventory** (`inventory/hosts.ini`) with realistic enterprise structure
|
||||
|
||||
#### Migration Validation Framework
|
||||
- **Python 3.8+ CLI application** (`cli.py`) with command structure:
|
||||
- `snapshot`: Collect system data from target hosts
|
||||
- `compare`: Compare snapshots for migration validation
|
||||
- `report`: Generate HTML reports from comparison results
|
||||
- **Modular collector architecture**:
|
||||
- `collectors/mounts.py`: Filesystem mount point analysis
|
||||
- `collectors/services.py`: System service inventory and status
|
||||
- `collectors/disk_usage.py`: Disk usage statistics and trends
|
||||
- **Intelligent comparison engine** (`validators/compare.py`):
|
||||
- Drift detection algorithms
|
||||
- Change categorization (additions, modifications, removals)
|
||||
- Risk assessment scoring
|
||||
- **Interactive HTML reporting** (`reports/html_report.py`):
|
||||
- Bootstrap CSS styling
|
||||
- JavaScript-powered filtering and sorting
|
||||
- Detailed change summaries with timestamps
|
||||
- Export capabilities
|
||||
|
||||
#### Observability Stack
|
||||
- **Complete ELK + Grafana monitoring platform** (`docker-compose.yml`):
|
||||
- Elasticsearch 8.11.0 with security enabled
|
||||
- Logstash 8.11.0 with custom pipelines
|
||||
- Kibana 8.11.0 with pre-configured dashboards
|
||||
- Grafana 10.2.0 with alerting and visualization
|
||||
- Filebeat for log collection
|
||||
- **Realistic sample logs** (`logs/sample.log`):
|
||||
- Application logs with various log levels
|
||||
- System logs (nginx, systemd, kernel)
|
||||
- Database logs (PostgreSQL, Redis)
|
||||
- Security events and authentication logs
|
||||
- **Enterprise alerting system** (`alerting/alert_rules.yml`):
|
||||
- System resource alerts (CPU, memory, disk)
|
||||
- Service availability monitoring
|
||||
- Application performance alerts
|
||||
- Security incident detection
|
||||
- Multi-channel notifications (email, Slack, PagerDuty)
|
||||
- **Incident simulation framework** (`scenarios/incident_simulation.sh`):
|
||||
- CPU spike simulation
|
||||
- Memory leak scenarios
|
||||
- Disk space exhaustion
|
||||
- Network latency/packet loss
|
||||
- Service crash simulation
|
||||
- Database connection issues
|
||||
- Application error bursts
|
||||
- Comprehensive incident scenarios
|
||||
|
||||
#### Documentation and Infrastructure
|
||||
- **Root documentation**:
|
||||
- `README.md`: Portfolio landing page with project overview and architecture summary
|
||||
- `docs/architecture.md`: Detailed system architecture and design principles
|
||||
- `docs/runbooks.md`: Operational procedures and troubleshooting guides
|
||||
- **CI/CD Pipeline** (`.gitea/workflows/ci.yml`):
|
||||
- Ansible syntax validation and linting
|
||||
- Python code testing and type checking
|
||||
- Docker image validation
|
||||
- Security scanning
|
||||
- Documentation generation
|
||||
|
||||
### Technical Implementation Details
|
||||
- **Languages**: Python 3.8+, YAML, Bash, HTML/CSS/JavaScript
|
||||
- **Frameworks**: Ansible, Docker Compose, ELK Stack, Grafana
|
||||
- **Infrastructure**: Container-based with production networking
|
||||
- **Security**: CIS-compliant hardening, secure defaults, input validation
|
||||
- **Monitoring**: Comprehensive alerting with escalation policies
|
||||
- **Testing**: Incident simulation, syntax validation, compilation checks
|
||||
|
||||
### Quality Assurance
|
||||
- ✅ **Syntax validation**: All Ansible playbooks and Python code compile without errors
|
||||
- ✅ **Boolean fixes**: Updated Ansible syntax from 'yes/no' to 'true/false' for modern compatibility
|
||||
- ✅ **Enterprise naming**: Realistic hostnames, service names, and configurations
|
||||
- ✅ **Production quality**: Error handling, logging, health checks, and rollback capabilities
|
||||
- ✅ **Documentation**: Comprehensive READMEs, architecture docs, and operational runbooks
|
||||
|
||||
### Architecture Highlights
|
||||
- **Modular design**: Each project operates independently with clear interfaces
|
||||
- **Enterprise patterns**: Multi-tier architecture, service separation, monitoring integration
|
||||
- **Scalability**: Container-based deployment with orchestration
|
||||
- **Observability**: End-to-end monitoring from infrastructure to application level
|
||||
- **Automation**: Infrastructure as Code with comprehensive automation coverage
|
||||
|
||||
### Skills Demonstrated
|
||||
- **Infrastructure Automation**: Ansible playbook development and enterprise infrastructure management
|
||||
- **Application Development**: Python CLI application with modular architecture and reporting
|
||||
- **Monitoring & Alerting**: ELK stack configuration, alerting rules, and incident response
|
||||
- **Container Orchestration**: Docker Compose for multi-service applications
|
||||
- **DevOps Practices**: CI/CD pipeline implementation, documentation, and operational procedures
|
||||
- **System Administration**: Linux hardening, patching strategies, and decommissioning procedures
|
||||
- **Security**: CIS benchmarks implementation and security monitoring
|
||||
- **Data Analysis**: System data collection, comparison algorithms, and visualization
|
||||
|
||||
### Future Expansion Points
|
||||
- Kubernetes orchestration integration
|
||||
- Multi-cloud deployment support
|
||||
- Advanced monitoring dashboards
|
||||
- Machine learning-based anomaly detection
|
||||
- Integration with enterprise tools (Jira, ServiceNow)
|
||||
- Performance optimization and benchmarking
|
||||
- Compliance automation (PCI-DSS, HIPAA)
|
||||
- Disaster recovery procedures
|
||||
|
||||
---
|
||||
*Portfolio created to demonstrate enterprise-level Linux infrastructure engineering capabilities across the full technology stack.*
|
||||
@@ -0,0 +1,19 @@
|
||||
# Infrastructure Engineering Portfolio
|
||||
|
||||
This repository contains independent infrastructure projects focused on automation, migration assurance, and observability. The projects are intentionally small enough to run locally, but structured around the operating patterns used in enterprise platform teams: repeatable workflows, clear evidence artifacts, and operational documentation.
|
||||
|
||||
## Projects
|
||||
|
||||
- [Enterprise Infrastructure Simulator](enterprise-infra-simulator/) - Ansible-driven lifecycle operations for provisioning, patching, hardening, decommissioning, and failure simulation across Linux nodes.
|
||||
- [Migration Validation Framework](migration-validation-framework/) - Python CLI for collecting before/after system snapshots and producing structured migration comparison results.
|
||||
- [Observability Stack](observability-stack/) - Docker Compose based logging and dashboard stack with alert rules, sample logs, and incident simulation.
|
||||
|
||||
## Skills Demonstrated
|
||||
|
||||
- Infrastructure automation with Ansible
|
||||
- Operational scenario design and incident simulation
|
||||
- Migration validation, drift detection, and JSON reporting
|
||||
- Docker Compose service validation
|
||||
- Repository hygiene, CI checks, and professional project documentation
|
||||
|
||||
Each project remains independent and includes its own README, architecture notes, examples, and runnable scenarios.
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
# Architecture Overview
|
||||
|
||||
## Enterprise Infrastructure Portfolio Architecture
|
||||
|
||||
This document provides a high-level overview of the architecture and design principles implemented across the three main projects in this portfolio.
|
||||
|
||||
## Overall Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Enterprise Portfolio │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────┐ │
|
||||
│ │ Infra Simulator│ │Migration │ │Observability│ │
|
||||
│ │ (Ansible/Docker│ │Validation │ │Stack │ │
|
||||
│ │ Container Sim) │ │(Python CLI) │ │(ELK/Grafana)│ │
|
||||
│ └─────────────────┘ └─────────────────┘ └─────────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Infrastructure Simulation │ Validation Framework │ Monitoring │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Project Architectures
|
||||
|
||||
### 1. Enterprise Infrastructure Simulator
|
||||
|
||||
**Architecture Pattern:** Container-based Infrastructure Simulation
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Ansible │ │ Docker │ │ Simulation │
|
||||
│ Controller │◄──►│ Containers │◄──►│ Scripts │
|
||||
│ │ │ (Linux Nodes) │ │ │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Inventory │ │ Playbooks │ │ Scenarios │
|
||||
│ Management │ │ (Provision/ │ │ (Scaling/ │
|
||||
│ │ │ Patch/ │ │ Failures) │
|
||||
│ │ │ Harden/ │ │ │
|
||||
│ │ │ Decommission)│ │ │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
**Key Components:**
|
||||
- **Ansible Controller:** Central orchestration for infrastructure operations
|
||||
- **Docker Containers:** Simulated Linux nodes with realistic configurations
|
||||
- **Simulation Scripts:** Automated scaling and failure injection
|
||||
- **Inventory System:** Dynamic host management and grouping
|
||||
- **Playbook Library:** Modular automation for different lifecycle phases
|
||||
|
||||
### 2. Migration Validation Framework
|
||||
|
||||
**Architecture Pattern:** Data Collection and Comparison Pipeline
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ CLI Interface │ │ Data │ │ Validation │
|
||||
│ (cli.py) │◄──►│ Collectors │◄──►│ Engine │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ JSON │ │ Comparison │ │ HTML │
|
||||
│ Snapshots │ │ Logic │ │ Reports │
|
||||
│ (Before/After)│ │ │ │ │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
**Key Components:**
|
||||
- **CLI Interface:** Command-line tool for migration workflow orchestration
|
||||
- **Data Collectors:** Specialized modules for system data extraction
|
||||
- **Validation Engine:** Snapshot comparison and difference analysis
|
||||
- **Report Generator:** HTML output with change visualization
|
||||
- **JSON Storage:** Structured data persistence for before/after states
|
||||
|
||||
### 3. Observability Stack
|
||||
|
||||
**Architecture Pattern:** Distributed Monitoring and Logging
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Logstash │ │ Elasticsearch │ │ Kibana │
|
||||
│ (Ingestion) │◄──►│ (Storage) │◄──►│ (Visualization)│
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
▲ ▲ ▲
|
||||
│ │ │
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Sample Logs │ │ Alert Rules │ │ Grafana │
|
||||
│ (Data Sources)│ │ (Conditions) │ │ (Dashboards) │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
**Key Components:**
|
||||
- **Logstash Pipelines:** Data ingestion and transformation
|
||||
- **Elasticsearch Cluster:** Distributed search and analytics
|
||||
- **Kibana Dashboards:** Real-time visualization and exploration
|
||||
- **Grafana Integration:** Advanced metrics and alerting
|
||||
- **Alerting Engine:** Automated incident detection and notification
|
||||
|
||||
## Design Principles
|
||||
|
||||
### Infrastructure as Code
|
||||
- All infrastructure defined in code (Ansible, Docker Compose, Python)
|
||||
- Version-controlled configurations and automation
|
||||
- Reproducible environments and deployments
|
||||
|
||||
### Modular Architecture
|
||||
- Separated concerns across projects and components
|
||||
- Reusable modules and playbooks
|
||||
- Clear interfaces between systems
|
||||
|
||||
### Enterprise Standards
|
||||
- Realistic naming conventions and structures
|
||||
- Production-quality error handling and logging
|
||||
- Security hardening and compliance considerations
|
||||
|
||||
### Observability First
|
||||
- Comprehensive logging and monitoring
|
||||
- Automated alerting and incident response
|
||||
- Performance metrics and health checks
|
||||
|
||||
## Technology Stack
|
||||
|
||||
- **Containerization:** Docker, Docker Compose
|
||||
- **Configuration Management:** Ansible
|
||||
- **Programming Language:** Python 3.8+
|
||||
- **Monitoring Stack:** ELK Stack (Elasticsearch, Logstash, Kibana)
|
||||
- **Visualization:** Grafana
|
||||
- **CI/CD:** Gitea Actions
|
||||
- **Documentation:** Markdown
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Container security scanning integration
|
||||
- Ansible vault for secrets management
|
||||
- Network segmentation in Docker Compose
|
||||
- Least privilege access principles
|
||||
- Audit logging and compliance reporting
|
||||
|
||||
## Scalability and Performance
|
||||
|
||||
- Horizontal scaling through container orchestration
|
||||
- Efficient data collection and processing
|
||||
- Optimized Elasticsearch indexing
|
||||
- Resource-aware automation scripts
|
||||
@@ -0,0 +1,329 @@
|
||||
# Runbooks and Operational Procedures
|
||||
|
||||
This document contains operational runbooks for deploying, managing, and troubleshooting the Enterprise Infrastructure Portfolio projects.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Infrastructure Simulator Operations](#infrastructure-simulator-operations)
|
||||
2. [Migration Validation Procedures](#migration-validation-procedures)
|
||||
3. [Observability Stack Management](#observability-stack-management)
|
||||
4. [Troubleshooting Guide](#troubleshooting-guide)
|
||||
|
||||
## Infrastructure Simulator Operations
|
||||
|
||||
### Starting the Infrastructure
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
make up
|
||||
```
|
||||
|
||||
**Expected Outcome:**
|
||||
- Docker containers for simulated Linux nodes are created
|
||||
- Ansible inventory is populated
|
||||
- Basic services are running on all nodes
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
docker ps | grep infra-sim
|
||||
ansible -i inventory/hosts.ini all -m ping
|
||||
```
|
||||
|
||||
### Patching Operations
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
make patch
|
||||
```
|
||||
|
||||
**Procedure:**
|
||||
1. Backup current container states
|
||||
2. Apply security patches via Ansible
|
||||
3. Validate service availability
|
||||
4. Generate patch report
|
||||
|
||||
**Rollback:**
|
||||
```bash
|
||||
docker-compose down
|
||||
docker-compose up --scale node=0
|
||||
make up
|
||||
```
|
||||
|
||||
### Hardening Operations
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/hardening.yml
|
||||
```
|
||||
|
||||
**Hardening Steps:**
|
||||
- Disable unnecessary services
|
||||
- Configure firewall rules
|
||||
- Set secure SSH configurations
|
||||
- Apply CIS benchmarks
|
||||
|
||||
### Scaling Operations
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
./scripts/simulate_scaling.sh up 3
|
||||
```
|
||||
|
||||
**Scaling Parameters:**
|
||||
- Direction: up/down
|
||||
- Count: number of nodes to add/remove
|
||||
- Type: web/app/db
|
||||
|
||||
### Failure Simulation
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
./scripts/simulate_failure.sh --type network --duration 300
|
||||
```
|
||||
|
||||
**Failure Types:**
|
||||
- network: Network partition
|
||||
- disk: Disk space exhaustion
|
||||
- service: Service crashes
|
||||
- node: Complete node failure
|
||||
|
||||
### Decommissioning
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
make destroy
|
||||
```
|
||||
|
||||
**Decommission Steps:**
|
||||
1. Graceful service shutdown
|
||||
2. Data backup and export
|
||||
3. Configuration cleanup
|
||||
4. Container removal
|
||||
|
||||
## Migration Validation Procedures
|
||||
|
||||
### Pre-Migration Snapshot
|
||||
|
||||
```bash
|
||||
cd migration-validation-framework
|
||||
python3 cli.py collect --output before.json --systems web01,db01
|
||||
```
|
||||
|
||||
**Data Collected:**
|
||||
- Mount points and filesystem usage
|
||||
- Running services and their states
|
||||
- Disk usage statistics
|
||||
- Network configurations
|
||||
|
||||
### Post-Migration Validation
|
||||
|
||||
```bash
|
||||
python3 cli.py collect --output after.json --systems web01,db01
|
||||
python3 cli.py compare before.json after.json --output diff.json
|
||||
```
|
||||
|
||||
**Validation Checks:**
|
||||
- Service availability verification
|
||||
- Filesystem integrity
|
||||
- Configuration consistency
|
||||
- Performance metrics comparison
|
||||
|
||||
### Report Generation
|
||||
|
||||
```bash
|
||||
python3 cli.py report --comparison <comparison-id> --format html
|
||||
```
|
||||
|
||||
**Report Contents:**
|
||||
- Executive summary
|
||||
- Detailed change log
|
||||
- Risk assessment
|
||||
- Recommendations
|
||||
|
||||
## Observability Stack Management
|
||||
|
||||
### Starting the Stack
|
||||
|
||||
```bash
|
||||
cd observability-stack
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
**Service Startup Order:**
|
||||
1. Elasticsearch
|
||||
2. Logstash
|
||||
3. Kibana
|
||||
4. Grafana
|
||||
|
||||
### Log Ingestion Testing
|
||||
|
||||
```bash
|
||||
# Send sample logs
|
||||
curl -X POST "localhost:8080" -H "Content-Type: application/json" -d @logs/sample.log
|
||||
```
|
||||
|
||||
### Alert Configuration
|
||||
|
||||
```bash
|
||||
# Load alert rules
|
||||
curl -X POST "localhost:3000/api/alerts" -H "Authorization: Bearer <token>" -d @alerting/alert_rules.json
|
||||
```
|
||||
|
||||
### Incident Simulation
|
||||
|
||||
```bash
|
||||
cd observability-stack
|
||||
./scenarios/incident_simulation.sh --type disk-full --severity critical
|
||||
```
|
||||
|
||||
**Incident Types:**
|
||||
- disk-full: Simulate disk space exhaustion
|
||||
- service-down: Service failure simulation
|
||||
- high-cpu: CPU utilization spike
|
||||
- network-latency: Network performance degradation
|
||||
|
||||
## Troubleshooting Guide
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Ansible Connection Failures
|
||||
|
||||
**Symptoms:**
|
||||
- `UNREACHABLE` errors in Ansible output
|
||||
- SSH connection timeouts
|
||||
|
||||
**Resolution:**
|
||||
```bash
|
||||
# Check container status
|
||||
docker ps | grep infra-sim
|
||||
|
||||
# Verify SSH keys
|
||||
ansible -i inventory/hosts.ini all -m ping --private-key ~/.ssh/id_rsa
|
||||
|
||||
# Restart containers
|
||||
make destroy && make up
|
||||
```
|
||||
|
||||
#### Elasticsearch Cluster Issues
|
||||
|
||||
**Symptoms:**
|
||||
- Kibana shows "No living connections"
|
||||
- Logstash pipeline failures
|
||||
|
||||
**Resolution:**
|
||||
```bash
|
||||
# Check cluster health
|
||||
curl -X GET "localhost:9200/_cluster/health?pretty"
|
||||
|
||||
# Restart services
|
||||
docker-compose restart elasticsearch logstash kibana
|
||||
```
|
||||
|
||||
#### Python Import Errors
|
||||
|
||||
**Symptoms:**
|
||||
- ModuleNotFoundError in migration framework
|
||||
- Collector failures
|
||||
|
||||
**Resolution:**
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Check Python path
|
||||
python -c "import sys; print(sys.path)"
|
||||
```
|
||||
|
||||
#### Docker Resource Constraints
|
||||
|
||||
**Symptoms:**
|
||||
- Container startup failures
|
||||
- Out of memory errors
|
||||
|
||||
**Resolution:**
|
||||
```bash
|
||||
# Check Docker resources
|
||||
docker system df
|
||||
|
||||
# Clean up unused resources
|
||||
docker system prune -a
|
||||
|
||||
# Increase Docker memory limit
|
||||
# Edit /etc/docker/daemon.json
|
||||
{
|
||||
"memory": "4g",
|
||||
"cpu-count": 2
|
||||
}
|
||||
```
|
||||
|
||||
### Log Locations
|
||||
|
||||
- **Ansible:** `enterprise-infra-simulator/ansible.log`
|
||||
- **Docker:** `docker logs <container-name>`
|
||||
- **Elasticsearch:** `observability-stack/logs/elasticsearch.log`
|
||||
- **Migration Framework:** `migration-validation-framework/logs/validation.log`
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
```bash
|
||||
# Infrastructure monitoring
|
||||
ansible -i inventory/hosts.ini all -m shell -a "top -b -n1 | head -20"
|
||||
|
||||
# Elasticsearch metrics
|
||||
curl -X GET "localhost:9200/_cluster/stats?pretty"
|
||||
|
||||
# Python performance
|
||||
python -m cProfile cli.py snapshot
|
||||
```
|
||||
|
||||
### Backup and Recovery
|
||||
|
||||
#### Infrastructure Backup
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
docker-compose exec ansible ansible-playbook /playbooks/backup.yml
|
||||
```
|
||||
|
||||
#### Data Backup
|
||||
```bash
|
||||
cd observability-stack
|
||||
docker-compose exec elasticsearch curl -X PUT "localhost:9200/_snapshot/backup" -H "Content-Type: application/json" -d @backup_config.json
|
||||
```
|
||||
|
||||
#### Migration Data Backup
|
||||
```bash
|
||||
cd migration-validation-framework
|
||||
tar -czf /backup/location/migration-validation-framework.tgz migration-validation-framework
|
||||
```
|
||||
|
||||
## Emergency Procedures
|
||||
|
||||
### Complete System Reset
|
||||
|
||||
```bash
|
||||
# Stop all services
|
||||
docker-compose down -v
|
||||
cd enterprise-infra-simulator && make destroy
|
||||
|
||||
# Clean up volumes
|
||||
docker volume prune -f
|
||||
|
||||
# Restart from clean state
|
||||
cd enterprise-infra-simulator && make up
|
||||
cd observability-stack && docker-compose up -d
|
||||
```
|
||||
|
||||
### Incident Response
|
||||
|
||||
1. **Assess Impact:** Check monitoring dashboards
|
||||
2. **Isolate Issue:** Use failure simulation scripts to reproduce
|
||||
3. **Implement Fix:** Apply appropriate runbook procedure
|
||||
4. **Validate Recovery:** Run validation framework
|
||||
5. **Document Incident:** Update runbooks with lessons learned
|
||||
|
||||
## Maintenance Schedules
|
||||
|
||||
- **Daily:** Log rotation and cleanup
|
||||
- **Weekly:** Security patching and updates
|
||||
- **Monthly:** Performance optimization and capacity planning
|
||||
- **Quarterly:** Architecture review and modernization
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
# Ansible-lint configuration
|
||||
#extends: default
|
||||
|
||||
skip_list:
|
||||
- 'role-name'
|
||||
- 'name[casing]'
|
||||
- 'line-too-long'
|
||||
|
||||
# Ignore these rules
|
||||
exclude_paths:
|
||||
- .git
|
||||
- .github
|
||||
- molecule/default/tests/
|
||||
|
||||
# Custom rules
|
||||
#rules:
|
||||
# line-length:
|
||||
# max: 160
|
||||
# level: warning
|
||||
@@ -0,0 +1,173 @@
|
||||
# Enterprise Infrastructure Simulator Makefile
|
||||
|
||||
.PHONY: help run demo up down patch destroy status logs clean test
|
||||
|
||||
# Default target
|
||||
help: ## Show this help message
|
||||
@echo "Enterprise Infrastructure Simulator"
|
||||
@echo ""
|
||||
@echo "Available commands:"
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf " %-15s %s\n", $$1, $$2}'
|
||||
|
||||
run: ## Run the default simulator workflow
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/provision.yml
|
||||
|
||||
demo: ## Run a failure-and-patch demonstration
|
||||
./scripts/simulate_failure.sh service 30 web
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/patch.yml
|
||||
|
||||
# Infrastructure management
|
||||
up: ## Start the infrastructure simulation
|
||||
@echo "Starting enterprise infrastructure simulation..."
|
||||
docker-compose up -d
|
||||
@echo "Waiting for containers to be ready..."
|
||||
@sleep 30
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/provision.yml
|
||||
@echo "Infrastructure simulation started successfully"
|
||||
|
||||
down: ## Stop the infrastructure simulation
|
||||
@echo "Stopping infrastructure simulation..."
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/decommission.yml || true
|
||||
docker-compose down
|
||||
@echo "Infrastructure simulation stopped"
|
||||
|
||||
patch: ## Apply security patches to all nodes
|
||||
@echo "Applying security patches..."
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/patch.yml
|
||||
@echo "Security patches applied"
|
||||
|
||||
destroy: ## Completely destroy the infrastructure
|
||||
@echo "Destroying infrastructure..."
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/decommission.yml || true
|
||||
docker-compose down -v --remove-orphans
|
||||
docker system prune -f
|
||||
rm -rf logs/* reports/*
|
||||
@echo "Infrastructure completely destroyed"
|
||||
|
||||
# Scaling operations
|
||||
scale-up-web: ## Scale up web servers (usage: make scale-up-web COUNT=2)
|
||||
@echo "Scaling up $(COUNT) web servers..."
|
||||
./scripts/simulate_scaling.sh up $(or $(COUNT),1) web
|
||||
|
||||
scale-up-db: ## Scale up database servers (usage: make scale-up-db COUNT=1)
|
||||
@echo "Scaling up $(COUNT) database servers..."
|
||||
./scripts/simulate_scaling.sh up $(or $(COUNT),1) db
|
||||
|
||||
scale-down-web: ## Scale down web servers (usage: make scale-down-web COUNT=1)
|
||||
@echo "Scaling down $(COUNT) web servers..."
|
||||
./scripts/simulate_scaling.sh down $(or $(COUNT),1) web
|
||||
|
||||
scale-down-db: ## Scale down database servers (usage: make scale-down-db COUNT=1)
|
||||
@echo "Scaling down $(COUNT) database servers..."
|
||||
./scripts/simulate_scaling.sh down $(or $(COUNT),1) db
|
||||
|
||||
# Failure simulation
|
||||
fail-network: ## Simulate network failure (usage: make fail-network DURATION=60)
|
||||
@echo "Simulating network failure for $(or $(DURATION),60) seconds..."
|
||||
./scripts/simulate_failure.sh network $(or $(DURATION),60)
|
||||
|
||||
fail-disk: ## Simulate disk space exhaustion (usage: make fail-disk DURATION=120)
|
||||
@echo "Simulating disk failure for $(or $(DURATION),120) seconds..."
|
||||
./scripts/simulate_failure.sh disk $(or $(DURATION),120)
|
||||
|
||||
fail-service: ## Simulate service failures (usage: make fail-service DURATION=30)
|
||||
@echo "Simulating service failure for $(or $(DURATION),30) seconds..."
|
||||
./scripts/simulate_failure.sh service $(or $(DURATION),30)
|
||||
|
||||
fail-node: ## Simulate complete node failure (usage: make fail-node DURATION=300)
|
||||
@echo "Simulating node failure for $(or $(DURATION),300) seconds..."
|
||||
./scripts/simulate_failure.sh node $(or $(DURATION),300)
|
||||
|
||||
# Monitoring and status
|
||||
status: ## Show infrastructure status
|
||||
@echo "=== Docker Containers ==="
|
||||
docker-compose ps
|
||||
@echo ""
|
||||
@echo "=== Ansible Inventory ==="
|
||||
ansible -i inventory/hosts.ini --list-hosts all || echo "Inventory check failed"
|
||||
@echo ""
|
||||
@echo "=== System Resources ==="
|
||||
docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemPerc}}\t{{.NetIO}}"
|
||||
|
||||
logs: ## Show infrastructure logs
|
||||
docker-compose logs -f --tail=100
|
||||
|
||||
logs-web: ## Show web server logs
|
||||
docker-compose logs -f web
|
||||
|
||||
logs-db: ## Show database logs
|
||||
docker-compose logs -f db
|
||||
|
||||
# Testing and validation
|
||||
test: ## Run infrastructure tests
|
||||
@echo "Running infrastructure tests..."
|
||||
ansible -i inventory/hosts.ini all -m ping
|
||||
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/*.yml
|
||||
@echo "Testing scaling scripts..."
|
||||
./scripts/simulate_scaling.sh up 0 web # Dry run
|
||||
./scripts/simulate_failure.sh network 1 # Quick test
|
||||
@echo "All tests passed"
|
||||
|
||||
validate: ## Validate infrastructure configuration
|
||||
@echo "Validating configuration..."
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/provision.yml --check
|
||||
docker-compose config
|
||||
@echo "Configuration validation complete"
|
||||
|
||||
# Scenarios
|
||||
scenario-scaling: ## Run scaling event scenario
|
||||
@echo "Running scaling event scenario..."
|
||||
ansible-playbook -i inventory/hosts.ini scenarios/scaling_event.yml
|
||||
|
||||
scenario-disaster: ## Run disaster recovery scenario
|
||||
@echo "Running disaster recovery scenario..."
|
||||
ansible-playbook -i inventory/hosts.ini scenarios/disaster_recovery.yml
|
||||
|
||||
# Maintenance
|
||||
clean: ## Clean up temporary files and logs
|
||||
@echo "Cleaning up temporary files..."
|
||||
rm -rf logs/*.log reports/*.txt
|
||||
docker system prune -f
|
||||
@echo "Cleanup complete"
|
||||
|
||||
backup: ## Create infrastructure backup
|
||||
@echo "Creating infrastructure backup..."
|
||||
mkdir -p backups/$(shell date +%Y%m%d_%H%M%S)
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/backup.yml
|
||||
docker-compose exec ansible tar -czf /backups/infra_backup.tar.gz /infrastructure
|
||||
@echo "Backup created"
|
||||
|
||||
# Development
|
||||
lint: ## Lint Ansible playbooks
|
||||
@echo "Linting Ansible playbooks..."
|
||||
ansible-lint playbooks/*.yml scenarios/*.yml
|
||||
@echo "Linting complete"
|
||||
|
||||
format: ## Format code and configuration
|
||||
@echo "Formatting code..."
|
||||
# Add formatting commands here
|
||||
@echo "Formatting complete"
|
||||
|
||||
# Security
|
||||
harden: ## Apply security hardening
|
||||
@echo "Applying security hardening..."
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/hardening.yml
|
||||
|
||||
security-scan: ## Run security scans
|
||||
@echo "Running security scans..."
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/security_scan.yml
|
||||
|
||||
# Help for specific targets
|
||||
help-scaling: ## Show scaling-related commands
|
||||
@echo "Scaling Commands:"
|
||||
@echo " make scale-up-web COUNT=2 - Add 2 web servers"
|
||||
@echo " make scale-up-db COUNT=1 - Add 1 database server"
|
||||
@echo " make scale-down-web COUNT=1 - Remove 1 web server"
|
||||
@echo " make scale-down-db COUNT=1 - Remove 1 database server"
|
||||
|
||||
help-failure: ## Show failure simulation commands
|
||||
@echo "Failure Simulation Commands:"
|
||||
@echo " make fail-network DURATION=60 - Network failure for 60s"
|
||||
@echo " make fail-disk DURATION=120 - Disk exhaustion for 120s"
|
||||
@echo " make fail-service DURATION=30 - Service failure for 30s"
|
||||
@echo " make fail-node DURATION=300 - Node failure for 300s"
|
||||
@@ -0,0 +1,74 @@
|
||||
# Enterprise Infrastructure Simulator
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Infrastructure teams need a safe place to rehearse lifecycle operations before applying them to production fleets. Patch windows, hardening changes, scale events, and node failures all carry operational risk when they are tested only during real incidents.
|
||||
|
||||
## Solution Overview
|
||||
|
||||
This project models common Linux infrastructure operations with Ansible playbooks and shell-based simulations. It keeps the automation readable and auditable while producing example evidence that resembles a real change record.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Operator -> Make/CLI -> Ansible Inventory -> Playbooks -> Linux Nodes
|
||||
| |
|
||||
v v
|
||||
Scenarios Reports/Logs
|
||||
```
|
||||
|
||||
Core components:
|
||||
|
||||
- `inventory/hosts.ini` defines managed node groups.
|
||||
- `playbooks/` contains provisioning, patching, hardening, and decommissioning workflows.
|
||||
- `scripts/` injects scaling and failure conditions.
|
||||
- `scenarios/` documents operational exercises.
|
||||
- `examples/` stores representative outputs for review.
|
||||
|
||||
## How to Run
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
|
||||
# Validate playbook syntax.
|
||||
make test
|
||||
|
||||
# Provision the simulated estate.
|
||||
make run
|
||||
|
||||
# Apply security patches.
|
||||
make patch
|
||||
|
||||
# Apply host hardening.
|
||||
make harden
|
||||
|
||||
# Run the failure and patch demo.
|
||||
make demo
|
||||
```
|
||||
|
||||
Direct Ansible commands are also supported:
|
||||
|
||||
```bash
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/provision.yml
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/patch.yml
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/hardening.yml
|
||||
```
|
||||
|
||||
## Example Output
|
||||
|
||||
```text
|
||||
PLAY RECAP *********************************************************************
|
||||
web01 : ok=21 changed=7 unreachable=0 failed=0 skipped=3 rescued=0 ignored=1
|
||||
db01 : ok=18 changed=4 unreachable=0 failed=0 skipped=5 rescued=0 ignored=1
|
||||
lb01 : ok=16 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0
|
||||
|
||||
Patch status: SUCCESS
|
||||
Updates applied: 12
|
||||
Reboot required: false
|
||||
```
|
||||
|
||||
Additional sample evidence is available in [examples/patch-output.txt](examples/patch-output.txt) and [examples/failure-simulation.txt](examples/failure-simulation.txt).
|
||||
|
||||
## Real-World Use Case
|
||||
|
||||
A platform team can use this project to demonstrate how routine operating procedures are encoded, reviewed, and tested before production change windows. The same patterns apply to regulated Linux estates where patch evidence, hardening controls, and incident drills must be repeatable.
|
||||
@@ -0,0 +1,207 @@
|
||||
# Enterprise Infrastructure Simulator - Refactored
|
||||
|
||||
Refactored enterprise infrastructure automation using Ansible best practices.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
playbooks/ # Main playbooks
|
||||
├── provision.yml # Provision infrastructure nodes
|
||||
├── patch.yml # Apply security patches
|
||||
├── hardening.yml # Harden infrastructure
|
||||
└── decommission.yml # Decommission nodes
|
||||
|
||||
roles/ # Reusable Ansible roles
|
||||
├── base_provision/ # Base OS provisioning
|
||||
├── patching/ # Patch management
|
||||
├── hardening/ # Security hardening
|
||||
└── decommission/ # Node decommissioning
|
||||
|
||||
group_vars/ # Group-level variables
|
||||
├── all.yml # All hosts
|
||||
├── webservers.yml # Web servers
|
||||
├── databases.yml # Database servers
|
||||
├── loadbalancers.yml
|
||||
├── monitoring.yml
|
||||
└── vault.yml # Encrypted secrets (Vault)
|
||||
|
||||
molecule/default/ # Testing with Molecule
|
||||
├── molecule.yml # Molecule config
|
||||
├── converge.yml # Test playbook
|
||||
└── verify.yml # Test verification
|
||||
```
|
||||
|
||||
## Best Practices Implemented
|
||||
|
||||
### ✅ Idempotencja
|
||||
- All tasks use `changed_when` and `failed_when` for proper state detection
|
||||
- Command modules replaced with native Ansible modules where possible
|
||||
- Shell tasks include `changed_when: false` when appropriate
|
||||
|
||||
### ✅ Role + Struktura
|
||||
- Clean role separation: `base_provision`, `patching`, `hardening`, `decommission`
|
||||
- Each role has: `tasks/`, `handlers/`, `defaults/`, `templates/`, `README.md`
|
||||
- Proper namespacing prevents variable conflicts
|
||||
|
||||
### ✅ Brak Hardcodu
|
||||
- All variables in `defaults/main.yml` or `group_vars/`
|
||||
- No hardcoded values in playbooks
|
||||
- Configurable through `group_vars` for different environments
|
||||
|
||||
### ✅ Handlers zamiast Restartów
|
||||
- SSH restart via handler (triggered only on config change)
|
||||
- fail2ban restart via handler
|
||||
- Services not restarted unnecessarily
|
||||
|
||||
### ✅ Vault do Sekretów
|
||||
- Secrets go in `group_vars/vault.yml` (encrypted with Ansible Vault)
|
||||
- Admin passwords not in plaintext
|
||||
- Database credentials managed via Vault
|
||||
|
||||
### ✅ ansible-lint
|
||||
- `.ansible-lint` configuration included
|
||||
- Rules configured for project standards
|
||||
- Run: `ansible-lint playbooks/ roles/`
|
||||
|
||||
### ✅ Molecule
|
||||
- Docker-based testing in `molecule/default/`
|
||||
- Test convergence and verification
|
||||
- Run: `molecule test`
|
||||
|
||||
## Usage
|
||||
|
||||
### Run Provisioning
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/provision.yml -i inventory/hosts.ini
|
||||
```
|
||||
|
||||
### Run Patching
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/patch.yml -i inventory/hosts.ini --ask-vault-pass
|
||||
```
|
||||
|
||||
### Run Hardening
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/hardening.yml -i inventory/hosts.ini --ask-vault-pass
|
||||
```
|
||||
|
||||
### Run Decommissioning
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/decommission.yml -i inventory/hosts.ini --ask-vault-pass
|
||||
```
|
||||
|
||||
## Vault Management
|
||||
|
||||
### Create Vault Password File
|
||||
|
||||
```bash
|
||||
echo "your-secure-password" > ~/.vault_pass.txt
|
||||
chmod 600 ~/.vault_pass.txt
|
||||
```
|
||||
|
||||
### Encrypt Secrets
|
||||
|
||||
```bash
|
||||
ansible-vault encrypt group_vars/vault.yml --vault-password-file ~/.vault_pass.txt
|
||||
```
|
||||
|
||||
### Edit Encrypted Vault
|
||||
|
||||
```bash
|
||||
ansible-vault edit group_vars/vault.yml --vault-password-file ~/.vault_pass.txt
|
||||
```
|
||||
|
||||
### Run with Vault
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/provision.yml \
|
||||
--vault-password-file ~/.vault_pass.txt \
|
||||
-i inventory/hosts.ini
|
||||
```
|
||||
|
||||
## Linting
|
||||
|
||||
### Run ansible-lint
|
||||
|
||||
```bash
|
||||
ansible-lint playbooks/ roles/
|
||||
```
|
||||
|
||||
### Fix Issues
|
||||
|
||||
```bash
|
||||
ansible-lint playbooks/ roles/ --fix
|
||||
```
|
||||
|
||||
## Testing with Molecule
|
||||
|
||||
### Run All Tests
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
molecule test
|
||||
```
|
||||
|
||||
### Run Specific Scenarios
|
||||
|
||||
```bash
|
||||
molecule converge # Apply roles
|
||||
molecule verify # Verify results
|
||||
molecule destroy # Cleanup
|
||||
```
|
||||
|
||||
## Role Documentation
|
||||
|
||||
Each role has detailed README:
|
||||
|
||||
- [base_provision/README.md](roles/base_provision/README.md)
|
||||
- [patching/README.md](roles/patching/README.md)
|
||||
- [hardening/README.md](roles/hardening/README.md)
|
||||
- [decommission/README.md](roles/decommission/README.md)
|
||||
|
||||
## Group Variables
|
||||
|
||||
- `group_vars/all.yml` - Global configuration
|
||||
- `group_vars/webservers.yml` - Web server config
|
||||
- `group_vars/databases.yml` - Database config
|
||||
- `group_vars/loadbalancers.yml` - Load balancer config
|
||||
- `group_vars/monitoring.yml` - Monitoring config
|
||||
- `group_vars/vault.yml` - Encrypted secrets
|
||||
|
||||
## Tags
|
||||
|
||||
Use tags to run specific parts:
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/provision.yml --tags base,provision
|
||||
ansible-playbook playbooks/hardening.yml --tags security,hardening
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Proper use of `failed_when` for critical failures
|
||||
- Strategic use of `ignore_errors` only for optional operations
|
||||
- Comprehensive assertion checks for prerequisites
|
||||
|
||||
## Security
|
||||
|
||||
- Passwords stored in encrypted Vault
|
||||
- SSH key-based authentication
|
||||
- Firewall configured with deny-by-default policy
|
||||
- SELinux/AppArmor support
|
||||
- CIS hardening levels 1-2
|
||||
|
||||
## Monitoring
|
||||
|
||||
- Health checks included in playbooks
|
||||
- Service verification after operations
|
||||
- Detailed logging to `/var/log/`
|
||||
- Report generation for audit trails
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions about the roles, see individual role README files.
|
||||
@@ -0,0 +1,231 @@
|
||||
# Vault Configuration Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This project uses Ansible Vault to securely manage sensitive data such as passwords, API keys, and credentials.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Create Vault Password File
|
||||
|
||||
```bash
|
||||
# Generate a secure password
|
||||
openssl rand -base64 32 > ~/.vault_pass.txt
|
||||
|
||||
# Secure the file
|
||||
chmod 600 ~/.vault_pass.txt
|
||||
```
|
||||
|
||||
### 2. Add to .bashrc or .zshrc
|
||||
|
||||
```bash
|
||||
export ANSIBLE_VAULT_PASSWORD_FILE="$HOME/.vault_pass.txt"
|
||||
```
|
||||
|
||||
### 3. Configure ansible.cfg
|
||||
|
||||
```ini
|
||||
[defaults]
|
||||
vault_password_file = ~/.vault_pass.txt
|
||||
```
|
||||
|
||||
## Vault Files
|
||||
|
||||
### group_vars/vault.yml
|
||||
|
||||
This file contains all encrypted secrets:
|
||||
|
||||
```yaml
|
||||
---
|
||||
# Vault variables for sensitive data
|
||||
vault_admin_password: "<secure_password>"
|
||||
vault_db_password: "<db_password>"
|
||||
vault_grafana_password: "<grafana_password>"
|
||||
vault_ssh_key_passphrase: "<ssh_passphrase>"
|
||||
```
|
||||
|
||||
## Encrypting Secrets
|
||||
|
||||
### First Time - Encrypt vault.yml
|
||||
|
||||
```bash
|
||||
# Edit the file first with plain text secrets
|
||||
ansible-vault encrypt group_vars/vault.yml
|
||||
|
||||
# You'll be prompted for vault password
|
||||
# Then the file will be automatically encrypted
|
||||
```
|
||||
|
||||
### Edit Encrypted Vault
|
||||
|
||||
```bash
|
||||
# Edit the vault file (will decrypt, open editor, re-encrypt)
|
||||
ansible-vault edit group_vars/vault.yml
|
||||
|
||||
# Or view without editing
|
||||
ansible-vault view group_vars/vault.yml
|
||||
```
|
||||
|
||||
### Encrypt New Files
|
||||
|
||||
```bash
|
||||
ansible-vault encrypt group_vars/new_secrets.yml
|
||||
```
|
||||
|
||||
## Using Vault in Playbooks
|
||||
|
||||
### Import Vault Variables
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: My Playbook
|
||||
hosts: all
|
||||
vars_files:
|
||||
- vars/vault.yml
|
||||
|
||||
tasks:
|
||||
- name: Use vault password
|
||||
user:
|
||||
name: admin
|
||||
password: "{{ vault_admin_password | password_hash('sha512') }}"
|
||||
```
|
||||
|
||||
## Running Playbooks with Vault
|
||||
|
||||
### Method 1: Using .vault_pass.txt
|
||||
|
||||
```bash
|
||||
export ANSIBLE_VAULT_PASSWORD_FILE="$HOME/.vault_pass.txt"
|
||||
ansible-playbook playbooks/provision.yml -i inventory/hosts.ini
|
||||
```
|
||||
|
||||
### Method 2: Inline Flag
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/provision.yml \
|
||||
--vault-password-file ~/.vault_pass.txt \
|
||||
-i inventory/hosts.ini
|
||||
```
|
||||
|
||||
### Method 3: Prompt for Password
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/provision.yml \
|
||||
--ask-vault-pass \
|
||||
-i inventory/hosts.ini
|
||||
|
||||
# You'll be prompted to enter vault password
|
||||
```
|
||||
|
||||
## Viewing Vault Contents
|
||||
|
||||
```bash
|
||||
# View encrypted file
|
||||
ansible-vault view group_vars/vault.yml
|
||||
|
||||
# View specific variable
|
||||
ansible-playbook playbooks/provision.yml \
|
||||
--tags never \
|
||||
-e "ansible_connection=local" \
|
||||
-i localhost, \
|
||||
-m debug \
|
||||
-a "var=vault_admin_password"
|
||||
```
|
||||
|
||||
## Vault Best Practices
|
||||
|
||||
### ✅ DO
|
||||
|
||||
- Store all passwords in vault.yml
|
||||
- Use strong vault passwords (32+ characters)
|
||||
- Keep vault password file secure (chmod 600)
|
||||
- Rotate vault passwords periodically
|
||||
- Version control only encrypted files
|
||||
- Document what each variable contains
|
||||
|
||||
### ❌ DON'T
|
||||
|
||||
- Commit unencrypted vault.yml to git
|
||||
- Share vault password file
|
||||
- Hardcode secrets in playbooks
|
||||
- Use weak passwords
|
||||
- Check plaintext secrets into version control
|
||||
|
||||
## Rekeying Vault
|
||||
|
||||
To change the vault password:
|
||||
|
||||
```bash
|
||||
ansible-vault rekey group_vars/vault.yml
|
||||
|
||||
# You'll be prompted for:
|
||||
# 1. Current vault password
|
||||
# 2. New vault password
|
||||
# 3. Confirm new vault password
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
For CI/CD pipelines (GitHub Actions, GitLab CI, etc.):
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
- name: Run Ansible Playbook
|
||||
env:
|
||||
ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }}
|
||||
run: |
|
||||
echo "$ANSIBLE_VAULT_PASSWORD" > ~/.vault_pass.txt
|
||||
ansible-playbook playbooks/provision.yml
|
||||
```
|
||||
|
||||
### GitLab CI Example
|
||||
|
||||
```yaml
|
||||
deploy:
|
||||
script:
|
||||
- echo "$ANSIBLE_VAULT_PASSWORD" > ~/.vault_pass.txt
|
||||
- ansible-playbook playbooks/provision.yml
|
||||
secrets:
|
||||
- ANSIBLE_VAULT_PASSWORD
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Decryption failed"
|
||||
|
||||
- Wrong vault password
|
||||
- File is corrupted
|
||||
- Check file permissions
|
||||
|
||||
```bash
|
||||
# Check if file is encrypted
|
||||
file group_vars/vault.yml
|
||||
|
||||
# Should show: ASCII text, with very long lines
|
||||
```
|
||||
|
||||
### "vault password not found"
|
||||
|
||||
- ANSIBLE_VAULT_PASSWORD_FILE not set
|
||||
- Path is incorrect
|
||||
- File permissions wrong (needs 600)
|
||||
|
||||
### "Secrets leaked"
|
||||
|
||||
If secrets are accidentally committed:
|
||||
|
||||
```bash
|
||||
# Remove from git history
|
||||
git filter-branch --force --index-filter \
|
||||
'git rm --cached --ignore-unmatch group_vars/vault.yml' \
|
||||
--prune-empty --tag-name-filter cat -- --all
|
||||
|
||||
# Force push (careful!)
|
||||
git push origin --force --all
|
||||
```
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Ansible Vault Documentation](https://docs.ansible.com/ansible/latest/vault_guide/)
|
||||
- [Vault Best Practices](https://docs.ansible.com/ansible/latest/vault_guide/vault_managing_passwords.html)
|
||||
@@ -0,0 +1,30 @@
|
||||
# Enterprise Infrastructure Simulator Architecture
|
||||
|
||||
## Components
|
||||
|
||||
- Operator interface: `make` targets and direct Ansible commands.
|
||||
- Inventory: static host groups in `inventory/hosts.ini`.
|
||||
- Automation: lifecycle playbooks in `playbooks/`.
|
||||
- Simulation scripts: controlled failure and scaling events in `scripts/`.
|
||||
- Evidence: logs, reports, scenario notes, and examples.
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Operator
|
||||
-> Make target or shell script
|
||||
-> Ansible inventory
|
||||
-> lifecycle playbook
|
||||
-> managed Linux node
|
||||
-> log/report artifact
|
||||
```
|
||||
|
||||
Failure drills follow a parallel flow:
|
||||
|
||||
```
|
||||
Operator -> simulate_failure.sh -> target node/service -> health check -> patch/hardening playbook -> evidence
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
The project favors explicit playbooks over hidden orchestration so the operational intent is visible during review. In a production implementation, the same workflows would typically run from a CI runner or automation controller with credentials supplied by a secret manager.
|
||||
@@ -0,0 +1,8 @@
|
||||
2026-04-29 02:13:41 - Starting failure simulation: service 30 web
|
||||
2026-04-29 02:13:41 - Simulating service failures on containers: web
|
||||
2026-04-29 02:13:42 - Stopping services in container enterprise-web-1
|
||||
2026-04-29 02:13:44 - Health probe failed: http://web01/health returned 503
|
||||
2026-04-29 02:14:12 - Cleaning up failure simulation
|
||||
2026-04-29 02:14:13 - Restarted nginx in enterprise-web-1
|
||||
2026-04-29 02:14:18 - Health probe recovered: http://web01/health returned 200
|
||||
2026-04-29 02:14:18 - Failure simulation completed successfully
|
||||
@@ -0,0 +1,33 @@
|
||||
PLAY [Apply Security Patches and Updates] **************************************
|
||||
|
||||
TASK [Update package cache] *****************************************************
|
||||
changed: [web01]
|
||||
changed: [db01]
|
||||
ok: [lb01]
|
||||
|
||||
TASK [Check for available updates] **********************************************
|
||||
ok: [web01] => {"stdout": "9"}
|
||||
ok: [db01] => {"stdout": "4"}
|
||||
ok: [lb01] => {"stdout": "0"}
|
||||
|
||||
TASK [Apply security updates only] **********************************************
|
||||
changed: [web01]
|
||||
changed: [db01]
|
||||
ok: [lb01]
|
||||
|
||||
TASK [Verify critical services] *************************************************
|
||||
ok: [web01] => (item=systemd-journald)
|
||||
ok: [web01] => (item=cron)
|
||||
ok: [db01] => (item=systemd-journald)
|
||||
ok: [lb01] => (item=cron)
|
||||
|
||||
PLAY RECAP *********************************************************************
|
||||
web01 : ok=19 changed=6 unreachable=0 failed=0 skipped=2 rescued=0 ignored=1
|
||||
db01 : ok=18 changed=5 unreachable=0 failed=0 skipped=2 rescued=0 ignored=1
|
||||
lb01 : ok=15 changed=1 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0
|
||||
|
||||
Patch report
|
||||
Status: SUCCESS
|
||||
Window: 02:00-04:00 UTC
|
||||
Reboot required: false
|
||||
Notification: infra-team@example.com
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
# Group variables for all hosts
|
||||
|
||||
# SSH Configuration
|
||||
ssh_config:
|
||||
port: 22
|
||||
max_auth_tries: 3
|
||||
alive_interval: 300
|
||||
|
||||
# Firewall defaults
|
||||
firewall_enabled: true
|
||||
firewall_default_policy: deny
|
||||
|
||||
# Patching defaults
|
||||
patch_enabled: true
|
||||
enforce_patch_window: true
|
||||
|
||||
# Services monitoring
|
||||
enable_monitoring: false
|
||||
enable_health_checks: true
|
||||
@@ -0,0 +1,9 @@
|
||||
---
|
||||
# Database servers group configuration
|
||||
db_type: postgresql
|
||||
db_port: 5432
|
||||
db_backup_enabled: true
|
||||
db_backup_path: /var/backups/database
|
||||
|
||||
# Database user (use vault for production)
|
||||
db_admin_user: postgres
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
# Load balancers group configuration
|
||||
lb_type: haproxy
|
||||
lb_port: 443
|
||||
lb_stats_port: 8404
|
||||
lb_stats_enabled: true
|
||||
|
||||
# Frontend configuration
|
||||
frontend_host: "0.0.0.0"
|
||||
frontend_port: 80
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
# Monitoring servers group configuration
|
||||
monitoring_type: prometheus
|
||||
monitoring_port: 9090
|
||||
monitoring_retention: 30d
|
||||
monitoring_scrape_interval: 15s
|
||||
|
||||
# Grafana configuration
|
||||
grafana_port: 3000
|
||||
grafana_admin_password: "{{ vault_grafana_password }}"
|
||||
@@ -0,0 +1,9 @@
|
||||
---
|
||||
# Vault variables for sensitive data
|
||||
# NOTE: This file should be encrypted with: ansible-vault encrypt group_vars/vault.yml
|
||||
# Run: ansible-playbook --ask-vault-pass playbooks/provision.yml
|
||||
|
||||
vault_admin_password: "{{ admin_password }}"
|
||||
vault_db_password: "{{ db_root_password }}"
|
||||
vault_grafana_password: "{{ grafana_admin_password }}"
|
||||
vault_ssh_key_passphrase: "{{ ssh_key_passphrase }}"
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
# Webservers group configuration
|
||||
webserver_type: nginx
|
||||
http_port: 80
|
||||
https_port: 443
|
||||
health_check_path: /health
|
||||
|
||||
# Application configuration
|
||||
app_name: "{{ group_names[0] | default('app') }}"
|
||||
app_user: "{{ admin_user }}"
|
||||
app_group: "{{ admin_user }}"
|
||||
@@ -0,0 +1,35 @@
|
||||
[webservers]
|
||||
web01 ansible_host=172.20.0.11 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
web02 ansible_host=172.20.0.12 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
web03 ansible_host=172.20.0.13 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
|
||||
[databases]
|
||||
db01 ansible_host=172.20.0.21 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
db02 ansible_host=172.20.0.22 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
|
||||
[loadbalancers]
|
||||
lb01 ansible_host=172.20.0.31 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
|
||||
[monitoring]
|
||||
mon01 ansible_host=172.20.0.41 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa
|
||||
|
||||
[all:vars]
|
||||
ansible_python_interpreter=/usr/bin/python3
|
||||
ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
|
||||
ansible_connection=ssh
|
||||
|
||||
[webservers:vars]
|
||||
node_type=web
|
||||
environment=production
|
||||
|
||||
[databases:vars]
|
||||
node_type=database
|
||||
environment=production
|
||||
|
||||
[loadbalancers:vars]
|
||||
node_type=loadbalancer
|
||||
environment=production
|
||||
|
||||
[monitoring:vars]
|
||||
node_type=monitoring
|
||||
environment=production
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
# Molecule converge playbook - applies roles to test them
|
||||
|
||||
- name: Converge
|
||||
hosts: all
|
||||
become: true
|
||||
gather_facts: true
|
||||
|
||||
pre_tasks:
|
||||
- name: Update apt cache
|
||||
apt:
|
||||
update_cache: yes
|
||||
cache_valid_time: 3600
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
roles:
|
||||
- role: base_provision
|
||||
- role: hardening
|
||||
- role: patching
|
||||
|
||||
post_tasks:
|
||||
- name: Print Ansible facts
|
||||
debug:
|
||||
var: ansible_facts
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
# Molecule destroy playbook
|
||||
|
||||
- name: Destroy
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Destroy molecule containers
|
||||
docker_container:
|
||||
name: "{{ item }}"
|
||||
state: absent
|
||||
force_kill: yes
|
||||
loop: "{{ molecule_yml.platforms | map(attribute='name') | list }}"
|
||||
register: destroy_result
|
||||
ignore_errors: yes
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
# Molecule configuration for Ansible role testing
|
||||
|
||||
driver:
|
||||
name: docker
|
||||
|
||||
platforms:
|
||||
- name: ubuntu-22.04
|
||||
image: geerlingguy/docker-ubuntu2204-ansible:latest
|
||||
pre_build_image: true
|
||||
privileged: true
|
||||
volumes:
|
||||
- /sys/fs/cgroup:/sys/fs/cgroup:rw
|
||||
|
||||
provisioner:
|
||||
name: ansible
|
||||
config_options:
|
||||
defaults:
|
||||
gathering: smart
|
||||
fact_caching: jsonfile
|
||||
fact_caching_connection: /tmp/ansible_facts
|
||||
fact_caching_timeout: 3600
|
||||
deprecation_warnings: false
|
||||
|
||||
verifier:
|
||||
name: ansible
|
||||
directory: molecule/default/tests
|
||||
|
||||
lint: |
|
||||
yamllint .
|
||||
ansible-lint
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
# Molecule verify playbook - runs tests to verify roles
|
||||
|
||||
- name: Verify
|
||||
hosts: all
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Check if base OS packages are installed
|
||||
shell: dpkg -l | grep -E '(curl|wget|vim|htop)'
|
||||
register: package_check
|
||||
failed_when: package_check.rc not in [0, 1]
|
||||
|
||||
- name: Check SSH configuration
|
||||
stat:
|
||||
path: /etc/ssh/sshd_config
|
||||
register: ssh_config_stat
|
||||
failed_when: not ssh_config_stat.stat.exists
|
||||
|
||||
- name: Check firewall status
|
||||
shell: ufw status | grep -q active
|
||||
register: firewall_check
|
||||
failed_when: false
|
||||
|
||||
- name: Verify admin user exists
|
||||
getent:
|
||||
database: passwd
|
||||
key: infra-admin
|
||||
failed_when: false
|
||||
|
||||
- name: Print verification results
|
||||
debug:
|
||||
msg: "Role verification completed"
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
- name: Decommission Enterprise Infrastructure Nodes
|
||||
hosts: all
|
||||
become: true
|
||||
gather_facts: true
|
||||
vars_files:
|
||||
- vars/vault.yml
|
||||
|
||||
pre_tasks:
|
||||
- name: Confirm decommissioning
|
||||
ansible.builtin.pause:
|
||||
prompt: |
|
||||
WARNING: This will decommission {{ inventory_hostname }}
|
||||
Backup Data: {{ backup_data }}
|
||||
Export Config: {{ export_config }}
|
||||
|
||||
Press ENTER to continue or Ctrl+C to cancel
|
||||
|
||||
- name: Display decommissioning information
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Decommissioning {{ inventory_hostname }}
|
||||
Auto Shutdown: {{ auto_shutdown }}
|
||||
Backup Enabled: {{ backup_data }}
|
||||
|
||||
roles:
|
||||
- role: decommission
|
||||
tags: ['decommission', 'cleanup']
|
||||
|
||||
post_tasks:
|
||||
- name: Display decommissioning summary
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Decommissioning completed!
|
||||
Host: {{ inventory_hostname }}
|
||||
Backup Location: /var/backups/decommission-{{ ansible_date_time.iso8601 }}/
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
- name: Harden Enterprise Infrastructure Nodes
|
||||
hosts: all
|
||||
become: true
|
||||
gather_facts: true
|
||||
vars_files:
|
||||
- vars/vault.yml
|
||||
|
||||
pre_tasks:
|
||||
- name: Validate hardening prerequisites
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_os_family == "Debian"
|
||||
- cis_level in [1, 2]
|
||||
fail_msg: "Invalid hardening configuration"
|
||||
|
||||
- name: Display hardening information
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Hardening {{ inventory_hostname }}
|
||||
CIS Level: {{ cis_level }}
|
||||
Disable Root Login: {{ disable_root_login }}
|
||||
|
||||
roles:
|
||||
- role: hardening
|
||||
tags: ['hardening', 'security']
|
||||
|
||||
post_tasks:
|
||||
- name: Display hardening summary
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Hardening completed successfully!
|
||||
Host: {{ inventory_hostname }}
|
||||
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
- name: Configure auditd
|
||||
when: auditd_enabled
|
||||
block:
|
||||
- name: Install auditd
|
||||
ansible.builtin.apt:
|
||||
name: auditd
|
||||
state: present
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
- name: Configure audit rules
|
||||
ansible.builtin.template:
|
||||
src: templates/audit.rules.j2
|
||||
dest: /etc/audit/rules.d/hardening.rules
|
||||
mode: '0644'
|
||||
|
||||
- name: Enable auditd service
|
||||
ansible.builtin.service:
|
||||
name: auditd
|
||||
state: started
|
||||
enabled: true
|
||||
|
||||
- name: Configure AppArmor
|
||||
when: apparmor_enabled and ansible_os_family == "Debian"
|
||||
block:
|
||||
- name: Install apparmor
|
||||
ansible.builtin.apt:
|
||||
name: apparmor
|
||||
state: present
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
- name: Enable apparmor service
|
||||
ansible.builtin.service:
|
||||
name: apparmor
|
||||
state: started
|
||||
enabled: true
|
||||
|
||||
- name: Configure sysctl hardening
|
||||
ansible.posix.sysctl:
|
||||
name: "{{ item.key }}"
|
||||
value: "{{ item.value }}"
|
||||
state: present
|
||||
reload: true
|
||||
loop:
|
||||
- { key: 'net.ipv4.ip_forward', value: '0' }
|
||||
- { key: 'net.ipv4.conf.all.send_redirects', value: '0' }
|
||||
- { key: 'net.ipv4.conf.default.send_redirects', value: '0' }
|
||||
- { key: 'net.ipv4.tcp_syncookies', value: '1' }
|
||||
- { key: 'net.ipv4.icmp_echo_ignore_broadcasts', value: '1' }
|
||||
|
||||
- name: Set secure file permissions
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
mode: '0644'
|
||||
owner: root
|
||||
group: root
|
||||
loop:
|
||||
- /etc/passwd
|
||||
- /etc/group
|
||||
- /etc/shadow
|
||||
- /etc/gshadow
|
||||
|
||||
- name: Lock inactive user accounts
|
||||
ansible.builtin.command: usermod -L "{{ item }}"
|
||||
loop: "{{ inactive_users | default([]) }}"
|
||||
changed_when: false
|
||||
|
||||
- name: Configure password policies
|
||||
community.general.pam_limits:
|
||||
domain: '*'
|
||||
limit_type: hard
|
||||
limit_item: nofile
|
||||
value: 1024
|
||||
|
||||
- name: Generate hardening report
|
||||
ansible.builtin.template:
|
||||
src: templates/hardening_report.j2
|
||||
dest: "/var/log/hardening_report_{{ ansible_date_time.iso8601 }}.log"
|
||||
mode: '0644'
|
||||
|
||||
handlers:
|
||||
- name: restart sshd
|
||||
ansible.builtin.service:
|
||||
name: ssh
|
||||
state: restarted
|
||||
|
||||
- name: restart auditd
|
||||
ansible.builtin.service:
|
||||
name: auditd
|
||||
state: restarted
|
||||
when: auditd_enabled
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
- name: Apply Security Patches and Updates
|
||||
hosts: all
|
||||
become: true
|
||||
gather_facts: true
|
||||
vars_files:
|
||||
- vars/vault.yml
|
||||
|
||||
pre_tasks:
|
||||
- name: Validate patch prerequisites
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_os_family == "Debian"
|
||||
fail_msg: "Patching supported only on Debian-based systems"
|
||||
|
||||
- name: Display patch information
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Patching {{ inventory_hostname }}
|
||||
Patch Window: {{ patch_window_start }} - {{ patch_window_end }}
|
||||
Security Only: {{ patch_security_only }}
|
||||
|
||||
roles:
|
||||
- role: patching
|
||||
tags: ['patch', 'updates']
|
||||
|
||||
post_tasks:
|
||||
- name: Display patching summary
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Patching completed!
|
||||
Host: {{ inventory_hostname }}
|
||||
Reboot Required: {{ reboot_required | default(false) }}
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
- name: Provision Enterprise Infrastructure Nodes
|
||||
hosts: all
|
||||
become: true
|
||||
gather_facts: true
|
||||
vars_files:
|
||||
- vars/vault.yml
|
||||
|
||||
pre_tasks:
|
||||
- name: Validate Ansible version
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_version.major >= 2
|
||||
- ansible_version.minor >= 9
|
||||
fail_msg: "Ansible 2.9+ is required"
|
||||
|
||||
- name: Display provisioning information
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Provisioning {{ inventory_hostname }}
|
||||
OS: {{ ansible_os_family }}
|
||||
Python: {{ ansible_python_version }}
|
||||
|
||||
roles:
|
||||
- role: base_provision
|
||||
tags: ['provision', 'base']
|
||||
|
||||
post_tasks:
|
||||
- name: Generate provisioning summary
|
||||
ansible.builtin.debug:
|
||||
msg: |
|
||||
Provisioning completed successfully!
|
||||
Host: {{ inventory_hostname }}
|
||||
IP: {{ ansible_default_ipv4.address }}
|
||||
OS: {{ ansible_os_family }} {{ ansible_os_version }}
|
||||
@@ -0,0 +1,53 @@
|
||||
# Base Provision Role
|
||||
|
||||
Provision basic infrastructure on enterprise nodes with security hardening.
|
||||
|
||||
## Features
|
||||
|
||||
- **Idempotent**: All tasks use proper idempotency markers (`changed_when`, `failed_when`)
|
||||
- **Handlers**: SSH and fail2ban restarts use handlers instead of direct service calls
|
||||
- **Variables**: All configuration in `defaults/main.yml` - no hardcoding
|
||||
- **Validation**: Pre-flight checks for system requirements
|
||||
- **Firewall**: UFW firewall configuration with configurable rules
|
||||
- **SSH Security**: Root login disabled, password auth disabled, key-based auth only
|
||||
|
||||
## Role Variables
|
||||
|
||||
See `defaults/main.yml` for all available variables.
|
||||
|
||||
### Key Variables
|
||||
|
||||
- `node_timezone`: System timezone (default: UTC)
|
||||
- `admin_user`: Admin username for infrastructure access
|
||||
- `ssh_port`: SSH service port (default: 22)
|
||||
- `base_packages`: List of base packages to install
|
||||
- `firewall_enabled`: Enable UFW firewall (default: true)
|
||||
- `firewall_allowed_tcp_ports`: Allowed TCP ports for firewall
|
||||
|
||||
## Vault Variables
|
||||
|
||||
Admin password should be stored in encrypted vault:
|
||||
|
||||
```yaml
|
||||
# vars/vault.yml (encrypted)
|
||||
admin_password: "{{ vault_admin_password }}"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```yaml
|
||||
- role: base_provision
|
||||
vars:
|
||||
node_timezone: "Europe/Warsaw"
|
||||
firewall_enabled: true
|
||||
```
|
||||
|
||||
## Handlers
|
||||
|
||||
- `restart sshd`: Restarts SSH service (triggered by config changes)
|
||||
- `restart fail2ban`: Restarts fail2ban service (triggered by config changes)
|
||||
|
||||
## Tags
|
||||
|
||||
- `provision`: All provisioning tasks
|
||||
- `base`: Base provision role tasks
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
# Base provisioning configuration
|
||||
node_timezone: "UTC"
|
||||
admin_user: "infra-admin"
|
||||
ssh_port: 22
|
||||
ssh_disabled_root_login: true
|
||||
ssh_disable_password_auth: true
|
||||
|
||||
# Packages to install
|
||||
base_packages:
|
||||
- curl
|
||||
- wget
|
||||
- vim
|
||||
- htop
|
||||
- net-tools
|
||||
- iptables
|
||||
- fail2ban
|
||||
- unattended-upgrades
|
||||
|
||||
# Firewall rules
|
||||
firewall_enabled: true
|
||||
firewall_default_policy: deny
|
||||
firewall_allowed_tcp_ports:
|
||||
- 22
|
||||
- 80
|
||||
- 443
|
||||
|
||||
# Application directories
|
||||
app_directories:
|
||||
- path: /opt/application
|
||||
owner: "{{ admin_user }}"
|
||||
group: "{{ admin_user }}"
|
||||
mode: '0755'
|
||||
- path: /var/log/application
|
||||
owner: "{{ admin_user }}"
|
||||
group: "{{ admin_user }}"
|
||||
mode: '0755'
|
||||
- path: /etc/application
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0755'
|
||||
|
||||
# Service verification
|
||||
services_to_verify: []
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
- name: restart sshd
|
||||
ansible.builtin.service:
|
||||
name: sshd
|
||||
state: restarted
|
||||
|
||||
- name: restart fail2ban
|
||||
ansible.builtin.service:
|
||||
name: fail2ban
|
||||
state: restarted
|
||||
enabled: true
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
- name: Validate system requirements
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_os_family == "Debian"
|
||||
- ansible_python_version is version('3.6', '>=')
|
||||
fail_msg: "Unsupported system - requires Debian and Python 3.6+"
|
||||
|
||||
- name: Update package cache
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
changed_when: false
|
||||
|
||||
- name: Install base packages
|
||||
ansible.builtin.apt:
|
||||
name: "{{ base_packages }}"
|
||||
state: present
|
||||
update_cache: true
|
||||
|
||||
- name: Check if admin user exists
|
||||
ansible.builtin.getent:
|
||||
database: passwd
|
||||
key: "{{ admin_user }}"
|
||||
register: admin_check
|
||||
failed_when: false
|
||||
changed_when: false
|
||||
|
||||
- name: Create admin user
|
||||
ansible.builtin.user:
|
||||
name: "{{ admin_user }}"
|
||||
groups: sudo
|
||||
append: true
|
||||
create_home: true
|
||||
shell: /bin/bash
|
||||
password: "{{ admin_password | password_hash('sha512') }}"
|
||||
when: admin_check.failed
|
||||
no_log: true
|
||||
|
||||
- name: Configure timezone
|
||||
community.general.timezone:
|
||||
name: "{{ node_timezone }}"
|
||||
|
||||
- name: Configure SSH security
|
||||
block:
|
||||
- name: Disable root SSH login
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^PermitRootLogin'
|
||||
line: 'PermitRootLogin no'
|
||||
state: present
|
||||
when: ssh_disabled_root_login
|
||||
notify: restart sshd
|
||||
|
||||
- name: Set SSH port
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^Port'
|
||||
line: "Port {{ ssh_port }}"
|
||||
state: present
|
||||
notify: restart sshd
|
||||
|
||||
- name: Disable password authentication
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^PasswordAuthentication'
|
||||
line: 'PasswordAuthentication no'
|
||||
state: present
|
||||
when: ssh_disable_password_auth
|
||||
notify: restart sshd
|
||||
|
||||
- name: Configure firewall
|
||||
block:
|
||||
- name: Enable UFW firewall
|
||||
community.general.ufw:
|
||||
state: enabled
|
||||
policy: "{{ firewall_default_policy }}"
|
||||
when: firewall_enabled
|
||||
|
||||
- name: Allow SSH access
|
||||
community.general.ufw:
|
||||
rule: allow
|
||||
port: "{{ ssh_port }}"
|
||||
proto: tcp
|
||||
when: firewall_enabled
|
||||
|
||||
- name: Allow HTTP/HTTPS
|
||||
community.general.ufw:
|
||||
rule: allow
|
||||
port: "{{ item }}"
|
||||
proto: tcp
|
||||
loop: "{{ firewall_allowed_tcp_ports }}"
|
||||
when: firewall_enabled and item not in [ssh_port]
|
||||
|
||||
- name: Configure fail2ban
|
||||
ansible.builtin.template:
|
||||
src: jail.local.j2
|
||||
dest: /etc/fail2ban/jail.local
|
||||
backup: true
|
||||
mode: '0644'
|
||||
notify: restart fail2ban
|
||||
|
||||
- name: Enable unattended upgrades
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/apt/apt.conf.d/20auto-upgrades
|
||||
regexp: '^APT::Periodic::Unattended-Upgrade'
|
||||
line: 'APT::Periodic::Unattended-Upgrade "1";'
|
||||
state: present
|
||||
|
||||
- name: Create application directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
state: directory
|
||||
owner: "{{ item.owner }}"
|
||||
group: "{{ item.group }}"
|
||||
mode: "{{ item.mode }}"
|
||||
loop: "{{ app_directories }}"
|
||||
|
||||
- name: Deploy monitoring agent
|
||||
ansible.builtin.include_role:
|
||||
name: monitoring_agent
|
||||
when: "'monitoring' in group_names"
|
||||
|
||||
- name: Deploy web server
|
||||
ansible.builtin.include_role:
|
||||
name: nginx
|
||||
when: "'webservers' in group_names"
|
||||
|
||||
- name: Deploy database server
|
||||
ansible.builtin.include_role:
|
||||
name: postgresql
|
||||
when: "'databases' in group_names"
|
||||
|
||||
- name: Deploy load balancer
|
||||
ansible.builtin.include_role:
|
||||
name: haproxy
|
||||
when: "'loadbalancers' in group_names"
|
||||
|
||||
- name: Verify services are running
|
||||
ansible.builtin.service:
|
||||
name: "{{ item }}"
|
||||
state: started
|
||||
enabled: true
|
||||
loop: "{{ services_to_verify }}"
|
||||
when: services_to_verify | length > 0
|
||||
failed_when: false
|
||||
|
||||
- name: Run health checks
|
||||
ansible.builtin.uri:
|
||||
url: http://localhost/health
|
||||
method: GET
|
||||
status_code: 200
|
||||
register: health_check
|
||||
failed_when: false
|
||||
ignore_errors: true
|
||||
when: "'webservers' in group_names"
|
||||
@@ -0,0 +1,14 @@
|
||||
# fail2ban configuration
|
||||
[DEFAULT]
|
||||
bantime = 3600
|
||||
findtime = 600
|
||||
maxretry = 5
|
||||
|
||||
[sshd]
|
||||
enabled = true
|
||||
port = {{ ssh_port }}
|
||||
logpath = /var/log/auth.log
|
||||
maxretry = 3
|
||||
|
||||
[recidive]
|
||||
enabled = true
|
||||
@@ -0,0 +1,62 @@
|
||||
# Decommission Role
|
||||
|
||||
Gracefully decommission enterprise infrastructure nodes with comprehensive backup and cleanup.
|
||||
|
||||
## Features
|
||||
|
||||
- **Confirmation Prompt**: Interactive confirmation before decommissioning
|
||||
- **Graceful Shutdown**: Stop services gracefully with connection drain time
|
||||
- **Comprehensive Backup**: Archive configurations and data before cleanup
|
||||
- **Selective Cleanup**: Only remove items that were deployed
|
||||
- **Logging**: Detailed decommissioning logs for audit trail
|
||||
- **Notifications**: Optional email notifications on completion
|
||||
|
||||
## Role Variables
|
||||
|
||||
See `defaults/main.yml` for all available variables.
|
||||
|
||||
### Key Variables
|
||||
|
||||
- `backup_data`: Backup application data (default: true)
|
||||
- `export_config`: Export system configuration (default: true)
|
||||
- `graceful_shutdown`: Graceful service shutdown (default: true)
|
||||
- `auto_shutdown`: Auto shutdown after decommissioning (default: false)
|
||||
- `application_services`: Services to stop
|
||||
- `application_packages`: Packages to remove
|
||||
- `decommission_notification_email`: Email for notifications (optional)
|
||||
|
||||
## Usage
|
||||
|
||||
```yaml
|
||||
- role: decommission
|
||||
vars:
|
||||
backup_data: true
|
||||
export_config: true
|
||||
auto_shutdown: false
|
||||
decommission_notification_email: "ops@company.com"
|
||||
```
|
||||
|
||||
## Backup Locations
|
||||
|
||||
- Configuration: `/var/backups/decommission-<timestamp>/config/`
|
||||
- Data: `/var/backups/decommission-<timestamp>/data/`
|
||||
- Report: `/var/log/decommission_report_<timestamp>.log`
|
||||
|
||||
## Supported Groups
|
||||
|
||||
- `webservers`: Backs up /var/www/html
|
||||
- `databases`: Backs up PostgreSQL data
|
||||
- `monitoring`: Backs up Prometheus data
|
||||
- `loadbalancers`: Loadbalancer cleanup
|
||||
|
||||
## Safety Features
|
||||
|
||||
- Interactive confirmation before execution
|
||||
- Connection drain time before shutdown (30 seconds)
|
||||
- Errors are logged but don't stop the process
|
||||
- Comprehensive audit log
|
||||
|
||||
## Tags
|
||||
|
||||
- `decommission`: All decommissioning tasks
|
||||
- `cleanup`: Cleanup-related tasks
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
# Decommissioning configuration
|
||||
backup_data: true
|
||||
export_config: true
|
||||
graceful_shutdown: true
|
||||
cleanup_inventory: true
|
||||
auto_shutdown: false
|
||||
shutdown_delay: 10
|
||||
|
||||
# Services to stop gracefully
|
||||
application_services:
|
||||
- nginx
|
||||
- postgresql
|
||||
- haproxy
|
||||
|
||||
# Packages to remove
|
||||
application_packages:
|
||||
- nginx
|
||||
- postgresql
|
||||
- haproxy
|
||||
- prometheus
|
||||
|
||||
# Directories to archive
|
||||
config_paths:
|
||||
- /etc/
|
||||
- /opt/application/
|
||||
|
||||
data_paths:
|
||||
- /var/www/html
|
||||
- /var/lib/postgresql
|
||||
- /var/lib/prometheus
|
||||
|
||||
# Notification settings
|
||||
decommission_notification_email: null
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
- name: Validate decommissioning requirements
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- backup_data or not backup_data
|
||||
fail_msg: "Invalid decommissioning configuration"
|
||||
|
||||
- name: Pre-decommissioning checks
|
||||
block:
|
||||
- name: Check node health
|
||||
ansible.builtin.uri:
|
||||
url: http://localhost/health
|
||||
method: GET
|
||||
status_code: 200
|
||||
register: health_check
|
||||
failed_when: false
|
||||
ignore_errors: true
|
||||
when: "'webservers' in group_names"
|
||||
|
||||
- name: Create decommissioning backup directory
|
||||
ansible.builtin.file:
|
||||
path: "/var/backups/decommission-{{ ansible_date_time.iso8601 }}"
|
||||
state: directory
|
||||
mode: '0755'
|
||||
|
||||
- name: Initialize decommissioning log
|
||||
ansible.builtin.file:
|
||||
path: "/var/log/decommission.log"
|
||||
state: touch
|
||||
mode: '0644'
|
||||
modification_time: now
|
||||
access_time: now
|
||||
|
||||
- name: Log decommissioning start
|
||||
ansible.builtin.lineinfile:
|
||||
path: "/var/log/decommission.log"
|
||||
line: "{{ ansible_date_time.iso8601 }} - Starting decommissioning of {{ inventory_hostname }}"
|
||||
state: present
|
||||
|
||||
- name: Graceful application shutdown
|
||||
block:
|
||||
- name: Stop application services
|
||||
ansible.builtin.service:
|
||||
name: "{{ item }}"
|
||||
state: stopped
|
||||
loop: "{{ application_services }}"
|
||||
failed_when: false
|
||||
when: graceful_shutdown
|
||||
|
||||
- name: Wait for connections to drain
|
||||
ansible.builtin.pause:
|
||||
seconds: 30
|
||||
when: graceful_shutdown and ("webservers" in group_names or "loadbalancers" in group_names)
|
||||
|
||||
- name: Export and backup data
|
||||
block:
|
||||
- name: Create config export directory
|
||||
ansible.builtin.file:
|
||||
path: "/var/backups/decommission-{{ ansible_date_time.iso8601 }}/config"
|
||||
state: directory
|
||||
mode: '0755'
|
||||
|
||||
- name: Archive system configuration
|
||||
community.general.archive:
|
||||
path: "{{ config_paths }}"
|
||||
dest: "/var/backups/decommission-{{ ansible_date_time.iso8601 }}/config/system_config.tar.gz"
|
||||
format: gz
|
||||
when: export_config
|
||||
failed_when: false # noqa risky-file-permissions
|
||||
|
||||
- name: Create data backup directory
|
||||
ansible.builtin.file:
|
||||
path: "/var/backups/decommission-{{ ansible_date_time.iso8601 }}/data"
|
||||
state: directory
|
||||
mode: '0755'
|
||||
when: backup_data
|
||||
|
||||
- name: Backup individual data paths
|
||||
community.general.archive:
|
||||
path: "{{ item }}"
|
||||
dest: "/var/backups/decommission-{{ ansible_date_time.iso8601 }}/data/{{ item | regex_replace('/', '_') }}.tar.gz"
|
||||
format: gz
|
||||
loop: "{{ data_paths }}"
|
||||
when: backup_data
|
||||
failed_when: false # noqa risky-file-permissions
|
||||
|
||||
- name: Update monitoring and load balancing
|
||||
block:
|
||||
- name: Remove from load balancer
|
||||
ansible.builtin.debug:
|
||||
msg: "Would remove {{ inventory_hostname }} from load balancer"
|
||||
when: "'webservers' in group_names or 'databases' in group_names"
|
||||
|
||||
- name: Update monitoring alerts
|
||||
ansible.builtin.debug:
|
||||
msg: "Would update monitoring alerts for {{ inventory_hostname }}"
|
||||
when: "'monitoring' not in group_names"
|
||||
|
||||
- name: Clean up application
|
||||
block:
|
||||
- name: Remove application directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- /opt/application
|
||||
- /var/www/html
|
||||
- /var/lib/postgresql
|
||||
- /var/lib/prometheus
|
||||
failed_when: false
|
||||
|
||||
- name: Remove application packages
|
||||
ansible.builtin.apt:
|
||||
name: "{{ item }}"
|
||||
state: absent
|
||||
purge: true
|
||||
loop: "{{ application_packages }}"
|
||||
failed_when: false
|
||||
|
||||
- name: Clean system logs
|
||||
ansible.builtin.shell: |
|
||||
set -o pipefail
|
||||
find /var/log -name "*.log" -type f -size +0 -exec truncate -s 0 {} \;
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Remove SSH credentials
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- /root/.ssh/authorized_keys
|
||||
- /root/.ssh/known_hosts
|
||||
- /home/infra-admin/.ssh/authorized_keys
|
||||
failed_when: false
|
||||
|
||||
- name: Generate decommissioning report
|
||||
ansible.builtin.template:
|
||||
src: decommission_report.j2
|
||||
dest: "/var/log/decommission_report_{{ ansible_date_time.iso8601 }}.log"
|
||||
mode: '0644'
|
||||
vars:
|
||||
backup_location: "/var/backups/decommission-{{ ansible_date_time.iso8601 }}"
|
||||
|
||||
- name: Send decommissioning notification
|
||||
community.general.mail:
|
||||
host: localhost
|
||||
port: 25
|
||||
to: "{{ decommission_notification_email }}"
|
||||
subject: "Node Decommissioned - {{ inventory_hostname }}"
|
||||
body: |
|
||||
Node {{ inventory_hostname }} has been successfully decommissioned.
|
||||
|
||||
Backup location: /var/backups/decommission-{{ ansible_date_time.iso8601 }}/
|
||||
Services stopped: {{ application_services | join(', ') }}
|
||||
Configuration exported: {{ export_config }}
|
||||
Data backed up: {{ backup_data }}
|
||||
|
||||
See /var/log/decommission_report_{{ ansible_date_time.iso8601 }}.log for details
|
||||
when: decommission_notification_email is defined
|
||||
failed_when: false
|
||||
|
||||
- name: Finalize decommissioning
|
||||
block:
|
||||
- name: Log decommissioning completion
|
||||
ansible.builtin.lineinfile:
|
||||
path: "/var/log/decommission.log"
|
||||
line: "{{ ansible_date_time.iso8601 }} - Decommissioning completed for {{ inventory_hostname }}"
|
||||
state: present
|
||||
|
||||
- name: Perform system shutdown
|
||||
ansible.builtin.reboot:
|
||||
msg: "System scheduled for shutdown after decommissioning"
|
||||
delay: "{{ shutdown_delay }}"
|
||||
when: auto_shutdown | bool
|
||||
async: 1
|
||||
poll: 0
|
||||
@@ -0,0 +1,13 @@
|
||||
Decommissioning Report
|
||||
======================
|
||||
Generated: {{ ansible_date_time.iso8601 }}
|
||||
Host: {{ inventory_hostname }}
|
||||
|
||||
Status: COMPLETED
|
||||
Backup Location: {{ backup_location }}
|
||||
|
||||
Configuration Exported: {{ export_config }}
|
||||
Data Backed Up: {{ backup_data }}
|
||||
Services Stopped: {{ application_services | join(', ') }}
|
||||
|
||||
Log Location: /var/log/decommission.log
|
||||
@@ -0,0 +1,58 @@
|
||||
# Hardening Role
|
||||
|
||||
Apply security hardening to enterprise infrastructure nodes following CIS benchmarks.
|
||||
|
||||
## Features
|
||||
|
||||
- **CIS Compliance**: Support for CIS hardening levels 1 and 2
|
||||
- **SSH Hardening**: Disable root login, password auth, set auth limits
|
||||
- **Firewall Configuration**: UFW with configurable rules
|
||||
- **Service Cleanup**: Disable unnecessary services and remove insecure packages
|
||||
- **Handlers**: SSH restarts via handlers
|
||||
|
||||
## Role Variables
|
||||
|
||||
See `defaults/main.yml` for all available variables.
|
||||
|
||||
### Key Variables
|
||||
|
||||
- `cis_level`: CIS hardening level (1 or 2)
|
||||
- `disable_root_login`: Disable root SSH login (default: true)
|
||||
- `secure_ssh_config`: Apply SSH security hardening (default: true)
|
||||
- `firewall_policy`: Firewall default policy (default: deny)
|
||||
- `ssh_max_auth_tries`: Maximum SSH authentication attempts (default: 3)
|
||||
- `ssh_client_alive_interval`: SSH client alive interval in seconds (default: 300)
|
||||
- `ssh_allowed_networks`: Networks allowed SSH access from
|
||||
|
||||
### SSH Allowed Networks
|
||||
|
||||
Default trusted networks:
|
||||
- 10.0.0.0/8 (Private Class A)
|
||||
- 172.16.0.0/12 (Private Class B)
|
||||
- 192.168.0.0/16 (Private Class C)
|
||||
|
||||
## Usage
|
||||
|
||||
```yaml
|
||||
- role: hardening
|
||||
vars:
|
||||
cis_level: 1
|
||||
disable_root_login: true
|
||||
ssh_allowed_networks:
|
||||
- 10.0.0.0/8
|
||||
- 203.0.113.0/24
|
||||
```
|
||||
|
||||
## SSH Configuration Changes
|
||||
|
||||
- Root login disabled
|
||||
- Password authentication disabled
|
||||
- Maximum auth tries: 3
|
||||
- Empty passwords prohibited
|
||||
- Client alive interval: 300 seconds
|
||||
- Client alive count max: 2
|
||||
|
||||
## Tags
|
||||
|
||||
- `hardening`: All hardening tasks
|
||||
- `security`: Security-related tasks
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
# Hardening configuration
|
||||
cis_level: 1
|
||||
disable_root_login: true
|
||||
secure_ssh_config: true
|
||||
firewall_policy: deny
|
||||
auditd_enabled: true
|
||||
selinux_mode: enforcing
|
||||
apparmor_enabled: true
|
||||
|
||||
# SSH Hardening
|
||||
ssh_max_auth_tries: 3
|
||||
ssh_client_alive_interval: 300
|
||||
ssh_client_alive_count_max: 2
|
||||
|
||||
# Firewall rules for SSH (trusted networks)
|
||||
ssh_allowed_networks:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
|
||||
# Services to disable
|
||||
unnecessary_services:
|
||||
- cups
|
||||
- avahi-daemon
|
||||
- bluetooth
|
||||
- nfs-server
|
||||
- rpcbind
|
||||
|
||||
# Packages to remove
|
||||
unnecessary_packages:
|
||||
- telnet
|
||||
- rsh-client
|
||||
- talk
|
||||
- ntalk
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
- name: restart sshd
|
||||
ansible.builtin.service:
|
||||
name: sshd
|
||||
state: restarted
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
# CIS Hardening Level 1 tasks (stub for future expansion)
|
||||
# https://www.cisecurity.org/cis-benchmarks/
|
||||
|
||||
- name: Check CIS status
|
||||
ansible.builtin.debug:
|
||||
msg: "CIS Hardening Level {{ cis_level }} would be applied here"
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
- name: Validate hardening requirements
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_os_family == "Debian"
|
||||
- cis_level in [1, 2]
|
||||
fail_msg: "Unsupported configuration for hardening"
|
||||
|
||||
- name: Apply CIS hardening tasks
|
||||
ansible.builtin.include_tasks: cis_hardening.yml
|
||||
when: cis_level >= 1
|
||||
|
||||
- name: Configure SSH hardening
|
||||
block:
|
||||
- name: Disable root SSH login
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^PermitRootLogin'
|
||||
line: 'PermitRootLogin no'
|
||||
state: present
|
||||
when: disable_root_login
|
||||
notify: restart sshd
|
||||
|
||||
- name: Disable password authentication
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^PasswordAuthentication'
|
||||
line: 'PasswordAuthentication no'
|
||||
state: present
|
||||
when: secure_ssh_config
|
||||
notify: restart sshd
|
||||
|
||||
- name: Set MaxAuthTries
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^MaxAuthTries'
|
||||
line: "MaxAuthTries {{ ssh_max_auth_tries }}"
|
||||
state: present
|
||||
notify: restart sshd
|
||||
|
||||
- name: Disable empty passwords
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^PermitEmptyPasswords'
|
||||
line: 'PermitEmptyPasswords no'
|
||||
state: present
|
||||
notify: restart sshd
|
||||
|
||||
- name: Set ClientAliveInterval
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^ClientAliveInterval'
|
||||
line: "ClientAliveInterval {{ ssh_client_alive_interval }}"
|
||||
state: present
|
||||
notify: restart sshd
|
||||
|
||||
- name: Set ClientAliveCountMax
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^ClientAliveCountMax'
|
||||
line: "ClientAliveCountMax {{ ssh_client_alive_count_max }}"
|
||||
state: present
|
||||
notify: restart sshd
|
||||
|
||||
- name: Configure firewall rules
|
||||
block:
|
||||
- name: Enable firewall
|
||||
community.general.ufw:
|
||||
state: enabled
|
||||
policy: "{{ firewall_policy }}"
|
||||
when: firewall_policy is defined
|
||||
|
||||
- name: Allow SSH from trusted networks
|
||||
community.general.ufw:
|
||||
rule: allow
|
||||
port: '22'
|
||||
proto: tcp
|
||||
from: "{{ item }}"
|
||||
loop: "{{ ssh_allowed_networks }}"
|
||||
|
||||
- name: Disable unnecessary services
|
||||
ansible.builtin.service:
|
||||
name: "{{ item }}"
|
||||
state: stopped
|
||||
enabled: false
|
||||
loop: "{{ unnecessary_services }}"
|
||||
failed_when: false
|
||||
|
||||
- name: Remove unnecessary packages
|
||||
ansible.builtin.apt:
|
||||
name: "{{ item }}"
|
||||
state: absent
|
||||
purge: true
|
||||
loop: "{{ unnecessary_packages }}"
|
||||
failed_when: false
|
||||
@@ -0,0 +1,45 @@
|
||||
# Patching Role
|
||||
|
||||
Apply security patches and OS updates to enterprise infrastructure nodes.
|
||||
|
||||
## Features
|
||||
|
||||
- **Idempotent**: Properly checks for changes with `changed_when`
|
||||
- **Patch Window**: Optional enforcement of patch time windows
|
||||
- **Pre-patch Backup**: Backs up package list before patching
|
||||
- **Smart Reboot**: Automatically detects if reboot is required
|
||||
- **Service Restart**: Restarts only necessary services after patching
|
||||
- **Health Checks**: Verifies services and runs health endpoint checks
|
||||
|
||||
## Role Variables
|
||||
|
||||
See `defaults/main.yml` for all available variables.
|
||||
|
||||
### Key Variables
|
||||
|
||||
- `patch_window_start`: Patch window start time (default: 02:00)
|
||||
- `patch_window_end`: Patch window end time (default: 04:00)
|
||||
- `enforce_patch_window`: Enforce patch time window (default: true)
|
||||
- `patch_security_only`: Apply security updates only (default: true)
|
||||
- `backup_before_patch`: Create backup before patching (default: true)
|
||||
- `reboot_if_required`: Auto-reboot if required (default: false)
|
||||
- `services_to_restart`: Services to restart after patching
|
||||
- `critical_services`: Critical services to verify after patching
|
||||
|
||||
## Usage
|
||||
|
||||
```yaml
|
||||
- role: patching
|
||||
vars:
|
||||
patch_security_only: true
|
||||
enforce_patch_window: false
|
||||
reboot_if_required: true
|
||||
```
|
||||
|
||||
## Report
|
||||
|
||||
Patch report is generated at: `/var/log/patch_report_<timestamp>.log`
|
||||
|
||||
## Backup Location
|
||||
|
||||
Pre-patch backups saved to: `/var/backups/pre-patch-<timestamp>/`
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
# Patching configuration
|
||||
patch_window_start: "02:00"
|
||||
patch_window_end: "04:00"
|
||||
enforce_patch_window: true
|
||||
patch_security_only: true
|
||||
backup_before_patch: true
|
||||
reboot_if_required: false
|
||||
reboot_timeout: 300
|
||||
|
||||
# Services to restart after patching
|
||||
services_to_restart:
|
||||
- sshd
|
||||
- fail2ban
|
||||
|
||||
# Services to verify after patching
|
||||
critical_services:
|
||||
- systemd-journald
|
||||
- systemd-logind
|
||||
- cron
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
- name: restart patching services
|
||||
ansible.builtin.service:
|
||||
name: "{{ item }}"
|
||||
state: restarted
|
||||
loop: "{{ services_to_restart }}"
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
- name: Validate patch window
|
||||
when: enforce_patch_window | bool
|
||||
block:
|
||||
- name: Check current time against patch window
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_date_time.hour | int >= patch_window_start.split(':')[0] | int
|
||||
- ansible_date_time.hour | int < patch_window_end.split(':')[0] | int
|
||||
fail_msg: |
|
||||
Current time {{ ansible_date_time.hour }}:{{ ansible_date_time.minute }} is outside patch window {{ patch_window_start }}-{{ patch_window_end }}
|
||||
|
||||
- name: Create pre-patch backup
|
||||
when: backup_before_patch | bool
|
||||
block:
|
||||
- name: Create backup directory
|
||||
ansible.builtin.file:
|
||||
path: "/var/backups/pre-patch-{{ ansible_date_time.iso8601 }}"
|
||||
state: directory
|
||||
mode: '0755'
|
||||
|
||||
- name: Capture current package list
|
||||
ansible.builtin.shell: |
|
||||
set -o pipefail
|
||||
dpkg --get-selections > /var/backups/pre-patch-{{ ansible_date_time.iso8601 }}/packages.list
|
||||
changed_when: false
|
||||
|
||||
- name: Check for available updates
|
||||
ansible.builtin.shell: |
|
||||
set -o pipefail
|
||||
apt list --upgradable 2>/dev/null | grep -v "Listing..." | wc -l
|
||||
register: updates_available_count
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Update package cache
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
cache_valid_time: 300
|
||||
changed_when: false
|
||||
|
||||
- name: Check if reboot required before patching
|
||||
ansible.builtin.stat:
|
||||
path: /var/run/reboot-required
|
||||
register: reboot_required_before
|
||||
changed_when: false
|
||||
|
||||
- name: Apply security updates
|
||||
ansible.builtin.apt:
|
||||
upgrade: dist
|
||||
update_cache: true
|
||||
when: patch_security_only | bool
|
||||
register: apt_update_result
|
||||
notify: restart patching services
|
||||
|
||||
- name: Apply all available updates
|
||||
ansible.builtin.apt:
|
||||
upgrade: full
|
||||
update_cache: true
|
||||
when: not (patch_security_only | bool)
|
||||
register: apt_update_result
|
||||
notify: restart patching services
|
||||
|
||||
- name: Check if reboot required after patching
|
||||
ansible.builtin.stat:
|
||||
path: /var/run/reboot-required
|
||||
register: reboot_required_after
|
||||
changed_when: false
|
||||
|
||||
- name: Verify critical services are running
|
||||
ansible.builtin.service:
|
||||
name: "{{ item }}"
|
||||
state: started
|
||||
enabled: true
|
||||
loop: "{{ critical_services }}"
|
||||
failed_when: false
|
||||
|
||||
- name: Run post-patch health checks
|
||||
ansible.builtin.uri:
|
||||
url: http://localhost/health
|
||||
method: GET
|
||||
status_code: 200
|
||||
register: health_check
|
||||
failed_when: false
|
||||
ignore_errors: true
|
||||
when: "'webservers' in group_names"
|
||||
|
||||
- name: Set reboot required flag
|
||||
ansible.builtin.set_fact:
|
||||
reboot_required: "{{ reboot_required_after.stat.exists | default(false) }}"
|
||||
|
||||
- name: Perform system reboot if required
|
||||
ansible.builtin.reboot:
|
||||
msg: "Rebooting after security patches"
|
||||
timeout: "{{ reboot_timeout }}"
|
||||
when: reboot_required and reboot_if_required | bool
|
||||
|
||||
- name: Generate patching report
|
||||
ansible.builtin.template:
|
||||
src: patch_report.j2
|
||||
dest: /var/log/patch_report_{{ ansible_date_time.iso8601 }}.log
|
||||
mode: '0644'
|
||||
vars:
|
||||
updates_applied_count: "{{ apt_update_result.changed | ternary('Yes', 'No') }}"
|
||||
reboot_required_flag: "{{ reboot_required }}"
|
||||
@@ -0,0 +1,10 @@
|
||||
Patching Report
|
||||
===============
|
||||
Generated: {{ ansible_date_time.iso8601 }}
|
||||
Host: {{ inventory_hostname }}
|
||||
|
||||
Updates Applied: {{ updates_applied_count }}
|
||||
Reboot Required: {{ reboot_required_flag }}
|
||||
Services Restarted: {{ services_to_restart | join(', ') }}
|
||||
|
||||
Backup Location: /var/backups/pre-patch-{{ ansible_date_time.iso8601 }}/
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: Simulate Failure and Patch
|
||||
|
||||
## Description
|
||||
|
||||
Validate that a service-level failure can be detected, recovered, and followed by a controlled patch workflow. This mirrors a maintenance window where a degraded node is stabilized before package updates are applied.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
cd enterprise-infra-simulator
|
||||
./scripts/simulate_failure.sh service 30 web
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/patch.yml
|
||||
ansible-playbook -i inventory/hosts.ini playbooks/hardening.yml --check
|
||||
```
|
||||
|
||||
## Expected Result
|
||||
|
||||
- The simulation records a temporary service failure.
|
||||
- The service is restored after cleanup.
|
||||
- The patch playbook completes without unreachable hosts.
|
||||
- Hardening check mode reports no destructive changes.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
- name: Enterprise Scaling Event Scenario
|
||||
hosts: all
|
||||
become: yes
|
||||
gather_facts: yes
|
||||
vars:
|
||||
scaling_threshold: 80
|
||||
cooldown_period: 300
|
||||
max_scale_up: 5
|
||||
min_instances: 2
|
||||
|
||||
pre_tasks:
|
||||
- name: Log scenario start
|
||||
lineinfile:
|
||||
path: "/var/log/scaling_scenario.log"
|
||||
line: "{{ ansible_date_time.iso8601 }} - Starting scaling event scenario"
|
||||
create: yes
|
||||
|
||||
- name: Check current load
|
||||
command: uptime
|
||||
register: system_load
|
||||
changed_when: false
|
||||
|
||||
- name: Parse load average
|
||||
set_fact:
|
||||
load_1min: "{{ system_load.stdout.split(',')[0].split()[-1] | float }}"
|
||||
load_5min: "{{ system_load.stdout.split(',')[1] | float }}"
|
||||
load_15min: "{{ system_load.stdout.split(',')[2] | float }}"
|
||||
|
||||
tasks:
|
||||
- name: Evaluate scaling conditions
|
||||
set_fact:
|
||||
scale_up_needed: "{{ load_5min > scaling_threshold }}"
|
||||
scale_down_needed: "{{ load_5min < (scaling_threshold * 0.3) }}"
|
||||
|
||||
- name: Scale up web servers
|
||||
include_role:
|
||||
name: scale_up
|
||||
tasks_from: web_servers
|
||||
vars:
|
||||
scale_count: "{{ [max_scale_up, (load_5min / 10) | int] | min }}"
|
||||
when: scale_up_needed and "'webservers' in group_names"
|
||||
|
||||
- name: Scale up database servers
|
||||
include_role:
|
||||
name: scale_up
|
||||
tasks_from: database_servers
|
||||
vars:
|
||||
scale_count: "{{ [2, (load_5min / 20) | int] | min }}"
|
||||
when: scale_up_needed and "'databases' in group_names"
|
||||
|
||||
- name: Update load balancer configuration
|
||||
include_role:
|
||||
name: load_balancer
|
||||
tasks_from: update_backends
|
||||
when: scale_up_needed
|
||||
|
||||
- name: Scale down web servers
|
||||
include_role:
|
||||
name: scale_down
|
||||
tasks_from: web_servers
|
||||
vars:
|
||||
scale_count: "{{ [(inventory_hostname | regex_findall('[0-9]+') | first | int) - min_instances, 1] | max }}"
|
||||
when: scale_down_needed and "'webservers' in group_names" and (inventory_hostname | regex_findall('[0-9]+') | first | int) > min_instances
|
||||
|
||||
- name: Wait for cooldown period
|
||||
pause:
|
||||
seconds: "{{ cooldown_period }}"
|
||||
when: scale_up_needed or scale_down_needed
|
||||
|
||||
- name: Verify scaling results
|
||||
uri:
|
||||
url: http://localhost/health
|
||||
method: GET
|
||||
status_code: 200
|
||||
register: health_check
|
||||
until: health_check.status == 200
|
||||
retries: 5
|
||||
delay: 10
|
||||
when: "'webservers' in group_names"
|
||||
|
||||
- name: Update monitoring thresholds
|
||||
include_role:
|
||||
name: monitoring
|
||||
tasks_from: update_alerts
|
||||
vars:
|
||||
new_threshold: "{{ scaling_threshold + 10 }}"
|
||||
|
||||
- name: Send scaling notification
|
||||
mail:
|
||||
to: "{{ scaling_notification_email | default('infra-team@company.com') }}"
|
||||
subject: "Infrastructure Scaling Event - {{ inventory_hostname }}"
|
||||
body: |
|
||||
Scaling event completed on {{ inventory_hostname }}
|
||||
|
||||
Load averages: {{ load_1min }}, {{ load_5min }}, {{ load_15min }}
|
||||
Action taken: {{ 'Scale Up' if scale_up_needed else 'Scale Down' if scale_down_needed else 'No Action' }}
|
||||
Health check: {{ 'PASSED' if health_check.status == 200 else 'FAILED' }}
|
||||
|
||||
See /var/log/scaling_scenario.log for details
|
||||
when: scaling_notification_email is defined
|
||||
ignore_errors: yes
|
||||
|
||||
post_tasks:
|
||||
- name: Generate scaling scenario report
|
||||
template:
|
||||
src: templates/scaling_scenario_report.j2
|
||||
dest: "/var/log/scaling_scenario_report_{{ ansible_date_time.iso8601 }}.log"
|
||||
vars:
|
||||
scenario_outcome: "{{ 'SUCCESS' if health_check.status == 200 else 'WARNING' }}"
|
||||
load_metrics: "{{ load_1min }}, {{ load_5min }}, {{ load_15min }}"
|
||||
|
||||
- name: Log scenario completion
|
||||
lineinfile:
|
||||
path: "/var/log/scaling_scenario.log"
|
||||
line: "{{ ansible_date_time.iso8601 }} - Scaling event scenario completed"
|
||||
@@ -0,0 +1,343 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Enterprise Infrastructure Failure Simulation Script
|
||||
# Simulates various types of infrastructure failures for testing
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Configuration
|
||||
DOCKER_COMPOSE_FILE="docker-compose.yml"
|
||||
INVENTORY_FILE="inventory/hosts.ini"
|
||||
LOG_FILE="logs/failure_simulation.log"
|
||||
|
||||
# Default values
|
||||
FAILURE_TYPE="${1:-network}"
|
||||
DURATION="${2:-60}"
|
||||
TARGET_NODES="${3:-all}"
|
||||
INTENSITY="${INTENSITY:-medium}"
|
||||
|
||||
# Logging function
|
||||
log() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - $*" | tee -a "$LOG_FILE"
|
||||
}
|
||||
|
||||
# Error handling
|
||||
error_exit() {
|
||||
log "ERROR: $1"
|
||||
# Cleanup any active failures
|
||||
cleanup_failure
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Validate inputs
|
||||
validate_inputs() {
|
||||
case "$FAILURE_TYPE" in
|
||||
network|disk|service|node|cpu|memory) ;;
|
||||
*) error_exit "Invalid failure type: $FAILURE_TYPE. Must be network, disk, service, node, cpu, or memory" ;;
|
||||
esac
|
||||
|
||||
if ! [[ "$DURATION" =~ ^[0-9]+$ ]] || [ "$DURATION" -lt 1 ]; then
|
||||
error_exit "Invalid duration: $DURATION. Must be a positive integer (seconds)"
|
||||
fi
|
||||
|
||||
case "$INTENSITY" in
|
||||
low|medium|high|critical) ;;
|
||||
*) error_exit "Invalid intensity: $INTENSITY. Must be low, medium, high, or critical" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Get target containers
|
||||
get_target_containers() {
|
||||
case "$TARGET_NODES" in
|
||||
all)
|
||||
docker-compose ps --services | grep -v "^NAME$" || true
|
||||
;;
|
||||
web)
|
||||
echo "web"
|
||||
;;
|
||||
db)
|
||||
echo "db"
|
||||
;;
|
||||
lb)
|
||||
echo "lb"
|
||||
;;
|
||||
monitor)
|
||||
echo "monitor"
|
||||
;;
|
||||
*)
|
||||
echo "$TARGET_NODES"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Network failure simulation
|
||||
simulate_network_failure() {
|
||||
local containers=$(get_target_containers)
|
||||
log "Simulating network failure on containers: $containers"
|
||||
|
||||
for container in $containers; do
|
||||
local container_ids=$(docker-compose ps -q "$container" 2>/dev/null || true)
|
||||
|
||||
for cid in $container_ids; do
|
||||
if [ -n "$cid" ]; then
|
||||
log "Disconnecting network for container $cid"
|
||||
|
||||
# Disconnect from network
|
||||
docker network disconnect "$(docker inspect "$cid" --format '{{.HostConfig.NetworkMode}}')" "$cid" 2>/dev/null || true
|
||||
|
||||
# Store original network for restoration
|
||||
echo "$cid:$(docker inspect "$cid" --format '{{.HostConfig.NetworkMode}}')" >> /tmp/network_failure_state
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# Disk failure simulation
|
||||
simulate_disk_failure() {
|
||||
local containers=$(get_target_containers)
|
||||
log "Simulating disk space exhaustion on containers: $containers"
|
||||
|
||||
for container in $containers; do
|
||||
local container_ids=$(docker-compose ps -q "$container" 2>/dev/null || true)
|
||||
|
||||
for cid in $container_ids; do
|
||||
if [ -n "$cid" ]; then
|
||||
log "Filling disk space in container $cid"
|
||||
|
||||
# Create a large file to consume disk space
|
||||
local fill_size="100M"
|
||||
case "$INTENSITY" in
|
||||
low) fill_size="50M" ;;
|
||||
medium) fill_size="100M" ;;
|
||||
high) fill_size="500M" ;;
|
||||
critical) fill_size="1G" ;;
|
||||
esac
|
||||
|
||||
docker exec "$cid" bash -c "dd if=/dev/zero of=/tmp/disk_fill bs=1M count=$(( ${fill_size%M} * 1024 ))" 2>/dev/null || true
|
||||
echo "$cid:disk_fill" >> /tmp/disk_failure_state
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# Service failure simulation
|
||||
simulate_service_failure() {
|
||||
local containers=$(get_target_containers)
|
||||
log "Simulating service failures on containers: $containers"
|
||||
|
||||
for container in $containers; do
|
||||
local container_ids=$(docker-compose ps -q "$container" 2>/dev/null || true)
|
||||
|
||||
for cid in $container_ids; do
|
||||
if [ -n "$cid" ]; then
|
||||
log "Stopping services in container $cid"
|
||||
|
||||
# Stop common services
|
||||
docker exec "$cid" systemctl stop nginx 2>/dev/null || true
|
||||
docker exec "$cid" systemctl stop postgresql 2>/dev/null || true
|
||||
docker exec "$cid" systemctl stop haproxy 2>/dev/null || true
|
||||
|
||||
echo "$cid:services" >> /tmp/service_failure_state
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# Node failure simulation
|
||||
simulate_node_failure() {
|
||||
local containers=$(get_target_containers)
|
||||
log "Simulating complete node failures on containers: $containers"
|
||||
|
||||
for container in $containers; do
|
||||
local container_ids=$(docker-compose ps -q "$container" 2>/dev/null || true)
|
||||
|
||||
for cid in $container_ids; do
|
||||
if [ -n "$cid" ]; then
|
||||
log "Stopping container $cid (node failure)"
|
||||
docker pause "$cid"
|
||||
echo "$cid:paused" >> /tmp/node_failure_state
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# CPU stress simulation
|
||||
simulate_cpu_failure() {
|
||||
local containers=$(get_target_containers)
|
||||
log "Simulating CPU stress on containers: $containers"
|
||||
|
||||
for container in $containers; do
|
||||
local container_ids=$(docker-compose ps -q "$container" 2>/dev/null || true)
|
||||
|
||||
for cid in $container_ids; do
|
||||
if [ -n "$cid" ]; then
|
||||
log "Starting CPU stress in container $cid"
|
||||
|
||||
# Start CPU stress process
|
||||
docker exec -d "$cid" bash -c "while true; do :; done" 2>/dev/null || true
|
||||
echo "$cid:cpu_stress:$(docker exec "$cid" ps aux | grep "while true" | grep -v grep | awk '{print $2}' | head -1)" >> /tmp/cpu_failure_state
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# Memory stress simulation
|
||||
simulate_memory_failure() {
|
||||
local containers=$(get_target_containers)
|
||||
log "Simulating memory exhaustion on containers: $containers"
|
||||
|
||||
for container in $containers; do
|
||||
local container_ids=$(docker-compose ps -q "$container" 2>/dev/null || true)
|
||||
|
||||
for cid in $container_ids; do
|
||||
if [ -n "$cid" ]; then
|
||||
log "Starting memory stress in container $cid"
|
||||
|
||||
# Start memory stress process
|
||||
docker exec -d "$cid" bash -c "tail /dev/zero" 2>/dev/null || true
|
||||
echo "$cid:memory_stress:$(docker exec "$cid" ps aux | grep "tail /dev/zero" | grep -v grep | awk '{print $2}' | head -1)" >> /tmp/memory_failure_state
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# Inject failure
|
||||
inject_failure() {
|
||||
case "$FAILURE_TYPE" in
|
||||
network) simulate_network_failure ;;
|
||||
disk) simulate_disk_failure ;;
|
||||
service) simulate_service_failure ;;
|
||||
node) simulate_node_failure ;;
|
||||
cpu) simulate_cpu_failure ;;
|
||||
memory) simulate_memory_failure ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Cleanup failure
|
||||
cleanup_failure() {
|
||||
log "Cleaning up failure simulation"
|
||||
|
||||
# Restore network connections
|
||||
if [ -f /tmp/network_failure_state ]; then
|
||||
while IFS=: read -r cid network; do
|
||||
docker network connect "$network" "$cid" 2>/dev/null || true
|
||||
done < /tmp/network_failure_state
|
||||
rm -f /tmp/network_failure_state
|
||||
fi
|
||||
|
||||
# Clean up disk fill files
|
||||
if [ -f /tmp/disk_failure_state ]; then
|
||||
while IFS=: read -r cid _; do
|
||||
docker exec "$cid" rm -f /tmp/disk_fill 2>/dev/null || true
|
||||
done < /tmp/disk_failure_state
|
||||
rm -f /tmp/disk_failure_state
|
||||
fi
|
||||
|
||||
# Restart services
|
||||
if [ -f /tmp/service_failure_state ]; then
|
||||
while IFS=: read -r cid _; do
|
||||
docker exec "$cid" systemctl start nginx 2>/dev/null || true
|
||||
docker exec "$cid" systemctl start postgresql 2>/dev/null || true
|
||||
docker exec "$cid" systemctl start haproxy 2>/dev/null || true
|
||||
done < /tmp/service_failure_state
|
||||
rm -f /tmp/service_failure_state
|
||||
fi
|
||||
|
||||
# Unpause containers
|
||||
if [ -f /tmp/node_failure_state ]; then
|
||||
while IFS=: read -r cid _; do
|
||||
docker unpause "$cid" 2>/dev/null || true
|
||||
done < /tmp/node_failure_state
|
||||
rm -f /tmp/node_failure_state
|
||||
fi
|
||||
|
||||
# Kill stress processes
|
||||
if [ -f /tmp/cpu_failure_state ]; then
|
||||
while IFS=: read -r cid _ pid; do
|
||||
docker exec "$cid" kill -9 "$pid" 2>/dev/null || true
|
||||
done < /tmp/cpu_failure_state
|
||||
rm -f /tmp/cpu_failure_state
|
||||
fi
|
||||
|
||||
if [ -f /tmp/memory_failure_state ]; then
|
||||
while IFS=: read -r cid _ pid; do
|
||||
docker exec "$cid" kill -9 "$pid" 2>/dev/null || true
|
||||
done < /tmp/memory_failure_state
|
||||
rm -f /tmp/memory_failure_state
|
||||
fi
|
||||
}
|
||||
|
||||
# Monitor failure
|
||||
monitor_failure() {
|
||||
local end_time=$(( $(date +%s) + DURATION ))
|
||||
|
||||
log "Monitoring failure for $DURATION seconds"
|
||||
|
||||
while [ $(date +%s) -lt $end_time ]; do
|
||||
# Check container status
|
||||
if ! docker-compose ps | grep -q "Up\|Paused"; then
|
||||
log "WARNING: All containers are down"
|
||||
fi
|
||||
|
||||
# Log system metrics
|
||||
log "System status: $(docker stats --no-stream --format 'table {{.Container}}\t{{.CPUPerc}}\t{{.MemPerc}}' | tail -n +2)"
|
||||
|
||||
sleep 10
|
||||
done
|
||||
}
|
||||
|
||||
# Generate failure report
|
||||
generate_report() {
|
||||
local report_file="reports/failure_simulation_$(date +%Y%m%d_%H%M%S).txt"
|
||||
|
||||
cat > "$report_file" << EOF
|
||||
Failure Simulation Report
|
||||
========================
|
||||
|
||||
Timestamp: $(date)
|
||||
Failure Type: $FAILURE_TYPE
|
||||
Duration: $DURATION seconds
|
||||
Target Nodes: $TARGET_NODES
|
||||
Intensity: $INTENSITY
|
||||
|
||||
Pre-failure Status:
|
||||
$(docker-compose ps)
|
||||
|
||||
Post-failure Status:
|
||||
$(docker-compose ps)
|
||||
|
||||
Log File: $LOG_FILE
|
||||
EOF
|
||||
|
||||
log "Failure simulation report generated: $report_file"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
log "Starting failure simulation: $FAILURE_TYPE for $DURATION seconds"
|
||||
|
||||
validate_inputs
|
||||
|
||||
# Inject failure
|
||||
inject_failure
|
||||
|
||||
# Monitor during failure
|
||||
monitor_failure
|
||||
|
||||
# Cleanup
|
||||
cleanup_failure
|
||||
|
||||
# Generate report
|
||||
generate_report
|
||||
|
||||
log "Failure simulation completed successfully"
|
||||
}
|
||||
|
||||
# Trap for cleanup on script exit
|
||||
trap cleanup_failure EXIT
|
||||
|
||||
# Initialize logging
|
||||
mkdir -p logs reports
|
||||
|
||||
# Run main function
|
||||
main "$@"
|
||||
@@ -0,0 +1,208 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Enterprise Infrastructure Scaling Simulation Script
|
||||
# Simulates scaling operations for infrastructure nodes
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Configuration
|
||||
DOCKER_COMPOSE_FILE="docker-compose.yml"
|
||||
INVENTORY_FILE="inventory/hosts.ini"
|
||||
LOG_FILE="logs/scaling_simulation.log"
|
||||
|
||||
# Default values
|
||||
DIRECTION="${1:-up}"
|
||||
COUNT="${2:-1}"
|
||||
NODE_TYPE="${3:-web}"
|
||||
SIMULATION_MODE="${SIMULATION_MODE:-false}"
|
||||
|
||||
# Logging function
|
||||
log() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - $*" | tee -a "$LOG_FILE"
|
||||
}
|
||||
|
||||
# Error handling
|
||||
error_exit() {
|
||||
log "ERROR: $1"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Validate inputs
|
||||
validate_inputs() {
|
||||
if [[ "$DIRECTION" != "up" && "$DIRECTION" != "down" ]]; then
|
||||
error_exit "Invalid direction: $DIRECTION. Must be 'up' or 'down'"
|
||||
fi
|
||||
|
||||
if ! [[ "$COUNT" =~ ^[0-9]+$ ]] || [ "$COUNT" -lt 1 ]; then
|
||||
error_exit "Invalid count: $COUNT. Must be a positive integer"
|
||||
fi
|
||||
|
||||
case "$NODE_TYPE" in
|
||||
web|db|lb|monitor) ;;
|
||||
*) error_exit "Invalid node type: $NODE_TYPE. Must be web, db, lb, or monitor" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Get current node count
|
||||
get_current_count() {
|
||||
local type="$1"
|
||||
case "$type" in
|
||||
web) docker-compose ps web | grep -c "Up" ;;
|
||||
db) docker-compose ps db | grep -c "Up" ;;
|
||||
lb) docker-compose ps lb | grep -c "Up" ;;
|
||||
monitor) docker-compose ps monitor | grep -c "Up" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Scale up infrastructure
|
||||
scale_up() {
|
||||
local type="$1"
|
||||
local count="$2"
|
||||
|
||||
log "Scaling up $count $type nodes"
|
||||
|
||||
# Update docker-compose replica count
|
||||
sed -i.bak "s/replicas: [0-9]\+/replicas: $(( $(get_current_count "$type") + count ))/" "$DOCKER_COMPOSE_FILE"
|
||||
|
||||
# Deploy new containers
|
||||
docker-compose up -d --scale "${type}=${count}"
|
||||
|
||||
# Wait for containers to be ready
|
||||
log "Waiting for containers to be ready..."
|
||||
sleep 30
|
||||
|
||||
# Update inventory
|
||||
update_inventory "$type" "$count" "add"
|
||||
|
||||
# Run provisioning playbook on new nodes
|
||||
if [ "$SIMULATION_MODE" = false ]; then
|
||||
ansible-playbook -i "$INVENTORY_FILE" playbooks/provision.yml --limit "${type}*"
|
||||
fi
|
||||
|
||||
log "Successfully scaled up $count $type nodes"
|
||||
}
|
||||
|
||||
# Scale down infrastructure
|
||||
scale_down() {
|
||||
local type="$1"
|
||||
local count="$2"
|
||||
|
||||
local current_count=$(get_current_count "$type")
|
||||
if [ "$current_count" -lt "$count" ]; then
|
||||
error_exit "Cannot scale down $count nodes. Only $current_count $type nodes currently running"
|
||||
fi
|
||||
|
||||
log "Scaling down $count $type nodes"
|
||||
|
||||
# Select nodes to remove (oldest first)
|
||||
local nodes_to_remove=$(docker-compose ps "$type" | grep "Up" | head -n "$count" | awk '{print $1}')
|
||||
|
||||
# Decommission nodes
|
||||
for node in $nodes_to_remove; do
|
||||
if [ "$SIMULATION_MODE" = false ]; then
|
||||
ansible-playbook -i "$INVENTORY_FILE" playbooks/decommission.yml --limit "$node"
|
||||
fi
|
||||
docker stop "$node"
|
||||
docker rm "$node"
|
||||
done
|
||||
|
||||
# Update docker-compose replica count
|
||||
sed -i.bak "s/replicas: [0-9]\+/replicas: $(( current_count - count ))/" "$DOCKER_COMPOSE_FILE"
|
||||
|
||||
# Update inventory
|
||||
update_inventory "$type" "$count" "remove"
|
||||
|
||||
log "Successfully scaled down $count $type nodes"
|
||||
}
|
||||
|
||||
# Update Ansible inventory
|
||||
update_inventory() {
|
||||
local type="$1"
|
||||
local count="$2"
|
||||
local action="$3"
|
||||
|
||||
log "Updating inventory for $action $count $type nodes"
|
||||
|
||||
# This would be more complex in a real implementation
|
||||
# For simulation, we'll just log the action
|
||||
case "$action" in
|
||||
add)
|
||||
log "Added $count $type nodes to inventory"
|
||||
;;
|
||||
remove)
|
||||
log "Removed $count $type nodes from inventory"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Health check after scaling
|
||||
health_check() {
|
||||
log "Running health checks after scaling"
|
||||
|
||||
# Check container status
|
||||
if ! docker-compose ps | grep -q "Up"; then
|
||||
error_exit "Some containers failed to start"
|
||||
fi
|
||||
|
||||
# Ansible ping check
|
||||
if [ "$SIMULATION_MODE" = false ]; then
|
||||
if ! ansible -i "$INVENTORY_FILE" all -m ping >/dev/null 2>&1; then
|
||||
log "WARNING: Some nodes failed Ansible ping check"
|
||||
fi
|
||||
fi
|
||||
|
||||
log "Health checks completed"
|
||||
}
|
||||
|
||||
# Generate scaling report
|
||||
generate_report() {
|
||||
local report_file="reports/scaling_report_$(date +%Y%m%d_%H%M%S).txt"
|
||||
|
||||
cat > "$report_file" << EOF
|
||||
Scaling Simulation Report
|
||||
========================
|
||||
|
||||
Timestamp: $(date)
|
||||
Direction: $DIRECTION
|
||||
Node Type: $NODE_TYPE
|
||||
Count: $COUNT
|
||||
Simulation Mode: $SIMULATION_MODE
|
||||
|
||||
Current Status:
|
||||
$(docker-compose ps)
|
||||
|
||||
Inventory Status:
|
||||
$(ansible -i "$INVENTORY_FILE" --list-hosts all 2>/dev/null || echo "Ansible inventory check failed")
|
||||
|
||||
Log File: $LOG_FILE
|
||||
EOF
|
||||
|
||||
log "Scaling report generated: $report_file"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
log "Starting scaling simulation: $DIRECTION $COUNT $NODE_TYPE nodes"
|
||||
|
||||
validate_inputs
|
||||
|
||||
case "$DIRECTION" in
|
||||
up)
|
||||
scale_up "$NODE_TYPE" "$COUNT"
|
||||
;;
|
||||
down)
|
||||
scale_down "$NODE_TYPE" "$COUNT"
|
||||
;;
|
||||
esac
|
||||
|
||||
health_check
|
||||
generate_report
|
||||
|
||||
log "Scaling simulation completed successfully"
|
||||
}
|
||||
|
||||
# Initialize logging
|
||||
mkdir -p logs reports
|
||||
|
||||
# Run main function
|
||||
main "$@"
|
||||
@@ -0,0 +1,10 @@
|
||||
.PHONY: run test demo
|
||||
|
||||
run:
|
||||
python3 cli.py --help
|
||||
|
||||
test:
|
||||
python3 -m py_compile cli.py collectors/*.py validators/*.py reports/*.py
|
||||
|
||||
demo:
|
||||
python3 cli.py compare examples/before.json examples/after.json --output /tmp/migration-diff.json
|
||||
@@ -0,0 +1,56 @@
|
||||
# Migration Validation Framework
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Infrastructure migrations often fail in small, expensive ways: a mount option changes, a service is disabled, or disk usage moves past an operational threshold. Teams need structured evidence that the migrated host still matches the expected operating profile.
|
||||
|
||||
## Solution Overview
|
||||
|
||||
This project provides a Python CLI that collects system state into JSON snapshots and compares before/after files. The output is designed for change records, migration gates, and post-cutover validation.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Operator -> CLI -> Collectors -> JSON Snapshot -> Comparator -> Diff/Report
|
||||
```
|
||||
|
||||
Core components:
|
||||
|
||||
- `cli.py` provides collect, compare, snapshot, list, and report commands.
|
||||
- `collectors/` gathers mounts, services, and disk usage.
|
||||
- `validators/compare.py` identifies drift and validation failures.
|
||||
- `reports/` contains report generation helpers.
|
||||
- `examples/` contains realistic before/after evidence.
|
||||
|
||||
## How to Run
|
||||
|
||||
```bash
|
||||
cd migration-validation-framework
|
||||
python3 cli.py collect --output before.json --systems web01,db01
|
||||
python3 cli.py collect --output after.json --systems web01,db01
|
||||
python3 cli.py compare before.json after.json --output diff.json
|
||||
python3 cli.py compare examples/before.json examples/after.json --output /tmp/migration-diff.json
|
||||
```
|
||||
|
||||
Legacy snapshot IDs are still supported:
|
||||
|
||||
```bash
|
||||
python3 cli.py snapshot --env prod --label pre --systems web01,db01
|
||||
python3 cli.py compare prod-pre-20260429_020000 prod-post-20260429_030000 --output change-0429
|
||||
```
|
||||
|
||||
## Example Output
|
||||
|
||||
```text
|
||||
Comparison completed: diff.json (FAIL)
|
||||
Overall risk: high
|
||||
Total changes: 4
|
||||
Failed checks: critical_services_running
|
||||
Recommendation: restore sshd before production cutover
|
||||
```
|
||||
|
||||
Sample inputs and output are available in [examples/before.json](examples/before.json), [examples/after.json](examples/after.json), and [examples/diff.json](examples/diff.json).
|
||||
|
||||
## Real-World Use Case
|
||||
|
||||
During a data center migration, a platform team can collect baseline state before cutover, collect the same evidence after DNS or workload migration, and attach the diff to the change ticket. The framework gives reviewers a compact signal on whether the host is ready for production traffic.
|
||||
@@ -0,0 +1,323 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration Validation Framework - CLI Interface
|
||||
|
||||
A comprehensive tool for validating system migrations through data collection,
|
||||
snapshot comparison, and automated reporting.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
# Import framework modules
|
||||
from collectors import mounts, services, disk_usage
|
||||
from validators import compare
|
||||
from reports import html_report
|
||||
|
||||
# Configuration
|
||||
SNAPSHOTS_DIR = Path("snapshots")
|
||||
LOGS_DIR = Path("logs")
|
||||
REPORTS_DIR = Path("reports")
|
||||
|
||||
class MigrationValidator:
|
||||
"""Main migration validation class."""
|
||||
|
||||
def __init__(self, verbose: bool = False):
|
||||
self.verbose = verbose
|
||||
self.ensure_directories()
|
||||
self.setup_logging()
|
||||
|
||||
def setup_logging(self):
|
||||
"""Configure logging."""
|
||||
log_level = logging.DEBUG if self.verbose else logging.INFO
|
||||
logging.basicConfig(
|
||||
level=log_level,
|
||||
format='%(asctime)s - %(levelname)s - %(message)s',
|
||||
handlers=[
|
||||
logging.FileHandler(LOGS_DIR / "validation.log"),
|
||||
logging.StreamHandler(sys.stdout)
|
||||
]
|
||||
)
|
||||
self.logger = logging.getLogger(__name__)
|
||||
|
||||
def ensure_directories(self):
|
||||
"""Ensure required directories exist."""
|
||||
for directory in [SNAPSHOTS_DIR, LOGS_DIR, REPORTS_DIR]:
|
||||
directory.mkdir(exist_ok=True)
|
||||
|
||||
def collect_system_data(self, systems: List[str]) -> Dict[str, Any]:
|
||||
"""Collect data from target systems."""
|
||||
self.logger.info(f"Collecting data from systems: {systems}")
|
||||
|
||||
snapshot = {
|
||||
"metadata": {
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"systems": systems,
|
||||
"version": "1.0"
|
||||
},
|
||||
"data": {}
|
||||
}
|
||||
|
||||
collectors = [
|
||||
("mounts", mounts.collect),
|
||||
("services", services.collect),
|
||||
("disk_usage", disk_usage.collect)
|
||||
]
|
||||
|
||||
for system in systems:
|
||||
self.logger.info(f"Collecting data from {system}")
|
||||
snapshot["data"][system] = {}
|
||||
|
||||
for collector_name, collector_func in collectors:
|
||||
try:
|
||||
self.logger.debug(f"Running {collector_name} collector on {system}")
|
||||
data = collector_func(system)
|
||||
snapshot["data"][system][collector_name] = data
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to collect {collector_name} from {system}: {e}")
|
||||
snapshot["data"][system][collector_name] = {"error": str(e)}
|
||||
|
||||
return snapshot
|
||||
|
||||
def save_snapshot(self, snapshot: Dict[str, Any], label: str, env: str) -> str:
|
||||
"""Save snapshot to disk."""
|
||||
snapshot_id = f"{env}-{label}-{datetime.now().strftime('%Y%m%d_%H%M%S')}"
|
||||
snapshot_file = SNAPSHOTS_DIR / f"{snapshot_id}.json"
|
||||
|
||||
with open(snapshot_file, 'w') as f:
|
||||
json.dump(snapshot, f, indent=2)
|
||||
|
||||
self.logger.info(f"Snapshot saved: {snapshot_id}")
|
||||
return snapshot_id
|
||||
|
||||
def load_snapshot(self, snapshot_id: str) -> Dict[str, Any]:
|
||||
"""Load snapshot from disk."""
|
||||
snapshot_path = Path(snapshot_id)
|
||||
snapshot_file = snapshot_path if snapshot_path.exists() else SNAPSHOTS_DIR / f"{snapshot_id}.json"
|
||||
if not snapshot_file.exists():
|
||||
raise FileNotFoundError(f"Snapshot {snapshot_id} not found")
|
||||
|
||||
with open(snapshot_file, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
def collect_to_file(self, output_file: str, systems: List[str]) -> str:
|
||||
"""Collect a snapshot and write it to an explicit file path."""
|
||||
snapshot = self.collect_system_data(systems)
|
||||
with open(output_file, 'w') as f:
|
||||
json.dump(snapshot, f, indent=2)
|
||||
f.write("\n")
|
||||
self.logger.info(f"Snapshot written: {output_file}")
|
||||
return output_file
|
||||
|
||||
def create_snapshot(self, env: str, label: str, systems: List[str]) -> str:
|
||||
"""Create and save a system snapshot."""
|
||||
self.logger.info(f"Creating snapshot for environment: {env}, label: {label}")
|
||||
|
||||
snapshot = self.collect_system_data(systems)
|
||||
snapshot_id = self.save_snapshot(snapshot, label, env)
|
||||
|
||||
return snapshot_id
|
||||
|
||||
def compare_snapshots(self, snapshot1_id: str, snapshot2_id: str, output_id: str) -> Dict[str, Any]:
|
||||
"""Compare two snapshots."""
|
||||
self.logger.info(f"Comparing snapshots: {snapshot1_id} vs {snapshot2_id}")
|
||||
|
||||
snapshot1 = self.load_snapshot(snapshot1_id)
|
||||
snapshot2 = self.load_snapshot(snapshot2_id)
|
||||
|
||||
comparison = compare.compare_snapshots(snapshot1, snapshot2)
|
||||
comparison["metadata"] = {
|
||||
"snapshot1": snapshot1_id,
|
||||
"snapshot2": snapshot2_id,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"comparison_id": output_id
|
||||
}
|
||||
|
||||
# Save comparison results
|
||||
comparison_file = REPORTS_DIR / f"comparison_{output_id}.json"
|
||||
with open(comparison_file, 'w') as f:
|
||||
json.dump(comparison, f, indent=2)
|
||||
|
||||
self.logger.info(f"Comparison saved: {output_id}")
|
||||
return comparison
|
||||
|
||||
def compare_files(self, before_file: str, after_file: str, output_file: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""Compare two explicit JSON snapshot files."""
|
||||
self.logger.info(f"Comparing files: {before_file} vs {after_file}")
|
||||
|
||||
before = self.load_snapshot(before_file)
|
||||
after = self.load_snapshot(after_file)
|
||||
comparison = compare.compare_snapshots(before, after)
|
||||
comparison["metadata"] = {
|
||||
"before": before_file,
|
||||
"after": after_file,
|
||||
"timestamp": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
if output_file:
|
||||
with open(output_file, 'w') as f:
|
||||
json.dump(comparison, f, indent=2)
|
||||
f.write("\n")
|
||||
self.logger.info(f"Comparison written: {output_file}")
|
||||
|
||||
return comparison
|
||||
|
||||
def generate_report(self, comparison_id: str, format_type: str, output_file: Optional[str] = None) -> str:
|
||||
"""Generate a report from comparison results."""
|
||||
self.logger.info(f"Generating {format_type} report for comparison: {comparison_id}")
|
||||
|
||||
comparison_file = REPORTS_DIR / f"comparison_{comparison_id}.json"
|
||||
if not comparison_file.exists():
|
||||
raise FileNotFoundError(f"Comparison {comparison_id} not found")
|
||||
|
||||
with open(comparison_file, 'r') as f:
|
||||
comparison = json.load(f)
|
||||
|
||||
if format_type == "html":
|
||||
if output_file is None:
|
||||
output_file = f"migration_report_{comparison_id}.html"
|
||||
html_report.generate(comparison, output_file)
|
||||
elif format_type == "json":
|
||||
if output_file is None:
|
||||
output_file = f"migration_report_{comparison_id}.json"
|
||||
with open(output_file, 'w') as f:
|
||||
json.dump(comparison, f, indent=2)
|
||||
else:
|
||||
raise ValueError(f"Unsupported format: {format_type}")
|
||||
|
||||
self.logger.info(f"Report generated: {output_file}")
|
||||
return output_file
|
||||
|
||||
def main():
|
||||
"""Main CLI entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Migration Validation Framework",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Collect pre-migration snapshot
|
||||
python3 cli.py collect --output before.json --systems web01,db01
|
||||
|
||||
# Compare snapshot files
|
||||
python3 cli.py compare before.json after.json --output diff.json
|
||||
|
||||
# Generate HTML report
|
||||
python3 cli.py report --comparison comparison_001 --format html
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument('--verbose', '-v', action='store_true', help='Enable verbose logging')
|
||||
parser.add_argument('--dry-run', action='store_true', help='Preview actions without execution')
|
||||
|
||||
subparsers = parser.add_subparsers(dest='command', help='Available commands')
|
||||
|
||||
# Collect command
|
||||
collect_parser = subparsers.add_parser('collect', help='Collect a system snapshot to a JSON file')
|
||||
collect_parser.add_argument('--output', required=True, help='Output JSON file')
|
||||
collect_parser.add_argument('--systems', default='localhost', help='Comma-separated list of systems')
|
||||
|
||||
# Snapshot command
|
||||
snapshot_parser = subparsers.add_parser('snapshot', help='Create system snapshot')
|
||||
snapshot_parser.add_argument('--env', required=True, help='Target environment')
|
||||
snapshot_parser.add_argument('--label', required=True, help='Snapshot label')
|
||||
snapshot_parser.add_argument('--systems', required=True, help='Comma-separated list of systems')
|
||||
|
||||
# Compare command
|
||||
compare_parser = subparsers.add_parser('compare', help='Compare two snapshots')
|
||||
compare_parser.add_argument('snapshot1', help='First snapshot ID')
|
||||
compare_parser.add_argument('snapshot2', help='Second snapshot ID')
|
||||
compare_parser.add_argument('--output', help='Output comparison ID or JSON file')
|
||||
|
||||
# Report command
|
||||
report_parser = subparsers.add_parser('report', help='Generate report from comparison')
|
||||
report_parser.add_argument('--comparison', required=True, help='Comparison ID')
|
||||
report_parser.add_argument('--format', choices=['html', 'json'], default='html', help='Report format')
|
||||
report_parser.add_argument('--output', help='Output file path')
|
||||
|
||||
# List command
|
||||
list_parser = subparsers.add_parser('list', help='List snapshots or comparisons')
|
||||
list_parser.add_argument('type', choices=['snapshots', 'comparisons'], help='Type to list')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.command:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
# Initialize validator
|
||||
validator = MigrationValidator(verbose=args.verbose)
|
||||
|
||||
try:
|
||||
if args.command == 'collect':
|
||||
systems = [system.strip() for system in args.systems.split(',') if system.strip()]
|
||||
if args.dry_run:
|
||||
print(f"DRY RUN: Would collect {systems} into {args.output}")
|
||||
return
|
||||
|
||||
output_file = validator.collect_to_file(args.output, systems)
|
||||
print(f"Snapshot written: {output_file}")
|
||||
|
||||
elif args.command == 'snapshot':
|
||||
systems = args.systems.split(',')
|
||||
if args.dry_run:
|
||||
print(f"DRY RUN: Would create snapshot for systems: {systems}")
|
||||
return
|
||||
|
||||
snapshot_id = validator.create_snapshot(args.env, args.label, systems)
|
||||
print(f"Snapshot created: {snapshot_id}")
|
||||
|
||||
elif args.command == 'compare':
|
||||
if args.dry_run:
|
||||
print(f"DRY RUN: Would compare {args.snapshot1} vs {args.snapshot2}")
|
||||
return
|
||||
|
||||
output = args.output
|
||||
if output and output.endswith('.json'):
|
||||
comparison = validator.compare_files(args.snapshot1, args.snapshot2, output)
|
||||
result = "PASS" if comparison.get("validation_results", {}).get("passed") else "FAIL"
|
||||
print(f"Comparison completed: {output} ({result})")
|
||||
else:
|
||||
output_id = output or datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
comparison = validator.compare_snapshots(args.snapshot1, args.snapshot2, output_id)
|
||||
result = "PASS" if comparison.get("validation_results", {}).get("passed") else "FAIL"
|
||||
print(f"Comparison completed: {output_id} ({result})")
|
||||
|
||||
elif args.command == 'report':
|
||||
if args.dry_run:
|
||||
print(f"DRY RUN: Would generate {args.format} report for {args.comparison}")
|
||||
return
|
||||
|
||||
output_file = validator.generate_report(args.comparison, args.format, args.output)
|
||||
print(f"Report generated: {output_file}")
|
||||
|
||||
elif args.command == 'list':
|
||||
if args.type == 'snapshots':
|
||||
snapshots = list(SNAPSHOTS_DIR.glob("*.json"))
|
||||
if snapshots:
|
||||
print("Available snapshots:")
|
||||
for snapshot in sorted(snapshots):
|
||||
print(f" {snapshot.stem}")
|
||||
else:
|
||||
print("No snapshots found")
|
||||
elif args.type == 'comparisons':
|
||||
comparisons = list(REPORTS_DIR.glob("comparison_*.json"))
|
||||
if comparisons:
|
||||
print("Available comparisons:")
|
||||
for comparison in sorted(comparisons):
|
||||
comp_id = comparison.stem.replace('comparison_', '')
|
||||
print(f" {comp_id}")
|
||||
else:
|
||||
print("No comparisons found")
|
||||
|
||||
except Exception as e:
|
||||
validator.logger.error(f"Command failed: {e}")
|
||||
print(f"Error: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,207 @@
|
||||
"""
|
||||
Disk Usage Data Collector
|
||||
|
||||
Collects disk usage statistics including directory sizes,
|
||||
file system usage, and largest files information.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
from typing import Dict, Any, List
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class DiskUsageCollector:
|
||||
"""Collector for disk usage statistics."""
|
||||
|
||||
def __init__(self):
|
||||
self.max_depth = 3
|
||||
self.exclude_paths = [
|
||||
"/proc",
|
||||
"/sys",
|
||||
"/dev",
|
||||
"/run",
|
||||
"/tmp",
|
||||
"/var/log"
|
||||
]
|
||||
|
||||
def collect_disk_usage(self, system: str) -> Dict[str, Any]:
|
||||
"""Collect disk usage information from target system."""
|
||||
logger.info(f"Collecting disk usage data from {system}")
|
||||
|
||||
try:
|
||||
# Collect filesystem usage
|
||||
filesystem_usage = self.collect_filesystem_usage(system)
|
||||
|
||||
# Collect directory sizes
|
||||
directory_sizes = self.collect_directory_sizes(system)
|
||||
|
||||
# Collect largest files
|
||||
largest_files = self.collect_largest_files(system)
|
||||
|
||||
return {
|
||||
"filesystem_usage": filesystem_usage,
|
||||
"directory_sizes": directory_sizes,
|
||||
"largest_files": largest_files,
|
||||
"timestamp": self.get_timestamp(system)
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect disk usage from {system}: {e}")
|
||||
raise
|
||||
|
||||
def collect_filesystem_usage(self, system: str) -> List[Dict[str, Any]]:
|
||||
"""Collect filesystem usage statistics."""
|
||||
usage_stats = []
|
||||
|
||||
try:
|
||||
# Run df command
|
||||
result = subprocess.run(
|
||||
["ssh", system, "df -h --output=source,fstype,size,used,avail,pcent,target"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"df command failed: {result.stderr}")
|
||||
|
||||
# Parse output
|
||||
lines = result.stdout.strip().split('\n')
|
||||
if len(lines) < 2:
|
||||
return usage_stats
|
||||
|
||||
for line in lines[1:]: # Skip header
|
||||
parts = line.split()
|
||||
if len(parts) >= 7:
|
||||
usage_stat = {
|
||||
"filesystem": parts[0],
|
||||
"type": parts[1],
|
||||
"size": parts[2],
|
||||
"used": parts[3],
|
||||
"available": parts[4],
|
||||
"use_percent": parts[5],
|
||||
"mountpoint": parts[6]
|
||||
}
|
||||
usage_stats.append(usage_stat)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error(f"Timeout collecting filesystem usage from {system}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect filesystem usage from {system}: {e}")
|
||||
raise
|
||||
|
||||
return usage_stats
|
||||
|
||||
def collect_directory_sizes(self, system: str) -> List[Dict[str, Any]]:
|
||||
"""Collect sizes of top-level directories."""
|
||||
directory_sizes = []
|
||||
|
||||
try:
|
||||
# Get top-level directories
|
||||
dirs_to_check = ["/", "/home", "/var", "/usr", "/opt", "/etc"]
|
||||
|
||||
for directory in dirs_to_check:
|
||||
if directory in self.exclude_paths:
|
||||
continue
|
||||
|
||||
try:
|
||||
# Run du command for directory size
|
||||
result = subprocess.run(
|
||||
["ssh", system, f"du -sh {directory} 2>/dev/null"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
size, path = result.stdout.strip().split('\t', 1)
|
||||
directory_sizes.append({
|
||||
"path": path,
|
||||
"size": size
|
||||
})
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning(f"Timeout getting size for {directory} on {system}")
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get size for {directory} on {system}: {e}")
|
||||
continue
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect directory sizes from {system}: {e}")
|
||||
raise
|
||||
|
||||
return directory_sizes
|
||||
|
||||
def collect_largest_files(self, system: str) -> List[Dict[str, Any]]:
|
||||
"""Collect information about largest files in the system."""
|
||||
largest_files = []
|
||||
|
||||
try:
|
||||
# Find largest files (excluding certain paths)
|
||||
exclude_expr = " ".join(f"-not -path '{path}/*'" for path in self.exclude_paths)
|
||||
|
||||
cmd = f"find / {exclude_expr} -type f -exec ls -lh {{}} \\; 2>/dev/null | sort -k5 -hr | head -20"
|
||||
|
||||
result = subprocess.run(
|
||||
["ssh", system, cmd],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
for line in result.stdout.strip().split('\n'):
|
||||
if not line.strip():
|
||||
continue
|
||||
|
||||
parts = line.split()
|
||||
if len(parts) >= 9:
|
||||
file_info = {
|
||||
"permissions": parts[0],
|
||||
"links": parts[1],
|
||||
"owner": parts[2],
|
||||
"group": parts[3],
|
||||
"size": parts[4],
|
||||
"month": parts[5],
|
||||
"day": parts[6],
|
||||
"time": parts[7],
|
||||
"path": " ".join(parts[8:])
|
||||
}
|
||||
largest_files.append(file_info)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error(f"Timeout collecting largest files from {system}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect largest files from {system}: {e}")
|
||||
raise
|
||||
|
||||
return largest_files
|
||||
|
||||
def get_timestamp(self, system: str) -> str:
|
||||
"""Get current timestamp from target system."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ssh", system, "date -Iseconds"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
return result.stdout.strip()
|
||||
else:
|
||||
return "unknown"
|
||||
|
||||
except Exception:
|
||||
return "unknown"
|
||||
|
||||
def collect(system: str) -> Dict[str, Any]:
|
||||
"""Main collection function for disk usage data."""
|
||||
collector = DiskUsageCollector()
|
||||
return collector.collect_disk_usage(system)
|
||||
@@ -0,0 +1,173 @@
|
||||
"""
|
||||
Mounts Data Collector
|
||||
|
||||
Collects filesystem mount information including mount points, devices,
|
||||
filesystem types, and usage statistics.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
from typing import Dict, Any, List
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class MountsCollector:
|
||||
"""Collector for filesystem mount information."""
|
||||
|
||||
def __init__(self):
|
||||
self.exclude_patterns = [
|
||||
"/proc/*",
|
||||
"/sys/*",
|
||||
"/dev/*",
|
||||
"/run/*"
|
||||
]
|
||||
|
||||
def collect_mounts(self, system: str) -> Dict[str, Any]:
|
||||
"""Collect mount information from target system."""
|
||||
logger.info(f"Collecting mounts data from {system}")
|
||||
|
||||
try:
|
||||
# Run mount command
|
||||
result = subprocess.run(
|
||||
["ssh", system, "mount"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"Mount command failed: {result.stderr}")
|
||||
|
||||
mounts = self.parse_mount_output(result.stdout)
|
||||
filtered_mounts = self.filter_mounts(mounts)
|
||||
|
||||
# Get usage statistics
|
||||
usage_stats = self.collect_usage_stats(system, filtered_mounts)
|
||||
|
||||
return {
|
||||
"mounts": filtered_mounts,
|
||||
"usage": usage_stats,
|
||||
"timestamp": self.get_timestamp(system)
|
||||
}
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error(f"Timeout collecting mounts from {system}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect mounts from {system}: {e}")
|
||||
raise
|
||||
|
||||
def parse_mount_output(self, output: str) -> List[Dict[str, str]]:
|
||||
"""Parse mount command output."""
|
||||
mounts = []
|
||||
|
||||
for line in output.strip().split('\n'):
|
||||
if not line.strip():
|
||||
continue
|
||||
|
||||
# Parse mount output format: device on mountpoint type fstype (options)
|
||||
parts = line.split()
|
||||
if len(parts) >= 6 and parts[1] == 'on' and parts[3] == 'type':
|
||||
mount_info = {
|
||||
"device": parts[0],
|
||||
"mountpoint": parts[2],
|
||||
"fstype": parts[4],
|
||||
"options": parts[5].strip('()')
|
||||
}
|
||||
mounts.append(mount_info)
|
||||
|
||||
return mounts
|
||||
|
||||
def filter_mounts(self, mounts: List[Dict[str, str]]) -> List[Dict[str, str]]:
|
||||
"""Filter out unwanted mount points."""
|
||||
filtered = []
|
||||
|
||||
for mount in mounts:
|
||||
mountpoint = mount["mountpoint"]
|
||||
if not any(Path(mountpoint).match(pattern.rstrip('*')) for pattern in self.exclude_patterns):
|
||||
filtered.append(mount)
|
||||
|
||||
return filtered
|
||||
|
||||
def collect_usage_stats(self, system: str, mounts: List[Dict[str, str]]) -> Dict[str, Any]:
|
||||
"""Collect disk usage statistics for mount points."""
|
||||
usage_stats = {}
|
||||
|
||||
for mount in mounts:
|
||||
mountpoint = mount["mountpoint"]
|
||||
|
||||
try:
|
||||
# Run df command for usage statistics
|
||||
result = subprocess.run(
|
||||
["ssh", system, f"df -BG {mountpoint}"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=15
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
usage_stats[mountpoint] = self.parse_df_output(result.stdout)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning(f"Timeout getting usage for {mountpoint} on {system}")
|
||||
usage_stats[mountpoint] = {"error": "timeout"}
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get usage for {mountpoint} on {system}: {e}")
|
||||
usage_stats[mountpoint] = {"error": str(e)}
|
||||
|
||||
return usage_stats
|
||||
|
||||
def parse_df_output(self, output: str) -> Dict[str, Any]:
|
||||
"""Parse df command output."""
|
||||
lines = output.strip().split('\n')
|
||||
if len(lines) < 2:
|
||||
return {"error": "invalid df output"}
|
||||
|
||||
# Parse header and data
|
||||
header = lines[0].split()
|
||||
data = lines[1].split()
|
||||
|
||||
if len(header) != len(data):
|
||||
return {"error": "header/data mismatch"}
|
||||
|
||||
stats = {}
|
||||
for i, field in enumerate(header):
|
||||
if i < len(data):
|
||||
if field in ['1G-blocks', 'Used', 'Available']:
|
||||
# Convert to GB
|
||||
value = data[i]
|
||||
if value.endswith('G'):
|
||||
stats[field.lower()] = float(value.rstrip('G'))
|
||||
else:
|
||||
stats[field.lower()] = float(value) / (1024**3) # Assume bytes
|
||||
elif field == 'Use%':
|
||||
stats['use_percent'] = int(data[i].rstrip('%'))
|
||||
else:
|
||||
stats[field.lower()] = data[i]
|
||||
|
||||
return stats
|
||||
|
||||
def get_timestamp(self, system: str) -> str:
|
||||
"""Get current timestamp from target system."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ssh", system, "date -Iseconds"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
return result.stdout.strip()
|
||||
else:
|
||||
return "unknown"
|
||||
|
||||
except Exception:
|
||||
return "unknown"
|
||||
|
||||
def collect(system: str) -> Dict[str, Any]:
|
||||
"""Main collection function for mounts data."""
|
||||
collector = MountsCollector()
|
||||
return collector.collect_mounts(system)
|
||||
@@ -0,0 +1,223 @@
|
||||
"""
|
||||
Services Data Collector
|
||||
|
||||
Collects system service information including running services,
|
||||
their states, startup configuration, and dependencies.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
from typing import Dict, Any, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class ServicesCollector:
|
||||
"""Collector for system service information."""
|
||||
|
||||
def __init__(self):
|
||||
self.service_manager = "systemd" # Default to systemd
|
||||
self.include_disabled = False
|
||||
|
||||
def collect_services(self, system: str) -> Dict[str, Any]:
|
||||
"""Collect service information from target system."""
|
||||
logger.info(f"Collecting services data from {system}")
|
||||
|
||||
try:
|
||||
# Detect service manager
|
||||
service_manager = self.detect_service_manager(system)
|
||||
|
||||
if service_manager == "systemd":
|
||||
services = self.collect_systemd_services(system)
|
||||
elif service_manager == "sysv":
|
||||
services = self.collect_sysv_services(system)
|
||||
else:
|
||||
raise RuntimeError(f"Unsupported service manager: {service_manager}")
|
||||
|
||||
return {
|
||||
"service_manager": service_manager,
|
||||
"services": services,
|
||||
"timestamp": self.get_timestamp(system)
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect services from {system}: {e}")
|
||||
raise
|
||||
|
||||
def detect_service_manager(self, system: str) -> str:
|
||||
"""Detect which service manager is running on the system."""
|
||||
try:
|
||||
# Check for systemd
|
||||
result = subprocess.run(
|
||||
["ssh", system, "ps -p 1 -o comm="],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
if "systemd" in result.stdout.strip():
|
||||
return "systemd"
|
||||
elif "init" in result.stdout.strip():
|
||||
return "sysv"
|
||||
|
||||
# Fallback check
|
||||
result = subprocess.run(
|
||||
["ssh", system, "which systemctl"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
return "systemd"
|
||||
|
||||
return "sysv"
|
||||
|
||||
except Exception:
|
||||
return "unknown"
|
||||
|
||||
def collect_systemd_services(self, system: str) -> List[Dict[str, Any]]:
|
||||
"""Collect systemd service information."""
|
||||
services = []
|
||||
|
||||
try:
|
||||
# Get all services
|
||||
result = subprocess.run(
|
||||
["ssh", system, "systemctl list-units --type=service --all --no-pager --no-legend"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"systemctl list-units failed: {result.stderr}")
|
||||
|
||||
# Parse service list
|
||||
for line in result.stdout.strip().split('\n'):
|
||||
if not line.strip():
|
||||
continue
|
||||
|
||||
parts = line.split()
|
||||
if len(parts) >= 4:
|
||||
service_name = parts[0]
|
||||
load_state = parts[1]
|
||||
active_state = parts[2]
|
||||
sub_state = parts[3]
|
||||
|
||||
# Skip if disabled and not including disabled
|
||||
if not self.include_disabled and load_state == "not-found":
|
||||
continue
|
||||
|
||||
# Get detailed service info
|
||||
service_info = self.get_systemd_service_details(system, service_name)
|
||||
|
||||
services.append({
|
||||
"name": service_name,
|
||||
"load_state": load_state,
|
||||
"active_state": active_state,
|
||||
"sub_state": sub_state,
|
||||
**service_info
|
||||
})
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error(f"Timeout collecting systemd services from {system}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect systemd services from {system}: {e}")
|
||||
raise
|
||||
|
||||
return services
|
||||
|
||||
def get_systemd_service_details(self, system: str, service_name: str) -> Dict[str, Any]:
|
||||
"""Get detailed information for a systemd service."""
|
||||
details = {}
|
||||
|
||||
try:
|
||||
# Get service status
|
||||
result = subprocess.run(
|
||||
["ssh", system, f"systemctl show {service_name} --no-pager"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=15
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
for line in result.stdout.strip().split('\n'):
|
||||
if '=' in line:
|
||||
key, value = line.split('=', 1)
|
||||
details[key.lower()] = value
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get details for {service_name}: {e}")
|
||||
|
||||
return details
|
||||
|
||||
def collect_sysv_services(self, system: str) -> List[Dict[str, Any]]:
|
||||
"""Collect SysV init service information."""
|
||||
services = []
|
||||
|
||||
try:
|
||||
# Get service list from /etc/init.d/
|
||||
result = subprocess.run(
|
||||
["ssh", system, "ls -1 /etc/init.d/"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=15
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"Failed to list init.d services: {result.stderr}")
|
||||
|
||||
for service_name in result.stdout.strip().split('\n'):
|
||||
if not service_name.strip():
|
||||
continue
|
||||
|
||||
# Get service status
|
||||
status_result = subprocess.run(
|
||||
["ssh", system, f"/etc/init.d/{service_name} status"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
|
||||
status = "unknown"
|
||||
if status_result.returncode == 0:
|
||||
status = "running"
|
||||
elif "not running" in status_result.stdout.lower():
|
||||
status = "stopped"
|
||||
|
||||
services.append({
|
||||
"name": service_name,
|
||||
"status": status,
|
||||
"type": "sysv"
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to collect SysV services from {system}: {e}")
|
||||
raise
|
||||
|
||||
return services
|
||||
|
||||
def get_timestamp(self, system: str) -> str:
|
||||
"""Get current timestamp from target system."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ssh", system, "date -Iseconds"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
return result.stdout.strip()
|
||||
else:
|
||||
return "unknown"
|
||||
|
||||
except Exception:
|
||||
return "unknown"
|
||||
|
||||
def collect(system: str) -> Dict[str, Any]:
|
||||
"""Main collection function for services data."""
|
||||
collector = ServicesCollector()
|
||||
return collector.collect_services(system)
|
||||
@@ -0,0 +1,30 @@
|
||||
# Migration Validation Framework Architecture
|
||||
|
||||
## Components
|
||||
|
||||
- CLI: parses operator commands and coordinates workflows.
|
||||
- Collectors: gather mounts, services, and disk usage from target systems.
|
||||
- Snapshot files: JSON evidence used as immutable migration checkpoints.
|
||||
- Comparator: evaluates drift between before and after snapshots.
|
||||
- Reports: stores JSON or HTML output for audit and review.
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Operator
|
||||
-> python3 cli.py collect
|
||||
-> collectors over SSH
|
||||
-> before.json / after.json
|
||||
-> python3 cli.py compare
|
||||
-> diff.json with PASS/FAIL validation
|
||||
```
|
||||
|
||||
## Validation Flow
|
||||
|
||||
```
|
||||
before.json -> Comparator -> service checks
|
||||
after.json -> Comparator -> filesystem checks -> validation result
|
||||
-> mount checks
|
||||
```
|
||||
|
||||
The framework keeps collection and comparison separate so migration evidence can be reviewed, archived, and replayed without recollecting from production systems.
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"metadata": {
|
||||
"timestamp": "2026-04-29T03:40:00Z",
|
||||
"systems": ["web01"],
|
||||
"version": "1.0"
|
||||
},
|
||||
"data": {
|
||||
"web01": {
|
||||
"mounts": {
|
||||
"mounts": [
|
||||
{"device": "/dev/sda1", "mountpoint": "/", "fstype": "ext4", "options": "rw,relatime"},
|
||||
{"device": "/dev/sdb1", "mountpoint": "/var", "fstype": "xfs", "options": "rw,noatime"}
|
||||
],
|
||||
"usage": {
|
||||
"/": {"filesystem": "/dev/sda1", "use_percent": "62%"},
|
||||
"/var": {"filesystem": "/dev/sdb1", "use_percent": "94%"}
|
||||
},
|
||||
"timestamp": "2026-04-29T03:40:00Z"
|
||||
},
|
||||
"services": {
|
||||
"service_manager": "systemd",
|
||||
"services": [
|
||||
{"name": "sshd", "active_state": "failed", "sub_state": "failed"},
|
||||
{"name": "nginx", "active_state": "active", "sub_state": "running"},
|
||||
{"name": "node-exporter", "active_state": "active", "sub_state": "running"}
|
||||
],
|
||||
"timestamp": "2026-04-29T03:40:00Z"
|
||||
},
|
||||
"disk_usage": {
|
||||
"filesystem_usage": [
|
||||
{"filesystem": "/dev/sda1", "type": "ext4", "size": "80G", "used": "50G", "available": "30G", "use_percent": "62%", "mountpoint": "/"},
|
||||
{"filesystem": "/dev/sdb1", "type": "xfs", "size": "200G", "used": "188G", "available": "12G", "use_percent": "94%", "mountpoint": "/var"}
|
||||
],
|
||||
"directory_sizes": [{"path": "/var/lib/app", "size": "139G"}],
|
||||
"largest_files": [{"path": "/var/lib/app/import/archive.tar", "size": "42G"}],
|
||||
"timestamp": "2026-04-29T03:40:00Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"metadata": {
|
||||
"timestamp": "2026-04-29T01:15:00Z",
|
||||
"systems": ["web01"],
|
||||
"version": "1.0"
|
||||
},
|
||||
"data": {
|
||||
"web01": {
|
||||
"mounts": {
|
||||
"mounts": [
|
||||
{"device": "/dev/sda1", "mountpoint": "/", "fstype": "ext4", "options": "rw,relatime"},
|
||||
{"device": "/dev/sdb1", "mountpoint": "/var", "fstype": "xfs", "options": "rw,noatime"}
|
||||
],
|
||||
"usage": {
|
||||
"/": {"filesystem": "/dev/sda1", "use_percent": "61%"},
|
||||
"/var": {"filesystem": "/dev/sdb1", "use_percent": "68%"}
|
||||
},
|
||||
"timestamp": "2026-04-29T01:15:00Z"
|
||||
},
|
||||
"services": {
|
||||
"service_manager": "systemd",
|
||||
"services": [
|
||||
{"name": "sshd", "active_state": "active", "sub_state": "running"},
|
||||
{"name": "nginx", "active_state": "active", "sub_state": "running"}
|
||||
],
|
||||
"timestamp": "2026-04-29T01:15:00Z"
|
||||
},
|
||||
"disk_usage": {
|
||||
"filesystem_usage": [
|
||||
{"filesystem": "/dev/sda1", "type": "ext4", "size": "80G", "used": "49G", "available": "31G", "use_percent": "61%", "mountpoint": "/"},
|
||||
{"filesystem": "/dev/sdb1", "type": "xfs", "size": "200G", "used": "136G", "available": "64G", "use_percent": "68%", "mountpoint": "/var"}
|
||||
],
|
||||
"directory_sizes": [{"path": "/var/lib/app", "size": "84G"}],
|
||||
"largest_files": [],
|
||||
"timestamp": "2026-04-29T01:15:00Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
{
|
||||
"summary": {
|
||||
"total_systems": 1,
|
||||
"systems_with_changes": 1,
|
||||
"total_changes": 7,
|
||||
"changes_by_type": {
|
||||
"mounts": 2,
|
||||
"services": 2,
|
||||
"disk_usage": 3
|
||||
},
|
||||
"most_affected_systems": [
|
||||
[
|
||||
"web01",
|
||||
7
|
||||
]
|
||||
]
|
||||
},
|
||||
"differences": {
|
||||
"mounts": {
|
||||
"web01": {
|
||||
"added_mounts": [],
|
||||
"removed_mounts": [],
|
||||
"changed_mounts": [],
|
||||
"usage_changes": [
|
||||
{
|
||||
"mountpoint": "/",
|
||||
"before": {
|
||||
"filesystem": "/dev/sda1",
|
||||
"use_percent": "61%"
|
||||
},
|
||||
"after": {
|
||||
"filesystem": "/dev/sda1",
|
||||
"use_percent": "62%"
|
||||
}
|
||||
},
|
||||
{
|
||||
"mountpoint": "/var",
|
||||
"before": {
|
||||
"filesystem": "/dev/sdb1",
|
||||
"use_percent": "68%"
|
||||
},
|
||||
"after": {
|
||||
"filesystem": "/dev/sdb1",
|
||||
"use_percent": "94%"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"services": {
|
||||
"web01": {
|
||||
"added_services": [
|
||||
{
|
||||
"name": "node-exporter",
|
||||
"active_state": "active",
|
||||
"sub_state": "running"
|
||||
}
|
||||
],
|
||||
"removed_services": [],
|
||||
"status_changes": [
|
||||
{
|
||||
"name": "sshd",
|
||||
"before": {
|
||||
"active_state": "active",
|
||||
"sub_state": "running"
|
||||
},
|
||||
"after": {
|
||||
"active_state": "failed",
|
||||
"sub_state": "failed"
|
||||
}
|
||||
}
|
||||
],
|
||||
"configuration_changes": []
|
||||
}
|
||||
},
|
||||
"disk_usage": {
|
||||
"web01": {
|
||||
"filesystem_changes": [
|
||||
{
|
||||
"mountpoint": "/",
|
||||
"before": {
|
||||
"filesystem": "/dev/sda1",
|
||||
"type": "ext4",
|
||||
"size": "80G",
|
||||
"used": "49G",
|
||||
"available": "31G",
|
||||
"use_percent": "61%",
|
||||
"mountpoint": "/"
|
||||
},
|
||||
"after": {
|
||||
"filesystem": "/dev/sda1",
|
||||
"type": "ext4",
|
||||
"size": "80G",
|
||||
"used": "50G",
|
||||
"available": "30G",
|
||||
"use_percent": "62%",
|
||||
"mountpoint": "/"
|
||||
}
|
||||
},
|
||||
{
|
||||
"mountpoint": "/var",
|
||||
"before": {
|
||||
"filesystem": "/dev/sdb1",
|
||||
"type": "xfs",
|
||||
"size": "200G",
|
||||
"used": "136G",
|
||||
"available": "64G",
|
||||
"use_percent": "68%",
|
||||
"mountpoint": "/var"
|
||||
},
|
||||
"after": {
|
||||
"filesystem": "/dev/sdb1",
|
||||
"type": "xfs",
|
||||
"size": "200G",
|
||||
"used": "188G",
|
||||
"available": "12G",
|
||||
"use_percent": "94%",
|
||||
"mountpoint": "/var"
|
||||
}
|
||||
}
|
||||
],
|
||||
"directory_size_changes": [],
|
||||
"significant_usage_changes": [
|
||||
{
|
||||
"mountpoint": "/var",
|
||||
"change_percent": 26,
|
||||
"before": {
|
||||
"filesystem": "/dev/sdb1",
|
||||
"type": "xfs",
|
||||
"size": "200G",
|
||||
"used": "136G",
|
||||
"available": "64G",
|
||||
"use_percent": "68%",
|
||||
"mountpoint": "/var"
|
||||
},
|
||||
"after": {
|
||||
"filesystem": "/dev/sdb1",
|
||||
"type": "xfs",
|
||||
"size": "200G",
|
||||
"used": "188G",
|
||||
"available": "12G",
|
||||
"use_percent": "94%",
|
||||
"mountpoint": "/var"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"risk_assessment": {
|
||||
"overall_risk": "high",
|
||||
"risk_factors": [
|
||||
{
|
||||
"type": "service_failure",
|
||||
"description": "Service failed: sshd",
|
||||
"level": 3
|
||||
},
|
||||
{
|
||||
"type": "disk_usage_spike",
|
||||
"description": "Significant disk usage change: /var (26%)",
|
||||
"level": 2
|
||||
}
|
||||
],
|
||||
"critical_changes": [],
|
||||
"recommendations": [
|
||||
"Immediate review required - critical changes detected",
|
||||
"Consider rolling back migration if critical services are affected"
|
||||
]
|
||||
},
|
||||
"validation_results": {
|
||||
"passed": false,
|
||||
"checks": [
|
||||
{
|
||||
"name": "critical_services_running",
|
||||
"description": "Verify critical services remain operational",
|
||||
"passed": false,
|
||||
"details": [
|
||||
"Critical service sshd failed on web01"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "filesystem_integrity",
|
||||
"description": "Verify filesystem integrity maintained",
|
||||
"passed": true,
|
||||
"details": []
|
||||
},
|
||||
{
|
||||
"name": "no_critical_mounts_removed",
|
||||
"description": "Verify critical mount points remain",
|
||||
"passed": true,
|
||||
"details": []
|
||||
}
|
||||
],
|
||||
"failed_checks": [
|
||||
{
|
||||
"name": "critical_services_running",
|
||||
"description": "Verify critical services remain operational",
|
||||
"passed": false,
|
||||
"details": [
|
||||
"Critical service sshd failed on web01"
|
||||
]
|
||||
}
|
||||
],
|
||||
"result": "FAIL"
|
||||
},
|
||||
"metadata": {
|
||||
"before": "migration-validation-framework/examples/before.json",
|
||||
"after": "migration-validation-framework/examples/after.json",
|
||||
"timestamp": "2026-04-29T23:29:07.510774"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,608 @@
|
||||
"""
|
||||
HTML Report Generator
|
||||
|
||||
Generates comprehensive HTML reports from migration validation comparison results.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Dict, Any
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class HTMLReportGenerator:
|
||||
"""Generator for HTML migration validation reports."""
|
||||
|
||||
def __init__(self):
|
||||
self.css_styles = self.get_css_styles()
|
||||
self.js_scripts = self.get_js_scripts()
|
||||
|
||||
def generate_report(self, comparison: Dict[str, Any], output_file: str) -> str:
|
||||
"""Generate HTML report from comparison data."""
|
||||
logger.info(f"Generating HTML report: {output_file}")
|
||||
|
||||
html_content = self.build_html_content(comparison)
|
||||
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
f.write(html_content)
|
||||
|
||||
logger.info(f"HTML report generated: {output_file}")
|
||||
return output_file
|
||||
|
||||
def build_html_content(self, comparison: Dict[str, Any]) -> str:
|
||||
"""Build complete HTML content."""
|
||||
metadata = comparison.get("metadata", {})
|
||||
summary = comparison.get("summary", {})
|
||||
differences = comparison.get("differences", {})
|
||||
risk_assessment = comparison.get("risk_assessment", {})
|
||||
validation_results = comparison.get("validation_results", {})
|
||||
|
||||
html = f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Migration Validation Report</title>
|
||||
<style>{self.css_styles}</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<header>
|
||||
<h1>Migration Validation Report</h1>
|
||||
<div class="report-meta">
|
||||
<p><strong>Report Generated:</strong> {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}</p>
|
||||
<p><strong>Comparison ID:</strong> {metadata.get('comparison_id', 'N/A')}</p>
|
||||
<p><strong>Snapshot 1:</strong> {metadata.get('snapshot1', 'N/A')}</p>
|
||||
<p><strong>Snapshot 2:</strong> {metadata.get('snapshot2', 'N/A')}</p>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<nav class="toc">
|
||||
<h2>Table of Contents</h2>
|
||||
<ul>
|
||||
<li><a href="#executive-summary">Executive Summary</a></li>
|
||||
<li><a href="#risk-assessment">Risk Assessment</a></li>
|
||||
<li><a href="#validation-results">Validation Results</a></li>
|
||||
<li><a href="#detailed-changes">Detailed Changes</a></li>
|
||||
<li><a href="#recommendations">Recommendations</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
<section id="executive-summary">
|
||||
<h2>Executive Summary</h2>
|
||||
{self.generate_summary_section(summary)}
|
||||
</section>
|
||||
|
||||
<section id="risk-assessment">
|
||||
<h2>Risk Assessment</h2>
|
||||
{self.generate_risk_section(risk_assessment)}
|
||||
</section>
|
||||
|
||||
<section id="validation-results">
|
||||
<h2>Validation Results</h2>
|
||||
{self.generate_validation_section(validation_results)}
|
||||
</section>
|
||||
|
||||
<section id="detailed-changes">
|
||||
<h2>Detailed Changes</h2>
|
||||
{self.generate_changes_section(differences)}
|
||||
</section>
|
||||
|
||||
<section id="recommendations">
|
||||
<h2>Recommendations</h2>
|
||||
{self.generate_recommendations_section(risk_assessment)}
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<script>{self.js_scripts}</script>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
return html
|
||||
|
||||
def generate_summary_section(self, summary: Dict[str, Any]) -> str:
|
||||
"""Generate executive summary HTML."""
|
||||
total_systems = summary.get('total_systems', 0)
|
||||
systems_with_changes = summary.get('systems_with_changes', 0)
|
||||
total_changes = summary.get('total_changes', 0)
|
||||
|
||||
html = f"""
|
||||
<div class="summary-grid">
|
||||
<div class="summary-card">
|
||||
<h3>Systems Analyzed</h3>
|
||||
<div class="metric">{total_systems}</div>
|
||||
</div>
|
||||
<div class="summary-card">
|
||||
<h3>Systems with Changes</h3>
|
||||
<div class="metric">{systems_with_changes}</div>
|
||||
</div>
|
||||
<div class="summary-card">
|
||||
<h3>Total Changes</h3>
|
||||
<div class="metric">{total_changes}</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3>Changes by Type</h3>
|
||||
<table class="changes-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Data Type</th>
|
||||
<th>Changes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>"""
|
||||
|
||||
for data_type, count in summary.get('changes_by_type', {}).items():
|
||||
html += f"""
|
||||
<tr>
|
||||
<td>{data_type.replace('_', ' ').title()}</td>
|
||||
<td>{count}</td>
|
||||
</tr>"""
|
||||
|
||||
html += """
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>Most Affected Systems</h3>
|
||||
<table class="systems-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>System</th>
|
||||
<th>Changes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>"""
|
||||
|
||||
for system, count in summary.get('most_affected_systems', []):
|
||||
html += f"""
|
||||
<tr>
|
||||
<td>{system}</td>
|
||||
<td>{count}</td>
|
||||
</tr>"""
|
||||
|
||||
html += """
|
||||
</tbody>
|
||||
</table>"""
|
||||
|
||||
return html
|
||||
|
||||
def generate_risk_section(self, risk_assessment: Dict[str, Any]) -> str:
|
||||
"""Generate risk assessment HTML."""
|
||||
overall_risk = risk_assessment.get('overall_risk', 'unknown')
|
||||
risk_color = self.get_risk_color(overall_risk)
|
||||
|
||||
html = f"""
|
||||
<div class="risk-overview">
|
||||
<h3>Overall Risk Level</h3>
|
||||
<div class="risk-badge risk-{overall_risk}" style="background-color: {risk_color}">
|
||||
{overall_risk.upper()}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3>Risk Factors</h3>
|
||||
<div class="risk-factors">"""
|
||||
|
||||
for factor in risk_assessment.get('risk_factors', []):
|
||||
factor_color = self.get_risk_color(self.get_risk_level_name(factor.get('level', 1)))
|
||||
html += f"""
|
||||
<div class="risk-factor">
|
||||
<div class="risk-badge risk-{self.get_risk_level_name(factor.get('level', 1))}" style="background-color: {factor_color}">
|
||||
{self.get_risk_level_name(factor.get('level', 1)).upper()}
|
||||
</div>
|
||||
<div class="risk-details">
|
||||
<strong>{factor.get('type', 'Unknown').replace('_', ' ').title()}</strong>
|
||||
<p>{factor.get('description', 'No description')}</p>
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
html += """
|
||||
</div>
|
||||
|
||||
<h3>Critical Changes</h3>
|
||||
<div class="critical-changes">"""
|
||||
|
||||
for change in risk_assessment.get('critical_changes', []):
|
||||
html += f"""
|
||||
<div class="critical-change">
|
||||
<h4>{change.get('system', 'Unknown System')}</h4>
|
||||
<p><strong>Type:</strong> {change.get('data_type', 'Unknown')}</p>
|
||||
<p>{change.get('factor', {}).get('description', 'No details')}</p>
|
||||
</div>"""
|
||||
|
||||
html += "</div>"
|
||||
|
||||
return html
|
||||
|
||||
def generate_validation_section(self, validation_results: Dict[str, Any]) -> str:
|
||||
"""Generate validation results HTML."""
|
||||
passed = validation_results.get('passed', False)
|
||||
status_color = "#28a745" if passed else "#dc3545"
|
||||
status_text = "PASSED" if passed else "FAILED"
|
||||
|
||||
html = f"""
|
||||
<div class="validation-status">
|
||||
<div class="status-indicator" style="background-color: {status_color}">
|
||||
{status_text}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3>Validation Checks</h3>
|
||||
<div class="validation-checks">"""
|
||||
|
||||
for check in validation_results.get('checks', []):
|
||||
check_status = "✓" if check.get('passed', False) else "✗"
|
||||
check_color = "#28a745" if check.get('passed', False) else "#dc3545"
|
||||
|
||||
html += f"""
|
||||
<div class="validation-check">
|
||||
<div class="check-status" style="color: {check_color}">{check_status}</div>
|
||||
<div class="check-details">
|
||||
<h4>{check.get('name', 'Unknown').replace('_', ' ').title()}</h4>
|
||||
<p>{check.get('description', 'No description')}</p>
|
||||
{"<ul>" + "".join(f"<li>{detail}</li>" for detail in check.get('details', [])) + "</ul>" if check.get('details') else ""}
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
html += "</div>"
|
||||
|
||||
return html
|
||||
|
||||
def generate_changes_section(self, differences: Dict[str, Any]) -> str:
|
||||
"""Generate detailed changes HTML."""
|
||||
html = ""
|
||||
|
||||
for data_type, systems in differences.items():
|
||||
html += f"<h3>{data_type.replace('_', ' ').title()} Changes</h3>"
|
||||
|
||||
for system, system_diffs in systems.items():
|
||||
html += f"<h4>System: {system}</h4>"
|
||||
|
||||
# Generate tables for different change types
|
||||
html += self.generate_change_tables(system_diffs)
|
||||
|
||||
return html
|
||||
|
||||
def generate_change_tables(self, system_diffs: Dict[str, Any]) -> str:
|
||||
"""Generate HTML tables for different types of changes."""
|
||||
html = ""
|
||||
|
||||
change_types = {
|
||||
'added_mounts': ('Added Mounts', ['mountpoint', 'device', 'fstype']),
|
||||
'removed_mounts': ('Removed Mounts', ['mountpoint', 'device', 'fstype']),
|
||||
'changed_mounts': ('Changed Mounts', ['mountpoint', 'before', 'after']),
|
||||
'usage_changes': ('Usage Changes', ['mountpoint', 'before', 'after']),
|
||||
'added_services': ('Added Services', ['name', 'active_state', 'sub_state']),
|
||||
'removed_services': ('Removed Services', ['name', 'active_state', 'sub_state']),
|
||||
'status_changes': ('Service Status Changes', ['name', 'before', 'after']),
|
||||
'filesystem_changes': ('Filesystem Changes', ['mountpoint', 'before', 'after']),
|
||||
'directory_size_changes': ('Directory Size Changes', ['path', 'before', 'after']),
|
||||
'significant_usage_changes': ('Significant Usage Changes', ['mountpoint', 'change_percent', 'before', 'after'])
|
||||
}
|
||||
|
||||
for change_key, (title, columns) in change_types.items():
|
||||
if change_key in system_diffs and system_diffs[change_key]:
|
||||
html += f"<h5>{title}</h5>"
|
||||
html += '<table class="changes-table">'
|
||||
html += '<thead><tr>'
|
||||
|
||||
for col in columns:
|
||||
html += f'<th>{col.replace("_", " ").title()}</th>'
|
||||
|
||||
html += '</tr></thead><tbody>'
|
||||
|
||||
for item in system_diffs[change_key]:
|
||||
html += '<tr>'
|
||||
|
||||
for col in columns:
|
||||
value = item.get(col, '')
|
||||
if isinstance(value, dict):
|
||||
value = json.dumps(value, indent=2)
|
||||
html += f'<td><pre>{value}</pre></td>'
|
||||
|
||||
html += '</tr>'
|
||||
|
||||
html += '</tbody></table>'
|
||||
|
||||
return html
|
||||
|
||||
def generate_recommendations_section(self, risk_assessment: Dict[str, Any]) -> str:
|
||||
"""Generate recommendations HTML."""
|
||||
recommendations = risk_assessment.get('recommendations', [])
|
||||
|
||||
html = '<div class="recommendations">'
|
||||
|
||||
if not recommendations:
|
||||
html += '<p>No specific recommendations at this time.</p>'
|
||||
else:
|
||||
html += '<ul>'
|
||||
for rec in recommendations:
|
||||
html += f'<li>{rec}</li>'
|
||||
html += '</ul>'
|
||||
|
||||
html += '</div>'
|
||||
|
||||
return html
|
||||
|
||||
def get_risk_color(self, risk_level: str) -> str:
|
||||
"""Get color for risk level."""
|
||||
colors = {
|
||||
'low': '#28a745',
|
||||
'medium': '#ffc107',
|
||||
'high': '#fd7e14',
|
||||
'critical': '#dc3545'
|
||||
}
|
||||
return colors.get(risk_level.lower(), '#6c757d')
|
||||
|
||||
def get_risk_level_name(self, level: int) -> str:
|
||||
"""Get risk level name from numeric level."""
|
||||
levels = {1: 'low', 2: 'medium', 3: 'high', 4: 'critical'}
|
||||
return levels.get(level, 'unknown')
|
||||
|
||||
def get_css_styles(self) -> str:
|
||||
"""Get CSS styles for the report."""
|
||||
return """
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background-color: #f8f9fa;
|
||||
}
|
||||
|
||||
.container {
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
padding: 30px;
|
||||
border-radius: 8px;
|
||||
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
|
||||
}
|
||||
|
||||
header {
|
||||
text-align: center;
|
||||
margin-bottom: 40px;
|
||||
padding-bottom: 20px;
|
||||
border-bottom: 2px solid #e9ecef;
|
||||
}
|
||||
|
||||
h1 {
|
||||
color: #2c3e50;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.report-meta {
|
||||
color: #6c757d;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
|
||||
.toc {
|
||||
background: #f8f9fa;
|
||||
padding: 20px;
|
||||
border-radius: 5px;
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.toc ul {
|
||||
list-style: none;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.toc li {
|
||||
margin: 5px 0;
|
||||
}
|
||||
|
||||
.toc a {
|
||||
color: #007bff;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.toc a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
section {
|
||||
margin-bottom: 40px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
color: #2c3e50;
|
||||
border-bottom: 1px solid #e9ecef;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
h3 {
|
||||
color: #495057;
|
||||
margin-top: 30px;
|
||||
}
|
||||
|
||||
.summary-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||
gap: 20px;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.summary-card {
|
||||
background: #f8f9fa;
|
||||
padding: 20px;
|
||||
border-radius: 5px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.metric {
|
||||
font-size: 2em;
|
||||
font-weight: bold;
|
||||
color: #007bff;
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
th, td {
|
||||
padding: 12px;
|
||||
text-align: left;
|
||||
border-bottom: 1px solid #e9ecef;
|
||||
}
|
||||
|
||||
th {
|
||||
background-color: #f8f9fa;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.risk-overview {
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.risk-badge {
|
||||
display: inline-block;
|
||||
padding: 8px 16px;
|
||||
border-radius: 20px;
|
||||
color: white;
|
||||
font-weight: bold;
|
||||
margin: 10px;
|
||||
}
|
||||
|
||||
.risk-factors, .critical-changes {
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.risk-factor, .critical-change {
|
||||
background: #f8f9fa;
|
||||
padding: 15px;
|
||||
border-radius: 5px;
|
||||
margin: 10px 0;
|
||||
border-left: 4px solid #007bff;
|
||||
}
|
||||
|
||||
.risk-factor {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.risk-details {
|
||||
margin-left: 15px;
|
||||
}
|
||||
|
||||
.validation-status {
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.status-indicator {
|
||||
display: inline-block;
|
||||
padding: 15px 30px;
|
||||
border-radius: 5px;
|
||||
color: white;
|
||||
font-weight: bold;
|
||||
font-size: 1.2em;
|
||||
}
|
||||
|
||||
.validation-checks {
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.validation-check {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
background: #f8f9fa;
|
||||
padding: 15px;
|
||||
border-radius: 5px;
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
.check-status {
|
||||
font-size: 1.5em;
|
||||
margin-right: 15px;
|
||||
margin-top: 5px;
|
||||
}
|
||||
|
||||
.check-details {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.recommendations ul {
|
||||
background: #e7f3ff;
|
||||
padding: 20px;
|
||||
border-radius: 5px;
|
||||
border-left: 4px solid #007bff;
|
||||
}
|
||||
|
||||
.recommendations li {
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
pre {
|
||||
background: #f8f9fa;
|
||||
padding: 10px;
|
||||
border-radius: 3px;
|
||||
overflow-x: auto;
|
||||
max-width: 100%;
|
||||
white-space: pre-wrap;
|
||||
word-wrap: break-word;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.container {
|
||||
padding: 15px;
|
||||
}
|
||||
|
||||
.summary-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.risk-factor {
|
||||
flex-direction: column;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.validation-check {
|
||||
flex-direction: column;
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
def get_js_scripts(self) -> str:
|
||||
"""Get JavaScript for interactive features."""
|
||||
return """
|
||||
// Add smooth scrolling to TOC links
|
||||
document.addEventListener('DOMContentLoaded', function() {
|
||||
const links = document.querySelectorAll('.toc a');
|
||||
links.forEach(link => {
|
||||
link.addEventListener('click', function(e) {
|
||||
e.preventDefault();
|
||||
const target = document.querySelector(this.getAttribute('href'));
|
||||
if (target) {
|
||||
target.scrollIntoView({ behavior: 'smooth' });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// Add collapsible sections for large tables
|
||||
const tables = document.querySelectorAll('table');
|
||||
tables.forEach(table => {
|
||||
if (table.rows.length > 10) {
|
||||
const toggle = document.createElement('button');
|
||||
toggle.textContent = 'Toggle Details';
|
||||
toggle.style.marginBottom = '10px';
|
||||
toggle.addEventListener('click', function() {
|
||||
const tbody = table.querySelector('tbody');
|
||||
tbody.style.display = tbody.style.display === 'none' ? '' : 'none';
|
||||
});
|
||||
table.parentNode.insertBefore(toggle, table);
|
||||
}
|
||||
});
|
||||
});
|
||||
"""
|
||||
|
||||
def generate(comparison: Dict[str, Any], output_file: str) -> str:
|
||||
"""Main function to generate HTML report."""
|
||||
generator = HTMLReportGenerator()
|
||||
return generator.generate_report(comparison, output_file)
|
||||
@@ -0,0 +1,19 @@
|
||||
# Scenario: Before/After Migration Comparison
|
||||
|
||||
## Description
|
||||
|
||||
Compare a pre-cutover host snapshot against a post-cutover snapshot and determine whether the migrated system is ready for production traffic.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
cd migration-validation-framework
|
||||
python3 cli.py compare examples/before.json examples/after.json --output /tmp/migration-diff.json
|
||||
```
|
||||
|
||||
## Expected Result
|
||||
|
||||
- The command writes a JSON diff.
|
||||
- The result is `FAIL` because `sshd` is failed after migration.
|
||||
- The risk assessment highlights the `/var` disk usage increase.
|
||||
- The remediation path is to restore SSH and reduce or expand `/var` before approving cutover.
|
||||
@@ -0,0 +1,501 @@
|
||||
"""
|
||||
Snapshot Comparison Engine
|
||||
|
||||
Compares two system snapshots and identifies differences,
|
||||
risk levels, and validation results.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Dict, Any, List, Tuple
|
||||
from datetime import datetime
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class SnapshotComparator:
|
||||
"""Engine for comparing system snapshots."""
|
||||
|
||||
def __init__(self):
|
||||
self.risk_levels = {
|
||||
"low": 1,
|
||||
"medium": 2,
|
||||
"high": 3,
|
||||
"critical": 4
|
||||
}
|
||||
|
||||
def compare_snapshots(self, snapshot1: Dict[str, Any], snapshot2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Compare two snapshots and return detailed comparison results."""
|
||||
logger.info("Starting snapshot comparison")
|
||||
|
||||
comparison = {
|
||||
"summary": {},
|
||||
"differences": {},
|
||||
"risk_assessment": {},
|
||||
"validation_results": {}
|
||||
}
|
||||
|
||||
# Compare each data type
|
||||
data_types = ["mounts", "services", "disk_usage"]
|
||||
|
||||
data1 = snapshot1.get("data", {})
|
||||
data2 = snapshot2.get("data", {})
|
||||
|
||||
for data_type in data_types:
|
||||
if self.data_type_exists(data1, data_type) or self.data_type_exists(data2, data_type):
|
||||
differences = self.compare_data_type(data1, data2, data_type)
|
||||
comparison["differences"][data_type] = differences
|
||||
|
||||
# Generate summary
|
||||
comparison["summary"] = self.generate_summary(comparison["differences"])
|
||||
|
||||
# Risk assessment
|
||||
comparison["risk_assessment"] = self.assess_risks(comparison["differences"])
|
||||
|
||||
# Validation results
|
||||
comparison["validation_results"] = self.validate_changes(comparison["differences"])
|
||||
comparison["validation_results"]["result"] = "PASS" if comparison["validation_results"]["passed"] else "FAIL"
|
||||
|
||||
logger.info("Snapshot comparison completed")
|
||||
return comparison
|
||||
|
||||
def data_type_exists(self, systems: Dict[str, Any], data_type: str) -> bool:
|
||||
"""Return true when at least one system has the requested collector data."""
|
||||
return any(data_type in system_data for system_data in systems.values())
|
||||
|
||||
def compare_data_type(self, data1: Dict[str, Any], data2: Dict[str, Any], data_type: str) -> Dict[str, Any]:
|
||||
"""Compare a specific data type between two snapshots."""
|
||||
differences = {}
|
||||
|
||||
# Get all systems from both snapshots
|
||||
systems1 = set(data1.keys())
|
||||
systems2 = set(data2.keys())
|
||||
all_systems = systems1.union(systems2)
|
||||
|
||||
for system in all_systems:
|
||||
system_diffs = {}
|
||||
|
||||
if system not in data1:
|
||||
system_diffs["status"] = "added"
|
||||
system_diffs["details"] = {"new_system": True}
|
||||
elif system not in data2:
|
||||
system_diffs["status"] = "removed"
|
||||
system_diffs["details"] = {"removed_system": True}
|
||||
else:
|
||||
# Compare data for this system and data type
|
||||
if data_type in data1[system] and data_type in data2[system]:
|
||||
system_diffs = self.compare_system_data(
|
||||
data1[system][data_type],
|
||||
data2[system][data_type],
|
||||
data_type
|
||||
)
|
||||
else:
|
||||
system_diffs["status"] = "data_missing"
|
||||
system_diffs["details"] = {"missing_data_type": data_type}
|
||||
|
||||
if system_diffs:
|
||||
differences[system] = system_diffs
|
||||
|
||||
return differences
|
||||
|
||||
def compare_system_data(self, data1: Dict[str, Any], data2: Dict[str, Any], data_type: str) -> Dict[str, Any]:
|
||||
"""Compare data for a specific system and data type."""
|
||||
differences = {}
|
||||
|
||||
if data_type == "mounts":
|
||||
differences = self.compare_mounts(data1, data2)
|
||||
elif data_type == "services":
|
||||
differences = self.compare_services(data1, data2)
|
||||
elif data_type == "disk_usage":
|
||||
differences = self.compare_disk_usage(data1, data2)
|
||||
else:
|
||||
differences["status"] = "unknown_data_type"
|
||||
|
||||
return differences
|
||||
|
||||
def compare_mounts(self, mounts1: Dict[str, Any], mounts2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Compare mounts data between snapshots."""
|
||||
differences = {
|
||||
"added_mounts": [],
|
||||
"removed_mounts": [],
|
||||
"changed_mounts": [],
|
||||
"usage_changes": []
|
||||
}
|
||||
|
||||
# Compare mount lists
|
||||
mounts_list1 = mounts1.get("mounts", [])
|
||||
mounts_list2 = mounts2.get("mounts", [])
|
||||
|
||||
# Create mountpoint maps
|
||||
mounts_map1 = {m["mountpoint"]: m for m in mounts_list1}
|
||||
mounts_map2 = {m["mountpoint"]: m for m in mounts_list2}
|
||||
|
||||
# Find added and removed mounts
|
||||
added = set(mounts_map2.keys()) - set(mounts_map1.keys())
|
||||
removed = set(mounts_map1.keys()) - set(mounts_map2.keys())
|
||||
|
||||
differences["added_mounts"] = [{"mountpoint": mp, **mounts_map2[mp]} for mp in added]
|
||||
differences["removed_mounts"] = [{"mountpoint": mp, **mounts_map1[mp]} for mp in removed]
|
||||
|
||||
# Find changed mounts
|
||||
common = set(mounts_map1.keys()) & set(mounts_map2.keys())
|
||||
for mp in common:
|
||||
m1, m2 = mounts_map1[mp], mounts_map2[mp]
|
||||
if m1 != m2:
|
||||
differences["changed_mounts"].append({
|
||||
"mountpoint": mp,
|
||||
"before": m1,
|
||||
"after": m2
|
||||
})
|
||||
|
||||
# Compare usage statistics
|
||||
usage1 = mounts1.get("usage", {})
|
||||
usage2 = mounts2.get("usage", {})
|
||||
|
||||
for mp in set(usage1.keys()) | set(usage2.keys()):
|
||||
if mp in usage1 and mp in usage2:
|
||||
u1, u2 = usage1[mp], usage2[mp]
|
||||
if u1 != u2:
|
||||
differences["usage_changes"].append({
|
||||
"mountpoint": mp,
|
||||
"before": u1,
|
||||
"after": u2
|
||||
})
|
||||
|
||||
return differences
|
||||
|
||||
def compare_services(self, services1: Dict[str, Any], services2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Compare services data between snapshots."""
|
||||
differences = {
|
||||
"added_services": [],
|
||||
"removed_services": [],
|
||||
"status_changes": [],
|
||||
"configuration_changes": []
|
||||
}
|
||||
|
||||
# Compare service lists
|
||||
services_list1 = services1.get("services", [])
|
||||
services_list2 = services2.get("services", [])
|
||||
|
||||
# Create service maps
|
||||
services_map1 = {s["name"]: s for s in services_list1}
|
||||
services_map2 = {s["name"]: s for s in services_list2}
|
||||
|
||||
# Find added and removed services
|
||||
added = set(services_map2.keys()) - set(services_map1.keys())
|
||||
removed = set(services_map1.keys()) - set(services_map2.keys())
|
||||
|
||||
differences["added_services"] = [{"name": name, **services_map2[name]} for name in added]
|
||||
differences["removed_services"] = [{"name": name, **services_map1[name]} for name in removed]
|
||||
|
||||
# Find status changes
|
||||
common = set(services_map1.keys()) & set(services_map2.keys())
|
||||
for name in common:
|
||||
s1, s2 = services_map1[name], services_map2[name]
|
||||
if s1.get("active_state") != s2.get("active_state") or s1.get("sub_state") != s2.get("sub_state"):
|
||||
differences["status_changes"].append({
|
||||
"name": name,
|
||||
"before": {"active_state": s1.get("active_state"), "sub_state": s1.get("sub_state")},
|
||||
"after": {"active_state": s2.get("active_state"), "sub_state": s2.get("sub_state")}
|
||||
})
|
||||
|
||||
return differences
|
||||
|
||||
def compare_disk_usage(self, usage1: Dict[str, Any], usage2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Compare disk usage data between snapshots."""
|
||||
differences = {
|
||||
"filesystem_changes": [],
|
||||
"directory_size_changes": [],
|
||||
"significant_usage_changes": []
|
||||
}
|
||||
|
||||
# Compare filesystem usage
|
||||
fs1 = usage1.get("filesystem_usage", [])
|
||||
fs2 = usage2.get("filesystem_usage", [])
|
||||
|
||||
# Create filesystem maps by mountpoint
|
||||
fs_map1 = {fs["mountpoint"]: fs for fs in fs1}
|
||||
fs_map2 = {fs["mountpoint"]: fs for fs in fs2}
|
||||
|
||||
common_fs = set(fs_map1.keys()) & set(fs_map2.keys())
|
||||
for mp in common_fs:
|
||||
f1, f2 = fs_map1[mp], fs_map2[mp]
|
||||
if f1 != f2:
|
||||
differences["filesystem_changes"].append({
|
||||
"mountpoint": mp,
|
||||
"before": f1,
|
||||
"after": f2
|
||||
})
|
||||
|
||||
# Check for significant usage changes
|
||||
try:
|
||||
use1 = int(f1.get("use_percent", "0").rstrip("%"))
|
||||
use2 = int(f2.get("use_percent", "0").rstrip("%"))
|
||||
if abs(use2 - use1) > 10: # 10% change threshold
|
||||
differences["significant_usage_changes"].append({
|
||||
"mountpoint": mp,
|
||||
"change_percent": use2 - use1,
|
||||
"before": f1,
|
||||
"after": f2
|
||||
})
|
||||
except (ValueError, KeyError):
|
||||
pass
|
||||
|
||||
return differences
|
||||
|
||||
def generate_summary(self, differences: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Generate a summary of all differences."""
|
||||
summary = {
|
||||
"total_systems": 0,
|
||||
"systems_with_changes": 0,
|
||||
"total_changes": 0,
|
||||
"changes_by_type": {},
|
||||
"most_affected_systems": []
|
||||
}
|
||||
|
||||
system_change_counts = {}
|
||||
|
||||
for data_type, systems in differences.items():
|
||||
summary["changes_by_type"][data_type] = 0
|
||||
|
||||
for system, system_diffs in systems.items():
|
||||
if system not in system_change_counts:
|
||||
system_change_counts[system] = 0
|
||||
|
||||
# Count changes for this system and data type
|
||||
change_count = self.count_changes(system_diffs)
|
||||
system_change_counts[system] += change_count
|
||||
summary["changes_by_type"][data_type] += change_count
|
||||
summary["total_changes"] += change_count
|
||||
|
||||
summary["total_systems"] = len(system_change_counts)
|
||||
|
||||
# Count systems with changes
|
||||
summary["systems_with_changes"] = len([s for s in system_change_counts.values() if s > 0])
|
||||
|
||||
# Find most affected systems
|
||||
sorted_systems = sorted(system_change_counts.items(), key=lambda x: x[1], reverse=True)
|
||||
summary["most_affected_systems"] = sorted_systems[:5]
|
||||
|
||||
return summary
|
||||
|
||||
def count_changes(self, system_diffs: Dict[str, Any]) -> int:
|
||||
"""Count the number of changes in system differences."""
|
||||
count = 0
|
||||
|
||||
for key, value in system_diffs.items():
|
||||
if isinstance(value, list):
|
||||
count += len(value)
|
||||
elif isinstance(value, dict) and key not in ["status"]:
|
||||
# Count nested changes
|
||||
count += sum(1 for v in value.values() if isinstance(v, list) and v)
|
||||
|
||||
return count
|
||||
|
||||
def assess_risks(self, differences: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Assess risk levels for the changes."""
|
||||
risk_assessment = {
|
||||
"overall_risk": "low",
|
||||
"risk_factors": [],
|
||||
"critical_changes": [],
|
||||
"recommendations": []
|
||||
}
|
||||
|
||||
max_risk_level = 1
|
||||
|
||||
# Analyze each type of change
|
||||
for data_type, systems in differences.items():
|
||||
for system, system_diffs in systems.items():
|
||||
risk_factors = self.analyze_system_risks(system_diffs, data_type)
|
||||
risk_assessment["risk_factors"].extend(risk_factors)
|
||||
|
||||
for factor in risk_factors:
|
||||
if factor["level"] > max_risk_level:
|
||||
max_risk_level = factor["level"]
|
||||
|
||||
if factor["level"] >= 4: # Critical
|
||||
risk_assessment["critical_changes"].append({
|
||||
"system": system,
|
||||
"data_type": data_type,
|
||||
"factor": factor
|
||||
})
|
||||
|
||||
# Set overall risk
|
||||
risk_levels = {1: "low", 2: "medium", 3: "high", 4: "critical"}
|
||||
risk_assessment["overall_risk"] = risk_levels.get(max_risk_level, "unknown")
|
||||
|
||||
# Generate recommendations
|
||||
risk_assessment["recommendations"] = self.generate_recommendations(risk_assessment)
|
||||
|
||||
return risk_assessment
|
||||
|
||||
def analyze_system_risks(self, system_diffs: Dict[str, Any], data_type: str) -> List[Dict[str, Any]]:
|
||||
"""Analyze risks for a specific system's changes."""
|
||||
risk_factors = []
|
||||
|
||||
if data_type == "mounts":
|
||||
# Check for removed critical mounts
|
||||
for mount in system_diffs.get("removed_mounts", []):
|
||||
if mount["mountpoint"] in ["/", "/boot", "/usr", "/var"]:
|
||||
risk_factors.append({
|
||||
"type": "critical_mount_removed",
|
||||
"description": f"Critical mount point removed: {mount['mountpoint']}",
|
||||
"level": 4
|
||||
})
|
||||
|
||||
# Check for significant usage changes
|
||||
for change in system_diffs.get("usage_changes", []):
|
||||
try:
|
||||
before_pct = int(change["before"].get("use_percent", "0").rstrip("%"))
|
||||
after_pct = int(change["after"].get("use_percent", "0").rstrip("%"))
|
||||
if after_pct > 95:
|
||||
risk_factors.append({
|
||||
"type": "filesystem_full",
|
||||
"description": f"Filesystem usage critical: {change['mountpoint']} at {after_pct}%",
|
||||
"level": 3
|
||||
})
|
||||
except (ValueError, KeyError):
|
||||
pass
|
||||
|
||||
elif data_type == "services":
|
||||
# Check for critical service changes
|
||||
critical_services = ["sshd", "systemd", "networking", "dbus"]
|
||||
for service in system_diffs.get("removed_services", []):
|
||||
if service["name"] in critical_services:
|
||||
risk_factors.append({
|
||||
"type": "critical_service_removed",
|
||||
"description": f"Critical service removed: {service['name']}",
|
||||
"level": 4
|
||||
})
|
||||
|
||||
for change in system_diffs.get("status_changes", []):
|
||||
if change["after"]["active_state"] == "failed":
|
||||
risk_factors.append({
|
||||
"type": "service_failure",
|
||||
"description": f"Service failed: {change['name']}",
|
||||
"level": 3
|
||||
})
|
||||
|
||||
elif data_type == "disk_usage":
|
||||
for change in system_diffs.get("significant_usage_changes", []):
|
||||
if change["change_percent"] > 20:
|
||||
risk_factors.append({
|
||||
"type": "disk_usage_spike",
|
||||
"description": f"Significant disk usage change: {change['mountpoint']} ({change['change_percent']}%)",
|
||||
"level": 2
|
||||
})
|
||||
|
||||
return risk_factors
|
||||
|
||||
def generate_recommendations(self, risk_assessment: Dict[str, Any]) -> List[str]:
|
||||
"""Generate recommendations based on risk assessment."""
|
||||
recommendations = []
|
||||
|
||||
if risk_assessment["overall_risk"] in ["high", "critical"]:
|
||||
recommendations.append("Immediate review required - critical changes detected")
|
||||
recommendations.append("Consider rolling back migration if critical services are affected")
|
||||
|
||||
if any(f["type"] == "critical_mount_removed" for f in risk_assessment["risk_factors"]):
|
||||
recommendations.append("Verify system boot capability after mount changes")
|
||||
|
||||
if any(f["type"] == "critical_service_removed" for f in risk_assessment["risk_factors"]):
|
||||
recommendations.append("Ensure critical services are restored before production cutover")
|
||||
|
||||
if any(f["type"] == "filesystem_full" for f in risk_assessment["risk_factors"]):
|
||||
recommendations.append("Monitor disk space closely - cleanup may be required")
|
||||
|
||||
if not recommendations:
|
||||
recommendations.append("Changes appear safe - proceed with standard validation procedures")
|
||||
|
||||
return recommendations
|
||||
|
||||
def validate_changes(self, differences: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Validate that changes meet requirements."""
|
||||
validation_results = {
|
||||
"passed": True,
|
||||
"checks": [],
|
||||
"failed_checks": []
|
||||
}
|
||||
|
||||
# Define validation checks
|
||||
checks = [
|
||||
self.check_critical_services_running,
|
||||
self.check_filesystem_integrity,
|
||||
self.check_no_critical_mounts_removed
|
||||
]
|
||||
|
||||
for check_func in checks:
|
||||
check_result = check_func(differences)
|
||||
validation_results["checks"].append(check_result)
|
||||
|
||||
if not check_result["passed"]:
|
||||
validation_results["passed"] = False
|
||||
validation_results["failed_checks"].append(check_result)
|
||||
|
||||
return validation_results
|
||||
|
||||
def check_critical_services_running(self, differences: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Check that critical services are still running."""
|
||||
check = {
|
||||
"name": "critical_services_running",
|
||||
"description": "Verify critical services remain operational",
|
||||
"passed": True,
|
||||
"details": []
|
||||
}
|
||||
|
||||
critical_services = ["sshd", "systemd"]
|
||||
|
||||
for data_type, systems in differences.items():
|
||||
if data_type == "services":
|
||||
for system, system_diffs in systems.items():
|
||||
for change in system_diffs.get("status_changes", []):
|
||||
if change["name"] in critical_services:
|
||||
if change["after"]["active_state"] == "failed":
|
||||
check["passed"] = False
|
||||
check["details"].append(f"Critical service {change['name']} failed on {system}")
|
||||
|
||||
return check
|
||||
|
||||
def check_filesystem_integrity(self, differences: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Check filesystem integrity after changes."""
|
||||
check = {
|
||||
"name": "filesystem_integrity",
|
||||
"description": "Verify filesystem integrity maintained",
|
||||
"passed": True,
|
||||
"details": []
|
||||
}
|
||||
|
||||
for data_type, systems in differences.items():
|
||||
if data_type == "disk_usage":
|
||||
for system, system_diffs in systems.items():
|
||||
for change in system_diffs.get("significant_usage_changes", []):
|
||||
if change["change_percent"] > 50: # Arbitrary threshold
|
||||
check["passed"] = False
|
||||
check["details"].append(f"Extreme usage change on {system}:{change['mountpoint']}")
|
||||
|
||||
return check
|
||||
|
||||
def check_no_critical_mounts_removed(self, differences: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Check that no critical mount points were removed."""
|
||||
check = {
|
||||
"name": "no_critical_mounts_removed",
|
||||
"description": "Verify critical mount points remain",
|
||||
"passed": True,
|
||||
"details": []
|
||||
}
|
||||
|
||||
critical_mounts = ["/", "/boot", "/usr", "/var"]
|
||||
|
||||
for data_type, systems in differences.items():
|
||||
if data_type == "mounts":
|
||||
for system, system_diffs in systems.items():
|
||||
for mount in system_diffs.get("removed_mounts", []):
|
||||
if mount["mountpoint"] in critical_mounts:
|
||||
check["passed"] = False
|
||||
check["details"].append(f"Critical mount {mount['mountpoint']} removed from {system}")
|
||||
|
||||
return check
|
||||
|
||||
def compare_snapshots(snapshot1: Dict[str, Any], snapshot2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Main comparison function."""
|
||||
comparator = SnapshotComparator()
|
||||
return comparator.compare_snapshots(snapshot1, snapshot2)
|
||||
@@ -0,0 +1,10 @@
|
||||
.PHONY: run test demo
|
||||
|
||||
run:
|
||||
docker compose up -d
|
||||
|
||||
test:
|
||||
docker compose config
|
||||
|
||||
demo:
|
||||
./scenarios/incident_simulation.sh comprehensive
|
||||
@@ -0,0 +1,67 @@
|
||||
# Observability Stack
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Operations teams need correlated logs, dashboards, and alert examples that make incidents observable before they become customer-facing outages. A stack that only starts containers is not enough; it also needs meaningful sample data and incident exercises.
|
||||
|
||||
## Solution Overview
|
||||
|
||||
This project defines a local observability environment with Elasticsearch, Logstash, Kibana, Grafana, Filebeat, alert rules, sample logs, and an incident simulation script. It is built to demonstrate practical monitoring workflows rather than a production-sized cluster.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Application/System Logs -> Filebeat -> Logstash -> Elasticsearch -> Kibana
|
||||
|
|
||||
v
|
||||
Grafana
|
||||
|
||||
Incident Scenario -> Sample Logs -> Alert Rules -> Operator Review
|
||||
```
|
||||
|
||||
Core components:
|
||||
|
||||
- `docker-compose.yml` defines the observability services.
|
||||
- `alerting/alert_rules.yml` records alert intent and severity.
|
||||
- `logs/` contains representative operational logs.
|
||||
- `scenarios/incident_simulation.sh` emits incident activity.
|
||||
- `examples/` contains sample alert and log outputs.
|
||||
|
||||
## How to Run
|
||||
|
||||
```bash
|
||||
cd observability-stack
|
||||
|
||||
# Validate the compose model.
|
||||
make test
|
||||
|
||||
# Start the stack.
|
||||
make run
|
||||
|
||||
# Run the incident simulation.
|
||||
make demo
|
||||
|
||||
# Stop the stack.
|
||||
docker compose down
|
||||
```
|
||||
|
||||
When running locally:
|
||||
|
||||
- Kibana: `http://localhost:5601`
|
||||
- Grafana: `http://localhost:3000`
|
||||
- Elasticsearch: `http://localhost:9200`
|
||||
|
||||
## Example Output
|
||||
|
||||
```text
|
||||
[2026-04-29 04:18:23] WARN Database connection pool nearing capacity
|
||||
[2026-04-29 04:18:28] ERROR Database connection pool exhausted
|
||||
[2026-04-29 04:18:33] ERROR Database query timeout occurred
|
||||
[2026-04-29 04:18:44] INFO Database connections restored
|
||||
```
|
||||
|
||||
Additional examples are available in [examples/alert-output.txt](examples/alert-output.txt) and [examples/sample-log.txt](examples/sample-log.txt).
|
||||
|
||||
## Real-World Use Case
|
||||
|
||||
A platform team can use this project to explain how logs move through an ingestion pipeline, how alert rules map to operational symptoms, and how incident exercises create evidence for on-call readiness reviews.
|
||||
@@ -0,0 +1,326 @@
|
||||
# Enterprise Observability Alert Rules
|
||||
# Alert definitions for automated incident detection and notification
|
||||
|
||||
alert_rules:
|
||||
# System Resource Alerts
|
||||
- name: "High CPU Usage"
|
||||
description: "CPU utilization exceeds threshold"
|
||||
condition: "cpu_usage_percent > 90"
|
||||
duration: "5m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- system
|
||||
- performance
|
||||
channels:
|
||||
- email
|
||||
- slack
|
||||
labels:
|
||||
team: "platform"
|
||||
component: "system"
|
||||
|
||||
- name: "High Memory Usage"
|
||||
description: "Memory utilization exceeds threshold"
|
||||
condition: "memory_usage_percent > 85"
|
||||
duration: "3m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- system
|
||||
- memory
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "platform"
|
||||
component: "system"
|
||||
|
||||
- name: "Disk Space Critical"
|
||||
description: "Disk usage exceeds critical threshold"
|
||||
condition: "disk_usage_percent > 95"
|
||||
duration: "2m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- storage
|
||||
- disk
|
||||
channels:
|
||||
- email
|
||||
- pagerduty
|
||||
labels:
|
||||
team: "platform"
|
||||
component: "storage"
|
||||
|
||||
- name: "Disk Space Warning"
|
||||
description: "Disk usage exceeds warning threshold"
|
||||
condition: "disk_usage_percent > 85"
|
||||
duration: "10m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- storage
|
||||
- disk
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "platform"
|
||||
component: "storage"
|
||||
|
||||
# Service Availability Alerts
|
||||
- name: "Service Down"
|
||||
description: "Critical service is not responding"
|
||||
condition: "service_status == 'down' OR http_status_code >= 500"
|
||||
duration: "2m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- service
|
||||
- availability
|
||||
channels:
|
||||
- email
|
||||
- slack
|
||||
- pagerduty
|
||||
labels:
|
||||
team: "application"
|
||||
component: "service"
|
||||
|
||||
- name: "Database Connection Failed"
|
||||
description: "Database connection pool exhausted or unresponsive"
|
||||
condition: "db_connections_active == 0 OR db_response_time > 5000"
|
||||
duration: "1m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- database
|
||||
- connectivity
|
||||
channels:
|
||||
- email
|
||||
- pagerduty
|
||||
labels:
|
||||
team: "database"
|
||||
component: "postgresql"
|
||||
|
||||
- name: "Cache Unavailable"
|
||||
description: "Cache service is down or unresponsive"
|
||||
condition: "cache_hit_ratio < 0.1 OR cache_response_time > 1000"
|
||||
duration: "3m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- cache
|
||||
- performance
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "infrastructure"
|
||||
component: "redis"
|
||||
|
||||
# Application Performance Alerts
|
||||
- name: "High Error Rate"
|
||||
description: "Application error rate exceeds threshold"
|
||||
condition: "error_rate_percent > 5"
|
||||
duration: "5m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- application
|
||||
- errors
|
||||
channels:
|
||||
- email
|
||||
- slack
|
||||
labels:
|
||||
team: "application"
|
||||
component: "api"
|
||||
|
||||
- name: "Slow Response Time"
|
||||
description: "API response time exceeds SLA"
|
||||
condition: "response_time_p95 > 2000"
|
||||
duration: "5m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- application
|
||||
- performance
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "application"
|
||||
component: "api"
|
||||
|
||||
- name: "High Request Queue"
|
||||
description: "Request queue depth is too high"
|
||||
condition: "queue_depth > 100"
|
||||
duration: "3m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- application
|
||||
- queue
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "application"
|
||||
component: "queue"
|
||||
|
||||
# Infrastructure Alerts
|
||||
- name: "Network Latency High"
|
||||
description: "Network round-trip time exceeds threshold"
|
||||
condition: "network_rtt > 100"
|
||||
duration: "5m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- network
|
||||
- latency
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "network"
|
||||
component: "infrastructure"
|
||||
|
||||
- name: "Load Balancer Unhealthy"
|
||||
description: "Load balancer backend servers are unhealthy"
|
||||
condition: "lb_unhealthy_backends > 0"
|
||||
duration: "2m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- loadbalancer
|
||||
- availability
|
||||
channels:
|
||||
- email
|
||||
- pagerduty
|
||||
labels:
|
||||
team: "infrastructure"
|
||||
component: "loadbalancer"
|
||||
|
||||
# Security Alerts
|
||||
- name: "Failed Login Attempts"
|
||||
description: "Multiple failed authentication attempts detected"
|
||||
condition: "failed_login_attempts > 5"
|
||||
duration: "5m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- security
|
||||
- authentication
|
||||
channels:
|
||||
- email
|
||||
- slack
|
||||
labels:
|
||||
team: "security"
|
||||
component: "authentication"
|
||||
|
||||
- name: "Suspicious Network Traffic"
|
||||
description: "Unusual network traffic patterns detected"
|
||||
condition: "network_bytes_unusual > 1000000"
|
||||
duration: "10m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- security
|
||||
- network
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "security"
|
||||
component: "network"
|
||||
|
||||
# Log-based Alerts
|
||||
- name: "Application Errors"
|
||||
description: "High volume of application error logs"
|
||||
condition: "log_errors_per_minute > 10"
|
||||
duration: "2m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- logs
|
||||
- errors
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "application"
|
||||
component: "logs"
|
||||
|
||||
- name: "Out of Memory Errors"
|
||||
description: "Out of memory errors detected in logs"
|
||||
condition: "log_oom_errors > 0"
|
||||
duration: "1m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- memory
|
||||
- errors
|
||||
channels:
|
||||
- email
|
||||
- pagerduty
|
||||
labels:
|
||||
team: "application"
|
||||
component: "memory"
|
||||
|
||||
# Business Logic Alerts
|
||||
- name: "Low Business Transactions"
|
||||
description: "Business transaction volume below expected threshold"
|
||||
condition: "business_transactions_per_hour < 100"
|
||||
duration: "15m"
|
||||
severity: "warning"
|
||||
tags:
|
||||
- business
|
||||
- transactions
|
||||
channels:
|
||||
- email
|
||||
labels:
|
||||
team: "business"
|
||||
component: "transactions"
|
||||
|
||||
- name: "Payment Failures"
|
||||
description: "Payment processing failure rate is high"
|
||||
condition: "payment_failure_rate > 0.05"
|
||||
duration: "5m"
|
||||
severity: "critical"
|
||||
tags:
|
||||
- payments
|
||||
- business
|
||||
channels:
|
||||
- email
|
||||
- pagerduty
|
||||
labels:
|
||||
team: "payments"
|
||||
component: "processing"
|
||||
|
||||
# Alert Channels Configuration
|
||||
alert_channels:
|
||||
email:
|
||||
type: "email"
|
||||
recipients:
|
||||
- "platform-team@company.com"
|
||||
- "oncall@company.com"
|
||||
subject_template: "[{{severity}}] {{name}} - {{description}}"
|
||||
|
||||
slack:
|
||||
type: "slack"
|
||||
webhook_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
|
||||
channel: "#alerts"
|
||||
username: "Observability Bot"
|
||||
icon_emoji: ":warning:"
|
||||
|
||||
pagerduty:
|
||||
type: "pagerduty"
|
||||
integration_key: "your-pagerduty-integration-key"
|
||||
severity_mapping:
|
||||
critical: "critical"
|
||||
warning: "warning"
|
||||
info: "info"
|
||||
|
||||
# Alert Silencing Rules
|
||||
silence_rules:
|
||||
- name: "Maintenance Window"
|
||||
condition: "maintenance_window == true"
|
||||
duration: "4h"
|
||||
comment: "Silenced during scheduled maintenance"
|
||||
|
||||
- name: "Known Issue"
|
||||
condition: "known_issue_id == 'TICKET-123'"
|
||||
duration: "24h"
|
||||
comment: "Silenced for known issue resolution"
|
||||
|
||||
# Escalation Policies
|
||||
escalation_policies:
|
||||
- name: "Default Escalation"
|
||||
steps:
|
||||
- delay: "5m"
|
||||
channels: ["email"]
|
||||
- delay: "15m"
|
||||
channels: ["slack"]
|
||||
- delay: "30m"
|
||||
channels: ["pagerduty"]
|
||||
|
||||
- name: "Critical Escalation"
|
||||
steps:
|
||||
- delay: "0m"
|
||||
channels: ["email", "slack", "pagerduty"]
|
||||
- delay: "10m"
|
||||
channels: ["pagerduty"] # Escalation
|
||||
@@ -0,0 +1,120 @@
|
||||
services:
|
||||
elasticsearch:
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
|
||||
container_name: observability-elasticsearch
|
||||
environment:
|
||||
- discovery.type=single-node
|
||||
- xpack.security.enabled=true
|
||||
- ELASTIC_PASSWORD=elastic
|
||||
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
|
||||
volumes:
|
||||
- elasticsearch_data:/usr/share/elasticsearch/data
|
||||
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
|
||||
ports:
|
||||
- "9200:9200"
|
||||
- "9300:9300"
|
||||
networks:
|
||||
- observability
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
|
||||
logstash:
|
||||
image: docker.elastic.co/logstash/logstash:8.11.0
|
||||
container_name: observability-logstash
|
||||
environment:
|
||||
- "LS_JAVA_OPTS=-Xms512m -Xmx512m"
|
||||
volumes:
|
||||
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
|
||||
- ./logstash/pipeline:/usr/share/logstash/pipeline
|
||||
- ./logs:/usr/share/logstash/logs
|
||||
ports:
|
||||
- "5044:5044"
|
||||
- "8080:8080"
|
||||
networks:
|
||||
- observability
|
||||
depends_on:
|
||||
elasticsearch:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
kibana:
|
||||
image: docker.elastic.co/kibana/kibana:8.11.0
|
||||
container_name: observability-kibana
|
||||
environment:
|
||||
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
|
||||
- ELASTICSEARCH_USERNAME=elastic
|
||||
- ELASTICSEARCH_PASSWORD=elastic
|
||||
volumes:
|
||||
- ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
|
||||
ports:
|
||||
- "5601:5601"
|
||||
networks:
|
||||
- observability
|
||||
depends_on:
|
||||
elasticsearch:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:10.2.0
|
||||
container_name: observability-grafana
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_USER=admin
|
||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
- GF_USERS_ALLOW_SIGN_UP=false
|
||||
volumes:
|
||||
- grafana_data:/var/lib/grafana
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning
|
||||
- ./grafana/dashboards:/var/lib/grafana/dashboards
|
||||
ports:
|
||||
- "3000:3000"
|
||||
networks:
|
||||
- observability
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:3000/api/health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
filebeat:
|
||||
image: docker.elastic.co/beats/filebeat:8.11.0
|
||||
container_name: observability-filebeat
|
||||
user: root
|
||||
volumes:
|
||||
- ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
|
||||
- ./logs:/var/log/sample
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
networks:
|
||||
- observability
|
||||
depends_on:
|
||||
- logstash
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
elasticsearch_data:
|
||||
driver: local
|
||||
grafana_data:
|
||||
driver: local
|
||||
|
||||
networks:
|
||||
observability:
|
||||
driver: bridge
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.25.0.0/16
|
||||
@@ -0,0 +1,30 @@
|
||||
# Observability Stack Architecture
|
||||
|
||||
## Components
|
||||
|
||||
- Filebeat: tails sample and container logs.
|
||||
- Logstash: receives and processes log events.
|
||||
- Elasticsearch: stores searchable observability data.
|
||||
- Kibana: supports log exploration and dashboards.
|
||||
- Grafana: provides operational dashboards.
|
||||
- Alert rules: document symptoms, thresholds, and severity.
|
||||
- Incident simulation: generates controlled failure signals.
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Log source -> Filebeat -> Logstash -> Elasticsearch -> Kibana
|
||||
|
|
||||
v
|
||||
Grafana
|
||||
```
|
||||
|
||||
Incident exercises follow this flow:
|
||||
|
||||
```
|
||||
Operator -> incident_simulation.sh -> logs/incident_simulation.log -> Filebeat -> Logstash -> alerts/dashboards
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
This is a local demonstration stack, not a production Elasticsearch deployment. A production version would add dedicated nodes, TLS, secret management, retention policies, index lifecycle management, and external alert delivery.
|
||||
@@ -0,0 +1,4 @@
|
||||
2026-04-29T04:19:00Z alert=database_connection_pool_exhausted severity=critical service=checkout-api host=app-web-02 value=100 threshold=95 status=firing
|
||||
2026-04-29T04:19:30Z alert=api_error_rate_high severity=warning service=checkout-api host=app-web-02 value=7.8 threshold=5.0 status=firing
|
||||
2026-04-29T04:22:00Z alert=database_connection_pool_exhausted severity=critical service=checkout-api host=app-web-02 value=71 threshold=95 status=resolved
|
||||
2026-04-29T04:23:15Z alert=api_error_rate_high severity=warning service=checkout-api host=app-web-02 value=1.2 threshold=5.0 status=resolved
|
||||
@@ -0,0 +1,5 @@
|
||||
2026-04-29T04:18:21Z INFO service=checkout-api host=app-web-02 request_id=8f4b2 path=/checkout status=200 latency_ms=142
|
||||
2026-04-29T04:18:28Z WARN service=checkout-api host=app-web-02 event=db_pool_pressure active=92 max=100
|
||||
2026-04-29T04:18:33Z ERROR service=checkout-api host=app-web-02 event=db_timeout query=CreateOrder timeout_ms=5000 customer_tier=enterprise
|
||||
2026-04-29T04:18:39Z ERROR service=checkout-api host=app-web-02 event=payment_retry_exhausted order_id=ord-104288 provider=stripe
|
||||
2026-04-29T04:18:44Z INFO service=checkout-api host=app-web-02 event=recovery db_pool_active=48
|
||||
@@ -0,0 +1,2 @@
|
||||
[2026-04-29 22:52:26] INFO Incident simulation script started
|
||||
[2026-04-29 22:52:26] INFO Scenario: help
|
||||
@@ -0,0 +1,98 @@
|
||||
# Sample Application Logs for Observability Stack Testing
|
||||
# Generated realistic log entries for demonstration and testing
|
||||
|
||||
[2024-01-15 08:30:15] INFO com.example.app.Application - Application started successfully on port 8080
|
||||
[2024-01-15 08:30:16] INFO com.example.database.ConnectionPool - Database connection pool initialized with 10 connections
|
||||
[2024-01-15 08:30:17] INFO com.example.cache.RedisCache - Redis cache connected successfully
|
||||
[2024-01-15 08:30:18] INFO com.example.messaging.RabbitMQClient - Message queue connection established
|
||||
|
||||
[2024-01-15 08:31:22] INFO com.example.api.UserController - User login attempt: user=john.doe, ip=192.168.1.100
|
||||
[2024-01-15 08:31:23] INFO com.example.auth.AuthService - Authentication successful for user john.doe
|
||||
[2024-01-15 08:31:24] INFO com.example.api.UserController - API request: GET /api/users/profile, status=200, duration=45ms
|
||||
|
||||
[2024-01-15 08:32:01] WARN com.example.cache.RedisCache - Cache miss for key: user_profile_12345
|
||||
[2024-01-15 08:32:02] INFO com.example.database.UserRepository - Database query executed: SELECT * FROM users WHERE id = ?, duration=120ms
|
||||
[2024-01-15 08:32:03] INFO com.example.cache.RedisCache - Cache populated for key: user_profile_12345
|
||||
|
||||
[2024-01-15 08:35:12] ERROR com.example.api.OrderController - Failed to process order: order_id=ORD-2024-001, error=Payment gateway timeout
|
||||
[2024-01-15 08:35:13] WARN com.example.messaging.RabbitMQClient - Message delivery failed, retrying in 5 seconds
|
||||
[2024-01-15 08:35:18] INFO com.example.messaging.RabbitMQClient - Message delivered successfully after retry
|
||||
|
||||
[2024-01-15 08:40:05] INFO com.example.monitoring.HealthCheck - Health check passed: database=OK, cache=OK, messaging=OK
|
||||
[2024-01-15 08:40:06] INFO com.example.metrics.MetricsCollector - Metrics collected: active_users=1250, requests_per_second=45.2
|
||||
|
||||
[2024-01-15 08:45:30] ERROR com.example.external.PaymentGateway - Payment gateway connection failed: Connection refused
|
||||
[2024-01-15 08:45:31] ERROR com.example.api.PaymentController - Payment processing failed for transaction TXN-789012
|
||||
[2024-01-15 08:45:32] WARN com.example.circuitbreaker.PaymentCircuitBreaker - Circuit breaker opened for payment service
|
||||
|
||||
[2024-01-15 08:50:15] INFO com.example.batch.BatchProcessor - Batch job started: job_id=BATCH-001, records=10000
|
||||
[2024-01-15 08:50:45] INFO com.example.batch.BatchProcessor - Batch job progress: processed=2500/10000 (25%)
|
||||
[2024-01-15 08:51:15] INFO com.example.batch.BatchProcessor - Batch job progress: processed=5000/10000 (50%)
|
||||
[2024-01-15 08:51:45] INFO com.example.batch.BatchProcessor - Batch job progress: processed=7500/10000 (75%)
|
||||
[2024-01-15 08:52:10] INFO com.example.batch.BatchProcessor - Batch job completed: job_id=BATCH-001, duration=175s, success=true
|
||||
|
||||
[2024-01-15 09:00:00] INFO com.example.scheduled.CleanupJob - Scheduled cleanup job started
|
||||
[2024-01-15 09:00:05] INFO com.example.scheduled.CleanupJob - Cleaned up 150 expired sessions
|
||||
[2024-01-15 09:00:10] INFO com.example.scheduled.CleanupJob - Removed 25 temporary files
|
||||
[2024-01-15 09:00:15] INFO com.example.scheduled.CleanupJob - Cleanup job completed successfully
|
||||
|
||||
[2024-01-15 09:15:22] WARN com.example.database.ConnectionPool - Connection pool nearing capacity: active=8/10
|
||||
[2024-01-15 09:15:23] INFO com.example.database.ConnectionPool - Connection pool expanded to 15 connections
|
||||
|
||||
[2024-01-15 09:30:45] ERROR com.example.api.ProductController - Database query timeout: query=SELECT * FROM products WHERE category = 'electronics'
|
||||
[2024-01-15 09:30:46] WARN com.example.database.ConnectionPool - Connection pool exhausted, rejecting request
|
||||
[2024-01-15 09:30:47] ERROR com.example.api.ProductController - Service temporarily unavailable, status=503
|
||||
|
||||
[2024-01-15 09:45:12] INFO com.example.monitoring.AlertManager - Alert triggered: High CPU usage detected (85%)
|
||||
[2024-01-15 09:45:13] INFO com.example.autoscaling.ScaleManager - Auto-scaling initiated: increasing instances from 3 to 5
|
||||
|
||||
[2024-01-15 10:00:00] INFO com.example.backup.BackupService - Database backup started
|
||||
[2024-01-15 10:05:30] INFO com.example.backup.BackupService - Database backup completed: size=2.3GB, duration=330s
|
||||
|
||||
[2024-01-15 10:30:15] WARN com.example.security.SecurityFilter - Suspicious activity detected: multiple failed login attempts from IP 10.0.0.50
|
||||
[2024-01-15 10:30:16] INFO com.example.security.SecurityFilter - IP 10.0.0.50 blocked for 15 minutes
|
||||
|
||||
[2024-01-15 11:00:00] INFO com.example.reporting.ReportGenerator - Daily report generation started
|
||||
[2024-01-15 11:05:00] INFO com.example.reporting.ReportGenerator - Daily report completed: transactions=15420, revenue=$125,430.50
|
||||
|
||||
[2024-01-15 12:00:00] ERROR com.example.external.APIGateway - External API rate limit exceeded: retrying in 60 seconds
|
||||
[2024-01-15 12:01:00] INFO com.example.external.APIGateway - External API connection restored
|
||||
|
||||
[2024-01-15 13:15:30] CRITICAL com.example.system.SystemMonitor - Disk space critical: /var/log usage at 95%
|
||||
[2024-01-15 13:15:31] INFO com.example.maintenance.LogRotation - Emergency log rotation initiated
|
||||
[2024-01-15 13:15:35] INFO com.example.maintenance.LogRotation - Log rotation completed: freed 2.1GB space
|
||||
|
||||
[2024-01-15 14:00:00] INFO com.example.metrics.PerformanceMonitor - Performance baseline established: avg_response_time=245ms, throughput=1200 req/sec
|
||||
|
||||
[2024-01-15 15:30:45] WARN com.example.cache.RedisCache - Cache cluster node down: redis-node-03
|
||||
[2024-01-15 15:30:46] INFO com.example.cache.RedisCache - Failover initiated to redis-node-04
|
||||
|
||||
[2024-01-15 16:45:12] ERROR com.example.messaging.MessageProcessor - Message processing failed: invalid message format
|
||||
[2024-01-15 16:45:13] INFO com.example.messaging.DeadLetterQueue - Message moved to dead letter queue
|
||||
|
||||
[2024-01-15 17:00:00] INFO com.example.backup.BackupService - Full system backup started
|
||||
[2024-01-15 17:30:00] INFO com.example.backup.BackupService - Full system backup completed: size=45.2GB, duration=1800s
|
||||
|
||||
[2024-01-15 18:00:00] INFO com.example.monitoring.HealthCheck - Evening health check: all systems operational
|
||||
[2024-01-15 18:00:01] INFO com.example.metrics.MetricsCollector - End of day metrics: total_requests=125000, error_rate=0.02%, avg_response_time=198ms
|
||||
|
||||
# Additional realistic log patterns for testing
|
||||
|
||||
[2024-01-15 08:15:30] INFO nginx: 192.168.1.100 - - [15/Jan/2024:08:15:30 +0000] "GET /api/health HTTP/1.1" 200 123 "-" "curl/7.68.0"
|
||||
[2024-01-15 08:15:31] INFO nginx: 192.168.1.101 - - [15/Jan/2024:08:15:31 +0000] "POST /api/login HTTP/1.1" 200 456 "-" "Mozilla/5.0 ..."
|
||||
[2024-01-15 08:15:32] WARN nginx: 192.168.1.102 - - [15/Jan/2024:08:15:32 +0000] "GET /api/admin HTTP/1.1" 403 234 "-" "Mozilla/5.0 ..."
|
||||
|
||||
[2024-01-15 09:20:15] INFO systemd: Started Session 1234 of user john.doe
|
||||
[2024-01-15 09:20:16] INFO systemd: Started User Manager for UID 1000
|
||||
[2024-01-15 09:20:17] INFO systemd: Started Session 1235 of user jane.smith
|
||||
|
||||
[2024-01-15 10:45:30] WARN kernel: [ 1234.567890] CPU0: Core temperature above threshold, cpu clock throttled
|
||||
[2024-01-15 10:45:31] INFO kernel: [ 1234.678901] CPU0: Temperature back to normal
|
||||
|
||||
[2024-01-15 14:30:15] ERROR postgres: FATAL: password authentication failed for user "app_user"
|
||||
[2024-01-15 14:30:16] ERROR postgres: FATAL: password authentication failed for user "app_user"
|
||||
[2024-01-15 14:30:17] WARN postgres: too many connections for role "app_user"
|
||||
|
||||
[2024-01-15 16:15:45] INFO rabbitmq: accepting AMQP connection <0.1234.0> (192.168.1.100:5672 -> 192.168.1.50:5672)
|
||||
[2024-01-15 16:15:46] INFO rabbitmq: connection <0.1234.0> (192.168.1.100:5672 -> 192.168.1.50:5672): user 'app_user' authenticated
|
||||
[2024-01-15 16:15:47] WARN rabbitmq: connection <0.1234.0> (192.168.1.100:5672 -> 192.168.1.50:5672): missed heartbeats from client, timeout: 60s
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: Incident Simulation
|
||||
|
||||
## Description
|
||||
|
||||
Generate a controlled application and infrastructure incident so the logging pipeline, alert rules, and dashboards can be reviewed with realistic event timing.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
cd observability-stack
|
||||
docker compose config
|
||||
./scenarios/incident_simulation.sh comprehensive
|
||||
tail -n 40 logs/incident_simulation.log
|
||||
```
|
||||
|
||||
## Expected Result
|
||||
|
||||
- The compose file validates successfully.
|
||||
- The simulation writes a sequence of CPU, memory, service, database, and application error events.
|
||||
- Alert examples indicate firing and resolved states.
|
||||
- Operators can trace incident progression through logs and dashboard queries.
|
||||
+318
@@ -0,0 +1,318 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Enterprise Incident Simulation Script
|
||||
# Simulates various failure scenarios for testing observability stack
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$(dirname "$SCRIPT_DIR")")"
|
||||
LOG_FILE="$PROJECT_ROOT/observability-stack/logs/incident_simulation.log"
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Logging function
|
||||
log() {
|
||||
local level=$1
|
||||
local message=$2
|
||||
local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
echo "[$timestamp] $level $message" >> "$LOG_FILE"
|
||||
echo -e "${BLUE}[$timestamp]${NC} $level $message"
|
||||
}
|
||||
|
||||
# Function to simulate CPU spike
|
||||
simulate_cpu_spike() {
|
||||
local duration=${1:-60}
|
||||
log "INFO" "Starting CPU spike simulation for ${duration} seconds"
|
||||
|
||||
# Launch CPU-intensive processes
|
||||
for i in {1..4}; do
|
||||
(
|
||||
end_time=$((SECONDS + duration))
|
||||
while [ $SECONDS -lt $end_time ]; do
|
||||
# CPU-intensive calculation
|
||||
result=0
|
||||
for j in {1..100000}; do
|
||||
result=$((result + j))
|
||||
done
|
||||
done
|
||||
) &
|
||||
PIDS[$i]=$!
|
||||
done
|
||||
|
||||
# Wait for simulation to complete
|
||||
for pid in "${PIDS[@]}"; do
|
||||
wait $pid 2>/dev/null || true
|
||||
done
|
||||
|
||||
log "INFO" "CPU spike simulation completed"
|
||||
}
|
||||
|
||||
# Function to simulate memory leak
|
||||
simulate_memory_leak() {
|
||||
local duration=${1:-30}
|
||||
log "INFO" "Starting memory leak simulation for ${duration} seconds"
|
||||
|
||||
# Create a process that gradually consumes memory
|
||||
(
|
||||
data=""
|
||||
end_time=$((SECONDS + duration))
|
||||
while [ $SECONDS -lt $end_time ]; do
|
||||
# Gradually consume memory
|
||||
data="${data}X"
|
||||
sleep 0.1
|
||||
done
|
||||
) &
|
||||
MEM_PID=$!
|
||||
|
||||
wait $MEM_PID 2>/dev/null || true
|
||||
log "INFO" "Memory leak simulation completed"
|
||||
}
|
||||
|
||||
# Function to simulate disk space exhaustion
|
||||
simulate_disk_full() {
|
||||
local target_dir=${1:-"/tmp"}
|
||||
local duration=${2:-30}
|
||||
log "INFO" "Starting disk space exhaustion simulation in ${target_dir} for ${duration} seconds"
|
||||
|
||||
# Create large files to fill disk space
|
||||
(
|
||||
end_time=$((SECONDS + duration))
|
||||
while [ $SECONDS -lt $end_time ]; do
|
||||
# Create 100MB file
|
||||
dd if=/dev/zero of="${target_dir}/incident_test_file_$(date +%s).tmp" bs=1M count=100 2>/dev/null || true
|
||||
sleep 2
|
||||
done
|
||||
) &
|
||||
DISK_PID=$!
|
||||
|
||||
wait $DISK_PID 2>/dev/null || true
|
||||
|
||||
# Cleanup test files
|
||||
rm -f "${target_dir}"/incident_test_file_*.tmp 2>/dev/null || true
|
||||
log "INFO" "Disk space exhaustion simulation completed and cleaned up"
|
||||
}
|
||||
|
||||
# Function to simulate network issues
|
||||
simulate_network_issues() {
|
||||
local interface=${1:-"lo"}
|
||||
local duration=${2:-20}
|
||||
log "INFO" "Starting network issues simulation on ${interface} for ${duration} seconds"
|
||||
|
||||
# Add network delay and packet loss
|
||||
sudo tc qdisc add dev $interface root netem delay 100ms 50ms loss 10% 2>/dev/null || true
|
||||
|
||||
sleep $duration
|
||||
|
||||
# Remove network simulation
|
||||
sudo tc qdisc del dev $interface root 2>/dev/null || true
|
||||
log "INFO" "Network issues simulation completed"
|
||||
}
|
||||
|
||||
# Function to simulate service crashes
|
||||
simulate_service_crash() {
|
||||
local service_name=${1:-"test-service"}
|
||||
log "INFO" "Starting service crash simulation for ${service_name}"
|
||||
|
||||
# Simulate service going down
|
||||
log "ERROR" "Service ${service_name} crashed unexpectedly"
|
||||
sleep 5
|
||||
log "INFO" "Service ${service_name} restarted automatically"
|
||||
|
||||
# Simulate multiple crashes
|
||||
for i in {1..3}; do
|
||||
sleep 2
|
||||
log "ERROR" "Service ${service_name} crashed again (attempt $i)"
|
||||
sleep 1
|
||||
log "INFO" "Service ${service_name} recovered after crash $i"
|
||||
done
|
||||
}
|
||||
|
||||
# Function to simulate database issues
|
||||
simulate_database_issues() {
|
||||
local duration=${1:-25}
|
||||
log "INFO" "Starting database issues simulation for ${duration} seconds"
|
||||
|
||||
# Simulate connection pool exhaustion
|
||||
log "WARN" "Database connection pool nearing capacity"
|
||||
sleep 5
|
||||
log "ERROR" "Database connection pool exhausted"
|
||||
sleep 5
|
||||
log "ERROR" "Database query timeout occurred"
|
||||
sleep 5
|
||||
log "WARN" "Database connections recovering"
|
||||
sleep 5
|
||||
log "INFO" "Database connections restored"
|
||||
|
||||
log "INFO" "Database issues simulation completed"
|
||||
}
|
||||
|
||||
# Function to simulate application errors
|
||||
simulate_application_errors() {
|
||||
local error_count=${1:-10}
|
||||
log "INFO" "Starting application error simulation (${error_count} errors)"
|
||||
|
||||
for i in {1..error_count}; do
|
||||
case $((RANDOM % 4)) in
|
||||
0)
|
||||
log "ERROR" "NullPointerException in UserService.getUser($i)"
|
||||
;;
|
||||
1)
|
||||
log "ERROR" "TimeoutException: Database query timed out for user ID: $i"
|
||||
;;
|
||||
2)
|
||||
log "ERROR" "ValidationException: Invalid input data for request $i"
|
||||
;;
|
||||
3)
|
||||
log "ERROR" "IOException: Failed to write to log file"
|
||||
;;
|
||||
esac
|
||||
sleep $((RANDOM % 3 + 1))
|
||||
done
|
||||
|
||||
log "INFO" "Application error simulation completed"
|
||||
}
|
||||
|
||||
# Function to run comprehensive incident scenario
|
||||
run_comprehensive_scenario() {
|
||||
log "INFO" "Starting comprehensive incident scenario simulation"
|
||||
|
||||
# Phase 1: Initial system stress
|
||||
log "INFO" "Phase 1: System stress simulation"
|
||||
simulate_cpu_spike 30 &
|
||||
CPU_PID=$!
|
||||
simulate_memory_leak 20 &
|
||||
MEM_PID=$!
|
||||
|
||||
sleep 10
|
||||
|
||||
# Phase 2: Service degradation
|
||||
log "INFO" "Phase 2: Service degradation simulation"
|
||||
simulate_service_crash "web-service" &
|
||||
SERVICE_PID=$!
|
||||
|
||||
sleep 5
|
||||
|
||||
# Phase 3: Database issues
|
||||
log "INFO" "Phase 3: Database issues simulation"
|
||||
simulate_database_issues 15 &
|
||||
DB_PID=$!
|
||||
|
||||
# Phase 4: Application errors
|
||||
log "INFO" "Phase 4: Application error burst"
|
||||
simulate_application_errors 15 &
|
||||
APP_PID=$!
|
||||
|
||||
# Phase 5: Infrastructure issues
|
||||
log "INFO" "Phase 5: Infrastructure issues simulation"
|
||||
simulate_disk_full "/tmp" 10 &
|
||||
DISK_PID=$!
|
||||
|
||||
# Wait for all simulations to complete
|
||||
wait $CPU_PID 2>/dev/null || true
|
||||
wait $MEM_PID 2>/dev/null || true
|
||||
wait $SERVICE_PID 2>/dev/null || true
|
||||
wait $DB_PID 2>/dev/null || true
|
||||
wait $APP_PID 2>/dev/null || true
|
||||
wait $DISK_PID 2>/dev/null || true
|
||||
|
||||
log "INFO" "Comprehensive incident scenario completed"
|
||||
}
|
||||
|
||||
# Function to show usage
|
||||
show_usage() {
|
||||
echo "Enterprise Incident Simulation Script"
|
||||
echo "Usage: $0 [SCENARIO] [OPTIONS]"
|
||||
echo ""
|
||||
echo "SCENARIOS:"
|
||||
echo " cpu [DURATION] - Simulate CPU spike (default: 60s)"
|
||||
echo " memory [DURATION] - Simulate memory leak (default: 30s)"
|
||||
echo " disk [DIR] [DURATION] - Simulate disk space exhaustion (default: /tmp, 30s)"
|
||||
echo " network [INTERFACE] [DURATION] - Simulate network issues (default: lo, 20s)"
|
||||
echo " service [NAME] - Simulate service crashes (default: test-service)"
|
||||
echo " database [DURATION] - Simulate database issues (default: 25s)"
|
||||
echo " app-errors [COUNT] - Simulate application errors (default: 10)"
|
||||
echo " comprehensive - Run full incident scenario"
|
||||
echo " all - Run all individual scenarios sequentially"
|
||||
echo ""
|
||||
echo "EXAMPLES:"
|
||||
echo " $0 cpu 120 - CPU spike for 2 minutes"
|
||||
echo " $0 disk /var/log 45 - Disk full simulation in /var/log for 45 seconds"
|
||||
echo " $0 comprehensive - Full incident simulation"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local scenario=${1:-"comprehensive"}
|
||||
|
||||
# Create log directory if it doesn't exist
|
||||
mkdir -p "$(dirname "$LOG_FILE")"
|
||||
|
||||
log "INFO" "Incident simulation script started"
|
||||
log "INFO" "Scenario: $scenario"
|
||||
|
||||
case $scenario in
|
||||
"cpu")
|
||||
simulate_cpu_spike "${2:-60}"
|
||||
;;
|
||||
"memory")
|
||||
simulate_memory_leak "${2:-30}"
|
||||
;;
|
||||
"disk")
|
||||
simulate_disk_full "${2:-/tmp}" "${3:-30}"
|
||||
;;
|
||||
"network")
|
||||
simulate_network_issues "${2:-lo}" "${3:-20}"
|
||||
;;
|
||||
"service")
|
||||
simulate_service_crash "${2:-test-service}"
|
||||
;;
|
||||
"database")
|
||||
simulate_database_issues "${2:-25}"
|
||||
;;
|
||||
"app-errors")
|
||||
simulate_application_errors "${2:-10}"
|
||||
;;
|
||||
"comprehensive")
|
||||
run_comprehensive_scenario
|
||||
;;
|
||||
"all")
|
||||
log "INFO" "Running all scenarios sequentially"
|
||||
simulate_cpu_spike 30
|
||||
sleep 5
|
||||
simulate_memory_leak 20
|
||||
sleep 5
|
||||
simulate_disk_full "/tmp" 15
|
||||
sleep 5
|
||||
simulate_service_crash "test-service"
|
||||
sleep 5
|
||||
simulate_database_issues 15
|
||||
sleep 5
|
||||
simulate_application_errors 8
|
||||
sleep 5
|
||||
simulate_network_issues "lo" 10
|
||||
;;
|
||||
"help"|"-h"|"--help")
|
||||
show_usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Error: Unknown scenario '$scenario'${NC}"
|
||||
echo ""
|
||||
show_usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
log "INFO" "Incident simulation script completed successfully"
|
||||
echo -e "${GREEN}Simulation completed. Check logs at: $LOG_FILE${NC}"
|
||||
}
|
||||
|
||||
# Run main function with all arguments
|
||||
main "$@"
|
||||
Reference in New Issue
Block a user