# Log Observability ELK/Grafana

## Problem

Operations teams need searchable logs and reviewable incident evidence in addition to simple OS checks. Zabbix is useful for host and service health signals; ELK/Grafana is better suited for log ingestion, error analysis, dashboards, and environment-level observability.

## CV Relevance

This project supports the monitoring and troubleshooting part of the CV by showing how incident logs can be collected, parsed, searched, and reviewed. It is separate from the Zabbix project: Zabbix handles simple checks, while this project focuses on logs and observability evidence.

## What This Project Demonstrates

- A local Docker Compose scaffold for Elasticsearch, Logstash, Kibana, Grafana, and Filebeat.
- Minimal configs required for the stack to validate independently.
- Sample logs and alert intent that can be reviewed without starting the full stack.
- An incident simulation script for generating operational log evidence.

This is a local demo stack. The default credentials are for non-production use only.

## Architecture

```
Application/System Logs -> Filebeat -> Logstash -> Elasticsearch -> Kibana
                                                       |
                                                       v
                                                    Grafana

Incident Scenario -> Sample Logs -> Alert Rules -> Operator Review
```

Core components:

- `docker-compose.yml` defines the observability services.
- `alerting/alert_rules.yml` records alert intent and severity.
- `examples/` contains representative operational logs and alert output.
- `scenarios/incident_simulation.sh` emits incident activity.
- `grafana/`, `kibana/`, `logstash/`, `filebeat/`, and `elasticsearch/` contain minimal local configs.

## Quickstart

```bash
cd professional-infra/log-observability-elk-grafana
make test
make demo
```

Start the full local stack with Docker:

```bash
make test
make run
make down
```

When running locally:

- Kibana: `http://localhost:5601`
- Grafana: `http://localhost:3000`
- Elasticsearch: `http://localhost:9200`

Default demo credentials:

- Elasticsearch/Kibana: `elastic` / `elastic`
- Grafana: `admin` / `admin`

## Validation

```bash
make test
docker compose config --quiet
```

`make test` also checks that all bind-mounted config files and directories exist.

## Example Output

```text
[2026-04-29 04:18:23] WARN Database connection pool nearing capacity
[2026-04-29 04:18:28] ERROR Database connection pool exhausted
[2026-04-29 04:18:33] ERROR Database query timeout occurred
[2026-04-29 04:18:44] INFO Database connections restored
```

Additional examples are available in [examples/alert-output.txt](examples/alert-output.txt) and [examples/sample-log.txt](examples/sample-log.txt).

## Interview Talking Points

- When to use Zabbix checks versus ELK log analysis.
- How Filebeat, Logstash, and Elasticsearch fit into a basic log pipeline.
- How incident simulations create evidence for troubleshooting discussions.
- Why local demo credentials and single-node Elasticsearch are not production architecture.

## Roadmap

- Add curated Grafana and Kibana dashboards.
- Add Prometheus metrics collection.
- Add distributed tracing with Jaeger or OpenTelemetry.
- Add synthetic monitoring checks.