Add incident log summary tool

2026-05-11 17:03:31 +00:00
parent 61483c233f
commit 5dde403ce3
5 changed files with 765 additions and 0 deletions
@@ -0,0 +1,8 @@
+2026-05-11 09:48:12 app01 api[4150]: INFO request_id=7f3a status=200 path=/health
+2026-05-11 10:01:03 app01 api[4150]: ERROR request_id=8b21 HTTP 500 path=/checkout duration_ms=942
+2026-05-11 10:03:19 app01 api[4150]: WARNING request_id=8b22 database unavailable for payments cluster
+2026-05-11 10:05:44 app01 api[4150]: ERROR request_id=8b25 timeout waiting for inventory service
+2026-05-11 10:07:02 app01 api[4150]: ERROR request_id=8b29 connection refused connecting to redis-cache:6379
+2026-05-11T10:11:33 app01 api[4150]: CRITICAL request_id=8b31 TLS handshake failed: certificate expired
+2026-05-11 10:13:58 app01 api[4150]: ERROR request_id=8b44 HTTP 503 path=/checkout upstream unavailable
+2026-05-11 12:10:01 app01 api[4150]: INFO request_id=9001 status=200 path=/health
@@ -0,0 +1,144 @@
+# Incident Log Summary
+
+## CRITICAL: certificate expired
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:11:33
+- Last seen: 2026-05-11 10:11:33
+
+Sample log lines:
+
+```text
+2026-05-11T10:11:33 app01 api[4150]: CRITICAL request_id=8b31 TLS handshake failed: certificate expired
+```
+
+## CRITICAL: CRITICAL
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:11:33
+- Last seen: 2026-05-11 10:11:33
+
+Sample log lines:
+
+```text
+2026-05-11T10:11:33 app01 api[4150]: CRITICAL request_id=8b31 TLS handshake failed: certificate expired
+```
+
+## CRITICAL: database unavailable
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:03:19
+- Last seen: 2026-05-11 10:03:19
+
+Sample log lines:
+
+```text
+2026-05-11 10:03:19 app01 api[4150]: WARNING request_id=8b22 database unavailable for payments cluster
+```
+
+## CRITICAL: HTTP 500
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:01:03
+- Last seen: 2026-05-11 10:01:03
+
+Sample log lines:
+
+```text
+2026-05-11 10:01:03 app01 api[4150]: ERROR request_id=8b21 HTTP 500 path=/checkout duration_ms=942
+```
+
+## CRITICAL: HTTP 503
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:13:58
+- Last seen: 2026-05-11 10:13:58
+
+Sample log lines:
+
+```text
+2026-05-11 10:13:58 app01 api[4150]: ERROR request_id=8b44 HTTP 503 path=/checkout upstream unavailable
+```
+
+## CRITICAL: TLS handshake failed
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:11:33
+- Last seen: 2026-05-11 10:11:33
+
+Sample log lines:
+
+```text
+2026-05-11T10:11:33 app01 api[4150]: CRITICAL request_id=8b31 TLS handshake failed: certificate expired
+```
+
+## WARNING: ERROR
+
+- Occurrences: 4
+- First seen: 2026-05-11 10:01:03
+- Last seen: 2026-05-11 10:13:58
+
+Sample log lines:
+
+```text
+2026-05-11 10:01:03 app01 api[4150]: ERROR request_id=8b21 HTTP 500 path=/checkout duration_ms=942
+2026-05-11 10:05:44 app01 api[4150]: ERROR request_id=8b25 timeout waiting for inventory service
+2026-05-11 10:07:02 app01 api[4150]: ERROR request_id=8b29 connection refused connecting to redis-cache:6379
+```
+
+## WARNING: unavailable
+
+- Occurrences: 2
+- First seen: 2026-05-11 10:03:19
+- Last seen: 2026-05-11 10:13:58
+
+Sample log lines:
+
+```text
+2026-05-11 10:03:19 app01 api[4150]: WARNING request_id=8b22 database unavailable for payments cluster
+2026-05-11 10:13:58 app01 api[4150]: ERROR request_id=8b44 HTTP 503 path=/checkout upstream unavailable
+```
+
+## WARNING: connection refused
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:07:02
+- Last seen: 2026-05-11 10:07:02
+
+Sample log lines:
+
+```text
+2026-05-11 10:07:02 app01 api[4150]: ERROR request_id=8b29 connection refused connecting to redis-cache:6379
+```
+
+## WARNING: failed
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:11:33
+- Last seen: 2026-05-11 10:11:33
+
+Sample log lines:
+
+```text
+2026-05-11T10:11:33 app01 api[4150]: CRITICAL request_id=8b31 TLS handshake failed: certificate expired
+```
+
+## WARNING: timeout
+
+- Occurrences: 1
+- First seen: 2026-05-11 10:05:44
+- Last seen: 2026-05-11 10:05:44
+
+Sample log lines:
+
+```text
+2026-05-11 10:05:44 app01 api[4150]: ERROR request_id=8b25 timeout waiting for inventory service
+```
+
+## Operational Summary
+
+- Total lines scanned: 8
+- Total findings: 15
+- Critical finding groups: 6
+- Warning finding groups: 5
+- Overall status: CRITICAL
@@ -0,0 +1,7 @@
+May 11 09:57:01 ops-node-01 systemd[1]: Started Session 443 of user svc_backup.
+May 11 10:02:14 ops-node-01 systemd[1]: failed to start nightly-report.service: Unit entered failed state.
+May 11 10:04:22 ops-node-01 sudo[18442]: svc_backup : command not allowed ; permission denied
+May 11 10:16:07 ops-node-01 kernel: EXT4-fs warning: no space left on device while writing /var/log/messages
+May 11 10:21:45 ops-node-01 kernel: out of memory: killed process 2517 (java) total-vm:2048000kB
+May 11 10:22:03 ops-node-01 systemd[1]: service restart scheduled for app-worker.service
+May 11 10:30:31 ops-node-01 sshd[19210]: Accepted publickey for admin from 192.0.2.15 port 52210 ssh2