Add Slurm AI/HPC cluster platform project

platform-projects/hpc-slurm-ai-cluster/docs/interview-cheatsheet.md

@@ -0,0 +1,22 @@
+# Interview Cheatsheet: Slurm AI/HPC Lab
+
+## One-minute summary
+
+I built an Ansible-managed Slurm AI/HPC lab with a controller, CPU compute nodes and a GPU node. The lab includes Munge authentication, cgroup-based CPU/GPU enforcement, GRES GPU scheduling, SlurmDBD accounting backed by MariaDB, QOS/fairshare/priority policies, rolling OS upgrades, node provisioning/decommissioning and health remediation workflows.
+
+## Topics I can discuss
+
+- How Slurm schedules CPU and GPU workloads.
+- Difference between GRES scheduling and cgroup device enforcement.
+- Why Munge key consistency matters.
+- How `slurmdbd`, `sacct`, `sacctmgr` and `sreport` fit together.
+- How QOS, account associations, fairshare and multifactor priority work.
+- Operational workflows: drain, decommission, provision, rolling upgrade, canary test and auto-remediation.
+
+## Real troubleshooting examples
+
+- `IDLE+NOT_RESPONDING` after node reprovisioning.
+- Accounting delay where `sacct` temporarily showed `PENDING` while job output existed.
+- Missing `gres/gpu` TRES before QOS GPU limits could be configured.
+- `sacctmgr` idempotency issues such as `Nothing new added`.
+- Slurm version differences around state transitions such as `RESUME`, `UNDRAIN` and `IDLE`.

platform-projects/hpc-slurm-ai-cluster/docs/runbook.md

@@ -0,0 +1,62 @@
+# Slurm AI/HPC Lab Runbook
+
+## Standard deployment order
+
+```bash
+ansible-playbook playbooks/bootstrap/bootstrap-ansible.yml --ask-pass --ask-become-pass
+ansible-playbook playbooks/bootstrap/slurm-hosts.yml
+ansible-playbook playbooks/bootstrap/slurmuser-ssh-mesh.yml
+ansible-playbook playbooks/bootstrap/slurmuser-sudoers-fix.yml
+
+ansible-playbook playbooks/core/manage-munge.yml
+ansible-playbook playbooks/core/manage-slurm-config.yml --check --diff
+ansible-playbook playbooks/core/manage-slurm-config.yml --diff
+ansible-playbook playbooks/core/restart-slurm-safe.yml
+
+ansible-playbook playbooks/tests/validate-slurm-operator.yml
+ansible-playbook playbooks/tests/test-cpu-job.yml
+ansible-playbook playbooks/tests/test-gpu-job.yml
+ansible-playbook playbooks/tests/test-gpu-deny-without-gres.yml
+
+ansible-playbook playbooks/accounting/setup-slurmdbd.yml
+ansible-playbook playbooks/accounting/initialize-slurm-accounting.yml
+ansible-playbook playbooks/accounting/backup-slurmdbd.yml
+ansible-playbook playbooks/accounting/restore-check-slurmdbd.yml
+ansible-playbook playbooks/accounting/validate-slurm-accounting.yml
+
+ansible-playbook playbooks/qos/configure-slurm-qos.yml
+ansible-playbook playbooks/qos/validate-slurm-qos-priority.yml
+
+ansible-playbook playbooks/health/check-slurm-health.yml
+```
+
+## Node lifecycle
+
+Provision a node:
+
+```bash
+ansible-playbook playbooks/lifecycle/provision-slurm-node.yml -e target_node=slurm-c02
+```
+
+Decommission a node:
+
+```bash
+ansible-playbook playbooks/lifecycle/decommission-slurm-node.yml -e target_node=slurm-c02 -e "decom_reason=planned maintenance"
+```
+
+Repair a node:
+
+```bash
+ansible-playbook playbooks/health/repair-slurm-node.yml -e target_node=slurm-c02
+```
+
+## Rolling OS upgrade
+
+```bash
+ansible-playbook playbooks/upgrade/canary-slurm-node-upgrade.yml -e canary_node=slurm-c02
+ansible-playbook playbooks/upgrade/rolling-upgrade-slurm-workers.yml -e canary_node=slurm-c02 -e skip_canary=true
+ansible-playbook playbooks/upgrade/upgrade-slurm-controller.yml
+ansible-playbook playbooks/upgrade/validate-after-os-upgrade.yml
+```
+
+If `upgrade-slurm-controller.yml` is not present, create it from the documented controller upgrade workflow or keep controller upgrades manual.

platform-projects/hpc-slurm-ai-cluster/docs/troubleshooting-cases.md

@@ -0,0 +1,28 @@
+# Troubleshooting Cases
+
+## `IDLE+NOT_RESPONDING` after node maintenance
+
+Symptoms: `sinfo` shows `idle*` or `scontrol show node` shows `IDLE+NOT_RESPONDING`.
+
+Actions:
+
+```bash
+systemctl restart munge
+systemctl restart slurmd
+systemctl restart slurmctld
+scontrol update NodeName=<node> State=RESUME || true
+scontrol update NodeName=<node> State=UNDRAIN || true
+scontrol update NodeName=<node> State=IDLE || true
+```
+
+## Missing GPU TRES
+
+Symptoms: `sacctmgr` fails with `no TRES known by type gres/gpu`.
+
+Fix: add `AccountingStorageTRES=...,gres/gpu`, restart/reconfigure Slurm, run a GPU job and verify with `sacctmgr show tres`.
+
+## SlurmDBD objects already exist
+
+Symptoms: `sacctmgr` returns `Nothing new added` or `Already existing`.
+
+Fix: make Ansible tasks idempotent: attempt the change, tolerate known existing-object messages, then normalize state with `modify`.