mateusz/portfolio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document Slurm AI/HPC cluster project
lint / shell-yaml-ansible (push) Failing after 16s
Details

Browse Source
This commit is contained in:
Mateusz Suski
2026-06-04 19:54:43 +00:00
parent d300d490f5
commit 83877fb598
5 changed files with 239 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
platform-projects/README.md
+8 -2
View File
@@ -1,8 +1,14 @@
# platform-projects
This directory is reserved for larger infrastructure platform topics and future case studies. The current implemented project is [infra-run](../infra-run/).
This directory contains larger infrastructure platform topics and case studies. Most subdirectories are planning areas unless their own README says otherwise.
Current subdirectories are intentionally light and should be read as planning areas unless their own README says otherwise:
## Implemented platform projects
- [hpc-slurm-ai-cluster](./hpc-slurm-ai-cluster/) - Slurm AI/HPC cluster automation covering Ansible-managed Slurm operations, GPU scheduling with GRES, cgroup enforcement, SlurmDBD accounting, QOS/fairshare/priority, node lifecycle operations, rolling upgrades, and health remediation.
## Planning areas
These subdirectories are intentionally light and should be read as planning areas unless their own README says otherwise:
- `monitoring-zabbix`
- `elk-log-analysis`
platform-projects/hpc-slurm-ai-cluster/README.md
+215 -38
View File
@@ -1,59 +1,236 @@
# Ansible Slurm AI/HPC Lab
# Slurm AI/HPC Cluster Automation Lab
Ansible automation for a small Slurm AI/HPC lab with CPU nodes, a GPU node, Munge, cgroups, GRES, SlurmDBD accounting, QOS/fairshare, node lifecycle workflows, rolling OS upgrades and health remediation.
## Executive summary
This repository is sanitized for publication. Replace the example inventory values under `inventories/lab/` with your own hostnames, IP addresses and users before running it.
This project builds and operates a small production-like Slurm AI/HPC cluster in a sanitized lab. It uses Ansible to bootstrap hosts, manage Munge authentication, deploy Slurm controller and worker configuration, integrate a GPU node through GRES, enable cgroup enforcement, configure accounting, apply QOS/fairshare policy, and run operational validation jobs.
## What this lab covers
The goal is not to present a certified production platform. The goal is to show practical Linux, HPC, and SRE-style operational work: controlled automation, repeatable workflows, explicit checks, recovery steps, and evidence that the cluster behaves as expected.
- Slurm controller and worker configuration
- Munge key distribution
- GPU GRES configuration
- cgroup CPU/GPU/device enforcement
- SlurmDBD + MariaDB accounting
- `sacct`, `sreport`, `sacctmgr` validation
- QOS, limits, fairshare and priority/multifactor
- Node provisioning and decommissioning
- Rolling OS upgrades with canary validation
- Health checks and node auto-remediation
## What this project demonstrates
- Slurm controller and worker node management.
- Munge authentication across the cluster.
- GPU node integration through Slurm GRES.
- cgroup CPU, memory, and GPU device enforcement.
- SlurmDBD with MariaDB-backed accounting.
- `sacct`, `sreport`, and `sacctmgr` workflows.
- QOS, fairshare, and multifactor priority configuration.
- Node provisioning and decommissioning workflows.
- Rolling OS upgrades with canary validation.
- Health checks and auto-remediation.
- Backup and restore-check workflow for the accounting database.
- Operational validation jobs for CPU, GPU, cgroup, accounting, and reporting behavior.
## Architecture overview
```mermaid
flowchart LR
operator[Ansible control node]
munge[Munge authentication]
controller[Slurm controller<br/>slurmctld]
db[MariaDB + SlurmDBD<br/>accounting]
shared[Shared filesystem<br/>site dependency]
cpu_part[CPU partition]
gpu_part[GPU partition]
cpu_nodes[CPU compute nodes<br/>slurmd]
gpu_node[GPU node<br/>slurmd + GRES]
jobs[User jobs<br/>sbatch / srun]
operator -->|bootstrap and configure| controller
operator -->|configure workers| cpu_nodes
operator -->|configure GPU worker| gpu_node
operator -->|deploy key and service| munge
munge --> controller
munge --> cpu_nodes
munge --> gpu_node
controller -->|accounting RPC| db
jobs -->|submit to Slurm| controller
controller -->|schedule CPU jobs| cpu_part
controller -->|schedule GPU jobs| gpu_part
cpu_part --> cpu_nodes
gpu_part --> gpu_node
cpu_nodes --- shared
gpu_node --- shared
controller --- shared
```
The lab models a common Slurm pattern: an Ansible control node manages a Slurm controller, CPU workers, a GPU worker, Munge authentication, SlurmDBD accounting, and policy configuration. CPU and GPU jobs flow through Slurm partitions; GPU access is declared through GRES and constrained with cgroups.
## Repository layout
```text
inventories/lab/ Example inventory and group variables
templates/ Slurm, cgroup, gres and slurmdbd templates
playbooks/bootstrap/ Initial SSH, sudo and /etc/hosts setup
playbooks/core/ Munge, Slurm config and safe restart workflows
playbooks/accounting/ SlurmDBD, backup/restore-check and accounting validation
playbooks/qos/ QOS, fairshare and priority configuration
playbooks/lifecycle/ Provisioning and decommissioning nodes
playbooks/upgrade/ Rolling OS upgrade and canary workflow
playbooks/health/ Health checks and auto-remediation
playbooks/tests/ CPU/GPU/cgroup/accounting validation jobs
playbooks/backup/ Slurm config backup helpers
docs/ Runbooks and interview notes
prompts/codex/ Prompts for generating or expanding documentation
inventories/lab/ Sanitized lab inventory and group variables
playbooks/bootstrap/ Initial SSH, sudo, operator user, and host setup
playbooks/core/ Munge, Slurm config, and safe restart workflows
playbooks/accounting/ SlurmDBD, MariaDB, backup, restore-check, and reporting validation
playbooks/qos/ QOS, fairshare, and priority configuration
playbooks/lifecycle/ Node provisioning, inspection, and decommissioning
playbooks/upgrade/ Canary and rolling OS upgrade workflows
playbooks/health/ Health checks, repair, and auto-remediation
playbooks/tests/ CPU, GPU, cgroup, accounting, and reporting validation jobs
playbooks/backup/ Slurm and Munge state backup helpers
templates/ Slurm, cgroup, GRES, and SlurmDBD templates
docs/ Runbook, interview notes, and troubleshooting cases
prompts/ Documentation prompts used to expand this project
```
## Quick start
## Main operational workflows
1. Edit `inventories/lab/inventory.yml`.
2. Edit `inventories/lab/group_vars/slurm_cluster.yml`.
3. Create and encrypt a vault file for database credentials:
Run commands from `platform-projects/hpc-slurm-ai-cluster/`. Review inventory and variables before running any playbook.
### Bootstrap access
```bash
cp inventories/lab/group_vars/vault.example.yml inventories/lab/group_vars/vault.yml
ansible-vault encrypt inventories/lab/group_vars/vault.yml
ansible-playbook playbooks/bootstrap/bootstrap-ansible.yml --ask-pass --ask-become-pass
ansible-playbook playbooks/bootstrap/slurm-hosts.yml
ansible-playbook playbooks/bootstrap/slurmuser-ssh-mesh.yml
ansible-playbook playbooks/bootstrap/slurmuser-sudoers-fix.yml
```
4. Run syntax checks:
### Deploy Munge
```bash
find playbooks -name '*.yml' -print0 | xargs -0 -n1 ansible-playbook --syntax-check
ansible-playbook playbooks/core/manage-munge.yml
```
5. Run the bootstrap/core workflows in the order described in `docs/runbook.md`.
### Deploy Slurm config
## Security notes
```bash
ansible-playbook playbooks/core/manage-slurm-config.yml --check --diff
ansible-playbook playbooks/core/manage-slurm-config.yml --diff
ansible-playbook playbooks/core/restart-slurm-safe.yml
```
Do not commit real inventories, backup archives, SQL dumps, Munge keys, private SSH keys or Ansible Vault files. This repository intentionally excludes generated backup artifacts.
### Validate CPU jobs
```bash
ansible-playbook playbooks/tests/validate-slurm-operator.yml
ansible-playbook playbooks/tests/test-cpu-job.yml
```
### Validate GPU jobs
```bash
ansible-playbook playbooks/tests/test-gpu-job.yml
ansible-playbook playbooks/tests/test-gpu-deny-without-gres.yml
```
### Enable accounting
```bash
ansible-playbook playbooks/accounting/setup-slurmdbd.yml
ansible-playbook playbooks/accounting/initialize-slurm-accounting.yml
ansible-playbook playbooks/accounting/validate-slurm-accounting.yml
ansible-playbook playbooks/tests/test-sreport-usage.yml
```
### Configure QOS and fairshare
```bash
ansible-playbook playbooks/qos/configure-slurm-qos.yml
ansible-playbook playbooks/qos/validate-slurm-qos-priority.yml
```
### Provision a node
```bash
ansible-playbook playbooks/lifecycle/provision-slurm-node.yml -e target_node=<node>
ansible-playbook playbooks/tests/test-specific-node.yml -e target_node=<node>
```
### Decommission a node
```bash
ansible-playbook playbooks/lifecycle/decommission-slurm-node.yml \
-e target_node=<node> \
-e "decom_reason=planned maintenance"
```
### Rolling OS upgrade
```bash
ansible-playbook playbooks/upgrade/canary-slurm-node-upgrade.yml -e canary_node=<node>
ansible-playbook playbooks/upgrade/rolling-upgrade-slurm-workers.yml \
-e canary_node=<node> \
-e skip_canary=true
ansible-playbook playbooks/upgrade/upgrade-slurm-controller.yml
ansible-playbook playbooks/upgrade/validate-after-os-upgrade.yml
```
### Health check and auto-remediation
```bash
ansible-playbook playbooks/health/check-slurm-health.yml
ansible-playbook playbooks/health/auto-remediate-slurm-health.yml
ansible-playbook playbooks/health/repair-slurm-node.yml -e target_node=<node>
```
### Accounting backup and restore-check
```bash
ansible-playbook playbooks/accounting/backup-slurmdbd.yml
ansible-playbook playbooks/accounting/restore-check-slurmdbd.yml
```
## Operational maturity
This is more than a toy lab because it includes operational controls around the cluster, not only a static `slurm.conf` example.
- Ansible workflows are designed to be repeatable and readable.
- Configuration deployment supports check and diff review before applying changes.
- Validation jobs prove CPU scheduling, GPU scheduling, cgroup behavior, accounting, and reporting.
- SlurmDBD and MariaDB accounting are configured with `sacct`, `sreport`, and `sacctmgr` validation.
- QOS, fairshare, priority, and association workflows show resource governance.
- Node lifecycle playbooks drain, decommission, reprovision, resume, and validate nodes.
- Rolling upgrade playbooks include canary validation before broader worker upgrades.
- Health and repair playbooks document remediation paths for common node states.
- Backup and restore-check playbooks verify that accounting data can be dumped and imported into a test database.
- Troubleshooting cases document real lab failure modes without exposing private infrastructure details.
## Tested capabilities
- [x] CPU job scheduling.
- [x] GPU job scheduling.
- [x] GPU denial when no GRES is requested.
- [x] CPU cgroup enforcement.
- [x] SlurmDBD accounting setup.
- [x] `sacct` job history visibility.
- [x] `sreport` usage reporting.
- [x] QOS creation and validation.
- [x] Fairshare and priority visibility.
- [x] Node decommission and reprovision workflow.
- [x] Rolling upgrade canary workflow.
- [x] Node health check and auto-remediation workflow.
These checks represent sanitized lab validation, not a claim of production certification.
## Safety and sanitization
This repository is prepared for public portfolio review. Inventory values are examples, and the sample `10.10.10.x` addresses are sanitized lab placeholders.
Do not commit real inventories, internal hostnames, private IP plans, Munge keys, SSH private keys, database dumps, generated backup archives, or Ansible Vault files. Real credentials, including SlurmDBD database passwords, belong in Ansible Vault or another approved secret store.
Generated backup artifacts are intentionally excluded from the repository. Treat backup paths and database names in playbooks as examples that must be reviewed before use in a real environment.
## Why this matters for AI/HPC infrastructure roles
AI and HPC platforms depend on more than GPU hardware. They need Linux system ownership, scheduler operations, authentication, resource isolation, accounting, upgrade discipline, and a clear recovery path when nodes drift or fail.
This project demonstrates practical understanding of:
- Linux systems operations.
- Slurm cluster operations.
- GPU infrastructure and GRES scheduling.
- Job scheduling and resource isolation.
- Accounting, reporting, QOS, fairshare, and priority policy.
- Automation and repeatability with Ansible.
- Troubleshooting and operational ownership.
## Deeper docs
- [Runbook](docs/runbook.md)
- [Interview cheatsheet](docs/interview-cheatsheet.md)
- [Troubleshooting cases](docs/troubleshooting-cases.md)
platform-projects/hpc-slurm-ai-cluster/docs/runbook.md
+13
View File
@@ -50,6 +50,19 @@ Repair a node:
ansible-playbook playbooks/health/repair-slurm-node.yml -e target_node=slurm-c02
```
Run health remediation for nodes that can be recovered by the automated workflow:
```bash
ansible-playbook playbooks/health/auto-remediate-slurm-health.yml
```
Back up Slurm and Munge state before planned lifecycle work:
```bash
ansible-playbook playbooks/backup/backup-slurm-state.yml
ansible-playbook playbooks/backup/fetch-slurm-backups.yml
```
## Rolling OS upgrade
```bash