29 lines
874 B
Markdown
29 lines
874 B
Markdown
|
# Troubleshooting Cases
|
||
|
|
|
||
|
|
## `IDLE+NOT_RESPONDING` after node maintenance
|
||
|
|
|
||
|
|
Symptoms: `sinfo` shows `idle*` or `scontrol show node` shows `IDLE+NOT_RESPONDING`.
|
||
|
|
|
||
|
|
Actions:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
systemctl restart munge
|
||
|
|
systemctl restart slurmd
|
||
|
|
systemctl restart slurmctld
|
||
|
|
scontrol update NodeName=<node> State=RESUME || true
|
||
|
|
scontrol update NodeName=<node> State=UNDRAIN || true
|
||
|
|
scontrol update NodeName=<node> State=IDLE || true
|
||
|
|
```
|
||
|
|
|
||
|
|
## Missing GPU TRES
|
||
|
|
|
||
|
|
Symptoms: `sacctmgr` fails with `no TRES known by type gres/gpu`.
|
||
|
|
|
||
|
|
Fix: add `AccountingStorageTRES=...,gres/gpu`, restart/reconfigure Slurm, run a GPU job and verify with `sacctmgr show tres`.
|
||
|
|
|
||
|
|
## SlurmDBD objects already exist
|
||
|
|
|
||
|
|
Symptoms: `sacctmgr` returns `Nothing new added` or `Already existing`.
|
||
|
|
|
||
|
|
Fix: make Ansible tasks idempotent: attempt the change, tolerate known existing-object messages, then normalize state with `modify`.
|