Commit ef60d538 authored by goekce's avatar goekce
Browse files

slurm details

parent 101b8361
# 2021-08-04
## [Tutorial Slurm workload manager 2012](https://www.open-mpi.org/video/?category=slurm), [slides](https://www.open-mpi.org/video/slurm/Slurm_EMC_Dec2012.pdf) by Morris Jette.
Main components:
- `slurmctld` central controller, one per cluster
- `slurmd` compute node daemon, one per node
- `slurmdbd` database, one per enterprise
Run on demand:
- `slurmstepd` to manage a job step, one per job step
Commands:
- `sbatch` submit for later execution, batch mode
- `salloc` allocate right now, starts a shell, interactive mode
- `srun` create a job allocation, and launch a job step (an MPI job)
- `sattach` connect to an existing job (like in `screen` or `tmux`)
## Archlinux
- `pacman -Qqe` lists explicitly installed packages. Useful for backup and fresh installation
- `paccheck --md5sum --quiet` (from `pacutils` package) lists changed files. Useful to see changes from default configuration
- for a single package: `paccheck --md5sum --quiet slurm-llnl`
## Installation Slurm:
- [Slurm Workload Manager - Quick Start Administrator Guide](https://slurm.schedmd.com/quickstart_admin.html)
- interesting plugins and their deps
- *cgroup task affinity* for `task/cgroup`, for binding processes to specific processors? => `hwloc`
- *IPMI energy consumption* for `acct_gather_energy/ipmi` for monitoring energy cons. => `freeipmi` dev. library
- *MUNGE* for `auth/munge`, this is the default authentication mechanism for auth. between nodes, e.g., process of user1 on comp1 can authenticate and run on comp2
- *MySQL* for accounting, could be interesting to introduce a weekly limit
- *PAM* for restricting access to specific compute resources, e.g., Slurm can prevent a user from accessing a node which they has not been allocated if desired => pam dev. library
- *NUMA affinity* for NUMA-aware affinity? makes more sense when binding to multiple processors => `numactl`
- *readline* for interactive `scontrol` and `sacctmgr`
- *RRD external sensor data* for `ext_sensors/rrd` plugin => `rrdtool`
- *sview* GUI => `gtk2`
- install packages
```sh
pikaur -S munge gtk2 --noconfirm
```
- https://slurm.schedmd.com/configurator.easy.html for `slurm.conf`. Currently used [`slurm.conf`](slurm.conf) is in the repo. Both `slurmd` and `slurmctld` use `slurm.conf`.
- `systemctl enable --now slurmd`
- `slurmdbd` for job accounting. The guide states:
> defer adding accounting support until after basic Slurm functionality is established on your system
## [Configless Slurm](https://slurm.schedmd.com/configless_slurm.html)
to avoid config file distribution to the nodes. They just have to be started using `--confserver slurmctld-primary-server-name` or using DNS. The last option can be specified in `/etc/sysconfig/slurm-llnl` (or `/etc/sysconfig/slurm-llnl`, look into the service file)
Note that in a large cluster, main node should better not accept computing jobs to be available for configuration file requests
## Testing
on one of the machines:
```
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
p* up infinite 2 idle c[0-1]
$ sview
# opens up a GUI
```
## Installation other:
- if `slurmctld` does not start after boot automatically, [add `network-online.target` to the service files](https://github.com/SchedMD/slurm/commit/e88f7ff11a3c01d092628a3f73ef6cf64ce77a35#diff-89f02e71ee43332477dac1af7c5fd141776a5d132b4a5ac752bd81649453b076)
- `xorg-xauth` for X11-forwarding
- `/etc/systemd/network/20-en.network` including NTP:
```
[Match]
Name=eno0
[Network]
DHCP=yes
NTP=ntp.th-deg.de
```
- `timedatectl set-ntp true`
# 2021-06-18
STAR resource utilization using `/usr/bin/time -v ...` for indexing and mapping a human genome sample:
......@@ -64,10 +152,20 @@ Command being timed: "make mapping"
- STAR supports only OpenMP but not OpenMPI
- Unix as host, because clustering using the kernel is easier
- Server Modell: S2600CP
- [supported RAM](https://ark.intel.com/content/www/us/en/ark/products/56333/intel-server-board-s2600cp2.html#tab-blade-1-0-2)
- DDR3 ECC UDIMM 1333/1600, RDIMM 1066/1333/1600/1866, LRDIMM 1333/1600/1866
- max 512G
- 16 DIMMs 8 channels (32G per DIMM max)
- https://geizhals.de/?cat=ramddr3&xf=15903_DDR3%7E15903_mitECC
- 32G ECC DDR 1866 kostet 70 EUR
## Goal
- specs for hardware
presenting an example with `slurm`
- running an application like STAR in
- interactive mode
- batch mode (many runs with many parameters)
## use case
......
# Manually created using the configuration parameters in
# https://slurm.schedmd.com/configurator.easy.html
# 2021-08-04 goekce.aydos@th-deg.de
ClusterName=c
## Control machines
SlurmctldHost=c0
## Compute machines
# info using `slurmd -C`
# if memory should also be allocated, then `RealMemory` should be included
NodeName=c0 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 #RealMemory=56233
NodeName=c1 CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=12 ThreadsPerCore=2 #RealMemory=64293
## Partitions
# if no partitions are specified, then slurmctld won't start
# See *partition configuration* on https://slurm.schedmd.com/slurm.conf.html
PatitionName=DEFAULT Default=YES
## Slurm User
SlurmUser=slurm
## State Preservation
# automatically return service if the compute server was non-responsive
ReturnToService=1
# AUR package creates the following state files
StateSaveLocation=/var/spool/slurm/slurmctld
SlurmdSpoolDir=/var/spool/slurm/slurmd
## Resource selection
SelectType=select/cons_tres
# default is `select/linear` which makes whole nodes selectable but not single
# CPUs. `select/tres` makes cores, memory, and all other trackable resources
# allocable
SelectTypeParameters=CR_CPU
# default is `CR_Core`, but the parameter has to be provided manually:
# https://bugs.schedmd.com/show_bug.cgi?id=12214
# If also memory should be allocable use `CR_Core_Memory`
## Logging
# according to slurm.conf(5) `Slurm{d,ctld}Debug` is `quiet` as default,
# because they are running in the foreground using -D switch. Set them to info
# to get messages in journalctl
SlurmdSyslogDebug=verbose
SlurmctldSyslogDebug=verbose
## Configless Slurm
SlurmctldParameters=enable_configless
# Slurm
Slurm is a resource manager and job scheduler. Functions:[^slurm-overview]
- resource allocation to many resources
- framework for managing parallel jobs
- arbitration in case there is more work than resources
[^slurm-overview](https://slurm.schedmd.com/overview.html)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment