Commit ef60d538 authored by goekce's avatar goekce
Browse files

slurm details

parent 101b8361
# 2021-08-04
## [Tutorial Slurm workload manager 2012](, [slides]( by Morris Jette.
Main components:
- `slurmctld` central controller, one per cluster
- `slurmd` compute node daemon, one per node
- `slurmdbd` database, one per enterprise
Run on demand:
- `slurmstepd` to manage a job step, one per job step
- `sbatch` submit for later execution, batch mode
- `salloc` allocate right now, starts a shell, interactive mode
- `srun` create a job allocation, and launch a job step (an MPI job)
- `sattach` connect to an existing job (like in `screen` or `tmux`)
## Archlinux
- `pacman -Qqe` lists explicitly installed packages. Useful for backup and fresh installation
- `paccheck --md5sum --quiet` (from `pacutils` package) lists changed files. Useful to see changes from default configuration
- for a single package: `paccheck --md5sum --quiet slurm-llnl`
## Installation Slurm:
- [Slurm Workload Manager - Quick Start Administrator Guide](
- interesting plugins and their deps
- *cgroup task affinity* for `task/cgroup`, for binding processes to specific processors? => `hwloc`
- *IPMI energy consumption* for `acct_gather_energy/ipmi` for monitoring energy cons. => `freeipmi` dev. library
- *MUNGE* for `auth/munge`, this is the default authentication mechanism for auth. between nodes, e.g., process of user1 on comp1 can authenticate and run on comp2
- *MySQL* for accounting, could be interesting to introduce a weekly limit
- *PAM* for restricting access to specific compute resources, e.g., Slurm can prevent a user from accessing a node which they has not been allocated if desired => pam dev. library
- *NUMA affinity* for NUMA-aware affinity? makes more sense when binding to multiple processors => `numactl`
- *readline* for interactive `scontrol` and `sacctmgr`
- *RRD external sensor data* for `ext_sensors/rrd` plugin => `rrdtool`
- *sview* GUI => `gtk2`
- install packages
pikaur -S munge gtk2 --noconfirm
- for `slurm.conf`. Currently used [`slurm.conf`](slurm.conf) is in the repo. Both `slurmd` and `slurmctld` use `slurm.conf`.
- `systemctl enable --now slurmd`
- `slurmdbd` for job accounting. The guide states:
> defer adding accounting support until after basic Slurm functionality is established on your system
## [Configless Slurm](
to avoid config file distribution to the nodes. They just have to be started using `--confserver slurmctld-primary-server-name` or using DNS. The last option can be specified in `/etc/sysconfig/slurm-llnl` (or `/etc/sysconfig/slurm-llnl`, look into the service file)
Note that in a large cluster, main node should better not accept computing jobs to be available for configuration file requests
## Testing
on one of the machines:
$ sinfo
p* up infinite 2 idle c[0-1]
$ sview
# opens up a GUI
## Installation other:
- if `slurmctld` does not start after boot automatically, [add `` to the service files](
- `xorg-xauth` for X11-forwarding
- `/etc/systemd/network/` including NTP:
- `timedatectl set-ntp true`
# 2021-06-18
STAR resource utilization using `/usr/bin/time -v ...` for indexing and mapping a human genome sample:
......@@ -64,10 +152,20 @@ Command being timed: "make mapping"
- STAR supports only OpenMP but not OpenMPI
- Unix as host, because clustering using the kernel is easier
- Server Modell: S2600CP
- [supported RAM](
- DDR3 ECC UDIMM 1333/1600, RDIMM 1066/1333/1600/1866, LRDIMM 1333/1600/1866
- max 512G
- 16 DIMMs 8 channels (32G per DIMM max)
- 32G ECC DDR 1866 kostet 70 EUR
## Goal
- specs for hardware
presenting an example with `slurm`
- running an application like STAR in
- interactive mode
- batch mode (many runs with many parameters)
## use case
# Manually created using the configuration parameters in
# 2021-08-04
## Control machines
## Compute machines
# info using `slurmd -C`
# if memory should also be allocated, then `RealMemory` should be included
NodeName=c0 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 #RealMemory=56233
NodeName=c1 CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=12 ThreadsPerCore=2 #RealMemory=64293
## Partitions
# if no partitions are specified, then slurmctld won't start
# See *partition configuration* on
PatitionName=DEFAULT Default=YES
## Slurm User
## State Preservation
# automatically return service if the compute server was non-responsive
# AUR package creates the following state files
## Resource selection
# default is `select/linear` which makes whole nodes selectable but not single
# CPUs. `select/tres` makes cores, memory, and all other trackable resources
# allocable
# default is `CR_Core`, but the parameter has to be provided manually:
# If also memory should be allocable use `CR_Core_Memory`
## Logging
# according to slurm.conf(5) `Slurm{d,ctld}Debug` is `quiet` as default,
# because they are running in the foreground using -D switch. Set them to info
# to get messages in journalctl
## Configless Slurm
# Slurm
Slurm is a resource manager and job scheduler. Functions:[^slurm-overview]
- resource allocation to many resources
- framework for managing parallel jobs
- arbitration in case there is more work than resources
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment