miaplpy unwrap memory usage

[falkamelung]

This seems to work:

1st job file

cat /scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0.job
#! /bin/bash
#SBATCH -J run_05_miaplpy_unwrap_ifgram_0
#SBATCH -A TG-EAR200012
#SBATCH --mail-user=falk.amelung@gmail.com
#SBATCH --mail-type=fail
#SBATCH -N 2
#SBATCH -n 96
#SBATCH -o /scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0_%J.o
#SBATCH -e /scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0_%J.e
#SBATCH -p skx-dev
#SBATCH -t 2:00:00
################################################
# execute tasks with launcher
################################################
export OMP_NUM_THREADS=1
export LAUNCHER_PPN=48
export LAUNCHER_NHOSTS=2
export LAUNCHER_JOB_FILE=/scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0
export LAUNCHER_WORKDIR=/dev/shm
cd /dev/shm
$LAUNCHER_DIR/paramrun

after resizing jobfile
#! /bin/bash
#SBATCH -J run_05_miaplpy_unwrap_ifgram_0
#SBATCH -A TG-EAR200012
#SBATCH --mail-user=falk.amelung@gmail.com
#SBATCH --mail-type=fail
#SBATCH -N 2
#SBATCH -n 96
#SBATCH -o /scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0_%J.o
#SBATCH -e /scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0_%J.e
#SBATCH -p skx-dev
#SBATCH -t 2:00:00
################################################

execute tasks with launcher

################################################
export OMP_NUM_THREADS=1
export LAUNCHER_PPN=48
export LAUNCHER_NHOSTS=2
export LAUNCHER_JOB_FILE=/scratch/05861/tg851601/EtnaSenD124/miaplpy_qqBig6_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0
export LAUNCHER_WORKDIR=/dev/shm
cd /dev/shm
$LAUNCHER_DIR/paramrun

top - 01:11:37 up 15 days, 14:37, 2 users, load average: 17.00, 15.46, 9.88
Tasks: 802 total, 18 running, 784 sleeping, 0 stopped, 0 zombie
%Cpu(s): 35.3 us, 0.0 sy, 0.0 ni, 64.6 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
GiB Mem : 187.0 total, 9.1 free, 161.4 used, 19.3 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 25.7 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

1796868 tg851601 20 0 9487796 9.0g 2324 R 99.7 4.8 10:23.81 snaphu
1796869 tg851601 20 0 9487796 9.0g 2344 R 99.7 4.8 10:24.54 snaphu
1796871 tg851601 20 0 9487796 9.0g 2368 R 99.7 4.8 10:24.50 snaphu
1796872 tg851601 20 0 9487796 9.0g 2368 R 99.7 4.8 10:24.47 snaphu
1796873 tg851601 20 0 9487796 9.0g 2340 R 99.7 4.8 10:24.48 snaphu
1796896 tg851601 20 0 9487796 9.0g 2344 R 99.7 4.8 10:23.97 snaphu
1796932 tg851601 20 0 9487796 9.0g 2328 R 99.7 4.8 10:23.05 snaphu
1796953 tg851601 20 0 9487796 9.0g 2364 R 99.7 4.8 10:22.43 snaphu
1797030 tg851601 20 0 9487796 9.0g 2324 R 99.7 4.8 10:21.09 snaphu
1797031 tg851601 20 0 9487796 9.0g 2324 R 99.7 4.8 10:21.33 snaphu
1797072 tg851601 20 0 9487796 9.0g 2340 R 99.7 4.8 10:20.20 snaphu
1797073 tg851601 20 0 9487796 9.0g 2308 R 99.7 4.8 10:20.34 snaphu
1797094 tg851601 20 0 9487796 9.0g 2340 R 99.7 4.8 10:19.51 snaphu
1796870 tg851601 20 0 9487796 9.0g 2332 R 99.3 4.8 10:24.42 snaphu
1796931 tg851601 20 0 9487796 9.0g 2308 R 99.3 4.8 10:23.34 snaphu
1797028 tg851601 20 0 9487796 9.0g 2328 R 99.3 4.8 10:21.30 snaphu
1797029 tg851601 20 0 9487796 9.0g 2332 R 99.3 4.8 10:21.00 snaphu
1798388 tg851601 20 0 11252 5032 3468 R 0.3 0.0 0:00.87 top

top - 01:36:54 up 92 days, 9:50, 1 user, load average: 17.08, 17.04, 15.67
Tasks: 832 total, 18 running, 814 sleeping, 0 stopped, 0 zombie
%Cpu(s): 35.3 us, 0.0 sy, 0.0 ni, 64.6 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
GiB Mem : 187.0 total, 18.9 free, 159.4 used, 10.6 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 27.6 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

4130624 tg851601 20 0 9487796 9.0g 2392 R 99.7 4.8 35:34.14 snaphu
4130625 tg851601 20 0 9487796 9.0g 2356 R 99.7 4.8 35:37.00 snaphu
4130698 tg851601 20 0 9487796 9.0g 2344 R 99.7 4.8 35:35.61 snaphu
4130699 tg851601 20 0 9487796 9.0g 2396 R 99.7 4.8 35:35.77 snaphu
4130739 tg851601 20 0 9487796 9.0g 2344 R 99.7 4.8 35:34.61 snaphu
4130816 tg851601 20 0 9487796 9.0g 2352 R 99.7 4.8 35:33.91 snaphu
4130880 tg851601 20 0 9487796 9.0g 2376 R 99.7 4.8 35:32.07 snaphu
4130922 tg851601 20 0 9487796 9.0g 2376 R 99.7 4.8 35:30.66 snaphu
4130623 tg851601 20 0 9487796 9.0g 2396 R 99.3 4.8 35:37.13 snaphu
4130646 tg851601 20 0 9487796 9.0g 2352 R 99.3 4.8 35:36.62 snaphu
4130697 tg851601 20 0 9487796 9.0g 2376 R 99.3 4.8 35:34.01 snaphu
4130740 tg851601 20 0 9487796 9.0g 2400 R 99.3 4.8 35:34.72 snaphu
4130815 tg851601 20 0 9487796 9.0g 2352 R 99.3 4.8 35:33.97 snaphu
4130817 tg851601 20 0 9487796 9.0g 2360 R 99.3 4.8 35:31.35 snaphu
4130818 tg851601 20 0 9487796 9.0g 2396 R 99.3 4.8 35:31.07 snaphu
4130839 tg851601 20 0 9487796 9.0g 2356 R 99.3 4.8 35:32.72 snaphu
4130881 tg851601 20 0 9487796 9.0g 2340 R 99.3 4.8 35:31.90 snaphu
4132590 tg851601 20 0 11300 5068 3460 R 0.7 0.0 0:00.13 top

[falkamelung]

We hace incidents that only one snafu jobs needs to terminate, with 3 remaining nodes running empty.Tiling the snafu jobs and running them one-after-the-other in one node would solve this.

check_job.bash 3364937
From job file: /scratch/05861/tg851601/EtnaSenD124/miaplpy_qBig5_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0.job
Nodes: 4 Ntasks: 192 CPUs/task: 1 Walltime: 2:00:00 Partition: skx-dev
Total requested CPUs: 192
OMP_NUM_THREADS: 1
LAUNCHER_JOB_FILE: /scratch/05861/tg851601/EtnaSenD124/miaplpy_qBig5_201412_201512/network_delaunay_4/run_files/run_05_miaplpy_unwrap_ifgram_0
Launcher file lines: 64

NODES CPU LOAD L/CPU IDLE% WA% MEM% GB/core STATUS

c454-081 48 1.0 0.02 97.8 0.0 5.9% 0.23 UNDERUSED
c454-082 48 0.0 0.00 99.8 0.0 3.2% 0.12 UNDERUSED
c454-083 48 0.1 0.00 99.8 0.0 3.2% 0.12 UNDERUSED
c454-084 48 0.1 0.00 99.8 0.0 3.7% 0.15 UNDERUSED