using supercomputers

How to use the computers

This section is intended to guide the users of IBIS servers in their management of computing resources.

By computing resources we mean processors (CPU / core), memory (RAM) and
than the time required to complete a task.

It is very important to use these resources in the most optimal way. The
goal is to allow as many users as possible to benefit the computer power.
It is important to request the amount of resources as accurately as possible.
If you do not ask enough, your calculation will take longer to complete (not enough CPU)
or it can be interrupted by the system (not enough memory) and if you ask too much,
you are unnecessarily monopolizing resources that other users might need (CPU and memory).

For example: a user reserve 10 cores which are given by the allocation system. In reality,
the software can use only 1 core. Or he forget to put the option specifying th number of
threads to use. In these 2 cases, there are 9 cores that will not be available to other
users and even, some users could be deprive the capability to launch their calculation.
It is the same thing for memory. If you book a lot of what you actually need, you will
prevent other users from using the computer as well.

So it is essential to evaluate as precisely as possible the amount of
resources you will need to carry out your analyzes.

Please do not hesitate to consult other users who have already done the same analyzes or
to ask the staff of the bioinformatics platform.

Tips before launching an analysis

The first step is to understand how the software you want to use works. Most important,
does it have an option to specify the number of cores it can use? If you don’t know, run the
software with the option -h or -help to see all options available. If no option exist, don’t
request more than 1 core.

Also, even if the software can use multiple cores, it is possible that this is not
an optimal use. In addition, we must think of other users. If you monopolize several dozen
of cores for several hours, you can prevent other users from using the servers. Do some tests
before asking for more than 10 cores and check if the load on the computers is high.
In other times, you can also benefit from this generosity.

Regarding the memory and the calculation time, for most bioinformatics software, there are
no options to specify the amount of memory needed or the computation time. These values must
be estimated empirically. It depends a lot on how coded was the software and on the size of
the file you want to analyze. The type of operation on the file is also an important factor
to be considered.

We suggest to do tests on reduced dataset of different size and to see how it scale. You
can also play on the number of core.

In case of sub-optimal use of resources

If you receive an email from Bioinformatics Platform, this means that you have used less
than 50% of the resources you have been granted.

Read the message carefully to know which resource it is. Then make the necessary adjustments
if you have to restart this analysis. The message will look like this:

Job Runtime JobID Partition CPU CPU% Mem(Go) Used Unused Mem%
ASSEMBLY_TRINITY 93h08m 13952958 soyagen 32 27.46% 400 243.45 156.55 60.86%

Submiting a job with SLURM

To enable optimal use of resources (CPU and RAM), users must submit their work to a system
of job submission. The one used on the IBIS’ computers is SLURM (Simple Linux Utility for Resource Management).

Using a system such as SLURM avoids overloading the computer and
a risk for the operating system (OS) to run out of memory. The results
can go from a simple slowdown in running tasks to a complete shutdown of the server.

The operating principle is simple: the user fills a parameter file
containing analysis information. An example of such a parameter file (SLURM_example.sh)
was put in your account when it was created. Here is the content of such a file.


#!/bin/bash
#SBATCH -D /home/jimlogan/analyse01
#SBATCH -J blast_nt
#SBATCH -o blast_nt-%j.log
#SBATCH –p ibis_mini
#SBATCH –c 10
#SBATCH --mail-type=ALL
#SBATCH --mail-user=jim.logan@ibis.ulaval.ca
#SBATCH --time=2-08:34:21
#SBATCH --mem=10240
#creating a variable for the blast database
nt="/biodata/blastdb/nt"
#to load the executables
module load ncbiblast/2.4.0
#command to execute
blastn -db $nt -query contigs.fasta -num_threads 10 -evalue 1e-5 -outfmt 6 -out contigs_blast_nt_fmt6.out

The user then submits the parameter file to SLURM who will execute the commands when the
requested resources are available:

sbatch SLURM_example.sh

The system will then return a job id.

An example of such a parameter file (SLURM_example.sh) was put in your account when it was
created. Here is the content of such a file.

Explanation of the content of the SLURM file

Format of the time variable:

-t [min]
-t [minutes:secondes]
-t [hours:minutes:secondes]
-t [days-hours]
-t [days-hours:minutes]
-t [days-hours:minutes:secondes]

The memory (–mem) unit is in megabyte: 10240 = 10 Gb

To know the meaning of other variables contained in the lines beginning with #SBATCH,
enter the following command:

man sbatch

To view information about jobs located in the SLURM scheduling queue:
squeue

and for a particular user:
squeue -u

To cancel a job:
scancel

To open the SLURM task manager through a graphical user interface to view all the current jobs:

sview

It is important to know that until your task has been started, you can change the settings.

  1. Right-click on the job you want to edit
  2. Select the Edit Job menu
  3. Scroll down to setting you want to change

But one it has been started, it’s only possible to change the time limit, and this only by the administrator. The other parameters can’t be modified.

Description of queues configured on the computers

thoth

Nom Time Limit Max Cores Max cores per user Max Memory Max Memory per user Priority Advantageous for
small 1 day 240 80 5000G 600G Highest Fast and priority
medium 7 days 100 40 4000G 1000G Moderate One week
large 21 days 80 20 3000G 400G lowest Longuer analysis

manitou

Name Time Limit Max Cores Max Cores per user Max Memory Max Memory per user Priority Advantageous for
rclevesq 21 days 128 40 1000G 1000G Higher Group rclevesq
soyagen 21 days 128 40 1000G 1000G Higher Group soyagen
small 1 day 128 40 700G 500G High Short tasks
medium 7 days 128 30 1000G 1000G Moderate High memory
large 21 days 128 20 600G 400G lower Long tasks

katak

Name Time Limit Max Cores Max Cores per user Max Memory Max Memory per user Priority Advantageous for
small 1 day 60 20 500G 250G Higher Short tasks
medium 7 days 30 16 400G 300G Moderate High memory
large 21 days 30 10 300G 200G lower Long tasks

Limits per users

Server Max Cores Max Memory
thot 120 1000G
manitou 40 1000G
katak 20 300G

small or ibis_small are the default queues for all servers

Graphical Processing Unit (GPU)

The servers thoth and manitou are equiped respectively with 6 and 2 graphic cards NVIDIA Tesla 16G P100. To get access to one or two
of this GPU, you have to use this SLURM syntax within your batch file or with srun:

To use 1 GPU:
#SBATCH --gres=gpu:1

To use 2 GPU:
#SBATCH --gres=gpu:2

In interactive mode:
srun --pty --gres=gpu:1 bash

Usage of GPU can be viewed with the command “`gpu_usage“` (you have to source the file “`/prg/bashrc/commands.sh“`).

It is very important to know that only software code in CUDA programming language can make use of GPU.

To use the server in interactive mode

This mode is interesting when you want to submit small jobs without continuously edit a file
and submit it with the “`sbatch“` command. It is also strongly recommanded when you need to use
a GUI software like CLC Workbenk or Geneious or need to work directly through an command interpreter
like R or python.

The basic command is:
srun --pty bash
By default you will get 1 core, 10G of RAM and 24 hours of walltime limit.

To specify other limits, use these options:
srun -c 4 --mem=102400 --time=2-0:00:00 --pty bash

Si votre tâche requiert une intervention clavier de votre part (ex : demande un mot de passe),
vous devez utiliser l’option –pty de srun. Cette option simule un terminal et vous pourrez
alors entrer des informations au clavier.

Exemple :
srun --pty rsync rep1 toto@machine.ca:
(besoin de –pty puisque rsync vous demanderas un mot de passe)

srun cp a.fasta b.fasta
(pas de –pty puisque cp ne vous demanderas pas d’informations au clavier)

Don’t forget to quit the interactive mode when you have finish

To get info about your past and present jobs

All information about your tasks is stored in a database. It is therefore possible to
consult your history at any time with the command:

sacct

By default, all tasks for the current day will be displayed. To step back in time,
use the -S option:

sacct -S 2016-01-01
will shows tasks since 1 January 2016.

To see the resources used for a task, you can use the -l option and send it all to a file:
sacct -l > Liste_taches.txt

Set of homemade commands

You can add this line at the bottom of your .bashrc file on manitou to have access to a set
of useful commands:

source /prg/bashrc/commands.sh

After the modification, you must close all terminal windows that are actually open. After
that, re-open the windows and type this command to see what are the name of the new useful
command:

useful_ibis_commands

Comments are closed.