Compiling with MPI - runs not recognized

Compiler issues (Intel compiler, mpicc, etc.), data libraries and everything needed to get Serpent running is discussed here
Post Reply
scorpadorp
Posts: 8
Joined: Fri Feb 21, 2020 11:31 pm
Security question 1: No
Security question 2: 7

Compiling with MPI - runs not recognized

Post by scorpadorp » Thu May 13, 2021 9:14 pm

I compiled Serpent 2 with the makefile flags:
CFLAGS = -Wall -ansi -ffast-math -O3 -Wunused
CFLAGS += -DOPEN_MP
CFLAGS += -fopenmp
CFLAGS += -Wno-unused-but-set-variable
CFLAGS += -pedantic
CFLAGS += -fpic
LDFLAGS += -fopenmp
LDFLAGS = -lm
CC = mpicc
CFLAGS += -DMPI)

It compiles fine, but when I submit a job using 1 node (with 44 processors):
-mpi 44 input.inp

After a bit I get as the job output file which only says:

"MPI parallel calculation not available in compilation."

Any ideas why? And how to fix it?

Ana Jambrina
Posts: 685
Joined: Tue May 26, 2020 5:32 pm
Security question 1: No
Security question 2: 7

Re: Compiling with MPI - runs not recognized

Post by Ana Jambrina » Thu May 13, 2021 10:00 pm

That means that you have not compiled Serpent in MPI mode, enabling MPI parallelisation (the preprocessor variable called “MPI” is not defined).
Which Serpent version are you compiling? OS/compiler? Is MPI available? MPI implementation? (some installations require defining the location of the associated libraries).

Note: there is an extra parenthesis following the DMPI flag.
- Ana

scorpadorp
Posts: 8
Joined: Fri Feb 21, 2020 11:31 pm
Security question 1: No
Security question 2: 7

Re: Compiling with MPI - runs not recognized

Post by scorpadorp » Fri May 14, 2021 4:57 pm

I got it to run fine, actually, but the problem I am having is due to memory (on the cluster). When I increase the number of source particles above a certain number, I get this error when running ion OpenMP:

Code: Select all

Input error:

Out of memory error (allocating 149.9Gb, limit 149.9Gb = 0.80 * 187.4Gb)

Now when I just ran MPI I get:

Code: Select all

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 221494 RUNNING AT current
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
I'm running with the MPI with 1 node and 44 processors, with "ss2-mpi 44 input.inp" for the flags.

Any idea?

Ana Jambrina
Posts: 685
Joined: Tue May 26, 2020 5:32 pm
Security question 1: No
Security question 2: 7

Re: Compiling with MPI - runs not recognized

Post by Ana Jambrina » Fri May 14, 2021 5:55 pm

You are reaching the maximum memory available in the system/node. You could use the ‘set memfrac’ option that defines the fraction of the total system that Serpent can use (default 0.8) to squeeze a bit more of memory.
In terms of memory demand:
- Depending on the calculation you are running you can turn-off some options (e.g. 'set gcu -1', etc. ) in order to not allocate unnecessary memory.
- Reduce the buffer size (for neutrons 'set nbuf'), for large number of neutrons per generation, the default buffer might lead to unnecessarily high memory consumption.
- Check the optimization mode, 'set opti' option, which defines the performance and memory usage (default mode = 4).
- For specific calculation modes, some additional features might help in terms of memory demand, e.g. in depletion: domain decomposition (see ‘set dd’) that splits burnable materials among MPI tasks (lowering the memory demand per node).
- Ana

scorpadorp
Posts: 8
Joined: Fri Feb 21, 2020 11:31 pm
Security question 1: No
Security question 2: 7

Re: Compiling with MPI - runs not recognized

Post by scorpadorp » Thu May 20, 2021 5:50 pm

Ana Jambrina wrote:
Fri May 14, 2021 5:55 pm
You are reaching the maximum memory available in the system/node. You could use the ‘set memfrac’ option that defines the fraction of the total system that Serpent can use (default 0.8) to squeeze a bit more of memory.
In terms of memory demand:
- Depending on the calculation you are running you can turn-off some options (e.g. 'set gcu -1', etc. ) in order to not allocate unnecessary memory.
- Reduce the buffer size (for neutrons 'set nbuf'), for large number of neutrons per generation, the default buffer might lead to unnecessarily high memory consumption.
- Check the optimization mode, 'set opti' option, which defines the performance and memory usage (default mode = 4).
- For specific calculation modes, some additional features might help in terms of memory demand, e.g. in depletion: domain decomposition (see ‘set dd’) that splits burnable materials among MPI tasks (lowering the memory demand per node).
Just wanted to say thanks and I now have several problems running smoothly.

However, I'm still confused about my MPI error. Any ideas?

Ana Jambrina
Posts: 685
Joined: Tue May 26, 2020 5:32 pm
Security question 1: No
Security question 2: 7

Re: Compiling with MPI - runs not recognized

Post by Ana Jambrina » Thu May 20, 2021 7:22 pm

I might be wrong but, it is most likely related to being running out of memory. There is no straight answer to what might be causing that. E.g. might be due to how the CPU tasks are mapped within the node, MPI library version (and installation) and compiler, etc.
You would need to traceback the error.

In general, Serpent 2 tends to perform better within in-node multi-threading than node-to-node message-passing (or single node message-passing). However, it is machine/case (operations and pattern) dependent on whether the intra- or inter-node cost would bias the performance and scalability of the simulation or lead to memory problems.

Note that reducing the number of CPUs per node might help to keep the simulation running.
- Ana

Riku Tuominen
Posts: 73
Joined: Tue Feb 27, 2018 10:52 am
Security question 1: No
Security question 2: 93

Re: Compiling with MPI - runs not recognized

Post by Riku Tuominen » Thu May 20, 2021 8:31 pm

Regarding the MPI error I'm not sure if the -mpi command line parameter works properly. I would use mpirun (or similar) or srun if using SLURM. You can check Running parallel calculations section from here.

Post Reply