Memory limitation: 16 GB

Report any suspected bugs and unexpected behavior here
User avatar
Tuomas Viitanen
Posts: 65
Joined: Thu Mar 25, 2010 1:50 pm
Security question 2: 0
Location: VTT Technical Research Centre of Finland, Espoo

Memory limitation: 16 GB

Post by Tuomas Viitanen » Thu Jun 17, 2010 9:27 am

Hi!

I noticed an undocumented property concerning memory allocation and pointers in Serpent. As I made some Serpent calculations that required huge amounts of memory (because of multiple temperatures appearing in problem geometry) Serpent suddenly crashed and gave an error message:

"ReallocMem: Memory allocation failed."

This happens, because all the array index variables in Serpent are of type int and, hence, can only contain numbers up to 2^31-1. If one of the three main data arrays of Serpent (DATA, RESULTS, ACE) exceeds the size of (2^31-1)*8 bytes = 16 GB (8 bytes being the size of double in bytes), problems emerge.

The first appearance of this problem is always in function ReallocMem(..), which stores the new size of array into an int variable. The variable becomes negative as the 2^31-1 limit is reached and, consequently, the memory reallocation fails resulting in the previously mentioned error message.

Tuomas

User avatar
Jaakko Leppänen
Site Admin
Posts: 2447
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Memory limitation: 16 GB

Post by Jaakko Leppänen » Thu Jun 17, 2010 10:12 am

This is a serious problem, thanks for pointing it out!

Fixing the bug is trivial, but it will take time because all local pointer variables used in over 200 subroutines are of type int, and they need to be changed to long int. Otherwise the extra bits are lost to typecasting.

Do you know if there is a compiler option in gcc that extends all ints to long? That could be a quick fix to the problem.

I'll add the bug in the list of known issues.
- Jaakko

User avatar
Jaakko Leppänen
Site Admin
Posts: 2447
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Memory limitation: 16 GB

Post by Jaakko Leppänen » Thu Jul 01, 2010 3:45 pm

The problem was corrected in update 1.1.12 by changing the 32-bit int variables to 64-bit long int. This required a lot of modifications in the subroutines, so be aware that some problems may still persist when the code allocates more than 64 Gb of memory.
- Jaakko

ssmh
Posts: 6
Joined: Fri Apr 23, 2010 5:31 pm

Re: Memory limitation: 16 GB

Post by ssmh » Thu Aug 19, 2010 3:39 pm

Jaakko Leppänen wrote:The problem was corrected in update 1.1.12 by changing the 32-bit int variables to 64-bit long int. This required a lot of modifications in the subroutines, so be aware that some problems may still persist when the code allocates more than 64 Gb of memory.
Hello,

The problem of Memory allocation failed is stile not corrected.
It exactly says:

Fatal error in function ReallocMem:

Memory allocation failed

Sideeg

User avatar
Jaakko Leppänen
Site Admin
Posts: 2447
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Memory limitation: 16 GB

Post by Jaakko Leppänen » Thu Aug 19, 2010 3:42 pm

ssmh wrote:
Jaakko Leppänen wrote:The problem was corrected in update 1.1.12 by changing the 32-bit int variables to 64-bit long int. This required a lot of modifications in the subroutines, so be aware that some problems may still persist when the code allocates more than 64 Gb of memory.
Hello,

The problem of Memory allocation failed is stile not corrected.
It exactly says:

Fatal error in function ReallocMem:

Memory allocation failed
This may not be the same problem. You sometimes get the same error if the code tries to allocate more memory than what is available in the system. Could you provide more information on the calculation case and the computer in which you running the calculation.
- Jaakko

ssmh
Posts: 6
Joined: Fri Apr 23, 2010 5:31 pm

Re: Memory limitation: 16 GB

Post by ssmh » Thu Aug 19, 2010 4:39 pm

I'm running a complicated BWR pin cell model. It consist of fuel, clad and coolant.
It was divided into twenty five segments each has different void fraction and hence a significant drop in the water density with elevation in the core.
I'm using the detector capability to calculate: total flux, fission reaction rate, capture reaction rate, total fission and the total fission neutron production in each segment.
The error message appear when you ask the code to plot and mesh!
And of course same thing for assembly case.
My laptop is two gaga of ram, 1.66 GHz.

Sideeg

User avatar
Jaakko Leppänen
Site Admin
Posts: 2447
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Memory limitation: 16 GB

Post by Jaakko Leppänen » Thu Aug 19, 2010 4:52 pm

ssmh wrote:I'm running a complicated BWR pin cell model. It consist of fuel, clad and coolant.
It was divided into twenty five segments each has different void fraction and hence a significant drop in the water density with elevation in the core.
I'm using the detector capability to calculate: total flux, fission reaction rate, capture reaction rate, total fission and the total fission neutron production in each segment.
The error message appear when you ask the code to plot and mesh!
And of course same thing for assembly case.
My laptop is two gaga of ram, 1.66 GHz.

Sideeg
Well, with 2 Gb the pointers shouldn't exceed the maximum value for an integer variable, so the problem must be related to something else.

25 burnable materials sounds quite a lot for a 2 Gb machine. Are you using any grid thinning (set egrid).

Can you monitor your memory usage before the calculation stops (with top or similar)? Does the computer start swapping before the end? At which point does the calculation stop, what is the last message?
- Jaakko

ssmh
Posts: 6
Joined: Fri Apr 23, 2010 5:31 pm

Re: Memory limitation: 16 GB

Post by ssmh » Thu Aug 19, 2010 6:02 pm

I do not use any grid thinning.
And sorry I do not know how to monitor memory usage before calculation stops, what you mean?
and also I'm not sure if the computer started swapping or no!!
Anyway finalizing XS data...OK, but the calculation stops when preparing statistics started.
Tomorrow I will post the last message because I'm not able to run the code now.

Sideeg

ssmh
Posts: 6
Joined: Fri Apr 23, 2010 5:31 pm

Re: Memory limitation: 16 GB

Post by ssmh » Fri Aug 20, 2010 12:42 pm

Hello,

The last message after the calculation stops is:


Finalizing XS data...
OK.

Preparing statistics...

***** Fri Aug 20 11:33:29 2010 (seed = 1282295306)

Fatal error in function ReallocMem:

Memory allocation failed

Segmentation fault

could you please provide me what the problem !?!

Sideeg

User avatar
Jaakko Leppänen
Site Admin
Posts: 2447
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Memory limitation: 16 GB

Post by Jaakko Leppänen » Fri Aug 20, 2010 2:21 pm

The problem is most likely related to insufficient memory. Serpent reads the available memory size from /proc/meminfo, and stops the calculation if it takes more than 80% of the total size. This is to avoid the system going into swapping and hanging completely. In some systems (OSX?) the info is not available, the check is not performed, and the calculation may crash when memory allocation fails.

First thing you could do is to set the available memory size manually in the input:

set availmem 2.0

and see if that changes anyhing. If the problem is there, you should get a different error message ("Out of memory error, limit set to 2.0 Gb")

To reduce the memory demand, you could try using grid thinning ("set egrid 1E-4") or so, but as I said, 25 burnable materials may simply be too much for a 2Gb machine. The next update will have options to significantly reduce the memory demand per burnable material (wih the cost of CPU time) and with those options you should be able to run the calculation.
- Jaakko

Post Reply