Issues with coupled-transients calculations

Report all good and bad behavior here
Diego
Posts: 73
Joined: Wed Jun 01, 2011 8:49 pm

Issues with coupled-transients calculations

Post by Diego » Thu Mar 22, 2018 6:15 pm

Dear colleagues;
I'm trying to perform some simple transients calculations for a 3x3 minicore (PWR type), where the intention is to combine it with Multi-physics capabilities.
I've followed the http://serpent.vtt.fi/mediawiki/index.p ... ound_on_PN and http://serpent.vtt.fi/mediawiki/index.p ... _interface, to develop the inputs, but several issues arise:
0. I build a .prec and .live source from a previous critical configuration and I can run some simple transients w/o problem (despite some nbuf and pbuf adjustements).
1. Then, for my foreseen transient calculations (w/coupling) I include the "set comfile com.i com.o" instruction. It looks like it's working (It seems that the total nps are divided by the number of time bins, the code stops after each time binning interval and and com files are written and new interfaces are requested), but I get a warning that looks like the dynamic source has not been updated for the next time bin:

Code: Select all

Warning message from function NormalizeDynSrc:
Source is empty
In addition, for some runs, I get an error related to memory when I approach to the last time bins steps, such as:

Code: Select all

Out of memory error (allocating 302.9Gb, limit 302.9Gb = 0.80 * 378.6Gb)
Both thinks looks strange as far as the results looks similar:
transient.png
transient.png (31.29 KiB) Viewed 7398 times

2. Then I try to include an IFC type 2 in this problem, in order to be able to update the temperature and density profiles after each time bin, including the ifc files as:

Code: Select all

ifc "file_for_fuel.dat"
ifc "file_for_cool.dat"
.. etc
But in this second case the code hangs for several minutes and (almost always) asks for increase in pbuf and/or nbuf. I've tried to simplify the IFC file, even considering 1 only zone for all the core with the same temperature as for the case w/o ifc, but the problem remains. It looks like the TMS (or something) required for IFC changes the initial condition available in my saved .main and .prec files and some population is diverging.
Any ideas to solve this?
Best,
Diego

Ville Valtavirta
Posts: 549
Joined: Fri Sep 07, 2012 1:43 pm
Security question 1: No
Security question 2: 92

Re: Issues with coupled-transients calculations

Post by Ville Valtavirta » Wed Mar 28, 2018 12:20 pm

Hi Diego,

it's nice to hear you've started to move into the interesting world of coupled transients. Some comments:

** I don't think the total nps should be divided by the number of time-intervals. It should be simply divided with the number of batches and the number of MPI-tasks (if Serpent is compiled in MPI_MODE1), but tell me if it seems to be working differently.

** The warning about an empty source only means that no live neutrons survived until the time-interval boundary, but the delayed neutron precursors should be fine and the delayed neutron emission will then create the source for the next time interval. Without delayed neutrons (such as in external source mode), this would simply terminate the simulation.

** The memory allocation problem is a bit more interesting and most likely happens due to Serpent automatically allocating larger neutron or precursor stacks during simulation at line 418 of normalizedynsrc.c and/or line 550 of precursorpopcontrol.c. You can try commenting out those function calls to see if it helps, but you might run into problems with insufficient neutron/precursor buffers as they are not expanded automatically.

If the memory allocation problems seem more prevalent in calculations with coupling, its reasons might require some additional investigation.

** For the interface files, are you bringing in the same temperatures and densities that you were using in the source generation? It sounds like the new system with the interface data is quite supercritical and the simulation of the first time-interval takes a very long time because of that ending only when Serpent runs out of neutron or precursor buffer. Did you run a separate criticality source simulation with the interfaces to check for the k-eff?

You might want to run the simulation with the interfaces for, e.g. 100 ms with 10 intervals to see how the neutron population behaves.

-Ville

Diego
Posts: 73
Joined: Wed Jun 01, 2011 8:49 pm

Re: Issues with coupled-transients calculations

Post by Diego » Wed Apr 04, 2018 11:15 am

Thanks Ville for your answer!
Just a couple of comments related to your suggestions:
- Regarding nps, It looks like when you include the set com files the neutrons are followed up to the end of time bin (and somehow saved in a buffer I suppose), thus actually for each step the calculation time is around the total (w/o set com) divided by the number of bins. But, as you said, the nps are not divided by intervals.
- Regarding last point, I cannot figure out how the SIGUSR2 is interpreted by Serpent in this calculations. I have set ccmaxiter to 1 and each time a time bin is calculated the ifc files update are requested 2 times (despite overwriting com.i to 12 -SIGUSR2-). Maybe I'm missing something.
- Related to memory problem, I am experiencing some issues for Serpent2v1.30 similar to http://ttuki.vtt.fi/serpent/viewtopic.p ... b288#p8290, that do not arise for S2.1.29 (or for S2.1.30 for non-transients). If I get any hint of the origin of such problem I will post it.
- Finally, are you planning to include the option for some "initial normalization" of the saved source (live + precursors) in order to avoid problems if the transient case were you load this source is slightly different from critical ? (f.e. have the option to perform an initial criticality calculation on this transient case and adjust the source population with the keffs, in order to avoid saving many sources for problems that are very similar).
Thanks again,
Diego

Ville Valtavirta
Posts: 549
Joined: Fri Sep 07, 2012 1:43 pm
Security question 1: No
Security question 2: 92

Re: Issues with coupled-transients calculations

Post by Ville Valtavirta » Thu Apr 05, 2018 4:58 pm

Hi,

**Regarding nps, you're correct. The neutrons (and precursors) are saved to disk after each time-interval and the simulation proceeds interval-by-interval.

**Regarding the communication: After each solution for a time-interval, Serpent will send a SIGUSR1 to indicate that a new neutronics solution has been obtained and written to the interface output files. This allows the external code to update the end-of-interval coupled fields based on the new solution. Then, after all of the iterations have been finished (or after receiving SIGUSR2 from the external code), Serpent will prepare the next time-interval and send a SIGUSR2 in order to indicate to the external code that the next neutronics solution will be for the next time interval. Here the external code can actually provide a prediction for the end-of-interval fields for the next time interval (e.g. assuming constant power or constant derivative of power). If the external code doesn't want to predict the fields at end-of-interval, you don't need to update the files, simply reply with SIGUSR1.

**I'll be happy if you can catch some more information on the memory allocation problem. Those are tricky to pinpoint as they can be quite difficult to reproduce.

**We have plans to include an additional normalization for the fission neutron production and delayed neutron precursor production in transients to allow starting the simulation from a non-critical system. The idea is to simply use the (inverse of the) k-eff calculated in the initial configuration as a fixed normalization factor for the neutron and precursor production during the transient. I imagine you could then generate the neutron and precursor distributions once, and use the same sources in slightly perturbed systems by simply changing the value of initial k-eff based on criticality source simulations for the perturbed configurations.

-Ville

Ville Valtavirta
Posts: 549
Joined: Fri Sep 07, 2012 1:43 pm
Security question 1: No
Security question 2: 92

Re: Issues with coupled-transients calculations

Post by Ville Valtavirta » Fri Apr 06, 2018 2:54 pm

Actually there is already the possibility to apply a k-eff normalization to the fission neutron production (see http://serpent.vtt.fi/mediawiki/index.p ... l#set_keff), but this currently does not affect the production of delayed neutron precursors.

In order to apply the k-eff normalization also to the delayed neutron precursor production, you'll need to add some lines in precdet.c. After lines 127-143

Code: Select all

      /* Get delayed neutron nubar for reaction */
      /* so we don't have to think about beta   */

      if ((ptr = (long)RDB[rea + REACTION_PTR_DNUBAR]) > VALID_PTR)
        {
          dnu = Nubar(ptr, E0, id);
        }
      else
        {
          /* Next reaction */

          rls = NextItem(rls);

          /* Cycle loop */

          continue;
        }
add the scaling:

Code: Select all

      /* Scale nubar with external source k-eff */

      dnu = dnu*RDB[DATA_EXT_SRC_NUBAR_F];
This should allow you to simply set the k-eff of the initial system using "set keff" and the inverse of that will be used to scale both the prompt neutron and delayed neutron precursor production cross sections.

-Ville

Diego
Posts: 73
Joined: Wed Jun 01, 2011 8:49 pm

Re: Issues with coupled-transients calculations

Post by Diego » Fri Apr 13, 2018 3:36 pm

Thanks again Ville!
Diego

Diego
Posts: 73
Joined: Wed Jun 01, 2011 8:49 pm

Re: Issues with coupled-transients calculations

Post by Diego » Mon Apr 23, 2018 11:28 am

Hi Ville!
I think that I could (probably) give some hint on the memory problem for Hybrid MPI-OpenMP calculations. I compiled in debug mode for several combinations (gcc 4.7 and 7 / icc 17 + OMP 1.8/2.1) and I always get the following error for some MPI slaves at the beginning of calculation:

Code: Select all

 - MPI task         = 16
 - OpenMP thread    = 2
 - RNG parent seed  = 1524466389
 - RNG history seed = 6978237607956021973
 - RNG history idx  = 4

Fatal error in function DopMicroXS:

Value -1.337674E-15 of parameter "xs" below lower limit 0.000000E+00 

NOTE: This value check was performed because the code was compiled in the
      debug mode and it may or may not be an indication of an actual problem.
      The debugger mode can be switched off by recompiling the source code
      without the -DDEBUG option (see Makefile).

Simulation aborted.
When I compile without the -DEBUG option the calculations advence some time steps but then I just get a end of execution in the main MPI stderr that identifies that one of the MPI slaves just ended with an error status 255. The output running with /usr/bin/time is like (running as: mpirun -n 20 --bind-to core --map-by node:PE=20 -report-bindings -output-filename stdout.dat -tag-output -timestamp-output /usr/time/bin --verbose sss2 input -omp 20) for the MPI that failed look like:

Code: Select all

<stderr>:   Command exited with non-zero status 255
<stderr>:	Command being timed: "sss2 input -omp 20"
<stderr>:	User time (seconds): 11159.73
<stderr>:	System time (seconds): 1207.38
<stderr>:	Percent of CPU this job got: 1489%
<stderr>:	Elapsed (wall clock) time (h:mm:ss or m:ss): 13:50.43
<stderr>:	Average shared text size (kbytes): 0
<stderr>:	Average unshared data size (kbytes): 0
<stderr>:	Average stack size (kbytes): 0
<stderr>:	Average total size (kbytes): 0
<stderr>:	Maximum resident set size (kbytes): 1449728
<stderr>:	Average resident set size (kbytes): 0
<stderr>:	Major (requiring I/O) page faults: 34
<stderr>:	Minor (reclaiming a frame) page faults: 3255311
<stderr>:	Voluntary context switches: 35126
<stderr>:	Involuntary context switches: 17499
<stderr>:	Swaps: 0
<stderr>:	File system inputs: 14848
<stderr>:	File system outputs: 59656
<stderr>:	Socket messages sent: 0
<stderr>:	Socket messages received: 0
<stderr>:	Signals delivered: 0
<stderr>:	Page size (bytes): 4096
<stderr>:	Exit status: 255
Hope this helps,
Diego

Ville Valtavirta
Posts: 549
Joined: Fri Sep 07, 2012 1:43 pm
Security question 1: No
Security question 2: 92

Re: Issues with coupled-transients calculations

Post by Ville Valtavirta » Wed May 02, 2018 10:32 am

Thank you Diego,

can you try replacing the CheckValue-statement at line 253 of dopmicroxs.c with

Code: Select all

  if (xs < 0)
    xs = ZERO;
for future calculations and see if that takes care of some of the problems. I'm guessing the on-the-fly thermal scattering treatment makes the cross section negative in rare cases when using TMS-temperature treatment for the resonance region and this breaks something later in the code.

-Ville

Diego
Posts: 73
Joined: Wed Jun 01, 2011 8:49 pm

Re: Issues with coupled-transients calculations

Post by Diego » Tue May 22, 2018 12:58 pm

Hi Ville!
I replaced the CheckValue-statement at line 253 of dopmicroxs.c, but now the problem arises afterwards (compiling with icc17 + OMP 2.1 in debug mode):

First I get a warning message like this:

Code: Select all

   ***** Tue May 22 11:17:06 2018 (seed = 3, MPI task = 16)
  Warning message from function SampleReaction:

  TMS majorant exceeded 92238.06c E: 5.170202E-04 Er: 5.178067E-04 T 979.411765 f: 1.057362
This happens for several isotopes on each MPI ( f.e. in other MPI I get the error for 92238.06c, in a third one for the 1001.06c and so on). And then the run is aborted. But the problem is that depending on the number of MPI runs I get the error identifying the reason or not (even, for some case with low number of MPI tasks the problem seems to run ok). The error I get is like (quite strange for me):

Code: Select all

***** Tue May 22 11:50:47 2018:

 - MPI task         = 24
 - OpenMP thread    = 13
 - RNG parent seed  = 3
 - RNG history idx  = 0

Fatal error in function RandF:

Value 0.000000E+00 of parameter "seed" below lower limit 1.000000E+00 

NOTE: This value check was performed because the code was compiled in the
      debug mode and it may or may not be an indication of an actual problem.
      The debugger mode can be switched off by recompiling the source code
      without the -DDEBUG option (see Makefile).

Simulation aborted.

Any ideas?
Best,
Diego

User avatar
Jaakko Leppänen
Site Admin
Posts: 2447
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Issues with coupled-transients calculations

Post by Jaakko Leppänen » Wed May 23, 2018 1:48 pm

I think it might be useful to get some more information on the error from dopmicroxs. Could you add:

Code: Select all

  if (xs < 0.0)
    printf("%s %ld %E %E %E %E %E\n", GetText(nuc + NUCLIDE_PTR_NAME), 
	   (long)RDB[rea + REACTION_MT], E, *Er, g, T, kT)
before the CheckValue() that terminates the run?

The error from RandF is most likely related to a memory leak, and not any primary cause.
- Jaakko

Post Reply