Lost Particles in Geometry

Report all good and bad behavior here
jesse.johns
Posts: 114
Joined: Wed Apr 20, 2011 5:19 am

Lost Particles in Geometry

Post by jesse.johns » Fri Feb 05, 2016 11:11 pm

Hello,

I am hoping to get some when with this cryptic geometry error:

Code: Select all

Warning message from function MoveST:

Possible overlap at (-NAN, -NAN, -NAN)

***** Mon Feb  1 13:01:52 2016 (seed = 1454357279, MPI task = 0, OMP thread = 18)

Tracking error: material pointer lost

(x,y,z)  =         -NAN         -NAN         -NAN
(u,v,w)  =         -NAN         -NAN         -NAN

Checking geometry errors (this may fail)...

Geometry error: overlap.

Geometry structure at last position is printed below.
Some levels may not be printed correctly. Look for the
first level that fails: undefined region, suspicious
local coordinates, etc., and check the corresponding
geometry definition in the input.

Level 0 / universe 0 :
Cell universe / overlap or undefined cell
(x,y,z)  =         -NAN         -NAN         -NAN
(u,v,w)  =         -NAN         -NAN         -NAN

Simulation aborted.
It seems more prevalent the more processors that I am using, but I don't know if that's just because I can't wait around for 1 process to finish up in 4 months. I am using 28 OpenMP processes, so maybe I am seeing some errors from overlapping calls to the shared memory?

I am using version 2.1.24.

Thanks,

Jesse

Ville Valtavirta
Posts: 457
Joined: Fri Sep 07, 2012 1:43 pm
Security question 1: No
Security question 2: 92

Re: Lost Particles in Geometry

Post by Ville Valtavirta » Wed Feb 10, 2016 4:06 pm

Hi Jesse,

the location and velocity of the neutron turning into NaNs certainly is a problem. Is your simulation using any reflective or periodic boundary conditions? Is it a criticality source or a time dependent simulation?

Based on the error message it seems likely that the position and the velocity are contaminated by NaN at some point of the tracking loop and are caught only at MoveST. There are not so many routines that modify both the position and the velocity (or do operations that might cross contaminate them), the major ones include:

-MoveST and MoveDT (which should not modify the velocity, however)
-TimeCutoff (should only do something when time cutoff is specified with "set tcut" or by setting up a dynamic simulation)
-BoundaryConditions (does do arithmetic operations between location and velocity)
-Collision (should not modify the position of the particle)

Have you tried to recompile with the -DDEBUG switch and rerun the simulations? There are many value checks in the source code that are only compiled with the DEBUG switch and they might help you catch the problem closer to it's creation.

As for the effect of OpenMP-threading, that might have something to do with the problem, but I've routinely run simulations with 32 OpenMP threads and never come up with this problem. It might, of course be a combination of the OpenMP threading and your specific simulation.

Can you send us your input?

User avatar
Jaakko Leppänen
Site Admin
Posts: 2377
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Lost Particles in Geometry

Post by Jaakko Leppänen » Wed Feb 10, 2016 4:34 pm

The energy and direction cosines should be checked every time they are changed when the source code is compiled with the -DDEBUG option, so like Ville said, running the calculation in debug mode should tell you where the problem first apperar. The NaN:s can get to the coordinates from the direction cosines when the position is updated.

Are you using any transformations or symmetry options in the calculation?
- Jaakko

jesse.johns
Posts: 114
Joined: Wed Apr 20, 2011 5:19 am

Re: Lost Particles in Geometry

Post by jesse.johns » Thu Feb 11, 2016 9:49 pm

Hi Ville and Jaako,

Thank you for the reply.

The boundary condition is a black condition and this is a normal, steady-state criticality simulation.

I do have a run going with -DDEBUG, but I'm running with 1.5e9 histories, trying to converge reaction rates on a spatial mesh. So, that will take some time.

I am using a transformation for the control rod banks - just a vertical transition. I'll try manually modifying the surfaces to handle that transformation in the mean time as well to see if that cleans up the problem.

I can't share this input file.

And I should of also mentioned that this occurs only on some inputs (due to the random number - no other modifications besides changing some nuclide inventories between runs) - others run to completion without issue.

Best,

jesse.johns
Posts: 114
Joined: Wed Apr 20, 2011 5:19 am

Re: Lost Particles in Geometry

Post by jesse.johns » Fri Feb 12, 2016 8:06 pm

Results with DEBUG mode:

Code: Select all

***** Fri Feb 12 05:18:47 2016 (seed = 1455229631, MPI task = 0, OMP thread = 14)

Fatal error in function AziRot:

Invalid value -NAN of parameter "final direction cosines"

NOTE: This value check was performed because the code was compiled in the
      debug mode and it may or may not be an indication of an actual problem.
      The debugger mode can be switched off by recompiling the source code
      without the -DDEBUG option (see Makefile).

Simulation aborted.
It appears the cosines calculation is a concern here.

I haven't had the opportunity to remove the transformations, but I'll get to that later today and run over the weekend. Also, I didn't answer the question about symmetry - there's no symmetry in this problem.

User avatar
Jaakko Leppänen
Site Admin
Posts: 2377
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Lost Particles in Geometry

Post by Jaakko Leppänen » Sat Feb 13, 2016 8:33 pm

It could be a physics, rather than geometry problem. The AziRot() function is probably getting a NaN nubar passed as an input variable. This is one of the values that should be checked before passing it on, but looking at the subroutines it seems that this is not always the case. Could you try adding value check:

Code: Select all

CheckValue(FUNCTION_NAME, "mu", "", mu, -1.01, 1.01);
in functions:

elasticscattering.c
fission.c
inelasticscattering.c
levelscattering.c
otfsabscattering.c
sabscattering.c
samplesrcpoint.c
targetvelocity.c

before the call to AziRot() to see where the bad value comes from?

This is a neutron transport problem, right?
- Jaakko

jesse.johns
Posts: 114
Joined: Wed Apr 20, 2011 5:19 am

Re: Lost Particles in Geometry

Post by jesse.johns » Mon Feb 15, 2016 8:01 pm

Hi Jaakko,

I made the changes, but the following functions:

elasticscattering.c
levelscattering.c
targetvelocity.c

did not have "mu" defined, so I commented them out for now. I'm not sure if just adding a static "double mu" would cause any issues.

This is a neutron transport problem and I just put them on the cluster to run. It's pretty busy, so it might take some time.

User avatar
Jaakko Leppänen
Site Admin
Posts: 2377
Joined: Thu Mar 18, 2010 10:43 pm
Security question 2: 0
Location: Espoo, Finland
Contact:

Re: Lost Particles in Geometry

Post by Jaakko Leppänen » Tue Feb 16, 2016 12:59 pm

Sorry... For elasticscattering.c and levelscattering.c the variable is "muc", and in targetvelocity.c it is "c". The checks should be placed right before the call to AziRot().
- Jaakko

jesse.johns
Posts: 114
Joined: Wed Apr 20, 2011 5:19 am

Re: Lost Particles in Geometry

Post by jesse.johns » Sat Feb 20, 2016 12:09 am

None of the DEBUG cases have crashed yet - now that we're trying to obverse the event, it doesn't happen. However, in an alternate simulation, with the same geometry, I got a little bit more information from the crash:

Code: Select all

***** Fri Feb 19 11:10:09 2016 (seed = 1455903186)
Warning message from function MoveST:

Possible overlap at (-NAN, -NAN, -NAN)


***** Fri Feb 19 11:10:11 2016 (seed = 1455903186, MPI task = 0, OMP thread = 13)

Tracking error: infinite loop (after 1000000 cycles)

Particle type : neutron

Collision = 650602707
(x,y,z)   =         -NAN         -NAN         -NAN
(u,v,w)   =         -NAN         -NAN         -NAN
E         =  5.21943E-09
material  =  void / undefined
totxs     =  0.00000E+00
majorant  =  3.68526E+01

Checking geometry errors (this may fail)...

Geometry error: overlap.

Geometry structure at last position is printed below.

Geometry error: overlap.

Geometry structure at last position is printed below.
Some levels may not be printed correctly. Look for the
first level that fails: undefined region, suspicious
local coordinates, etc., and check the corresponding
geometry definition in the input.

Level 0 / universe 0 :
Cell universe / overlap or undefined cell
(x,y,z)  =         -NAN         -NAN         -NAN
(u,v,w)  =         -NAN         -NAN         -NAN

Simulation aborted.

jesse.johns
Posts: 114
Joined: Wed Apr 20, 2011 5:19 am

Re: Lost Particles in Geometry

Post by jesse.johns » Mon Feb 22, 2016 6:49 pm

Alright, with the code changes, the cases failed with the following error which is the same as before:

Code: Select all

***** Fri Feb 19 18:11:29 2016 (seed = 1455659194, MPI task = 0, OMP thread = 17)

Fatal error in function AziRot:

Invalid value -NAN of parameter "final direction cosines"

NOTE: This value check was performed because the code was compiled in the
      debug mode and it may or may not be an indication of an actual problem.
      The debugger mode can be switched off by recompiling the source code
      without the -DDEBUG option (see Makefile).

Simulation aborted.

Post Reply