Qualifying COMSOL for Nuclear-Safety-Related Procedures

COMSOL NEWS 2009

The Department of Energy’s (DOE) Oak Ridge National Laboratory (ORNL) High Flux Isotope Reactor (HFIR) is the highest-flux reactor-based source of neutrons for condensed-matter research in the US. Thermal and cold neutrons produced by HFIR are used to study physics, chemistry, materials science, engineering, and biology, as well as produce unique radioactive isotopes for industry and research. As mandated by the DOE, ORNL must perform software quality assurance (SQA) procedures with special attention to nuclear-safety-related software applicable to the HFIR. The SQA process is performed to assure that the software used to perform an analysis is producing the intended results. The DOE requires that all software in their facilities satisfy a graded approach to SQA. For software used to perform nuclear-safety-related analysis, these requirements can be more extensive.

"COMSOL has become a principal code that is being used for some very important, safety-related design and safety calculations."

James D. Freels, Ph.D., a senior research staff member in ORNL’s Research Reactors Division (RRD), is a co-developer of the SQA procedure (called SBP-1300) for implementing these DOE-mandated SQA requirements. Dr. Freels said, “The procedure was approved for use on June 6, 2001. I developed the procedure along with Max Gildner, a QA specialist and also an employee of ORNL. Since that time, there have been approximately 51 separate computer codes approved through this procedure on about 30 separate computers.” Recently, he applied SBP-1300 to COMSOL so that nuclear-safety-related calculations may be performed using the COMSOL Multiphysics software.

Figure 1: (a) This 2D model shows the fuel plate (which contains the nuclear fuel) and coolant temperature contours overlaid with the velocity contour lines. Shown is the effect on the temperature distribution caused by the effects of entrance (top), exit (bottom), and main channel coolant flow. (b) The full 3D extension to this same problem is being developed. This graph shows the ”total temperature” across a centrally located arc line at several axial locations down the fuel plate length.

Nuclear-Safety-Related Applications

COMSOL is currently undergoing review at ORNL’s HFIR facility to perform nuclear-safety-related applications. Dr. Freels said, “COMSOL has become a principal code that is being used for some very important, safety-related design and safety calculations for RRD, and hence, for DOE. In order for the calculations to be approved by the RRD, the software QA must be in place.”

Examples of the equipment at the HFIR facility that COMSOL would be applied to designated as nuclear-safety-related include safety plates that get inserted when the reactor needs to be shut down, pipes or valves that need to operate to keep the facility safe, or fuel plates that contain the nuclear fuel. “Any of this [nuclear] safety-related equipment being analyzed with a computer code has to have a corresponding safety-related calculation that goes along with it,” said Dr. Freels. “What makes a safety-related calculation different is that, not only does it go through a formal check and review process, but it also requires another independent review. The independent review process strongly encourages an alternative calculation be performed, or better yet, a test or experiment be performed to demonstrate the validity of the calculation.”

Senior nuclear safety analysts at HFIR are (left to right) Cliff Hyman, Jim Freels, Fred Griffin, and Trent Primm. Freels holds a slice of an inert HFIR inner fuel element showing several of the involute-shaped fuel plates attached together. In front is a full-scale mockup of an entire HFIR fuel element assembly.

The SQA Procedure

The verification that a specific software code is installed and producing the results expected by the developer is the main focus of the SQA procedure. “What we do in our SQA procedure is just the verification step. [We] take out representative problems from the software manuals that are typical of what we do here and run those on our computer and verify that we get the same results that are intended by the code developers,” said Dr. Freels. “For COMSOL, because it is a multiphysics code and we want to use it for many applications, it is important to try to cover a broad range of verification problems over which to document the SQA. Therefore, it takes a little longer to both create the SQA documentation and to perform the review. It takes about a man-month to perform both steps on average for this first time.”

In addition to the representative problems extracted from the COMSOL manual (the results of which were found to be identical), Dr. Freels created two additional problems to perform the SQA. These two problems were developed to verify both the finite-element convergence rate and the parallel-processing performance of COMSOL. The problems are typical of what might be analyzed at HFIR.

Figure 2: Convergence curves provide an idea of what level of noding at which to stop to achieve a given level of accuracy. The convergence rate exhibited by COMSOL follows the expected pattern of a finite-element-based code.

According to Dr. Freels, finite-element convergence is a special quality that gives the user a desired numerical accuracy with the minimal level of resources required (number of nodes, amount of memory, CPU time, etc.). The convergence curves also give the user an idea of what level of noding to stop at to achieve a given level of accuracy (see Figure 2). “The convergence rate exhibited by COMSOL follows the expected pattern of a finite-element based code. This means that COMSOL truly is a finite-element based code. Many of the popular CFD codes would not be able to do this because they are not finite-element codes,” he stated.

Figure 3: A plot of speedup (y-axis) versus the number of processors (x-axis). The slope of the line dramatically changes after four processors because there are two separate processors and four cores per processor, and the data transfer between the processors is slower than between the cores. The CPU time required to complete the job with eight processors is shown at the end of each curve. Even though the speed-up is lower, the segregated indirect solver is much faster than the other solvers.

Parallel processing can be categorized into shared-memory and distributed types. “Presently, COMSOL can utilize shared memory parallel processing. A future COMSOL version will include both shared-memory type on each cluster node, and then distributed across multiple cluster nodes,” explained Dr. Freels. Both shared-memory and distributed types of parallel processing require a certain level of overhead (wasted processor time). “A perfect parallel processing code would have a speed-up factor equal to the same number of processors. The speed-up achieved for COMSOL is shown by Figure 3.

The RRD recently acquired a new computer that has eight processors per node. “This has allowed me to generate a larger table and more meaningful results for COMSOL than what was done last fall for the Boston Conference, which used only four processors. We found that the speed-up increase reduces in moving from the first processor four cores to the second processor of four cores. Thus, for a given problem, adding more shared-memory processors will eventually not significantly increase the turnaround for the problem,” said Dr. Freels. “We are expecting a new version of COMSOL to be out this year, enabling us to use multiple nodes of our 9-node cluster and be able to use up to 40 processors at a time when running COMSOL. Such a distributed parallel processing capability will likely produce a much different efficiency. It will be very interesting to see what type of configuration is optimum for running COMSOL.”

What’s Next?

Along with the new computer, the RRD recently received a new version of COMSOL; therefore, the SQA procedure for nuclear-safety-related applications is once again being repeated. “We will qualify COMSOL for all the conceivable uses we have here in RRD. I think the opportunity for COMSOL at ORNL is tremendous. I am absolutely amazed at the breadth of applications that are being simulated with COMSOL,” said Dr. Freels. “I can foresee that all the existing COMSOL modules could be fully utilized at ORNL, and perhaps new modules developed in the future.”