One billion equations

Sunday 20th January 2013 is a historic date for ParaFEM. During the middle of the night, we managed to solve a problem with more than 1 billion equations, 1,023,468,720 equations to be precise!

The problem was a trivial one, a linear elastic analysis comprising a grid of 440 by 440 by 440 twenty noded hexahedral elements. However, the program did not take advantage of the regular geometry and therefore should be able to solve an arbitrarily shaped domain that is just as large.

We used HECToR to run the analysis. On 4096 MPI processes, the problem solved in less than 15 minutes. Using 8192 MPI processes, run time was 7 minutes. The compute nodes needed to be run in an "under-populated" state in order to access sufficient memory. One core on each of 4096 16-core processors was used for the 4096 process run. Two cores per processor were used in the 8192 process run.

Later in the year, we hope to report results for an arbitrary geometry and visualize the results!