Operational weather forecasting is key to informing many important decisions that are made in agriculture, marine, air transport, environment, military, financial, and other sectors. The WRF (Weather Research and Forecasting) model is a widely used US model that can simulate forecasts based on actual atmospheric conditions (observations). Real-time forecasting applications can then use these results to make informed decisions.
Over the last few years weather researchers have been investigating the use of cloud resources to run their models, with WRF [1] being a popular example. Access to such flexible resources is expected to enable more researchers the ability to perform advanced WRF simulations [2]. In this blog, we discuss running WRF on the AWS Graviton2 processor, based on Arm's Neoverse N1 cores.
The WRF model is a numerical weather prediction system designed for atmospheric research and operational forecasting applications. WRF has been developed since the late 1990s through a collaborative partnership between.
WRF features two dynamical cores.
This enables WRF to support a wide range of meteorological applications across scales from 0.001-1000kms. There are over 30,000 registered WRF users world-wide.
Releases of the WRF source code are available to download here. This blog discusses version 3.9.1.1, so that we can compare performance with existing results and use readily available data sets for reproducibility. The latter is not possible from WRF version 4.
WRF uses both MPI and OpenMP for parallelization, and also requires the following external libraries: an MPI distribution (for example, Open MPI, MPICH, and so on), HDF5, NetCDF-C NetCDF-Fortran.
Instructions how to build WRF are given on the Arm community GitLab pages. This is a straightforward process. The environment variables HDFDIR and NETCDF need to be set to the location of the HDF5 and NetCDF installations and the stanza set in the file.
WRFV3-3.9.1.1/arch/configure_new.defaults
Then execute the configure script (in dm+SM mode), followed by the build script. To facilitate a fair comparison we will use the Gnu compiler toolchain (GCC) version 9.3.
The two WRF benchmarks we use are the Continental U.S. (CONUS) 2.5km and 12km: CONUS 2.5km and CONUS 12km. CONUS 12km is a 48-hour simulation with a time step of 72 seconds at October 24, 2001. The benchmark period is for hours 25-27 starting from a restart file at the end of hour 24. CONUS 2.5km is a 9-hour simulation with a time step of 15 seconds at June 4, 2005. The benchmark period is for hours 6-9, starting from a restart file at the end of hour 6. Since CONUS 2.5km is at a much higher resolution than CONUS 12km, this is the larger case, making it much better for demonstrating processor scalability.
For performance testing, we ran CONUS 2.5km and CONUS 12km on up to 12 AWS Graviton2 HPC instances (c6g.16xlarge). Each instance has 64 vCPUs (with a vCPU corresponding to 1 CPU core), 128GiB Memory, AWS Enhanced Network running at 25Gbps and Elastic Block Store (EBS) filesystem running at 19Gbps.
To optimize single-node performance we first determined the best combination of MPI task/OMP thread to use for each test. For CONUS 12km Figure 1 shows the lowest time per simulation time-step (0.33 elapsed seconds) could be achieved by using 8 MPI tasks with 8 OMP threads.
Figure 1: Mean Time per Step for CONUS 12km
Similarly, for CONUS 2.5km, Figure 2 shows the lowest time per simulation time-step (4.97 elapsed seconds) could be achieved by using 32 MPI tasks with 2 OMP threads. This can be attributed to vectorization of the key loops helping more, whereas the (smaller) CONUS 12km test can benefit from memory-bandwidth alone.
Figure 2: Mean Time per Step for CONUS 2.5km
CONUS 2.5km is known to scale well and is therefore suitable to test performance of the AWS Enhanced Network across several nodes. Figure 3 shows results running this benchmark on up to 12 nodes with AWS Graviton2 instances. Per-node we use 32 MPI tasks with 2 OMP threads. As CONUS 2.5km is of fixed size, the number of grid points per MPI task decrease as we use more nodes. Performance of the AWS Enhanced Network enables excellent scalability, and this agrees with the almost linear scalability reported in [3].
Figure 3: Strong Scaling for CONUS 2.5km
For a brief performance comparison with other AWS instances, we also tested WRF on c5a.16xlarge (AMD EPYC 7571), c5.18xlarge (Intel® Xeon® Platinum 8175m) and c5n.18xlarge (Intel® Xeon® Platinum 8259CL). Again, using the same Gnu built software stack with the following flags for WRF.
-Ofast -march=native -fopenmp -frecursive -funroll-loops
Results for CONUS 12km (Figure 4) and CONUS 2.5km (Figure 5) show the best performance is achieved with the AWS Graviton2. This is due to both CPU capability and maturity of the Gnu compiler for Arm architectures.
Figure 4: Comparison of CONUS 12km Across AWS Instances
Figure 5: Comparison of CONUS 2.5km Across AWS Instances
We have demonstrated how easy it is to run a widely used weather forecasting application, WRF on AWS Graviton2, which is based on Arm Neoverse N1 cores. The AWS platform as a whole enables excellent scalability. By comparing performance of WRF with a selection of other AWS instances, the AWS Graviton2 shows the overall fastest time to solution.
At the time of writing, c6g instances are available at a 20% lower price than equivalently configured c5 instances. In terms of cost per simulation, for WRF this means at least 45% lower cost than the c5 instances.
[CTAToken URL = "https://www.arm.com/solutions/infrastructure" target="_blank" text="See Arm Infrastructure solutions for HPC" class ="green"]
[1] Goga, K., Parodi, A., Ruiu, P., Terzo, O., Performance Analysis of WRF Simulations in a Public Cloud and HPC environment, Volume 611, p. 384-396, 2017.
[2] Powers, G. Jordan, and others, The Weather Research and Forecasting Model: Overview, System Efforts, and Future Directions, Volume 98(8), Bull. Amer. Meteor. Soc., p. 1717–1737, 2017.
[3] https://www.amd.com/system/files/documents/wrf-and-amd-epyc-the-right-combination-for-weather-modeling.pdf
hi, I tried this conus 12 on AWS c6g.16xlarge following the arm gitlab setup: https://gitlab.com/arm-hpc/packages/-/wikis/packages/wrf-modeler. it would run into seg fault with this config with gcc. but after added -O3 to FCBASEOPTS, it can work, however, the score is 0.362, not 0.33, is there any special config to get this good score ?
Hi, can you please try with FCOPTIM=-mcpu=neoverse-n1 -Ofast -fopenmp -frecursive, with similar for FCNOOPT (but with -O0 in place of -Ofast) and FCBASEOPTS.
(From GCC10 also add -fallow-argument-mismatch -fallow-invalid-boz to these flags and to the flags for FORMAT_FIXED and FORMAT_FREE)
Thanks
Thanks Phil, I suppose that all these scores are based on gcc 9.3 as mentioned in this blog. if there is the case, I'm wondering any other config I'm missing to replicate
The config should be okay, that's as far as it can go. This test case does rely on IO during the time steps and it's possibly giving different performance than on the system I used.
OK, Thanks Phil, I‘m using ubuntu 20.04, may i know which system you are using
That's interesting, I've been using RHEL 8 based OSs. Thanks, Phil.
Hi Phil, is there a reason -L<path to Arm Performance Libraries installation> -larmpl was not included in your response? I'm testing for performance boost with ARMPL in general and wrf and I was wondering if "-lamath" is only relevant here. I presume the "-larmpl" routines contain BLAS/LAPACK and FFT.
Hi, you're correct, the "-larmpl" for the Arm Performance Libraries refers to BLAS/LAPACK and FFT. For WRF, only the "-lamath" and possibly "-lastring" are relevant (I say possibly because it can depend upon which base OS you're using and which platform, sometimes the default versions for memset and memcpy are okay. There's no particular reason why the ArmPLs were not used at the time of this post and so I'd certainly recommend trying with "-lamath". You might also like to try the Arm Compiler for Linux, (that will build by default with the routines from "-lamath").
Thanks for your prompt response! I did recompile WRF by modifying the configure script and adding the "-L<path> -lamath" to LDFLAGS_LOCAL and FCOPTIM lines. Then linked those lib locations (via LD_LIBRARY_PATH) to the compiled wrf.exe since it's dependent on it. I ran a few data points on graviton2 instances and there was no speedup/time difference if comparing to the previous compiled wrf.exe (w/o ARMPL). I did use the conus 12km model and I did read that the 2.5km model might more easily show a performance boost. So I will try the 2.5km model. Or I wonder if I missed anything else on this process.
No problem. Thanks for reporting back on the results. Unfortunately, it looks like you're not going to get any further performance with the ArmPLs. As this is an AWS Graviton2 you're using, it looks like the standard OS supplied routines are already optimized to the same level. It's always worth trying all options though.
Indeed, there are a few other things I missed. First, I only re-compiled pointing to armpls on LDFLAGS_LOCAL. I did not also point to FCOPTIM. I'll recompile it with FCOPTIM. Also, I've been running "mpirun -np #CORES wrf.exe" and no other arguments but based on another blog post of yours, even though the armpls are required when checking "ldd wrf.exe", they still need to be linked with "-lm" flag. I hope I encounter improvements here. What's your suggestion to measure wall clock time? I notice it takes about 5-10 minutes for the model to be loaded (especially the CONUS. 2.5km ), before the computation starts. I didn't find an explicit walltime/elapsed time output in any of the logs.