Operational weather forecasting is key to informing many important decisions that are made in agriculture, marine, air transport, environment, military, financial, and other sectors. The WRF (Weather Research and Forecasting) model is a widely used US model that can simulate forecasts based on actual atmospheric conditions (observations). Real-time forecasting applications can then use these results to make informed decisions.
Over the last few years weather researchers have been investigating the use of cloud resources to run their models, with WRF  being a popular example. Access to such flexible resources is expected to enable more researchers the ability to perform advanced WRF simulations . In this blog, we discuss running WRF on the AWS Graviton2 processor, based on Arm's Neoverse N1 cores.
The WRF model is a numerical weather prediction system designed for atmospheric research and operational forecasting applications. WRF has been developed since the late 1990s through a collaborative partnership between.
WRF features two dynamical cores.
This enables WRF to support a wide range of meteorological applications across scales from 0.001-1000kms. There are over 30,000 registered WRF users world-wide.
Releases of the WRF source code are available to download here. This blog discusses version 188.8.131.52, so that we can compare performance with existing results and use readily available data sets for reproducibility. The latter is not possible from WRF version 4.
WRF uses both MPI and OpenMP for parallelization, and also requires the following external libraries: an MPI distribution (for example, Open MPI, MPICH, and so on), HDF5, NetCDF-C NetCDF-Fortran.
Instructions how to build WRF are given on the Arm community GitLab pages. This is a straightforward process. The environment variables HDFDIR and NETCDF need to be set to the location of the HDF5 and NetCDF installations and the stanza set in the file.
Then execute the configure script (in dm+SM mode), followed by the build script. To facilitate a fair comparison we will use the Gnu compiler toolchain (GCC) version 9.3.
The two WRF benchmarks we use are the Continental U.S. (CONUS) 2.5km and 12km: CONUS 2.5km and CONUS 12km. CONUS 12km is a 48-hour simulation with a time step of 72 seconds at October 24, 2001. The benchmark period is for hours 25-27 starting from a restart file at the end of hour 24. CONUS 2.5km is a 9-hour simulation with a time step of 15 seconds at June 4, 2005. The benchmark period is for hours 6-9, starting from a restart file at the end of hour 6. Since CONUS 2.5km is at a much higher resolution than CONUS 12km, this is the larger case, making it much better for demonstrating processor scalability.
For performance testing, we ran CONUS 2.5km and CONUS 12km on up to 12 AWS Graviton2 HPC instances (c6g.16xlarge). Each instance has 64 vCPUs (with a vCPU corresponding to 1 CPU core), 128GiB Memory, AWS Enhanced Network running at 25Gbps and Elastic Block Store (EBS) filesystem running at 19Gbps.
To optimize single-node performance we first determined the best combination of MPI task/OMP thread to use for each test. For CONUS 12km Figure 1 shows the lowest time per simulation time-step (0.33 elapsed seconds) could be achieved by using 8 MPI tasks with 8 OMP threads.
Figure 1: Mean Time per Step for CONUS 12km
Similarly, for CONUS 2.5km, Figure 2 shows the lowest time per simulation time-step (4.97 elapsed seconds) could be achieved by using 32 MPI tasks with 2 OMP threads. This can be attributed to vectorization of the key loops helping more, whereas the (smaller) CONUS 12km test can benefit from memory-bandwidth alone.
Figure 2: Mean Time per Step for CONUS 2.5km
CONUS 2.5km is known to scale well and is therefore suitable to test performance of the AWS Enhanced Network across several nodes. Figure 3 shows results running this benchmark on up to 12 nodes with AWS Graviton2 instances. Per-node we use 32 MPI tasks with 2 OMP threads. As CONUS 2.5km is of fixed size, the number of grid points per MPI task decrease as we use more nodes. Performance of the AWS Enhanced Network enables excellent scalability, and this agrees with the almost linear scalability reported in .
Figure 3: Strong Scaling for CONUS 2.5km
For a brief performance comparison with other AWS instances, we also tested WRF on c5a.16xlarge (AMD EPYC 7571), c5.18xlarge (Intel® Xeon® Platinum 8175m) and c5n.18xlarge (Intel® Xeon® Platinum 8259CL). Again, using the same Gnu built software stack with the following flags for WRF.
-Ofast -march=native -fopenmp -frecursive -funroll-loops
Results for CONUS 12km (Figure 4) and CONUS 2.5km (Figure 5) show the best performance is achieved with the AWS Graviton2. This is due to both CPU capability and maturity of the Gnu compiler for Arm architectures.
Figure 4: Comparison of CONUS 12km Across AWS Instances
Figure 5: Comparison of CONUS 2.5km Across AWS Instances
We have demonstrated how easy it is to run a widely used weather forecasting application, WRF on AWS Graviton2, which is based on Arm Neoverse N1 cores. The AWS platform as a whole enables excellent scalability. By comparing performance of WRF with a selection of other AWS instances, the AWS Graviton2 shows the overall fastest time to solution.
At the time of writing, c6g instances are available at a 20% lower price than equivalently configured c5 instances. In terms of cost per simulation, for WRF this means at least 45% lower cost than the c5 instances.
See Arm Infrastructure solutions for HPC
 Goga, K., Parodi, A., Ruiu, P., Terzo, O., Performance Analysis of WRF Simulations in a Public Cloud and HPC environment, Volume 611, p. 384-396, 2017.
 Powers, G. Jordan, and others, The Weather Research and Forecasting Model: Overview, System Efforts, and Future Directions, Volume 98(8), Bull. Amer. Meteor. Soc., p. 1717–1737, 2017.
hi, I tried this conus 12 on AWS c6g.16xlarge following the arm gitlab setup: https://gitlab.com/arm-hpc/packages/-/wikis/packages/wrf-modeler. it would run into seg fault with this config with gcc. but after added -O3 to FCBASEOPTS, it can work, however, the score is 0.362, not 0.33, is there any special config to get this good score ?
Hi, can you please try with FCOPTIM=-mcpu=neoverse-n1 -Ofast -fopenmp -frecursive, with similar for FCNOOPT (but with -O0 in place of -Ofast) and FCBASEOPTS.
(From GCC10 also add -fallow-argument-mismatch -fallow-invalid-boz to these flags and to the flags for FORMAT_FIXED and FORMAT_FREE)
Thanks a lot for the reply, I've tried following your advise, the best score with these flags is 0.35 stably, not as good as 0.33.
The gcc version is 9.3.0, the same as the one on the arm gitlab, any other advise or config I should use to get this good score ?
Hi, thanks for the update. Can you please try with GCC10.2
The only other thing to try would be the free Arm Performance Libraries (https://developer.arm.com/tools-and-software/server-and-hpc/downloads/arm-performance-libraries). The libamath library contains better optimized versions of log, exp and pow and sometimes these can help. So LDFLAGS_LOCAL and FCOPTIM would need extra -L<path to Arm Performance Libraries installation> -lamath
Thanks Phil, I suppose that all these scores are based on gcc 9.3 as mentioned in this blog. if there is the case, I'm wondering any other config I'm missing to replicate
The config should be okay, that's as far as it can go. This test case does rely on IO during the time steps and it's possibly giving different performance than on the system I used.
OK, Thanks Phil, I‘m using ubuntu 20.04, may i know which system you are using
That's interesting, I've been using RHEL 8 based OSs. Thanks, Phil.
Hi Phil, is there a reason -L<path to Arm Performance Libraries installation> -larmpl was not included in your response? I'm testing for performance boost with ARMPL in general and wrf and I was wondering if "-lamath" is only relevant here. I presume the "-larmpl" routines contain BLAS/LAPACK and FFT.