Assessing AWS Graviton2 for running WRF

Operational weather forecasting is key to informing many important decisions that are made in agriculture, marine, air transport, environment, military, financial, and other sectors. The WRF (Weather Research and Forecasting) model is a widely used US model that can simulate forecasts based on actual atmospheric conditions (observations). Real-time forecasting applications can then use these results to make informed decisions.

Over the last few years weather researchers have been investigating the use of cloud resources to run their models, with WRF [1] being a popular example. Access to such flexible resources is expected to enable more researchers the ability to perform advanced WRF simulations [2]. In this blog, we discuss running WRF on the AWS Graviton2 processor, based on Arm's Neoverse N1 cores.

Background to WRF

The WRF model is a numerical weather prediction system designed for atmospheric research and operational forecasting applications. WRF has been developed since the late 1990s through a collaborative partnership between.

• National Center for Atmospheric Research (NCAR)
• The National Oceanic and Atmospheric Administration (NOAA)
• The U.S. Air Force (USAF)
• The Naval Research Laboratory (NRL)
• The University of Oklahoma (OU)
• The Federal Aviation Administration (FAA)

WRF features two dynamical cores.

• The ARW (Advanced Research WRF)
• The NMM (Nonhydrostatic Mesoscale Model)

This enables WRF to support a wide range of meteorological applications across scales from 0.001-1000kms. There are over 30,000 registered WRF users world-wide.

The WRF Source Code

Releases of the WRF source code are available to download here. This blog discusses version 3.9.1.1, so that we can compare performance with existing results and use readily available data sets for reproducibility. The latter is not possible from WRF version 4.

WRF uses both MPI and OpenMP for parallelization, and also requires the following external libraries: an MPI distribution (for example, Open MPI, MPICH, and so on), HDF5, NetCDF-C NetCDF-Fortran.

Building WRF

Instructions how to build WRF are given on the Arm community GitLab pages. This is a straightforward process. The environment variables HDFDIR and NETCDF need to be set to the location of the HDF5 and NetCDF installations and the stanza set in the file.

WRFV3-3.9.1.1/arch/configure_new.defaults

Then execute the configure script (in dm+SM mode), followed by the build script. To facilitate a fair comparison we will use the Gnu compiler toolchain (GCC) version 9.3.

WRF Performance Benchmarks

The two WRF benchmarks we use are the Continental U.S. (CONUS) 2.5km and 12km: CONUS 2.5km and CONUS 12km. CONUS 12km is a 48-hour simulation with a time step of 72 seconds at October 24, 2001. The benchmark period is for hours 25-27 starting from a restart file at the end of hour 24. CONUS 2.5km is a 9-hour simulation with a time step of 15 seconds at June 4, 2005. The benchmark period is for hours 6-9, starting from a restart file at the end of hour 6. Since CONUS 2.5km is at a much higher resolution than CONUS 12km, this is the larger case, making it much better for demonstrating processor scalability.

For performance testing, we ran CONUS 2.5km and CONUS 12km on up to 12 AWS Graviton2 HPC instances (c6g.16xlarge). Each instance has 64 vCPUs (with a vCPU corresponding to 1 CPU core), 128GiB Memory, AWS Enhanced Network running at 25Gbps and Elastic Block Store (EBS) filesystem running at 19Gbps.

Single Node Performance

To optimize single-node performance we first determined the best combination of MPI task/OMP thread to use for each test. For CONUS 12km Figure 1 shows the lowest time per simulation time-step (0.33 elapsed seconds) could be achieved by using 8 MPI tasks with 8 OMP threads.

Figure 1: Mean Time per Step for CONUS 12km

Similarly, for CONUS 2.5km, Figure 2 shows the lowest time per simulation time-step (4.97 elapsed seconds) could be achieved by using 32 MPI tasks with 2 OMP threads. This can be attributed to vectorization of the key loops helping more, whereas the (smaller) CONUS 12km test can benefit from memory-bandwidth alone.

 Figure 2: Mean Time per Step for CONUS 2.5km

Multi Node Performance

CONUS 2.5km is known to scale well and is therefore suitable to test performance of the AWS Enhanced Network across several nodes. Figure 3 shows results running this benchmark on up to 12 nodes with AWS Graviton2 instances. Per-node we use 32 MPI tasks with 2 OMP threads. As CONUS 2.5km is of fixed size, the number of grid points per MPI task decrease as we use more nodes. Performance of the AWS Enhanced Network enables excellent scalability, and this agrees with the almost linear scalability reported in [3].

Figure 3: Strong Scaling for CONUS 2.5km

Performance Comparison

For a brief performance comparison with other AWS instances, we also tested WRF on c5a.16xlarge (AMD EPYC 7571), c5.18xlarge (Intel® Xeon® Platinum 8175m) and c5n.18xlarge (Intel® Xeon® Platinum 8259CL). Again, using the same Gnu built software stack with the following flags for WRF.

-Ofast -march=native -fopenmp -frecursive -funroll-loops  

Results for CONUS 12km (Figure 4) and CONUS 2.5km (Figure 5) show the best performance is achieved with the AWS Graviton2. This is due to both CPU capability and maturity of the Gnu compiler for Arm architectures.

Figure 4: Comparison of CONUS 12km Across AWS Instances

Figure 5: Comparison of CONUS 2.5km Across AWS Instances

Summary

We have demonstrated how easy it is to run a widely used weather forecasting application, WRF on AWS Graviton2, which is based on Arm Neoverse N1 cores. The AWS platform as a whole enables excellent scalability. By comparing performance of WRF with a selection of other AWS instances, the AWS Graviton2 shows the overall fastest time to solution.

At the time of writing, c6g instances are available at a 20% lower price than equivalently configured c5 instances. In terms of cost per simulation, for WRF this means at least 45% lower cost than the c5 instances.

[CTAToken URL = "https://www.arm.com/solutions/infrastructure" target="_blank" text="See Arm Infrastructure solutions for HPC" class ="green"]

References

[1] Goga, K., Parodi, A., Ruiu, P., Terzo, O., Performance Analysis of WRF Simulations in a Public Cloud and HPC environment, Volume 611, p. 384-396, 2017.

[2] Powers, G. Jordan, and others, The Weather Research and Forecasting Model: Overview, System Efforts, and Future Directions, Volume 98(8), Bull. Amer. Meteor. Soc., p. 1717–1737, 2017.

[3] https://www.amd.com/system/files/documents/wrf-and-amd-epyc-the-right-combination-for-weather-modeling.pdf