Arm Community
Arm Community
  • Site
  • User
  • Site
  • Search
  • User
  • Groups
    • Arm Research
    • DesignStart
    • Education Hub
    • Graphics and Gaming
    • High Performance Computing
    • Innovation
    • Multimedia
    • Open Source Software and Platforms
    • Physical
    • Processors
    • Security
    • System
    • Software Tools
    • TrustZone for Armv8-M
    • 中文社区
  • Blog
    • Announcements
    • Artificial Intelligence
    • Automotive
    • Healthcare
    • HPC
    • Infrastructure
    • Innovation
    • Internet of Things
    • Machine Learning
    • Mobile
    • Smart Homes
    • Wearables
  • Forums
    • All developer forums
    • IP Product forums
    • Tool & Software forums
  • Support
    • Open a support case
    • Documentation
    • Downloads
    • Training
    • Arm Approved program
    • Arm Design Reviews
  • Community Help
  • More
  • Cancel
High Performance Computing
  • Developer Community
  • Tools and Software
  • High Performance Computing
  • Jump...
  • Cancel
High Performance Computing
HPC blog Arm demonstrates excellent performance on Ceph storage at significantly lower cost than x86
  • HPC blog
  • HPC forum
  • Server & HPC events
  • Mentions
  • Sub-Groups
  • Tags
  • Jump...
  • Cancel
  • New
More blogs in High Performance Computing
  • HPC blog

Tags
  • storage
  • node
  • Neoverse
Actions
  • RSS
  • More
  • Cancel
Related blog posts
Related forum threads

Arm demonstrates excellent performance on Ceph storage at significantly lower cost than x86

Steve Demski
Steve Demski
July 27, 2020

 This is a graphic to show CEPH Storage collaboration.

A key trend in modern data centers is implementation of software defined storage, like the open-source software distribution, Ceph. The open source software community was an early adopter in moving workloads to Arm Neoverse. All types of application users are experiencing performance benefits and cost savings by switching to Arm-based platforms, like Lenovo platforms that are based on Ampere Computing CPUs.

Delivering better-than-x86 performance at lower TCO is a value proposition that we set out to establish with Arm Neoverse. Our news today is no exception. Last November, a group within SUSE submitted the first Ceph-based result for a storage and metadata benchmark called the IO500 10 Node Challenge, achieving a score of 12.43 using a Xeon Gold 6142-based cluster. Through a six-way collaboration between Arm, Ampere, the same group at SUSE, Mellanox (Nvidia), Micron, and Broadcom, we thought we could do better.  And today we're excited to announce that an Arm-based cluster, using Ampere eMAG CPUs, achieved a Ceph-based score of 15.61, consuming far less power and at considerably lower price, on the IO500 10 Node Challenge benchmark.

If you are new to Ceph, here’s some background. Ceph enables deployment of distributed storage systems that are designed for scalability, reliability, and performance. A Ceph cluster can be run on commodity servers over a common network like Ethernet. Ceph clusters scale up well to thousands of servers and into the petabyte range. 

Although achieving this result was a group effort, Arm contributed with multiple years of incremental improvements on Ceph and other related open-source software projects. These contributions include:
• 100+ upstreamed patches to improve Ceph storage ecosystem on Arm servers, covering multiple open source communities includes Ceph, Ceph-CSI, SPDK, DPDK, ISA-L, and OpenStack.
• Boosted Ceph performance on Arm with optimizations in its common routines like string handling, dcache hashing, and CRC32.
• Added 64KB kernel page support to Ceph. This support is a unique feature on Arm, which enhanced SPDK integration with considerable performance uplift achieved.

The IO500 benchmark was established in 2017 to compliment the TOP500 benchmark (recently topped by the Arm-based Fugaku supercomputer) but with a focus on storage sub-system performance.  While the IO500 test aims for maximum performance from an unbounded number of clients and servers, the ten-node challenge limits clients to ten. This challenge focuses on achieving the best storage throughput and metadata performance from a smaller set of systems. This challenge also demonstrates that, if the performance of Ceph on Arm is good enough for HPC workloads, it should also be suitable for a large portion of the enterprise storage market. 

Ceph may not be the first name you think of when it comes to high-performance computing storage filesystems. However, Ceph is seeing broader consideration and adoption in HPC but also in media, telecommunications, cloud computing, and elsewhere. Similarly, Ampere Computing may not be a household name (yet) but on the IO500 benchmark 10 Node Challenge, Ampere Computing’s eMAG CPU has shown that it can offer more performance on a Ceph-based cluster (see Figure 1) while offering significant CapEx savings (see Figure 2) over last November's Xeon-based alternative1.

 Figure 1:IO500 10 Node Challenge Score Figure 2: IO500 Node Challenge Cluster Price
The test cluster setup that we used for this benchmark includes:
• Memory and NVMe-based SSDs from Micron

• An NVMe storage controller from Broadcom
• Dual 100GbE networking from Mellanox (Nvidia)

We chose NVMe-based storage for this test because that is what more and more customers are choosing. Although 100GbE networking might still be on the leading edge, there is broad acknowledgement that faster networking will be required to keep up with an ever increasing deluge of data.

So what did we learn? First, out-of-the-box Ceph runs well on the Ampere eMAG CPU, showing a 26% performance improvement over the Intel Xeon Gold 6142 comparison cluster. It also consumes far less power under test. The Arm-based cluster consumed, at most, 152 Watts per server. This is more than 50% lower than the 310 W that SUSE observed on the Xeon-based servers. This is important for storage environments, because reduced ambient temperatures can greatly improve the reliability of HDD and SSD-based storage devices. And have I mentioned the potential 40% CapEx savings? 

We are grateful to all of the partners involved in achieving this result. We would like to send a special thanks to the team at SUSE who maintained the cluster and performed all of the testing. You can read more details about setting up, tuning and running the cluster in SUSE's CephFS blog.

Learn about Arm Neoverse

Cluster configuration details:

Role Qty Component Notes
Admin, Monitor, Gateway, MDS Nodes 6 Ampere 1U Servers (Lenovo HR330A) • 1x Ampere eMAG 8180 32Core 3.3GHz
• 32GB DRAM 4x8 DIMM 2667
• 2x Micron 7300 Pro NVMe M.2 480GB
• 1x Mellanox  MCX653105A-HDAT Dual Port 100GbE
OSD Nodes 10 Ampere 2U Servers (Lenovo HR350A) • 1x Ampere eMAG 8180 32Core 3.3GHz
• 128GB 8x16 DIMM 2667
• 2x Micron 240GB 5300 NVMe M.2
• 4x Micron 3.84TB 7300 PRO NVMe U.2 SSD
• 1x BRCM 9500-16i HBA
• 1x Mellanox  MCX653105A-HDAT Dual Port 100GbE
Network Switch 1 Mellanox Spectrum 2 100G Switch

 

References:

1. https://www.vi4io.org/assets/io500/2019-11/2019-11-07-SUSESUSEEnterpriseStorage-CephFS-10-8.zip

2. Cluster price is estimated “street pricing” for both clusters obtained from public sources such as CDW.com, Lenovo.com and Newegg.com during the month of July 2020.

Anonymous
HPC blog
  • Designing Arm Cortex-M55 CPU on Arm Neoverse powered AWS Graviton2 Processors

    Tim Thornton
    Tim Thornton
    In this blog, read how Arm made the transition from on-prem EDA to running EDA in the Cloud on AWS Graviton2.
    • December 17, 2020
  • Ocean Modeling with HYCOM on AWS Graviton2

    Lucas Pettey
    Lucas Pettey
    AWS Graviton2 based c6g instances offer the fastest resolution time on HYCOM.
    • December 10, 2020
  • Trends to Watch in HPC

    Brent Gorda
    Brent Gorda
    In this blog, Brent Gorda discusses a few key trends that will impact the future of high-performance computing.
    • November 4, 2020