Arm Community
Arm Community
  • Site
  • User
  • Site
  • Search
  • User
  • Groups
    • Arm Research
    • DesignStart
    • Education Hub
    • Innovation
    • Open Source Software and Platforms
  • Forums
    • AI and ML forum
    • Architectures and Processors forum
    • Arm Development Platforms forum
    • Arm Development Studio forum
    • Arm Virtual Hardware forum
    • Automotive forum
    • Compilers and Libraries forum
    • Graphics, Gaming, and VR forum
    • High Performance Computing (HPC) forum
    • Infrastructure Solutions forum
    • Internet of Things (IoT) forum
    • Keil forum
    • Morello Forum
    • Operating Systems forum
    • SoC Design and Simulation forum
    • 中文社区论区
  • Blogs
    • AI and ML blog
    • Announcements
    • Architectures and Processors blog
    • Automotive blog
    • Graphics, Gaming, and VR blog
    • High Performance Computing (HPC) blog
    • Infrastructure Solutions blog
    • Innovation blog
    • Internet of Things (IoT) blog
    • Mobile blog
    • Operating Systems blog
    • Research Articles
    • SoC Design and Simulation blog
    • Smart Homes
    • Tools, Software and IDEs blog
    • Works on Arm blog
    • 中文社区博客
  • Support
    • Open a support case
    • Documentation
    • Downloads
    • Training
    • Arm Approved program
    • Arm Design Reviews
  • Community Help
  • More
  • Cancel
Arm Community blogs
Arm Community blogs
High Performance Computing (HPC) blog Debugging and Profiling HPC Applications while Working Remotely
  • Blogs
  • Mentions
  • Sub-Groups
  • Tags
  • Jump...
  • Cancel
More blogs in Arm Community blogs
  • AI and ML blog

  • Announcements

  • Architectures and Processors blog

  • Automotive blog

  • Embedded blog

  • Graphics, Gaming, and VR blog

  • High Performance Computing (HPC) blog

  • Infrastructure Solutions blog

  • Internet of Things (IoT) blog

  • Operating Systems blog

  • SoC Design and Simulation blog

  • Tools, Software and IDEs blog

Tags
  • High Performance Computing (HPC)
  • Profiling
  • Arm Forge
  • Software Development Tools
  • Debugging
Actions
  • RSS
  • More
  • Cancel
Related blog posts
Related forum threads

Debugging and Profiling HPC Applications while Working Remotely

Nick Forrington
Nick Forrington
July 20, 2020

The ongoing impact of the COVID-19 pandemic means that more and more scientific research is being conducted by teams working remotely.

While remote access to compute resources is nothing new, visual tasks such as debugging and profiling can become difficult as network latencies increase, and remote graphics solutions become unresponsive.

The good news is that the Arm Forge cross-platform tools suite provides various GUI and command-line methods for remote debugging and profiling, including the following components:

  • Arm DDT
  • Arm MAP
  • Arm Performance Reports

Here we describe some of those methods that you can use to quickly get up and running.

Arm Forge remote client

The Arm Forge remote client allows you to debug and profile remote jobs, while running the GUI on your local machine. This is faster than remote-X11 (particularly for slow connections) and provides a native GUI.

The remote client is available for Windows, Mac, and Linux, and can also be used as a local viewer for collected MAP profiles.

The Arm Forge remote client will connect and authenticate using SSH, and use existing licensing from your remote compute resource, so minimal setup is required.

 Remote Client Configuration

When you are connected, Forge looks and behaves as it does when running locally, but launches jobs, browses for files, and uses the configuration found on the remote system.

Additionally, the Reverse Connect feature allows you to easily launch jobs with DDT and MAP using your usual terminal.

For example, once connecting your remote client, run the following in your remote terminal:

ddt --connect mpirun -np 24 ./a.out

Or

map --connect mpirun -np 24 ./a.out

When you execute the DDT or MAP --connect command, a connection is made to your existing remote client, and the specified mpirun command is executed to start up your program.

This provides a convenient way for the remote client to work with batch systems and avoids the need to tell DDT or MAP about any program parameters, environment variables, or module files required.

For more information on the Forge Remote Client see:

  • Download the Arm Forge remote client
  • Forge remote client setup and usage
  • Connecting to a remote system

Using Arm Forge from the command-line

While the previous section describes how to better use the Forge GUI on remote connections, sometimes working without a GUI can be preferable.

Not only is this useful when working remotely, it can be useful to drive the tools inside batch scripts, or integrate them into Continuous Integration workflows.

Debugging Offline with DDT

Arm DDT is best known as an interactive debugger. But whether an unreliable connection makes using a GUI difficult, or you are not sure if you are at your desk when your job is scheduled, offline debugging can be a very useful alternative.

Offline debugging provides the complete breadth of DDT debugging capabilities but without user interaction, and without using the GUI.

Instead, DDT generates a report when your job completes, detailed any crashes or areas of interest (specified when launching), along with relevant variables. This mode can simplify the debugging process when you are working in the easily interrupted WFH world.

For example:

ddt --offline mpirun -n 4 PROGRAM [ARGUMENTS]...

This shows an example detecting a program crash, highlighting the offending line of code, along with variables.

 DDT Offline Crash Log

With command-line arguments or session files, you can define breakpoints, watchpoints, and tracepoints to gather more extensive data.

For more information about offline debugging, see:

  • Blog: Debugging while you sleep
  • Blog: More debugging while you sleep
  • Arm Forge user guide: Offline Debugging

Collecting MAP profiles from the command line

When you are debugging, it is common to submit runs from inside a debugger. For profiling, the common approach would be to run the program offline, producing a profile file that can be inspected later. To do this, replace your usual program invocation with a MAP command such as:

map --profile mpirun -n 4 PROGRAM [ARGUMENTS]...

MAP runs without a GUI, gathering data to a .map profile file.

When you have collected your profile, you can:

  1. Connect the remote client, and browse to open it (see previous)
  2. Copy it to your local machine, and open it with the remote client
  3. Convert it to a Performance Report (see the following)

Characterise performance with Performance Reports

Arm Performance Reports is a low-overhead tool that produces one-page text and HTML reports summarizing and characterizing both scalar and MPI application performance.

These reports can be particularly useful when working remotely, because no remote GUI is required – all that is needed is a web browser.

Reports are generated by simple modifications to your launch command:

perf-report mpirun -n 4 PROGRAM [ARGUMENTS]...

Or, if you have an existing MAP file, you can generate a report from that:

perf-report profile.map

You can copy the resulting report to your local machine and open it with your web browser. Here you’ll see summary information:

 Performance Reports Summary

As well as sections displaying more detail and advice for CPU usage, MPI, I/O, Threading/OpenMP usage, memory usage, and energy usage.

 Performance Reports CPU Information

For more information about performance reports, see:

  • Arm Forge user guide
  • Interpreting performance reports
  • Arm Developer website: Debug and Profile

Summary

In this article, we have introduced potential solutions to some of the issues faced when debugging and profiling HPC code when working remotely.

Using Arm Forge, we have covered how to improve the performance of GUI solutions, by using the remote client rather than generic solutions like X11-forwarding.

We have also discussed how to bypass the GUI entirely, and use DDT, MAP, and Performance reports non-interactively from the command line, which can help with slow connections, and dealing with busy batch systems.

Request free Arm Forge trial

Anonymous
High Performance Computing (HPC) blog
  • Key workloads demonstrate how Arm servers are changing HPC

    David Lecomber
    David Lecomber
    In the blog we look at the progress made in the Arm HPC application ecosystem and give a preview of our activities at ISC'22.
    • May 24, 2022
  • Arm Compilers and Performance Libraries for HPC developers now available for free

    Ashok Bhat
    Ashok Bhat
    Arm C/C++/Fortran Compilers and Arm Performance Libraries, aimed at HPC application developers, are now available for free. You no longer need license files to use the tools.
    • May 9, 2022
  • Stoking the Fire in Arm HPC

    David Lecomber
    David Lecomber
    In this blog we look at the growth of Arm in HPC - from humble beginnings to the number one ranked supercomputer in the world
    • May 3, 2022