The nature of HPC is on the cusp of a revolution with the rapid adoption of the Python programming language. Python is a rich, object-oriented, interpreted programming language with extensive support for technical computing, data visualization, and most recently multi-process distributed computing. The language is very approachable providing a productive environment for scientists and engineers. Development time is typically a small fraction of the time required using traditional HPC languages such as C and FORTRAN. Almost more exciting, the ease of development in Python is expanding the viability of HPC to many disciplines and researchers which were previously excluded because of the high technical barriers.
Python has already proven itself as a sound solution in another technical computing revolution; that is the enabling of practicing domain scientists in software development without the need for specialized computer scientists. The start of my career in technical computing was largely focused on writing software solutions in languages like C++ to enable the experimental or modelling vision of domain specific scientists such as geophysicists and oceanographers. This process was extremely slow and error prone. Obtaining scientifically and algorithmically correct solutions would take many iterations with myriad error prone steps along the way. With the advance of tools like Matlab, IDL, and PV-Wave in the late 80s this process was greatly accelerated and simplified.
The approachability of these tools enabled domain scientists to create their own solutions in far less time and much less cost. Much of the time the interpreted language solution would be adequate but sometimes part of the interpreted code would have to be written in a compiled language such as C or Fortran. In these cases, however, the cost would still be a fraction of writing the complete solution from scratch because the developer would have a working, tested implementation, and typically only a small fraction of the application would need to be re-written.
In the late 90s Python expanded this model by providing open source solutions in addition to the commercial alternatives. A rich ecosystem of add-ons such as numpy and scipy has evolved to support that model, and vendors have emerged to provide in-depth Python technical support.
Python already has a long, successful history of use in a broad range of IT tasks. System administrators and database administrators use the language for a broad range of maintenance tasks. Python is frequently used to simplify and accelerate the development of GUI and web-based applications. Python has even been used for some time within HPC for pre-processing data before feeding it to a traditional HPC application.
So, what is ‘HPC Python’?
HPC Python is the coupling of numerical toolkits such as numpy that provides a broad range of the mathematical functions typically needed in HPC codes with a process distribution framework such as mpi4py, mpi4py. The mpi4py toolkit is essentially a wrapper around MPI enabling Python programs to be designed with a very similar architecture as would be used in C or FORTRAN.
Arm is proud to be part of this HPC revolution! We have recently added Python profiling to our industry leading cross-platform HPC source code profiler MAP.
The MAP User Interface Analyzing a Python Code
Our staff has a broad range of HPC and domain expertise that can help you get started, get the most out of our tools, and help you on the path to a successful solution. We continue to survey HPC Python users and investigate how our HPC tools can be leveraged to support this revolution.
Start with the Arm python product brief at Python Profiling with Arm MAP. This site provides in depth tutorials, how to obtain and work with the most common Python toolkits used in HPC python, and numerous webinars on using MAP in general and using MAP with Python specifically.
Learn about Python profiling in Arm MAP