OpenCL support for Mali-T628 MP6 on Arndale Octa?

Hi Bramv,

At the risk of beating a dead horse, by my math the peak throughput of the Mali T628 MP6 in the Arndale Octa is:

17 FLOPS * 2 ALUs per core * 6 cores * 0.533Ghz (i think this is the max frequency for this board?) = 108.732 GFLOPS. But, as those cores are distributed across 2 core groups (4+2) simply using the first device available will mean you're only running on 4 cores, so that comes down to 72.488GFLOPS. That's the peak possible throughput of the A pipes, if you were to just keep pumping instructions through it which perfectly exercise all the functional units. In the real world, where kernels tend not to map perfectly to the hardware, tend to include some LS activity, tends to try and load stuff not in the cache, bandwidth is finite etc, it is not expected that you will actually achieve this in real world use cases. Asking for peak flops is the wrong question if you're really interested in peak real world performance.

I am seeing 33.27 and 33.17 SP GFLOPS for float2 and float4 respectively, or 45% of max theoretical peak (this is very good, considering its probably just using the VMUL and VADD units, havent looked into it yet), without looking at the actual code of the benchmark, which seems to indicate your environment has some issue. I will investigate why this benchmark reports almost identical perf numbers for float2 and float4. I also have to state that I've never seen clpeak before and am in no way vouching for its accuracy or usefulness when comparing GPU products.

Hth,

Chris

UPDATE: I've taken a look at the kernels and sure enough apart from right at the end they only touch the VMUL and VADD units. The workload is purely vector add and multiply, and absolutely does not touch the scalar or dot units, so you can take that 17 FLOPS down to 8, and our peak realistic performance becomes:

8 FLOPS * 2 ALUs * 4 cores (in one core group, which is 1 CL device) * 0.533 = 34.112 GFLOPS peak realistic performance.

We're seeing 33.27 GFLOPS, which is 97.5% of peak realistic performance, so not too shabby.

Hi Chris, I am glad that you did reply, you bring up some very interesting topics!

For me only one OpenCL platform containing only one device is found and when I request the number of compute units and clock frequency:

cl::Device device

device.getInfo<CL_DEVICE_MAX_COMPUTE_UNITS>();

device.getInfo<CL_DEVICE_MAX_CLOCK_FREQUENCY>()

I get 6 and 600 respectively. So I would assume that I am running on all six cores at 600 Mhz. Is there anything I can do to verify which core group is actually used?

Somewhere else mentioned the following formula for peak floating points performance:

16FP x 2 Vec4 x 6 Clusters x 0.600 GHz = 115.2 GFLOPS

Here the number of flops is 16 instead of 17, so I wonder where those numbers come from. Is there any reference document (by Mali?) that lists all important specs, like the number of FLOPS (including the exact number of vector, scalar and dot units per compute unit?

I ran clpeak on another architecture (Nvidia Tesla K40) and the program indeed measured peak performance (of about 4000 GFLOPS), therefore I assumed that I could also use this benchmark to measure peak performance of this Mali GPU. SInce this clearly doesn't work, I wonder how if I we could come up with some kernel that runs at +/- 100 GFLOPS. What benchmark are you using to measure those 33.27 GFLOPS?

You might be wondering why I am so determined to run the GPU at peak performance, but that is to ultimately be able to directly compare the efficiency of this presumably energy-efficient GPU to more common accelerators like the Nvidia Tesla K40, AMD Firepro W9100 and Intel Xeon Phi.

Hi Bramv,

I get 6 and 600 respectively. So I would assume that I am running on all six cores at 600 Mhz. Is there anything I can do to verify which core group is actually used?

Just to add to what Chris has said about this, this is indeed what we see with older versions of the driver.  As you are seeing, with this version it does return 6 cores, though it really will only run on 4 of them.  This particular SoC configures the 6 cores into 2 core-groups, 4 + 2.  With the driver version you are using you can only target the 4-core group with CL... it is only later drivers that allow access to both core groups, and then it will be as separate devices so you will need to manually split the work if you want to use all the cores.  (Graphics works differently and can automatically spread the workload across the entire GPU).

Hope that helps, Tim

I cannot stress enough that if you are interested in real world performance, you should move away from these synthetic benchmarks and look at actual data for proper use cases. There are some real-world oriented benchmarks out there already, which already have Mali powered devices in their results.

I totally agree that synthetic benchmarks are no proper measure for real world performance, but these benchmarks do help to determine whether the environment is set-up properly, which is clearly not (yet) the case.

Have spoken to someone from the driver team, if you're only seeing one device then it must be an old driver, it's worth asking Insignal what their roadmap is for providing updates. On current drivers you will see 2 devices, one with 4 cores and one with 2.

By now the r4p0-02rel0 drivers are available. I tried running the benchmarks using them instead of the r3p0-02rel0 drivers that I used before, however the runtime fails to create an OpenCL context.

Someone opened a topic on the Insignal forum about the latest drivers, it seems that I have to compile my own kernel using the latest kernel driver to get it working.

Hi Bram,

Yes thats correct the old kernel driver will not work with the new userspace driver, the kernel needs recompiling with the new kernel driver.

Hth,

Chris

Hi Bramv,

Are you able use all 6 cores T628?. If you are succeeded can you give complete steps to builds required  binaries.

Thanks,

Veeranna

Hi Bramv,

/dev/mali should show as soon as you insmod a correctly built Mali kernel module.  If /dev/mali is not appearing then the kernel module is not correctly built and configured for the platform.  Even if you successfully build and insert a correct kernel module for the platform userspace functionality, such as OpenCL, will not be available without Mali userspace binaries matching the version of the Mali kernel module.

Can you confirm the kernel you are trying to compile with/against and the versions of Mali kernel and userspace you have available to you, as ARM have only currently released the r4p0 userspace binaries, though r4p1 should be available through the malideveloper site in the not too distant future.

Could you also confirm the steps you have taken to configure the Mali kernel source for use with the Arndale Octa board, especially as I can see you are trying to build it out of tree?  You need to configure the kernel device tree and add platform specific configuration to the kernel to set up interrupts and memory addresses etc.  The easiest way to integrate the latest version of the kernel would be to download the Linaro kernel for Arndale Octa and extract the r4p1 kernel source INTO the kernel source, overwriting the included r4p0 kernel source as this will keep the platform integration files intact.  Looking at the kernel source you mentioned in your earlier post "II-arndale-octa" I can see that the platform integration files are present, for example in /drivers/gpu/arm/midgard/platform/5420.  You could try extracting the r4p1 kernel source into this kernel tree and building that way.

Hope this helps,

Rich

Hi Rich,

In fact, I didn't configure anything at all but just ran:

CONFIG_MALI_MIDGARD=m make

And ended up with this module for the r4p1 kernel driver:

filename:       /lib/modules/3.15.0-1-linaro-arndale-octa/kernel/drivers/gpu/arm/midgard/mali_kbase.ko

version:        r4p1-00rel0

license:        GPL

srcversion:     29BCA495EB0E26992A9C01E

alias:          of:N*T*Carm,mali-midgard*

alias:          of:N*T*Carm,malit6xx*

depends:       

vermagic:       3.15.0-1-linaro-arndale-octa SMP mod_unload ARMv7 p2v8

And this one for r4p0:

filename:       /lib/modules/3.15.0-1-linaro-arndale-octa/kernel/drivers/gpu/arm/midgard/mali_kbase.ko

version:        r4p1-00rel0

license:        GPL

srcversion:     29BCA495EB0E26992A9C01E

alias:          of:N*T*Carm,mali-midgard*

alias:          of:N*T*Carm,malit6xx*

depends:       

vermagic:       3.15.0-1-linaro-arndale-octa SMP mod_unload ARMv7 p2v8

Next I downloaded the following kernel: linux-linaro-3.15-2014.06, because it seems to be the closest match to the running kernel: 3.15.0-1-linaro-arndale-octa. I enabled the Mali options in the kernel configuration:

Enable Mali GPU support in Gator

-Mali-400MP or Mali-450MP

+Mali-T604 or Mali-T658

Path to Mali driver: drivers/gpu/arm/midgard

To do so I had to enable some timers and performance events options as well. The path points to the r4p0 driver that I copied into the kernel tree. Why does the configuration option only mention T604 and T658, and not T628?

After running "make modules" I had once again a mali_kbase.ko, but the result is the same as for the out-of-tree build: no /dev/mail.

The next step was to build and flash the full kernel. Apparently the kernel I used is not fully compatible since the -arndale-octa postfix is missing. As a result I had to modify some filenames to get the new kernel installed. Unfortunately this didn't work, the board does not boot any more. I have been looking for the proper linaro arndale-octa kernel source so that I can try it again, but haven't found it yet since it appears that only the binary hwpack and rootfs are available. Will be continued.

Sorry, I mentioned the Arndale board in my previous comment but you're using an Arndale Octa which has a Mali-T628 GPU.  You'll need to use a different version of the binary Linaro Releases, which also contain a different user-side Mali driver binary.  The kernel source and configuration I mentioned are correct for Arndale Octa.

Guillaume