Hello,
We have an SoC which contains the MALI G78AE GPU. In the SoC, we have a linux image with the needed drivers for MALI and we are gathering metrics from the Performance Monitor Counters available through Streamline application.
The GPU has 4 slices, each slice with 2 shader cores. According to the document "Arm® Immortalis and Mali GPU Virtualization Guide", it is possible to change the partitioning of the GPU dynamically (or on the fly, without needing to rebuild the Linux image) through the utilization of sysfs interface. This means basically that the current configuration can be read using a "cat /sys/path/to/gpu/" or it can be written using a "echo 0x1 > /sys/path/to/gpu/", for instance.
We are interested in controlling the partitioning so that we can be sure we are exercising a single slice of the GPU and check its performance metrics. The document provides a few examples of possible modifications, but the logic to control the partitioning is not further explained. One of the examples is pasted below:
The following example shows the use of the sysfs interface to allocate slices 0, 1, 2, and 3 tothe first partition. In the example, all literal values are hex bitmasks, where each bit representsa slice or access window.
echo "0xF" > /sys/bus/platform/devices/6e0a0000.gpu_resource_group/arbiter/partitions/partition0/active_slices
This example is basic as probably the F means 1111 in binary, which asserts every slice to the first partition defined with the path ".../partitions/partition0/active_slices".
But, for instance, how could I change the partitioning of the system after if I want multiple partitions?
The system currently has only the partition0 directory, should I do a "mkdir partition1" in partitions directory and then create a new "active_slices" file? Will that work? I am not very familiarized with the sysfs interface to have a proper understanding on the topic.
Can anyone point me towards the documentation which explains this definition?
Any reply will be appreciated,
thanks, Luca.
Hi Luca,
You are correct in your analysis that "echo "0xF" > .../partition0/active_slices" assigns all 4 slices to Partition0.
To dynamically re-assign Slices to different Partitions, you must first remove Slices from any existing Partition:e.g. "echo "0x3" > .../partition0/active_slices".This then leaves 2 Slices free to assign to Partition1:"echo "0xC" > .../partition1/active_slices"
If partition1 (2, and 3) do not exist, then we need to look into your device tree.The sysfs entries are created by mali_arbiter / mali_gpu_resource_group based on the device tree entries.
Please could you share a little more about the system and device tree? e.g. is this all in a single OS, or are we using a Hypervisor?What are the DT entries for the mali-*?
Kind regards,Richard.
Hi Richard,
Thanks for replying to this providing another example.
The system I am running is an SoC that is subdivided into two different domains, called main-cluster and sub-cluster. Both domains are running embedded Linux with images that were generated through Yocto.
About the entries to the device tree, I need to know exactly what you want so that I can tell you, here is some info though:
$ ls /proc/device-tree/ grep -i "gpu"fbus_gpugpu_3d_cfg@10e10000gpu_aw@10100000gpu_aw_message@1011ffc0gpu_partition_config@10040000gpu_partition_control@10050000gpu_resource_group@100c0000
$ zcat /proc/config.gz | grep "MALI" shows the following for the system partitioning:CONFIG_MALI_DEFAULT_PTM_CONFIG="\",B:S0:P0:W0,A:S1:S2:S3:P1:W1\""
This means we should 2 partitions (P0 and P1), with P0 having one slice S0 and P1 having 3 slices, distributed in different access windows W0 and W1, and having two different AXI interfaces connected to each of them (A and B). The same configuration is shown both on main-cluster and sub-cluster.
So, let me know if you need more specific information regarding the device-tree or that is enough to help me modify dynamically the system partitioning.
Thanks for you help on the issue.
Kind regards, Luca.
Thank you for the details.
The configuration you describe splits the GPU into two isolated partitions, which appear as two independent GPUs: one on AXI-A and one on AXI-B. It is not possible to dynamically move slices / partitions between different resource groups.
In the configuration above you have: Resource Group 1 on AXI-B with Access Window 0, Partition 0, and Slice 0.Resource Group 2 on AXI-A with Access Window 1, Partition 1, and Slices 1-3.
Assuming this is a fixed configuration / constraint, I believe you can do the following: For the Resource Group with 3 Slices (sub cluster?), you can adjust the value written to active_slices in order to dynamically vary the number of slices assigned to that partition.Other than gathering metrics, I do not see this as a sensible use case, as you can't use the slices anywhere else.
To demonstrate dynamic re-partitioning, it would be better to have all resources in a single Resource Group, where you can dynamically move Slices between Partitions.Adding multiple Access Windows to a Resource Group also allows you to time slice multiple VMs on a single Partition.
edit: I missed the first comma in PTM_CONFIG. Updated the above with RG1 and RG2 rather than RG0 and RG1.
Thanks for your reply and clear explanations on the matter.
Is it possible to dynamically change the AXI interfaces for different resource groups or is this interface assigned only during compilation time of the kernel?
I also read on the ARM virtualization document (page 45 of 364, version 3.2 - Documentation – Arm Developer) that it is possible to assign (re-assign?) module parameters:
I am interested in assigning more access windows to each resource group dynamically, so I tried to add it:
$ echo ",B:S0:P0:W0:W2,A:S1:S2:S3:P1:W1:W3" > /sys/module/mali_gpu_assign/parameters/ptm_config
$ modprobe mali_gpu_power
But, when I tried to see if the windows had been assigned, I didn't see any modification.
$ cat /sys/bus/platform/devices/104b0000.gpu_resource_group/arbiter/partitions/partition0/assigned_access_windows0x1
Can you tell me if it is possible and how this is supposed to be done if possible?
edit: added question about dynamically assigning access windows.