• Locked Locked
  • Replies 8 replies
  • Subscribers 136 subscribers
  • Views 6523 views
  • Users 0 members are here
Options
Related
How was your experience today?
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

bad performance on 3.8 kernel

Marian Mihailescu
Note: This was originally posted on 18th June 2013 at http://forums.arm.com

Mali 400 on an exynos-based board:

with 3.0 kernel, EGL working fine, with up to 600fps in es2gears

ported drivers to 3.8 kernel, and mali acceleration working, however, the performance is roughly 50%.

I have debugged the issue at the gp job start wrapper - _mali_ukk_gp_start_job, which is called now 50% more times than on the 3.0 kernel...

Here is a comparison between the 2 kernels:

1) with SKIP_GP_JOBS and retuning the job straight away from _mali_ukk_gp_start_job, both 3.0 and 3.8 kernel results in the same number of mali_ioctl calls and the same performance - 650fps in es2gears
2) i modified es2gears to stop after 600 frames and here are my results (from bottom to top):

      GP jobs actually done - calls to "mali_gp_job_start": 299 on 3.0 kernel, 302 on 3.8 kernel
      calls to mali_group_start_gp_job (which calls mali_gp_job_start): 299 on 3.0, 302 on 3.8 kernel
      executions of mali_gp_scheduler_schedule (which calls mali_group_start_gp_job): 299 on 3.0, 302 on 3.8 kernel -- appears as "mali_gp_scheduler_schedule() {" in ftrace
      calls to mali_gp_scheduler_schedule: 0 on 3.0, 299 on 3.8 kernel -- appears as "mali_gp_scheduler_schedule();" in ftrace
     
      system calls served (mali_ioctl) : 960 on 3.0 kernel, 1373 on 3.8 kernel

results: ~600fps on 3.0 kernel, ~380fps on 3.8 kernel

So the conclusion is that the slowdown is due to a much larger number (almost double) of mali_ioctls for MALI_IOC_GP2_START_JOB.

Since I don't have the code for libMali to debug why exactly it's making so many syscalls, I hope somebody here can help me and give me an idea where to look.

A strange thing is the job numbers assigned.
In the 3.0 kernel, they are all multiples of 4, like: Mali GP scheduler: Job 2405 (0xE6581B80) queued; 2409, 2413, 2417, 2421, 2425, ...
In the 3.8 kernel, they increment either by 2, 4 or 6: 8825, 8829, 8833, 8835, 8841, 8843, 8849, 8853, ...
Parents
  • Note: This was originally posted on 15th July 2013 at http://forums.arm.com


    [/size]

    Thanks for the reply.
    I have tried both a Debian Wheezy image (http://forum.odroid.....php?f=9&t=1608) with LXDE and a Ubuntu 13.04 image with XFCE (not Linaro).
    The kernel is compiled straight from the hardkernel repository (https://github.com/h...ee/odroid-3.8.y), the drivers are in drivers/gpu/arm/ (https://github.com/h...drivers/gpu/arm) and have been integrated by the maintainer, working but having bad performance. The framebuffer driver I think it's at https://github.com/h.../videobuf2-fb.c

    I have been looking for the cause of the performance drop and debugged the drivers with ftrace, and found the issue described above: there is one GP job / ioctl started, whereas in the 3.0 kernel there are 2 GP jobs/ioctl started.

    My understanding is that in 3.0 you have:
    ioctl -> GP start job from frame register 1 -> schedule job -> submit job -> (...libMali.so binary blob...) -> send job to user -> GP start job frame register 2 -> schedule job -> submit job -> (...libMali.so binary blob...)  -> send job to user -> end ioctl -> new ioctl -> repeat

    while in 3.8 the behaviour is:
    ioctl -> GP start job from frame register 1 -> schedule job -> submit job -> (...libMali.so binary blob...)  ->send job to user -> end ioctl -> new ioctl -> repeat
    ...... (in the meantime) ioctl -> GP start job from frame register 2 -> schedule job -> slot busy -> end ioctl -> new ioctl -> repeat

    this gives 2x ioctls and 2x more locks , and scheduling for frame register 2 always results in slot busy, so you have an ioctl wasted just for putting a job in the queue. The mali code is exactly the same as before, so that's not the issue. The maintainer also thinks the UMP integration is at fault, but can't find a root cause. I was hoping for somebody here to have a better idea what exactly is causing this.

    As a side note, es2gears gives ~300fps on Ubuntu+XFCE(even worse with compositor enabled) and ~600 on Debian+LXDE. I did not look at the xorg server version, just went on with the Debian image :)




    Hi Memeka,
    Apologies for the delayed response.


    Could you run strings on mali.ko to find the API level, failing that, the revision number from libmali.so?  The issue with reduced performance could be caused by attempting to use kernel driver modules known to be incompatible with this revision of the Linux kernel.

    Kind Regards,
    Rich
Reply
  • Note: This was originally posted on 15th July 2013 at http://forums.arm.com


    [/size]

    Thanks for the reply.
    I have tried both a Debian Wheezy image (http://forum.odroid.....php?f=9&t=1608) with LXDE and a Ubuntu 13.04 image with XFCE (not Linaro).
    The kernel is compiled straight from the hardkernel repository (https://github.com/h...ee/odroid-3.8.y), the drivers are in drivers/gpu/arm/ (https://github.com/h...drivers/gpu/arm) and have been integrated by the maintainer, working but having bad performance. The framebuffer driver I think it's at https://github.com/h.../videobuf2-fb.c

    I have been looking for the cause of the performance drop and debugged the drivers with ftrace, and found the issue described above: there is one GP job / ioctl started, whereas in the 3.0 kernel there are 2 GP jobs/ioctl started.

    My understanding is that in 3.0 you have:
    ioctl -> GP start job from frame register 1 -> schedule job -> submit job -> (...libMali.so binary blob...) -> send job to user -> GP start job frame register 2 -> schedule job -> submit job -> (...libMali.so binary blob...)  -> send job to user -> end ioctl -> new ioctl -> repeat

    while in 3.8 the behaviour is:
    ioctl -> GP start job from frame register 1 -> schedule job -> submit job -> (...libMali.so binary blob...)  ->send job to user -> end ioctl -> new ioctl -> repeat
    ...... (in the meantime) ioctl -> GP start job from frame register 2 -> schedule job -> slot busy -> end ioctl -> new ioctl -> repeat

    this gives 2x ioctls and 2x more locks , and scheduling for frame register 2 always results in slot busy, so you have an ioctl wasted just for putting a job in the queue. The mali code is exactly the same as before, so that's not the issue. The maintainer also thinks the UMP integration is at fault, but can't find a root cause. I was hoping for somebody here to have a better idea what exactly is causing this.

    As a side note, es2gears gives ~300fps on Ubuntu+XFCE(even worse with compositor enabled) and ~600 on Debian+LXDE. I did not look at the xorg server version, just went on with the Debian image :)




    Hi Memeka,
    Apologies for the delayed response.


    Could you run strings on mali.ko to find the API level, failing that, the revision number from libmali.so?  The issue with reduced performance could be caused by attempting to use kernel driver modules known to be incompatible with this revision of the Linux kernel.

    Kind Regards,
    Rich
Children
No data