Hi,
We are working with Mali-400 driver r3p2-01rel0 on Exynos4412, running gnome-shell under Linux/X11.
base: BUILD=RELEASE ARCH=arch_011_udd PLATFORM=default_7a TRACE=0 THREAD= GEOM= CORES=MALI400 USING_MALI400=1 TARGET_CORE_REVISION=0x0101 TOPLEVEL_REPO_URL=Linux-r3p2-01rel0 REVISION=Linux-r3p2-01rel0 CHANGED_REVISION=Linux-r3p2-01rel0 REPO_URL=Linux-r3p2-01rel0 BUILD_DATE=Fri Jan 11 14:58:31 UTC 2013 CHANGE_DATE=Linux-r3p2-01rel0 TARGET_TOOLCHAIN=gcc HOST_TOOLCHAIN=gcc TARGET_TOOLCHAIN_VERSION=gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) HOST_TOOLCHAIN_VERSION=gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) TARGET_SYSTEM=gcc-arm-linux-gnueabihf HOST_SYSTEM=gcc-arm-linux-gnueabihf CPPFLAGS= CUSTOMER=internal VARIANT=mali400-r3p2-gles11-gles20-linux-ump-x11 HOSTLIB=direct INSTRUMENTED=FALSE USING_MRI=FALSE MALI_TEST_API= UDD_OS=linux
We are facing a problem with gnome-shell that is easy to reproduce: the UI often hangs while minimizing windows or opening new windows. I have traced this down to a deadlock.
At the point of hang, one thread is waiting for a reply from X:
#0 0xb656ed30 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb587dfa2 in poll (__timeout=-1, __nfds=1, __fds=0xb40fe988)
at /usr/include/arm-linux-gnueabihf/bits/poll2.h:46
#2 _xcb_conn_wait (c=c@entry=0x17ae08, cond=cond@entry=0xb40fe9d8,
vector=0x0, count=0x0) at ../../src/xcb_conn.c:400
#3 0xb587edb0 in wait_for_reply (c=c@entry=0x17ae08,
request=, e=e@entry=0xb40fea7c) at ../../src/xcb_in.c:395
#4 0xb587ef3a in xcb_wait_for_reply (c=0x17ae08, request=36, e=0xb40fea7c)
at ../../src/xcb_in.c:425
#5 0xb5e22644 in _XReply () from /usr/lib/arm-linux-gnueabihf/libX11.so.6
#6 0xb5627b9a in DRI2SwapBuffers ()
from /usr/lib/arm-linux-gnueabihf/libEGL.so.1
The main gnome-shell thread is hung trying to acquire a mali lock:
#0 __libc_do_syscall ()
at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:43
#1 0xb6630c44 in __lll_lock_wait (futex=futex@entry=0x10180c, private=0)
at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
#2 0xb662d4b0 in __GI___pthread_mutex_lock (mutex=0x10180c)
at pthread_mutex_lock.c:64
#3 0xb5b9219e in _mali_osu_lock_wait ()
from /usr/lib/arm-linux-gnueabihf/libEGL.s
#4 0xb5bd2e80 in glDeleteTextures ()
from /usr/lib/arm-linux-gnueabihf/libEGL.so
#5 0xb5fff5ca in _cogl_delete_gl_texture (gl_texture=42)
at ./driver/gl/cogl-pipeline-opengl.c:212
#6 0xb6024ae4 in _cogl_texture_2d_free (tex_2d=0x19f6858)
at ./cogl-texture-2d.c:72
#7 _cogl_object_texture_2d_indirect_free (obj=0x19f6858)
at ./cogl-texture-2d.c:56
#8 0xb600b950 in _cogl_object_default_unref (object=0x19f6858)
at ./cogl-object.c:96
#9 0xb600b8c4 in cogl_object_unref (obj=<optimized out>)
at ./cogl-object.c:104
#10 0xb601d646 in _cogl_pipeline_layer_free (layer=0x19e7f60)
at ./cogl-pipeline-layer.c:630
#11 _cogl_object_pipeline_layer_indirect_free (obj=0x19e7f60)
at ./cogl-pipeline-layer.c:52
#12 0xb600b950 in _cogl_object_default_unref (object=0x19e7f60)
#13 0xb600b8c4 in cogl_object_unref (obj=<optimized out>)
What has happened here is the following race:
X is deaf to the message sent in step 4, since another client (in step 2) issued GrabServer. So the DRI2SwapBuffers thread sits around forever waiting for a response, with the Mali lock held. The client that issued GrabServer itself is hung trying to obtain the Mali lock to do some GL op, so it will never ungrab the server. Deadlock!
Any solutions or workarounds appreciated. The best I can think of is to make sure no client ever does any kind of GL operation while it has the server grabbed. As the scope of that is enormous, it does not seem optimal.
May I ask what version of gnome you use in your project?
So you are porting Ubuntu onto the mali based device or something else? The hanged UI is developed by yourselft, or just the stock one?
As we can see in the description of XGrabServer:
The XGrabServer function disables processing of requests and close downs on all other connections than the one this request arrived on. You should not grab the X server any more than is absolutely necessary.
How do you use XGrabServer in this case?
Are the main thread and DRI2SwapBuffers thread in the same process?
And does main thread call some opengl API with server grab hold?
Thanks,
Frank.
We are using GNOME 3.8 and 3.10. The hang has been seen on custom and generic GNOME shells.
XGrabServer is used by the window manager (mutter), and this is quite normal for a window manager. In this case, mutter emits signals to the desktop shell which does rendering in response, all while Mutter is holding the server grab.
Maybe you can clarify how mali behaves here, as I am having trouble finding a way to solidly determine which library starts each thread and from which point. However, from what I have gathered, the window manager and desktop shell is single-threaded, and I believe that Mali creates an extra thread and a new X connection, and for some reason, it is sending SwapBuffers requests to the server from that thread. The main WM/shell thread does do rendering while a server grab is held on that thread's X connection, but then we hit the deadlock.
In the mean time, I have been improving mutter to not emit any signals (which could cause GL rendering) while holding server grabs. But, assuming my statement above is correct (Mali creating its own thread and separate X client and doing SwapBuffers there), this problem could be seen in other contexts.
Hi dsd,
Can you please let me know what device you are testing the driver with?. We cannot reproduce the problem on the Odroid-U2 running kernel 3.8 and r3p2-01rel4 Mali driver.
r3p2-01rel0 driver was released end of last year, our latest kernel driver is r4p0-00rel0 and is available from malideveloper.com .
Tu
Thanks for looking into this. We are testing on the ODROID-U2. Unfortunately the bug is not easy to reproduce, you have to try a lot of window operations (minimize, maximize, open, close) before it bites.
Nevertheless, I believe I have posted enough technical information above to highlight the bug in Mali. Mali's threaded design is incompatible with any X client that might perform GL operations while holding a server grab. If ARM is not prepared to support such scenario, or is not prepared to fix this, then it would be unfortunate but actually not a big issue from our end any more. I have now improved Mutter (the GNOME window manager) so that it never takes server grabs and my changes were accepted upstream for the next version.
However, I believe this threaded model that has bitten here is part of a wider design problem that is biting us in other areas such as in Bad interaction with DRI2 for vsync
So it would still be worth working on the larger problem. Right now Mali starts a new thread and a new X connection, and uses that new thread/connection to perform SwapBuffers. This seems bizarre, given that SwapBuffers is an asynchronous operation that should always return immediately, but as we see in the other thread, it looks like there was some confusion here and the original Mali stack has implemented this as a synchronous, blocking op (which explains why libMali would then create a new thread and connection). Would be great if that can be fixed.
We would love to try r4p0 and in fact I already downloaded the source code you mentioned, compiled it and checked that it loads - great. However we cannot actually test this until someone provides us with the r4p0 version of libMali/libEGL/libGLESv2. Hardkernel have some DDK licensing difficulties at the moment so they can't provide this. If ARM could provide a new binary libMali for Mali-400/armhf/Linux/X11/UMP it would be ideal.
Daniel
dsd wrote: We would love to try r4p0 and in fact I already downloaded the source code you mentioned, compiled it and checked that it loads - great. However we cannot actually test this until someone provides us with the r4p0 version of libMali/libEGL/libGLESv2. Hardkernel have some DDK licensing difficulties at the moment so they can't provide this. If ARM could provide a new binary libMali for Mali-400/armhf/Linux/X11/UMP it would be ideal.
dsd wrote:
Thanks for your feedback, hopefully HardKernel can release soon as we typically do not release binaries to anyone who isn't under at least an LUL licence agreement.
Chris