C++ Standard library support in Morello linux pure-cap

Hi Yury, sorry it took so long but I have now tried out the new C++ support on Linux pure-cap.  However I have had problems building example C++ code because the C++ exception table entries are not being built properly.  This means any exception thrown in my code gets unhandled.  Specifically, I am getting warnings like the following at the link stage, for all symbols:

ld.lld: warning: Could not find a real symbol for .gcc_except_table._ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE5flushEv+0x70 in /llvm-morello-build/musl-bin/lib/libc++.a(ios.instantiations.cpp.o)

I presume I am making some silly mistake but I am not sure what? - hope you can help.

How I built the toolchain
I followed your steps in build-morello-clang.rst however note that:
- I did not build clang from source, instead I downloaded the morello llvm-project-release v1.6
- I had to define an env var LLVM in order to build the kernel headers (ref __download_kernel_headers() from build_morello.sh) after extracting the tar file
- Note I am cross-compiling, but it shouldn't make any difference

How I build example C++ file
It's pretty much the same way as per the Morello C examples... Compiling:

clang++ march=morello+c64 --target=aarch64-linux-musl_purecap --sysroot /path/to/musl-bin -g -c example.cpp -o example.cpp.o

Then linking:

clang -fuse-ld=lld march=morello+c64 --target=aarch64-linux-musl_purecap -nostdlib -L/path/to/musl/bin -lc -lc++ -lc++abi -lunwind
   <clang_abs_path>/clang_rt.ctbegin.o   <clang_abs_path>/clang_rt.ctend.o  <clang_abs_path>/libclang_rt.builtins.a
   <musl_bin>/lib/crt1.o  <musl_bin>/lib/crti.o  <musl_bin>/lib/crtn.o
    -Wl,--gc--sections   -static
    -o example.out
     example.cpp.o

Note that all else seems to work ok apart from exception handling.
(Limited testing so far: from C++ stdlib I have used iostream, string, sstream, unordered_map.  Also have been using templates (basic), global operator new & delete.  Note I am targeting c++14).

Thanks
Pete 

Hi Yury, sorry it took so long but I have now tried out the new C++ support on Linux pure-cap.  However I have had problems building example C++ code because the C++ exception table entries are not being built properly.  This means any exception thrown in my code gets unhandled.  Specifically, I am getting warnings like the following at the link stage, for all symbols:

ld.lld: warning: Could not find a real symbol for .gcc_except_table._ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE5flushEv+0x70 in /llvm-morello-build/musl-bin/lib/libc++.a(ios.instantiations.cpp.o)

I presume I am making some silly mistake but I am not sure what? - hope you can help.

How I built the toolchain
I followed your steps in build-morello-clang.rst however note that:
- I did not build clang from source, instead I downloaded the morello llvm-project-release v1.6
- I had to define an env var LLVM in order to build the kernel headers (ref __download_kernel_headers() from build_morello.sh) after extracting the tar file
- Note I am cross-compiling, but it shouldn't make any difference

How I build example C++ file
It's pretty much the same way as per the Morello C examples... Compiling:

clang++ march=morello+c64 --target=aarch64-linux-musl_purecap --sysroot /path/to/musl-bin -g -c example.cpp -o example.cpp.o

Then linking:

clang -fuse-ld=lld march=morello+c64 --target=aarch64-linux-musl_purecap -nostdlib -L/path/to/musl/bin -lc -lc++ -lc++abi -lunwind
   <clang_abs_path>/clang_rt.ctbegin.o   <clang_abs_path>/clang_rt.ctend.o  <clang_abs_path>/libclang_rt.builtins.a
   <musl_bin>/lib/crt1.o  <musl_bin>/lib/crti.o  <musl_bin>/lib/crtn.o
    -Wl,--gc--sections   -static
    -o example.out
     example.cpp.o

Note that all else seems to work ok apart from exception handling.
(Limited testing so far: from C++ stdlib I have used iostream, string, sstream, unordered_map.  Also have been using templates (basic), global operator new & delete.  Note I am targeting c++14).

Thanks
Pete 

Hi Pete and thanks for such a comprehensive report!

The linker warning

ld.lld: warning: Could not find a real symbol for .gcc_except_table

is a known issue and you should be able to just ignore it. At some point it should be fixed. You may add  -Wl,-warning-limit=1 to your command like to see fewer warnings.

It is totally fine to use the prebuilt LLVM binaries for the v1.6 release, in fact, this should save you some time and allow to skip first few steps from the guide. These binaries are generated according to the guide anyway.

Could you elaborate a bit more on what you mean by

I had to define an env var LLVM in order to build the kernel headers

What exactly didn't work here?

Cross-compiling is fine: you should be able to use the build script successfully both x86 and arm64 hosts.

The way you compile and link your test executable seems correct, though you should be able to simplify it:

clang++ -target aarch64-unknown-linux-musl_purecap \
        -march=morello+c64 \
        --sysroot /path/to/sysroot \
        hello.cpp -o hello \
        -lunwind -lc++abi -static

If you use prebuilt binaries, the right compiler runtime and linker will be selected, the same can be achieved by using -fuse-ld=lld and --rtlib=compiler-rt flags.

Thanks,
Yury

Hi Yury, thanks for coming back.  Re. compiling yes that is easier, thanks, I had just copied the morello example Makefile scripts.

Re. the warnings, I had assumed these were related to the fact exception handling is not working... seems you already know about those warnings, so I guess exception handling is a bug.  I investigated this issue further, so here are some more details...

Exceptions Issue

Here is a simple bit of C++ code, I cross-compile it to a linux pure-cap program using the same clang++ command line as you used above then I transfer it to the morello board and execute it.

#include <iostream>

struct CMyException {};

int main()
{
    std::cout << "Start of exception test" << std::endl;
    try
    {
        throw CMyException();
        std::cout << "Should not print this" << std::endl;
    }
    catch(CMyException)
    {
    	std::cout << "Exception!" << std::endl;
    }
    std::cout << "End of exception test" << std::endl;
    return 0;
}

We expect on stdout to see "Start of exception test", "Exception!", "End of exception test" and in fact this happens if you build with g++.  But instead we get a segmentation fault at the throw() instruction.

The reason for this is there is a dereference of a capability memory address which has an invalid tag (i.e tag==0).  And looking at the generated assembler code the reason for this would appear to be because an address has been dereferenced as an integer not a intptr_t i.e the compiler has used an x-register not the c-register.

This all occurs in dl_iterate_phdr()... I had a quick look at the source code but I cannot obviously figure out what has gone wrong.  Anyway, I will hand it over to you as you should be able to recreate it (or alternatively tell me if I have done something silly!)

Explaining LLVM variable in build

Ok, to explain this...

1. I am running build-morello.sh libcxx as per your instructions, note I am using morello/master branch of musl-libc
2. This script ends up calling a function __download_kernel_headers()
3. This function downloads & extracts the tarball and then runs: make -C linux-${VERSION} headers_install ARCH=arm64 INSTALL_HDR_PATH=${BUILD_PATH}/kernel
4. If we look at the Makefile that is being used (from the extracted tarball) we can see that it tries to work out whether to use clang or gcc.. it's around line 426, if LLVM is not defined then it falls back to gcc which is what was happening in my case.
5. So basically it is necessary to define LLVM=<path> then run the command through bash, so that the linux kernel build can figure out where your clang & clang++ are found.

So in summary I think probably you just need to update your document to add instructions to set LLVM variable and then run the script.  Also BTW there look like a few copy/paste errors in the document as it refers to building libunwind in the instructions for building libc++ and libc++abi, and finally there are no instructions in the bash_morello.sh file itself although the other targets have instructions in that file.

Hope this makes sense.

That being said, these are small points and the process was quite ok and the instructions were very helpful.  The only problem I found was this need to set LLVM variable to build the libcxx case, although as I say I did not rebuild clang itself from source so I cannot comment if that step works or not.

Thanks
Pete

Hi Pete, thanks for the detailed report! We've made some updates on the Musl side and if you rebuild it from the top of the "morello/master" branch, you should get two improvements:

1. The need for extra env vars for installing kernels from source has been removed: we now download kernel headers for Morello and just unpack them.
2. With a fix to Musl dynamic linker, exceptions in C++ should now work in static binaries.

Hope this helps!

Thanks Yury, I look forward to giving it another try sometime next week!
Pete

Hi Yury, just reporting back, I updated from master and re-tested and indeed the two changes you described worked well - many thanks!

I appreciate there is still a little bit more work to do (e.g all those build warnings are still there), but do you have an updated estimate for when there will be an official release of this MUSL libc++ support?

Best Regards
Pete