C++ Standard library support in Morello linux pure-cap

Hi Yury, thanks for coming back.  Re. compiling yes that is easier, thanks, I had just copied the morello example Makefile scripts.

Re. the warnings, I had assumed these were related to the fact exception handling is not working... seems you already know about those warnings, so I guess exception handling is a bug.  I investigated this issue further, so here are some more details...

Exceptions Issue

Here is a simple bit of C++ code, I cross-compile it to a linux pure-cap program using the same clang++ command line as you used above then I transfer it to the morello board and execute it.

#include <iostream>

struct CMyException {};

int main()
{
    std::cout << "Start of exception test" << std::endl;
    try
    {
        throw CMyException();
        std::cout << "Should not print this" << std::endl;
    }
    catch(CMyException)
    {
    	std::cout << "Exception!" << std::endl;
    }
    std::cout << "End of exception test" << std::endl;
    return 0;
}

We expect on stdout to see "Start of exception test", "Exception!", "End of exception test" and in fact this happens if you build with g++.  But instead we get a segmentation fault at the throw() instruction.

The reason for this is there is a dereference of a capability memory address which has an invalid tag (i.e tag==0).  And looking at the generated assembler code the reason for this would appear to be because an address has been dereferenced as an integer not a intptr_t i.e the compiler has used an x-register not the c-register.

This all occurs in dl_iterate_phdr()... I had a quick look at the source code but I cannot obviously figure out what has gone wrong.  Anyway, I will hand it over to you as you should be able to recreate it (or alternatively tell me if I have done something silly!)

Explaining LLVM variable in build

Ok, to explain this...

1. I am running build-morello.sh libcxx as per your instructions, note I am using morello/master branch of musl-libc
2. This script ends up calling a function __download_kernel_headers()
3. This function downloads & extracts the tarball and then runs: make -C linux-${VERSION} headers_install ARCH=arm64 INSTALL_HDR_PATH=${BUILD_PATH}/kernel
4. If we look at the Makefile that is being used (from the extracted tarball) we can see that it tries to work out whether to use clang or gcc.. it's around line 426, if LLVM is not defined then it falls back to gcc which is what was happening in my case.
5. So basically it is necessary to define LLVM=<path> then run the command through bash, so that the linux kernel build can figure out where your clang & clang++ are found.

So in summary I think probably you just need to update your document to add instructions to set LLVM variable and then run the script.  Also BTW there look like a few copy/paste errors in the document as it refers to building libunwind in the instructions for building libc++ and libc++abi, and finally there are no instructions in the bash_morello.sh file itself although the other targets have instructions in that file.

Hope this makes sense.

That being said, these are small points and the process was quite ok and the instructions were very helpful.  The only problem I found was this need to set LLVM variable to build the libcxx case, although as I say I did not rebuild clang itself from source so I cannot comment if that step works or not.

Thanks
Pete

Hi Yury, thanks for coming back.  Re. compiling yes that is easier, thanks, I had just copied the morello example Makefile scripts.

Re. the warnings, I had assumed these were related to the fact exception handling is not working... seems you already know about those warnings, so I guess exception handling is a bug.  I investigated this issue further, so here are some more details...

Exceptions Issue

Here is a simple bit of C++ code, I cross-compile it to a linux pure-cap program using the same clang++ command line as you used above then I transfer it to the morello board and execute it.

#include <iostream>

struct CMyException {};

int main()
{
    std::cout << "Start of exception test" << std::endl;
    try
    {
        throw CMyException();
        std::cout << "Should not print this" << std::endl;
    }
    catch(CMyException)
    {
    	std::cout << "Exception!" << std::endl;
    }
    std::cout << "End of exception test" << std::endl;
    return 0;
}

We expect on stdout to see "Start of exception test", "Exception!", "End of exception test" and in fact this happens if you build with g++.  But instead we get a segmentation fault at the throw() instruction.

The reason for this is there is a dereference of a capability memory address which has an invalid tag (i.e tag==0).  And looking at the generated assembler code the reason for this would appear to be because an address has been dereferenced as an integer not a intptr_t i.e the compiler has used an x-register not the c-register.

This all occurs in dl_iterate_phdr()... I had a quick look at the source code but I cannot obviously figure out what has gone wrong.  Anyway, I will hand it over to you as you should be able to recreate it (or alternatively tell me if I have done something silly!)

Explaining LLVM variable in build

Ok, to explain this...

1. I am running build-morello.sh libcxx as per your instructions, note I am using morello/master branch of musl-libc
2. This script ends up calling a function __download_kernel_headers()
3. This function downloads & extracts the tarball and then runs: make -C linux-${VERSION} headers_install ARCH=arm64 INSTALL_HDR_PATH=${BUILD_PATH}/kernel
4. If we look at the Makefile that is being used (from the extracted tarball) we can see that it tries to work out whether to use clang or gcc.. it's around line 426, if LLVM is not defined then it falls back to gcc which is what was happening in my case.
5. So basically it is necessary to define LLVM=<path> then run the command through bash, so that the linux kernel build can figure out where your clang & clang++ are found.

So in summary I think probably you just need to update your document to add instructions to set LLVM variable and then run the script.  Also BTW there look like a few copy/paste errors in the document as it refers to building libunwind in the instructions for building libc++ and libc++abi, and finally there are no instructions in the bash_morello.sh file itself although the other targets have instructions in that file.

Hope this makes sense.

That being said, these are small points and the process was quite ok and the instructions were very helpful.  The only problem I found was this need to set LLVM variable to build the libcxx case, although as I say I did not rebuild clang itself from source so I cannot comment if that step works or not.

Thanks
Pete

Hi Pete, thanks for the detailed report! We've made some updates on the Musl side and if you rebuild it from the top of the "morello/master" branch, you should get two improvements:

1. The need for extra env vars for installing kernels from source has been removed: we now download kernel headers for Morello and just unpack them.
2. With a fix to Musl dynamic linker, exceptions in C++ should now work in static binaries.

Hope this helps!

Thanks Yury, I look forward to giving it another try sometime next week!
Pete

Hi Yury, just reporting back, I updated from master and re-tested and indeed the two changes you described worked well - many thanks!

I appreciate there is still a little bit more work to do (e.g all those build warnings are still there), but do you have an updated estimate for when there will be an official release of this MUSL libc++ support?

Best Regards
Pete