Morello toolchain compilation

Hi,

I'm trying to compile the Morello (baremetal cross-compilation) toolchains from source. (I know binary releases exist, but I can't use those.)

Initially I tried compiling an LLVM-based toolchain from https://git.morello-project.org/morello/llvm-project (as seems to be most recommended). I managed to get a clang binary that nominally works, but has a few annoying quirks that are annoying to deal with:

  • By default, the compiler wants to use my system linker at /usr/bin/ld, instead of the lld that just got compiled. This can be fixed by adding -DCLANG_DEFAULT_LINKER=lld to the CMake configuration flags, but apparently this is "discouraged".
  • By default, clang wans to link against a libc, crt0 and clang_rt that do not exist, unless -ffreestanding -nostartfiles -nostdlib are supplied (and all other LLVM/clang runtime components seem to rely on a libc being present). Which is understandable, except that:
    • LLVM libc can't really be compiled because it's still to experimental to begin with
    • The binary releases seem to use newlib, but it's unclear  how these got built:

As I am more familiar with these, I then attempted to use the GNU-based one (from https://git.morello-project.org/morello/gnu-toolchain/ and https://git.morello-project.org/morello/newlib ). Compiling binutils and gcc seems to work, but then the newlib build chokes on errors like these:

../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2354:25: error: unknown type name 'capability'; did you mean '__capability'?
 2354 | static Void_t* mem2heap(capability Void_t* mem)
      |                         ^~~~~~~~~~
      |                         __capability
../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2378:13: error: '__cheri_fromcap' undeclared (first use in this function)
 2378 |     return (__cheri_fromcap Void_t *)mem;

Looking at the 'mallocr.c' file, this all seems to happen inside an #ifdef block for CHERI support (so the compiler does support those), but somehow the new magic keywords don't seem to be supported. I did try to use a multilib configuration for GCC and newlib (to compile purecap and hybridcap versions in one go), but I'm not sure if these errors are caused by that, or by something else.

So what's going on here? Are there any 'official' docs on what configuration flags etc. to use to build a cross-compiler?

Thanks.

P.S. here are the configuration flags I've been using:

LLVM: 

cmake ../llvm-project/llvm/ -G Ninja \
        -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_INSTALL_PREFIX:STRING="$INSTALLDIR" \
        -DLLVM_ENABLE_PROJECTS="clang;lld;lldb;clang-tools-extra" \
        -DLLVM_DEFAULT_TARGET_TRIPLE="aarch64-none-elf" -DCLANG_DEFAULT_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD:STRING="AArch64" \
        -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
        -DCMAKE_SKIP_BUILD_RPATH:BOOL=OFF \
        -DCMAKE_INSTALL_RPATH:STRING=\$ORIGIN/../lib \
        -DLLVM_ENABLE_ASSERTIONS:BOOL=ON \
        -DCMAKE_INSTALL_RPATH:STRING=\$ORIGIN/../lib \
        -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON \
        -DLLVM_ENABLE_LIBCXX:BOOL=OFF -DLIBUNWIND_ENABLE_THREADS:BOOL=ON \
        -DLLVM_ENABLE_EH:BOOL=ON -DLLVM_ENABLE_RTTI:BOOL=ON -DLLVM_ENABLE_Z3_SOLVER:BOOL=OFF \
        -DLIBCXX_CXX_ABI:STRING=libcxxabi \
        -DLIBCXX_CXX_ABI_INCLUDE_PATHS:STRING=../llvm-project/libcxxabi/include \
        -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=ON -DLIBCXX_ENABLE_THREADS:BOOL=ON \
        -DLIBCXXABI_ENABLE_THREADS:BOOL=ON \
        -DBUILD_SHARED_LIBS:BOOL=ON

Binutils: 

../binutils-gdb/configure --target="aarch64-none-elf" \
        --disable-nls --enable-deterministic-archives --enable-ld --enable-multilib \
        --prefix="$INSTDIR" --with-gnu-as --with-gnu-ld \
        --with-sysroot="$INSTDIR/aarch64-none-elf" --with-system-zlib --without-isl

GCC: 

../gcc/configure --target="aarch64-none-elf" \
        --enable-languages=c,c++,lto --enable-multilib \
        --with-multilib-generator='aarch64-ilp32--;morello-hybridcap--;morello+c64-purecap--' \
        \
        --libdir="$INSTDIR/lib" --libexecdir="$INSTDIR/lib" \
        --prefix="$INSTDIR" --with-headers="$INSTDIR/aarch64-none-elf/include" \
        --with-sysroot="$INSTDIR/aarch64-none-elf" \
        --with-native-system-header-dir=/include --with-newlib --with-python-dir=share/gcc-aarch64-none-elf \
        \
        --disable-decimal-float --disable-libffi --disable-libgomp --disable-libmudflap \
        --disable-libquadmath --disable-libssp --disable-libstdcxx-pch --disable-nls \
        --disable-shared --disable-threads --disable-tls --disable-werror \
        --disable-gcov \
        --enable-__cxa_atexit --enable-c99 --enable-gnu-indirect-functions \
        --enable-long-long --enable-plugins --enable-lto \
        --with-gmp --with-gnu-as --with-gnu-ld --with-host-libstdcxx='-static-libgcc' \
        --without-isl --with-libelf --with-mpc --with-mpfr --with-system-zlib

newlib: 

../newlib/configure --target="aarch64-none-elf" \
        --prefix="$INSTDIR" \
        --with-gnu-as --with-gnu-ld --disable-nls --enable-interwork \
        --disable-newlib-supplied-syscalls --enable-newlib-retargetable-locking \
        --enable-newlib-io-long-long --enable-newlib-register-fini

  • The last time I built newlib (  https://git.morello-project.org/morello/newlib  ) with Morello LLVM was a while back.

    The LLVM newlib build scripts were never released as far as I know, nor are there any official docs on building it.

     

    The type name 'capability' is defined in /lib/clang/13.0.0/include/cheri.h and is conditionally defined only when WANT_CHERI_QUALIFIER_MACROS is defined.

     

    Looking at my logs, the way newlib was built was with

      CFLAGS_FOR_TARGET="-O2 -Qunused-arguments -DWANT_CHERI_QUALIFIER_MACROS" 

  • Interestingly, I'm still getting errors like these, when adding that flag (when building with GCC):

    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2354:1: warning: use of '__capability' before the pointer type is deprecated [-Wdeprecated-declarations]
     2354 | static Void_t* mem2heap(capability Void_t* mem)
          | ^~~~~~
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c: In function 'mem2heap':
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2375:13: error: '__cheri_fromcap' undeclared (first use in this function)
     2375 |     return (__cheri_fromcap Void_t *)mem;
          |             ^~~~~~~~~~~~~~~
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2375:13: note: each undeclared identifier is reported only once for each function it appears in
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2375:28: error: expected ')' before 'void'
     2375 |     return (__cheri_fromcap Void_t *)mem;
          |            ~               ^
          |                            )
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2375:38: error: expected ';' before 'mem'
     2375 |     return (__cheri_fromcap Void_t *)mem;
          |                                      ^~~
          |                                      ;
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c: At top level:
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2381:1: warning: use of '__capability' before the pointer type is deprecated [-Wdeprecated-declarations]
     2381 | {
          | ^
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c: In function 'heap2mem':
    ../../../../../../newlib/newlib/libc/stdlib/mallocr.c:2385:5: warning: use of '__capability' before the pointer type is deprecated [-Wdeprecated-declarations]
     2385 |     capability Void_t* m = (capability Void_t*)hmem;
          |     ^~~~~~~~~~

    LLVM/clang seems to fare better, though I've had to apply the following patch to have it built for normal, hybridcap and purecap ABIs and installed as with the binary LLVM releases:

    diff --git a/libgloss/aarch64/configure b/libgloss/aarch64/configure
    index b45256f3c..af17cca46 100755
    --- a/libgloss/aarch64/configure
    +++ b/libgloss/aarch64/configure
    @@ -2521,7 +2521,7 @@ test "${CCASFLAGS+set}" = set || CCASFLAGS=$CFLAGS
    
    
     case "${target}" in
    -  *-*-elf)
    +  *-*-elf*)
            objtype=elf-
            ;;
     esac
    diff --git a/libgloss/aarch64/configure.in b/libgloss/aarch64/configure.in
    index 73d11b54d..45e80b7a3 100644
    --- a/libgloss/aarch64/configure.in
    +++ b/libgloss/aarch64/configure.in
    @@ -29,7 +29,7 @@ AC_PROG_RANLIB
     LIB_AM_PROG_AS
    
     case "${target}" in
    -  *-*-elf)
    +  *-*-elf*)
            objtype=elf-
            ;;
     esac
    diff --git a/libgloss/libnosys/configure b/libgloss/libnosys/configure
    index fbe7db764..7b7653fc1 100755
    --- a/libgloss/libnosys/configure
    +++ b/libgloss/libnosys/configure
    @@ -2054,7 +2054,7 @@ case "${target}" in
     esac
    
     case "${target}" in
    -  *-*-elf)
    +  *-*-elf*)
             $as_echo "#define HAVE_ELF 1" >>confdefs.h
    
    
    diff --git a/libgloss/libnosys/configure.in b/libgloss/libnosys/configure.in
    index 1d4846b17..be2058192 100644
    --- a/libgloss/libnosys/configure.in
    +++ b/libgloss/libnosys/configure.in
    @@ -88,7 +88,7 @@ esac
    
     dnl Make sure we know if elf format used
     case "${target}" in
    -  *-*-elf)
    +  *-*-elf*)
             AC_DEFINE(HAVE_ELF)
    
             AC_CACHE_CHECK([for .previous assembler directive],

    I'll try compiling the LLVM runtime libraries, and report back on how that works. Once it's finished, I'll also post my buildscript here. But thanks for the help at least.

  • The prebuilt baremetal toolchain is built with Morello LLVM (which is far more mature than Morello GCC). AFAIU the released Morello GCC does not implement __cheri_{from,to}cap, hence the errors you're getting.

  • Alright, I managed to finalize a working buildscript. Turns out, most of the work had already been done at https://github.com/ARM-software/LLVM-embedded-toolchain-for-Arm/blob/main/arm-runtimes/CMakeLists.txt , but I discovered this only relatively late. But basically, if you combine it with the above newlib stuff (and steal the .cfg files from the binary release), you get a working compiler.

    Do note that the supplied .cfg files by default pull in libgloss by its entirety, which, depending on the usage, might end up being a footgun.

    llvm-morello-baremetal-buildscript.zip