Baremetal program jumps to 0x200

In attempt to run a baremetal program using C standard library function on hardware, I also tried using GNU Morello toolchain (arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf.tar.xz),

I am not familiar with integrating standard library to embedded systems so I tried to follow this tutorial:

https://www.youtube.com/watch?v=AzYC5hO2uJk

I used the following makefile, linker script and "syscalls.c" trying to compile a "hello_world" program (I uploaded all files to Github):

- Makefile

- linker_script.ld

- hello_world.c

- syscalls.c (with modified _read and _write functions to use 0x2a400000 address, and modified _sbrk to use the HEAP_START symbol from the linker script, I verified that this kind of setup worked well using riscv-unknown-elf toolchain before)

During compilation I got error about undefined functions like this:

michal@ubuntu22-04:~/morello/standalone_build/helloworld$ make
aarch64-none-elf-gcc -march=morello+c64 -mabi=purecap -static -fvisibility=hidden -I../include -g -o hello_world hello_world.o ./syscalls.o -T linker_script.ld
/home/michal/arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/10.1.0/../../../../aarch64-none-elf/bin/ld: /home/michal/arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/10.1.0/../../../../aarch64-none-elf/lib/purecap/c64/crt0.o: in function `.pure':
/data/jenkins/workspace/GNU-toolchain/morello-trunk/src/newlib-cygwin/libgloss/aarch64/crt0.S:243: undefined reference to `__init_global_caps'
/data/jenkins/workspace/GNU-toolchain/morello-trunk/src/newlib-cygwin/libgloss/aarch64/crt0.S:243:(.text+0x184): relocation truncated to fit: R_MORELLO_CALL26 against undefined symbol `__init_global_caps'
/home/michal/arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/10.1.0/../../../../aarch64-none-elf/bin/ld: /data/jenkins/workspace/GNU-toolchain/morello-trunk/src/newlib-cygwin/libgloss/aarch64/crt0.S:244: undefined reference to `__processRelocs'
/data/jenkins/workspace/GNU-toolchain/morello-trunk/src/newlib-cygwin/libgloss/aarch64/crt0.S:244:(.text+0x188): relocation truncated to fit: R_MORELLO_CALL26 against undefined symbol `__processRelocs'
/home/michal/arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/10.1.0/../../../../aarch64-none-elf/bin/ld: hello_world.o(.rela.got+0xf0): warning: relocation R_MORELLO_ADR_GOT_PAGE against symbol 'SBSS_START' in section without permission flags '.sbss'.  Assuming Read-Write.
/home/michal/arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/10.1.0/../../../../aarch64-none-elf/bin/ld: hello_world.o(.rela.got+0x10): warning: relocation R_MORELLO_ADR_GOT_PAGE against symbol 'SBSS_END' in section without permission flags '.sbss'.  Assuming Read-Write.
collect2: error: ld returned 1 exit status
make: *** [Makefile:43: hello_world] Error 1

From what I see, the __init_global_caps and __process_relocs functions are in the following file (assumming that the newlib from link below is the same newlib used in the GNU toolchain):

https://git.morello-project.org/morello/newlib/-/blob/morello/master/libgloss/aarch64/relocs.c

But for some reason the error message says these functions/symbols are undefined.

https://community.arm.com/support-forums/f/morello-forum/52923/baremetal-tfa-payload-built-for-capability-mode-still-in-pstate-a64-on-el2-entry

The good news is that since then, Silviu Baranga added support for booting at EL2 in Newlib:

https://git.morello-project.org/morello/newlib/-/merge_requests/17

As far as I can tell there is however no easy way to make the LLD baremetal driver choose the EL2 boot code instead of EL3, for reference:

https://git.morello-project.org/morello/llvm-project/-/blob/morello/master/clang/lib/Driver/ToolChains/AArch64BareMetal.cpp?ref_type=heads#L255

I think you could work around that by passing -nostdlib to LLD, and then explicitly specify all startup files. With LIB the local path to this directory:

https://git.morello-project.org/morello/llvm-project-releases/-/tree/morello/baremetal-release-1.6/aarch64-none-elf/lib

The flags to pass to LLD would be:

-nostdlib LIB/morello-el2-crt0.o LIB/crti.o LIB/cpu-init/morello-init-el2.o --start-group -lc -lgloss-morello-el2 --end-group -lclang_rt.builtins-aarch64 LIB/crtn.o

Note that I have not actually tested this command line, but hopefully it will work as intended.

Is there some flag for GCC that could be added in order to not use EL3 and use EL2 instead? I tried adding "-D__MORELLO_EL2__" (by looking at the changes from this commit) but it keeps producing "MRS x0, CPTR_EL3".

Does it mean that the compiler itself must be recompiled with that flag?

vshcmd: > # Recommend using a separate untar of the toolchain.
vshcmd: > # (Replacing bundled binaries with newly built binaries.  I don't
vshcmd: > # know for certain these binaries would work -- though I think it's a
vshcmd: > # good enough chance to try.)
vshcmd: > tar -xaf arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf.tar.xz
morello-distro [14:06:32] $
vshcmd: > # Just to de-mystify this helper function I happen to use.
vshcmd: > type newdir
newdir is a function
newdir ()
{
    mkdir -p "$@";
    cd "${@: -1}"
}
morello-distro [14:06:34] $
vshcmd: > newdir rebuild-newlib
rebuild-newlib [14:06:36] $
vshcmd: > git clone git.morello-project.org/.../newlib.git
Cloning into 'newlib'...
<snip>
vshcmd: > newdir newlib-build
newlib-build [14:07:32] $
vshcmd: > PATH="$(pwd)/../../arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin:$PATH" \
vshcmd: > CC_FOR_TARGET=aarch64-none-elf-gcc \
vshcmd: >     CFLAGS="-O1 -g" \
vshcmd: >     CFLAGS_FOR_TARGET="-ffunction-sections -fdata-sections -O2 -g -DWANT_CHERI_QUALIFIER_MACROS -D__cheri_fromcap=" \
vshcmd: >     CXXFLAGS_FOR_TARGET="-ffunction-sections -fdata-sections -O2 -g -DWANT_CHERI_QUALIFIER_MACROS -D__cheri_fromcap=" \
vshcmd: > ../newlib/configure --disable-newlib-supplied-syscalls \
vshcmd: >                     --enable-newlib-retargetable-locking \
vshcmd: >                     --enable-newlib-reent-check-verify \
vshcmd: >                     --enable-newlib-io-long-long \
vshcmd: >                     --enable-newlib-io-c99-formats \
vshcmd: >                     --enable-newlib-register-fini \
vshcmd: >                     --enable-newlib-mb \
vshcmd: >                     --target=aarch64-none-elf \
vshcmd: >                     --prefix=/ --with-pkgversion=unknown
<snip>
vshcmd: > # N.b. I happen to build the entirety of newlib.
vshcmd: > # I expect that we could get away with only building libgloss, but
vshcmd: > # building everything doesn't take too long.
vshcmd: > PATH="$(pwd)/../../arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin:$PATH" \
vshcmd: >       make -j8 all-target-newlib all-target-libgloss
<snip>
vshcmd: > # One option is to install in a local directory.
vshcmd: > PATH="$(pwd)/../../arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin:$PATH" \
vshcmd: > make DESTDIR=$(pwd)/../newlib-install \
vshcmd: >     INSTALL="/usr/bin/install -C" \
vshcmd: >     install-target-newlib \
vshcmd: >     install-target-libgloss
<snip>
vshcmd: > # I believe you would be able to directly use these binaries using
vshcmd: > # `-nostartfiles` to avoid the non-EL2 crt0.o file and then manually
vshcmd: > # specifying the crti.o, crtbegin.o, and morello-el2-crt0.o file.
vshcmd: > # (N.b. don't worry about the warning -- it's not something that
vshcmd: > # should stop things working).
vshcmd: > cd ../../
vshcmd: > ./arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/aarch64-none-elf-gcc \
vshcmd: >   -march=morello+c64 -mabi=purecap -nostartfiles -L rebuild-newlib/newlib-install/aarch64-none-elf/lib/purecap/c64/ \
vshcmd: >   -Wl,arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/lib/gcc/aarch64-none-elf/10.1.0/purecap/c64/crti.o,arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/lib/gcc/aarch64-none-elf/10.1.0/purecap/c64/crtbegin.o,./rebuild-newlib/newlib-install/aarch64-none-elf/lib/purecap/c64/morello-el2-crt0.o,./rebuild-newlib/newlib-install/aarch64-none-elf/lib/purecap/c64/cpu-init/morello-init-el2.o,--start-group,-lc,-lgloss-morello-el2,--end-group,arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/lib/gcc/aarch64-none-elf/10.1.0/purecap/c64/crtend.o,arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/lib/gcc/aarch64-none-elf/10.1.0/purecap/c64/crtn.o test.c -o test-direct-alt
morello-distro [14:28:06] $ > > <snip>/aarch64-none-elf/bin/ld: rebuild-newlib/newlib-install/aarch64-none-elf/lib/purecap/c64//libgloss-morello-el2.a(morello-el2-uart.o)(.rela.got+0x1c0): warning: relocation R_MORELLO_ADR_GOT_PAGE against symbol '__UART' in section without permission flags '*ABS*'.  Assuming Read-Write.
morello-distro [14:28:06] $
vshcmd: > # However, it seems like it would be neater to instead install into
vshcmd: > # the GCC install directory.  Then create a new `specs` file so you
vshcmd: > # don't have to always specify the required crt0 startup object
vshcmd: > # files.
vshcmd: > cd rebuild-newlib/newlib-build/
vshcmd: > PATH="$(pwd)/../../arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin:$PATH" \
vshcmd: > make DESTDIR=$(pwd)/../../arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf \
vshcmd: >     INSTALL="/usr/bin/install -C" \
vshcmd: >     install-target-newlib \
vshcmd: >     install-target-libgloss
<snip>
vshcmd: > cd ../../
morello-distro [14:29:51] $
vshcmd: > # Manually defined specs file (i.e. write the below in the relevant file).
vshcmd: > # Includes the new crt0.o file and the new morello-init-el2.o file.
vshcmd: > cat arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/aarch64-none-elf/lib/morello-board-el2.specs
%rename lib libc

*libgloss:
-lgloss-morello-el2

*startfile:
crti.o%s crtbegin.o%s morello-el2-crt0.o%s

*lib:
cpu-init/morello-init-el2.o%s --start-group %(libc) %(libgloss) --end-group

morello-distro [14:30:05] $
vshcmd: > ./arm-gnu-toolchain-10.1.morello-alp2-x86_64-aarch64-none-elf/bin/aarch64-none-elf-gcc \
vshcmd: >   -march=morello+c64 -mabi=purecap -specs=morello-board-el2.specs test.c -o test
> <snip>/aarch64-none-elf/bin/ld: <snip>/aarch64-none-elf/lib/purecap/c64/libgloss-morello-el2.a(morello-el2-uart.o)(.rela.got+0x1c0): warning: relocation R_MORELLO_ADR_GOT_PAGE against symbol '__UART' in section without permission flags '*ABS*'.  Assuming Read-Write.
morello-distro [14:31:11] $


https://github.com/michalmonday/morello_baremetal/blob/5c17d0600a0ae10549e414f70b3bf86fc58a253e/Makefile

And this permalink to the objdump it produced:

https://github.com/michalmonday/morello_baremetal/blob/5c17d0600a0ae10549e414f70b3bf86fc58a253e/hello_world.dump

I tried running it with and without ".ds" script from this reply about semihosting (because curr_sp0_fiq was mentioned in that thread):

set semihosting heap-base 0
set semihosting heap-limit 0x80800000
set semihosting stack-limit 0x80800000
set semihosting stack-base 0x81000000
set semihosting enabled on

But in both cases the behaviour was the same.

https://git.morello-project.org/morello/newlib/-/blob/morello/master/libgloss/aarch64/crt0.S#L180

Hard to say why without more information. Have a look at the value of the ESR_EL2 system register just after the exception is taken, feel free to copy it here.

-_cpu_init_hook (the cpu_init_hook seems to have "ret" instruction only in the source code, but in the disassembler it calls _init_vectors and _flat_map, I attached image of it below)

- _init_vectors

- _flat_map

- memset

I didn't test using F6 with any other functions, but every function I tried to step-over resulted in exception.

I tried to use breakpoints to find part of code that potentially cause this exception, but working with breakpoints seems unstable (sometimes running the code until breakpoint worked well, sometimes it crashed the ARM Development studio, sometimes it made program jump at address 0 and required rebooting the board).

In this thread the same behaviour was described (where stepping through code worked, and running it resulted in exception), and the suggested solution was to introduce 2 ISB instructions, but from what I see in the "rebuild-newlib" I used, these 2 ISB instructions are already in the following file:

newlib/libgloss/aarch64/crt0.S

I checked system registers after the program went to "curr_sp0_fiq" (following F6/step-over functions) and the values were always the following:

• _cpu_init_hook is actually defined here. The definition you were looking at is a fallback (weak symbol), in case it is not otherwise defined.
• The ESR_EL2 value in your screenshot corresponds to a data abort, with the DFSC (0x2a) indicating a capability bound fault (DS should give you the decoding if you click on the + icon next to it). The address at which the access (write) failed is indicated by FAR_EL2, and I would guess that is somewhere on the stack. Maybe a store via CSP, whose bounds are not appropriate? The address of the instruction at which the fault occurred is indicated by ELR_EL2. That might tell you enough to figure out what happened.