CheriBSD Morello FVP Pure-Capability - How to cross-compile from source?

Several points:

1. Downloading Ubuntu's package sources is not a great idea. If you download and extract the .dsc then you'll get all the Ubuntu patches applied plus all the debian/ folder containing Debian/Ubuntu packaging stuff that's irrelevant. If you just download the .orig.tar.gz/xz then you might as well just get the sources from upstream. Moreover, it's generally best to clone upstream git repositories; if you need to make modifications then you actually can do so easily and push to a fork, without being stuck without a VCS.

2. That page lists the binary run-time dependencies, not the build-time dependencies, which are on the *source* package page at https://packages.ubuntu.com/source/focal/curl. However, those dependencies are just for whatever Ubuntu's curl configuration needs, they don't reflect the minimal set required by upstream. Given the wide portability of curl my guess is you don't need any dependencies for the core functionality beyond what's in the FreeBSD (and thus CheriBSD) base system (which includes OpenSSL), but when looking at things like this it's generally best to look at upstream documentation as it often lists what's required and what's optional, rather than just an arbitrary list a distribution decided on.

3. You can absolutely use ./configure just as with any other cross-compiler, you just have to make sure you set CC, CFLAGS and LDFLAGS appropriately, and whatever else the configure script might need, to point at your Morello LLVM and tell Morello LLVM to target pure-capability Morello (and with --sysroot=/path/to/rootfs-morello-purecap so all the headers and libraries needed during the build can be found). The autoconf-specific part of this is widely documented online, and the Morello-specific flags you need are documented in Arm's Morello LLVM user guide at https://git.morello-project.org/morello/llvm-project-releases/-/blob/morello/release-docs-1.1/UserGuide.pdf. Of course, cheribuild deals with all this for you if you want to use that instead. The wiki page's libffi example should be a good starting point, as both curl and libffi are autotools-based projects whose git repositories require you to first generate the configure script (though curl doesn't have a script for it, the documentation just says to run autoreconf -fi).

Since it's a simple target to add and other people could plausibly want it, I just went ahead and added a curl target to cheribuild so you can now just `cheribuild curl-morello-purecap` and get a /usr/local/morello-purecap/bin/curl. The upstream source built and ran just fine as is, no changes needed. It's been lightly tested against http://www.google.com and https://www.google.com.

samDobson said:

Thanks so much for this information. Because I will be cross-compiling quite large projects, I think using the generated ./configure script is the best way to go for me. So I've cloned the curl git repo and ran autoconf -fi to generate all the configuration files. Now, let's say I'm wanting to compile for morello purecap, would I run something similar to:

./configure CC=aarch64-unknown-freebsd-clang --build=x86_64 --host=aarch64-unknown-freebsd13 --sysroot=/home/ubuntu/cheri/output/rootfs-morello-purecap

Or have I got the wrong idea here?

Yeah, that's the right kind of idea.

CC="$HOME/cheri/output/morello-sdk/bin/clang --target=aarch64-unknown-freebsd13 --sysroot=$HOME/cheri/output/rootfs-morello-purecap" CFLAGS="-march=morello+c64 -mabi=purecap -femulated-tls" LDFLAGS="-fuse-ld=lld" PKG_CONFIG_PATH= PKG_CONFIG_LIBDIR= ../curl/configure --host=aarch64-unknown-freebsd13 --with-openssl --with-ca-path=/etc/ssl/certs

works for me on my Mac, as a from-scratch set of minimal flags needed for curl. Note that --build is autodetected and --sysroot is an argument to Clang not configure. The PKG_CONFIG_* being set to the empty string are to avoid the configure script's calls to pkg-config finding my native system libraries for various things it searches for.

(Note also the use of `-femulated-tls`, which is a temporary workaround; until recently the thread-local storage ABI for Morello was entirely undocumented so we couldn't implement the run-time side of it in CheriBSD and instead resorted to emulated TLS. There's a pull request against the spec now to document the current ABI so we could support it in CheriBSD but the current ABI is more experimental than the rest of the toolchain and I raised some concerns with the current approach, so given emulated TLS works we're sticking with that for now until things are more settled. If you don't have `-femulated-tls` you may see the odd undefined symbol reference, in this case `_ThreadRuneLocale`.)

samDobson said:

Just for my own learning going forward, could you walk me through the steps you took to add the target to cheribuild? There are a lot more pieces of software I wish to add in the future, so knowing this process would mean it's less likely for me to have to keep asking questions here on this topic :)

The short answer is "find an existing target that does what you want and copy the bits you need from it". At this point, most things you could want to do are already done somewhere in cheribuild. In this case, I knew curl was using autotools as its build system so copied the skeleton for an autotools-based project from elsewhere, in this case libffi. We've never really documented cheribuild before because it's been a tool we've developed that users have previously just needed to use as-is, without needing new targets. But the general principles are:

1. Define a new class BuildMyTarget that inherits from CrossCompileFooProject, for some build system Foo (they're all in pycheribuild/projects/cross/crosscompileproject.py if you want to see what's supported).

2. Set the repository member to the Git repository

3. If needed, set the target member to the name of your target (by default, BuildMyTarget_Name gives a target name of mytarget-name, so in fact I didn't need to specify it for curl)

4. Set supported_architectures to CompilationTargets.ALL_FREEBSD_AND_CHERIBSD_TARGETS + [CompilationTargets.NATIVE] (hopefully self-explanatory) unless there's a good reason why that won't work

5. If your target depends on other targets (other than the implied default SDK targets, i.e. a compiler and sysroot) then set dependencies to a list of target names *without any architecture suffix*. If you're in one of the rare situations where the list of dependencies changes based on the architecture being compiled for then you can also declare dependencies as a @classmethod that returns the list and can dynamically create the list as needed.

6. Override the setup method if you need to pass additional environment variables, configure arguments or make arguments; self.configure_arguments is a list you can add to, some build system superclasses give self.add_foo_options methods (e.g. self.add_cmake_options(FOO="bar") will pass -DFOO=bar to cmake), self.configure_environment is a dictionary, self.make_args has a set method (for normal arguments) and an env_vars dictionary field (for environment variables, as you might guess, though slightly weird it's under a thing called make_args). If you just grep the sources for those various things it'll hopefully become clear

7. If you need to modify any of the configure, build or install steps beyond just arguments and environment variables then configure, compile (not build) and install are methods you can override to do whatever you need (e.g. see curl where configure needs to run autoreconf before running the normal configure method).

8. If you need to query information (where various directories are, what the target is, compiler flags) for something not automatically handled for you, the base SimpleProject class has a whole load of methods you can call (see pycheribuild/projects/project.py). self.target_info and self._xtarget (prefer the former if it has the information you need) also exist for additional information about the architecture you're compiling for (I always forget the exact difference between them, so just grep for examples of using both to find the information you need).

We have some ideas for how to make this easier (e.g. potentially adding a special set of `cheribuild build-current-directory-as-autotools-morello-purecap` targets so you can reuse the standard CrossCompileFooProject infrastructure in cheribuild without having to write a target for your project), but none of that exists today unfortunately.

samDobson said:

Once I have created the myproject.py file in cheribuild/pycheribuild/projects/cross, do I have to run the script with ./myproject.py

That will not work, and that's not how you use any of the other targets so that's not how you use an additional target you've added.

samDobson said:

or does the file only have to be in that directory so it can be used by cheribuild?

Any .py file in projects/ and projects/cross/ is automatically picked up.

samDobson said:

Another question, is there a way I can keep the machine online, so that I can close the terminal and not have to wait 10 minutes for ./cheribuild run-fvp-morello-purecap to run when I want to boot up again?

screen/tmux? There's nothing special about cheribuild in this regard.

Jessica Clarke said:

samDobson said:

Another question, is there a way I can keep the machine online, so that I can close the terminal and not have to wait 10 minutes for ./cheribuild run-fvp-morello-purecap to run when I want to boot up again?

screen/tmux? There's nothing special about cheribuild in this regard.

Although you may be interested in our Morello QEMU port. cheribuild morello-qemu will build it, and cheribuild run-morello-purecap (no -fvp) will run it using the same disk image. It is still a bit experimental but it has been able to run CheriBSD for a couple of months now. There are still some bugs in more obscure parts of the ISA, but rarely anything that should affect userspace development, and given it is many times faster than the FVP you may find it a much more pleasant environment to work in. We are using it in our day-to-day work rather than Arm's FVP, and it is being used in our CI environment for testing CheriBSD changes.