Home
September 15, 2019

Portably Cross Assembling and Linking Raw ARM Instructions

Building a cross compiler for a desired target architecture is a rite of passage – a dense, dark thicket of forest that the village elders push you towards with only a high-level discourse on expectations, leaving you responsible for the particulars of your own existence. After reading “How to Build a GCC Cross-Compiler” by one such elder, I felt enlightened for approximately three seconds. But after returning to the particulars of my own requirements, I immediately lost the path.

I enjoy a good round of mental masochism, but I really wanted to avoid building, from source, an entire cross compiler for ARM. I had two excuses:

  1. I didn’t actually need the compiler. In the jcompiler project, I have a nascent backend emitting raw ARM instructions. Therefore, all I really needed was as and ld.
  2. I wanted portability for use in CI builds. I needed whatever toolchain I used locally to be portable to Travis CI builds, and ideally not take forever to setup. crosstool-ng, although quite pleasant to use, took over an hour to build an ARM toolchain locally, and thus would likely have taken 2+ hours on a less capable Travis instance.

Package Manager: A Viable Shortcut?

At first I attempted to rely on apt‘s comprehensive package availability; it would have sufficed had my local system and the distributions offered by Travis aligned. The latest Travis instance, as of this writing, is Ubuntu 18.04 bionic, which offers gcc-8-arm-linux-gnueabihf as its latest version, while Debian unstable offers gcc-9-arm-linux-gnueabihf.

On the Travis machine, installing gcc-8-arm-linux-gnueabihf binutils-arm-linux-gnueabihf libc6-dev-armhf-cross was sufficient to assemble and link raw ARM instructions successfully. But what to do locally? I can hear you say, “Just install version 8 alongside version 9 locally.” And I appreciate your sage advice, fellow traveler. However, after doing so the issue I promptly ran into, and could not resolve, was the presence of multiple symbol definitions that ran arm-linux-gnueabihf-ld straight into the ground. Nothing I said or did made things right. Eventually I took a step back and realized I needed to find a portable, self-contained, pre-built toolchain that worked everywhere and could be setup quickly on the Travis instance.

The Linaro Toolchain

Stumbling upon the Linaro Toolchain Releases was my good fortune. I’m using amd64 hosts so gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf.tar.xz was the one for me. Unpacking that and giving the GCC contained within a simple C file went smoothly. But since I didn’t actually need the compiler front-end, I needed to find out how the front-end was calling the assembler and linker.

I first wrote out the simplest possible C file main.c, making sure to include the <stdio.h> library, because some of the ARM instructions emitted by jcompiler rely on the presence of symbols in that library:

#include <stdio.h>

void main() {
    printf("Why try harder?");
}

Then, after placing the path to Linaro in an environment variable:

export ARM_LINARO_CROSS_COMPILER_PATH="/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf"

I compiled main.c using GCC within the Linaro release:

$ARM_LINARO_CROSS_COMPILER_PATH/bin/arm-linux-gnueabihf-gcc -v -static main.c -o main

with -static to specify static linking, and -v for verbose because we want to know how the assembler and linker are called. That will dump a whole bunch of information to stdout; after eliding the nonessential parts, here’s what I got:

...
 /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/7.4.1/../../../../arm-linux-gnueabihf/bin/as -v -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -meabi=5 -o /tmp/cc7YqyyP.o /tmp/ccj4YQc0.s
...
 /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/7.4.1/collect2 -plugin /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/7.4.1/liblto_plugin.so -plugin-opt=/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/7.4.1/lto-wrapper -plugin-opt=-fresolution=/tmp/ccpO8WXE.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc --sysroot=/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc --build-id -Bstatic -X -m armelf_linux_eabi -o main /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib/crt1.o /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib/crti.o /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/7.4.1/crtbeginT.o -L/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/7.4.1 -L/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf -L/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc -L/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/7.4.1/../../../../arm-linux-gnueabihf/lib -L/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/lib -L/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib /tmp/cc7YqyyP.o --start-group -lgcc -lgcc_eh -lc --end-group /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/7.4.1/crtend.o /opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib/crtn.o
...

At first I was extremely confused by the lack of any calls to ld in this output. Eventually I learned that collect2 is calling out to the real ld, hence my elision of everything above except for the two calls we care about: the first to arm-linux-gnueabihf/bin/as, and the second to arm-linux-gnueabihf/7.4.1/collect2, both provided by the Linaro toolchain.

Now we can now clean this up and condense it into a two-statement script assemble-and-link-armv7.sh that will assemble and link any *.s ARM instruction file that we want a machine binary for:

#!/bin/sh

export ARM_LINARO_CROSS_COMPILER_PATH="/opt/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf"

$ARM_LINARO_CROSS_COMPILER_PATH/bin/arm-linux-gnueabihf-as \
        -march=armv7-a \
        -mfloat-abi=hard \
        -mfpu=vfpv3-d16 \
        -meabi=5 \
        -o $1.o \
        $1

$ARM_LINARO_CROSS_COMPILER_PATH/bin/arm-linux-gnueabihf-ld \
        -plugin $ARM_LINARO_CROSS_COMPILER_PATH/libexec/gcc/arm-linux-gnueabihf/7.4.1/liblto_plugin.so \
        -plugin-opt=$ARM_LINARO_CROSS_COMPILER_PATH/libexec/gcc/arm-linux-gnueabihf/7.4.1/lto-wrapper \
        -plugin-opt=-fresolution=/tmp/cc2CC7OQ.res \
        -plugin-opt=-pass-through=-lgcc \
        -plugin-opt=-pass-through=-lgcc_eh \
        -plugin-opt=-pass-through=-lc \
        --sysroot=$ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/libc \
        --build-id \
        -Bstatic \
        -X \
        --hash-style=gnu \
        -m armelf_linux_eabi \
        -o $2 \
        $ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/libc/usr/lib/crt1.o \
        $ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/libc/usr/lib/crti.o \
        $ARM_LINARO_CROSS_COMPILER_PATH/lib/gcc/arm-linux-gnueabihf/7.4.1/crtbeginT.o \
        -L$ARM_LINARO_CROSS_COMPILER_PATH/lib/gcc/arm-linux-gnueabihf/7.4.1 \
        -L$ARM_LINARO_CROSS_COMPILER_PATH/lib/gcc/arm-linux-gnueabihf \
        -L$ARM_LINARO_CROSS_COMPILER_PATH/lib/gcc \
        -L$ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/lib \
        -L$ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/libc/lib \
        -L$ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/libc/usr/lib \
        $1.o \
        --start-group -lgcc -lgcc_eh -lc --end-group \
        $ARM_LINARO_CROSS_COMPILER_PATH/lib/gcc/arm-linux-gnueabihf/7.4.1/crtend.o \
        $ARM_LINARO_CROSS_COMPILER_PATH/arm-linux-gnueabihf/libc/usr/lib/crtn.o

For example, given this in mystery.s:

.arch armv7-a
.data
intfmt: .asciz "%d"
nlfmt:  .asciz "\n"
spacefmt:  .asciz " "
.text
.global main
.extern printf
.syntax unified
main:
push {ip, lr}
mov r7, 8
sub sp, 4
str r7, [sp]
ldr r0, =intfmt
ldr r1, [sp, 0]
bl printf
ldr r0, =nlfmt
bl printf
add sp, 4
pop {ip, pc}

we can assemble and link like so:

$ ./assemble-and-link-armv7.sh mystery.s mystery

With stdout remaining gloriously silent, we proceed to $ ./mystery. Assuming the presence of package qemu-user-static on the host system, which provides a runtime emulator for ARM binaries, we expect to be greeted with:

8

CI Portability

This is all quite portable in a way that is easy to use on CI machines. Here’s all the relevant YAML one needs to get this setup on Travis:

before_install:
- sudo apt-get update
- sudo apt-get install -y qemu-user-static

install:
- export LINARO_CROSS_COMPILER_NAME=gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf.tar.xz
- wget https://releases.linaro.org/components/toolchain/binaries/latest-7/arm-linux-gnueabihf/$LINARO_CROSS_COMPILER_NAME
- sudo tar xf $LINARO_CROSS_COMPILER_NAME -C /opt/
← return home