FreeBSD on ARM64


#1

So I’ve been thinking about this since 2017, but only yesterday signed up for an account and played around with the ARM64 offering.
Turns out it’s pretty great! KVM boots into UEFI, there’s a local VirtIO disk attached, no NBD junk required. So we can definitely run FreeBSD.

I managed to “depenguinate” a running instance, the notes are below.
Would be great if Scaleway offered an official image instead :wink:


For some reason, unlike on x86, mounting additional volumes is not allowed on ARM64 instances. So we’ll have to move the running Linux to a ramdisk using pivot_root and then we can do whatever to our one and only disk.

Spin up an instance with Ubuntu Zesty and ssh in.

Prepare the system and change the root to a tmpfs:

apt install gdisk
mount -t tmpfs tmpfs /tmp
cp -r /bin /sbin /etc /dev /root /home /lib /run /usr /var /tmp
mkdir /tmp/proc /tmp/sys /tmp/oldroot
mount /dev/vda /tmp/oldroot
mount --make-rprivate /
pivot_root /tmp /tmp/oldroot
for i in dev proc sys run; do mount --move /oldroot/$i /$i; done
systemctl daemon-reload
systemctl restart sshd

Now reconnect to ssh from a second terminal (note: rm the connection file if you use ControlPersist in ssh config), then exit the old session. Kill the old sshd process, restart or stop the rest of the stuff using the old disk:

pkill -f notty
sed -ibak 's/RefuseManualStart.*$//g' /lib/systemd/system/dbus.service
systemctl daemon-reload
systemctl restart dbus
systemctl daemon-reexec
systemctl stop user@0 ntp cron systemd-logind
systemctl restart systemd-journald systemd-udevd
pkill agetty
pkill rsyslogd

Check that nothing is touching /oldroot:

lsof | grep oldroot

There will probably be an old dbus-daemon, kill it.

And finally, unmount the old root and overwrite the hard disk with a memstick image:

umount -R /oldroot
wget https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/12.0/FreeBSD-12.0-CURRENT-arm64-aarch64-20180719-r336479-mini-memstick.img.xz
xzcat FreeBSD-12.0-CURRENT-arm64-aarch64-20180719-r336479-mini-memstick.img.xz | dd if=/dev/stdin of=/dev/vda bs=1M

(Look for the newest snapshot here, don’t copy paste the July 19 link above if you’re reading this in the future. Actually maybe use a release instead of CURRENT…)

Now, fix the GPT: move the secondary table to the end of the disk and resize the table.
It’s important to resize here, as FreeBSD does not do that and silently creates partitions that won’t persist across reboots :smiley:

gdisk /dev/vda
x
e
s
4
w
y

And reboot. (You might actually want to hard reboot here: for some reason on the first reboot from Linux, pressing the any-key to enter the prompt in the loader hangs the console for me.)

I didn’t have to go into the ESC menu and choose the local disk in the boot manager, it seems to boot from disk automatically.

Now we’re in the FreeBSD EFI loader.
For some reason, the (recently fixed?) serial autodetection from EFI is not working correctly. Or something.
So you don’t get console output by default.
To fix, you have to run these commands in the boot loader command prompt:

set console=comconsole,efi
boot

(Ignore the warning about comconsole not being a valid console.
Since there’s at least one (efi) that the loader thinks is valid, it sets the whole variable.)

(UPD: shouldn’t be necessary in the next snapshot)

Now it’s a regular installation process!
When asked about partitioning, choose Shell, and manually add a partition and set up a root filesystem:

gpart add -t freebsd-zfs -a 4k -l zroot vtbd0
zpool create -R /mnt -O mountpoint=none -O atime=off zroot /dev/gpt/zroot
zfs create -o canmount=off -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/usr zroot/ROOT/default/usr
zfs create -o mountpoint=/var zroot/ROOT/default/var
zfs create -o mountpoint=/var/log zroot/ROOT/default/var/log
zfs create -o mountpoint=/usr/home zroot/home
zpool set bootfs=zroot/ROOT/default zroot
exit

(In this example, I set up ZFS with a beadm-compatible layout which allows me to use Boot Environments.)

In the post-install chroot shell, fix some configs like so:

echo 'opensolaris_load="YES"' >> /boot/loader.conf
echo 'console="comconsole,efi"' >> /boot/loader.conf
echo 'vfs.zfs.arc_max="512M"' >> /boot/loader.conf
sysrc zfs_enable=YES
exit

(Yeah, for some reason, the loader does not load zfs.ko's dependency opensolaris.ko automatically here. idk what even. It does on my desktop and laptop.)

Now you can reboot into the installed system!!

Here’s how you can set up IPv6 (and root’s ssh key) auto configuration on boot:

pkg
pkg install curl
curl https://raw.githubusercontent.com/scaleway/image-tools/master/bases/overlay-common/usr/local/bin/scw-metadata > /usr/local/bin/scw-metadata
chmod +x /usr/local/bin/scw-metadata
echo '#\!/bin/sh' > /etc/rc.local
echo 'PATH=/usr/local/bin:$PATH' >> /etc/rc.local
echo 'eval $(scw-metadata)' >> /etc/rc.local
echo 'echo $SSH_PUBLIC_KEYS_0_KEY > /root/.ssh/authorized_keys' >> /etc/rc.local
echo 'chmod 0400 /root/.ssh/authorized_keys' >> /etc/rc.local
echo 'ifconfig vtnet0 inet6 $IPV6_ADDRESS/$IPV6_NETMASK' >> /etc/rc.local
echo 'route -6 add default $IPV6_GATEWAY' >> /etc/rc.local
mkdir /run
mkdir /root/.ssh
sh /etc/rc.local

One remaining issue: incoming TCP connections (like ssh, or simply testing with nc) from the public IPv4 address don’t work — I see incoming packets in tcpdump but no replies o_0 (IPv6 works perfectly, at least)

And to fix incoming TCP connections, configure the DHCP client to change the broadcast address:

echo 'interface "vtnet0" { supersede broadcast-address 255.255.255.255; }' >> /etc/dhclient.conf
killall dhclient
dhclient vtnet0

Other random notes:

  • keep in mind that -CURRENT snapshots come with a debugging kernel by default, which limits syscall performance by a lot, you might want to build your own with config GENERIC-NODEBUG
  • also disable heavy malloc debugging features by running ln -s 'abort:false,junk:false' /etc/malloc.conf (yes that’s storing config in a symlink)
  • you can reuse the installer’s partition for swap

P.S. Thanks to openattic.org for good information about Linux pivot_root!


FreeBSD on x86
How to run a custom kernel on a VPS (VC1S, etc.)
#2

@myfreeweb great job! :slight_smile:

Have you tried changing the broadcast-addess in /etc/dhclient.conf ?

interface "vtnet0" {
  supersede broadcast-address 255.255.255.255;
}

It works on x86.


#3

Thanks a lot!! Looks like I still have some things to learn about networking basics :smiley: