CI/CD for FreeM on Real UNIX

FreeM is a highly portable M implementation. As of this writing, it builds and runs on the following systems:

  • GNU+Linux (Ubuntu, Debian, Slackware, OpenSUSE, Raspbian) on i386, armv6l, armv7l, aarch64, x86-64, and s390x
  • Sun/Oracle Solaris 10 and 11 on i86pc and sparc64
  • HP Tru64 UNIX (a.k.a. Digital UNIX, a.k.a. OSF/1) on alpha
  • SCO OpenServer 5.0.7 on i386
  • IBM AIX 5L 5.1 on ppc
  • GNU HURD 0.9 on i386
  • NetBSD/amd64
  • OpenBSD/amd64
  • FreeBSD/amd64
  • Mac OS X on amd64

As the current FreeM maintainer, and an avid retrocomputing enthusiast, I am committed to supporting all of these–and more–permanently. However, being a single developer, building and testing each of these architecture/OS combinations for each increment of the FreeM codebase is a hugely difficult task if done in a manual way. CI/CD platforms (like GitLab CI, Jenkins, and Rally), have no build agent support for many of these systems, and even getting SSH working can be a real challenge–and when you do, you may not have the ability to support the most modern encryption protocols.

Yet, a solution was needed. I would have to develop such a solution myself.

I began investigating the problem early on in my stewardship of the FreeM codebase, and decided that I needed to find out the lowest common denominator of automation, networking, and scripting capabilities all of these systems could support. This is what I arrived upon:

  • TCP/IP (using IPv4) is universally available
  • All of them have some support for cron
  • NFS v2 or greater, though NFS v3 is spotty and NFS v4 is rare
  • Vanilla Bourne shell (some variant of ksh is also relatively common, but I saw no reason to dig into its specifics, as all of the ksh variants will support vanilla Bourne shell constructs if you’re careful)

FreeM is developed from a locally-hosted GitLab git repository. It became obvious early on that doing a git pull as a core mechanic from each build host of my CI solution would not be feasible, as the git software has extensive prerequisites that many old UNIX systems are incapable of providing.

A central file server, using NFS v2, exports a filesystem for use by all build farm hosts. It contains a master list of build farm hosts in a file called servers.list, each line of which contains the short hostname (equivalent to hostname -s) of one build farm host. The filesystem also has a subdirectory corresponding to each of the build farm hosts, where the code of FreeM will be deposited and worked on. Each build farm host mounts this filesystem at /var/nas/freem-build.

There are a number of files corresponding to the current build status on each host (success, failure, running), of which only one will ever exist concurrently. Each host also has a log file containing the output of the most recent build attempt, and potentially a file to indicate that a build has been requested on that host.

I developed a series of Bourne shell scripts:

  • frm-bldall will request a build on all build hosts in servers.list by creating empty files with a certain naming convention in /var/nas/freem-build
  • frm-bldlog will display the latest build log output for a particular build host
  • frm-bldstat will display the build status (success, failure, running) for a particular build host
  • frm-build will attempt a configure and make on the current host
  • frm-chkbuild will check for the existence “build requested” file for the current build host in /var/nas/freem-build, and run frm-build if it exists (run from each host’s root user crontab every five minutes)
  • frm-cloneall will git clone the FreeM code repository for all build farm hosts (run from a post-commit hook on the GitLab server when a tagged release is pushed)
  • frm-commit will make sure FreeM can be successfully built on the local machine, and if so, will update the semantic version info in the appropriate files, update the change log, prepare a tagged commit, and push it to GitLab, which will run frm-cloneall and frm-bldall in its post-commit hook
  • frm-reclone re-clones the source repository for a requested build host (will not run if the requested build host is currently running a build)
  • frm-reqbuild requests a new build from a specific build host

The various elements generated by this CI system are also used to populate the build status page on the FreeM website.

The system, while already quite useful, has a number of glitches still to be ironed out:

  • Since all the build hosts run the build as root, there are permissions issues that have yet to be ironed out. In a future release, there will be a user account for the CI system with matching UIDs on each system.
  • There are occasional race conditions.

Eventually, I will enhance the system to be more generic (supporting projects other than FreeM), and also extend it to generate native binary packages for each supported platform.

In spite of GNU+Linux dominance, I am committed to supporting 1990s-style portability across all UNIX systems, and I hope that these tools will eventually enable others to do the same for their own projects.