Automating Hugo with GitHub Actions on a Raspberry Pi

Over the last few months, I've been studying and working on getting a website up with Hugo, a static site builder based on Markdown files. It doesn't take months to learn Hugo--it's pretty straight-forward and there's a great YouTube video course that gets you up to speed quickly. But I also took time to learn the Go programming language in which Hugo is written and built a program, first in Go then in Delphi (and shared at ODUG), to build content files more efficiently than hand-editing. Now, as the content files are built and website assets are put in place, I need Hugo to build the entire site locally and upload the generated HTML/JS/CSS files to the website whenever there's new content. Naturally, I wanted to automate this as much as possible.

The files Hugo needs are stored in a repository on GitHub which offers something called GitHub Actions. GitHub Actions are triggered by certain events to launch pre-defined processes--such as rebuilding and uploading files for a website. The trigger in my case needs to be any push of new commits to the repository. When that happens, the GitHub Action launches a GitHub Runner and processes steps in a defined Workflow. 

You can use GitHub's standard runners that spin up containerized virtual machines, download your repository, install prerequisite tools and libraries, build and deploy your app or website, then destroy itself; or you can use your own self-hosted runners installed and running on your own hardware. Using GitHub's runners will incur charges after a certain amount of free monthly minutes are used and since I'm building a personal photo album with thousands of pictures and frequent updates, I know it would quickly exceed GitHub's usage limits. So I dove in the deep end to figure out how to configure Actions and Workflows and set up my own self-hosted Action Runner.

This was quite a learning curve but after reading lots of documentation and blogs and wading through some trial-and-error, I got it working on an old Linux box I had sitting in the corner of my office. Once the concept was proven, I decided to move everything to a Raspberry Pi, partially to save space (I wanted to get rid of spare computers sitting in office corners) and because I wanted to see if it could be done. I took notes along the way and now that it's done and working, I wanted to officially document the steps in case I ever need to do this again--and in case anyone else is interested in what's involved.

Selecting the OS

The operating system of choice for a Raspberry Pi is often Raspbian because it has fairly low resources and provides the basics most users would like to have to get started exploring their small Linux device. But getting a GitHub Actions-Runner installed on Raspbian is problematic--it's not impossible if you understand how to manually install special versions of .NET core, but I didn't want to spend that much time if there was a simpler path. Also, once set up, I won't need a GUI at all which would take extra CPU and memory, precious resources on a Raspberry Pi.

So, I chose Ubuntu Server. Ubuntu is very popular and thus is supported by a wide variety of applications, including self-hosted GitHub runners; plus, I don't need a GUI as command-line driven, automated processes don't need to be constantly monitored once they're up and running (besides, the GitHub Action web interface has a pretty good real-time view of actions being processed, I found out). For my Raspberry Pi 3, I used the 64-bit ARM version and with the handy Raspberry Pi Imager, soon had a bootable microSD card ready. 

Setting up the User

Once booted up on the Pi, I logged in as the default user; for a new install of Ubuntu, the username is just ubuntu. I could've stuck with that but I wanted to use a personal login so created my own account while logged in as the initial ubuntu account:

sudo adduser david

Since I will be needing to run commands as a root user and since, by default, new users are not trusted, I needed to add my new user account to the list of users that can run sudo:

sudo usermod -aG sudo david

Again, these steps are completely optional but I wanted to make it consistent with other Linux environments I have.

Attaching an External Drive

The amount and size of files I will be dealing with will quickly exceed the available space on the 64-GB SD card I am using to boot the Raspberry Pi so I attached a 500 GB external drive via USB. To get this to be automatically mounted each time the Pi is started, I had to add an entry to the /etc/fstab file. I won't go into all the details of how to do this because I followed the well-written instructions I found at this site: How to Write an fstab File on Linux. One word of warning from personal experience: be sure and test your addition to the fstab configuration before rebooting because if you mess that up, it's possible you won't be able to start the OS. There are ways to mount and modify the configuration from a Windows computer but it's easier to just re-image the SD card (don't ask me how I know!).

Installing Hugo

The Hugo site recommends using snap but it restricts folder use to your user's folders. If all you're ever going to do is run it from your own user folders, then I suppose that's fine but I'm using an external drive mounted outside of my user area. Besides, installing Hugo on Ubuntu is a very simple step:

sudo apt install hugo

Adding an SSH Key for Your Git Repository

If you use SSH to access your git repository (which is highly recommended) you need to set up an SSH key and add it to your repository. In order to prevent being prompted for an SSH key, you need to create a deploy key.

• On the Raspberry Pi, run ssh-keygen (do not set a passphrase--it's only used on this one machine and needs to be automated; yes, there are ways to automatically load a passphrase and avoid the prompt but it's very tricky and not necessary as this is localized to this one repository, one machine, and it's a read-only key)
• You need to upload the generated public key to your GitHub account. The easiest way for me was to copy the key file (.pub) to a removable USB disk (or thumb-drive) that I temporarily mounted in the Raspberry Pi, then move the thumb-drive to my Windows machine where I could open the key file in a text editor.
• In your GitHub repository's settings' sidebar, under Deploy keys, click Add deploy key. Then copy the text from the key file and paste it into the new deploy key area with an appropriate title.
• Back on the Raspberry Pi, you need to load the private key when it boots so add the following lines to the .bashrcin your user's $HOME folder:

if [ -z "$SSH_AUTH_SOCK" ] ; then
  eval `ssh-agent -s`
  ssh-add
fi

Installing the GitHub Runner

Finally, it's time to download and configure the GitHub action runner. There are clear instructions that are dynamically customized for your repository on the GitHub website. Under your repository's Settings area, expand Actions in the sidebar, and click Runners. This will bring up a list of your currently installed runners--none, at first, of course. Click New self-hosted runner and you'll see a script with three platform buttons above it, macOS, Linux, and Windows, along with an Architecture drop-down. As you click each of these platform buttons and select the architecture corresponding to the computer where your runner will live, the command-line scripts to will change.

The commands are rather long and since the Raspberry Pi doesn't have a GUI, you'll want to copy these to a text file, transport the text file to the Raspberry Pi on the thumb-drive in the same manner you brought the public SSH key file over. (Of course, if you're really adept at using a command-line oriented web browser on Linux, you can skip this step.)

One thing to remember when creating text files on Windows and porting them to Linux is that standard text files on Windows save both CR and LF for each line whereas Linux uses only LF. To execute these scripts directly on the Raspberry Pi, you'll want to make sure they're using the right text format when saving them on Windows. Most text editors have a simple option to change this.

Once the action runner scripts are downloaded to your Raspberry Pi, run them to install, configure, and start your runner. Then refresh your web browser's view of the action runners associated with your GitHub repository and notice the new registered runner.

Creating a Workflow

Now that you have an Action Runner running, you need to tell GitHub two things:

• What triggers it?
• What to do when it launches?

These are both defined in a Workflow file (written in YAML) that resides in the .github/workflows folder of your repository. You can have multiple workflow files in a repository and they can specify different action runners to launch and be triggered by different events. For my case, it was simple: one workflow that launches one runner.

The workflow file starts off with a name to identify it, declares the GitHub event that will trigger it, then lists one or more jobs it will run, each with one or more steps. Here is the one for my personal photo website:

name: Update Personal Photo Website
on: [push]
jobs:
  rebuild-website:
    runs-on: self-hosted
    steps:
      - name: Refresh website files from repository
        run: cd ${{ vars.LOCAL_REPO }} && git pull origin master --recurse-submodules
        shell: bash
          
      - name: Build HTML files for publishing
        run: cd ${{ vars.LOCAL_REPO }} && hugo --minify
        shell: bash

      - name: Push to the web
        run: ${{ vars.LOCAL_REPO }}/upload.sh ${{ secrets.PHOTO_SITE_HOST }} ${{ secrets.PHOTO_SITE_USER }} ${{ secrets.PHOTO_SITE_PASS }} ${{ vars.LOCAL_SRC }} "/"
        shell: bash  

The first line is the name of the workflow, "Update Personal Photo Website".

The next line defines the trigger event, in this case [push] or when GitHub detects I've used the git push command to upload commits from my local repository. There are several other events you can choose from.

Then comes the list of jobs. For my simple workflow, there's just one job: "rebuild-website" which specifies that it runs only on a self-hosted action runner and has three steps. Each step has a name and the details of how to accomplish that step. There are many things you can do in jobs and many pre-defined processes that can be launched in these steps; again, GitHub's documentation on Using Jobs is pretty thorough. Since I'm controlling everything on my own machine, I wrote out the commands explicitly using the run keyword and specified the environment (or shell) in which it runs. Each run starts the shell afresh--when you change directory in one run command, it does not stay in that directory for the next command; that's why each of these three run statements either starts off with a cd command or references a file with the full path and filename.

Note the last run command launches a script, upload.sh. This script runs an ftp client to upload files and it took a bit of testing to get it to work before I even added it to this workflow, so I just left it in a script and called it from this workflow. This shows the flexibility of using GitHub Actions--they can use pre-defined standard actions, or open-source scripts others have written, or low-level Linux commands--it all depends on your needs, your environment, and how much time you have!

Using Secrets and Variables

Each of the run commands reference ${{ secrets.___ }} or ${{ vars.___ }}. These get replaced when the workflow runs with values you've defined in a special section of the settings of your repository. Let's say you have a team of several developers and they use a local in-house website for their testing but they don't know the login credentials for the public FTP site--only the manager knows that. The manager can add special "secret" values that require extra authentication to change without exposing this critical information to a wider audience. Additionally, "variables" can be defined that are not hidden but may be used to customize a particular workflow that is used or copied for multiple purposes.

Testing Your Workflow

Now that your workflow is established and your self-hosted runner is registered and listening for triggered events, you are ready to test! There are two ways to do this, the first is obvious: you can perform the action that causes the workflow to be triggered, such as pushing a commit to the repository if, like mine, its trigger event is [push]. Additionally, you can go to your repository's settings sidebar, click on Actions, select the action, and launch it manually.

While it runs, you can watch the progress of various jobs and steps on the GitHub page as the workflow progresses; it also counts up the amount of time that has elapsed. Here's a screenshot of a finished job run:

I had to make several changes to my scripts, the workflow, and even a configuration change in the Hugo files until I got everything working consistently. Running Linux commands through an action runner is different than running them yourself from the command-line, so logging becomes an invaluable aid in trouble-shooting.

There were two things in particular I needed to change because I was working on a Raspberry Pi rather than a desktop computer with more resources. First, the timeout setting in the Hugo configuration needed to be increased to several minutes from its default of 30 seconds because of the large amount of image files it was processing for my photo site on the slower CPU.

Second, it ran out of memory.

Adding Virtual Memory

The Raspberry Pi 3 only has 1 GB of RAM. I am using very little of the 64-GB SD card used to boot and I don't care about the performance hit using a swap file will take because it's just a back-end process that runs occasionally when I upload a new photo album--I just don't want it to crash! Generally, the size of the swap should be at least equal to or twice the amount of RAM on your system, but it can also depend on your specific use case. For me, I added 4 GB, four times the amount of RAM (this may be overkill but it doesn't crash any more!)

Here are the steps to create a swap file that adds virtual memory:

• Create a 4 GB swap file: sudo dd if=/dev/zero of=/swapfile bs=4M count=1024
• Restrict read/write permissions to the root user: sudo chmod 600 /swapfile
• Designate the file as a swap area: sudo mkswap /swapfile
• Enable swapping on your system: sudo swapon /swapfile
• Verify the swap file is now active: sudo swapon --show
• Enable the swap file at start-up by adding /swapfile none swap sw 0 0 to /etc/fstab

Conclusion

My Raspberry Pi 3 is now a very useful part of my office, sitting on a shelf with nothing more than a power cord, an external hard disk, and a network cable. But it allows me to simply commit new photos to my repository--and my website is magically updated! Later, as I convert some of my existing static sites from Drupal to Hugo, there will be several more websites added in like manner.