Tips for maintaining small to medium projects with Yocto

The Yocto Project is generally pretty good at forcing you to have a clean infrastructure of your whole embedded Linux software, compartmentalizing everything from the bootloader, kernel, and userspace applications to your custom applications in well-defined meta-layers and recipes. 

But, for long-term projects, people do forget about some things that could make their future selves significantly happier engineers. Let me tell you three tips that I think are pretty important for the medium to long-term health of your project.

Tip #1: Keep the downloads folder cached somewhere safe

Yocto has a caching mechanism that downloads tarballs and source code from remote servers. This is done through the SRC_URI variable. Let's take a look at an example from the docker compose recipe from meta-toradex-torizon:

SRC_URI = "git://github.com/docker/compose;name=compose;branch=v2;protocol=https"

This will actually translate into a git clone being executed in the do_fetch() task when the recipe gets executed, and the result of this clone is stored in a local folder called, you guessed it, downloads:

➜  downloads ls | grep docker.compose

git2_github.com.docker.compose.tar.gz

Now, what happens if GitHub is down? Or if the repository migrated? Or if literally, everyone thought having 'master' as the default branch name is unacceptable and switched to 'main'?

Simple: cache your downloads folder somewhere safe and you'll always be able to reproducibly rebuild your whole system reproducibly, just by updating the DL_DIR variable in your local.conf. It'll barely cost 50~60 gigs at maximum.

Tip #2: Keep your changes in your meta-layer

There's this phenomenon I like to call 'repository mania' which is when a team won't stop creating new git repos for literally everything. 

It can start by, for example, wanting to change how a recipe behaves from their standard board support package, and instead of creating a new meta-layer with a .bbappend, developers fork the upstream meta-layer and introduce changes from there. This is a very bad pattern because if there are upstream updates, you will have a very hard time rebasing and re-testing everything.

If you fork something, it becomes your responsibility to take care of it.

The Yocto Project has a number of mechanisms to help you keep changes within your layer: kernel fragments, kernel features (especially useful if you have multiple targets within a family), and applying patches instead of forking, just to name a few.

Tip #2: Use containerized builds with CROPS in a dedicated, on-premise, server

Seriously, it's dead easy to build a Yocto distribution using containers, in this case, crops poky-container.

People in embedded tend to not want to adopt these new technologies, but by using a container and any software like Jenkins, GitLab CI, or GitHub Actions, within 10 minutes or less, you get amazing infrastructure that you can schedule to run every night to provide your team and clients with nightly builds, for example.

And you get Tip #1 for free if you mount the downloads folder and sstate cache from a volume.

Now, why use an on-premise server? Two reasons: 

1. It's easier to include automated integration testing if your build server is a physical machine that you can control and physically connect boards to. This should work really well for a small to medium-scale project with a few board variations, as I have done for a couple of years already.

2. It's super easy to onboard new team members and if you're physically close (or have a very good internet connection), it's probably faster to make your local build point to use the on-premise server as a sstate mirror than waiting the whole day for your first build to finish. This is also a pretty good idea because, if setup properly, the cache gets updated dynamically with every build, and the build system is smart enough to generally not break everything.

And that's pretty much it. Shoot me an e-mail if you think I forgot something. Hopefully, I can write more about the technical aspects of testing with hardware-in-the-loop someday, but that's probably a much bigger topic than this short article. If you're interested in that, I highly recommend the Automated Testing and Continuous Integration/Delivery: Lessons Learned talk, which partially inspired this blog post.