👨🏻‍💻 Thanks for reading! This blog has been archived and won't have content updates. You can follow my latest work at trujillo.io.
CHROMABITS

Eduardo Trujillo
3 minutes

On a previous post, I covered how to setup continuous integration for Haskell projects using a combination of Stack, Travis CI, and Docker. But what about documentation?

Sweet auto-generated docs!

If you already have CI setup for your project with Stack and Travis CI, it is actually pretty easy. In fact, you can make use of GitHub’s Pages feature, which hosts statics sites based on the content of the gh-pages branch to host your documentation for free.

The following is a bash script I’ve been using on a couple of projects on GitHub. It takes care of collecting the documentation and coverage reports generated while building the application, triggered using Stack’s --haddock and --coverage respectively.

#!/bin/bash

# .travis/assemble-docs.sh
#
# Make sure you install the Travis CLI and encrypt a GitHub API token with
# access to the repository: `travis encrypt GH_TOKEN=xxxxxxx --add`.
#
# This script is meant to be run during the `after_success` build step.

# Copy haddocks to a separate directory.
mkdir -p ../gh-pages
cp -R "$(stack path --local-doc-root)" ../gh-pages
cp -R "$(stack path --local-hpc-root)" ../gh-pages
cd ../gh-pages

# Set identity.
git config --global user.email "travis@travis-ci.org"
git config --global user.name "Travis"

# Add branch.
git init
git remote add origin https://${GH_TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git > /dev/null
git checkout -B gh-pages

# Push generated files.
git add .
git commit -m "Haddocks updated" git push origin gh-pages -fq > /dev/null

To do its job, the script relies on some neat tricks/hacks to keep the process as simple as possible:

  • Stack: When using --haddock and --coverage flags, Stack will place documentation and coverage reports on specific paths. You can query these paths using stack path. On the script above, we use special flags so the output of the program is a single line with the requested path. This avoids having to think about or trying to find where the compiler placed the documentation and related files.
  • GitHub API Tokens: Managing SSH keys inside a build job is probably something doable but probably not easy. Thankfully, GitHub allows you to push commits to a repository using just an API token.
  • Travis CI encrypted variables: This allows us to conveniently store the aforementioned token in a secure manner and easily access it as an environment variable while the job runs. We do have to use > /dev/null on a couple of places so the key is not leaked on build logs.
  • Bare Git branch: Given that keeping track of history is not a priority and could break the build process if somebody else pushed to the documentation branch, we simply keep a single commit on the gh-pages branch. One can easily do this by initializing a new repository, committing, and force-pushing into the branch.

If you would like to see a public working example, checkout this repository and its build logs on Travis CI. The resulting documentation is available as a GitHub pages website and coverage reports can be found under /hpc on the site.


Eduardo Trujillo
2 minutes

After using Gentoo for a while, I’ve noticed that while the main portage tree has many up-to-date ebuilds, there are still some that lag behind upstream.

The good news is that, if you’re a bit adventurous, you can add overlays, which, like the name implies, are ebuild trees that are overlaid on top of your local tree.

Overlays usually contain software or package versions that are not available on the main tree. However, it also means that they might not have gone through the same amount of quality assurance than packages from the main tree.

Overlays may be personal or maintained by a small group of developers. They can also exist just locally on your machine or they can be hosted on a git repository.

There’s also tools like layman which can help you manage overlays used on a machine.

So far, my favorite overlays are gnome, steam-overlay, and haskell, which provide a recent version of GNOME (3.20), Steam, and up-to-date Haskell packages respectively.

However, not everything has an ebuild, even if you look on overlays, so you might have to write your own ebuilds.

I’ve been collecting a couple of these and putting them on my own personal overlay hosted on git. This has allowed me to version them and share them across Gentoo machines.

If you are feeling adventurous, you can try it by adding the following file to /etc/portage/repos.conf/chromabits.conf:

[chromabits]
location = /usr/local/overlay/chromabits
sync-type = git
sync-uri = https://phabricator.chromabits.com/diffusion/PORT/portage.git
auto-sync = yes

However, I cannot guarantee that they will build, install correctly, or not start a fire given that I lack the time resources to perform something more advanced than “It works on my machine(s)” in terms of QA.

Here a quick summary of a few goodies included:

  • iio-sensor-proxy: Allows GNOME to automatically adjust the screen brightness.
  • nm-applet: A recent version (1.2) of the NetworkManager applet.
  • corebird: A recent version (1.2) of a great GTK+ Twitter client.
  • slack: The official Slack client for Linux.
  • networkmanager: The latest version of NetworkManager, which has nice features like Wi-Fi power save and MAC address randomization.
  • softether-vpn: The SoftEther VPN server and client.
  • syncthing-gtk: A GTK GUI for Syncthing.

Eduardo Trujillo
5 minutes

At my current job, we are starting to adopt Haskell to write some of our backend APIs and, like other projects at the company, we are using Docker for deploying them to staging and production environments.

Working with Haskell required some re-thinking on how we do certain parts of our workflow, but it also led to a much improved image size, with some extra room for improvement.

haskell containers

Unlike PHP or JavaScript, Haskell is a compiled language, meaning that some of previous approaches we were using for building Docker images did not port that well.

For dynamic languages, we followed roughly these steps to build our images:

  • Use a base image with packages for the software we need (Debian, Alpine).
  • Install server software, such as Nginx, Node.js, and PHP.
  • Install language tools and package managers.
  • Copy application sources.
  • Download dependencies.
  • Final optimizations and cleanup.

This is a hybrid approach where we have both build and runtime dependencies on the same Docker image, which is useful because image builders such as Quay.io can automatically build the docker image for you on every commit without the need of an additional step in the CI pipeline, and this also has the slight advantage of having enough tooling inside a container for light debugging.

As a result, the image size is not the smallest it could be since there is a lot of things that are not commonly used during runtime. However, the added convenience sort of out weighs the issues of having a larger image for these cases.

For Haskell projects, though, things are bit different, and not in a bad way:

The community has made an excellent build tool called Stack. stack takes care of mostly everything related to setting up your project: Installing GHC, pulling dependencies, building, testing, coverage reports, documentation. When paired with Nix, it can even pull non-Haskell dependencies for reproducible builds.

Stack takes care of mostly everything related to setting up your project.

If we try to do a hybrid image approach like above using Stack, we mainly have to do the following on a Dockerfile:

  • Download and install Stack.
  • Copy application sources.
  • Install GHC (stack setup).
  • Compile project (stack build).

This works, but it is extremely slow and the resulting images are huge (+800MB!).

On every Docker build, stack would have to download and install GHC, and then proceed to download and compile every dependency of the project, which tended to a good 30 minutes on a Quay.io worker node.

When developing locally, you only have to go through this process every now and then because most of it is cached in directories such as ~/.stack and .stack-work.

Looking for faster build times and smaller images, I decided to experiment with splitting the build and runtime aspects of the project.

The build part was already setup since we were using Travis CI for running unit and some integration tests. When compared to basic Docker image builders, Travis CI has the clear benefit of having a cache that can be reused across builds without too much work. This cache allowed us to keep our built dependencies across builds, which reduced the build time to under 5 minutes.

This enable caching of Stack builds, you just need to add the work directories to the cache section of .travis.yml:

cache:
  directories:
    - '$HOME/.stack'
    - .stack-work

So, getting the runtime half working meant taking the resulting build from the previous steps and building a Docker container with enough dependencies and data files for running the application.

FPComplete has a great article out there on how to create minimal Docker images for Haskell projects.

The main difficulty with this process is that Haskell programs are not built statically by default, meaning that you have to identify all the libraries the binary is linked against and include them in the final runtime image.

The main difficulty with this process is that Haskell programs are not built statically...

In order to keep things simple, I decided to stick to using a base image, which we could use to pull in any dependencies we don’t have, like libcurl-dev.

I initially tried to use Alpine, since its known for being one of the smallest images out there. However, getting a Haskell program running in it was not trivial since it requires cross-compiling GHC.

So I settled with debian, which is a larger image, but has almost everything we need out of the box.

Building a Docker image on Travis CI is a fairly simple process. Pushing it to a registry and correctly tagging it was the annoying part. After a couple of hours of trial and error, I made a small shell script for authenticating with the registry and pushing a tagged image matching the current git tag and branch.

This script is called on the after-success step of the Travis CI build:

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

docker build -t myapp .

# If this is not a pull request, update the branch's docker tag.
if [ $TRAVIS_PULL_REQUEST = 'false' ]; then
  docker tag myapp quay.io/myorg/myapp:${TRAVIS_BRANCH/\//-} \
    && docker push quay.io/myorg/myapp:${TRAVIS_BRANCH/\//-};

  # If this commit has a tag, use on the registry too.
  if ! test -z $TRAVIS_TAG; then
    docker tag myapp quay.io/myorg/myapp:${TRAVIS_TAG} \
      && docker push quay.io/myorg/myapp:${TRAVIS_TAG};
  fi
fi

As a result, we now have Docker images for our Haskell projects that are about 80 MB, which is not terrible, but can definitely be improved on.

The next steps for me are investigating how to make our images even smaller by using a smaller base image, and automate the deployment of development and staging environments by having Travis CI notify a scheduler that a new image has been built.

I’m including some of my scripts and an example Dockerfile on a GitHub Gist for reference. You will most likely have to modify them to meet your needs.

…or you can find more in the archives.

CHROMABITS
Copyright © 2015-2021 - Eduardo Trujillo
Except where otherwise noted, content on this site is licensed under a Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.
Site generated using Gatsby.