Korkeala.info - web log

Reproducible development boxes

2026-01-03T14:00:00Z

For some time I've experimented and looked for a good solution for building (reproducible) development boxes. These will be virtual machines aimed for development. In the beginning they will be local virtual machines, but in the future they could also be remote machines. Main reason for using the development boxes would be added security by separating and isolating projects/customer work from each other. Each VM would have projects for one customer and only required SSH/access keys. The actual builds are also run in isolated containers and have even more reduced access to resources. This will protect against supply-chain malware as they reduce significantly resources the malware can read, like SSH keys and access tokens. Here are requirements for my setup:

Easy way to create and destroy virtual machines
Have automated way to set up basic configuration (installed packages, network configuration and user accounts)
Share folders from host computer to the virtual machine

I will use multiple layers and tools for protection, using separate virtual machines is just one of them. In this post, I'll focus on automating the creation and management of these development boxes, in the next post I will describe some other methods I use inside the VM for isolating builds.

Automating local virtual machines

Long time ago I used Vagrant to have reproducible test environments on local virtual machines. This was the first tool on my list to test for automating virtual machine management. First thing I noticed when looking for suitable image for the VM was, that many boxes have not been updated for some time. Maybe the community is shrinking and not active anymore?

I created OpenSUSE boxes with Vagrant and used it for a couple of weeks. Vagrant is over 10 years old so it has quite a lot of documentation and examples. Simple things are easy to configure, like shared folders between host and guest. It also has good support running different provisioning tools to configure the VM on creation.

During the couple of weeks test run I trouble suspending and resuming my box, it seems every time I had to suspend/shutdown my laptop I had to also bring the Vagrant box up. Also, Vagrant isn't fast, I had to wait long time for certain operations on Vagrant box to finish. So I decided to look for tools.

Opentofu

Opentofu has a provider for creating KVM virtual machines. I use Opentofu on other projects, so this could be good solution. With little HCL-configuration I was able to create myself an OpenSUSE virtual machine. Searching for the right type of image was cumbersome, would be nice to have a better way to get a list of images to use. Again booting virtual machines took a while, though this mostly just my impression.

I had a script for sharing folders from host computers with SSHFS. It worked alright, though it disconnects now and then. While I was testing this solution, I found another interesting project for lightweight virtual machines, incus.

Incus

Incus is a successor of an older project, LXD. Incus is a platform running containers and lightweight VMs. The lightweight VMs seemed interesting option. Fedora also had packages for it, so I decided to test it out.

After setting initial configurations, I checked available images:

incus image list images

There are images available for most current Linux distributions. After choosing to run OpenSUSE Tumbleweed, I started a test VM to experiment different configuration options and sure the lightweight VM's booted very fast!

But initially they would not start properly, I could not connect to it via SSH or incus shell. I was also experimenting with different configuration options, so perhaps I had put a bad configuration option? I could not query information about the running system, although the incus documentation said I should be able to. After a while, I noticed others had similar issues: incus-agent fails to run in virtual machines after recent update. So the issue was with incus-agent on the virtual machine. After I was able to set a working incus-agent on the VM and proper cloud-init configuration, I could access it and start experimenting.

During testing, I noticed it is faster than previous tools I tested. Vagrant has different options to provision configurations for the boxes. With incus, I settled for cloud-init, which cloud variants of the incus images support. Incus also supports sharing folders from host computer with simple command (incus config device add <instance_name> <device_name> disk source=<path_on_host> path=<path_in_instance> shift=true) and if I need further provisioning, I can execute pyinfra/ansible over SSH. It's not fully reproducible system like GuixOS or NixOS, but it is a good start for my current needs.

Incus provider for Opentofu

When I was writing this post, I noticed that incus has an Opentofu provider as well. I moved the configuration to the Opentofu to have a better management of the virtual machines. Now I can have the VM configuration in code, which makes provisioning similar development boxes trivial. Here is an example configuration for an OpenSUSE box:

terraform {
  required_providers {
    incus = {
      source = "lxc/incus"
      version = "1.0.2"
    }
  }
}

provider "incus" {
  default_remote = "local"

  remote {
    name    = "local"
    address = "unix://"
  }

}

resource "incus_image" "opensuse_cloud" {
  source_image = {
    remote = "images"
    name   = "opensuse/tumbleweed/cloud"
    type   = "virtual-machine" 
  }
}
data "template_file" "cloudinit" {
  template = file("${path.module}/cloud_init.cfg")
  vars = {
    user_account = "user"
  }
}

resource "incus_instance" "devbox1" {
  name      = "devbox1"
  image     = incus_image.opensuse_cloud.fingerprint
  type      = "virtual-machine"
  ephemeral = false
  profiles  = ["default"]
  config = {
    "user.user-data" = data.template_file.cloudinit.rendered
    "limits.cpu" = 2
    "limits.memory" = "8192MiB"
  }
  wait_for {
    type = "agent"
  }
}

And an example cloud-init configuration (cloud_init.cfg):

#cloud-config
package_update: true
package_upgrade: true
timezone: Europe/Helsinki
ssh_pwauth: true
packages:
  - git
  - openssh
  - helix
  - vim
  - helix-bash-completion
  - make
  - lazygit
  - git-delta
  - chezmoi
users:
  - name: user
    groups: users, wheel
    password: <hash>
    ssh_pwauth: true
    lock_passwd: false
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 <ssh_key_pub>

Current status and future

The incus VM has worked well this autumn. I use OpenSUSE Tumbbleweed as the OS, Opentofu to define the VMs and I have cloud-init configurations to have some automation for setting up the box.

The actual VMs haven't required much configuration, mostly just installing different development tools. After setting base configuration with cloud-init, I use Chezmoi to deploy same configurations for dotfiles. You can combine that with podman quadlets configured in user's home directory to have container services for databases and other services.

Next step is to install Guix package manager on to have isolated Guix shell environments. I would also like to see Guix image available for incus.

Signing and verifying Git commits with SSH keys

2024-06-02T20:30:00Z

SSH commit signing support was added to Git version 2.34.0. In this post I look at how to sign Git commits with SSH keys, preferably backed with FIDO2 security keys. There are different opinions whether signing commits is worth the extra effort. For example just recently Harley Watson wrote how commit signing is still kinda wack and Ludovic Courtès wrote on the importance of signing and verifying commits. This post is about the technical aspects of signing commits.

TLDR version

You can sign Git commits with SSH keys:

# git config --local gpg.format ssh
# git config --local user.signingkey ~/.ssh/id_ed25519-sk.pub
# git commit -S

To verify the Git commit create allowed signers file and configure your Git repository to use it. Then run git verify-commit:

# git config --local gpg.ssh.allowedSignersFile ./allowed_signers
# git verify-commit $commitid

With little bit of scripting you can verify all new commits in a Git branch:

commits=$(git log --pretty="%H" main..$branch)
for commit in $commits; do
   git verify-commit $commit 
done

You can use FIDO2 security keys (like Solokeys) with OpenSSH. Create a non discoverable SSH key with key type ed25519-sk or ecdsa-sk:

# ssh-keygen  -t ed25519-sk

SSH does not have similar Web of Trust (WoT) as the PGP ecosystem has. This limits the discoverability and distribution of SSH keys. In an organisation, you can use SSH CA to sign users SSH keys and make it easier to verify SSH signature and SSH key usage.

Getting started

Before you can sign Git commits, you need a SSH key. Probably you already have one, but if not, you can create one with the following command:

# ssh-keygen  -t ed25519

If you have a FIDO2 security token, you can create SSH key with SSH key pairing. See some notes about it in the Advanced usage section below.

Signing commits

When you have a SSH key, we can configure Git to sign commmits with it. First configure your local Git repository to sign commits with SSH keys and set your user.signingkey to the public SSH key. My examples use SSH keytype ed25519-sk:

# git config --local user.email <you_email@example.com>
# git config --local gpg.format ssh
# git config --local user.signingkey ~/.ssh/id_ed25519-sk.pub

After that make a Git commit with option -S, which is the option for signing commits.

# git commit -S

If you want to use this configuration in all your Git repositories, you can make the configuration global by replacing --local with --global.

Verifying commits

Verifying a Git commit does two things. First, the actual commit is verified against the cryptographic signature in the commit to see that the signature is valid. Second, it verifies that the commit signature is made with an allowed signing key. So now if you verify the commit, it should print something like this:

# git verify-commit <commitid>
Good "git" signature with ED25519-SK key SHA256:sxZkSe2TJJOKv3dQjadJ05RkCqXWWMB0HkCBU3D+pEM
No principal matched.

So the actual signature is valid, but it did not find a principal for it. It means Git does not know which keys are allowed to sign commits. When you verify Git commits signed with a GPG key, the signing keys are checked against the GnuPG keyring. The SSH ecosystem does not have similar Web Of Trust (WoT) to distribute and verify keys. You must define separately the SSH keys which are allowed to sign Git commits. These are defined in an allowed signers file. The format is defined in manual page for ssh-keygen. The file matches email addresses to the public SSH keys for users that are allowed to sign Git commits. You can also set valid-after and valid-before timestamps for the keys. Example for the allowed signers file:

# Comment here
user@example.org sk-ssh-ed25519@openssh.com AAA41...

Create the allowed signers file to the root of your Git repository as allowed_signers, use your email address, ssh key type and ssh public key in it. Then configure the Git repository to use it in the root of the Git repository:

# git config --local gpg.ssh.allowedSignersFile ./allowed_signers

After you have created and configured the allowed signers file you can verify Git commits with the following command:

# git verify-commit fe155257ece409c670c96a11e912538acf2b085d
Good "git" signature for user@example.org with ED25519-SK key SHA256:sxZkSe2TJJOKv3dQjadJ05RkCqXWWMB0HkCBU3D+pE

Advanced usage

Now let's look into more advanced usage. These include verifying Git commits in different use cases and ways to secure the SSH keys.

SSH keys paired with security keys

OpenSSH introduced support for SSH-keys backed with security keys in version 8.2. These two new key types are "ecdsa-sk" and "ed25519-sk":

FIDO/U2F Support
----------------

This release adds support for FIDO/U2F hardware authenticators to
OpenSSH. U2F/FIDO are open standards for inexpensive two-factor
authentication hardware that are widely used for website
authentication.  In OpenSSH FIDO devices are supported by new public
key types "ecdsa-sk" and "ed25519-sk", along with corresponding
certificate types.

So creating and using using these new key types requires FIDO2 hardware security keys: when you create a new SSH key it will pair the security key to our SSH key and the hardware key is required when using the SSH key. It essentially acts as MFA (Multi Factor Authentication) for your ssh key: if you have set a password for your ssh key then that is the first factor, something you know. Then the FIDO2 hardware token acts as a second factor, something you have. The security keys usually are small tokens, that you plug into USB port or use through bluetooth or NFC. Using the key requires some action, like pressing a button on the device or giving a PIN-code.

You can generate a security key backed SSH key with normal ssh-keygen flow, plug in the security key, run ssh-keygen -t ed25519-sk and follow the instructions.

You can also hook GnuPG keys to security keys that have smartcard features, like Yubikeys, but it requires quite a bit work. I've used drduh's yubikey-guide in the past and it is a frustrating process.

After you have created the key you can use it like a regular SSH key, but you have to have the security key plugged in. One caveat is that the new SSH key types will require support in the SSH-server as well, if you use to SSH connections. So services which use older OpenSSH versions or some other SSH server software might not support the new key types (I'm especially looking at you Bitbucket).

Verifying commits in pull-request flow

For verifying Git commits from a branch in a pull-request flow, you can use the approach below. It first fetches the main branch from the Git origin and loads allowed_signers file from the main branch. This makes sure that allowed signing keys are not updated in the pull-request branch, so the signing keys for new committers must first be added to the main branch before their changes can be accepted.

After loading allowed_signers, script gets commitIDs of new commits that are not in the main branch and verifies them.

git fetch origin main:main
git show main:allowed_signers > ~/gitsigners
git config --local gpg.ssh.allowedSignersFile ~/gitsigners
commits=$(git log --pretty="%H" main..$branch)
for commit in $commits; do
   echo "Verifying commit $commit"
   git verify-commit $commit 
done

If verify-commit fails for any of the commits, it should fail the build.

Guix git authenticate flow

Guix takes verifying Git commits further and offers way to authenticate the whole git tree from given Git commit. It also uses a separate file in the repository, .guix-authorizations, which has a list of allowed signing keys for users. The signing key must be added to file in previous commits before the signing key can be used to sign a commit. Currently guix git authenticate only supports PGP keys.

Below is a proof of concept of a bash function for similar behaviour. It iterates through the Git commits from the given commit id and verifies the commits. It fetches the allowed signers file from the parent commit and configures the allowed signers file to the Git repository for the verification. Saving the allowed signers file to a temp directory for the verification is quite hackish.

verify_commit() {
    # temp directory where extra files are stored for the verification
    local work_dir=$1
    #commit to start, for example: $(git log --pretty="%H" | head -n 1 )
    local commitid=$2
    # original signers file: $(git config --get --local gpg.ssh.allowedSignersFile)
    local orig_signers=$3
    local parents=$(git log --pretty=%P -n 1 $commitid)
    if [ -z "$parents" ]; then
        echo "Parent ($parents) empty, we are finished"
        echo "Set back allow signers file to $orig_signers"
        git config --local gpg.ssh.allowedSignersFile $orig_signers
    else
        while IFS= read -r parentid; do
            GITSIGNERS=$(git show $parentid:allowed_signers)
            #echo -e "GIT signers $parentid \n$GITSIGNERS"
            echo "$GITSIGNERS" > $work_dir/$parentid.gitsigners
            git config --local gpg.ssh.allowedSignersFile $work_dir/$parentid.gitsigners
            git verify-commit $commitid
            local ret=$?
            if [ $ret -ne 0 ]; then
               echo "Verification failed for commit $commitid"
               echo "Set back allow signers file to $orig_signers"
               git config --local gpg.ssh.allowedSignersFile $orig_signers
               echo "Removing directory $work_dir"
               rm -r $work_dir
               exit 1
            fi
            verify_commit $work_dir $parentid $signers
        done <<< "$parents"
    fi

}

SSH CA support

The allowed signers file supports also SSH certificate authority. In an organisation you can have a central CA which you use to create SSH certificates for users. The CA creates a SSH certificate by signing the given SSH key with certificate information, such as username identity of the user and certificate valid times. You can have the following in the allowed signers file to indicate SSH CA signing key:

# Organisations cert-authoriy, public key for for the SSH CA
*@example.org cert-authority ssh-ed25519 AAA...

Then if the Git commit principal matches the "*@example.org", Git tries to verify that the commit signing certificate is signed by the CA. Here is one article by Step CA about using SSH certificates in an organisation. This simplifies the allowed signers file and can be make verification easier in an organization, but requires setting up the CA for SSH keys.

Conclusion

After using GPG keys with Yubikey for couple of years, I've been impressed how easy Fido2 hardware tokens with SSH keys have been to use. After the SSH key signing support was added to Git, Git commit signing has been easy. Usually users already have SSH keys, which makes it easier to start to signing Git commits with SSH keys. One obstacle is that not all software in the Git ecosystem support SSH keys yet, like libgit2, but slowly different tools have added support for it. For the commit signing to succeed, better tools are required, not just for the signing but for the verification of commits as well.

Adding DNS based ad blocking to Openwrt

2023-05-18T20:00:00Z

I consider ads on the Internet a major hindrance. They usually are large and flashy (especially video ads) and hence distract user from the task at hand. They have are also being used to track users and even being used to spread malware. I have used ad blocking on the browser for a long time, currently I use UBlock Origin.

I became aware of the Pi-Hole-project a few years ago, that seemed like a good way to have ad blocking for all devices on the local network. It operates as a DNS-level ad blocker, so every DNS-request to the DNS-server is blocked if it matches an entry the ad block list. I myself use Openwrt on my routers and noticed it has adblock packages available that would make it operate like the pi-hole. OpenWRT has seperate AD-blocking guide you can use. I installed the following packages:

adblock
luci-app-adblock and luci-i18n-adblock-en (optional, for the LUCI web interface)

In addition to the adblock package, you will need SSL-support to fetch block lists over HTTPS. I have the following packages installed:

libustream-wolfssl20201210
libwolfssl5.5.4.ee39414e

After installing packages you can update the blocklist sources configuration, quite a few different sources are supported. Just be aware, including many big blocklist sources might slow the router down.

You will also want refresh the list from time to time. The adblock package refreshes the list on startup, so you can add crontab file to reload the adblock service. Like this:

0 06 * * *    /etc/init.d/adblock reload

I have been running the setup for few years and I am happy with it.

Linux on HP Dragonfly Elitebook G3

2023-01-08T20:00:00Z

I recently got myself a new laptop, HP Dragonfly Elitebook Max G3. Here are some impressions of the laptop and few thoughts about the Linux support for it.

TLDR version

For the busy people, here is the short summary: Linux support is good for the laptop, it is powerful enough for my development work and light, hence very portable. I had few issues with it, mainly getting the suspend to ram/resume to work, but otherwise I can recommend it. For longer description and issues I had with it, please read on.

Previous laptop

I do not upgrade laptops very often. My previous main laptop was over 3 years old, it was a Lenovo T480s. It started showing signs of failure, so before it would fail completely I decided to get a new one. I wanted a powerful, lightweight and easily portable laptop. Due to the COVID supply chain issues there were not many options available when I was browsing the local retailers. I was interested in Framework laptops, but at the time they did not deliver to my country. Previously I had been using Lenovo laptops and had been quite satisfied with them. Their Linux support has been getting better with programs like the preinstalled Linux on Lenovo. But then again they have a history of putting questionable software on their computers (see for example arstechnica article) or just search lenovo malware. So this time I decided to try other manufactor. With not many options, I decided on the HP Dragonfly Elitebook Max G3.

Hardware and Linux support

The hardware is quite nice, 32 gigs of RAM, touchscreen and pen support (pen included), 13 inch screen, videocam + shutter to hide the camera and enough CPU power for my development work. The laptop weights about 1kg, so very portable and easy to travel with.

I first installed Fedora 36 KDE Spin on the machine. I am a long time Gnome user, but I recently wanted to see how well KDE performs. Installation itself was a breeze, like it has been for years with Fedora and most things worked out of the box, like pen support, wifi and graphics. But two things did not work: the touchscreen and suspend/resume. As I searched the touchscreen issue it seemed that KDE does not yet support touchscreens, altough the pen worked fine. So I installed Gnome and switched back to the Gnome desktop environment, which supports touchscreen and pen input.

The suspend to RAM issue was a little bit trickier. I use the functionality a lot, so it was important to find a solution. After searching around and I eventually found a blog post about a similar issue on a another HP laptop: Frederik Himpe's blog post. It seems to require a firmware update. The Fedora already has fwupd installed by default so running:

# fwupdtool update

updated the firmware. After reboot the issue was fixed and suspending to RAM worked. Though sometimes returning from suspension, the clock was wrong, it was 2 hours ahead (computer itself is in UTC time and my timezone is UTC + 2, so something related to it). Restarting the NTP-service corrected the time. There were also some strange sound issues, sometimes after resuming sound had periodic glitches and one could not enjoy listening to music or watching videos. I tried to restart pipewire for no gain, but then noticed the clock was always off in those times. Restarting the NTP-service fixed this issue as well.

After that I tested the rest of the hardware as well and everything seemed to work well: WIFI, graphics, fingerprint reader, webcam and bluetooth. Also the automatic screen rotation works when rotating the screen and putting the laptop to stand up position.

Daily usage

So after getting everything working, this laptop has performed very well and I have been very happy with it. The keyboard is okey, I mostly use external mechanical keyboards, but the laptop's keyboard is good enough for typing a full day with it. The battery has been good enough for my usage. The laptop's portability is good. The only thing I miss from the Lenovo laptops is the trackpoint.

I have noticed that I have not used much the tablet mode, meaning using the laptop with touchscreen and pen. I will try to explore it more thoroughly and write a separate post about it. The few times I have tested it, it has worked smoothly and Gnome has a good support for touchscreen and pen usage.