Reproducible development boxes

For some time I've experimented and looked for a good solution for building (reproducible) development boxes. These will be virtual machines aimed for development. In the beginning they will be local virtual machines, but in the future they could also be remote machines. Main reason for using the development boxes would be added security by separating and isolating projects/customer work from each other. Each VM would have projects for one customer and only required SSH/access keys. The actual builds are also run in isolated containers and have even more reduced access to resources. This will protect against supply-chain malware as they reduce significantly resources the malware can read, like SSH keys and access tokens. Here are requirements for my setup:

1. Easy way to create and destroy virtual machines
2. Have automated way to set up basic configuration (installed packages, network configuration and user accounts)
3. Share folders from host computer to the virtual machine

I will use multiple layers and tools for protection, using separate virtual machines is just one of them. In this post, I'll focus on automating the creation and management of these development boxes, in the next post I will describe some other methods I use inside the VM for isolating builds.

Automating local virtual machines

Long time ago I used Vagrant to have reproducible test environments on local virtual machines. This was the first tool on my list to test for automating virtual machine management. First thing I noticed when looking for suitable image for the VM was, that many boxes have not been updated for some time. Maybe the community is shrinking and not active anymore?

I created OpenSUSE boxes with Vagrant and used it for a couple of weeks. Vagrant is over 10 years old so it has quite a lot of documentation and examples. Simple things are easy to configure, like shared folders between host and guest. It also has good support running different provisioning tools to configure the VM on creation.

During the couple of weeks test run I trouble suspending and resuming my box, it seems every time I had to suspend/shutdown my laptop I had to also bring the Vagrant box up. Also, Vagrant isn't fast, I had to wait long time for certain operations on Vagrant box to finish. So I decided to look for tools.

Opentofu

Opentofu has a provider for creating KVM virtual machines. I use Opentofu on other projects, so this could be good solution. With little HCL-configuration I was able to create myself an OpenSUSE virtual machine. Searching for the right type of image was cumbersome, would be nice to have a better way to get a list of images to use. Again booting virtual machines took a while, though this mostly just my impression.

I had a script for sharing folders from host computers with SSHFS. It worked alright, though it disconnects now and then. While I was testing this solution, I found another interesting project for lightweight virtual machines, incus.

Incus

Incus is a successor of an older project, LXD. Incus is a platform running containers and lightweight VMs. The lightweight VMs seemed interesting option. Fedora also had packages for it, so I decided to test it out.

After setting initial configurations, I checked available images:

incus image list images

There are images available for most current Linux distributions. After choosing to run OpenSUSE Tumbleweed, I started a test VM to experiment different configuration options and sure the lightweight VM's booted very fast!

But initially they would not start properly, I could not connect to it via SSH or incus shell. I was also experimenting with different configuration options, so perhaps I had put a bad configuration option? I could not query information about the running system, although the incus documentation said I should be able to. After a while, I noticed others had similar issues: incus-agent fails to run in virtual machines after recent update. So the issue was with incus-agent on the virtual machine. After I was able to set a working incus-agent on the VM and proper cloud-init configuration, I could access it and start experimenting.

During testing, I noticed it is faster than previous tools I tested. Vagrant has different options to provision configurations for the boxes. With incus, I settled for cloud-init, which cloud variants of the incus images support. Incus also supports sharing folders from host computer with simple command (incus config device add <instance_name> <device_name> disk source=<path_on_host> path=<path_in_instance> shift=true) and if I need further provisioning, I can execute pyinfra/ansible over SSH. It's not fully reproducible system like GuixOS or NixOS, but it is a good start for my current needs.

Incus provider for Opentofu

When I was writing this post, I noticed that incus has an Opentofu provider as well. I moved the configuration to the Opentofu to have a better management of the virtual machines. Now I can have the VM configuration in code, which makes provisioning similar development boxes trivial. Here is an example configuration for an OpenSUSE box:

terraform {
  required_providers {
    incus = {
      source = "lxc/incus"
      version = "1.0.2"
    }
  }
}

provider "incus" {
  default_remote = "local"

  remote {
    name    = "local"
    address = "unix://"
  }

}

resource "incus_image" "opensuse_cloud" {
  source_image = {
    remote = "images"
    name   = "opensuse/tumbleweed/cloud"
    type   = "virtual-machine" 
  }
}
data "template_file" "cloudinit" {
  template = file("${path.module}/cloud_init.cfg")
  vars = {
    user_account = "user"
  }
}

resource "incus_instance" "devbox1" {
  name      = "devbox1"
  image     = incus_image.opensuse_cloud.fingerprint
  type      = "virtual-machine"
  ephemeral = false
  profiles  = ["default"]
  config = {
    "user.user-data" = data.template_file.cloudinit.rendered
    "limits.cpu" = 2
    "limits.memory" = "8192MiB"
  }
  wait_for {
    type = "agent"
  }
}

And an example cloud-init configuration (cloud_init.cfg):

#cloud-config
package_update: true
package_upgrade: true
timezone: Europe/Helsinki
ssh_pwauth: true
packages:
  - git
  - openssh
  - helix
  - vim
  - helix-bash-completion
  - make
  - lazygit
  - git-delta
  - chezmoi
users:
  - name: user
    groups: users, wheel
    password: <hash>
    ssh_pwauth: true
    lock_passwd: false
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 <ssh_key_pub>

Current status and future

The incus VM has worked well this autumn. I use OpenSUSE Tumbbleweed as the OS, Opentofu to define the VMs and I have cloud-init configurations to have some automation for setting up the box.

The actual VMs haven't required much configuration, mostly just installing different development tools. After setting base configuration with cloud-init, I use Chezmoi to deploy same configurations for dotfiles. You can combine that with podman quadlets configured in user's home directory to have container services for databases and other services.

Next step is to install Guix package manager on to have isolated Guix shell environments. I would also like to see Guix image available for incus.