Signing and verifying Git commits with SSH keys

June 02, 2024
Tags:sshfido2git

SSH commit signing support was added to Git version 2.34.0. In this post I look at how to sign Git commits with SSH keys, preferably backed with FIDO2 security keys. There are different opinions whether signing commits is worth the extra effort. For example just recently Harley Watson wrote how commit signing is still kinda wack and Ludovic Courtès wrote on the importance of signing and verifying commits. This post is about the technical aspects of signing commits.

TLDR version

You can sign Git commits with SSH keys:

# git config --local gpg.format ssh
# git config --local user.signingkey ~/.ssh/id_ed25519-sk.pub
# git commit -S

To verify the Git commit create allowed signers file and configure your Git repository to use it. Then run git verify-commit:

# git config --local gpg.ssh.allowedSignersFile ./allowed_signers
# git verify-commit $commitid

With little bit of scripting you can verify all new commits in a Git branch:

commits=$(git log --pretty="%H" main..$branch)
for commit in $commits; do
   git verify-commit $commit 
done

You can use FIDO2 security keys (like Solokeys) with OpenSSH. Create a non discoverable SSH key with key type ed25519-sk or ecdsa-sk:

# ssh-keygen  -t ed25519-sk

SSH does not have similar Web of Trust (WoT) as the PGP ecosystem has. This limits the discoverability and distribution of SSH keys. In an organisation, you can use SSH CA to sign users SSH keys and make it easier to verify SSH signature and SSH key usage.

Getting started

Before you can sign Git commits, you need a SSH key. Probably you already have one, but if not, you can create one with the following command:

# ssh-keygen  -t ed25519

If you have a FIDO2 security token, you can create SSH key with SSH key pairing. See some notes about it in the Advanced usage section below.

Signing commits

When you have a SSH key, we can configure Git to sign commmits with it. First configure your local Git repository to sign commits with SSH keys and set your user.signingkey to the public SSH key. My examples use SSH keytype ed25519-sk:

# git config --local user.email <you_email@example.com>
# git config --local gpg.format ssh
# git config --local user.signingkey ~/.ssh/id_ed25519-sk.pub

After that make a Git commit with option -S, which is the option for signing commits.

# git commit -S

If you want to use this configuration in all your Git repositories, you can make the configuration global by replacing --local with --global.

Verifying commits

Verifying a Git commit does two things. First, the actual commit is verified against the cryptographic signature in the commit to see that the signature is valid. Second, it verifies that the commit signature is made with an allowed signing key. So now if you verify the commit, it should print something like this:

# git verify-commit <commitid>
Good "git" signature with ED25519-SK key SHA256:sxZkSe2TJJOKv3dQjadJ05RkCqXWWMB0HkCBU3D+pEM
No principal matched.

So the actual signature is valid, but it did not find a principal for it. It means Git does not know which keys are allowed to sign commits. When you verify Git commits signed with a GPG key, the signing keys are checked against the GnuPG keyring. The SSH ecosystem does not have similar Web Of Trust (WoT) to distribute and verify keys. You must define separately the SSH keys which are allowed to sign Git commits. These are defined in an allowed signers file. The format is defined in manual page for ssh-keygen. The file matches email addresses to the public SSH keys for users that are allowed to sign Git commits. You can also set valid-after and valid-before timestamps for the keys. Example for the allowed signers file:

# Comment here
user@example.org sk-ssh-ed25519@openssh.com AAA41...

Create the allowed signers file to the root of your Git repository as allowed_signers, use your email address, ssh key type and ssh public key in it. Then configure the Git repository to use it in the root of the Git repository:

# git config --local gpg.ssh.allowedSignersFile ./allowed_signers

After you have created and configured the allowed signers file you can verify Git commits with the following command:

# git verify-commit fe155257ece409c670c96a11e912538acf2b085d
Good "git" signature for user@example.org with ED25519-SK key SHA256:sxZkSe2TJJOKv3dQjadJ05RkCqXWWMB0HkCBU3D+pE

Advanced usage

Now let's look into more advanced usage. These include verifying Git commits in different use cases and ways to secure the SSH keys.

SSH keys paired with security keys

OpenSSH introduced support for SSH-keys backed with security keys in version 8.2. These two new key types are "ecdsa-sk" and "ed25519-sk":

FIDO/U2F Support
----------------

This release adds support for FIDO/U2F hardware authenticators to
OpenSSH. U2F/FIDO are open standards for inexpensive two-factor
authentication hardware that are widely used for website
authentication.  In OpenSSH FIDO devices are supported by new public
key types "ecdsa-sk" and "ed25519-sk", along with corresponding
certificate types.

So creating and using using these new key types requires FIDO2 hardware security keys: when you create a new SSH key it will pair the security key to our SSH key and the hardware key is required when using the SSH key. It essentially acts as MFA (Multi Factor Authentication) for your ssh key: if you have set a password for your ssh key then that is the first factor, something you know. Then the FIDO2 hardware token acts as a second factor, something you have. The security keys usually are small tokens, that you plug into USB port or use through bluetooth or NFC. Using the key requires some action, like pressing a button on the device or giving a PIN-code.

You can generate a security key backed SSH key with normal ssh-keygen flow, plug in the security key, run ssh-keygen -t ed25519-sk and follow the instructions.

You can also hook GnuPG keys to security keys that have smartcard features, like Yubikeys, but it requires quite a bit work. I've used drduh's yubikey-guide in the past and it is a frustrating process.

After you have created the key you can use it like a regular SSH key, but you have to have the security key plugged in. One caveat is that the new SSH key types will require support in the SSH-server as well, if you use to SSH connections. So services which use older OpenSSH versions or some other SSH server software might not support the new key types (I'm especially looking at you Bitbucket).

Verifying commits in pull-request flow

For verifying Git commits from a branch in a pull-request flow, you can use the approach below. It first fetches the main branch from the Git origin and loads allowed_signers file from the main branch. This makes sure that allowed signing keys are not updated in the pull-request branch, so the signing keys for new committers must first be added to the main branch before their changes can be accepted.

After loading allowed_signers, script gets commitIDs of new commits that are not in the main branch and verifies them.

git fetch origin main:main
git show main:allowed_signers > ~/gitsigners
git config --local gpg.ssh.allowedSignersFile ~/gitsigners
commits=$(git log --pretty="%H" main..$branch)
for commit in $commits; do
   echo "Verifying commit $commit"
   git verify-commit $commit 
done

If verify-commit fails for any of the commits, it should fail the build.

Guix git authenticate flow

Guix takes verifying Git commits further and offers way to authenticate the whole git tree from given Git commit. It also uses a separate file in the repository, .guix-authorizations, which has a list of allowed signing keys for users. The signing key must be added to file in previous commits before the signing key can be used to sign a commit. Currently guix git authenticate only supports PGP keys.

Below is a proof of concept of a bash function for similar behaviour. It iterates through the Git commits from the given commit id and verifies the commits. It fetches the allowed signers file from the parent commit and configures the allowed signers file to the Git repository for the verification. Saving the allowed signers file to a temp directory for the verification is quite hackish.

verify_commit() {
    # temp directory where extra files are stored for the verification
    local work_dir=$1
    #commit to start, for example: $(git log --pretty="%H" | head -n 1 )
    local commitid=$2
    # original signers file: $(git config --get --local gpg.ssh.allowedSignersFile)
    local orig_signers=$3
    local parents=$(git log --pretty=%P -n 1 $commitid)
    if [ -z "$parents" ]; then
        echo "Parent ($parents) empty, we are finished"
        echo "Set back allow signers file to $orig_signers"
        git config --local gpg.ssh.allowedSignersFile $orig_signers
    else
        while IFS= read -r parentid; do
            GITSIGNERS=$(git show $parentid:allowed_signers)
            #echo -e "GIT signers $parentid \n$GITSIGNERS"
            echo "$GITSIGNERS" > $work_dir/$parentid.gitsigners
            git config --local gpg.ssh.allowedSignersFile $work_dir/$parentid.gitsigners
            git verify-commit $commitid
            local ret=$?
            if [ $ret -ne 0 ]; then
               echo "Verification failed for commit $commitid"
               echo "Set back allow signers file to $orig_signers"
               git config --local gpg.ssh.allowedSignersFile $orig_signers
               echo "Removing directory $work_dir"
               rm -r $work_dir
               exit 1
            fi
            verify_commit $work_dir $parentid $signers
        done <<< "$parents"
    fi

}

SSH CA support

The allowed signers file supports also SSH certificate authority. In an organisation you can have a central CA which you use to create SSH certificates for users. The CA creates a SSH certificate by signing the given SSH key with certificate information, such as username identity of the user and certificate valid times. You can have the following in the allowed signers file to indicate SSH CA signing key:

# Organisations cert-authoriy, public key for for the SSH CA
*@example.org cert-authority ssh-ed25519 AAA...

Then if the Git commit principal matches the "*@example.org", Git tries to verify that the commit signing certificate is signed by the CA. Here is one article by Step CA about using SSH certificates in an organisation. This simplifies the allowed signers file and can be make verification easier in an organization, but requires setting up the CA for SSH keys.

Conclusion

After using GPG keys with Yubikey for couple of years, I've been impressed how easy Fido2 hardware tokens with SSH keys have been to use. After the SSH key signing support was added to Git, Git commit signing has been easy. Usually users already have SSH keys, which makes it easier to start to signing Git commits with SSH keys. One obstacle is that not all software in the Git ecosystem support SSH keys yet, like libgit2, but slowly different tools have added support for it. For the commit signing to succeed, better tools are required, not just for the signing but for the verification of commits as well.