Organise QEMU Images Infrastructure Efficiently with Packer

Problem

Let's try to solve the following problem: we need qcow2 images for all popular linux distros for our inner cloud infrastructure that doesn't support VM snapshots. These images are used mostly for tests and can be a subject to change when new tests are added and old ones retire. It would be nice if these images can also be reused for other needs as well.

Technologies

Packer is a popular tool that is commonly used for building images for almost everything. There is as well a plugin for building qemu images. (https://developer.hashicorp.com/packer/plugins/builders/qemu).
For installing packages, dependencies, doing something with the linux distro people usually use Ansible. And at last - to work with any kind of APIs, to make wrappers around existing tools, it is very handy to use Python scripts.

So let's combine all three tools into one working solution that will help us to deal with the stated problem.

Solution

First of all, we need a basic packer config file that can be reused for building of all images. Here is what I have come up with:

variable "iso_checksum" {
  type    = string
  default = ""
}

variable "iso_url" {
  type    = string
  default = ""
}

variable "boot_command" {
  type    = list(string)
  default = [""]
}

variable "name" {
  type    = string
  default = ""
}

variable "cpus" {
  type    = string
  default = "30"
}

variable "memory" {
  type    = string
  default = "8192"
}

variable "datastore" {
  type    = string
  default = "108"
}

variable "disk_size" {
  type    = string
  default = "12G"
}

variable "disk_image" {
  type    = bool
  default = false
}

variable "deploy_image_params" {
  type    = string
  default = ""
}

variable "playbook" {
  type = string
  default = ""
}

variable "AWS_ACCESS_KEY_ID" {
  type    = string
  default = "${env("AWS_ACCESS_KEY_ID")}"
}

variable "AWS_SECRET_ACCESS_KEY" {
  type    = string
  default = "${env("AWS_SECRET_ACCESS_KEY")}"
}

source "qemu" "qemu_source" {
  boot_command     = var.boot_command
  boot_wait        = "10s"
  disk_discard     = "unmap"
  disk_interface   = "virtio-scsi"
  disk_size        = var.disk_size
  disk_image       = var.disk_image
  format           = "qcow2"
  http_directory   = "${path.root}"
  iso_checksum     = var.iso_checksum
  iso_url          = var.iso_url
  output_directory = "${path.root}/output/${var.name}"
  shutdown_command = "echo 'my-fav-username'|sudo -S /sbin/halt -h -p"
  ssh_handshake_attempts = 1000
  ssh_password     = "my-fav-ssh-password"
  ssh_port         = 22
  ssh_username     = "my-fav-username"
  ssh_wait_timeout = "10000s"
  net_device       = "virtio-net"
  vm_name          = "${var.name}.qcow2"
  cpus             = var.cpus
  accelerator      = "kvm"
  disk_compression = true
  headless         = true
  vnc_bind_address = "0.0.0.0"
  qemuargs = [[ "-cpu", "host" ]]
  memory           = var.memory
}

build {
  sources = ["source.qemu.qemu_source"]

  provisioner "ansible-local" {
    extra_arguments = [
      "--extra-vars", "ansible_become_pass=my-fav-username"
    ]
    playbook_file   = "${path.root}/${var.playbook}"
    playbook_dir = "${path.root}/ansible"
  }

  post-processor "shell-local" {
    environment_vars = [
      "EXEC_DIR=${path.root}/post-processors",
      "IMAGE=${path.root}/output/${var.name}/${var.name}.qcow2",
      "DEPLOY_IMAGE_PARAMS=${var.deploy_image_params}",
      "AWS_ACCESS_KEY_ID=${var.AWS_ACCESS_KEY_ID}",
      "AWS_SECRET_ACCESS_KEY=${var.AWS_SECRET_ACCESS_KEY}"
    ]
    execute_command  = ["/bin/bash", "-c", "{{ .Vars }} {{.Script}}"]
    inline           = [
      "$EXEC_DIR/deploy-image.py -i $IMAGE $DEPLOY_IMAGE_PARAMS"
    ]
    name             = "cloud_deploy"
  }
}

Variables defined in the file can be redefined during the actual execution of Packer. Notice variables for the qemu builder plugin, defined under block source "qemu" "qemu_source", most important are - boot_command, used upon virtual machine startup and iso_url, which specifies the location of the image for the virtual machine. For further descriptions of the remaining variables, please refer to this link: https://developer.hashicorp.com/packer/plugins/builders/qemu. Other user defined variables are defined above the block for qemu builder. We will use them to redefine behaviour for specific images during packer execution.

Moving on, let's discuss the provisioner "ansible-local" and the post-processor "shell-local" blocks. The 'ansible-local' provisioner is responsible for running the Ansible script immediately after the installation of the Linux distribution. Its purpose is to install all the required dependencies. As you see - the actual playbook is a subject to change. Additionally, note that this is a 'local' provisioner, which assumes Ansible is already installed on the virtual machine following its installation. This can be accomplished using seed files, which I will explain in more detail shortly.

The post-processor block is utilized after the provisioner completes its tasks and the image is ready for use. We will employ this post-processor to execute a Python script that uploads the newly built image to our internal cloud.

Okay, let's proceed to the next component of our solution. We now need to define the specific configuration with actual user-defined variables for a particular distro. Here is the configuration for Ubuntu 22.04:

iso_url = "http://releases.ubuntu.com/22.04/ubuntu-22.04-live-server-amd64.iso"
iso_checksum = "84aeaf7823c8c61baa0ae862d0a06b03409394800000b3235854a6b38eb4856f"
name = "ubuntu-22.04-base"
disk_size = "64G"
playbook = "ubuntu/22.04/base/provision.yml"
deploy_image_params = "inner_cloud -bc base_cloud_config.json"
boot_command = [
  "<esc><esc><esc><esc>e<wait>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "<del><del><del><del><del><del><del><del>",
  "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;
   seedfrom=http://{{ .HTTPIP }}:
   {{ .HTTPPort }}/ubuntu/22.04/base/preseed/\"<enter><wait>",
  "initrd /casper/initrd<enter><wait>",
  "boot<enter>",
  "<enter><f10><wait>"
]

The playbook parameter refers to the path of the Ansible playbook file used in the provisioner block of the base HCL Packer configuration. Packer automatically copies this playbook to the virtual machine since we employ the 'ansible-local' provisioner.

The deploy_image_params parameter refers to the parameters passed in the post-processor block of the base HCL configuration to our custom Python script responsible for uploading the built image. You can check the base config file to locate the usage of these parameters.
The seemingly complex parameter, boot_command, represents a sequence executed upon virtual machine startup to initiate the installation process of the Linux distro. The boot command will differ for various distros. Pay attention to the following line within the boot command:

"linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;
seedfrom=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ubuntu/22.04/base/preseed/\"<enter><wait>",

To install Ubuntu 22.04, we utilize a seed file, which is served via HTTP. In the path provided, you can observe the location of this file. It is stored in the local filesystem and the path starts from the Packer working directory. The seed file contains the necessary information for the image installation process:

#cloud-config

autoinstall:
  version: 1
  apt:
    geoip: true
    disable_components: []
    preserve_sources_list: false
    primary:
      - arches: [amd64, i386]
        uri: http://us.archive.ubuntu.com/ubuntu
      - arches: [default]
        uri: http://ports.ubuntu.com/ubuntu-ports
  early-commands:
    - sudo systemctl stop ssh
  locale: en_US
  keyboard:
    layout: us
  identity:
    hostname: jammy
    username: my-fav-username
    password: "$6$rounds=4096$Uua67h5t4Fi2QpL6$0Y8f4YDDpTlwC.
    02F3WvWmKBq305uV0cTMXzugpyLYNCwYziCkfE0mIHiBTbc.ZgJhxTp3uonZ22yUtNyHv9x1"
  ssh:
    install-server: true
    allow-pw: true
  packages:
    - openssh-server
    - open-vm-tools
    - cloud-init
    - whois
    - zsh
    - wget
    - tasksel
    - ansible
  user-data:
    disable_root: false
    timezone: UTC
  late-commands:
    - sed -i -e 's/^#\?PermitRootLogin.*/PermitRootLogin yes/g' 
      /target/etc/ssh/sshd_config
    - sed -i -e 's/^#\?PasswordAuthentication.*/PasswordAuthentication 
      yes/g' /target/etc/ssh/sshd_config
    - echo 'vmadmin ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/my-fav-username
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/my-fav-username
    - "lvresize -v -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv"
    - "resize2fs -p /dev/mapper/ubuntu--vg-ubuntu--lv"

identity:password here is the string encrypted with SHA-512. Packages block here is what will be installed during the installation process and late-commands are the commands to be run after the installation completes. Note that this is the place where we install Ansible, which will be used for our 'ansible-local' provisioner. Actually, we can do a lot of things here and completely get rid of ansible, but utilizing Ansible provides greater clarity and enables easier migration to other images whereas these commands inside seed file are specific to this particular distro.

The next item to consider is the Ansible playbook that we will execute once the initial installation commands are completed. The playbook follows a standard format, so I won't go into specific details here. You can find all the code on my GitHub repository. For the actual Ansible playbook, please refer to the following location: images/ubuntu/22.04/base/provision.yml. Additionally, the playbook references Ansible roles, which can be found here: images/ansible/roles

The last component is a Python script used for deploying the image to our internal cloud and saving it to S3. Here is an example of the parse_args() method from the script:

def parse_args():
    argparser = argparse.ArgumentParser()
    argparser.add_argument('-i', '--image', required=True, help='path to image file')
    argparser.add_argument('--s3-path', help='skip deployment to s3, 
                            just use predefined path')

    subparsers = argparser.add_subparsers()
    nebula_parser = subparsers.add_parser('cloud', help='cloud specific params')
    nebula_parser.add_argument
    ('-c', '--config', help='path to nebula image config file')
    nebula_parser.add_argument('-bc', '--base-config', 
            help='path to nebula image base config file')

    return argparser.parse_args()

The parameters for this script are passed from our specific HCL file for a particular distro, see deploy_image_params. Regarding the cloud-specific parameters, I follow a similar approach to the Packer parameters, where I have a base cloud config and a separate config for each specific distro with overridden variables. With this script, we have the flexibility to perform various operations on the images. For instance, we can implement a versioning scheme or remove obsolete images from S3 or the cloud. In other words, anything that needs to be done upon the arrival of a new image can be achieved here.

Now that we have all our building blocks, let's take a look at how the file structure is organized:

>>> images/
>>>    ansible/
>>>       roles/
>>>          ...
>>>    post-processors/
>>>       deploy-image.py
>>>    ubuntu/
>>>       22.04/
>>>          base/
>>>             preseed/
>>>                meta-data
>>>                user-data
>>>             custom.pkrvars.hcl
>>>             provision.yml
>>>          ...
>>>    base.pkr.hcl

In the file structure, you'll notice the base folder under ubuntu/22.04. This implies that we can have other images built on top of the base image. This can be achieved by using the disk_image parameter in the Packer qemu builder. If disk_image is set to true, the installation process will be skipped. Instead, Packer will start the virtual machine with the image specified in iso_url and run the provisioner and post-processor on it. Example configuration for a non-base image:

iso_url = "base_image"
iso_checksum = "none"
disk_image = true
name = "ubuntu-22.04-jenkins-agent"
disk_size = "64G"
playbook = "ubuntu/22.04/jenkins-agent/provision.yml"
deploy_image_params = "inner_cloud -bc base_cloud_config.json"

We don't have a boot_command because it is unnecessary. The base image already has the Linux distro installed. In this case, we simply add a custom Ansible playbook. The iso_url should be provided during the actual execution of Packer. I used another Python script to obtain this iso_url. I employed a specific naming convention, so if we have a distro named ubuntu-22.04, the base distro name would be ubuntu-22.04-base. The path in the S3 storage is fixed.

The command to start packer build of a base image:

packer build -var-file images/ubuntu/22.04/base/custom.pkrvars.hcl base.pkr.hcl

The command to start packer build of a non base image:

packer build -var iso_url=http://path_to_s3_with_base_image 
-var-file images/ubuntu/22.04/jenkins-agent/custom.pkrvars.hcl base.pkr.hcl

Another important aspect not mentioned here is the Jenkins job that detects changes in the images/ansible directory and generates a set of Packer build commands to rebuild the affected images. To accomplish this, we utilize another Python script that relies on the playbooks and roles structure. First, we determine the role to which the changed file belongs, and then we identify the playbook associated with that role. Finally, we generate a Packer command for each image where this playbook is located.

That's it! For the actual code, please refer to the GitHub repository.