Setup Terraform to work with Proxmox

In my previous post I talked about what Terraform is, how to get it.  Now we’re going to use it.

First step – Plan

My plan is to incorporate Terraform and Ansible so my plan is as follows:

• Terraform is to deploy
• Ansible is to configure

So step one – install Terraform and configure

On my CentOS Cloud 10 VM (already deployed) let’s get the OS up to date:

sudo dnf update && sudo dnf upgrade

You can either add the Hashicorp repository to dnf or you can add the binary manually.  In CentOS Cloud 10, there is no repo for Hashicorp that’s native, so you’d have to use the RHEL9 version:

sudo dnf config-manager –add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

sudo dnf install -y terraform

I just installed the binary manually:

cd /tmp

wget https://releases.hashicorp.com/terraform/1.10.1/terraform_1.10.1_linux_amd64.zip

unzip terraform_1.10.1_linux_amd64.zip

sudo mv terraform /usr/local/bin

Now verify terraform

terraform version

Terraform v1.14.0
on linux_amd64

 

 

The plan is to test using Terraform for a very basic function – deploy VMs to Proxmox

To do this, we need to link terraform via a provider file (provider.tf) to a Proxmox host.  The host needs to be able to accept a connection from terraform with the right permissions to be permitted to deploy.  Also, it’s assumed at this point that you already have a template to clone, but if not, we’ll cover that, too.

Set up Proxmox

We need to create a token that can be used by Terraform.  On one of your Proxmox hosts:

pveum user token add root@pam terraform-token –privsep=0

In my example I just used root@pam, but you can set this to any user with admin privileges.  This just adds a token, similar to an App Password, with privileges (privsep=0), to the account.

When you execute this command you’ll be provided with key/value data. Very important! Copy this information somewhere (hopefully secure)! You will never see this token information again and you’ll need it to set up terraform to communicate with the host.  Here’s an example of the output (values are not real):

key :                       value
full-tokenid:           root@pam!terraform-token
info:                       {“privsep”:”0″}
value:                     0fa26ab7-3267-4915-bc21-56945b807140

If you don’t already have a template in Proxmox, let’s create one quickly.  On one of your Proxmox hosts using local-lvm storage (or pick whatever storage will support your image):

cd /var/lib/vz/template/iso

Download a run-ready image:

wget https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2

Now create a VM and inject the image:

qm create 9000 –name rocky9-template –memory 2048 –cores 2 –net0 virtio,bridge=vmbr0

qm importdisk 9000 /var/lib/vz/template/iso/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 local-lvm

Again – set your image location to whatever you have supported.  For many the local-lvm storage capacity is limited, so use zfs or whatever else you have with adequate room.  I’m only using local-lvm as an example.

Attach the drive:

qm set 9000 –scsihw virtio-scsi-pci –scsi0 local-lvm:vm-9000-disk-0

Add the Cloud-Init drive:

qm set 9000 –ide2 local-lvm:cloudinit

Set the boot order:

qm set 9000 –boot c –bootdisk scsi0

Set the display (I set the display output to –vga std but typically with a fully matured system you would assume the VM runs headless and outputs to remote console (SSH – use serial0)

qm set 9000 –serial0 socket –vga std

Set whether the QEMU agent is enabled.  It may not automatically install but that’s ok, because we’re going to use terraform to add it.

qm set 9000 –agent enabled=1

At this point you should be preparing your your image.  Use Cloud-Init to set login values, including username, password, and DNS.  Boot up the VM, log in with the values you provided, and ensure that the image is up to date, that you have your qeumu agent installed (check VM in Proxmox to ensure you see IP addresses – this is an indicator that the agent is running properly).  Install whatever other software you want this image to have.

When you’re done prepping the image, shut it down.

Finally, template it:

qm template 9000

Done.  You should show on your Proxmox host a new VM with a new icon called 9000 (rocky9-template)

Use this process (you could script it even – if I get around to it I’ll add a script) for any bootable image.  I set up both Rocky Linux 9 and Ubuntu 24 templates.

It’s also important to note that before you ‘template’ the image, you can launch the VM and edit it.  However, to do so you’ll need to edit the Cloud-Init for the VM so a username and password or SSH Key is injected, otherwise you won’t be able to log in.  Remember, your terraform template will override any template settings – you will be setting a different cloud-init configuration.

At this point we don’t really have any automations in place.  Yes, we could technically use ansible (and I will add playbooks to do exactly that) but it’s important to understand the purpose of each tool in the set.

Terraform and Ansible both overlap in many respects, but the power of each tool is characterized by their primary function.  In this case, Terraform has power with rapid deployment at scale (orchestration), while Ansible has power to configure (deploy applications, patching, automations).

Also, the template is bare bones.  We deployed it with a Cloud-Init drive, and while we could configure it manually in Proxmox, our goal is to standardize our operations while having the flexibility to make changes.  Terraform cloud-init config will override whatever we put apply to the template manually in Proxmox, so it’s probably safer to not store credentials in Proxmox.

Despite this being a lab, where security is generally lax, we should always be trying to follow best practices and develop the habits of doing things properly, securely, regardless of the environment.

As with any project, production or otherwise, it pays to plan.

Create a logical folder structure to house your files.  Optionally use git – this can come in handy later when you want to replicate your work.

For Terraform it’s important to understand how it works.  You create a series of .tf files, put them in a folder, then execute.  Terraform then reads those files as a “Plan” and merges all the info together.  You have a lot of freedom here on how you construct these files, so think logically.  Technically you could put everything into a single file, but there’s no flexibility.  You may be deploying to Azure, or Proxmox, or somewhere else entirely.  Separating out your files into functions makes the most logical sense.

NOTE:
For non-production I found it easier to work from my home folder.  I don’t have to sudo every command and I’m not sharing my work with anyone on my network.  For production, move your focus to use /opt.

mkdir -p ~/projects/terraform/proxmox-lab

OPTION – USE GIT

cd ~/projects/terraform

sudo dnf install git

git init

git config –global user.name “Your Name”

git config –global user.email name@domain.com

Now:

cd ~/projects/terraform/proxmox-lab

Create your first .tf file:  provider-proxmox-lab.tf.  This is also important – check your Proxmox versioning.  My initial setup was for an earlier version of Proxmox which didn’t work – I had to redo it to match my current version of Proxmox VE 9.01.  If things are working, then suddenly start breaking, the first thing you should review is versioning changes in the environment.

nano provider-proxmox-lab.tf

Copy/Paste:

terraform {
required_providers {
proxmox = {
source  = “bpg/proxmox”
version = “0.66.3”
}
}
}

provider “proxmox” {
endpoint = “<https://<IP or DNS Name of your Proxmox Server:8006/>”
api_token = “root@pam!terraform-token=YOUR_SECRET”
insecure = true }

Edit the values to match your environment.  See that token field? You recorded this earlier.

This file is the link to your Proxmox environment, and what Terraform will use to connect and create.  Your token gives access to the API offered by your Proxmox host specifically so you can do this.

NOTE:
Please review before trying to apply.  Default values in Proxmox can very from host to host – specify values (disk, network, BIOS, CPU) that fits your deployment environment

Now, we want to deploy from our newly minted template:

nano rocky-linux-dep.tf

Copy/Paste:

resource “proxmox_vm_qemu” “test-vm” {
name        = “terraform-test”
target_node = “proxmox4”
clone       = “rocky9-template”
cores   = 2
memory  = 2048

# specify CPU type – some linux distros do not like the Proxmox default and some Proxmox servers don’t support all CPU types
cpu {
type = “x86-64-v2-AES”
}

disks {
scsi {
scsi0 {
disk {
storage = “local-lvm”
size    = 20
}
}
}
}

network {
model  = “virtio”
bridge = “vmbr0”
}
}

First thing you’ll want to do is properly format the file.  While not necessary as terraform doesn’t particularly care about white space and formatting, it’s a lot easier to digest a properly formatted document:

terraform fmt

Now for the cloud-init component.  Remember we created a template in Proxmox manually, and we may or may not have set any cloud-init parameters.  Our best practice is to set those parameters in an initialization brace.  Insert to the bottom just above the final closing curly brace [ } ]:

# Cloud-init configuration
initialization {

ip_config {
ipv4 { address = “dhcp”
}
}
user_account {
username = “<your designated username>”
password = “<password for designated username>” # Or use Terraform variables
keys = [ “ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC… your_public_key_here“ ]
}
dns { servers = [“192.168.XXX.XXX”, “192.168.XXX.XXX”]
domain = “corp.domain.com”
}
}
}

This is where some best practices in security should be kicking in.  Do you really want a username and password and/or public key exposed in a plain text file?  (The correct answer is no). Also, if your DNS servers are pretty static, and for most they would be, do we need to enter this every single time? And if we do have a DNS change, do we want to edit every single template every time?  Again, no.

Terraform allows you to create variable files, and secure those variable files.  Remember, we’re creating templates for rapid deployment, so let’s break some of this out:

nano dns-variables.tf

Copy/paste:

variable “dns_servers” {
description = “DNS servers”
type = list(string)
default = [“192.168.XXX.XXX”, “192.168.XXX.XXX”] }

Save file

Now update the template:

nano rocky-linux-dep.tf

Replace to look like this:

dns {
servers = var.dns_servers
domain = “corp.domain.com”
}

Because terraform compiles/merges all the .tf files in the folder your variables file.  Pretty much anything that is reusable can be defined as a variable once and referenced in a variables file.  So if we want to add another common variable – the domain, we just add it to the mix:

nano domain.tf

variable “domain” {
description = “Domain name for VMs”
type = string
default = “corp.domain.com”
}

Update your template:

dns {
servers = var.dns_servers
domain = var.domain
}

Sensitive information, like credentials, can be locked down.

Create a secrets file:

nano mysecrets.tfvars

Insert your secrets:

ssh_public_key = “ssh-rsa AAAAB3NzaC1yc2EAAAADAQA…”
vm_password = “YourSecurePassword”
proxmox_token = “root@pam!terraform-token=abc123…”

Protect your secrets:

chmod 0600 mysecrets.tfvars
echo “mysecrets.tfvars” >> .gitignore

Create a usersecretsvar.tf where you specify your account information without revealing your secrets:

variable “ssh_public_key” {
description = “SSH public key for VMs”
type = string
sensitive = true # Hides from logs
}

variable “vm_password” {
description = “Default VM password”
type = string
sensitive = true
}

variable “proxmox_token” {
description = “Proxmox API token”
type = string
sensitive = true
}

Now you can edit your template for the VM creation with an initialization section that looks like this:

nano rocky-linux-dep.tf

Edit user account info:

user_account {
username = “<your designated username>”
password = var.vm_password
keys = [ var.ssh_public_key ]
}

Same with the provider file “provider-proxmox-lab.tf”:

api_token = var.proxmox_token

When you execute your plan you include your mysecrets.tfvars securely:

terraform plan -var-file=”mysecrets.tfvars”

And when you execute your application, provided the plan works:

terraform apply -var-file=”mysecrets.tfvars”

There are other ways this can be done.  You could embed these credentials to your ~/.bashrc or create a source_secrets file.  If you are a lone wolf practitioner, this is probably a safer option.  Another option is that Hashicorp offers an Enterprise Vault.  We’ll cover that in another post.

As you can see there is a lot of flexibility with terraform to match to your environment.  Try to use best practices – secure your sensitive info – parse your projects into logical folder structures – and clean up after yourself.

The last thing I want to talk about is the terraform destroy feature.

The purpose of destroy is to quickly tear down the infrastructure you’ve created – either entirely (lab, test, pre-prod) or selectively (one of the VMs didn’t deploy the way I expected…).

So first thing, your ran terraform plan, and terraform apply, then worked in the infra, added more stuff, and reapplied (terraform apply).  For whatever reason, you now need to tear it all down.  Well, if that was a complex Azure infrastructure, it can take a while to rip it all out again.

Be careful with this… make sure you aren’t removing something you actually need to keep:

terraform plan -destroy

Gives you a human readable map of exactly what running a destroy will do.

terraform destroy

Runs the destroy process, at which point you will need to type in the word ‘yes’ to agree.  No backsies!

Even more useful, let’s say a full deployment is done and you realize one VM wasn’t what you wanted.  You can manually delete it, or you can selectively delete it, fix your files, and redeploy.

terraform destroy -target=<name of item to destroy>

Really handy for removal of VMs that you don’t want or need, but leave the infrastructure behind.

I went through this process exactly how I documented it, all the while with an eye towards “What would I do in a production environment?”.

Remember I said terraform can deploy ‘at scale’?  It doesn’t seem very efficient to only deploy a single VM every time.  In my next post we’ll explore how to deploy multiple VMs from a single template using a single plan.  Terraform – Scaling