Migrating from Helmfile to Terraform

Jan 22, 2023 · 7 min read · Infrastructure Kubernetes Helm Helmfile Home-Lab Terraform ·

Share on:

I'm back from a bit of a hectic time in my life! My wife and I had a new baby girl, and parents who've gone through the newborn phase know, that there's precious little time to be hacking away at your computer outside work when you have a little one crying upstairs. But now that she's kind of sleep trained, I'm back with more DevOps blogging content.

Let's talk about Terraform. At my day job, I started working with another similar DevOps team in my org and learned they used Terraform to provision the workloads in their K8s clusters rather than something like Helmfile. I thought about it for a good long while, and having worked with Helmfile and struggled through their lack of online community, I decided why not try it out in my home lab. Plus with Terraform, you can theoretically codify your actual K8s cluster with your initial workloads, which is pretty cool I think!

Let's get started with the nitty gritty of moving from Helmfile to Terraform. First thing is to create an actual Terraform module to house my helm deployments. That looks a bit like this:

 1provider "helm" {
 2}
 3
 4resource "helm_release" "grafana" {
 5    name = "grafana"
 6    namespace = "observability"
 7    repository = "grafana"
 8    chart = "grafana"
 9    version = "6.23.1"
10
11    values = [
12        "${file("grafana/values.yaml")}"
13    ]
14
15  depends_on = [
16      resource.helm_release.metallb
17  ]
18}
19
20resource "helm_release" "gitlab_runner" {
21    name = "gitlab-runner"
22    namespace = "default"
23    repository = "gitlab"
24    chart = "gitlab-runner"
25    version = "0.38.1"
26
27    values = [
28        "${file("gitlab-runner/values.yaml")}"
29    ]
30  
31}
32
33resource "helm_release" "metallb" {
34    name = "metallb"
35    namespace = "default"
36    repository = "bitnami"
37    chart = "metallb"
38    version = "3.0.2"
39
40    values = [
41        "${file("metallb/values.yaml")}"
42    ]
43}
44
45resource "helm_release" "prometheus_operator" {
46    name = "prometheus-operator"
47    namespace = "observability"
48    repository = "prometheus-community"
49    chart = "kube-prometheus-stack"
50    version = "33.1.0"
51
52    values = [
53        "${file("prometheus-operator/values.yaml")}"
54    ]
55}
56
57resource "helm_release" "prometheus_blackbox_exporter" {
58    name = "prometheus-blackbox-exporter"
59    namespace = "observability"
60    repository = "prometheus-community"
61    chart = "prometheus-blackbox-exporter"
62    version = "5.4.1"
63
64    values = [
65        "${file("prometheus-blackbox-exporter/values.yaml")}"
66    ]
67
68    depends_on = [
69        resource.helm_release.prometheus_operator
70    ]
71}

Note, this is very much directly taking my Helmfile configuration and porting it to Terraform. Some follow up work I'd like to do is introduce a fair amount of this into their own variables, and maybe even cut down the length of the module with a Terraform for_each construct for certain things.

Nonetheless, we have our Terraform module now. Next up, is to find a place to store the Terraform state. For folks who haven't used Terraform before, the state file is a very important file when working with Terraform. It contains the current state of your infrastructure so that when you run terraform plan's, it knows what has already been deployed and the current state of everything before trying to figure out what to do next. Another very important thing to note about the state file is that its usually considered a very bad idea to keep it as a regular file in your repo. You typically want some sort of remote state file location so multiple developers can work on it without having to worry about merge conflicts and such. For this particular migration, I opted to use GitLab Terraform state repos since most of my personal projects are already kept in GitLab.

There will be a few steps in moving to GitLab for state management. The first step will be creating the backend.tf file that describes what my state backend looks like, and the other will be importing all my currently existing Helm releases that reside in my K8s cluster into Terraform so Terraform knows they already exist in some form.

Let's start with the backend.tf file. That should look something like this:

1terraform {
2  backend "http" {
3  }
4}

Now you may be thinking, there's nothing in there that says we want to use GitLab. But bear with me, as there's more we're going to do here. Secondly, we'll want to import the Helm releases into Terraform. That first involves initializing our Terraform module with the remote backend locally like this:

 1PROJECT_ID="<your GitLab project ID. Can be found at the top of your project homepage>"
 2TF_USERNAME="<your GitLab username. Can you found on your profile page>"
 3TF_PASSWORD="<a personal access token that was generated for the above user>"
 4TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/<project name>"
 5
 6terraform init \
 7  -backend-config=address=${TF_ADDRESS} \
 8  -backend-config=lock_address=${TF_ADDRESS}/lock \
 9  -backend-config=unlock_address=${TF_ADDRESS}/lock \
10  -backend-config=username=${TF_USERNAME} \
11  -backend-config=password=${TF_PASSWORD} \
12  -backend-config=lock_method=POST \
13  -backend-config=unlock_method=DELETE \
14  -backend-config=retry_wait_min=5

To go over some of the above, we grab the project ID from the project homepage. Second, we need a personal access token that we generated for our GitLab user. Third, we need to specify the address of the Terraform state, that will use the project ID from the above as well as a custom state file name. Mine for example is just called "home-lab-charts". Lastly, we use all of those above variables in a long terraform init line that takes all of that information and initializes Terraform with the backend we want.

Now that we have our backend configuration, we'll want to import all of our resources. Now one thing to know about Terraform is that most if not all of the resource types you can codify with Terraform have some form of an import statement for this exact purpose. Usually you can find it at the very bottom of the Terraform documentation page for that resource type. For example, you can fien the helm_release import command here. Or for the lazy who don't want to click on the link, its basically this:

1terraform import helm_release.example default/example-name

So an example command from one of my home lab releaes would be something like this:

1terraform import helm_release.gitlab_runner default/gitlab-runner

We can repeat the above process for all of the resources I maintain currently in my K8s cluster.

The last step will be changing over my GitLab CI/CD pipeline to use Terraform rather than Helmfile. That's actually pretty straightforward as GitLab maintains a specialized docker image for this exact purpose. So my .gitlab-ci.yml file ends up looking like this:

 1---
 2stages:
 3  - plan
 4  - apply
 5
 6variables:
 7  TF_ADDRESS: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/terraform/state/home-lab-charts"
 8
 9default:
10  image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
11  tags:
12    - home-lab
13
14plan:
15  stage: plan
16  script:
17  - |
18    cd helm-charts
19    gitlab-terraform init -backend-config=address=${TF_ADDRESS}
20    gitlab-terraform plan
21  except:
22    - main
23
24apply:
25  stage: apply
26  script:
27  - |
28    cd helm-charts
29    gitlab-terraform init -backend-config=address=${TF_ADDRESS}
30    gitlab-terraform apply
31  only:
32    - main

To explain some of the parts here:

We're using an environment variable called TF_ADDRESS, which is the same path that we used in our terraform import command. This will tell Terraform the path to our state file in GitLab
We're now using a default image of GitLab's Terraform image. Using this image, we can take Terraform commands, prefix them with gitlab-terraform and they'll know to use GitLab for their Terraform state.
We're finally passing the TF_ADDRESS variable to our Terraform plan and apply commands so that the gitlab-terraform utility knows where to find our state file

And that's about it! We've covered creating the initial module, setting up the backend in your code, importing existing resources into GitLab's state management repo, and finally incorporating GitLab Terraform state into your CI/CD. I hope you all enjoyed reading through this! Hopefully you won't have to wait 6 months for the next article :)