{"id":26197,"date":"2024-02-29T18:02:39","date_gmt":"2024-02-29T17:02:39","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=26197"},"modified":"2024-02-29T18:02:41","modified_gmt":"2024-02-29T17:02:41","slug":"terraform-x-go-challenges-when-interacting-with-terraform-through-go","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2024\/02\/29\/terraform-x-go-challenges-when-interacting-with-terraform-through-go\/","title":{"rendered":"Terraform x Go: Challenges when interacting with Terraform through Go"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n
<\/span>\"\"
\n

goTerra<\/p>\n<\/div><\/div>\n\n\n\n

In a recent project, some of my fellow students and I developed a basic hosting provider that allows a user to spin up Docker containers on a remote server, which is realized by using Terraform locally on the server.<\/p>\n\n\n\n

During this project, we developed a Go-based backend service that provided a REST API to the client and handled the connection to Terraform via RabbitMQ, authentication via Keycloak and data persistence with DynamoDB. As mentioned earlier, Terraform runs locally on the remote server and is fully automated by Go’s power. This sometimes became a challenge. This blog post briefly describes how we interacted with Terraform using Go and the problems we have encountered and those we have identified but not yet resolved, while working with Terraform.<\/p>\n\n\n\n

Terraform<\/h3>\n\n\n\n

Terraform, an open-source tool created by HashiCorp for provisioning cloud infrastructure, allows describing infrastructure as code (IaC) in a simple, human-readable language called HCL (HashiCorp Configuration Language). This approach provides version-controllable, reusable, and collaborative means to manage infrastructure.<\/p>\n\n\n\n

Traditionally, managing Terraform involves writing HCL files, running terraform init to initialize the Terraform working directory, terraform plan to create an execution plan, and terraform apply to apply the desired changes. This process, while straightforward, can become complex and time-consuming as your infrastructure grows.<\/p>\n\n\n\n

This is where Go comes into play. In this blog post, I will share my experience of leveraging Go\u2019s features to automate the process of managing Terraform through HashiCorp’s Terraform Go package<\/a>.<\/p>\n\n\n\n

Go: How to Interact with Terraform<\/h2>\n\n\n\n
\n

Through HashiCorp’s Terraform go package<\/a>, no direct interaction with the binary with OS commands is necessary.<\/p>\n<\/blockquote>\n\n\n\n

Avoidance of requiring the provision of binaries<\/h3>\n\n\n\n

Terraform thought about that as well and does supply a procedure to download the required binaries through the code.<\/p>\n\n\n\n

import (\n    "context"\n\n    version "github.com\/hashicorp\/go-version"\n    product "github.com\/hashicorp\/hc-install\/product"\n    releases "github.com\/hashicorp\/hc-install\/releases"\n    tfexec "github.com\/hashicorp\/terraform-exec\/tfexec"\n    log "github.com\/rs\/zerolog\/log"\n)\n\nfunc main() {\n    terraform_version := "1.7.1"\n    binary_install_dir := "\/path\/to\/bins"\n    terraform_cwd_dir := "\/path\/to\/terraformFiles"\n\n    \/\/ This specifies the exact version of the product, in this case Terraform, to install\n    installer := &releases.ExactVersion{\n        Product:    product.Terraform,\n        Version:    version.Must(version.NewVersion(terraform_version)),\n        InstallDir: binary_install_dir,\n    }\n\n    \/\/ This installs the product and returns the path to the binary\n    \/\/ The binary will be installed in the directory specified by InstallDir \/ binary_install_dir\n    execPath, err := installer.Install(context.TODO())\n    if err != nil { log.Error().Msgf("Error occured %s", err) }\n\n    \/\/ This ensures that the binary is installed and returns a Terraform struct\n    \/\/ The Terraform struct is used to interact with the Terraform binary\n    tf, err := tfexec.NewTerraform(terraform_cwd_dir, execPath)\n    if err != nil { log.Error().Msgf("Error occured %s", err) }\n\n    \/\/ This returns the version of the installed Terraform binary\n    tfVersion, _, err := tf.Version(context.TODO(), true)\n    if err != nil { log.Error().Msgf("Error occured %s", err) }\n\n    log.Debug().Msgf("Terraform installed in version: %s", tfVersion)\n}<\/code><\/pre>\n\n\n\n
Invoking Terraform commands<\/h3>\n\n\n\n
The basic command palette of Terraform, e.g. init, plan, apply, and destroy, is easily accessible. As shown in the code example below, which extends the code above, this is made very easy.<\/p>\n\n\n\n
    var varFiles []string = []string{\n        "\/path\/to\/terraform.tfvars.json",\n    }\n\n    \/\/ This runs the Terraform plan command, with the specified varFiles\n    \/\/ Sadly, there exists no generic PlanOption, so we have to create one for every Terraform command\n    tfPlanOpts := []tfexec.PlanOption{}\n    for _, varFile := range varFiles {\n        varFileOpt := tfexec.VarFile(varFile)\n        tfPlanOpts = append(tfPlanOpts, varFileOpt)\n    }\n\n    \/\/ This plans the Terraform configuration, which is the equivalent of running terraform plan\n    \/\/ This returns a boolean indicating whether the Terraform plan has a difference between the current state and the plan\n    plan_diff, err := tf.Plan(context.TODO(), tfPlanOpts...)\n    if err != nil { log.Error().Msgf("Error occured %s", err) }\n\n    if plan_diff {\n        log.Info().Msg("Terraform plan has a difference between the current state and the plan")\n    } else {\n        log.Info().Msg("Terraform plan has no difference between the current state and the plan")\n    }\n\n    tfApplyOpts := []tfexec.ApplyOption{}\n    for _, varFile := range varFiles {\n        varFileOpt := tfexec.VarFile(varFile)\n        tfApplyOpts = append(tfApplyOpts, varFileOpt)\n    }\n\n    \/\/ This applies the Terraform plan, which is the equivalent of running terraform apply\n    err = tf.Apply(context.TODO(), tfApplyOpts...)\n    if err != nil { log.Error().Msgf("Error occured %s", err) }\n\n    \/\/ This shows the Terraform state, which is the equivalent of running terraform show\n    state, err := tf.Show(context.TODO())\n    if err != nil { log.Error().Msgf("Error occured %s", err) }\n\n    log.Info().Msgf("New Terraform state %v", state)<\/code><\/pre>\n\n\n\n
Problems I encountered<\/h2>\n\n\n\n
No dependency for binaries<\/h3>\n\n\n\n
As already mentioned in “Avoidance of requiring the provision of binaries”, Terraform supplies a solution to automate the process of downloading the necessary binaries. This can also be done through HashiCorp’s tool, hc-install<\/a>, available on GitHub.<\/p>\n\n\n\n
Handling multiple Terraform states<\/h3>\n\n\n\n
When managing changes within the same environment concurrently in Terraform, consider using Terraform modules or leveraging multiple Terraform workspaces. Modules allow you to encapsulate and reuse configurations, promoting consistency and collaboration. Workspaces provide environmental isolation, making it easy to switch between different states. Additionally, changing the working directory dynamically using variables can help manage distinct environments effectively. Code-wise, this is easily appliable through changing Terraform’s current working directory, or, as can be seen in the code example, the value of the variable terraform_cwd_dir<\/code>.<\/p>\n\n\n\n
Clear text credentials in Terraform files<\/h3>\n\n\n\n
When working with Terraform, you often need to define sensitive information such as passwords, API keys, or access tokens. However, storing these secrets directly in your Terraform configuration or environment files is not a best practice.<\/p>\n\n\n\n
With the use of Terraform Vault, which is a secrets’ management tool that securely stores and manages sensitive data. Credentials can be dynamically accessed by defining placeholders (references) in the Terraform configuration. During runtime, Terraform communicates with the Vault to dynamically generate short-lived based on policies and roles or access existing credentials.<\/p>\n\n\n\n
When retrieving secrets from the Vault, the data is still written in clear text to the following artifacts:<\/p>\n\n\n\n