Skip to content

alt text

IaC Variable Management

Part of our Infrastructure as Code (IaC) efforts are to ensure that our cloud systems are able to be restored from backup easily and reliably. This relies on having a system for using secret values within our code that satisfies the following conditions:

  • Security: Variables must be stored in locations that are protected by authentication and authorization layers
  • Accessibility: Variables must be pulled from their storage location as needed at runtime
  • Recovery: Variables must have the ability to be restored from a backup location in the case of lost credentials

The solution I suggest makes use of Azure Key Vault for storage of secrets and keys, with Delinea set as our backup in case of failure.

Unsecured Variables

Let's say you have an IaC component that will need to rely on a insecure variable's value. This can be for any variety of reasons, including a difference of environments (dev vs. prod) or updates to the application (v1 vs. v2 vs. latest). The first thing you should ask yourself is whether there is any disadvantage towards keeping the variable stored locally as part of the code package. Examples of such values include:

  • Resource group names
  • Virtual network names
  • Model names

If it is truly a variable that is ok to be stored in an unsecured format (like plaintext), it is better to keep it directly in the code itself. This way, it can be tracked using Git. Terraform and Pulumi accomplish this in different ways:

Terraform Local Vars

You have two ways to go about this:

Method 1

Store the variable as part of your locals.tf file as in this example below. They can then be referenced by using local.var1 or local.var2 in the code:

    locals {
        var1 = "hello-world!"
        var2 = "foo-bar-baz"
    }

Method 2

Store the variable in your .tfvars files. These files are not committed to Git. Up until this point, we have been storing them using SharePoint, which will no longer be allowed. If you choose to use this method, it is up to you to justify why and provide a method of accessing these files other than Git that is acceptable to all other parties. Referencing these variables in code is done using var.var1 or var.var2:

    var1 = "hello-world!
    var2 = "foo-bar-bas"

Pulumi Local Vars

Pulumi variable management is more straightforward. These variables are stored according to the Pulumi stack being used in the Pulumi.<stack>.yaml file. These variables can either be encrypted or left as plaintext. The advantage of keeping things as plaintext is that changes made to variables can be tracked with source control in a readable fashion. As such, it comes with the responsibility of not keeping important values as plaintext. Instead, they should be added as "secure" values that will be encrypted using a passphrase (stored in environment vars as PULUMI_CONFIG_PASSPHRASE).

An example of this might look like this:

encryptionsalt: v1:<long_hex_string>
config:
    project-name:var1: hello-world
    project-name:var2:
        secure: v1:<long_hex_string>

Keep in mind that these values will tracked by source control. To add a new variable to a project:

export PULUMI_CONFIG_PASSPHRASE=<passphrase_value>
pulumi config set <key> <value>

To add an encrypted variable value:

pulumi config set --secret <key> <value>

The <passphrase_value> can be acquired by talking to a developer. It is also stored in our Azure Key Vault.

To use the value in Python code:

pulumi_config = pulumi.Config()
var1 = pulumi_config.require("var1")
var2 = pulumi_config.require_secret("var2")

Secure Variables

There may also the need to store certain sensitive variables using more secured methods. Examples of these values include:

  • Service principal credentials
  • API keys
  • Usernames/passwords

Although Pulumi offers the option to store encrypted values, there may be times when such values require additional protection. In this case, these values should be stored in our Azure Key Vault, where they can be audited, rotated, and governed. This also necessitates that a Key Vault be created in the right resource group ahead of deploying further IaC components. Don't skip this step as it will also tie into the recovery process!

Key Vault Setup

A Key Vault should be set up in each resource group for each environment that will be targeted by IaC. For example, there should be a dev Key Vault in the dev resource group and a prod Key Vault in the prod resource group before anything else can be created. You can accomplish this setup by any method you choose (IaC, Azure Portal, CLI, etc.). Here is an example of how to accomplish this via the Azure CLI:

az keyvault create \
    --name "<your-unique-keyvault-name>-dev" \
    --resource-group "myResourceGroup-dev"

The important thing is that the Key Vault should initially be empty and populated with secrets as needed. We will be syncing this Key Vault with Delinea as described later in this document.

Ensure that you have set up a service principal for this Key Vault that is capable of reading secrets (Key Vault Secrets User) and permissions for the development team to upload secrets (Key Vault Contributor).

Populating Key Vault

For now, this is done manually. All necessary values must be entered via the Azure Portal based on previously known Key Vault values or desired configuration. My proposal is that we use Delinea (already used at CNH) to support backing up our Key Vaults regularly to a known, secured third-party location. It will also support synchronized entry so there can be a single place for the team to maintain secrets.

The documentation for Delinea integration is available online, but may require further conversations with the admins to allow integrations to be installed. We also need the ability to create Azure App Registrations in our resource group. If this goes through, we will gain the following capabilities:

  • Create secrets in Delinea that are pushed up automatically to a registered Azure Key Vault
  • Manage secrets that synchronize automatically to a registered Azure Key Vault
  • Rotate secrets with automatic synchronization
  • Recover Key Vaults by setting new ones to synchronize to a Delinea backup
  • Assign ownership over backed-up secrets at the group level

Terraform Remote Vars

To pull values from Azure Key Vault and use them in code:

data "azurerm_key_vault" "kv" {
    name                = "key_vault_name"
    resource_group_name = "resource_group_name"
}

data "azurerm_key_vault_secret" "var1" {
    name         = "var1"
    key_vault_id = data.azurerm_key_vault.kv.id
}

output "secret_value" {
    value = data.azurerm_key_vault_secret.example.value
}

The first data block sets up a reference to a Key Vault object that is capable of pulling secret values. These values can then be used in the rest of the code wherever a secure variable is needed by referencing the name of that variable.

Pulumi Remote Vars

To pull values from Azure Key Vault and use them in code:

from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

pulumi_config = pulumi.Config()
key_vault_name = pulumi_config.require("key-vault-name")
resource_group_name = pulumi_config.require("resource-group-name")

credential = DefaultAzureCredential()
client = SecretClient(
    vault_url=f"https://{key_vault_name}.vault.azure.net", 
    credential=credential
)

var1 = secret_client.get_secret("var1")
var2 = secret_client.get_secret("var2")

This requires that you have stored key-vault-name and resource-group-name in your stack's configuration:

encryptionsalt: v1:<long_hex_string>
config:
    project-name:key-vault-name: key_vault_name
    project-name:resource-group-name: resource_group_name

We won't be able to use Pulumi's azure-native method of referencing Key Vault objects as that capability has not been supported yet. It is currently open as an issue on GitHub. As such, you must take care during deployment not to expose any secrets through print statements or logs.