Chef Habitat + Vault = vault-helper

One of the ongoing challenges for any Chef Habitat deployment is efficient and safe management of secrets. After much discussion with the internal team at Indellient, Inc., we decided Chef Habitat needed a service like consul-template, that allowed for parsing of tokens in arbitrary hooks and files in the Chef Habitat package when the service is started. It must include the ability to use a different left and right delimiter when parsing secrets, like: (( .Username )), so it doesn’t conflict with Chef Habitat handle-bar helpers, and should be 100% self-contained (no external dependencies). It also must interact with HashiCorp’s Vault service. Below is a run through of a solution, Vault-Helper, that I created to solve this.

Vault Setup

This particular case assumes that a Vault server is already running and you have some secrets at the locations shown below:

Vault Secrets

vault kv get -format=json secret/jenkins/prod/git-credential
{
  "request_id": "139fd6e6-2682-185e-5b36-298024621f10",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "username": "administrator",
      "password": "bacon"
    },
    "metadata": {
      "created_time": "2019-06-18T19:59:42.704428423Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "warnings": null
}

Detailed vault configuration and secrets management is out of scope for this article, however, I strongly recommend that if you do use vault-helper, do not use two different paths for secrets in the same file (as two separate invocation of vault-helper using parse will result in two writes of the file, leading to a race condition). Make sure to specify all secrets for a given file (or even multiple files) at the same Vault URL.

Why Not Use consul-template?

That’s all well and good, but why not use consul-template as-is? At the time we were evaluating consul-template to be used in the Chef Habitat hooks, you could only change the left and right delimiters by specifying a HCL configuration file with the delimiters overridden (see below), and whenever you invoked consul-template, it would have to point at that configuration file. Additionally, if Chef Habitat saw handle-bar like template values like {{ .Username }} in a hook, it would choke and not be able to start the service: Failed to compile configuration: Error rendering "init.groovy" line 111, col 30: Invalid JSON path.

vault-template/habitat/config/template.hcl

template {
  [... snip ...]
  
  left_delimiter  = "(("
  right_delimiter = "))"
}

This was a bit more cumbersome than I wanted (generate the template configuration file, and then invoke consul-template against the target file specifying the configuration file), as I really wanted something to meet all of these requirements:

A single binary that has zero external libraries or dependencies.
Could parse a single text-token like ((.username)) or a whole file with multiple text/template declarations.
No external configuration files; all options can be specified at invocation time or using environment variables.
It will automagically revoke a generated token once it’s done parsing a file via the parse sub command.

vault-helper

vault-helper meets all of the above requirements, can be invoked in a number of ways, and supports the full Golang text/template functionality, using (( & )) for delimiters.

vault-helper is available on the public Chef Habitat depot:

hab pkg install indellient/vault-helper

Examples

Generate a new approle token:
        vault-helper token create --addr="http://somewhere:8200" --role-id="dead-beef" --secret-id="ea7-beef"

Renew an existing token (non-zero exit if the token cannot be renewed):
        vault-helper token renew --addr="http://somewhere:8200" --token="dead-c0de"

Revoke an existing token (non-zero exit if the token cannot be revoked):
        vault-helper token revoke --addr="http://somewhere:8200" --token="dead-c0de"

Fetch a secret:
        vault-helper secret --addr="http://somewhere:8200" --token="dead-c0de" --path="secret/data/jenkins/dev/user/admin" --selector="((.username))"

Parse a file:
        vault-helper parse --addr="http://somewhere:8200" --role-id="dead-beef" --secret-id="ea7-beef" --path="secret/data/jenkins/dev/user/admin" --file="init.groovy"

See vault-helper --help for more information.

Example – Jenkins

Let’s assume for a minute you’ve created a Jenkins Chef Habitat package. You created an init.groovy that gets run by Jenkins when it’s first started up, and it has some content like this:

jenkins/habitat/config/init.groovy

[... snip ...]
if ("{{toLowercase cfg.admin.strategy}}" == "default") {
  def hudsonRealm = new HudsonPrivateSecurityRealm(false)
  hudsonRealm.createAccount("((.username))", "((.password))")
  instance.setSecurityRealm(hudsonRealm)

  def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
  strategy.setAllowAnonymousRead(false)
  instance.setAuthorizationStrategy(strategy)
}
[... snip ...]

Basically, you want your default Jenkins administrator username/password to come from Vault, but you also need to be able to test locally in Test Kitchen. You also want a Vault Approle to be used when communicating with Vault to fetch the credentials, and to automatically revoke the token used.

In order to meet the requirements above for vault-helper, you need to set up your run hook like this:

jenkins/habitat/hooks/run

[... snip ...]
# We only want this codepath to run if the vault address, role_id, and secret_id are all specified
if [ "{{cfg.vault.address}}" != "" ] && [ "{{cfg.vault.role_id}}" != "" ] && [ "{{cfg.vault.secret_id}}" != "" ] ; then
    export VAULT_ADDR="{{cfg.vault.address}}"
    export VAULT_ROLE_ID="{{cfg.vault.role_id}}"
    export VAULT_SECRET_ID="{{cfg.vault.secret_id}}"
    vault-helper --log-level=debug parse --path={{cfg.vault.base}}/{{cfg.vault.environment}}/{{cfg.vault.admin_secrets.path}} --file={{pkg.svc_data_path}}/init.groovy
else
    # In the default case, replace with the specified username/password from TOML
    sed -i "s/((.username))/{{cfg.admin.username}}/; s/((.password))/{{cfg.admin.password}}/" {{pkg.svc_data_path}}/init.groovy
fi

# Start Jenkins
exec java \
  {{cfg.java.opts}} \
  -jar "${JENKINS_WAR}" \
  --prefix={{cfg.jenkins.prefix}} \
  [...]

The important parts are line 7 and line 10. Line 7 actually parses the init.groovy file and writes the specified text file back to disk after pulling the secrets from Vault using the specified path and VAULT_ADDR/VAULT_ROLE_ID/VAULT_SECRET_ID from the environment.

If you are not using Vault (e.g., running a local Test Kitchen instance), line 10 will replace the ((.username)) and ((.password)) tokens in the init.groovy with values from either default.toml or user.toml.

The other added benefit to this approach, is that you can update the secret value in Vault and trigger a re-start of the Jenkins Chef Habitat service to have it re-fetch the default ((.username)) / ((.password)), and parse them in to your init.groovy.

Caveats

Right now, the binary does not support being run as a process inside of a Chef Habitat service hook. The downside to this is that vault-helper can’t “watch” a location in Vault for secrets updates, and when they change, parse a file with the new secrets and send a restart signal to a dependent Chef Habitat package. That might be a good extension in the future .

As mentioned earlier, don’t parse the same file with two different URL locations in Vault, using two separate vault-helper parse … invocations, as bad things will happen.

Interested in vault-helper?

If you’re interested in using vault-helper, you can access it here:
https://github.com/Indellient/vault-helper

Indellient is a Canadian Software Development Company that specializes in Data Analytics, DevOps Services, and Business Process Management