HashiCorp Vault CA

You can configure Kong Mesh to communicate with a Vault.

The default mTLS policy in Kong Mesh supports the following backends:

• builtin: Kong Mesh automatically generates the Certificate Authority (CA) root certificate and key that will be used to generate the data plane certificates.
• provided: the CA root certificate and key can be provided by the user.

Kong Mesh adds:

• vault: Kong Mesh generates Data Plane certificates using a CA root certificate and key stored in a HashiCorp Vault server.
• acmpca: Kong Mesh generates Data Plane certificates using Amazon Certificate Manager Private CA.
• certmanager: Kong Mesh generates Data Plane certificates using Kubernetes cert-manager certificate controller.

In vault mTLS mode, Kong Mesh communicates with the HashiCorp Vault PKI, which generates the Data Plane proxy certificates automatically. Kong Mesh does not retrieve private key of the CA to generate Data Plane proxy certificates, which means that private key of the CA is secured by Vault and not exposed to third parties.

In vault mode, you point Kong Mesh to the Vault server and provide the appropriate credentials. Kong Mesh uses these parameters to authenticate the Control Plane and generate the Data Plane certificates.

When Kong Mesh is running in vault mode, the backend communicates with Vault and ensures that Vault’s PKI automatically issues Data Plane certificates and rotates them for each proxy.

If Kong Mesh is configured to authenticate to Vault using a renewable token, it will handle keeping the token renewed.

The vault mTLS backend expects a configured PKI and role for generating Data Plane proxy certificates.

The following steps show how to configure Vault for Kong Mesh with a mesh named default. For your environment, replace default with the appropriate mesh name.

Configure the Certificate Authority

Kong Mesh works with a Root CA or an Intermediate CA.

vault secrets enable -path=kmesh-pki-default pki

vault secrets tune -max-lease-ttl=87600h kmesh-pki-default

vault write -field=certificate kmesh-pki-default/root/generate/internal \
  common_name="Kong Mesh Default" \
  uri_sans="spiffe://default" \
  ttl=87600h

vault secrets enable pki

vault secrets tune -max-lease-ttl=87600h pki

vault write -field=certificate pki/root/generate/internal \
  common_name="Organization CA" \
  ttl=87600h > ca.pem

vault secrets enable -path=kmesh-pki-default pki

vault write -format=json kmesh-pki-default/intermediate/generate/internal \
    common_name="Kong Mesh Mesh Default" \
    uri_sans="spiffe://default" \
    | jq -r '.data.csr' > pki_intermediate.csr

vault write -format=json pki/root/sign-intermediate csr=@pki_intermediate.csr \
  format=pem_bundle \
  ttl="43800h" \
  | jq -r '.data.certificate' > intermediate.cert.pem

cat intermediate.cert.pem > bundle.pem
echo "" >> bundle.pem
cat ca.pem >> bundle.pem
vault write kmesh-pki-default/intermediate/set-signed certificate=@bundle.pem

Create a role for generating Data Plane proxy certificates

Run the following to create a role for generating Data Plane proxy certificates:

vault write kmesh-pki-default/roles/dataplane-proxies \
  allowed_uri_sans="spiffe://default/*,kuma://*" \
  key_usage="KeyUsageKeyEncipherment,KeyUsageKeyAgreement,KeyUsageDigitalSignature" \
  ext_key_usage="ExtKeyUsageServerAuth,ExtKeyUsageClientAuth" \
  client_flag=true \
  require_cn=false \
  allowed_domains="mesh" \
  allow_subdomains=true \
  basic_constraints_valid_for_non_ca=true \
  max_ttl="720h" \
  ttl="720h"

Note: Use the allowed_domains and allow_subdomains parameters only when commonName is set in the mTLS Vault backend.

Create a policy to use the new role

Run the following to create a policy to use the new role:

cat > kmesh-default-dataplane-proxies.hcl <<- EOM
path "/kmesh-pki-default/issue/dataplane-proxies"
{
  capabilities = ["create", "update"]
}
EOM
vault policy write kmesh-default-dataplane-proxies kmesh-default-dataplane-proxies.hcl

Configure authentication method

To authorize Kong Mesh to access Vault using a token, generate the following orphan token and pass it to the mesh:

vault token create -type=service -orphan -format=json -policy="kmesh-default-dataplane-proxies" | jq -r ".auth.client_token"

We suggest using an orphan token to avoid surprising behavior around expiration in token hierarchies. You need root/sudo permissions to execute the previous command. The output should print a Vault token that you then provide as the conf.fromCp.auth.token value of the Mesh object.

Note: There are some failure modes where the vault CLI still returns a token even though an error was encountered and the token is invalid. For example, if the policy creation fails in the previous step, then the vault token create command both returns a token and exposes an error. In such situations, using jq to parse the output hides the error message provided in the vault CLI output. Manually parse the output instead of using jq so that the full output of the vault CLI command is available.

Kong Mesh also supports AWS Instance Role authentication to Vault. Vault must be configured to accept EC2 or IAM role authentication. See Vault documentation for details. With Vault configured, select AWS authentication in the Mesh object by setting conf.fromCp.auth.aws. Kong Mesh will authenticate using the instance or IRSA role available within the environment.

Configure Mesh

kuma-cp communicates directly with Vault. To connect to Vault, you must provide credentials in the configuration of the mesh object of kuma-cp.

You can authenticate with the token, with client certificates by providing clientKey and clientCert, or by AWS role-based authentication.

You can provide these values inline for testing purposes only, as a path to a file on the same host as kuma-cp, or contained in a secret. When using a secret, it should be a [mesh-scoped secret][secrets]. On Kubernetes, this mesh-scoped secret should be stored in the system namespace (kong-mesh-system by default) and should be configured as type: system.kuma.io/secret.

Here’s an example of a configuration with a vault-backed CA:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: vault-1
    backends:
      - name: vault-1
        type: vault
        dpCert:
          rotation:
            expiration: 1d # must be lower than max_ttl in Vault role
        conf:
          fromCp:
            address: https://vault.8200
            agentAddress: "" # optional
            namespace: "" # optional
            pki: kmesh-pki-default # name of the configured PKI
            role: dataplane-proxies # name of the role that will be used to generate Data Plane proxy certificates
            commonName: '{{ tag "kuma.io/service" }}.mesh' # optional. If set, then commonName is added to the certificate. You can use "tag" directive to pick a tag which will be base for commonName.

            tls: # options for connecting to Vault via TLS
              skipVerify: false   # if set to true, caCert is optional, should only be used in development
              caCert:             # caCert is used to verify the TLS certificate presented by Vault
                secret: sec-1     # one of secret, inline, or inlineString
              serverName: ""      # optional. The SNI to use when connecting to Vault

            auth: # how to authenticate Kong Mesh when connecting to Vault
              token:
                secret: token-1  # one of secret, inline, or inlineString
              tls:
                clientKey:
                  secret: sec-2  # can be file, secret or inline
                clientCert:
                  file: /tmp/cert.pem # can be file, secret or inlineString
              aws: # AWS role-based authentication. May be empty to use defaults.
                type: "IAM" or "EC2" # Optional AWS authentication type. Default is IAM.
                role: role-name # Optional role name to use for IAM authentication
                iamServerIdHeader: example.com # Optional server ID header value

type: Mesh
name: default
mtls:
  enabledBackend: vault-1
  backends:
  - name: vault-1
    type: vault
    dpCert:
      rotation:
        expiration: 24h # must be lower than max_ttl in Vault role
    conf:
      fromCp:
        address: https://vault.8200
        agentAddress: "" # optional
        namespace: "" # optional
        pki: kmesh-pki-default # name of the configured PKI
        role: dataplane-proxies # name of the role that will be used to generate Data Plane proxy certificates
        commonName: '{{ tag "kuma.io/service" }}.mesh' # optional. If set, then commonName is added to the certificate. You can use "tag" directive to pick a tag which will be base for commonName.
        tls:
          caCert:
            secret: sec-1
          skipVerify: false # if set to true, caCert is optional. Set to true only for development
          serverName: "" # verify sever name
        auth: # how to authenticate Kong Mesh when connecting to Vault
          token:
            secret: token-1  # can be file, secret or inlineString
          tls:
            clientKey:
              secret: sec-2  # can be file, secret or inlineString
            clientCert:
              file: /tmp/cert.pem # can be file, secret or inline
            aws: # AWS role-based authentication. May be empty to use defaults.
              type: "IAM" or "EC2" # Optional AWS authentication type. Default is IAM.
              role: role-name # Optional role name to use for IAM authentication
              iamServerIdHeader: example.com # Optional server ID header value

Apply the configuration with kumactl apply -f [..].

If you’re running in Universal mode, you can also use the [HTTP API][http-api] to apply configuration.

Kong Mesh uses Service Alternative Name with spiffe:// format to verify secure connection between Services. In this case, the common name in the certificate is not used. You may need to set a common name in the certificate, for compliance reasons. To do this, set the commonName field in the Vault mTLS backend configuration. The value contains the template that will be used to generate the name.

For example, assuming that the template is '{{ tag "kuma.io/service" }}.mesh', a Data Plane proxy with kuma.io/service: backend tag will receive a certificate with the backend.mesh common name.

You can also use the replace function to replace _ with -. For example, '{{ tag "kuma.io/service" | replace "_" "-" }}.mesh' changes the common name of kuma.io/service: my_backend from my_backend.mesh to my-backend.mesh.

In a multi-zone environment, the global Control Plane provides the Mesh to the zone Control Planes. However, you must make sure that each zone Control Plane communicates with Vault over the same address. This is because certificates for Data Plane proxies are issued from the zone Control Plane, not from the global Control Plane.