Kubernetes | 30 November 2023

Using Mutual Transport Layer Security (mTLS) for Agents with the connectware-agent Helm Chart

By default, agents use a password for authentication. As an alternative to password-based authentication you can use mutual TLS (mTLS) as the authentication mechanism for agents. mTLS is an X.509 certificate-based authentication and provides better performance compared to password-based authentication. We recommend using mTLS when handling a large number of agents.

Prerequisites

• mTLS is enabled for Connectware.
• You are familiar with Public Key Infrastructure (PKI) and openssl.

Procedure

To configure agents for mTLS, do the following:

1. Extracting Certificate Authority key pairs
2. Signing certificate key pairs for your agents
3. Configuring agents for key pairs
4. Configuring your agent to use the Certificate Authority
5. Activating mTLS in Connectware
6. Enabling mTLS for the agent

Extracting Certificate Authority Key Pairs

In order to use mTLS authentication, you must extract the Certificate Authority (CA) that Connectware uses to sign certificates that are created for you agents. You can extract the certificate from Connectware or replace it with a CA certificate that you already have. In both cases, you must extract the truststore that Connectware uses, as well as the affiliated private key.

The steps in this section are executed in your Connectware installation, not your connectware-agent installation

Note: For production setups, we recommend that you replace the Public Key Infrastructure (PKI) that is generated during the Connectware installation with a PKI that is managed and approved by your company.

Extracting the CA Key Pair of Connectware

To extract the existing CA key pair, use kubectl cp to copy the certificate from the running auth-server pod via the following commands. Make sure to specify the Kubernetes namespace that contains the Connectware installation.

namespace=<namespace>
pod=$(kubectl -n ${namespace} get pod -o name -lapp.kubernetes.io/name=auth-server | head -1 | sed 's/pod\///g');
kubectl -n ${namespace} cp -c auth-server $pod:/connectware_certs/cybus_ca.key cybus_ca.key
kubectl -n ${namespace} cp -c auth-server $pod:/connectware_certs/cybus_ca.crt cybus_ca.crt
Code-Sprache: YAML (yaml)

Result

The files cybus_ca.crt and cybus_ca.key are created in your current directory.

Using a Custom Certificate Authority (Optional)

For production setups, we recommend that you use a Certificate Authority (CA) that is managed and approved by your company. You can append the certificate of your CA or a valid intermediate CA to the certificate truststore that Connectware uses.

Prerequisites

The following files are available:

• The cybus_ca.crt and cybus_ca.key from Connectware
• The ca-chain.pem CA certificate chain in PEM format
• The server.crt server certificate for Connectware in PEM format, signed by your custom CA
• The server.key matching server.crt
• Connectware is a planned maintenance downtime

Procedure

1. Append your ca-chain.crt to cybus_ca.crt:

<code>cat ca-chain.pem >> cybus_ca.crt</code>
Code-Sprache: YAML (yaml)

2. Upload the following files to Connectware. Make sure to specify the Connectware namespace:
   • Modified cybus_ca.crt
   • Your new server certificate
   • Your new server key

namespace=<namespace>
pod=$(kubectl -n ${namespace} get pod -o name -lapp.kubernetes.io/name=auth-server | head -1 | sed 's/pod\///g');
kubectl -n ${namespace} cp -c auth-server cybus_ca.crt $pod:/connectware_certs/cybus_ca.crt 
kubectl -n ${namespace} cp -c auth-server server.crt $pod:/connectware_certs/cybus_server.crt 
kubectl -n ${namespace} cp -c auth-server server.key $pod:/connectware_certs/cybus_server.key 
kubectl -n ${namespace} exec $pod -c auth-server -- chown -R root.root /connectware_certs
kubectl -n ${namespace} exec $pod -c auth-server -- chmod 664 /connectware_certs/cybus_ca.crt
kubectl -n ${namespace} exec $pod -c auth-server -- chmod 664 /connectware_certs/cybus_ca.key
kubectl -n ${namespace} exec $pod -c auth-server -- chmod 664 /connectware_certs/cybus_server.crt
kubectl -n ${namespace} exec $pod -c auth-server -- chmod 664 /connectware_certs/cybus_server.key
Code-Sprache: YAML (yaml)

3. To apply the new server certificate, restart the deployment of the Connectware Ingress proxy:

<code>namespace=<namespace> kubectl -n ${namespace} rollout restart deployment connectware</code>
Code-Sprache: YAML (yaml)

Signing Certificate Key Pairs for Your Agents

Every agent needs a certificate key pair that is signed by the Certificate Authority (CA) that you want to use. We will assume that the certificate CA files are named cybus_ca.crt and cybus_ca.key and in your current directory.

Note: If you have already signed certificates for your agents, skip this task and continue with Configuring the agent to use your key pair.

The exact parameters for the key pair are subject to your preferences and security requirements. The commands used here are meant as an example.

1. To generate a new key for your agent, enter the following command (Do not set a password for the key):

<code>openssl genrsa -out tls.key 4096</code>
Code-Sprache: YAML (yaml)

2. To generate a Certificate Signing Request (CSR), enter the following command:

openssl req -new -key tls.key -out tls.csr
Code-Sprache: YAML (yaml)

3. Fill out the details for the certificate. Make sure to set Common Name (e.g. server FQDN or YOUR name) to the exact name of the agent that you are generating a certificate for.
4. To sign the CSR, use the following command:

openssl x509 -req -in tls.csr -CA cybus_ca.crt -CAkey cybus_ca.key -CAcreateserial -out tls.crt -days 365 -sha256
Code-Sprache: YAML (yaml)

If you are using other file names than the ones that we are using in this documentation, make sure to use them in the command.

Result

The certificate is created. The certificate is valid for one year. Make sure to create new certificates before the old certificates expire to avoid impact on the operation of the corresponding agents.

Key Pairs for Agents

You can configure key pairs for agents in your values.yaml file or you can create a Kubernetes Secret before you deploy your agent.

Each method has its advantages. However, the most important difference is that a key is considered private data, like a password. If you do not want to store this information in your unencrypted values.yaml file, we recommend that you use the Kubernetes Secret.

Configuring the Agent to Use your Key Pair

You can configure the key pair in the mTLS.keyPair section inside the agents entry in protocolMapperAgents context of your values.yaml file. Alternatively you can create a Kubernetes secret before you deploy your agent.

Each method has its advantages. However, the most important difference is that a key is considered private data, like a password. If you do not want to store this information in your plain text values.yaml file, we recommend that you use the Kubernetes secret.

Configuring Key Pair via Helm Values

To add the key pair to your Helm values, add the respective files as literal block scalars to these Helm values inside the agents entry in protocolMapperAgents context of your values.yaml file:

Value	Content
mTLS.keyPair.cert	The certificate you generated for the agent (tls.crt) as a literal block scalar
mTLS.keyPair.key	The key you generated for the agent (tls.key)  as a literal block scalar

Make sure to stick to the YAML indentation rules. Indent the certificate and key by two spaces relative to cert/key, see the example.

Example

protocolMapperAgents:
  - name: bender-robots
    mTLS:
      keyPair:
        cert: |
          -----BEGIN CERTIFICATE-----
          IIEgTCCAmkCFCN+Wi9RpeajIunZnxdIhvdZep6ZMA0GCSqGSIb3DQEBCwUAMIGN
          [skipped for brevity - include whole certificate]
          sD9hY3o=
          -----END CERTIFICATE-----
        key: |
          -----BEGIN PRIVATE KEY-----
          IIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCg+mC1iGmz+qCO
          [skipped for brevity - include whole private key]
          nJn5oNH9lcodhXcuOYVg3kQ=
          -----END PRIVATE KEY-----
Code-Sprache: YAML (yaml)

Configuring Key Pair via Kubernetes Secret

Creating the Kubernetes secret

If you want to manually manage the certificate and key as a Kubernetes secret you will need to create it before configuring your agent. You can use any process you like, as long as the result is a Kubernetes secret that:

• Is of the type “tls”
• Contains a file called tls.crt containing the signed certificate for the agent
• Contains a file called tls.key containing the matching key for the agent
• Is in the same namespace as your connectware-agent installation

We will demonstrate how to create this secret through kubectl. Please ensure that your agent certificate is stored in a file named tls.crt and your key is stored in a file named tls.key. 

Define a name for your secret. You will need this name later to configure the agent, so keep it at hand. We recommend choosing a name following the scheme “<chart-name>-<release-name>-<agent-name>-mtls”. Some characters, most prominently the “.” character will be replaced by “-” characters in the agent name.

For example if your agent is named bender-robots and you named your installation “connectware-agent” as used in our docs, the name should be “connectware-agent-bender-robots-mtls”. If you follow this naming convention you will need to include less configuration in your values.yaml. 

If you are unsure how to name the secret, you can first deploy your agent without the mTLS configuration, and check the name of its StatefulSet using kubectl get sts. The secret should have the same name with the suffix “-mtls”.

Example

<code>kubectl -n <namespace> create secret tls <secret-name> --key="./tls.key" --cert="./tls.crt"</code>
Code-Sprache: YAML (yaml)

Configuring the agent through Helm values

If you followed the naming convention of “<chart-name>-<release-name>-<agent-name>-mtls” you don’t need to configure the name of the secret, as this name will be assumed. If you have chosen a different name you need to specify it in the Helm value mTLS.keyPair.existingSecret inside the agents entry in protocolMapperAgents context of your values.yaml file.

Example

protocolMapperAgents:
  - name: bender-robots
    mTLS:
      keyPair:
        existingSecret: <secret-name>
Code-Sprache: YAML (yaml)

Certificate Authority for Agents

There are two ways you can configure your agent to use the Certificate Authority (CA):

• Directly through including it in your values.yaml file
• By creating a Kubernetes ConfigMap before deploying your agent

Configuring Certificate Authority for Agents via Helm Values

To add the CA certificate to your Helm values, add the file as literal block scalar to the Helm value mTLS.caChain.cert inside the agents entry in protocolMapperAgents section of your values.yaml file.

Make sure to pay attention to indentation of your CA certificate, it needs to be indented by two spaces relative to cert and keep this indentation (see Example).

If you configure more than one agent, it is recommended to provide the CA certificate through protocolMapperAgentDefaults instead of protocolMapperAgents, because it should be the same for all agents.

Example

protocolMapperAgents:
  - name: bender-robots
    mTLS:
      caChain:
        cert: |
          -----BEGIN CERTIFICATE-----
          MIIFpTCCA40CFGFL86145m7JIg2RaKkAVCOV1H71MA0GCSqGSIb3DQEBCwUAMIGN
          [skipped for brevity - include whole certificate]
          SKnBS1Y1Dn2e
          -----END CERTIFICATE-----
Code-Sprache: YAML (yaml)

Configuring Certificate Authority for Agents via Manual Kubernetes ConfigMap

Alternatively, you can provide the CA certificate as a Kubernetes ConfigMap.

Creating the Kubernetes ConfigMap

If you want to manually manage the CA certificate as a Kubernetes ConfigMap you will need to create it before configuring your agent. You can use any process you like, as long as the result is a Kubernetes ConfigMap that:

• Contains a file called ca-chain.pem containing the CA certificate 
• Is in the same namespace as your connectware-agent installation

We will demonstrate how to create this ConfigMap through kubectl. Please ensure that your CA certificate is stored in a file named ca-chain.pem. Because the CA certificate extracted from Connectware it is named cybus_ca.crt we will create a copy in our example.

Define a name for your ConfigMap. You will need this name later to configure the agent, so keep it at hand. We recommend choosing a name following the scheme “<chart-name>-<release-name>-<agent-name>-mtls-ca-cert”. Some characters, most prominently the “.” character will be replaced by “-” characters in the agent name.

For example if you named your installation “connectware-agent” as used in our docs, the name should be “connectware-agent-mtls-ca-cert”. If you follow this naming convention you will need to include less configuration in your values.yaml. 

Example

cp cybus_ca.crt ca-chain.pem
kubectl create configmap <configmap-name> --from-file ca-chain.pem
Code-Sprache: YAML (yaml)

Configuring the agent through Helm values

If you followed the naming convention of “<chart-name>-<release-name>-<agent-name>-mtls-ca-cert” you don’t need to configure the name of the ConfigMap, as this name will be assumed. If you have chosen a different name you need to specify it in the Helm value mTLS.caChain.existingConfigMap inside the agents entry in protocolMapperAgents context of your values.yaml file

If you configure more than one agent, it is recommended to provide the CA certificate through protocolMapperAgentDefaults instead of protocolMapperAgents, because it should be the same for all agents.

Example

protocolMapperAgents:
  - name: bender-robots
    mTLS:
      caChain:
        existingConfigMap: <configmap-name>
Code-Sprache: YAML (yaml)

Enabling mTLS for the Agent

Finally you will need to enable mTLS for the agent. To do this set the Helm value mTLS.enabled to true inside the agents entry in protocolMapperAgents section of your values.yaml file.

Example

protocolMapperAgents:
  - name: bender-robots
    mTLS:
      enabled: true
Code-Sprache: YAML (yaml)

To apply this configuration to your agent you need to use Helm upgrade on your connectware-agent installation with the same parameters you originally used.

Example

<code>helm upgrade connectware-agent cybus/connectware-agent -f values.yaml -n <namespace></code>
Code-Sprache: YAML (yaml)

Replacing mTLS Certificates and Keys for the connectware-agent Helm Chart

If you want to replace certificates or keys you follow the same steps as when adding them, however the agents will not automatically start using the new certificates. You will need to manually restart the Kubernetes StatefulSets associated with the agents for which you replaced certificates. This StatefulSet is named “<chart-name>-<release-name>-<agent-name>”. Some characters, most prominently the “.” character will be replaced by “-” characters in the agent name.

If you followed the recommendations in these docs the first two parts are abbreviated to “connectware-agent”. An agent named “bender-robots” for example would then be deployed as a StatefulSet named “connectware-agent-bender-robots”.

<code>kubectl -n <namespace> rollout restart sts <statefulset-name></code>
Code-Sprache: YAML (yaml)

This will restart the agent and apply the new certificates/key.

If you want to restart all agents from your installation, you can use this command in combination with the name you gave to your agent deployment:

<code>kubectl -n <namespace> rollout restart sts -l app.kubernetes.io/instance=<release-name></code>
Code-Sprache: YAML (yaml)

If you followed the recommendations in these docs, <release-name> is “connectware-agent”.

Full mTLS Examples for the connectware-agent Helm Chart

Two Agents with Manually Created Kubernetes Secrets/Configmap with Default Names

This example assumes:

• You created a ConfigMap with the name “connectware-agent-mtls-ca-cert” containing the CA certificate as a file called “ca-chain.pem”.
• You created a secret named “connectware-agent-bender-robots-mtls” containing a certificate for the Common Name (CN) “bender-robots”, signed by the CA, as a file called “tls.crt” and the matching key as a file called “tls.key”.
• You created a secret named “connectware-agent-welder-robots-mtls” containing a certificate for the Common Name (CN) “welder-robots”, signed by the CA, as a file called “tls.crt” and the matching key as a file called “tls.key”.
• You named your connectware-agent Helm installation “connectware-agent”.
• You Connectware installation is located in the namespace “cybus”.

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus
  mTLS:
    enabled: true
protocolMapperAgents:
  - name: bender-robots
  - name: welder-robots
Code-Sprache: YAML (yaml)

Two Agents with Manually Created Kubernetes Secrets/Configmap with Custom Names

This example assumes:

• You created a ConfigMap with the name “my-ca-cert” containing the CA certificate as a file called “ca-chain.pem”.
• You created a secret named “mtls-keypair-1” containing a certificate for the Common Name (CN) “bender-robots”, signed by the CA, as a file called “tls.crt” and the matching key as a file called “tls.key”.
• You created a secret named “mtls-keypair-2” containing a certificate for the Common Name (CN) “welder-robots”, signed by the CA, as a file called “tls.crt” and the matching key as a file called “tls.key”.
• You named your connectware-agent Helm installation “connectware-agent”.
• You Connectware installation is located in the namespace “cybus”.

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus
  mTLS:
    enabled: true
    caChain:
      existingConfigMap: my-ca-cert
protocolMapperAgents:
  - name: bender-robots
    mTLS:
      keypair:
        existingSecret: mtls-keypair-1
  - name: welder-robots
    mTLS:
      keypair:
        existingSecret: mtls-keypair-2
Code-Sprache: YAML (yaml)

Two Agents with Manually Created CA Certificate Configmap but Key Pair in Helm Values

This example assumes:

• You created a ConfigMap with the name “connectware-agent-mtls-ca-cert” containing the CA certificate as a file called “ca-chain.pem”.
• You named your connectware-agent Helm installation “connectware-agent”.
• You Connectware installation is located in the namespace “cybus”.

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus
  mTLS:
    enabled: true
protocolMapperAgents:
  - name: bender-robots
    mTLS:
      keyPair:
        cert: |
          -----BEGIN CERTIFICATE-----
          MIIEgTCCAmkCFCN+Wi9RpeajIunZnxdIhvdZep6ZMA0GCSqGSIb3DQEBCwUAMIGN
          [skipped for brevity - include whole certificate]
          sD9hY3o=
          -----END CERTIFICATE-----
        key: |
          -----BEGIN PRIVATE KEY-----
          MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCg+mC1iGmz+qCO
          [skipped for brevity - include whole private key]
          nJn5oNH9lcodhXcuOYVg3kQ=
          -----END PRIVATE KEY-----
  - name: welder-robots
    mTLS:
      keyPair:
        cert: |
          -----BEGIN CERTIFICATE-----
          MIIFcjCCA1oCFFgO7SgdLBuU6YBOuZxhQg0eW5f+MA0GCSqGSIb3DQEBCwUAMIGN
          [skipped for brevity - include whole certificate]
          VM6E0Lqy
          -----END CERTIFICATE-----
        key: |
          -----BEGIN PRIVATE KEY-----
          MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDvmp+v3+x1am6m
          [skipped for brevity - include whole private key]
          Y6vWPIuRCwum9DxjrdIva6Z6Pqkdyed9
          -----END PRIVATE KEY-----
Code-Sprache: YAML (yaml)

Two Agents with Manually Created CA Certificate and Key Pair in Helm Values

This example assumes:

• You named your connectware-agent Helm installation “connectware-agent”.
• You Connectware installation is located in the namespace “cybus”.

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus
  mTLS:
    enabled: true
    caChain:
      cert: |
        -----BEGIN CERTIFICATE-----
        MIIFpTCCA40CFGFL86145m7JIg2RaKkAVCOV1H71MA0GCSqGSIb3DQEBCwUAMIGN
        [skipped for brevity - include whole certificate]
        SKnBS1Y1Dn2e
        -----END CERTIFICATE-----
protocolMapperAgents:
  - name: bender-robots
    mTLS:
      keyPair:
        cert: |
          -----BEGIN CERTIFICATE-----
          MIIEgTCCAmkCFCN+Wi9RpeajIunZnxdIhvdZep6ZMA0GCSqGSIb3DQEBCwUAMIGN
          [skipped for brevity - include whole certificate]
          sD9hY3o=
          -----END CERTIFICATE-----
        key: |
          -----BEGIN PRIVATE KEY-----
          MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCg+mC1iGmz+qCO
          [skipped for brevity - include whole private key]
          nJn5oNH9lcodhXcuOYVg3kQ=
          -----END PRIVATE KEY-----
  - name: welder-robots
    mTLS:
      keyPair:
        cert: |
          -----BEGIN CERTIFICATE-----
          MIIFcjCCA1oCFFgO7SgdLBuU6YBOuZxhQg0eW5f+MA0GCSqGSIb3DQEBCwUAMIGN
          [skipped for brevity - include whole certificate]
          VM6E0Lqy
          -----END CERTIFICATE-----
        key: |
          -----BEGIN PRIVATE KEY-----
          MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDvmp+v3+x1am6m
          [skipped for brevity - include whole private key]
          Y6vWPIuRCwum9DxjrdIva6Z6Pqkdyed9
          -----END PRIVATE KEY-----
Code-Sprache: YAML (yaml)