This Cybus Kubernetes guide provides a detailed procedure to help admins adjust the persistent volume content permissions to ensure a smooth upgrade to Connectware 1.5.0 in Kubernetes environments.

Important: Connectware 1.5.0 introduces a significant change regarding container privileges. Now, containers are started without root privileges. This change causes an issue where files that persisted on volumes that were created by a user other than the one accessing them, cannot easily have their permissions changed. As a result, this upgrade requires the admin to manually update the permissions.

Prerequisites

Overview of the Upgrade

Before you begin the upgrade process, ensure that you have made the necessary preparations and that all relevant stakeholders are involved.

The following steps give you an overview of what you need to do. See below for detailed step-by-step instructions.

  1. Create the YAML snippets: Create YAML snippets for each PersistentVolumeClaim used by Connectware. This step is essential for identifying the volumes that need permission adjustments.
  2. Filter volumes: Carefully review the created snippets to ensure that only related volumes are included. This is critical to avoid unnecessary modifications and potential disruptions.
  3. Update the Kubernetes job file: Integrate the snippets into the kubernetes-job.yml file as described.
  4. Shutdown Connectware: Securely shut down all Connectware and agent pods to prevent any access conflicts during the permissions update. This step is crucial to ensure that the permissions changes are applied without interference.
  5. Run the Kubernetes job: Run the job to adjust permissions on all relevant volumes. This action addresses the core challenge posed by the upgrade and ensures that the new, non-root containers can access the files they need.
  6. Delete Kubernetes services: Remove all services used by Connectware in preparation for the new version. This step is necessary to accommodate updates that are incompatible with the existing service configurations.
  7. Apply Helm upgrade: Proceed with the Helm upgrade to Connectware version 1.5.0. This step completes the upgrade process and implements the new security measures and permissions model.

Important Notes

Upgrading Procedure

1. Creating YAML Snippets for relevant PersistentVolumeClaim names

#!/usr/bin/env bash
#
# Creates a Kubernetes resource list of volumeMounts and volumes for Connectware (agent) deployments
#
function usage {
    printf "Usage:\n"
    printf "$0 [--kubeconfig|-kc <kubeconfig_file>] [--context|-ctx <target_context>] [--namespace|-n <target_namespace>] [--no-external-agents] [--no-connectware]\n"
    exit 1
}

function argparse {

  while [ $# -gt 0 ]; do
    case "$1" in
      --kubeconfig|-kc)
        # a kube_config file for the Kubernetes cluster access
        export KUBE_CONFIG="${2}"
        shift
        ;;
      --context|-ctx)
        # a KUBE_CONTEXT for the Kubernetes cluster access
        export KUBE_CONTEXT="${2}"
        shift
        ;;
      --namespace|-n)
        # the Kubernetes cluster namespace to operate on
        export NAMESPACE="${2}"
        shift
        ;;
      --no-external-agents)
        export NO_EXTERNAL_AGENTS=true
        shift
        ;;
      --no-connectware)
        export NO_CONNECTWARE=true
        shift
        ;;
      *)
        printf "ERROR: Parameters invalid\n"
        usage
    esac
    shift
  done
}

#
# init
export NO_EXTERNAL_AGENTS=false
export NO_CONNECTWARE=false
export KUBECTL_BIN=$(command -v kubectl)

argparse $*

shopt -s expand_aliases
# Check for kubectl paramaters and construct the ${KUBECTL_CMD} command to use
if [ ! -z $KUBE_CONFIG ]; then
  KUBECONFIG_FILE_PARAM=" --kubeconfig=${KUBE_CONFIG}"
fi
if [ ! -z $KUBE_CONTEXT ]; then
  KUBECONFIG_CTX_PARAM=" --context=${KUBE_CONTEXT}"
fi
if [ ! -z $NAMESPACE ]; then
  NAMESPACE_PARAM=" -n${NAMESPACE}"
fi

KUBECTL_CMD=${KUBECTL_BIN}${KUBECONFIG_FILE_PARAM}${KUBECONFIG_CTX_PARAM}${NAMESPACE_PARAM}

volumes=$(${KUBECTL_CMD} get pvc -o name | sed -e 's/persistentvolumeclaim\///g' )
valid_pvcs=""
volume_yaml=""
volumemounts_yaml=""

if [[ "${NO_CONNECTWARE}" == "false" ]]; then
  valid_pvcs=$(cat << EOF
system-control-server-data
certs
brokerdata-*
brokerlog-*
workbench
postgresql-postgresql-*
service-manager
protocol-mapper-*
EOF
)
fi

if [[ "${NO_EXTERNAL_AGENTS}" == "false" ]]; then
  # Add volumes from agent Helm chart
  pvc_volumes=$(${KUBECTL_CMD} get pvc -o name -l connectware.cybus.io/service-group=agent | sed -e 's/persistentvolumeclaim\///g' )
  valid_pvcs="${valid_pvcs}
  ${pvc_volumes}"
fi

# Collect volumeMounts
for pvc in $volumes; do
  for valid_pvc in $valid_pvcs; do
      if [[ "$pvc" =~ $valid_pvc ]]; then
        volumemounts_yaml="${volumemounts_yaml}
          - name: $pvc
            mountPath: /mnt/connectware_$pvc"
        break
      fi
  done
done

# Collect volumes
for pvc in $volumes; do
  for valid_pvc in $valid_pvcs; do
      if [[ "$pvc" =~ $valid_pvc ]]; then
        volume_yaml="${volume_yaml}
        - name: $pvc
          persistentVolumeClaim:
            claimName: $pvc"
          break
      fi
  done
done


# print volumeMounts
echo
echo "Copy this as the \"volumeMounts:\" section:"
echo "######################################"
echo -n "        volumeMounts:"
echo "$volumemounts_yaml"

# print volumes
echo
echo "Copy this as the \"volumes:\" section:"
echo "######################################"
echo -n "      volumes:"
echo "$volume_yaml"
Code-Sprache: YAML (yaml)
kubectl config use-context <my-cluster>
kubectl config set-context --current --namespace <my-connectware-namespace>
Code-Sprache: YAML (yaml)
chmod u+x create-pvc-yaml.sh
./create-pvc-yaml.sh
Code-Sprache: YAML (yaml)

Example output:

Copy this as the "volumeMounts:" section:
######################################
        volumeMounts:
          - name: brokerdata-broker-0
            mountPath: /mnt/connectware_brokerdata-broker-0
          - name: brokerdata-control-plane-broker-0
            mountPath: /mnt/connectware_brokerdata-control-plane-broker-0
          - name: brokerlog-broker-0
            mountPath: /mnt/connectware_brokerlog-broker-0
          - name: brokerlog-control-plane-broker-0
            mountPath: /mnt/connectware_brokerlog-control-plane-broker-0
          - name: certs
            mountPath: /mnt/connectware_certs
          - name: postgresql-postgresql-0
            mountPath: /mnt/connectware_postgresql-postgresql-0
          - name: protocol-mapper-agent-001-0
            mountPath: /mnt/connectware_protocol-mapper-agent-001-0
          - name: service-manager
            mountPath: /mnt/connectware_service-manager
          - name: system-control-server-data
            mountPath: /mnt/connectware_system-control-server-data
          - name: workbench
            mountPath: /mnt/connectware_workbench

Copy this as the "volumes:" section:
######################################
      volumes:
        - name: brokerdata-broker-0
          persistentVolumeClaim:
            claimName: brokerdata-broker-0
        - name: brokerdata-control-plane-broker-0
          persistentVolumeClaim:
            claimName: brokerdata-control-plane-broker-0
        - name: brokerlog-broker-0
          persistentVolumeClaim:
            claimName: brokerlog-broker-0
        - name: brokerlog-control-plane-broker-0
          persistentVolumeClaim:
            claimName: brokerlog-control-plane-broker-0
        - name: certs
          persistentVolumeClaim:
            claimName: certs
        - name: postgresql-postgresql-0
          persistentVolumeClaim:
            claimName: postgresql-postgresql-0
        - name: protocol-mapper-agent-001-0
          persistentVolumeClaim:
            claimName: protocol-mapper-agent-001-0
        - name: service-manager
          persistentVolumeClaim:
            claimName: service-manager
        - name: system-control-server-data
          persistentVolumeClaim:
            claimName: system-control-server-data
        - name: workbench
          persistentVolumeClaim:
            claimName: workbench
Code-Sprache: YAML (yaml)

2. Updating and Applying the Kubernetes Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: connectware-fix-permissions
  labels:
    app: connectware-fix-permissions
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      imagePullSecrets:
      - name: cybus-docker-registry
      containers:
      - image: registry.cybus.io/cybus/connectware-fix-permissions:1.5.0
        securityContext:
          runAsUser: 0
        imagePullPolicy: Always
        name: connectware-fix-permissions
        # Insert the volumeMounts section here
        # volumeMounts:
        # - name: brokerdata-broker-0
        #   mountPath: /mnt/connectware_brokerdata-broker-0
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
      # Insert the volumes section here
      # volumes:
      # - name: brokerdata-broker-0
      #   persistentVolumeClaim:
      #     claimName: brokerdata-broker-0


Code-Sprache: YAML (yaml)
kubectl scale sts,deploy -lapp.kubernetes.io/instance=<connectware-installation-name> --replicas 0
kubectl scale sts -lapp.kubernetes.io/instance=<connectware-agent-installation-name> --replicas 0
Code-Sprache: YAML (yaml)
kubectl apply -f kubernetes-job.yml
Code-Sprache: YAML (yaml)

3. Verifying and Cleaning Up

kubectl logs -f job/connectware-fix-permissions
Code-Sprache: SAS (sas)

Example output:

Found directory: connectware_brokerdata-broker-0. Going to change permissions
Found directory: connectware_brokerdata-broker-1. Going to change permissions
Found directory: connectware_brokerdata-control-plane-broker-0. Going to change permissions
Found directory: connectware_brokerdata-control-plane-broker-1. Going to change permissions
Found directory: connectware_brokerlog-broker-0. Going to change permissions
Found directory: connectware_brokerlog-broker-1. Going to change permissions
Found directory: connectware_brokerlog-control-plane-broker-0. Going to change permissions
Found directory: connectware_brokerlog-control-plane-broker-1. Going to change permissions
Found directory: connectware_certs. Going to change permissions
Found directory: connectware_postgresql-postgresql-0. Going to change permissions
Postgresql volume identified, using postgresql specific permissions
Found directory: connectware_service-manager. Going to change permissions
Found directory: connectware_system-control-server-data. Going to change permissions
Found directory: connectware_workbench. Going to change permissions
All done. Found 13 volumes.
Code-Sprache: YAML (yaml)
kubectl delete -f kubernetes-job.yml
Code-Sprache: YAML (yaml)
kubectl delete svc -l app.kubernetes.io/instance=<connectware-installation-name>
Code-Sprache: YAML (yaml)

4. Upgrading Connectware

helm upgrade -n <namespace> <installation-name> <repo-name>/connectware --version 1.5.0 -f values.yaml
Code-Sprache: YAML (yaml)

Result: You have successfully upgraded Connectware to 1.5.0.

Functionality of Provided Files

create-pvc-yaml.sh

This script is designed to automatically generate Kubernetes resource lists, specifically volumeMounts and volumes, for Connectware deployments on Kubernetes. It simplifies the preparation process for upgrading Connectware by:

The script outputs sections that can be directly copied and pasted into the kubernetes-job.yml file, ensuring that the correct volumes are mounted with the appropriate permissions for the upgrade process.

kubernetes-job.yml

This YAML file defines a Kubernetes job responsible for adjusting the permissions of the volumes identified by the create-pvc-yaml.sh script. The job is tailored to run with root privileges, enabling it to modify ownership and permissions of files and directories within PVCs that are otherwise inaccessible due to the permission changes introduced in Connectware 1.5.0.

The job’s purpose is to ensure that all persistent volumes used by Connectware are accessible by the new, non-root container user, addressing the core upgrade challenge without compromising on security by avoiding the use of root containers in the Connectware deployment itself.

Similar Articles

Prerequisites

Adding the Helm Chart Repository

Add the Cybus connectware-helm repository to your local Helm installation to use the connectware-agent Helm chart to install Connectware agents in Kubernetes:

helm repo add cybus https://repository.cybus.io/repository/connectware-helm
Code-Sprache: YAML (yaml)

To verify that the Helm chart is available you can execute a Helm search:

helm search repo connectware-agent
Code-Sprache: YAML (yaml)

NAME CHART VERSIONAPP VERSION DESCRIPTION
cybus/connectware-agent
standalone agents		1.0.0 1.1.5Cybus Connectware

Creating Your values.yaml File

As with all Helm charts, the connectware-agent chart is configured using a YAML file. This file can have any name, however we will refer to it as the values.yaml file.

Create this file to start configuring your agent installation by using your preferred editor:

vi values.yaml
Code-Sprache: YAML (yaml)

Installing a Single Agent

To quickly install a single agent you only need to add your Connectware license key to your values.yaml file as the Helm value licenseKey:

licenseKey: <your-connectware-license-key>
Code-Sprache: YAML (yaml)

You can now use this file to deploy your Connectware agent in a Kubernetes namespace of your choice:

helm upgrade -i connectware-agent cybus/connectware-agent -f values.yaml -n <namespace>
Code-Sprache: YAML (yaml)

Output

Release "connectware-agent" does not exist. Installing it now.

NAME: connectware-agent

LAST DEPLOYED: Mon Mar 13 14:31:39 2023

NAMESPACE: connectware

STATUS: deployed

REVISION: 1

TEST SUITE: None

NOTES:

Thank you for using the Cybus Connectware agent Helm chart!

For additional information visit https://cybus.io/

Number of agents: 1

--------------------

- agent

If any of these agents are new, please remember to visit Connectware's client registry to set up the connection to Connectware.

Hint: If you have agents stuck in a status other than "Running", you need to delete the stuck pods before a pod with your new configuration will be created.
Code-Sprache: YAML (yaml)

This will start a single Connectware agent named “agent”, which will connect to a Connectware installation deployed in the same namespace. Unlock the Client Registry in your Connectware admin UI to connect this agent. Refer to Client Registry — Connectware 1.3.2 documentation to learn how to use the Client Registry to connect agents.

You can repeat the same command to apply any changes to your values.yaml file configuration in the future.

Changing the Target Connectware

If you are not deploying the agent in the same Kubernetes namespace, or even inside the same Kubernetes cluster, you need to specify the hostname under which Connectware is reachable for this agent.

In the default configuration, the following network ports on Connectware must be reachable for the agent:

Specify the hostname of Connectware to which the agent connects to by setting the Helm value connectwareHost inside the protocolMapperAgentDefaults context of your values.yaml file. For Connectware deployments in a different Kubernetes namespace this is “connectware.<namespace>”.

Example

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus # adjust to actual hostname of Connectware
Code-Sprache: YAML (yaml)

Connectware with Separate control-plane-broker

To connect to a Connectware that uses the separate control-plane-broker, you need to set the Helm value controlPlaneBrokerEnabled to true inside the protocolMapperAgentDefaults section of your values.yaml file.

Example

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus # adjust to actual hostname of Connectware
  controlPlaneBrokerEnabled: true
Code-Sprache: YAML (yaml)

Note: This adds TCP/1884 to required network ports.

Installing Multiple Agents with a Single Helm Installation

You can use the agent chart to install multiple Connectware agents. Every agent you configure needs to be named using the Helm value name in a collection entry inside the context protocolMapperAgents. This way, the default name “agent” will be replaced by the name you give the agent.

Example

licenseKey: <your-connectware-license-key>
protocolMapperAgentDefaults:
  connectwareHost: connectware.cybus # adjust to actual hostname of Connectware
protocolMapperAgents:
- name: bender-robots
- name: welder-robots
Code-Sprache: YAML (yaml)

This will deploy two Connectware agents, named “bender-robots” and “welder-robots”, both of which will contact the Client Registry of Connectware inside the Kubernetes namespace “cybus”, as described in Client Registry — Connectware 1.3.2 documentation

Introduction

This quick start guide describes the steps to install the Cybus Connectware onto a Kubernetes cluster.
Please consult the article Installing Cybus Connectware for the basic requirements to run the software, like having access to the Cybus Portal to acquire a license key.

The following topics are covered by this article:

Prerequisites

We assume that you are already familiar with the Cybus Portal and that you have obtained a license key or license file. Also see the prerequisites in the article Installing Cybus Connectware.

This guide does not introduce Kubernetes, Docker, containerization or tooling knowledge, we expect the system admin to know about their respective Kubernetes environment, which brings – besides wellknown standards – a certain specific custom complexity, e.g. the choice of certain load balancers, the management environment, storage classes and the like, which are up to the customer’s operations team and should not affect the reliability of Cybus Connectware deployed there, if the requirements are met.

Besides a Kubernetes cluster the following tools and resources are required:

Quick Installation Guide

To be able to start with Cybus Connectware on a Kubernetes cluster, use the prepared helm chart and the following steps:

  1. Add the public Helm V3 repository to the local configuration using the Helm CLI:
<code>helm repo add cybus https://repository.cybus.io/repository/connectware-helm</code>
Code-Sprache: YAML (yaml)
  1. Update repository and verify available Connectware Helm Chart releases (e.g. 1.1.0)
helm repo update
helm search repo connectware [-l]
Code-Sprache: YAML (yaml)
  1. Create a file called values.yaml. This file will be used to configure your installation of Connectware. Initially fill this file with this YAML content:
global:
  licensekey: <YOUR-CONNECTWARE-LICENSE-KEY>
  setImmutableLabels: true
  broker:
    clusterSecret: <SOME-RANDOM-SECRET-STRING>
Code-Sprache: YAML (yaml)
storage:
    storageClassName: “san-storage” # example value
Code-Sprache: YAML (yaml)
<code>helm show values cybus/connectware > default-values.yaml</code>
Code-Sprache: YAML (yaml)
helm install <YOURDEPLOYMENTNAME> cybus/connectware -f ./values.yaml --dry-run --debug -n<YOURNAMESPACE> --create-namespace
Code-Sprache: YAML (yaml)

Example

helm install connectware cybus/connectware -f ./values.yaml --dry-run --debug -ncybus --create-namespace
Code-Sprache: YAML (yaml)
helm install <YOURDEPLOYMENTNAME> cybus/connectware -f ./values.yaml --n<YOURNAMESPACE> --create-namespace
Code-Sprache: YAML (yaml)
helm upgrade <YOURDEPLOYMENTNAME> cybus/connectware -f ./values.yml -n<YOURNAMESPACE>
Code-Sprache: YAML (yaml)

Important values.yml Parameters

When taking a look at the default-values.yaml file you should check out these important values within the global section:

Important Notes on Setting Values When Upgrading to Newer Versions of the Chart

Helm allows setting values by both specifying a values file (using -f or --values) and the --set flag. When upgrading this chart to newer versions you should use the same arguments for the Helm upgrade command to avoid conflicting values being set for the Chart; this is especially important for the value of global.broker.clusterSecret, which would cause the nodes not to form the cluster correctly, if not set to the same value used during install or upgrade.

For more information about value merging, see the respective Helm documentation.

Using Connectware

After following all the steps above Cybus Connectware is now installed. You can access the Admin UI by opening your browser and entering the Kubernetes application URL https://<external-ip> with the initial login credentials:

Username: admin
Password: admin

To determine this data, the following kubectl command can be used:

kubectl get svc connectware --namespace=<YOURNAMESPACE> -o jsonpath={.status.loadBalancer.ingress}
Code-Sprache: YAML (yaml)

Should this value be empty your Kubernetes cluster load-balancer might need further configuration, which is beyond the scope of this document, but you can take a first look at Connectware by port-forwarding to your local machine:

kubectl --namespace=<YOURNAMESPACE> port-forward svc/connectware 10443:443 1883:1883 8883:8883
Code-Sprache: YAML (yaml)

You can now access the admin UI at: https://localhost:10443/

If you would like to learn more how to use Connectware, check out our docs at https://docs.cybus.io/ or see more guides here.

The Cybus Connectware Helm Chart

The Kubernetes version of Cybus Connectware comes with a Helm Umbrella chart, describing the instrumentation of the Connectware images for deployment in a Kubernetes cluster.

It is publicly available in the Cybus Repository for download or direct use with Helm.

System Requirements

Cybus Connectware expects a regular Kubernetes cluster and was tested for Kubernetes 1.22 or higher.

This cluster needs to be able to provide load-balancer ingress functionality and persistent volumes in ReadWriteOnce and ReadWriteMany access modes provided by a default StorageClass unless you specify another StorageClass using the global.storage.storageClassName Helm value.

For Kubernetes 1.25 and above Connectware needs a privileged namespace or a namespace with PodSecurityAdmission labels for warn mode. In case of specific boundary conditions and requirements in customer clusters, a system specification should be shared to evaluate them for secure and stable Cybus Connectware operations.

Resource Limits

Connectware specifies default limits for CPU and memory in its Helm values that need to be at least fulfilled by the Kubernetes cluster for production use. Variations need to be discussed with Cybus, depending on the specific demands and requirements in the customer environment, e.g., the size of the broker cluster for the expected workload with respect to the available CPU cores and memory.

Smaller resource values are often enough for test or POC environments and can be adjusted using the global.podResources section of the Helm values.

Permissions

In order to run Cybus Connectware in Kubernetes clusters, two new RBAC roles are deployed through the Helm chart and will provide Connectware with the following namespace permissions:

Permissions for the Broker
resource(/subresource)/actionpermission
pods/list list all containers
get status of all containers
pods/get
pods/watch		inspect containers
statefulsets/listlist all StatefulSets
get status of all StatefulSets
statefulsets/get
statefulsets/watch		inspect StatefulSets

Permissions for the Container/Pod-Manager
resource(/subresource)/actionpermission
pods/listlist all containers
get status of all containers
pods/get
pods/watch		inspect containers
pods/log/get
pods/log/watch		inspect containers
get a stream of container logs
deployments/createcreate Deployments
deployments/deletedelete Deployments
deployments/update
deployments/patch
to restart containers (since we rescale deployments)

Specialties

The system administrator needs to be aware of certain characteristics of the Connectware deployment:

Deploying Protocol-Mapper Agents

The default-values.yaml file contains a protocolMapperAgents section representing a list of Connectware agents to deploy. The general configuration for these agents is the same as described in the Connectware documentation.

You can copy this section to your local values.yaml file to easily add agents to your Connectware installation

The only required property of the list items is name; if only this property is specified the chart assumes some defaults:

You can check out the comments of that section in default-values.yaml to see further configuration options.

You can find further information in the general Connectware Agent documentation.

Ihr Browser unterstützt diese Webseite nicht.

Liebe Besucher:innen, Sie versuchen unsere Website über den Internet Explorer zu besuchen. Der Support für diesen Browser wurde durch den Hersteller eingestellt, weshalb er moderne Webseiten nicht mehr richtig darstellen kann.
Um die Inhalte dieser Website korrekt anzeigen zu können, benötigen Sie einen modernen Browser.

Unter folgenden Links finden Sie Browser, für die unsere Webseite optimiert wurde:

Google Chrome Browser herunterladen Mozilla Firefox Browser herunterladen

Sie können diese Website trotzdem anzeigen lassen, müssen aber mit erheblichen Einschränkungen rechnen.

Diese Website trotzdem anzeigen.