This Cybus Kubernetes guide provides a detailed procedure to help admins adjust the persistent volume content permissions to ensure a smooth upgrade to Connectware 1.5.0 in Kubernetes environments.

Important: Connectware 1.5.0 introduces a significant change regarding container privileges. Now, containers are started without root privileges. This change causes an issue where files that persisted on volumes that were created by a user other than the one accessing them, cannot easily have their permissions changed. As a result, this upgrade requires the admin to manually update the permissions.

Prerequisites

Overview of the Upgrade

Before you begin the upgrade process, ensure that you have made the necessary preparations and that all relevant stakeholders are involved.

The following steps give you an overview of what you need to do. See below for detailed step-by-step instructions.

  1. Create the YAML snippets: Create YAML snippets for each PersistentVolumeClaim used by Connectware. This step is essential for identifying the volumes that need permission adjustments.
  2. Filter volumes: Carefully review the created snippets to ensure that only related volumes are included. This is critical to avoid unnecessary modifications and potential disruptions.
  3. Update the Kubernetes job file: Integrate the snippets into the kubernetes-job.yml file as described.
  4. Shutdown Connectware: Securely shut down all Connectware and agent pods to prevent any access conflicts during the permissions update. This step is crucial to ensure that the permissions changes are applied without interference.
  5. Run the Kubernetes job: Run the job to adjust permissions on all relevant volumes. This action addresses the core challenge posed by the upgrade and ensures that the new, non-root containers can access the files they need.
  6. Delete Kubernetes services: Remove all services used by Connectware in preparation for the new version. This step is necessary to accommodate updates that are incompatible with the existing service configurations.
  7. Apply Helm upgrade: Proceed with the Helm upgrade to Connectware version 1.5.0. This step completes the upgrade process and implements the new security measures and permissions model.

Important Notes

Upgrading Procedure

1. Creating YAML Snippets for relevant PersistentVolumeClaim names

#!/usr/bin/env bash
#
# Creates a Kubernetes resource list of volumeMounts and volumes for Connectware (agent) deployments
#
function usage {
    printf "Usage:n"
    printf "$0 [--kubeconfig|-kc <kubeconfig_file>] [--context|-ctx <target_context>] [--namespace|-n <target_namespace>] [--no-external-agents] [--no-connectware]n"
    exit 1
}

function argparse {

  while [ $# -gt 0 ]; do
    case "$1" in
      --kubeconfig|-kc)
        # a kube_config file for the Kubernetes cluster access
        export KUBE_CONFIG="${2}"
        shift
        ;;
      --context|-ctx)
        # a KUBE_CONTEXT for the Kubernetes cluster access
        export KUBE_CONTEXT="${2}"
        shift
        ;;
      --namespace|-n)
        # the Kubernetes cluster namespace to operate on
        export NAMESPACE="${2}"
        shift
        ;;
      --no-external-agents)
        export NO_EXTERNAL_AGENTS=true
        shift
        ;;
      --no-connectware)
        export NO_CONNECTWARE=true
        shift
        ;;
      *)
        printf "ERROR: Parameters invalidn"
        usage
    esac
    shift
  done
}

#
# init
export NO_EXTERNAL_AGENTS=false
export NO_CONNECTWARE=false
export KUBECTL_BIN=$(command -v kubectl)

argparse $*

shopt -s expand_aliases
# Check for kubectl paramaters and construct the ${KUBECTL_CMD} command to use
if [ ! -z $KUBE_CONFIG ]; then
  KUBECONFIG_FILE_PARAM=" --kubeconfig=${KUBE_CONFIG}"
fi
if [ ! -z $KUBE_CONTEXT ]; then
  KUBECONFIG_CTX_PARAM=" --context=${KUBE_CONTEXT}"
fi
if [ ! -z $NAMESPACE ]; then
  NAMESPACE_PARAM=" -n${NAMESPACE}"
fi

KUBECTL_CMD=${KUBECTL_BIN}${KUBECONFIG_FILE_PARAM}${KUBECONFIG_CTX_PARAM}${NAMESPACE_PARAM}

volumes=$(${KUBECTL_CMD} get pvc -o name | sed -e 's/persistentvolumeclaim///g' )
valid_pvcs=""
volume_yaml=""
volumemounts_yaml=""

if [[ "${NO_CONNECTWARE}" == "false" ]]; then
  valid_pvcs=$(cat << EOF
system-control-server-data
certs
brokerdata-*
brokerlog-*
workbench
postgresql-postgresql-*
service-manager
protocol-mapper-*
EOF
)
fi

if [[ "${NO_EXTERNAL_AGENTS}" == "false" ]]; then
  # Add volumes from agent Helm chart
  pvc_volumes=$(${KUBECTL_CMD} get pvc -o name -l connectware.cybus.io/service-group=agent | sed -e 's/persistentvolumeclaim///g' )
  valid_pvcs="${valid_pvcs}
  ${pvc_volumes}"
fi

# Collect volumeMounts
for pvc in $volumes; do
  for valid_pvc in $valid_pvcs; do
      if [[ "$pvc" =~ $valid_pvc ]]; then
        volumemounts_yaml="${volumemounts_yaml}
          - name: $pvc
            mountPath: /mnt/connectware_$pvc"
        break
      fi
  done
done

# Collect volumes
for pvc in $volumes; do
  for valid_pvc in $valid_pvcs; do
      if [[ "$pvc" =~ $valid_pvc ]]; then
        volume_yaml="${volume_yaml}
        - name: $pvc
          persistentVolumeClaim:
            claimName: $pvc"
          break
      fi
  done
done


# print volumeMounts
echo
echo "Copy this as the "volumeMounts:" section:"
echo "######################################"
echo -n "        volumeMounts:"
echo "$volumemounts_yaml"

# print volumes
echo
echo "Copy this as the "volumes:" section:"
echo "######################################"
echo -n "      volumes:"
echo "$volume_yaml"
Code-Sprache: YAML (yaml)
kubectl config use-context <my-cluster>
kubectl config set-context --current --namespace <my-connectware-namespace>
Code-Sprache: YAML (yaml)
chmod u+x create-pvc-yaml.sh
./create-pvc-yaml.sh
Code-Sprache: YAML (yaml)

Example output:

Copy this as the "volumeMounts:" section:
######################################
        volumeMounts:
          - name: brokerdata-broker-0
            mountPath: /mnt/connectware_brokerdata-broker-0
          - name: brokerdata-control-plane-broker-0
            mountPath: /mnt/connectware_brokerdata-control-plane-broker-0
          - name: brokerlog-broker-0
            mountPath: /mnt/connectware_brokerlog-broker-0
          - name: brokerlog-control-plane-broker-0
            mountPath: /mnt/connectware_brokerlog-control-plane-broker-0
          - name: certs
            mountPath: /mnt/connectware_certs
          - name: postgresql-postgresql-0
            mountPath: /mnt/connectware_postgresql-postgresql-0
          - name: protocol-mapper-agent-001-0
            mountPath: /mnt/connectware_protocol-mapper-agent-001-0
          - name: service-manager
            mountPath: /mnt/connectware_service-manager
          - name: system-control-server-data
            mountPath: /mnt/connectware_system-control-server-data
          - name: workbench
            mountPath: /mnt/connectware_workbench

Copy this as the "volumes:" section:
######################################
      volumes:
        - name: brokerdata-broker-0
          persistentVolumeClaim:
            claimName: brokerdata-broker-0
        - name: brokerdata-control-plane-broker-0
          persistentVolumeClaim:
            claimName: brokerdata-control-plane-broker-0
        - name: brokerlog-broker-0
          persistentVolumeClaim:
            claimName: brokerlog-broker-0
        - name: brokerlog-control-plane-broker-0
          persistentVolumeClaim:
            claimName: brokerlog-control-plane-broker-0
        - name: certs
          persistentVolumeClaim:
            claimName: certs
        - name: postgresql-postgresql-0
          persistentVolumeClaim:
            claimName: postgresql-postgresql-0
        - name: protocol-mapper-agent-001-0
          persistentVolumeClaim:
            claimName: protocol-mapper-agent-001-0
        - name: service-manager
          persistentVolumeClaim:
            claimName: service-manager
        - name: system-control-server-data
          persistentVolumeClaim:
            claimName: system-control-server-data
        - name: workbench
          persistentVolumeClaim:
            claimName: workbench
Code-Sprache: YAML (yaml)

2. Updating and Applying the Kubernetes Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: connectware-fix-permissions
  labels:
    app: connectware-fix-permissions
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      imagePullSecrets:
      - name: cybus-docker-registry
      containers:
      - image: registry.cybus.io/cybus/connectware-fix-permissions:1.5.0
        securityContext:
          runAsUser: 0
        imagePullPolicy: Always
        name: connectware-fix-permissions
        # Insert the volumeMounts section here
        # volumeMounts:
        # - name: brokerdata-broker-0
        #   mountPath: /mnt/connectware_brokerdata-broker-0
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
      # Insert the volumes section here
      # volumes:
      # - name: brokerdata-broker-0
      #   persistentVolumeClaim:
      #     claimName: brokerdata-broker-0


Code-Sprache: YAML (yaml)
kubectl scale sts,deploy -lapp.kubernetes.io/instance=<connectware-installation-name> --replicas 0
kubectl scale sts -lapp.kubernetes.io/instance=<connectware-agent-installation-name> --replicas 0
Code-Sprache: YAML (yaml)
kubectl apply -f kubernetes-job.yml
Code-Sprache: YAML (yaml)

3. Verifying and Cleaning Up

kubectl logs -f job/connectware-fix-permissions
Code-Sprache: SAS (sas)

Example output:

Found directory: connectware_brokerdata-broker-0. Going to change permissions
Found directory: connectware_brokerdata-broker-1. Going to change permissions
Found directory: connectware_brokerdata-control-plane-broker-0. Going to change permissions
Found directory: connectware_brokerdata-control-plane-broker-1. Going to change permissions
Found directory: connectware_brokerlog-broker-0. Going to change permissions
Found directory: connectware_brokerlog-broker-1. Going to change permissions
Found directory: connectware_brokerlog-control-plane-broker-0. Going to change permissions
Found directory: connectware_brokerlog-control-plane-broker-1. Going to change permissions
Found directory: connectware_certs. Going to change permissions
Found directory: connectware_postgresql-postgresql-0. Going to change permissions
Postgresql volume identified, using postgresql specific permissions
Found directory: connectware_service-manager. Going to change permissions
Found directory: connectware_system-control-server-data. Going to change permissions
Found directory: connectware_workbench. Going to change permissions
All done. Found 13 volumes.
Code-Sprache: YAML (yaml)
kubectl delete -f kubernetes-job.yml
Code-Sprache: YAML (yaml)
kubectl delete svc -l app.kubernetes.io/instance=<connectware-installation-name>
Code-Sprache: YAML (yaml)

4. Upgrading Connectware

helm upgrade -n <namespace> <installation-name> <repo-name>/connectware --version 1.5.0 -f values.yaml
Code-Sprache: YAML (yaml)

Result: You have successfully upgraded Connectware to 1.5.0.

Functionality of Provided Files

create-pvc-yaml.sh

This script is designed to automatically generate Kubernetes resource lists, specifically volumeMounts and volumes, for Connectware deployments on Kubernetes. It simplifies the preparation process for upgrading Connectware by:

The script outputs sections that can be directly copied and pasted into the kubernetes-job.yml file, ensuring that the correct volumes are mounted with the appropriate permissions for the upgrade process.

kubernetes-job.yml

This YAML file defines a Kubernetes job responsible for adjusting the permissions of the volumes identified by the create-pvc-yaml.sh script. The job is tailored to run with root privileges, enabling it to modify ownership and permissions of files and directories within PVCs that are otherwise inaccessible due to the permission changes introduced in Connectware 1.5.0.

The job’s purpose is to ensure that all persistent volumes used by Connectware are accessible by the new, non-root container user, addressing the core upgrade challenge without compromising on security by avoiding the use of root containers in the Connectware deployment itself.

Similar Articles

This Cybus Docker guide provides a detailed procedure to help admins adjust the persistent volume content permissions to ensure a smooth upgrade to Connectware 1.5.0 in Docker Compose environments.

Important: Connectware 1.5.0 introduces a significant change regarding container privileges. Now, containers are started without root privileges. This change causes an issue where files that persisted on volumes that were created by a user other than the one accessing them, cannot easily have their permissions changed. As a result, this upgrade requires the admin to manually update the permissions.

Attention: The following protocols may not operate as expected in some constellations:

If you experience problems with these protocols after upgrading to Connectware 1.5.0 (or higher), these affected services may require root permissions and are no longer supported by the internal protocol-mapper or agents without root permissions.

In this case, you should transfer the service that uses these protocols to a separate agent. This agent can have higher permissions but in a controlled manner.

To learn how to configure an agent to run with root permissions, see the section on agent orchestration.

Upgrading Procedure

As an additional step to this upgrade, we provide a docker-compose.override.yml file to help you configure volume permissions.

The upgrade procedure consists of the following steps:

  1. Shut down Connectware.
  2. Save the provided file as docker-compose.override.yml in the same folder as your Connectware docker-compose.yml.
  3. Start the the Docker composition to adjust the volume permissions.
  4. Remove the docker-compose.override.yml file.
  5. Upgrade Connectware using the connectware-online-installer.sh script.

1. Shutting Down Connectware

All containers should be stopped and removed:

Containers stopped and removed

2. Downloading and Saving the Override File

version: "2.0"
services:
  connectware-fix-permissions:
    user: root
    image: registry.cybus.io/cybus/connectware-fix-permissions:1.5.0
    volumes:
      - certs:/mnt/connectware_certs
      - brokerLog:/mnt/connectware_brokerLog
      - brokerData:/mnt/connectware_brokerData
      - postgresql:/mnt/connectware_postgresql
      - service-manager:/mnt/connectware_service-manager
      - systemControlServerData:/mnt/connectware_systemControlServerData
      - workbench:/mnt/connectware_workbench
  admin-web-app:
    profiles:
    - do-not-start
  auth-server:
    profiles:
    - do-not-start
  broker:
    profiles:
    - do-not-start
  connectware:
    profiles:
    - do-not-start
  container-manager:
    profiles:
    - do-not-start
  doc-server:
    profiles:
    - do-not-start
  ingress-controller:
    profiles:
    - do-not-start
  postgresql:
    profiles:
    - do-not-start
  protocol-mapper:
    profiles:
    - do-not-start
  service-manager:
    profiles:
    - do-not-start
  system-control-server:
    profiles:
    - do-not-start
  workbench:
    profiles:
    - do-not-start
Code-Sprache: YAML (yaml)

Download: upgrade-1-5-docker-compose.override.yml

If you already have a docker-compose.override.yml file for your Connectware installation, make sure to rename it before saving it as docker-compose.override.yml.

3. Adjusting Volume Permissions

Example output:

[+] Building 0.0s (0/0)
[+] Running 1/0
✔ Container connectware-connectware-fix-permissions-1  Created                              0.0s
Attaching to connectware-connectware-fix-permissions-1
connectware-connectware-fix-permissions-1  | Found directory: connectware_brokerData. Going to change permissions
connectware-connectware-fix-permissions-1  | Found directory: connectware_brokerLog. Going to change permissions
connectware-connectware-fix-permissions-1  | Found directory: connectware_certs. Going to change permissions
connectware-connectware-fix-permissions-1  | Found directory: connectware_postgresql. Going to change permissions
connectware-connectware-fix-permissions-1  | Postgresql volume identified, using postgresql specific permissions
connectware-connectware-fix-permissions-1  | Found directory: connectware_service-manager. Going to change permissions
connectware-connectware-fix-permissions-1  | Found directory: connectware_systemControlServerData. Going to change permissions
connectware-connectware-fix-permissions-1  | Found directory: connectware_workbench. Going to change permissions
connectware-connectware-fix-permissions-1  | All done. Found 7 volumes.
connectware-connectware-fix-permissions-1 exited with code 0
Code-Sprache: YAML (yaml)

4. Removing Override File

5. Upgrading Connectware

wget https://download.cybus.io/1.5.0/connectware-online-installer.sh
chmod +x connectware-online-installer.sh
./connectware-online-installer.sh
Code-Sprache: YAML (yaml)

Note: You may need sudo access to install/update Connectware, this is normal, however the new Connectware containers will not have root access once they are running.

After upgrading, you can start Connectware as usual.

Optional: To verify that Connectware services are running with an unprivileged user, you can run this command:

docker ps -qf="label=io.cybus.connectware=core" | xargs -I % docker exec % sh -c 'echo "$(hostname) runs with user ID $(id -u)"'
Code-Sprache: YAML (yaml)

Only the service container-manager should use the user ID 0.

Upgrading Connectware Agents

Connectware agents are ideally orchestrated using docker-compose as described in agent orchestration.

To upgrade your agents, do one of the following:

To benefit from the security improvements, we recommend that you adjust volume permissions rather than continuing to run with elevated privileges.

Important: If you are experiencing problems after upgrading to Connectware 1.5.0 (or higher), the affected services may require root permissions. In such cases, it is advisable to choose to run your agents with elevated privileges (see below).

Similar to adjusting permissions for the Connectware instance, we provide a docker-compose.override.yml to help you adjust volume permissions.

The upgrade consists of the following steps:

  1. Shut down the agent.
  2. Save the provided file as docker-compose.override.yml in the same folder as your agent’s docker-compose.yml.
  3. Start the the Docker composition to adjust volume permissions.
  4. Remove the docker-compose.override.yml file.
  5. Upgrade the agent by adjusting the image tag in your docker-compose.yml file.

1. Shutting Down the Agent

2. Downloading and Saving the Override File

version: '2.0'
services:
  connectware-fix-permissions:
    user: root
    image: registry.cybus.io/cybus/connectware-fix-permissions:1.5.0
    volumes:
    - protocol-mapper-agent:/mnt/connectware_agent_data
  protocol-mapper-agent:
    profiles:
    - do-not-start
Code-Sprache: YAML (yaml)

Download: upgrade-1-5-agent-docker-compose.override.yml

If you use a different name for your agent Docker service and volume than our example, you need to adjust the name of the agent to your actual agent’s name (lines 7 and 8 of the example above) and its volume in your docker-compose.yml file.

3. Adjusting Volume Permissions

Example output:

[+] Building 0.0s (0/0)
[+] Running 2/2
✔ Network agent_default                          Created                                                                                                0.1s
✔ Container agent-connectware-fix-permissions-1  Created                                                                                                0.1s
Attaching to agent-connectware-fix-permissions-1
agent-connectware-fix-permissions-1  | Found directory: connectware_agent_data. Going to change permissions.
agent-connectware-fix-permissions-1  | All done. Found 1 volumes.
agent-connectware-fix-permissions-1 exited with code 0
Code-Sprache: YAML (yaml)

4. Removing the Override File

5. Upgrading the Agent

Example of an agent YAML file for agent running 1.5.0:

version: '2.0'
services:
  protocol-mapper-agent:
    image: registry.cybus.io/cybus/protocol-mapper:1.5.0
    environment:
      CYBUS_AGENT_MODE: distributed
      CYBUS_AGENT_NAME: myAgent
      CYBUS_MQTT_HOST: 172.17.0.1
    volumes:
      - protocol-mapper-agent:/data
    restart: unless-stopped
    network_mode: host
    hostname: myAgent
volumes:
  protocol-mapper-agent:
Code-Sprache: YAML (yaml)

You can now start your agent as usual, for example by running docker compose up -d.

Upgrading by Using Root Privileges

As an alternative to modifying volume permissions, or if you are using a protocol that requires elevated permissions, you can modify your agent’s docker-compose.yml file to specify the user which is used for the container.

For example (see line 4):

version: '2.0'
services:
  protocol-mapper-agent:
    user: root
    image: registry.cybus.io/cybus/protocol-mapper:1.5.0
    environment:
      CYBUS_AGENT_MODE: distributed
      CYBUS_AGENT_NAME: myAgent
      CYBUS_MQTT_HOST: 172.17.0.1
    volumes:
      - protocol-mapper-agent:/data
    restart: unless-stopped
    network_mode: host
    hostname: myAgent
volumes:
  protocol-mapper-agent:
Code-Sprache: YAML (yaml)

Related Articles

Ihr Browser unterstützt diese Webseite nicht.

Liebe Besucher:innen, Sie versuchen unsere Website über den Internet Explorer zu besuchen. Der Support für diesen Browser wurde durch den Hersteller eingestellt, weshalb er moderne Webseiten nicht mehr richtig darstellen kann.
Um die Inhalte dieser Website korrekt anzeigen zu können, benötigen Sie einen modernen Browser.

Unter folgenden Links finden Sie Browser, für die unsere Webseite optimiert wurde:

Google Chrome Browser herunterladen Mozilla Firefox Browser herunterladen

Sie können diese Website trotzdem anzeigen lassen, müssen aber mit erheblichen Einschränkungen rechnen.

Diese Website trotzdem anzeigen.