How to | 03 Januar 2022

Installing Cybus Connectware on Kubernetes clusters

Introduction

This quick start guide describes the steps to install the Cybus Connectware on Kubernetes clusters.
Please consult the article Installing Cybus Connectware for the basic requirements to run the software.

The following topics are covered by this article:

Prerequisites
Quick Installation Guide
The Cybus Connectware Helm Chart
System requirements, resource limits and permissions
Specialties
Deploying Protocol Mapper Agents

Prerequisites

Please consult the prerequisites chapter in the article Installing Cybus Connectware, we assume that you are already familiar with the Cybus Portal and obtaining a license key and a license file.

This guide does not introduce Kubernetes, Docker, containerization or tooling knowledge, we expect the system admin to know about their respective Kubernetes environment, which brings – besides wellknown standards – a certain specific custom complexity, e.g. the choice of certain load balancers, the management environment, storage classes and the like, which are up to the customer’s operations team and should not affect the reliability of Cybus Connectware deployed there, if the requirements are met.

Besides a Kubernetes cluster the following tools and resources are required:

the kubectl CLI
the Helm V3 CLI
Ability to reach your Kubernetes cluster
Ability to download Docker images from registry.cybus.io directly or through a mirror/proxy, if the internet connection is restricted.

Quick Installation Guide

Start with Connectware on Kubernetes Clusters

To be able to start with Cybus Connectware on a Kubernetes cluster use the prepared helm chart please follow the following steps:

Add the public Helm V3 repository to the local configuration using the Helm CLI:
helm repo add cybus https://repository.cybus.io/repository/connectware-helm
Update repository and verify available Connectware Helm Chart releases (e.g. 1.7.0)
helm repo update helm search repo connectware [-l]
Extract the values.yaml file from the helm package and define custom values (see specification below). A connectware licenseKey and a broker.clusterSecret are essential. Beside this at least the cpu/memory resource definitions at the end of the chart values list need to be adjusted for the runtime environment (in case of restrictions).
helm show values cybus/connectware > values.yaml
Optionally perform a dry-run with debug options to validate all settings:
helm install <YOURDEPLOYMENTNAME> <REPONAME>/connectware -f ./values.yaml --dry-run --debug -n<YOURNAMESPACE> --create-namespace
(shows user supplied values, along with computed values and the rendered template)

Example:
sudo helm install connectware-1.0 cybus/connectware -f ./values.preview.yaml --dry-run --debug -ncybus --create-namespace
Then initially execute accordingly:
helm install <YOURDEPLOYMENTNAME> <REPONAME>/connectware -f ./values.yaml --n<YOURNAMESPACE> --create-namespace
Subsequent updates can be executed by
helm upgrade <YOURDEPLOYMENTNAME> <REPONAME>/connectware -f ./values.yml -n<YOURNAMESPACE>

Important values.yml parameters

The Helm Chart contains a values.schema.json file with more detailed information about these available options:

The licenseKey value handles the licensekey of the Connectware installation. This needs to be a production license key.
The licenseFile value is used to activate the Connectware in offline mode. The content of a license file downloaded from the Cybus Portal has to be set (this is a single line of a base64 encoded json object).
The image section specify the connectware image source and version
The broker section specifies MQTT broker related settings:
- broker.clusterSecret: the authentication secret for the MQTT broker cluster
- broker.replicaCount: the number of broker instances
- broker.enableMutualTLS: true/false for mandatory TLS for client connections
The loadBalancer section allows pre-configuration for a specific load balancer
The podResources set of values allows you to configure the number of resources per pod; by default some minimal values are set as a starting point but depending on the particular use case they need to be tuned in relation to the expected load in the system
The protocolMapperAgents section allows you to configure additional protocol-mapper instances in Agent mode. See the documentation below for more details

Important notes on setting values when upgrading to newer versions of the chart

Helm allows setting values by both specifying a values file (using -f or --values) and the --set flag. When upgrading this chart to newer versions you should use the same arguments for the Helm upgrade command to avoid conflicting values being set for the Chart; this is especially important for the value of global.broker.clusterSecret, which would cause the nodes not to form the cluster correctly, if not set to the same value used during install or upgrade.

For more information about value merging see the respective Helm documentation.

Using the Connectware

After following all the steps above Cybus Connectware is now installed. You can access the Admin UI by opening your browser and entering the Kubernetes application URL https://<external-ip> with the initial login credentials username=admin and password=admin.

To determine this data, the following kubectl command can be used:
kubectl get svc connectware --namespace=<YOURNAMESPACE> -o jsonpath={.status.loadBalancer.ingress}

If you would like to learn more how to use the Connectware, check out our docs at https://docs.cybus.io/ or see more guides here.

The Cybus Connectware Helm Chart

The Kubernetes version of Cybus Connectware comes with a Helm Umbrella chart, describing the instrumentation of the Connectware images for deployment in a Kubernetes cluster.

It is publicly available in the Cybus Repository for download resp. direct use with Helm.

System requirements for Kubernetes clusters

Cybus Connectware expects regular Kubernetes clusters and was tested for Kubernetes 1.18 or higher.
In case of specific boundary conditions and requirements in customer Kubernetes clusters, a system specification should be shared to evaluate them for secure and stable Cybus Connectware operations.

Resource limits

Connectware specifies default limits for CPU and memory in the values.yml, that need to be at least fulfilled by the Kubernetes cluster. Variations need to be discussed with Cybus, depending on the specific demands and requirements in the customer environment, e.g., the size of the broker cluster for the expected workload with respect to the available CPU cores and memory.

The additional file values.preview.yaml (for limited cluster environments) show some typical resource limit differences, which are defined to guarantee proper operation of Cybus Connectware.

Permissions

To be able to run Cybus Connectware in Kubernetes clusters the implemented compatibility layer requires the following permissions in form of resource[/sub-resource]/action for the required features.

Current required permissions defined in the role definitions for broker and container/pod-manager are:

Broker:

resource(/subresource)/action	permission
pods/list	list all containersget status of all containers
pods/get pods/watch	inspect containers

Container/Pod-manager:

resource(/subresource)/action	permission
pods/list	list all containersget status of all containers
pods/get pods/watch	inspect containers
pods/log/get pods/log/watch	inspect containersget a stream of container logs
deployments/create	create containers
deployments/delete	delete containers
deployments/update deployments/patch	to restart containers (since we rescale deployments)

Specialties

The system administrator needs to be aware of certain characteristics of the Connectware deployment:

It requires a storage class marked as default, otherwise the volumes fail to be deployed.
In case of Longhorn storage be aware that it is still ReadWriteOnce, but just on node level.
The Helm Chart requires a licenseKey to download images from the cybus registry, which requires a proxy or direct connection to the registry.cybus.io host.
To download a license file for the key a connection to portal.cybus.io is required.
This can be skipped for an offline setup by defining the licenseFile parameter in the helm chart values (see licenseFile above)
Dynamic scaling is just possible for the broker deployment.
Multiple ProtocolMapper instances are possible by explicit definition of additional protocolMapperAgents in the helm chart values.
In case of a MetalLB the values.yaml can define an addressPoolName to assign an external IP address range managed by MetalLB.

Deploying Protocol-Mapper Agents

The values.yaml file contains a protocolMapperAgents section representing a list of Connectware agents to deploy. The general configuration for these agents is the same as described in the Connectware documentation.

The only required property of the list items is name; if only this property is specified the chart assumes some defaults:

CYBUS_AGENT_NAME is set to the same value as that of name
CYBUS_MQTT_HOST is set to connectware which is the DNS name of the Connectware Note: this DNS name must be a DNS name reachable by each of the agents being deployed. In case of some deployment distribution strategy of the target Kubernetes cluster this name has to be configured accordingly.
storageSize is set to 40 MB by default. The agents use some local storage which needs to be configured based on each use case. If a larger number of services is going to be deployed, this value should be specified and set to bigger values.

Each entry on the list of agents accepts env properties as environment variables injected into each agent:

CYBUS_MQTT_USERNAME
CYBUS_MQTT_SCHEME
CYBUS_MQTT_PORT

The complete documentation for these values can be found in the Connectware Agent documentation.