Prerequisites

In this lesson, we will send data from the Connectware MQTT Broker to Azure IoT Hub.

It is required to have a working Connectware instance, and also a working Azure IoT Hub setup. We assume you are already familiar with Connectware and its service concept. If not, we recommend reading the articles Connectware Technical Overview and Service Basics for a quick introduction. Furthermore, this lesson requires basic understanding of MQTT and how to publish data on a MQTT topic. If you want to refresh your MQTT knowledge, we recommend reading the lessons MQTT Basics and How to connect an MQTT client to publish and subscribe data.

Introduction

This article is divided into two parts. In the first part, the general background of Azure IoT services and their differences is described. Feel free to skip this section if you are familiar with Azure IoT Hub, in particular with the difference between IoT Edge Runtime and IoT Device SDKs. In the second part, the recommended integration mechanisms between Connectware and Azure IoT Hub are explained in a hands-on approach.

Azure IoT Hub

Azure IoT Hub is a core component of the Azure IoT offerings. It is a fully managed service that enables reliable and secure bi-directional communication between IoT devices and an application back-end. Azure IoT Hub offers reliable device-to-cloud and cloud-to-device hyper-scale messaging, enables secure communications using per-device security credentials and access control, and includes device libraries for most popular programming languages and platforms.

There are two ways you can send data to Azure IoT Hub which are described below.

Azure IoT Hub Edge

Azure IoT Edge lets you offload parts of your IoT workload from your Azure cloud services to your devices. IoT Edge can reduce latency in your solution, reduce the amount of data your devices exchange with the cloud, and enable off-line scenarios. You can manage IoT Edge devices from IoT Central and some solution accelerators.

Azure IoT Edge is made up of three components:

Azure IoT Edge
Azure IoT Edge (adapted from Microsoft, 2020)

Azure IoT Hub Device SDKs

IoT Hub Device SDKs enable you to build apps that run on your IoT devices. These apps send telemetry to your IoT Hub, and optionally receive messages, job, method, or twin updates from your IoT Hub. The device SDKs support multiple operating systems, such as Linux, Windows, and real-time operating systems. There are SDKs for multiple programming languages, such as C, Node.js, Java, .NET, and Python.

Connectware & Azure IoT Hub Integration

Before proceeding further, first you need to complete the Azure IoT Hub setup. It is required to obtain the device connection string. You can follow the instructions on this page for doing so.

There are multiple possibilities to enable data exchange between Azure IoT Hub and Connectware. These possibilities are described below.

Cybus Azure IoT Hub connector service

Cybus Azure-IoT-Connector is a Connectware service provided by Cybus and available as a docker image. It uses the Azure IoT Hub Device Node.js SDK, and MQTT as the transport protocol. This is a simple solution to send data to the IoT Hub. Most of the complexity is hidden and a simple interface is provided for configuration in the form of „connector configuration“. This service is capable of sending telemetry data and blob data to the IoT Hub. However, the service does not support receiving data from the IoT Hub. As an additional advantage, data buffering is built-in, so that data is stored locally when there is a temporary connection failure to the IoT Hub.

The commissioning file below sends data published on topics ‚/machine/temperature‘ and ‚/machine/vibration‘ as telemetry data to Azure IoT Hub. Before installing the service, make sure you are publishing data on the Connectware broker on topics ‚/machine/temperature‘ and ‚/machine/vibration‘. Further details on setting up the suitable MQTT topics can be found in this article: How to connect an MQTT client to publish and subscribe data.

description: >
    Azure IoT Hub connector service

metadata:
  name: Azure IoT Hub connector
  icon: https://www.cybus.io/wp-content/uploads/2019/03/Cybus-logo-Claim-lang.svg
  provider: cybus
  homepage: https://www.cybus.io
  version: 0.0.1

definitions:
  MQTT_ROOT_TOPIC : !ref Mqtt_Root_Topic

parameters:

  Azure_Iot_Hub_Connection_String:
    type: string

  Mqtt_Root_Topic:
    type: string
    default: machine

resources:

  #----------------------------------------------------------------------------
  # Cybus Azure IoT Hub Connector Service
  #----------------------------------------------------------------------------

  azureIoTConnector:
    type: Cybus::Container
    properties:
      image: registry.cybus.io/cybus-services/azure-iot-connector:0.0.5
      environment:
        LOG_LEVEL: 'info'
        CONNECTOR_CONFIG: !sub  |
          {
            "general": {
              "name": "Azure Connector"
            },
            "source": {
              "driver": "azure.iot",
              "connection": {
                "connectionString": "${Azure_Iot_Hub_Connection_String}"
              },
              "defaults": {
                "operation": "write"
              }
            },
            "target": {
              "driver": "mqtt",
              "connection": {
                "protocol": "mqtt",
                "host": ${Cybus::MqttHost},
                "port": ${Cybus::MqttPort},
                "username": ${Cybus::MqttUser},
                "password": ${Cybus::MqttPassword}
              },
              "defaults": {
                "operation": "subscribe",
                "topicPrefix": "${Mqtt_Root_Topic}"
              }
            },
            "mappings": [
              {
                "source": {
                  "name": "temperature",
                  "type": "telemetry"
                },
                "target": {
                  "topic": "temperature"
                }
              },
              {
                "source": {
                  "name": "vibration",
                  "type": "telemetry"
                },
                "target": {
                  "topic": "vibration"
                }
              }
            ]
          }

The resource azureIoTConnector describes the Docker container with the connector service. This service is configured by passing the CONNECTOR_CONFIG environment variable, which in turn contains a JSON object. This JSON object has three important parts (in addition to the general section with the name):

In the source section, the only configurable parameter is the device connectionString to connect to a particular device on Azure IoT Hub. In this example, this string is defined as a parameter for the service commissioning file and asked during the installation of the service in the Connectware.

In this section, the MQTT broker connection parameters are defined. Theoretically, any arbitrary MQTT broker can be specified, but in this example, the local Connectware broker is used. The connection settings to the Connectware broker are set by using the global parameters of the Connectware for these values. Optionally, a default MQTT topic prefix can be specified within this section, which is useful when installing multiple services on the same Connectware but which should not interfere with each other.

Every object within the mappings array defines a mapping between source topic which corresponds to IoT Hub device parameter and target MQTT topic (on the Connectware broker). In the above example, we are sending data published on topics temperature and vibration to IoT Hub as telemetry data with corresponding name.

Cybus workbench service

The workbench service is basically a Node-RED instance running securely on the Connectware as a service. This opens up the possibility to install any Node-RED nodes within the service container for quick prototyping as well as for production environment. If your use-case cannot be achieved with the service commissioning file of the previous section, using the workbench might be a suitable choice with higher flexibility and additional tools to prototype your solution using third party Node-RED modules.

node-red-contrib-azure-iot-hub is a Node-RED module that allows you to send messages and register devices with Azure IoT Hub. It contains a total of four Node-RED cloud nodes: Azure IoT Hub, Azure IoT Registry, Azure IoT Hub Receiver and Azure IoT Hub Device Twin. These nodes can be used for configuring more complex data handling.

Tools

Now that we are successfully sending the data to the IoT Hub, we can monitor the transmitted data using various tools as described below.

Azure IoT Tools for Visual Studio Code

Azure IoT Tools is a collection of Visual Studio Code extensions for working with Azure IoT Hub. Using these extensions, you can interact with an Azure IoT Hub, manage connected devices, and enable distributed tracing for your Azure IoT applications. Further, you can easily subscribe to telemetry messages sent to the IoT Hub for quick testing. Please go to Visual Studio Marketplace to find further documentation regarding installation and usage of the Azure IoT tools.

Extension IoT Hub

Azure IoT Explorer

Azure IoT Explorer is an open source cross-platform user interface for interacting with Azure IoT Hub without logging into the Azure portal. This tool can be used to perform tasks like creating, deleting and querying devices within the IoT Hub. Device functionalities such as sending and receiving telemetry and editing device and module twin configuration is also possible with this tool. You can find more information about the explorer on its GitHub page.

Azure IoT Explorer

Summary

This lesson started with a quick introduction to Azure IoT Hub and its various components and SDKs available for integration with other services. Then an example setup to send data from the Connectware MQTT Broker to Azure IoT Hub was described, using a commissioning file with the azure-iot-connector service. Additionally, the IoT Hub Node-RED modules were described, which can be used with Cybus workbench service for prototyping of complex scenarios. Eventually, some of the monitoring tools were mentioned that can be used to monitor data flow between Azure IoT Hub and Connectware.

Where to go from here

You can take a look at more advanced scenarios involving Azure IoT Hub such as Machine Condition Monitoring in Connectware documentation pages.

References

Microsoft. 2020. What Is Azure Iot Edge. [online] Available at: https://docs.microsoft.com/en-us/azure/iot-edge/about-iot-edge [Accessed 27 October 2020].

Prerequisites

In this lesson, we will send data from the Connectware MQTT Broker to AWS IoT.

It is required to set up a Connectware instance and at least one AWS IoT Device. In case of using AWS IoT at the edge, an AWS IoT Greengrass Core has to be set up.

We assume you are already familiar with the Connectware and its service concept. If not, we recommend reading the articles Connectware Technical Overview and Service Basics for a quick introduction. Furthermore, this lesson requires basic understanding of MQTT and how to publish data on an MQTT topic. If you want to refresh your MQTT knowledge, we recommend the lessons MQTT Basics and How to connect an MQTT client to publish and subscribe data.

Introduction

This article is divided into three parts.

First, it provides general information about AWS IoT services and their differences. Feel free to skip this section if you are familiar with AWS IoT and the differences between AWS IoT Core and IoT Greengrass.
Then, the current integration mechanisms between the Connectware and AWS IoT are explained through a hands-on approach.
Finally, the article describes the tools to work with your MQTT use case to prototype, review and monitor the integration scenario.

AWS IoT

AWS IoT is a managed cloud platform that lets connected devices interact easily and securely with cloud applications and other devices. AWS IoT practically supports a nearly unlimited number of devices and messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely.

For AWS IoT, Amazon offers a software development kit available for most popular programming languages and platforms.

AWS IoT Core

AWS IoT Core is the main component to manage devices, their certificates, shadows, Greengrass resources and integration rules to subsequent AWS resources like IoT Analytics. It also offers ways to audit and test your IoT use cases.

AWS IoT Greengrass

AWS IoT Greengrass extends AWS Cloud resources to edge devices, so they can act locally on the generated data, while still using the cloud for management, analytics, and durable storage. It is possible for connected devices to interact with AWS Lambda functions and Docker containers, execute predictions based on machine learning models, keep device data in sync, and communicate with other devices – even when not connected to the Internet.

Greengrass has the following advantages:

Although in many scenarios these advantages are very significant, one could also mention some drawbacks to make the picture more complete:

Connectware & AWS IoT Integration

Before proceeding further, first set up AWS IoT Core (and AWS IoT Greengrass for an edge deployment) by following the respective instructions:

To integrate AWS IoT with the Cybus Connectware, the built-in MQTT connector with TLS support is the simplest, most reliable and secure way of communication. For a successful AWS IoT integration, the Connectware does not require more than that. As an additional advantage, the Connectware MQTT connector has also data buffering built-in, so that data is stored locally when there is a temporary connection failure with AWS IoT Core or Greengrass Core.

There can be two integration scenarios.

In the first integration scenario, the Connectware connects directly to the AWS cloud:

Cybus AWS IoT Core
Cybus AWS IoT Core

In the second integration scenario, the Connectware is connected to Greengrass Core, which is meant to be deployed as a gateway to the AWS cloud next to the Connectware IIoT Edge Gateway:

Cybus AWS IoT Greengrass
Cybus AWS IoT Greengras

Cybus Connectware Service for AWS IoT

For AWS IoT connections using the Connectware, the following has to be configured:

For details on how to get this information, see the article How to connect AWS IoT and Greengrass. Use the example below to implement a simple AWS IoT service transmitting any data structure in the selected MQTT topic.

The definitions part requires PEM formatted certificates:

You may then configure Endpoint and Mapping resources following the Cybus resource documentation.

The commissioning file below sends any data published on topics ${Cybus::MqttRoot}/test/#topic to AWS IoT into topics TestDevice/$topic with a simple transformation rule.

Make sure you are publishing data on the Connectware broker on the respective topic. The placeholder ${Cybus::MqttRoot} represents the root topic defined as services/<serviceId> after the service is successfully started. The notation #topic/$topic represents a wildcard mapping from any topic name used in subscribe to the same topic name in publish, which has the effect of an MQTT bridge with applied rules like the transformation in the example.

Further details on MQTT topic transformations can be found in the article How to connect an MQTT client to publish and subscribe data.

description: >
  Cybus Connectware to AWS IoT Core
metadata:
  name: AWS IoT Core Test
  version: 1.0.0
  provider: cybus
  homepage: https://www.cybus.io

parameters:
  Aws_IoT_Endpoint_Address:
    type: string
    description: The ATS endpoint to reach your AWS account's AWS IoT Core
    default: <your-aws-account-endpoint-id>-ats.iot.eu-central-1.amazonaws.com

definitions:

  # The root CA certificate as PEM format (AmazonRootCA1.pem)
  caCert: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  # The device certificate in PEM CRT format
  clientCert: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  # The device private key in PEM format
  clientPrivateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    -----END RSA PRIVATE KEY-----

resources:

  awsMqttConnection:
    type: Cybus::Connection
    properties:
      protocol: Mqtt
      connection:
        host: !ref Aws_IoT_Endpoint_Address
        port: 8883
        scheme: mqtts
        clientId: !sub "${Cybus::ServiceId}-awsMqttConnection"
        mutualAuthentication: true
        caCert: !ref caCert
        clientCert: !ref clientCert
        clientPrivateKey: !ref clientPrivateKey

  sourceTargetMapping:
    type: Cybus::Mapping
    properties:
      mappings:
        - subscribe:
            topic: !sub "${Cybus::MqttRoot}/test/#topic"
          publish:
            connection: !ref awsMqttConnection
            topic: TestDevice/$topic
          rules:
            - transform:
                expression: |
                  (
                    {
                      "deviceId": "TestDevice",
                      "payload": $
                    }
                  )

Changes for AWS IoT Greengrass

In order to connect to a Greengrass Core, the example service commissioning file needs several changes:

See the article How to connect AWS IoT and Greengrass about how to get the Greengrass Group Certificate Authority.

parameters:
...
  awsGreengrassClientId:
    type: string
    default: TestDeviceEdge
...
resources:
  greengrassTestDeviceEdgeMqttConnection:
    type: Cybus::Connection
    properties:
      protocol: Mqtt
      connection:
        host: !ref Greengrass_Core_Endpoint_Address
        port: 8883
        scheme: mqtts
        clientId: !ref awsGreengrassClientId
        mutualAuthentication: true
        caCert: !ref caCert
        clientCert: !ref clientCert
        clientPrivateKey: !ref clientPrivateKey
...

Tools

To implement or maintain a new IIoT Edge integration use case as fast and reliable as possible, there are suitable tools for working with MQTT, Connectware and AWS IoT.

AWS Command Line Interface (CLI)

The AWS CLI generally helps with any task on AWS. In this case we have at least two tasks being most efficiently completed using the CLI:

1) Find out the AWS IoT ATS endpoint defined for your AWS account:

aws iot describe-endpoint --endpoint-type iot:Data-ATS

The response contains the AWS account specific ATS (Amazon Trust Services) endpoint address to be used as the MQTT hostname:

{
   "endpointAddress": "a7t9...1pi-ats.iot.eu-central-1.amazonaws.com"
}

2) Get the Greengrass Group Certificate Authority certificate in case of using AWS IoT Greengrass. You then need the following for the caCert setting in the service commissioning file instead of the Amazon Root CA:

aws greengrass list-groups 
aws greengrass list-group-certificate-authorities --group-id "4824ea5c-f042-42be-addc-fcbde34587e7" 
aws greengrass get-group-certificate-authority --group-id "4824ea5c-f042-42be-addc-fcbde34587e7" \ 
--certificate-authority-id "3e60c373ee3ab10b039ea4a99eaf667746849e3fd87940cb3afd3e1c8de054af"

The JSON Output of the latter call has a field PemEncodedCertificate containing the requested information which needs to be set as the caCert parameter similar to this:

-----BEGIN CERTIFICATE-----
MIIC1TCCAb2gAwIBAgIJANXVxedsqvdKMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMDEwMDUwNTM4MzRaFw0zMDEwMDMwNTM4MzRaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM/0NrS45cm0ovF3+8q8TUzj+E3UH8ldnJJPCQFGMaL+7PoxbO0fYf3ETkEW+dijIZOfus9dSPX7qBDbfilz/HtNppGDem4IjgC52iQl3B1R7TvU8yLNliv43uDDUd+PkzW1cWbUuykr5QPG2sIDSANukosvRdFKO4ydP0Hr9iUdOfbg4k6hMFCrzJubKQqhcBTSsxGtl78abx0Q49shuWr9RRjzqE6mRFa4h0DrKBstgAfmsDRGm4ySBCM7lwxphSsoejb6l39WI/MNU7/U7cGj26ghWHAWp8VCksBOqma8tmr/0BuqcCgKJYaDr1tf4SVxlwU20K+jz0pphdEwSj0CAwEAAaMeMBwwGgYDVR0RBBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQBkcKC3cgHJGna6OxA5QM3dGM5pEiSXyZt5HWoW8z6wUlYtir6U+mWIb9yg7zaSy9nUOqU4sizQh1HG/Mq9K2WbflGafvfN0wW16uyINdjcfGYDh43UDkXHr5Xzky5XIgt0Fx4BWmjgbLYsza7qpbeIg5ekUYPYQw1Ic2sNpyncmS0eutg4tAO7uzDu1x84WPcZzUjDHKYfupuDXkWroPnHTAxlJ6vtgW976c3Z5rQ5l8bUysWhLBEM8q2OP/zmGDo7fpUHYOKo5qU4h7vGD3t0Pb4ufPOd7XtHuY6HsI2cAPV3tpuetHH6wyAQTG9luhdYrZjAp+ZvlwBm+9nXYp/Y
-----END CERTIFICATE-----

Cybus workbench service

The Workbench service is basically a Node-RED application running securely on the Connectware as a service. This opens up the possibility to install any Node-RED nodes within the service container for quick prototyping as well as for the production environment. If your use-case cannot be achieved with the above service commissioning file, using the workbench will give you some flexibility and additional tools to prototype your solution using Node-RED modules.

In case of AWS IoT, MQTT connection is enough for most integration scenarios. You may use simple injection nodes and some random value generator in order to implement and test the use northbound to AWS IoT:

Cybus Workbench AWS IoT

If there are other requirements such as working with shadow devices and other AWS resources, e.g. as part of the IoT Greengrass Core deployment, you may want to use additional Node-RED modules supporting AWS.

AWS IoT SDK

If it comes to more complex data management and handling, you may want to use the AWS IoT Device SDK to create a specific Connector Service for Connectware to cover your requirements.

In most cases, it is enough to process any kind of device data and apply rules to them on the Connectware as the most powerful edge gateway tool. Similar capabilities can be used on the near-cloud gateway AWS IoT Greengrass or AWS IoT Core itself to manage rules and transformations near the shadow devices definitions.

What works best depends on your business strategy and technical constraints.

AWS IoT Core, Analytics and other AWS resources

Now that we are successfully sending the data to the IoT Core, we can monitor the transmitted data using various AWS resources.

The obvious tool is the AWS IoT Core MQTT Client offered on the AWS IoT console. With this tool you regularly subscribe to your topic defined in the service commissioning file for outgoing data:

AWS IoT Core MQTT client

In order to make use of AWS resources, you define AWS IoT rules and define actions appropriately, e.g. transmission to IoT Analytics and a DynamoDB table:

AWS IoT rules

The AWS IoT Console helps to quickly implement data transfer to these endpoints.

An example of how to work with these resources could be a change to the transformation mentioned above to better meet the requirements using the fast and easy mapping support of the Connectware. Given a requirement to flatten an original data object injected into the internal topic, you can easily transform that data using a Connectware transformation rule using Jsonata:

Given a structured object:

"DeviceData": {
    "Temperature": <decimal>,
    "Position": {
        "X": <decimal>,
        "Y": <decimal>,
        "Z": <decimal>
    }
}

As an example, the above mentioned mapping could be then enhanced for flattening the elements and adding a timestamp:

  sourceTargetMapping:
...
          rules:
            - transform:
                expression: |
                  (
                    {
                      "deviceId": "TestDeviceEdge",
                      "payload": $
                    }
                  )
            - transform:
                expression: |
                  (
                    {
                      "deviceId": "TestDeviceEdge",
                      "timestamp": $now(),
                      "temperature": $.payload.DeviceData.Temperature,
                      "position_x": $.payload.DeviceData.Position.X,
                      "position_y": $.payload.DeviceData.Position.Y,
                      "position_z": $.payload.DeviceData.Position.Z
                    }
                  )

After implementing the use case, you may see the options to shorten things a bit. The Connectware then plays its strength with fast integration processes near the connected devices, where most of the data pre-processing can be realized with low latency and fewer costs before transmitting it to the cloud.

The enhanced transformation rule within the Connectware mentioned above may be inspired by a requirement to write the data in a well-structured database:

AWS IoT Rule Dynamo DB

Or the requirement was to create some graph with Amazon Quicksight:

AWS IoT Analytics Quicksight

If it comes to the AWS Cloud, there is a vast amount of resources that can be useful to create your IoT Application. You should especially have a look at lambda functions that could be deployed to your IoT Greengrass Core instance.

Other new tools like AWS IoT SiteWise or AWS IoT Things Graph may be useful to build your IoT applications faster with easier management and monitoring.

Summary

This lesson first offered a brief introduction to AWS IoT and its components available for integration with other services. Then it explained how to send data from the Connectware MQTT Broker to AWS IoT Core or Greengrass Core with a simple commissioning file using the built-in MQTT connector of the Connectware. Furthermore, the Cybus workbench service for prototyping more advanced scenarios was presented. The lesson finished with a description of some basic and advanced tools used to monitor data flow between AWS IoT and Connectware.

Where to go from here

Cybus provides an example with sample service commissioning files, and some more technical details in the Github project How to connect AWS IoT and Greengrass.

From there you may have further ideas on how to benefit from the IIoT Edge capabilities of the Connectware compared to AWS IoT and Greengrass.

Ihr Browser unterstützt diese Webseite nicht.

Liebe Besucher:innen, Sie versuchen unsere Website über den Internet Explorer zu besuchen. Der Support für diesen Browser wurde durch den Hersteller eingestellt, weshalb er moderne Webseiten nicht mehr richtig darstellen kann.
Um die Inhalte dieser Website korrekt anzeigen zu können, benötigen Sie einen modernen Browser.

Unter folgenden Links finden Sie Browser, für die unsere Webseite optimiert wurde:

Google Chrome Browser herunterladen Mozilla Firefox Browser herunterladen

Sie können diese Website trotzdem anzeigen lassen, müssen aber mit erheblichen Einschränkungen rechnen.

Diese Website trotzdem anzeigen.