In this lesson we will set up a CI/CD Pipeline for Connectware using Ansible and GitLab. Our Connectware infrastructure will be composed of:
- 10 Siemens S7 connections services
- 10 Modbus connections services
- 1 data aggregation and transformation service
- 1 SAP Cloud connection service
As a prerequisite, it is necessary to have a valid Connectware License on hand. In addition it is necessary to have a Git environment. This can be either hosted on the cloud or on premise. We’ll be using GitLab for this example.
Finally, the deployment environment (i.e. the machine that will run Connectware) must have Docker and Docker Compose installed.
Info: Deployment environments can be hosted on premise or use a cloud solution. For advice on setting up your deployment environment please get in contact at Cybus Support
We assume you are already familiar with Cybus Connectware and its service concept. If not, we recommend reading the articles Connectware Technical Overview and Service Basics for a quick introduction. Furthermore this lesson requires basic understanding of Docker and Ansible. If you want to refresh your knowledge, we recommend looking at the lesson Docker Basics and this Ansible Getting started guide.
Introduction
GitLab CI/CD is a tool for software development that uses the concepts of Continuous Integration (CI) and Continuous Deployment / Delivery (CD), both are fundamental parts of modern DevOps.
Continuous integration is the practice where developers frequently collaborate and merge code changes into a central repository where automated builds, tasks and tests run. The build process helps to ensure that the application is not broken whenever there are new changes in the commits. Continuous Deployment is an extension of CI, as it deploys the latest changes to a test or production environment.
CI/CD helps deliver applications faster, with more quality and less integration problems. It also removes development pressure on decisions for small changes and promotes faster iterations.
Project Structure
To deploy Cybus Connectware using Ansible and the CI/CD pipeline we will set up a GitLab project with the following structure:
├── README.md
├── .gitlab-ci.yml
├── ansible
│ ├── ansible.cfg
│ ├── hosts.yaml
│ └── playbook.yaml
└── services
├── data-aggregation.yml
├── modbus.yml
├── s7.yml
└── sap.yml
Code-Sprache: YAML (yaml)Under services we place the Connectware services‘ commissioning files. Under ansible we place our playbook and a host configuration (more on this later). Finally, .gitlab-ci.yml is our pipeline configuration file.
Pipeline Configuration
To provide GitLab with all the information that is required to run the CI/CD pipeline, we need to write a short configuration file .gitlab-ci.yml.
Here we describe:
- The structure and order of jobs that we want to be executed (e.g. tasks, tests).
- The decisions the pipeline should make when specific conditions are encountered.
Note: You need to ensure that you have Runners available. If you’re using gitlab.com , you can skip this step. gitlab.com provides shared runners for you.
For self hosted runners, please refer to Registering runners | GitLab
variables:
CONNECTWARE_VERSION: "1.1.3"
stages:
- deploy
deploy-connectware:
rules:
- if: '$CI_PIPELINE_SOURCE == "web"' # Trigger from Web-UI
when: always # Always run
- when: never # Else never run
image:
name: registry.cybus.io/cybus/cybus-connectware-ansible:latest
entrypoint: [""]
stage: deploy
script:
- "chmod o-w ansible"
- "chmod 600 $SSH_KEY"
- "cd ansible && ansible-playbook playbook.yaml"
Code-Sprache: YAML (yaml)The file uses the YAML text format, and follows the GitLab keyword reference. Inside .gitlab-ci.yml we define variables, stages (i.e. tasks) and the pipeline’s trigger.
- Variables: Connectware version to install.
- Stages: deploy is our 1 and only stage in our pipeline. In this stage we run deploy-connectware job.
- Jobs: deploy-connectware consist of rules, an image and a script.
- Rules: When is the pipeline triggered.
- Script: Makes Ansible executable and executes the ansible/playbook.yml (which deploys Connectware and its services, see Using Ansible for Connectware orchestration).
- Image: Latest Ansible image from Cybus‘ registry
- Pipeline trigger: In this case through the GitLab Web UI (Run Pipeline Button ). This can also be configured to be triggered by a new commit or merge event.
Ansible Playbook and Host Configuration
To deploy Connectware and its services we need to write a short playbook. In principle with the same structure as seen in Using Ansible for Connectware orchestration | Tutorial . One key difference might be the host. For on-premise self-hosted Git environments, the host name should be the IP address where you want Connetware to run, for example: localhost.
However, if you are using a cloud solution, such as gitlab.com, the host needs to point to a defined resource, such as a cloud environment.
In our example we’ll use AWS EC2 Instance to deploy Connectware.
Create an ansible/playbook.yml with the following content:
- name: Connectware Infrastructure Playbook
hosts: AWS
vars:
connectwareVersion: "{{ lookup('env','CONNECTWARE_VERSION') }}"
adminPassword: "{{ lookup('env','CONNECTWARE_ADMIN_PASSWORD') }}"
license: "{{ lookup('env','CYBUS_CONNECTWARE_LICENCE') }}"
tasks:
- name: "Deploy Connectware"
become: yes
cybus.connectware.instance:
version: "{{ connectwareVersion }}"
license: "{{ license }}"
admin_password: "{{ adminPassword }}"
- name: Install Services (S7)
loop:
- host: dev01
- host: dev02
- host: dev03
- host: dev04
- host: dev05
- host: dev06
- host: dev07
- host: dev08
- host: dev09
- host: dev10
loop_control:
loop_var: s7
cybus.connectware.service:
admin_password: "{{ adminPassword }}"
id: "{{ s7.host }}_S7"
commissioning_file: ../services/s7.yml
parameters:
ipAddress: "{{ s7.host }}"
- name: Install Services (Modbus)
loop:
- host: dev11
- host: dev12
- host: dev13
- host: dev14
- host: dev15
- host: dev16
- host: dev17
- host: dev18
- host: dev19
- host: dev20
loop_control:
loop_var: modbus
cybus.connectware.service:
admin_password: "{{ adminPassword }}"
id: "{{ modbus.host }}_Modbus"
commissioning_file: ../services/modbus.yml
parameters:
ipAddress: "{{ modbus.host }}"
- name: Install SAP Connector Service
cybus.connectware.service:
admin_password: "{{ adminPassword }}"
id: "demo_sapconnector"
commissioning_file: ../services/sap.yml
- name: Install Data Mapping Service
cybus.connectware.service:
admin_password: "{{ adminPassword }}"
id: "demo_datamapping"
commissioning_file: ../services/data-aggregation.yml
Code-Sprache: YAML (yaml)Notice that our host points to AWS.
Info: In our playbook.yml we are using environment variables such as :
"{{ lookup('env','CYBUS_CONNECTWARE_LICENCE') }}"
- To set up the values of these variables, go to GitLab and select Settings > CI/CD > Variables.
Note: It is best practice to place sensitive information as environment variables and NOT in the code base.
Create another file called host.yaml with the following content:
---
all:
hosts:
AWS:
ansible_connection: ssh
ansible_host: demo-devops.cybus.io
ansible_port: 22
ansible_user: ubuntu
ansible_ssh_private_key_file: "{{ lookup('env','SSH_KEY') }}"
Code-Sprache: YAML (yaml)This will help Ansible deploy the Connectware infrastructure in our cloud endpoint (demo-devops.cybus.io)
Run the Pipeline
Make sure your project has:
- Ansible playbook and host configuration under /ansible
- Connectware service commissioning files under /services
- CI/CD configuration file .git.ci.yml
- Necessary environmental variables
- Valid deployment endpoint (host)
- Available Runners (either cloud based or self-hosted)
Then on the left panel, go to CI/CD → Pipelines → Run Pipeline
Congratulations, your pipeline should be running soon.
After a few minutes, the build will finish and GitLab will update the status to Passed. You should be able to open your host endpoint and login to Connectware.
In case the Pipeline fails, the status will be updated to Failed. You are able to see more information by clicking on the status button. This will display the logs from the execution.
Prerequisites
In this lesson we will set up a local Cybus Connectware Instance using Ansible.
As a prerequisite, it is necessary to have Ansible, Docker and Docker Compose installed on your system as well as a valid Connectware License on hand.
Docker shouldn’t be installed using snapcraft!
We assume you are already familiar with Cybus Connectware and its service concept. If not, we recommend reading the articles Connectware Technical Overview and Service Basics for a quick introduction. Furthermore this lesson requires basic understanding of Docker and Ansible. If you want to refresh your knowledge, we recommend looking at the lesson Docker Basics and this Ansible Getting started guide.
Introduction
Ansible is an open-source provisioning tool enabling infrastructure as code. Cybus provides a set of custom modules (Collection) exclusively developed to manage every part of a Connectware Deployment, seamlessly integrated into the Ansible workflow. With only a few lines of code you can describe and roll out your whole infrastructure, including services, users and many more.
The collection provides the following modules:
- instance | Manages Connectware Instances
- service | Manages Connectware Services
- role | Manages Connectware Roles
- user | Manages Connectware Users
- agent | Manages Agent Instances
Ansible Collection Installation
First of all we have to make sure that the Connectware Ansible Collection is present on our system. The Collection is available for Download on Ansible Galaxy.
Installing the Collection is fairly easy by using Ansible Tools:
$ ansible-galaxy collection install cybus.connectware
Code-Sprache: YAML (yaml)To get a list of all installed collections you can use the following command:
$ ansible-galaxy collection list
Code-Sprache: YAML (yaml)If you already have installed the collection you can force an update like this:
$ ansible-galaxy collection install cybus.connectware --force
Code-Sprache: YAML (yaml)Writing Our First Playbook
To provide Ansible with all the information that is required to perform the Connectware installation, we need to write a short playbook. Create a empty folder and a file called playbook.yaml with the following content:
- name: Connectware Deployment
hosts: localhost
tasks:
- name: Deploy Connectware
cybus.connectware.instance:
license: ***
install_path: ./
Code-Sprache: YAML (yaml)Taking the file apart we define one play named Connectware Deployment which will be executed on the given hosts. The only host in our case is the localhost.
Then we define all the tasks to be executed. We only have one task called Deploy Connectware. The task takes a module to be executed along with some configuration parameters. For now we only specify parameters for the licence and the install_path. Make sure to replace the *** with your actual licence key. The install_path will be the one you created your playbook in.
There are a lot more parameters to use with this module, but the only required one is license. To see a full list use this command:
$ ansible-doc cybus.connectware.instance
Code-Sprache: YAML (yaml)For running the playbook open a shell in the newly created folder and execute this command (execution may take a few minutes):
$ ansible-playbook playbook.yaml
Code-Sprache: YAML (yaml)The output should look somewhat similar to this.
Notice that the state of the Deploy Connectware task is marked as changed. This indicates that the Connectware is now running and is reachable at https://localhost.
Beside the log output you should now find two new files beside the playbook.yaml. One file is the actual docker compose file managing the Connectware Containers and the other one is holding some additional configurations.
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [Connectware Deployment] ***************************************************************************
TASK [Gathering Facts] ***************************************************************************
ok: [localhost]
TASK [Deploy Connectware] ***************************************************************************
changed: [localhost]
PLAY RECAP ***************************************************************************
localhost: ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Code-Sprache: YAML (yaml)If you like you can rerun the exact same command, which will result in no operation, since the Connectware already has the desired state. As you can see, this time there is no changed state.
$ ansible-playbook playbook.yaml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [Connectware Deployment] ***************************************************************************
TASK [Gathering Facts] ***************************************************************************
ok: [localhost]
TASK [Deploy Connectware] ***************************************************************************
ok: [localhost]
PLAY RECAP ***************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Code-Sprache: YAML (yaml)Installing a Service
Now that we have a running Cybus Connectware instance, we go a step further and install a Service. First thing we have to do is to create the Service Commissioning File. We will use a very simple Service for demonstration purposes. Create a file called example-service.yml and paste in the following content:
---
description: Example Service
metadata:
name: example_service
resources:
mqttConnection:
type: Cybus::Connection
properties:
protocol: Mqtt
connection:
host: !ref Cybus::MqttHost
port: !ref Cybus::MqttPort
scheme: mqtt
username: !ref Cybus::MqttUser
password: !ref Cybus::MqttPassword
Code-Sprache: YAML (yaml)There is really not much going on here than simply creating a connection to the internal Connectware Broker.
Next we have to enrich our playbook by appending another task like this:
- name: Install Service
cybus.connectware.service:
id: example_service
commissioning_file: ./example-service.yml
Code-Sprache: YAML (yaml)The task uses another module of the collection for managing services. The required parameters are id and commissioning_file.
You can learn more about the module by using this command:
$ ansible-doc cybus.connectware.service
Code-Sprache: YAML (yaml)When executing the playbook again, you should see similar output to this:
$ ansible-playbook playbook.yaml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [Connectware Deployment] ***************************************************************************
TASK [Gathering Facts] ***************************************************************************
ok: [localhost]
TASK [Deploy Connectware] ***************************************************************************
ok: [localhost]
TASK [Install Service] ***************************************************************************
changed: [localhost]
PLAY RECAP ***************************************************************************
localhost: ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Code-Sprache: YAML (yaml)When visiting the Connectware UI Services page you should now see the service being installed and enabled (https://localhost/admin/#/services).
Stopping Connectware
Every module supports different states. The default state of the instance module for example is started. To stop the Connectware we define another task, use the instance module and define the state to be stopped.
- name: Stopping Connectware
cybus.connectware.instance:
state: stopped
license: ***
install_path: ./
Code-Sprache: YAML (yaml)After executing the playbook once more, the Cybus Connectware should no longer be running.
Next Steps for Using Ansible
All the steps described above are really basic and there is so much more to learn and discover.
The recommended next steps are:
- Ansible Connectware Collection inside Docker #add Link
- Automated Connectware Deployment using GitLab CI/CD and Ansible