{"id":45257,"date":"2024-10-29T17:16:09","date_gmt":"2024-10-29T16:16:09","guid":{"rendered":"https:\/\/beta.cybus.io\/?p=45257"},"modified":"2025-06-24T15:29:17","modified_gmt":"2025-06-24T13:29:17","slug":"data-integration-chemical-industry","status":"publish","type":"post","link":"https:\/\/www.cybus.io\/en\/data-integration-chemical-industry\/","title":{"rendered":"Integration of Shopfloor Data"},"content":{"rendered":"\n
\n

International chemical group integrates production data with Azure Data Lake Storage Gen2 (ADLS)<\/h1>\n\n\n\n

More efficiency and secure data management for industrial environments thanks to simplified data pipeline<\/strong>

For a PoC, the large chemical company collected production and laboratory data at high frequency to use it for analysis and BI via the cloud. The PoC successfully demonstrated how Cybus Connectware can seamlessly synchronize unstructured data such as images and files with Azure Data Lake. Additionally, scalability was proven by extending the integration from one to multiple endpoints.<\/p>\n<\/div><\/div><\/div><\/div>\n\n\n\n

\n

What is Azure Data Lake Storage Gen2?<\/strong><\/h3>\n\n\n\n

Azure Data Lake Storage Gen2 is Microsoft\u2019s Azure scalable and secure data lake solution, designed to handle massive amounts of data in a cost-effective manner. Built on Azure Blob storage, it provides hierarchical namespace support and is optimized for big data analytics, making it ideal for organizations looking to store, process, and analyze large volumes of structured and unstructured data.<\/p>\n\n\n\n

Seamless data transfer to the cloud with Cybus Connectware<\/strong><\/h3>\n\n\n\n

The Group’s Manufacturing Data Integration team relies on Azure Data Lake as the primary repository for all production and manufacturing data. This includes structured data from production and laboratory environments, such as OPC UA devices, as well as unstructured data such as images or test files. The ability to collect and analyze data from different sources is an essential prerequisite for making informed decisions.<\/p>\n\n\n\n

However, the challenge lies in the seamless transfer of this data from on-site devices to a cloud environment where it can be further analyzed and used. Traditional methods often involve complex and time-consuming processes that create a bottleneck in the data pipeline. This is where Connectware comes in, providing a reliable and efficient way to collect, manage, and transfer this data to the cloud.<\/p>\n\n\n

\n
\"\"
Connecting to the Cloud: The Architecture<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

Integration and data synchronization via the protocol connector<\/strong><\/span><\/h3>\n\n\n\n

To overcome this challenge, Connectware’s protocol connector was used, which can be seamlessly integrated into Azure Data Lake Storage Gen2. Connectware’s native HTTP request capabilities also enabled data to be uploaded directly from OPC UA devices in labs and workshops to the cloud-based data lake.<\/p>\n\n\n\n

One of the key components that could be used in Azure was the blob storage artifact. Specifically to secure the data generated throughout the day in a single CSV file within the blob. To achieve this, we implemented the Blob Append feature, which allows new data to be continuously added to the existing file without creating multiple files. This approach ensures a lean and organized data storage, which is crucial for efficient data analysis.<\/p>\n\n\n\n

In addition, Azure’s AZCopy tool was used in conjunction with a Connectware agent to securely synchronize other file types, such as images, and ensure that all relevant data was transferred to the cloud securely and efficiently.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\n

Secure and efficient data integration with Connectware<\/strong><\/h3>\n\n\n\n

Cybus Connectware is characterized by its ability to integrate seamlessly into existing industrial infrastructures and offers a flexible, scalable solution for efficient data management. By connecting directly to Azure Data Lake Storage Gen2, Connectware enables reliable and secure transfer of production data to the cloud without manual intervention or complex customization. This combination offers companies decisive advantages that go far beyond conventional data solutions.
<\/p>\n\n\n\n