One of the most common concerns I hear regarding the Unified Namespace (UNS) is the architecture lacks the versatility needed to address diverse downstream data consumers.
Suppose quality, maintenance, and process engineers are all doing their part to support a production line. Quality teams need inspection results, maintenance teams need asset performance data, and process engineers need lot and process parameters. The teams need data sets that both overlap and differ by use case. These engineers are using different applications and services—anything from ERP modules for quality and maintenance to specialized ML platforms in the cloud—that each require very different data structures. Many of these applications and services do not easily interoperate with the UNS and its architectural conventions. They may not natively interface with MQTT brokers, nor should one expect them to. They may not consume payloads that were oriented around rigid asset hierarchy and publishing telemetry data from process control nodes. They may have completely different needs than what was envisioned when factory automation was installed and integrated. Their data needs can transcend how the UNS was initially architected and organized.
HighByte Intelligence Hub can overcome these challenges. Through data modeling and pipelines, the Intelligence Hub enables the full potential of the UNS, delivering contextualized manufacturing data to the cloud. Let’s look at a sample architecture to see how.
Suppose quality, maintenance, and process engineers are all doing their part to support a production line. Quality teams need inspection results, maintenance teams need asset performance data, and process engineers need lot and process parameters. The teams need data sets that both overlap and differ by use case. These engineers are using different applications and services—anything from ERP modules for quality and maintenance to specialized ML platforms in the cloud—that each require very different data structures. Many of these applications and services do not easily interoperate with the UNS and its architectural conventions. They may not natively interface with MQTT brokers, nor should one expect them to. They may not consume payloads that were oriented around rigid asset hierarchy and publishing telemetry data from process control nodes. They may have completely different needs than what was envisioned when factory automation was installed and integrated. Their data needs can transcend how the UNS was initially architected and organized.
HighByte Intelligence Hub can overcome these challenges. Through data modeling and pipelines, the Intelligence Hub enables the full potential of the UNS, delivering contextualized manufacturing data to the cloud. Let’s look at a sample architecture to see how.
Data from the line to the cloud
In our sample architecture, we will deliver usable data from industrial sensors, machines, and systems to cloud applications through the UNS.
Cloud-based data consumers have historically been somewhat difficult for the UNS to reach, exemplifying what is often referred to as “the OT-IT gap.” A UNS architecture is ideal for industrial systems that use the publish-subscribe pattern for exchanging telemetry data, but anything else requires a robust abstraction layer (the part of the UNS in which datasets are assembled, contextualized, and standardized). The Intelligence Hub can not only build and govern the UNS; it can also serve as the UNS and subscribe to the UNS, providing data to cloud services and other applications according to their unique data requirements. From top to bottom, that means the Intelligence Hub can:
Cloud-based data consumers have historically been somewhat difficult for the UNS to reach, exemplifying what is often referred to as “the OT-IT gap.” A UNS architecture is ideal for industrial systems that use the publish-subscribe pattern for exchanging telemetry data, but anything else requires a robust abstraction layer (the part of the UNS in which datasets are assembled, contextualized, and standardized). The Intelligence Hub can not only build and govern the UNS; it can also serve as the UNS and subscribe to the UNS, providing data to cloud services and other applications according to their unique data requirements. From top to bottom, that means the Intelligence Hub can:
- Normalize and contextualize source data according to the broad needs of the UNS and its consumers, knowing these needs will evolve over time.
- Publish data to subscribers via the embedded MQTT broker or third-party MQTT broker infrastructure.
- Create payloads that transform and blend data from multiple sources to meet the needs of the consuming system.
- Deliver the data payload, at the right moment or frequency, directly to the consuming system.
- Manage acknowledgments, and monitor and govern these data pipelines to and from the UNS as repeatable, mission-critical business processes.
This diagram provides an overview of what data movement might look like for Intelligence Hub users operating a UNS alongside AWS and Snowflake. In the blog Abstraction puts the ‘unified’ in Unified Namespace, my colleague John Harrington covers the basics of how machine data can be contextualized for the UNS. On the downstream side, let’s discuss how the Intelligence Hub contextualizes UNS data for the cloud, using Snowflake Data Cloud and AWS IoT SiteWise as examples.
UNS and Snowflake
Snowflake Data Cloud is one of the most popular target systems in the architectures of HighByte Intelligence Hub users. The Data Cloud platform enables instant, frictionless secure sharing of live data within and between organizations with scalability, concurrency, and performance.
In previous versions of the Intelligence Hub, users could connect to Snowflake Data Cloud using general interfaces such as JDBC (SQL), MQTT Sparkplug, or Kafka, or through a cloud object storage service such as Amazon S3 or Azure Blob Storage. As part of Intelligence Hub version 3.3, HighByte has unveiled two dedicated Snowflake connectors: Snowflake Streaming and Snowflake SQL. These two connectors help bring the power of Snowflake’s Data Cloud to the UNS.
The first connector, Snowflake Streaming, utilizes the Snowflake Snowpipe Streaming API. When a user publishes modeled data to Snowflake, the connector creates or updates Snowflake’s table schema based on the model defined in the Intelligence Hub then hydrates it with consistent and contextualized industrial data. This interface enables direct publishing to Snowflake tables without the need for staging files, staging tables, or third-party applications. The direct publishing lowers the compute, storage, latency, and cost of historizing frequent telemetry events in Snowflake. Ideal for the UNS, HighByte Intelligence Hub’s Snowflake Streaming connector allows users to stream contextualized data payloads quickly and easily without needing to manually maintain payload definitions in both OT systems and Snowflake Data Cloud. Snowflake Streaming is a simple, direct path between Snowflake cloud applications and the UNS.
The second connector, Snowflake SQL, enables Intelligence Hub users to directly query Snowflake tables. Rather than only publishing to Snowflake, HighByte Intelligence Hub can make insights and context derived through Snowflake available for industrial devices and applications. In the context of the UNS, this means that advanced calculations using tasks, scripts, or Snowpark libraries in Snowflake can be written back to a UNS as KPIs, advanced process parameters, and events for use by SCADA or HMI nodes. For example, suppose a manufacturer is using the Intelligence Hub to deliver contextualized industrial data to Snowflake and it is hosting ML/AI workloads in Snowflake for anomaly detection related to quality, maintenance, or process engineering use cases. These anomalies are valuable for not just IT services like business intelligence, but also for OT services on the plant floor like alarm management and frontline operations platforms. Intelligence Hub can operationalize these events and insights by making them available for a wide variety of consumers through the UNS.
In previous versions of the Intelligence Hub, users could connect to Snowflake Data Cloud using general interfaces such as JDBC (SQL), MQTT Sparkplug, or Kafka, or through a cloud object storage service such as Amazon S3 or Azure Blob Storage. As part of Intelligence Hub version 3.3, HighByte has unveiled two dedicated Snowflake connectors: Snowflake Streaming and Snowflake SQL. These two connectors help bring the power of Snowflake’s Data Cloud to the UNS.
The first connector, Snowflake Streaming, utilizes the Snowflake Snowpipe Streaming API. When a user publishes modeled data to Snowflake, the connector creates or updates Snowflake’s table schema based on the model defined in the Intelligence Hub then hydrates it with consistent and contextualized industrial data. This interface enables direct publishing to Snowflake tables without the need for staging files, staging tables, or third-party applications. The direct publishing lowers the compute, storage, latency, and cost of historizing frequent telemetry events in Snowflake. Ideal for the UNS, HighByte Intelligence Hub’s Snowflake Streaming connector allows users to stream contextualized data payloads quickly and easily without needing to manually maintain payload definitions in both OT systems and Snowflake Data Cloud. Snowflake Streaming is a simple, direct path between Snowflake cloud applications and the UNS.
The second connector, Snowflake SQL, enables Intelligence Hub users to directly query Snowflake tables. Rather than only publishing to Snowflake, HighByte Intelligence Hub can make insights and context derived through Snowflake available for industrial devices and applications. In the context of the UNS, this means that advanced calculations using tasks, scripts, or Snowpark libraries in Snowflake can be written back to a UNS as KPIs, advanced process parameters, and events for use by SCADA or HMI nodes. For example, suppose a manufacturer is using the Intelligence Hub to deliver contextualized industrial data to Snowflake and it is hosting ML/AI workloads in Snowflake for anomaly detection related to quality, maintenance, or process engineering use cases. These anomalies are valuable for not just IT services like business intelligence, but also for OT services on the plant floor like alarm management and frontline operations platforms. Intelligence Hub can operationalize these events and insights by making them available for a wide variety of consumers through the UNS.
UNS and AWS
AWS IoT SiteWise is an asset-oriented time series database service that collects, models, stores, and monitors industrial data. The capability to create models, assets, and asset relationships in AWS IoT SiteWise and hydrate it with industrial data has long been available in HighByte Intelligence Hub. Using its modeling capabilities, the Intelligence Hub can normalize a wide variety of industrial data sources into a consistent set of measurements within modeled assets in IoT SiteWise.
New in HighByte Intelligence Hub version 3.3, users can browse and target the IoT SiteWise asset namespace from the Intelligence Hub as well as extract model, instance, and relationship definitions as metadata using the new instance expression functions. This can be used to further contextualize payloads or dynamically route their delivery.
Importantly, the Intelligence Hub can subscribe to the UNS and historize modeled data, a necessary step in delivering data from a UNS to a time series database like IoT SiteWise. Functioning on behalf of the consuming system, the Intelligence Hub gathers UNS data to make it easily ingestible by IoT SiteWise, delivering UNS data at the optimal interval or event and with the right context.
New in HighByte Intelligence Hub version 3.3, users can browse and target the IoT SiteWise asset namespace from the Intelligence Hub as well as extract model, instance, and relationship definitions as metadata using the new instance expression functions. This can be used to further contextualize payloads or dynamically route their delivery.
Importantly, the Intelligence Hub can subscribe to the UNS and historize modeled data, a necessary step in delivering data from a UNS to a time series database like IoT SiteWise. Functioning on behalf of the consuming system, the Intelligence Hub gathers UNS data to make it easily ingestible by IoT SiteWise, delivering UNS data at the optimal interval or event and with the right context.
Wrap Up
The UNS architecture continues to grow in both interest and adoption. Despite the heterogeneity of Operational Technology (OT) within and across industry verticals, a UNS architecture enables organizations to harmonize distribution of industrial data in one place and in a structure that resembles their business. Rather than being constrained to the way OT solutions present data, a UNS provides the freedom to publish and consume industrial data in the way organizations operate.
I hope you’ve found the examples presented in this post to be useful in your understanding of the UNS and how to put UNS data to work for a broad set of applications, services, and business use cases.
In addition to the new and improved connectors for cloud services, many new features and capabilities are included in HighByte Intelligence Hub version 3.3. To learn more about the latest release, please check out these additional resources:
I hope you’ve found the examples presented in this post to be useful in your understanding of the UNS and how to put UNS data to work for a broad set of applications, services, and business use cases.
In addition to the new and improved connectors for cloud services, many new features and capabilities are included in HighByte Intelligence Hub version 3.3. To learn more about the latest release, please check out these additional resources:
- Watch HighByte Office Hours to learn how to architect an Enterprise UNS with live demos of Intelligence Hub version 3.3.
- Explore the release notes for details on all new features and fixes.
- Request a free trial or log in to your existing account to test and deploy the software in your unique environment.
