Since the third industrial revolution, manufacturers have been slowly but steadily automating their operations. From the PLC to ERP to Computer-Integrated Manufacturing (CIM), industry has adopted myriad advancements in technology to gain an edge and, sometimes, even to survive. Rapid technological advancement led to a slew of devices, applications, and services that were rarely introduced or managed under a holistic technology strategy.
Until the advent of the fourth industrial revolution, it was significantly challenging or even impossible to integrate across these silos of industrial automation. The recent confluence of networking and computing technology and their mainstream use have fundamentally shifted industry. Rather than approaching industrial automation as individual functions, they can now be connected to form integrated, holistic, digital business processes. With this interoperability, industrial automation is not just performing more efficiently; it is being applied in entirely new ways and wholly transforming operations.
This is where Industrial DataOps comes in. To intelligently utilize disparate devices, applications, and services together and across domains, there must be a common data infrastructure. This infrastructure must be adaptable to its nodes rather than nodes adapting to it. It not only must be able to exchange data across many connections, but, most importantly, make data usable for consuming systems regardless of how data was produced.
In the past twelve months, we have observed a significant maturation in the adoption of Industrial DataOps and a shift in industrial organizations’ digital strategies. As users discover how data can be transformed and used, their needs for data grow in both volume and diversity. Increasingly complex use cases drive a need to connect to devices, applications, and services in more sophisticated ways, such as intelligent data browsing, monitoring, and aggregation within data sources prior to data acquisition. These use cases also require a holistic approach to data delivery. Industrial enterprises may need to target hot/warm/cold-tier storage, drive schema, or set metadata fields in consuming systems, requiring far more than simply publishing updated values.
In HighByte Intelligence Hub version 3.4, we’re excited to introduce new advanced connection capabilities to more intelligently source and deliver data. This includes SQL Change Data Capture, OPC UA Alarms & Conditions, AWS IoT SiteWise buffered ingestion, Google BigQuery, and more. These are complemented with new data engineering and governance capabilities including several Pipeline enhancements and Secrets Management.
This is where Industrial DataOps comes in. To intelligently utilize disparate devices, applications, and services together and across domains, there must be a common data infrastructure. This infrastructure must be adaptable to its nodes rather than nodes adapting to it. It not only must be able to exchange data across many connections, but, most importantly, make data usable for consuming systems regardless of how data was produced.
In the past twelve months, we have observed a significant maturation in the adoption of Industrial DataOps and a shift in industrial organizations’ digital strategies. As users discover how data can be transformed and used, their needs for data grow in both volume and diversity. Increasingly complex use cases drive a need to connect to devices, applications, and services in more sophisticated ways, such as intelligent data browsing, monitoring, and aggregation within data sources prior to data acquisition. These use cases also require a holistic approach to data delivery. Industrial enterprises may need to target hot/warm/cold-tier storage, drive schema, or set metadata fields in consuming systems, requiring far more than simply publishing updated values.
In HighByte Intelligence Hub version 3.4, we’re excited to introduce new advanced connection capabilities to more intelligently source and deliver data. This includes SQL Change Data Capture, OPC UA Alarms & Conditions, AWS IoT SiteWise buffered ingestion, Google BigQuery, and more. These are complemented with new data engineering and governance capabilities including several Pipeline enhancements and Secrets Management.
Advanced Connections
Intelligence Hub version 3.4 has a plethora of new capabilities in existing connectors and the addition of a new Google BigQuery connector.
SQL: Change Data Capture
Change Data Capture (CDC) is now supported in the Microsoft SQL Server and PostgreSQL connections, enabling data collection from databases with CDC enabled. CDC is native functionality in SQL databases that tracks inserts, edits, deletes, and other table modifications server-side. Without CDC, modifications must be detected client-side which requires frequent extraction and comparisons of data sets from the database, a complex, limiting, and resource intensive process. As a result, many industrial data users settle and end up periodically replicating their databases. This also has limitations. The replication is less frequent than they would like, typically resulting in stale data long after events of interest occur.
With version 3.4, the Intelligence Hub’s inclusion of CDC support ensures that incremental changes to industrial datasets can be efficiently synchronized across technology platforms in near real time. The Intelligence Hub can query for only the changes to a database table, using this powerful tool for DataOps to unlock new possibilities for users. For example:
With version 3.4, the Intelligence Hub’s inclusion of CDC support ensures that incremental changes to industrial datasets can be efficiently synchronized across technology platforms in near real time. The Intelligence Hub can query for only the changes to a database table, using this powerful tool for DataOps to unlock new possibilities for users. For example:
- Querying for changes in a surface mount technology (SMT) machine’s internal database and publishing those as events to a Unified Namespace (UNS).
- Taking transactions from a PostgreSQL database embedded within a lab instrumentation application and replicating them to a Microsoft SQL Server database used for business intelligence.
- Monitoring the backend databases of many site-level MES applications and aggregating those changes in a data lake in the cloud for enterprise-wide visibility into manufacturing operations.
OPC UA: Structured Data + Alarms & Conditions
The OPC UA Client connection now supports browsing and subscribing to both structured data and Alarms & Conditions (A&C), enabling data collection and normalization for analysis in external systems. This functionality was specifically developed for the varied implementations of OPC UA A&C among modern PLC, SCADA, DCS solutions in the market. This is a powerful tool for unifying disparate alarm data in one place. Additionally, with structured data support, the Intelligence Hub can now gather UDTs within OPC UA servers and log them to SQL databases or even model-aware solutions such as AVEVA PI System, AWS IoT SiteWise, or MQTT Sparkplug B applications.
[NEW] Google BigQuery
For Intelligence Hub users that have adopted Google Cloud Platform (GCP) as their cloud services provider, Google BigQuery is a popular service. To increase the data engineering capabilities available to GCP users, the new connector allows users to create and update BigQuery tables based on Intelligence Hub model definitions. This connector allows users to interface with BigQuery data as easily as any other database.
AWS IoT SiteWise: Buffered Ingestion
In addition to support for streaming data into AWS IoT SiteWise, the Intelligence Hub now also supports AWS’ new buffered ingestion method. Introduced as a cost-efficient option for IoT SiteWise, buffered ingestion delivers large datasets by publishing in bulk to warm-tier storage via Amazon S3 and, within minutes, IoT SiteWise extracting the contents. Complementing streaming ingestion for near-real-time monitoring, buffered ingestion is ideal for analytical use cases that can tolerate some ingestion latency.
AVEVA PI System: PI Point Metadata & Point Browsing
The PI System connector in the Intelligence Hub now returns additional PI Point metadata attributes, including descriptor, point type, engineering units, etc. The new Point Browse capability can be used to browse and query PI Points directly from the PI Archive by their metadata; a useful feature for organizations that have not yet implemented PI Asset Framework.
Data Engineering
Pipelines has had a number of recent updates and improvements and has become a core data engineering tool of the Intelligence Hub. In version 3.4, we again return to Pipelines with a new stage, Switch, and improvements to the Write stages.
The Switch stage simplifies the configuration of Pipelines we commonly observe with complex conditional logic. Just as it sounds, the Switch stage adds switch statement logic to the growing array of Pipelines capabilities, introducing the ability to direct and divert data flowing through Pipelines according to conditions specified by users. The Switch stage is an ideal companion to the new success and failure paths in the Write and Write New stages. In the event of a failed write, users can easily handle and remediate the error using conditional logic.
The Switch stage simplifies the configuration of Pipelines we commonly observe with complex conditional logic. Just as it sounds, the Switch stage adds switch statement logic to the growing array of Pipelines capabilities, introducing the ability to direct and divert data flowing through Pipelines according to conditions specified by users. The Switch stage is an ideal companion to the new success and failure paths in the Write and Write New stages. In the event of a failed write, users can easily handle and remediate the error using conditional logic.
Data Governance
Secrets Management
As Intelligence Hub users continue to deploy more hubs with more connections, managing all the connection credentials and tokens to various target systems can become difficult.
To accelerate deployment and configuration across the industrial enterprise, Intelligence Hub version 3.4 introduces Secrets Management. Compromising on security has never been an option for Intelligence Hub users, but the diversity of connection credentials across different environments meant that what worked on one site or environment may not work at another, presenting a challenge for those who wished to share configuration or deploy with a DevOps approach. To address this, the Intelligence Hub now offers support for external secrets and includes a new configuration construct for securely referencing secrets, effectively decoupling them from project configuration.
This new approach is ideal when moving project configurations across environments with different connection credentials (e.g., DEV/QA/PROD, sites, areas) and is particularly valuable for enterprises that have adopted DevOps methodologies and deploy the Intelligence Hub using Infrastructure-as-Code (IaC).
To accelerate deployment and configuration across the industrial enterprise, Intelligence Hub version 3.4 introduces Secrets Management. Compromising on security has never been an option for Intelligence Hub users, but the diversity of connection credentials across different environments meant that what worked on one site or environment may not work at another, presenting a challenge for those who wished to share configuration or deploy with a DevOps approach. To address this, the Intelligence Hub now offers support for external secrets and includes a new configuration construct for securely referencing secrets, effectively decoupling them from project configuration.
This new approach is ideal when moving project configurations across environments with different connection credentials (e.g., DEV/QA/PROD, sites, areas) and is particularly valuable for enterprises that have adopted DevOps methodologies and deploy the Intelligence Hub using Infrastructure-as-Code (IaC).
Advanced Debugging
As Industrial DataOps use cases and deployments grow in sophistication, so does the need to effectively understand state and troubleshoot configuration of the Intelligence Hub. Additionally, this needs to be easily accessible for larger teams. Continuing a relentless focus on ease-of-use and observability, Intelligence Hub version 3.4 introduces advanced debugging. By enabling the Debug Logging option from the settings menu, users can capture in-depth system telemetry of the Intelligence Hub runtime. This is an invaluable tool for optimizing configuration as well as troubleshooting issues that are challenging to reproduce or test.
Wrap Up
As industrial organizations continue to digitally mature, their teams will need tools that are smarter and more sophisticated to continuously solve evolving interoperability problems and enable their digital transformation outcomes. The features available in HighByte Intelligence Hub version 3.4 further support manufacturers as well as energy, utilities, and logistics providers in their Industry 4.0 journey.
Beyond what you’ve read about here, many new features and capabilities are included in HighByte Intelligence Hub version 3.4. To learn more about the latest release, please check out these additional resources:
Beyond what you’ve read about here, many new features and capabilities are included in HighByte Intelligence Hub version 3.4. To learn more about the latest release, please check out these additional resources:
- Explore the release notes for details on all new features and fixes.
- Request a free trial or log in to your existing account to test and deploy the software in your unique environment.
