Data modeling deep dive: Explicit vs implicit approaches

Data contextualization and standardization has always been a cornerstone and differentiator of HighByte Intelligence Hub. It’s our reason for being. The secret sauce of contextualization and standardization at scale is using data models. Over time, customer use cases and data modeling requirements have evolved, and so the Intelligence Hub has evolved in kind. This post dives deep into today’s data modeling capabilities and explains multiple methodologies for modeling data in the Intelligence Hub. If you’re a data engineer—or really anyone using the Intelligence Hub—this post is for you.

Background on Data Modeling Maturity 

As the industry digitally matures, more data is moving between systems, and more systems are consuming structured data. Data models are essentially groupings and contextualization of related pieces of data. By grouping data, the relationships are clear. By adding context, the consuming system can differentiate different data sets. And by creating these consolidated objects, the management and communication burden is reduced. Data-driven enterprises are increasingly modeling data depicting operations, products, and processes as curated data structures rather than primitive “tags” from industrial control systems.

While the Intelligence Hub has long supported data point modeling, Intelligence Hub users have become more sophisticated in their use of data structures. They are not just grouping together primitive data points. They want to:

• nest them or compose them into complex, inheritable, and extendable data structures;
• blend models to strategically normalize their disparate data sources while also accounting for the differences between them;
• implicitly model incoming data structures from data sources with a rules-based approach of validation and remediation.

With the release of version 4.0, the Intelligence Hub’s modeling engine received major enhancements to make it easier to create and manage more complex models. Let’s take a deeper look at both explicit and implicit modeling in the Intelligence Hub.

Explicit Data Modeling  

Our focus on contextualizing and standardizing industrial data at scale became even clearer with the launch of version 4.0 in October 2024. Now, you can:

• Add a description to each model attribute and define default attribute values within a model rather than within each instance. This is a simple but invaluable tool for larger teams managing many models and their use across the enterprise. 
• Visually depict models in an attribute tree. From a single compact view, you can visually intuit the shape of their model as you build it. You can efficiently add, remove, reorder, rename, and manage datatypes of the model attributes and leverage attribute type iconography to comprehend and interrogate large, complex model definitions in a clean, condensed view.
• Use the new initialization block to sequence complex parameterized input reads across attributes within an Instance.
• Nest model attributes to create depth within a payload. Models can reference other models to extend their definition with greater flexibility and scale. You have the flexibility of driving payload depth inherently from within a model or from an external model. For example, a use case involving inspection data may need to contain inspection results as well as metadata about the inspection process and what was inspected. The results could be child attributes under a parent “results” attribute while the metadata could come from another model definition used to describe processes and products. Modeling provides the flexibility to coalesce data structures and models together.

The latest release of HighByte Intelligence Hub gives you more granular control over complex and hierarchal data structures. Use Models and Instances to explicitly create structured data sets from data points. Now let’s look at working with inbound structured data by implicitly modeling in the Intelligence Hub. 

Implicit Data Modeling 

A bifurcation in approaches to data modeling has emerged.  

As discussed above, the first approach involves explicitly modeling incoming data points into consistent data sets. This aligns with the problem of organizing inconsistent OPC UA tags into a data set. Since their inception, this is largely what Models and Instances have addressed. From the drag-and-drop reference pane to templating to expressions and functions, Instances are very powerful to facilitate this approach to modeling.

The second approach involves implicitly modeling incoming data structures by validating and tailoring them to the needs of an external system or use case. This includes appending, transforming, or ensuring consistency of already-structured data. Intelligence Hub users have increasingly leaned on Pipelines and its Transform stage for this implicit modeling approach. The Transform stage requires JavaScript expressions to manipulate incoming data into a consistent shape. This allowed data leaving the pipeline to meet model definitions or other data standards.

To holistically support both explicit and implicit approaches, HighByte Intelligence Hub version 4.0 expanded its modeling capabilities with the introduction of Pipeline modeling. Pipeline modeling includes two new Pipeline stages: Model and Model Validation. The Model and Model Validation stages can take incoming data payloads within a Pipeline and validate them against a model definition without the use of JavaScript. 

• The Model stage defines logic for associating incoming payloads with a model. This allows you to use a model definition to reshape incoming data structures. It works by mapping (and transforming) preceding event and metadata contents into model attributes. 

• The Model Validation stage compares incoming payloads against model definitions. You can configure the Model Validation stage to ensure event values match (or partially match) one or more model definitions. And with its “valid” and “invalid” egress paths, the new stage is a powerful tool for enforcing data standards and facilitating remediation.  

For example, suppose there were multiple AGV assets that published their data as a Webhook or MQTT message. While the payload from the AGV was structured and somewhat standardized, it was not in the format and naming the business intelligence team needs. The new Pipeline Modeling stages could be used to reshape payloads and validate them. This is a far more productive way to enforce data standards than breaking up the AGV data sets into individual attributes and explicitly associating all of them with instances of a model.

Complementing the Intelligence Hub’s explicit modeling capabilities to form data structures, these new features enable you to implicitly enforce modeling standards on already-structured data with ease.

Conclusion  

HighByte Intelligence Hub enables industrial enterprises to define models and put them to use both explicitly and implicitly. Whether its explicitly associating individual tags from an OPC server to asset models or ensuring already-structured data from many coordinate-measurement machine (CMM) inspection programs adhere to data contextualization requirements, the Intelligence Hub can help. The table below offers a summary of what we’ve discussed in this post.

Beyond what you’ve read here, many new features and capabilities were included in the latest release of the Intelligence Hub. To learn more, please check out these additional resources:

• Read the product launch post. 

• Explore the release notes for details on all new features and fixes. 

• Request a free trial or log in to your existing account to test and deploy the software in your unique environment.