Data quality management features in the DataVera EKG Platform include format-logical control and data normalization.
Format-logical control and integrity control rules are formulated in the model using SHACL constraints. These rules are stored in the platform data and can be edited through the EKG Explorer interface. Each rule describes specific conditions that objects of a given class must meet. Rule conditions can:
Rules are applied automatically when objects are created or edited. You can also manually apply rules to a group of objects. In the EKG Explorer interface, table cells whose values violate rule conditions are highlighted in red. Hovering over such a cell displays the error message. Violations can also be viewed in the object card.
Above the list of objects in EKG Explorer, there's a data quality monitor that displays information about the number of objects with and without violations. Clicking this indicator filters the list to show only those objects that violate the rule conditions. In this mode, a data quality engineer can review all objects requiring attention and resolve the errors.
To verify that control relations are met, you can use the SHACL function mechanism. It allows you to define arithmetic functions whose results can be used within constraints. For example, suppose you want to verify that the price of a product with VAT equals the price without VAT plus 12%. First, define a function that will increase the value of the argument passed to it by 12%. Then, use this function in the rule. To do this, bind the function's result to a variable that is compared to the value of one of the data object's properties.
Data normalization functions are also defined in the model and are applied when an object is created or edited. Each function describes a rule by which a normalized property value is obtained from the original value. For example, a normalization function can use a regular expression to remove spaces, retain only digits in a phone number, or separate comma-separated values in a single string. The result of the normalization function is written to the value of another property that cannot be edited directly. For example, a request to the platform specifies a phone number in the "Phone" property, and the normalization result is stored in the "Normalized Phone" property.
Using the DataVera EKG Platform tools discussed above, an organization can establish business processes for continuously monitoring and improving data quality.