Data quality management features in the DataVera EKG Platform include format-logical control and data normalization.

Format-logical control and integrity control capabilities

Format-logical control and integrity control rules are formulated in the model using SHACL constraints. These rules are stored in the platform data and can be edited through the EKG Explorer interface. Each rule describes specific conditions that objects of a given class must meet. Rule conditions can:

  • Compare the property values ​​of a single object or several related objects. For example, you can check that a person's date of birth or a document's issue date is within a certain range.
  • Perform operations on string values ​​– extract and concatenate parts of strings, and compare them. For example, the first six characters of the IIN must match the year, month, and day of birth.
  • Use regular expressions to check for pattern matching. For example, you can check that the IIN consists of 12 digits.
  • Use built-in platform functions to perform specialized checks. One such function can check that the seventh character of the IIN matches a person's gender and century of birth.
  • Check for duplicate objects by matching certain properties. For example, a rule can check that the system does not contain another person with the same IIN, or another document with the same number and date.

Using Data Quality Control Tools

Rules are applied automatically when objects are created or edited. You can also manually apply rules to a group of objects. In the EKG Explorer interface, table cells whose values ​​violate rule conditions are highlighted in red. Hovering over such a cell displays the error message. Violations can also be viewed in the object card.

Above the list of objects in EKG Explorer, there's a data quality monitor that displays information about the number of objects with and without violations. Clicking this indicator filters the list to show only those objects that violate the rule conditions. In this mode, a data quality engineer can review all objects requiring attention and resolve the errors.

To verify that control relations are met, you can use the SHACL function mechanism. It allows you to define arithmetic functions whose results can be used within constraints. For example, suppose you want to verify that the price of a product with VAT equals the price without VAT plus 12%. First, define a function that will increase the value of the argument passed to it by 12%. Then, use this function in the rule. To do this, bind the function's result to a variable that is compared to the value of one of the data object's properties.

Normalization and Automatic Data Cleansing

Data normalization functions are also defined in the model and are applied when an object is created or edited. Each function describes a rule by which a normalized property value is obtained from the original value. For example, a normalization function can use a regular expression to remove spaces, retain only digits in a phone number, or separate comma-separated values ​​in a single string. The result of the normalization function is written to the value of another property that cannot be edited directly. For example, a request to the platform specifies a phone number in the "Phone" property, and the normalization result is stored in the "Normalized Phone" property.

Using the DataVera EKG Platform tools discussed above, an organization can establish business processes for continuously monitoring and improving data quality.