The Task of Normalizing the Product Catalog

A poorly structured, duplicate-stuffed product catalog is a common problem among manufacturing companies. A lack of high-quality data on equipment, spare parts, and materials can lead to errors in inventory management, the selection of suboptimal supplier offers, and the inability to build analytics for management decision-making.

The master data management system addresses the challenges of creating a master product catalog and exchanging product information between corporate systems.

The artificial intelligence tools in the DataVera EKG Platform automate the tasks of cleaning and structuring the product catalog.

Grouping and creating classes

The first phase of processing involves dividing products into groups using machine learning tools. This is a challenging task: we need products to be placed into distinct groups, even if their names contain many overlapping words. Thus, a lamp and a lamp bulb should be placed in different categories, despite sharing a common name.

We achieve this using a special algorithm for calculating the semantic similarity of names.

The next step is to form product classes based on product groups. At this stage, we once again evaluate the similarity of all products in each group, merge or split some of them, and move some products to more appropriate categories.

To complete the class hierarchy, we add superclasses for the resulting groups. The class tree is ready – our catalog is now organized!

Defining Product Properties

Next, we need to determine which properties can be extracted from product labeling. These could be physical characteristics, such as current or voltage, or qualitative descriptions, such as color or design variant. The algorithm builds a general list of properties and then determines which properties products in each class may have. At this stage, we obtain a complete conceptual model of our catalog, which contains descriptions of product groups and their sets of properties.

Next, we iterate through the entire list of products to determine property values ​​for each. Values ​​can be extracted from both the text properties of a product, such as its label or description, and from external documents, such as a product data sheet or questionnaire. At this stage, the percentage of property values ​​filled and other data quality metrics are also determined.

For each product class, you can set up a custom template for generating its name or label. Standard names and labels can include property values ​​or codes. The automatic label generation feature helps ensure consistency in the designations of all products in certain categories.

Finding Duplicate Items and Using the Catalog

The final processing stage is searching for possible duplicates. A fuzzy search algorithm determines which catalog items may be equivalent to each other. Duplicates can arise due to typos, abbreviations in product names, and the inclusion or omission of values ​​for certain characteristics in the labeling.

The prepared catalog is saved to the MDM system. Now the data steward's job is to verify its correctness and confirm or cancel the merger of duplicates.

The catalog is ready! Now the MDM can synchronize it with corporate systems that handle inventory control, purchasing, and repair and maintenance management. The MDM will save each product code from each system to preserve the data structure, if necessary. When new product items appear, they can be entered in one system and distributed to other applications using the MDM, or they can be entered into the MDM centrally.

Once the task of creating a reference catalog is completed, you can move on to the next steps of business process automation using AI. You can set up automated processing of technical procurement specifications, verification of compliance with supplier proposals, searching for similar products, and extracting product information from manufacturer catalogs.