Poor-quality data in product catalogs, purchased items, equipment, and materials is a common problem for industrial and commercial enterprises. Over the years, a company's inventory catalog typically accumulates a large number of outdated, duplicate, and poorly populated records. This hinders high-quality analytics and interferes with inventory planning and procurement. In such cases, normalization, or cleansing, of the inventory catalog is necessary, which can be performed either as part of a master data management (MDM) implementation project or as a standalone service.

Building a Catalog Hierarchy. Attributes and Templates

It's necessary to begin by creating a reference structure for the inventory catalog. To achieve this, a modeling methodology is developed: the number of catalog levels is specified and the basis for classifying products at each level is described. Products can be categorized by purpose, method of use, applicability in specific business processes, and functional and design characteristics. A consistent and concise hierarchy will help quickly find the desired catalog items and reduce the likelihood of cluttering it with duplicate and incorrectly classified items. Our experts typically manually create one or two top levels of the hierarchy, while lower-level classes are identified using ML models. Clusters (groups) of products that the ML model considers similar are constructed. A specific algorithm then rebalances the clusters until the specified values ​​of semantic similarity of products within each group are achieved. An expert processes the resulting results: confirms the class identification, merges or separates some of them, and assigns them to upper-level classes. Both the methodology and classification structure must be agreed upon with the Customer's business units.

Then, an attribute composition is formed for each classifier branch: the properties that products in certain groups may possess are determined. Based on these sets of properties, templates for standard product names for each class are created.

If the Customer has a ready-made inventory catalog hierarchy, the task is simplified—they can immediately proceed to processing product data.

Normalization of Product Names and Characteristics

Inventory catalogs in accounting systems often do not store product property values, or only a small set of them are stored. Meanwhile, having property values ​​specific to products of different classes (such as Power, Volume, Voltage, etc.) stored as numeric or string values ​​in separate fields is important for searching for analogs, interchangeable components, duplicates, and solving analytical problems. Product property values ​​can often be extracted from the product name or found in open sources such as supplier catalogs. During normalization, we process product names using AI models (without transferring them to cloud services) to identify as many characteristics of each product as possible.

Furthermore, a standard name and additional non-standardized characteristics are determined for each product. The names and values ​​of all characteristics are automatically cleared of unnecessary characters, converted to a single case, and checked for alternating Latin and Cyrillic characters (for example, the English letters "a" or "o" may be overlooked in a Cyrillic product name). Based on the cleaned values, a full standard product name is then generated in accordance with the naming template defined for the class to which it belongs.

Data Enrichment

Data often needs to be supplemented with missing information: determining the class for each item using external classifiers (such as TN VED or OKTRU), linking it to the National Catalogue Identification Number (NTIN) or international catalog codes. We solve such problems using a combination of AI models and expert-verified rules. This significantly reduces the labor intensity of data enrichment, ensuring high reliability of the results.

Finding and eliminating duplicates

After the data is normalized, automatic rules are used to search for duplicates—item catalog records that describe the same product item. One reference record is generated from each group of duplicates, and, if necessary, the product property values ​​from multiple cards are merged. Typically, only one reference product card remains active in the accounting system for each group of duplicates, while all others are marked as inactive.

In some cases, duplicate searches are performed using fuzzy rules: in this case, an expert or AI model reviews each group of duplicates and determines whether the found items are actually duplicates or simply describe similar products (analogs, replacements). In the latter case, links can be created between product cards to describe the nature of their relationships.

Normalization Result

The result of normalization is a cleaned inventory catalog, where all products are assigned to the appropriate hierarchical groups according to clear and defined classification criteria, their property values ​​are highlighted, standard inventory names are generated using templates, and links to external data sources are established.

To further maintain the catalog's relevance, we recommend maintaining it in an MDM master data management system. MDM provides tools for processing new and modified product approvals, automatically classifies and normalizes new catalog items, distributes up-to-date master data to all accounting and analytical systems, links records for the same product across different sources, and enables centralized management of auxiliary directories and classifiers.

As a result, businesses receive high-quality inventory data necessary for building end-to-end analytics and supporting decision-making in business processes such as inventory management, procurement and sales planning and execution, repairs, and maintenance.

Our company has experience normalizing complex inventory catalogs for large industrial enterprises. Contact us if your company faces a similar challenge – we will offer solutions.