Re-Use MDM Best Practices for Big Data
One of our group members on Master Data Central raised the question about using our automated data standardization, normalization, attribution, rationalization and enrichment processes with transactional data – specifically large volumes of variable length of messages between systems.
The challenge – the data in the messages had relatively well defined formats, (about 400 different formats) yet had variable content length for any given format.
Seems like a job for a rules processing engine, but the catch? New message formats are being identified monthly – at a rate of 20 per month.
Ok – blatant self-promotion warning – using our AI based approach to pattern identification our Harmonize® SaaS platform allows people to normalize and standardize information without writing rules. They find the pattern, they break it into pieces and then Harmonize finds similar information records, messages, or packets, (anything that can be represented in bits and bytes) and applies smart filtering to apply the same standardization, attribute/characteristic extraction, data normalization, and data rationalization (aggregation or unique-ing) to the data sets.
In general though, the best practices associated with the data cleansing and enrichment associated with many forms of master data can be well leveraged for big data challenges like high performance analysis and reporting, near real-time data enrichment.
Using Artificial Intelligence based componentry like fuzzy logic for match and search, or proximity and frequency algorithms for trending and predictive forecasting are just the jumping off point.
If you have successfully re-used best practices from your master data management projects in the transactional, big data or theoretical physics – let us know…
Blog: Strengthening your E-BOM with MDM
E-Book: Sailing smooth through Data Vortex – Master data management
Latest posts by Arthur Raguette (see all)
- The need for effective Material Master Data Management – Part 2 - January 16, 2018
- The need for effective Material Master Data Management - January 4, 2018
- Material Master Data Governance Teams - December 26, 2017