MDMAI
Home › MDM Guides › Data Quality
Data Quality

Master Data Quality Profiling: Metrics, Validation Rules, and Anomaly Detection

Build quantitative data quality scorecards using ISO/IEC 25012 dimensions, preventive validation gates, and drift alerts before bad records pollute golden tables.

Why Profiling Must Precede Matching

Poor inputs produce false matches. When an ERP encodes unknown phones as '999-999-9999' or blank birthdates as '1900-01-01', naive matching treats those dummy tokens as shared identity evidence.

Automated profiling scans incoming feeds before matching runs—exposing null rates, dummy placeholders, format inconsistencies, and unexpected key cardinality.

  • Frequency scans expose disguised nulls like 'N/A', 'TBD', or '000000000'.
  • Pattern checks detect inconsistent date orders and unparsed country codes.
  • Cardinality alerts flag supposedly unique keys repeating across unrelated rows.

Six Core Master Data Quality Dimensions (ISO/IEC 25012)

ISO/IEC 25012 standardizes quality measurement across Completeness, Validity, Uniqueness, Consistency, Accuracy, and Timeliness so business and engineering teams share objective metrics.

Tracking dimensions separately pinpoints whether trust drops stem from stale sync lag, missing tax keys, or format drift.

  • Completeness: Share of active records holding populated required attributes.
  • Validity: Share passing regex, ISO code lists, and check-digit rules.
  • Uniqueness: Ratio of golden entities to raw contributing source records.
  • Timeliness & Consistency: Adherence to sync SLAs and cross-system parity.

Preventive Gates vs. Statistical Anomaly Alerts

Resilient pipelines pair preventive ingestion gates with statistical anomaly detection. Preventive gates quarantine malformed keys before they overwrite golden attributes, returning diagnostics to source owners.

Detective monitors watch rolling baselines and trigger alerts whenever daily null rates, match rates, or batch volumes shift abruptly.

  • Hard Quarantine: Holds records failing legal check-digits or mandatory keys.
  • Soft Scoring: Admits partial records while tagging low-trust flags downstream.
  • Drift Alerts: Fires when daily null or validity metrics deviate >3 sigma.

Publishing Source-Level Quality Scorecards

Quality improves sustainably when upstream application owners see their feed scores. Domain scorecards turn abstract hygiene into clear operational SLAs across sales, finance, and supply chain.

ISO/IEC 25012 Master Data Quality Scorecard Framework

Quality DimensionMetric FormulaTarget ThresholdRemediation Action
CompletenessValid populated required / Total>= 98.0% core keysEnforce required intake fields
ValidityPassing format & check-digit / Total>= 99.5% identifiersQuarantine malformed IDs at gate
Uniqueness1 - (Duplicate clusters / Source rows)< 1.5% residual dupesRun AI matching & merge sweep
ConsistencyCross-system key parity / Linked>= 97.0% sync parityDispatch reconciliation webhook
TimelinessUpdates within SLA / Total updates>= 99.0% < 5 minScale CDC workers or alert lag

Reference scorecard linking ISO/IEC 25012 dimensions to concrete operational thresholds.

Worked Illustration: Catching Silent Upstream Schema Drift

This worked illustration shows profiling and drift gates blocking corrupt CRM updates.

  1. Step 1 (Worked Illustration — Baseline): Over 30 days, CRM imports average 98.4% valid 2-letter ISO-3166 country codes.
  2. Step 2 (Worked Illustration — Drift): A Tuesday CRM form update writes full names ('United States') into the 2-char column across 4,200 rows, dropping Validity to 61.0%.
  3. Step 3 (Worked Illustration — Quarantine): Preventive validation quarantines the 1,638 invalid rows while leaving existing golden country codes untouched.
  4. Step 4 (Worked Illustration — Replay): Engineers add a normalization map ('United States' -> 'US') and replay the batch cleanly within 2 hours.

Worked Illustration Summary: Preventive validation and drift alerts isolate upstream UI changes before corrupt data reaches ERP and billing.

Data Quality Profiling & Validation Checklist

  • Profile every new source feed for dummy placeholder tokens prior to matching.
  • Maintain a blocklist of dummy values ('TBD', '999999999') excluded from survivorship.
  • Enforce syntax, ISO code-list, and check-digit gates on critical identifiers.
  • Configure rolling anomaly alerts on null spikes, match-rate drops, and batch volume.
  • Quarantine failed rows non-destructively with actionable payloads for replay.
  • Review source-by-source ISO/IEC 25012 scorecards monthly with domain owners.

Frequently Asked Questions

How does data profiling differ from data cleansing?

Profiling analyzes datasets to uncover nulls, format drift, and anomalies. Cleansing applies standardization and validation rules based on those findings.

Should malformed incoming records be dropped or quarantined?

Quarantine them so raw payloads remain recoverable for upstream correction and replay without losing business transactions.

How do dummy placeholders hurt AI matching?

If hundreds of rows share '000-000-0000', naive matching treats that phone as a strong link signal, creating false candidates and erroneous merges.

Which ISO standard governs data quality dimensions?

ISO/IEC 25012 defines the core data quality model including completeness, accuracy, consistency, uniqueness, validity, and timeliness.

Primary Standards & Regulatory References

  • ISO/IEC 25012:2008 — Software engineering: Software product Quality Requirements and Evaluation (SQuaRE) — Data quality model.
  • ISO 8000-61 — Data quality: Data quality management: Process reference model.
  • ISO 3166-1 & ISO 4217 — Standard Country and Currency Codes for Reference Data Validation.

Related MDM Technical Guides