Did this work during my tenure at The Elite Group.
Coupled with the DataMart I implemented a framework that would automatically check the source data for data quality issues present. These issues were not causing errors in their respective source systems, but would cause issues on the EDI’s being developed.
The DataQualityControl database had a few initial requirements:
- Speed – It was in the critical path for loading the DataMart, and it needed to get out of the way as quickly as possible.
- Flexibility – Additional tests would need to be created over time, and be able to be integrated into DataMart as needed.
- Reporting – There are two types of data recorded. First was an aggregate for historical purposes, and second was a daily log showing that days issues. Thus trends could be displayed over time to show how the errors were being addressed system wide, and the latest example of such an error could be quickly displayed.
All of these were met in the initial pass, and over time additional quality checks were added as the need arose.