Ask HN: Data quality monitoring?
https://news.ycombinator.com/item?id=9801552We have a data quality problem.
We run a large analytics and real time decisioning pipeline operating on thousands of different parameters per event. We source this data from many different source both inside and outside of our company. For some events, some parameters are optional vs required, for other events it's a different set. The mix of parameters for each event changes over time but not frequently.
Is there anything out there that can monitor the expected set of parameters for each event, including expected range for each parameters, and alert when either the parameter is missing or out of bounds?
I do not want to define this manually but would prefer something that includes machine learning and statistical probability.