You just found out that the quality of your analysis data is poor — What are your options?

10 min readMar 31, 2022

You find out that the data quality of your analysis data does not fulfill the defined requirements. What should you do? Close the eyes and walk through? Stop the project and wait until things get better? Can analytics help you to overcome these obstacles?

This articles discusses your options and illustrates examples.

In a perfect world

First take a look at the optimal flow of the analysis process. You start with a business question, define and check the data quality criteria, you find out that the quality of your analysis data is good and your perform the analysis.

Press enter or click to view image in full size

Note that the chart does not include all necessary steps for the analysis. For example data preparation, data exploration/data understanding and also model validation, model ops and model governance have been left out for more clarity. The goal here is to focus on the important aspects for data quality.

1. The business question — “Can you calculate for me …”

Usually the journey starts with a business question. Consider the following two examples for business questions:

Demand Forecasting and Planning: Provide a forecast for the weekly demand pattern of my consumers for the next 26 weeks. Detail this forecast not only on a global level, but also on a country level. Also update the forecast for short term planning on a weekly basis every Monday 7am.
Customer Retention: Predict the cancellation risk of my business customers in telecommunication business on a monthly basis and provide a list of those with high cancellation risk to be included in a retention campaign.

2. Define the data quality criteria — “Let me first check …”

In the next step you are going to define the data quality criteria which your analysis data should fulfill. Such criteria for example can include data availability, data completeness, data correctness, or data quantity. See more details in the article “Is your data ready for data science? — Motivating this topic from a sail-race-analysis example”

For the demand planning example this could be requirements like

Having a demand history of >52 weeks
Having history data also on a country level, not only on a global level
Being able to have the actual demand data available in the system within 3 hours

For the customer retention example criteria like

Having a sufficient amount of cancelations (e.g. 250 cases) to be able to train a supervised machine learning model
Data Completeness with less then 10% of missing values

3. Data Quality Profiling — “Here we see that …”

In the profiling step you check the data quality status of your data. This can include descriptive statistics like counting the number of missing values, checking the available demand history across product groups, checking the availability of historic data on the appropriate detail level in your data.

4. Performing the analysis — “That’s why we (data scientists) are here …”

After you find out that the data quality status of your data is fine, you can start with the analysis. For example training an supervised machine learning model with a binary target “Cancellation yes/no” on the customer data or applying timeseries forecasting techniques like ARIMA, exponential smoothing, Unobserved Components Models (UCM), or recurring neural networks on the historic weekly demand data.

In real world

In many cases the above mentioned flow it too optimistic. In practice you often find out that your data does not fit to the predefined requirements. This should however not kill your project idea. It is important to be able to assess the situation and decide what your options are and how you can proceed.

Some of the options are displayed in the diagram below, they can be grouped in two major categories:

Changing the parameters or the status of the analysis project (green arrows)
Moving forward with the actual data quality status (red arrows)

1. Changing the parameters or the status of the analysis project

They include re-collecting or re-measuring the data for example to improve the correctness or completeness of the data. In the customer retention example you might find out that the data from the customer care system are not available for all customer segments. You have to go back to the IT department to make sure that they are made available for your analysis and the periodic application of your predictive model.
In some cases you need to re-formulate your business question, e.g. you might be able to run the demand forecasting analysis on a global level but you can’t detail the analysis (at the moment) on a country level.
If you however find out that some of your major requirements are not fulfilled at all, you might need to cancel or postpone your project. If you did not collect historic cancellation events of your customers you won’t be able to feed your machine learning model with labeled data and might have to postpone the analysis to a later point in time, when you collected a sufficient amount of data.

2. Moving forward with the actual data quality status

In some cases you find out that the data quality cannot be improved retrospectively. E.g. customers with missing values in a survey cannot not be contacted again to provide full data. So you might lower the data quality threshold and accept a larger proportion of missing values or a smaller number of events or a short time history in demand forecasting.
You can also use analytics to improve the data quality of your analysis data. For example you can impute missing values for example by using decision trees. You can use analytics methods to detect and flag outliers in your data. And you can also handle (small) data quantity for example by applying special methods for the analysis of rare events.

Conclusion

This overview has shown you the typical options you have when you find out that your data quality does not fit to your requirements. You can use the flow presented in the above diagram to discuss with your peers in data science, the business department or the IT department different options. The flow also allows you to highlight the importance of good data quality for analysis because it illustrates the actions that need to be taken, if data quality is bad or the situation you need to accept when you use the data for the analysis.

The next section provides more details on selected options as present above.

Selected details on the options presented above

Lowering the threshold and accepting the data for the analysis

In the context of the book “Data Quality for Analytics” I have performed a number of simulation studies to assess the effect of lower data quality on the model accuracy in supervised machine learning and in time series forecasting. The findings of these simulation studies have been published in three medium.com articles:

Article “Quantifying the Effect of Missing Values on Model Accuracy in Supervised Machine Learning Models” provides you the following insight and allow you to (roughly) quantify the decrease in model accuracy.

summary of 4 different simulation scenarios, hit rate per missing value percentage

Random missing values in training data only have limited effect (red line)
Missing values that occur also in the scoring data have a larger effect (orange vs. red, light green vs. green)
Systematic missing values have a much larger effect (green line)

Article “The effect of (bad) Data Quality on Model Accuracy in Supervised Machine Learning — Results from Simulation Studies” illustrates the loss in model accuracy if for example major input variables (like customer age) are not available in the analysis data.

Accepting a shorter data history in time series forecasting

The article “Determining the best length of the history of your timeseries data for timeseries forecasting” shows results that confirm to intuition that, on average, with increasing data quantity in terms of available time history, the quality of the forecasting model increases.

It can also be seen that the availability of the two first full years indicates a strong contribution to model quality. Additional months still provide quality improvement, which is, however, relatively lower. Also, this finding is intuitive: the marginal effect on model quality is higher when only a few months are available for analysis.

Improving data quality with analytics

The following points offer an overview of the typical features of analytics that are relevant for data quality.

Outlier Detection

Simple profiling of univariate data: Analytics play an important role in detecting outliers based on statistical measures like standard deviations or quantiles.
Outlier detection with methods of cluster analysis and distance metrics. These methods allow you to identify outliers in the data from a multivariate viewpoint.
Individual outlier detection with predictive models and time series methods. These methods allow you to calculate validation limits and optimal correction values on an individual basis. As an overall average might introduce unwanted bias into the analysis, a within-group average might be a better choice for replacement. See a youtube webinar with an example analysis.
Analytics not only provide methods for profiling and identifying outliers and non-plausible values, they also can provide a suggestion for a most probable value that should be entered here.

Missing Value Imputation

Analytics can deliver replacement values for missing values. For example missing values in one-row-per-subject data marts and time series data marts can be imputed.
These imputation methods range from average-based imputation values to analysis subject individual imputation values, which are based on analytic methods like decision trees or spline interpolations for time series.

Data standardization and de-duplication

Identification and elimination of duplicates in a database where no unique key for the analysis subjects is available can be based on statistical methods that describe the similarity between records.
These methods provide a measure of the closeness and similarity between records that is based on information like addresses, names, phone numbers, and account numbers, among others.

Handling data quantity

Analytics allow you to plan the optimal number of observations for a controlled experiment with sample size and power calculation methods.
In the case of small samples or small numbers of events in predictive modeling, methods for modeling rare events are provided.
In the case of time series forecasting, so-called intermittent demand models are provided that model time series with only occasional non-zero quantities.

Analytic transformation of input variables

Analytical methods are used to transform variables to a distribution that is suited for the respective analysis method. Log and square root transformations are, for example, used to transfer right-skewed data to a normal distribution.
For variables with many categories, analytics provide methods to combine categories. Here, the combination logic for these categories depends on the number of observations in each category and the relationship to the target variables. Examples of these methods include decision trees or weight of evidence calculations.
Text mining allows you to convert free-form text into structured information that can then be processed by analytical methods.

Variable Selection

Various methods for variable selection allow you to identify the subset of variables that have a strong relationship with the target variable in predictive modeling. These methods include simple metrics like R-square and metrics like LARS, LASSO.
Many analytical methods allow different methods of variable selection in the respective analysis model itself. Consider, for example, the forward, backward, and stepwise model selection in regression.

Defining and reformulating a business question

Consider the business question as introduced above: Can you build a machine learning model that predicts the cancellation risk of our customers?

There are many aspects that need to be detailed before the analysis can be run which have a large impact on data preparation and data quality. For example: What do you mean by “cancellation”?

Do you mean the full cancellation of the product or a decline in usage?
Do you want to include customers that have canceled the product but have started to use a more (or less) advanced product?
Do you also want to consider customers who did not cancel themselves but were canceled by our company?
What is the business process for contacting customers?
What additional attributes and explanations do you need?
Which latency period should we consider between the availability of the scores and the execution of the marketing campaign?

In my webinar “Conceptual Considerations when Preparing Data” these aspects have been discussed in more detail.

Mlearning.ai Submission Suggestions

How to become a writer on Mlearning.ai

medium.com