Articles

Hitting the target: confidence intervals for your Machine Learning models

3 min read

In this article, we explain what confidence intervals are in Machine Learning models and the great value they bring to business environments. We also include a video of our talk on this topic at PyCon Spain 2022.

Today, more and more companies rely on Machine Learning models and Advanced Analytics techniques to make their business decisions.

There is no doubt that data is the new oil of this era, and making decisions based on it provides a considerable competitive advantage. However, there are still scenarios (stock market fluctuations, energy and commodity prices, etc.) whose high complexity makes them difficult to model even with state-of-the-art algorithms.

In these situations, the most viable solution is usually to quantify the model’s uncertainty or assign confidence intervals to predictions in order to make more precise business decisions.

What confidence intervals are

If someone asked you how old a person is, what would you answer? You might say a specific number based on physical appearance, the environment, and comparing with others you know (28 years old), but the truth is you have no idea. Wouldn’t it be better to give an age range around your initial estimate to ensure your answer is correct? Between 26 and 30 years. This was the example we used in our talk to explain the concept of confidence intervals.

In Machine Learning, we could say that a confidence interval is a range of values within which a certain unknown value inferred by our predictive model is estimated to lie. This way, you ensure that the prediction provided is accurate.

As we mentioned in the talk, we find this approach very often in our daily lives: every time we order something from Amazon, Glovo, Cabify, or check a flight departure time. These companies estimate arrival times as a range whose width depends on the uncertainty of the estimate. An Amazon package or a Glovo dinner has relatively narrow delivery windows, while scenarios with higher uncertainty produce wider estimated ranges.

Providing a narrow delivery window is extremely valuable for customers, as it allows them to better organize themselves, ensure the order will arrive on time, and be home for the delivery. Consequently, it is also very beneficial for the company: customer experience improves, and failed deliveries decrease.

The great business value of confidence intervals

Predicting confidence intervals has great value in business, especially in strategic sectors such as banking or energy, where even small errors may lead to major losses.

During the talk we used the example of prediction models used by some companies for buying and selling electricity in the energy market. As you can see in the explanation, when estimating energy prices it is very different to provide a single value versus providing the full distribution of estimated prices. A single estimate of 120€ may belong to intervals of very different sizes: (100€, 140€) or (119€, 121€). If we only consider the single prediction, we cannot measure the uncertainty, an essential variable in sectors where decisions are made directly based on model outputs. Strategies such as “playing it safe” by acting only on low-uncertainty predictions to minimize losses are common, which makes predictive intervals absolutely necessary.

Take a look at the video where we dive deeper into these use cases, explain the different approaches to “hitting the target” in ML models (confidence interval, prediction interval, and quartile-based predictions), and describe the advantages and disadvantages of each.

“Hitting the Target: Confidence Intervals for Your Machine Learning Models” Ángel Luengo and Guillermo Delgado at PyCon Spain 2022.

CONTACT US

Our experts can help you

Tell us the constraint you are working against: the regulator, the core system, the data, and we will tell you where we would start.