Top Ensemble Techniques in Machine Learning

An all-encompassing meta-approach to machine learning called ensemble learning aims to improve predictive performance by pooling predictions from many models.

There are three techniques that rule the world of ensemble learning, even though there are about an infinite amount of ensembles you can create for your predictive modelling issue. So much so that it is a topic of study that has given rise to numerous specialised approaches rather than algorithms per see.

 

Why Use Ensemble Learning?

 

The accuracy of predictions is increased by lowering the variance component of the prediction error. In order to achieve higher prediction performance, such as a lower error on regression or high classification accuracy, we can explicitly apply ensemble learning.

 

Ensemble Modelling Techniques

 

Below are the popular ensemble modelling techniques in machine learning:

 

Boosting Technique

 

Here multiple models are added sequentially. The errors in model one are corrected by model two, etc. Each model corrects consequently and we may have to stop overfitting later. In AdaBoost, a weighted dataset is given. Here the emphasis is more on the data where the models went wrong rather than the correct ones.  

In this boosting, there is only one node decision tree, and it is called a decision stump. As an extension of AdaBoost, loss functions are also added to minimize overfitting and error. XGBoost and LightGradientBoost are two more methods that involve sequential boosting plus the loss functions.

 

Bagging Technique 

 

Here samples(rows or also called ensembles) randomly are given as inputs to multiple decision trees. The samples that are given to the dataset are also returned to the original dataset. This is called replacement or bootstrapping.  The final prediction of all those decision trees is combined. Using statistical techniques such as averaging or voting, a final decision is made.

As an extension Random forest of the ensemble technique, here bootstrapping happens with features also. That is to say that some features + ensembles (rows) are taken for the first tree, then it is returned to the original data set and another set of features (may include the features that were selected first, but not always) are again sent to the second tree. Likewise, the process is repeated for n trees. Again the final decision is made by average or voting of output from all trees. In both these models, the models work in parallel.

 

Voting Ensembles

Here we take a majority rule on predictions from multiple models. For the same dataset, multiple models are trained, and then a prediction or classification is made. The class or the prediction with maximum votes is the output.  This is also termed hard voting. Soft voting happens in classification problems, where each model gives a probability value for the various classes. The label with the largest sum of all probabilities is the final output.

 

Stacking Ensembles

 

Here a new ML algorithm is set up for which the input is nothing but the output of various ML algorithms. This could be linear regression in the case of prediction or logistic regression in the case of classification. Although it is not a hard and fast rule to use the same.

The output of various ML algorithms in an ensemble is fed as input to another ML algorithm which, in turn, makes a decision.

To sum it all up, Bagging is the process of fitting numerous decision trees to various samples of the same dataset and averaging the resulting predictions. When numerous distinct model types are fitted to the same data, stacking is used to learn how to combine the predictions most effectively. A weighted average of the predictions is produced by boosting, which entails adding ensemble members in a sequential manner that corrects the predictions provided by earlier models.