Project 5

Q.Identify the working principle of all 5 GBM and list down its advantages and disadvantages?

AdaBoost
AdaBoost also called Adaptive Boosting is a technique in Machine Learning used as an Ensemble Method. The most common algorithm used with AdaBoost is decision trees with one level that means with Decision trees with only 1 split. These trees are also called Decision Stumps.
Advantages and Disadvantages
Adaboost is less prone to overfitting as the input parameters are not jointly optimized. The accuracy of weak classifiers can be improved by using Adaboost. Nowadays, Adaboost is being used to classify text and images rather than binary classification problems.
The main disadvantage of Adaboost is that it needs a quality dataset. Noisy data and outliers have to be avoided before adopting an Adaboost algorithm.

Gradient Boosting
Gradient boosting is a type of machine learning boosting. It relies on the intuition that the best possible next model, when combined with previous models, minimizes the overall prediction error. The key idea is to set the target outcomes for this next model in order to minimize the error.
Advantages and Disadvantages
Lots of flexibility - can optimize on different loss functions and provides several hyper parameter tuning options that make the function fit very flexible.
No data pre-processing required - often works great with categorical and numerical values as is.
Gradient Boosting Models will continue improving to minimize all errors. This can overemphasize outliers and cause overfitting.
Computationally expensive - often require many trees (>1000) which can be time and memory exhaustive.

XGBoosting
XGBoost minimizes a regularized (L1 and L2) objective function that combines a convex loss function (based on the difference between the predicted and target outputs) and a penalty term for model complexity (in other words, the regression tree functions).
Advantages and Disadvantages
Regularization: XGBoost has in-built L1 (Lasso Regression) and L2 (Ridge Regression) regularization which prevents the model from overfitting
Handling Missing Values: XGBoost has an in-built capability to handle missing values.
XGBoost does not perform so well on sparse and unstructured data.

Light GBM
LightGBM splits the tree leaf-wise as opposed to other boosting algorithms that grow tree level-wise. It chooses the leaf with maximum delta loss to grow. Since the leaf is fixed, the leaf-wise algorithm has lower loss compared to the level-wise algorithm.
Advantages and Disadvantages
lightGBM has the advantages of training efficiency, low memory usage, high accuracy, parallel learning, corporate support, and scale-ability.
its’ newness is its main disadvantage because there is little community support.

CatBoost
CatBoost is based on gradient boosted decision trees. During training, a set of decision trees is built consecutively. Each successive tree is built with reduced loss compared to the previous trees. The number of trees is controlled by the starting parameters.
Advantages and Disadvantages
Performance: CatBoost provides state of the art results and it is competitive with any leading machine learning algorithm on the performance front.
Easy-to-use: You can use CatBoost from the command line, using an user-friendly API for both Python and R.
Catboost's weaknesses are its training and optimization times.