Optuna

XGBoost: eXtreme Gradient Boosting

Overview
XGBoost is a scalable, distributed gradient-boosted decision tree (GBDT) library. For nearly a decade, it has been the "King of Kaggle," winning more competitions than any other algorithm on tabular data.

Why is it so good?

1. Regularization
It includes L1 and L2 regularization in the objective function, preventing overfitting better than standard Gradient Boosting.

2. Speed
- Column Block Structure: Parallelizes tree construction.
- Hardware Optimization: Cache-aware access patterns.

3. Handling Missing Values
It automatically learns the best direction (left or right) to handle missing values ('NaN') in the data.

Usage (Python)

import xgboost as xgb

# DMatrix (Internal efficient format)
dtrain = xgb.DMatrix(X_train, label=y_train)

# Parameters
param = {'max_depth': 2, 'eta': 1, 'objective': 'binary:logistic'}

# Train
bst = xgb.train(param, dtrain, num_boost_round=10)

# Predict
preds = bst.predict(dtest)

Competition
Recently, LightGBM (Microsoft) and CatBoost (Yandex) have challenged XGBoost's dominance by offering faster training speeds and better categorical handling, but XGBoost remains the gold standard baseline.

Optuna

Take optuna further