Xgb vs Lgb Benchmarks

Terminal CPUs below

The only metric was AUC for its following properties:

• Not a calibrable (aka post-optimizable) metric (unlike logarithmic loss (logloss), RMSE, etc.)
• Not relying on absolute output values (based on relative ranking only)
• Rare classes cannot be ignored by the classifier to score high (cannot optimize for one class)
• Not threshold-based (there are as many thresholds as observations, unlike Accuracy, etc.)

• Source code for reproducibility: https://github.com/Laurae2/gbt_benchmarks
• LightGBM bagging is number of threads -dependent. For instance, with 8 threads, results are reproducible only using 8 threads. AUC and final model are affected.
• LightGBM has major performance issues when bagging with less than 50% samples. See Microsoft/LightGBM#628 for the known issue.
• LightGBM is known faster using Visual Studio than MinGW in Windows, as shown in Microsoft/LightGBM#542. Pull Request Microsoft/LightGBM#584 allows to use a custom-made DLL using Visual Studio, which was the method used for this benchmark for LightGBM.
• xgboost fast histogram could not scale properly with more than 10 threads on Reputation dataset. More than 10 threads was not tested for speed purposes.
• xgboost fast histogram cannot be used with MinGW 4.9 since Pull Request dmlc/xgboost#2104 of xgboost. See dmlc/xgboost#2165 for the no-solution. The issue is still unsolved other than changing compiler version. This leaves all R Windows users who cannot compile R by themselves in limbo.
• We did not use all features on reputation dataset: it would have exploded the construction time of datasets for xgboost and LightGBM. It also may explode RAM and CPU usage altogether. See Microsoft/LightGBM#536 and dmlc/xgboost#2326. The latter issue (xgboost) has no known solution.
• There were no negative numbers after preprocessing.
• xgboost is known "cheating" when constructing binary datasets in RAM. To avoid this multithreading cheat, we removed the dataset construction time. They are denoted as 0 for every first iterations.
• For reproducibility of results, we leave the user compiling R 3.4 with MinGW 7.1 and getting the appropriate license for Windows Server 2012 R2, along setting up a proper virtualization environment from KVM / Linux. NUMA nodes were optimized (pinning, sockets, topology, etc.) for maximum performance on the Dual Xeon server.
• Reputation dataset was heavily modified to add synthetic noise where the algorithm must find the proper synthetic signal to generalize correctly. This also means there are two mixed synthetic noises: a random noise which means nothing, and a random noise which means something (but is rarer, up to 100 times more).
• To avoid overfitting, the number of iterations was dramatically reduced in certain cases compared to the previous mega benchmark. Therefore, it underestimates the maximum AUC possible.