Xgb vs Lgb Benchmarks
February 2017 Edition
Datasets Used
Bosch
- 1,000,000 training observations (first)
- 183,747 testing observations (last)
- 969 features (ID removed, rescaled, NAs as zeros, one feature with count of NAs)
- Metric: AUC (because it is a global metric)
Higgs
- 10,000,000 training observations (first)
- 1,000,000 testing observations (last)
- 29 features (rescaled, NAs as zeros, one feature with count of NAs)
- Metric: AUC (because it is a global metric)
- Reality: dropped due to convergence issues, but you can still see them
Servers Used (Hardware / Software)
1-6 + 12 thread runs
- CPU: i7-3930K (3.9/3.5 GHz, 6c/12t)
- RAM: 64GB 1600MHz RAM (54GB RAM child)
- OS: Ubuntu 16.04 (4.4 kernel)
- Virtualization: KVM + Windows Server 2012 R2 Datacenter
- Microsoft R Client 3.3.2
- Rtools 3.4
20 + 40 + gpu threads runs
- CPU: Dual Quanta Freedom Ivy Bridge (3.3/2.7 GHz, 20c/40t)
- RAM: 96GB 1600MHz RAM (80GB RAM child)
- OS: Ubuntu 16.04 (4.4 kernel)
- Virtualization: KVM + Windows Server 2012 R2 Datacenter
- Microsoft R Client 3.3.2
- Rtools 3.4
- GPU: Intel OpenCL on CPU
- Reality: dropped as we know too many threads/GPU is bad, but you can still see them
Gradient Boosted Trees Algorithms Used
xgboost
- Versions used: commit
b4d97d3
- Flags used:
- Default:
-O2 -mtune=core2
- Extra 1:
-O3 -funroll-loops -march=native
- Extra 2:
-O3 -funroll-loops -march=native -ffast-math
+-Wl,-O1 -O3 -Wl,-ffast-math
on linker
- Default:
LightGBM
- Versions used: commit
ea6bc0a
(v1) and1bf7bbd
(v2) - Flags used:
- Default:
-O2 -mtune=core2
- Extra 1:
-O3 -march=native
- Extra 2:
-O3 -march=native -ffast-math
+-Wl,-O1 -O3 -Wl,-ffast-math
on linker - Extra 3:
-O2 -march=native
- Extra 4:
-Os
- Default:
Installation of Gradient Boosted Trees Algorithms
xgboost
- Manual installation of commit
dmlc/xgboost@b4d97d3
(Pull Request 2045, Feb 20 2017) or devtools::install_github("Laurae2/ez_xgb/R-package@2017-02-15-v1", force = TRUE)
LightGBM
- LightGBM v1:
devtools::install_github("Microsoft/LightGBM@v1@ea6bc0a", subdir = "R-package", force = TRUE)
- LightGBM v2:
devtools::install_github("Microsoft/LightGBM@1bf7bbd", subdir = "R-package", force = TRUE)
- LightGBM v2 + Intel OpenCL:
devtools::install_github("Laurae2/LightGBM@7514fde", subdir = "R-package", force = TRUE)
(or make an installation using https://github.com/Microsoft/LightGBM/pull/448
Hyperparameters Used (Full list)
Bosch
Leaves Hyperparameters:
- Depth: ∞
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: {2K, 1.5K, 1K, 1K, 500, 400, 400, 400, 400, 400}
- Learning Rate: 0.02
Depth Hyperparameters:
- Depth: {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: {2K, 1.5K, 1K, 1K, 500, 400, 400, 400, 400, 400}
- Learning Rate: 0.02
Pruning Hyperparameters:
- Depth: 10
- Leaves: 1023
- Hessian: {1, 5, 25, 125}
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 400
- Learning Rate: 0.02
Sampling Hyperparameters:
- Depth: 6
- Leaves: 63
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: {100%, 80%, 60%, 40%}
- Iterations: 1000
- Learning Rate: 0.04
Higgs
Leaves Hyperparameters:
- Depth: ∞
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 500
- Learning Rate: 0.25
Depth Hyperparameters:
- Depth: {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 500
- Learning Rate: 0.25
Pruning Hyperparameters:
- Depth: 10
- Leaves: 1023
- Hessian: {1, 5, 25, 125}
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 500
- Learning Rate: 0.25
Sampling Hyperparameters:
- Depth: 6
- Leaves: 63
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: {100%, 80%, 60%, 40%}
- Iterations: 1000
- Learning Rate: 0.25
Metric Performance Analysis
Use the Metric Performance Analysis if you expect to compare AUC metric data. LightGBM is not reproducible when Stochastic.
Check interactively on Tableau Public:
https://public.tableau.com/views/gbt_benchmarks/AUC-Data?:showVizHome=no
Provided dynamic and interactive filters:
- Type
- Dataset
- Algorithm
- Parameter
- Threads
- CPU
Master Data Analysis
Use the Master Analysis if you expect all data at the same time.
Check interactively on Tableau Public:
https://public.tableau.com/views/gbt_benchmarks/Master-Data?:showVizHome=no
Provided dynamic and interactive filters:
- Type
- Dataset
- Algorithm
- Parameter
- Threads
- CPU
Disaggregated Data Analysis
Use the Disaggregated Data Analysis if you want fully detailed results.
Check interactively on Tableau Public:
Provided dynamic and interactive filters:
- Type
- Dataset
- Algorithm
- Parameter
- Threads
- CPU
- Iteration
Leaves Analysis
Bosch
Leaves Hyperparameters:
- Depth: ∞
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: {2K, 1.5K, 1K, 1K, 500, 400, 400, 400, 400, 400}
- Learning Rate: 0.02
Higgs
Leaves Hyperparameters:
- Depth: ∞
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 500
- Learning Rate: 0.25
Use the Leaves Analysis if you expect only Leaves data to show up.
Check interactively on Tableau Public:
https://public.tableau.com/views/gbt_benchmarks/Leaves-Data?:showVizHome=no
Provided dynamic and interactive filters:
- Dataset
- Algorithm
- Parameter
- Threads
- CPU
Depth Analysis
Bosch
Depth Hyperparameters:
- Depth: {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: {2K, 1.5K, 1K, 1K, 500, 400, 400, 400, 400, 400}
- Learning Rate: 0.02
Higgs
Depth Hyperparameters:
- Depth: {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
- Leaves: {7, 15, 31, 63, 127, 255, 511, 1023, 2047, 4095}
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 500
- Learning Rate: 0.25
Use the Depth Analysis if you expect only Depth data to show up.
Check interactively on Tableau Public:
https://public.tableau.com/views/gbt_benchmarks/Depth-Data?:showVizHome=no
Provided dynamic and interactive filters:
- Dataset
- Algorithm
- Parameter
- Threads
- CPU
Pruning Analysis
Bosch
Pruning Hyperparameters:
- Depth: 10
- Leaves: 1023
- Hessian: {1, 5, 25, 125}
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 400
- Learning Rate: 0.02
Higgs
Pruning Hyperparameters:
- Depth: 10
- Leaves: 1023
- Hessian: {1, 5, 25, 125}
- Column Sampling: 100%
- Row Sampling: 100%
- Iterations: 500
- Learning Rate: 0.25
Use the Pruning Analysis if you expect only Pruning data to show up.
Check interactively on Tableau Public:
https://public.tableau.com/views/gbt_benchmarks/Pruning-Data?:showVizHome=no
Provided dynamic and interactive filters:
- Dataset
- Algorithm
- Parameter
- Threads
- CPU
Sampling Analysis
Bosch
Sampling Hyperparameters:
- Depth: 6
- Leaves: 63
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: {100%, 80%, 60%, 40%}
- Iterations: 1000
- Learning Rate: 0.04
Higgs
Sampling Hyperparameters:
- Depth: 6
- Leaves: 63
- Hessian: 1
- Column Sampling: 100%
- Row Sampling: {100%, 80%, 60%, 40%}
- Iterations: 1000
- Learning Rate: 0.25
Use the Sampling Analysis if you expect only Sampling data to show up.
Check interactively on Tableau Public:
https://public.tableau.com/views/gbt_benchmarks/Sampling-Data?:showVizHome=no
Provided dynamic and interactive filters:
- Dataset
- Algorithm
- Parameter
- Threads
- CPU