XGBoost debug in Visual Studio

by Ginko Balboa

Posted on November 11, 2021



1. General info

The famous library for machine learning XGBoost is developed mainly for Linux. In that manner, maybe less attention is given to compiling XGBoost under Visual Studio in Debug mode. One of the reasons Linux is preferred is because of CUDA support (for example, CUDA doesn't support NCCL under Windows).

For generating VS solutions, XGBoost uses CMake. So there is no possibility to generate Release and Debug configuration in the same project. One of the two configurations must be set manually.

Manual setup of a Debug version starting from the Release solution is explained in this post in the hope to help others struggling with the same problem.

2. Clone the repo and generate VS solution

According to the documentation for the installation for dmlc XGBoost. First you will need to clone the XGBoost repo the following command:

git clone --recurse-submodules https://github.com/dmlc/xgboost.git

If you pass --recurse-submodules to the git clone command, it will automatically initialize and update each submodule in the repository, including nested submodules, if any of the submodules in the repository have submodules themselves.

Now you should have the xgboost folder with the latest source code.


NOTE

If you cloned previously without the submodules, you can update them afterwards buy running (from the xgboost folder):

git submodule init
git submodule update

Create a build directory inside the xgboost folder where the .sln will be created. From inside the build directory, run CMake to create a .sln. The above steps are performed by running the following commands from PowerShell:

mkdir build
cd build
cmake .. -G "Visual Studio 16 2019" -A x64 -T v142,cuda=11.5 -DUSE_CUDA=ON -DGOOGLE_TEST=ON -DUSE_DMLC_GTEST=ON -DBUILD_WITH_CUDA_CUB=ON

The command for generating the solution is in the format that suits my VS configuration:

  • Visual Studio 2019 (v142) on 64bit machine.
  • Installed CUDA toolkit version is 11.5.
  • For CUDA toolkit >= 11.4, BUILD_WITH_CUDA_CUB is required.

Distributed GPU training depends on NCCL. Since NCCL is only available for Linux machines, distributed GPU training is not available in Windows. The following flag will NOT work under Windows -DUSE_NCCL=ON. Anyway, the NCCL is by default turned off.

When CMake finishes, you can open /xgboost/build/xgboost.sln with your Visual Studio and compile the library (right click on xgboost project and Build) in Release mode. This is the starting point to set up the XGBoost for the Debug mode.

3. Set up the debug mode

We will set up the debug mode going step by step, by setting up and compiling standalone parts of the project. But first, you must turn off the CMake automatic run, or all the settings that you made will be gone.

  • Step 1: Tools->Options->CMake->Never run configure step automatically.

3.1. Setting up the dmlc core

We want to run dmlc_unit_tests in Debug mode. These depend on dmlc and gtest projects.

  • Step 2: Add debug configuration
    • We add a new configuration by clicking on Configuration Manager (dmlc->Properties->Configuration Manager). Add new and name it Debug and copy settings from Release. Only this first time we leave the box under Create new solution configurations checked. Now the whole solution has an option Debug.
    • Add debug mode for dmlc_unit_tests
    • Add debug mode for gtest

We choose All Configurations and replace the hardcoded word "Release" with "$(Configuration)" in the following places:

  • Step 3: Replace "Release" for All Configurations mode:
    • dmlc->Properties->General->Output Directory
    • dmlc->Properties->General->Intermediate Directory
    • dmlc_unit_tests->Properties->General->Output Directory
    • dmlc_unit_tests->Properties->General->Intermediate Directory
    • dmlc_unit_tests->Linker->Input->Additional Dependencies
    • gtest->Properties->General->Output Directory
    • gtest->Properties->General->Intermediate Directory

Next we set the debug options in the mentioned projects, this is set only on the Debug mode.

  • Step 4: Set debug options for Debug mode:
    • dmlc->Properties->C/C++->General->Debug Information Format to Program Database (/Zi)
    • dmlc->Properties->C/C++->Code Generation->Runtime Library to Multi-threaded Debug (/MTd)
    • dmlc->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)
    • dmlc_unit_tests->Properties->C/C++->General->Debug Information Format to Program Database (/Zi)
    • dmlc_unit_tests->Properties->C/C++->Code Generation->Runtime Library to Multi-threaded Debug (/MTd)
    • dmlc_unit_tests->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)
    • dmlc_unit_tests->Properties->Linker->Debugging->Generate Debug Info to Generate Debug Information (/DEBUG)
    • gtest->Properties->C/C++->General->Debug Information Format to Program Database (/Zi)
    • gtest->Properties->C/C++->Code Generation->Runtime Library to Multi-threaded Debug (/MTd)
    • gtest->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)

NOTE

We choose Debug mode and build dmlc_unit_tests this builds all the mentioned libraries in debug mode. We can now run tests and breakpoint inside the code. If the program doesn't stop on breakpoint some of the following things should be set:

  • In Tools->Options->Debugging->Symbols the option Microsoft Symbol Servers should be checked.
  • In Tools->Options->Debugging->General the options:
    • Enable Just My Code - unchecked
    • Enable address-level debugging and item from sublist - checked
    • Enable source server support and all items from sublist - checked
    • Enable Source Link support - checked
    • Load dll exports - checked

3.2. Setting up the xgboost library

Lets do the same settings for the rest of the projects needed to compile XGBoost library.

  • Step 5: Add debug configuration (same as step 2)

    • Debug mode for objxgboost
    • Debug mode for test_xgboost
    • Debug mode for runxgboost
    • Debug mode for xgboost
  • Step 6: Replace "Release" for All Configurations mode (same as step 3):

    • objxgboost->Properties->General->Output Directory
    • objxgboost->Properties->General->Intermediate Directory
    • testxgboost->Properties->General->Intermediate Directory
    • testxgboost->Linker->Input->Additional Dependencies
    • runxgboost->Properties->General->Intermediate Directory
    • runxgboost->Linker->Input->Additional Dependencies
    • xgboost->Properties->General->Intermediate Directory
    • xgboost->Linker->Input->Additional Dependencies
  • Step 7: Set debug options for Debug mode (same as step 4):

    • objxgboost->Properties->C/C++->General->Debug Information Format to Program Database (/Zi)
    • objxgboost->Properties->C/C++->Code Generation->Runtime Library to Multi-threaded Debug (/MTd)
    • objxgboost->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)
    • testxgboost->Properties->C/C++->General->Debug Information Format to Program Database (/Zi)
    • testxgboost->Properties->C/C++->Code Generation->Runtime Library to Multi-threaded Debug (/MTd)
    • testxgboost->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)
    • testxgboost->Linker->Debugging->Generate Debug Info to Generate Debug Information (/DEBUG)
    • runxgboost->Properties->C/C++->General->Debug Information Format to Program Database (/Zi)
    • runxgboost->Properties->C/C++->Code Generation->Runtime Library to Multi-threaded Debug (/MTd)
    • runxgboost->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)
    • runxgboost->Linker->Debugging->Generate Debug Info to Generate Debug Information (/DEBUG)
    • xgboost->Properties->CUDA C/C++->Host->Runtime Library to Multi-threaded Debug (/MTd)
    • xgboost->Linker->Debugging->Generate Debug Info to Generate Debug Information (/DEBUG)

We compile xgboost and we get the debug version of the dll in xgboost\lib\xgboost.dll.

Next we can pack the xgboost library using wheel.

pip uninstall xgboost
cd python-package
rm -rf dist
python setup.py bdist_wheel
pip install ./dist/xgboost-1.6.0.dev0-cp38-cp38-win_amd64.whl

A somewhat better option is to use the generated files directly, without installing. To do this run the following commands:

pip uninstall xgboost
cd python-package
pip install -e .

You can debug both Python and C++ code from Visual Studio Code using the following addon Python C++ Debugger.