Building llvm with -O0 for better Debugging

Beginners
#1

The version of llvm that comes with my system seems to optimize out a lot of operational details; i.e., I cannot step into certain llvm routines.

For this reason I am trying to build a debug version of llvm with no optimization; i.e. -O0.

Trying to do this with g++ resulted in a signal 15 while trying to build a library,
so I switched to using clang.

This seemed to build fine. Then I went into my build directory and executed

make Kaleidoscope-Ch5

and that seems to work but, when I tried to run the Kaleidoscope example I got:

build>bin/Kaleidoscope-Ch5 
bin/Kaleidoscope-Ch5: symbol lookup error: bin/Kaleidoscope-Ch5: undefined symbol: _ZN4llvm23EnableABIBreakingChecksE, version LLVM_11

?? What I did wrong (more details are provided below) ??

I am using llvm release/11.x brandh because that is what is also installed on my system.
I am assuming that llvm can build correctly even it it is previously already installed.
Here is the cmake command I am using to configure llvm:

cmake \
    -D CMAKE_C_COMPILER=clang \
    -D CMAKE_CXX_COMPILER=clang++ \
    \
    -D BUILD_SHARED_LIBS=OFF \
    -D LLVM_PARALLEL_LINK_JOBS=1 \
    -D CMAKE_BUILD_TYPE=Debug \
    -D CMAKE_SKIP_RPATH=ON \
    -D CMAKE_C_FLAGS="-O0" \
    -D CMAKE_CXX_FLAGS="-O0" \
    -D LLVM_LIBDIR_SUFFIX=64 \
    -D LLVM_TARGETS_TO_BUILD='Native' \
    -D LLVM_ENABLE_LIBCXX=OFF \
    -D LLVM_ENABLE_ZLIB=ON \
    -D LLVM_ENABLE_FFI=ON \
    -D LLVM_ENABLE_RTTI=ON \
    -D LLVM_BUILD_RUNTIME=ON \
    -D LLVM_INCLUDE_TOOLS=ON \
    -D LLVM_BUILD_TOOLS=ON \
    -D LLVM_INCLUDE_TESTS=ON \
    -D LLVM_BUILD_TESTS=ON \
    -D LLVM_INCLUDE_EXAMPLES=ON \
    -D LLVM_BUILD_EXAMPLES=OFF \
    -D LLVM_INCLUDE_UTILS=ON \
    -D LLVM_INSTALL_UTILS=ON \
    -D LLVM_TOOLS_INSTALL_DIR:PATH=bin \
    -D LLVM_INCLUDE_DOCS=OFF \
    -D LLVM_BUILD_DOCS=OFF \
    -D LLVM_ENABLE_SPHINX=OFF \
    -D LLVM_ENABLE_DOXYGEN=OFF \
    -D LLVM_BUILD_LLVM_DYLIB=ON \
    -D LLVM_DYLIB_EXPORT_ALL=ON \
    -D LLVM_LINK_LLVM_DYLIB=ON \
    -D LLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
    -D LLVM_INSTALL_TOOLCHAIN_ONLY=OFF \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    ../llvm

Here is a brief description of the system I am using:

llvm.git>uname -a
Linux localhost.localdomain 5.9.13-200.fc33.x86_64 #1 SMP Tue Dec 8 15:42:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
``
#2

This error:

undefined symbol: _ZN4llvm23EnableABIBreakingChecksE, version LLVM_11

Usually means that your binary (bin/Kaleidoscope-Ch5) has been linked to a version of LLVM built in a different configuration that the one found at runtime.
I suspect at runtime it tries to use a version of LLVM installed globally on the system. You should be able to verify this with ldd bin/Kaleidoscope-Ch5. The LD_LIBRARY_PATH environment variable can likely help here.

You’re using a lot of options, how did you come up with this list? Are these all really intended?

For example you could just not use a shared library for LLVM (Remove -D LLVM_LINK_LLVM_DYLIB=ON), or not disable the rpath ( -D CMAKE_SKIP_RPATH=ON, which is likely the cause of the wrong library being loaded).

#3

It is really nice when the default

cmake ..

works for beginners. I tried this first, bur without any options, llvm builds for LLVM_TARGETS_TO_BUILD=all which takes forever and uses lots of space.

After that, I decided I would have to choose the cmake options, and there are lots of them, so I went to
https://src.fedoraproject.org/rpms/llvm/blob/master/f/llvm.spec
for suggestion (since I was build on a fedora system).

#4

Changing the cmake command as suggested above got the program to link, but now when I run it I get a segmentation fault.

I do not get a segmentation fault when I used the installed version.

build>gdb --quiet bin/Kaleidoscope-Ch5 
Reading symbols from bin/Kaleidoscope-Ch5...
(gdb) run 
Starting program: /home/bradbell/repo/llvm.git/build/bin/Kaleidoscope-Ch5 
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.32-2.fc33.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ready> 1 + 2

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7d40316 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /lib64/libstdc++.so.6
Missing separate debuginfos, use: dnf debuginfo-install libgcc-10.2.1-9.fc33.x86_64 libstdc++-10.2.1-9.fc33.x86_64 ncurses-libs-6.2-3.20200222.fc33.x86_64 zlib-1.2.11-23.fc33.x86_64
(gdb) where
#0  0x00007ffff7d40316 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /lib64/libstdc++.so.6
#1  0x00007ffff7d406ae in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /lib64/libstdc++.so.6
#2  0x00000000007c9320 in llvm::DataLayout::operator= (this=0x221b158, DL=...) at /home/bradbell/repo/llvm.git/llvm/include/llvm/IR/DataLayout.h:209
#3  0x00000000007c91ee in llvm::DataLayout::DataLayout (this=0x221b158, DL=...) at /home/bradbell/repo/llvm.git/llvm/include/llvm/IR/DataLayout.h:203
#4  0x00000000007c89ed in llvm::TargetMachine::createDataLayout (this=0x0) at /home/bradbell/repo/llvm.git/llvm/include/llvm/Target/TargetMachine.h:153
#5  0x00000000007e60a4 in llvm::orc::KaleidoscopeJIT::KaleidoscopeJIT (this=0x221b060) at /home/bradbell/repo/llvm.git/llvm/examples/Kaleidoscope/Chapter5/../include/KaleidoscopeJIT.h:52
#6  0x00000000007c51e3 in std::make_unique<llvm::orc::KaleidoscopeJIT> () at /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/unique_ptr.h:962
#7  0x00000000007bbc3b in main () at /home/bradbell/repo/llvm.git/llvm/examples/Kaleidoscope/Chapter5/toy.cpp:966

Here is what happens if I build the same source using the installed version of llvm-devel:

clang++ -g -O0 toy.cpp -I/usr/include -std=c++14 -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -L/usr/lib64 -lLLVM-11 -o toy
apter5>./toy
ready> 1 + 2
ready> Evaluated to 3.000000
ready> Chapter5>
#5

I tried cmake .. again and it failed after a long time with a signal 15 error.

#6

Using -DCMAKE_BUILD_TYPE=Debug is enough, no need to specify -O0.

If that is the problem, -DLLVM_TARGETS_TO_BUILD=‘Native’ should be enough.

Packaging is done in a separate environment with different assumptions. You don’t need these options, best is to take a look at LLVM CMake options documentation and decide what you want.

Using -DLLVM_USE_LINKER=lld can decrease the linking time.

#7

I tried the following options:

cmake \
    -D CMAKE_C_COMPILER='clang' \
    -D CMAKE_CXX_COMPILER='clang++' \
    \
    -D CMAKE_BUILD_TYPE='Debug' \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    -D LLVM_LIBDIR_SUFFIX='64' \
    -D CMAKE_C_FLAGS="-O0" \
    -D CMAKE_CXX_FLAGS="-O0" \
    -D CMAKE_CXX_STANDARD='14' \
    -D LLVM_TARGETS_TO_BUILD='Native' \
    ../llvm
make
make Kaleidoscope-Ch5

and got the same problem:

build>bin/Kaleidoscope-Ch5 
ready> 1 + 2
Segmentation fault (core dumped)
#8

The same thing happened with the following configuration:

if [ $(nproc) == 1 ]
then
    n_job=1
else
    n_job=$(expr $(nproc) - 1)
fi
cmake \
    -G Ninja \
    -D CMAKE_C_COMPILER='clang' \
    -D CMAKE_CXX_COMPILER='clang++' \
    \
    -D LLVM_PARALLEL_COMPILE_JOBS="$n_job" \
    -D LLVM_PARALLEL_LINK_JOBS="$n_job" \
    \
    -D CMAKE_BUILD_TYPE='Debug' \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    -D LLVM_TARGETS_TO_BUILD='Native' \
    ../llvm | tee cmake.log
#
cmake --build . --target Kaleidoscope-Ch5

This at least compiles much faster and is easier to test ideas with.

#9

OK I see the issue, the CMake configuration for “Native” is broken with respect to the JIT.
You should see something like -- Native target X86 is not selected; lli will not JIT code in the CMake output.

To build you “native” target the option is : -DLLVM_TARGETS_TO_BUILD="host".

With LLVM 12 instead of crashing you would see Unable to find target for this triple (no targets are registered).

I’ll see if I can fix this at head.

Ninja default to use the core-count, you don’t really need to add this here: -DLLVM_PARALLEL_LINK_JOBS is used to limit the number of parallel link (which can be memory heavy, especial when you’re not using lld)

#10

Thanks for the fix.
The following works for me and now in the debugger I can step into the llvm routines without optimizaiton:

cmake \
    -G Ninja \
    -D CMAKE_BUILD_TYPE='Debug' \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    -D LLVM_TARGETS_TO_BUILD='host' \
    ../llvm | tee cmake.log
#
cmake --build . --target Kaleidoscope-Ch5