Building llvm with -O0 for better Debugging

The version of llvm that comes with my system seems to optimize out a lot of operational details; i.e., I cannot step into certain llvm routines.

For this reason I am trying to build a debug version of llvm with no optimization; i.e. -O0.

Trying to do this with g++ resulted in a signal 15 while trying to build a library,
so I switched to using clang.

This seemed to build fine. Then I went into my build directory and executed

make Kaleidoscope-Ch5

and that seems to work but, when I tried to run the Kaleidoscope example I got:

build>bin/Kaleidoscope-Ch5 
bin/Kaleidoscope-Ch5: symbol lookup error: bin/Kaleidoscope-Ch5: undefined symbol: _ZN4llvm23EnableABIBreakingChecksE, version LLVM_11

?? What I did wrong (more details are provided below) ??

I am using llvm release/11.x brandh because that is what is also installed on my system.
I am assuming that llvm can build correctly even it it is previously already installed.
Here is the cmake command I am using to configure llvm:

cmake \
    -D CMAKE_C_COMPILER=clang \
    -D CMAKE_CXX_COMPILER=clang++ \
    \
    -D BUILD_SHARED_LIBS=OFF \
    -D LLVM_PARALLEL_LINK_JOBS=1 \
    -D CMAKE_BUILD_TYPE=Debug \
    -D CMAKE_SKIP_RPATH=ON \
    -D CMAKE_C_FLAGS="-O0" \
    -D CMAKE_CXX_FLAGS="-O0" \
    -D LLVM_LIBDIR_SUFFIX=64 \
    -D LLVM_TARGETS_TO_BUILD='Native' \
    -D LLVM_ENABLE_LIBCXX=OFF \
    -D LLVM_ENABLE_ZLIB=ON \
    -D LLVM_ENABLE_FFI=ON \
    -D LLVM_ENABLE_RTTI=ON \
    -D LLVM_BUILD_RUNTIME=ON \
    -D LLVM_INCLUDE_TOOLS=ON \
    -D LLVM_BUILD_TOOLS=ON \
    -D LLVM_INCLUDE_TESTS=ON \
    -D LLVM_BUILD_TESTS=ON \
    -D LLVM_INCLUDE_EXAMPLES=ON \
    -D LLVM_BUILD_EXAMPLES=OFF \
    -D LLVM_INCLUDE_UTILS=ON \
    -D LLVM_INSTALL_UTILS=ON \
    -D LLVM_TOOLS_INSTALL_DIR:PATH=bin \
    -D LLVM_INCLUDE_DOCS=OFF \
    -D LLVM_BUILD_DOCS=OFF \
    -D LLVM_ENABLE_SPHINX=OFF \
    -D LLVM_ENABLE_DOXYGEN=OFF \
    -D LLVM_BUILD_LLVM_DYLIB=ON \
    -D LLVM_DYLIB_EXPORT_ALL=ON \
    -D LLVM_LINK_LLVM_DYLIB=ON \
    -D LLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
    -D LLVM_INSTALL_TOOLCHAIN_ONLY=OFF \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    ../llvm 

Here is a brief description of the system I am using:

llvm.git>uname -a
Linux localhost.localdomain 5.9.13-200.fc33.x86_64 #1 SMP Tue Dec 8 15:42:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
``

This error:

undefined symbol: _ZN4llvm23EnableABIBreakingChecksE, version LLVM_11

Usually means that your binary (bin/Kaleidoscope-Ch5) has been linked to a version of LLVM built in a different configuration that the one found at runtime.
I suspect at runtime it tries to use a version of LLVM installed globally on the system. You should be able to verify this with ldd bin/Kaleidoscope-Ch5. The LD_LIBRARY_PATH environment variable can likely help here.

You’re using a lot of options, how did you come up with this list? Are these all really intended?

For example you could just not use a shared library for LLVM (Remove -D LLVM_LINK_LLVM_DYLIB=ON), or not disable the rpath ( -D CMAKE_SKIP_RPATH=ON, which is likely the cause of the wrong library being loaded).

It is really nice when the default

cmake ..

works for beginners. I tried this first, bur without any options, llvm builds for LLVM_TARGETS_TO_BUILD=all which takes forever and uses lots of space.

After that, I decided I would have to choose the cmake options, and there are lots of them, so I went to
https://src.fedoraproject.org/rpms/llvm/blob/master/f/llvm.spec
for suggestion (since I was build on a fedora system).

Changing the cmake command as suggested above got the program to link, but now when I run it I get a segmentation fault.

I do not get a segmentation fault when I used the installed version.

build>gdb --quiet bin/Kaleidoscope-Ch5 
Reading symbols from bin/Kaleidoscope-Ch5...
(gdb) run 
Starting program: /home/bradbell/repo/llvm.git/build/bin/Kaleidoscope-Ch5 
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.32-2.fc33.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ready> 1 + 2

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7d40316 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /lib64/libstdc++.so.6
Missing separate debuginfos, use: dnf debuginfo-install libgcc-10.2.1-9.fc33.x86_64 libstdc++-10.2.1-9.fc33.x86_64 ncurses-libs-6.2-3.20200222.fc33.x86_64 zlib-1.2.11-23.fc33.x86_64
(gdb) where
#0  0x00007ffff7d40316 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /lib64/libstdc++.so.6
#1  0x00007ffff7d406ae in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /lib64/libstdc++.so.6
#2  0x00000000007c9320 in llvm::DataLayout::operator= (this=0x221b158, DL=...) at /home/bradbell/repo/llvm.git/llvm/include/llvm/IR/DataLayout.h:209
#3  0x00000000007c91ee in llvm::DataLayout::DataLayout (this=0x221b158, DL=...) at /home/bradbell/repo/llvm.git/llvm/include/llvm/IR/DataLayout.h:203
#4  0x00000000007c89ed in llvm::TargetMachine::createDataLayout (this=0x0) at /home/bradbell/repo/llvm.git/llvm/include/llvm/Target/TargetMachine.h:153
#5  0x00000000007e60a4 in llvm::orc::KaleidoscopeJIT::KaleidoscopeJIT (this=0x221b060) at /home/bradbell/repo/llvm.git/llvm/examples/Kaleidoscope/Chapter5/../include/KaleidoscopeJIT.h:52
#6  0x00000000007c51e3 in std::make_unique<llvm::orc::KaleidoscopeJIT> () at /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/unique_ptr.h:962
#7  0x00000000007bbc3b in main () at /home/bradbell/repo/llvm.git/llvm/examples/Kaleidoscope/Chapter5/toy.cpp:966

Here is what happens if I build the same source using the installed version of llvm-devel:

clang++ -g -O0 toy.cpp -I/usr/include -std=c++14 -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -L/usr/lib64 -lLLVM-11 -o toy
apter5>./toy
ready> 1 + 2
ready> Evaluated to 3.000000
ready> Chapter5>

I tried cmake .. again and it failed after a long time with a signal 15 error.

Using -DCMAKE_BUILD_TYPE=Debug is enough, no need to specify -O0.

If that is the problem, -DLLVM_TARGETS_TO_BUILD=‘Native’ should be enough.

Packaging is done in a separate environment with different assumptions. You don’t need these options, best is to take a look at LLVM CMake options documentation and decide what you want.

Using -DLLVM_USE_LINKER=lld can decrease the linking time.

I tried the following options:

cmake \
    -D CMAKE_C_COMPILER='clang' \
    -D CMAKE_CXX_COMPILER='clang++' \
    \
    -D CMAKE_BUILD_TYPE='Debug' \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    -D LLVM_LIBDIR_SUFFIX='64' \
    -D CMAKE_C_FLAGS="-O0" \
    -D CMAKE_CXX_FLAGS="-O0" \
    -D CMAKE_CXX_STANDARD='14' \
    -D LLVM_TARGETS_TO_BUILD='Native' \
    ../llvm
make
make Kaleidoscope-Ch5

and got the same problem:

build>bin/Kaleidoscope-Ch5 
ready> 1 + 2
Segmentation fault (core dumped)

The same thing happened with the following configuration:

if [ $(nproc) == 1 ]
then
    n_job=1
else
    n_job=$(expr $(nproc) - 1)
fi
cmake \
    -G Ninja \
    -D CMAKE_C_COMPILER='clang' \
    -D CMAKE_CXX_COMPILER='clang++' \
    \
    -D LLVM_PARALLEL_COMPILE_JOBS="$n_job" \
    -D LLVM_PARALLEL_LINK_JOBS="$n_job" \
    \
    -D CMAKE_BUILD_TYPE='Debug' \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    -D LLVM_TARGETS_TO_BUILD='Native' \
    ../llvm | tee cmake.log
#
cmake --build . --target Kaleidoscope-Ch5

This at least compiles much faster and is easier to test ideas with.

OK I see the issue, the CMake configuration for “Native” is broken with respect to the JIT.
You should see something like -- Native target X86 is not selected; lli will not JIT code in the CMake output.

To build you “native” target the option is : -DLLVM_TARGETS_TO_BUILD="host".

With LLVM 12 instead of crashing you would see Unable to find target for this triple (no targets are registered).

I’ll see if I can fix this at head.

Ninja default to use the core-count, you don’t really need to add this here: -DLLVM_PARALLEL_LINK_JOBS is used to limit the number of parallel link (which can be memory heavy, especial when you’re not using lld)

Thanks for the fix.
The following works for me and now in the debugger I can step into the llvm routines without optimizaiton:

cmake \
    -G Ninja \
    -D CMAKE_BUILD_TYPE='Debug' \
    -D CMAKE_INSTALL_PREFIX="$HOME/prefix/llvm" \
    -D LLVM_TARGETS_TO_BUILD='host' \
    ../llvm | tee cmake.log
#
cmake --build . --target Kaleidoscope-Ch5