Thoughts on running the sanitizer tests with "real" MSVC

Runtimes Sanitizers
1

Hi, I work on the Microsoft Visual C++ team, on Asan. We currently have a fork of LLVM14 with modifications to make asan work better on windows, and we’re gearing up to “update” to main, regularly take changes from main, and generally prefer to make new changes by contributing them to upstream LLVM first.

I’ve mentioned this before here: Upstreaming the Microsoft asan-on-MSVC changes - Runtimes / Sanitizers - LLVM Discussion Forums.

I wanted to solicit feedback about something a bit different though. We have our own code to set up lit and run asan’s lit tests on windows, with MSVC, it depends somewhat on lit internals and requires frequent poking when taking upstream test changes. Does anyone have strong feelings against changes that would allow running the asan tests with “real” MSVC? The idea is that this’ll allow us to easily upstream additional test coverage (we have about 120 “new” tests) and will prevent us from getting broken by “unrelated” test changes.

I have made modifications to LLVM’s cmake and lit configuration to allow msvc to be used as the “COMPILER_RT_TEST_COMPILER” when building compiler-rt with an “external” LLVM, and I’m porting over all the test changes we’ve made. Here are some questions that have come up:

compiler command line style

Currently asan tests can be one of:

  • normal clang for windows: clang and clang-cl with windows-msvc ABI
  • mingw tests, with only clang.exe, and windows-gnu ABI, here %clang_cl, %clang_cl_asan, and friends actually expand to clang.exe and there are substitutions for common CL style arguments to replace them with non-CL-style equivalents.

I am adding:

  • “real” MSVC, where all of %clang, %clangxx, %clang_cl, %clang_cl_asan, %clangxx_asan, %clang_asan expand to cl.exe, and there are the “reverse” substitutions. So %MD, %LD, etc, expand to -MD, -LD like with clang-cl.exe, but additionally there are substitutions replacing -O0 with /Od and -O1 with /O1i-, and so on.

This starts to get pretty confusing but I couldn’t think of a cleaner way to do it. Substitutions that only apply for a few tests are done with %if MSVC %{ ... %}.

Does anyone have a better idea here?

Test requirements

With my changes to run the tests with MSVC you need to first build a copy of LLVM (at least lit and FileCheck, but really a lot more since it needs to be installed) then build compiler-rt with the external LLVM you installed. Is there a way to do this “standalone”, where compiler-rt is built as part of LLVM, but the COMPILER_RT_TEST_COMPILER is not the “just built” clang and clang isn’t depended on by the compiler-rt tests? Ideally LLVM itself wouldn’t be included either, just lit and FileCheck.

Spaces

This isn’t critical for us, but it seems like lit has trouble running commands with spaces in their paths. This causes problems using the “official” installed MSVC with a fully resolved path. (since it’s in “C:\Program Files”). I tried various quoting styles and they didn’t seem to work. Does anyone know a way around this off the top of their head? This seems worth fixing but only if there’s no easy way to quote things that I’m missing.

1 Like
2

FWIW, it is possible to run the compiler-rt/asan tests without doing a full/installed build of LLVM - I do that as part of my testing for mingw, like this:

I do build a handful of executables from LLVM, the executables not FileCheck llvm-config KillTheDoctor count split-file, but I don’t install it. Instead I pass LLVM_CONFIG_PATH and LLVM_CMAKE_DIR to the compiler-rt build, pointing into directories in the build tree from that earlier partial LLVM build.

It’s not very elegant, but works pretty well, and building the few test executables in the LLVM tree doesn’t take very long (and seems to only build ~210 files): Builds · mstorsjo/llvm-mingw@2312d6e · GitHub