CI failing with "llvm-lit: not found"

Does anyone know why CI is failing to find lit all of a sudden? This doesn’t look correlated to the change.

-Chris

Hmm that is strange… I had a big LLVM update last week, but the CI checks seemed to be finding llvm-lit during and after that change…

Yeah it just randomly (and persistently) started failing. Do we need to clean out or rebuild the LLVM it is using?

I’m trying a full re-build and a few other changes now. The LLVM compile takes forever, so if it’s successful overnight, feel free to merge: Trying to fix build · llvm/circt@1c58b17 (github.com)

I wonder if it has to do with GitHub workflows updating the ubuntu-latest to use Ubuntu 20.04. All of the successful builds had a warning that this switch was coming soon… and the failing builds don’t :slight_smile:

Here’s the issue linked from the warning: https://github.com/actions/virtual-environments/issues/1816.

It does include a Clang upgrade… maybe we need to manually install llvm-lit in this version of Ubuntu?

I take that back, I do see the warning on the failing builds still.

I think this could still be the issue.

I confirmed that the failing builds are now using Ubuntu 20.04 as GitHub rolls that out for ubuntu-latest. I followed the instructions from the linked issue to lock it back to ubuntu-18.04 for now, and that seemed to work: Try locking GitHub workflows to ubuntu-18.04. · llvm/circt@f87250d · GitHub. I’ll merge it into main.

2 Likes

Thanks!

I surprised the integration tests aren’t failing. That image is based on 20.04. Maybe we should switch to using a docker image for the CI workflow just for something more predictable.

1 Like

Yeah, I chalked the CI integration tests working up to the fact that we carefully manage the installed dependencies in those containers. +1 to use a docker image for the CI workflow… could we use the same one?

The integration test image has all the dependencies installed (e.g. capnp) so it builds slightly differently which has caught issues in the past (and I seem to recall one having slipped through). So short answer: I don’t think that’s a good idea. I think it should match a vanilla build environment pretty closely.

I haven’t really looked in to this, but I think it is supposed to be using the llvm-lit from the cached llvm build. One of the cmake args is -DLLVM_EXTERNAL_LIT=pwd/../llvm/build/bin/llvm-lit and the print for missing lit is /bin/sh: 1: ../../llvm/build/bin/llvm-lit: not found. Maybe there is a mismatch between what OS LLVM was built+cached on vs CIRCT?

Fair enough.

Yes, I think something is going awry here. It does seem like the intention is to use the llvm-lit that just got built with LLVM (or restored from cache). I didn’t think that would have anything to do with OS version, and I was a little surprised that the buildAndTest.yml workflow was fixed by switching back to Ubuntu 18.04… Maybe that LLVM_EXTERNAL_LIT flag wasn’t actually working and we were just picking up the OS llvm-lit? And that isn’t installed by default on the newer Ubuntu?