TL;DR: I’d like to set up nightly testing based on Docker images. This would be a non-disruptive, non-PR-gating addition.
Docker seems to be the way modern projects assemble their environment for testing and production. (I personally have used it for development also.) As such, I think we should use it for running our nightly integration tests. I have very successfully done this several times in the past. Given that this is relatively standard these days, I’m going to assume that everyone either doesn’t care or is in agreement for the rest of this post.
I’ve put together a prototype workflow which runs the integration tests out of a Docker container with all the necessary software pre-installed. The prototype workflow builds the image (with the Docker cache, so rarely re-builds it) then runs the build and test inside the container. This ensures that the pre-installed software matches the scripts in the CIRCT repo.
Setting up this workflow different repos (different organizations really) is a bit involved since it involves an organization admin configuring permissions for the container registry then a workflow secret. It’s not too much work but after doing it this way, I’m thinking it’s not worth the effort.
Instead, I think we should set up a CIRCT Docker repo in the circt github repo and use it to build the image(s) which are used by the CIRCT integration builds. This method would make forking while maintaining operation of the nightly tests easier. It would, however, mean when adding/updating a test requirement (e.g. updating Verilator) changing both the Docker repo to generate a new image (with the upgraded Verilator) and the CIRCT nightly workflow to point to the new image. That’s a fair trade-off (IMO) given that these requirements rarely change.
I do not think we should include MLIR in said image since we are deeply coupled to it for build and it changes relatively frequently. Also, we’ve already got a caching solution for the llvm submodule which works well. The Docker image should only contain the all the optional requirements for building and integration testing of MLIR and CIRCT.
Does everyone agree that this is the way to go? If so, I can set it up if I’m made an owner for the CIRCT organization, then PR into the CIRCT repo with an additional workflow to do the nightly (and on-demand) integration testing.