[VectorOps] [RFC] Add suite of "integration" tests for Vector dialect operations

MLIR
1

In parallel to the microbenchmarking effort for the Vector dialect (see AVX512 case study under Talks - MLIR), Nicolas and I wrote a test suite of “integration” tests for most of the Vector dialect operations. The test suite currently consists of tests for broadcast, constant-mask, contraction, create-mask, extract-slices, extract-strided-slice, flat-transpose-col/row, fma, insert-slices, insert-strided-slice, matrix-multiply-col/row, outer product, reductions, shape-cast, shuffle, transfer-read, transfer-write, and transpose.

Each test sets up test vectors, performs various different forms of one particular operation, and, driven by a RUN command line (mlir-opt/mlir-cpu-runner and small runtime support library for printing), verifies the result using vector.print and CHECK. An example is shown below. These tests serve several purposes:

  • verify that the lowering to LLVM IR for CPU yields correct code
  • ensure future changes do not break the lowering
  • supplement the documentation with real-working and illustrative examples
  • provide a test suite for other lowering paths

As such, I would really like to move this suite from a closed repository into the MLIR repository. This way, others can benefit from and contribute to the suite. Only small vectors are tested, keeping the suite’s running overhead very light.

Questions.

  1. is there an interest in such a suite?
  2. what would be a good place to put the suite?

Some suggestions for such a location:
llvm/llvm-project/mlir/test/mlir-cpu-runner/vector
llvm/llvm-project/mlir/test/Dialect/Vector/integration
llvm/llvm-project/mlir/test/Target/CPU/vector

SAMPLE TEST

// RUN: mlir-opt %s ... | mlir-cpu-runner ... | FileCheck %s

func @entry() {
  %f1 = constant 1.0: f32
  %f2 = constant 2.0: f32
  %v1 = vector.broadcast %f1 : f32 to vector<2x4xf32>
  %v2 = vector.broadcast %f2 : f32 to vector<2x4xf32>
  vector.print %v1 : vector<2x4xf32>
  vector.print %v2 : vector<2x4xf32>
  //
  // test vectors:
  //
  // CHECK: ( ( 1, 1, 1, 1 ), ( 1, 1, 1, 1 ) )
  // CHECK: ( ( 2, 2, 2, 2 ), ( 2, 2, 2, 2 ) )

  %v3 = vector.shuffle %v1, %v2 [3, 1, 2] : vector<2x4xf32>, vector<2x4xf32>
  vector.print %v3 : vector<3x4xf32>
  // CHECK: ( ( 2, 2, 2, 2 ), ( 1, 1, 1, 1 ), ( 2, 2, 2, 2 ) )

  return
}
2

Thanks Aart!
I would very much like to expose those and stop having them in our private repo (and fix asynchronously). In the past, this would have been useful to catch issues related to e.g. alignment.

If they end up taking too long upstream, and people object, we could consider putting them in integration_tests and have those only trigger conditionally.

1 Like
3

Having a separate “integration_tests” directory, not enabled by default during development, would be my favorite choice here.

We could update the MLIR bot to pass a CMake option enable these. Seems similar to the LLVM_BUILD_BENCHMARKS CMake option (default Off).

4

+1. I would strongly prefer to keep these types of tests outside of test/. Some of that has unfortunately already started creeping in with the particularly long sgemm_naive_codegen.mlir test.

5

+1 to having a separate “integration_tests” directory in-tree that is non default. As Nicolas says, we’re accumulating a fair amount of these kinds of bits in private repos, which is costing us both velocity and collaboration potential.

6

+1 to moving test/integration_tests into the same top-level integration_tests

7

Okay, it sounds there is interest, as long as we keep this outside our regular testing sub-tree.
Then next, are there preferences on the exact location in a new “integration_test” directory (or should it be plural “integration_tests”; we have singular “test” now for unit tests)?

Some ideas are

llvm/llvm-project/mlir/integration_test/...
     .../Dialect/Vector           ; reflects the dialect ops
     .../Dialect/Vector/CPU       ; also reflects target of lowering path
     .../Target/CPU/Vector        ; directly reflects target, then dialect
     .../Conversion/VectorToLLVM/ ; emphasizes lowering
8

The structure you propose SGTM, I would additionally add an “integration_test/runners” subdir (or better named as it’s been proven time and again I can’t name things…) to move all the CPU/GPU/SPIRV/ROCDL etc stuff.

9

I have completed a first migration from the private repo into LLVM/MLIR (I had to hold back a few integration tests, since they still expose bugs in the backend).

Selected location of the suite:

llvm/llvm-project/mlir/integration_test/Dialect/Vector/CPU

Running the integration tests currently only invokes this particular vector integration test suite and is still rather light-weight. But hopefully it sets up a testing infrastructure for a lot more.

$ cmake --build . --target check-mlir-integration
[0/1] Running the MLIR integration tests

Testing Time: 0.24s
  Passed: 22
10

+1 to integration_test/, it would be great if we could move sgemm_naive_codegen and other slow “integration tests” there too.

11

How many of them are there? Can you just pull them in with XFAIL and a comment as to what is blocking turning them on?

–River

12

Two, related to mask creation. They exposed various back-end bugs in the past (see e.g. https://bugs.llvm.org/show_bug.cgi?id=45995), and still don’t seem to run fully clean in opt mode. I prefer to add them later when I have resolved all issues, rather than convolute the new framework with immediate failures :slight_smile: Or at the very least, I would like to add them later as XFAIL.

13

Note, the corresponding differential https://reviews.llvm.org/D81626 has been merged. I look forward to community contributions to the integration tests: additions, improvements, new suites, migrating test from unit to this new place, etc.!

14

Do I need to add a special flag when building LLVM/MLIR? I currently build it as recommended in the “Getting Started” page and

$ cmake --build . --target check-mlir-integration

is not recognized as a target in the top build directory. Asking this as I am trying to add a new test for vector.transfer_read using 2D vectors.

15

On the initial setup cmake command, make sure to pass the following

cmake ... \
  -DMLIR_INCLUDE_INTEGRATION_TESTS=ON \
  ...

After that

cmake --build . --target check-mlir-integration

will run just the integration tests. Also

cmake --build . --target check-mlir

will run all tests, including the integration test.

Thanks for contributing to the integration tests suite!

16

Added a test, in Phabricator now. Thanks for the pointers on how to build the integration suite. Examples are probably the best help to understand the code at this time.

1 Like
17

I moved it under test/Integration instead of a top-level directory, see https://reviews.llvm.org/D97241

It does not change the fact that they don’t run unless MLIR is configured with -DMLIR_INCLUDE_INTEGRATION_TESTS=ON.

1 Like