[Meeting Notes] MLIR Build/Install and Shared Libraries

MLIR
1

Several of us met earlier today as an offshoot of the thread about installing and releasing Python-based MLIR projects. Here are some rough notes and outcomes (from memory - please correct if mis-stated).

Attendees:

Notes:

  • General Discussion:

    • Treat installing/release as a supply chain problem: there are a lot of ways people downstream will need to assemble things, and it is best if the project provides the components for these to happen and has the right divisions in place for know possibilities.
    • Especially when considering Windows, forethought is needed since not all paths are tractable ~months down the road.
    • Agreement that the baseline support needed for Python deployability of core and dependent python projects is to install the C++ deps (headers/libs) needed to build the dependent project into the dist directory of the upstream project alongside the installed python sources/extensions.
    • Different release channels (OS packaging, Anaconda, etc) can specialize from there with various system packaging approaches, but this kind of baseline setup can be a good starting point for the core project, leaving differentiation to others.
    • The Swift side may need to solve the OS-package situation at the get-go.

  • Topic: Improvements to MLIR installability

    • ninja install-mlir should work
    • It would be nice to extricate MLIR from its current nesting as an “LLVM Tool” (groups it under installation for other tools which are unrelated)
    • Not fundamentally “hard” but the last time ~an hour was taken on this, it was more than a quick thing.
    • Likely needs some other CMake attention.
    • Several things need this worked out in the mid-term (@badenh by ~Feb, me in the next few months)
    • @marbre going to look at this

  • Topic: Shared library rework

    • Options include:
      a. Do nothing: likely difficult for downstream projects to add components to increasingly monolithic shared libraries.
      b. Use BUILD_SHARED_LIBS mode: Every CMake library becomes a shared library.
      c. Design the shared library boundaries: Treat shared libraries as a key part of the API and draw the boxes around the intersections that make sense as usable components and dependencies.
    • For (b) this is really mostly a developer feature at present. Might be worth “productionizing” slightly as it is the lowest friction path to get something working (i.e. “If I had a weekend to do this, I’d just use that”). Doesn’t work on Windows today but could probably be made to after a fashion.
    • For (c), this is the canonical way that a project is a good shared library citizen.
      • If drawing the boxes they might include the following breakdowns:
        • MLIR:
          1. core: IR, Transforms, maybe standard dialect
          2. Library per dialect
          3. Execution engine, runners, etc
        • LLVM: Likely need to create dedicated librarys for Support/ADT and demangle.
      • If could be split in this way, MLIR core would not depend on any of the heavy parts of LLVM.
      • Discussed seeing if we could do the CMake plumbing for the LLVM Support library first and see if there is a pattern there that could be applied to the rest. (correct if wrong) @marbre expressed he might be able to look at this.
      • Stella may look at some work to make BUILD_SHARED_LIBS usable beyond dev/across projects as a stop-gap.

  • Next step: A followup meeting before the end of year would be good. Someone not-Stella should speak up and organize.

2

Would be interested in joining the call next time around. I am currently using libMLIR.so for an experimental Julia project so I am interested in having it work well.

Regarding the disabled on windows issue, anyone know why that is? The same is true for the clang shlib, but I have found it to work okay using mingw32 https://reviews.llvm.org/D73156 (We cross-compile everything from Linux).

3

+1 for factoring out LLVMSupport. It might be good to have someone who manages one of the downstream packages to be part of this to make sure that whatever solutions are amenable. I’m happy to be involved too.

4

Iirc, mingw32 takes some lengths to emulate the implicit export behavior of Unix-like compilers/linkers. It doesn’t hide everything but often works out of the box. There are some new cmake features that do this emulation for native vc projects. I haven’t checked the details, but this may be entwined with the discussion to upgrade cmake. I don’t believe there is a hard block here.

I am aware of bugs pending though and there may be others. They tend to come from the fact that vague linkage across DLLs does not work in the same generality as on unix-like systems. I have a TODO in the mlir typeid code that needs a special case for windows (it needs to fall back to type name check via static rtti). I expect there may be another bug or two that will be exposed when trying to combine DLLs containing MLIR code.

5

It didn’t make it into the notes but we discussed wishing you had been there to shed more light on those issues. Any pointers?

6

FYI - I started a thread on llvm-dev related to this: http://lists.llvm.org/pipermail/llvm-dev/2021-January/147567.html