@bhack I think we may be talking about two different levels of abstraction, and from that perspective, no I don’t think that ONNX represents a common subset factored at a level that I think is the most important to focus on for this initiative. You asked a direct question, so I’m giving a direct opinion, but I do think there is much more nuance than can be conveyed in such a way.
Like the TensorFlow op set (and numpy, scipy, otherd), it is the union of many fundamental and composite operations from various domains and implies a number of structural conventions, that while popular in present day python-derived numeric computing environments, are by no means universal.
This is a potentially good normalization point if you are trying to build some flavors of present day dominant frameworks, but it isn’t quite the same as saying that the job is to build tooling and representations that underlie and produce meaningful structural simplifications of such frameworks in a way that aids the construction of various compiler functions (i.e. the tool versus toolkit distinction).
I think it is important to focus first on the simplifications in MLIR-core, while organizing things like ONNX, et al to layer on top. We may still arrive at a very wide/comprehensive opset like ONNX in core, but the way we get there is important to me (and I am one voice here, speaking my opinion). In my/TF experience, many useful properties emerge by focusing on the decomposition of components in the toolkit while analyzing/implementing the high level mappings to it.
From my side of the work, we’ve had success with some elements of XLA/HLO and LinAlg as structural simplifications that can be used as concentration points; however the distinction is important: LinAlg was co-designed with MLIR to be such a thing whereas HLO was imported into the ecosystem. When we started the project, we were aware that while HLO got some things right for something like MLIR, it also has a lot of unrelated baggage and assumes some things that were biases of previous representations/goals which it would be good to revisit. We could have chosen to just baseline on it by importing it en-masse but did not – which has given us the design space to explore the structural components.
I’d also point to other threads where the nGraph folks are looking to upgrade the core abstractions (like Loops) which should underpin their ops. We’re also having similar discussions on the shape side as we are struggling to come to terms with how much (and where) to bake broadcasting assumptions into the core abstractions.
That kind of exercise is what I was referring to when I said “gap analysis”. I don’t think we should just be importing wide opsets en-masse without having gone through the process to distill the assumptions they’ve made and map those to core abstractions. As my examples at the beginning with a sample MLP showed (and has been shown in the various proposals for handling dynamic shapes in things like HLO), it doesn’t take very many ops (even things you might take for granted like “add” and “matmul”, “conv”, “einsum”) to interrogate the opinions and levels of abstraction involved – and arrive at useful intermediate design points.
We don’t necessarily need to reduce everything down to non-overlapping primitives, nor do we need to globally select one intermediate abstraction that underlies everything in an area (many things have a small N of reasonable approaches), but if we don’t provide intermediate simplifications as part of the design process, we’re missing an opportunity.
History shows that once an opset reaches a certain span, it grows indefinitely based on the starting core abstractions and it is very hard to change its level of abstraction. I personally want to see MLIR-core be a place where we have a well implemented catalog of such simplification points (and supporting analyses /transformations) more than I want to see it embed the breadth of something like ONNX.
As I’ve said before: I’m super supportive of such a breadth-first thing based on ONNX existing and being done well. As just one voice, I could even be convinced that such a thing should literally be part of the LLVM project, but I just want to see the MLIR-core be more toolkit than tool and am worried that we are skipping steps if we jump straight to the breadth of ONNX (or TensorFlow, or HLO, etc).
