Hey James,
One can effectively do so by introducing a placeholder type. E.g., introduce a type !my_dialect.undetermined and then one can still have functions, operations and run passes on them. Now for most ops today that would fail verification (e.g., op X verifies that it is operating on integer type, and as undetermined isn’t integer, it fails verification - rather than not failing as there is insufficient information to fail) and so it wouldn’t work well with existing ops. For your own dialect it would work fine, but wouldn’t compose as well [perhaps that is fine and you are operating on a “template”'esque level and so these define with ops in your dialect and get instantiated & specialized before lowering to other dialects].
A proposal/discussion that is similar is [RFC] Tensors with unknown element types - #24 by mehdi_amini which could have been used here to say all functions operating on tensors of unknown element & unranked size. One can still represent scalars as tensors, but could be cumbersome.
There is also another option: decoupling type verification from op verification. So that one can have operations that operate on arbitrary types and only verify supported types when it matters (e.g., why not allow std.addi to operate on type !my_dialect.banana and only verify when it matters). This is a little bit extreme, Allowing Types to have interfaces and/or traits (like ops do) is less so [I have an idea in that direction different than the one proposed, with less coupling to the type which I should try and just do a scratch work of so that holes could be poked at 
]
This seems to have come up a few times in different context (thinking of another internal discussion from this past week), so might be worth a ODM discussion actually.
– Jacques