HandshakeToFIRRTL verifications

As we are wrapping up the lowerings for the various Handshake ops, @hanchenye mentioned we should probably have a strict verification for each of the components in this issue: [HandshakeToFIRRTL] Lowering of MemoryOp/LoadOp/StoreOp · Issue #337 · llvm/circt · GitHub.

We thought we’d open up a thread for discussion here to see if anyone in the community has some feedback.

My two cents is it might be worth creating an integration test, similar to ESI, that runs all the lowerings to FIRRTL, then runs the FIRRTL lowerings to System Verilog, and checks the output using Verilator.

The upside is this lets us verify the hardware we are generating does in fact have the properties we care about (latency insensitive, etc.).

The downside is we are testing the FIRRTL lowering path as well. Since this is a functional test, hopefully changes to the FIRRTL implementation don’t break it (too often).

What do others think about testing Handshake at the RTL level?

Another option is lower to FIRRTL, run the FIRRTL lowerings to RTL, convert the RTL to LLHD, and check the output using llhd-sim. I started working on a conversion from RTL to LLHD already, but we would need to flesh that out more to support checking all the Handshake components.

I’m a big fan of creating end-to-end integration tests, so I’d encourage you to setup integration tests for both the handshake -> sv and handshake -> llhd routes. Since it seems like the handshake -> sv path should already be working (for simple designs at least) so I would do that sooner to reveal bugs/gotchas sooner rather than later.

Indeed, we could write a handshake -> sv integration test today (for programs without memory accesses). We can do a basic test first, then add more advanced tests as the integrations mature: a test with loops, a test with memory accesses, a test with Affine loops and memory accesses, etc.

Since you brought it up, I’m curious what your thoughts are @hanchenye. Is testing with Verilator what you had in mind?

I think we all agree that having an integration test sooner rather than later is important, so I’m happy to set up some of the boilerplate to do a Verilator test if others can chime in to help write the actual test cases.

Exactly what I’m thinking! Handshake-to-SystemVerilog and Verilator integration test sounds great to me. I’ll be happy to help on creating some test cases. Meanwhile, some cases from test/handshake-runner may be found useful as well.

In a longer term, something in my mind is a software & hardware co-simulation. In that regard, the Handshake-to-LLHD flow is also interesting to me, as we can lower the hardware part through Handshake->FIRRTL->LLHD->LLVM while the software testbenches directly lowered to LLVM. In this way, we can conduct an integrated simulation with some interfacing codes connecting the software testbench and hardware.

Great, I will start on a Verilator integration test for Handshake, and we can take it from there.

The co-simulation topic is very interesting, and I’ve been wondering if we can make use of ESI for this. I’m definitely interested in digging into this more down the road.

1 Like

I’ve got a working, very limited cosim prototype running in a branch. It depends on some patches and clean ups, so it may be awhile till it gets merged.