Feature request - add caching of Verilator binaries

[oharboe]

I have a test that takes ca. 10-20 minutes to prepare(Elaboration + generate FIR + Verilator + g++), but the elaboration itself takes only 42 seconds. The test itself is only 3000 cycles and quickly completed.

If Chisel could create a hash after 42 seconds(when elaboration is complete), then that could be used as a key in a cache of the Verilator binaries.

This could bring down the turnaround time for these tests from 10-20 minutes to 60 seconds. Bringing down the turnaround time to 60 seconds would matter enormously, because there is then no need to switch tasks to try to make use of the 10-20 minute wait.

[ducky64]

Interesting. I'm a bit wary about introducing external state (like the hash db) in the core, since there's more potential for failure, and when it fails it's difficult to debug. It might be an experimental / power use feature though. There would also be a not-insignificant amount of plumbing work, both to build the cache, and to refactor the Verilator backend to use it.

Side note: I'm thinking of adding an API for separating simulator compilation and test execution (to address #34), for example:

class BasicTest extends FlatSpec with ChiselScalatestTester with Matchers {
  val precompiled = test(new StaticModule(42.U))
  // or perhaps even manually creating the backend / simulator interface object

  it should "test a thing" in {
    precompiled.test { c =>
      c.out.expect(42.U)
    }
  }

  it should "test it again w/o recompiling" in {
    precompiled.test { c =>
      c.out.expect(42.U)
    }
  }
}

(names TBD, this could be compile and run instead of just test)

In theory, with the simulator interface constructor exposed, this could allow you to specify a Chisel elaborated design with a pre-existing Verilator binary (though there would still be a bit of plumbing and refactoring that needs to happen). The caching layer would be separate, but you could probably build it?

[oharboe]

Having given it some thought, I believe the above design is the right one: it separates the concern of how to cache things on disk from the ability to do so.

[sequencer]

Maybe we can maintain map of (chiselElabroatedCircuit -> verilatorModelFile) as annoation? Let user decide how to cache these thing.

_{Sent with GitHawk}

[oharboe]

To be able to re-use the Verilator binary between invocations of the test would be great during development. It would make for very rapid turnaround time when developing the Scala peek/poke code on integration test when the device under test is not changing. A bit of manual hacking is quite acceptable during this phase.

[ryan-lund]

It's been a while since this is active but I've developed a simple annotation based method (CachingAnnotation) to indicate if the testbench should attempt to reuse binaries between runs. The cache looks at high FIRRTL, elaborated annotations, as well as Scala, Java, and FIRRTL versions. If they all match it then loads a saved store of the simulator command and paths, and then uses the existing simulator binary rather than re-generating one.

In testing on the VerilatorBasicTest suite, it reduced runtime from 35 seconds on the first run to 5 seconds on the cached run.

The code can be found here: https://github.com/ryan-lund/chisel-testers2 and I can make a PR if there's a willingness to include this feature through development.

[oharboe]

Nice! Does this extend to caching elaborstion too?

[ekiwi]

Verilator binaries can now be cached.