The Case for vhdlUnit

Testing. It’s the most important part of engineering, whether its software or hardware. Yet all too often, it ends up getting sidestepped to get product out the door. It is woefully under-estimated in budgets and schedules. Then once the schedule starts slipping, engineers have trouble fighting the urge to skimp on testing to get things “done” faster. Most frustrating of all is the pressure from management to cut down testing hours in project plan estimates, because it “costs too much”, “takes too long”, or “doesn’t add enough value.”

The last 9 months of my life have been enlightening, jealousy inspiring, and ultimately motivating. Through the course of a number of side-projects I’ve been learning and using Java, PHP, Python, Javascript, and Tcl. The most poignant revelation to come from my varied experiences is that rigorous testing doesn’t have to be too difficult or costly to implement, and in fact, most languages have made it so easy to automate unit testing that the only excuse left to not do it is laziness. All of these languages have unit testing frameworks (most notably JUnit and PyUnit), which make writing tests and reporting results beautifully simple.

By day, I’m an FPGA developer and write mostly VHDL. Despite what some software developers might think about FPGA developers, we do quite a bit of unit testing, albeit without a formal framework. And that’s the problem. As schedules start slipping, the stress and anxiety of missing a deadline cause testing rigor to decrease. Ironically, if the project had a formal unit test suite that could be relied upon, stress and anxiety would be lessened because engineers wouldn’t have to fear breaking something in the frantic push to ship on time. They could simply launch their regression tests and review the results.

The most difficult part with engineering that takes place inside a text editor is that once our creations “live” in the physical world, they become a lot more difficult to test and diagnose problems.

Suppose you’re an analog engineer designing a bandpass filter using discrete components. You just got the manufactured boards back. How do you know the filter works? Do you just look at it and say “well, it looks like a bandpass filter, so it probably works.” Of course not. You connect a signal generator up to the input, place a spectrum analyzer on the output, and you sweep across input frequencies and measure the output signal to confirm your filter has the correct attenuation for out-of-band frequencies, etc. If you were to discover a problem, you’d grab a Fluke and check to confirm that the passive components have the proper quantities. This is in essence, a unit test.

Test suites written with frameworks like JUnit give you that warm, fuzzy feeling that all of the little pieces that make up your design are working correctly. This reduces the stress on your engineers, which I believe has a strong, direct correlation with product quality.

VHDL desperately needs an open-source unit testing framework modeled after JUnit. In general, FPGA and ASIC development methodologies have severely lagged innovations in the software world. Kent Beck, the father of Test Driven Development, wrote his landmark Simple Smalltalk Testing: With Patterns paper in 1989. It’s now 21 years later, and there is no vhdlUnit.

So, who wants to get started?

6 thoughts on “The Case for vhdlUnit”

  1. What about PSL’s vunit in VHDL flavour? Not sure about what JUnit really does, but I guess vunit could probably do the same thing for VHDL. Anyone tried this?

  2. I think the problem is that VHDL is too weak a language to build the whole thing it – it can’t inspect itself for example (unlike Python’s many unit test suites). Hence it would have to be written in another language, which then fragments the number of people who will integrate it into their flow and reduces even further those who might contribute to it.

    My current method is to write all the tests in VHDL, and have an external simple test file which lists all the configurations (and their generic params) which need testing. These files exist at every level of the directory hierarchy that i need them and a top-level Python script pulls them all together and runs them one after the other.

    It’s not perfect, but it gets me 90% of where I want to be for big regression runs!

    1. tekphile:

      I want vhdlUnit too. How do you suggest one get started?

      Martin:

      Do you have an example that I could take a look at?

    2. I actually found an implementation of vhdlUnit here:

      http://kik.weii.tu.koszalin.pl/vhdlunit/

      It’s in Polish but Google Translate did a decent job. I downloaded the source here:

      http://kik.ie.tu.koszalin.pl/vhdlunit

      Looks decent. It says on the site that it is open-source. I propose we try to contact the developer to see if we can use it as a launching point? It looks pretty decent in fact.

      It’s even generating reports in HTML directly from VHDL :-P.

      1. stantonk :
        Looks decent. It says on the site that it is open-source. I propose we try to contact the developer to see if we can use it as a launching point? It looks pretty decent in fact.

        Any luck with that? It indeed sounds like a useful starting point.

      2. Indeed… Any ideas? A decent open-source test framework would be very darn useful. After all… This VHDL stuff really is really just software when it boils down to it.

Leave a Reply

Your email address will not be published. Required fields are marked *