I have a shameful confession to make: roughly 80% of the code of one of the projects I work on lack unit testing. Not even for regressions.

I could blame it to the fact that there are only two programmers in the entire team and we must act as arquitects, designers, implementers, testers, deployers, coffe makers, etc… but I would be fooling myself.

Truth is, we just don’t master the art of convincing managment that time spent on writting good unit test is much better than time spent hunting the cause of bugs customers found.

Or maybe that isn’t really true.. maybe we don’t want to invest time on proper testing, eager to see results fast.

In any case, I was trying to identify practical issues with unit testing, surely just to convince myself that is is way too difficult, but also, at least to some extent, to find simpler ways to do it.

The way I see it, conventional unit testing has three drawbacks:

• It takes so much external code to instrument it.

• It takes so much imagination to come up with significant fabricated input.

• It takes so much time to fix the expected results that match the fabricated input.

Eventually, some fundamental questions came to me: why do I need to create external testing code whose job is to call the subject functions with some fabricated input and validate the output or resulting state? Aren’t functions supposed to have a contract? Why isn’t it sufficient to validate the preconditions, invariants and postconditions inside each funtion itself.

There are plenty answers to those questions, so yes, testing only by means of contract assertion is a naive idea, but an idea that sparks an interesting insight nevertheless.

Typical unit testing code validates results for fabricated input, yet in most cases, the validity of the results can be tested algorithmically, that is, there is no real need to create a fixed map between input and output. That’s what contract assertion is all about.

So, why do we put most valdation code in the unit test instead?

From my experience, some validation code can be highly involved and complex, specially if it must be expressed algorithmically, so it might be just too time consuming. Thus, if such a code is to be embedded into the function themselves, it needs specialized asserts that can be turn on/off in a much clever way that is normally done to distinguish release vs debug configurations. But such smart asserts are missing in a typical toolbox, so one just uses “conservative” asserts.

And why most validation code uses fixed expected results rather than algorithms?

I guess the cannonical answer would be: because by principle, the testing instrumentation must be itself reliable, so normally you don’t depend on the codebase being tested to obtain expected results.

But I’m unconvinced: can’t fabricated expected results be in error? sure they can. In fact, many times I had to invest effort fixing wrongly calculated expected results.

Also, I wonder, how bad it really is, in practice not theory, to depend on other parts of the code during validation. Consider the most nasty case possible: a cyclic dependency.

object decode(encoding e )
{
  object r = some_decoding_algo(e);   

  TEST( encode(r) == e ) ;   

  return r ;
}
encoding encode ( object o )
{
  encoding e = some_encoding_algo(o);   

 TEST ( decode(e) == o ) ;   

  return e ;
}

Is that really bad? There is a cycle, yes, but only two items are involved. If the test indeed fails, and only after that, I can always resort to conventional external unit testing to break the cycle.

So, I started playing with this thing, which I’m calling “Intrisinc Unit Testing“. That is, I’m using specialized asserts to embeed as much validation code I can in the functions themselves. Of course, there are still the external drivers that put the units at play in a controlled, determined context and with fixed input.

The specialized asserts differ from conventional asserts not only in the way they report errors but in the way they determine whether they need to be executed or not (in some next post I’ll give more details on this).

One upside of Intrinsic Unit Testing is that it allows the application to be tested while it is being used: that is, with real data. Of course you still need to run the automated testing because that would cover, if not all, much more paths; but the embeeded validation code can catch errores from real data just like the automated test can.

Another possible advantage of this is that the specialized assertions, given they nature, can become a good mechanism for collecting interesting input data that can be logged and integreated into the test drivers. For example, it is often recommended to add as interesting input every item that had once found to produce an error. This is particularly important, at least to me, because in my experience, when someone finds a bug, even a manual tester, it often reports just the observable defect, and you seldom get a hand on the input that produced the problem.  Test-oriented validation code, unlike conventional asserts, would log as much data as available when a defect is found, from stack trace to input values.