 |
IconUnit: A Unit Testing Framework
For The Icon Programming Language
Abstract
This paper describes a unit testing framework for
Icon (website: http://www.cs.arizona.edu/icon/),
a programming language with novel features such as expressions
that may produce sequences of results and goal-directed
evaluation. Most unit testing frameworks are written
for object-oriented languages with exception handling
mechanims. By contrast, Icon is a procedural languange
whose unusual semantics make a straightforward implementation
of a testing framework challenging. The testing framework
uses Icon's sophisticated control flow facilities and
its primitive preprocessor to create a functional, albeit
aesthetically crude, unit testing framework.
Introduction
JavaUnit
Probably the most well-known unit testing framework is JUnit, a unit testing framework
for Java written by Kent Beck and Erich Gamma (website:
http://www.junit.org/).
Using this framework is simple, as the following example
shows:
public class Example extends TestCase {
// The two tests.
public void testAdd() {
assertEquals(3 + 4, 7);
}
public void testVector() {
assert(new Vector().isEmpty())
}
// The test suite.
public static TestSuite suite() {
TestSuite suite = new TestSuite();
suite.addTest(new Example("testAdd");
suite.addTest(new Example("testVector");
return suite;
}
}
The TestRunner program looks for the suite method in the
supplied class file and uses it to run the tests. When TestRunner has
finished, it reports what tests have failed or encountered an error,
along with information about the nature of the failures (like where
each one occured).
The implementation of the Java testing framework is straightforward.
The assertEquals method throws an AssertionFailedError exception if an
assertion fails. The TestCase method used to actually run the tests
catches exceptions and records the exception as a failure or error, as
appropriate. Other test frameworks, like those for Python or Smalltalk,
use a similar technique.
The Icon Implementation
Icon Limitations
The approach used to implement the Java testing framework is unsuitable
for the Icon programming language, however:
- Icon has no exception handling mechanism. If the Icon interpreter
encounters an error, it simply prints an error message and a stack trace
and then exits.
- Unlike conventional programming languages, comparison operators
do not return a boolean value as a result of the comparison. Rather, a
comparison, or any other expression, either succeeds with the
expression's result (or results; see below), or fails. Thus, in
the expression
assert(member(set(), 82))
which tests to see if 82 is in a new (empty) set, the assert()
function will only be called if 82 is a member of an empty set! Since 82
is not a member of an empty set, the member() function
fails, and thus the assert(...) expression fails without assert()
ever getting called.
- Icon does have a macro preprocessor similar to the C
preprocessor. However, Icon macros cannot be parameterized, so there is
no way to define an "assert" preprocessor macro that behaves like
JavaUnit's
assert() method.
The Implementation
The syntax to test to see if an expression holds true (the equivalent
of JavaUnit's assert() method) is obtained by putting the token
"shouldSucceed" after the expression. For example,
member(set(), 82) shouldSucceed
The word "shouldSucceed" is a macro that expands into a use of Icon's
alternation syntax. To understand the alternation syntax, consider the
following example:
procedure SingleDigitPrime(i)
return i = (2 | 3 | 5 | 7)
end
The vertical bar is the alternation operator, and is known in Icon
terminology as a generator, because it can generate more than
one result. In the case of an alternation operator, the result sequence
of the alternation is the result sequence of the expression to the left
of the "|" followed by the result sequence of the expression to the
right of the "|".
The example above operates like this:
- The expression (2 | 3 | 5 | 7) generates 2 as its first result,
and is then suspended.
- The variable i is compared to 2; if they are equal,
SingleDigitPrime terminates with the result of the comparison (2, in
this case).
- If they are not equal, the expression (2 | 3 | 5 | 7) is resumed.
It generates the value 3 and is then suspended.
- The variable i is compared to 3, and SingleDigitPrime terminates
with a result of 3 if they are equal.
- The alternation expression continues to be used to generate
results until either the comparison succeeds or it can generate no more
values. In the latter case, the return statement fails, so
SingleDigitPrime fails because it has no more statements to execute.
The shouldSucceed macro expands to
| (return TestFailure(&file, &line))
That is to say, either the expression preceeding shouldSucceed
succeeds, causing the entire expresion containing "shouldSucceed"
to succeed, or a TestFailure record is constructed and
returned as a value of the test. As a result of this construction,
test procedures cannot return a value -- they must instead
fail if all the checks in the procedure are successful.
Clearly, the shouldSucceed macro is somewhat fragile because of its poor hygiene.
For example, the statement
a = 3 & b = 32 shouldSucceed
is equivalent to
(a = 3) & (b = 32 | (return TestFailure(&file, &line)))
Therefore, the test succeeds even if a is
not equal to 3, since the right-hand side of conjunction
expression is not evaluated if the left-hand side fails.
In practice, this is not a serious limitation. Using a conjunction as a test
expression is a bad idea in any programming language.
The above statement would be better written as
a = 3 shouldSucceed
b = 32 shouldSucceed
so that, should the test fail, the line number of the test failure message
will indicate if it was the value of a
or b that caused the test to fail. All
the other operators with a lower precedence than "|"
(mostly keyword expressions like "while i > 3" and
"1 to 100") "do what I mean" when used with shouldSucceed.
To handle errors in the test procedure (like division by zero), the testing
framework sets the Icon keyword &error
to -1. If &error is non-zero when an
error occurs, the Icon interpreter does not stop program
execution but instead assigns the keywords &errornumber
and &errortext to indicate there is
an error. The test framework clears these keywords before
running each test, and uses their values after the test
to see if an error occured.
Using IconUnit
For comparison, below is the IconUnit version of the JavaUnit test
program at the start of this paper. The "*" operator produces the number
of elements in a data structure, so it is used to determine if a new
list with no elements is in fact empty.
$include "testMacros.icn"
# The two tests
procedure testAdd()
3 + 4 = 7 shouldSucceed
end
procedure testVector()
*list(0) = 0 shouldSucceed
end
# Run the test suite.
procedure main(args)
suite := [testAdd, testVector]
DisplayResult(runTest(suite))
end
The tests themselves are nearly identical to the Java versions, the
only difference being one of syntax. The construction and running of the
test suite, however, is much simpler, thanks to Icon's powerful
aggregate data types.
Future Work
Non-Termination
In the author's experience, it is far easier to write expressions in Icon that
fail to terminate (that is, hang forever) than it is in
other programming languages. Icon appears not to provide
any way to limit the time that an expression takes to
execute. While this limitation has not been a problem
in the small projects it has been used in so far, it seriously
jeopardizes the framework's usefulness in large projects
with thousands of unit tests where the entire test suite
may take several minutes to run. It may be possible to
work around this limitation using MT-Icon, a variant
Icon implementation that permits Icon programs to create
another Icon interpreter that runs some other program.
UI — "green bar"
Unit testing frameworks like JavaUnit typically include
a simple but easy to use user interface referred to as
"the green bar." The green bar is a progress indicator.
When the programmer presses the "run" button, the bar
indicates how many tests have run. Should a problem occur
during testing, the bar changes color, and the location
that the error or test failure occured is displayed in
a pane beneath the bar.
Pressing the "run" button always causes the latest set of tests and
code to be run. This is a problem for an Icon interface, as Icon does
not natively support the linking of code into a running system. The Icon
utility "qei", which interactively evaluates Icon expressions, addresses
this limitation by using the system() call to run another copy of the
Icon interpreter -- hardly an elegant solution. Another possibility
might be to use MT-Icon.
Locating Errors
While test failure messages indicate where the error occured, errors (like division
by zero) simply indicate the nature of the error. It would
be helpful if the file name and line number of the source
code line that caused the error were included in the message,
but Icon appears to provide no mechanism for doing so.
Availability
IconUnit can be downloaded from Bill's home page at
www.ease.com/~trost/IconUnit/.
It is in the public domain.
|
|
