Unit tests

[cardillan]

So, today's session of debugging some Mindcode has given me another idea: unit testing of Mindcode.

We already have a processor emulator, so we can run compiled Mindcode. Unit tests might be, for simplicity, contained in the same source file as the code being tested, perhaps in some separate section, perhaps just functions marked with some annotation.

All test functions would be run. If they return anything else than 0, the test would fail and the output of the text buffer would be printed.

The processor emulator can already emulate memory cells and memory banks. In the future, more parts of the Mindustry World could be emulated (mocked). The mocked world would be static - certainly at the beginning.

As a first step, the mocked objects would just return values when sensed (this could be easily provided as a collection of property-value pairs) and would accept any calls. Later the calls might be somehow tracked too.

[cardillan]

Expanding on the idea:

The functionality will be built upon modules (#149). Unit tests will be code blocks annotated with a #test keyword (compiler directive, doesn't pollute keyword space). It can be included in the same source file, or in another module. A module needed for unit tests is loaded using a #require directive (instead of the plain require keyword). Web app won't run unit tests. Command-line compiler will run them by default, but it can be switched off.

Compiling without unit tests:

• #require directives are ignored.
• #test code blocks are parsed (no way to avoid it), but not compiled.
• Putting the test code into separate modules avoids the unnecessary parse.

Compiling with unit tests:

• #require directives are parsed and processed, recursively in the whole tree.
  • Parse trees from regular directives are merged separately and stored for later compilation
  • #require directions in system libraries are not processed (a system library is a library required using the module name and not file name). These will be unit tested whenever Mindcode is built.
  • #test code blocks are compiled as functions (anonymous, so no name clashes with existing functions) and stored in a list of functions to call during tests.
• Parse tree containing regular modules and unit tests is built and compiled
  • No optimizations at all are made - faster compiling. We don't mind slower executions: the emulated processor is fast, and we aim to test Mindcode logic, not the compiler.
  • No practical limit on mlog code size (this is also why we don't want to run optimizations).
  • There will be a limit (maybe configurable) on the number of processor instructions executed (e.g. 10K or maybe 100K steps) to detect infinite loops.
  • There might be an option to switch on optimizations for unit test runs, at least for testing system libraries.
  • Unit test output with #print and #println: when compiled for unit tests, handled just like print and println, otherwise ignored
    • Must not call with arguments producing side effects, e.g. #println($"Status: ${}", status = "error");. Probably won't be enforced.
    • Alternative: create a unit test library (e.g. tests) and have it define tests.print/tests.println functions.
• All #test code blocks are run. When a test fails, the failure is reported and unit test output is printed.
  • Code coverage on the mlog and with limitations on the source code lines can be emitted. Code coverage on the mlog is already implemented.
• When all unit tests succeed, the regular parse tree is compiled again and optimized
  • No need to parse it again - saves time on parse (will be significant once the system libraries are large)
  • New compilation is better than trying to reuse code compiled earlier - the compiler is usually pretty fast
  • Optimization as usual

Mocking the environment:

• Either a standalone mock setup (#mock directive), or part of the unit test definition
• A mock is a static object with properties. The definition specifies a set of properties with values.
• Static properties might be predefined based on object type from Mindustry metadata (e.g. unit's speed or itemCapacity).
• Types of objects to mock
  • Units: will respond to ubind instruction in the way Mindustry Logic does.
  • Blocks: will be accessible in the code using a linked variable name (e.g. switch1), or using @links and getblock.
• Mocked objects properties
  • Mock can define values for sensed properties
  • When a list of values is present, values from the list are returned and consumed every time a property is sensed, the last value in the list is returned after the list is consumed
  • Sensing undefined property causes the unit test to fail
  • Contents of a memory cell/bank might be specified as a list of values, or perhaps generated by a function call in some way.

A unit test might look like this:

// Standalone mock setup
#mock memory(                                                   // Name of the mock setup
    @memory-cell cell1(slots: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)),  // cell1 with its contents
    @memory-bank bank1(slots: setupCell2())                     // contents of bank1 created by a function call(?)    
);

// Unit test
#test(
    name = "Name/description of the test",
    step-limit = 1000,                              // Max. number of steps executed in this unit test
    environment = (
        memory,                                     // Reference to a standalone mock setup 
        @poly(dead: (0, 0, 0, 1), controlled: 0),   // Additional definitions just for this test
        @switch switch1(enabled: (1, 0))
        @sorter sorter1(config: (@coal, @copper))
    ),
)
begin
    // do some tests
    #fail("reason");            // explicitly reports failure 
    #assert(expr, "reason");    // if expression is not true, reports given failure
                                // probably done as a virtual executable instruction

    // Need also a way to test that the output of the text buffer is equal to the expected value,
    // as mlog cannot manipulate/compare texts. Something like this:
    #assertOutput("text", "reason");    
    
    // Maybe also a way to test output of each print instruction separately.
end;

[cardillan]

Unit tests were implemented in version 2.6.0 for the system library. The unit tests depend on the text output created by the test code, which allows testing not just values of the variables, but also actual output generated by the program.

[cardillan]

Some rudimentary support for unit testing will be included in the next release.

The Mindcode compiler will support two new functions: assertEquals(expected, actual, message) and assertPrints(expected, actual, message). These functions get compiled down to special instructions recognized by the processor emulator, which will compare the expected and actual values and will produce a separate output of all executed assertions (both true and false). The assertion results are evaluated by the extended testing tool (#179) and by the command line Mindcode compiler when running the code.

assertEquals works pretty much as expected: the two values can be any expressions, and the assertion fails if the two values do not match exactly (using strict comparison, not the Mindustry Logic one where 1e-10 is equal to 0.)

assertPrints is designed to compare the expected value to the contents of the text buffer generated by the actual expression or function call. For this reason it allows a call to a void function to be used in place of the actual argument, which would otherwise be unsupported. The actual text produced by the tested code is taken from the text buffer

Example of a code utilizing these functions:

#set target = ML8A;
require math;
require printing;

assertEquals(5, distance(1,1,4,5), "distance(1,1,4,5)");
assertEquals(1, sum(1), "sum(1)");
assertEquals(5, median(2,6,4,8), "median(2,6,4,8)");
assertPrints("1,234,567", printNumber(1234567), "printNumber(1234567)");
assertPrints("0", printExactFast(0), "printExactFast(0)");
assertPrints("1.012345678900000", printExactSlow(1.0123456789), "printExactSlow(1.0123456789)");

The special instructions generated by the assertEquals and assertPrints functions are present even in the generated mlog code, which, of course, cannot be used on the Logic processor within the game.

To create a basic unit test, a separate source file containing just the assertions should be created. This file would use the require directive to include the file being tested. The tested files should contain only functions, as any code present in the tested file would be executed before the unit tests, which isn't desirable.

Unit tests would be executed by compiling and running the unit test file.

It's a far cry from a finished unit tests suite, but it still could be useful when developing and maintaining a set of library functions.