How can we make the Runner UI better?

[kazk]

Let me know if you have ideas to improve the Runner UI:

I'll briefly explain how we get the test results rendered:

1. Each test framework outputs the results in a custom format or some other formats that the Runner can post-process (e.g., JSON)
2. The Runner Service parses the results into a tree
3. The Runner UI receives the tree as JSON
4. The Runner UI renders the tree

We've kept it minimal for years, but it's actually pretty flexible on how to display the results, and it can be extended. For example, by adding a step between 3 and 4 or making it more interactive.

Possible ideas:

• Filtering (only failed, only passed, only logs, etc)
• Collapsing (change the level to collapse the passing group/test)

If you have any questions on how it works in more detail, feel free to ask in a separate discussion.

Please post one idea per comment, so it's easier to discuss. We can also move to independent ideas after discussing it here first.

[ejini6969]

I prefer the collapsing way (It sounds more easy to implement than filtering as the usage of it blocks can already made multiple tests collapsed together). Just my humble opinion, correct me if I'm wrong.

[kazk]

Yeah, collapsing will be easier to add because we just make that configurable and/or change the default behavior.

Filtering can complement collapsing. Maybe toggling by types using buttons at the top.

I also wanted to know if there are any other ideas. Those two were something we (Qualified) already had come up with (it's been on our list to do).

[ejini6969]

Actually I'm not sure how filtering can complement collapsing. Is filtering meant to filter by displaying error test cases or correct ones or both depending on the situation? And toggling means what? Like those expandable and minimizable features / buttons?

And I don't have much more ideas thoo since I haven't get in touch with industry mechanism that deeply and thoroughly.

[kazk]

Yeah.

Filter by button/toggle, can have multiple on:

[passed] [failed] [errors] [logs]

The depth to start collapsing passing group made configurable:

collapse_passed: 0 (depth)

---

For example:

- Group 1
  - Test Failed
    - Log
    - Failure Details
  - Test Failed
    - Log
    - Failure Details
  - Test Passed
- Group 2
  - Test Passed

(the passed tests have logs too, but the test case is collapsed)

When configured to collapse passed groups from depth 0:

- Group 1
  - Test Failed
    - Log
    - Failure Details
  - Test Failed
    - Log
    - Failure Details
  - Test Passed
- Group 2

Filtered to show only failed tests:

- Group 1
  - Test Failed
    - Log
    - Failure Details
  - Test Failed
    - Log
    - Failure Details

Filtered to show only passed tests:

- Group 1
  - Test Passed
- Group 2
  - Test Passed

Filtered to show only logs:

- Log
- Log
- Log
- Log

[hobovsky]

My main grudge with the test output panel is that it encourages (or even sometimes enforces) writing non-idiomatic/suboptimal test suites. Main features I see responsible for this fact are: performance of the UI, and verbosity of the output.

---

Disclaimer: on a daily basis I work with NUnit and JUnit. Sometimes I use them for TDD, sometimes for fine-grained unit tests, but usually to create tests of quite high-level SUTs to provide test coverage and safe refactoring of a legacy code base. I do not have too much experience with BDD frameworks or property-based testing frameworks outside of CW.

---

In my work I use things like [TestCase], [TestCaseSource], [Values] and [ValuesSorce] NUnit annotations a lot. I like how they provide a separation between definition of tests and actual testing logic. When working with Codewars tests, my wheels grind a little when I see test suite like this:

[Test]
public void FixedTests() {
    Assert.AreEqual("I", KataSolution.ToRoman(1));
    Assert.AreEqual("V", KataSolution.ToRoman(5));
    Assert.AreEqual("C", KataSolution.ToRoman(100));
}

[Test]
public void RandomTestsWithFullCoverage() {

    int[] inputs = Enumerable.Range(1, 4000).Shuffled().ToArray();
    foreach(var input in inputs) {
        Assert.AreEqual(ReferenceSolution.ToRoman(input), KataSolution.ToRoman(input));
    }
}

Basically, in many test suites of CW kata, something what is supposed to be a test actually is a set of tests. Something what is considered a bad practice, antipattern, and rarely seen in the wild.

I am pretty well aware that tests of a CW kata are (can be) a specific topic, and not all CW use cases can be easily expressed with frameworks for unit test, but these would be probably more of an exception rater than rules.

Ideally, I would see the above test suite broken down: FixedTests would turn into three separate methods, or a method with [Values] or similar. RandomTestsWithFullCoverage would turn into a method with a parameter and [TestCaseSource]. But it would not work well on Codewars.

Verbosity

Generated test cases make the output very noisy. Authors like looping over inputs in a test because it keeps output small.

Log entry per assertion

This one I think is a real WTF. Having a Test Passed per every passed assertion is, umm... not very helpful. I think there might be some technical reasons behind this, but it's a pain.

One collapsible header per test (it)

With generated tests, when all of them pass, users are hit with a green wall of text. All of the titles are usually identical, sometimes authors put information about inputs there (which is also sometimes problematic, see blow). But generally information about each and every individual passed test is not really necessary. I think it would be enough to have just a summary of passed tests in a group ("4000 tests passed in 3.2s" as a summary of a describe, or something like this).

On the other side, it could cause problems with existing test suites. Some of them might not look good in such set up, or maybe sometimes users/authors would like to have a full overview of test results. If test output would be filtered, I think it would have to be performed by user's request, and not by a config entry in kata or general setting for the framework.

Another situation are failing tests. When tests fail, you usually like to have as many details as possible. But it also means that a solution with a bug produces 5k red assertion messages, and some frameworks (Elixir?) add a ton of additional junk, stack traces, and whatnot. Many frameworks also miss the "stop on the first failure" functionality. I do not have a good idea for handling and presenting large amount of failed tests in a good way. But maybe it does not have to be handled? Errors are errors, so fix them and try to get to the clean, green output.

Performance

Even if we got a good idea of presenting and filtering tests, the problem is with browser performance. There will be scenarios when generated test cases would produce significant amount of output, and test output panel does not like this. When user toggles from "filtered" to "full", or when there's a bug in submitted solution and many red entries would be produced, the panel would die.

Other presentation issues

For some tests I like to put inputs in the test headers. Some frameworks do this automatically, and title of the test case generated by the framework cannot be modified easily. It works well for very small inputs, like a single number, but for something even slightly more complex, like an array with just a couple of elements, titles start to look ugly. And I do not mean that they are verbose, because this behavior can be limited only to, for example, fixed tests with very small arrays or edge cases. But I do not like how arrays are presented in titles with current font and styling.
This problem could be solved if authors have possibility of overriding a test title, but I am not sure if it's possible in every framework. NUnit allows for this when using [TestCaseSource] which returns TestCaseData, but AFAIK titles cannot be customized for [TestCase] or [ValueSource]. Have an array as an input, even a small one, and things start to look ugly.

Edit: uh, pushed the button too early, forgot to about another point:

CW - specific testing

There are some testing scenarios which might not go well with use cases the testing frameworks were designed for: random tests, and kata performance tests.
Kata performance tests, especially when not created optimally, tend to generate a shitload of inputs in a loop which goes over nine thousands. I am not sure whether performance tests implemented in terms of dynamic test cases would be a good idea, because:

• testing frameworks handle scheduling of tests, what might affect performance (framework which uses parallellization might require four times amount of test cases than framework which does not)
• performance tests will create really bad output, especially on failures.

[hobovsky]

Can you create a kumite and show me how it currently looks like?

Here.

[kazk]

NUnit output is currently based on its XML output because when I added .Net Core support, Event Listeners didn't work as documented. That's why we can't see partial results either on timeout/crash. It'd like to try again and improve when adding the new C# version.

What's your ideal output for this?

[hobovsky]

I am not sure if that's what you're asking, but I'd like something like this (forgive me my Paint-art):

Individual passed test cases would not be there at all and it would not bother me. But, if output panel would be able to handle them, they could appear after expanding the most nested group.

When there are failed tests, they can be presented individually:

Problems with UX start when there's many failed tests, for example solution is buggy or user submitted just the initial solution. 1000 failed tests do not carry too much information either. When this happens, maybe just expand a couple of them, and keep the rest collapsed?

Generally my opinion is similar to Johan's: successful test cases should be not even visible (either not present at all, or filtered by default, or collapsed out of view, whatever). Failed test cases, on the other hand, should be present, but I am not sure how to handle large amount of them.

[kazk]

Started working on some improvements and bug fixes. Here's a summary of what I did over the weekend (not deployed yet):

1. Fixed the triangle glyph displayed as emoji on some mobile browsers by using SVG icons instead
2. Added filtering test case by results (passed / failed / error) controlled by clicking the summary at the top (see below). The filter persists while solving a kata, so you don't have to keep clicking after each test.
3. Changed to collapse passing nested test groups (non-root test groups)
4. Performance improvements. To prevent freezing, rendering of test results is now delayed. Child nodes of the collapsed nodes (e.g., passing test case and nested test groups) are not rendered until expanding. Previously, it was rendered and hidden with CSS. When there are more than 10 child nodes, those are rendered only when it comes near the viewport (i.e., rendered as you scroll).

test-filtering.mp4

With the filter, a solution that fails only when the input is 0 tested against Hobovsky's example with generated tests increased to 10K looks like this:

[kazk]

Even if we got a good idea of presenting and filtering tests, the problem is with browser performance. There will be scenarios when generated test cases would produce significant amount of output, and test output panel does not like this. When user toggles from "filtered" to "full", or when there's a bug in submitted solution and many red entries would be produced, the panel would die.

Toggling the filters of 10K test results takes a few seconds, but should be usable.

filtering-10k.mp4

filtering-large.mp4

[hobovsky]

Performance

Yeah, it's not optimized at all. Jake wrote it years ago, and I've recently refactored a bit (removed like 70% of the code), but that's about it. This might need a significant rewrite, but it's not a huge app.

If performance of the output panel would be improved without filtering, I take it. I personally am not bothered by the verbosity of the test output, and when creating test suites, the only thing which stops me from creating a test case generator of 10k inputs is performance on the client.

Even if test output panel would display 10k collapsible blocks for every it, but would do it without killing my browser, I take it. For me, creating a good/idiomatic test suite is more important than presentation. Presentation can be improved later, and poorly written test suites will pile up. Besides, I am probably used to reading (or skimming over and ignoring) large reports :)

The only problem I see is hunting this one failing tet case among the long list of passing ones. But I think it's not a very common scenario. In majority of cases, either everything passes, or a noticeable part fails. Except when crappy rounding in reference solution, or something like that.

[hobovsky]

Also related to test output panel: test runs which crash, get aborted, or tests which pass, but not: #69

[JohanWiltink]

I have crashed too many not just panels, not just browser tabs, but whole browsers and even complete PCs on verbose test results of many tests.

Tests really should quit an it-block on failure ( which might be the user solution throwing where not expected ). If existing frameworks don't do that for certain languages, have authors program it in. Can't be that hard. People don't complain they have to do their own memory management in, like, C; they can do their own test management.

Hobo doesn't seem to agree, but I am very much in favour of multiple asserts per it ( and multiple its per describe ). The lowest level of clustering should not create output, so you can have 10 000 tests without having 10 000 lines of output ( headers, logging or otherwise ). Because I've lost work to having 10 000 lines of output. Collapsing by default 10 000 lines doesn't help - under the waterline they're still in memory - they need to not exist. Maybe I don't understand how frameworks are supposed to be used, and maybe I don't actually have to have multiple asserts per it - the thing I want is an invisible lowest level of clustering. For succeeding tests. For failing tests, see above paragraph.

[hobovsky]

To be clear: my concern is not amount of asserts per it, or amount of it per describe, but amount of invocations of a user solution per it. With some rare exceptions, "industry best practices" state that each test should invoke a SUT (subject under test) only once (for Codewars it would more or less translate to one invocation of a user solution per it). When SUT returns, test can do as many assertions as necessary to verify its correctness.

What I do not like, and what I fight against, is the fact that Codewars enforces bad coding (testing) practices, and makes it difficult to apply good coding practices.

Because I've lost work to having 10 000 lines of output.

Well, I have some very specific opinion on trusting results of my work to a web browser or a web page. But maybe that's because I'm just old :) It's out of scope of this discussion though. Besides, if these 10 000 lines could be handled gracefully, it would become a no-argument.

Collapsing by default 10 000 lines doesn't help - under the waterline they're still in memory - they need to not exist.

I am not sure about this. Not being able to keep 10k of data entries in memory sounds like a WTF on itself. Like, seriously? Since when is remembering 10k of anything a problem to a computer?

I understand that things can become challenging when having thousands of UI elements representing the data, but this problem is long solved: virtual views, on-demand loading, remote data sources... I think it should be perfectly possible to have thousands of test results in memory or on a server, and put in the DOM tree only the viewed ones. I am not a front-end developer though, so maybe my view is too optimistic :)

[JohanWiltink]

In a perfect world,I too would rather have 10k its, collapsed.

But ours is an imperfect world. Several of my computers ( cq. browsers ) have crashed on "just" a thousand it-headers ( running tests repeatedly. some kind of memory leak probably ), and I am not in a position to replace all of them.

In the real world, few its with lots of asserts is less of an assault on limited available resources.

[kazk]

With some rare exceptions, "industry best practices" state that each test should invoke a SUT (subject under test) only once (for Codewars it would more or less translate to one invocation of a user solution per it). When SUT returns, test can do as many assertions as necessary to verify its correctness.

Hmm, this might be "best practice" in some languages/test framework/"industry", but I don't think it's as universal as you think. Searching for unit testing best practice doesn't show any definitive answer. It depends.

One concept/requirement/feature per test case is fine in my opinion. As long as it's easy enough to identify what's broken, it's isolated enough.

To be clear, I know some kata have a test case like works with 100 assertions. I'm against them.

[awesomeAD1]

+1 for filtering.

A suggestion I'd have pertains to logging. 100% of logging is related to debugging. 99% of debugging needs to happen on failed tests. Usually, I only need to see the inputs. In the best case, inputs are small. Often they're not. If I'm lucky, I won't exceed the 1.5 MB limit logging all inputs of hundreds of tests, most of which pass and are therfore uninteresting, but even in those cases, I now have to scroll and look for that tiny red blip in a sea of white text. In the worst case I can't debug at all because I can log everything or nothing.
So it would be cool if there was automatic logging of inputs that I could display on a failed test, and only there. Perhaps they could be initially hidden in a collapsed textbox, or even dynamically requested when the user clicks on it (if that's technically feasable).
Bonus: add a small "copy" button next to a test that copies log data to the clipboard. I'm fairly certain I'm not alone in preferring to code in my local IDE and doing so manually is often finicky.

On a related note, it would be nice if the UI clearly grouped the test result (e.g. "xxx should equal yyy") together with any logging I performed. Currently, the log messages are in their own framed box above the result, and even to this day I often have to pause for a minute and remember if the result applies to the textbox above or below. Extending the frame to include the test, with a divider between them, or something similar would improve the UX. It's a small thing, but small things take a UX from good to great :)

Final note, regarding @JohanWiltink 's remark on multiple asserts per test: I disagree. It's not helpful when a test tells me "You've failed one ore more of 5 things. I won't tell you which though." I thought this was a bad design choice on the test author's part, not related to the test runner itself.

[awesomeAD1]

Sure, but it's up to the test author to make sensible use out of it, no? I've had a few katas where multiple asserts where grouped under a single message and didn't distinguish the exact point of failure. As I said, probably nothing to do with the test runner, and more related to plain bad test authoring.

[JohanWiltink]

True, and I have been and still am guilty of that myself.

But shouldn't that problem be fixed where the problem is, instead of elsewhere?

In my defence, Haskell shouldBe ( ie, assert.equal ) has no custom error messages ( at all !! ). Sometimes I write my own, sometimes I don't. Haskell makes it .. ++UnEasy .. to have optional arguments. Maybe I should look into writing a shouldBe that optionally takes a custom error message.

[awesomeAD1]

But shouldn't that problem be fixed where the problem is, instead of elsewhere?

Totally, so it's not within the scope of this discussion. Except maybe that the test framework could perhaps enforce proper authoring or, conversely, disallow ambiguous messages. I'm not familiar enough with the system, not having authored tests myself. So I'll remove myself from this discussion :)

[hobovsky]

+1 on better grouping of outputs coming from a single tests. Newbies trip up on this all the time, mismatching logs and related assertions.

[kazk]

So it would be cool if there was automatic logging of inputs that I could display on a failed test, and only there.

This depends on the test framework and how the tests are written. So it's out of scope. There's no way to do this universally.
A better assertion library may solve some issues about finding the input. For example, some test frameworks and assertion libraries can show something like this on failure:

  assert(ary.indexOf(zero) === two)
         |   |       |     |   |
         |   |       |     |   2
         |   -1      0     false
         [1,2,3]

[kazk]

+1 on better grouping of outputs coming from a single tests. Newbies trip up on this all the time, mismatching logs and related assertions.

@hobovsky Can you elaborate? I know the problem exists, but how can it be improved?

The logs from a solution should be shown under the relevant test case when possible (test framework dependent). Logs come before the assertion failure message because tests call the solution before testing the result.
If you mean to associate the log with the assertion somehow, the test probably need to add some marker to indicate the grouping.

print("<---::>") // some marker
assert square(2) == 4
// imagine multiple assertions passing

print("<---::>")
assert square(3) == 9

The last log (if any) is associated with the failure

- collapsed logs
- log
- assertion failure

[hobovsky]

I mean here the problem which this paragraph of the FAQ attempts to explain. Users often mismatch the log (values they printed) with assertion message, and they raise issues about buggy tests because they are convinced that input X should result in a green line, but panel shows red. But it turns out that they look at the wrong assertion line (above the log).

I do not know what would be the best way to solve it, because i'm not UX expert, but I think that just some kind of a small visual hint, like divider or spacer, which would group elements from the same test case and separate elements of different test cases, would be enough. I think that the confusion results from the fact that heavy text of assertion looks like a header, and not like a summary, so users mentally place it at the top.

If you mean to associate the log with the assertion somehow, the test probably need to add some marker to indicate the grouping.

I am not really sure what this sentence implies. "tests probably need to [do something]" means that the author should print CW-specific markers, or that test framework should take care of this?

[kazk]

"tests probably need to [do something]" means that the author should print CW-specific markers, or that test framework should take care of this?

Yeah, I don't know how else to associate logs to assertions without markers.

assert solution(1) == 1 # pass
assert solution(2) == 2 # fail

will output

1
2
<FAILED::>

There's no way to know if 1 was from the first assertion. Did I misunderstand you?

---

group elements from the same test case and separate elements of different test cases, would be enough.

It already groups by test case (in most test frameworks). The FAQ example isn't using it, and it's much more confusing because of how Python test framework shows passed assertions :(

[awesomeAD1]

@kazk consider:

vs

Console output and test result are clearly grouped together. For some reason it seems (to me and apparent others as well) intuitive that the log should appear after/below the test result. But yes, since a test calls a solution before it can be evaluated, the console output appears first.
Purely from a HTML/CSS standpoint, it should be trivial to switch them around in the rendering if this is indeed preferred. In any case, the frame around console output should grow to encompass the test result.

[kazk]

@awesomeAD1 Yeah, I agree with that. I was just thinking how to implement something similar for proper test frameworks that doesn't log the successful assertions because the logs will be combined.

Currently, Python will output like this:

- test case
  - log
  - passed
  - log
  - failed

This can be styled like your example.

Other languages with proper test framework outputs like this:

- test case
  - log + log
  - failed

The first log is from the assertion that passed. There's no way to ungroup log + log because it's impossible to know where to split. I thought this would be confusing, but I realized I can apply the style only when there are multiple failed/passed messages within a test case.

[kazk]

Tests really should quit an it-block on failure ( which might be the user solution throwing where not expected ).

@JohanWiltink Can you elaborate on this? I'm not sure what you mean by "quit an it-block on failure".

[hobovsky]

I think it means "exit early, on the first failure". As opposed to, for example, Python (which by default does not stop on failed assertions), but most importantly, to generated its: with test-in-a-loop, tests always stop on first failure. With generated tests, you always get lots of them.

Maybe Johan can confirm, but that's what I think is the most favorable argument towards loops in it blocks.

[JohanWiltink]

That is what I meant.

QuickCheck does many tests in a single it, and it will stop ( and fail ) the block immediately upon any failure.

Related: being used to that, I don't see the problem of calling the user solution multiple times in a single it-header.