Releases: joakim-brannstrom/dextool
Hematite
There are no new major features in this release but rather small improvements to the existing features. The most significant change is that this release contains a crucial bug fix when using a newer D compiler.
Fixes
- Bindings updated to support llvm-16 and 17.
- Fix failed parsing of internal compiler flags when the console languages
isn't English. - Fix segfault with dmd-2.106.0+
- Officially dropped support for llvm v4-13 because they are no longer tested
by CI. They may or may not work.
Fixes for dextool mutate
- It is problematic to inject mutant schema in C++ templates because it may be
highly likely that the resulting schema fail to compile. If it is likely or
not depend on the source code. Sometimes it works fine, sometimes it doesn't
Because dextool mutate is intended to be generically applicable this release
changes the schema generator to be conservative. If it is a C++ template
then do not inject a schema code in them. This may result in slower mutation
testing for some and faster for others. If it turns out to be a problem this
behavior may become configurable. - The unique identifier calculated by the relaxed algorithm didn't work
correctly for mutants in headers. It often lead to duplication of mutants
which lead to a misleading mutation score and made the tool many times
slower than it should have been. - Stopped generated statement delete mutants of
return;
because those
mutants overlap in behavior with other mutants thus they are redundant. An
example of such a mutant is the early return programming pattern.
void fn(int x) {
if (x > 42) { return; }
/* other stuff */
}
The mutant x>42
to false
is the same as deleting return;
. Because
mutation testing is a somewhat slow process it is better to generate less
mutants wherein every mutant that is generated are useful (exhibit a new
behavior giving useful insights to how the test suite can be improved).
- Add support for textual control structure in C++ such as
and
,or
. - The CR mutation operator replaced floats with an integer
0
which may
result in compilation errors. Fix now by replace with0.0
. - Fix schema phase that could stop prematurely when the database is accessed
from multiple instances in parallel. The database locks in that case but do
not retry at a later time which lead to the schema not finding any files to
inject the schema runtime in. - Remove deletion of whole if-statement. They are pretty redundant considering
that the bodies of each branch are deleted in turn. Less mutants that are
"productive" is better. - Timeout scale factor is now always applied in the schema phase.
- Fix bug where the timeout worklist wasn't purged of mutants that no longer
had thetimeout
status. - Add support for the spaceship operator in C++. Should improve the success
rate of compiling mutant scheman which in the end mean that the time to
finish mutation testing is faster. - Major fixes to mutant schema generation. Many of the scheman led to
compilation errors. All known problems have been fixed which should speedup
mutation testing alot. - Update jquery to v3.7.1 to fix three potential security vulnerabilities.
- Allow analysis of more complext code by not discarding the whole AST because
some AST branches have a "too high" depth. - Fix crash when running
dextool mutate test
with schemata activated but
there are no scheman.
Limestone
Lots of changes. The user visible highlight is the significant improvements to
the HTML report. Everything has been improved from the overview to the file
mutant navigation. It is now a much more pleasent experience to read the
report.
The significant behavior change with this release is the removal of --mutant
.
Allowing test
and report
to use other mutantion operators than what where
used during the analyze phase both resulted in a higher implementation
complexity and surprising user behavior. The new behavior is that what has been
analyzed is tested and reported. It is both more logical as implemented and how
it is used.
Fixes
- Updated support to llvm-14. This fix should make dextool automatically work
with future versions of llvm as long as the C bindings that are used by
dextool are unchanged. - Bindings updated to support llvm-15.
New features for dextool mutate
- Annotating each line with
NOMUT
for mutants to ignore may end up being too
much clutter in the source code. It may even be unfeasible to add on the line
because it may become too long. Therefor two new annotation types are added.
NOMUTNEXT
to ignore the mutants on the next line andNOMUTBEGIN
/NOMUTEND
to ignore mutants in a whole block of code. - (html report) Auto detect when javascript is deactivated to print a warning.
This is mainly for Jenkins servers wherein the security settings may have
activatedCSP
. The HTML report look really bad and isn't working as
expected. For a new user it may seem like it should be "this bad". The auto
detect thus print show a warning to the user and a hint of how to solve it
if CSP is the problem. - Add support for constant replacement of floating point numbers.
- (html report) Improve the sortable tables
- (html report) Improve how mutants that span multiple lines are displayed.
- (html report) Remove buggy treemap.
- (html report) Display an explanation for the mutation operator abbreviations
when hovering with the mouse. - (html report) Display how an individual files mutation score have changed
compared to the last 7 days average. This is to help in identifying negative
trends. - (html report) Display coverage information.
- (html report) Move help text and explanations to a popup dialog (I) to
reduce the clutter. - (html report) When clicking on a mutant choose the smallest mutants under
the cursors. This is to fix the annoying behavior wherein the first mutant
where chosen which could be one that span multiple lines. It is hardly ever
that one that is of interest. If the mouse is over e.g. a+
then the
expected behavior is to show the AOR mutants under the cursor (-
). - (html report) Deduplicate mutants that have the same source code
transformation to make it easier to find the relevant mutant to look. - (html report) Fix scrolling in a file report. Scrolling with the mouse do
not change the mutant except when the mouse is over the window showing the
mutants on the current line. - (html report) Fix the web browsers "back" button when looking at a file report.
- The CLI parameter
--mutant
has been removed fromtest
andreport
command groups. - (html report) The test cases in the info box in the file report is now a
link to the detailed test case report. - Automatically trigger a full analyze when the configuration is changed.
Fixes for dextool mutate
- (html report) Fix sortable tables in chrome.
- Coverage info could accumulate infinitely in the database, wrong foreign key.
- Changed the checksum of mutants from 128bit to 64bit to simplify the DB
schema, implementation and slightly improve the performance. - Fixed a bug when sorting mutants by datetime which resulted in e.g. "testing
oldest mutant" not working as it should. - The trend graph for file changes is redesigned to show when a files mutation
score last changed. It should make it easier to understand why the mutation
score have changed because it points to the files that are changed. --load-behavior
halt sometimes failed because the threadpool used for
tests did not properly shutdown. Added a background thread that triggers after
10 minutes to forcefully shutdown dextool. This is mainly an issue when
dextool is running on a shared CI server and it becomes overloaded. It is hard
to reproduce but the server probably run out of memory.- Fix coverage bug where a mutant where not re-tested if only tests changed
such that the mutant is not covered. - Improve scheman. Moved the schema generator from analyze to test phase to
make use of the internal worklist. The worklist contains what mutants to test.
The scheman are now built up from fragments containing mutants that are in the
worklist. This result in more mutants that are tested for each schema. A rough
test show a 50% reduction of the number of scheman needed which mean a
sometimes huge reduction in test time. A side effect of this is that the
analyze phase use a lot less memory. - Analyze phase where not stable. The same mutants where not saved to the
database even if no source code where changed. Luckily the bug where in the
SQL schema definition which prohibited the same mutant kind to affect one and
the same source code location.
Deep Steel
Fixes
- Fix compilation with dmd-compiler (2.100.0).
- Removing unused fuzzer plugin.
New features for dextool mutate
- test_cmd_dir where previously always searched recursively but it has turned out to be a problem because some test suites contain executable test data. Thus the new configuration option test_cmd_dir_search is added to allow the search to be configured to be shallow (non-recursive).
- Improved speed when using diff mode by skipping the coverage instrumentation. The user has already specified exactly what to test thus it shouldn't be necessary to run it through the potentially slow coverage run. It is assumed that a diff is usually pretty "small" and the change have entry point coverage thus coverage data wouldn't help.
Fixes for dextool mutate
- Update moment.js to fix security vulnerabilities.
Impact
This vulnerability impacts npm (server) users of moment.js, especially if user
provided locale string, eg fr is directly used to switch moment locale.
The vulnerability have a low or no impact on dextool as of date but problems should be fixed. - Analyze never finish if the number of threads are set to one (--threads 1). It is now changed to be a minimum of four.
Cat Gold
Fixes
- Fix compilation with latest dmd-compiler (2.099.0). Required the min version
to be updated too though because the older one segfaults when compiling
miniorm.
Fixes for dextool mutate
- Schemas may, and do, affect the runtime of a test suite. This mean that
the timeout need to be measured and adjusted per schema. This feature exists
and is done but the timeout where set after all test workers where spawned
which meant it had no effect. The fix is to send a signal to all test workers
to update the timeout, if it is needed. The end result is that schemas should
result in less mutants withtimeout
status.
New features for dextool mutate
- Add an alternative algorithm for calculating IDs for mutants (
relaxed
).
The new algorithm is scope aware. Mutants are unique within a file and
within a scope. Only mutants within a scope are re-tested if the source code
in the scope is changed. This thus basically mean that the algorithm works
like "git diff". It only test mutants in scopes that have changed. It makes it
faster to test a code change and not lose the report for the whole program
compared to a "git diff" that only test and report for the changed lines.
The option is activated viaid_algo
in the configuration file. - Changed the default ID algorithm to
relaxed
. After extensive testing it is
now mature enough to be used by default. Its main benefit is that diff based
testing of changes are activated by default which reduces the overall test
time when small changes are made to the source code.
Polished Gem
This is a long overdue release. The focus have been on polishing existing features and improve the user integration and interaction.
This release have now been battle tested in production for two month without running into any significant problems thus it is time to release an update.
There are still lots of features to fix and improve in the future but those will require some re-engineering so stay tuned for future fun.
Fixes
- Fixed build instructions and cmake config to allow greater control over what
llvm/clang libs are linked. This allows dextool to be built for Fedora-34. - Support llvm/clang 13.
- Fix compilation with latest dmd-compiler (2.097.2)
New features for dextool mutate
- Add a page to HTML report which show the worklist. This is to help a user to
understand what is being worked on by the tool. - Parallel testing of mutants in a schema. Previously the total number of
cores where not fully utilized because only one mutant where tested at a
time. This usually meant that there ended up being idle cores because the
test suite had one or more slow tests that "held back" the testing of
mutants. This is now fixed by running multiple mutants at the same time
which mean that there will probably always be tests that are being
executed on all cores.
From some casual testing this lead to 2-10x faster testing.
Fixes for dextool mutate
- HTML file report could fail. Added code to catch exceptions during report
generation which should fix the problem. - Running parallel instances didn't work that great when testing mutants in
large schemas because an instances didn't update the mutants to test with
those that had already been tested by another instances. This has now been
fixed by polling the database each minute to remove those mutants that have
been tested.
Multiple instances have also been changed to test the mutants in a schema in
random order. This should further reduce the "overlap". The final result is
that for a schema that have 500 mutants and there are 5 instances running in
parallel then each should only need to test ~100 mutants each. - Retest of old mutants could get stuck on those marked with
noCoverage
because the logic would add them to the worklist during initialization and
then be removed by before the test phase started. It
is now changed to only add mutants not with thenoCoverage
status to the
worklist. - Populate the worklist with re-test of old mutants if it is below the
specified retest number. - Fix memory overload check. It failed to parse
/proc/meminfo
. - Block schemas inside constexpr template functions. This reduces the number
of failed schemas thus improve the overall speed.
This fix is only available with clang-12+. - If
test
is run without finishing and thenanalyze
is ran the handling of
timeout mutants will lead to an ever increasing list.
The system basically live lock over time. This is now fixed by tracking the
timeout iteration per mutant instead of globally. - Fix an infinite recursion bug that lead to crash when analyzing. Happened
for example when analyzing llvm. - (html) The trend graph would look like a "hill" /\ when there where a
significant amount of mutants with the timeout status while the timeout
algorithm where still testing them. Because first they where classified as
timeout (killed) and then later on changed to alive when the timeout where
increased. This where detrimental to a teams "motivational drive" to work
with mutation testing and improving the test suite. They expected the first
sample, which showed an improvement, to be "true" and which lead to a
positive feeling. Then later on it started to go down without the team
actually "doing" anything with the test suite.
The change is to only update the trend score graph when there are no timeout
mutants to re-test. - (html) Make the trend graphs prediction color "robust" to small changes
(fluctuations) by using grey color when it is below one 1%. This thus
mean that when it is red the team know that "we have to act", and vice versa
when it is green they know that "it is going well now!". - A binary that contains a mutation schema could sometimes result in a
significant slower test suite. This would lead to a high number of mutants
being initially tagged as "timeout" which slowed down the mutation testing
process significantly. This is now fixed by updating the test suite timeout
by measuring the test suite with the schema injected. - Fixed implementation of pointer aware ROR when generating mutants for
==
and!=
.
Smooth Road
New features for dextool mutate
- Speedup the test phase by skipping mutants inside SDL mutants. Because a SDL
delete the code it is higly unlikely that mutants inside such a region would
be killed. This is basically an extension of using coverage to skip mutants
to test. - Automatic classification of test cases as either unique, bad or buggy. It
hopefully makes it easier to read the test case report and fast find the
interesting test case for further inspection without having to read all test
cases in detail. - (html) A page with all mutants, their status etc has been added.
- (html) Allow a user to provide additional metadata about test cases such as
a link to the implementation and text about the test case. This is in a
separate json file. The intention is that tools are used to extract the
information from the implementation of the test cases.
[{"name": "tc_1", "redundant": true, "text": "<p>a text with <br> html <a href=\"foo.html\">links</a>.</p>", "location": {"file": "../../tc_1.cpp", "line": 42}}]
- When
test_cmd
s are executed the host could run out of memry. It resulted
in either the tool hanging, the linux OOM killer reaping processes etc. In
summary the system and tool became unstable. The tool now have support for
specifying the max total used memory of the host via
max_mem_usage_percentage
. If the limit is reachedtest_cmd
s will be
terminated. - Use
__builtin_expect
to improve performance when executing binaries with
schemas by marking all mutant branches asunlikely
. It may reduce the test
suite execution time when running with schemas by up to 20%. - Somewhat solve the problem of scheman that fail to compile by letting a
failed schema influence how future scheman are generated. "Adapt" future
scheman based on passed failures. This allows the schema generator to be
less pessimistic about pointers because a code base that use "pointer
tricks" which lead to many failing scheman will over time adjust how the
scheman are generated. - Speedup testing of heavy templated libraries by allowing scheman to also
cover the templated functions. This may initially lead to many failing
scheman but this will lead to adaption of how the scheman are composed which
should over time self heal. - The slowdown mode now reacts faster to a system overload by using loadavg1
instead of 15.
Fixes for dextool mutate
- Fix invalid scheman when a constexpr is used as a label in a case stmt.
- Fix broken test case suggestion SQL query.
- Fix scattered test case report. Users have reported that the information
about test cases are spread out in the html report. This constructs an
individual page per test case and a summary table inindex.html
for easy
navigation. - Updated llvm compatiblity to version 12. The bindings for version 11 worked
most of the time except fornew
expressions which could lead to mutant
schematan that didn't compile. - Fix bug when the path to the database is a symlink.
Emerald
General
- Fix compatibility with D language frontend 2.079.0+
New features for dextool mutate
- The configuration file for coverage and schema is changed to allow a user to
specify that the runtime should not be injected by the tool. The user, you,
manually link it. This is because the automatic injection do not always work
thus this option fixes the problem by allowing a user to change the build
system and link with a pre-compiled version of the runtime. - Install static libraries of the coverage and schema runtimes. See
README_tutorial_autotools.md for an example of how to use them. - Reduce which
test_cmd
are executed when the optiontest_cmd_checksum
is
activated. This option also activate a checksum table oftest_cmd
-
mutation status. If the checksum of thetest_cmd
match what is already
known then that mutation status is used instead.
Fixes for dextool mutate
- Fix labels being duplicated in scheman and thus leading to compilation errors.
- Fix that a UTF-8 BOM lead to compilation errors. The tool now find and warn about it.
It further detect when the source code is in something else than UTF-8 and warn. - Reduce memory usage by stack allocating AST nodes.
- Fix C++17 if-statement initialization making scheman not compile.
- Fix slow re-analyze on a large project by optimizing the sqlite schema (use indexes).
- Fix GC-crash on long analysis. Dangling object where collected.
Topaz
New features for dextool mutate
- Speedup execution of tests by only running those that are affected by the
mutant when running the slow source code modification engine. This is done
by saving the checksum of alltest_cmd
afterbuild_cmd
is executed and
no mutant is inserted. When a mutant is later on inserted the resulting
test_cmd
s are compared to the unmodified. Only those that are different
are classified as affected and thus need to be executed. If notest_cmd
is affected by the inserted mutant the mutation is classified as
equivalent. Settest_cmd_checksum
to true to activate the feature. It is
by default turned off. - Changed the default test order of mutants to
bySize
after a thorough
testing phase. It means that the mutants are tested in random order but
weighted by how much they affect the program (SUT). This mean that e.g.
large sdl mutants will be prioritized above aor mutants.
Fixes for dextool mutate
- The captured output from the test cases wasn't possible to limit which could
lead to a memory explosion and consequential crash of the tool because it
ran out of memory. This has now been fixed by introducing a configuration
option (max_test_cmd_output
) and default limit of 10 Mbyte. - Fix console spam when the plugin compile the software under test. The spam
is replaced by a timestamp that is printed once every minute to indicate to
a CI service that it is alive and to the user how long it has been
compiling. If the compilation fail it will automatically re-start but this
time print all output to the console for the user to understand what went
wrong. - Fix lockup when there are test cases with the same name.
Drizzle
New features for dextool mutate
- Check that all files to mutate are writable.
- Check periodically that the test suite, without any mutants, execute OK. If
not, rollback the last tested mutants. - Save what test_cmd killed a mutant. This is only activated if test case
analyzers are used. - Prioritize testing of unknown mutants.
- Check overload/halt conditions when running a schema for each mutant that is
tested. Previously it was only done when selecting a new schema. - Construct larger mutant scheman. Previously they where at most one
translation unit. Now they can span multiple translation units (e.g. *.cpp).
This mean that each schema contain more mutants thus the overall mutation
testing time is reduced. - Continuously save the result (every 20minute) the tested mutants of a
schema. This should reduce the overall memory usage and means that if the
schema is interrupted only the result from the last 20 minutes (at the most)
are lost. - Modernize the html layout theme.
- Prioritize mutants to test based on how much they affect the source code in
order to quicker give actionable feedback to the user. Based on the
assumption that a large mutant that survive is important to kill. - Change high interest mutants to reporting those that affect the most src
code and has survive. Previously it was those that had survived the most
test suite executions but this was never used by the users. The hope is that
those that are reported as high interest are actually useful, interesting
to look at and kill.
Also allows the number of high interest mutants that are shown to be
changed. - Automatically force a re-analyze of all files when a new version of the tool
is installed anddextool mutate analyze
is executed. - Automatically save the mutation score after each test run when all mutants
are tested. This is then used to plot a trend. - The HTML report and console print a SyncStatus. It shows how "in sync" the
tested mutants are with the code changes and test suite changes.
The further apart they are the less "trustworthy" is the report. Because
lets say the test suite is changed to now kill a mutant that is marked as
alive.
Fixes for dextool mutate
- Fix bugs in schema code generation such that more scheman compile.
- Fix bug where the dependency tracking via included headers used the wrong
path for headers included from the current directory (#include "foo.hpp"
). - Fix bug where a mutant that span multiple lines in the html report is shown
on one line. It makes it hard to read. - Removed delete of case branches in switch statements. The language C/C++ and
the Clang AST make it hard to generate correct mutants and schematan. The
effort needed to make it work in all cases is still great. It is further a
bit unnecessary because SDL will delete individual statements in the
branches thus well. It doesnt't really give that much to be able to delete
whole branches. - Not all mutant schematan for binary operators where generated which meant
that speedup opportunities where lost. With this fix all binary operators
should have a schema generated for them which speeds up dcr, ror, aor mutant
operators. - Fix the size of the database when schematas are saved by compressing them on
the fly. Reduced the size up to 90% of a database. - Add support for C++17 structural binding in schematas.
- Enable scheman which mutate const variables in C/C++.
- Fix mutation of C++ lamba expressions.
- Fix libclang bindings. The enum CXCursorKind is not backward or forward
compatible. Because of this dextool now have a binding for each version it
supports. What happend where that the analyzed AST could, if the wrong
binding is used, contain seemingly random nodes which mean that the
mutations where waaaay off. - Fix schematas inside return-statements and when a condition contain pointers.
- Measuring the runtime of the test suite is now done multithreaded. It is too
slow and pessimistic to do single threaded. It was previously changed to
single threaded because the load of the host computer could vary and lead to
falsely classify mutants as timeout. But with the feature--load-behavior slowdown
this is no longer a problem. Thus changing back to multithreaded
to speedup both the start and overall mutation testing. - Remove returnFalse/returnTrue mutants because they are redundant, overlap
with normal true/false. - Stop generating the equivalent mutant sdl when it deletes empty scopes.
Nice Weather
This release, as the previous once, has focused on the mutation testing plugin.
The focus has been on improving the mutation testing speed for software that
have just started to write tests to those that have a full test suite with 100%
statement coverage. The road has been long but we are finally at the finish line.
New features for dextool mutate
- Replaced
restrict
in the configuration withinclude
andexclude
to
allow a user to both box in what to test and to selectively exclude parts. - Automatically run SQL vacuum on the database to reduce its size. Currently
triggers when there is a new tool release or schematan are removed. - An estimation (trend) is calculated using a kalman filter which is ran over
the historical data of when a mutant where discovered. The intention is to
give the user a feeling of how the mutation score and thus test suite
quality is trending based on the latest code changes. - Add an admin command to reset a mutant subtype.
- Add an admin command to clear the worklist.
- Allow a user to specify retest of old mutants as a percentage of the total.
This is to allow a user to easier predict when all mutants have been
re-tested. - Reduce the database size by only saving those mutants to the database that
the user has specified. - Save the test suites execution time in the database to reduce the number of
times it needs to be executed during the warmup phase of testing mutants. - Save the mutation testing score in the database when all mutants have been tested (worklist is zero).
Use this mutation score show a bar graph of how it has changed over time. - Add
--load-threshold
and--load-behavior
to easier integrate the plugin
in Jenkins as a good citizen by allowing the plugin to detect when the
build server is overloaded and either slow down or stop mutation testing.
Mainly intended to be used for mutation testing of themaster
branch which
should have a lower priority than pull requests. - Add test case stats, uniqueness and test cases to the JSON report.
- Save the exit code of the test suite to the database. This will be used in
the future to improve the reports because users are interested in the
difference between them such as when the test suite SEGFAULT verses exits
with a failure. - Add tracking of the test suite to allow dextool to know if old mutants need
to be tested because the test suite have changed. - Use coverage information to quickly mark mutants that exists in functions
that are never executed by the test suite asalive
. - Add AORs as a simplified AOR. Instead of producing four mutants when an
arithmetic operator is used it only produce one. - Add dependency tracking to reduce the number of files that need to be
re-analysed. It is expensive to analyze modern C++ thus by tracking the
#include
it is possible to see exactly which files need to be analysed
instead of as previously, always analyse every file. - Enable analysis of code that isn't parsed correctly by the clang frontend by
the analysis to proceed even though there are compilation errors.
Fixes for dextool mutate
- Fix the table sorting in the HTML report to be numerical instead of lexical.
- Fix the mutation score constantly "resetting" because tests are added/removed.
- Fix schemata bug for switch statements. The scheman where not equivalent to
the source code mutants. - Fix bug where the admin command to stop the timeout testing didn't actually work.
- Removed case branch delete from mutation operator DCR because it is a
duplicate of what SDL do for case branches. - Fix memory usage when analysing. A pool is now used for the mutant AST where
all nodes are allocated. The pool is then disposed as soon as it isn't
needed. - Fix bug in MemFree which calls malloc_trim and GC free to reduce the heap.
- Always interpret the build command as a shell command to execute. Reduced or
removes the need for abuild.sh
because the most common use where just to
change directory and call make. - Separately record the time it took to test a mutant as in compile and
execute test suite time. - Print what is wrong in the configuration file when it fails to parse.
- Fix SQL query for retrieving the oldest mutants by telling it to interpret
as a datetime and not string. - Remove mutants that only delete whitespaces. These are obviously equivalent
mutants thus not interesting. - Fix AOR mutation operator using the modulo operator when either side is a
floating point type by not generating this type of mutants. - Fix bug when counting the number of mutants a test case killed. Mutants
where duplicated. - Fix mutation checksum calculation which thus repaired the previously broken
comment robustness. - Fix UOI negation by using double negation instead of removing
!
. This is
because!
changes the type to a boolean and C++ is picky about the types.
Breaking changes to dextool mutate
- Removed the mutation operator COR and DCC.
- Removed the deprecated
--level
for the report subcommand. - Remove
restrict
from the configuration file.