ZTS: include microsecond timestamps on all output

[robn]

[Sponsors: Klara, Inc., Wasabi Technology, Inc.]

Motivation and Context

When reviewing test output after a failure, it's often quite difficult to work out the order and timing of events, and to correlate test suite output with kernel logs.

Description

This adds timestamps to ZTS output to help correlate events:

• all of the standard log_XXX functions ultimately end up in _printline, which now prefixes output with a timestamp.
• the test runner logging function log() also now prefixes its output with a timestamp.
• on failure, when capturing the kernel log in zfs_dmesg.ksh, the iso time format is requested. This is gated behind a platform check, as FreeBSD's dmesg has no switches to control timestamps.

How Has This Been Tested?

Manually, running a test I know fails in my test VMs and considering the output.

$ zfs-tests.sh -DvKx -t zfs_load-key_https
...
/usr/local/share/zfs/test-runner/bin/test-runner.py  -D  -K -c "/var/tmp/zfs-tests.481.run" -T "functional" -i "/usr/local/share/zfs/zfs-tests" -I "1"
[    6.960295] ZTS run /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zfs_load-key/setup
[2025-02-12T01:05:34.353115] NOTE: begin default_setup_noexit
[2025-02-12T01:05:34.398547] SUCCESS: zpool create -f testpool loop0
[2025-02-12T01:05:34.432899] SUCCESS: zfs create testpool/testfs
[2025-02-12T01:05:34.463772] SUCCESS: zfs set mountpoint=/var/tmp/testdir testpool/testfs
[2025-02-12T01:05:35.520185] SUCCESS: openssl req -x509 -newkey rsa:4096 -sha256 -days 1 -nodes -keyout /testpool/snakeoil.key -out /testpool/snakeoil.crt -subj /CN=localhost
Traceback (most recent call last):
  File "<string>", line 16, in <module>
  File "<string>", line 10, in <module>
  File "/usr/lib/python3.11/socketserver.py", line 456, in __init__
    self.server_bind()
  File "/usr/lib/python3.11/http/server.py", line 136, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib/python3.11/socketserver.py", line 472, in server_bind
    self.socket.bind(self.server_address)

[2025-02-12T01:06:25.290486] NOTE: Performing test-fail callback (/usr/local/share/zfs/zfs-tests/callbacks/zfs_dbgmsg.ksh)
...
[2025-02-12T01:06:25.315555] NOTE: Performing test-fail callback (/usr/local/share/zfs/zfs-tests/callbacks/zfs_dmesg.ksh)
=================================================================
 Tailing last 200 lines of dmesg log
=================================================================
2025-02-12T01:05:28,971544+00:00 .... node  #0, CPUs:      #1 #2 #3 #4 #5
2025-02-12T01:05:28,971544+00:00 smp: Brought up 1 node, 6 CPUs
2025-02-12T01:05:28,971544+00:00 smpboot: Max logical packages: 1
2025-02-12T01:05:28,971544+00:00 smpboot: Total of 6 processors activated (23904.00 BogoMIPS)
2025-02-12T01:05:29,090172+00:00 node 0 deferred pages initialised in 120ms
...
=================================================================
 End of dmesg log
=================================================================
[2025-02-12T01:06:25.334713] NOTE: Performing test-fail callback (/usr/local/share/zfs/zfs-tests/callbacks/zfs_mmp.ksh)
[2025-02-12T01:06:25.340432] Test: /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zfs_load-key/setup (run as root) [00:51] [FAIL]
[2025-02-12T01:06:25.345063] Test: /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_https.ksh (run as root) [00:00] [SKIP]
[   58.006623] ZTS run /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zfs_load-key/cleanup
[2025-02-12T01:06:25.551120] SUCCESS: zpool destroy -f testpool
[2025-02-12T01:06:25.583079] SUCCESS: rm -rf /var/tmp/testdir
[2025-02-12T01:06:25.594580] Test: /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zfs_load-key/cleanup (run as root) [00:00] [PASS]
...

(fyi, this test fails because the VM has no network support at all, not even a loopback. It's no big deal)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

"03:04:46.89 [2025-02-16T03:04:46.898708]" looks duplicate and excessively verbose. I suspect you meant to add the timestamps in some places where they weren't, but seems now there are places when they are duplicate.

And BTW, the CI seems very unhappy for some reason.

[robn]

03:04:46.89 [2025-02-16T03:04:46.898708]

This doesn't happen locally. Might be something in the CI runner adding timestamps? I'll look soon.