MDEV-33763 : Buffered error logging for Galera

[janlindstrom]

This is adaptation of feature by Zsolt Parragi [email protected] for Percona XtraDB Cluster to MariaDB. Original feature is documented at https://www.percona.com/blog/introducing-buffered-error-logging-in-percona-server-for-mysql/.

In this adaptation variable names are changed and integration to server code is different.

Commit adds new variables
wsrep_buffered_error_log_buffer_size ulong default 0 dynamic
Size of buffer used in buffered error logging. Because
wsrep debug setting produces large number of messages
it is recomended to set this variable around 1-100Mb.

wsrep_buffered_error_log_filename string default NULL static
  Name of the file where buffered error logging is done.

wsrep_buffered_error_log_file_size ulong default 0 dynamic
  Maximum size of one buffered error log file. This variable
  should be configured to be 10-100 times bigger than buffer
  size.

wsrep_buffered_error_log_rotations uint default 9 dynamic.
  Number of buffered error log files kept before deleting
  them.

When wsrep_buffered_error_log_buffer_size is specified and non zero, a buffer is
allocated with the specified size, and log messages are
written there. This includes also wsrep debug messages.

When this buffer gets full, the server shuts down, or crashes, the
buffer is written to wsrep_buffered_error_log_filename.

When log files reaches configured size limit log is rotated and
only configured amount of log files are kept.

This is useful in situations where debug logs create lots of output and
slow down normal server operation, but are helpful for figuring out
issues, as writing to a memory buffer has less overhead.

Warning: When this feature is used server will write big number
of messages to error log. Therefore, it is recomended to store
these logs on separate location to avoid filling up your
database disk. Furthermore, it is recomended to set up
logrotate to delete old log files.

Detailed changes:

wsrep.h
Log printing is now controlled by wsrep_debug_mode variable and
controlled by wsrep_debug and wsrep_buffered_error_log_buffer_size
variables.

sql/log.cc
Added sql_print_debug to log wsrep debug messages (WSREP_DEBUG()).

print_buffer_to_file
Actual buffered error logging if enabled will happen in this
function.

sql/mysqld.cc
Buffered error logging is initialized and shutdown here if it is
configured.

sql/signal_handler.cc
If buffered error logging is enabled remaining buffer is written
before continuing signal handling.

sql/sys_vars.cc
New variables are defined here.

sql/wsrep_buffered_error_log.cc
sql/wsrep_buffered_error_log.h
Buffered error log implementation using logger API (mysys/file_logger.cc).

sql/wsrep_var.cc
New variables update and check functions.

mysys/file_logger.cc
Added support for buffered error logging, file name changing and
buffer resizing.

[sciascid]

Is this file used somewhere?

[janlindstrom]

Its seems it is not used

[sciascid]

In plugin/server_audit we have pretty much duplicated service_logger.h and file_logger.c?
Is that correct?

[janlindstrom]

They are not same, I did not want to change audit log code, so I made copy of original API and used new API on my implementation that uses C++. Audit plugin uses C and changing that is out of scope for this PR.

[sciascid]

Two of the new tests are not working:
galera.galera_var_buffered_error_log_file_size
galera_var_buffered_error_log_buffer_size

And this one needs to be re-recorded:
galera.galera_defaults

[sciascid]

Why would one want to set the buffer size? And what would be a good number?
Can't we just pick a fixed number, based on some basic performance testing?`

[janlindstrom]

Naturally we could, buffer size configuration depends wsrep-debug level, higher lever produces more messages. In wsrep-debug it is around 2000kb/s (sysbench). I use this variable to indicate that buffered error logging is used, naturally we could use filename as well.