Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add check for "rsyslogd: error" in /var/log/messages in all tests #388

Merged
merged 7 commits into from
Jun 5, 2024
Merged

fix: Add check for "rsyslogd: error" in /var/log/messages in all tests #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2186ad9
Add check for "rsyslogd: error" in /var/log/messages in all tests
spetrosi May 27, 2024
408a8c1
Fix tests on RHEL 7
spetrosi May 27, 2024
941b12d
Do not start service earlier than necessary
spetrosi May 29, 2024
947916e
Fix omitting on EL 7
spetrosi May 30, 2024
1a9e3c6
Don't check errors in ovirt_elasticsearch.yml due to expected errors
spetrosi May 30, 2024
4ed0583
Fix EL 7 maxdatasize incorrect condition
spetrosi Jun 4, 2024
a789904
Print possible errors in {{ __default_system_log }}
spetrosi Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion roles/rsyslog/tasks/main_core.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,6 @@
service:
name: rsyslog
enabled: true
state: started
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What problem is being addressed by this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On latest Fedora this causes error messages in /var/log/messages as such:

May 25 09:43:31 7e33d79e-6b6e-4e22-97fc-fa84e46170bf rsyslogd: error during parsing file /etc/rsyslog.d/00-global.conf, on or before line 12: parameter 'workdirectory' specified more than once - one instance is ignored. Fix config [v8.2312.0-1.fc40 try https://www.rsyslog.com/e/2207 ]
May 25 09:43:31 7e33d79e-6b6e-4e22-97fc-fa84e46170bf rsyslogd: error during parsing file /etc/rsyslog.d/00-global.conf, on or before line 12: parameter 'workDirectory' not known -- typo in config file? [v8.2312.0-1.fc40 try https://www.rsyslog.com/e/2207 ]
May 25 09:43:31 7e33d79e-6b6e-4e22-97fc-fa84e46170bf rsyslogd: parameters for built-in module builtin:omfile already set - ignored  [v8.2312.0-1.fc40 try https://www.rsyslog.com/e/2220 ]
May 25 09:43:31 7e33d79e-6b6e-4e22-97fc-fa84e46170bf rsyslogd: module 'imuxsock' already in this config, cannot be added  [v8.2312.0-1.fc40 try https://www.rsyslog.com/e/2221 ]
May 25 09:43:31 7e33d79e-6b6e-4e22-97fc-fa84e46170bf rsyslogd: error during parsing file /etc/rsyslog.d/90-input-basics-basic_input.conf, on or before line 8: parameter 'SysSock.Use' not known -- typo in config file? [v8.2312.0-1.fc40 try https://www.rsyslog.com/e/2207 ]
May 25 09:43:31 7e33d79e-6b6e-4e22-97fc-fa84e46170bf rsyslogd: error during parsing file /etc/rsyslog.d/90-input-basics-basic_input.conf, on or before line 8: parameter 'SysSock.RateLimit.Interval' not known -- typo in config file? [v8.2312.0-1.fc40 try https://www.rsyslog.com/e/2207 ]

This is because the main config file was not generated yet. Why this does not fail on RHEL 8 and 9 - I have no idea.
But it seems like a good practice to start a service using a handler after the role invocation finishes.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On latest Fedora this causes error messages in /var/log/messages as such

Only on Fedora 40, but not 39, or on other platforms? Sounds like an issue with the fedora 40 rsyslog package.

This is because the main config file was not generated yet

The rsyslog package provides a default /etc/rsyslog.conf (and possibly files in /etc/rsyslog.d/), and the service should start with the default files. If this isn't working, it is a problem with the rsyslog package.

But it seems like a good practice to start a service using a handler after the role invocation finishes.

In general, yes, but in some cases the role needs the service to be running before it can take further actions.

when:
- __rsyslog_enabled | bool
- not rsyslog_in_image | default(false) | bool
Expand Down
3 changes: 3 additions & 0 deletions roles/rsyslog/templates/input_relp.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
input(name="{{ __rsyslog_input.name }}"
type="imrelp"
port="{{ __rsyslog_input.port | d(20514) | int }}"
{% if ansible_distribution in ['CentOS', 'RedHat'] and
ansible_distribution_major_version is version('7', '>') %}
maxDataSize="{{ __rsyslog_input.max_data_size | d(logging_max_message_size) | int }}"
{% endif %}
spetrosi marked this conversation as resolved.
Show resolved Hide resolved
{% if __rsyslog_input.tls | default(true) %}
{% if __rsyslog_input.ca_cert is defined %}
{% set __cacert = __rsyslog_input.ca_cert %}
Expand Down
26 changes: 26 additions & 0 deletions tests/tasks/assert_varlogmessages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# SPDX-License-Identifier: MIT
---
- name: Get content of {{ __default_system_log }}
shell: cat {{ __default_system_log }} || true
register: __default_system_log_content
changed_when: false
no_log: true

- name: Print possible errors in {{ __default_system_log }}
vars:
errors: >-
{{ __default_system_log_content.stdout
| regex_search('rsyslogd: error.*') }}
debug:
var: errors
when: errors | length > 0

# /var/log/messages sometimes contains errors not visible in journalctl
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give me an example of this happening? It seems like a journald configuration issue e.g. journal logs are being trimmed too quickly. Note that all log messages in /var/log/messages come from journal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, tests_ovirt_elasticsearch.yml prints error messages only on EL 7. I fixed other tests in this PR and I am looking into this.

2024-05-29T11:46:14.984650-04:00 ibm-p8-kvm-03-guest-02 rsyslogd: error: 'tls.cacert' file /etc/rsyslog.d/es-ca.crt couldn't be accessed: No such file or directory  [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2039 ]
2024-05-29T11:46:14.984810-04:00 ibm-p8-kvm-03-guest-02 rsyslogd: error: 'tls.mycert' file /etc/rsyslog.d/es-cert.pem couldn't be accessed: No such file or directory  [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2039 ]
2024-05-29T11:46:14.984979-04:00 ibm-p8-kvm-03-guest-02 systemd: Started System Logging Service.
2024-05-29T11:46:14.985136-04:00 ibm-p8-kvm-03-guest-02 rsyslogd: error: 'tls.myprivkey' file /etc/rsyslog.d/es-key.pem couldn't be accessed: No such file or directory  [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2039 ]
2024-05-29T11:46:14.985322-04:00 ibm-p8-kvm-03-guest-02 rsyslogd: error: 'tls.cacert' file /etc/rsyslog.d/es-ca.crt couldn't be accessed: No such file or directory  [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2039 ]
2024-05-29T11:46:14.985492-04:00 ibm-p8-kvm-03-guest-02 rsyslogd: error: 'tls.mycert' file /etc/rsyslog.d/es-cert.pem couldn't be accessed: No such file or directory  [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2039 ]
2024-05-29T11:46:14.985633-04:00 ibm-p8-kvm-03-guest-02 rsyslogd: error: 'tls.myprivkey' file /etc/rsyslog.d/es-key.pem couldn't be accessed: No such file or directory  [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2039 ]

- name: Ensure no errors in {{ __default_system_log }}
assert:
that: "'rsyslogd: error' not in __default_system_log_content.stdout"

- name: Remove {{ __default_system_log }}
file:
path: "{{ __default_system_log }}"
state: absent
3 changes: 3 additions & 0 deletions tests/tests_basics_files.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,3 +392,6 @@
file:
path: /tmp/__testfile__
state: absent

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
4 changes: 4 additions & 0 deletions tests/tests_basics_forwards.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@
target: host.domain
- name: forwards_no_severity_and_facility_protocol_port_target
type: forwards
target: host.domain
- target: no_name.localdomain
type: forwards
logging_inputs:
Expand Down Expand Up @@ -582,6 +583,9 @@
rmdir /tmp/rsyslog.d-backup
changed_when: false

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml

# TEST CASE 3
- name: Test case 3
block:
Expand Down
9 changes: 8 additions & 1 deletion tests/tests_combination.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,11 @@
- name: "{{ __test_tag }}"
type: files
input_log_path: "{{ __test_inputfiles_dir }}/*.log"
endmsg_regex: xyz
spetrosi marked this conversation as resolved.
Show resolved Hide resolved
# Not supported on EL 7
endmsg_regex: "{{ omit
if ansible_distribution in ['CentOS', 'RedHat'] and
ansible_distribution_major_version is version('7', '==')
else 'xyz' }}"
- name: basic_input
type: basics
ratelimit_burst: 33333
Expand Down Expand Up @@ -577,3 +581,6 @@
- name: "Force all notified handlers to run at this point,
not waiting for normal sync points"
meta: flush_handlers

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
4 changes: 3 additions & 1 deletion tests/tests_files_elasticsearch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
false in the configuration named elasticsearch_output"
__certdir: /etc/pki/tls/certs/
__keydir: /etc/pki/tls/private/

__default_system_log: /var/log/messages
tasks:
- name: Run test
block:
Expand Down Expand Up @@ -458,6 +458,8 @@
not waiting for normal sync points"
meta: flush_handlers

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
always:
- name: Remove tempdir
file:
Expand Down
3 changes: 3 additions & 0 deletions tests/tests_files_files.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,6 @@

- name: Check ports managed by firewall and selinux
include_tasks: tasks/check_firewall_selinux.yml

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
16 changes: 2 additions & 14 deletions tests/tests_imuxsock_files.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,8 @@
__logging_file: "{{ __default_system_log }}"
include_tasks: tasks/test_logger.yml

- name: Get content of {{ __default_system_log }}
command: cat {{ __default_system_log }}
register: __default_system_log_content
changed_when: false

- name: Ensure no errors in {{ __default_system_log }}
assert:
that: "'rsyslogd: error' not in __default_system_log_content.stdout"
- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml

- name: Check ports managed by firewall and selinux
include_tasks: tasks/check_firewall_selinux.yml
Expand Down Expand Up @@ -148,12 +142,6 @@
command: journalctl -ex
changed_when: false

# When imuxsock is configured, errors are not visible in journalctl
- name: Print errors in {{ __default_system_log }}
command: >-
grep "rsyslogd: error" {{ __default_system_log }}
changed_when: false

- name: Fail
fail:
msg: "{{ ansible_failed_result }}"
10 changes: 9 additions & 1 deletion tests/tests_ovirt_elasticsearch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
__test_logs_index: project.ovirt-logs
__test_engine_input: /var/log/ovirt-engine/engine.log
__test_vdsm_input: /var/log/vdsm/vdsm.log

__default_system_log: /var/log/messages
tasks:
- name: TEST CASE 0; Ensure basic ovirt default configuration works
vars:
Expand Down Expand Up @@ -376,3 +376,11 @@
- name: "Force all notified handlers to run at this point,
not waiting for normal sync points"
meta: flush_handlers

# This tests results in expected error in /var/log/messages on EL 7
# Errors are that /etc/rsyslog.d/es-ca.crt and /etc/rsyslog.d/es-cert.pem
# do not exist.
- name: Remove {{ __default_system_log }}
file:
path: "{{ __default_system_log }}"
state: absent
4 changes: 4 additions & 0 deletions tests/tests_purge_reset.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
hosts: all
vars:
__test_default_files_conf: /etc/rsyslog.d/30-output-files-default_files.conf
__default_system_log: /var/log/messages
tasks:
- name: Determine if system is ostree and set flag
when: not __logging_is_ostree is defined
Expand Down Expand Up @@ -151,3 +152,6 @@

- name: Check ports managed by firewall and selinux
include_tasks: tasks/check_firewall_selinux.yml

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
9 changes: 8 additions & 1 deletion tests/tests_relp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
__test_relp_global: /etc/rsyslog.d/00-global.conf
logging_max_message_size: 16384
logging_preserve_fqdn: true

__default_system_log: /var/log/messages
tasks:
# TEST CASE 0
# Note: Create a self-signed cert just for the "unit" test.
Expand Down Expand Up @@ -181,13 +181,17 @@
failed_when: __result.stdout != "1"
# yamllint enable rule:line-length

# maxDataSize is not supported on EL 7
- name: Check maxDataSize is logging_max_message_size
command: >-
grep -c 'maxDataSize="{{ logging_max_message_size }}"'
{{ __test_relp_server0 }}
changed_when: false
register: __result
failed_when: __result.stdout != "1"
when:
- ansible_distribution in ['CentOS', 'RedHat']
- ansible_distribution_major_version is version('7', '>')

- name: Check preserveFQDN is on
command: grep -c 'preserveFQDN="on"' {{ __test_relp_global }}
Expand Down Expand Up @@ -388,3 +392,6 @@

- name: Check ports managed by firewall and selinux
include_tasks: tasks/check_firewall_selinux.yml

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
5 changes: 4 additions & 1 deletion tests/tests_remote.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
/etc/rsyslog.d/30-output-files-remote_files_output0.conf
__test_output_remote1: >-
/etc/rsyslog.d/30-output-files-remote_files_output1.conf

__default_system_log: /var/log/messages
tasks:
# TEST CASE 0
- name: "TEST CASE 0; Test configuration inputs from the remote rsyslog
Expand Down Expand Up @@ -242,3 +242,6 @@

- name: Check ports managed by firewall and selinux
include_tasks: tasks/check_firewall_selinux.yml

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
5 changes: 4 additions & 1 deletion tests/tests_server.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
__test_server_tcp: /etc/rsyslog.d/11-input-remote-remote_tcp.conf
__test_server_udp: /etc/rsyslog.d/11-input-remote-remote_udp.conf
__expected_error: "Error: remote_tcp_0 and remote_tcp_1 conflict."

__default_system_log: /var/log/messages
tasks:
# TEST CASE 0
# Note: Create a self-signed cert just for the "unit" test.
Expand Down Expand Up @@ -223,3 +223,6 @@
- "{{ __test_ca_cert }}"
- "{{ __test_key }}"
- "{{ __test_cert }}"

- name: Assert {{ __default_system_log }}
include_tasks: tasks/assert_varlogmessages.yml
Loading