Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition of TimedRotatingFileHandler to log_config.yaml for iCtrl's logging configuration #46

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Addition of TimedRotatingFileHandler to log_config.yaml for iCtrl's logging configuration #46

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1905de1
Addition of timedRotatingHandler to log_config.yaml for messages to b…
li-ruihao Jan 18, 2025
0c148d0
Direct all log files to the logs directory
li-ruihao Jan 19, 2025
5cac6ff
all changes related to logging configurations in log_config.yaml base…
li-ruihao Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions application/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@
try:
with open('log_config.yaml') as config_file:
config = yaml.safe_load(config_file.read())

os.makedirs(os.path.join(os.getcwd(), 'logs'), exist_ok=True)
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
logging.config.dictConfig(config)
except Exception:
# Fallback to a basic configuration
Expand Down
12 changes: 10 additions & 2 deletions log_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
version: 1
disable_existing_loggers: True
disable_existing_loggers: False
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved

formatters:
default:
Expand All @@ -12,7 +12,15 @@ handlers:
level: DEBUG
formatter: default
stream: ext://sys.stderr
file_rotating:
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
class: logging.handlers.TimedRotatingFileHandler
level: DEBUG
filename: logs/ictrl_log.log
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
formatter: default
when: 'midnight'
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
interval: 1
backupCount: 14
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did we arrive at this count? Do we have an estimate of how many logs iCtrl generates per hour? Then how many (bytes of) logs do we plan to keep on the user's machine? Note this should not exceed the application environment requirements in your previous analysis, say in the Project Proposal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially, I just thought it was a good idea to keep a backup for the past 2 weeks' logs. However let's do some calculation given what we wrote in the project proposal.

We are using the following format for our log message:
'%(asctime)s %(levelname)s [%(name)s:%(lineno)d] %(message)s'.

Let's make some very bad assumptions for each value to estimate the length of a very long log message.

  • %(asctime)s: This value is fixed to be 23 characters
  • %(levelname)s: the longest level name is CRITICAL, which is 8 characters
  • %(name)s: Assume the longest Python file name to be 20 characters
  • %(lineno)d: Assume the longest line number to be 5 digits(characters)
  • %(message)s: Assume a very detailed log message that has 200 characters
  • Other characters such as space sum up to a total of 6 characters, and a newline character consists of CR and LF which adds 2 to 6 becomes 8 characters

In total, there are 264 characters, so each message has 528 bytes assuming each character is 2 bytes according to the UTF standard, which by default is 2 bytes each.

Assuming 5 log messages are generated each second, which is 2640 bytes/second. Then each 3-hour period will generate 60 x 60 x 3 x 2640 = 28,512,000 bytes of data which is 27.1912 MB. Let's round it up to 28 MB to include some metadata generated from file rotation and make the calculation easier.

The requirement on the project proposal said that the size of the logs generated by iCtrl cannot exceed 32GB. Originally we are backing up 2 weeks of data, that is 112 rotation periods. The total size for log backups would be 112 x 28 = 3,136 MB = 3.0625 GB, which is far from breaking the constraint requirement.

Therefore, I think 2 weeks of backup data is appropriate, and the calculations above show that the system can allow more backups, if the assumptions are bad enough.


root:
level: DEBUG
handlers: [console]
handlers: [console, file_rotating]
li-ruihao marked this conversation as resolved.
Show resolved Hide resolved
Loading