cfn-hup in daemon mode doesn't work on any currently supported Python version, because of dependency on obsolete python-daemon version

[Aneurin]

aws-cfn-bootstrap requires python-daemon<2.2, but version 2.2.4 (from 2019) is the oldest version that supports Python 3.8+. Thus, any OS with a current version of Python (eg 3.10 in Ubuntu 22.04) will get the following error when attempting to launch cfn-hup in daemon mode:

Traceback (most recent call last):
  File "/usr/local/bin/cfn-hup", line 226, in <module>
    with daemon.DaemonContext(pidfile=pidfile.TimeoutPIDLockFile('/var/run/cfn-hup.pid', 300),
  File "/usr/local/lib/python3.10/dist-packages/daemon/daemon.py", line 280, in __init__
    detach_process = is_detach_process_context_required()
  File "/usr/local/lib/python3.10/dist-packages/daemon/daemon.py", line 801, in is_detach_process_context_required
    if is_process_started_by_init() or is_process_started_by_superserver():
  File "/usr/local/lib/python3.10/dist-packages/daemon/daemon.py", line 778, in is_process_started_by_superserver
    if is_socket(stdin_fd):
  File "/usr/local/lib/python3.10/dist-packages/daemon/daemon.py", line 743, in is_socket
    file_socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_RAW)
  File "/usr/lib/python3.10/socket.py", line 546, in fromfd
    return socket(family, type, proto, nfd)
  File "/usr/lib/python3.10/socket.py", line 232, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 88] Socket operation on non-socket

Ignoring the dependency error and forcing a newer python-daemon version seems to lead to cfn-hup silently failing, which I've not investigated further, so currently the best workaround I know of is to run it with --no-daemon in a cronjob or systemd timer.

[mleziva]

Hi Aneurin, thanks for bringing this to our attention. This repo just supports the sample templates. I will see if I can find the correct team that supports aws-cfn-bootstrap

[mleziva]

Hi @Aneurin, were you following the guidance from this post: https://repost.aws/knowledge-center/install-cloudformation-scripts?

[aneurin-price]

(Originally filed under my personal account, but this is my work account)

Apologies, it's been some time and it looks like I no longer have my notes from this, but I believe I found some documentation that took me to https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/solutions/OperatingSystems/Ubuntu22.04_cfn-hup.yaml - it could well have been that page although it doesn't look hugely familiar.

Ultimately I just ended up doing

            /lib/systemd/system/cfn-hup.service:
              content: !Sub |
                [Unit]
                Description=cfn-hup
                [Service]
                Type=oneshot
                ExecStart=/usr/local/bin/cfn-hup -c /etc/cfn --no-daemon
                [Install]
                WantedBy=multi-user.target
              mode: '000400'
              owner: root
              group: root
            /lib/systemd/system/cfn-hup.timer:
              content: !Sub |
                [Unit]
                Description=Run cfn-hup every minute
                [Timer]
                OnCalendar=*-*-* *:*:00
                [Install]
                WantedBy=timers.target
              mode: '000400'
              owner: root
              group: root

which is inelegant but gets the job done.

I'm guessing what you're really looking for is trivial steps to reproduction - I have some vague recollection of eventually realising that it only crashed after restarting or something like that, and thus the template could appear to work correctly at first run. I'm a bit swamped but I can try to take a look next week if I get the time.

[guilleojeda]

I'm experiencing the same problem.
@aneurin-price maybe you were led to this repo from the official docs? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent-New-Instances-CloudFormation.html#installing-CloudWatch-Agent-using-CloudFormation-Templates-inline
That's how I got here

[mleziva]

Looks like the latest version of aws-cfn-bootstrap does support newer versions of python, but the S3 account link to the latest version is not actually pointing to the latest version.

aws-cfn-bootstrap documentation (to access links to the latest versions):
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/releasehistory-aws-cfn-bootstrap.html

Incorrect latest link:
https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz => points to version 1.4 which is not the latest

However the py3 latest link points to a version that is closer to the latest:
https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz

Can you verify if the issue is fixed when using a newer version?

[aneurin-price]

That's definitely the version with the issue, because its setup.py requires python-daemon>=2.1,<2.2.

I've just spent an hour or so trying to replicate this again but it's tough because it originally came up in a large stack with a lot of dependencies and I've not yet been able to minimise it. I've also not had any success in using https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/solutions/OperatingSystems/Ubuntu22.04_cfn-hup.yaml as a starting point, because that template doesn't work. Even after fixing the obvious copy/paste errors in it I still haven't been able to get anything that successfully triggers cfn-hup after a stack update, and I'm not certain if it's something to do with this problem or if I've just made a mistake. It's unfortunate that there aren't any working minimal examples to start from.

I have to go away for a bit but I can probably find a couple of hours the week after next to try properly debugging this by paring down a known-working stack, and then switching it back into daemon mode.

[github-actions]

Due to inactivity this issue will be closed in 7 days

[ericzbeard]

I think this is the same issue as #411

[DawidKrysiak]

Hi,
I am trying to deploy Ubuntu 24.04 from AWS' offering, and I have the same issue.
I tried venv approach, but it's falling apart further down the line anyway.
Any ETA on a RELIABLE solution for cfn-init on modern Linux systems?

[ericzbeard]

Have a look at this one, it was updated and tested recently: https://github.com/aws-cloudformation/aws-cloudformation-templates/blob/main/AutoScaling/AutoScalingMultiAZWithNotifications.yaml

[DawidKrysiak]

@ericzbeard not even remotely a solution - it relies on the tool to be working.
I found a workaround - as expected - using venv, essentially installing two python versions side-by-side with 3.8 for cfn and 3.12 for the rest of the system, and set up venv to use 3.8, set up the paths so cfn-init,cfn-init were used from that venv..
It's ridiculous that AWS is shipping OS without ensuring their bloody tools are working properly. But I guess that's not glamorous, nothing for the sales guys to brag about during keynotes, so no one cares...