Nextcloud AIO fails to start, seems to be issues with Collabora and potentially local firewall

[panomitrius]

Steps to reproduce

1. Setup nextcloud-aio with docker. I use a nginx reverse proxy on the host, not on docker. Also running csf firewall on the host. I have several other containers running fine with this setup, including non-aio nextcloud on docker (which I've stopped during setup of aio). I had to use SKIP_DOMAIN_VALIDATION=true to get the initial setup to function well.
2. Start it up

Expected behavior

Nextcloud starts when clicking "Open your Nextcloud ↗"

Actual behavior

Nextcloud doesn't start when clicking "Open your Nextcloud ↗"

Host OS

Ubuntu 22.04.3 LTS

Nextcloud AIO version

6.4.0 (Nextcloud 27.0.2)

Current channel

Other valuable info

nextcloud-aio-collabora logs with csf enabled:

me%22:%22Collabora%20Office%22%2C%22ProductVersion%22:%2223.05%22%2C%22tunnelled_dialog_image_cache_size%22:%22100%22%7D]| wsd/COOLWSD.cpp:3524
wsd-00001-00018 2023-08-10 13:00:40.396408 +0200 [ prisoner_poll ] TRC  #18: New child spawned after 0ms of requesting| wsd/COOLWSD.cpp:3550
wsd-00001-00018 2023-08-10 13:00:40.396622 +0200 [ prisoner_poll ] INF  #18: New child [21], jailId: TjCFR2Tbz6IS3hyK| wsd/COOLWSD.cpp:3582
wsd-00001-00018 2023-08-10 13:00:40.396668 +0200 [ prisoner_poll ] TRC  #18: Calling make_shared<ChildProcess>, for NewChildren?| wsd/COOLWSD.cpp:3590
wsd-00001-00018 2023-08-10 13:00:40.396706 +0200 [ prisoner_poll ] TRC  #18: Upgrading to WebSocket| net/WebSocketHandler.hpp:922
wsd-00001-00018 2023-08-10 13:00:40.396749 +0200 [ prisoner_poll ] INF  #18: WebSocket version: 13, key: [Czr+T22gXTA7BY5L068bMQ==], protocol: [chat]| net/WebSocketHandler.hpp:933
wsd-00001-00018 2023-08-10 13:00:40.426929 +0200 [ prisoner_poll ] TRC  #18: Sending WS Upgrade response: Date: Thu, 10 Aug 2023 11:00:40
Server: COOLWSD HTTP Server 23.05.1.3
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: k7uisQpf2xOv59S0KXExzAQl+gg=
| net/WebSocketHandler.hpp:944
wsd-00001-00018 2023-08-10 13:00:40.427083 +0200 [ prisoner_poll ] TRC  #18: Wrote 201 bytes of 201 buffered data| net/Socket.hpp:1438
wsd-00001-00018 2023-08-10 13:00:40.427321 +0200 [ prisoner_poll ] INF  ChildProcess ctor [21].| wsd/COOLWSD.hpp:59
wsd-00001-00018 2023-08-10 13:00:40.427376 +0200 [ prisoner_poll ] TRC  #18: Resetting thread affinity while in transit (was 0x7faf6be586c0)| net/Socket.hpp:331
wsd-00001-00018 2023-08-10 13:00:40.427421 +0200 [ prisoner_poll ] TRC  #18: Calling addNewChild in disposition's move thing to add to NewChildren| wsd/COOLWSD.cpp:3605
wsd-00001-00018 2023-08-10 13:00:40.427453 +0200 [ prisoner_poll ] TRC  Adding a new child 21 to NewChildren, have 0 outstanding requests| wsd/COOLWSD.cpp:547
wsd-00001-00018 2023-08-10 13:00:40.427485 +0200 [ prisoner_poll ] INF  Have 1 spare child after adding [21]. Notifying.| wsd/COOLWSD.cpp:553
wsd-00001-00018 2023-08-10 13:00:40.506120 +0200 [ prisoner_poll ] TRC  #17: Handling poll events of prisoner_poll at index 1 (of 3): 0x4| net/Socket.cpp:457
wsd-00001-00001 2023-08-10 13:00:40.506158 +0200 [ coolwsd ] TRC  Have 1 new children.| wsd/COOLWSD.cpp:5684
wsd-00001-00001 2023-08-10 13:00:40.506219 +0200 [ coolwsd ] INF  WSD initialization complete: setting log-level to [warning] as configured.| wsd/COOLWSD.cpp:5700
wsd-00001-00018 2023-08-10 13:00:40.506225 +0200 [ prisoner_poll ] TRC  #17: Wrote 10 bytes of 10 buffered data| net/Socket.hpp:1438
frk-00019-00019 2023-08-10 13:00:40.506628 +0200 [ forkit ] WRN  The systemplate directory [/opt/cool/systemplate] is read-only, and at least [/opt/cool/systemplate//etc/hosts] is out-of-date. Will have to copy sysTemplate to jails. To restore optimal performance, make sure the files in [/opt/cool/systemplate/etc] are up-to-date.| common/JailUtil.cpp:519
Ready to accept connections on port 9980.

wsd-00001-00022 2023-08-10 13:00:43.191680 +0200 [ remotefontconfig_poll ] ERR  Remote config server has response status code: Found| wsd/COOLWSD.cpp:1174
wsd-00001-00022 2023-08-10 13:01:43.305296 +0200 [ remotefontconfig_poll ] ERR  Remote config server has response status code: Found| wsd/COOLWSD.cpp:1174

nextcloud-aio-collabora logs with csf disabled:

Ready to accept connections on port 9980.

wsd-00001-00016 2023-08-10 13:09:42.531644 +0200 [ remotefontconfig_poll ] ERR  Failed to lookup host [mynextcloud.domain.com]. Skipping (EAGAIN: Resource temporarily unavailable)| net/NetUtil.cpp:157
wsd-00001-00016 2023-08-10 13:09:42.935779 +0200 [ remotefontconfig_poll ] ERR  #-1: Failed to connect to mynextcloud.domain.com:443| net/HttpRequest.hpp:1257
wsd-00001-00016 2023-08-10 13:09:42.935842 +0200 [ remotefontconfig_poll ] ERR  Remote config server has response status code: Unknown| wsd/COOLWSD.cpp:1174

From this discussion

Go into the container

sudo docker exec -it nextcloud-aio-nextcloud bash

Now inside the container

curl -vvv https://$NC_DOMAIN:443/hosting/discovery
exit

*   Trying 9.18.12.20:443...
* Connected to mynextcloud.domain.com (9.18.12.20) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=formernextcloudadress.domain.com
*  start date: Aug  9 06:15:16 2023 GMT
*  expire date: Nov  7 06:15:15 2023 GMT
*  subjectAltName: host "mynextcloud.domain.com" matched cert's "mynextcloud.domain.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: mynextcloud.domain.com]
* h2 [:path: /hosting/discovery]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7f25998d8af0)
> GET /hosting/discovery HTTP/2
> Host: mynextcloud.domain.com
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/2 302 
< server: nginx
< date: Thu, 10 Aug 2023 11:05:21 GMT
< content-type: text/html; charset=UTF-8
< content-length: 0
< x-powered-by: PHP/8.2.8
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< cache-control: no-store, no-cache, must-revalidate
< pragma: no-cache
< location: /
< set-cookie: PHPSESSID=8c8299b2ecad1d3d9071db353d8cd1bb; path=/
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< 
* Connection #0 to host mynextcloud.domain.com left intact

nextcloud-aio-nextcloud logs:

Added mynextcloud.domain.com:3478.
Added mynextcloud.domain.com:3478.
There is nothing to delete.
Added signaling server https://mynextcloud.domain.com/standalone-signaling/.
Config value recording_servers of app spreed deleted
System config value enabledPreviewProviders => 0 set to string OC\Preview\Imaginary
System config value preview_imaginary_url set to string http://nextcloud-aio-imaginary:9000
fulltextsearch disabled
fulltextsearch 27.0.1 removed
[10-Aug-2023 13:07:15] NOTICE: fpm is running, pid 1079
[10-Aug-2023 13:07:15] NOTICE: ready to handle connections
Activating collabora config...
An unhandled exception has been thrown:
Error: Call to a member function xpath() on bool in /var/www/html/custom_apps/richdocuments/lib/WOPI/Parser.php:51
Stack trace:
#0 /var/www/html/custom_apps/richdocuments/lib/Command/ActivateConfig.php(67): OCA\Richdocuments\WOPI\Parser->getUrlSrc('Capabilities')
#1 /var/www/html/3rdparty/symfony/console/Command/Command.php(298): OCA\Richdocuments\Command\ActivateConfig->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 /var/www/html/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 /var/www/html/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Richdocuments\Command\ActivateConfig), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#4 /var/www/html/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#5 /var/www/html/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 /var/www/html/console.php(100): OC\Console\Application->run()
#7 /var/www/html/occ(11): require_once('/var/www/html/c...')

[szaimen]

issues with Collabora and potentially local firewall
Also running csf firewall on the host

Can you for a test disable the csf firewall and check if that makes it work? If so, you need to fix your firewall.

[panomitrius]

Well, I did before and posted the output log of both csf enabled and disabled.

However, now when I stop csf (csf -x) I can't even get the nextcloud-aio to start at all:

Output from running docker restart nextcloud-aio-mastercontainer after csf -x

Error response from daemon: Cannot restart container nextcloud-aio-mastercontainer: driver failed programming external connectivity on endpoint nextcloud-aio-mastercontainer (0d420719308e6db287165fec6e453f2f699ff79925a0ffc70c6fa654391ce0dd):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9025 -j DNAT --to-destination 172.23.0.2:8080 ! -i br-27e90a56807e: iptables: No chain/target/match by that name.
 (exit status 1))

Output from running docker-compose down; docker-compose up -d with csf disabled:

Creating network "nextcloud-aio_default" with the default driver
ERROR: Failed to Setup IP tables: Unable to enable SKIP DNAT rule:  (iptables failed: iptables --wait -t nat -I DOCKER -i br-0f399d47835e -j RETURN: iptables: No chain/target/match by that name.
 (exit status 1))

Upon running docker-compose up -d after re-enabling csf, with csf -e:

Creating network "nextcloud-aio_default" with the default driver
Creating nextcloud-aio-mastercontainer ... done

After having csf enabled, starting the mastercontainer, then stopping csf, restarting all the containers I get this from the collabora logs:

wsd-00001-00022 2023-08-11 12:41:16.082491 +0200 [ remotefontconfig_poll ] ERR  Failed to lookup host [mynextcloud.domain.com]. Skipping (EAGAIN: Resource temporarily unavailable)| net/NetUtil.cpp:157
wsd-00001-00022 2023-08-11 12:41:16.197122 +0200 [ remotefontconfig_poll ] ERR  #-1: Failed to connect to mynextcloud.domain.com:443| net/HttpRequest.hpp:1257
wsd-00001-00022 2023-08-11 12:41:16.197193 +0200 [ remotefontconfig_poll ] ERR  Remote config server has response status code: Unknown| wsd/COOLWSD.cpp:1174
wsd-00001-00001 2023-08-11 12:42:16.139125 +0200 [ coolwsd ] ERR  Failed to lookup host [rating.collaboraonline.com]. Skipping (EAGAIN: Resource temporarily unavailable)| net/NetUtil.cpp:157
wsd-00001-00001 2023-08-11 12:42:16.139175 +0200 [ coolwsd ] ERR  #-1: Failed to connect to rating.collaboraonline.com:443| net/HttpRequest.hpp:1257
wsd-00001-00001 2023-08-11 12:42:16.139214 +0200 [ coolwsd ] WRN  Failed to update the infobar. Got: Unknown | wsd/COOLWSD.cpp:5548

So, basically the same as the logs in my first post.

I got these lines in the /usr/local/include/csf/post.d/docker.sh (taken from suggestions in this discussion)

/usr/sbin/iptables -I DOCKER-USER -p tcp -i enp0s25 ! -s 192.168.0.0/24 --dport 9000 -j DROP
with multiple such lines for all the open docker ports

Does this give any further clues?

[szaimen]

Failed to Setup IP tables: Unable to enable SKIP DNAT rule

I've found this: https://stackoverflow.com/questions/53766558/iptables-no-chain-target-match-error-with-docker-network-create

[panomitrius]

Failed to Setup IP tables: Unable to enable SKIP DNAT rule

I've found this: https://stackoverflow.com/questions/53766558/iptables-no-chain-target-match-error-with-docker-network-create

When this is working I'm back at

Ready to accept connections on port 9980.

wsd-00001-00016 2023-08-10 13:09:42.531644 +0200 [ remotefontconfig_poll ] ERR  Failed to lookup host [mynextcloud.domain.com]. Skipping (EAGAIN: Resource temporarily unavailable)| net/NetUtil.cpp:157
wsd-00001-00016 2023-08-10 13:09:42.935779 +0200 [ remotefontconfig_poll ] ERR  #-1: Failed to connect to mynextcloud.domain.com:443| net/HttpRequest.hpp:1257
wsd-00001-00016 2023-08-10 13:09:42.935842 +0200 [ remotefontconfig_poll ] ERR  Remote config server has response status code: Unknown| wsd/COOLWSD.cpp:1174

So I paused trying to fix my setup of nextcloud AIO, as I managed to install a separate nextcloud office container now and got that to work..