Nextcloud AIO v6.2.1 via Docker runs out of database connections with "FATAL: sorry, too many clients already" even with minimal load

[alinradut]

Steps to reproduce

1. Install Nextcloud AIO v6.2.1 via Docker on a DigitalOcean 2vCPU 2GB RAM 40GB storage VM
2. Select the bare essentials to get a file server running:

• Apache
• Database
• Nextcloud
• Notify Push
• Redis
• Collabora

3. Disable all unnecessary apps (entire app list below)
4. Create one or more additional non-admin accounts and log in with it in private browsing window
5. Upload 20 documents using the non-admin account
6. Select all the documents in the listing, right click and delete. If everything works correctly, immediately re-do the upload/delete scenario.

Expected behavior

The files are deleted.

Actual behavior

One or more files are deleted, but a good chunk of them will error out because the server will be complaining about running out of Postgres connections. The Nextcloud installation will then be inaccessible for the next ~10 minutes (probably until the db connections are shut down).

Host OS

Ubuntu 22.04.2 LTS

Nextcloud AIO version

v6.2.1

Current channel

latest

Other valuable info

The number of connections based on the formula will be ~110 connections I don't understand what could cause all of them to be depleted.

Deleting the files one by one will not cause this behavior.

I am the only person using this server, so it's not like many other users are hammering it.

nextcloud-aio-database log

2023-07-17 10:03:00.790 UTC [170698] FATAL:  sorry, too many clients already
2023-07-17 10:03:00.793 UTC [170699] FATAL:  sorry, too many clients already
2023-07-17 10:03:00.796 UTC [170700] FATAL:  sorry, too many clients already
2023-07-17 10:03:01.596 UTC [170701] FATAL:  sorry, too many clients already
2023-07-17 10:03:01.600 UTC [170702] FATAL:  sorry, too many clients already
2023-07-17 10:03:01.604 UTC [170703] FATAL:  sorry, too many clients already
2023-07-17 10:03:01.607 UTC [170704] FATAL:  sorry, too many clients already
2023-07-17 10:03:01.610 UTC [170705] FATAL:  sorry, too many clients already

....

2023-07-17 10:48:52.599 UTC [171794] FATAL:  remaining connection slots are reserved for non-replication superuser connections
2023-07-17 10:49:22.675 UTC [171805] FATAL:  remaining connection slots are reserved for non-replication superuser connections
2023-07-17 10:49:52.758 UTC [171815] FATAL:  remaining connection slots are reserved for non-replication superuser connections
2023-07-17 10:50:22.831 UTC [171826] FATAL:  remaining connection slots are reserved for non-replication superuser connections
2023-07-17 10:50:52.922 UTC [171836] FATAL:  remaining connection slots are reserved for non-replication superuser connections
2023-07-17 10:51:23.009 UTC [171847] FATAL:  remaining connection slots are reserved for non-replication superuser connections
2023-07-17 10:51:53.086 UTC [171857] FATAL:  remaining connection slots are reserved for non-replication superuser connections

occ app:list

Enabled:
  - activity: 2.18.0
  - admin_audit: 1.16.0
  - bruteforcesettings: 2.6.0
  - cloud_federation_api: 1.9.0
  - comments: 1.16.0
  - contacts: 5.3.2
  - contactsinteraction: 1.7.0
  - dav: 1.25.0
  - encryption: 2.14.0
  - federatedfilesharing: 1.16.0
  - files: 1.21.1
  - files_pdfviewer: 2.7.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_trashbin: 1.16.0
  - files_versions: 1.19.1
  - firstrunwizard: 2.15.0
  - groupfolders: 14.0.2
  - logreader: 2.11.0
  - lookup_server_connector: 1.14.0
  - nextcloud-aio: 0.4.0
  - notify_push: 0.6.3
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - privacy: 1.10.0
  - provisioning_api: 1.16.0
  - quota_warning: 1.17.0
  - richdocuments: 8.0.2
  - serverinfo: 1.16.0
  - settings: 1.8.0
  - sharebymail: 1.16.0
  - support: 1.9.0
  - suspicious_login: 4.4.0
  - systemtags: 1.16.0
  - text: 3.7.2
  - theming: 2.1.1
  - twofactor_backupcodes: 1.15.0
  - twofactor_totp: 8.0.0
  - viewer: 1.10.0
  - workflowengine: 2.8.0
Disabled:
  - calendar: 4.4.3 (installed 4.4.3)
  - circles: 26.0.0 (installed 26.0.0)
  - dashboard: 7.6.0 (installed 7.6.0)
  - deck: 1.9.2 (installed 1.9.2)
  - federation: 1.16.0 (installed 1.16.0)
  - files_external: 1.18.0
  - nextcloud_announcements: 1.15.0 (installed 1.15.0)
  - notes: 4.8.0 (installed 4.8.0)
  - notifications: 2.14.0 (installed 2.14.0)
  - photos: 2.2.0 (installed 2.2.0)
  - recommendations: 1.5.0 (installed 1.5.0)
  - related_resources: 1.1.0-alpha1 (installed 1.1.0-alpha1)
  - survey_client: 1.14.0 (installed 1.14.0)
  - tasks: 0.15.0 (installed 0.15.0)
  - user_ldap: 1.16.0
  - user_status: 1.6.0 (installed 1.6.0)
  - weather_status: 1.6.0 (installed 1.6.0)

occ log:tail

An unhandled exception has been thrown:
Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] connection to server at "nextcloud-aio-database" (172.18.0.3), port 5432 failed: FATAL:  sorry, too many clients already in /var/www/html/lib/private/DB/Connection.php:140
Stack trace:
#0 /var/www/html/3rdparty/doctrine/dbal/src/Connection.php(1531): OC\DB\Connection->connect()
#1 /var/www/html/3rdparty/doctrine/dbal/src/Connection.php(1029): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /var/www/html/lib/private/DB/Connection.php(262): Doctrine\DBAL\Connection->executeQuery('SELECT * FROM "...', Array, Array, NULL)
#3 /var/www/html/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\DB\Connection->executeQuery('SELECT * FROM "...', Array, Array)
#4 /var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php(280): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /var/www/html/lib/private/AppConfig.php(418): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /var/www/html/lib/private/AppConfig.php(184): OC\AppConfig->loadConfigValues()
#7 /var/www/html/lib/private/AppConfig.php(374): OC\AppConfig->getApps()
#8 /var/www/html/lib/private/legacy/OC_App.php(976): OC\AppConfig->getValues(false, 'installed_versi...')
#9 /var/www/html/lib/private/Server.php(731): OC_App::getAppVersions()
#10 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\Server->OC\{closure}(Object(OC\Server))
#11 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#12 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OC\\Memcache\\Fac...')
#13 /var/www/html/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OC\\Memcache\\Fac...', true)
#14 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OC\\Memcache\\Fac...')
#15 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(193): OC\AppFramework\Utility\SimpleContainer->get('OC\\Memcache\\Fac...')
#16 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#17 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#18 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OCP\\ICacheFacto...')
#19 /var/www/html/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OCP\\ICacheFacto...', true)
#20 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OCP\\ICacheFacto...')
#21 /var/www/html/lib/private/Server.php(1112): OC\AppFramework\Utility\SimpleContainer->get('OCP\\ICacheFacto...')
#22 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\Server->OC\{closure}(Object(OC\Server))
#23 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#24 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OCP\\Lock\\ILocki...')
#25 /var/www/html/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OCP\\Lock\\ILocki...', true)
#26 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OCP\\Lock\\ILocki...')
#27 /var/www/html/lib/private/Server.php(2067): OC\AppFramework\Utility\SimpleContainer->get('OCP\\Lock\\ILocki...')
#28 /var/www/html/lib/private/Files/View.php(119): OC\Server->getLockingProvider()
#29 /var/www/html/lib/private/Server.php(465): OC\Files\View->__construct()
#30 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\Server->OC\{closure}(Object(OC\Server))
#31 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#32 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OC\\Files\\Node\\H...')
#33 /var/www/html/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OC\\Files\\Node\\H...', true)
#34 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OC\\Files\\Node\\H...')
#35 /var/www/html/lib/private/Server.php(1469): OC\AppFramework\Utility\SimpleContainer->get('OC\\Files\\Node\\H...')
#36 /var/www/html/lib/base.php(627): OC\Server->boot()
#37 /var/www/html/lib/base.php(1169): OC::init()
#38 /var/www/html/console.php(48): require_once('/var/www/html/l...')
#39 /var/www/html/occ(11): require_once('/var/www/html/c...')

Thanks for any ideas on how I could fix this! I intend to allow this server to be used by up to 10 people to occasionally upload and share files. Checking htop reveals that there is plenty of CPU/RAM to spare. I even set up an additional 2GB swap in case it would help, but it made no impact.

[vsaw]

I have the same issue, Postgres refuses additional connections. My feeling is this happens when someone is using Collabora in the Browser, but I can not be certain.

[szaimen]

Hi, with collabora enabled you need 3gb ram as recommended in the AIO interface (2gb for starting correctly +1gb ram for stable operation.)

Also see #1335

[vsaw]

Not sure this is the only reason this may be happening. My system has 8 GB RAM. And it was working fine with Collabora enabled for a few months now (always NextCloud AIO Latest with updates enabled). The outages started happening a few days ago, so I suspect an software issue, not resource exhaustion.

[vsaw]

@szaimen I checked resource usage of my system. The overall system uses roughly 50% of available RAM, with more then 3 GB of unused free RAM.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       3.7Gi       391Mi       271Mi       3.7Gi       3.5Gi
Swap:             0B          0B          0B

Nextcloud and Docker itself do not have a RAM limit set up and can use all of existing RAM if necessary.

$ docker stats
CONTAINER ID   NAME                              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
81745ea305ec   nextcloud-aio-apache              0.57%     49.7MiB / 7.763GiB    0.63%     6.19MB / 19.5MB   1.68MB / 1.74MB   123
51b0ef14dfef   nextcloud-aio-notify-push         0.04%     2.066MiB / 7.763GiB   0.03%     220kB / 158kB     61.4kB / 0B       6
7ea85e46d350   nextcloud-aio-nextcloud           9.92%     251.7MiB / 7.763GiB   3.17%     149MB / 35.2MB    9.13MB / 95.5MB   17
b4887e63b751   nextcloud-aio-imaginary           0.00%     6.258MiB / 7.763GiB   0.08%     3.78kB / 0B       20.5kB / 0B       10
2be2645bdd75   nextcloud-aio-fulltextsearch      0.24%     1.345GiB / 7.763GiB   17.32%    76.7kB / 7.94kB   2.83MB / 22.2MB   68
32f53ac8677f   nextcloud-aio-redis               0.33%     4.48MiB / 7.763GiB    0.06%     7.85MB / 4.93MB   385kB / 24.6MB    5
b9425d32db41   nextcloud-aio-database            3.16%     64.2MiB / 7.763GiB    0.81%     22.5MB / 69.9MB   40.6MB / 190MB    7
cdb17975898e   nextcloud-aio-talk                5.74%     39.61MiB / 7.763GiB   0.50%     251MB / 573MB     336kB / 213kB     66
432f3e31c426   nextcloud-aio-collabora           0.03%     260.2MiB / 7.763GiB   3.27%     1.14MB / 364kB    41kB / 442MB      10

I haven't checked RAM or other resource usage when I get the database connection error, but I doubt it is because of resource exhaustion. And if it is because of resource exhaustion, I wonder why? (My instance has only 3 registered users, so I don't know how they could bring down the entire system by opening a few documents)

Other docker services removed for brevity.

[Jayd603]

Experienced this same issue today... similar set up to OP. I will increase RAM but it looked like there was enough. Oh and in my case it never recovered and the service never came back without intervention.

[mangabs]

I have the same issue. My system has 4GB RAM. My instance has only 3 registered users. My setup is AIO 7.2.1 with default options (Collabora, Imaginary, Nextcloud Talk). For me as well, the service never came back without intervention. I am also concerned by the problem #3378 . Same issue? It seems that It has something to do with Collabora: if I remove it, I haven't this problem.