Issues with updates in Nextcloud AIO v7.2.1 - Containers unhealthy after backup. #3423

EgonHeuson · 2023-09-27T14:24:14Z

EgonHeuson
Sep 27, 2023

Hello all,
It's me again ^^ I have had few issues on the last 3-4 weeks with my NCAIO instance running on my small server. At some point, apparently during the backup (according to the logs (see bellow)), it seems that there is an error happening that prevents the containers to relaunch properly, which make them end up into un unhealthy state for most of them. The cloud consequently is not anymore available, and I have issues with stoping manually the containers. Usually what I do to solve it is that I restart the VM on my Proxmox server and start the containers manually again.

I have checked the logs of all containers, and the 2 ones where I could find interesting/error messages are:

nextcloud-aio-mastercontainer

+ export BACKUP_TIME
+ export DAILY_BACKUP=1
+ DAILY_BACKUP=1
++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
+ '[' '' '!=' automaticUpdatesAreNotEnabled ']'
+ export AUTOMATIC_UPDATES=1
+ AUTOMATIC_UPDATES=1
+ set +x
Daily backup script has started
grep: write error: Broken pipe
Starting mastercontainer update...
(The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
Waiting for watchtower to stop
Waiting for watchtower to stop
Creating daily backup...
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Starting and updating containers...
Sending backup notification...
Daily backup script has finished
Total reclaimed space: 0B
++ head -1 /mnt/docker-aio-config/data/daily_backup_time
+ BACKUP_TIME=03:00
+ export BACKUP_TIME
+ export DAILY_BACKUP=1
+ DAILY_BACKUP=1
++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
+ '[' '' '!=' automaticUpdatesAreNotEnabled ']'
+ export AUTOMATIC_UPDATES=1
+ AUTOMATIC_UPDATES=1
+ set +x
Daily backup script has started
grep: write error: Broken pipe
Starting mastercontainer update...
(The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
Waiting for watchtower to stop
Waiting for watchtower to stop
Waiting for watchtower to stop
Waiting for watchtower to stop
Waiting for watchtower to stop
Creating daily backup...
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Starting and updating containers...
Fatal error: Uncaught GuzzleHttp\Exception\ServerException: Server error: `POST http://localhost/v1.41/containers/nextcloud-aio-nextcloud/start` resulted in a `500 Internal Server Error` response:
{"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process (truncated...)
 in /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
�������
Stack trace:
#0 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Middleware.php(72): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response), NULL, Array, NULL)
#1 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(209): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))
#2 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(158): GuzzleHttp\Promise\Promise::callHandler(1, Object(GuzzleHttp\Psr7\Response), NULL)
#3 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#4 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Promise\TaskQueue->run(true)
#5 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
#6 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
#7 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
#8 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
#9 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Client.php(189): GuzzleHttp\Promise\Promise->wait()
#10 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/ClientTrait.php(95): GuzzleHttp\Client->request('POST', 'http://localhos...', Array)
#11 /var/www/docker-aio/php/src/Docker/DockerActionManager.php(189): GuzzleHttp\Client->post('http://localhos...')
#12 /var/www/docker-aio/php/src/Controller/DockerController.php(48): AIO\Docker\DockerActionManager->StartContainer(Object(AIO\Container\Container))
#13 /var/www/docker-aio/php/src/Controller/DockerController.php(33): AIO\Controller\DockerController->PerformRecursiveContainerStart('nextcloud-aio-n...', true)
#14 /var/www/docker-aio/php/src/Controller/DockerController.php(193): AIO\Controller\DockerController->PerformRecursiveContainerStart('nextcloud-aio-a...', true)
#15 /var/www/docker-aio/php/src/Cron/StartAndUpdateContainers.php(17): AIO\Controller\DockerController->startTopContainer(true)
#16 {main}
  thrown in /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php on line 113
Something seems to be wrong: Nextcloud should be started at this step.
Sending backup notification...
Daily backup script has finished
Total reclaimed space: 0B
++ head -1 /mnt/docker-aio-config/data/daily_backup_time
+ BACKUP_TIME=03:00
+ export BACKUP_TIME
+ export DAILY_BACKUP=1
+ DAILY_BACKUP=1
++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
+ '[' '' '!=' automaticUpdatesAreNotEnabled ']'
+ export AUTOMATIC_UPDATES=1
+ AUTOMATIC_UPDATES=1
+ set +x

And nextcloud-aio-redis:

Memory overcommit is disabled but necessary for safe operation
See https://github.com/nextcloud/all-in-one/discussions/1731 how to enable overcommit
7:C 27 Sep 2023 06:50:53.363 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

I guess both are not related, but maybe you have an idea on this? In any case I'll follow the procedure listed in #1731 to fix the memory issue with redis, to improve the overall stability :-). But for the other issue I'm not sure how to get rid of it.

Thanks in advance for your help!

Answered by szaimen

Sep 27, 2023

Indeed, your issue is something different. I found this: https://www.google.com/search?q=failed%20to%20create%20shim%20task%3A%20OCI%20runtime%20create%20failed%3A%20runc%20create%20failed%3A%20unable%20to%20start%20container%20process%20proxmox

View full answer

szaimen · 2023-09-27T14:49:55Z

szaimen
Sep 27, 2023
Maintainer

This reminds me of #3385

5 replies

EgonHeuson Sep 27, 2023
Author

In the case of #3385 restarting the containers was the solution, but it happens almost once a week, so I'm not sure this is a normal behavior 😅

Ad the author didn't mention this type of error:

Starting and updating containers...
Fatal error: Uncaught GuzzleHttp\Exception\ServerException: Server error: `POST http://localhost/v1.41/containers/nextcloud-aio-nextcloud/start` resulted in a `500 Internal Server Error` response:
{"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process (truncated...)
 in /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
�������

szaimen Sep 27, 2023
Maintainer

Indeed, your issue is something different. I found this: https://www.google.com/search?q=failed%20to%20create%20shim%20task%3A%20OCI%20runtime%20create%20failed%3A%20runc%20create%20failed%3A%20unable%20to%20start%20container%20process%20proxmox

Answer selected by szaimen

EgonHeuson Sep 27, 2023
Author

However the same error was reported here: #2991
So I'll try to "reset" properly the instance :-) And see what I get :-)

szaimen Sep 27, 2023
Maintainer

However the same error was reported here: #2991

this is a different issue IIRC...

EgonHeuson Sep 27, 2023
Author

Also based on this:

#14 /var/www/docker-aio/php/src/Controller/DockerController.php(193): AIO\Controller\DockerController->PerformRecursiveContainerStart('nextcloud-aio-a...', true)
#15 /var/www/docker-aio/php/src/Cron/StartAndUpdateContainers.php(17): AIO\Controller\DockerController->startTopContainer(true)
#16 {main}
  thrown in /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php on line 113
Something seems to be wrong: Nextcloud should be started at this step.
Sending backup notification...

I wonder if it doesn't come from the AIO-mastercontainer.

EgonHeuson · 2023-09-27T14:56:59Z

EgonHeuson
Sep 27, 2023
Author

Indeed, your issue is something different. I found this: https://www.google.com/search?q=failed%20to%20create%20shim%20task%3A%20OCI%20runtime%20create%20failed%3A%20runc%20create%20failed%3A%20unable%20to%20start%20container%20process%20proxmox

Not sure either. I had no issue for months before this started to happen (no changed done on the instance, just classical regular updates). And I have only issue with my NC VM on proxmox

1 reply

szaimen Sep 27, 2023
Maintainer

Maybe these threads help then, always AIO was not the underlying issue: https://github.com/nextcloud/all-in-one/discussions?discussions_q=is%3Aopen+failed+to+create+shim+task%3A+OCI+runtime+create+failed%3A+runc+create+failed%3A+unable+to+start+container+process

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues with updates in Nextcloud AIO v7.2.1 - Containers unhealthy after backup. #3423

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issues with updates in Nextcloud AIO v7.2.1 - Containers unhealthy after backup. #3423

EgonHeuson Sep 27, 2023

Replies: 2 comments · 6 replies

szaimen Sep 27, 2023 Maintainer

EgonHeuson Sep 27, 2023 Author

szaimen Sep 27, 2023 Maintainer

EgonHeuson Sep 27, 2023 Author

szaimen Sep 27, 2023 Maintainer

EgonHeuson Sep 27, 2023 Author

EgonHeuson Sep 27, 2023 Author

szaimen Sep 27, 2023 Maintainer

EgonHeuson
Sep 27, 2023

Replies: 2 comments 6 replies

szaimen
Sep 27, 2023
Maintainer

EgonHeuson Sep 27, 2023
Author

szaimen Sep 27, 2023
Maintainer

EgonHeuson Sep 27, 2023
Author

szaimen Sep 27, 2023
Maintainer

EgonHeuson Sep 27, 2023
Author

EgonHeuson
Sep 27, 2023
Author

szaimen Sep 27, 2023
Maintainer