Upgrade to 2.40 fails with error message

[clarkac1]

Upgrading to 2.40 fails with the error message:

Error: Class "CRM_Contactlayout_DAO_Base" not found in require_once() (line 15 of /home/acivior2/sa.acivi.org.uk/files/civicrm/ext/org.civicrm.contactlayout/CRM/Contactlayout/DAO/ContactLayout.php).

I encountered this on systems at 5.78.3 and also at 5.80.3. Both systems are running Backdrop.

[colemanw]

That's a puzzling one. It works fine for me locally (running Civi-standalone master) but I can't think of any reason it wouldn't work on Backdrop as well. That require_once is just a standard part of the entity-framework v2 civix upgrade, which has also been done for searchkit and afform extensions, so if it works for those why wouldn't it work here?

[clarkac1]

I just tried the upgrade on the last D7 site I have and the upgrade failed with the same message. Well, 'failed' might be inaccurate as afterwards it seems to have been upgraded. This site is running 5.78.3. On one of the Backdrop sites the same thing happened, but on the other the upgrade didn't succeed. However the problem is not peculiar to Backdrop.

[clarkac1]

I just tried another Backdrop site, cleared caches first, ran the upgrade, same crash and now it thinks the upgrade has been done. But... it happened too quickly for the upgrade to have worked. However the extension still seems to be functioning OK.

[colemanw]

Ah, so it's not a problem once the upgrade is done, but the error happens in the in-between zone when the new code is in place but the upgrade hasn't happened yet. Interesting...

[clarkac1]

Yes, that seems to be the case - it crashes very quickly and then in the list of extensions it's now at the new level.

[colemanw]

What if you clear caches after replacing the code but before running the upgrader?

[clarkac1]

I manually replaced the code on another site, cleared caches, but then the upgrade wouldn't run because it thought the new version was there. I should have opened up another screen to clear caches... Now I've run out of sites to play with.

[colemanw]

The other piece of this is that there isn't actually any upgrade to run. There hasn't been any upgrade script needed for this extension since v2.0. So your last attempt IMO behaved correctly. After replacing the code and clearing caches, there's nothing else to do, the extension is upgraded.

[totten]

I've got a dev-loop to elicit the error.

• Setup:
  • Install BD 1.29, Civi 5.78.3, and ContactLayout 2.3.1.
  • Keep a copy of the zip file for 2.3.1.
• Loop: Re-run this script to elicit the error

  rm org.civicrm.contactlayout -rf
  unzip org.civicrm.contactlayout-2.3.1.zip
  civibuild restore bcmaster --no-test
  cv en contactlayout
  cv ev 'echo "OK\n";'
  cv dl org.civicrm.contactlayout -f -v

Full trace:

Using extension feed "https://civicrm.org/extdir/ver=5.78.3|uf=Backdrop|status=stable|ready=ready"
Downloading extension "org.civicrm.contactlayout" (https://github.com/civicrm/org.civicrm.contactlayout/archive/2.4.0.zip)

In ContactLayout.php line 15:
                                                
  [Error]                                       
  Class "CRM_Contactlayout_DAO_Base" not found  
                                                

Exception trace:
  at /home/totten/bknix/build/bcmaster/web/files/civicrm/ext/org.civicrm.contactlayout/CRM/Contactlayout/DAO/ContactLayout.php:15
 require_once() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/CRM/Core/ClassLoader.php:170
 CRM_Core_ClassLoader->loadClass() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/AbstractEntity.php:101
 Civi\Api4\Generic\AbstractEntity::getEntityTitle() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/AbstractEntity.php:146
 Civi\Api4\Generic\AbstractEntity::getInfo() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Provider/ActionObjectProvider.php:166
 Civi\Api4\Provider\ActionObjectProvider->getEntities() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Utils/CoreUtil.php:91
 Civi\Api4\Utils\CoreUtil::getInfoItem() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/AbstractBatchAction.php:73
 Civi\Api4\Generic\AbstractBatchAction->getSelect() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/AbstractBatchAction.php:60
 Civi\Api4\Generic\AbstractBatchAction->getBatchAction() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/AbstractBatchAction.php:40
 Civi\Api4\Generic\AbstractBatchAction->getBatchRecords() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/DAODeleteAction.php:36
 Civi\Api4\Generic\DAODeleteAction->_run() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Provider/ActionObjectProvider.php:70
 Civi\Api4\Provider\ActionObjectProvider->invoke() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/API/Kernel.php:153
 Civi\API\Kernel->runRequest() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/Api4/Generic/AbstractAction.php:256
 Civi\Api4\Generic\AbstractAction->execute() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/CRM/Core/Config.php:290
 CRM_Core_Config->cleanupCaches() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/CRM/Core/Invoke.php:406
 CRM_Core_Invoke::rebuildMenuAndCaches() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/CRM/Extension/Manager.php:231
 CRM_Extension_Manager->replace() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/CRM/Extension/Downloader.php:152
 CRM_Extension_Downloader->download() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/api/v3/Extension.php:240
 civicrm_api3_extension_download() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php:89
 Civi\API\Provider\MagicFunctionProvider->invoke() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/API/Kernel.php:153
 Civi\API\Kernel->runRequest() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/Civi/API/Kernel.php:79
 Civi\API\Kernel->runSafe() at /home/totten/bknix/build/bcmaster/web/modules/civicrm/api/api.php:28
 civicrm_api() at phar:///home/totten/bknix/bin/cv/src/Util/VerboseApi.php:17
 Civi\Cv\Util\VerboseApi::callApi3Success() at phar:///home/totten/bknix/bin/cv/src/Command/ExtensionDownloadCommand.php:111
 Civi\Cv\Command\ExtensionDownloadCommand->execute() at phar:///home/totten/bknix/bin/cv/vendor/symfony/console/Command/Command.php:155
 Cvphar\Symfony\Component\Console\Command\Command->run() at phar:///home/totten/bknix/bin/cv/vendor/symfony/console/Application.php:680
 Cvphar\Symfony\Component\Console\Application->doRunCommand() at phar:///home/totten/bknix/bin/cv/vendor/symfony/console/Application.php:218
 Cvphar\Symfony\Component\Console\Application->doRun() at phar:///home/totten/bknix/bin/cv/lib/src/BaseApplication.php:98
 Civi\Cv\BaseApplication->doRun() at phar:///home/totten/bknix/bin/cv/vendor/symfony/console/Application.php:124
 Cvphar\Symfony\Component\Console\Application->run() at phar:///home/totten/bknix/bin/cv/lib/src/BaseApplication.php:65
 Civi\Cv\BaseApplication->run() at phar:///home/totten/bknix/bin/cv/lib/src/BaseApplication.php:27
 Civi\Cv\BaseApplication::main() at phar:///home/totten/bknix/bin/cv/bin/cv:29
 require() at /home/totten/bknix/bin/cv:15

A few thoughts/observations:

(1) In the middle of the low-level cleanupCaches(), it calls out to APIv4 (Civi\Api4\UserJob::delete()). This forces it to resolve all entity metadata. From the comments, it's not the first time that this step has led to trouble for the extension-lifecycle.

I don't fundamentally understand why we would need to call this step as part of the extension-lifecycle. It could be deleted/moved. However, that's not enough to fix this use-case.

(2) There is a long-standing structural risk in how the download-upgrade step (eg CRM_Extension_Manager::replace()) combines with rebuildMenuAndCaches():

• When you swap-in new code, you cannot immediately reload the PHP files. (You cannot re-define live functions and classes.) Even if you delete the files and make new ones, the old code remains active.
  • For example, old implementations of mymodule.php, mymodule.civix.php, and Civi/MyModule/MySubscriber.php would stay live for the rest of the PHP process.
  • For brevity, let's say the replace() operation means we are still running defunct PHP files.
• The remit of rebuildMenuAndCaches() is two-part.
  • One part is to clear caches. That's fine. Even if you've got defunct PHP files in memory, you can delete existing caches.
  • Another part is re-building caches. Here, defunct PHP files are a major risk.
    • Risk 1: You will directly run the old code, which gives wrong information. (Say, mymodule_civicrm_entityTypes gives an old list instead of a new list.)
    • Risk 2: As a matter of circumstance, you get interactions between defunct PHP files (which were loaded before the call to replace()) and new PHP files (which are loaded after the call to replace()). But these often aren't designed to be mixed.

In this case, I think we're specifically seeing a bad mix of old+new files:

• You've got the old copy of contactlayout.civix.php (which doesn't support EFv2).
• You've got the new copy of CRM/Contactlayout/DAO/ContactLayout.php (which requires EFv2).

(3) In the design of EFv2, we made this change to the DAO classes:

• In the old CRM_*_DAO_* classes, the files were basically self-sufficient and lazy-loaded.
• In the new CRM_*_DAO_* classes, the files depend on a bigger metadata service that resolves more upfront.

I'm not exactly sure how to say it, but (intuitively) I feel like this has increased the risk of provoking the old-file/new-file problem. (Or maybe it was just a very delicate balance to begin with - and anything non-trivial is likely to disrupt it.)

---

Aside: The defunct file problem has only affected the built-in downloader. If you're a developer who uses git or curl or composer to manage all the downloads externally, then you don't notice it.

---

In the past, we've been able to find clever work-arounds for the defunct file problem. And maybe we can find one here too. (That may need some marinating...)

However, IMHO, what's really needed is to split the download/upgrade into two subprocesses -- e.g. two AJAX calls;or two exec('php...') calls; or POST+REDIRECT+GET. The steps are like:

1. Download the new code. Invalidate all the current caches/metadata.
2. Rebuild caches/metadata.

[totten]

Filed a more general issue about the in-app upgrader: https://lab.civicrm.org/dev/core/-/issues/5700. I've got working draft of a fix, but it's a little ugly. I'll post a link on 5700 when it's tidier.

When testing that, I observed a secondary problem with [email protected] on Backdrop:

• Backdrop CLI allows you to read PHAR files. So all the automated civix testing with [email protected] had passed on Backdrop. Great, but...
• Backdrop Web UI completely blocks usage of PHAR files. 🤦

The fix here is to extract the PHAR and provide an equivalent subfolder. (civix will need an update to make the change automatic.)

[totten]

Recap: This highlighted two issues. Solutions for each:

1. The in-app downloader has trouble with Entity Framework v2: This affects any contrib extension with EFv2 on any CMS and any version. (The problem is not unique to EFv2, but it's more visible.) As a categorical solution, the site-builder can apply a hotfix: https://lab.civicrm.org/extensions/hotfix_extup/
2. The Backdrop Web UI is incompatible with EFv2 templates in civix (circa v24.09.1 - v25.01.0): To fix this, the extension author should update to civix v25.01.1+ and tag a new release.

[clarkac1]

This problem also affects the recent update to Geocoder - now 1.14 - which crashes in the same way. Civi at 5.81.2, Backdrop at 1.30. As I understand it from @colemanw remarks above, the crash occurs after the code has been replaced but before any update. Therefore the problem is that I have no idea whether the new release of the extension required an update after replacing the code, so the only safe course of action is to stop updating all extensions until this problem is fixed. Since the fix mentioned above is dependent on extension authors that could be a very long time?? Is there another approach?