Staging site no longer finds components without registry_url in idf_component.yml (PACMAN-1062)

[gojimmypi]

The Component Manager version

2.1.1

ESP-IDF Version

v5.2

python Version

Python 3.10.12

Operating System

WSL

Browser (for https://components.espressif.com Issues)

No response

Description

This issue is related to #78, in particular #78 (comment).

TL;DR: without explicit registry_url: https://components-staging.espressif.com in main/idf_component.yml, staging site components are not found.

I believe this was introduced in a recent version of the idf_component_manager. I'm using v2.1.1

I can fix this moving forward, but all prior staging site components are now inoperable.

To Reproduce

Details

Without the IDF_COMPONENT_REGISTRY_URL staging site setting, the component is of course not found at https://components.espressif.com/

$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1c:wolfssh_echoserver"
Executing action: create-project-from-example
ERROR: Component "gojimmypi/mywolfssh" not found
gojimmypi:/mnt/c/testc
$  echo $IDF_COMPONENT_REGISTRY_URL

And of course with a IDF_COMPONENT_REGISTRY_URL staging site value, the library is found:

gojimmypi:/mnt/c/testc
$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1c:wolfssh_echoserver"
Executing action: create-project-from-example
NOTICE: Example "wolfssh_echoserver" successfully downloaded to /mnt/c/testc/wolfssh_echoserver
Done
gojimmypi:/mnt/c/testc
$ echo $IDF_COMPONENT_REGISTRY_URL
https://components-staging.espressif.com

BUT - try the same thing with a prior version, mywolfssh=1.4.19-preview1a:

gojimmypi:/mnt/c/testc
$ cd test1a
gojimmypi:/mnt/c/testc/test1a
$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1a:wolfssh_echoserver"
Executing action: create-project-from-example
NOTICE: Example "wolfssh_echoserver" successfully downloaded to /mnt/c/testc/test1a/wolfssh_echoserver
Done

The project is fetched, but no component source code yet. (fetched at first-build time)

$ cd test1a
gojimmypi:/mnt/c/testc/test1a


$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1a:wolfssh_echoserver"
Executing action: create-project-from-example
NOTICE: Example "wolfssh_echoserver" successfully downloaded to /mnt/c/testc/test1a/wolfssh_echoserver
Done


gojimmypi:/mnt/c/testc/test1a
$ ls
wolfssh_echoserver
gojimmypi:/mnt/c/testc/test1a
$ ls wolfssh_echoserver
CMakeLists.txt  README.md  VisualGDB  main  partitions_singleapp_large.csv  sdkconfig.defaults
gojimmypi:/mnt/c/testc/test1a
$ ls wolfssh_echoserver/main
CMakeLists.txt  echoserver.c  idf_component.yml  include  main.c  time_helper.c  wifi_connect.c
gojimmypi:/mnt/c/testc/test1a

These are known to be bad version, we'll edit, later:

$ cat wolfssh_echoserver/main/idf_component.yml
dependencies:
  gojimmypi/mywolfssh: ^1.4.17-Preview6
  gojimmypi/mywolfssl: ^5.7.1-Preview2d
  idf:
    version: '>=4.1.0'
gojimmypi:/mnt/c/testc/test1a

Edit the wolfssh_echoserver/main/idf_component.yml with recent libraries known to exist:

$ nano wolfssh_echoserver/main/idf_component.yml
gojimmypi:/mnt/c/testc/test1a
$ cat wolfssh_echoserver/main/idf_component.yml
dependencies:
  gojimmypi/mywolfssh: ^1.4.19-preview1c
  gojimmypi/mywolfssl: ^5.7.4-preview1i
  idf:
    version: '>=4.1.0'
gojimmypi:/mnt/c/testc/test1a

See

• gojimmypi/mywolfssh/versions/1.4.19-preview1c
• gojimmypi/mywolfssl/versions/5.7.4-preview1i

gojimmypi:/mnt/c/testc/test1a
$ cd wolfssh_echoserver
gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ idf.py build
Executing action: all (aliases: build)
Running cmake in directory /mnt/c/testc/test1a/wolfssh_echoserver/build

[ ... snip ... ]

NOTICE: Dependencies lock doesn't exist, solving dependencies.
.HINT: Please check manifest file of the following component(s): main
CMake Error at /mnt/c/SysGCC/esp32/esp-idf/v5.2/tools/cmake/build.cmake:544 (message):
  WARNING: Component "gojimmypi/mywolfssh" not found

  WARNING: Component "gojimmypi/mywolfssl" not found

  ERROR: Because project depends on gojimmypi/mywolfssl (^5.7.4-preview1i)
  which doesn't match any versions, version solving failed.

Call Stack (most recent call first):
  /mnt/c/SysGCC/esp32/esp-idf/v5.2/tools/cmake/project.cmake:605 (idf_build_process)
  CMakeLists.txt:122 (project)


-- Configuring incomplete, errors occurred!
See also "/mnt/c/testc/test1a/wolfssh_echoserver/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1, output of the command is in the /mnt/c/testc/test1a/wolfssh_echoserver/build/log/idf_py_stderr_output_4255 and /mnt/c/testc/test1a/wolfssh_echoserver/build/log/idf_py_stdout_output_4255


Edit the `main/idf_component.yml` to explicitly name the `registry_url` (even though we already have `IDF_COMPONENT_REGISTRY_URL` set)


gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ cat main/idf_component.yml
dependencies:
  gojimmypi/mywolfssh:
    version: ^1.4.19-preview1c
    registry_url: https://components-staging.espressif.com
  gojimmypi/mywolfssl:
    version: ^5.7.4-preview1i
    registry_url: https://components-staging.espressif.com
  idf:
    version: '>=4.1.0'

$ echo $IDF_COMPONENT_REGISTRY_URL
https://components-staging.espressif.com


Observe components now all found, and project builds successfully:


gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ idf.py build
Executing action: all (aliases: build)
Running cmake in directory /mnt/c/testc/test1a/wolfssh_echoserver/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/home/gojimmypi/.espressif/python_env/idf5.2_py3.10_env/bin/python -DESP_PLATFORM=1 -DCCACHE_ENABLE=0 /mnt/c/testc/test1a/wolfssh_echoserver"...

[ ... snip ... ]

-- Building ESP-IDF components for target esp32
NOTICE: Dependencies lock doesn't exist, solving dependencies.
...NOTICE: Updating lock file at /mnt/c/testc/test1a/wolfssh_echoserver/dependencies.lock
NOTICE: Processing 3 dependencies:
NOTICE: [1/3] gojimmypi/mywolfssh (1.4.19-preview1c)
NOTICE: [2/3] gojimmypi/mywolfssl (5.7.4-preview7aj)
NOTICE: [3/3] idf (5.2.0)

[ ... snip ... ]

[1053/1054] Generating binary image from built executable
esptool.py v4.8.1
Creating esp32 image...
Merged 2 ELF sections
Successfully created esp32 image.
Generated /mnt/c/testc/test1a/wolfssh_echoserver/build/wolfssh_echoserver.bin
[1054/1054] cd /mnt/c/testc/test1a/wolfssh_echoserver/build/es...c/testc/test1a/wolfssh_echoserver/build/wolfssh_echoserver.bi
wolfssh_echoserver.bin binary size 0xf7e40 bytes. Smallest app partition is 0x177000 bytes. 0x7f1c0 bytes (34%) free.

Project build complete. To flash, run:
 idf.py flash
or
 idf.py -p PORT flash
or
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_size 2MB --flash_freq 40m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/wolfssh_echoserver.bin
or from the "/mnt/c/testc/test1a/wolfssh_echoserver/build" directory
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash "@flash_args"

This is all with the same ISP-IDF and component manager versions:

$  idf.py --version
ESP-IDF v5.2-dev-3903-g66992aca7a-dirty
gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ python -m idf_component_manager --help
Usage: python -m idf_component_manager [OPTIONS] COMMAND [ARGS]...

Options:
  -W, --warnings-as-errors  Treat warnings as errors.
  -h, --help                Show this message and exit.

Commands:
  autocomplete  Generate tab-completion scripts for the specified shell.
  cache         Group of commands to manage cache of the IDF Component...
  component     Group of commands to interact with components.
  manifest      Group of commands to manage manifest of the project.
  project       Group of project related commands
  registry      Group of commands to work with component registry
  version       Print version of the IDF Component Manager.
gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ python -m idf_component_manager version
2.1.1

Expected behaviour

Previously, staging components would be found without registry_url setting in idf_component.yml.

If there's a IDF_COMPONENT_REGISTRY_URL environment setting, that should be the default for missing registry_url values.

It is a potentially cool feature (mix staging and release components? alternate, provider-specific sources?).

Additional info

See #78 (comment)

Test on Staging Registry Documentation at

https://docs.espressif.com/projects/idf-component-manager/en/latest/guides/packaging_components.html#test-on-staging-registry

I have checked existing issues and online Documentation

• I confirm I have checked existing issues and online Documentation.

[kumekay]

Hello @gojimmypi

First, I apologize that you have issues with the registry, and it took you time to figure out what is going on.

The most common use case for the staging registry is to quickly test uploading and work of a single component before publishing it on the main registry. With the old behavior the staging environment was not very usable for many users, because it was impossible to solve dependencies for components on staging to have non-direct dependencies on the components from production.

So you guessed the intention correctly, we want to allow mixing staging and production components together.

However, it looks like there is an issue with the implementation.

The expected behavior in component manager 2.1 is : Setting of IDF_COMPONENT_REGISTRY_URL only affects direct dependencies specified in idf_component.yml files of your project, but not the ones fetched from the registry. So you don't need to specify it for every dependency.

We will fix this issue, but older components which depend on other components on staging and do not have registry_url set explicitly will stay broken. But demo projects should work with the env variable configured.