Failed to resolve link 'nats' with type 'nats'

[eyett]

What is this issue about?

I'm trying to update my pretty old bosh and cf versions to the latest, and facing this issue.

I've started updating cf first, then when I'm trying to redeploy logsearch and prometheus, there's this error

Task 816 | 06:36:42 | Error: Failed to resolve links from deployment 'prometheus'. See errors below:
  - Failed to resolve link 'nats' with type 'nats' from job 'route_registrar' in instance group 'nginx'. Details below:
    - No link providers found

Task 816 Started  Mon Jun  5 06:36:41 UTC 2023
Task 816 Finished Mon Jun  5 06:36:42 UTC 2023
Task 816 Duration 00:00:01
Task 816 error

Updating deployment:
  Expected task '816' to succeed but state is 'error'

Exit code 1

I can see from my updated cf-deployment.yml that job nats is removed and there's only nats-tls left

  jobs:
  - name: nats-tls
    release: nats
    provides:
      nats-tls: {as: nats-tls, shared: true}
    properties:

But prometheus-boshrelease seems to reference nats still. I can see from the enable-cf-route-registrar.yml

- type: replace
  path: /instance_groups/name=nginx/jobs/-
  value:
    name: route_registrar
    release: routing
    consumes:
      nats:
        from: nats
        deployment: ((cf_deployment_name))

Does this mean I can not use the latest cf version to use with logsearch-boshrelease and prometheus-boshrelease?
I've tried to change the nats alias to nats-tls but it's not working anyway.

- type: replace
  path: /instance_groups/name=nginx/jobs/-
  value:
    name: route_registrar
    release: routing
    consumes:
      nats:
        from: nats-tls
        deployment: ((cf_deployment_name))

+ name: prometheus
Task 1379

Task 1379 | 11:58:56 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 1379 | 11:58:57 | Preparing deployment: Preparing deployment (00:00:01)
Task 1379 | 11:58:58 | Preparing deployment: Rendering templates (00:00:04)
                    L Error: Unable to render instance groups for deployment. Errors are:
  - Unable to render jobs for instance group 'nginx'. Errors are:
    - Unable to render templates for job 'route_registrar'. Errors are:
      - Error filling in template 'registrar_settings.json.erb' (line 8: Can't find link 'nats')
Task 1379 | 11:59:02 | Error: Unable to render instance groups for deployment. Errors are:
  - Unable to render jobs for instance group 'nginx'. Errors are:
    - Unable to render templates for job 'route_registrar'. Errors are:
      - Error filling in template 'registrar_settings.json.erb' (line 8: Can't find link 'nats')

Tried to restore nats job and link in cf-deployment.yml side, but deploying cf fails with the following error

Task 1512 | 20:33:37 | Preparing deployment: Preparing deployment (00:00:31)
Task 1512 | 20:34:08 | Preparing deployment: Rendering templates (00:00:20)
                    L Error: Unable to render instance groups for deployment. Errors are:
  - Unable to render jobs for instance group 'nats'. Errors are:
    - Unable to render templates for job 'nats'. Errors are:
      - Error filling in template 'ca.pem.erb' (line 1: Can't find property '["nats.migrate_server.tls.ca"]')
      - Error filling in template 'certificate.pem.erb' (line 1: Can't find property '["nats.migrate_server.tls.certificate"]')
      - Error filling in template 'private_key.pem.erb' (line 1: Can't find property '["nats.migrate_server.tls.private_key"]')
      - Error filling in template 'ca.pem.erb' (line 1: Can't find property '["nats.migrate_client.tls.ca"]')
      - Error filling in template 'certificate.pem.erb' (line 5: nats.migrate_client.tls.certificate not provided in nats job properties)
      - Error filling in template 'private_key.pem.erb' (line 5: nats.migrate_client.tls.private_key not provided in nats job properties)
Task 1512 | 20:34:28 | Error: Unable to render instance groups for deployment. Errors are:
  - Unable to render jobs for instance group 'nats'. Errors are:
    - Unable to render templates for job 'nats'. Errors are:
      - Error filling in template 'ca.pem.erb' (line 1: Can't find property '["nats.migrate_server.tls.ca"]')
      - Error filling in template 'certificate.pem.erb' (line 1: Can't find property '["nats.migrate_server.tls.certificate"]')
      - Error filling in template 'private_key.pem.erb' (line 1: Can't find property '["nats.migrate_server.tls.private_key"]')
      - Error filling in template 'ca.pem.erb' (line 1: Can't find property '["nats.migrate_client.tls.ca"]')
      - Error filling in template 'certificate.pem.erb' (line 5: nats.migrate_client.tls.certificate not provided in nats job properties)
      - Error filling in template 'private_key.pem.erb' (line 5: nats.migrate_client.tls.private_key not provided in nats job properties)

Task 1512 Started  Mon Jun  5 20:33:37 UTC 2023
Task 1512 Finished Mon Jun  5 20:34:28 UTC 2023
Task 1512 Duration 00:00:51
Task 1512 error

Updating deployment:
  Expected task '1512' to succeed but state is 'error'

Exit code 1

What version of cf-deployment are you using?

cf-deployment v30.2.0

Please include the bosh deploy... command, including all the operations files (plus any experimental operation files you're using):

I'm deploying this cf-deployment manifest.

bosh int upstream/cf_deployment/cf-deployment.yml \
-o upstream/cf_deployment/operations/use-external-blobstore.yml \
-o upstream/cf_deployment/operations/use-s3-blobstore.yml \
-o upstream/logsearch-boshrelease/deployment/operations/cf/add-firehose-to-syslog-uaa-clients.yml \ 
-o upstream/logsearch-boshrelease/deployment/operations/cf/add-grafana-uaa-clients.yml \ 
-o upstream/prometheus-boshrelease/manifests/operations/cf/add-prometheus-uaa-clients.yml \ 
-o upstream/cf_deployment/operations/override-app-domains.yml \
-o upstream/cf_deployment/operations/scale-database-cluster.yml \
-o upstream/cf_deployment/operations/use-compiled-release.yml \

What IaaS is this issue occurring on?

AWS

Is there anything else unique or special about your setup?

Tag your pair, your PM, and/or team!

It's helpful to tag a few other folks on your team or your team alias in case we need to follow up later.

[maxmoehl]

Most likely caused by #932. I faintly remember that there were some issues during the migration, I'll take a look and see if I can find some references for this.

[maxmoehl]

If I remember correctly you have to ensure that both nats and nats-tls are available, migrate all deployments from nats to nats-tls and only after that you can remove the nats job (=> cf-d v17). But since the most recent cf-d is missing nats already you are seeing those issues.

In any case you are right: deployments relying on the nats link are no longer compatible with recent versions of cf-d and have to be migrated.

[eyett]

Appreciate for the info.

I was able to bypass the issue and go on with deployments by restoring plaintext nats jobs in nats node on the cf side.
But I had to add migrate_client and migrate server properties in nats properties additionally, which had not been there at first...
I have not tested functionality yet but at least it removed the error at the rendering template step of deployments.