Better handling of resource naming/metadata

[TonyWildish-BH]

There are a number of tickets related to scalability and naming of things like resource groups or storage accounts, all of which boil down to the fact that relying on naming conventions for resources limits the namespace and causes collisions, which limits scalability. E.g., #3921, #2893, #3666, #3862, #3243, #1073.

The fixes proposed for these issues have all relied on increasing randomness in the suffix, appending unique ids which then get passed around etc. I think this is not the best approach going forwards for two main reasons:

1. using deterministic names inevitably means reduced namespace for resources, since there are fewer characters left to carry the randomness, which will lead (has led) to collisions. Some proposals include embedding the tre_id in the name, but that's not adequate either, it assumes we'll all have unique tre_id's, whereas it's likely we're all calling them sde or tre.
2. using deterministic names which are calculated in several places in the code leads to brittle code. Changing the calculated name becomes very difficult as every place where it's calculated has to be found and changed.

A better approach would be to calculate the name once, store it somewhere (e.g. terraform state, cosmos DB, ...), then reference it from there. Where the value gets stored will depend on the downstream uses for it, but the principle should be to have a SSOT wherever possible.

This can make the code more complex in some ways, of course, since looking something up rather than calculating it is more work, but it's essential to fix this problem or we can't scale much beyond where we are today.

[marrobi]

Hi @TonyWildish-BH.

Thank you, can we use one of the existing issues mentioned, rather than creating another thread?

Worth noting, there does need to be a unique identified for each deployment, and that's the intention of tre_id - see https://microsoft.github.io/AzureTRE/latest/tre-admins/environment-variables/#for-azure-tre-instance-in-configyaml .

A globally unique identifier.

There is a workaround in place for the workspace storage account clash, and have not seen a clash for a long while. If you have instances where other resource names clash please do post the logs so we can triage.

However I do agree we need a longer term solution, should be passing names through rather than calculating in various places. Can we focus on this issue to discuss the technical resolution #2893 . Feel free to add your suggestions as to how to resolve (from a code perspective).

Closing as duplicate.