[bugfix] workqueue metrics are mising

[RainbowMango]

What happened:
We received reports that the controller's work queue metrics can not be emitted. (See #5945 for more details, thanks to @CharlesQQ for reporting this).

This issue occurred before and was fixed in release 1.0 in #945, and then we made some subsequent impairments in #2899 and #3012. (Thanks to @Garrybest for doing this)

However after #2998 and #4706, this issue was introduced again in release-1.10.

This issue is used to track any relevant fix to ensure the problem is fully resolved without introducing any potential risk.

Root Cause:
TBD

Iteration tasks:

• Revist #2998, fix potential risk or twist solutions.
• Clarify the usage of Service Reference in the ResourceInterpreterWebhookConfiguration API.
• Clarify the usage documentation of Service Reference in the ResourceInterpreterWebhookConfiguration. (#5969, @XiShanYongYe-Chang)
• Minimum fix on master. (Fix the issue of missing workqueue metrics in the Karmada controller #5972, @XiShanYongYe-Chang )
• Backport to release branches. (release-1.10~release-1.12) (Cherrypick #5972: fix the issue of missing workqueue metrics in the Karmada controller #5979)
• Cleanup workaround made on release-1.0. (remove workaround code for resolving workqueue metrics loss by modifying import order #5945, @chaosi-zju)
• Introduce tests to cover the metrics. (introduce tests to check whether workqueue metrics exist #6003, @chaosi-zju)
• Expose metrics of karmada-metrics-adapter. (expose metrics of karmada-metrics-adapter #6013, @chaosi-zju)

Other Potential risks

• karmada-webhook panic in case of invalid configuration. (#5960, @XiShanYongYe-Chang)
• Cannot access karmada-interpreter-webhook by configuring Service in ResourceInterpreterWebhookConfiguration #5958

References:

• From Go 1.21, the program initialization order was specified more precisely. See Go 1.21 release notes for more details.

Echo from the release notes:

Package initialization order is now specified more precisely. The new algorithm is:

• Sort all packages by import path.
• Repeat until the list of packages is empty:
  • Find the first package in the list for which all imports are already initialized.
  • Initialize that package and remove it from the list.

• Go Program initialization
• Issue in controller-runtime: Add a new metrics to indicates the the current queue length. kubernetes-sigs/controller-runtime#1969
• Issue in Kubernetes: There is a conflict between the metrics of controller-runtime and component-base, the metric workqueue_depth of controller-runtime repository not take effect kubernetes/kubernetes#128326

[chaosi-zju]

the PR #6003 works for the task Introduce tests to cover the metrics

[RainbowMango]

Got it.