HDDS-11959. Remove tests for non-Ratis SCM

[chungen0126]

What changes were proposed in this pull request?

Remove tests that exercise non-Ratis SCM.

Please describe your PR in detail:

1. Remove tests for non-Ratis SCM
2. Remove unnecessary "Enable SCM Ratis" config.
3. Set OZONE_SCM_RATIS_PORT_KEY and OZONE_SCM_GRPC_PORT_KEY as a free port to avoid port conflict from MiniOzoneClusterProvider.
4. Wait for pipeline ready in some tests to fix tests failure.(TestSCMContainerManagerMetrics, TestContainerStateMachine, and TestReconTasks)
5. Disable test cases in SCM integration tests where test implementation is specific to non-Ratis mode (should be updated in HDDS-11989, HDDS-12202, and HDDS-12023)
6. Disable test cases in SCM unit tests where test implementation is specific to non-Ratis mode (should be updated in HDDS-11986)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11959

How was this patch tested?

https://github.com/chungen0126/ozone/actions/runs/12621318024

[adoroszlai]

Thanks @chungen0126 for the patch.

There are some leftover references in tests:

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMSnapshot.java

Line 47 in 5b27f6d

conf.setBoolean(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY, true);

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMInstallSnapshot.java

Line 69 in 5b27f6d

conf.setBoolean(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY, true);

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java

Line 235 in 5b27f6d

conf.setBoolean(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY, false);

(please check if TestSecureOzoneCluster works without this setting)

Also in MiniOzoneClusterImpl:

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java

Lines 769 to 780 in 5b27f6d

	//TODO: HDDS-6897
	//Disabling Ratis for only of MiniOzoneClusterImpl.
	//MiniOzoneClusterImpl doesn't work with Ratis enabled SCM
	if (StringUtils.isNotEmpty(
	conf.get(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY))
	&& SCMHAUtils.isSCMHAEnabled(conf)) {
	scmStore.setSCMHAFlag(true);
	scmStore.persistCurrentState();
	SCMRatisServerImpl.initialize(clusterId, scmId,
	SCMHANodeDetails.loadSCMHAConfig(conf, scmStore)
	.getLocalNodeDetails(), conf);
	}

[chungen0126]

Thanks @adoroszlai for reviewing.

There are some leftover references in tests:

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMSnapshot.java

Line 47 in 5b27f6d

conf.setBoolean(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY, true);

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMInstallSnapshot.java

Line 69 in 5b27f6d

conf.setBoolean(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY, true);

(please check if TestSecureOzoneCluster works without this setting)

Also in MiniOzoneClusterImpl:

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java

Lines 769 to 780 in 5b27f6d

	//TODO: HDDS-6897
	//Disabling Ratis for only of MiniOzoneClusterImpl.
	//MiniOzoneClusterImpl doesn't work with Ratis enabled SCM
	if (StringUtils.isNotEmpty(
	conf.get(ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY))
	&& SCMHAUtils.isSCMHAEnabled(conf)) {
	scmStore.setSCMHAFlag(true);
	scmStore.persistCurrentState();
	SCMRatisServerImpl.initialize(clusterId, scmId,
	SCMHANodeDetails.loadSCMHAConfig(conf, scmStore)
	.getLocalNodeDetails(), conf);
	}

I'm leaving the reference in MiniOzoneClusterImpl because that enabling SCM Ratis in all tests with MiniOzoneClusterImpl leads to many tests failures. TestSCMSnapshot.java and TestSCMInstallSnapshot.java should set OZONE_SCM_HA_ENABLE_KEY to true because MiniOzoneClusterImpl is non-Ratis SCM by default.
I believe we should fix this in HDDS-6897.

(please check if TestSecureOzoneCluster works without this setting)

TestSecureOzoneCluster doesn't work with OZONE_SCM_HA_ENABLE_KEY set to true but works without it. However, MiniOzoneClusterImpl is non-Ratis SCM by default.

I'll convert this to draft and fix HDDS-6897 first. I think that would make the change clearer.

[chungen0126]

There is only one OZONE_SCM_HA_ENABLE_KEY in all the tests which is in TestOzoneConfigurationFields. It should be removed after the config is removed.

[adoroszlai]

It should be removed after the TestOzoneConfigurationFields is removed.

I guess you mean when the config itself is removed.

[chungen0126]

It should be removed after the TestOzoneConfigurationFields is removed.

I guess you mean when the config itself is removed.

Yes, you're right, I wrote the wrong word. Thanks for the reminder.

[adoroszlai]

Thanks @chungen0126 for the patch, mostly LGTM.

[jojochuang]

Both TestReconScmHASnapshot and TestReconScmNonHASnapshot are covered by TestReconScmSnapshot.

[jojochuang]

Merged. Thanks @chungen0126 and @adoroszlai for review.