[Bug]: Nightly snapshot release is failing due to Gradle 8 upgrade

[ahmedabu98]

What happened?

Around the same time that Gradle 8 got upgraded (#28756), the nightly snapshot has been failing on both Jenkins and GHA. For both, I'm seeing errors with many Gradle task configurations that go like:

Some problems were found with the configuration of task ':sdks:java:core:testSourcesJar' (type 'Jar').
- Gradle detected a problem with the following location: '/home/jenkins/jenkins-agent/26ff11f4/workspace/beam_Release_NightlySnapshot/src/sdks/java/core/build/generated-test-avro-java'.
 
Reason: Task ':sdks:java:core:testSourcesJar' uses this output of task ':sdks:java:core:generateTestAvroJava' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.

Possible solutions:
  1. Declare task ':sdks:java:core:generateTestAvroJava' as an input of ':sdks:java:core:testSourcesJar'.
  2. Declare an explicit dependency on ':sdks:java:core:generateTestAvroJava' from ':sdks:java:core:testSourcesJar' using Task#dependsOn.
  3. Declare an explicit dependency on ':sdks:java:core:generateTestAvroJava' from ':sdks:java:core:testSourcesJar' using Task#mustRunAfter.

For more information, please refer to https://docs.gradle.org/8.3/userguide/validation_problems.html#implicit_dependency in the Gradle documentation.

Example: https://ci-beam.apache.org/job/beam_Release_NightlySnapshot/2041/console

Looks like Gradle 8 is more strict on declaring dependencies.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[ahmedabu98]

Some relevant links:
Default source configuration:

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Lines 1787 to 1797 in d170634

	project.task('sourcesJar', type: Jar) {
	from project.sourceSets.main.allSource
	archiveClassifier = 'sources'
	}
	project.artifacts.archives project.sourcesJar
	project.task('testSourcesJar', type: Jar) {
	from project.sourceSets.test.allSource
	archiveClassifier = 'test-sources'
	}
	project.artifacts.archives project.testSourcesJar

Flink module has copy and merge source overrides:

beam/runners/flink/flink_runner.gradle

Lines 46 to 78 in d170634

	/*
	* Copy & merge source overrides into build directory.
	*/
	def sourceOverridesBase = "${project.buildDir}/source-overrides/src"
	def copySourceOverrides = tasks.register('copySourceOverrides', Copy) {
	it.from main_source_overrides
	it.into "${sourceOverridesBase}/main/java"
	it.duplicatesStrategy DuplicatesStrategy.INCLUDE
	}
	compileJava.dependsOn copySourceOverrides
	def copyResourcesOverrides = tasks.register('copyResourcesOverrides', Copy) {
	it.from main_resources_overrides
	it.into "${sourceOverridesBase}/main/resources"
	it.duplicatesStrategy DuplicatesStrategy.INCLUDE
	}
	processResources.dependsOn copyResourcesOverrides
	def copyTestSourceOverrides = tasks.register('copyTestSourceOverrides', Copy) {
	it.from test_source_overrides
	it.into "${sourceOverridesBase}/test/java"
	it.duplicatesStrategy DuplicatesStrategy.INCLUDE
	}
	compileTestJava.dependsOn copyTestSourceOverrides
	def copyTestResourcesOverrides = tasks.register('copyTestResourcesOverrides', Copy) {
	it.from test_resources_overrides
	it.into "${sourceOverridesBase}/test/resources"
	it.duplicatesStrategy DuplicatesStrategy.INCLUDE
	}
	processTestResources.dependsOn copyTestResourcesOverrides

[Abacn]

Thanks for triaging. This additional check in Gradle 8 has broken several test (e.g. #28845) - they are existing problem in our build but begins to be enforced in Gradle 8.

From the migration guide, I ran gradle help --scan to detect possible breakages for migrating to gradle 8; it detected many, but not this specific issue, causing many cases get missed in #28756

[Abacn]

Also note - flink runner errors in https://ci-beam.apache.org/job/beam_Release_NightlySnapshot/2041/console are likely the cause of the timing issue of #28419