[Failing Test]: Several example failing on Dataflow legacy runner

[Abacn]

What happened?

Though happening after #29353, these tests were accidently get exercised on Legacy runner after #28565, specifically, the line of mistake:

https://github.com/Abacn/beam/blob/08e0cbf626a57e001e892ab0ff2d82c3bc62b160/runners/google-cloud-dataflow-java/examples/build.gradle#L87

which is recently fixed in #29353. However, some examples begin to fail:

https://github.com/apache/beam/actions/runs/6820493465

all except one has fail reason: "java.lang.RuntimeException: Failed to create job with prefix beam_bq_job_LOAD_testpipelinerunner"

"errorResult" : {
      "message" : "Not found: Files /bigstore/temp-storage-for-end-to-end-tests/testpipeline-runner-1110040417-2f5b96cf/output/results/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinerunner11100404172f5b96cf_63f2d5d686a34ca59a84aa4cdabaf8c5/a74945b0-8393-4ca1-b1b2-143f96599d2a",
      "reason" : "notFound"
    },
    "errors" : [ {
      "message" : "Not found: Files /bigstore/temp-storage-for-end-to-end-tests/testpipeline-runner-1110040417-2f5b96cf/output/results/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinerunner11100404172f5b96cf_63f2d5d686a34ca59a84aa4cdabaf8c5/a74945b0-8393-4ca1-b1b2-143f96599d2a",
      "reason" : "notFound"
    }

basically the temp file is not found

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

#28565 merged on Sep 22, after the release cut of 2.51.0

[Abacn]

It's actual pretty redherring. It is caused by #19191. The reason is that the tempLocation is set to

gs://temp-storage-for-end-to-end-tests/

then the temp file has location said in log

Opening TableRowWriter to gs://temp-storage-for-end-to-end-tests//testpipeline-runner-1110034644-b64d623c/output/results/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinerunner1110034644b64d623c_a9fb169b414047fa99d8844f5eadbde1/1f11ac86-936a-483b-8244-88fc49ff3aac

notice that there are double "//"

Then the load job has error

"Not found: Files /bigstore/temp-storage-for-end-to-end-tests/testpipeline-runner-1110034644-b64d623c/output/results/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinerunner1110034644b64d623c_a9fb169b414047fa99d8844f5eadbde1/1f11ac86-936a-483b-8244-88fc49ff3aac"

the double "//" get parsed to "/bigstore/"

For Dataflow runner v2 job the gradle task is in different place, and the temp location is

beam/runners/google-cloud-dataflow-java/build.gradle

Line 145 in f47d473

def dataflowPostCommitTempRoot = project.findProperty('dataflowTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests'

does not have trailing slash.