[Failing Test]: Go Rrecommit failed for the Github action due to "directories_disallowed"

[chamikaramj]

What happened?

https://github.com/apache/beam/actions/runs/6780185302/job/18499776500?pr=29322

2023/11/08 20:50:29 ERROR process bundle failed for instruction inst001 using plan stage-001 : while executing Process for Plan[stage-001]:
1: DataSource[S[stage-001_source@localhost:37423], i0] Out:6 Coder:W;cwv_n1<bytes;c0>!GWC
6: ParDo[beam.createFn] Out:[5] Sig: func([]uint8, func(typex.T)) error, SideInputs: []
5: PCollection[n2] Out:[4]
4: ParDo[fileio.readFn] Out:[3] Sig: func(fileio.FileMetadata, func(fileio.ReadableFile)) error, SideInputs: []
3: PCollection[n3] Out:[2]
2: ParDo[passert.failFn] Out:[] Sig: func(typex.X) error, SideInputs: []
caused by:
DoFn[UID:4, PID:fileio.ReadMatches/fileio.readFn, Name: github.com/apache/beam/sdks/v2/go/pkg/beam/io/fileio.readFn] failed:
path to directory not allowed: "dir/" source=/runner/_work/beam/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:484 time=2023-11-08T20:50:29.226Z worker.ID=job-012[go-job-12-1699476629184984274]go worker.endpoint=localhost:37423
--- FAIL: TestReadMatches (0.16s)
--- FAIL: TestReadMatches/Error-directories_disallowed (0.05s)
--- FAIL: TestReadMatches/Error-directories_disallowed/Error-_directories_disallowed (0.05s)
read_test.go:145: ReadMatches() error = , wantErr true
FAIL
FAIL github.com/apache/beam/sdks/v2/go/pkg/beam/io/fileio 0.524s

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[chamikaramj]

cc: @lostluck @damccorm

[johannaojeling]

Hm.. I've been trying to troubleshoot it but the DoFn's ProcessElement method seems to return an error as expected. The DoFn behavior I could think of causing the flakiness is if the condition to check whether the path is a directory is not true for this CI environment.

beam/sdks/go/pkg/beam/io/fileio/read.go

Lines 135 to 139 in 6bf871f

	func isDirectory(path string) bool {
	if strings.HasSuffix(path, "/") || strings.HasSuffix(path, "\\") {
	return true
	}
	return false

However I checked another job for the same type of runner and environment, and all fileio tests were successful.

I can reproduce the failing test behavior by introducing some delay:

beam/sdks/go/pkg/beam/core/runtime/harness/harness.go

Lines 483 to 485 in 6bf871f

	if err != nil {
	return fail(ctx, instID, "process bundle failed for instruction %v using plan %v : %v", instID, bdID, err)
	}

		if err != nil {
+			time.Sleep(100 * time.Millisecond)
			return fail(ctx, instID, "process bundle failed for instruction %v using plan %v : %v", instID, bdID, err)
		}

In this case the pipeline does not fail as it should. There might be some cancelled context having to do with it but I lack enough knowledge of this part of the code base to understand what is going on.

[lostluck]

I think this got resolved in my deflake pass: 200aa46#diff-af6e698c535cbcccdd7da6594a1b1833b11bf30d4405c9afeb4236abb28681ea

Basically, we nested t.Run unnecessarily. I don't know if that's the root cause, but removing the extra layer made it not fail in my deflake runs.