[Bug]: mode/pipeline grpc generated Java compilation fail

[turb]

What happened?

On master, Apache Beam Java compilation fail on some configurations (here Apple Mac Book Pro with M3 chip), since commit f03e0c0 introduced by #33858. cc @robertwb

For whatever reason, model/pipeline/build/classes/java/main/org/apache/beam/model/pipeline/v1/RunnerApi.class is present but not included in the classpath. I'm not familiar enough with gradle to find out why.

------------------------------------------------------------
Gradle 8.4
------------------------------------------------------------

Build time:   2023-10-04 20:52:13 UTC
Revision:     e9251e572c9bd1d01e503a0dfdf43aedaeecdc3f

Kotlin:       1.9.10
Groovy:       3.0.17
Ant:          Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM:          11.0.26 (Homebrew 11.0.26+0)
OS:           Mac OS X 15.2 aarch64

Error log:

> Task :model:pipeline:compileJava
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:18: error: cannot find symbol
  private static volatile io.grpc.MethodDescriptor<org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest,
                                                                                              ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:26: error: cannot find symbol
  public static io.grpc.MethodDescriptor<org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest,
                                                                                    ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:102: error: cannot find symbol
    default void events(org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest request,
                                                                   ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:169: error: cannot find symbol
        org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest request) {
                                                   ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:140: error: cannot find symbol
    public void events(org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest request,
                                                                  ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:23: error: cannot find symbol
      requestType = org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest.class,
                                                               ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:28: error: cannot find symbol
    io.grpc.MethodDescriptor<org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest, org.apache.beam.model.pipeline.v1.RunnerApi.TestStreamPayload.Event> getEventsMethod;
                                                                        ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:33: error: cannot find symbol
              io.grpc.MethodDescriptor.<org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest, org.apache.beam.model.pipeline.v1.RunnerApi.TestStreamPayload.Event>newBuilder()
                                                                                   ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:38: error: cannot find symbol
                  org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest.getDefaultInstance()))
                                                             ^
  symbol:   variable EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:212: error: cannot find symbol
          serviceImpl.events((org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest) request,
                                                                         ^
  symbol:   class EventsRequest
  location: class RunnerApi
beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:237: error: cannot find symbol
              org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest,
                                                         ^
  symbol:   class EventsRequest
  location: class RunnerApi
11 errors

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shunping]

Hi @turb, could you run ./gradlew clean first before building your pipeline again? There is some update on the proto, so you will need a clean build.

[turb]

Hello @shunping and thanks for the answer,

Indeed I didn't notice the ./gradlew clean I executed previously had failed. For whatever reason I had to delete build dir first because Python was unable to find grpcio-tools.

After doing ./gradlew clean successfully :

• I still have generated Java code in model/pipeline/build/generated/source/proto/main/java/org/apache/beam/model/pipeline/v1/RunnerApi.java
• however now model/pipeline/build/classes/java/main is empty

And the error is still the same:

beam/model/pipeline/build/generated/source/proto/main/grpc/org/apache/beam/model/pipeline/v1/TestStreamServiceGrpc.java:18: error: cannot find symbol
  private static volatile io.grpc.MethodDescriptor<org.apache.beam.model.pipeline.v1.RunnerApi.EventsRequest,
                                                                                              ^
  symbol:   class EventsRequest
  location: class RunnerApi

...and indeed there is no EventsRequest in model/pipeline/build/generated/source/proto/main/java/org/apache/beam/model/pipeline/v1/RunnerApi.java.

I have the same error on a freshly cloned beam repo.

[shunping]

What if you remove the entire folder of "beam/model/pipeline/build/"?

Another thing you can try is to remove gradle build cache under "~/.gradle/caches/".

I have the same error on a freshly cloned beam repo.

I am not able to reproduce that with a newly cloned repo. Could you let me know what your steps are to get the error?

[turb]

Another thing you can try is to remove gradle build cache under "~/.gradle/caches/".

That did the trick. Sorry for bothering you, and thanks!

[shunping]

Another thing you can try is to remove gradle build cache under "~/.gradle/caches/".

That did the trick. Sorry for bothering you, and thanks!

No problem at all. Glad it helps.