Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Getting an error related to SchemaCoder after upgrading to 2.48 #26981

Closed
1 of 15 tasks
Amraneze opened this issue Jun 2, 2023 · 5 comments
Closed
1 of 15 tasks

Comments

@Amraneze
Copy link
Contributor

Amraneze commented Jun 2, 2023

What happened?

After upgrading Apache Beam to the today's latest version 2.48.0, we encountered some issues with our tests. All tests were failing because of this following error:

Caused by: java.lang.IllegalArgumentException: org.apache.beam.model.Function$SchemaCodeGen$9JxP0J5v must be defined in the same package as org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function
image

It only happens when you have a nested schema. If we remove the nested schema it works fine.

I created this repository beam-schema-bug that you can run with DirectRunner to reproduce the issue.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@Amraneze
Copy link
Contributor Author

Any progress related to this issue. We can't run the pipeline with the version 2.48

@marcin-sucharski
Copy link

marcin-sucharski commented Jun 27, 2023

The same problem (however with different package prefix for Function) occurs with custom generated protobuf classes, e.g.:

org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalArgumentException: com.my.pacakge.with.proto.Function$SchemaCodeGen$bWrbfTA2 must be defined in the same package as org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function

Version 2.47 works, so the bug was introduced in 2.48

@dopieralad
Copy link
Contributor

I have the same issue. I've generated classes from an Avro schema using avro-compiler at build time and Apache Beam in version 2.48.0 can't handle them properly.

@damondouglas
Copy link
Contributor

This seems to be caused by #25578.

@marcin-sucharski
Copy link

Version 2.51 seems to be working as expected

@github-actions github-actions bot added this to the 2.53.0 Release milestone Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants