[Feature Request]: Upgrade transforms without upgrading the pipeline using the Transform Service

[chamikaramj]

What would you like to happen?

Starting Beam 2.49.0 Beam ships with a Docker Compose based transform service that can be used to expand transforms.

This service can be used to expand portable transforms available in Beam and such expanded transforms will come with a SDK harness containers to be used during runtime.

With Beam portability framework, it should be possible to use different Beam versions for the pipeline and for the transforms expanded using the transform service.

This should allow Beam users to upgrade Beam version of Beam transforms without upgrading the Beam version of the pipeline.

This feature can be used by Beam users that wish to upgrade individual transforms early without migrating full pipelines (for example to use new features or to apply critical fixes to transforms).

We have to introduce APIs and utilities to Beam make this upgrade process easy.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner