feat: Add offline scaling support for background ddl with arrangement backfill

[shanicky]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

WIP STATE

It seems that No Shuffle backfill is still difficult to support in offline scaling, and this PR is still in a WIP state.

I found it somewhat challenging to directly support background jobs in online scaling, so I proposed this PR to add support for background jobs in offline scaling. At the same time, I attempted to skip jobs related to background DDL jobs in online scaling instead of completely omitting them.

This PR makes several specific changes.

1. Due to changes in the stream job fragment after offline scaling, it needs to be retrieved again.
2. Since the scaling occurs before the created event, the FE will receive the scaling mapping first.

Checklist

• I have written necessary rustdoc comments.

[gru-agent]

This pull request has been modified. If you want me to regenerate unit test for any of the files related, please find the file in "Files Changed" tab and add a comment @gru-agent. (The github "Comment on this file" feature is in the upper right corner of each file in "Files Changed" tab.)

[yezizp2012]

I think we can use fragment_type_mask instead to reduce memory usage.

[shanicky]

It seems that the fragment_type_mask cannot determine the type of stream scan; at least it couldn't in previous versions. We only have the stream node's node body to make that judgment. cc @kwannoel

[kwannoel]

We can shortcut for snapshot backfill

[kwannoel]

Snapshot backfill uses fragment_type_mask iirc

[shanicky]

I found that performing offline scaling directly during the recovery process causes the background DDL job status to remain at 100%. Do I need to do anything else? cc @kwannoel

 Id |                   Statement                   |     Progress
----+-----------------------------------------------+-------------------
  7 | CREATE MATERIALIZED VIEW m AS SELECT * FROM t | 100.00% (100/100)
(1 row)

[kwannoel]

I found that performing offline scaling directly during the recovery process causes the background DDL job status to remain at 100%. Do I need to do anything else? cc @kwannoel

 Id |                   Statement                   |     Progress
----+-----------------------------------------------+-------------------
  7 | CREATE MATERIALIZED VIEW m AS SELECT * FROM t | 100.00% (100/100)
(1 row)

Did the jobs get created successfully? If not, is the state inside the CreateMviewProgressTracker up to date? Did all the new actors get included in it?

Can also add some tracing logs to see which actor did not report that they're finished yet. You can share the test, I'll try to take a look tomorrow.

[shanicky]

I found that performing offline scaling directly during the recovery process causes the background DDL job status to remain at 100%. Do I need to do anything else? cc @kwannoel

 Id |                   Statement                   |     Progress
----+-----------------------------------------------+-------------------
  7 | CREATE MATERIALIZED VIEW m AS SELECT * FROM t | 100.00% (100/100)
(1 row)

Did the jobs get created successfully? If not, is the state inside the CreateMviewProgressTracker up to date? Did all the new actors get included in it?

Can also add some tracing logs to see which actor did not report that they're finished yet. You can share the test, I'll try to take a look tomorrow.

Ah, I've found the problem. If the meta undergoes recovery without a restart, the states in the progress tracker are not updated. These states should be reconstructed.

[yezizp2012]

lgtm