-
Notifications
You must be signed in to change notification settings - Fork 537
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Remove support for Batch Compression without Batch Grouping (#23017)
## Description We can and should get rid of the mode where we send ops with empty content for Compression. Configuration-wises, at this point both compression & op grouping are ON in main, and OFF in LTS. So, we can group them together into one modality and stop mentioning (and dealing with) a modality where compression is on, but op grouping is off. Acceptance Criteria The code that compresses batches of 2 or more messages by creating empty message placeholders should be removed. The code to reconstitute such batches must remain, of course. Execution Plan There are two exceptions we need to deal with first as part of this: * Stop compressing the blobAttach batch, since Grouping is disabled (See discussion in ADO for justification of this change) * Always group a batch of 2 or more messages - Remove the ability to configure this Then it's just a matter of updating OpCompressor to require single-message batches and remove the empty placeholder stuff. ## Reviewer Guidance Let me know if there's a better way to accomplish the task. Also review the tests to make sure that the flow the tests are trying to recreate is still being followed. Fixes: [AB#28649](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/28649)
- Loading branch information
1 parent
a4fc086
commit f55ab97
Showing
12 changed files
with
185 additions
and
151 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
61 changes: 61 additions & 0 deletions
61
packages/runtime/container-runtime/src/test/opLifecycle/legacyCompression.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| /*! | ||
| * Copyright (c) Microsoft Corporation and contributors. All rights reserved. | ||
| * Licensed under the MIT License. | ||
| */ | ||
|
|
||
| import { IsoBuffer } from "@fluid-internal/client-utils"; | ||
| import { compress } from "lz4js"; | ||
|
|
||
| import { CompressionAlgorithms } from "../../index.js"; | ||
| import type { BatchMessage, IBatch } from "../../opLifecycle/index.js"; | ||
|
|
||
| /** | ||
| * The code in this file recreates the legacy flow for batches that have multiple messages. | ||
| * It should not be removed since we still need to be able to read batches that were compressed with the old flow. | ||
| * */ | ||
|
|
||
| /** | ||
| * Combine the batch's content strings into a single JSON string (a serialized array) | ||
| */ | ||
| function serializeBatchContents(batch: IBatch): string { | ||
| // Yields a valid JSON array, since each message.contents is already serialized to JSON | ||
| return `[${batch.messages.map(({ contents }) => contents).join(",")}]`; | ||
| } | ||
|
|
||
| /** | ||
| * This is a helper function that replicates the now deprecated process for compressing a batch that creates empty placeholder messages. | ||
| * It was added since the new process cannot compress a batch with multiple messages, it now only compresses individual messages (which can be a regular message or a grouped one). | ||
| * But we need to ensure the current code still supports READING the old op format (where an old client compressed a multi-message batch) | ||
| * @internal | ||
| * @param batch - batch with messages that are going to be compressed | ||
| * @returns compresed batch with empty placeholder messages | ||
| */ | ||
| export function compressMultipleMessageBatch(batch: IBatch): IBatch { | ||
| const contentsAsBuffer = new TextEncoder().encode(serializeBatchContents(batch)); | ||
| const compressedContents = compress(contentsAsBuffer); | ||
| const compressedContent = IsoBuffer.from(compressedContents).toString("base64"); | ||
|
|
||
| const messages: BatchMessage[] = []; | ||
| messages.push({ | ||
| ...batch.messages[0], | ||
| contents: JSON.stringify({ packedContents: compressedContent }), | ||
| metadata: batch.messages[0].metadata, | ||
| compression: CompressionAlgorithms.lz4, | ||
| }); | ||
|
|
||
| // Add empty placeholder messages to reserve the sequence numbers | ||
| for (const message of batch.messages.slice(1)) { | ||
| messages.push({ | ||
| localOpMetadata: message.localOpMetadata, | ||
| metadata: message.metadata, | ||
| referenceSequenceNumber: message.referenceSequenceNumber, | ||
| }); | ||
| } | ||
|
|
||
| const compressedBatch: IBatch = { | ||
| contentSizeInBytes: compressedContent.length, | ||
| messages, | ||
| referenceSequenceNumber: batch.referenceSequenceNumber, | ||
| }; | ||
| return compressedBatch; | ||
| } |
Oops, something went wrong.