Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keyed Window Buffer #32

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Goggin
Copy link

@Goggin Goggin commented Jun 13, 2024

  • Implement a keyed window buffer
  • Tests for keyed window buffer
  • Docs for keyed window buffer

Keyed window buffer works similar to system window but batches based on a key mapping, and has an optional timeout for batches. When a new key is seen, a batch is started for that key, and as new messages come in they are added to their respective keys. If a new batch is to be created, but max pending keys is reached, then the message is nacked to allow for completing the current batches. After each message, if a check mapping is supplied, it is run against the batch to determine if the batch completion needs are matched. Once a batch is complete or the specified timeout is hit, then the batch is pushed through the pipeline. On normal shutdown, any pending batches are nacked for processing upon next startup.

- Implement a keyed window buffer
- Tests for keyed window buffer
- Docs for keyed window buffer

Keyed window buffer works similar to system window but batches based on a key
mapping, and has an optional timeout for batches.  When a new key is seen, a
batch is started for that key, and as new messages come in they are added to
their respective keys. If a new batch is to be created, but max pending keys
is reached, then the message is nacked to allow for completing the current
batches. After each message, if a check mapping is supplied, it is run against
the batch to determine if the batch completion needs are matched. Once a batch
is complete or the specified timeout is hit, then the batch is pushed through
the pipeline. On normal shutdown, any pending batches are nacked for
processing upon next startup.
@CLAassistant
Copy link

CLAassistant commented Jun 13, 2024

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants