Feature: Allow stream custom maxsize per batch

[simon28082]

Problem

I'm trying to read the data using badger and use grpc for node sync, but I can't change the max cut size of stream because it's restricted hardcode

// maxStreamSize is the maximum allowed size of a stream batch. This is a soft limit
// as a single list that is still over the limit will have to be sent as is since it
// cannot be split further. This limit prevents the framework from creating batches
// so big that sending them causes issues (e.g running into the max size gRPC limit).
var maxStreamSize = uint64(100 << 20) // 100MB

Solution

Add a property to Stream to allow custom slice size

type Stream struct {
	// Prefix to only iterate over certain range of keys. If set to nil (default), Stream would
	// iterate over the entire DB.
	Prefix []byte

	// Number of goroutines to use for iterating over key ranges. Defaults to 8.
	NumGo int

	// Badger would produce log entries in Infof to indicate the progress of Stream. LogPrefix can
	// be used to help differentiate them from other activities. Default is "Badger.Stream".
	LogPrefix string

	// ChooseKey is invoked each time a new key is encountered. Note that this is not called
	// on every version of the value, only the first encountered version (i.e. the highest version
	// of the value a key has). ChooseKey can be left nil to select all keys.
	//
	// Note: Calls to ChooseKey are concurrent.
	ChooseKey func(item *Item) bool

	// MaxSize is the maximum allowed size of a stream batch
	MaxSize uint64
}

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[netlify]

✅ Deploy Preview for badger-docs canceled.

Name	Link
🔨 Latest commit	28cbe82
🔍 Latest deploy log	https://app.netlify.com/sites/badger-docs/deploys/6719e479d4c87600085b04ba

[mangalaman93]

@simon28082 the linter is not happy. Could you check that please? and also pull in latest changes from the main branch?

[simon28082]

@mangalaman93 will do it

[simon28082]

@mangalaman93 I done it, please review

[mangalaman93]

could you try upgrading the action version in .github/workflows/ci-golang-lint.yml:29 to

uses: golangci/[email protected]