Headers fetching via NeoFS BlockFetcher service #3789

AliceInHunterland · 2025-01-20T08:47:31Z

AnnaShaleva

HeadersFetcher is started in the wrong place. Ensure that nothing prevents StateSync module to request blocks/headers during the node start.

pkg/network/bqueue/queue.go

pkg/network/server.go

AnnaShaleva · 2025-01-20T16:02:12Z

pkg/network/server.go

 	var err error
-	s.blockFetcher, err = blockfetcher.New(chain, s.NeoFSBlockFetcherCfg, log, s.bFetcherQueue.PutBlock,
-		sync.OnceFunc(func() { close(s.blockFetcherFin) }))
+	s.headerFetcher, err = blockfetcher.New(chain, s.NeoFSBlockFetcherCfg, log, s.bFetcherQueue.PutBlock, s.bFetcherQueue.PutHeader,


Provide s.stateSync instead of chain as the first argument to blockfetcher.New.

The height is different for s.stateSync. It starts from 3 mln, so the block/header can persist, and the queue becomes locked. That's why I rolled it back to the chain. Why can't it be the chain? Blockfetcher uses it only for config and BlockHeight(). will recheck why its s.stateSync starts from the middle of the sequence.

We need BlockHeight and HeaderHeight to be aligned with statesync module vision, that's the main idea.

pkg/core/statesync/module.go

AnnaShaleva

Does the service start work as expected?

AnnaShaleva · 2025-01-21T08:53:08Z

pkg/core/blockchain.go

@@ -285,9 +285,6 @@ func NewBlockchain(s storage.Store, cfg config.Blockchain, log *zap.Logger) (*Bl
 			zap.Uint32("MaxValidUntilBlockIncrement", cfg.MaxValidUntilBlockIncrement))
 	}
 	if cfg.P2PStateExchangeExtensions {
-		if !cfg.StateRootInHeader {
-			return nil, errors.New("P2PStatesExchangeExtensions are enabled, but StateRootInHeader is off")
-		}


We need more sophisticated check. If BlockFetcher is on, then it's OK, but if Blockfetcher is off and StateRootInHeader is also off, then it's an error.

Fix error message.

Now, when we have NeoFSStateSyncExtensions this conversation is not relevant. Untie NeoFSStateSyncExtensions from P2PStateExchangeExtensions, these are two different settings. Implement esparate verification logic for NeoFSStateSyncExtensions.

AnnaShaleva · 2025-01-21T08:55:09Z

pkg/network/server.go

 		if s.chain.HeaderHeight() < p.LastBlockIndex() {
 			return s.requestHeaders(p)
 		}


This if check must be adjusted. Right now it's related to peers only, but if blockfetcher is used, then we don't care about peer height because another source of data is used (NeoFS storage).

AnnaShaleva · 2025-01-21T09:10:21Z

pkg/network/server.go

 	if s.stateSync.NeedHeaders() {
 		if s.chain.HeaderHeight() < p.LastBlockIndex() {
 			return s.requestHeaders(p)
 		}
 		return nil
 	}
+	s.headerFetcher.Shutdown()
+	if s.ServerConfig.NeoFSBlockFetcherCfg.Enabled {
+		err := s.blockFetcher.Start()


Blockfetcher should fetch exactly the range of blocks that is required by statesymc module if statesync module is enabled. If statesync module is disabled, then blockfetcher should fetch all blocks starting from 0. Check that this logic is present in the blockfetcher.

AnnaShaleva · 2025-01-21T09:15:42Z

pkg/network/server.go

+	if s.ServerConfig.NeoFSBlockFetcherCfg.Enabled {
+		err := s.headerFetcher.Start()
+		if err != nil {
+			s.log.Error("skipping NeoFS BlockFetcher", zap.Error(err))
+		}
+	}
+	if s.headerFetcher.IsActive() {
+		return nil
+	}


This condition may lead to deadlock, consider the situation: NeoFS storage is missing the latest headers. Once started, headerfetcher will fetch all available headers and shutdown. However, if some latest headers are still required by statesync, then we need to switch to P2P. Right now this code will restart headersfetcher anyway. Setting s.headerFetcher to nil after the first headerfetcher shutdown (and checking for nil in dependent places) may do the trick.

with nil we cant use isActive.

Without nil we can't distinguish finished service from not started. So additional nil checks is a trade-off.

I have added isShutdown to the blockfetcher. if you don't like it, will look more precisely to the nil.

AnnaShaleva · 2025-01-21T09:18:27Z

pkg/services/blockfetcher/blockfetcher.go

-			err = bfs.enqueueBlock(b)
+			if !bfs.cfg.BlocksOnly {
+				err = bfs.enqueueHeader(&b.Header)


We need the code that will read header from stream without transactions reading.

AnnaShaleva · 2025-01-21T09:25:37Z

pkg/network/server.go

-	s.bFetcherQueue = bqueue.New(chain, log, nil, s.NeoFSBlockFetcherCfg.BQueueSize, updateBlockQueueLenMetric, bqueue.Blocking)
+	s.bFetcherQueue = bqueue.New(s.stateSync, log, nil, s.NeoFSBlockFetcherCfg.BQueueSize, updateBlockQueueLenMetric, bqueue.Blocking)


We may need a switch here. The main idea is that blockfetcher should work both with statesync module enabled/disabled. If statesync module is disabled then we need a usual blockfetcher operation flow that will fetch blocks, put them directly into chain-backed queue and shutdown. If statesync module is enabled then blockfetcher should use statesync-backed queue and headersfetcher should also be started.

codecov · 2025-01-21T15:13:40Z

Codecov Report

Attention: Patch coverage is 41.21864% with 164 lines in your changes missing coverage. Please review.

Project coverage is 82.82%. Comparing base (72e0cff) to head (d94eb81).

Files with missing lines	Patch %	Lines
pkg/services/blockfetcher/blockfetcher.go	21.25%	61 Missing and 2 partials ⚠️
pkg/network/server.go	27.53%	41 Missing and 9 partials ⚠️
pkg/core/statesync/module.go	67.44%	13 Missing and 1 partial ⚠️
pkg/core/blockchain.go	0.00%	10 Missing and 2 partials ⚠️
pkg/network/bqueue_adapters.go	44.44%	10 Missing ⚠️
pkg/network/bqueue/queue.go	73.52%	8 Missing and 1 partial ⚠️
pkg/core/block/header.go	73.68%	4 Missing and 1 partial ⚠️
pkg/consensus/consensus.go	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3789      +/-   ##
==========================================
- Coverage   83.02%   82.82%   -0.20%     
==========================================
  Files         336      337       +1     
  Lines       47090    47286     +196     
==========================================
+ Hits        39095    39164      +69     
- Misses       6395     6510     +115     
- Partials     1600     1612      +12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

AliceInHunterland · 2025-01-21T20:54:53Z

sometimes:

2025-01-21T23:53:55.597+0300    INFO    persisted to disk       {"blocks": 0, "keys": 1543, "headerHeight": 625633, "blockHeight": 0, "took": "15.490041ms"}
^C2025-01-21T23:53:56.155+0300  INFO    shutting down server    {"peers": 18}
2025-01-21T23:53:56.155+0300    INFO    shutting down NeoFS BlockFetcher service        {"force": true}
panic: send on closed channel

goroutine 533 [running]:
github.com/nspcc-dev/neo-go/pkg/network/bqueue.(*Queue).PutHeader(0x140000a5d40, 0x140050b9d40)
        github.com/nspcc-dev/neo-go/pkg/network/bqueue/queue.go:301 +0x228
github.com/nspcc-dev/neo-go/pkg/services/blockfetcher.(*Service).blockDownloader(0x140018cdce0)
        github.com/nspcc-dev/neo-go/pkg/services/blockfetcher/blockfetcher.go:314 +0x408
created by github.com/nspcc-dev/neo-go/pkg/services/blockfetcher.(*Service).Start in goroutine 171
        github.com/nspcc-dev/neo-go/pkg/services/blockfetcher/blockfetcher.go:229 +0x62c

pkg/network/server.go

AnnaShaleva

Better, but still needs polishing.

AnnaShaleva · 2025-01-23T09:57:23Z

pkg/core/blockchain.go

@@ -285,9 +285,6 @@ func NewBlockchain(s storage.Store, cfg config.Blockchain, log *zap.Logger) (*Bl
 			zap.Uint32("MaxValidUntilBlockIncrement", cfg.MaxValidUntilBlockIncrement))
 	}
 	if cfg.P2PStateExchangeExtensions {
-		if !cfg.StateRootInHeader {
-			return nil, errors.New("P2PStatesExchangeExtensions are enabled, but StateRootInHeader is off")
-		}


Fix error message.

AnnaShaleva · 2025-01-23T10:01:37Z

pkg/core/statesync/module.go

@@ -131,6 +136,9 @@ func (s *Module) Init(currChainHeight uint32) error {
 	if p < 2*s.syncInterval {
 		// chain is too low to start state exchange process, use the standard sync mechanism
 		s.syncStage = inactive
+		if s.syncStage != oldStage {
+			s.notifyStageChanged()
+		}


Move it to the defered statement.

AnnaShaleva · 2025-01-23T10:02:55Z

pkg/core/statesync/module.go

+
+	// stageCallback is an optional callback that is triggered whenever
+	// the sync stage changes.
+	stageCallback func()


stageChangedCallback?

AnnaShaleva · 2025-01-23T10:04:10Z

pkg/core/statesync/module.go

+// notifyStageChanged triggers stage callback if it's set.
+func (s *Module) notifyStageChanged() {
+	if s.stageCallback != nil {
+		go s.stageCallback()


NACK, not a proper place for goroutine, make it synchronious on the module side.

is it fixed in the current implementation? moved it to go notifyStageChanged()

AnnaShaleva · 2025-01-23T10:23:16Z

pkg/core/statesync/module.go

+		if s.syncStage != oldStage {
+			s.notifyStageChanged()
+			oldStage = s.syncStage
+		}


defineSyncStage is an atomic operation, you can't notify in the middle of it.

AnnaShaleva · 2025-01-23T12:25:23Z

pkg/services/blockfetcher/blockfetcher.go

@@ -140,6 +143,18 @@ func New(chain Ledger, cfg config.NeoFSBlockFetcher, logger *zap.Logger, putBloc
 	if err != nil {
 		return nil, err
 	}
+
+	getCfg := chain.GetConfig()
+	m := smartcontract.GetDefaultHonestNodeCount(int(getCfg.ValidatorsCount))


You need to make it adaptive. Validators count depends on the node's setting, we have pre-defined list of heights where validators count may be updated. See CommitteeUpdateHistory.

not yet implemented.

As we do Get on oid we do not know the index, so we can't check the relevant for this index Validation count.

we can do something like this: pass to the oidsCh oid and Index, but its not really cool

pass to the oidsCh oid and Index

Do it. Implement GETRANGE-based solution for index-files-based headers search and for single-object header search, keep the old solution for SEARCH-based headers search (just set the index to -1 to reuse the same oidsCh). Create an issue for the latter case.

AnnaShaleva · 2025-01-23T12:26:48Z

pkg/services/blockfetcher/blockfetcher.go

+				if isContextCanceledErr(err) {
+					return
+				}
+				bfs.log.Error("failed to decode block from stream", zap.String("oid", blkOid.String()), zap.Error(err))


Adjust error text, it's not block anymore.

pkg/services/blockfetcher/blockfetcher.go

pkg/core/block/block.go

AliceInHunterland · 2025-01-28T08:34:46Z

pkg/core/statesync/module.go

@@ -509,7 +568,7 @@ func (s *Module) GetUnknownMPTNodesBatch(limit int) []util.Uint256 {
 	return s.mptpool.GetBatch(limit)
 }

-// HeaderHeight returns the height of the latest header.


AnnaShaleva · 2025-01-28T07:51:39Z

pkg/network/bqueue/queue.go

-// New creates an instance of BlockQueue.
-func New(bc Blockqueuer, log *zap.Logger, relayer func(*block.Block), cacheSize int, lenMetricsUpdater func(l int), mode OperationMode) *Queue {
+// New creates an instance of Queue that handles both blocks and headers.
+func New(bc Blockqueuer, log *zap.Logger, relayer any, cacheSize int, lenMetricsUpdater func(l int), mode OperationMode, objectMode TypeMode) *Queue {


relayer is not any, it's a function with pre-defined signature.

AnnaShaleva · 2025-01-28T08:02:41Z

pkg/network/bqueue/queue.go

+func (bl *Queue) add(item Queueble) error {
+	if b, ok := item.(*block.Block); ok {
+		return bl.chain.AddBlock(b)
+	}
+	if h, ok := item.(*block.Header); ok {
+		return bl.chain.AddHeaders(h)
+	}
+	return nil
+}


NACK. I literally mean make the queue generic:

var bQueue := New[Block](blockAdapter{chain}, ...) var bSyncQueue := New[Block](blockAdapter{stateSyncModule}, ...) var hSyncQueue := New[Header](headerAdapter{stateSyncModule}, ...)

With func New[Q Queueable](bc Blockqueuer[Q], ...). Then implement Blockqueuer over blockAdapter and headerAdapter. And then inside the queue use bq.chain.AddItem(item) for a single block/header or bq.chain.AddItems(item) for a multiple headers, that's it.

You will need to modify Blockqueuer:

type Blockqueuer[Q Queueable] interface { AddItem(Q) error AddItems(...Q) error Height() uint32 }

AnnaShaleva · 2025-01-28T08:14:15Z

pkg/core/block/block.go

+// GetIndex returns the index of the block.
+func (b *Block) GetIndex() uint32 {
+	return b.Header.Index
+}
+


You don't need that, Header already has this definition.

pkg/core/block/header.go

pkg/consensus/consensus.go

AnnaShaleva · 2025-01-28T08:37:08Z

pkg/core/statesync/module.go

+	s.stageChangedCallback = cb
+}
+
+// notifyStageChanged triggers stage callback if it's set.


It's not "stage callback". It notifies about state changes.

AnnaShaleva · 2025-01-28T08:39:26Z

pkg/core/statesync/module.go

@@ -351,6 +399,9 @@ func (s *Module) AddBlock(block *block.Block) error {
 		s.syncStage |= blocksSynced
 		s.log.Info("blocks are in sync",
 			zap.Uint32("blockHeight", s.blockHeight))
+		if s.syncStage != oldStage {
+			s.notifyStageChanged()
+		}


What I mean is that you need to move notification to the checkSyncIsCompleted and pass the old stage as an argument.

AnnaShaleva · 2025-01-28T08:43:17Z

pkg/core/statesync/module.go

@@ -351,6 +399,9 @@ func (s *Module) AddBlock(block *block.Block) error {
 		s.syncStage |= blocksSynced
 		s.log.Info("blocks are in sync",
 			zap.Uint32("blockHeight", s.blockHeight))
+		if s.syncStage != oldStage {
+			s.notifyStageChanged()
+		}


But it actually won't work properly because we need to notify only after mutex is removed. Hence, just move this to the defered statement to the start of the function.

AnnaShaleva · 2025-01-28T08:45:58Z

pkg/network/server.go

@@ -1136,7 +1209,7 @@ func (s *Server) handleGetHeadersCmd(p Peer, gh *payload.GetBlockByIndex) error

 // handleHeadersCmd processes headers payload.
 func (s *Server) handleHeadersCmd(p Peer, h *payload.Headers) error {
-	if s.blockFetcher.IsActive() {
+	if s.syncHeaderFetcher.IsActive() {


AnnaShaleva · 2025-01-28T08:49:16Z

pkg/core/transaction/witness.go

@@ -16,6 +16,9 @@ const (
 	// MaxVerificationScript is the maximum allowed length of verification
 	// script. It should be appropriate for 11/21 multisignature committee.
 	MaxVerificationScript = 1024
+
+	// DefaultInvocationScriptSize is the default length of invocation script with one signature.
+	DefaultInvocationScriptSize = 66


Line width. And it's relevant not only for Transaction, think about better place for it. Comment must be more specific, it's not about "default", tell something about Crypto.System.CheckSig. VM has definition of this constant, may be we can export.

AnnaShaleva · 2025-01-28T08:58:10Z

pkg/services/blockfetcher/blockfetcher.go

 }

 // New creates a new BlockFetcher Service.
-func New(chain Ledger, cfg config.NeoFSBlockFetcher, logger *zap.Logger, putBlock func(*block.Block) error, shutdownCallback func()) (*Service, error) {
+func New(chain Ledger, cfg config.NeoFSBlockFetcher, logger *zap.Logger, put func(any) error, shutdownCallback func(), typeMode bqueue.TypeMode) (*Service, error) {


You need to define this mode for blockfetcher, not for queue.

And why typeMode when it's an operationMode?

AnnaShaleva · 2025-01-28T09:21:20Z

pkg/services/blockfetcher/blockfetcher.go

 		account:           account,
 		stateRootInHeader: chain.GetConfig().StateRootInHeader,
+		headerSize:        getHeaderSize(chain.GetConfig()),


It must be a mapping from height to header size, it's not a single value.

AnnaShaleva · 2025-01-29T08:24:16Z

pkg/network/server.go

@@ -59,7 +59,7 @@ type (
 	Ledger interface {
 		extpool.Ledger
 		mempool.Feer
-		bqueue.Blockqueuer
+		bqueue.Blockqueuer[bqueue.Queueable]


You need new interface for that. Starting from this commit bqueue.Blockqueuer serves only queue needs.

AnnaShaleva · 2025-01-29T08:26:37Z

pkg/network/server.go


-	s.bSyncQueue = bqueue.New(s.stateSync, log, nil, bqueue.DefaultCacheSize, updateBlockQueueLenMetric, bqueue.NonBlocking)
+	s.bSyncQueue = bqueue.New[*block.Block](blockAdapter{s.stateSync}, log, nil, bqueue.DefaultCacheSize, updateBlockQueueLenMetric, bqueue.NonBlocking)


You will have two adapters for statesync module. So it's more like stateSyncBlockQueueAdapter and stateSyncHeaderQueueAdapter for statesync module and chainBlockQueueAdapter.

AnnaShaleva · 2025-01-29T08:27:33Z

pkg/network/server.go

@@ -282,6 +279,48 @@ func newServerFromConstructors(config ServerConfig, chain Ledger, stSync StateSy
 	return s, nil
 }

+type blockAdapter struct {
+	bc StateSync


It's not bc here, it's stateSync.

AnnaShaleva · 2025-01-29T08:30:09Z

pkg/network/server.go

@@ -282,6 +279,48 @@ func newServerFromConstructors(config ServerConfig, chain Ledger, stSync StateSy
 	return s, nil
 }

+type blockAdapter struct {


Move all adapters to a separate file. Declare a variable to ensure it implements proper interface, like `var _ = (queue.Blockqueuer[*block.Block])(&blockAdapter{})

AnnaShaleva · 2025-01-29T08:31:16Z

pkg/network/server.go

+}
+
+func (a blockAdapter) AddItem(b *block.Block) error {
+	return a.bc.AddItem(b)


Statesync module itself must not implement any Item-related things. That's exactly the purpose of adapters. Use AddBlock method here.

AnnaShaleva · 2025-01-29T08:31:33Z

pkg/network/server.go

+func (a blockAdapter) AddItems(bs ...*block.Block) error {
+	for _, b := range bs {
+		if err := a.bc.AddItem(b); err != nil {
+			return err
+		}
+	}
+	return nil


This must not be supported for blocks, it's only designated for headers. Panic.

AnnaShaleva · 2025-01-29T08:32:18Z

pkg/network/server.go

+}
+
+func (a chainAdapter) AddItem(b *block.Block) error {
+	return a.chain.AddItem(b)


Ditto for the blockchain, no AddItem is defined on the blockchain.

AnnaShaleva · 2025-01-29T08:33:05Z

pkg/network/server.go

+}
+
+func (a chainAdapter) Height() uint32 {
+	return a.chain.Height()


Ditto for height. It must be BlockHeight() for blocks adapter and HeaderHeight() for headers adapter.

AnnaShaleva · 2025-01-29T08:34:18Z

pkg/network/server.go

+	for _, header := range h.Hdrs {
+		err := s.stateSync.AddItems(header)


You must use AddItems(h.Hrs...), otherwise it's useless. This operation must be batched.

AnnaShaleva · 2025-01-29T08:35:05Z

pkg/network/state_sync.go

@@ -9,12 +10,15 @@ import (
 // StateSync represents state sync module.
 type StateSync interface {
 	AddMPTNodes([][]byte) error
-	bqueue.Blockqueuer
+	bqueue.Blockqueuer[bqueue.Queueable]


Ditto for statesync, bqueue.Blockqueuer[bqueue.Queueable] is now queue-related. Create one more interface with specific methods and reuse it for statesync and for Ledger.

AliceInHunterland · 2025-01-29T17:07:58Z

pkg/services/blockfetcher/blockfetcher.go

+
+	for height := range chain.CommitteeHistory {
+		m = smartcontract.GetDefaultHonestNodeCount(chain.GetNumOfCNs(height))
+		verification, _ = smartcontract.CreateDefaultMultiSigRedeemScript(vs[:chain.GetNumOfCNs(height)])


should it be new vs here? in CommitteeHistory we do not have public keys. or it should be ValidatorsHistory?

You don't really care about particular keys, size estimators doesn't need it.

AliceInHunterland · 2025-01-29T17:10:53Z

pkg/services/blockfetcher/blockfetcher.go

 				}
+				index++ //Won't work properly if neofs.ObjectSearch results are not ordered.


full get for this mode

or headers as an object(s) in the container

or adapt batchSize to headerSizeMap

AnnaShaleva

Process queue feedback first, then update PR and I'll review the rest, there's a lot of things that need to be fixed.

AnnaShaleva · 2025-01-30T07:17:00Z

internal/fakechain/fakechain.go

@@ -49,6 +49,10 @@ type FakeStateSync struct {
 	AddMPTNodesFunc   func(nodes [][]byte) error
 }

+func (s *FakeStateSync) HeaderHeight() uint32 {
+	return s.HeaderHeight()


What exactly is s.HeaderHeight() in this context? You're implementing interface, you can't use methods of this interface in the implementation, it just won't work.

internal/fakechain/fakechain.go

AnnaShaleva · 2025-01-30T07:18:15Z

pkg/consensus/consensus.go

@@ -58,7 +59,7 @@ type Ledger interface {

 // BlockQueuer is an interface to the block queue manager sufficient for Service.
 type BlockQueuer interface {
-	PutBlock(block *coreb.Block) error
+	Put(block bqueue.Queueable) error


It's not block anymore. It's something queueable.

*coreb.Block can be saved?

AnnaShaleva · 2025-01-30T07:19:56Z

pkg/consensus/consensus_test.go

+func (bq testBlockQueuer) Put(b bqueue.Queueable) error {
+	blk, ok := b.(*coreb.Block)
+	if !ok {
+		return nil
+	}
+	return bq.bc.AddBlock(blk)


Use unsafe cast here, because nothing else may be used here except the *coreb.Block.

AnnaShaleva · 2025-01-30T07:20:10Z

pkg/core/block/block.go

+// GetIndex returns the index of the block.
+func (b *Block) GetIndex() uint32 {
+	return b.Header.Index
+}
+


AnnaShaleva · 2025-01-30T09:09:36Z

pkg/network/server.go

-	s.bQueue = bqueue.New(chain, log, func(b *block.Block) {
-		s.tryStartServices()
-	}, bqueue.DefaultCacheSize, updateBlockQueueLenMetric, bqueue.NonBlocking)
+	s.bQueue = bqueue.New[*block.Block](chainBlockQueueAdapter{chain}, log, func(b bqueue.Queueable) { s.tryStartServices() }, bqueue.DefaultCacheSize, updateBlockQueueLenMetric, bqueue.NonBlocking)


func(b bqueue.Queueable)

It's not. You need to practice with generics and learn the specification. It's func(b *block.Block)

pkg/network/server.go

pkg/network/state_sync.go

roman-khimov · 2025-02-03T13:28:38Z

pkg/core/block/block.go

@@ -51,6 +51,11 @@ type auxBlockIn struct {
 	Transactions []json.RawMessage `json:"tx"`
 }

+// GetIndex returns the index of the block.


Some additional clarification on why it's needed would be helpful.

roman-khimov · 2025-02-03T13:54:58Z

pkg/network/bqueue/queue.go

+}
+
+// Queue is the queue of queueable elements.
+type Queue[Q queueable] struct {


You can't use non-exported type for an exported thing. Why queueable at all? Queueable should be sufficient.

because then i can't use put func(item bqueue.Queueable) as Cannot use type comparable outside a type constraint: interface is (or embeds) comparable. Made it any for blockfetcher.

roman-khimov · 2025-02-03T13:59:46Z

pkg/network/bqueue/queue.go

 						zap.String("error", err.Error()),
-						zap.Uint32("blockHeight", bq.chain.BlockHeight()),
-						zap.Uint32("nextIndex", b.Index))
+						zap.Uint32("Height", bq.chain.Height()),


s/Height/height/, I doubt we start with capitals anywhere.

roman-khimov · 2025-02-03T14:45:02Z

pkg/network/server.go

@@ -87,6 +87,13 @@ type (
 		Shutdown()
 	}

+	// ledger is a minimal subset of Ledger interface.


I think ledger is too vague for it. blockHeaderQueuer maybe? At least we could recognize queuer there to match with its real use.

roman-khimov · 2025-02-03T15:07:55Z

pkg/core/block/header.go

@@ -85,6 +85,16 @@ func (b *Header) GetIndex() uint32 {
 	return b.Index
 }

+// GetExpectedHeaderSize returns the expected header size with empty witness.
+func (b *Header) GetExpectedHeaderSize() int {


I'm not sure it's very useful, you rather need some GetExpectedHeaderSize(stateRootInHeader bool, numOfValidators int) that is a function, not a method. stateRootInHeader will be replaced by version after #3566.

pkg/core/blockchain.go

roman-khimov · 2025-02-03T15:11:40Z

pkg/core/blockchain.go

+			return nil, errors.New("NeoFSStateSyncExtensions are enabled, but NeoFSBlockFetcher is off")
+		}
+		if cfg.KeepOnlyLatestState && !cfg.RemoveUntraceableBlocks {
+			return nil, errors.New("NeoFSStateSyncExtensions can be enabled either on MPT-complete node (KeepOnlyLatestState=false) or on light GC-enabled node (RemoveUntraceableBlocks=true)")


It is similar to P2PStateExchangeExtensions, shouldn't it be like that?

P2P state exchange requires nodes to have some old MPTs to answer GetMPTData requests, it won't work if every node is doing KeepOnlyLatestState. Quite the contrary, NeoFS state sync is possible if node is stripping the state completely, we don't need data from other nodes. And this doesn't require any special support, just drop this check.

roman-khimov · 2025-02-03T15:14:20Z

pkg/core/statesync/module.go

 	defer s.lock.Unlock()
-


roman-khimov · 2025-02-03T15:21:50Z

pkg/services/blockfetcher/blockfetcher.go

 // Start runs the NeoFS BlockFetcher service.
 func (bfs *Service) Start() error {
-	if !bfs.isActive.CompareAndSwap(false, true) {
+	if !bfs.isActive.CompareAndSwap(false, true) || bfs.IsShutdown() {


If you can't reuse this instance, Start() should return an error. If you can it should just start.

roman-khimov · 2025-02-03T15:25:39Z

pkg/services/blockfetcher/blockfetcher.go

+
+	for height := range chain.CommitteeHistory {
+		m = smartcontract.GetDefaultHonestNodeCount(chain.GetNumOfCNs(height))
+		verification, _ = smartcontract.CreateDefaultMultiSigRedeemScript(vs[:chain.GetNumOfCNs(height)])


You don't really care about particular keys, size estimators doesn't need it.

The queue can operate with any item that implements proper interface. Signed-off-by: Ekaterina Pavlova <[email protected]>

Close #3574 Signed-off-by: Ekaterina Pavlova <[email protected]>

Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the headers branch 2 times, most recently from ee083c5 to 6b9d29e Compare January 20, 2025 15:52

AnnaShaleva reviewed Jan 20, 2025

View reviewed changes

AliceInHunterland force-pushed the headers branch 3 times, most recently from e867abb to 26c546e Compare January 21, 2025 08:04

AliceInHunterland requested a review from AnnaShaleva January 21, 2025 08:08

AliceInHunterland force-pushed the headers branch from 26c546e to 3b2b9a6 Compare January 21, 2025 08:10

AnnaShaleva reviewed Jan 21, 2025

View reviewed changes

AliceInHunterland force-pushed the headers branch from 3b2b9a6 to f4d006f Compare January 21, 2025 14:45

AliceInHunterland force-pushed the headers branch from f4d006f to b2c9187 Compare January 21, 2025 18:06

AliceInHunterland requested a review from AnnaShaleva January 22, 2025 07:30

AnnaShaleva reviewed Jan 22, 2025

View reviewed changes

pkg/network/server.go Outdated Show resolved Hide resolved

AliceInHunterland force-pushed the headers branch 2 times, most recently from 2112fe0 to f3629c4 Compare January 22, 2025 20:10

AliceInHunterland requested a review from AnnaShaleva January 22, 2025 20:10

AliceInHunterland marked this pull request as ready for review January 23, 2025 09:22

AliceInHunterland requested a review from roman-khimov as a code owner January 23, 2025 09:22

AnnaShaleva reviewed Jan 23, 2025

View reviewed changes

This was referenced Jan 23, 2025

Sync state via NeoFS block storage #3793

Open

Integrate StateSync module with RemoveUntraceableHeaders option #3795

Open

AliceInHunterland force-pushed the headers branch 2 times, most recently from 495af54 to 49fe75d Compare January 27, 2025 17:18

AliceInHunterland requested a review from AnnaShaleva January 27, 2025 17:19

AliceInHunterland commented Jan 28, 2025

View reviewed changes

AnnaShaleva reviewed Jan 28, 2025

View reviewed changes

AliceInHunterland force-pushed the headers branch 2 times, most recently from 231f7d7 to c3af507 Compare January 28, 2025 16:51

AnnaShaleva requested changes Jan 29, 2025

View reviewed changes

AliceInHunterland force-pushed the headers branch from c3af507 to 4a9c2eb Compare January 29, 2025 15:14

AliceInHunterland mentioned this pull request Jan 29, 2025

Move standard System.Crypto.CheckSig invocation script length to a designated constant #3797

Open

AliceInHunterland force-pushed the headers branch from 4a9c2eb to 646625b Compare January 29, 2025 15:52

AliceInHunterland requested a review from AnnaShaleva January 29, 2025 15:52

AliceInHunterland commented Jan 29, 2025

View reviewed changes

AnnaShaleva reviewed Jan 30, 2025

View reviewed changes

AliceInHunterland mentioned this pull request Jan 30, 2025

Redundant Header.GetIndex() method #3798

Open

AliceInHunterland force-pushed the headers branch 2 times, most recently from dd6c338 to 0997a77 Compare January 30, 2025 12:29

AliceInHunterland requested a review from AnnaShaleva January 30, 2025 12:33

AliceInHunterland force-pushed the headers branch from 0997a77 to 18c8d20 Compare January 31, 2025 08:23

roman-khimov reviewed Feb 3, 2025

View reviewed changes

AliceInHunterland force-pushed the headers branch from 18c8d20 to 0165688 Compare February 4, 2025 13:45

AliceInHunterland added 3 commits February 4, 2025 16:48

bqueue: make queue generic

de50d0b

The queue can operate with any item that implements proper interface. Signed-off-by: Ekaterina Pavlova <[email protected]>

network: integrate state sync module with blockfetcher

ee1989f

Close #3574 Signed-off-by: Ekaterina Pavlova <[email protected]>

blockfetcher: add headerSizeMap to GetRange headers accordingly

d94eb81

Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the headers branch from 0165688 to d94eb81 Compare February 4, 2025 13:50

AliceInHunterland requested a review from roman-khimov February 4, 2025 13:50

		s.bFetcherQueue = bqueue.New(chain, log, nil, s.NeoFSBlockFetcherCfg.BQueueSize, updateBlockQueueLenMetric, bqueue.Blocking)
		s.bFetcherQueue = bqueue.New(s.stateSync, log, nil, s.NeoFSBlockFetcherCfg.BQueueSize, updateBlockQueueLenMetric, bqueue.Blocking)


		s.bSyncQueue = bqueue.New(s.stateSync, log, nil, bqueue.DefaultCacheSize, updateBlockQueueLenMetric, bqueue.NonBlocking)
		s.bSyncQueue = bqueue.New[*block.Block](blockAdapter{s.stateSync}, log, nil, bqueue.DefaultCacheSize, updateBlockQueueLenMetric, bqueue.NonBlocking)

		for _, header := range h.Hdrs {
		err := s.stateSync.AddItems(header)

		}
		index++ //Won't work properly if neofs.ObjectSearch results are not ordered.

Headers fetching via NeoFS BlockFetcher service #3789

Are you sure you want to change the base?

Headers fetching via NeoFS BlockFetcher service #3789

Conversation

AliceInHunterland commented Jan 20, 2025

AnnaShaleva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 21, 2025 • edited Loading

Codecov Report

AliceInHunterland commented Jan 21, 2025

AnnaShaleva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AliceInHunterland Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

AnnaShaleva Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AliceInHunterland Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 21, 2025 •

edited

Loading

AliceInHunterland Jan 27, 2025 •

edited

Loading

AnnaShaleva Jan 28, 2025 •

edited

Loading

AliceInHunterland Jan 29, 2025 •

edited

Loading