Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get block notifications API #3781

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/rpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,10 @@ block. It can be removed in future versions, but at the moment you can use it
to see how much GAS is burned with a particular block (because system fees are
burned).

#### `getblocknotifications` call

This method returns notifications from a block organized by trigger type. Supports filtering by contract and event name.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, preserve common line width (~80 symbols).


#### Historic calls

A set of `*historic` extension methods provide the ability of interacting with
Expand Down
106 changes: 105 additions & 1 deletion pkg/core/block/block.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,20 @@
Transactions []json.RawMessage `json:"tx"`
}

type TrimmedBlock struct {
Header
TxHashes []util.Uint256
}

type auxTrimmedBlockOut struct {
TxHashes []util.Uint256 `json:"tx"`
}

// auxTrimmedBlockIn is used for JSON i/o.
type auxTrimmedBlockIn struct {
TxHashes []json.RawMessage `json:"tx"`
}
Comment on lines +59 to +66
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON marshallers are not needed for this structure. DecodeBinary is enough for our purposes.


// ComputeMerkleRoot computes Merkle tree root hash based on actual block's data.
func (b *Block) ComputeMerkleRoot() util.Uint256 {
hashes := make([]util.Uint256, len(b.Transactions))
Expand Down Expand Up @@ -221,7 +235,6 @@
return size
}

// ToStackItem converts Block to stackitem.Item.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useless change.

func (b *Block) ToStackItem() stackitem.Item {
items := []stackitem.Item{
stackitem.NewByteArray(b.Hash().BytesBE()),
Expand All @@ -241,3 +254,94 @@

return stackitem.NewArray(items)
}

// NewTrimmedBlockFromReader creates a block with only the header and transaction hashes.
func NewTrimmedBlockFromReader(stateRootEnabled bool, br *io.BinReader) (*TrimmedBlock, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse NewTrimmedFromReader for that and remove this function. And we don't even need this new TrimmedBlock structure at all, because we're fine with Block with Trimmed set to true.

block := &TrimmedBlock{
Header: Header{
StateRootEnabled: stateRootEnabled,
},
}

block.Header.DecodeBinary(br)
lenHashes := br.ReadVarUint()
if lenHashes > MaxTransactionsPerBlock {
return nil, ErrMaxContentsPerBlock
}

Check warning on line 270 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L269-L270

Added lines #L269 - L270 were not covered by tests
if lenHashes > 0 {
block.TxHashes = make([]util.Uint256, lenHashes)
for i := range lenHashes {
block.TxHashes[i].DecodeBinary(br)
}
}

return block, br.Err
}

func (b TrimmedBlock) MarshalJSON() ([]byte, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also should be removed, why do we need this code?

Copy link
Author

@lock9 lock9 Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first, I considered adding a get trimmed block endpoint. I also had some issues to debug the RPC endpoints (it doesn't stop on the debugger). I had an error that I didn't know where it was coming from. Later I realized that it was due to the notification object serialization, not the block. The same for the 'toStackItem'. I didn't know if that was being used (now I know).

I also got confused because the tests 'were passing', when in practice, they weren't. I 'suspected' that this was related to some inner serilization / deserialization that I wasn't aware of. That's why I included these methods. I'll remove this and other parts that aren't being used.

abo := auxTrimmedBlockOut{
TxHashes: b.TxHashes,
}

if abo.TxHashes == nil {
abo.TxHashes = []util.Uint256{}
}

Check warning on line 288 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L287-L288

Added lines #L287 - L288 were not covered by tests
auxb, err := json.Marshal(abo)
if err != nil {
return nil, err
}

Check warning on line 292 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L291-L292

Added lines #L291 - L292 were not covered by tests
baseBytes, err := json.Marshal(b.Header)
if err != nil {
return nil, err
}

Check warning on line 296 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L295-L296

Added lines #L295 - L296 were not covered by tests

// Does as the 'normal' block does
if baseBytes[len(baseBytes)-1] != '}' || auxb[0] != '{' {
return nil, errors.New("can't merge internal jsons")
}

Check warning on line 301 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L300-L301

Added lines #L300 - L301 were not covered by tests
baseBytes[len(baseBytes)-1] = ','
baseBytes = append(baseBytes, auxb[1:]...)
return baseBytes, nil
}

func (b *TrimmedBlock) UnmarshalJSON(data []byte) error {
auxb := new(auxTrimmedBlockIn)
err := json.Unmarshal(data, auxb)
if err != nil {
return err
}

Check warning on line 312 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L311-L312

Added lines #L311 - L312 were not covered by tests
err = json.Unmarshal(data, &b.Header)
if err != nil {
return err
}

Check warning on line 316 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L315-L316

Added lines #L315 - L316 were not covered by tests
if len(auxb.TxHashes) != 0 {
b.TxHashes = make([]util.Uint256, len(auxb.TxHashes))
for i, hashBytes := range auxb.TxHashes {
err = json.Unmarshal(hashBytes, &b.TxHashes[i])
if err != nil {
return err
}

Check warning on line 323 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L322-L323

Added lines #L322 - L323 were not covered by tests
}
}
return nil
}

func (b *TrimmedBlock) ToStackItem() stackitem.Item {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this necessary?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, remove it. We're only interested in proper deserialization from bytes.

items := []stackitem.Item{
stackitem.NewByteArray(b.Hash().BytesBE()),
stackitem.NewBigInteger(big.NewInt(int64(b.Version))),
stackitem.NewByteArray(b.PrevHash.BytesBE()),
stackitem.NewByteArray(b.MerkleRoot.BytesBE()),
stackitem.NewBigInteger(big.NewInt(int64(b.Timestamp))),
stackitem.NewBigInteger(new(big.Int).SetUint64(b.Nonce)),
stackitem.NewBigInteger(big.NewInt(int64(b.Index))),
stackitem.NewBigInteger(big.NewInt(int64(b.PrimaryIndex))),
stackitem.NewByteArray(b.NextConsensus.BytesBE()),
stackitem.NewBigInteger(big.NewInt(int64(len(b.TxHashes)))),
}
if b.StateRootEnabled {
items = append(items, stackitem.NewByteArray(b.PrevStateRoot.BytesBE()))
}

Check warning on line 344 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L329-L344

Added lines #L329 - L344 were not covered by tests

return stackitem.NewArray(items)

Check warning on line 346 in pkg/core/block/block.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/block/block.go#L346

Added line #L346 was not covered by tests
}
41 changes: 41 additions & 0 deletions pkg/core/block/block_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,47 @@ func TestTrimmedBlock(t *testing.T) {
}
}

func TestNewTrimmedBlockFromReader(t *testing.T) {
block := getDecodedBlock(t, 1)

buf := io.NewBufBinWriter()
block.EncodeTrimmed(buf.BinWriter)
require.NoError(t, buf.Err)

r := io.NewBinReaderFromBuf(buf.Bytes())
trimmedBlock, err := NewTrimmedBlockFromReader(false, r)
require.NoError(t, err)

assert.Equal(t, block.Version, trimmedBlock.Version)
assert.Equal(t, block.PrevHash, trimmedBlock.PrevHash)
assert.Equal(t, block.MerkleRoot, trimmedBlock.MerkleRoot)
assert.Equal(t, block.Timestamp, trimmedBlock.Timestamp)
assert.Equal(t, block.Index, trimmedBlock.Index)
assert.Equal(t, block.NextConsensus, trimmedBlock.NextConsensus)

assert.Equal(t, block.Script, trimmedBlock.Script)
assert.Equal(t, len(block.Transactions), len(trimmedBlock.TxHashes))
for i := range block.Transactions {
assert.Equal(t, block.Transactions[i].Hash(), trimmedBlock.TxHashes[i])
}

data, err := json.Marshal(trimmedBlock)
require.NoError(t, err)

var decoded TrimmedBlock
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)

assert.Equal(t, trimmedBlock.Version, decoded.Version)
assert.Equal(t, trimmedBlock.PrevHash, decoded.PrevHash)
assert.Equal(t, trimmedBlock.MerkleRoot, decoded.MerkleRoot)
assert.Equal(t, trimmedBlock.Timestamp, decoded.Timestamp)
assert.Equal(t, trimmedBlock.Index, decoded.Index)
assert.Equal(t, trimmedBlock.NextConsensus, decoded.NextConsensus)
assert.Equal(t, trimmedBlock.Script, decoded.Script)
assert.Equal(t, trimmedBlock.TxHashes, decoded.TxHashes)
}

func newDumbBlock() *Block {
return &Block{
Header: Header{
Expand Down
5 changes: 5 additions & 0 deletions pkg/core/blockchain.go
Original file line number Diff line number Diff line change
Expand Up @@ -3149,3 +3149,8 @@ func (bc *Blockchain) GetStoragePrice() int64 {
}
return bc.contracts.Policy.GetStoragePriceInternal(bc.dao)
}

// GetTrimmedBlock returns a block with only the header and transaction hashes.
func (bc *Blockchain) GetTrimmedBlock(hash util.Uint256) (*block.TrimmedBlock, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return (*block.Block, error) from this method.

return bc.dao.GetTrimmedBlock(hash)
}
15 changes: 15 additions & 0 deletions pkg/core/dao/dao.go
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,21 @@
return block, nil
}

// GetTrimmedBlock returns a block with only the header and transaction hashes.
func (dao *Simple) GetTrimmedBlock(hash util.Uint256) (*block.TrimmedBlock, error) {
key := dao.makeExecutableKey(hash)
b, err := dao.Store.Get(key)
if err != nil {
return nil, err
}

r := io.NewBinReaderFromBuf(b)
if r.ReadB() != storage.ExecBlock {
return nil, storage.ErrKeyNotFound
}

Check warning on line 453 in pkg/core/dao/dao.go

View check run for this annotation

Codecov / codecov/patch

pkg/core/dao/dao.go#L452-L453

Added lines #L452 - L453 were not covered by tests
return block.NewTrimmedBlockFromReader(dao.Version.StateRootInHeader, r)
}

// Version represents the current dao version.
type Version struct {
StoragePrefix storage.KeyPrefix
Expand Down
50 changes: 50 additions & 0 deletions pkg/core/dao/dao_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

Expand Down Expand Up @@ -123,6 +124,55 @@ func TestPutGetBlock(t *testing.T) {
require.Error(t, err)
}

func TestGetTrimmedBlock(t *testing.T) {
dao := NewSimple(storage.NewMemoryStore(), false)
tx := transaction.New([]byte{byte(opcode.PUSH1)}, 0)
tx.Signers = []transaction.Signer{{Account: util.Uint160{1, 2, 3}}}
tx.Scripts = []transaction.Witness{{}}

b := &block.Block{
Header: block.Header{
Timestamp: 42,
Script: transaction.Witness{
VerificationScript: []byte{byte(opcode.PUSH1)},
InvocationScript: []byte{byte(opcode.NOP)},
},
},
Transactions: []*transaction.Transaction{tx},
}
hash := b.Hash()
appExecResult1 := &state.AppExecResult{
Container: hash,
Execution: state.Execution{
Trigger: trigger.OnPersist,
Events: []state.NotificationEvent{},
Stack: []stackitem.Item{},
},
}
err := dao.StoreAsBlock(b, appExecResult1, nil)
require.NoError(t, err)

trimmedBlock, err := dao.GetTrimmedBlock(hash)
require.NoError(t, err)
require.NotNil(t, trimmedBlock)

assert.Equal(t, b.Version, trimmedBlock.Version)
assert.Equal(t, b.PrevHash, trimmedBlock.PrevHash)
assert.Equal(t, b.MerkleRoot, trimmedBlock.MerkleRoot)
assert.Equal(t, b.Timestamp, trimmedBlock.Timestamp)
assert.Equal(t, b.Index, trimmedBlock.Index)
assert.Equal(t, b.NextConsensus, trimmedBlock.NextConsensus)
assert.Equal(t, b.Script, trimmedBlock.Script)

assert.Equal(t, len(b.Transactions), len(trimmedBlock.TxHashes))
for i := range b.Transactions {
assert.Equal(t, b.Transactions[i].Hash(), trimmedBlock.TxHashes[i])
}

_, err = dao.GetTrimmedBlock(util.Uint256{1, 2, 3})
require.Error(t, err)
}

func TestGetVersion_NoVersion(t *testing.T) {
dao := NewSimple(storage.NewMemoryStore(), false)
version, err := dao.GetVersion()
Expand Down
12 changes: 12 additions & 0 deletions pkg/neorpc/result/block_notifications.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package result

import (
"github.com/nspcc-dev/neo-go/pkg/core/state"
)

// BlockNotifications represents notifications from a block organized by trigger type.
type BlockNotifications struct {
PrePersistNotifications []state.ContainedNotificationEvent `json:"prepersist,omitempty"`
TxNotifications []state.ContainedNotificationEvent `json:"transactions,omitempty"`
PostPersistNotifications []state.ContainedNotificationEvent `json:"postpersist,omitempty"`
Comment on lines +7 to +11
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/PrePersistNotifications/OnPersist
s/TxNotifications/Application
s/PostPersistNotifications/PostPersist

And I'd request @roman-khimov's opinion on this structure, do we need this explicit separation? ContainedNotificationEvent has container hash, may be it's enough for our purposes wrt #3781 (comment).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[]ContainedNotificationEvent is sufficient to me, it will be ordered natually, it has triggers inside to separate pre/post. We better not introduce additional types if we can.

Copy link
Member

@AnnaShaleva AnnaShaleva Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it has triggers inside

It doesn't, that's the case. But it has Container, technically it's possible to distinguish block/transaction executions (but not OnPersist/PostPersist in case if there's no Application-related notifications in between).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, OK. Likely we don't care for notifications, if something happened it happened. Container is here anyway.

}
9 changes: 9 additions & 0 deletions pkg/rpcclient/rpc.go
Original file line number Diff line number Diff line change
Expand Up @@ -972,3 +972,12 @@
}
return resp, nil
}

// GetBlockNotifications returns notifications from a block organized by trigger type.
func (c *Client) GetBlockNotifications(blockHash util.Uint256, filters ...*neorpc.NotificationFilter) (*result.BlockNotifications, error) {
var resp = &result.BlockNotifications{}
if err := c.performRequest("getblocknotifications", []any{blockHash.StringLE(), filters}, resp); err != nil {
return nil, err
}
return resp, nil

Check warning on line 982 in pkg/rpcclient/rpc.go

View check run for this annotation

Codecov / codecov/patch

pkg/rpcclient/rpc.go#L977-L982

Added lines #L977 - L982 were not covered by tests
}
58 changes: 58 additions & 0 deletions pkg/services/rpcsrv/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@
VerifyWitness(util.Uint160, hash.Hashable, *transaction.Witness, int64) (int64, error)
mempool.Feer // fee interface
ContractStorageSeeker
GetTrimmedBlock(hash util.Uint256) (*block.TrimmedBlock, error)
}

// ContractStorageSeeker is the interface `findstorage*` handlers need to be able to
Expand Down Expand Up @@ -185,6 +186,14 @@
// Item represents Iterator stackitem.
Item stackitem.Item
}

notificationComparatorFilter struct {
id neorpc.EventID
filter neorpc.SubscriptionFilter
}
notificationEventContainer struct {
ntf *state.ContainedNotificationEvent
}
Comment on lines +189 to +196
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move it to a separate file. Ref. #3781 (comment).

)

const (
Expand Down Expand Up @@ -219,6 +228,7 @@
"getblockhash": (*Server).getBlockHash,
"getblockheader": (*Server).getBlockHeader,
"getblockheadercount": (*Server).getBlockHeaderCount,
"getblocknotifications": (*Server).getBlockNotifications,
AnnaShaleva marked this conversation as resolved.
Show resolved Hide resolved
AnnaShaleva marked this conversation as resolved.
Show resolved Hide resolved
"getblocksysfee": (*Server).getBlockSysFee,
"getcandidates": (*Server).getCandidates,
"getcommittee": (*Server).getCommittee,
Expand Down Expand Up @@ -3202,3 +3212,51 @@
}
return tx.Bytes(), nil
}

// getBlockNotifications returns notifications from a specific block with optional filtering.
func (s *Server) getBlockNotifications(reqParams params.Params) (any, *neorpc.Error) {
param := reqParams.Value(0)
hash, respErr := s.blockHashFromParam(param)
if respErr != nil {
return nil, respErr
}

block, err := s.chain.GetTrimmedBlock(hash)
if err != nil {
return nil, neorpc.ErrUnknownBlock
}

var filter *neorpc.NotificationFilter
if len(reqParams) > 1 {
filter = new(neorpc.NotificationFilter)
err := json.Unmarshal(reqParams[1].RawMessage, filter)
if err != nil {
return nil, neorpc.WrapErrorWithData(neorpc.ErrInvalidParams, fmt.Sprintf("invalid filter: %s", err))
}

Check warning on line 3235 in pkg/services/rpcsrv/server.go

View check run for this annotation

Codecov / codecov/patch

pkg/services/rpcsrv/server.go#L3234-L3235

Added lines #L3234 - L3235 were not covered by tests
if err := filter.IsValid(); err != nil {
return nil, neorpc.WrapErrorWithData(neorpc.ErrInvalidParams, fmt.Sprintf("invalid filter: %s", err))
}

Check warning on line 3238 in pkg/services/rpcsrv/server.go

View check run for this annotation

Codecov / codecov/patch

pkg/services/rpcsrv/server.go#L3237-L3238

Added lines #L3237 - L3238 were not covered by tests
}

notifications := &result.BlockNotifications{}

aers, err := s.chain.GetAppExecResults(block.Hash(), trigger.OnPersist)
if err == nil && len(aers) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return this error if it's not nil. Ref. #3781 (comment).

notifications.PrePersistNotifications = processAppExecResults([]state.AppExecResult{aers[0]}, filter)
}

for _, txHash := range block.TxHashes {
aers, err := s.chain.GetAppExecResults(txHash, trigger.Application)
if err != nil {
return nil, neorpc.NewInternalServerError("failed to get app exec results")
}
notifications.TxNotifications = append(notifications.TxNotifications, processAppExecResults(aers, filter)...)

Check warning on line 3253 in pkg/services/rpcsrv/server.go

View check run for this annotation

Codecov / codecov/patch

pkg/services/rpcsrv/server.go#L3249-L3253

Added lines #L3249 - L3253 were not covered by tests
}

aers, err = s.chain.GetAppExecResults(block.Hash(), trigger.PostPersist)
if err == nil && len(aers) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

notifications.PostPersistNotifications = processAppExecResults([]state.AppExecResult{aers[0]}, filter)
}

return notifications, nil
}
Loading
Loading