Skip to content

Commit

Permalink
blockchain: Convert to full block index in mem.
Browse files Browse the repository at this point in the history
This reworks the block index code such that it loads all of the headers
in the main chain at startup and constructs the full block index
accordingly.

Since the full index from the current best tip all the way back to the
genesis block is now guaranteed to be in memory, this also removes all
code related to dynamically loading the nodes and updates some of the
logic to take advantage of the fact traversing the block index can no
longer potentially fail.  There are also many more optimizations and
simplifications that can be made in the future as a result of this.

Due to removing all of the extra overhead of tracking the dynamic state,
and ensuring the block node structs are aligned to eliminate extra
padding, the end result of a fully populated block index now takes quite
a bit less memory than the previous dynamically loaded version.

It also speeds up the initial startup process by roughly 2x since it is
faster to bulk load the nodes in order as opposed to dynamically loading
only the nodes near the tip in backwards order.

For example, here is some startup timing information before and after
this commit on a node that contains roughly 238,000 blocks:

7200 RPM HDD:
-------------
Startup time before this commit: ~7.71s
Startup time after this commit: ~3.47s

SSD:
----
Startup time before this commit: ~6.34s
Startup time after this commit: ~3.51s

Some additional benefits are:

- Since every block node is in memory, the code which reconstructs
  headers from block nodes means that all headers can always be served
  from memory which will be important since the network will be moving
  to header-based semantics
- Several of the error paths can be removed since they are no longer
  necessary
- It is no longer expensive to calculate CSV sequence locks or median
  times of blocks way in the past
- It is much less expensive to calculate the initial states for the
  various intervals such as the stake and voter version
- It will be possible to create much more efficient iteration and
  simplified views of the overall index

An overview of the logic changes are as follows:

- Move AncestorNode from blockIndex to blockNode and greatly simplify
  since it no longer has to deal with the possibility of dynamically
  loading nodes and related failures
- Replace nodeAtHeightFromTopNode from BlockChain with RelativeAncestor
  on blockNode and define it in terms of AncestorNode
- Move CalcPastMedianTime from blockIndex to blockNode and remove no
  longer necessary test for nil
- Remove findNode and replace all of its uses with direct queries of the
  block index
- Remove blockExists and replace all of its uses with direct queries of
  the block index
- Remove all functions and fields related to dynamically loading nodes
  - children and parentHash fields from blockNode
  - depNodes from blockIndex
  - loadBlockNode from blockIndex
  - PrevNodeFromBlock from blockIndex
  - {p,P}revNodeFromNode from blockIndex
  - RemoveNode
- Replace all instances of iterating backwards through nodes to directly
  access the parent now that nodes don't potentially need to be
  dynamically loaded
- Introduce a lookupNode function on blockIndex which allows the
  initialization code to locklessly query the index
- No longer take the chain lock when only access to the block index,
  which has its own lock, is needed
- Removed the error paths from several functions that can no longer fail
  - getReorganizeNodes
  - findPrevTestNetDifficulty
  - sumPurchasedTickets
  - findStakeVersionPriorNode
- Removed all error paths related to node iteration that can no longer
  fail
- Modify FetchUtxoView to return an empty view for the genesis block
davecgh committed Jun 1, 2018
1 parent 36b4fdc commit fc91d2c
Showing 17 changed files with 431 additions and 1,025 deletions.
16 changes: 4 additions & 12 deletions blockchain/accept.go
Original file line number Diff line number Diff line change
@@ -114,34 +114,26 @@ func IsFinalizedTransaction(tx *dcrutil.Tx, blockHeight int64, blockTime time.Ti
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) maybeAcceptBlock(block *dcrutil.Block, flags BehaviorFlags) (int64, error) {
// Get a block node for the block previous to this one. Will be nil
// if this is the genesis block.
prevNode, err := b.index.PrevNodeFromBlock(block)
if err != nil {
log.Debugf("PrevNodeFromBlock: %v", err)
return 0, err
}

// This function should never be called with orphan blocks or the
// genesis block.
prevHash := &block.MsgBlock().Header.PrevBlock
prevNode := b.index.LookupNode(prevHash)
if prevNode == nil {
prevHash := &block.MsgBlock().Header.PrevBlock
str := fmt.Sprintf("previous block %s is not known", prevHash)
return 0, ruleError(ErrMissingParent, str)
}

// There is no need to validate the block if an ancestor is already
// known to be invalid.
if b.index.NodeStatus(prevNode).KnownInvalid() {
prevHash := &block.MsgBlock().Header.PrevBlock
str := fmt.Sprintf("previous block %s is known to be invalid",
prevHash)
return 0, ruleError(ErrInvalidAncestorBlock, str)
}

// The block must pass all of the validation rules which depend on the
// position of the block within the block chain.
err = b.checkBlockContext(block, prevNode, flags)
err := b.checkBlockContext(block, prevNode, flags)
if err != nil {
return 0, err
}
@@ -197,7 +189,7 @@ func (b *BlockChain) maybeAcceptBlock(block *dcrutil.Block, flags BehaviorFlags)

// Grab the parent block since it is required throughout the block
// connection process.
parent, err := b.fetchBlockByHash(&newNode.parentHash)
parent, err := b.fetchBlockByHash(&newNode.parent.hash)
if err != nil {
return 0, err
}
Loading

0 comments on commit fc91d2c

Please sign in to comment.