indexers: Implement optional tx/address indexes.

This introduces a new indexing infrastructure for supporting optional
indexes using the new database and blockchain infrastructure along with
two concrete indexer implementations which provide both a
transaction-by-hash and a transaction-by-address index.

The new infrastructure is mostly separated into a package named indexers
which is housed under the blockchain package.  In order to support this,
a new interface named IndexManager has been introduced in the blockchain
package which provides methods to be notified when the chain has been
initialized and when blocks are connected and disconnected from the main
chain.  A concrete implementation of an index manager is provided by the
new indexers package.

The new indexers package also provides a new interface named Indexer
which allows the index manager to manage concrete index implementations
which conform to the interface.

The following is high level overview of the main index infrastructure
changes:

- Define a new IndexManager interface in the blockchain package and
  modify the package to make use of the interface when specified
- Create a new indexers package
  - Provides an Index interface which allows concrete indexes to plugin
    to an index manager
  - Provides a concrete IndexManager implementation
    - Handles the lifecycle of all indexes it manages
    - Tracks the index tips
    - Handles catching up disabled indexes that have been reenabled
    - Handles reorgs while the index was disabled
    - Invokes the appropriate methods for all managed indexes to allow
      them to index and deindex the blocks and transactions
  - Implement a transaction-by-hash index
    - Makes use of internal block IDs to save a significant amount of
      space and indexing costs over the old transaction index format
  - Implement a transaction-by-address index
    - Makes use of a leveling scheme in order to provide a good tradeoff
      between space required and indexing costs
- Supports enabling and disabling indexes at will
- Support the ability to drop indexes if they are no longer desired

The following is an overview of the btcd changes:

- Add a new index logging subsystem
- Add new options --txindex and --addrindex in order to enable the
  optional indexes
  - NOTE: The transaction index will automatically be enabled when the
    address index is enabled because it depends on it
- Add new options --droptxindex and --dropaddrindex to allow the indexes
  to be removed
  - NOTE: The address index will also be removed when the transaction
    index is dropped because it depends on it
- Update getrawtransactions RPC to make use of the transaction index
- Reimplement the searchrawtransaction RPC that makes use of the address
  index
- Update sample-btcd.conf to include sample usage for the new optional
  index flags

blockchain/chain.go

@@ -176,6 +176,7 @@ type BlockChain struct {
 	chainParams         *chaincfg.Params
 	notifications       NotificationCallback
 	sigCache            *txscript.SigCache
+	indexManager        IndexManager
 
 	// subsidyCache is the cache that provides quick lookup of subsidy
 	// values.
@@ -983,7 +984,7 @@ func (b *BlockChain) calcPastMedianTime(startNode *blockNode) (time.Time, error)
 		var err error
 		iterNode, err = b.getPrevNodeFromNode(iterNode)
 		if err != nil {
-			log.Errorf("getPrevNodeFromNode: %v", err)
+			log.Errorf("getPrevNodeFromNode failed to find node: %v", err)
 			return time.Time{}, err
 		}
 	}
@@ -1085,6 +1086,20 @@ func (b *BlockChain) pushMainChainBlockCache(block *dcrutil.Block) {
 	b.mainchainBlockCacheLock.Unlock()
 }
 
+// dbMaybeStoreBlock stores the provided block in the database if it's not
+// already there.
+func dbMaybeStoreBlock(dbTx database.Tx, block *dcrutil.Block) error {
+	hasBlock, err := dbTx.HasBlock(block.Sha())
+	if err != nil {
+		return err
+	}
+	if hasBlock {
+		return nil
+	}
+
+	return dbTx.StoreBlock(block)
+}
+
 // connectBlock handles connecting the passed node/block to the end of the main
 // (best) chain.
 //
@@ -1200,21 +1215,32 @@ func (b *BlockChain) connectBlock(node *blockNode, block *dcrutil.Block,
 		}
 
 		// Insert the block into the database if it's not already there.
-		hasBlock, err := dbTx.HasBlock(block.Sha())
+		err = dbMaybeStoreBlock(dbTx, block)
 		if err != nil {
 			return err
 		}
-		if !hasBlock {
-			return dbTx.StoreBlock(block)
+
+		// Allow the index manager to call each of the currently active
+		// optional indexes with the block being connected so they can
+		// update themselves accordingly.
+		if b.indexManager != nil {
+			err := b.indexManager.ConnectBlock(dbTx, block, parent, view)
+			if err != nil {
+				return err
+			}
 		}
 
 		return nil
 	})
 	if err != nil {
+		log.Errorf("Failed to insert block %v: %s", node.hash, err.Error())
+
 		// Attempt to restore TicketDb if this fails.
-		_, _, _, errRemove := b.tmdb.RemoveBlockToHeight(node.height - 1)
-		if errRemove != nil {
-			return errRemove
+		if node.height >= b.chainParams.StakeEnabledHeight {
+			_, _, _, errRemove := b.tmdb.RemoveBlockToHeight(node.height - 1)
+			if errRemove != nil {
+				return errRemove
+			}
 		}
 
 		return err
@@ -1353,6 +1379,16 @@ func (b *BlockChain) disconnectBlock(node *blockNode, block *dcrutil.Block,
 			return err
 		}
 
+		// Allow the index manager to call each of the currently active
+		// optional indexes with the block being disconnected so they
+		// can update themselves accordingly.
+		if b.indexManager != nil {
+			err := b.indexManager.DisconnectBlock(dbTx, block, parent, view)
+			if err != nil {
+				return err
+			}
+		}
+
 		return nil
 	})
 	if err != nil {
@@ -1956,6 +1992,23 @@ func (b *BlockChain) BestSnapshot() *BestState {
 	return snapshot
 }
 
+// IndexManager provides a generic interface that the is called when blocks are
+// connected and disconnected to and from the tip of the main chain for the
+// purpose of supporting optional indexes.
+type IndexManager interface {
+	// Init is invoked during chain initialize in order to allow the index
+	// manager to initialize itself and any indexes it is managing.
+	Init(*BlockChain) error
+
+	// ConnectBlock is invoked when a new block has been connected to the
+	// main chain.
+	ConnectBlock(database.Tx, *dcrutil.Block, *dcrutil.Block, *UtxoViewpoint) error
+
+	// DisconnectBlock is invoked when a block has been disconnected from
+	// the main chain.
+	DisconnectBlock(database.Tx, *dcrutil.Block, *dcrutil.Block, *UtxoViewpoint) error
+}
+
 // Config is a descriptor which specifies the blockchain instance configuration.
 type Config struct {
 	// DB defines the database which houses the blocks and will be used to
@@ -1990,6 +2043,13 @@ type Config struct {
 	// This field can be nil if the caller is not interested in using a
 	// signature cache.
 	SigCache *txscript.SigCache
+
+	// IndexManager defines an index manager to use when initializing the
+	// chain and connecting and disconnecting blocks.
+	//
+	// This field can be nil if the caller does not wish to make use of an
+	// index manager.
+	IndexManager IndexManager
 }
 
 // New returns a BlockChain instance using the provided configuration details.
@@ -2020,6 +2080,7 @@ func New(config *Config) (*BlockChain, error) {
 		chainParams:             params,
 		notifications:           config.Notifications,
 		sigCache:                config.SigCache,
+		indexManager:            config.IndexManager,
 		root:                    nil,
 		bestNode:                nil,
 		index:                   make(map[chainhash.Hash]*blockNode),
@@ -2038,6 +2099,14 @@ func New(config *Config) (*BlockChain, error) {
 		return nil, err
 	}
 
+	// Initialize and catch up all of the currently active optional indexes
+	// as needed.
+	if config.IndexManager != nil {
+		if err := config.IndexManager.Init(&b); err != nil {
+			return nil, err
+		}
+	}
+
 	b.subsidyCache = NewSubsidyCache(b.bestNode.height, b.chainParams)
 
 	log.Infof("Blockchain database version %v loaded successfully",

blockchain/chainio.go

@@ -1579,6 +1579,11 @@ func dbMainChainHasBlock(dbTx database.Tx, hash *chainhash.Hash) bool {
 	return hashIndex.Get(hash[:]) != nil
 }
 
+// DBMainChainHasBlock is the exported version of dbMainChainHasBlock.
+func DBMainChainHasBlock(dbTx database.Tx, hash *chainhash.Hash) bool {
+	return dbMainChainHasBlock(dbTx, hash)
+}
+
 // MainChainHasBlock returns whether or not the block with the given hash is in
 // the main chain.
 //

blockchain/indexers/README.md

@@ -0,0 +1,48 @@
+indexers
+========
+
+[![Build Status](https://travis-ci.org/btcsuite/btcd.png?branch=master)]
+(https://travis-ci.org/btcsuite/btcd)
+
+Package indexers implements optional block chain indexes.
+
+These indexes are typically used to enhance the amount of information available
+via an RPC interface.
+
+## Supported Indexers
+
+- Transaction-by-hash (txbyhashidx) Index
+  - Creates a mapping from the hash of each transaction to the block that
+    contains it along with its offset and length within the serialized block
+- Transaction-by-address (txbyaddridx) Index
+  - Creates a mapping from every address to all transactions which either credit
+    or debit the address
+  - Requires the transaction-by-hash index
+- Address-ever-seen (existsaddridx) Index
+  - Stores a key with an empty value for every address that has ever existed 
+    and was seen by the client
+  - Requires the transaction-by-hash index
+
+## Documentation
+
+[![GoDoc](https://godoc.org/github.com/decred/dcrd/blockchain/indexers?status.png)]
+(http://godoc.org/github.com/decred/dcrd/blockchain/indexers)
+
+Full `go doc` style documentation for the project can be viewed online without
+installing this package by using the GoDoc site here:
+http://godoc.org/github.com/btcsuite/btcd/blockchain/indexers
+
+You can also view the documentation locally once the package is installed with
+the `godoc` tool by running `godoc -http=":6060"` and pointing your browser to
+http://localhost:6060/pkg/github.com/decred/dcrd/blockchain/indexers
+
+## Installation
+
+```bash
+$ go get -u github.com/decred/dcrd/blockchain/indexers
+```
+
+## License
+
+Package indexers is licensed under the [copyfree](http://copyfree.org) ISC
+License.