v10.3.0

Release Notes - Counterparty Core v10.3.0 (2024-07-12)

This version most notably introduces a major performance optimization for node API access: an additional SQLite3 database has been added which tracks the current state of all Counterparty objects (in addition to the primary database that is purely log-structured). Other major changes for this version include the removal of the kickstart functionality, which is possible now that start is performant, and numerous tweaks and improvements to the v2 API.

Upgrading

This update requires an automatic full reparse of the Counterparty transactions to populate the new database file. This process may take a number of hours to complete.

ChangeLog

Bugfixes

• Fix verbose logging of order matches
• Fix the order of NEW_TRANSACTION events on reparse
• Check the ZMQ rawblock topic more frequently on testnet
• Trigger NEW_TRANSACTION_OUTPUT events during a reparse
• Fix incorrect messages_hash after a reparse
• Fix silent errors in ZMQ follower
• Catch TimeoutError during version check; re-check on error
• Catch Bitcoin Core API errors correctly
• Show warning to user when Bitcoin Core cannot be reached

Codebase

• Remove UPDATE query for the addresses table
• Add NEW_ADDRESS_OPTIONS and ADDRESS_OPTIONS_UPDATE events
• Add tx_hash to DISPENSE_UPDATE event
• Add event_hash field to the messages table
• Add a new database optimized for the API. This new database is reconstructed only from events by the API Watcher, and a new field messages.event_hash ensures the correspondence between the two databases in the event of a blockchain reorganization.
• Parse transactions vouts with Rust
• Optimize get_matching_orders() with consolidated cache

API

• Introduce the following new routes:
  • /v2/transactions/<tx_hash>/sends
  • /v2/transactions/<tx_hash>/dispenses
  • /v2/dispenses
  • /v2/sends
  • /v2/issuances
  • /v2/issuances/<tx_hash>
  • /v2/sweeps
  • /v2/sweeps/<tx_hash>
  • /v2/broadcasts
  • /v2/broadcasts/<tx_hash>
  • /v2/assets/<asset>/info
• Add quantity_normalized to issuances endpoints
• Increase the detail for the InsufficientBTC error
• Inject unpacked_data into all results containing a data field when verbose=true
• Remove asset_info from the ASSET_ISSUANCE event
• Standardize on trailing zeros for divisible quantities
• /v2/orders/<order_hash>/matches returns all order matches by default
• Fix cache for /v2/blocks/last route
• Clean and enrich message_data for MPMA sends
• Support dispense message type
• Add supply_normalized to asset info object in API responses
• Add btc_amount to API responses returning dispenses objects
• Use all as the default status when returning orders objects
• Provide link to Apiary documentation in the root route for the v2 API

CLI

• Remove all kickstart functionality; start is now recommended for the initial catchup.
• Tweak RPS logging
• Fix erroneous Rust Fetcher errors on shutdown
• Use millisecond precision for logging

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett

v10.2.0

Release Notes - Counterparty Core v10.2.0 (2024-06-11)

This is a large release that includes significant refactoring and redesigns of critical node components, including the CLI and logging subsystems, mempool processing, and API database connection management. It also includes numerous updates and extensions to the v2 API, plus new ZeroMQ support. Of course, a large number of bugs have been resolved as well.

Upgrading

The counterparty-server process now uses ZeroMQ to communicate with Bitcoin Core, in addition to the traditional RPC API. You must configure Bitcoin Core with:

[main]
zmqpubrawtx=tcp://0.0.0.0:9332
zmqpubhashtx=tcp://0.0.0.0:9332
zmqpubsequence=tcp://0.0.0.0:9332
zmqpubrawblock=tcp://0.0.0.0:9333
[test]
zmqpubrawtx=tcp://0.0.0.0:19332
zmqpubhashtx=tcp://0.0.0.0:19332
zmqpubsequence=tcp://0.0.0.0:19332
zmqpubrawblock=tcp://0.0.0.0:19333

Note: This update requires a reparse from Block 819250, which will proceed automatically upon initialization.

ChangeLog

Bugfixes

• Fix circular imports
• Return 404 errors for undefined API routes
• Fix nested result value in the v1 API
• Fix burned, earned, and btc_amount_normalized fields in API results
• Do not cache non-cacheable v2 API routes (which could lead to a broken health check, in particular)
• Fix the execution interval of the APIStatusPoller thread
• Add Access-Control-Allow-Headers = * header to API v2 for CORS

Codebase

• Refactor mempool management and block tracking—catching up is now done via RPC, and tracking via ZeroMQ
• Introduce a new Rust module to fetch blocks from Bitcoin Core over RPC. Calls are now massively parallelized and buffered; block fetching no longer slows down block parsing
• Refactor the backend module; separate calls to Bitcoin Core and AddrIndexRs into two different modules
• Add the indexed tx_hash field to the messages table
• Update rowtracer so that apsw returns a boolean instead of an integer for BOOL type fields
• Delete the defunct implementation of rock-paper-scissors; introduce a replay_events() function to reconstruct the database from a hard-coded list of historical events
• Enable parsing blocks while Bitcoin Core is still catching up
• Optimize get_pending_dispensers() by adding the last_status_tx_source and close_block_index fields to the dispensers table; optimize is_dispensable() by caching a list of all dispenser addresses
• Add the transaction_count field to the blocks table
• Add the following new database indexes:
  • credits.calling_function
  • debits.action
  • transactions.source
  • credit.quantity
  • debit.quantity
  • balance.quantity
  • dispenser.give_quantity
  • order.get_quantity
  • order.give_quantity
  • dispense.dispense_quantity
• Add the new EVENT log level
• Disable the automatic SQLite ‘quick check’ for when the database has not been closed correctly. Display an error message when exiting and at the next restart.
• Add retries to get_oldest_tx() on a Addrindexrs timeout

API

• Introduce the following new routes:
  • /v2/assets/<asset>/dispenses
  • /v2/addresses/<address>/issuances
  • /v2/addresses/<address>/assets
  • /v2/addresses/<address>/transactions
  • /v2/addresses/<address>/dividends
  • /v2/addresses/<address>/orders
  • /v2/addresses/<address>/dispenses/sends
  • /v2/addresses/<address>/dispenses/receives
  • /v2/addresses/<address>/dispenses/sends/<asset>
  • /v2/addresses/<address>/dispenses/receives/<asset>
  • /v2/blocks/<block_hash>
  • /v2/blocks/last
  • /v2/transactions
  • /v2/transactions/<int:tx_index>/events
  • /v2/transactions/<tx_hash>/events
  • /v2/transactions/<int:tx_index>/events/<event>
  • /v2/transactions/<tx_hash>/events/<event>
  • /v2/transactions/<int:tx_index>
  • /v2/dividends
  • /v2/dividends/<dividend_hash>
  • /v2/dividends/<dividend_hash>/credits
  • /v2/mempool/transactions/<tx_hash>/events
  • /v2/events/<event>/count
  • /v2/bets
  • /v2/orders
  • /v2/dispensers
• Introduce the cursor API parameter
• Include next_cursor and result_count fields in all responses
• Accept cursor/offset and limit arguments in all queries that return lists from the database (see the Pagination paragraph of the API Documentation)
• Make the asset, assets, give_asset, and get_asset parameters case-insensitive
• Add the named=true|false parameter to /v2/assets to returning only named / numeric assets
• Publish events on the ZeroMQ Pub/Sub channel
• Implement database connection pooling for both API v1 and v2
• Enrich results containing block_index with block_time when verbose=true
• Add an action filter for the */credits and */debits routes
• Add an event_name filter for the */events routes
• Specify issuer=None within XCP and BTC asset information
• Exclude zero balances in the results of /v2/addresses/<address>/balances and zero quantities in the results of */credits and */debits
• Add BTC sent to the DISPENSE event
• Accept trailing slashes in routes
• Include first_issuance_block_index and last_issuance_block_index in asset information
• Add normalized quantities for the following fields:
  • fee_paid
  • fee_provided
  • fee_required
  • fee_required_remaining
  • fee_provided_remaining
  • fee_fraction_int
  • quantity_per_unit
  • btc_amount_normalized
  • burned
  • earned
  • dispense_quantity
• Return asset info for dispenses
• Add confirmation_target argument for compose endpoints

CLI

• Use -v for the DEBUG level, -vv for the EVENT level, and -vvv for the TRACE level (it is also possible to repeat the --verbose flag)
• Clean up and refactor CLI outputs for all commands
• Clean up log messages and add numerous additional logging statements

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello

v10.1.2

Release Notes - Counterparty Core v10.1.2 (2024-05-08)

This version of Counterparty Core marks the release of API v2, a new RESTful API—see the official project documentation. The new API is available at /v2/, while the old API is now available at /v1/ in addition to /.

Upgrading

There is a guide for migrating from the v1 to the v2 API in the documentation, which specifies equivalences between old and new functionality.

This release maintains full backwards-compatibility and includes no protocol changes.

ChangeLog

Bugfixes

• Fix logging of some raw tracebacks (#1715)
• Retry on ChunkedEncodingError with AddrIndexRs; break loop on all errors
• Fix bad logging of Rust module panic (#1721)

Codebase

• Release API v2
• Have both API v1 and v2 return ready if the last block is less than one minute old
• Add an index on the block_index field in the credits and debits tables
• Add TRACE level to Python logging
• Add basic anonymous node telemetry

Command-Line Interface

• Set default and minimum values for Backend Poll Interval to 3.0 seconds
• Update docker-compose.yml to use different profiles for mainnet and testnet
• Check that another process is not connected to the database before starting the server
• Launch database quick check on startup if the database has not been correctly shut down
• Support an additional level of verbosity with the CLI flags -vv
• Add the --no-telemetry flag to disable node telemetry

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello

v10.1.1

Release Notes - Counterparty Core v10.1.1 (2024-04-19)

This is a relatively small release with a number of bugfixes, one of which is critical---in v10.0.x and v10.1.0 there is a bug which can cause nodes to crash upon a blockchain reorganization.

Upgrading

To upgrade from v10.1.0 manually, you must first uninstall the following Counterparty Core Python packages:

pip3 uninstall counterparty-rs counterparty-lib counterparty-cli

This release contains no protocol changes, and the API has not been modified.

ChangeLog

Bugfixes

• Fix missing events (NEW_BLOCK and NEW_TRANSACTION) when kickstarting and reparsing. To correct the values in the messages table, a full reparse is required.
• Fix the current block index after a blockchain reorganisation.
• Fix database shutdown, which caused a recovery of the WAL file on each startup.
• Eliminate some extraneous error messages

Codebase

• Merge counterparty-lib and counterparty-core package into counterparty-core
• Integrate telemetry with optional Sentry service

Command-Line Interface

• Replace --no-check-asset-conservation with --check-asset-conservation
• Disable automatic DB integrity check on startup

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello

v10.1.0

Release Notes - Counterparty Core v10.1.0 (2024-04-16)

This release includes fixes for a number of bugs as well as a few regressions in v10.0.x.

Upgrade Procedure

This upgrade is optional but highly recommended. Upgrading from v10.0.x requires an automatic reparse from block 835,500 on mainnet, which should take a few minutes. If you are upgrading directly from v9.x.y, then there will be an automatic full database rebuild, which may take a long time (refer to the release notes for v10.0.0.)

In order to perform a manual installation, you must first uninstall all existing Counterparty Core Python packages:

pip3 uninstall counterparty-rs counterparty-lib counterparty-cli

ChangeLog

Bugfixes

• Validate non-empty block_indexes in call to api.get_blocks (fix for #1621)
• Reproduce order expiration bug in v9.61.x (fix for #1631)
• Fix get_blocks call when several block indexes are provided (fix for #1629)
• Fix create_send when one of the outputs is a dispenser (fix for #1119)
• Fix get_dispenser_info RPC call

Codebase

• Split out counterparty-cli package into counterparty-core and counterparty-wallet packages
• Implement heavy healthz probe (default to light)
• Automatic code checking and correction with Ruff
• Refactor transaction file singleton to class
• Run PRAGMA optimize on shutting down
• Run PRAGMA quick_check on database initialization
• Temporary disable asset conservation checking after each new block
• Add instrumentation for optional Sentry error and performance monitoring

Command-Line Interface

• Rename counterpary-client to counterparty-wallet
• Add --skip-db-check flag to skip database quick check
• Add --no-mempool flash to disable mempool parsing

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello

v10.1.0-rc.1: Merge pull request #1640 from CounterpartyXCP/develop

Release Notes - Counterparty Core v10.1.0-rc.1

Note: This update requires a reparse from block 835,500. (If you are upgrading from v9.x.y, then you need a full database rebuild.)

ChangeLog

Bugfixes

• Validate non-empty block_indexes in call to api.get_blocks (fix for #1621)
• Reproduce order expiration bug in v9.61 (fix for #1631)
• Fix get_blocks call when several block indexes are provided (fix for #1629)

Codebase

• Split out counterparty-cli package into counterparty-core and counterparty-wallet packages
• Implement light / heavy healthz probes
• Automatic code checking and correction with Ruff
• Refactor transaction file singleton to class
• Run PRAGMA optimize on shutting down

Command-Line Interface

• Rename counterpary-client to counterparty-wallet

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello

v10.0.1

Release Notes - Counterparty Core v10.0.1 (2024-04-09)

Hotfix release for #1619

ChangeLog

Stability and Correctness

• Add missing sanity check in address unpacking for dispensers that causes a complete network crash

Credits

• Adam Krellenstein

v9.61.3

Release Notes - Counterparty Core v9.61.3 (2024-04-09)

Hotfix release for #1619

ChangeLog

Stability and Correctness

• Add missing sanity check in address unpacking for dispensers that causes a complete network crash

Credits

• Adam Krellenstein

v10.0.0

Release Notes - Counterparty Core v10.0.0 (2024-04-09)

Counterparty Core v10.0.0 is a very large release comprising many improvements across different portions of the codebase. “Counterparty Core” is also the new name for the codebase and repository that is the result of a merge between counterparty-lib, counterparty-cli and a new Rust library, counterparty-rs.

Upgrade Procedure

This release does not include any protocol changes, so there is no deadline for upgrading. However it is strongly recommended that all users upgrade as soon as possible, in particular to avoid consensus problems due to non-determinism in previous versions. The Counterparty Core API is also unchanged for this release. The minimum required version of AddrIndexRs has been increased to v0.4.6.

Because this release includes numerous changes to the database schema, a full database rebuild is required and the major version number has been bumped from 9 to 10. You may upgrade either by using Docker Compose or by installing Counterparty Core manually, following the installation instructions available in the official project documentation. When using kickstart, the database rebuild will happen automatically and take between 8 and 24 hours to complete.

ChangeLog

Codebase

• Upgrade from Python 3.7 to Python >= 3.10
• Support Ubuntu 22.04 and macOS officially
• Upgrade packaging system: replace setup.py with pyproject.toml and use Hatchling as a build system
• Upgrade all pip dependencies to the latest available version
• Rename counterparty-lib repository to counterparty-core. NOTE: The normal redirect for GitHub URLs cannot be implemented.
• Merge the counterparty-cli repository into the counterparty-core repository
• Add Rust library, counterparty-rs, for performance-critical code
• Synchronize versions of counterparty-rs, counterparty-lib and counterparty-cli
• Update URL for hosting bootstrap files to https://bootstrap.counterparty.io/counterparty-*
• Update URL for hosting notifications for protocol changes to https://counterparty.io/protocol_changes.json

Documentation and Testing

• Fix test suite, with automated builds on supported operating systems
• Add GitHub Workflows for building, testing and running automated code scanners:
  • PyLint
  • Bandit
  • CodeQL
  • License Scanner
  • Build and publish Docker image
  • Enable testnet test book
  • Test docker-compose.yml in Google Compute Engine VM
• Add checkpoints for mainnet up to block 837,000 and for testnet up to block 2,580,000
• Rewrite README

Stability and Correctness

• Fix multiple sources of non-determinism caused by generic exception handling
• Fix source of non-determinism in AddrIndexRs caused by txid_limit and get_oldest_tx
• Fix consensus break due to missing support for segwit transactions in kickstart logic
• Fix crash in software version checking caused by a format change of protocol_changes.json

Deployment

• Rewrite Dockerfile and publish new official Docker images
• Create Docker Compose file as an alternative to Federated Node
• Change default bitcoind user from bitcoinrpc to rpc
• Change default port for communication with AddrIndexRs to 8432 (and 81432 for testnet)

Command-Line Interface

• Disable console logs except for with counterparty-server start
• Show fancy spinners for all discrete operations
• Rename checkdb command to check-db and refactor
• Rename debugconfig to show-config; clean up output
• Don't log values for transactions except with --verbose (for performance)
• Move noisy log messages to DEBUG
• Always log to a file, unless --no-log-files is set
• Fix and refactor log.set_up()
• Improve thread shutdown logic
• Accept config args before and after the command
• New flag --json-log, which replaces the human-readable logs in the console with the streaming content of the messages table in JSON format

Refactoring and Performance Optimizations

• Rewrite kickstart, splitting work across two Python processes using shared memory and queue for communication
• Activate write-ahead-log in SQLite and implement apsw.best_practices(), improving performance and fixing crashes from deadlocks
• Fix database version checking which launches a rebuild instead of a rollback / reparse
• Add numerous missing database indexes
• Fix collisions between existing database indexes
• DRY and refactor database index creation
• DRY and isolate all SQL queries in ledger.py
• Refactor log.messages and log.log heavily
• Add additional fields and rows in the messages table (the messages hash will change)
• Fix database integrity check and re-include assert conservation check
• Migrate to log-structured database for simpler and faster rollback and reparse
  • Add block_index field to the balances table
  • Remove all UPDATE queries—use the ledger.insert_update() function, which adds a new row with a new block_index
  • Update all SELECT queries—always use MAX(rowid)
  • Remove the undolog completely
  • Implement rollback and reparse by deleting table rows using the block_index field
• Migrate performance-critical logic to Rust library, counterparty-rs
  • b58_encode() and b58_decode()
  • script_to_asm()
  • script_to_address()
  • inverse_hash()
• Refactor connection logic for AddrIndexRs RPC
• Pre-fetch blocks with multiple threads for start
• DRY and refactor get_tx_info*() functions
  • Isolate transaction parsing inside gettxinfo.py module
  • Heavily refactor code; eliminate unused code blocks
  • Isolate dispenser logic in get_dispensers_outputs() and get_dispensers_tx_info()
• Change software version check interval to 24 hours
• Add the possibility to reparse from a given block on minor version change
• Add warning with confirmation dialogue to bootstrap command, and --no-confirm flag
• Add REST endpoints optimized for tables that were altered during the migration to a log-structured database:
  • GET /addresses/<address>/balances
  • GET /assets/<asset>/
  • GET /assets/<asset>/balances
  • GET /assets/<asset>/orders
  • GET /orders/<tx_hash>
  • GET /orders/<tx_hash>/matches
• Adjust the error message for when a user has an insufficient BTC balance but the unconfirmed flag has already been passed

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello

v10.0.0-rc.1

Release Notes - Counterparty Core v10.0.0-rc.1

Counterparty Core v10.0.0 is a very large release comprising many improvements across different portions of the codebase. “Counterparty Core” is also the new name for the codebase and repository that is the result of a merge between counterparty-lib, counterparty-cli and a new Rust library, counterparty-rs.

Upgrade Procedure

This release does not include any protocol changes, so there is no deadline for upgrading. However it is strongly recommended that all users upgrade as soon as possible, in particular to avoid consensus problems due to non-determinism in previous versions. The Counterparty Core API is also unchanged for this release.

Because this release includes numerous changes to the database schema, a full database rebuild is required and the major version number has been bumped from 9 to 10. Follow the updated installation instructions in the README to download and install the latest version of Counterparty Core, run counterparty-server kickstart (while bitcoind is not running), then start the server with counterparty-server start. The rebuild should happen automatically, and it should take between 8 and 24 hours hours to complete.

IMPORTANT Be certain that you are running the latest version of AddrIndexRs (v0.4.6).

ChangeLog

Codebase

• Upgrade from Python 3.7 to Python >= 3.10
• Support Ubuntu 22.04 and macOS officially
• Upgrade packaging system: replace setup.py with pyproject.toml and use Hatchling as a build system
• Upgrade all pip dependencies to the latest available version
• Rename counterparty-lib repository to counterparty-core. NOTE: The normal redirect for GitHub URLs cannot be implemented.
• Merge the counterparty-cli repository into the counterparty-core repository
• Add Rust library, counterparty-rs, for performance-critical code
• Synchronize versions of counterparty-rs, counterparty-lib and counterparty-cli
• Update URL for hosting bootstrap files to https://bootstrap.counterparty.io/counterparty-*
• Update URL for hosting notifications for protocol changes to https://counterparty.io/protocol_changes.json

Documentation and Testing

• Fix test suite, with automated builds on supported operating systems
• Add Github Workflows for building, testing and running automated code scanners:
  • PyLint
  • Bandit
  • CodeQL
  • License Scanner
  • Build and publish Docker image
  • Enable testnet test book
  • Test compose.yml in Google Compute Engine VM
• Add checkpoints for mainnet up to block 834,500 and for testnet up to block 2,580,000
• Rewrite README

Stability and Correctness

• Fix multiple sources of non-determinism caused by generic exception handling
• Fix source of non-determinism in AddrIndexRs caused by txid_limit and get_oldest_tx
• Fix consensus break due to missing support for segwit transactions in kickstart logic
• Fix crash in software version checking caused by a format change of protocol_changes.json

Deployment

• Rewrite Dockerfile and publish new official Docker images
• Create Docker Compose file as an alternative to Federated Node
• Change default bitcoind user from bitcoinrpc to rpc
• Changed default port for communication with AddrIndexRs to 8432 (and 81432 for testnet)

Command-Line Interface

• Disable console logs except for with counterparty-server start
• Show fancy spinners for all discrete operations
• Rename checkdb command to check-db
• Rename debugconfig to show-config; clean up output
• Don't log values for transactions except with --verbose (for performance)
• Move noisy log messages to DEBUG
• Always log to a file, unless --no-log-files is set
• Fix and refactor log.set_up()
• Improve thread shutdown logic
• Accept config args before and after the command
• New flag --json-log, which replaces the human-readable logs in the console with the streaming content of the messages table in JSON format

Refactoring and Performance Optimizations

• Rewrite kickstart, splitting work across two Python processes using shared memory and queue for communication
• Activate write-ahead-log in SQLite and implement apsw.best_pratices(), improving performance and fixing crashes from deadlocks
• Fix database version checking which launched a rebuilds instead of rollbacks / reparses
• Add numerous missing database indexes
• Fix collisions between existing database indexes
• DRY and refactor database index creation
• DRY and isolate all SQL queries in ledger.py
• Heavily refactor of log.messages and log.log.
• Add additional fields and rows in the messages table (the messages hash will change)
• Fix database integrity check and re-include assert conservation check
• Migrate to log-structured database for simpler and faster rollback and reparse
  • Add block_index field to the balances table
  • Remove all UPDATE queries—use the ledger.insert_update() function, which adds a new row with a new block_index
  • Update all SELECT queries—always use MAX(rowid)
  • Remove the undolog completely
  • Implement rollback and reparse by deleting table rows using the block_index field
• Migrate performance-critical logic to Rust library, counterparty-rs
  • b58_encode() and b58_decode()
  • script_to_asm()
  • script_to_address()
  • inverse_hash()
• Refactor connection logic for AddrIndexRs RPC
• Pre-fetch blocks with multiple threads for start
• DRY and refactor get_tx_info*() functions
  • Isolate transaction parsing inside gettxinfo.py module
  • Heavily refactor code; eliminate unused code blocks
  • Isolate dispenser logic in get_dispensers_outputs() and get_dispensers_tx_info()
• Activate check software version every 24H
• Add the possibility to reparse from a given block on minor version change
• Add warning with confirmation dialogue to bootstrap command, and --no-confirm flag
• Add REST endpoints optimized for tables that were altered during the migration to a log-structured database:
  • GET /addresses/<address>/balances
  • GET /assets/<asset>/
  • GET /assets/<asset>/balances
  • GET /assets/<asset>/orders
  • GET /orders/<tx_hash>
  • GET /orders/<tx_hash>/matches

Credits

• Ouziel Slama
• Adam Krellenstein
• Warren Puffett
• Matt Marcello