bump changelog for release

CHANGELOG.md

@@ -4,9 +4,57 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). See [MAINTAINERS.md](./MAINTAINERS.md)
 for instructions to keep up to date.
 
-## Unreleased
+## v2.4.0
 
-* Substreams server @v1.4.0: performance improvements: less redundant module execution (at the cost of more cache storage). See https://github.com/streamingfast/substreams/releases/tag/v1.4.0.
+
+## Substreams
+
+* Substreams bumped to @v1.5.0: See https://github.com/streamingfast/substreams/releases/tag/v1.5.0 for details.
+
+#### Chain-agnostic tier2
+
+* A single substreams-tier2 instance can now serve requests for multiple chains or networks. All network-specific parameters are now passed from Tier1 to Tier2 in the internal ProcessRange request.
+* This allows you to better use your computing resources by pooling all the networks together.
+
+> [!IMPORTANT]
+> Since the `tier2` services will now get the network information from the `tier1` request, you must make sure that the file paths and network addresses will be the same for both tiers.
+> ex: if `--common-merged-blocks-store-url=/data/merged` is set on tier1, make sure the merged blocks are also available from tier2 under the path `/data/merged`.
+> The flags `--substreams-state-store-url`, `--substreams-state-store-default-tag`,  `--common-merged-blocks-store-url`, `--substreams-rpc-endpoints stringArray` and `--substreams-rpc-gas-limit` are now ignored on tier2. 
+> The flag `--common-first-streamable-block` should be set to 0 to accommodate every chain.
+> Non-ethereum chains can query a `firehose-ethereum` tier2, but the opposite is not true, since only the `firehose-ethereum` implements the `eth_call` WASM extension.
+
+> [!TIP]
+> The cached 'partial' files no longer contain the "trace ID" in their filename, preventing accumulation of "unsquashed" partial store files. The system will delete files under '{modulehash}/state' named in this format`{blocknumber}-{blocknumber}.{hexadecimal}.partial.zst` when it runs into them.
+
+#### Performance improvements
+
+* All module outputs are now cached. (previously, only the last module was cached, along with the "store snapshots", to allow parallel processing). 
+* Tier2 will now read back mapper outputs (if they exist) to prevent running them again. Additionally, it will not read back the full blocks if its inputs can be satisfied from existing cached mapper outputs.
+* Tier2 will skip processing completely if it's processing the last stage and the `output_module` is a mapper that has already been processed (ex: when multiple requests are indexing the same data at the same time)
+* Tier2 will skip processing completely if it's processing a stage where all the stores and outputs have been processed and cached.
+* Scheduler modification: a stage now waits for the previous stage to have completed the same segment before running, to take advantage of the cached intermediate layers.
+* Improved file listing performance for Google Storage backends by 25%!
+
+> [!TIP]
+> Concurrent requests on the same module hashes may benefit from the other requests' work to a certain extent (up to 75%!) -- The very first request does most of the work for the other ones.
+
+> [!TIP]
+> More caches will increase disk usage and there is no automatic removal of old module caches. The operator is responsible for deleting old module caches.
+
+> [!TIP]
+> The cached 'partial' files no longer contain the "trace ID" in their filename, preventing accumulation of "unsquashed" partial store files.
+> The system will delete files under '{modulehash}/state' named in this format`{blocknumber}-{blocknumber}.{hexadecimal}.partial.zst` when it runs into them.
+
+#### Metrics
+
+* Readiness metric for Substreams tier1 app is now named `substreams_tier1` (was mistakenly called `firehose` before).
+* Added back readiness metric for Substreams tiere app (named `substreams_tier2`).
+* Added metric `substreams_tier1_active_worker_requests` which gives the number of active Substreams worker requests a tier1 app is currently doing against tier2 nodes.
+* Added metric `substreams_tier1_worker_request_counter` which gives the total Substreams worker requests a tier1 app made against tier2 nodes.
+
+### Flags
+
+* Added `--merger-delete-threads` to customize the number of threads the merger will use to delete files. It's recommended to increase this when using Ceph as S3 storage provider to 25 or higher (due to performance issues with deletes the merger might otherwise not be able to delete one-block files fast enough).
 
 ## v2.3.7
 

go.mod

@@ -21,12 +21,12 @@ require (
 	github.com/streamingfast/dstore v0.1.1-0.20240325191553-bcce8892a9bb
 	github.com/streamingfast/eth-go v0.0.0-20240312122859-216e183c0b7f
 	github.com/streamingfast/firehose v0.1.1-0.20240118135215-dcf04d40bfcd
-	github.com/streamingfast/firehose-core v1.2.6-0.20240402142301-912d47b6141d
+	github.com/streamingfast/firehose-core v1.3.0
 	github.com/streamingfast/jsonpb v0.0.0-20210811021341-3670f0aa02d0
 	github.com/streamingfast/logging v0.0.0-20230608130331-f22c91403091
 	github.com/streamingfast/pbgo v0.0.6-0.20240131193313-6b88bc7139db
 	github.com/streamingfast/shutter v1.5.0
-	github.com/streamingfast/substreams v1.4.1-0.20240402141712-b2a8afdd34cc
+	github.com/streamingfast/substreams v1.5.0
 	github.com/stretchr/testify v1.8.4
 	github.com/test-go/testify v1.1.4
 	github.com/tidwall/gjson v1.14.1

go.sum

@@ -607,8 +607,8 @@ github.com/streamingfast/eth-go v0.0.0-20240312122859-216e183c0b7f h1:1LwsmRVENf
 github.com/streamingfast/eth-go v0.0.0-20240312122859-216e183c0b7f/go.mod h1:UEm8dqibr3c3A1iIA3CHpkhN7j3X78prN7/55sXf3A0=
 github.com/streamingfast/firehose v0.1.1-0.20240118135215-dcf04d40bfcd h1:t5n8dDcgUi7t36Qwxm19K4H2vyOLJfY6MHxTbOvK1z8=
 github.com/streamingfast/firehose v0.1.1-0.20240118135215-dcf04d40bfcd/go.mod h1:du6tys2Q6X2pRQ3JbCziWiy7Y7KrOcl4CSb9uiGsVxA=
-github.com/streamingfast/firehose-core v1.2.6-0.20240402142301-912d47b6141d h1:cZmblsgCUbvber4VObh2Zzm8PoNQWWAhUa6GVaLR2IM=
-github.com/streamingfast/firehose-core v1.2.6-0.20240402142301-912d47b6141d/go.mod h1:fqONkRuvkqEEDvih18//LSCIfhlQKgAXu9NOFGcJva4=
+github.com/streamingfast/firehose-core v1.3.0 h1:33KqCOkqunvcC+KXggwklwhS/4Hd0NTGCrU3oOTYgwY=
+github.com/streamingfast/firehose-core v1.3.0/go.mod h1:MfhLyDdVt97yJtSVGyYKfxpZRejtNCWdXXhnT93zJDQ=
 github.com/streamingfast/jsonpb v0.0.0-20210811021341-3670f0aa02d0 h1:g8eEYbFSykyzIyuxNMmHEUGGUvJE0ivmqZagLDK42gw=
 github.com/streamingfast/jsonpb v0.0.0-20210811021341-3670f0aa02d0/go.mod h1:cTNObq2Uofb330y05JbbZZ6RwE6QUXw5iVcHk1Fx3fk=
 github.com/streamingfast/logging v0.0.0-20210811175431-f3b44b61606a/go.mod h1:4GdqELhZOXj4xwc4IaBmzofzdErGynnaSzuzxy0ZIBo=
@@ -630,8 +630,8 @@ github.com/streamingfast/shutter v1.5.0 h1:NpzDYzj0HVpSiDJVO/FFSL6QIK/YKOxY0gJAt
 github.com/streamingfast/shutter v1.5.0/go.mod h1:B/T6efqdeMGbGwjzPS1ToXzYZI4kDzI5/u4I+7qbjY8=
 github.com/streamingfast/snapshotter v0.0.0-20230316190750-5bcadfde44d0 h1:Y15G1Z4fpEdm2b+/70owI7TLuXadlqBtGM7rk4Hxrzk=
 github.com/streamingfast/snapshotter v0.0.0-20230316190750-5bcadfde44d0/go.mod h1:/Rnz2TJvaShjUct0scZ9kKV2Jr9/+KBAoWy4UMYxgv4=
-github.com/streamingfast/substreams v1.4.1-0.20240402141712-b2a8afdd34cc h1:pXQnwI5YzmzmHpiokba10MCs986RsatPa+YGh/g8DuQ=
-github.com/streamingfast/substreams v1.4.1-0.20240402141712-b2a8afdd34cc/go.mod h1:k9YNZTghlkEZdXEooNdnOYZx30y+W1u0OZMLU80bYcc=
+github.com/streamingfast/substreams v1.5.0 h1:kx+zKKljUPdVynrdrSVHhDyxjxzT/JRWQavnbUN1+p4=
+github.com/streamingfast/substreams v1.5.0/go.mod h1:k9YNZTghlkEZdXEooNdnOYZx30y+W1u0OZMLU80bYcc=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=