Crash on planned upgrade

[matthewdarwin]

Is it normal to crash on planned upgrade? Seems a clean exit would be better?

also seen on osmosis.

10:41PM INF commit synced commit=436F6D6D697449447B5B323134203132312038203136382031393120323431203131342039352036332032303820373320313734203133332033322032353520343220313439203539203139372037332031383020313932203138382035203139352034352031353020362032332035203132203230335D3A3134304543327D
10:41PM INF committed state app_hash=D67908A8BFF1725F3FD049AE8520FF2A953BC549B4C0BC05C32D960617050CCB height=1314498 module=state num_txs=4
10:41PM INF finalized dmlog block block=1314498 module=deepmind
10:41PM INF indexed block height=1314498 module=txindex
10:41PM INF client state updated client-id=07-tendermint-1450 height=3-12698 module=x/ibc/client
10:41PM INF executed block height=1314499 module=state num_invalid_txs=0 num_valid_txs=2
10:41PM INF commit synced commit=436F6D6D697449447B5B313530203230312031383020323232203332203232392033342031303020393620393620372039362031333020313935203433203738203233372031383520333320313434203232352031393320393220353520313739203137352032303920373120313435203138382035312039325D3A3134304543337D
10:41PM INF committed state app_hash=96C9B4DE20E522646060076082C32B4EEDB92190E1C15C37B3AFD14791BC335C height=1314499 module=state num_txs=2
10:41PM INF finalized dmlog block block=1314499 module=deepmind
10:41PM INF indexed block height=1314499 module=txindex
10:41PM ERR UPGRADE "v4" NEEDED at height: 1314500: https://raw.githubusercontent.com/osmosis-labs/networks/main/osmosis-1/upgrades/v4/mainnet/upgrade_4_binaries.json
panic: UPGRADE "v4" NEEDED at height: 1314500: https://raw.githubusercontent.com/osmosis-labs/networks/main/osmosis-1/upgrades/v4/mainnet/upgrade_4_binaries.json
goroutine 133 [running]:
github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{_, _}, _, {_, _}, {_, _}, _}, {{0x2029e18, 0xc000052050}, ...}, ...)
#011github.com/cosmos/[email protected]/x/upgrade/abci.go:70 +0xee8
github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
#011github.com/cosmos/[email protected]/x/upgrade/module.go:127
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2029e18, 0xc000052050}, {0x2034fc8, 0xc06ae3a900}, {{0xb, 0x1}, {0xc18e7891a4, 0x9}, 0x140ec4, ...}, ...}, ...)
#011github.com/cosmos/[email protected]/types/module/module.go:338 +0x3a2
github.com/osmosis-labs/osmosis/app.(*OsmosisApp).BeginBlocker(_, {{0x2029e18, 0xc000052050}, {0x2034fc8, 0xc06ae3a900}, {{0xb, 0x1}, {0xc18e7891a4, 0x9}, 0x140ec4, ...}, ...}, ...)
#011github.com/osmosis-labs/osmosis/app/app.go:562 +0xdb
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc37b276220, 0x20, 0x20}, {{0xb, 0x1}, {0xc18e7891a4, 0x9}, 0x140ec4, {0xb30b256, ...}, ...}, ...})
#011github.com/cosmos/[email protected]/baseapp/abci.go:193 +0x89c
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc37b276220, 0x20, 0x20}, {{0xb, 0x1}, {0xc18e7891a4, 0x9}, 0x140ec4, {0xb30b256, ...}, ...}, ...})
#011github.com/tendermint/[email protected]/abci/client/local_client.go:274 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc37b276220, 0x20, 0x20}, {{0xb, 0x1}, {0xc18e7891a4, 0x9}, 0x140ec4, {0xb30b256, ...}, ...}, ...})
#011github.com/tendermint/[email protected]/proxy/app_conn.go:81 +0x55
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x202a9e8?, 0xc0097f22a0}, {0x202f4a8, 0xc001eaa940}, 0xc148a08b40, {0x2035238, 0xc001eaaa90}, 0x140ec3?)
#011github.com/tendermint/[email protected]/state/execution.go:308 +0x3dd
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x1}, {0xc00977c5e0, 0x7}}, {0xc00977c5e7, 0x9}, 0x1, 0x140ec3, {{0xc368905800, ...}, ...}, ...}, ...)
#011github.com/tendermint/[email protected]/state/execution.go:141 +0x171
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000e7d180, 0x0)
#011github.com/tendermint/[email protected]/blockchain/v0/reactor.go:398 +0xb5a
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
#011github.com/tendermint/[email protected]/blockchain/v0/reactor.go:110 +0x7a
2022-08-21T22:41:25.412Z (ingestor) event logs reader finished (firehose-cosmos/ingestor.go:77){"error": "exit status 2"}
2022-08-21T22:41:25.419Z (ingestor) mindreader is stopping (mindreader/mindreader.go:241)
2022-08-21T22:41:25.420Z (ingestor) waiting until consume read flow (i.e. blocks) is actually done processing blocks... (mindreader/mindreader.go:255)
2022-08-21T22:41:25.420Z (console_reader) lines channel has been closed (codec/console_reader.go:74)
2022-08-21T22:41:25.422Z (ingestor) reached end of console reader stream, nothing more to do (mindreader/mindreader.go:228)
2022-08-21T22:41:25.423Z (ingestor) all blocks in channel were drained, exiting read flow (mindreader/mindreader.go:269)
2022-08-21T22:41:25.423Z (ingestor) archiver selector is terminating (mindreader/archiver_selector.go:88)
2022-08-21T22:41:25.423Z (ingestor) merger archiver is terminating (mindreader/merge_archiver.go:65)
2022-08-21T22:41:25.423Z (ingestor) terminating upload loop (mindreader/merge_archiver.go:100)
2022-08-21T22:41:27.263Z (ingestor) merger archiver is terminated (mindreader/merge_archiver.go:73)
2022-08-21T22:41:27.263Z (ingestor) one block archiver is terminating (mindreader/archiver.go:66)

Copy of issue: graphprotocol#35