Cosmos tx inclusion is not awaited for in the submit step

[tkporter]

Was poking around the explorer and noticed that some Neutron destination messages weren't immediately processed.

Taking a look at the logs for https://explorer.hyperlane.xyz/message/0x5eff3922f7cae967815a5ac19f934c5d7dde07136f613c4569c0affd26414d47:

We hit account sequence # mismatches pretty frequently (i.e. nonce issues). The reason is that we expect our call to mailbox.process to await tx inclusion (EVM and SVM impls do this), but on Cosmos we seemingly don't. Logs https://cloudlogging.app.goo.gl/GE2RfeUgMi1UPZ3R8 and https://cloudlogging.app.goo.gl/iEGwjyaGyM2ptnLV7.

We use broadcast mode Sync https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/main/chains/hyperlane-cosmos/src/providers/grpc.rs#L598, which according to cosmos/cosmos-sdk#4186 (comment) means that it just passes basic validity of the tx but not that it's landed in a block.

This also means that the gas used we get back from there etc isn't canonical.

So concretely we should:

• make sure the process fn for our Cosmos impl awaits tx inclusion, likely by having some logic that polls for delivery

Related: #5310