Async IO does not guarantee order of operations

[LtdJorge]

At least on io_uring, the order in which completions come back is not the order in which submissions were sent. This means that we could send a batch of writes, and finish it with an fsync, with the fsync happening first. Crashing right after or losing power would mean that data was supposed to be in storage but isn't.

Right now, I think this is not a big problem because, unless I'm mistaken, we are not sending concurrent IO (issuing a read, and while waiting for that, issuing a write, where the write could be completed before the read). Well, waiting for IO is not actually implemented yet. Someone with more knowledge of the VDBE, please correct me if I'm wrong.

I think, for io_uring at least, this could be fixed by adding IO hints, which I mentioned in #684 and would also resolve #203. This means that the IOs would have hints (e.g read page, but another read for the next page will come after, or sync but only after all previously submitted IOs have completed). For io_uring this could be done by using the IOSQE_IO_LINK flag. However, that requires to prepare all linked opcodes with the flag first and then send them in one submit call. For sync in particular, it could also be done by submitting fsync with the IOSQE_IO_DRAIN flag, which waits for all submissions to complete, and then completes the fsync.