arc: avoid possible deadlock in arc_read

[ixhamza]

Motivation and Context

In l2arc_evict(), the config lock may be acquired in reverse order (e.g., first the config lock (writer), then a hash lock) unlike in arc_read() during scenarios like L2ARC device removal. To avoid deadlocks, if the attempt to acquire the config lock (reader) fails in arc_read(), release the hash lock, wait for the config lock, and retry from the beginning.

Description

While I have not been able to reproduce the issue locally, I decoded the following kernel trace from a customer's logs that resulted in a complete system lockup during L2ARC vdev removal:

L2Cache Remove Context

l2arc_evict() => Tries to acquire `hash_lock` mutex it already acquired by arc_read() in below context
    l2arc_remove_vdev()
        spa_load_l2cache()
            spa_vdev_remove() => Acquired spa_config_lock, spa_namespace_lock
                zfs_ioc_vdev_remove()

ZFS Write context

spa_cofig_enter() => Waiting for spa_config_lock Lock to release but acquired by spa_vdev_remove() context
    zfs_blkptr_verify()
        arc_read() => Acquire hash_lock mutex
            dbuf_read_impl()
                dbuf_read()
                    dmu_tx_check_ioerr()
                        dmu_tx_count_write()
                            dmu_tx_hold_write_by_dnode()
                                zfs_write()
                                    zpl_iter_write()

How Has This Been Tested?

• CI Testing

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.