feature: migrate from pyfilesystem2 to fsspec

[judahrand]

Feature request

This change would allow for remote blob storage to be directly used for Model storage and Bento storage. fsspec has backends/implementations for S3, GCS, Azure Blob, Databricks Filesystem, FTP, etc.

This can only be a good thing! Additionally, integrating with universal-pathlib might be a good idea.

fsspec
universal-pathlib

Is this a move that would be accepted as a Pull Request?

Motivation

We'd like to be able to package Bentos and Models in CI but currently since BentoML only writes to the local filesystem this is not easily possible. Our workaround is to just not store the Bentos or the Models!

Other

No response

[ssheng]

Great suggestion. With remote storage, concurrency will likely have to be handled more explicitly compared to local storage.