Want to understand more details about disk type's impact on performance and data synchronization

[jejass]

Hello Team,

We are currently assisting cx in their efforts to leverage the blob CSI driver to invoke blobfuse for accessing a Storage account from their AKS cluster. During our investigation, we have encountered the following questions and would appreciate your insights:

Question 1: Synchronization Mechanism and Error Handling in Blobfuse

We would like to confirm our understanding of the blobfuse mechanism, where data is initially written to a local temporary path before being synchronized with the Storage server. Could you please clarify the following:

Are there any scenarios that may prevent successful synchronization? In the event of a sync operation failure, how does blobfuse handle this situation? Does the eviction of local cache operations verify whether the data has been successfully uploaded to the Storage server? How does blobfuse ensure that data in the local temporary path is ultimately written to Storage without the risk of data loss?

Question 2: Blobfuse Disk Type Selection Logic

Additionally, we observed a performance discrepancy in the AKS cluster, specifically related to the choice of pod SKU and the corresponding disk type. We test the performance by running below command in the pod.
while true; do BT=$(date +'%Y/%m/%d %T.%9N');dd bs=1M count=1000 if=/dev/zero of=/mnt/blob/testfile oflag=direct >>/root/log.txt 2>&1 ;echo "Begin at $BT, Sucess at $(date +'%Y/%m/%d %T.%9N') ">> /root/log.txt 2>&1 ;sleep 0.5 ; done

Below testing result is using 'Standard_D8s_v5' SKU. The initial upload speed is significantly higher but decreases after subsequent attempts.

Our findings indicate a correlation with pod SKUs, with "Standard_D8ds_v5" showing better performance due to its ephemeral OS disk.

We are curious about the logic behind blobfuse's choice of disk type. Could you provide insights into how blobfuse determines which disk type to use? How does disk type affect the performance? Are there any considerations or configurations that can be adjusted to optimize performance, especially when working with different pod SKUs?

Your expertise in addressing these queries is crucial for cx's successful implementation, and we appreciate your time and assistance. Please feel free to reach out if you require any further details.

[andyzhangx]

@jejass about the second question: the blob csi driver would set /mnt as cache dir by default, /mnt is a temp disk (with high performance) on Standard_D8ds_v5, while Standard_D8s_v5 does not have temp disk, so /mnt dir is actually on os disk, that's the reason why you could see higer perf on Standard_D8ds_v5.

@vibhansa-msft @gapra-msft can you answer the first question about Synchronization Mechanism and Error Handling in Blobfuse? Thanks.