How to tune NVM cache write latency when disk cache warms up

[tangliisu]

In our project, we build a hybrid cache with 100GB memory and 150GB SSD. We found the latency goes up very high when the cache starts to warm up and the latency goes to normal (writes drop) when disk fills up. We didn't see a similar behavior when we use DRAM cache (100GB memory) only. We believe the high latency is due to the SSD write cost.

Could we get some guidance on how to tune the hybrid cache to get rid of this high latency during cache warm up? Right now, nvmConfig we use is mostly as defaulted. We only set the fileSize = 150GB and fileName = ssd cache path.

One parameter we try to tune is maxConcurrentInserts but when we reduce it from default value 1000000 to 100, we didn't see any change in the disk write requests per seconds

[therealgymmy]

@tangliisu: hey sorry for the delay. We missed this question. Are you still seeing this issue?

Yeah your description sounds like the issue is due to cache warm-up. Do you have any stats backing this up? (For example, the write rate to SSD from beginning until cache is fully warmed up). If you have a replay workload that can run on cachebench, we can also try to repro this on our end.

Within Meta, services that have this type of behavior usually adopt a "slow-start" behavior. When a host cold starts, the clients will slowly ramp up the traffic. This mitigates the high write cost on warm-up you're seeing.

If this is not an option, you can enforce a write-cap via admission policy. (E.g. set max write rate to 100MB/s).

You can configure it here: https://fburl.com/zd6oiluw