Vulkan memory allocations with UMA architecture

[ddpasa]

Vulkan is a great blessing for those of us running on consumer hardware with iGPUs. I have an Intel Iris igpu that actually turns out to be pretty decent with Vulkan acceleration. Many thanks to @0cc4m and others who have made this happen. You are amazing!

I want to understand something that I'm seeing on my laptop. The setup has 16GB of main memory, but it looks like Vulkan can't access all of it. For example, there are models that fit into the combined memory when I lower the -ngl value, but fail to run at higher -ngl values due to failed Vulkan memory allocations (ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory).

This is surprising because regardless of whether the model layer is on Vulkan or on the CPU, it sits on the same system memory. This leads me to think that somehow the part of system memory that Vulkan can access is lower than the total amount of memory.

Does this make sense? Or is there something else going on? Are layers replicated twice (once for Vulkan and once again for CPU) even for UMA?

[0cc4m]

Layers are not duplicated, but the available memory to an iGPU is up to the driver. You can look at what your driver is doing by using vulkaninfo and checking the memoryHeaps. Usually on iGPUs there's a big chunk available, marked as host-visible and sometimes device-local, but it's often just half of the RAM or another fraction.

[ddpasa]

Layers are not duplicated, but the available memory to an iGPU is up to the driver. You can look at what your driver is doing by using vulkaninfo and checking the memoryHeaps. Usually on iGPUs there's a big chunk available, marked as host-visible and sometimes device-local, but it's often just half of the RAM or another fraction.

thanks @0cc4m ! So this is something I need to fix on the vulkan driver side? I'll follow up on that to see if it is possible. I see talk on various forums that the iGPU memory might be limited to half of the total availabla RAM.

[0cc4m]

Best check the output of vulkaninfo on your system, but yeah, that would have to be "fixed" on the driver side. I don't know if there's a hardware reason for it.