Fast(-er) way to let whisper.cpp & llama.cpp take turns in CUDA GPU usage?

[RhinoDevel]

On a Linux computer with a CUDA GPU, I'd like whisper.cpp and llama.cpp taking
turns in using the GPU for inference.

A kind of clumsy way to do this (this is what I am doing right now):

1. whisper.cpp: Reserve CUDA device (by reloading the model from file on disk).
2. whisper.cpp: Do STT inference.
3. whisper.cpp: Release CUDA device.
4. llama.cpp: Reserve CUDA device (by reloading the model from file on disk).
5. llama.cpp: Restore state (context, etc., if available).
6. llama.cpp: Do inference.
7. llama.cpp: Backup state to RAM (for next inference).
8. llama.cpp: Release CUDA device.
   => Back to step (1).

It would be nice, if the reloading of the model from disk and complete
reinitialization of llama.cpp (step 4) could be performance improved, e.g. by
copying the data from the CUDA device to RAM (as backup) before releasing the
GPU (step 8) and restoring the data back into the CUDA device from RAM instead
of doing a full reinitialization from file on disk (step 4).

After investigating the source code I am under the impression that there is no
straight forward solution for this and you would probably need to modify
the llama.cpp source code and maybe use the same ggml instance for whisper.cpp
and llama.cpp, but it can be that I missed something(?).

Thanks in advance.

[ExtReMLapin]

I think he's looking for something like pytorch where we can transfer data (including models) manualy across devices.

This is what we used at the office for our model memory manager, models are moved to CPU when it's possible and there is enough free ram, and moved back to GPU when it's required to be used.

[ggerganov]

Have you checked the talk-llama example in whisper.cpp?

[RhinoDevel]

Thanks for the hint, yes, I knew about talk-llama. The code looks as if whisper.cpp and llama.cpp can both use the GPU, I have not tested that, though. But even if this works, neither whisper.cpp nor llama.cpp could use the maximum possible layers the GPU supports - at least not in this example, if I understand correctly(?).