Mel-RoFormer (Kar) model?

[empz]

x-minus.pro has a model they call Mel-RoFormer (Kar) which does a very good job at keeping background vocals. Do you know if that model is available somewhere and could be added to audio-separator? If not, what's the best model available in this library that keeps background vocals with the instrumental part?

[empz]

MVSEP also seems to have it.

But I cannot find it anywhere (I'm not really sure where are these models published, if published at all...). I think it's the best one for karaoke nowadays (keep backing vocals).

Happy to spend time researching this, but I'm a total noob in this.

Any hints on where this model might have been published?

[empz]

Found it!
https://huggingface.co/jarredou/aufr33-viperx-karaoke-bsroformer-model/tree/main

Could this just be plugged into audio-separator or would it require quite some effort? Happy to put on the effort if I could get some guidance from you.

I'm going to see if simply using the .ckpt and the .yaml config from the link is good enough.

[empz]

Yeah, I was just taking a look at that. Shouldn't be better to copy that file over to this repo so that it doesn't depend on that? And also allows for adding other models.

Are Mel-RoFormer and BS-RoFormer the same thing or not the same thing, but similar?

[empz]

I'm asking because you have a bs_roformer.py and a mel_band_roformer.py. This model seems to be a version of bs_roformer, so I'm assuming that's the file we want to use for inferencing?

[beveradb]

It's complicated - by pulling in the file and configs from UVR, we get to use new models which are added to UVR (with existing architectures) without any additional work

I reckon the ideal solution would be a combination of models and configs stored somewhere we have a bit more control over plus pulling in anything new from UVR promptly, but I haven't given it tons of thought yet (I started out just trying to get things to work) 😅

For now the easiest solution would probably my be to just add a secondary location to pull supported models from (as part of this repo) and merge those with the UVR list.

As for Mel vs BS, they're similar but different architectures - Mel-Band Roformers were introduced in this paper, as a development building on the work by the same folks who previously worked on the Band-Split (BS) Roformer algorithm:
https://arxiv.org/abs/2310.01809

I would assume this mel-karaoke model should be inferenced using the logic in the Mel-RoFormer code. Only testing it will tell though ☺️

[empz]

Got it.

Whatever you think it's best. I'll try do something on my own, but you probably are going to be able to do it way faster than me.
There it goes a little sponsorship for motivation 😝

[beveradb]

FYI I'm finally working on this now 🙏

[beveradb]

I've just implemented and tested support for these new models:

• Mel-Roformer-Karaoke-Aufr33-Viperx: mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt

• Mel-Roformer-Crowd-Aufr33-Viperx: mel_band_roformer_crowd_aufr33_viperx_sdr_8.7144.ckpt

• Mel-Roformer-Denoise-Aufr33: denoise_mel_band_roformer_aufr33_sdr_27.9959.ckpt

• Mel-Roformer-Denoise-Aufr33-Aggr: denoise_mel_band_roformer_aufr33_aggr_sdr_27.9768.ckpt

All are available and working in the latest version of audio-separator, version 0.21.0 🎉

Hopefully they're useful for you 😄