On training textual inversion

[ZhihuaLiuEd]

Hi all,
I am new to diffusers and GenAI. Just want to ask a specific question on Textual-Inversion (TI).
(Please correct me if I understand something wrong, but what I reckon TI is to update the placeholder string embedding to the specific concept presented in the few-shot images?)

Thus the training for TI is one image each time? Though the batch can be enlarged, but it is still the same image \times batchsize plus prompt template with placeholder? Or it is similar images with similar visual concept?

What I have right now is a collection of different images, with different concepts. Is there a way to train these images together in one batch?

Thanks in advance!

[asomoza]

Hi, AFAIK you can't train multiple concepts with textual inversion, also our scripts are mostly basic so we don't even have the option for training multiple concepts for dreambooth loras as other training repositories do. Probably it would be best to train multiple TIs.

I must admit that I don't really use textual inversion so asking @linoytsaban if there's something to add or if this is possible.

[ZhihuaLiuEd]

Thank you so much for letting me know! My initial thought was like if I want to train a TI with multi images with different concept and different placeholder string in a same batch, I have to divide the loss properly to backpropagate and update the text embedding, but seems very hard to do it.
Anyway many thanks for your reply!

[linoytsaban]

Hey @ZhihuaLiuEd! with our scripts you could technically train more than one concept in a single train run when enabling pivotal tuning/ textual inversion - provided your image dataset comes with a caption column such that each caption contains unique identifiers assigned for the concept/s present in the image. This way you can pass in --token_abstraction a comma separated string containing the identifiers for each concept e.g/ - "TOK1,TOK2". These identifiers will then be converted to different new tokens whose embeddings will be optimized during training.
Generally multiple concept training tends to be quite tricky to nail, personally I haven't explored much multiple concept training with textual inversion. Other things that could have a significant effect on training in that context (and generally too) will be the quality of the dataset, how close these concepts are to the data the base model was trained on, the amount of images per concept, wether all images contain both concepts or they exist separately too etc.
See https://github.com/huggingface/diffusers/blob/main/examples/advanced_diffusion_training/README_flux.md for more examples on pivotal tuning that might be useful here 🤗

[ZhihuaLiuEd]

Hi @linoytsaban ! Thank you so so much for your reply. As I mentioned above, my initial though was try to divide the loss properly for backpropagation for updating the texutal embedding (but seems very hard to do). But I am very appreciate the hint on --token_abstraction and link on pivotal tuning, which is very useful for me to dive deeper into the TI code. Hope you have a nice day!
Best,
Zhihua