train_text_to_image_sdxl training effect is very poor

[wzhiyuan2016]

I use DeepSpeed for training: train_text_to_image_sdxl.py
1.The data volume is 231 pieces
2. deepspeed json

3.Training Script

4.After training, use the training prompt words again，The generated effect is as follows:

May I ask everyone, what is the reason for the poor generation effect？

[a-r-r-o-w]

cc @sayakpaul

[sayakpaul]

This should be a discussion as there are far too many things that are unknown here.

General comments:

• How do the samples look like in the dataset?
• What is the resolution of the samples in the dataset?
• Have you tried overfitting a single image and checked the results?

[wzhiyuan2016]

This should be a discussion as there are far too many things that are unknown here.

General comments:

• How do the samples look like in the dataset?
• What is the resolution of the samples in the dataset?
• Have you tried overfitting a single image and checked the results?

hi

• How do the samples look like in the dataset?
  The data is similar to:
  

• What is the resolution of the samples in the dataset?
  There are 213 pieces of training data, all of which are 1080p images, and the input for training is 1024 * 1024

• Have you tried overfitting a single image and checked the results?
  Train using one image as shown in the following figure:
  

"text": "Construction workers, Scaffolding, Building materials, Safety harness, Safety helmets, Yellow railings, Metal stairs, Concrete floor, Red and white safety tape, Tools and equipment"

The loss of training is as follows:

inference script

generate res:

[wzhiyuan2016]

@sayakpaul

[sayakpaul]

Does it work as expected without DeepSpeed? Have you tried varying the learning rate?

[wzhiyuan2016]

When I train train_text_to_image_stxl. py[Not applicable to Deepspeed], the parameter settings are as follows:[]




The single card only has 24GB, and when setting BF16 for training, it reported CUDA out of memory