How can I use a fine-tuned ControlNet?

[Artemis1111]

My goal is to 1. Utilize a fine-tuned transformer while 2. Applying ControlNet.

However, existing ControlNet models are optimized for base models, so applying them together results in significant degradation of both quality and performance.

To address this, I aimed to first fine-tune the transformer and then adapt ControlNet to work effectively with the fine-tuned transformer.

I referenced x-labs AI's methods and code to fine-tune ControlNet for my fine-tuned transformer. The resulting files include the following safetensors:

These files do not include a config.json. As a result, I attempted to use FluxControlNetModel.from_single_file, but unfortunately, FluxControlNetModel does not support from_single_file.

To the Diffusers team, would you consider adding support for from_single_file to FluxControlNetModel?

To others, I’m curious if there are alternative ways to resolve this issue.

Thank you for reading! 😊