Call method from OpenAiChatModel Class does not work for media generated from PNG images.

[flplvr-magie]

Hello!

When I use the "call" method from OpenAiChatModel Class (injected via the default constructor to @Autowired), passing as parameters a Prompt object instatied with a Usermessage with:

• String textContent; and
• Media media (generated from a PNG image)

don't get a result.

Like this:

@Autowired
private OpenAiChatModel chatModel;

...

public String myFunction(String textContent, File file) {
Media media = new Media(MimeTypeUtils.IMAGE_PNG, new FileSystemResource(file));
Prompt prompt = new Prompt(new UserMessage(textContent, media));
chatModel.call(prompt);
....
}

When I make this call, I don't get a result with a PNG image, but with a JPG image, it works well.

I also tested with the "stream" method from the same Class and it works well with all types of images.

Additionally:
If I manually instantiate the OpenAiChatModel by setting the RestClient.Builder of the OpenAiApi as: RestClient.builder(new RestTemplate()), then everything works well for the images.

[dev-jonghoonpark]

@flplvr-magie Hi, in my case, it works well. Could you please provide a more specific example?

[flplvr-magie]

@dev-jonghoonpark
I think I found what is causing the problem. It is not directly related to the image type, JPG or PNG.

However, it is related to the size. I noticed that both PNG and JPG images larger than ~ 700KB have an issue with the call() method of the OpenAiChatModel class. I send the request and do not receive a response, but the stream() method apparently works well.

I imagine this might be some restriction intentionally or unintentionally included in the RestClient.Builder used in the OpenAiApi class, but not in the WebClient.Builder.

So when I set the RestClient.Builder as RestClient.builder(new RestTemplate()), perhaps I bypass this image size restriction.