Function Call Response Is Not Inside ChatResponse Object

[oguzhantortop]

Hi,
I am using openAI and registered a function for setting SQL parameters. When I ask a related question such as : "draw a chart of top selling products" I can clearly see that, the function is being called. And I can see that expected parameters are also being set. But inside chatresponse object I can't see the values that I return from method.

import java.util.Arrays;
import java.util.function.Function;

public class GraphService implements Function<GraphService.Request, String> {

	public record Request(String[] measure, String[] aggregation) {}

	public String apply(Request request) {
		System.out.println("function call");
		StringBuffer sb = new StringBuffer();
		sb.append("graph(");
		sb.append(Arrays.toString(request.measure));
		sb.append(Arrays.toString(request.aggregation));
		sb.append(")");
		return sb.toString();
	}
}

Controller Class:

        GetMapping("/ai/generate3")
	public Map generate3(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
		
		UserMessage userMessage = new UserMessage(message);
		
		Prompt prompt = new Prompt(List.of(userMessage),OpenAiChatOptions
				.builder().withFunctionCallbacks(List.of(new FunctionCallbackWrapper<>(
						"DrawGraph", // name
						"Draws graph by defining measure fields and aggregation fields."
						+ " Possible fields are: AMOUNT which has id of 1, CREATE_BY which has id of 2,EMAIL "
						+ " which has id of 3,LOCATION  which has id of 4,ORDER_DATE  which has id of 5,PRODUCT "
						+ " which has id of 6,YEAR  which has id of 7,LOCATION  which has id of 8."
						+ " I want you to pass the field ids only.", // function description
						new GraphService()))).build());
		ChatResponse resp = chatClient.call(prompt);
		return Map.of("response", resp.getResult().getOutput().getContent());
	}

[oguzhantortop]

I closed the issue after seeing that my return value appended to the response. However despite calling my function callback each time sometimes i am seeing that I can't see my callbacks response each time. Also I want to know if it is possible to retrieve just function return value? So that I can return just my function out put to the client.

Thanks a lot!

example prompt for my service:

localhost:8080/ai/generate3?message=please draw a graph of top selling products aggregate by country

[oguzhantortop]

I have debugged the outgoing connections a little bit and realized that Spring AI makes two subsequent requests.
First for gathering required method call.
Second user prompt with gathered method call and params.

Is it intentionally? Cause the second output really messes my desired output. Is there a way to close this behavior?

[markpollack]

Hi. I'm not sure which two requests you are referring to. Can you please refer to the places in the code base that you are talking about and we can look into it. Thanks.

[oguzhantortop]

Hi,
I am using openAI and registered a function for setting SQL parameters. When I ask a related question such as : "draw a chart of top selling products" I can clearly see that, the function is being called. And I can see that expected parameters are also being set. But inside chatresponse object I can't see the values that I return from method.

import java.util.Arrays;
import java.util.function.Function;

public class GraphService implements Function<GraphService.Request, String> {

	public record Request(String[] measure, String[] aggregation) {}

	public String apply(Request request) {
		System.out.println("function call");
		StringBuffer sb = new StringBuffer();
		sb.append("graph(");
		sb.append(Arrays.toString(request.measure));
		sb.append(Arrays.toString(request.aggregation));
		sb.append(")");
		return sb.toString();
	}
}

Controller Class:

        	@GetMapping("/ai/generate3")
	public Map generate3(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
		
		UserMessage userMessage = new UserMessage(message);
		
		
		var promptOptions = OpenAiChatOptions.builder()
				.withFunctionCallbacks(List.of(FunctionCallbackWrapper.builder(new GraphService())
					.withName("DrawGraph")
					.withDescription("Draws graph by defining measure fields and aggregation fields.Possible fields are:  AMOUNT aka. \\\"Miktar\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d7f0006\\\" has type of measure, CREATE_BY aka. \\\"Ekleyen Kullanıcı\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d850007\\\"  has type of aggregation, EMAIL aka. \\\"Eposta\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d99000a\\\"  has type of aggregation, LOCATION aka. \\\"Şehir\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d940009\\\"  has type of aggregation,  ORDER_DATE aka. \\\"Sipariş Tarihi\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d8e0008\\\"  has type of date ,PRODUCT aka. \\\"Ürün\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d720004\\\"  has type of aggregation,YEAR aka. \\\"Yıl\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d7a0005\\\"  has type of aggregation, LOCATION aka. \\\"Konum\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb155a30000d\\\"  has type of aggregation. Don't return field names, I want id values of fields!.")
					.withResponseConverter((response) -> "" + response)
					.build()))
				.build();
		
		/*OpenAiChatOptions
				.builder().withFunctionCallbacks(List.of(new FunctionCallbackWrapper<>(
						"DrawGraph", // name
						"Draws graph by defining measure fields and aggregation fields.Possible fields are:  AMOUNT aka. \"Miktar\" which has id of \"ac120029-8dea-11a3-818d-eb119d7f0006\" has type of measure, CREATE_BY aka. \"Ekleyen Kullanıcı\" which has id of \"ac120029-8dea-11a3-818d-eb119d850007\"  has type of aggregation, EMAIL aka. \"Eposta\" which has id of \"ac120029-8dea-11a3-818d-eb119d99000a\"  has type of aggregation, LOCATION aka. \"Şehir\" which has id of \"ac120029-8dea-11a3-818d-eb119d940009\"  has type of aggregation,  ORDER_DATE aka. \"Sipariş Tarihi\" which has id of \"ac120029-8dea-11a3-818d-eb119d8e0008\"  has type of date ,PRODUCT aka. \"Ürün\" which has id of \"ac120029-8dea-11a3-818d-eb119d720004\"  has type of aggregation,YEAR aka. \"Yıl\" which has id of \"ac120029-8dea-11a3-818d-eb119d7a0005\"  has type of aggregation, LOCATION aka. \"Konum\" which has id of \"ac120029-8dea-11a3-818d-eb155a30000d\"  has type of aggregation. Don't return field names, I want id values of fields!.", // function description
						new GraphService()))).build()*/
		
		Prompt prompt = new Prompt(List.of(userMessage),promptOptions);
		
		
		ChatResponse resp = chatClient.call(prompt);
		return Map.of("response", resp.getResult().getOutput().getContent());
	}
		
		

I am using above code to make a function call with version 0.8.1-SNAPSHOT. And when I intercept the connection I can see that framework doing 2 subsequent calls to OpenAI.
First Request:

{
  "messages": [
    {
      "content": "Ürün ve şehir bazında satış miktarı grafiğini çizer misin?",
      "role": "user"
    }
  ],
  "model": "gpt-4-turbo-preview",
  "frequency_penalty": 0,
  "n": 1,
  "stream": false,
  "temperature": 0.8,
  "tools": [
    {
      "type": "function",
      "function": {
        "description": "Draws graph by defining measure fields and aggregation fields.Possible fields are:  AMOUNT aka. \\\"Miktar\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d7f0006\\\" has type of measure, CREATE_BY aka. \\\"Ekleyen Kullanıcı\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d850007\\\"  has type of aggregation, EMAIL aka. \\\"Eposta\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d99000a\\\"  has type of aggregation, LOCATION aka. \\\"Şehir\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d940009\\\"  has type of aggregation,  ORDER_DATE aka. \\\"Sipariş Tarihi\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d8e0008\\\"  has type of date ,PRODUCT aka. \\\"Ürün\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d720004\\\"  has type of aggregation,YEAR aka. \\\"Yıl\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d7a0005\\\"  has type of aggregation, LOCATION aka. \\\"Konum\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb155a30000d\\\"  has type of aggregation. Don't return field names, I want id values of fields!.",
        "name": "DrawGraph",
        "parameters": {
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "type": "object",
          "properties": {
            "aggregation": {
              "type": "array",
              "items": {
                "type": "string"
              }
            },
            "measure": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          }
        }
      }
    }
  ]
}

Second request which goes immediate after the first one:

{
  "messages": [
    {
      "content": "Ürün ve şehir bazında satış miktarı grafiğini çizer misin?",
      "role": "user"
    },
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "call_Z3Gbv5GrwSUgck2l3EEGjA8R",
          "type": "function",
          "function": {
            "name": "DrawGraph",
            "arguments": "{\"aggregation\":[\"ac120029-8dea-11a3-818d-eb119d940009\",\"ac120029-8dea-11a3-818d-eb119d720004\"],\"measure\":[\"ac120029-8dea-11a3-818d-eb119d7f0006\"]}"
          }
        }
      ]
    },
    {
      "content": "graph([ac120029-8dea-11a3-818d-eb119d7f0006],[ac120029-8dea-11a3-818d-eb119d940009, ac120029-8dea-11a3-818d-eb119d720004])",
      "role": "tool",
      "tool_call_id": "call_Z3Gbv5GrwSUgck2l3EEGjA8R"
    }
  ],
  "model": "gpt-4-turbo-preview",
  "frequency_penalty": 0,
  "n": 1,
  "stream": false,
  "temperature": 0.8,
  "tools": [
    {
      "type": "function",
      "function": {
        "description": "Draws graph by defining measure fields and aggregation fields.Possible fields are:  AMOUNT aka. \\\"Miktar\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d7f0006\\\" has type of measure, CREATE_BY aka. \\\"Ekleyen Kullanıcı\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d850007\\\"  has type of aggregation, EMAIL aka. \\\"Eposta\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d99000a\\\"  has type of aggregation, LOCATION aka. \\\"Şehir\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d940009\\\"  has type of aggregation,  ORDER_DATE aka. \\\"Sipariş Tarihi\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d8e0008\\\"  has type of date ,PRODUCT aka. \\\"Ürün\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d720004\\\"  has type of aggregation,YEAR aka. \\\"Yıl\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb119d7a0005\\\"  has type of aggregation, LOCATION aka. \\\"Konum\\\" which has id of \\\"ac120029-8dea-11a3-818d-eb155a30000d\\\"  has type of aggregation. Don't return field names, I want id values of fields!.",
        "name": "DrawGraph",
        "parameters": {
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "type": "object",
          "properties": {
            "aggregation": {
              "type": "array",
              "items": {
                "type": "string"
              }
            },
            "measure": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          }
        }
      }
    }
  ]
}

When I compare the first and second request i can see this difference:

And when I debug the code I can see that after function call Spring AI calls OpenAI again:

At this point this might be intentionally done, but sometimes OpenAI looses my functions output. So in cases where I just want to return output to user I am loosing my function response. If this is intentionally done I should be able to access my function response in ChatResponse class. Or is it possible to change the behavior to "not to call open ai again after calling my call back function" which can be set optionally ?

Thanks a lot!

[qinfengge]

I think this very useful. Sometimes I just need to call a function to access a simple interface, and I don't always need the model to summarize or process the return value for me. Perhaps we need a switch to decide whether the return content of each function should be processed by the model.

[samzhu]

Hello there

I completely agree with your point of view. Being able to access the raw response from the function call, so that we can decide how to further process it, rather than receiving a response that has already been processed by the LLM.

In the current implementation of Spring AI, the framework sends the function output back to the model for further processing, which, as you mentioned, may not always be ideal. Having the ability to retrieve the original function response would give us better control over the conversation flow.

[tzolov]

@oguzhantortop I think that 5017749 might help addressing this issue?
Please, check the OpenAiChatModelProxyToolCallsIT.java for examples how to run the function calling entirely on the client side.
Let me know if you have further questions.

[tzolov]

@oguzhantortop , @samzhu , @qinfengge I believe that 5017749 provides all the flexibilities you are looking for?
Now you can configure the Spring AI to proxy the function calls to the client instead of dispatch and response to them automatically (the default behavior).
The OpenAiChatModelProxyToolCallsIT.java shows how you can invoke manually the desired functions in side your clients and use the ToolCallHelper to respond to the model. For example:

spring-ai/models/spring-ai-openai/src/test/java/org/springframework/ai/openai/chat/OpenAiChatModelProxyToolCallsIT.java

Line 269 in 110a520

void functionCall2() throws JsonMappingException, JsonProcessingException {

Are those changes flexible enough for your use cases?

[ilayaperumalg]

Potentially, the short solution is to add the "ToolContext" into ChatGenerationMetadata's metadata. This can be related to this #2049