Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration Issue: Spring AI Embedding with VLLM OpenAI API Compatibility #2042

Open
icyclv opened this issue Jan 6, 2025 · 0 comments
Open

Comments

@icyclv
Copy link

icyclv commented Jan 6, 2025

Bug description
Unable to use embedding functionality when integrating Spring AI with VLLM using OpenAI API format. The issue appears to be related to HTTP protocol version compatibility.

Environment
Spring AI: 1.0.0-M4
JDK: 22
VLLM: 0.6.6.post1

Steps to reproduce

  1. Set up VLLM server
vllm serve Alibaba-NLP/gte-Qwen2-1.5B-instruct --task embed --tokenizer Alibaba-NLP/gte-Qwen2-1.5B-instruct
  1. Configure Spring AI to use VLLM endpoint
  2. Attempt to use embedding functionality
@RestController
@RequestMapping("/ai")
public class EmbeddingController {
    private final EmbeddingModel embeddingModel;

    @Autowired
    public EmbeddingController(EmbeddingModel embeddingModel) {
        this.embeddingModel = embeddingModel;
    }

    @GetMapping("/embedding")
    public Map embed(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
        EmbeddingResponse embeddingResponse = this.embeddingModel.embedForResponse(List.of(message));
        return Map.of("embedding", embeddingResponse);
    }

}

Expected behavior
Return embedding results.

Current Behavior
When sending requests to VLLM using Spring AI's OpenAI API implementation:

  1. VLLM logs:
WARNING: Unsupported upgrade request.
INFO: - "POST /v1/embeddings HTTP/1.1" 400 Bad Request
  1. Spring AI throws:
org.springframework.ai.retry.NonTransientAiException: 400 - {
    "object": "error",
    "message": "[{'type': 'missing', 'loc': ('body',), 'msg': 'Field required', 'input': None}]",
    "type": "BadRequestError",
    "param": null,
    "code": 400
}

The request being sent contains:

{
    "method": "POST",
    "headers": {
        "Connection": "Upgrade, HTTP2-Settings",
        "Host": "127.0.0.1:8080",
        "Http2-Settings": "AAEAAEAAAAIAAAABAAMAAABkAAQBAAAAAAUAAEAA",
        "Transfer-Encoding": "chunked",
        "Upgrade": "h2c",
        "User-Agent": "Java-http-client/22.0.1",
        "Authorization": "Bearer test",
        "Content-Type": "application/json"
    }
}

Current Workaround
I've implemented a temporary solution by forcing HTTP/1.1:

@Component
public class RestClientCustomizer {
    @Bean
    @Primary
    public RestClient.Builder customRestClientBuilder() {
        HttpClient httpClient = HttpClient.newBuilder()
                .version(HttpClient.Version.HTTP_1_1)
                .build();
        return RestClient.builder()
                .requestFactory(new JdkClientHttpRequestFactory(httpClient));
    }
}

🤔Perhaps a better solution would be for VLLM to provide support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant