Triton Example with AI Gateway

[FernandoDorado]

Hello,

We are currently evaluating open-source deployment tools and are particularly interested in the integration of TensorRT-LLM with Triton. We understand from the documentation that this can be achieved using a Custom Python Server, but the implementation details are not entirely clear.

Could we possibly collaborate on preparing a detailed demo or example showcasing this integration? Additionally, it might be beneficial to explore the creation of a step-by-step guide, which could serve as a valuable resource for both our team and the broader community.

[subnetmarco]

@fffonion I believe that this warrants support for a new LLM type in our system (TensorRT-LLM)?

[fffonion]

@subnetmarco I think TensorRT is another ML backend like pytorch. Kong doesn't talk to ML backend directly but
only care about the transport protocol.
@FernandoDorado Something I found online https://github.com/npuichigo/openai_trtllm, if you run this and setup an
OpenAI upstream in kong using ai-proxy or ai-proxy-advanced plugin then it wil work.

[fffonion]

It does seem the Triton inference server exposes a HTTP endpoint https://docs.nvidia.com/deeplearning/triton-inference-server/archives/tensorrt_inference_server_090_beta/tensorrt-inference-server-guide/docs/http_grpc_api.html but it seems it's not intended for consuming directly but mostly for low-level distributed computing.

[FernandoDorado]

Thanks for the details. I've already tested the proxy and works. However, I prefer not to depend on open-source technologies. I really appreciatte to have this combination on tensorrt-llm and triton inference server integration in Kong, similar to Ollama examples.