Doubt regarding GPU utilization and batch inference

[BakingBrains]

1. Can BentoML auto allocate the GPU resources based on the incoming requests in production? any code references.
2. Also, how can I do a batch inference for sequence of images? Like I want to send 8 images at once, any code references.

Any suggestion would be helpful.

Thanks and Regards.

[ssheng]

1. For autoscaling, please take a look if BentoCloud is what you need.
2. BentoML currently does not support sequence of images. We are looking to add image sequence IO descriptor in the future. Currently, could you convert the images to numpy and use the ndarray IO descriptor?

[visaals]

If you're using pytorch, you can also convert the images to pytorch.Tensor and BentoML supports batching those.