Supporting varying number of pods in radix-tree cache data structure.

[gangmuk]

🚀 Feature Description and Motivation

Currently, the radix tree cache does not support varying number of GPUs (pods). The corresponding tree nodes in RadixTree should be updated accordingly in real time with varying number of pods.

WARNING. I am worried about potential race condition between reflecting running pods in radixtree data structure and router makes decision based on outdated radixtree. It is more than performance. let's say a previously running pod is being terminated and the radix tree has been updated late. And the routing policy returns terminating pod as best prefix matching pod. I wonder how it is done in the current hash based prefix aware routing.

Use Case

Prefix and load aware routing

Proposed Solution

No response