Outlier Detection Success Rate Ejection Threshold Setting

[MengyingLiDD]

Description:
We are currently using the failure percentage mode for externally generated errors, and the success rate mode for locally generated errors in our service mesh's default outlier detection policy to identify bad pods. For externally generated errors, we use 99% for the error rate threshold as the default policy, and it works very well. However, for locally generated errors, we have observed a significant number of false positives, where pods with success rates as high as 99% are being ejected.
To mitigate this, we increased the standard deviation factor from the default value of 1.9 to 5. While this reduced the frequency of false positives, the issue persists. Increasing the standard deviation factor further is not a viable option for us, as it could lead to more false negatives.
In our environment, the issue occurs more frequently under the following conditions, which results in very low standard deviations:

• Services with a large number of hosts/pods.
• High RPS per pod (e.g., ~1k RPS per pod).
• High availability requirements (e.g., four 9s SLA).

We believe the current standard deviation-based approach could be more useful for high-RPS and highly available microservices by introducing an additional threshold. Specifically, outlier ejection should only occur if a pod's success rate falls below a static user-configurable threshold. Specifically in our case, we’d like to only eject a host when the success rate is lower than 90% AND it is X standard deviation lower than the average.
We'd like to get the community's thoughts on this. If we agree this is a reasonable ask, we'd be happy to contribute to upstream. Thanks!

[optional Relevant Links:]
Related issues: #18752

Any extra documentation required to understand the issue.

[adisuissa]

cc @alyssawilk who may know who this request should be routed to

[alyssawilk]

cc @cpakulski is the active dev with this most history here (though it's been a while and may be swapped out =P) and @Pawan-Bishnoi has touched it most recently

[cpakulski]

@MengyingLiDD Would you mind giving example of a config which includes proposed new parameter?

Thanks.

[MengyingLiDD]

@cpakulski Of course. I've also updated the description a bit. I want to have an extra parameter like success_rate_minimal_ejection_threshold (name subject to discussion).

Here's an example:

    outlier_detection:
      interval: 60s
      base_ejection_time: 120s
      max_ejection_percent: 10
      max_ejection_time: 300s
      max_ejection_time_jitter: 1s
      split_external_local_origin_errors: True
      enforcing_consecutive_gateway_failure: 0
      enforcing_consecutive_local_origin_failure: 0
      enforcing_consecutive_5xx: 0
      enforcing_success_rate: 0
      enforcing_local_origin_success_rate: 100
      enforcing_failure_percentage: 100
      enforcing_failure_percentage_local_origin: 0
      success_rate_minimum_hosts: 6
      success_rate_request_volume: 10
      success_rate_stdev_factor: 5000
      success_rate_minimal_ejection_threshold: 90 # this is the new parameter I would like to have where it only eject in success rate mode when the threshold is below 90.
      failure_percentage_minimum_hosts: 6
      failure_percentage_request_volume: 10
      failure_percentage_threshold: 99
      consecutive_gateway_failure: 1000000
      consecutive_local_origin_failure: 100000
      consecutive_5xx: 100000
      

[MengyingLiDD]

@cpakulski any update?

[cpakulski]

@MengyingLiDD Thanks for providing the example. I need to go to your original description to understand the need for this parameter.