Metrics endpoint for shortfin apps (llm / sd)

[renxida]

Like this: https://docs.vllm.ai/en/latest/serving/metrics.html (suggested by @ScottTodd )

[renxida]

At a minimum I think we should have metrics to figure out whether shortfin is waiting for IREE or if we're not saturating the device.

[ScottTodd]

I see two partially overlapping projects:

1. For developers working on technologies in the IREE/shortfin/ROCm/shark-ai/etc. tech stack, we want to be able to qualitatively and quantitatively analyze serving performance then test how changes impact performance.
   • We could consider serving performance to be an extension of inference performance, just with multiple requests in flight concurrently, but there are new metrics to consider that are unique to serving. For example, how long on average each request spends queued up, how long on average each request takes to be completed, how many requests are running at a given time, how many requests hit some fast path (e.g. parameters and program instructions already loaded into memory vs needing to be fetched from disk), etc.
   • Our existing profiling tools such as Tracy may be useful at this increased scale, either as-is today, with some additional instrumentation markers added (e.g. tagging each request), or even with deeper changes to the Tracy UI.
2. For production users, they will want to know how efficiently they are using the available hardware and what configuration changes they can make to achieve better results.
   • We shouldn't expect these users to run Tracy on serving snapshots that are short in duration (< 10 minutes), analyze trace files, or make source code changes to the compiler and runtime. They'll want something that can provide a summary and steer them towards actions they are capable of taking.

Some reference pages for inspiration:

• https://docs.vllm.ai/en/latest/serving/metrics.html
• https://pytorch.org/serve/metrics_api.html
• https://docs.bentoml.com/en/latest/build-with-bentoml/observability/metrics.html

Having each server expose a metrics endpoint will allow for monitor dashboards for long running servers and give us some way to extract metrics for automated tests too.