(shortfin-apps) Thin HTML Client

[monorimet]

To avoid using python-based UI libraries for demonstrating and deploying shortfin apps, we want to have a HTML-based, lightweight client that achieves the same functionality and user experience or improves upon current demos.

We would like the first iteration to have a simple chatbot interface for use with LLM serving, and a SDXL txt2img interface.

The client doesn't have to be smart -- it should build a JSON request based off UI states, send it to a shortfin server URL, and return an image (in SDXL). The chatbot is simpler, with conversation-style user experience and text only.

[bjacobgordon]

Overview

Will be used to demonstrate and deploy shortfin apps.

Criteria

Interfaces

1. text-to-text (LLM chatbot)
2. text-to-image (SDXL UI)

Requests

• shortfin server for text-to-text, text-to-image

Bottom lines

• Lightweight: final package measured in single-digit MBs
• Fast: network requests are the bulk of the latency
• Minimal dependencies
• easy to access

Future Considerations

• log telemetry
  • for speed
  • for errors
  • for hotspots

Tech Stacks for WebUI

Options

• Others?
• WASM-based:
  • eval:
    • ecosystem: young and sparse
    • package size: super tiny
    • speed: super fast
    • ergonomics: depends on the language
  • options:
    • SwiftWASM + Tokamak: ?
    • Dioxus in Rust: ?
    • Sycamore in Rust: ?
    • Yew in Rust: syntax still feels like raw HTML
    • Leptos in Rust: syntax still feels like raw HTML
• Native:
  • potentially not as "turn-key" as accessing some URL
  • options
    • swift cross ui
• HTML-, CSS-, JS-based
  • Ergonomics: med-low
    • Lots of configuration required for environment and development
    • Lots of dependencies required for things as simple as parsing API responses
    • Not inherently type-safe, requires TS as a dependency
  • Approaches
    • Plain HTML:
      • eval:
        • Super tiny, but:
        • will still require host to serve it to client
        • very tedious to write and debug HTML directly
      • options:
        • ssg with node.js + html, written in TypeScript: ?
        • ssr with node.js + some framework, written in TypeScript: ?
        • ssr with Vapor + Element, written in Swift: ?
    • Frameworks:
      • Easier to work with, but:
        • each has lots of history/baggage
        • each still requires JS/TS, leading to "dependency hell"
      • Options
        • Vue: easy to get going but
          • does not scale as well as Angular
          • is progressive, so not inherently opinionated enough
        • Angular: very scalable and opinionated, but
          • steep learning curve for non web devs
          • package size not as small as other frameworks, requires explicit optimization
          • speed not as fast as other frameworks, requires explicit optimization
        • React: super common and small, but
          • has a lot of cruft because of how far back the reach for support
          • would still be odd to learn for people on our team
• Python-based
  • Gradio:
    • use-case: prototypes, proof-of-concepts
    • package size: ~500MB
    • speed:
      • slow due to size
      • clumsy spinup
    • ergonomics: low
      • Python lacks strong typing and common UI customizations

Chosen

?