Public JWT keys become stale after some period of time

[KevinDCarlson]

Both prod and dev backends woke up grumpy this morning:

Feb 12 13:34:38 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:34:38.609675Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:34:46 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:34:46.018035Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:34:46 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:34:46.201151Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:34:46 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:34:46.389597Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:38:08 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:38:08.342531Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:38:08 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:38:08.713513Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:38:08 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:38:08.927380Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:38:09 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:38:09.158481Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:38:12 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:38:12.600675Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid
Feb 12 13:38:12 catcolab raqdsm32a825bs341wb3fb0sz0j76kqy-backend.sh[1831]: 2025-02-12T18:38:12.600658Z ERROR backend: Authentication error: Failed to verify token: NotfoundMatchKid

User experience is that it looks like you're logged in, but the permissions button flashes into the non-permissions button, so you can't create permissioned docs and your profile does not show your username. I assume that in this state your non-permissioned docs also aren't getting owned by you.

We were able to temporarily fix by running catcolab-restart in the servers pending investigation of some kind of cookie issue.

[epatters]

I have a guess as to what's going on. Apparently (and I don't know how JWT works), the public JWT keys that Google provides are periodically refreshed. firebase-auth, the third-party crate for Firebase auth that we use, spawns a task to periodically grab the latest keys. So my idea is that this task is either aborting or the request to get the public keys is failing, which causes token verification to start failing eventually.

To diagnose further, we'll need to increase the log-level of the service, then wait and see what happens?