add galaxy job radar visualization tool stats collection role and scripts #1395
Conversation
Yes, that needs to be done. I can ask and see if someone would be interested in contributing to it. |
| ] | ||
| - timeout = "240s" | ||
| - data_format = "influx" | ||
| - interval = "1h" |
There was a problem hiding this comment.
Those queries are very expensive arn't they? Do we really need to run them this often?
There was a problem hiding this comment.
No, The SQL queries I have added are optimized as much as possible (I think when I timed their executions, they completed in less than a few seconds (Yes, a few seconds might be an expensive operation)). The interval depends on how often we want the data in the Galaxy Job Radar to be updated for visualization and how "real" the "real-time" data should be for that.
There was a problem hiding this comment.
Stefan and me have the suspicion that one of our many queries for Grafana is causing our DB problems.
I'm just afraid of we add more of those. Please merge this in and observe how it is going, but my maybe we should also:
- increase the interval
- maybe we need to add new indices somewhere
- maybe we need to fill larger influx-tables and then do the selection in the influx side.
There was a problem hiding this comment.
I like dumping the data into the influx tables and then processing it there. Let me see the possibilities.
Issue: https://github.com/usegalaxy-eu/issues/issues/478