Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[clickhouse] Use chrono::UTC for distributed_ddl_queue endpoint #7001

Merged
merged 6 commits into from
Nov 7, 2024

Conversation

karencfv
Copy link
Contributor

@karencfv karencfv commented Nov 6, 2024

Modified the SQL query to set the date/time format to one we can parse.

$ curl http://[::1]:8888/distributed-ddl-queue | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1993  100  1993    0     0   4203      0 --:--:-- --:--:-- --:--:--  4213
[
  {
    "entry": "query-0000000001",
    "entry_version": 5,
    "initiator_host": "ixchel",
    "initiator_port": 22001,
    "cluster": "oximeter_cluster",
    "query": "CREATE DATABASE IF NOT EXISTS db1 UUID '701a3dd3-10f0-4f5d-b5b2-0ad11bcf2b17' ON CLUSTER oximeter_cluster",
    "settings": {
      "load_balancing": "random"
    },
    "query_create_time": "2024-11-01T03:17:08Z",
    "host": "::1",
    "port": 22001,
    "status": "Finished",
    "exception_code": 0,
    "exception_text": "",
    "query_finish_time": "2024-11-01T03:17:08Z",
    "query_duration_ms": 3
  },

@bnaecker
Copy link
Collaborator

bnaecker commented Nov 6, 2024

You can add query-specific settings by tacking them on at the end: https://clickhouse.com/docs/en/sql-reference/statements/select#syntax. So you would do SELECT … SETTINGS date_time_output_format = ‘iso’.

@karencfv
Copy link
Contributor Author

karencfv commented Nov 6, 2024

You can add query-specific settings by tacking them on at the end: https://clickhouse.com/docs/en/sql-reference/statements/select#syntax. So you would do SELECT … SETTINGS date_time_output_format = ‘iso’.

Nice! Thanks @bnaecker, I've made the modifications

@karencfv karencfv changed the title [clickhouse] Use chrono::NaiveDateTime for distributed_ddl_queue endpoint [clickhouse] Use chrono::UTC for distributed_ddl_queue endpoint Nov 6, 2024
Copy link
Collaborator

@bnaecker bnaecker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! One question about parsing, but otherwise good to go. Thanks!

@@ -1007,7 +1008,7 @@ pub struct DistributedDdlQueue {
/// Exception message
pub exception_text: String,
/// Query finish time
pub query_finish_time: String,
pub query_finish_time: DateTime<Utc>,
/// Duration of query execution (in milliseconds)
pub query_duration_ms: String,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is a UInt64, is it worth parsing into a stronger type during deserialization? Something like a Duration, or at least u64?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a string with a number in it :( I initially had it as a u64 and it didn't parse, but let me try with Duration!

Copy link
Collaborator

@bnaecker bnaecker Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this is because we're asking for JSON, and by default ClickHouse quotes 64-bit numbers. That causes serde to think it's a string, because it's written like "1", rather than just 1. You can set the output_format_json_quote_64bit_integers setting to ask it not to that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like Duration doesn't accept strings either :(

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my earlier comment. You can ask CH not to quote big integers, so that serde can parse it just fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ugh sorry, GitHub hid that comment from me, I see it now thanks!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, I figured :) Just didn't want you to lose track of it. I think that will work, but it's honestly up to you if you want to apply it. I don't have context for what this all will be used for!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That worked, thanks! It's definitely better with strong types. Wasn't able to get Duration to parse but at least it's u64 now. We're going to use this endpoint (and others) for long term monitoring and debugging of the ClickHouse cluster.

@karencfv karencfv enabled auto-merge (squash) November 6, 2024 23:45
@karencfv karencfv disabled auto-merge November 6, 2024 23:48
@karencfv karencfv enabled auto-merge (squash) November 7, 2024 00:30
@karencfv karencfv merged commit 5b66a15 into oxidecomputer:main Nov 7, 2024
18 checks passed
@karencfv karencfv deleted the fix-ddl-endpoint branch November 7, 2024 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants