From 87cd275266168fdb19db3709a086d74e5dd52cbf Mon Sep 17 00:00:00 2001 From: Manfred Moser Date: Wed, 29 Jan 2025 13:24:00 -0800 Subject: [PATCH] Add docs for Loki connector --- docs/src/main/sphinx/connector.md | 1 + docs/src/main/sphinx/connector/loki.md | 176 +++++++++++++++++++++++ docs/src/main/sphinx/static/img/loki.png | Bin 0 -> 9934 bytes 3 files changed, 177 insertions(+) create mode 100644 docs/src/main/sphinx/connector/loki.md create mode 100644 docs/src/main/sphinx/static/img/loki.png diff --git a/docs/src/main/sphinx/connector.md b/docs/src/main/sphinx/connector.md index c7e7a3b0402e..9c6259eb632f 100644 --- a/docs/src/main/sphinx/connector.md +++ b/docs/src/main/sphinx/connector.md @@ -26,6 +26,7 @@ Ignite JMX Kafka Kudu +Loki MariaDB Memory MongoDB diff --git a/docs/src/main/sphinx/connector/loki.md b/docs/src/main/sphinx/connector/loki.md new file mode 100644 index 000000000000..09c3847883be --- /dev/null +++ b/docs/src/main/sphinx/connector/loki.md @@ -0,0 +1,176 @@ +# Loki connector + +```{raw} html + +``` + +The Loki connector allows querying log data stored in [Grafana +Loki](https://grafana.com/oss/loki/). This document describes how to configure a +catalog with the Loki connector to run SQL queries against Loki. + +## Requirements + +To connect to Loki, you need: + +- Loki 3.1.0 or higher. +- Network access from the Trino coordinator and workers to Loki. Port 3100 is + the default port. + +## Configuration + +The connector can query log data in Loki. Create a catalog properties file that +specifies the Loki connector by setting the `connector.name` to `loki`. + +For example, to access a database as the `example` catalog, create the file +`etc/catalog/example.properties`. + +```text +connector.name=loki +loki.uri=http://loki.example.com:3100 +``` + +The following table contains a list of all available configuration properties. + +:::{list-table} Loki configuration properties +:widths: 40, 60 +:header-rows: 1 + +* - Property name + - Description +* - `loki.uri` + - The URI endpoint for the Loki server that Trino cluster nodes use to access + the Loki APIs. +* - `loki.query-timeout` + - [Duration](prop-type-duration) that Trino waits for a result from Loki + before the specific query request times out. Defaults to `10s`. A minimum of + `1s` is required. +::: + +(loki-type-mapping)= +## Type mapping + +Because Trino and Loki each support types that the other does not, this +connector [modifies some types](type-mapping-overview) when reading data. + +### Loki to Trino type mapping + +Each log line in Loki is split up by the connector into three columns: + +* `timestamp` +* `values` +* `labels` + +These are separately mapped to Trino types: + +:::{list-table} Loki log entry to Trino type mapping +:widths: 40, 60 +:header-rows: 1 + +* - Loki type + - Trino type +* - `timestamp` + - `TIMESTAMP WITH TIME ZONE` +* - `values` for [log queries](https://grafana.com/docs/loki/latest/query/log_queries/) + - `VARCHAR` +* - `values` for [metrics queries](https://grafana.com/docs/loki/latest/query/metric_queries/) + - `DOUBLE` +* - `labels` + - `MAP` with label names and values as `VARCHAR` key value pairs +::: + +No other types are supported. + +(loki-sql-support)= +## SQL support + +The Loki connector does not provide access to any schema or tables. Instead you +must use the [query_range](loki-query-range) table function to return a table +representation of the desired log data. Use the data in the returned table like +any other table in a SQL query, including use of functions, joins, and other SQL +functionality. + +(lok-table-functions)= +### Table functions + +The connector provides the following [table function](/functions/table) to +access Loki. + +(loki-query-range)= +### `query_range(varchar, timestamp, timestamp) -> table` + +The `query_range` function allows you to query the log data in Loki with the +following parameters: + +* The first parameter is a `varchar` string that uses valid + [LogQL](https://grafana.com/docs/loki/latest/query/) query. +* The second parameter is a `timestamp` formatted data and time representing the + start date and time of the log data range to query. +* The third parameter is a `timestamp` formatted data and time representing the + end date and time of the log data range to query. + +The table function is available in the `system` schema of the catalog using the +Loki connector, and returns a table with the columns `timestamp`, `value`, and +`labels` described in the [](loki-type-mapping) section. + +The following query invokes the `query_range` table function in the `example` +catalog. It uses the LogQL query string `{origin="CA"}` to retrieve all log data +with the value `CA` for the `origin` label on the log entries. The timestamp +parameters set a range of all log entries from the first of January 2025. + +```sql +SELECT timestamp, value +FROM + TABLE( + example.system.query_range( + '{origin="CA"}', + TIMESTAMP '2025-01-01 00:00:00', + TIMESTAMP '2025-01-02 00:00:00' + ) + ) +; +``` + +The query only returns the timestamp and value for each log entry, and omits the +label data in the `labels` column. The value is a `varchar` string since the +LoqQL query is a log query. + +## Examples + +The following examples show case combinations of +[LogQL](https://grafana.com/docs/loki/latest/query/) queries passed through the +table function with SQL accessing the data in the returned table. + +The following query uses a metrics query and therefore returns a `count` column +with double values, limiting the result data to the latest 100 values. + +```sql +SELECT value AS count +FROM + TABLE( + example.system.query_range( + 'count_over_time({test="metrics_query"}[5m])', + TIMESTAMP '2025-01-01 00:00:00', + TIMESTAMP '2025-01-02 00:00:00' + ) + ) +ORDER BY timestamp DESC +LIMIT 100; +``` + +The following query accesses the value of the label named `province` and returns +it as separate column. + +```sql +SELECT + timestamp, + value, + labels['province'] AS province +FROM + TABLE( + example.system.query_range( + '{origin="CA"}', + TIMESTAMP '2025-01-01 00:00:00', + TIMESTAMP '2025-01-02 00:00:00' + ) + ) +; diff --git a/docs/src/main/sphinx/static/img/loki.png b/docs/src/main/sphinx/static/img/loki.png new file mode 100644 index 0000000000000000000000000000000000000000..37692fd1093e9fc6b3b67b16a5863ade08bc28a6 GIT binary patch literal 9934 zcmeHrXH-*LxAqPoy>}2K22eVoLujE%la3%w2_zKhBy^AGBYbTf*>(WtkQv(1% zr?01N4*vImUo<5-_}hLxk_PBBYTUU@Tg)=?g|?siO=9uSo$%rbAVWYnY1GO?UN56-D#k@u zJ*SQ*+)+79_Ps@>+rvypuB#*5djp6FT9p6`qEoS*?C8^>S;(j)|89)b(BK~Y3W{u( zJXT}Rhxh^a^eSp;xMa8)P7LEYX%t_1 zo<7{g!CD~7RW=s!V%sbL0Euze(lXW8()!aM(2M+tG*!KJQ`T4$ixM3XdMidVO7nu% zB3McTwN5$Xl~^m@^OyE1jJVU;Idt0^K7Jf$C1zKP~35lu9?Is zsWROivd1(1jDe9GQ)=TNg&eMQ5nR8N@$q68nsF=0tE2F;84(DFHdhQB3Ko6jEexTkD4W6&|?#v zhbru%kttNm#~%Zgm5`Nyi|Yit2TH@#sG%zUXcvUJ_PIYGz&ll#8xH4-kdzDx3X%wt zk?`?%m6TFaQj&yAOG-Q+-vK zw8U@l;}%m-6wbxR-%Ax{gmQ+OSp3V@(%lPVfkPd!CZ!;wD5WSbt0*la4VRHq{Fjjx z#vcnR?g&&0E+O-~2kneF2STDi<+yvHTrrZq-mb?(j^csDn4{z2MN;rT!O{1J@*!x=#6e-rk}RYHF0BZY=a>oYuimqf8wclV#(_(j&AS0KR;xbvUQw-dj{A#j*mL6 zCDJX$Ktvx(T&%_Lj@EOIdadzM4SMNkHE5BSsb3E?Y6+}xPpcE_P2nvI_`#nglm?qo zJ(x=i-!ms{_8VU+3Ik5Cf4?m>mA_uFRj@UMrSeH}`04wpMRUto^Jg+p=JVvmH;oq} zx89pyrLt$4jTTS`C@Huj<`E7RFxq7d%W!ZnmKr8PnKO#8ITk8cCDhJ9GZm2J-HDPHGzF*&$@*ea+D{5Kj#M(@ODCVlog|%qn>I=3 z&mk+sp3O91r-Dq+GuIak3`|KHEqOf049(oa&Fouup4AX*_+n-WF|R;FT*@~OC;~t1 zS*l5IbXDq}c^(cERYP?u!-WH2!JN%1Xw zYK|a88b++lJx3K&&oj+Aca9fD*HHisL_*T(Z<&?RrRGn{Gd(&)3 z43!xDvNQE ztU|Q`1RJum#J-3I>F8dEH_HvzrDi=?Br7Q9q27|h3os4|(&R95-*E}S3k;e|2vZn+ z1dyoronhzVc;c*$%xlOBx9zJhak6*HBejD*-?><9FU8qpnonWYfSgNcqR`v)M6R6G zh#4JL0nR6ol>^MLc8k00@el8%V@OhoqpAi?xMAUvGlf7NX*lEQ`0<>>;G!>(bZvV7 z8BaQDn!r$jqJnzE`hH<{>tT$L-scD-Dp@0C40CLYbjZU22O|ngJNCnmySHG2^%in` z8en7+na4069G5)(PEj|=BM~At3Ny+VAife;`J$}o%Plb_E84tnw!-ZAP*usiVXvBd zACJOd4sps3t&P|y{gD;8uLb$WlPL!4`4v*B&Rk-q**MbQRbklIKxR8zPfKl=^=oOA zcX~|)$xO-UoE;i1eK3$KABtXViUhEOvW@Qz>EkSwoa>csmy%LgfZ+ULS`Z9$aJ;u!K8_0 zt7o4Y54l5EQe-P1l)YQ>z-FuTFwd1g`aD}KnyK(10il(tuKX}#S1nl5b{o)_Fz8s0 zarG#Xfo9Da8S&=NvRJS^6t?x0-zQXlz{r<$JIpd8igcU>fbp#NqOr~K6Y3iakaXJc z*88?o;V0U)pvGPKr~*Q-`TXwT#&(F1y?eAs%Nb1FZY(7Xz2aWCD; zSj)KLEpIBjn|(HZoi)#v+@j@Ye`+bUkH)j)tJ?Sesg~mN4;0usZ&B!_=Y;mw80-v* z5ChYST_c0O=rd+2I^{s?<6a7*uPk=$1r%RQX)b2oy~*_%dpA6h2uU|(c^}gK(5HfW z*0XX`0PdlaRBgZkxk65Am}_IoT^*)5hX~JME^8B_hD@!DUi$vBMs58Xs012lO}K!a zD(`A6bJ!i-Y;&LP-HF~=5!nSX;K_s!1%=6Yk#`oY9_6s~${ZS4A>kf%)x| zI=(MCI7gv_a#4Lt2NWxBf4V*+afxIuI+%8DZ{TeUaFoo3YXE3fC?cox3r$`TZrb9;1NJEyh~;WZcj8Dw^IK%Eg-cH@t%v zXXk?@a){?`HE<2WlvTwl(voA_H@?TYhUr= z=D`F_HTiJRapUddRz?IZ@8UbU#lLHtn#cLECWr|vilmyS#HT5%IvK|p7B<2ty2=++?|5~N06u|?2%Wr>MJYGT1?xa-pvqcP=uDKw0;KqfI?@>&xH1vQUbP`jR%lLEbD1Y9YPcPT1md zfU9>SHr*>*sp)Ze46?PI@-FkdD9Kpg*O`%utIDxZqbs^+kMhf5u;lWm3h~u{EQonH+vC$GEn`PQzjSI-efvk z;rVR;-Ns1=Q>DzRUzrb*r%80FCdF4eG9^UjL-uMB?)t<}a$p^$s((I{qeL-##qMg& z`NGMkHhKz^A}l%v$$AmnCs-(pn^7aR+DXtZ9puCBSVFK7SP{|_;#J?qX^i=WM=j^q z3wa-Q2(g*wj*SaXu=85b;BGGXR(?V+1zpm zE>l}vY`Cx06vUXA&IInE9ZRscc_3a`Za1L8Hd-ECs6>I)fH2>z3TPZ0t z@P5!umF7x@hkI51`?@e8q)^-FWrrsuecyMAH>bhf1d4Qgl{Ws|D`8H(zGYmQRrbRf z2U*vqXB8vT_ke;MrXJ=>hD$x{ocY6*;etmMM0oVmtVsNc15fOY6fG#!8M*_j`ULw=+WAP&_K5#do@?lzBprQ^(M5c0Ly|ArV=Y{(z zuJa6&n=B3ow+Mx0`J2cegJL8aHT$uGpBIMxhf(*D`UAJ$kp zyqs??lqwB@pH0|EZ(#EF)>fz%>Z4C6K9@gx8zzrT+>LM5v zctY1okF>p@TK|hibAD*(^mEtU3W}Zu{uMzW2pBSc8OrZqP_k0J=UBA8c&obqAauV^ zbSJTXmR^U1?z{Jxib_H2;HJXv6m7oR%&&OX=b|qkQ2Hfb?*~gAZk(-pSHM z^u`-C_xt)@tgBNv2Fb*bEtjlhw@1B-WbvQIFJ2r_?`lXl!paj|;DNq6!SnJiXTELHk6(}U#723SvZ(g(U7Lq@g55Lx^{o1tXPkE%C zbs&p3Az50t5@zl>2?>ANuD=LO<^aA`sfuT3sK~!R-YfzS>Wi5q4gDp&WKXHTWZr9! zR;f4jy8PgYX4s#MMM`^eUlIbWP04h@aXaC5nTw%H@*tMc$HQ~E zHjP)rQZ-^7v4fr%KYO>rTH-PZ93McrDY7kupQ{d&RdlwU@Og&tV+JzT!g~U6M@bWcm2K-$& zT(sllDE5^pdXk)SOep#a>Dh&cJiKo{n8hV-!6a)`3_nVF`NO~bbbjHB606*H(YQ`@?D@uV5@ zvi#YQ8al=KiLgeiFpCoglVK2>^4=0)uu0r-JZElugnZyjdqa6_7{0kLIUh1#{TUHX zJqJ7=-j@$sT#j7{@;3y?M};R`5C`PBP2VjhM-GMA6D!{fa)*blkA@-&Sn~2AG?vm3 zCHawEhZhm4Yx_N%8mPJ%hW9j(k%7)MMcqpd0FYN^x(Gr~x!$gFrJhBy4(Ywd ze96ge6kG>dwP>s_NmC}q-u|#T>m4s)iq`D0mk|^h!584&e_j1tmOOB94G-7&h^!Bq z$=Ei^K}8+v-Z+f%C2ytecxAerv242_z`&MWV`#7q;Jkb!bYmxgTzG4>ZRLw4rQnr| z2YUwhHBPIA((xSV;P2TnxL{ZB>;cOUS0@%vLB96v{L0TfdrqwwA7nsa<^=>q{fae( z7No!;Tzs%^JW|nL*OmF;{(`}!)`nZoDcKY~0S`;oZ>$a)Yy#a}0dsM6KDS>?yq=#= zO?~nBHYqdisqx(>mOp3GWPn|V1Glfu_UJqHO`$)$Eoy72K0Rnta?364EHEClb}wq{1F?KtOos$ZavM*TlsoZ8Oq;my8gmMI z(l45i=luJ5b9&?;Zmf2J%Ixx5ME?(t0uMHNrm<%!q$~S)9=s|+qt}mu&!exr)G$R- zhjINm`H&-9hggyKPo5qnhzWY=fS82^HADPZ$<-nOaqovSZGLo zOoqPUzEJLTGPwWc@H@U6_CRe!0YS)u_vAFGm{@&#s<{q-shzAMO%D~mGf;NP%z|f2 z6WrB&CSg=EcIDsAGi>s?^78u6_=%=TEKkSX9efXKH&72rcUi|7iCm8Fy6wb$H}^Qh zF^$u|&euL}=FYTyru;+V`}9((xS&GfR>yU+5OE~yRoZ%pIR#O+CxmiA{}m14ky%AA zgWBvNLkPaPs4G0AcE3GMrtpIKn(w#JJIQS;7C;V#g23xWJ-LOU8CuE?f(G