From f79cacd00e87e9d87e2c7aa89d9f7b7b7848992e Mon Sep 17 00:00:00 2001 From: qiancao Date: Wed, 30 Oct 2024 12:26:06 -0400 Subject: [PATCH] a few edits to JOSS paper --- paper/.paper.md.swp | Bin 0 -> 28672 bytes paper/paper.md | 8 +++++--- 2 files changed, 5 insertions(+), 3 deletions(-) create mode 100644 paper/.paper.md.swp diff --git a/paper/.paper.md.swp b/paper/.paper.md.swp new file mode 100644 index 0000000000000000000000000000000000000000..815ac14c3b1e6cbdc0a012c7ff73f43fa7075926 GIT binary patch literal 28672 zcmeI5YiuOhRmTf9$u0>YPsk>blDj(=V<+ym+w=BfGBX|z9x|St%6U0rqWx#ymH9{+QycWd#9D;wr)b;;w;D?RTePe0r~__X)Q_kR8X z@0rl=R~vpLU)||Xci8tAYj?WCez%Y`Vn1zk7P5gC8h-cLs2?m0{6P@g|5kg=yN@)z zV=95E1fG`!hH0yE_T}D*rP`uAckJPZ%xhl#`12ZKdh=8QQwdBZFqObm0#gZ0B`}r1 zR02~8{NI*9I(V`7gDl}&-7;3(-`_I!_lw-;i|+659sB;p?(-S<_xr}ae~J5i$^HG? zW8dHBK7YIW`zvGLyN}bqrV^M+U@C#B1f~+0N?vM&Cn~M`=Kv=hv$6``WW;a^c3{{ z&^hQ0(EZRCU+#H-2K^573()(a2)Y5CfDGh8{|Er^=g{v$ABR2y4WJ0}q0`V?pgHJ4 z=<6@@ystq20eu;o`!>&;g zlO%BLn7I*jLw_smhUuQU6#DJh?@^gHmxujE8b*CzzVN&&eUo+qv+0^eHqF3qZ2RqC z-s}cu5Jx*Zm>s7OMJ518B-!}SW95PZeY;=sDn9ZaS zZii{58~Aa5)A(^URc z^Q(SWrmXJ<{&roqj-{Qy7)On5E;a*E$|wm;hcS{)v@0t?1?+MnI=D>Ih;%dJVOK`) zvt)$po)hIj5#@GRj=DoV1G40}umi#{W3sRP#IT)x6SnjbyY(`4Cv11hBh2cB3y-hP z&6|)%{XNxwruN;)FjG2t2tfy{8Z1lC@9*h|l{iec`Pk?nCNzojBx>ak23;3IB3e1{F>uikO$ZTWocjF+N5b<~#XN*9H;rr- zJ#q4Y9e@*qIcfT$L+nZFC)@0(us+Ib)r<|Kb7O&WqY&))-Ju!;cH0-V8sD^Hi86O1 zj*_IpwA{ESB9v`0$umP4GKk|S7USgi!oIa&j^(B`SHmztmTmNS7uA?8_MCmn>=(Nx z7hcbYDr{Y#9T%U(~Y9- zpvi_DGdD3`_RO-Reh?5OZE_x@S8-GJ$2Zsa(hkPOxgxdfj@!%_m2Hr0es;{oDw=4R zV&7FO)H$&G$jELOdYKzxnMxcR103yAGHWC=muFSiqmE?!o6MZILpfn`Ot43Z=vT@W z)AUg?W>wRfYw_`fMX1%1!=MgB?UXLSbp5TMo5=A+>75sI&RM-eNq+rCci0S;QJq1c zCUCXn6W2to%nbU8T-YJ3~PAhoN` zbU}@661K(KTIUnQ34+xyN6u~7N6RkDaT4R>4vmx9yeNMxv#~GD_Y67wvH@FRU(Hd{ z=&)2P`>5TzI;Qt`IJt!@}0p+RSF z8ExjIRJmq6RDc~iVIFOyW@&NhB+;?K=peRbQ}ay4^sz_F=JD!HSxSUo!H!>whi!AA z$pWzeHpcP%R*RL}`ImJR6ZHJByKJ&9;Pb7fU+r{~YCGDISqOe&v~dvio9OMbIzzo) z8cLbSxp)VS%It46`7VQnX@t#Xv&f$cRI8jjed3`aQGO4SQ%Y z=O&5z_c-EGrB*w^*e4cGlt#Q9`~7Y6&Z@cT_l6vf=P}&r)2C~dlV|CwE*QMw?}X;< zI-1}0?~$`wbKz`4bv`X>o`YOJkGMY zFv;B%p>NqT9r83T2+&A-B+fXl20-%vUm{ojG9>vwDYgFjEB^79(5ImvBCmcQ)PvSu z>Uo!-dFX3A^Hulkm*n~BpQ!|<5|~P0DuJm4rV^M+U@C$CbrKNJ9iUR8(Zus4S$CnY z;B;NVb^2h%QNIQ1W$_8ka)B7D;3BJ&z8D<_Y`!1uc7tXc7?a^|fGhYhX!UiUptunZ zG!-;N(gODzhaawjJph16lzJ)H31LnX${T(&jJi=BwkoL4k`X^yMHE!F_;D0SKDF32Wlv9MpFy5hf!llE5WQz(JguMpX7xC%jBt;3}L=JCF zV<$T7&zS$gDG!2&O*=hhBE1gYQZ9-x$o7@Sy1E8fOVZTsD(Z)#v zI$ESX$sjF865Iy$OQZ&KlsHv?N>Au>IY>Ju>;xbyY*}5j6C63Lm7s5Gm7Syl%s~ld zz^0QFTG`7LS#5d8(78Zv(4nj5f&f(lxl--e8xtq6OFu#Ndpd{!c{*&*-?lqwrjyRyq$~T851^gG8mF}F+wAeCXYUMA-Ci{jbv7)i4ry|$6 zz%I5+)A;I z`@Msw_tOp|v0AE&kB$u-ZQ*EkQ1ur&@U?m`=QfR`z*UTBC$ZS^9oduPGP9A7Q;Y!f zoS!SW7$??EYcJQYRz&6qTWgfqLdr3$2AkH3z#w;l;bvw$D~k222khCcWZ0|k_f~2M zX5+xzYQ=tIf7Z#R;sihU^wPn;`}b_EVsCPv5B6E>!Q6piC?MurRPUZ%s_$=9mJVdD z@lr-!gs?>otvcn`qz)+jOOPY`T(;fpvDWsi;nZ@If`D3A(qWjn;;MLC6ndyR%UEh& zL=r@SR(tYI7Wqz4C8^I^#ZmW!u&q_>54&B{*&9T#Msd_CqqAMgid@+R7R~vVx?8p< ztuTfg**PslSz|i6UamLIsxhV7(Ih&YS)Y-DqP?WY)y%BU%;g?ISDK4a_&{_v4wBtp zCayR}U2(LkRNOE1B`gBx&a+rE99XN6JTJ2lG=h4ku1ArxK5u#>pV79QtGe9E*O&)8 zW0=%+b@f&pJhO5cHzK$)W=I@sJ&o#vZj`RntS#as)I~@uwMBbDq|VrNi43}tPa{B8 zJZEy`x7NeWdu6LuHhF3nCECZv6~bTzr3dO#Cnws!i-|k5=8}E$UU0EcRR9j9j#ja1 zkBRL>&BQ!;zCDbj;UVkHdF1>xKcI>w!hg#6oL2m~1>9{IzVRqSR?Xu@aZw8GRDqFW1SwL3qHqs=|hR&O-g~Dyviuvr^ri)p3JCx{*udYZpMH8mdeB)L> z+cZStQ<@?eATGf^lK@l5_xG%2w$4p*I7#(ojO;nJ{&JD@5O^S^p(_~MAvOhIjtu8d9MzXLq47<&RPTK1()WaSJD_I~4 z4z}pXNa_;|N&QrPu{!9t=Sr4Xt7a?5S2e$=mCcW~=?K^$3O$LD<7z;)p{2g%$19J~ zeq{|}VbSd8JH~oqcqefyrrS)i(R!^g(Wj0y!rxRAZQYDGh&o_RXXW9579R?)>N6Gy zU8J2LXwoIt?aE!%MzZs;ci6Gu19TAUX2xwl_=IbNGDpd6G#;j1jEu=e!xMC(>?|f} zA@li0I~^2FD!04<17NepPn5<{W5&ovFHh!8aqxj}drFN{8&qkKrL9@ICp%{oSZUkT zybztT=$5S8-miUCM|h65&;2%c4tj%-_E3U9@%=+ZtvH&a!BYt{QmK_G^w#S_0!XTi`nq=z+ZGF|E62qBwIX8GCw+Hl=xi9!;D7 zW6-?^$#*6H&)(nv+AHX(hklfv{0~4r6p^DJgC2$+g#MXa{hy!@Lj&jrbQY2v{^iii zp!=bJAZP!3=(nKXgnr{ip7$weAG!@aa38${fDoUD{s{U-=qI6%LLY%{Lq2o^dK>f* z^Z@h{=*7?%;Qi;I_d#3GIOvM~IxalFT%iv3cr-im?^c-Bff34|NI-DeG-|D6 zmt&gVxEaN&`?S&Tk%Y}^W|MIy5rB?u=z+^Ji%IBf4$Ri?qUJZt`|rd*tRl z{k9R>l%~#XaC`eOGh7^Dg%rizs~ouHPFPnl;lwZR1gXrHT!zC>S~-S*oVDjZ3-&Zx zoEhZbPqfxH$WnmaWCP%LhSKIwKt+FjVC|ljeZg_5mz98+M`*N8=*`pE9J8(FcXV6= zHr%Zh6 zDsq;p1-Jq$imoV}Qajl1A6TFzRG_7QV4iEu4(j1th5tM!k39x&VxDp35}KmXS=Pqo zyscAYNREcjasV~%B~J*-&sbNX{if>0b+y=Kfk6Hza=yH0S@xjG@m9Q7#EbdX9#5tP zjv)-Dh$R~_o;Qjolt_kXL;jg)a0Z?t?>S_vJwiLL*f%HWi)C`=1pSa=kV$$GSdEr; zf<~tw5?ko<(P3qwP*nGTLpneuVILPKTPd8dSKO|z zdt^{dQD`bs7O64dbm=0M^zO=Cb-)%TrJ0Y_xbQ)(lIp&?crjTe|eTWAfD8|#4s8hs!Y+~ z(HpURrSb)8X{#GjBNYWteE4y7OuLveQ`8ggnJEGU8#Hte&rmKvlwZoQa$x&vDpC&1t1>WS?CF*J0_T55Owv$(^koatyB8e(X_KNcqyDA?Gn;JC%4QFR8x<5GB zuT=&If-zgG*9-;$#~9c}Fd<)@xF%B!ambXrN5pZBKV;3LJz?^Eaj={zD=k1<1RSE% z7sGt7X7}Or(k$K7nMBbv2D6=GVc;tmHDIS!Iow?P^?vz8$xl$?G1>xiDY?+72 zXT=HIR6M^T8zZk%C?u%BR5Rp23rIyCOvP-(%5bw3P>ZQmj%1=41I4&7v8Y>NW z3|LO@HhHaV;to%)uIjE@&l`l||J-{@nRc&VwX#LIabh9rD0)t=={$^VxU5z8Tk^XY zsDOtpM@`8A6pRHd2%5;r=}k7QEfJ&iBEuQVP%{$oP)3vRF&$zVZGVB&^KYcol}F3j zj^VUy;i*abE5LW&!lP$JnoaZ^Pfn8M7`XBrMxs>38cC(~z}A4~$JS4bE!`|uIu!F{ zCMg;oF^-F(X1^Pqi_n|fDBJ??W0X_@;6!bCZ8R!Od5_%)_dPw>PUJGTt1M;%L zyXXn{0q7d^X6Q}O7r++&4U#^AUxq#a4WR@Q4B}}>`Ub8-KM%g}(@+~~K_{RkNZ$Yd z_Xlh*!DsmU*P%~ApMc&A9Y7uEgZw_gk3j?Iqx1)S2)YcdLazgx_%29f^Hq`0^v_fR z|F0xK8nrQ@DUMbe+fH%|z%sYt#6P$pZ3uLpL_EJZD;p@y$lb^9bi4CU?tZc6cLP)e zkqyH4gh9QSy>8!&l>W&_&X+FpwsDIq?P0weZc~OLWUT8;h>x7Vp0i$GfAq%9)pd*O zy3SOIE5$+zK-~kF%@(3*oV?7~3!l}jj(~IIjp!~QjY1jMj~{;d-bRNilLcu75#h`o z?9bK~29?^}tbER{&(RdZ+xo`^-q<8^arI28&=Sy!o77itGwB!W7l-XPY@36uqL7Ab z088?^bQT<_@NGa^+P$i|N#ZmbP~+vAf&I|{n~hKWK?2e@;&VyMMEpdlS*)CKJ3&{7 zsj2@utbiv*b2!{z-k3wsW~#9y1bX;{rh)pp&R<0}Exm%K(wL8=a9_fov3 zDl%+HrmxLq^a1;EO#Ql)qxb6$^rEe0(L(c%tCiybt;b(r624kMiW{~rU!@YA$}7!- zSzd#**Eq30}-=);foncKJY0Y;yj5!46__z6` zs)c0ifHF2TDed9NhTTJz zVL}ZeYfU;VK>n{)t^$+wlYg`K?MOTBe!N8zR9=!WM>K(8*YhPv7l?{HQ#|o}WP&{H zZviZhZDbDD1L%V9YCFWpaI$-u4rqG1;S^f$mH1|JbJI1OO;YCt)m)c8KZ56Sp#pCx^ntqIt!Ghy9@*5EC zcm{@z%22ILu>}7{#kEcJ-7Lkr`-}XXz}oCZ{QRxtnK<3I4Wv)_@qt_{3LUPSREr$f z%q&PB4@#MXNMT9-mI0*SG`quHetJpnGTfy%Nx*Qy9mvOSiVu3Xg$zo*TbdfV?PAU& zMV4#yuZ|2~Fu)AiV(xrn#;JL$Tl<9DVbZZGEFAu!4TX+muN=@Z`pZqYY5Bb&g$PET z6~9+IclI>V;g~uQ*AYBm)GQQ-2XuX{qVyN3>R9hCfzG5KCvv5Qk;(KIL4;9~)KVXQ zs~gzU#ZiUmS7PS)zsLv)!Sy>&a7MzQuo#u+iSVwNN iN8j-g@KT(Ci>o9w+=)U0Bc{u~BSz$>g!0c-@4o=Tok?^6 literal 0 HcmV?d00001 diff --git a/paper/paper.md b/paper/paper.md index 220182b..bc4418e 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -6,6 +6,8 @@ tags: - Artificial Intelligence - Calibration - Probablistic models + - Metric + - Evaluation authors: - name: Kwok Lung Fan orcid: 0000-0002-8246-4751 @@ -42,10 +44,10 @@ bibliography: paper.bib --- # Summary -`calzone` is a Python package for measuring calibration of probabilistic models for classification problems. It provides a set of functions and classes for calibration visualization and calibration metrics computation given a representative dataset with the model's predictions and the true labels. The metrics provided in `calzone` include the following: Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Hosmer-Lemeshow statistic (HL), Integrated Calibration Index (ICI), Spiegelhalter's Z-statistics and Cox's calibration slope/intercept. Some metrics come with variations such as binning scheme and top-class or class-wise. +`calzone` is a Python package for evaluating the calibration of probabilistic outputs of classifier models. It provides a set of functions and classes for visualizing calibration and computing calibration metrics given a representative dataset with the model's predictions and true class labels. The metrics provided in `calzone` include: Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Hosmer-Lemeshow (HL) statistic, Integrated Calibration Index (ICI), Spiegelhalter's Z-statistics and Cox's calibration slope/intercept. The package is designed with versatility in mind. For many of the metrics, users can adjust the binning scheme and toggle between top-class or class-wise calculations. # Statement of need -Classification is one of the most fundamental and important tasks in machine learning. The performance of classification models is often evaluated by a proper scoring rule, such as the cross-entropy or mean square error. Examination of the distinguishing power (resolution), such as AUC or Se/Sp are also used to evaluate the model performance. However, the reliability or calibration performance of the model is often overlooked. +Classification is one of the most fundamental tasks in machine learning. Classification models are often evaluated by a proper scoring rule, such as the cross-entropy or mean square error. Examination of the distinguishing power (resolution), such as AUC or Se/Sp are also used to evaluate the model performance. However, the reliability or calibration performance of the model is often overlooked. @Brocker_decompose has shown that the proper scoring rule can be decomposed into the resolution and reliability. That means even if the model has high resolution (high AUC), it may not be a reliable or calibrated model. In many high-risk machine learning applications, such as medical diagnosis, the reliability of the model is of paramount importance. @@ -257,4 +259,4 @@ The authors acknowledge the Research Participation Program at the Center for Dev # Conflicts of interest The authors declare no conflicts of interest. -# References \ No newline at end of file +# References