final corrections pdf

corrections on refs and spelling

03-DAC_to_rank_experts.Rmd

@@ -4,7 +4,7 @@
 \thispagestyle{empty}
 
 \blfootnote{This chapter is published as Veen, D., Stoel, D., Schalken, N., Mulder, K., \& van de Schoot, R. (2018). Using the Data Agreement Criterion to Rank Experts' Beliefs. \textit{Entropy}, 20(8). Doi: 10.3390/e20080592 \\ 
-\indent With correction published as Veen, D., Stoel, D., Schalken, N., Mulder, K., \& van de Schoot, R. (2019). Correction: Veen, D.; Stoel, D.; Schalken, N.; Mulder, K.; Van de Schoot, R. Using the Data Agreement Criterion to Rank Experts’ Beliefs. Entropy 2018, 20, 592. \textit{Entropy}, 21(3), 307. \\
+\indent With correction published as Veen, D., Stoel, D., Schalken, N., Mulder, K., \& van de Schoot, R. (2019). Correction: Veen, D.; Stoel, D.; Schalken, N.; Mulder, K.; van de Schoot, R. Using the Data Agreement Criterion to Rank Experts’ Beliefs. Entropy 2018, 20, 592. \textit{Entropy}, 21(3), 307. \\
 \indent DV, DS and RvdS. mainly contributed to the initial study design. DV and NS programmed and verified the statistical analyses for the $DAC_d$. DV programmed the elicitation software. The elicitations have been facilitated by DV and DS. DV wrote and revised the paper with feedback and input from DS, NS, RvdS. and KM. An anonymous reviewer suggested the comparison between the DAC and the BF and this conceptual comparison was carried out by KM and DV. RvdS supervised the project.}
 
 
@@ -29,7 +29,7 @@ Once expert knowledge is elicited and data is collected, it is desirable to find
 
 Prior-data (dis)agreement measures are currently used to evaluate, for example, the suitability of certain priors in the estimation of models or to uncover potential suitability problems with design, prior or both. Examples can be found in, for instance [@fu_adaptive_2017; @fu_bayesian_2015; @walley_advantages_2015]. We found no previous use of prior-data (dis)agreement measures to rank experts. However, when we have two experts, some very interesting questions can already be answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which expert's representation is more valid or useful? Can we reach convergence between expert judgement and data? Therefore, the main contribution of this paper will be to provide an application of prior-data (dis)agreement measures to expert ranking.
 
-Other measures that answer similar questions on different theoretical basis can be found. For instance, Cohen’s kappa [@cohen_coefficient_1960]] could be used to assess inter-rater agreement, intraclass correlations [@koch_intraclass_2004] could be used to asses rater reliability [@shrout_intraclass_1979] and Brier scores [@brier_verification_1950] can be used to asses discrepancy between experts’ estimated probability and actual outcomes [@barons_eliciting_2018]. These measures, however, do not account for the uncertainty of the experts over their provided estimates.
+Other measures that answer similar questions on different theoretical basis can be found. For instance, Cohen’s kappa [@cohen_coefficient_1960] could be used to assess inter-rater agreement, intraclass correlations [@koch_intraclass_2004] could be used to asses rater reliability [@shrout_intraclass_1979] and Brier scores [@brier_verification_1950] can be used to asses discrepancy between experts’ estimated probability and actual outcomes [@barons_eliciting_2018]. These measures, however, do not account for the uncertainty of the experts over their provided estimates.
 
 An alternative approach could be to use Bayes factors (BF) [@kass_bayes_1995] based on marginal likelihoods. One could imagine different experts' beliefs to be competing versions of models. When the differing views are expressed in different prior distributions, we could assess the likelihood of the data averaged across the prior distribution, which is what a marginal likelihood is [@liu_bayes_2008]. This likelihood depends on the model structure, such as parametrization, or the set of probability distributions that is used as the model [@wasserman_bayesian_2000]. If we keep this set of probability distributions, the model, equal across the experts and the same data is used, the marginal likelihood provides an indication of which experts' prior belief gives most probability to the data, and who is thus ranked most trustworthy. The BF, being a ratio of marginal likelihoods, could then provide us odds in favor of one expert's beliefs over another's. This approach warrants further comparison, which is given in Section \@ref(DACvsBF).
 
@@ -70,7 +70,7 @@ The DAC, as mentioned before, is a ratio of two KL divergences. A KL divergence
 It is necessary to expand on the choice for the benchmark prior $\pi^J(\theta)$ and in relation to this the posterior distribution $\pi^J(\theta|\textbf{y})$. Bousquet [@bousquet_diagnostics_2008] follows the reasoning Bernardo provided in discussion with Irony and Singpurwalla [@irony_noninformative_1997] to see $\pi^J(\theta|\textbf{y})$ as a non-subjective posterior that is representative of the situation that one's prior knowledge was dominated by the data. In other words, $\pi^J(\theta|\textbf{y})$ can be considered as a fictitious expert that is perfectly in agreement with the data, having no prior knowledge and being informed about the observations. $\pi^J(\theta|\textbf{y})$ can be considered to be a reference posterior conveying the inferential content of the data [@bernardo_reference_1979].
 
 If $\pi^J(\theta|\textbf{y})$ is taken to be a reference posterior, this would implicitly support the choice of $\pi^J(\theta)$ such that it is a reference prior as originally developed by @bernardo_reference_1979, further developed by Berger and Bernardo, e.g., [@berger_estimating_1989], described in @bernardo_bayesian_1994 and more formally worked out in @berger_formal_2009. Reference priors are not the only possible choice for priors that convey in some sense minimal information or affect the information of the likelihood as weakly as possible [@gelman_prior_2017]. An extensive overview can be found in @kass_selection_1996 and some notable options are
-Jeffrey's priors [@jeffreys_invariant_1946; @jeffreys_theory_1961] and maximum entropy priors [@jaynes_rationale_1982] to which the reference priors reduce in specific cases [@bernardo_bayesian_1994].
+Jeffreys priors [@jeffreys_invariant_1946; @jeffreys_theory_1961] and maximum entropy priors [@jaynes_rationale_1982] to which the reference priors reduce in specific cases [@bernardo_bayesian_1994].
 
 One notable problem for using reference priors as a choice for $\pi^J(\theta)$ is that they often are improper priors [@yang_catalog_1996] and KL divergences and thus the DAC are not well defined when one of the distributions is improper. An adaptation of the DAC could be used, however a choice for a more convenient prior that is proper and leads to a posterior $\pi^J(\theta|\textbf{y})$ closely resembling a reference posterior seems reasonable [@bousquet_diagnostics_2008].
 

06-Expert_Elicitation_LGM.Rmd

@@ -3,7 +3,7 @@
 \chaptermark{ELICITATION IN THE SOCIAL SCIENCES}
 \thispagestyle{empty}
 
-\blfootnote{This chapter is submitted for publication as Veen, D., Egberts, M. R., Van Loey, N. E. E. \& Van de Schoot, R. \textit{Expert Elicitation in the Social Sciences: The case of Posttraumatic Stress Symptoms Development in Children with Burn Injuries} \\
+\blfootnote{This chapter is submitted for publication as Veen, D., Egberts, M. R., van Loey, N. E. E. \& van de Schoot, R. \textit{Expert Elicitation in the Social Sciences: The case of Posttraumatic Stress Symptoms Development in Children with Burn Injuries} \\
 \indent All authors have been involved in the design of the study and the elicitation procedure. DV programmed the elicitation software. ME arranged the elicitation meetings with the experts. DV and ME conducted all elicitation procedures together. DV wrote and revised the paper with contributions and feedback provided by ME, NvL and RvdS.}
 
 

92-CV.Rmd

@@ -25,7 +25,7 @@ As of December 2019, Duco holds a post-doctoral position at Utrecht University t
   - Meershoek, A.J.A., de Vries, E.E., **Veen, D.**, den Ruijter, H.M., & de Borst, G.J. (2019).
 Meta‐analysis of the outcomes of treatment of internal carotid artery near occlusion. *British Journal of Surgery*, 106(6), 665-671.
 
-  - **Veen, D.**, Stoel, D., Schalken, N., Mulder, K., & van de Schoot, R. (2019). Correction: Veen, D.; Stoel, D.; Schalken, N.; Mulder, K.; Van de Schoot, R. Using the Data Agreement Criterion to Rank Experts’ Beliefs. Entropy 2018, 20, 592. *Entropy*, 21(3), 307.
+  - **Veen, D.**, Stoel, D., Schalken, N., Mulder, K., & van de Schoot, R. (2019). Correction: Veen, D.; Stoel, D.; Schalken, N.; Mulder, K.; van de Schoot, R. Using the Data Agreement Criterion to Rank Experts’ Beliefs. Entropy 2018, 20, 592. *Entropy*, 21(3), 307.
 
   - **Veen, D.**, Stoel, D., Schalken, N., Mulder, K., & van de Schoot, R. (2018). Using the Data Agreement Criterion to Rank Experts' Beliefs. *Entropy*, 20(8). Doi: 10.3390/e20080592
 
@@ -37,7 +37,7 @@ Meta‐analysis of the outcomes of treatment of internal carotid artery near occ
 
   - **Veen, D.**, & Egberts, M. R. (2020). The importance of collaboration in Bayesian analyses with small samples. In R. Van de Schoot & M. Miočević (Eds.), *Small sample size solutions: A guide for applied researchers and practitioners.* Routledge.
 
-  - Van de Schoot, R., **Veen, D.**, Smeets, L., Winter, S. D., & Depaoli, S. (2020). A tutorial on using the WAMBS-checklist to avoid the misuse of Bayesian statistics. In R. Van de Schoot & M. Miočević (Eds.), *Small sample size solutions: A guide for applied researchers and practitioners.* Routledge.
+  - van de Schoot, R., **Veen, D.**, Smeets, L., Winter, S. D., & Depaoli, S. (2020). A tutorial on using the WAMBS-checklist to avoid the misuse of Bayesian statistics. In R. Van de Schoot & M. Miočević (Eds.), *Small sample size solutions: A guide for applied researchers and practitioners.* Routledge.
 
 
 ## Technical Reports {-}

Ch06.bib

@@ -31,7 +31,7 @@ @article{veen_proposal_2017
 	title = {Proposal for a {Five}-{Step} {Method} to {Elicit} {Expert} {Judgement}},
 	volume = {8},
 	journal = {Frontiers in psychology},
-	author = {Veen, Duco and Stoel, Diederick and Zondervan-Zwijnenburg, Mariëlle and Van de Schoot, Rens},
+	author = {Veen, Duco and Stoel, Diederick and Zondervan-Zwijnenburg, Mariëlle and {van de Schoot}, Rens},
 	year = {2017},
 	pages = {2110}
 }
@@ -62,7 +62,7 @@ @article{veen_using_2018
 	volume = {20},
 	number = {8},
 	journal = {Entropy},
-	author = {Veen, Duco and Stoel, Diederick and Schalken, Naomi and Mulder, Kees and Van de Schoot, Rens},
+	author = {Veen, Duco and Stoel, Diederick and Schalken, Naomi and Mulder, Kees and {van de Schoot}, Rens},
 	year = {2018},
 	pages = {592}
 }
@@ -81,7 +81,7 @@ @article{zondervan-zwijnenburg_application_2017
 	title = {Application and evaluation of an expert judgment elicitation procedure for correlations},
 	volume = {8},
 	journal = {Frontiers in psychology},
-	author = {Zondervan-Zwijnenburg, Mariëlle and Van de Schoot-Hubeek, Wenneke and Lek, Kimberley and Hoijtink, Herbert and Van de Schoot, Rens},
+	author = {Zondervan-Zwijnenburg, Mariëlle and {van de Schoot}-Hubeek, Wenneke and Lek, Kimberley and Hoijtink, Herbert and {van de Schoot}, Rens},
 	year = {2017},
 	pages = {90}
 }
@@ -187,7 +187,7 @@ @incollection{van_de_schoot_tutorial_2020
 	title = {A tutorial on using the {WAMBS}-checklist to avoid the misuse {Bayesian} {Statistics}},
 	booktitle = {Small sample size solutions: {A} guide for applied researchers and practitioners},
 	publisher = {Routledge},
-	author = {Van de Schoot, Rens and Veen, Duco and Smeets, Laurent and Winter, Sonja and Depaoli, Sarah},
+	author = {{van de Schoot}, Rens and Veen, Duco and Smeets, Laurent and Winter, Sonja and Depaoli, Sarah},
 	year = {2020}
 }
 
@@ -366,7 +366,7 @@ @inproceedings{lek_development_2018
 	volume = {3},
 	isbn = {2504-284X},
 	publisher = {Frontiers},
-	author = {Lek, Kimberley and Van de Schoot, Rens},
+	author = {Lek, Kimberley and {van de Schoot}, Rens},
 	year = {2018},
 	pages = {82}
 }
@@ -417,7 +417,7 @@ @article{van_de_schoot_bayesian_2018
 	issn = {0027-3171},
 	number = {2},
 	journal = {Multivariate behavioral research},
-	author = {Van de Schoot, Rens and Sijbrandij, Marit and Depaoli, Sarah and Winter, Sonja D and Olff, Miranda and Van Loey, Nancy E},
+	author = {{van de Schoot}, Rens and Sijbrandij, Marit and Depaoli, Sarah and Winter, Sonja D and Olff, Miranda and {van Loey}, Nancy E},
 	year = {2018},
 	pages = {267--291}
 }
@@ -450,7 +450,7 @@ @article{van_de_schoot_systematic_2017
 	issn = {1433890747},
 	number = {2},
 	journal = {Psychological Methods},
-	author = {Van de Schoot, Rens and Winter, Sonja D and Ryan, Oisín and Zondervan-Zwijnenburg, M and Depaoli, Sarah},
+	author = {{van de Schoot}, Rens and Winter, Sonja D and Ryan, Oisín and Zondervan-Zwijnenburg, M and Depaoli, Sarah},
 	year = {2017},
 	pages = {217}
 }
@@ -509,7 +509,7 @@ @article{buist_developmental_2002
 	issn = {0047-2891},
 	number = {3},
 	journal = {Journal of youth and adolescence},
-	author = {Buist, Kirsten L and Dekovic, Maja and Meeus, Wim and Van Aken, Marcel AG},
+	author = {Buist, Kirsten L and Dekovic, Maja and Meeus, Wim and {van Aken}, Marcel AG},
 	year = {2002},
 	pages = {167--176}
 }
@@ -542,7 +542,7 @@ @article{egberts_mother_2018
 	issn = {0305-4179},
 	number = {4},
 	journal = {Burns},
-	author = {Egberts, Marthe R and Van de Schoot, Rens and Geenen, Rinie and Van Loey, Nancy EE},
+	author = {Egberts, Marthe R and {van de Schoot}, Rens and Geenen, Rinie and {van Loey}, Nancy EE},
 	year = {2018},
 	pages = {861--869}
 }
@@ -575,7 +575,7 @@ @article{lek_how_2019
 	volume = {21},
 	number = {5},
 	journal = {Entropy},
-	author = {Lek, Kimberley and Van de Schoot, Rens},
+	author = {Lek, Kimberley and {van de Schoot}, Rens},
 	year = {2019},
 	pages = {446}
 }

IntroDiscussionThesis.bib

@@ -51,7 +51,7 @@ @incollection{van_de_schoot_tutorial_2020
 	title = {A tutorial on using the {WAMBS}-checklist to avoid the misuse {Bayesian} {Statistics}},
 	booktitle = {Small sample size solutions: {A} guide for applied researchers and practitioners},
 	publisher = {Routledge},
-	author = {Van de Schoot, Rens and Veen, Duco and Smeets, Laurent and Winter, Sonja and Depaoli, Sarah},
+	author = {{van de Schoot}, Rens and Veen, Duco and Smeets, Laurent and Winter, Sonja and Depaoli, Sarah},
 	year = {2020}
 }
 
@@ -145,7 +145,7 @@ @book{lindley_understanding_2013
 
 @article{de_finetti_foresight:_1937,
 	title = {Foresight: {Its} {Logical} {Laws}, {Its} {Subjective} {Sources}},
-	author = {de Finetti, B},
+	author = {{de Finetti}, B},
 	year = {1937}
 }
 
@@ -192,7 +192,7 @@ @misc{cambridge_english_dictionary_expert_2019
 	abstract = {expert definition: 1. a person with a high level of knowledge or skill relating to a particular subject or activity…. Learn more.},
 	language = {en},
 	urldate = {2019-11-05},
-	author = {Cambridge English Dictionary},
+	author = {{Cambridge English Dictionary}},
 	year = {2019},
 	file = {Snapshot:C\:\\Users\\Administrator\\Zotero\\storage\\KTG4BMI6\\expert.html:text/html}
 }

Reader on statistics.bib

@@ -99,7 +99,7 @@ @article{veen_proposal_2017
 	title = {Proposal for a {Five}-{Step} {Method} to {Elicit} {Expert} {Judgement}},
 	volume = {8},
 	journal = {Frontiers in psychology},
-	author = {Veen, Duco and Stoel, Diederick and Zondervan-Zwijnenburg, Mariëlle and Van de Schoot, Rens},
+	author = {Veen, Duco and Stoel, Diederick and Zondervan-Zwijnenburg, Mariëlle and {van de Schoot}, Rens},
 	year = {2017},
 	pages = {2110}
 }
@@ -118,7 +118,7 @@ @article{zondervan-zwijnenburg_application_2017
 	title = {Application and evaluation of an expert judgment elicitation procedure for correlations},
 	volume = {8},
 	journal = {Frontiers in psychology},
-	author = {Zondervan-Zwijnenburg, Mariëlle and Van de Schoot-Hubeek, Wenneke and Lek, Kimberley and Hoijtink, Herbert and Van de Schoot, Rens},
+	author = {Zondervan-Zwijnenburg, Mariëlle and {van de Schoot}-Hubeek, Wenneke and Lek, Kimberley and Hoijtink, Herbert and {van de Schoot}, Rens},
 	year = {2017},
 	pages = {90}
 }
@@ -468,7 +468,7 @@ @article{raftery_approximate_1996
 @incollection{van_de_schoot_dealing_2018,
 	title = {Dealing with imperfect elicitation results},
 	booktitle = {Expert judgement in risk and decision analysis},
-	author = {Van de Schoot, Rens and Griffioen, Elian and Winter, Sonja},
+	author = {{van de Schoot}, Rens and Griffioen, Elian and Winter, Sonja},
 	editor = {Bedford, T and French, Simon and Hanea, A.M. and Nane, G.F.},
 	year = {2018},
 	note = {To be published in Springer’s International Series in Operations Research and Management Science},
@@ -503,7 +503,7 @@ @article{gelman_prior_2017
 
 @article{egberts_update_2018,
 	title = {Update this title},
-	author = {Egberts, Marthe and Veen, Duco and Van de Schoot, Rens and Van Loey, Nancy},
+	author = {Egberts, Marthe and Veen, Duco and {van de Schoot}, Rens and {van Loey}, Nancy},
 	year = {2018}
 }
 
@@ -562,7 +562,7 @@ @article{van_baar_reliability_2006
 	issn = {0305-4179},
 	number = {3},
 	journal = {Burns},
-	author = {Van Baar, ME and Essink-Bot, Marie-Louise and Oen, IMMH and Dokter, Jan and Boxma, Han and Hinson, MI and Van Loey, NEE and Faber, AW and van Beeck, Eduard F},
+	author = {Van Baar, ME and Essink-Bot, Marie-Louise and Oen, IMMH and Dokter, Jan and Boxma, Han and Hinson, MI and {van Loey}, NEE and Faber, AW and van Beeck, Eduard F},
 	year = {2006},
 	pages = {357--365}
 }
@@ -659,7 +659,7 @@ @misc{savage_modern_2017
 @article{veen_effects_2018,
 	title = {Effects {Coding} as {Unbiased} {Alternative} to {Scale} {Scores}},
 	journal = {Manuscript under review},
-	author = {Veen, Duco and Little, Todd D. and Van de Schoot, Rens},
+	author = {Veen, Duco and Little, Todd D. and {van de Schoot}, Rens},
 	year = {2018}
 }
 
@@ -677,7 +677,7 @@ @article{veen_using_2018
 	volume = {20},
 	number = {8},
 	journal = {Entropy},
-	author = {Veen, Duco and Stoel, Diederick and Schalken, Naomi and Mulder, Kees and Van de Schoot, Rens},
+	author = {Veen, Duco and Stoel, Diederick and Schalken, Naomi and Mulder, Kees and {van de Schoot}, Rens},
 	year = {2018},
 	pages = {592}
 }
@@ -695,7 +695,7 @@ @book{dirac_principles_1947
 	address = {Oxford},
 	title = {The principles of quantum mechanics},
 	isbn = {0-19-852011-5},
-	publisher = {CLARENDON PRESS},
+	publisher = {Clarendon Press},
 	author = {Dirac, Paul Adrien Maurice},
 	year = {1947}
 }
@@ -886,7 +886,7 @@ @article{zondervan-zwijnenburg_application_2017-1
 	title = {Application and evaluation of an expert judgment elicitation procedure for correlations},
 	volume = {8},
 	journal = {Frontiers in psychology},
-	author = {Zondervan-Zwijnenburg, M and Van de Schoot-Hubeek, Wenneke and Lek, Kimberley and Hoijtink, Herbert and Van de Schoot, Rens},
+	author = {Zondervan-Zwijnenburg, M and {van de Schoot}-Hubeek, Wenneke and Lek, Kimberley and Hoijtink, Herbert and {van de Schoot}, Rens},
 	year = {2017},
 	pages = {90}
 }
@@ -973,7 +973,7 @@ @book{de_finetti_theory_1974
 	title = {Theory of {Probability}},
 	volume = {1 and 2},
 	publisher = {Wiley},
-	author = {de Finetti, B},
+	author = {{de Finetti}, B},
 	year = {1974}
 }
 
@@ -1170,7 +1170,7 @@ @article{van_de_schoot_bayesian_2018
 	volume = {53},
 	number = {2},
 	journal = {Multivariate behavioral research},
-	author = {Van de Schoot, Rens and Sijbrandij, M and Depaoli, S and Winter, S D and Olff, M and Van Loey, N E},
+	author = {{van de Schoot}, Rens and Sijbrandij, M and Depaoli, S and Winter, S D and Olff, M and {van Loey}, N E},
 	year = {2018},
 	pages = {267--291}
 }
@@ -1190,7 +1190,7 @@ @article{zondervan-zwijnenburg_where_2017
 	volume = {14},
 	doi = {10.1080/15427609.2017.1370966},
 	journal = {Res. Hum. Dev.},
-	author = {Zondervan-Zwijnenburg, M and Peeters, M and Depaoli, S and Van de Schoot, Rens},
+	author = {Zondervan-Zwijnenburg, M and Peeters, M and Depaoli, S and {van de Schoot}, Rens},
 	year = {2017},
 	pages = {305--320}
 }