Academia.eduAcademia.edu

Figure 3 – uploaded by Lynne Stokes

See full PDFdownloadDownload figure
We present a network approach to study judge agreement based on pairwise com- parisons of wines. In the network graph, the wines to be ranked are the vertices, and the judges’ preferences are the edge flows connecting the vertices. Figure 2 provides an example of the network involving six wines. When judges all agree with one another, there will be no inconsistent triples in the network. The more inconsistent triples there are, the more serious is the lack of agreement among judges. So we can use the proportion of inconsistent triples among all triples as the test statistic (denoted as pj.) to measure judge agreement. The pairwise ranking matrix when there is no agreement between judges (denoted as V-andom) can be constructed by randomly assigning a 0 and 1 to each v; (i</), the upper diagonal elements in V;andom, and setting vj; = 1 — vj. The empirical null distri- bution of the proportion p,,. of inconsistencies is then obtained as the percentage of inconsistencies over all random choices of preferences. The p-value is the proportion of Pine under the empirical null distribution that is less than the observed pjn-. In the context of this example, the p-value is 0.047, which indicates that the judge agree- ment based on the triple-wise comparison is significant.

Figure 2 We present a network approach to study judge agreement based on pairwise com- parisons of wines. In the network graph, the wines to be ranked are the vertices, and the judges’ preferences are the edge flows connecting the vertices. Figure 2 provides an example of the network involving six wines. When judges all agree with one another, there will be no inconsistent triples in the network. The more inconsistent triples there are, the more serious is the lack of agreement among judges. So we can use the proportion of inconsistent triples among all triples as the test statistic (denoted as pj.) to measure judge agreement. The pairwise ranking matrix when there is no agreement between judges (denoted as V-andom) can be constructed by randomly assigning a 0 and 1 to each v; (i</), the upper diagonal elements in V;andom, and setting vj; = 1 — vj. The empirical null distri- bution of the proportion p,,. of inconsistencies is then obtained as the percentage of inconsistencies over all random choices of preferences. The p-value is the proportion of Pine under the empirical null distribution that is less than the observed pjn-. In the context of this example, the p-value is 0.047, which indicates that the judge agree- ment based on the triple-wise comparison is significant.

Related Figures (16)

Example of Wine-Rating System
Example of Wine-Rating System
Scores for 2 Judges on 12 Wines Table 2
Scores for 2 Judges on 12 Wines Table 2
Rank Data for 2 Judges and 12 Wines Let u; and v; denote the ranks given to wine i by judges | and 2, respectively, and let d;=u;—v;, i=1, ...,n. Then, rather than using Equation (1) applied to the ranks, Spearman’s rank correlation can be computed more easily as
Rank Data for 2 Judges and 12 Wines Let u; and v; denote the ranks given to wine i by judges | and 2, respectively, and let d;=u;—v;, i=1, ...,n. Then, rather than using Equation (1) applied to the ranks, Spearman’s rank correlation can be computed more easily as
Table 3 shows the ranks of the data in Table 2, where ties are shown as the mean of the tied ranks. Using Equation (4), we obtain r,= 0.596. The rank correlation is smaller than the product moment correlation, which in this case is due to ties. In general, ties decrease the correlation. A test that the population rank correlation is 0 can be carried out by calculating the test statistic z=r,/\/1/(n-—1) = 0.596/,/1/11 = 1.98, which has a p-value = 0.04. Therefore we can again reject (at level a = .05) the hypothesis of a rank correlation of 0 in the population.
Table 3 shows the ranks of the data in Table 2, where ties are shown as the mean of the tied ranks. Using Equation (4), we obtain r,= 0.596. The rank correlation is smaller than the product moment correlation, which in this case is due to ties. In general, ties decrease the correlation. A test that the population rank correlation is 0 can be carried out by calculating the test statistic z=r,/\/1/(n-—1) = 0.596/,/1/11 = 1.98, which has a p-value = 0.04. Therefore we can again reject (at level a = .05) the hypothesis of a rank correlation of 0 in the population.
with the resulting values for r,: 1, 4, 4, —1, —1, —1. Now use the average of these values, 7; = 0, as a measure of ion Bs calculation requires m! repetitions for each set of m ties. For Table 3 this requires (3!)(4!)(2!) for the ties for judge 1 and (5!)(2!)(2!) for the ties for judge 2, for a total of 138,240 computations of r, and yields 7, = 0.531. For this example, the computations are feasible. However, a complete enumeration of the permutations for n wines can be computationally inten- sive for large samples with many ties.
with the resulting values for r,: 1, 4, 4, —1, —1, —1. Now use the average of these values, 7; = 0, as a measure of ion Bs calculation requires m! repetitions for each set of m ties. For Table 3 this requires (3!)(4!)(2!) for the ties for judge 1 and (5!)(2!)(2!) for the ties for judge 2, for a total of 138,240 computations of r, and yields 7, = 0.531. For this example, the computations are feasible. However, a complete enumeration of the permutations for n wines can be computationally inten- sive for large samples with many ties.
Example of Ratings on Bordeaux Wines Table 4 provide only a group measure of consistency, while the other two provide estimates of individual judge characteristics. To illustrate their use, we examine data from the website bordoverview.com, which contains ratings assigned by world-renowned critics from the United States and Europe for hundreds of Bordeaux wines from 2004 to 2012 vintages. In this study, we use the dataset from 2012, which contains 108 wines, each tasted by five wine critics: Robert Parker (RP, American), Neal Martin (NM, English), Tim Atkin (TA, English), Jeff Leve (JL, American), and Jane Anson (JA, English). The ratings consist of exact scores (e.g., 85) and range data, which we converted to their maximum bound (e.g., 95 for 93-95). The wines were rated by each critic and placed into one of 17 ordered categories ranging from 82 to 98, where a higher score means a better wine quality. Table 4 shows a subset of the data for the 5 judges of the 108 wines.
Example of Ratings on Bordeaux Wines Table 4 provide only a group measure of consistency, while the other two provide estimates of individual judge characteristics. To illustrate their use, we examine data from the website bordoverview.com, which contains ratings assigned by world-renowned critics from the United States and Europe for hundreds of Bordeaux wines from 2004 to 2012 vintages. In this study, we use the dataset from 2012, which contains 108 wines, each tasted by five wine critics: Robert Parker (RP, American), Neal Martin (NM, English), Tim Atkin (TA, English), Jeff Leve (JL, American), and Jane Anson (JA, English). The ratings consist of exact scores (e.g., 85) and range data, which we converted to their maximum bound (e.g., 95 for 93-95). The wines were rated by each critic and placed into one of 17 ordered categories ranging from 82 to 98, where a higher score means a better wine quality. Table 4 shows a subset of the data for the 5 judges of the 108 wines.
Pairwise Pearson’s Correlation
Pairwise Pearson’s Correlation
Analysis Based on the Bayesian Ordinal Model Table 6 The quantity e,; is the judgment error made by judge j on his assessment of wine /, and is assumed to have a normal distribution with mean zero and a judge-specific variance oF for all wines. The parameter a; quantifies the bias for judge j, and £, measures the discrimination ability of judge j. Then y, takes the ordinal score of s if x; falls between category cutoffs c,_; and ¢,, that is
Analysis Based on the Bayesian Ordinal Model Table 6 The quantity e,; is the judgment error made by judge j on his assessment of wine /, and is assumed to have a normal distribution with mean zero and a judge-specific variance oF for all wines. The parameter a; quantifies the bias for judge j, and £, measures the discrimination ability of judge j. Then y, takes the ordinal score of s if x; falls between category cutoffs c,_; and ¢,, that is
From the Bordeaux wine data in Table 7 we generate three matrices of Pearson cor- relations from 2010, 2011, and 2012. Each matrix shows the correlation of the scores among the three judges, where the rows and columns correspond to RP, NM, and JA, in that
From the Bordeaux wine data in Table 7 we generate three matrices of Pearson cor- relations from 2010, 2011, and 2012. Each matrix shows the correlation of the scores among the three judges, where the rows and columns correspond to RP, NM, and JA, in that
Example of Ratings on Bordeaux Wines over Three Years Table 7 IV. Meta-Analysis for m Judges and n Wines
Example of Ratings on Bordeaux Wines over Three Years Table 7 IV. Meta-Analysis for m Judges and n Wines
Test for the Homogeneity of Correlation Matrices order. To investigate whether the years differ with respect to concordance, we test the homogeneity of correlation matrices. There are several alternative tests for the homogeneity of correlation matrices, pased on the assumption that the correlation matrices arise from a multivari: normal model. The likelihood ratio test is rather complicated, and, instead, — adopt a simpler test proposed by Larntz and Perlman (1988) that still provide: correct significance level. Apply the Fisher transformation he ee <7 -ach correlation coefficient 7 (i, j=1, ..., m) in the Ath (h=1, ..., k) ) sample pr co lation matrix R,, obtaining 2). Define ‘the element-wise means Zj = )>)_) zy and variances vy = (n— 3) hy (2 - —2,) , where n is the number of win [hen the test statistic is
Test for the Homogeneity of Correlation Matrices order. To investigate whether the years differ with respect to concordance, we test the homogeneity of correlation matrices. There are several alternative tests for the homogeneity of correlation matrices, pased on the assumption that the correlation matrices arise from a multivari: normal model. The likelihood ratio test is rather complicated, and, instead, — adopt a simpler test proposed by Larntz and Perlman (1988) that still provide: correct significance level. Apply the Fisher transformation he ee <7 -ach correlation coefficient 7 (i, j=1, ..., m) in the Ath (h=1, ..., k) ) sample pr co lation matrix R,, obtaining 2). Define ‘the element-wise means Zj = )>)_) zy and variances vy = (n— 3) hy (2 - —2,) , where n is the number of win [hen the test statistic is
The Empirical Null Distribution of Ry for the Three-Year Comparison than the observed RW, which is the marked area below the vertical line in the histo- gram. As mentioned above, the p-value is 0.525 for the three-year comparison.
The Empirical Null Distribution of Ry for the Three-Year Comparison than the observed RW, which is the marked area below the vertical line in the histo- gram. As mentioned above, the p-value is 0.525 for the three-year comparison.
Scores for 2 Wines and 10 Judges Table 9
Scores for 2 Wines and 10 Judges Table 9
The Network of Pairwise Comparisons for Six Wines
The Network of Pairwise Comparisons for Six Wines
BIB Design for the Configuration: n = 6, m= 10, 7 =3, w=5,A=2 Not all choices of n, m, j, w, 2 can be constructed to yield a BIB design. There are two restrictions:
BIB Design for the Configuration: n = 6, m= 10, 7 =3, w=5,A=2 Not all choices of n, m, j, w, 2 can be constructed to yield a BIB design. There are two restrictions:
The ANOVA Analysis MSE(judges)/ MSE(error) = 1.86/2.46 = 0.75 and yields a p-value from the F-test as 0.66. With a significance level of 0.05, we reject the null hypothesis that the mean effects of judges are the same.
The ANOVA Analysis MSE(judges)/ MSE(error) = 1.86/2.46 = 0.75 and yields a p-value from the F-test as 0.66. With a significance level of 0.05, we reject the null hypothesis that the mean effects of judges are the same.
Related topics:
MarketingWine EconomicsApplied EconomicsHorticultural production
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved