Figure 2 Use of criteria within relevance judgments.
Related Figures (13)
TABLE |. Examples of types of relevance. and passed on to a search intermediary, the user’s informa- tion need and the written query may no longer be closely tied: “The issue is not what the requester meant to ask but what the request itself actually said” (pp. 391-392). Howard (1994) elaborates on this by stating that objective relevance “is taken to be that relationship which is system-based and usually measured by topicality. That is, the crucial relation is how well the topic of the information request is repre- sented in the topics of the responses” (p. 172). Objective relevance, as defined by both Swanson and Howard, is the relationship between the stated request and the response to that request. This implies that all items containing one or more query terms could conceivably be objectively relevant although IR systems typically consider the number and frequency of query terms in items. However, the user’s perception of how those items relate to his or her informa- tion need is not considered when calculating objective rel- evance. ‘Category assumed but not studied directly. TABLE 2. Overlapping relevance categories identified in studies com- paring relevance criteria literature. FIG. 1. Data collection and analysis process. depending on whether the participant used the criterion as a positive or negative indication of relevance. The criteria identification was an iterative process. document relevance were compared to the criteria identified in Schamber (1991), Park (1992, 1993), Cool et al. (1993), Barry (1993, 1994), Wang and White (1994), and Spink, Greisdorf and Bateman (1998, 1999). The combined set of criteria identified in this literature did not appear to fully capture the information discussed by the participants in this study, although there was overlap. Therefore, as suggested by Stempel (1981), a new set of codes for the participants’ criteria was developed. TABLE 4. Use of category and criteria in both passage and document evaluation. Document relevance criterion focus TABLE 6. Distribution of positive and negative values of criteria in document judgments. As with the passage evaluation, the relationship between positive and negative values of the criteria varied across categories and types of documents relevance judgment (Fig. 4). In document relevance determination, abstract criteria were only mentioned positively while participant criteria were only mentioned negatively. This differed from passage evaluation where abstract criteria were mentioned more negatively in partially relevant judgments and more posi- tively in relevant judgments, and participant criteria were mentioned more positively in both partially relevant and relevant judgments. This may indicate that the most note- worthy aspects of these criteria in document evaluation are the positive aspects of the abstract but the negative aspects of participant criteria. related information for their jobs, making geographic prox- imity a very important criterion. The criteria identified in studies may also be influenced by the design of the study. For example, in this study, participants were promised full- text versions of articles they deemed relevant before they evaluated the document representation. Therefore, availabil- ity, unlike in other studies (e.g., Barry, 1994; Park, 1992; Schamber & Bateman, 1998), was not a criterion in this study. TABLE 7. Use of passage criteria and criterion values in document relevance judgments. FIG. 3. Positive and negative contributions of criteria used in evaluation of passages. FIG. 4. Positive and negative values of criteria used in evaluation of documents. TABLE 8. Synthesis of common concepts for relevance criteria in literature. * Category often assumed but not studied directly. TABLE 9. Frequency of category usage across relevance judgments. ipants indicated that they were unsatisfied with the infor- mativeness of the abstract more frequently in partially rel- evant than relevant document representations, and that they made more guesses about the content of the full-text docu- ment with partially relevant document representations. The results from this study also support theories from Bookstein (1983) and Janes (1993) that partially relevant documents are selected based on the same criteria as relevant docu- ments, they just do not meet as many criteria or do not satisfy the criteria to the same degree.
