Annotations and Digital Libraries: Designing Adequate Test-Beds
2007, Lecture Notes in Computer Science
https://doi.org/10.1007/978-3-540-77094-7_23Abstract
The increasing number of users and the diffusion of Digital Libraries (DLs) has increased the demand for newer and improved systems to give better assistance to the user during the search of resources in collections managed by Digital Library Systems (DLSs). In this perspective, the annotations made on documents offer an interesting possibility for improving both the user experience of the DLS and the retrieval performance of the system itself. However, while different approaches based on annotations have been proposed, they still lack a full experimental evaluation, mainly because an experimental collection with annotation is missing. Therefore, this paper addresses the problem of setting an adequate experimental test-bed for DL search algorithms which exploit annotations, and discusses a flexible strategy for creating test collections with annotated documents.
References (11)
- Agosti, M., Ferro, N.: Annotations: Enriching a Digital Library. In: Koch, T., Sølvberg, I.T. (eds.) ECDL 2003. LNCS, vol. 2769, pp. 88-100. Springer, Heidelberg (2003)
- Agosti, M., Ferro, N.: A Formal Model of Annotations of Digital Content. ACM Trans. on Information Systems (TOIS) 26(1), 1-55 (2008)
- Agosti, M., Ferro, N.: Annotations as Context for Searching Documents. In: Crestani, F., Ruthven, I. (eds.) CoLIS 2005. LNCS, vol. 3507, pp. 155-170.
- Springer, Heidelberg (2005)
- Golovchinsky, G., Price, M.N., Schilit, B.N.: From Reading to Retrieval: Freeform Ink Annotations as Queries. In: Proc. 22nd Annual Int. ACM SIGIR Conf. on Research and Development in IR, pp. 19-25. ACM Press, New York (1999)
- Frommholz, I., Brocks, H., Thiel, U., Neuhold, E., Iannone, L., Semeraro, G., Berardi, M., Ceci, M.: Document-Centered Collaboration for Scholars in the Humanities-The COLLATE System. In: Koch, T., Sølvberg, I.T. (eds.) ECDL 2003. LNCS, vol. 2769, pp. 434-445. Springer, Heidelberg (2003)
- Sanderson, M., Joho, H.: Forming test collections with no system pooling. In: SIGIR 2004: Proc. of the 27th Annual Int. ACM SIGIR Conf. on Research and Development in IR, pp. 33-40. ACM Press, New York (2004)
- Carterette, B., Allan, J., Sitaraman, R.: Minimal test collections for retrieval evalu- ation. In: SIGIR 2006: Proc. of the 29th Annual Int. ACM SIGIR Conf. on Research and development in IR, pp. 268-275. ACM Press, New York (2006)
- Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). MIT Press, Cambridge (2005)
- Coppotelli, T.: Creazione di una collezione sperimentale per la valutazione di sis- temi di reperimento dell'informazione che utilizzino le annotazioni (in Italian). Master's thesis, Dept of Information Engineering, Univ. Padua (2006)
- Agosti, M., Coppotelli, T., Ferro, N., Pretto, L.: Exploiting Relevance Assessment for the Creation of an Experimental Test Collection to Evaluate Systems that Use Annotations. In: Thanos, C., Borri, F., Candela, L. (eds.) DELOS 2007. LNCS, vol. 4877, pp. 195-202. Springer, Heidelberg (2007)