Test collection-based ir evaluation needs extension toward sessions - A case of extremely short queries

Heikki Keskustalo, Kalervo Järvelin, Ari Pirkola, Tarun Sharma, Marianne Lykke

Tutkimustuotos: KonferenssiartikkeliScientificvertaisarvioitu

28 Sitaatiot (Scopus)


There is overwhelming evidence suggesting that the real users of IR systems often prefer using extremely short queries (one or two individual words) but they try out several queries if needed. Such behavior is fundamentally different from the process modeled in the traditional test collection-based IR evaluation based on using more verbose queries and only one query per topic. In the present paper, we propose an extension to the test collection-based evaluation. We will utilize sequences of short queries based on empirically grounded but idealized session strategies. We employ TREC data and have test persons to suggest search words, while simulating sessions based on the idealized strategies for repeatability and control. The experimental results show that, surprisingly, web-like very short queries (including one-word query sequences) typically lead to good enough results even in a TREC type test collection. This finding motivates the observed real user behavior: as few very simple attempts normally lead to good enough results, there is no need to pay more effort. We conclude by discussing the consequences of our finding for IR evaluation.

OtsikkoInformation Retrieval Technology - 5th Asia Information Retrieval Symposium, AIRS 2009, Proceedings
DOI - pysyväislinkit
TilaJulkaistu - 2009
Julkaistu ulkoisestiKyllä
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
Tapahtuma5th Asia Information Retrieval Symposium, AIRS 2009 - Sapporo, Japani
Kesto: 21 lokak. 200923 lokak. 2009


NimiLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vuosikerta5839 LNCS
ISSN (painettu)0302-9743
ISSN (elektroninen)1611-3349


Conference5th Asia Information Retrieval Symposium, AIRS 2009

!!ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Sukella tutkimusaiheisiin 'Test collection-based ir evaluation needs extension toward sessions - A case of extremely short queries'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä