Document (#38105)

Author

Verberne, S.
Heijden, M. van der
Hinne, M.
Sappelli, M.
Koldijk, S.
Hoenkamp, E.
Kraaij, W.

Title

Reliability and validity of query intent assessments

Source

Journal of the American Society for Information Science and Technology. 64(2013) no.11, S.2224-2237

Year

2013

Abstract

In most intent recognition studies, annotations of query intent are created post hoc by external assessors who are not the searchers themselves. It is important for the field to get a better understanding of the quality of this process as an approximation for determining the searcher's actual intent. Some studies have investigated the reliability of the query intent annotation process by measuring the interassessor agreement. However, these studies did not measure the validity of the judgments, that is, to what extent the annotations match the searcher's actual intent. In this study, we asked both the searchers themselves and external assessors to classify queries using the same intent classification scheme. We show that of the seven dimensions in our intent classification scheme, four can reliably be used for query annotation. Of these four, only the annotations on the topic and spatial sensitivity dimension are valid when compared with the searcher's annotations. The difference between the interassessor agreement and the assessor-searcher agreement was significant on all dimensions, showing that the agreement between external assessors is not a good estimator of the validity of the intent classifications. Therefore, we encourage the research community to consider using query intent classifications by the searchers themselves as test data.

Theme

Suchtaktik

Similar documents (author)

1. Kraaij, W.; Pohlmann, R.: Evaluation of a Dutch stemming algorithm (1995) 4.95

   4.952564 = sum of:
     4.952564 = weight(author_txt:kraaij in 5798) [ClassicSimilarity], result of:
       4.952564 = fieldWeight in 5798, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.905128 = idf(docFreq=5, maxDocs=44218)
         0.5 = fieldNorm(doc=5798)
   

2. Hiemstra, D.; Kraaij, W.: ¬A language-modeling approach to TREC (2005) 4.95

   4.952564 = sum of:
     4.952564 = weight(author_txt:kraaij in 5091) [ClassicSimilarity], result of:
       4.952564 = fieldWeight in 5091, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.905128 = idf(docFreq=5, maxDocs=44218)
         0.5 = fieldNorm(doc=5091)
   

3. Sappelli, M.; Verberne, S.; Kraaij, W.: Evaluation of context-aware recommendation systems for information re-finding (2017) 3.71

   3.7144227 = sum of:
     3.7144227 = weight(author_txt:kraaij in 3528) [ClassicSimilarity], result of:
       3.7144227 = fieldWeight in 3528, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.905128 = idf(docFreq=5, maxDocs=44218)
         0.375 = fieldNorm(doc=3528)
   

4. Meij, E.; Trieschnigg, D.; Rijke, M. de; Kraaij, W.: Conceptual language models for domain-specific retrieval (2010) 3.10

   3.0953524 = sum of:
     3.0953524 = weight(author_txt:kraaij in 4238) [ClassicSimilarity], result of:
       3.0953524 = fieldWeight in 4238, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.905128 = idf(docFreq=5, maxDocs=44218)
         0.3125 = fieldNorm(doc=4238)
   

Similar documents (content)

1. Osman, D.J.; Yearwood, J.; Vamplew, P.: Automated opinion detection : implications of the level of agreement between human raters (2010) 0.14

   0.13515754 = sum of:
     0.13515754 = product of:
       0.6757877 = sum of:
         0.04730977 = weight(abstract_txt:assessments in 4232) [ClassicSimilarity], result of:
           0.04730977 = score(doc=4232,freq=3.0), product of:
             0.061625265 = queryWeight, product of:
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.008689752 = queryNorm
             0.7677009 = fieldWeight in 4232, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.0625 = fieldNorm(doc=4232)
         0.0043224366 = weight(abstract_txt:that in 4232) [ClassicSimilarity], result of:
           0.0043224366 = score(doc=4232,freq=2.0), product of:
             0.02063866 = queryWeight, product of:
               1.0023559 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008689752 = queryNorm
             0.20943399 = fieldWeight in 4232, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=4232)
         0.010182548 = weight(abstract_txt:process in 4232) [ClassicSimilarity], result of:
           0.010182548 = score(doc=4232,freq=1.0), product of:
             0.040217306 = queryWeight, product of:
               1.1424628 = boost
               4.0510116 = idf(docFreq=2091, maxDocs=44218)
               0.008689752 = queryNorm
             0.25318822 = fieldWeight in 4232, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0510116 = idf(docFreq=2091, maxDocs=44218)
               0.0625 = fieldNorm(doc=4232)
         0.43570322 = weight(abstract_txt:assessors in 4232) [ClassicSimilarity], result of:
           0.43570322 = score(doc=4232,freq=8.0), product of:
             0.28160232 = queryWeight, product of:
               3.7025366 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.008689752 = queryNorm
             1.5472288 = fieldWeight in 4232, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=4232)
         0.1782697 = weight(abstract_txt:agreement in 4232) [ClassicSimilarity], result of:
           0.1782697 = score(doc=4232,freq=3.0), product of:
             0.23688082 = queryWeight, product of:
               3.921169 = boost
               6.9519553 = idf(docFreq=114, maxDocs=44218)
               0.008689752 = queryNorm
             0.7525712 = fieldWeight in 4232, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.9519553 = idf(docFreq=114, maxDocs=44218)
               0.0625 = fieldNorm(doc=4232)
       0.2 = coord(5/25)
   

2. Smith, C.L.: Domain-independent search expertise : gaining knowledge in query formulation through guided practice (2017) 0.13

   0.13359487 = sum of:
     0.13359487 = product of:
       0.66797435 = sum of:
         0.007641061 = weight(abstract_txt:that in 3643) [ClassicSimilarity], result of:
           0.007641061 = score(doc=3643,freq=4.0), product of:
             0.02063866 = queryWeight, product of:
               1.0023559 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008689752 = queryNorm
             0.3702305 = fieldWeight in 3643, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=3643)
         0.039832644 = weight(abstract_txt:dimensions in 3643) [ClassicSimilarity], result of:
           0.039832644 = score(doc=3643,freq=1.0), product of:
             0.08604548 = queryWeight, product of:
               1.6710892 = boost
               5.925446 = idf(docFreq=320, maxDocs=44218)
               0.008689752 = queryNorm
             0.46292546 = fieldWeight in 3643, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.925446 = idf(docFreq=320, maxDocs=44218)
               0.078125 = fieldNorm(doc=3643)
         0.06234069 = weight(abstract_txt:searchers in 3643) [ClassicSimilarity], result of:
           0.06234069 = score(doc=3643,freq=1.0), product of:
             0.13277413 = queryWeight, product of:
               2.5423653 = boost
               6.009912 = idf(docFreq=294, maxDocs=44218)
               0.008689752 = queryNorm
             0.46952438 = fieldWeight in 3643, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.009912 = idf(docFreq=294, maxDocs=44218)
               0.078125 = fieldNorm(doc=3643)
         0.12595189 = weight(abstract_txt:query in 3643) [ClassicSimilarity], result of:
           0.12595189 = score(doc=3643,freq=6.0), product of:
             0.13845266 = queryWeight, product of:
               3.3516316 = boost
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.008689752 = queryNorm
             0.9097108 = fieldWeight in 3643, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.078125 = fieldNorm(doc=3643)
         0.43220806 = weight(abstract_txt:intent in 3643) [ClassicSimilarity], result of:
           0.43220806 = score(doc=3643,freq=1.0), product of:
             0.7211416 = queryWeight, product of:
               10.817599 = boost
               7.6715355 = idf(docFreq=55, maxDocs=44218)
               0.008689752 = queryNorm
             0.5993387 = fieldWeight in 3643, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6715355 = idf(docFreq=55, maxDocs=44218)
               0.078125 = fieldNorm(doc=3643)
       0.2 = coord(5/25)
   

3. Selvaretnam, B.; Belkhatir, M.: ¬A linguistically driven framework for query expansion via grammatical constituent highlighting and role-based concept weighting (2016) 0.10

   0.09994102 = sum of:
     0.09994102 = product of:
       0.6246314 = sum of:
         0.006617353 = weight(abstract_txt:that in 2876) [ClassicSimilarity], result of:
           0.006617353 = score(doc=2876,freq=3.0), product of:
             0.02063866 = queryWeight, product of:
               1.0023559 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008689752 = queryNorm
             0.320629 = fieldWeight in 2876, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2876)
         0.0315471 = weight(abstract_txt:scheme in 2876) [ClassicSimilarity], result of:
           0.0315471 = score(doc=2876,freq=1.0), product of:
             0.07365602 = queryWeight, product of:
               1.5461076 = boost
               5.4822793 = idf(docFreq=499, maxDocs=44218)
               0.008689752 = queryNorm
             0.42830306 = fieldWeight in 2876, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4822793 = idf(docFreq=499, maxDocs=44218)
               0.078125 = fieldNorm(doc=2876)
         0.1542589 = weight(abstract_txt:query in 2876) [ClassicSimilarity], result of:
           0.1542589 = score(doc=2876,freq=9.0), product of:
             0.13845266 = queryWeight, product of:
               3.3516316 = boost
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.008689752 = queryNorm
             1.1141635 = fieldWeight in 2876, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.078125 = fieldNorm(doc=2876)
         0.43220806 = weight(abstract_txt:intent in 2876) [ClassicSimilarity], result of:
           0.43220806 = score(doc=2876,freq=1.0), product of:
             0.7211416 = queryWeight, product of:
               10.817599 = boost
               7.6715355 = idf(docFreq=55, maxDocs=44218)
               0.008689752 = queryNorm
             0.5993387 = fieldWeight in 2876, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6715355 = idf(docFreq=55, maxDocs=44218)
               0.078125 = fieldNorm(doc=2876)
       0.16 = coord(4/25)
   

4. White, R.W.; Jose, J.M.; Ruthven, I.: Using top-ranking sentences to facilitate effective information access (2005) 0.10

   0.09954932 = sum of:
     0.09954932 = product of:
       0.3555333 = sum of:
         0.005293882 = weight(abstract_txt:that in 3881) [ClassicSimilarity], result of:
           0.005293882 = score(doc=3881,freq=3.0), product of:
             0.02063866 = queryWeight, product of:
               1.0023559 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008689752 = queryNorm
             0.2565032 = fieldWeight in 3881, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
         0.010182548 = weight(abstract_txt:process in 3881) [ClassicSimilarity], result of:
           0.010182548 = score(doc=3881,freq=1.0), product of:
             0.040217306 = queryWeight, product of:
               1.1424628 = boost
               4.0510116 = idf(docFreq=2091, maxDocs=44218)
               0.008689752 = queryNorm
             0.25318822 = fieldWeight in 3881, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0510116 = idf(docFreq=2091, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
         0.03473983 = weight(abstract_txt:actual in 3881) [ClassicSimilarity], result of:
           0.03473983 = score(doc=3881,freq=1.0), product of:
             0.091143794 = queryWeight, product of:
               1.719884 = boost
               6.0984654 = idf(docFreq=269, maxDocs=44218)
               0.008689752 = queryNorm
             0.3811541 = fieldWeight in 3881, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0984654 = idf(docFreq=269, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
         0.02526 = weight(abstract_txt:studies in 3881) [ClassicSimilarity], result of:
           0.02526 = score(doc=3881,freq=2.0), product of:
             0.06696039 = queryWeight, product of:
               1.8054698 = boost
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.008689752 = queryNorm
             0.37723795 = fieldWeight in 3881, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
         0.07053044 = weight(abstract_txt:searchers in 3881) [ClassicSimilarity], result of:
           0.07053044 = score(doc=3881,freq=2.0), product of:
             0.13277413 = queryWeight, product of:
               2.5423653 = boost
               6.009912 = idf(docFreq=294, maxDocs=44218)
               0.008689752 = queryNorm
             0.5312062 = fieldWeight in 3881, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.009912 = idf(docFreq=294, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
         0.05817468 = weight(abstract_txt:query in 3881) [ClassicSimilarity], result of:
           0.05817468 = score(doc=3881,freq=2.0), product of:
             0.13845266 = queryWeight, product of:
               3.3516316 = boost
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.008689752 = queryNorm
             0.4201774 = fieldWeight in 3881, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
         0.15135191 = weight(abstract_txt:searcher's in 3881) [ClassicSimilarity], result of:
           0.15135191 = score(doc=3881,freq=1.0), product of:
             0.2783114 = queryWeight, product of:
               3.6808383 = boost
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.008689752 = queryNorm
             0.54382217 = fieldWeight in 3881, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.0625 = fieldNorm(doc=3881)
       0.28 = coord(7/25)
   

5. Lee, W.M.; Sanderson, M.: Analyzing URL queries (2010) 0.09

   0.09425441 = sum of:
     0.09425441 = product of:
       0.5890901 = sum of:
         0.008086538 = weight(abstract_txt:that in 4105) [ClassicSimilarity], result of:
           0.008086538 = score(doc=4105,freq=7.0), product of:
             0.02063866 = queryWeight, product of:
               1.0023559 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008689752 = queryNorm
             0.3918151 = fieldWeight in 4105, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=4105)
         0.009744557 = weight(abstract_txt:classification in 4105) [ClassicSimilarity], result of:
           0.009744557 = score(doc=4105,freq=1.0), product of:
             0.039055604 = queryWeight, product of:
               1.1258416 = boost
               3.9920752 = idf(docFreq=2218, maxDocs=44218)
               0.008689752 = queryNorm
             0.2495047 = fieldWeight in 4105, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.9920752 = idf(docFreq=2218, maxDocs=44218)
               0.0625 = fieldNorm(doc=4105)
         0.08227142 = weight(abstract_txt:query in 4105) [ClassicSimilarity], result of:
           0.08227142 = score(doc=4105,freq=4.0), product of:
             0.13845266 = queryWeight, product of:
               3.3516316 = boost
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.008689752 = queryNorm
             0.5942206 = fieldWeight in 4105, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.7537646 = idf(docFreq=1035, maxDocs=44218)
               0.0625 = fieldNorm(doc=4105)
         0.48898762 = weight(abstract_txt:intent in 4105) [ClassicSimilarity], result of:
           0.48898762 = score(doc=4105,freq=2.0), product of:
             0.7211416 = queryWeight, product of:
               10.817599 = boost
               7.6715355 = idf(docFreq=55, maxDocs=44218)
               0.008689752 = queryNorm
             0.67807436 = fieldWeight in 4105, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.6715355 = idf(docFreq=55, maxDocs=44218)
               0.0625 = fieldNorm(doc=4105)
       0.16 = coord(4/25)