Document (#38105)

Author

Verberne, S.
Heijden, M. van der
Hinne, M.
Sappelli, M.
Koldijk, S.
Hoenkamp, E.
Kraaij, W.

Title

Reliability and validity of query intent assessments

Source

Journal of the American Society for Information Science and Technology. 64(2013) no.11, S.2224-2237

Year

2013

Abstract

In most intent recognition studies, annotations of query intent are created post hoc by external assessors who are not the searchers themselves. It is important for the field to get a better understanding of the quality of this process as an approximation for determining the searcher's actual intent. Some studies have investigated the reliability of the query intent annotation process by measuring the interassessor agreement. However, these studies did not measure the validity of the judgments, that is, to what extent the annotations match the searcher's actual intent. In this study, we asked both the searchers themselves and external assessors to classify queries using the same intent classification scheme. We show that of the seven dimensions in our intent classification scheme, four can reliably be used for query annotation. Of these four, only the annotations on the topic and spatial sensitivity dimension are valid when compared with the searcher's annotations. The difference between the interassessor agreement and the assessor-searcher agreement was significant on all dimensions, showing that the agreement between external assessors is not a good estimator of the validity of the intent classifications. Therefore, we encourage the research community to consider using query intent classifications by the searchers themselves as test data.

Theme

Suchtaktik

Similar documents (author)

1. Kraaij, W.; Pohlmann, R.: Evaluation of a Dutch stemming algorithm (1995) 4.95

   4.954854 = sum of:
     4.954854 = weight(author_txt:kraaij in 5866) [ClassicSimilarity], result of:
       4.954854 = fieldWeight in 5866, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.909708 = idf(docFreq=5, maxDocs=44421)
         0.5 = fieldNorm(doc=5866)
   

2. Hiemstra, D.; Kraaij, W.: ¬A language-modeling approach to TREC (2005) 4.95

   4.954854 = sum of:
     4.954854 = weight(author_txt:kraaij in 91) [ClassicSimilarity], result of:
       4.954854 = fieldWeight in 91, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.909708 = idf(docFreq=5, maxDocs=44421)
         0.5 = fieldNorm(doc=91)
   

3. Sappelli, M.; Verberne, S.; Kraaij, W.: Evaluation of context-aware recommendation systems for information re-finding (2017) 3.72

   3.7161405 = sum of:
     3.7161405 = weight(author_txt:kraaij in 4528) [ClassicSimilarity], result of:
       3.7161405 = fieldWeight in 4528, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.909708 = idf(docFreq=5, maxDocs=44421)
         0.375 = fieldNorm(doc=4528)
   

4. Meij, E.; Trieschnigg, D.; Rijke, M. de; Kraaij, W.: Conceptual language models for domain-specific retrieval (2010) 3.10

   3.0967836 = sum of:
     3.0967836 = weight(author_txt:kraaij in 238) [ClassicSimilarity], result of:
       3.0967836 = fieldWeight in 238, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.909708 = idf(docFreq=5, maxDocs=44421)
         0.3125 = fieldNorm(doc=238)
   

Similar documents (content)

1. Osman, D.J.; Yearwood, J.; Vamplew, P.: Automated opinion detection : implications of the level of agreement between human raters (2010) 0.13

   0.13385299 = sum of:
     0.13385299 = product of:
       0.6692649 = sum of:
         0.047217306 = weight(abstract_txt:assessments in 232) [ClassicSimilarity], result of:
           0.047217306 = score(doc=232,freq=3.0), product of:
             0.061551426 = queryWeight, product of:
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.008685918 = queryNorm
             0.7671196 = fieldWeight in 232, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.0625 = fieldNorm(doc=232)
         0.004298998 = weight(abstract_txt:that in 232) [ClassicSimilarity], result of:
           0.004298998 = score(doc=232,freq=2.0), product of:
             0.02056615 = queryWeight, product of:
               1.0011936 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008685918 = queryNorm
             0.20903271 = fieldWeight in 232, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=232)
         0.010170014 = weight(abstract_txt:process in 232) [ClassicSimilarity], result of:
           0.010170014 = score(doc=232,freq=1.0), product of:
             0.04018853 = queryWeight, product of:
               1.1427388 = boost
               4.048922 = idf(docFreq=2105, maxDocs=44421)
               0.008685918 = queryNorm
             0.25305763 = fieldWeight in 232, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.048922 = idf(docFreq=2105, maxDocs=44421)
               0.0625 = fieldNorm(doc=232)
         0.42889988 = weight(abstract_txt:assessors in 232) [ClassicSimilarity], result of:
           0.42889988 = score(doc=232,freq=8.0), product of:
             0.27869263 = queryWeight, product of:
               3.685567 = boost
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.008685918 = queryNorm
             1.5389711 = fieldWeight in 232, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.0625 = fieldNorm(doc=232)
         0.17867875 = weight(abstract_txt:agreement in 232) [ClassicSimilarity], result of:
           0.17867875 = score(doc=232,freq=3.0), product of:
             0.23726806 = queryWeight, product of:
               3.9267256 = boost
               6.9565353 = idf(docFreq=114, maxDocs=44421)
               0.008685918 = queryNorm
             0.753067 = fieldWeight in 232, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.9565353 = idf(docFreq=114, maxDocs=44421)
               0.0625 = fieldNorm(doc=232)
       0.2 = coord(5/25)
   

2. Smith, C.L.: Domain-independent search expertise : gaining knowledge in query formulation through guided practice (2017) 0.13

   0.13369751 = sum of:
     0.13369751 = product of:
       0.66848755 = sum of:
         0.0075996267 = weight(abstract_txt:that in 4643) [ClassicSimilarity], result of:
           0.0075996267 = score(doc=4643,freq=4.0), product of:
             0.02056615 = queryWeight, product of:
               1.0011936 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008685918 = queryNorm
             0.3695211 = fieldWeight in 4643, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=4643)
         0.039321095 = weight(abstract_txt:dimensions in 4643) [ClassicSimilarity], result of:
           0.039321095 = score(doc=4643,freq=1.0), product of:
             0.085316196 = queryWeight, product of:
               1.6649902 = boost
               5.899349 = idf(docFreq=330, maxDocs=44421)
               0.008685918 = queryNorm
             0.46088666 = fieldWeight in 4643, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.899349 = idf(docFreq=330, maxDocs=44421)
               0.078125 = fieldNorm(doc=4643)
         0.062397636 = weight(abstract_txt:searchers in 4643) [ClassicSimilarity], result of:
           0.062397636 = score(doc=4643,freq=1.0), product of:
             0.13286898 = queryWeight, product of:
               2.5447981 = boost
               6.011108 = idf(docFreq=295, maxDocs=44421)
               0.008685918 = queryNorm
             0.4696178 = fieldWeight in 4643, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.011108 = idf(docFreq=295, maxDocs=44421)
               0.078125 = fieldNorm(doc=4643)
         0.12604953 = weight(abstract_txt:query in 4643) [ClassicSimilarity], result of:
           0.12604953 = score(doc=4643,freq=6.0), product of:
             0.13853882 = queryWeight, product of:
               3.3546846 = boost
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.008685918 = queryNorm
             0.90984994 = fieldWeight in 4643, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.078125 = fieldNorm(doc=4643)
         0.43311965 = weight(abstract_txt:intent in 4643) [ClassicSimilarity], result of:
           0.43311965 = score(doc=4643,freq=1.0), product of:
             0.7222313 = queryWeight, product of:
               10.832261 = boost
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.008685918 = queryNorm
             0.5996966 = fieldWeight in 4643, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.078125 = fieldNorm(doc=4643)
       0.2 = coord(5/25)
   

3. Selvaretnam, B.; Belkhatir, M.: ¬A linguistically driven framework for query expansion via grammatical constituent highlighting and role-based concept weighting (2016) 0.10

   0.10011453 = sum of:
     0.10011453 = product of:
       0.62571585 = sum of:
         0.0065814694 = weight(abstract_txt:that in 3876) [ClassicSimilarity], result of:
           0.0065814694 = score(doc=3876,freq=3.0), product of:
             0.02056615 = queryWeight, product of:
               1.0011936 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008685918 = queryNorm
             0.32001466 = fieldWeight in 3876, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3876)
         0.031636234 = weight(abstract_txt:scheme in 3876) [ClassicSimilarity], result of:
           0.031636234 = score(doc=3876,freq=1.0), product of:
             0.07380248 = queryWeight, product of:
               1.548572 = boost
               5.4868593 = idf(docFreq=499, maxDocs=44421)
               0.008685918 = queryNorm
             0.42866087 = fieldWeight in 3876, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4868593 = idf(docFreq=499, maxDocs=44421)
               0.078125 = fieldNorm(doc=3876)
         0.15437852 = weight(abstract_txt:query in 3876) [ClassicSimilarity], result of:
           0.15437852 = score(doc=3876,freq=9.0), product of:
             0.13853882 = queryWeight, product of:
               3.3546846 = boost
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.008685918 = queryNorm
             1.114334 = fieldWeight in 3876, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.078125 = fieldNorm(doc=3876)
         0.43311965 = weight(abstract_txt:intent in 3876) [ClassicSimilarity], result of:
           0.43311965 = score(doc=3876,freq=1.0), product of:
             0.7222313 = queryWeight, product of:
               10.832261 = boost
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.008685918 = queryNorm
             0.5996966 = fieldWeight in 3876, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.078125 = fieldNorm(doc=3876)
       0.16 = coord(4/25)
   

4. White, R.W.; Jose, J.M.; Ruthven, I.: Using top-ranking sentences to facilitate effective information access (2005) 0.10

   0.09958169 = sum of:
     0.09958169 = product of:
       0.35564888 = sum of:
         0.005265176 = weight(abstract_txt:that in 4881) [ClassicSimilarity], result of:
           0.005265176 = score(doc=4881,freq=3.0), product of:
             0.02056615 = queryWeight, product of:
               1.0011936 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008685918 = queryNorm
             0.25601172 = fieldWeight in 4881, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
         0.010170014 = weight(abstract_txt:process in 4881) [ClassicSimilarity], result of:
           0.010170014 = score(doc=4881,freq=1.0), product of:
             0.04018853 = queryWeight, product of:
               1.1427388 = boost
               4.048922 = idf(docFreq=2105, maxDocs=44421)
               0.008685918 = queryNorm
             0.25305763 = fieldWeight in 4881, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.048922 = idf(docFreq=2105, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
         0.03470297 = weight(abstract_txt:actual in 4881) [ClassicSimilarity], result of:
           0.03470297 = score(doc=4881,freq=1.0), product of:
             0.09108891 = queryWeight, product of:
               1.7203971 = boost
               6.0956655 = idf(docFreq=271, maxDocs=44421)
               0.008685918 = queryNorm
             0.3809791 = fieldWeight in 4881, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0956655 = idf(docFreq=271, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
         0.02505704 = weight(abstract_txt:studies in 4881) [ClassicSimilarity], result of:
           0.02505704 = score(doc=4881,freq=2.0), product of:
             0.06660825 = queryWeight, product of:
               1.8017958 = boost
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.008685918 = queryNorm
             0.37618524 = fieldWeight in 4881, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
         0.07059486 = weight(abstract_txt:searchers in 4881) [ClassicSimilarity], result of:
           0.07059486 = score(doc=4881,freq=2.0), product of:
             0.13286898 = queryWeight, product of:
               2.5447981 = boost
               6.011108 = idf(docFreq=295, maxDocs=44421)
               0.008685918 = queryNorm
             0.53131187 = fieldWeight in 4881, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.011108 = idf(docFreq=295, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
         0.058219783 = weight(abstract_txt:query in 4881) [ClassicSimilarity], result of:
           0.058219783 = score(doc=4881,freq=2.0), product of:
             0.13853882 = queryWeight, product of:
               3.3546846 = boost
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.008685918 = queryNorm
             0.42024165 = fieldWeight in 4881, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
         0.15163901 = weight(abstract_txt:searcher's in 4881) [ClassicSimilarity], result of:
           0.15163901 = score(doc=4881,freq=1.0), product of:
             0.27869263 = queryWeight, product of:
               3.685567 = boost
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.008685918 = queryNorm
             0.54410845 = fieldWeight in 4881, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.0625 = fieldNorm(doc=4881)
       0.28 = coord(7/25)
   

5. Lee, W.M.; Sanderson, M.: Analyzing URL queries (2010) 0.09

   0.09442321 = sum of:
     0.09442321 = product of:
       0.5901451 = sum of:
         0.0080426885 = weight(abstract_txt:that in 105) [ClassicSimilarity], result of:
           0.0080426885 = score(doc=105,freq=7.0), product of:
             0.02056615 = queryWeight, product of:
               1.0011936 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008685918 = queryNorm
             0.39106438 = fieldWeight in 105, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=105)
         0.009748256 = weight(abstract_txt:classification in 105) [ClassicSimilarity], result of:
           0.009748256 = score(doc=105,freq=1.0), product of:
             0.039069604 = queryWeight, product of:
               1.1267185 = boost
               3.9921594 = idf(docFreq=2228, maxDocs=44421)
               0.008685918 = queryNorm
             0.24950996 = fieldWeight in 105, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.9921594 = idf(docFreq=2228, maxDocs=44421)
               0.0625 = fieldNorm(doc=105)
         0.08233521 = weight(abstract_txt:query in 105) [ClassicSimilarity], result of:
           0.08233521 = score(doc=105,freq=4.0), product of:
             0.13853882 = queryWeight, product of:
               3.3546846 = boost
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.008685918 = queryNorm
             0.5943115 = fieldWeight in 105, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.754492 = idf(docFreq=1039, maxDocs=44421)
               0.0625 = fieldNorm(doc=105)
         0.49001893 = weight(abstract_txt:intent in 105) [ClassicSimilarity], result of:
           0.49001893 = score(doc=105,freq=2.0), product of:
             0.7222313 = queryWeight, product of:
               10.832261 = boost
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.008685918 = queryNorm
             0.6784792 = fieldWeight in 105, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.0625 = fieldNorm(doc=105)
       0.16 = coord(4/25)