Document (#37394)

Author

Bilal, D.

Title

Ranking, relevance judgment, and precision of information retrieval on children's queries : evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids

Source

Journal of the American Society for Information Science and Technology. 63(2012) no.9, S.1879-1896

Year

2012

Abstract

This study employed benchmarking and intellectual relevance judgment in evaluating Google, Yahoo!, Bing, Yahoo! Kids, and Ask Kids on 30 queries that children formulated to find information for specific tasks. Retrieved hits on given queries were benchmarked to Google's and Yahoo! Kids' top-five ranked hits retrieved. Relevancy of hits was judged on a graded scale; precision was calculated using the precision-at-ten metric (P@10). Yahoo! and Bing produced a similar percentage in hit overlap with Google (nearly 30%), but differed in the ranking of hits. Ask Kids retrieved 11% in hit overlap with Google versus 3% by Yahoo! Kids. The engines retrieved 26 hits across query clusters that overlapped with Yahoo! Kids' top-five ranked hits. Precision (P) that the engines produced across the queries was P = 0.48 for relevant hits, and P = 0.28 for partially relevant hits. Precision by Ask Kids was P = 0.44 for relevant hits versus P = 0.21 by Yahoo! Kids. Bing produced the highest total precision (TP) of relevant hits (TP = 0.86) across the queries, and Yahoo! Kids yielded the lowest (TP = 0.47). Average precision (AP) of relevant hits was AP = 0.56 by leading engines versus AP = 0.29 by small engines. In contrast, average precision of partially relevant hits was AP = 0.83 by small engines versus AP = 0.33 by leading engines. Average precision of relevant hits across the engines was highest on two-word queries and lowest on one-word queries. Google performed best on natural language queries; Bing did the same (P = 0.69) on two-word queries. The findings have implications for search engine ranking algorithms, relevance theory, search engine design, research design, and information literacy.

Theme

Retrievalalgorithmen
Suchmaschinen

Object

Google
Yahoo!
Bing
Yahoo! Kids
ask Kids

Similar documents (author)

1. Bilal, D.: Children's use of the Yahooligans! Web search engine : I. Cognitive, physical, and affective behaviors on fact-based search tasks (2000) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bilal in 4776) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 4776, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=4776)
   

2. Bilal, D.: Children's use of the Yahooligans! Web search engine : II. Cognitive and physical behaviors on research tasks (2001) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bilal in 5583) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 5583, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=5583)
   

3. Bilal, D.: Web search engines for children : a comparative study and performance evaluation of Yahooligans!, AskJeeves for Kids, and Super Snooper (1999) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bilal in 6544) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 6544, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=6544)
   

4. Bilal, D.: Children's use of the Yahooligans! Web search engine : III. Cognitive and physical behaviors on fully self-generated search tasks (2002) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bilal in 5228) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 5228, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=5228)
   

5. Bilal, D.: Percentile ranks and the integrated impact indicator (I3) (2012) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bilal in 392) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 392, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=392)
   

Similar documents (content)

1. Bilal, D.: Web search engines for children : a comparative study and performance evaluation of Yahooligans!, AskJeeves for Kids, and Super Snooper (1999) 0.41

   0.41155243 = sum of:
     0.41155243 = product of:
       1.2861013 = sum of:
         0.027772177 = weight(abstract_txt:highest in 6544) [ClassicSimilarity], result of:
           0.027772177 = score(doc=6544,freq=1.0), product of:
             0.064769045 = queryWeight, product of:
               1.2138622 = boost
               6.8606052 = idf(docFreq=125, maxDocs=44218)
               0.0077774217 = queryNorm
             0.42878783 = fieldWeight in 6544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8606052 = idf(docFreq=125, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.028788554 = weight(abstract_txt:overlap in 6544) [ClassicSimilarity], result of:
           0.028788554 = score(doc=6544,freq=1.0), product of:
             0.06633979 = queryWeight, product of:
               1.228493 = boost
               6.943297 = idf(docFreq=115, maxDocs=44218)
               0.0077774217 = queryNorm
             0.43395606 = fieldWeight in 6544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.943297 = idf(docFreq=115, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.02188557 = weight(abstract_txt:relevance in 6544) [ClassicSimilarity], result of:
           0.02188557 = score(doc=6544,freq=2.0), product of:
             0.05020572 = queryWeight, product of:
               1.3089042 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0077774217 = queryNorm
             0.43591788 = fieldWeight in 6544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.031988036 = weight(abstract_txt:retrieved in 6544) [ClassicSimilarity], result of:
           0.031988036 = score(doc=6544,freq=1.0), product of:
             0.0896663 = queryWeight, product of:
               2.0198321 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.0077774217 = queryNorm
             0.35674536 = fieldWeight in 6544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.07769239 = weight(abstract_txt:engines in 6544) [ClassicSimilarity], result of:
           0.07769239 = score(doc=6544,freq=3.0), product of:
             0.13537198 = queryWeight, product of:
               3.2830997 = boost
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0077774217 = queryNorm
             0.5739178 = fieldWeight in 6544, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.051537942 = weight(abstract_txt:queries in 6544) [ClassicSimilarity], result of:
           0.051537942 = score(doc=6544,freq=1.0), product of:
             0.16147913 = queryWeight, product of:
               4.0658383 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0077774217 = queryNorm
             0.31916162 = fieldWeight in 6544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.2409485 = weight(abstract_txt:hits in 6544) [ClassicSimilarity], result of:
           0.2409485 = score(doc=6544,freq=1.0), product of:
             0.51036566 = queryWeight, product of:
               8.687264 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0077774217 = queryNorm
             0.47210953 = fieldWeight in 6544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
         0.8054881 = weight(abstract_txt:kids in 6544) [ClassicSimilarity], result of:
           0.8054881 = score(doc=6544,freq=5.0), product of:
             0.6311479 = queryWeight, product of:
               8.886538 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.0077774217 = queryNorm
             1.2762271 = fieldWeight in 6544, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.0625 = fieldNorm(doc=6544)
       0.32 = coord(8/25)
   

2. Eastman, C.M.: 30,000 hits may be better than 300 : precision anomalies in Internet searches (2002) 0.22

   0.22252257 = sum of:
     0.22252257 = product of:
       0.79472345 = sum of:
         0.032242704 = weight(abstract_txt:ranked in 5231) [ClassicSimilarity], result of:
           0.032242704 = score(doc=5231,freq=2.0), product of:
             0.056785636 = queryWeight, product of:
               1.1365927 = boost
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.0077774217 = queryNorm
             0.5677968 = fieldWeight in 5231, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
         0.015475436 = weight(abstract_txt:relevance in 5231) [ClassicSimilarity], result of:
           0.015475436 = score(doc=5231,freq=1.0), product of:
             0.05020572 = queryWeight, product of:
               1.3089042 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0077774217 = queryNorm
             0.3082405 = fieldWeight in 5231, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
         0.03921592 = weight(abstract_txt:ranking in 5231) [ClassicSimilarity], result of:
           0.03921592 = score(doc=5231,freq=3.0), product of:
             0.064703204 = queryWeight, product of:
               1.4859158 = boost
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0077774217 = queryNorm
             0.6060893 = fieldWeight in 5231, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
         0.044855725 = weight(abstract_txt:engines in 5231) [ClassicSimilarity], result of:
           0.044855725 = score(doc=5231,freq=1.0), product of:
             0.13537198 = queryWeight, product of:
               3.2830997 = boost
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0077774217 = queryNorm
             0.3313516 = fieldWeight in 5231, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
         0.072885655 = weight(abstract_txt:queries in 5231) [ClassicSimilarity], result of:
           0.072885655 = score(doc=5231,freq=2.0), product of:
             0.16147913 = queryWeight, product of:
               4.0658383 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0077774217 = queryNorm
             0.4513627 = fieldWeight in 5231, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
         0.17271297 = weight(abstract_txt:precision in 5231) [ClassicSimilarity], result of:
           0.17271297 = score(doc=5231,freq=7.0), product of:
             0.18903793 = queryWeight, product of:
               4.3991256 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0077774217 = queryNorm
             0.9136418 = fieldWeight in 5231, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
         0.41733503 = weight(abstract_txt:hits in 5231) [ClassicSimilarity], result of:
           0.41733503 = score(doc=5231,freq=3.0), product of:
             0.51036566 = queryWeight, product of:
               8.687264 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0077774217 = queryNorm
             0.8177177 = fieldWeight in 5231, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0625 = fieldNorm(doc=5231)
       0.28 = coord(7/25)
   

3. Li, L.; Shang, Y.; Zhang, W.: Improvement of HITS-based algorithms on Web documents 0.18

   0.18111351 = sum of:
     0.18111351 = product of:
       1.1319594 = sum of:
         0.033505306 = weight(abstract_txt:relevance in 2514) [ClassicSimilarity], result of:
           0.033505306 = score(doc=2514,freq=3.0), product of:
             0.05020572 = queryWeight, product of:
               1.3089042 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0077774217 = queryNorm
             0.6673603 = fieldWeight in 2514, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=2514)
         0.06442243 = weight(abstract_txt:queries in 2514) [ClassicSimilarity], result of:
           0.06442243 = score(doc=2514,freq=1.0), product of:
             0.16147913 = queryWeight, product of:
               4.0658383 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0077774217 = queryNorm
             0.39895204 = fieldWeight in 2514, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.078125 = fieldNorm(doc=2514)
         0.081599206 = weight(abstract_txt:precision in 2514) [ClassicSimilarity], result of:
           0.081599206 = score(doc=2514,freq=1.0), product of:
             0.18903793 = queryWeight, product of:
               4.3991256 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0077774217 = queryNorm
             0.4316552 = fieldWeight in 2514, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.078125 = fieldNorm(doc=2514)
         0.9524325 = weight(abstract_txt:hits in 2514) [ClassicSimilarity], result of:
           0.9524325 = score(doc=2514,freq=10.0), product of:
             0.51036566 = queryWeight, product of:
               8.687264 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0077774217 = queryNorm
             1.8661767 = fieldWeight in 2514, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.078125 = fieldNorm(doc=2514)
       0.16 = coord(4/25)
   

4. Bar-Ilan, J.: Evaluating the stability of the search tools Hotbot and Snap : a case study (2000) 0.15

   0.14937687 = sum of:
     0.14937687 = product of:
       0.74688435 = sum of:
         0.04318283 = weight(abstract_txt:overlap in 1180) [ClassicSimilarity], result of:
           0.04318283 = score(doc=1180,freq=1.0), product of:
             0.06633979 = queryWeight, product of:
               1.228493 = boost
               6.943297 = idf(docFreq=115, maxDocs=44218)
               0.0077774217 = queryNorm
             0.6509341 = fieldWeight in 1180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.943297 = idf(docFreq=115, maxDocs=44218)
               0.09375 = fieldNorm(doc=1180)
         0.047982056 = weight(abstract_txt:retrieved in 1180) [ClassicSimilarity], result of:
           0.047982056 = score(doc=1180,freq=1.0), product of:
             0.0896663 = queryWeight, product of:
               2.0198321 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.0077774217 = queryNorm
             0.53511804 = fieldWeight in 1180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.09375 = fieldNorm(doc=1180)
         0.067283586 = weight(abstract_txt:engines in 1180) [ClassicSimilarity], result of:
           0.067283586 = score(doc=1180,freq=1.0), product of:
             0.13537198 = queryWeight, product of:
               3.2830997 = boost
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0077774217 = queryNorm
             0.4970274 = fieldWeight in 1180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.09375 = fieldNorm(doc=1180)
         0.07730691 = weight(abstract_txt:queries in 1180) [ClassicSimilarity], result of:
           0.07730691 = score(doc=1180,freq=1.0), product of:
             0.16147913 = queryWeight, product of:
               4.0658383 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0077774217 = queryNorm
             0.47874242 = fieldWeight in 1180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.09375 = fieldNorm(doc=1180)
         0.51112896 = weight(abstract_txt:hits in 1180) [ClassicSimilarity], result of:
           0.51112896 = score(doc=1180,freq=2.0), product of:
             0.51036566 = queryWeight, product of:
               8.687264 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0077774217 = queryNorm
             1.0014956 = fieldWeight in 1180, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.09375 = fieldNorm(doc=1180)
       0.2 = coord(5/25)
   

5. Spink, A.; Jansen, B.J.; Blakely, C.; Koshman, S.: ¬A study of results overlap and uniqueness among major Web search engines (2006) 0.15

   0.14865701 = sum of:
     0.14865701 = product of:
       0.5309179 = sum of:
         0.05037997 = weight(abstract_txt:overlap in 993) [ClassicSimilarity], result of:
           0.05037997 = score(doc=993,freq=4.0), product of:
             0.06633979 = queryWeight, product of:
               1.228493 = boost
               6.943297 = idf(docFreq=115, maxDocs=44218)
               0.0077774217 = queryNorm
             0.7594231 = fieldWeight in 993, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.943297 = idf(docFreq=115, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
         0.028185375 = weight(abstract_txt:across in 993) [ClassicSimilarity], result of:
           0.028185375 = score(doc=993,freq=2.0), product of:
             0.07149978 = queryWeight, product of:
               1.803653 = boost
               5.097017 = idf(docFreq=734, maxDocs=44218)
               0.0077774217 = queryNorm
             0.39420226 = fieldWeight in 993, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.097017 = idf(docFreq=734, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
         0.039583173 = weight(abstract_txt:retrieved in 993) [ClassicSimilarity], result of:
           0.039583173 = score(doc=993,freq=2.0), product of:
             0.0896663 = queryWeight, product of:
               2.0198321 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.0077774217 = queryNorm
             0.44144982 = fieldWeight in 993, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
         0.029116476 = weight(abstract_txt:google in 993) [ClassicSimilarity], result of:
           0.029116476 = score(doc=993,freq=1.0), product of:
             0.09916567 = queryWeight, product of:
               2.3748507 = boost
               5.3689504 = idf(docFreq=559, maxDocs=44218)
               0.0077774217 = queryNorm
             0.29361448 = fieldWeight in 993, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3689504 = idf(docFreq=559, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
         0.14685541 = weight(abstract_txt:engines in 993) [ClassicSimilarity], result of:
           0.14685541 = score(doc=993,freq=14.0), product of:
             0.13537198 = queryWeight, product of:
               3.2830997 = boost
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0077774217 = queryNorm
             1.0848287 = fieldWeight in 993, product of:
               3.7416575 = tf(freq=14.0), with freq of:
                 14.0 = termFreq=14.0
               5.3016257 = idf(docFreq=598, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
         0.1275499 = weight(abstract_txt:queries in 993) [ClassicSimilarity], result of:
           0.1275499 = score(doc=993,freq=8.0), product of:
             0.16147913 = queryWeight, product of:
               4.0658383 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0077774217 = queryNorm
             0.78988475 = fieldWeight in 993, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
         0.10924758 = weight(abstract_txt:yahoo in 993) [ClassicSimilarity], result of:
           0.10924758 = score(doc=993,freq=1.0), product of:
             0.30168474 = queryWeight, product of:
               5.857972 = boost
               6.6217136 = idf(docFreq=159, maxDocs=44218)
               0.0077774217 = queryNorm
             0.36212498 = fieldWeight in 993, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6217136 = idf(docFreq=159, maxDocs=44218)
               0.0546875 = fieldNorm(doc=993)
       0.28 = coord(7/25)