Document (#37394)

Author

Bilal, D.

Title

Ranking, relevance judgment, and precision of information retrieval on children's queries : evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids

Source

Journal of the American Society for Information Science and Technology. 63(2012) no.9, S.1879-1896

Year

2012

Abstract

This study employed benchmarking and intellectual relevance judgment in evaluating Google, Yahoo!, Bing, Yahoo! Kids, and Ask Kids on 30 queries that children formulated to find information for specific tasks. Retrieved hits on given queries were benchmarked to Google's and Yahoo! Kids' top-five ranked hits retrieved. Relevancy of hits was judged on a graded scale; precision was calculated using the precision-at-ten metric (P@10). Yahoo! and Bing produced a similar percentage in hit overlap with Google (nearly 30%), but differed in the ranking of hits. Ask Kids retrieved 11% in hit overlap with Google versus 3% by Yahoo! Kids. The engines retrieved 26 hits across query clusters that overlapped with Yahoo! Kids' top-five ranked hits. Precision (P) that the engines produced across the queries was P = 0.48 for relevant hits, and P = 0.28 for partially relevant hits. Precision by Ask Kids was P = 0.44 for relevant hits versus P = 0.21 by Yahoo! Kids. Bing produced the highest total precision (TP) of relevant hits (TP = 0.86) across the queries, and Yahoo! Kids yielded the lowest (TP = 0.47). Average precision (AP) of relevant hits was AP = 0.56 by leading engines versus AP = 0.29 by small engines. In contrast, average precision of partially relevant hits was AP = 0.83 by small engines versus AP = 0.33 by leading engines. Average precision of relevant hits across the engines was highest on two-word queries and lowest on one-word queries. Google performed best on natural language queries; Bing did the same (P = 0.69) on two-word queries. The findings have implications for search engine ranking algorithms, relevance theory, search engine design, research design, and information literacy.

Theme

Retrievalalgorithmen
Suchmaschinen

Object

Google
Yahoo!
Bing
Yahoo! Kids
ask Kids

Similar documents (author)

1. Bilal, D.: Children's use of the Yahooligans! Web search engine : I. Cognitive, physical, and affective behaviors on fact-based search tasks (2000) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bilal in 5776) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 5776, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=5776)
   

2. Bilal, D.: Children's use of the Yahooligans! Web search engine : II. Cognitive and physical behaviors on research tasks (2001) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bilal in 6583) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 6583, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=6583)
   

3. Bilal, D.: Web search engines for children : a comparative study and performance evaluation of Yahooligans!, AskJeeves for Kids, and Super Snooper (1999) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bilal in 544) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 544, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=544)
   

4. Bilal, D.: Children's use of the Yahooligans! Web search engine : III. Cognitive and physical behaviors on fully self-generated search tasks (2002) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bilal in 228) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 228, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=228)
   

5. Bilal, D.: Percentile ranks and the integrated impact indicator (I3) (2012) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bilal in 1392) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 1392, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=1392)
   

Similar documents (content)

1. Bilal, D.: Web search engines for children : a comparative study and performance evaluation of Yahooligans!, AskJeeves for Kids, and Super Snooper (1999) 0.41

   0.4115895 = sum of:
     0.4115895 = product of:
       1.2862172 = sum of:
         0.027529689 = weight(abstract_txt:highest in 544) [ClassicSimilarity], result of:
           0.027529689 = score(doc=544,freq=1.0), product of:
             0.06438135 = queryWeight, product of:
               1.2105409 = boost
               6.8416553 = idf(docFreq=128, maxDocs=44421)
               0.007773551 = queryNorm
             0.42760345 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8416553 = idf(docFreq=128, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.02883196 = weight(abstract_txt:overlap in 544) [ClassicSimilarity], result of:
           0.02883196 = score(doc=544,freq=1.0), product of:
             0.06639601 = queryWeight, product of:
               1.2293354 = boost
               6.9478774 = idf(docFreq=115, maxDocs=44421)
               0.007773551 = queryNorm
             0.43424234 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9478774 = idf(docFreq=115, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.021844449 = weight(abstract_txt:relevance in 544) [ClassicSimilarity], result of:
           0.021844449 = score(doc=544,freq=2.0), product of:
             0.050134927 = queryWeight, product of:
               1.3083237 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.007773551 = queryNorm
             0.43571317 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.03192407 = weight(abstract_txt:retrieved in 544) [ClassicSimilarity], result of:
           0.03192407 = score(doc=544,freq=1.0), product of:
             0.089532636 = queryWeight, product of:
               2.0188549 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.007773551 = queryNorm
             0.35656348 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.07713067 = weight(abstract_txt:engines in 544) [ClassicSimilarity], result of:
           0.07713067 = score(doc=544,freq=3.0), product of:
             0.13469751 = queryWeight, product of:
               3.2757685 = boost
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.007773551 = queryNorm
             0.57262135 = fieldWeight in 544, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.05136281 = weight(abstract_txt:queries in 544) [ClassicSimilarity], result of:
           0.05136281 = score(doc=544,freq=1.0), product of:
             0.16108777 = queryWeight, product of:
               4.061972 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.007773551 = queryNorm
             0.31884983 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.24127325 = weight(abstract_txt:hits in 544) [ClassicSimilarity], result of:
           0.24127325 = score(doc=544,freq=1.0), product of:
             0.51074386 = queryWeight, product of:
               8.6927595 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.007773551 = queryNorm
             0.4723958 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
         0.80632037 = weight(abstract_txt:kids in 544) [ClassicSimilarity], result of:
           0.80632037 = score(doc=544,freq=5.0), product of:
             0.6314833 = queryWeight, product of:
               8.891227 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.007773551 = queryNorm
             1.2768673 = fieldWeight in 544, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.0625 = fieldNorm(doc=544)
       0.32 = coord(8/25)
   

2. Eastman, C.M.: 30,000 hits may be better than 300 : precision anomalies in Internet searches (2002) 0.22

   0.22244263 = sum of:
     0.22244263 = product of:
       0.79443794 = sum of:
         0.032296482 = weight(abstract_txt:ranked in 231) [ClassicSimilarity], result of:
           0.032296482 = score(doc=231,freq=2.0), product of:
             0.056839824 = queryWeight, product of:
               1.1374328 = boost
               6.428468 = idf(docFreq=194, maxDocs=44421)
               0.007773551 = queryNorm
             0.56820166 = fieldWeight in 231, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.428468 = idf(docFreq=194, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
         0.015446358 = weight(abstract_txt:relevance in 231) [ClassicSimilarity], result of:
           0.015446358 = score(doc=231,freq=1.0), product of:
             0.050134927 = queryWeight, product of:
               1.3083237 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.007773551 = queryNorm
             0.30809575 = fieldWeight in 231, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
         0.03915253 = weight(abstract_txt:ranking in 231) [ClassicSimilarity], result of:
           0.03915253 = score(doc=231,freq=3.0), product of:
             0.064623296 = queryWeight, product of:
               1.4853868 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.007773551 = queryNorm
             0.6058578 = fieldWeight in 231, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
         0.044531412 = weight(abstract_txt:engines in 231) [ClassicSimilarity], result of:
           0.044531412 = score(doc=231,freq=1.0), product of:
             0.13469751 = queryWeight, product of:
               3.2757685 = boost
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.007773551 = queryNorm
             0.3306031 = fieldWeight in 231, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
         0.072637975 = weight(abstract_txt:queries in 231) [ClassicSimilarity], result of:
           0.072637975 = score(doc=231,freq=2.0), product of:
             0.16108777 = queryWeight, product of:
               4.061972 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.007773551 = queryNorm
             0.45092174 = fieldWeight in 231, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
         0.17247567 = weight(abstract_txt:precision in 231) [ClassicSimilarity], result of:
           0.17247567 = score(doc=231,freq=7.0), product of:
             0.18883505 = queryWeight, product of:
               4.3979163 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.007773551 = queryNorm
             0.9133668 = fieldWeight in 231, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
         0.41789752 = weight(abstract_txt:hits in 231) [ClassicSimilarity], result of:
           0.41789752 = score(doc=231,freq=3.0), product of:
             0.51074386 = queryWeight, product of:
               8.6927595 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.007773551 = queryNorm
             0.8182135 = fieldWeight in 231, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.0625 = fieldNorm(doc=231)
       0.28 = coord(7/25)
   

3. Li, L.; Shang, Y.; Zhang, W.: Improvement of HITS-based algorithms on Web documents 0.18

   0.18125588 = sum of:
     0.18125588 = product of:
       1.1328492 = sum of:
         0.033442345 = weight(abstract_txt:relevance in 3514) [ClassicSimilarity], result of:
           0.033442345 = score(doc=3514,freq=3.0), product of:
             0.050134927 = queryWeight, product of:
               1.3083237 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.007773551 = queryNorm
             0.66704684 = fieldWeight in 3514, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=3514)
         0.06420351 = weight(abstract_txt:queries in 3514) [ClassicSimilarity], result of:
           0.06420351 = score(doc=3514,freq=1.0), product of:
             0.16108777 = queryWeight, product of:
               4.061972 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.007773551 = queryNorm
             0.39856228 = fieldWeight in 3514, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.078125 = fieldNorm(doc=3514)
         0.0814871 = weight(abstract_txt:precision in 3514) [ClassicSimilarity], result of:
           0.0814871 = score(doc=3514,freq=1.0), product of:
             0.18883505 = queryWeight, product of:
               4.3979163 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.007773551 = queryNorm
             0.43152526 = fieldWeight in 3514, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.078125 = fieldNorm(doc=3514)
         0.95371634 = weight(abstract_txt:hits in 3514) [ClassicSimilarity], result of:
           0.95371634 = score(doc=3514,freq=10.0), product of:
             0.51074386 = queryWeight, product of:
               8.6927595 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.007773551 = queryNorm
             1.8673085 = fieldWeight in 3514, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.078125 = fieldNorm(doc=3514)
       0.16 = coord(4/25)
   

4. Bar-Ilan, J.: Evaluating the stability of the search tools Hotbot and Snap : a case study (2000) 0.15

   0.14935866 = sum of:
     0.14935866 = product of:
       0.74679327 = sum of:
         0.04324794 = weight(abstract_txt:overlap in 2180) [ClassicSimilarity], result of:
           0.04324794 = score(doc=2180,freq=1.0), product of:
             0.06639601 = queryWeight, product of:
               1.2293354 = boost
               6.9478774 = idf(docFreq=115, maxDocs=44421)
               0.007773551 = queryNorm
             0.6513635 = fieldWeight in 2180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9478774 = idf(docFreq=115, maxDocs=44421)
               0.09375 = fieldNorm(doc=2180)
         0.047886103 = weight(abstract_txt:retrieved in 2180) [ClassicSimilarity], result of:
           0.047886103 = score(doc=2180,freq=1.0), product of:
             0.089532636 = queryWeight, product of:
               2.0188549 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.007773551 = queryNorm
             0.53484523 = fieldWeight in 2180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.09375 = fieldNorm(doc=2180)
         0.06679712 = weight(abstract_txt:engines in 2180) [ClassicSimilarity], result of:
           0.06679712 = score(doc=2180,freq=1.0), product of:
             0.13469751 = queryWeight, product of:
               3.2757685 = boost
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.007773551 = queryNorm
             0.49590462 = fieldWeight in 2180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.09375 = fieldNorm(doc=2180)
         0.07704421 = weight(abstract_txt:queries in 2180) [ClassicSimilarity], result of:
           0.07704421 = score(doc=2180,freq=1.0), product of:
             0.16108777 = queryWeight, product of:
               4.061972 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.007773551 = queryNorm
             0.47827476 = fieldWeight in 2180, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.09375 = fieldNorm(doc=2180)
         0.5118179 = weight(abstract_txt:hits in 2180) [ClassicSimilarity], result of:
           0.5118179 = score(doc=2180,freq=2.0), product of:
             0.51074386 = queryWeight, product of:
               8.6927595 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.007773551 = queryNorm
             1.0021029 = fieldWeight in 2180, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.09375 = fieldNorm(doc=2180)
       0.2 = coord(5/25)
   

5. Spink, A.; Jansen, B.J.; Blakely, C.; Koshman, S.: ¬A study of results overlap and uniqueness among major Web search engines (2006) 0.15

   0.148179 = sum of:
     0.148179 = product of:
       0.5292107 = sum of:
         0.05045593 = weight(abstract_txt:overlap in 1993) [ClassicSimilarity], result of:
           0.05045593 = score(doc=1993,freq=4.0), product of:
             0.06639601 = queryWeight, product of:
               1.2293354 = boost
               6.9478774 = idf(docFreq=115, maxDocs=44421)
               0.007773551 = queryNorm
             0.7599241 = fieldWeight in 1993, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.9478774 = idf(docFreq=115, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
         0.02791383 = weight(abstract_txt:across in 1993) [ClassicSimilarity], result of:
           0.02791383 = score(doc=1993,freq=2.0), product of:
             0.071028635 = queryWeight, product of:
               1.7981716 = boost
               5.081394 = idf(docFreq=749, maxDocs=44421)
               0.007773551 = queryNorm
             0.39299402 = fieldWeight in 1993, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.081394 = idf(docFreq=749, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
         0.039504018 = weight(abstract_txt:retrieved in 1993) [ClassicSimilarity], result of:
           0.039504018 = score(doc=1993,freq=2.0), product of:
             0.089532636 = queryWeight, product of:
               2.0188549 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.007773551 = queryNorm
             0.44122478 = fieldWeight in 1993, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
         0.029004036 = weight(abstract_txt:google in 1993) [ClassicSimilarity], result of:
           0.029004036 = score(doc=1993,freq=1.0), product of:
             0.09889465 = queryWeight, product of:
               2.3722246 = boost
               5.3628736 = idf(docFreq=565, maxDocs=44421)
               0.007773551 = queryNorm
             0.29328215 = fieldWeight in 1993, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3628736 = idf(docFreq=565, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
         0.14579365 = weight(abstract_txt:engines in 1993) [ClassicSimilarity], result of:
           0.14579365 = score(doc=1993,freq=14.0), product of:
             0.13469751 = queryWeight, product of:
               3.2757685 = boost
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.007773551 = queryNorm
             1.0823781 = fieldWeight in 1993, product of:
               3.7416575 = tf(freq=14.0), with freq of:
                 14.0 = termFreq=14.0
               5.2896495 = idf(docFreq=608, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
         0.12711646 = weight(abstract_txt:queries in 1993) [ClassicSimilarity], result of:
           0.12711646 = score(doc=1993,freq=8.0), product of:
             0.16108777 = queryWeight, product of:
               4.061972 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.007773551 = queryNorm
             0.78911304 = fieldWeight in 1993, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
         0.10942281 = weight(abstract_txt:yahoo in 1993) [ClassicSimilarity], result of:
           0.10942281 = score(doc=1993,freq=1.0), product of:
             0.30195978 = queryWeight, product of:
               5.8621774 = boost
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.007773551 = queryNorm
             0.36237544 = fieldWeight in 1993, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1993)
       0.28 = coord(7/25)