Document (#44290)

Author
Moffat, A.
Mackenzie, J.
Title
How much freedom does an effectiveness metric really have?
Source
Journal of the Association for Information Science and Technology. 75(2024) no.6, S.686-703
Year
2024
Abstract
It is tempting to assume that because effectiveness metrics have free choice to assign scores to search engine result pages (SERPs) there must thus be a similar degree of freedom as to the relative order that SERP pairs can be put into. In fact that second freedom is, to a considerable degree, illusory. That is because if one SERP in a pair has been given a certain score by a metric, fundamental ordering constraints in many cases then dictate that the score for the second SERP must be either not less than, or not greater than, the score assigned to the first SERP. We refer to these fixed relationships as innate pairwise SERP orderings. Our first goal in this work is to describe and defend those pairwise SERP relationship constraints, and tabulate their relative occurrence via both exhaustive and empirical experimentation. We then consider how to employ such innate pairwise relationships in IR experiments, leading to a proposal for a new measurement paradigm. Specifically, we argue that tables of results in which many different metrics are listed for champion versus challenger system comparisons should be avoided; and that instead a single metric be argued for in principled terms, with any relationships identified by that metric then reinforced via an assessment of the innate relationship as to whether other metrics are likely to yield the same system-versus-system outcome.
Content
Vgl.: https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24874. https://doi.org/10.1002/asi.24874.
Theme
Retrievalalgorithmen

Similar documents (author)

  1. Mackenzie, M.L.: ASIS to ASIS&T: A society in transition? (2006) 2.07 
    2.074138 = sum of:
  2.074138 = product of:
    4.148276 = sum of:
      4.148276 = weight(author_txt:mackenzie in 334) [ClassicSimilarity], result of:
        4.148276 = score(doc=334,freq=1.0), product of:
          0.6983451 = queryWeight, product of:
            9.504243 = idf(docFreq=8, maxDocs=44421)
            0.07347719 = queryNorm
          5.9401517 = fieldWeight in 334, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.504243 = idf(docFreq=8, maxDocs=44421)
            0.625 = fieldNorm(doc=334)
    0.5 = coord(1/2)
  2. Moffat, A.; Bell, T.A.H.: In situ generation of compressed inverted files (1995) 1.72 
    1.7217683 = sum of:
  1.7217683 = product of:
    3.4435365 = sum of:
      3.4435365 = weight(author_txt:moffat in 2716) [ClassicSimilarity], result of:
        3.4435365 = score(doc=2716,freq=1.0), product of:
          0.7157612 = queryWeight, product of:
            1.0123928 = boost
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.07347719 = queryNorm
          4.811013 = fieldWeight in 2716, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.5 = fieldNorm(doc=2716)
    0.5 = coord(1/2)
  3. Moffat, A.; Zobel, J.: Self-indexing inverted files for fast text retrieval (1996) 1.72 
    1.7217683 = sum of:
  1.7217683 = product of:
    3.4435365 = sum of:
      3.4435365 = weight(author_txt:moffat in 1009) [ClassicSimilarity], result of:
        3.4435365 = score(doc=1009,freq=1.0), product of:
          0.7157612 = queryWeight, product of:
            1.0123928 = boost
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.07347719 = queryNorm
          4.811013 = fieldWeight in 1009, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.5 = fieldNorm(doc=1009)
    0.5 = coord(1/2)
  4. Wan, R.; Moffat, A.: Block merging for off-line compression (2007) 1.72 
    1.7217683 = sum of:
  1.7217683 = product of:
    3.4435365 = sum of:
      3.4435365 = weight(author_txt:moffat in 1081) [ClassicSimilarity], result of:
        3.4435365 = score(doc=1081,freq=1.0), product of:
          0.7157612 = queryWeight, product of:
            1.0123928 = boost
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.07347719 = queryNorm
          4.811013 = fieldWeight in 1081, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.5 = fieldNorm(doc=1081)
    0.5 = coord(1/2)
  5. Moffat, A.; Isal, R.Y.K.: Word-based text compression using the Burrows-Wheeler transform (2005) 1.72 
    1.7217683 = sum of:
  1.7217683 = product of:
    3.4435365 = sum of:
      3.4435365 = weight(author_txt:moffat in 2044) [ClassicSimilarity], result of:
        3.4435365 = score(doc=2044,freq=1.0), product of:
          0.7157612 = queryWeight, product of:
            1.0123928 = boost
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.07347719 = queryNorm
          4.811013 = fieldWeight in 2044, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.622026 = idf(docFreq=7, maxDocs=44421)
            0.5 = fieldNorm(doc=2044)
    0.5 = coord(1/2)

Similar documents (content)

  1. Lewandowski, D.; Sünkler, S.; Kerkmann, F.: Are ads on Google search engine results pages labeled clearly enough? : the influence of knowledge on search ads on users' selection behaviour (2017) 0.11 
    0.1084679 = sum of:
  0.1084679 = product of:
    0.9038992 = sum of:
      0.08957204 = weight(abstract_txt:serps in 4567) [ClassicSimilarity], result of:
        0.08957204 = score(doc=4567,freq=1.0), product of:
          0.117525026 = queryWeight, product of:
            1.0485818 = boost
            9.755557 = idf(docFreq=6, maxDocs=44421)
            0.011488834 = queryNorm
          0.7621529 = fieldWeight in 4567, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.755557 = idf(docFreq=6, maxDocs=44421)
            0.078125 = fieldNorm(doc=4567)
      0.017681722 = weight(abstract_txt:that in 4567) [ClassicSimilarity], result of:
        0.017681722 = score(doc=4567,freq=3.0), product of:
          0.05525285 = queryWeight, product of:
            2.0335717 = boost
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.011488834 = queryNorm
          0.32001466 = fieldWeight in 4567, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.078125 = fieldNorm(doc=4567)
      0.79664546 = weight(abstract_txt:serp in 4567) [ClassicSimilarity], result of:
        0.79664546 = score(doc=4567,freq=2.0), product of:
          0.7276109 = queryWeight, product of:
            6.390905 = boost
            9.909708 = idf(docFreq=5, maxDocs=44421)
            0.011488834 = queryNorm
          1.0948784 = fieldWeight in 4567, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            9.909708 = idf(docFreq=5, maxDocs=44421)
            0.078125 = fieldNorm(doc=4567)
    0.12 = coord(3/25)
  2. Kenter, T.; Balog, K.; Rijke, M. de: Evaluating document filtering systems over time (2015) 0.08 
    0.08134842 = sum of:
  0.08134842 = product of:
    0.4067421 = sum of:
      0.0077731605 = weight(abstract_txt:system in 3672) [ClassicSimilarity], result of:
        0.0077731605 = score(doc=3672,freq=1.0), product of:
          0.042142686 = queryWeight, product of:
            1.0875741 = boost
            3.372775 = idf(docFreq=4140, maxDocs=44421)
            0.011488834 = queryNorm
          0.18444863 = fieldWeight in 3672, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            3.372775 = idf(docFreq=4140, maxDocs=44421)
            0.0546875 = fieldNorm(doc=3672)
      0.020211892 = weight(abstract_txt:that in 3672) [ClassicSimilarity], result of:
        0.020211892 = score(doc=3672,freq=8.0), product of:
          0.05525285 = queryWeight, product of:
            2.0335717 = boost
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.011488834 = queryNorm
          0.36580724 = fieldWeight in 3672, product of:
            2.828427 = tf(freq=8.0), with freq of:
              8.0 = termFreq=8.0
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.0546875 = fieldNorm(doc=3672)
      0.1006801 = weight(abstract_txt:metrics in 3672) [ClassicSimilarity], result of:
        0.1006801 = score(doc=3672,freq=3.0), product of:
          0.16115575 = queryWeight, product of:
            2.1267707 = boost
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.011488834 = queryNorm
          0.62473786 = fieldWeight in 3672, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.0546875 = fieldNorm(doc=3672)
      0.09645605 = weight(abstract_txt:score in 3672) [ClassicSimilarity], result of:
        0.09645605 = score(doc=3672,freq=2.0), product of:
          0.17928067 = queryWeight, product of:
            2.243182 = boost
            6.9565353 = idf(docFreq=114, maxDocs=44421)
            0.011488834 = queryNorm
          0.53801703 = fieldWeight in 3672, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            6.9565353 = idf(docFreq=114, maxDocs=44421)
            0.0546875 = fieldNorm(doc=3672)
      0.18162091 = weight(abstract_txt:metric in 3672) [ClassicSimilarity], result of:
        0.18162091 = score(doc=3672,freq=3.0), product of:
          0.2628493 = queryWeight, product of:
            3.1363213 = boost
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.011488834 = queryNorm
          0.6909697 = fieldWeight in 3672, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.0546875 = fieldNorm(doc=3672)
    0.2 = coord(5/25)
  3. Sakai, T.: On the reliability of information retrieval metrics based on graded relevance (2007) 0.08 
    0.07852654 = sum of:
  0.07852654 = product of:
    0.3926327 = sum of:
      0.011104516 = weight(abstract_txt:system in 1910) [ClassicSimilarity], result of:
        0.011104516 = score(doc=1910,freq=1.0), product of:
          0.042142686 = queryWeight, product of:
            1.0875741 = boost
            3.372775 = idf(docFreq=4140, maxDocs=44421)
            0.011488834 = queryNorm
          0.26349807 = fieldWeight in 1910, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            3.372775 = idf(docFreq=4140, maxDocs=44421)
            0.078125 = fieldNorm(doc=1910)
      0.028365985 = weight(abstract_txt:then in 1910) [ClassicSimilarity], result of:
        0.028365985 = score(doc=1910,freq=1.0), product of:
          0.07875069 = queryWeight, product of:
            1.4867055 = boost
            4.6105576 = idf(docFreq=1200, maxDocs=44421)
            0.011488834 = queryNorm
          0.3601998 = fieldWeight in 1910, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.6105576 = idf(docFreq=1200, maxDocs=44421)
            0.078125 = fieldNorm(doc=1910)
      0.017681722 = weight(abstract_txt:that in 1910) [ClassicSimilarity], result of:
        0.017681722 = score(doc=1910,freq=3.0), product of:
          0.05525285 = queryWeight, product of:
            2.0335717 = boost
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.011488834 = queryNorm
          0.32001466 = fieldWeight in 1910, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.078125 = fieldNorm(doc=1910)
      0.18568207 = weight(abstract_txt:metrics in 1910) [ClassicSimilarity], result of:
        0.18568207 = score(doc=1910,freq=5.0), product of:
          0.16115575 = queryWeight, product of:
            2.1267707 = boost
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.011488834 = queryNorm
          1.1521902 = fieldWeight in 1910, product of:
            2.236068 = tf(freq=5.0), with freq of:
              5.0 = termFreq=5.0
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.078125 = fieldNorm(doc=1910)
      0.1497984 = weight(abstract_txt:metric in 1910) [ClassicSimilarity], result of:
        0.1497984 = score(doc=1910,freq=1.0), product of:
          0.2628493 = queryWeight, product of:
            3.1363213 = boost
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.011488834 = queryNorm
          0.5699022 = fieldWeight in 1910, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.078125 = fieldNorm(doc=1910)
    0.2 = coord(5/25)
  4. Zimmerman, M.S.: Mapping literacies : comparing information horizons mapping to measures of information and health literacy (2020) 0.08 
    0.07770266 = sum of:
  0.07770266 = product of:
    0.32376108 = sum of:
      0.018607553 = weight(abstract_txt:relationship in 711) [ClassicSimilarity], result of:
        0.018607553 = score(doc=711,freq=1.0), product of:
          0.060268663 = queryWeight, product of:
            1.061935 = boost
            4.9398947 = idf(docFreq=863, maxDocs=44421)
            0.011488834 = queryNorm
          0.30874342 = fieldWeight in 711, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.9398947 = idf(docFreq=863, maxDocs=44421)
            0.0625 = fieldNorm(doc=711)
      0.03209245 = weight(abstract_txt:then in 711) [ClassicSimilarity], result of:
        0.03209245 = score(doc=711,freq=2.0), product of:
          0.07875069 = queryWeight, product of:
            1.4867055 = boost
            4.6105576 = idf(docFreq=1200, maxDocs=44421)
            0.011488834 = queryNorm
          0.40751955 = fieldWeight in 711, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            4.6105576 = idf(docFreq=1200, maxDocs=44421)
            0.0625 = fieldNorm(doc=711)
      0.025602242 = weight(abstract_txt:relationships in 711) [ClassicSimilarity], result of:
        0.025602242 = score(doc=711,freq=1.0), product of:
          0.0853456 = queryWeight, product of:
            1.5477055 = boost
            4.7997303 = idf(docFreq=993, maxDocs=44421)
            0.011488834 = queryNorm
          0.29998314 = fieldWeight in 711, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.7997303 = idf(docFreq=993, maxDocs=44421)
            0.0625 = fieldNorm(doc=711)
      0.0115496535 = weight(abstract_txt:that in 711) [ClassicSimilarity], result of:
        0.0115496535 = score(doc=711,freq=2.0), product of:
          0.05525285 = queryWeight, product of:
            2.0335717 = boost
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.011488834 = queryNorm
          0.20903271 = fieldWeight in 711, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            2.3649352 = idf(docFreq=11344, maxDocs=44421)
            0.0625 = fieldNorm(doc=711)
      0.06643164 = weight(abstract_txt:metrics in 711) [ClassicSimilarity], result of:
        0.06643164 = score(doc=711,freq=1.0), product of:
          0.16115575 = queryWeight, product of:
            2.1267707 = boost
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.011488834 = queryNorm
          0.41222012 = fieldWeight in 711, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.0625 = fieldNorm(doc=711)
      0.16947754 = weight(abstract_txt:metric in 711) [ClassicSimilarity], result of:
        0.16947754 = score(doc=711,freq=2.0), product of:
          0.2628493 = queryWeight, product of:
            3.1363213 = boost
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.011488834 = queryNorm
          0.64477074 = fieldWeight in 711, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.0625 = fieldNorm(doc=711)
    0.24 = coord(6/25)
  5. Haley, M.R.; McGee, M.K.: ¬A parametric "parent metric" approach for comparing maximum-normalized journal ranking metrics (2018) 0.08 
    0.07532578 = sum of:
  0.07532578 = product of:
    0.6277149 = sum of:
      0.13286328 = weight(abstract_txt:metrics in 4313) [ClassicSimilarity], result of:
        0.13286328 = score(doc=4313,freq=1.0), product of:
          0.16115575 = queryWeight, product of:
            2.1267707 = boost
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.011488834 = queryNorm
          0.82444024 = fieldWeight in 4313, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            6.595522 = idf(docFreq=164, maxDocs=44421)
            0.125 = fieldNorm(doc=4313)
      0.15589654 = weight(abstract_txt:score in 4313) [ClassicSimilarity], result of:
        0.15589654 = score(doc=4313,freq=1.0), product of:
          0.17928067 = queryWeight, product of:
            2.243182 = boost
            6.9565353 = idf(docFreq=114, maxDocs=44421)
            0.011488834 = queryNorm
          0.8695669 = fieldWeight in 4313, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            6.9565353 = idf(docFreq=114, maxDocs=44421)
            0.125 = fieldNorm(doc=4313)
      0.33895507 = weight(abstract_txt:metric in 4313) [ClassicSimilarity], result of:
        0.33895507 = score(doc=4313,freq=2.0), product of:
          0.2628493 = queryWeight, product of:
            3.1363213 = boost
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.011488834 = queryNorm
          1.2895415 = fieldWeight in 4313, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            7.2947483 = idf(docFreq=81, maxDocs=44421)
            0.125 = fieldNorm(doc=4313)
    0.12 = coord(3/25)