Document (#43671)

Author

Gao, R.
Ge, Y.
Sha, C.

Title

FAIR: Fairness-aware information retrieval evaluation

Source

Journal of the Association for Information Science and Technology. 73(2022) no.10, S.1461-1473

Year

2022

Abstract

With the emerging needs of creating fairness-aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness-aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR-based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

Content

https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24648. https://doi.org/10.1002/asi.24648.

Theme

Retrievalstudien

Similar documents (content)

1. An, X.; Huang, J.X.: geNov : a new metric for measuring novelty and relevancy in biomedical information retrieval (2017) 0.16

   0.16152723 = sum of:
     0.16152723 = product of:
       0.57688296 = sum of:
         0.07089225 = weight(abstract_txt:novelty in 4921) [ClassicSimilarity], result of:
           0.07089225 = score(doc=4921,freq=5.0), product of:
             0.06592862 = queryWeight, product of:
               7.694134 = idf(docFreq=54, maxDocs=44421)
               0.008568686 = queryNorm
             1.0752879 = fieldWeight in 4921, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.694134 = idf(docFreq=54, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
         0.0068250936 = weight(abstract_txt:information in 4921) [ClassicSimilarity], result of:
           0.0068250936 = score(doc=4921,freq=3.0), product of:
             0.026064545 = queryWeight, product of:
               1.2575291 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.008568686 = queryNorm
             0.26185355 = fieldWeight in 4921, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
         0.024403641 = weight(abstract_txt:ranking in 4921) [ClassicSimilarity], result of:
           0.024403641 = score(doc=4921,freq=1.0), product of:
             0.06976612 = queryWeight, product of:
               1.4547899 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.008568686 = queryNorm
             0.34979215 = fieldWeight in 4921, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
         0.025848482 = weight(abstract_txt:algorithm in 4921) [ClassicSimilarity], result of:
           0.025848482 = score(doc=4921,freq=1.0), product of:
             0.07249335 = queryWeight, product of:
               1.482952 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.008568686 = queryNorm
             0.35656348 = fieldWeight in 4921, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
         0.01171993 = weight(abstract_txt:results in 4921) [ClassicSimilarity], result of:
           0.01171993 = score(doc=4921,freq=1.0), product of:
             0.053905718 = queryWeight, product of:
               1.8084663 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.008568686 = queryNorm
             0.21741535 = fieldWeight in 4921, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
         0.1129684 = weight(abstract_txt:metrics in 4921) [ClassicSimilarity], result of:
           0.1129684 = score(doc=4921,freq=2.0), product of:
             0.19378172 = queryWeight, product of:
               3.4288573 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.008568686 = queryNorm
             0.5829673 = fieldWeight in 4921, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
         0.32422516 = weight(abstract_txt:metric in 4921) [ClassicSimilarity], result of:
           0.32422516 = score(doc=4921,freq=4.0), product of:
             0.35557103 = queryWeight, product of:
               5.688553 = boost
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.008568686 = queryNorm
             0.91184354 = fieldWeight in 4921, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.0625 = fieldNorm(doc=4921)
       0.28 = coord(7/25)
   

2. Kenter, T.; Balog, K.; Rijke, M. de: Evaluating document filtering systems over time (2015) 0.10

   0.09932508 = sum of:
     0.09932508 = product of:
       0.49662542 = sum of:
         0.0048760823 = weight(abstract_txt:information in 3672) [ClassicSimilarity], result of:
           0.0048760823 = score(doc=3672,freq=2.0), product of:
             0.026064545 = queryWeight, product of:
               1.2575291 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.008568686 = queryNorm
             0.18707721 = fieldWeight in 3672, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3672)
         0.014502673 = weight(abstract_txt:results in 3672) [ClassicSimilarity], result of:
           0.014502673 = score(doc=3672,freq=2.0), product of:
             0.053905718 = queryWeight, product of:
               1.8084663 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.008568686 = queryNorm
             0.26903775 = fieldWeight in 3672, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3672)
         0.12106278 = weight(abstract_txt:metrics in 3672) [ClassicSimilarity], result of:
           0.12106278 = score(doc=3672,freq=3.0), product of:
             0.19378172 = queryWeight, product of:
               3.4288573 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.008568686 = queryNorm
             0.62473786 = fieldWeight in 3672, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3672)
         0.110495105 = weight(abstract_txt:utility in 3672) [ClassicSimilarity], result of:
           0.110495105 = score(doc=3672,freq=2.0), product of:
             0.22483718 = queryWeight, product of:
               4.1293535 = boost
               6.35436 = idf(docFreq=209, maxDocs=44421)
               0.008568686 = queryNorm
             0.49144498 = fieldWeight in 3672, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.35436 = idf(docFreq=209, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3672)
         0.24568881 = weight(abstract_txt:metric in 3672) [ClassicSimilarity], result of:
           0.24568881 = score(doc=3672,freq=3.0), product of:
             0.35557103 = queryWeight, product of:
               5.688553 = boost
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.008568686 = queryNorm
             0.6909697 = fieldWeight in 3672, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3672)
       0.2 = coord(5/25)
   

3. Haley, M.R.: ¬A simple paradigm for augmenting the Euclidean index to reflect journal impact and visibility (2020) 0.09

   0.09293412 = sum of:
     0.09293412 = product of:
       0.46467057 = sum of:
         0.042812314 = weight(abstract_txt:jointly in 676) [ClassicSimilarity], result of:
           0.042812314 = score(doc=676,freq=1.0), product of:
             0.06941243 = queryWeight, product of:
               1.026081 = boost
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.008568686 = queryNorm
             0.61678165 = fieldWeight in 676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.078125 = fieldNorm(doc=676)
         0.004925587 = weight(abstract_txt:information in 676) [ClassicSimilarity], result of:
           0.004925587 = score(doc=676,freq=1.0), product of:
             0.026064545 = queryWeight, product of:
               1.2575291 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.008568686 = queryNorm
             0.18897653 = fieldWeight in 676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.078125 = fieldNorm(doc=676)
         0.03050455 = weight(abstract_txt:ranking in 676) [ClassicSimilarity], result of:
           0.03050455 = score(doc=676,freq=1.0), product of:
             0.06976612 = queryWeight, product of:
               1.4547899 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.008568686 = queryNorm
             0.43724018 = fieldWeight in 676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.078125 = fieldNorm(doc=676)
         0.0998509 = weight(abstract_txt:metrics in 676) [ClassicSimilarity], result of:
           0.0998509 = score(doc=676,freq=1.0), product of:
             0.19378172 = queryWeight, product of:
               3.4288573 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.008568686 = queryNorm
             0.5152751 = fieldWeight in 676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.078125 = fieldNorm(doc=676)
         0.28657722 = weight(abstract_txt:metric in 676) [ClassicSimilarity], result of:
           0.28657722 = score(doc=676,freq=2.0), product of:
             0.35557103 = queryWeight, product of:
               5.688553 = boost
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.008568686 = queryNorm
             0.8059634 = fieldWeight in 676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.078125 = fieldNorm(doc=676)
       0.2 = coord(5/25)
   

4. Katsaros, D.; Akritidis, L.; Bozanis, P.: ¬The f index : quantifying the impact of coterminal citations on scientists' ranking (2009) 0.08

   0.084884755 = sum of:
     0.084884755 = product of:
       0.53052974 = sum of:
         0.014649914 = weight(abstract_txt:results in 3805) [ClassicSimilarity], result of:
           0.014649914 = score(doc=3805,freq=1.0), product of:
             0.053905718 = queryWeight, product of:
               1.8084663 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.008568686 = queryNorm
             0.2717692 = fieldWeight in 3805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.078125 = fieldNorm(doc=3805)
         0.0998509 = weight(abstract_txt:metrics in 3805) [ClassicSimilarity], result of:
           0.0998509 = score(doc=3805,freq=1.0), product of:
             0.19378172 = queryWeight, product of:
               3.4288573 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.008568686 = queryNorm
             0.5152751 = fieldWeight in 3805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.078125 = fieldNorm(doc=3805)
         0.11161691 = weight(abstract_txt:utility in 3805) [ClassicSimilarity], result of:
           0.11161691 = score(doc=3805,freq=1.0), product of:
             0.22483718 = queryWeight, product of:
               4.1293535 = boost
               6.35436 = idf(docFreq=209, maxDocs=44421)
               0.008568686 = queryNorm
             0.4964344 = fieldWeight in 3805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.35436 = idf(docFreq=209, maxDocs=44421)
               0.078125 = fieldNorm(doc=3805)
         0.30441198 = weight(abstract_txt:fair in 3805) [ClassicSimilarity], result of:
           0.30441198 = score(doc=3805,freq=1.0), product of:
             0.5133292 = queryWeight, product of:
               7.8923435 = boost
               7.590594 = idf(docFreq=60, maxDocs=44421)
               0.008568686 = queryNorm
             0.59301513 = fieldWeight in 3805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.590594 = idf(docFreq=60, maxDocs=44421)
               0.078125 = fieldNorm(doc=3805)
       0.16 = coord(4/25)
   

5. Yi, K.: Harnessing collective intelligence in social tagging using Delicious (2012) 0.08

   0.08009039 = sum of:
     0.08009039 = product of:
       0.33370996 = sum of:
         0.0039404696 = weight(abstract_txt:information in 1515) [ClassicSimilarity], result of:
           0.0039404696 = score(doc=1515,freq=1.0), product of:
             0.026064545 = queryWeight, product of:
               1.2575291 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.008568686 = queryNorm
             0.15118122 = fieldWeight in 1515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0625 = fieldNorm(doc=1515)
         0.034511957 = weight(abstract_txt:ranking in 1515) [ClassicSimilarity], result of:
           0.034511957 = score(doc=1515,freq=2.0), product of:
             0.06976612 = queryWeight, product of:
               1.4547899 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.008568686 = queryNorm
             0.4946808 = fieldWeight in 1515, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0625 = fieldNorm(doc=1515)
         0.01171993 = weight(abstract_txt:results in 1515) [ClassicSimilarity], result of:
           0.01171993 = score(doc=1515,freq=1.0), product of:
             0.053905718 = queryWeight, product of:
               1.8084663 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.008568686 = queryNorm
             0.21741535 = fieldWeight in 1515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=1515)
         0.041544303 = weight(abstract_txt:evaluating in 1515) [ClassicSimilarity], result of:
           0.041544303 = score(doc=1515,freq=1.0), product of:
             0.113862194 = queryWeight, product of:
               2.2762156 = boost
               5.8378363 = idf(docFreq=351, maxDocs=44421)
               0.008568686 = queryNorm
             0.36486477 = fieldWeight in 1515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.8378363 = idf(docFreq=351, maxDocs=44421)
               0.0625 = fieldNorm(doc=1515)
         0.07988072 = weight(abstract_txt:metrics in 1515) [ClassicSimilarity], result of:
           0.07988072 = score(doc=1515,freq=1.0), product of:
             0.19378172 = queryWeight, product of:
               3.4288573 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.008568686 = queryNorm
             0.41222012 = fieldWeight in 1515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.0625 = fieldNorm(doc=1515)
         0.16211258 = weight(abstract_txt:metric in 1515) [ClassicSimilarity], result of:
           0.16211258 = score(doc=1515,freq=1.0), product of:
             0.35557103 = queryWeight, product of:
               5.688553 = boost
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.008568686 = queryNorm
             0.45592177 = fieldWeight in 1515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.0625 = fieldNorm(doc=1515)
       0.24 = coord(6/25)