Document (#43671)

Author

Gao, R.
Ge, Y.
Sha, C.

Title

FAIR: Fairness-aware information retrieval evaluation

Source

Journal of the Association for Information Science and Technology. 73(2022) no.10, S.1461-1473

Year

2022

Abstract

With the emerging needs of creating fairness-aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness-aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR-based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

Content

https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24648. https://doi.org/10.1002/asi.24648.

Theme

Retrievalstudien

Similar documents (content)

1. An, X.; Huang, J.X.: geNov : a new metric for measuring novelty and relevancy in biomedical information retrieval (2017) 0.16

   0.16146989 = sum of:
     0.16146989 = product of:
       0.57667816 = sum of:
         0.07065596 = weight(abstract_txt:novelty in 3921) [ClassicSimilarity], result of:
           0.07065596 = score(doc=3921,freq=5.0), product of:
             0.06559149 = queryWeight, product of:
               7.7079034 = idf(docFreq=53, maxDocs=44218)
               0.0085096415 = queryNorm
             1.0772122 = fieldWeight in 3921, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.7079034 = idf(docFreq=53, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
         0.0067831455 = weight(abstract_txt:information in 3921) [ClassicSimilarity], result of:
           0.0067831455 = score(doc=3921,freq=3.0), product of:
             0.025882447 = queryWeight, product of:
               1.2563449 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0085096415 = queryNorm
             0.26207513 = fieldWeight in 3921, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
         0.024219932 = weight(abstract_txt:ranking in 3921) [ClassicSimilarity], result of:
           0.024219932 = score(doc=3921,freq=1.0), product of:
             0.06921448 = queryWeight, product of:
               1.452746 = boost
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0085096415 = queryNorm
             0.34992582 = fieldWeight in 3921, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
         0.025629995 = weight(abstract_txt:algorithm in 3921) [ClassicSimilarity], result of:
           0.025629995 = score(doc=3921,freq=1.0), product of:
             0.07187547 = queryWeight, product of:
               1.4804085 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.0085096415 = queryNorm
             0.35658893 = fieldWeight in 3921, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
         0.011656261 = weight(abstract_txt:results in 3921) [ClassicSimilarity], result of:
           0.011656261 = score(doc=3921,freq=1.0), product of:
             0.05355473 = queryWeight, product of:
               1.8071955 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0085096415 = queryNorm
             0.21765138 = fieldWeight in 3921, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
         0.11365124 = weight(abstract_txt:metrics in 3921) [ClassicSimilarity], result of:
           0.11365124 = score(doc=3921,freq=2.0), product of:
             0.19399823 = queryWeight, product of:
               3.4395776 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0085096415 = queryNorm
             0.58583647 = fieldWeight in 3921, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
         0.32408166 = weight(abstract_txt:metric in 3921) [ClassicSimilarity], result of:
           0.32408166 = score(doc=3921,freq=4.0), product of:
             0.35443646 = queryWeight, product of:
               5.694047 = boost
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.0085096415 = queryNorm
             0.9143576 = fieldWeight in 3921, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.0625 = fieldNorm(doc=3921)
       0.28 = coord(7/25)
   

2. Kenter, T.; Balog, K.; Rijke, M. de: Evaluating document filtering systems over time (2015) 0.10

   0.099238455 = sum of:
     0.099238455 = product of:
       0.49619228 = sum of:
         0.0048461133 = weight(abstract_txt:information in 2672) [ClassicSimilarity], result of:
           0.0048461133 = score(doc=2672,freq=2.0), product of:
             0.025882447 = queryWeight, product of:
               1.2563449 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0085096415 = queryNorm
             0.18723552 = fieldWeight in 2672, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2672)
         0.014423887 = weight(abstract_txt:results in 2672) [ClassicSimilarity], result of:
           0.014423887 = score(doc=2672,freq=2.0), product of:
             0.05355473 = queryWeight, product of:
               1.8071955 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0085096415 = queryNorm
             0.26932985 = fieldWeight in 2672, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2672)
         0.12179455 = weight(abstract_txt:metrics in 2672) [ClassicSimilarity], result of:
           0.12179455 = score(doc=2672,freq=3.0), product of:
             0.19399823 = queryWeight, product of:
               3.4395776 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0085096415 = queryNorm
             0.6278127 = fieldWeight in 2672, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2672)
         0.109547645 = weight(abstract_txt:utility in 2672) [ClassicSimilarity], result of:
           0.109547645 = score(doc=2672,freq=2.0), product of:
             0.22290254 = queryWeight, product of:
               4.1221023 = boost
               6.3545527 = idf(docFreq=208, maxDocs=44218)
               0.0085096415 = queryNorm
             0.49145985 = fieldWeight in 2672, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.3545527 = idf(docFreq=208, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2672)
         0.24558009 = weight(abstract_txt:metric in 2672) [ClassicSimilarity], result of:
           0.24558009 = score(doc=2672,freq=3.0), product of:
             0.35443646 = queryWeight, product of:
               5.694047 = boost
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.0085096415 = queryNorm
             0.6928748 = fieldWeight in 2672, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2672)
       0.2 = coord(5/25)
   

3. Haley, M.R.: ¬A simple paradigm for augmenting the Euclidean index to reflect journal impact and visibility (2020) 0.09

   0.09296115 = sum of:
     0.09296115 = product of:
       0.46480572 = sum of:
         0.042730577 = weight(abstract_txt:jointly in 5676) [ClassicSimilarity], result of:
           0.042730577 = score(doc=5676,freq=1.0), product of:
             0.06912325 = queryWeight, product of:
               1.0265694 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.0085096415 = queryNorm
             0.6181795 = fieldWeight in 5676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.078125 = fieldNorm(doc=5676)
         0.0048953136 = weight(abstract_txt:information in 5676) [ClassicSimilarity], result of:
           0.0048953136 = score(doc=5676,freq=1.0), product of:
             0.025882447 = queryWeight, product of:
               1.2563449 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0085096415 = queryNorm
             0.18913643 = fieldWeight in 5676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.078125 = fieldNorm(doc=5676)
         0.030274915 = weight(abstract_txt:ranking in 5676) [ClassicSimilarity], result of:
           0.030274915 = score(doc=5676,freq=1.0), product of:
             0.06921448 = queryWeight, product of:
               1.452746 = boost
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0085096415 = queryNorm
             0.43740726 = fieldWeight in 5676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.078125 = fieldNorm(doc=5676)
         0.10045446 = weight(abstract_txt:metrics in 5676) [ClassicSimilarity], result of:
           0.10045446 = score(doc=5676,freq=1.0), product of:
             0.19399823 = queryWeight, product of:
               3.4395776 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0085096415 = queryNorm
             0.5178112 = fieldWeight in 5676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.078125 = fieldNorm(doc=5676)
         0.28645045 = weight(abstract_txt:metric in 5676) [ClassicSimilarity], result of:
           0.28645045 = score(doc=5676,freq=2.0), product of:
             0.35443646 = queryWeight, product of:
               5.694047 = boost
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.0085096415 = queryNorm
             0.8081856 = fieldWeight in 5676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.078125 = fieldNorm(doc=5676)
       0.2 = coord(5/25)
   

4. Katsaros, D.; Akritidis, L.; Bozanis, P.: ¬The f index : quantifying the impact of coterminal citations on scientists' ranking (2009) 0.08

   0.084944256 = sum of:
     0.084944256 = product of:
       0.5309016 = sum of:
         0.014570327 = weight(abstract_txt:results in 2805) [ClassicSimilarity], result of:
           0.014570327 = score(doc=2805,freq=1.0), product of:
             0.05355473 = queryWeight, product of:
               1.8071955 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0085096415 = queryNorm
             0.27206424 = fieldWeight in 2805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.078125 = fieldNorm(doc=2805)
         0.10045446 = weight(abstract_txt:metrics in 2805) [ClassicSimilarity], result of:
           0.10045446 = score(doc=2805,freq=1.0), product of:
             0.19399823 = queryWeight, product of:
               3.4395776 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0085096415 = queryNorm
             0.5178112 = fieldWeight in 2805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.078125 = fieldNorm(doc=2805)
         0.11065984 = weight(abstract_txt:utility in 2805) [ClassicSimilarity], result of:
           0.11065984 = score(doc=2805,freq=1.0), product of:
             0.22290254 = queryWeight, product of:
               4.1221023 = boost
               6.3545527 = idf(docFreq=208, maxDocs=44218)
               0.0085096415 = queryNorm
             0.49644944 = fieldWeight in 2805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3545527 = idf(docFreq=208, maxDocs=44218)
               0.078125 = fieldNorm(doc=2805)
         0.305217 = weight(abstract_txt:fair in 2805) [ClassicSimilarity], result of:
           0.305217 = score(doc=2805,freq=1.0), product of:
             0.51274425 = queryWeight, product of:
               7.9080906 = boost
               7.61935 = idf(docFreq=58, maxDocs=44218)
               0.0085096415 = queryNorm
             0.5952617 = fieldWeight in 2805, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.61935 = idf(docFreq=58, maxDocs=44218)
               0.078125 = fieldNorm(doc=2805)
       0.16 = coord(4/25)
   

5. Haley, M.R.; McGee, M.K.: ¬A parametric "parent metric" approach for comparing maximum-normalized journal ranking metrics (2018) 0.08

   0.08009852 = sum of:
     0.08009852 = product of:
       0.6674877 = sum of:
         0.048439864 = weight(abstract_txt:ranking in 3313) [ClassicSimilarity], result of:
           0.048439864 = score(doc=3313,freq=1.0), product of:
             0.06921448 = queryWeight, product of:
               1.452746 = boost
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0085096415 = queryNorm
             0.69985163 = fieldWeight in 3313, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.125 = fieldNorm(doc=3313)
         0.16072713 = weight(abstract_txt:metrics in 3313) [ClassicSimilarity], result of:
           0.16072713 = score(doc=3313,freq=1.0), product of:
             0.19399823 = queryWeight, product of:
               3.4395776 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0085096415 = queryNorm
             0.8284979 = fieldWeight in 3313, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.125 = fieldNorm(doc=3313)
         0.45832068 = weight(abstract_txt:metric in 3313) [ClassicSimilarity], result of:
           0.45832068 = score(doc=3313,freq=2.0), product of:
             0.35443646 = queryWeight, product of:
               5.694047 = boost
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.0085096415 = queryNorm
             1.2930969 = fieldWeight in 3313, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.125 = fieldNorm(doc=3313)
       0.12 = coord(3/25)