Document (#43647)

Author

Purpura, A.
Silvello, G.
Susto, G.A.

Title

Learning to rank from relevance judgments distributions

Source

Journal of the Association for Information Science and Technology. 73(2022) no.9, S.1236-1252

Year

2022

Abstract

LEarning TO Rank (LETOR) algorithms are usually trained on annotated corpora where a single relevance label is assigned to each available document-topic pair. Within the Cranfield framework, relevance labels result from merging either multiple expertly curated or crowdsourced human assessments. In this paper, we explore how to train LETOR models with relevance judgments distributions (either real or synthetically generated) assigned to document-topic pairs instead of single-valued relevance labels. We propose five new probabilistic loss functions to deal with the higher expressive power provided by relevance judgments distributions and show how they can be applied both to neural and gradient boosting machine (GBM) architectures. Moreover, we show how training a LETOR model on a sampled version of the relevance judgments from certain probability distributions can improve its performance when relying either on traditional or probabilistic loss functions. Finally, we validate our hypothesis on real-world crowdsourced relevance judgments distributions. Overall, we observe that relying on relevance judgments distributions to train different LETOR models can boost their performance and even outperform strong baselines such as LambdaMART on several test collections.

Content

https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24629. Vgl.: https://doi.org/10.1002/asi.24629.

Theme

Retrievalalgorithmen

Similar documents (author)

1. Silvello, G.: Learning to cite framework : how to automatically construct citations for hierarchical data (2017) 6.01

   6.0137663 = sum of:
     6.0137663 = weight(author_txt:silvello in 4647) [ClassicSimilarity], result of:
       6.0137663 = fieldWeight in 4647, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.625 = fieldNorm(doc=4647)
   

2. Silvello, G.: Theory and practice of data citation (2018) 6.01

   6.0137663 = sum of:
     6.0137663 = weight(author_txt:silvello in 6) [ClassicSimilarity], result of:
       6.0137663 = fieldWeight in 6, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.625 = fieldNorm(doc=6)
   

3. Ferro, N.; Silvello, G.: NESTOR: a formal model for digital archives (2013) 4.81

   4.811013 = sum of:
     4.811013 = weight(author_txt:silvello in 3707) [ClassicSimilarity], result of:
       4.811013 = fieldWeight in 3707, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.5 = fieldNorm(doc=3707)
   

4. Ferro, N.; Silvello, G.: Toward an anatomy of IR system component performances (2018) 4.81

   4.811013 = sum of:
     4.811013 = weight(author_txt:silvello in 35) [ClassicSimilarity], result of:
       4.811013 = fieldWeight in 35, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.5 = fieldNorm(doc=35)
   

5. Ferro, N.; Silvello, G.; Keskustalo, H.; Pirkola, A.; Järvelin, K.: ¬The twist measure for IR evaluation : taking user's effort into account (2016) 3.01

   3.0068831 = sum of:
     3.0068831 = weight(author_txt:silvello in 3771) [ClassicSimilarity], result of:
       3.0068831 = fieldWeight in 3771, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.3125 = fieldNorm(doc=3771)
   

Similar documents (content)

1. Bodoff, D.; Wu, B.; Wong, K.Y.M.: Relevance data for language models using maximum likelihood (2003) 0.25

   0.25450698 = sum of:
     0.25450698 = product of:
       1.0604458 = sum of:
         0.034840833 = weight(abstract_txt:document in 2822) [ClassicSimilarity], result of:
           0.034840833 = score(doc=2822,freq=2.0), product of:
             0.061196286 = queryWeight, product of:
               1.0076127 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.014143414 = queryNorm
             0.5693292 = fieldWeight in 2822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.030676087 = weight(abstract_txt:performance in 2822) [ClassicSimilarity], result of:
           0.030676087 = score(doc=2822,freq=1.0), product of:
             0.07082871 = queryWeight, product of:
               1.0840162 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.014143414 = queryNorm
             0.43310243 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.030743208 = weight(abstract_txt:models in 2822) [ClassicSimilarity], result of:
           0.030743208 = score(doc=2822,freq=1.0), product of:
             0.07093199 = queryWeight, product of:
               1.0848062 = boost
               4.623126 = idf(docFreq=1185, maxDocs=44421)
               0.014143414 = queryNorm
             0.43341807 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.623126 = idf(docFreq=1185, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.2969023 = weight(abstract_txt:distributions in 2822) [ClassicSimilarity], result of:
           0.2969023 = score(doc=2822,freq=1.0), product of:
             0.46393684 = queryWeight, product of:
               4.805307 = boost
               6.82627 = idf(docFreq=130, maxDocs=44421)
               0.014143414 = queryNorm
             0.6399628 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.82627 = idf(docFreq=130, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.43009868 = weight(abstract_txt:judgments in 2822) [ClassicSimilarity], result of:
           0.43009868 = score(doc=2822,freq=2.0), product of:
             0.47143146 = queryWeight, product of:
               4.843965 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.014143414 = queryNorm
             0.91232497 = fieldWeight in 2822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.23718461 = weight(abstract_txt:relevance in 2822) [ClassicSimilarity], result of:
           0.23718461 = score(doc=2822,freq=2.0), product of:
             0.36290634 = queryWeight, product of:
               5.2051663 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.014143414 = queryNorm
             0.65356976 = fieldWeight in 2822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
       0.24 = coord(6/25)
   

2. Spina, D.; Trippas, J.R.; Cavedon, L.; Sanderson, M.: Extracting audio summaries to support effective spoken document search (2017) 0.20

   0.19523996 = sum of:
     0.19523996 = product of:
       0.81349987 = sum of:
         0.029034028 = weight(abstract_txt:document in 4788) [ClassicSimilarity], result of:
           0.029034028 = score(doc=4788,freq=2.0), product of:
             0.061196286 = queryWeight, product of:
               1.0076127 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.014143414 = queryNorm
             0.47444102 = fieldWeight in 4788, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=4788)
         0.022101069 = weight(abstract_txt:show in 4788) [ClassicSimilarity], result of:
           0.022101069 = score(doc=4788,freq=1.0), product of:
             0.06427948 = queryWeight, product of:
               1.0326836 = boost
               4.400995 = idf(docFreq=1480, maxDocs=44421)
               0.014143414 = queryNorm
             0.34382772 = fieldWeight in 4788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.400995 = idf(docFreq=1480, maxDocs=44421)
               0.078125 = fieldNorm(doc=4788)
         0.088678956 = weight(abstract_txt:curated in 4788) [ClassicSimilarity], result of:
           0.088678956 = score(doc=4788,freq=1.0), product of:
             0.12882513 = queryWeight, product of:
               1.0337521 = boost
               8.811096 = idf(docFreq=17, maxDocs=44421)
               0.014143414 = queryNorm
             0.6883669 = fieldWeight in 4788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.811096 = idf(docFreq=17, maxDocs=44421)
               0.078125 = fieldNorm(doc=4788)
         0.22259393 = weight(abstract_txt:crowdsourced in 4788) [ClassicSimilarity], result of:
           0.22259393 = score(doc=4788,freq=1.0), product of:
             0.29978216 = queryWeight, product of:
               2.2301495 = boost
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.014143414 = queryNorm
             0.74251896 = fieldWeight in 4788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.078125 = fieldNorm(doc=4788)
         0.2534381 = weight(abstract_txt:judgments in 4788) [ClassicSimilarity], result of:
           0.2534381 = score(doc=4788,freq=1.0), product of:
             0.47143146 = queryWeight, product of:
               4.843965 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.014143414 = queryNorm
             0.53759265 = fieldWeight in 4788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.078125 = fieldNorm(doc=4788)
         0.19765384 = weight(abstract_txt:relevance in 4788) [ClassicSimilarity], result of:
           0.19765384 = score(doc=4788,freq=2.0), product of:
             0.36290634 = queryWeight, product of:
               5.2051663 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.014143414 = queryNorm
             0.5446415 = fieldWeight in 4788, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=4788)
       0.24 = coord(6/25)
   

3. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.17

   0.16751543 = sum of:
     0.16751543 = product of:
       0.8375771 = sum of:
         0.032848254 = weight(abstract_txt:document in 3591) [ClassicSimilarity], result of:
           0.032848254 = score(doc=3591,freq=4.0), product of:
             0.061196286 = queryWeight, product of:
               1.0076127 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.014143414 = queryNorm
             0.53676873 = fieldWeight in 3591, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.017680854 = weight(abstract_txt:show in 3591) [ClassicSimilarity], result of:
           0.017680854 = score(doc=3591,freq=1.0), product of:
             0.06427948 = queryWeight, product of:
               1.0326836 = boost
               4.400995 = idf(docFreq=1480, maxDocs=44421)
               0.014143414 = queryNorm
             0.27506217 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.400995 = idf(docFreq=1480, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.03786295 = weight(abstract_txt:topic in 3591) [ClassicSimilarity], result of:
           0.03786295 = score(doc=3591,freq=2.0), product of:
             0.08476239 = queryWeight, product of:
               1.185858 = boost
               5.053779 = idf(docFreq=770, maxDocs=44421)
               0.014143414 = queryNorm
             0.44669518 = fieldWeight in 3591, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.053779 = idf(docFreq=770, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.45336384 = weight(abstract_txt:judgments in 3591) [ClassicSimilarity], result of:
           0.45336384 = score(doc=3591,freq=5.0), product of:
             0.47143146 = queryWeight, product of:
               4.843965 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.014143414 = queryNorm
             0.961675 = fieldWeight in 3591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.2958212 = weight(abstract_txt:relevance in 3591) [ClassicSimilarity], result of:
           0.2958212 = score(doc=3591,freq=7.0), product of:
             0.36290634 = queryWeight, product of:
               5.2051663 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.014143414 = queryNorm
             0.8151447 = fieldWeight in 3591, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
       0.2 = coord(5/25)
   

4. Keikha, M.; Crestani, F.; Carman, M.J.: Employing document dependency in blog search (2012) 0.17

   0.16592519 = sum of:
     0.16592519 = product of:
       0.59259 = sum of:
         0.016424127 = weight(abstract_txt:document in 987) [ClassicSimilarity], result of:
           0.016424127 = score(doc=987,freq=1.0), product of:
             0.061196286 = queryWeight, product of:
               1.0076127 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.014143414 = queryNorm
             0.26838437 = fieldWeight in 987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
         0.017680854 = weight(abstract_txt:show in 987) [ClassicSimilarity], result of:
           0.017680854 = score(doc=987,freq=1.0), product of:
             0.06427948 = queryWeight, product of:
               1.0326836 = boost
               4.400995 = idf(docFreq=1480, maxDocs=44421)
               0.014143414 = queryNorm
             0.27506217 = fieldWeight in 987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.400995 = idf(docFreq=1480, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
         0.02892169 = weight(abstract_txt:performance in 987) [ClassicSimilarity], result of:
           0.02892169 = score(doc=987,freq=2.0), product of:
             0.07082871 = queryWeight, product of:
               1.0840162 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.014143414 = queryNorm
             0.40833285 = fieldWeight in 987, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
         0.02677315 = weight(abstract_txt:topic in 987) [ClassicSimilarity], result of:
           0.02677315 = score(doc=987,freq=1.0), product of:
             0.08476239 = queryWeight, product of:
               1.185858 = boost
               5.053779 = idf(docFreq=770, maxDocs=44421)
               0.014143414 = queryNorm
             0.3158612 = fieldWeight in 987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.053779 = idf(docFreq=770, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
         0.054840755 = weight(abstract_txt:rank in 987) [ClassicSimilarity], result of:
           0.054840755 = score(doc=987,freq=1.0), product of:
             0.13671175 = queryWeight, product of:
               1.5060313 = boost
               6.418264 = idf(docFreq=196, maxDocs=44421)
               0.014143414 = queryNorm
             0.4011415 = fieldWeight in 987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.418264 = idf(docFreq=196, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
         0.19793488 = weight(abstract_txt:distributions in 987) [ClassicSimilarity], result of:
           0.19793488 = score(doc=987,freq=1.0), product of:
             0.46393684 = queryWeight, product of:
               4.805307 = boost
               6.82627 = idf(docFreq=130, maxDocs=44421)
               0.014143414 = queryNorm
             0.42664188 = fieldWeight in 987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.82627 = idf(docFreq=130, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
         0.25001454 = weight(abstract_txt:relevance in 987) [ClassicSimilarity], result of:
           0.25001454 = score(doc=987,freq=5.0), product of:
             0.36290634 = queryWeight, product of:
               5.2051663 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.014143414 = queryNorm
             0.68892306 = fieldWeight in 987, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=987)
       0.28 = coord(7/25)
   

5. Efron, M.; Winget, M.: Query polyrepresentation for ranking retrieval systems without relevance judgments (2010) 0.15

   0.14660724 = sum of:
     0.14660724 = product of:
       0.73303616 = sum of:
         0.033466436 = weight(abstract_txt:topic in 456) [ClassicSimilarity], result of:
           0.033466436 = score(doc=456,freq=1.0), product of:
             0.08476239 = queryWeight, product of:
               1.185858 = boost
               5.053779 = idf(docFreq=770, maxDocs=44421)
               0.014143414 = queryNorm
             0.3948265 = fieldWeight in 456, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.053779 = idf(docFreq=770, maxDocs=44421)
               0.078125 = fieldNorm(doc=456)
         0.052288808 = weight(abstract_txt:single in 456) [ClassicSimilarity], result of:
           0.052288808 = score(doc=456,freq=2.0), product of:
             0.09058564 = queryWeight, product of:
               1.2259161 = boost
               5.2244954 = idf(docFreq=649, maxDocs=44421)
               0.014143414 = queryNorm
             0.57723063 = fieldWeight in 456, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2244954 = idf(docFreq=649, maxDocs=44421)
               0.078125 = fieldNorm(doc=456)
         0.068550944 = weight(abstract_txt:rank in 456) [ClassicSimilarity], result of:
           0.068550944 = score(doc=456,freq=1.0), product of:
             0.13671175 = queryWeight, product of:
               1.5060313 = boost
               6.418264 = idf(docFreq=196, maxDocs=44421)
               0.014143414 = queryNorm
             0.5014269 = fieldWeight in 456, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.418264 = idf(docFreq=196, maxDocs=44421)
               0.078125 = fieldNorm(doc=456)
         0.43896765 = weight(abstract_txt:judgments in 456) [ClassicSimilarity], result of:
           0.43896765 = score(doc=456,freq=3.0), product of:
             0.47143146 = queryWeight, product of:
               4.843965 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.014143414 = queryNorm
             0.9311378 = fieldWeight in 456, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.078125 = fieldNorm(doc=456)
         0.13976237 = weight(abstract_txt:relevance in 456) [ClassicSimilarity], result of:
           0.13976237 = score(doc=456,freq=1.0), product of:
             0.36290634 = queryWeight, product of:
               5.2051663 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.014143414 = queryNorm
             0.38511968 = fieldWeight in 456, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=456)
       0.2 = coord(5/25)