Document (#44341)

Author
Zhu, S.
Xie, X.
Ye, Z.
Ai, Q.
Liu, Y.
Title
Comparing point-wise and pair-wise relevance judgment with brain signals
Source
Journal of the Association for Information Science and Technology. 75(2024) no.9, S.957-971
Year
2024
Abstract
How to collect relevance judgment has long been an important problem in Information Retrieval (IR). A popular method is to collect relevance judgment in a point-wise manner, in which assessors examine and give an absolute relevance score for each item independently of the others. As an alternative, pair-wise relevance judgment, also named preference judgment, allows an assessor to compare two items side-by-side and express their preference for one over the other. Previous work has explored the differences between these two paradigms of relevance judgments from many different aspects. Most of these works are conducted through explicit/implicit feedback. However, few works investigate the underlying neurological mechanisms of the two paradigms. In this paper, we conduct a lab study to investigate and compare point-wise and pair-wise relevance judgment in image search scenarios. We study the neurological mechanisms of the two paradigms through an event-related potential (ERP) analysis of the users' brain signals while viewing images during a search process. We have obtained several observations, such as search engine users tend to pay more attention to preferred items in the point-wise paradigm but unpreferred items in the pair-wise paradigm. Furthermore, we test the adoption of brain signals as implicit feedback for predicting pair-wise relevance judgment, highlighting the feasibility of leveraging brain signals to understand users' relevance judgments.
Content
Vgl.: https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24936.

Similar documents (content)

  1. Jacucci, G.; Barral, O.; Daee, P.; Wenzel, M.; Serim, B.; Ruotsalo, T.; Pluchino, P.; Freeman, J.; Gamberini, L.; Kaski, S.; Blankertz, B.: Integrating neurophysiologic relevance feedback in intent modeling for information retrieval (2019) 0.17 
    0.17369162 = sum of:
  0.17369162 = product of:
    0.6203272 = sum of:
      0.007684723 = weight(abstract_txt:through in 356) [ClassicSimilarity], result of:
        0.007684723 = score(doc=356,freq=1.0), product of:
          0.030740311 = queryWeight, product of:
            3.9998152 = idf(docFreq=2211, maxDocs=44421)
            0.0076854327 = queryNorm
          0.24998845 = fieldWeight in 356, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            3.9998152 = idf(docFreq=2211, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
      0.012434648 = weight(abstract_txt:search in 356) [ClassicSimilarity], result of:
        0.012434648 = score(doc=356,freq=2.0), product of:
          0.038494516 = queryWeight, product of:
            1.3705375 = boost
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.3230239 = fieldWeight in 356, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
      0.061633296 = weight(abstract_txt:feedback in 356) [ClassicSimilarity], result of:
        0.061633296 = score(doc=356,freq=6.0), product of:
          0.067782246 = queryWeight, product of:
            1.4849226 = boost
            5.9394164 = idf(docFreq=317, maxDocs=44421)
            0.0076854327 = queryNorm
          0.90928376 = fieldWeight in 356, product of:
            2.4494898 = tf(freq=6.0), with freq of:
              6.0 = termFreq=6.0
            5.9394164 = idf(docFreq=317, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
      0.062490117 = weight(abstract_txt:implicit in 356) [ClassicSimilarity], result of:
        0.062490117 = score(doc=356,freq=3.0), product of:
          0.08618995 = queryWeight, product of:
            1.6744577 = boost
            6.697521 = idf(docFreq=148, maxDocs=44421)
            0.0076854327 = queryNorm
          0.7250279 = fieldWeight in 356, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            6.697521 = idf(docFreq=148, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
      0.1457518 = weight(abstract_txt:brain in 356) [ClassicSimilarity], result of:
        0.1457518 = score(doc=356,freq=2.0), product of:
          0.21862446 = queryWeight, product of:
            3.7714665 = boost
            7.5425844 = idf(docFreq=63, maxDocs=44421)
            0.0076854327 = queryNorm
          0.6666766 = fieldWeight in 356, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            7.5425844 = idf(docFreq=63, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
      0.1855811 = weight(abstract_txt:signals in 356) [ClassicSimilarity], result of:
        0.1855811 = score(doc=356,freq=2.0), product of:
          0.25682974 = queryWeight, product of:
            4.0877423 = boost
            8.175107 = idf(docFreq=33, maxDocs=44421)
            0.0076854327 = queryNorm
          0.7225842 = fieldWeight in 356, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            8.175107 = idf(docFreq=33, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
      0.14475147 = weight(abstract_txt:relevance in 356) [ClassicSimilarity], result of:
        0.14475147 = score(doc=356,freq=5.0), product of:
          0.21011269 = queryWeight, product of:
            5.5459795 = boost
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0076854327 = queryNorm
          0.68892306 = fieldWeight in 356, product of:
            2.236068 = tf(freq=5.0), with freq of:
              5.0 = termFreq=5.0
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0625 = fieldNorm(doc=356)
    0.28 = coord(7/25)
  2. Taylor, A.: User relevance criteria choices and the information search process (2012) 0.14 
    0.14449388 = sum of:
  0.14449388 = product of:
    0.6020578 = sum of:
      0.007684723 = weight(abstract_txt:through in 3745) [ClassicSimilarity], result of:
        0.007684723 = score(doc=3745,freq=1.0), product of:
          0.030740311 = queryWeight, product of:
            3.9998152 = idf(docFreq=2211, maxDocs=44421)
            0.0076854327 = queryNorm
          0.24998845 = fieldWeight in 3745, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            3.9998152 = idf(docFreq=2211, maxDocs=44421)
            0.0625 = fieldNorm(doc=3745)
      0.011564613 = weight(abstract_txt:users in 3745) [ClassicSimilarity], result of:
        0.011564613 = score(doc=3745,freq=2.0), product of:
          0.036677293 = queryWeight, product of:
            1.3377968 = boost
            3.5672934 = idf(docFreq=3408, maxDocs=44421)
            0.0076854327 = queryNorm
          0.31530717 = fieldWeight in 3745, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5672934 = idf(docFreq=3408, maxDocs=44421)
            0.0625 = fieldNorm(doc=3745)
      0.024869297 = weight(abstract_txt:search in 3745) [ClassicSimilarity], result of:
        0.024869297 = score(doc=3745,freq=8.0), product of:
          0.038494516 = queryWeight, product of:
            1.3705375 = boost
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.6460478 = fieldWeight in 3745, product of:
            2.828427 = tf(freq=8.0), with freq of:
              8.0 = termFreq=8.0
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0625 = fieldNorm(doc=3745)
      0.078257926 = weight(abstract_txt:judgments in 3745) [ClassicSimilarity], result of:
        0.078257926 = score(doc=3745,freq=4.0), product of:
          0.0909819 = queryWeight, product of:
            1.720376 = boost
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.86014825 = fieldWeight in 3745, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0625 = fieldNorm(doc=3745)
      0.19420448 = weight(abstract_txt:relevance in 3745) [ClassicSimilarity], result of:
        0.19420448 = score(doc=3745,freq=9.0), product of:
          0.21011269 = queryWeight, product of:
            5.5459795 = boost
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0076854327 = queryNorm
          0.92428726 = fieldWeight in 3745, product of:
            3.0 = tf(freq=9.0), with freq of:
              9.0 = termFreq=9.0
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0625 = fieldNorm(doc=3745)
      0.28547677 = weight(abstract_txt:judgment in 3745) [ClassicSimilarity], result of:
        0.28547677 = score(doc=3745,freq=3.0), product of:
          0.36029002 = queryWeight, product of:
            6.404806 = boost
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0076854327 = queryNorm
          0.7923527 = fieldWeight in 3745, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0625 = fieldNorm(doc=3745)
    0.24 = coord(6/25)
  3. Spink, A.: Towards a theoretical framework for information retrieval in an information seeking context (1999) 0.14 
    0.14313775 = sum of:
  0.14313775 = product of:
    0.5964073 = sum of:
      0.016354833 = weight(abstract_txt:users in 1300) [ClassicSimilarity], result of:
        0.016354833 = score(doc=1300,freq=4.0), product of:
          0.036677293 = queryWeight, product of:
            1.3377968 = boost
            3.5672934 = idf(docFreq=3408, maxDocs=44421)
            0.0076854327 = queryNorm
          0.44591168 = fieldWeight in 1300, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            3.5672934 = idf(docFreq=3408, maxDocs=44421)
            0.0625 = fieldNorm(doc=1300)
      0.017585248 = weight(abstract_txt:search in 1300) [ClassicSimilarity], result of:
        0.017585248 = score(doc=1300,freq=4.0), product of:
          0.038494516 = queryWeight, product of:
            1.3705375 = boost
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.45682475 = fieldWeight in 1300, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0625 = fieldNorm(doc=1300)
      0.035583995 = weight(abstract_txt:feedback in 1300) [ClassicSimilarity], result of:
        0.035583995 = score(doc=1300,freq=2.0), product of:
          0.067782246 = queryWeight, product of:
            1.4849226 = boost
            5.9394164 = idf(docFreq=317, maxDocs=44421)
            0.0076854327 = queryNorm
          0.5249752 = fieldWeight in 1300, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            5.9394164 = idf(docFreq=317, maxDocs=44421)
            0.0625 = fieldNorm(doc=1300)
      0.06777335 = weight(abstract_txt:judgments in 1300) [ClassicSimilarity], result of:
        0.06777335 = score(doc=1300,freq=3.0), product of:
          0.0909819 = queryWeight, product of:
            1.720376 = boost
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.74491024 = fieldWeight in 1300, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0625 = fieldNorm(doc=1300)
      0.12946966 = weight(abstract_txt:relevance in 1300) [ClassicSimilarity], result of:
        0.12946966 = score(doc=1300,freq=4.0), product of:
          0.21011269 = queryWeight, product of:
            5.5459795 = boost
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0076854327 = queryNorm
          0.6161915 = fieldWeight in 1300, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0625 = fieldNorm(doc=1300)
      0.32964018 = weight(abstract_txt:judgment in 1300) [ClassicSimilarity], result of:
        0.32964018 = score(doc=1300,freq=4.0), product of:
          0.36029002 = queryWeight, product of:
            6.404806 = boost
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0076854327 = queryNorm
          0.9149301 = fieldWeight in 1300, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0625 = fieldNorm(doc=1300)
    0.24 = coord(6/25)
  4. Kuhlthau, C.C.: Investigating patterns in information seeking : concepts in context (1999) 0.14 
    0.14313775 = sum of:
  0.14313775 = product of:
    0.5964073 = sum of:
      0.016354833 = weight(abstract_txt:users in 1301) [ClassicSimilarity], result of:
        0.016354833 = score(doc=1301,freq=4.0), product of:
          0.036677293 = queryWeight, product of:
            1.3377968 = boost
            3.5672934 = idf(docFreq=3408, maxDocs=44421)
            0.0076854327 = queryNorm
          0.44591168 = fieldWeight in 1301, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            3.5672934 = idf(docFreq=3408, maxDocs=44421)
            0.0625 = fieldNorm(doc=1301)
      0.017585248 = weight(abstract_txt:search in 1301) [ClassicSimilarity], result of:
        0.017585248 = score(doc=1301,freq=4.0), product of:
          0.038494516 = queryWeight, product of:
            1.3705375 = boost
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.45682475 = fieldWeight in 1301, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            3.654598 = idf(docFreq=3123, maxDocs=44421)
            0.0625 = fieldNorm(doc=1301)
      0.035583995 = weight(abstract_txt:feedback in 1301) [ClassicSimilarity], result of:
        0.035583995 = score(doc=1301,freq=2.0), product of:
          0.067782246 = queryWeight, product of:
            1.4849226 = boost
            5.9394164 = idf(docFreq=317, maxDocs=44421)
            0.0076854327 = queryNorm
          0.5249752 = fieldWeight in 1301, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            5.9394164 = idf(docFreq=317, maxDocs=44421)
            0.0625 = fieldNorm(doc=1301)
      0.06777335 = weight(abstract_txt:judgments in 1301) [ClassicSimilarity], result of:
        0.06777335 = score(doc=1301,freq=3.0), product of:
          0.0909819 = queryWeight, product of:
            1.720376 = boost
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.74491024 = fieldWeight in 1301, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0625 = fieldNorm(doc=1301)
      0.12946966 = weight(abstract_txt:relevance in 1301) [ClassicSimilarity], result of:
        0.12946966 = score(doc=1301,freq=4.0), product of:
          0.21011269 = queryWeight, product of:
            5.5459795 = boost
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0076854327 = queryNorm
          0.6161915 = fieldWeight in 1301, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0625 = fieldNorm(doc=1301)
      0.32964018 = weight(abstract_txt:judgment in 1301) [ClassicSimilarity], result of:
        0.32964018 = score(doc=1301,freq=4.0), product of:
          0.36029002 = queryWeight, product of:
            6.404806 = boost
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0076854327 = queryNorm
          0.9149301 = fieldWeight in 1301, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0625 = fieldNorm(doc=1301)
    0.24 = coord(6/25)
  5. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.14 
    0.14072435 = sum of:
  0.14072435 = product of:
    0.7036217 = sum of:
      0.007684723 = weight(abstract_txt:through in 3591) [ClassicSimilarity], result of:
        0.007684723 = score(doc=3591,freq=1.0), product of:
          0.030740311 = queryWeight, product of:
            3.9998152 = idf(docFreq=2211, maxDocs=44421)
            0.0076854327 = queryNorm
          0.24998845 = fieldWeight in 3591, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            3.9998152 = idf(docFreq=2211, maxDocs=44421)
            0.0625 = fieldNorm(doc=3591)
      0.06862074 = weight(abstract_txt:assessors in 3591) [ClassicSimilarity], result of:
        0.06862074 = score(doc=3591,freq=3.0), product of:
          0.072813064 = queryWeight, product of:
            1.0882672 = boost
            8.705735 = idf(docFreq=19, maxDocs=44421)
            0.0076854327 = queryNorm
          0.94242346 = fieldWeight in 3591, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            8.705735 = idf(docFreq=19, maxDocs=44421)
            0.0625 = fieldNorm(doc=3591)
      0.08749502 = weight(abstract_txt:judgments in 3591) [ClassicSimilarity], result of:
        0.08749502 = score(doc=3591,freq=5.0), product of:
          0.0909819 = queryWeight, product of:
            1.720376 = boost
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0076854327 = queryNorm
          0.961675 = fieldWeight in 3591, product of:
            2.236068 = tf(freq=5.0), with freq of:
              5.0 = termFreq=5.0
            6.881186 = idf(docFreq=123, maxDocs=44421)
            0.0625 = fieldNorm(doc=3591)
      0.17127225 = weight(abstract_txt:relevance in 3591) [ClassicSimilarity], result of:
        0.17127225 = score(doc=3591,freq=7.0), product of:
          0.21011269 = queryWeight, product of:
            5.5459795 = boost
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0076854327 = queryNorm
          0.8151447 = fieldWeight in 3591, product of:
            2.6457512 = tf(freq=7.0), with freq of:
              7.0 = termFreq=7.0
            4.929532 = idf(docFreq=872, maxDocs=44421)
            0.0625 = fieldNorm(doc=3591)
      0.36854896 = weight(abstract_txt:judgment in 3591) [ClassicSimilarity], result of:
        0.36854896 = score(doc=3591,freq=5.0), product of:
          0.36029002 = queryWeight, product of:
            6.404806 = boost
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0076854327 = queryNorm
          1.022923 = fieldWeight in 3591, product of:
            2.236068 = tf(freq=5.0), with freq of:
              5.0 = termFreq=5.0
            7.319441 = idf(docFreq=79, maxDocs=44421)
            0.0625 = fieldNorm(doc=3591)
    0.2 = coord(5/25)