Document (#34044)

Author

Qu, Y.
Furnas, G.W.

Title

Model-driven formative evaluation of exploratory search : a study under a sensemaking framework

Source

Information processing and management. 44(2008) no.2, S.511-533

Year

2008

Abstract

The evaluation of exploratory search relies on the ongoing paradigm shift from focusing on the search algorithm to focusing on the interactive process. This paper proposes a model-driven formative evaluation approach, in which the goal is not the evaluation of a specific system, per se, but the exploration of new design possibilities. This paper gives an example of this approach where a model of sensemaking was used to inform the evaluation of a basic exploratory search system(s) in the context of a sensemaking task. The model suggested that, rather than just looking at simple search performance measures, we should examine closely the interwoven, interactive processes of both representation construction and information seeking. Participants were asked to make sense of an unfamiliar topic using an augmented query-based search system. The processes of representation construction and information seeking were captured and analyzed using data from experiment notes, interviews, and a system log. The data analysis revealed users' sources of ideas for structuring representations and a tightly coupled relationship between search and representation construction in their exploratory searches. For example, users strategically used search to find useful structure ideas instead of just accumulating information facts. Implications for improving current search systems and designing new systems are discussed.

Footnote

Beitrag eines Themenschwerpunktes "Evaluating exploratory search systems"

Similar documents (content)

1. Pontis, S.; Blandford, A.: Understanding "influence" : an exploratory study of academics' processes of knowledge construction through iterative and interactive information seeking (2015) 0.32

   0.32358694 = sum of:
     0.32358694 = product of:
       1.0112092 = sum of:
         0.094361775 = weight(abstract_txt:unfamiliar in 3126) [ClassicSimilarity], result of:
           0.094361775 = score(doc=3126,freq=2.0), product of:
             0.13484201 = queryWeight, product of:
               1.058136 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.016095625 = queryNorm
             0.69979507 = fieldWeight in 3126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.049933188 = weight(abstract_txt:seeking in 3126) [ClassicSimilarity], result of:
           0.049933188 = score(doc=3126,freq=2.0), product of:
             0.1111469 = queryWeight, product of:
               1.3586029 = boost
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.016095625 = queryNorm
             0.449254 = fieldWeight in 3126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.05123112 = weight(abstract_txt:processes in 3126) [ClassicSimilarity], result of:
           0.05123112 = score(doc=3126,freq=2.0), product of:
             0.11306471 = queryWeight, product of:
               1.370274 = boost
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.016095625 = queryNorm
             0.4531133 = fieldWeight in 3126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.0461499 = weight(abstract_txt:interactive in 3126) [ClassicSimilarity], result of:
           0.0461499 = score(doc=3126,freq=1.0), product of:
             0.13287042 = queryWeight, product of:
               1.4854499 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.016095625 = queryNorm
             0.34733012 = fieldWeight in 3126, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.065346286 = weight(abstract_txt:construction in 3126) [ClassicSimilarity], result of:
           0.065346286 = score(doc=3126,freq=1.0), product of:
             0.19178979 = queryWeight, product of:
               2.1857588 = boost
               5.4514923 = idf(docFreq=517, maxDocs=44421)
               0.016095625 = queryNorm
             0.34071827 = fieldWeight in 3126, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4514923 = idf(docFreq=517, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.43910205 = weight(abstract_txt:sensemaking in 3126) [ClassicSimilarity], result of:
           0.43910205 = score(doc=3126,freq=3.0), product of:
             0.47353068 = queryWeight, product of:
               3.4345002 = boost
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.016095625 = queryNorm
             0.9272938 = fieldWeight in 3126, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.20602196 = weight(abstract_txt:exploratory in 3126) [ClassicSimilarity], result of:
           0.20602196 = score(doc=3126,freq=2.0), product of:
             0.36023852 = queryWeight, product of:
               3.459027 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.016095625 = queryNorm
             0.5719043 = fieldWeight in 3126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
         0.0590629 = weight(abstract_txt:search in 3126) [ClassicSimilarity], result of:
           0.0590629 = score(doc=3126,freq=1.0), product of:
             0.25858012 = queryWeight, product of:
               4.395899 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.016095625 = queryNorm
             0.22841237 = fieldWeight in 3126, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=3126)
       0.32 = coord(8/25)
   

2. Zhang, P.; Soergel, D.: Towards a comprehensive model of the cognitive process and mechanisms of individual sensemaking (2014) 0.25

   0.24638094 = sum of:
     0.24638094 = product of:
       1.0265872 = sum of:
         0.035308097 = weight(abstract_txt:seeking in 2344) [ClassicSimilarity], result of:
           0.035308097 = score(doc=2344,freq=1.0), product of:
             0.1111469 = queryWeight, product of:
               1.3586029 = boost
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.016095625 = queryNorm
             0.31767055 = fieldWeight in 2344, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.0625 = fieldNorm(doc=2344)
         0.06274506 = weight(abstract_txt:processes in 2344) [ClassicSimilarity], result of:
           0.06274506 = score(doc=2344,freq=3.0), product of:
             0.11306471 = queryWeight, product of:
               1.370274 = boost
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.016095625 = queryNorm
             0.5549482 = fieldWeight in 2344, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.0625 = fieldNorm(doc=2344)
         0.047365252 = weight(abstract_txt:ideas in 2344) [ClassicSimilarity], result of:
           0.047365252 = score(doc=2344,freq=1.0), product of:
             0.13519305 = queryWeight, product of:
               1.4983768 = boost
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.016095625 = queryNorm
             0.3503527 = fieldWeight in 2344, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.0625 = fieldNorm(doc=2344)
         0.020633627 = weight(abstract_txt:system in 2344) [ClassicSimilarity], result of:
           0.020633627 = score(doc=2344,freq=1.0), product of:
             0.0978832 = queryWeight, product of:
               1.8030715 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.016095625 = queryNorm
             0.21079844 = fieldWeight in 2344, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.0625 = fieldNorm(doc=2344)
         0.058848273 = weight(abstract_txt:model in 2344) [ClassicSimilarity], result of:
           0.058848273 = score(doc=2344,freq=3.0), product of:
             0.13649179 = queryWeight, product of:
               2.1291788 = boost
               3.9827821 = idf(docFreq=2249, maxDocs=44421)
               0.016095625 = queryNorm
             0.4311488 = fieldWeight in 2344, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.9827821 = idf(docFreq=2249, maxDocs=44421)
               0.0625 = fieldNorm(doc=2344)
         0.801687 = weight(abstract_txt:sensemaking in 2344) [ClassicSimilarity], result of:
           0.801687 = score(doc=2344,freq=10.0), product of:
             0.47353068 = queryWeight, product of:
               3.4345002 = boost
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.016095625 = queryNorm
             1.6929991 = fieldWeight in 2344, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.0625 = fieldNorm(doc=2344)
       0.24 = coord(6/25)
   

3. White, R.W.; Roth, R.A.: Exploratory search : beyond the query-response paradigm (2009) 0.24

   0.2426044 = sum of:
     0.2426044 = product of:
       0.8664443 = sum of:
         0.06672385 = weight(abstract_txt:unfamiliar in 1000) [ClassicSimilarity], result of:
           0.06672385 = score(doc=1000,freq=1.0), product of:
             0.13484201 = queryWeight, product of:
               1.058136 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.016095625 = queryNorm
             0.49482986 = fieldWeight in 1000, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
         0.061155412 = weight(abstract_txt:seeking in 1000) [ClassicSimilarity], result of:
           0.061155412 = score(doc=1000,freq=3.0), product of:
             0.1111469 = queryWeight, product of:
               1.3586029 = boost
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.016095625 = queryNorm
             0.5502215 = fieldWeight in 1000, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
         0.036225874 = weight(abstract_txt:processes in 1000) [ClassicSimilarity], result of:
           0.036225874 = score(doc=1000,freq=1.0), product of:
             0.11306471 = queryWeight, product of:
               1.370274 = boost
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.016095625 = queryNorm
             0.3203995 = fieldWeight in 1000, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
         0.033976067 = weight(abstract_txt:model in 1000) [ClassicSimilarity], result of:
           0.033976067 = score(doc=1000,freq=1.0), product of:
             0.13649179 = queryWeight, product of:
               2.1291788 = boost
               3.9827821 = idf(docFreq=2249, maxDocs=44421)
               0.016095625 = queryNorm
             0.24892388 = fieldWeight in 1000, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.9827821 = idf(docFreq=2249, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
         0.0604297 = weight(abstract_txt:evaluation in 1000) [ClassicSimilarity], result of:
           0.0604297 = score(doc=1000,freq=1.0), product of:
             0.21583809 = queryWeight, product of:
               2.9934907 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016095625 = queryNorm
             0.279977 = fieldWeight in 1000, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
         0.41204393 = weight(abstract_txt:exploratory in 1000) [ClassicSimilarity], result of:
           0.41204393 = score(doc=1000,freq=8.0), product of:
             0.36023852 = queryWeight, product of:
               3.459027 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.016095625 = queryNorm
             1.1438086 = fieldWeight in 1000, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
         0.19588949 = weight(abstract_txt:search in 1000) [ClassicSimilarity], result of:
           0.19588949 = score(doc=1000,freq=11.0), product of:
             0.25858012 = queryWeight, product of:
               4.395899 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.016095625 = queryNorm
             0.75755817 = fieldWeight in 1000, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=1000)
       0.28 = coord(7/25)
   

4. Blocks, D.; Cunliffe, D.; Tudhope, D.: ¬A reference model for user-system interaction in thesaurus-based searching (2006) 0.17

   0.16764957 = sum of:
     0.16764957 = product of:
       0.59874845 = sum of:
         0.07211473 = weight(abstract_txt:augmented in 327) [ClassicSimilarity], result of:
           0.07211473 = score(doc=327,freq=1.0), product of:
             0.12238094 = queryWeight, product of:
               1.0080585 = boost
               7.5425844 = idf(docFreq=63, maxDocs=44421)
               0.016095625 = queryNorm
             0.5892644 = fieldWeight in 327, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5425844 = idf(docFreq=63, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
         0.064038895 = weight(abstract_txt:processes in 327) [ClassicSimilarity], result of:
           0.064038895 = score(doc=327,freq=2.0), product of:
             0.11306471 = queryWeight, product of:
               1.370274 = boost
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.016095625 = queryNorm
             0.5663916 = fieldWeight in 327, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
         0.05767274 = weight(abstract_txt:system in 327) [ClassicSimilarity], result of:
           0.05767274 = score(doc=327,freq=5.0), product of:
             0.0978832 = queryWeight, product of:
               1.8030715 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.016095625 = queryNorm
             0.58919954 = fieldWeight in 327, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
         0.060269117 = weight(abstract_txt:representation in 327) [ClassicSimilarity], result of:
           0.060269117 = score(doc=327,freq=1.0), product of:
             0.15660349 = queryWeight, product of:
               1.9751052 = boost
               4.9261017 = idf(docFreq=875, maxDocs=44421)
               0.016095625 = queryNorm
             0.3848517 = fieldWeight in 327, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9261017 = idf(docFreq=875, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
         0.10403003 = weight(abstract_txt:model in 327) [ClassicSimilarity], result of:
           0.10403003 = score(doc=327,freq=6.0), product of:
             0.13649179 = queryWeight, product of:
               2.1291788 = boost
               3.9827821 = idf(docFreq=2249, maxDocs=44421)
               0.016095625 = queryNorm
             0.7621706 = fieldWeight in 327, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.9827821 = idf(docFreq=2249, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
         0.07553712 = weight(abstract_txt:evaluation in 327) [ClassicSimilarity], result of:
           0.07553712 = score(doc=327,freq=1.0), product of:
             0.21583809 = queryWeight, product of:
               2.9934907 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016095625 = queryNorm
             0.34997123 = fieldWeight in 327, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
         0.1650858 = weight(abstract_txt:search in 327) [ClassicSimilarity], result of:
           0.1650858 = score(doc=327,freq=5.0), product of:
             0.25858012 = queryWeight, product of:
               4.395899 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.016095625 = queryNorm
             0.63843197 = fieldWeight in 327, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.078125 = fieldNorm(doc=327)
       0.28 = coord(7/25)
   

5. Shah, C.; Hendahewa, C.; González-Ibáñez, R.: Rain or shine? : forecasting search process performance in exploratory search tasks (2016) 0.16

   0.16407764 = sum of:
     0.16407764 = product of:
       0.58599156 = sum of:
         0.014079807 = weight(abstract_txt:approach in 4012) [ClassicSimilarity], result of:
           0.014079807 = score(doc=4012,freq=1.0), product of:
             0.06021605 = queryWeight, product of:
               3.741144 = idf(docFreq=2864, maxDocs=44421)
               0.016095625 = queryNorm
             0.2338215 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.741144 = idf(docFreq=2864, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.035308097 = weight(abstract_txt:seeking in 4012) [ClassicSimilarity], result of:
           0.035308097 = score(doc=4012,freq=1.0), product of:
             0.1111469 = queryWeight, product of:
               1.3586029 = boost
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.016095625 = queryNorm
             0.31767055 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.082729 = idf(docFreq=748, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.06274506 = weight(abstract_txt:processes in 4012) [ClassicSimilarity], result of:
           0.06274506 = score(doc=4012,freq=3.0), product of:
             0.11306471 = queryWeight, product of:
               1.370274 = boost
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.016095625 = queryNorm
             0.5549482 = fieldWeight in 4012, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.126392 = idf(docFreq=716, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.020633627 = weight(abstract_txt:system in 4012) [ClassicSimilarity], result of:
           0.020633627 = score(doc=4012,freq=1.0), product of:
             0.0978832 = queryWeight, product of:
               1.8030715 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.016095625 = queryNorm
             0.21079844 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.0604297 = weight(abstract_txt:evaluation in 4012) [ClassicSimilarity], result of:
           0.0604297 = score(doc=4012,freq=1.0), product of:
             0.21583809 = queryWeight, product of:
               2.9934907 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016095625 = queryNorm
             0.279977 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.20602196 = weight(abstract_txt:exploratory in 4012) [ClassicSimilarity], result of:
           0.20602196 = score(doc=4012,freq=2.0), product of:
             0.36023852 = queryWeight, product of:
               3.459027 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.016095625 = queryNorm
             0.5719043 = fieldWeight in 4012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.18677329 = weight(abstract_txt:search in 4012) [ClassicSimilarity], result of:
           0.18677329 = score(doc=4012,freq=10.0), product of:
             0.25858012 = queryWeight, product of:
               4.395899 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.016095625 = queryNorm
             0.72230333 = fieldWeight in 4012, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
       0.28 = coord(7/25)