Document (#35315)

Author

Roitblat, H.L.
Kershaw, A.
Oot, P.

Title

Document categorization in legal electronic discovery : computer classification vs. manual review

Source

Journal of the American Society for Information Science and Technology. 61(2010) no.1, S.70-80

Year

2009

Abstract

In litigation in the US, the parties are obligated to produce to one another, when requested, those documents that are potentially relevant to issues and facts of the litigation (called discovery). As the volume of electronic documents continues to grow, the expense of dealing with this obligation threatens to surpass the amounts at issue and the time to identify these relevant documents can delay a case for months or years. The same holds true for government investigations and third-parties served with subpoenas. As a result, litigants are looking for ways to reduce the time and expense of discovery. One approach is to supplant or reduce the traditional means of having people, usually attorneys, read each document, with automated procedures that use information retrieval and machine categorization to identify the relevant documents. This study compared an original categorization, obtained as part of a response to a Department of Justice Request and produced by having one or more of 225 attorneys review each document with automated categorization systems provided by two legal service providers. The goal was to determine whether the automated systems could categorize documents at least as well as human reviewers could, thereby saving time and expense. The results support the idea that machine categorization is no less accurate at identifying relevant/responsive documents than employing a team of reviewers. Based on these results, it would appear that using machine categorization can be a reasonable substitute for human review.

Field

Rechtswissenschaft

Similar documents (content)

1. Yang, Y.; Wilbur, J.: Using corpus statistics to remove redundant words in text categorization (1996) 0.23

   0.22749993 = sum of:
     0.22749993 = product of:
       0.81249976 = sum of:
         0.007433302 = weight(abstract_txt:that in 4199) [ClassicSimilarity], result of:
           0.007433302 = score(doc=4199,freq=1.0), product of:
             0.040154994 = queryWeight, product of:
               1.0762373 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.01574633 = queryNorm
             0.18511525 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.012343019 = weight(abstract_txt:with in 4199) [ClassicSimilarity], result of:
           0.012343019 = score(doc=4199,freq=2.0), product of:
             0.04469127 = queryWeight, product of:
               1.1354018 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.01574633 = queryNorm
             0.27618414 = fieldWeight in 4199, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.029916227 = weight(abstract_txt:time in 4199) [ClassicSimilarity], result of:
           0.029916227 = score(doc=4199,freq=1.0), product of:
             0.09230886 = queryWeight, product of:
               1.4131576 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.01574633 = queryNorm
             0.32408836 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.069516554 = weight(abstract_txt:reduce in 4199) [ClassicSimilarity], result of:
           0.069516554 = score(doc=4199,freq=1.0), product of:
             0.14147094 = queryWeight, product of:
               1.4284232 = boost
               6.2897153 = idf(docFreq=222, maxDocs=44218)
               0.01574633 = queryNorm
             0.491384 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.2897153 = idf(docFreq=222, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.07372626 = weight(abstract_txt:automated in 4199) [ClassicSimilarity], result of:
           0.07372626 = score(doc=4199,freq=1.0), product of:
             0.1684174 = queryWeight, product of:
               1.9088085 = boost
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.01574633 = queryNorm
             0.43775916 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.08297266 = weight(abstract_txt:documents in 4199) [ClassicSimilarity], result of:
           0.08297266 = score(doc=4199,freq=2.0), product of:
             0.18221977 = queryWeight, product of:
               2.8079 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.01574633 = queryNorm
             0.4553439 = fieldWeight in 4199, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.53659177 = weight(abstract_txt:categorization in 4199) [ClassicSimilarity], result of:
           0.53659177 = score(doc=4199,freq=5.0), product of:
             0.46603814 = queryWeight, product of:
               4.490499 = boost
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.01574633 = queryNorm
             1.1513902 = fieldWeight in 4199, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
       0.28 = coord(7/25)
   

2. Kim, J.-H.; Choi, K.-S.: Patent document categorization based on semantic structural information (2007) 0.21

   0.2083892 = sum of:
     0.2083892 = product of:
       0.74424714 = sum of:
         0.0084098205 = weight(abstract_txt:that in 933) [ClassicSimilarity], result of:
           0.0084098205 = score(doc=933,freq=2.0), product of:
             0.040154994 = queryWeight, product of:
               1.0762373 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.01574633 = queryNorm
             0.20943399 = fieldWeight in 933, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
         0.006982266 = weight(abstract_txt:with in 933) [ClassicSimilarity], result of:
           0.006982266 = score(doc=933,freq=1.0), product of:
             0.04469127 = queryWeight, product of:
               1.1354018 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.01574633 = queryNorm
             0.15623334 = fieldWeight in 933, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
         0.023932982 = weight(abstract_txt:time in 933) [ClassicSimilarity], result of:
           0.023932982 = score(doc=933,freq=1.0), product of:
             0.09230886 = queryWeight, product of:
               1.4131576 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.01574633 = queryNorm
             0.2592707 = fieldWeight in 933, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
         0.037502076 = weight(abstract_txt:document in 933) [ClassicSimilarity], result of:
           0.037502076 = score(doc=933,freq=2.0), product of:
             0.098841436 = queryWeight, product of:
               1.4623065 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.01574633 = queryNorm
             0.37941656 = fieldWeight in 933, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
         0.044526633 = weight(abstract_txt:relevant in 933) [ClassicSimilarity], result of:
           0.044526633 = score(doc=933,freq=1.0), product of:
             0.15368742 = queryWeight, product of:
               2.1055105 = boost
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.01574633 = queryNorm
             0.28972206 = fieldWeight in 933, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
         0.114970304 = weight(abstract_txt:documents in 933) [ClassicSimilarity], result of:
           0.114970304 = score(doc=933,freq=6.0), product of:
             0.18221977 = queryWeight, product of:
               2.8079 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.01574633 = queryNorm
             0.63094306 = fieldWeight in 933, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
         0.50792307 = weight(abstract_txt:categorization in 933) [ClassicSimilarity], result of:
           0.50792307 = score(doc=933,freq=7.0), product of:
             0.46603814 = queryWeight, product of:
               4.490499 = boost
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.01574633 = queryNorm
             1.0898745 = fieldWeight in 933, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.0625 = fieldNorm(doc=933)
       0.28 = coord(7/25)
   

3. Goren-Bar, D.; Kuflik, T.: Supporting user-subjective categorization with self-organizing maps and learning vector quantization (2005) 0.20

   0.20222895 = sum of:
     0.20222895 = product of:
       0.8426206 = sum of:
         0.02308664 = weight(abstract_txt:human in 3325) [ClassicSimilarity], result of:
           0.02308664 = score(doc=3325,freq=1.0), product of:
             0.07872673 = queryWeight, product of:
               1.0655758 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.01574633 = queryNorm
             0.29325032 = fieldWeight in 3325, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.0625 = fieldNorm(doc=3325)
         0.010299884 = weight(abstract_txt:that in 3325) [ClassicSimilarity], result of:
           0.010299884 = score(doc=3325,freq=3.0), product of:
             0.040154994 = queryWeight, product of:
               1.0762373 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.01574633 = queryNorm
             0.2565032 = fieldWeight in 3325, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=3325)
         0.009874416 = weight(abstract_txt:with in 3325) [ClassicSimilarity], result of:
           0.009874416 = score(doc=3325,freq=2.0), product of:
             0.04469127 = queryWeight, product of:
               1.1354018 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.01574633 = queryNorm
             0.22094731 = fieldWeight in 3325, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=3325)
         0.053035945 = weight(abstract_txt:document in 3325) [ClassicSimilarity], result of:
           0.053035945 = score(doc=3325,freq=4.0), product of:
             0.098841436 = queryWeight, product of:
               1.4623065 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.01574633 = queryNorm
             0.53657603 = fieldWeight in 3325, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=3325)
         0.08129627 = weight(abstract_txt:documents in 3325) [ClassicSimilarity], result of:
           0.08129627 = score(doc=3325,freq=3.0), product of:
             0.18221977 = queryWeight, product of:
               2.8079 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.01574633 = queryNorm
             0.44614407 = fieldWeight in 3325, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=3325)
         0.66502744 = weight(abstract_txt:categorization in 3325) [ClassicSimilarity], result of:
           0.66502744 = score(doc=3325,freq=12.0), product of:
             0.46603814 = queryWeight, product of:
               4.490499 = boost
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.01574633 = queryNorm
             1.4269807 = fieldWeight in 3325, product of:
               3.4641016 = tf(freq=12.0), with freq of:
                 12.0 = termFreq=12.0
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.0625 = fieldNorm(doc=3325)
       0.24 = coord(6/25)
   

4. Han, K.; Rezapour, R.; Nakamura, K.; Devkota, D.; Miller, D.C.; Diesner, J.: ¬An expert-in-the-loop method for domain-specific document categorization based on small training data (2023) 0.19

   0.19358379 = sum of:
     0.19358379 = product of:
       0.60494936 = sum of:
         0.013297094 = weight(abstract_txt:that in 967) [ClassicSimilarity], result of:
           0.013297094 = score(doc=967,freq=5.0), product of:
             0.040154994 = queryWeight, product of:
               1.0762373 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.01574633 = queryNorm
             0.3311442 = fieldWeight in 967, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.026924502 = weight(abstract_txt:identify in 967) [ClassicSimilarity], result of:
           0.026924502 = score(doc=967,freq=1.0), product of:
             0.08722618 = queryWeight, product of:
               1.1216224 = boost
               4.9387927 = idf(docFreq=860, maxDocs=44218)
               0.01574633 = queryNorm
             0.30867454 = fieldWeight in 967, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9387927 = idf(docFreq=860, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.013964532 = weight(abstract_txt:with in 967) [ClassicSimilarity], result of:
           0.013964532 = score(doc=967,freq=4.0), product of:
             0.04469127 = queryWeight, product of:
               1.1354018 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.01574633 = queryNorm
             0.31246668 = fieldWeight in 967, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.023932982 = weight(abstract_txt:time in 967) [ClassicSimilarity], result of:
           0.023932982 = score(doc=967,freq=1.0), product of:
             0.09230886 = queryWeight, product of:
               1.4131576 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.01574633 = queryNorm
             0.2592707 = fieldWeight in 967, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.08341174 = weight(abstract_txt:automated in 967) [ClassicSimilarity], result of:
           0.08341174 = score(doc=967,freq=2.0), product of:
             0.1684174 = queryWeight, product of:
               1.9088085 = boost
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.01574633 = queryNorm
             0.49526796 = fieldWeight in 967, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.044526633 = weight(abstract_txt:relevant in 967) [ClassicSimilarity], result of:
           0.044526633 = score(doc=967,freq=1.0), product of:
             0.15368742 = queryWeight, product of:
               2.1055105 = boost
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.01574633 = queryNorm
             0.28972206 = fieldWeight in 967, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.06637813 = weight(abstract_txt:documents in 967) [ClassicSimilarity], result of:
           0.06637813 = score(doc=967,freq=2.0), product of:
             0.18221977 = queryWeight, product of:
               2.8079 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.01574633 = queryNorm
             0.36427513 = fieldWeight in 967, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
         0.33251372 = weight(abstract_txt:categorization in 967) [ClassicSimilarity], result of:
           0.33251372 = score(doc=967,freq=3.0), product of:
             0.46603814 = queryWeight, product of:
               4.490499 = boost
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.01574633 = queryNorm
             0.71349037 = fieldWeight in 967, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.0625 = fieldNorm(doc=967)
       0.32 = coord(8/25)
   

5. Collins-Thompson, K.; Callan, J.: Predicting reading difficulty with statistical language models (2005) 0.17

   0.17053255 = sum of:
     0.17053255 = product of:
       0.5329142 = sum of:
         0.013297094 = weight(abstract_txt:that in 4579) [ClassicSimilarity], result of:
           0.013297094 = score(doc=4579,freq=5.0), product of:
             0.040154994 = queryWeight, product of:
               1.0762373 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.01574633 = queryNorm
             0.3311442 = fieldWeight in 4579, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.026924502 = weight(abstract_txt:identify in 4579) [ClassicSimilarity], result of:
           0.026924502 = score(doc=4579,freq=1.0), product of:
             0.08722618 = queryWeight, product of:
               1.1216224 = boost
               4.9387927 = idf(docFreq=860, maxDocs=44218)
               0.01574633 = queryNorm
             0.30867454 = fieldWeight in 4579, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9387927 = idf(docFreq=860, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.009874416 = weight(abstract_txt:with in 4579) [ClassicSimilarity], result of:
           0.009874416 = score(doc=4579,freq=2.0), product of:
             0.04469127 = queryWeight, product of:
               1.1354018 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.01574633 = queryNorm
             0.22094731 = fieldWeight in 4579, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.026517972 = weight(abstract_txt:document in 4579) [ClassicSimilarity], result of:
           0.026517972 = score(doc=4579,freq=1.0), product of:
             0.098841436 = queryWeight, product of:
               1.4623065 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.01574633 = queryNorm
             0.26828802 = fieldWeight in 4579, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.058981005 = weight(abstract_txt:automated in 4579) [ClassicSimilarity], result of:
           0.058981005 = score(doc=4579,freq=1.0), product of:
             0.1684174 = queryWeight, product of:
               1.9088085 = boost
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.01574633 = queryNorm
             0.35020733 = fieldWeight in 4579, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.044526633 = weight(abstract_txt:relevant in 4579) [ClassicSimilarity], result of:
           0.044526633 = score(doc=4579,freq=1.0), product of:
             0.15368742 = queryWeight, product of:
               2.1055105 = boost
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.01574633 = queryNorm
             0.28972206 = fieldWeight in 4579, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.08129627 = weight(abstract_txt:documents in 4579) [ClassicSimilarity], result of:
           0.08129627 = score(doc=4579,freq=3.0), product of:
             0.18221977 = queryWeight, product of:
               2.8079 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.01574633 = queryNorm
             0.44614407 = fieldWeight in 4579, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
         0.27149633 = weight(abstract_txt:categorization in 4579) [ClassicSimilarity], result of:
           0.27149633 = score(doc=4579,freq=2.0), product of:
             0.46603814 = queryWeight, product of:
               4.490499 = boost
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.01574633 = queryNorm
             0.58256245 = fieldWeight in 4579, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.590942 = idf(docFreq=164, maxDocs=44218)
               0.0625 = fieldNorm(doc=4579)
       0.32 = coord(8/25)