Document (#31099)

Author

Thelwall, M.
Stuart, D.

Title

Web crawling ethics revisited : cost, privacy, and denial of service

Source

Journal of the American Society for Information Science and Technology. 57(2006) no.13, S.1771-1779

Year

2006

Abstract

Ethical aspects of the employment of Web crawlers for information science research and other contexts are reviewed. The difference between legal and ethical uses of communications technologies is emphasized as well as the changing boundary between ethical and unethical conduct. A review of the potential impacts on Web site owners is used to underpin a new framework for ethical crawling, and it is argued that delicate human judgment is required for each individual case, with verdicts likely to change over time. Decisions can be based upon an approximate cost-benefit analysis, but it is crucial that crawler owners find out about the technological issues affecting the owners of the sites being crawled in order to produce an informed assessment.

Theme

Suchmaschinen

Similar documents (author)

1. Angus, E.; Thelwall, M.; Stuart, D.: General patterns of tag usage among university groups in Flickr (2008) 4.17

   4.172987 = sum of:
     4.172987 = sum of:
       1.3594143 = weight(author_txt:thelwall in 3554) [ClassicSimilarity], result of:
         1.3594143 = score(doc=3554,freq=1.0), product of:
           0.52431554 = queryWeight, product of:
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.075834155 = queryNorm
           2.592741 = fieldWeight in 3554, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.375 = fieldNorm(doc=3554)
       2.8135724 = weight(author_txt:stuart in 3554) [ClassicSimilarity], result of:
         2.8135724 = score(doc=3554,freq=1.0), product of:
           0.851524 = queryWeight, product of:
             1.2743893 = boost
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.075834155 = queryNorm
           3.304161 = fieldWeight in 3554, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.375 = fieldNorm(doc=3554)
   

2. Thelwall, M.; Klitkou, A.; Verbeek, A.; Stuart, D.; Vincent, C.: Policy-relevant Webometrics for individual scientific fields (2010) 3.48

   3.477489 = sum of:
     3.477489 = sum of:
       1.1328453 = weight(author_txt:thelwall in 561) [ClassicSimilarity], result of:
         1.1328453 = score(doc=561,freq=1.0), product of:
           0.52431554 = queryWeight, product of:
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.075834155 = queryNorm
           2.1606174 = fieldWeight in 561, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.3125 = fieldNorm(doc=561)
       2.3446436 = weight(author_txt:stuart in 561) [ClassicSimilarity], result of:
         2.3446436 = score(doc=561,freq=1.0), product of:
           0.851524 = queryWeight, product of:
             1.2743893 = boost
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.075834155 = queryNorm
           2.7534676 = fieldWeight in 561, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.3125 = fieldNorm(doc=561)
   

3. Thelwall, M.; Kousha, K.; Abdoli, M.; Stuart, E.; Makita, M.; Wilson, P.; Levitt, J.: Do altmetric scores reflect article quality? : evidence from the UK Research Excellence Framework 2021 (2023) 2.78

   2.7819912 = sum of:
     2.7819912 = sum of:
       0.9062762 = weight(author_txt:thelwall in 1948) [ClassicSimilarity], result of:
         0.9062762 = score(doc=1948,freq=1.0), product of:
           0.52431554 = queryWeight, product of:
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.075834155 = queryNorm
           1.7284939 = fieldWeight in 1948, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.25 = fieldNorm(doc=1948)
       1.875715 = weight(author_txt:stuart in 1948) [ClassicSimilarity], result of:
         1.875715 = score(doc=1948,freq=1.0), product of:
           0.851524 = queryWeight, product of:
             1.2743893 = boost
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.075834155 = queryNorm
           2.202774 = fieldWeight in 1948, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.25 = fieldNorm(doc=1948)
   

4. Thelwall, M.; Kousha, K.; Abdoli, M.; Stuart, E.; Makita, M.; Wilson, P.; Levitt, J.: Why are coauthored academic articles more cited : higher quality or larger audience? (2023) 2.78

   2.7819912 = sum of:
     2.7819912 = sum of:
       0.9062762 = weight(author_txt:thelwall in 1997) [ClassicSimilarity], result of:
         0.9062762 = score(doc=1997,freq=1.0), product of:
           0.52431554 = queryWeight, product of:
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.075834155 = queryNorm
           1.7284939 = fieldWeight in 1997, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.25 = fieldNorm(doc=1997)
       1.875715 = weight(author_txt:stuart in 1997) [ClassicSimilarity], result of:
         1.875715 = score(doc=1997,freq=1.0), product of:
           0.851524 = queryWeight, product of:
             1.2743893 = boost
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.075834155 = queryNorm
           2.202774 = fieldWeight in 1997, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.25 = fieldNorm(doc=1997)
   

5. Thelwall, M.; Kousha, K.; Stuart, E.; Makita, M.; Abdoli, M.; Wilson, P.; Levitt, J.: In which fields are citations indicators of research quality? (2023) 2.78

   2.7819912 = sum of:
     2.7819912 = sum of:
       0.9062762 = weight(author_txt:thelwall in 2035) [ClassicSimilarity], result of:
         0.9062762 = score(doc=2035,freq=1.0), product of:
           0.52431554 = queryWeight, product of:
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.075834155 = queryNorm
           1.7284939 = fieldWeight in 2035, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.9139757 = idf(docFreq=119, maxDocs=44421)
             0.25 = fieldNorm(doc=2035)
       1.875715 = weight(author_txt:stuart in 2035) [ClassicSimilarity], result of:
         1.875715 = score(doc=2035,freq=1.0), product of:
           0.851524 = queryWeight, product of:
             1.2743893 = boost
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.075834155 = queryNorm
           2.202774 = fieldWeight in 2035, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.811096 = idf(docFreq=17, maxDocs=44421)
             0.25 = fieldNorm(doc=2035)
   

Similar documents (content)

1. Rubin, R.; Froehlich, T.J.: Ethical aspects of library and information science (2009) 0.12

   0.12177393 = sum of:
     0.12177393 = product of:
       0.76108706 = sum of:
         0.075087294 = weight(abstract_txt:conduct in 765) [ClassicSimilarity], result of:
           0.075087294 = score(doc=765,freq=2.0), product of:
             0.10275565 = queryWeight, product of:
               1.0581467 = boost
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.014682638 = queryNorm
             0.7307364 = fieldWeight in 765, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.078125 = fieldNorm(doc=765)
         0.056687944 = weight(abstract_txt:privacy in 765) [ClassicSimilarity], result of:
           0.056687944 = score(doc=765,freq=1.0), product of:
             0.107340895 = queryWeight, product of:
               1.0814978 = boost
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.014682638 = queryNorm
             0.52811134 = fieldWeight in 765, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.078125 = fieldNorm(doc=765)
         0.06423784 = weight(abstract_txt:ethics in 765) [ClassicSimilarity], result of:
           0.06423784 = score(doc=765,freq=1.0), product of:
             0.11667165 = queryWeight, product of:
               1.1275238 = boost
               7.0475073 = idf(docFreq=104, maxDocs=44421)
               0.014682638 = queryNorm
             0.5505865 = fieldWeight in 765, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0475073 = idf(docFreq=104, maxDocs=44421)
               0.078125 = fieldNorm(doc=765)
         0.565074 = weight(abstract_txt:ethical in 765) [ClassicSimilarity], result of:
           0.565074 = score(doc=765,freq=7.0), product of:
             0.41256806 = queryWeight, product of:
               4.2405367 = boost
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.014682638 = queryNorm
             1.3696504 = fieldWeight in 765, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.078125 = fieldNorm(doc=765)
       0.16 = coord(4/25)
   

2. MacFarlane, A.; Missaoui, S.; Makri, S.; Gutierrez Lopez, M.: Sender vs. recipient-orientated information systems revisited (2022) 0.11

   0.1076216 = sum of:
     0.1076216 = product of:
       0.672635 = sum of:
         0.031768005 = weight(abstract_txt:argued in 1608) [ClassicSimilarity], result of:
           0.031768005 = score(doc=1608,freq=1.0), product of:
             0.10256453 = queryWeight, product of:
               1.0571622 = boost
               6.6077175 = idf(docFreq=162, maxDocs=44421)
               0.014682638 = queryNorm
             0.30973676 = fieldWeight in 1608, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6077175 = idf(docFreq=162, maxDocs=44421)
               0.046875 = fieldNorm(doc=1608)
         0.0385427 = weight(abstract_txt:ethics in 1608) [ClassicSimilarity], result of:
           0.0385427 = score(doc=1608,freq=1.0), product of:
             0.11667165 = queryWeight, product of:
               1.1275238 = boost
               7.0475073 = idf(docFreq=104, maxDocs=44421)
               0.014682638 = queryNorm
             0.3303519 = fieldWeight in 1608, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0475073 = idf(docFreq=104, maxDocs=44421)
               0.046875 = fieldNorm(doc=1608)
         0.38036764 = weight(title_txt:revisited in 1608) [ClassicSimilarity], result of:
           0.38036764 = score(doc=1608,freq=1.0), product of:
             0.13419807 = queryWeight, product of:
               1.2092502 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.014682638 = queryNorm
             2.834375 = fieldWeight in 1608, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.375 = fieldNorm(doc=1608)
         0.22195667 = weight(abstract_txt:ethical in 1608) [ClassicSimilarity], result of:
           0.22195667 = score(doc=1608,freq=3.0), product of:
             0.41256806 = queryWeight, product of:
               4.2405367 = boost
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.014682638 = queryNorm
             0.537988 = fieldWeight in 1608, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.046875 = fieldNorm(doc=1608)
       0.16 = coord(4/25)
   

3. Frohmann, B.: Subjectivity and information ethics (2008) 0.09

   0.09081058 = sum of:
     0.09081058 = product of:
       0.56756616 = sum of:
         0.0561182 = weight(abstract_txt:privacy in 2360) [ClassicSimilarity], result of:
           0.0561182 = score(doc=2360,freq=2.0), product of:
             0.107340895 = queryWeight, product of:
               1.0814978 = boost
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.014682638 = queryNorm
             0.52280354 = fieldWeight in 2360, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2360)
         0.015016711 = weight(abstract_txt:between in 2360) [ClassicSimilarity], result of:
           0.015016711 = score(doc=2360,freq=2.0), product of:
             0.056159414 = queryWeight, product of:
               1.1062908 = boost
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.014682638 = queryNorm
             0.26739436 = fieldWeight in 2360, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2360)
         0.16212896 = weight(abstract_txt:ethics in 2360) [ClassicSimilarity], result of:
           0.16212896 = score(doc=2360,freq=13.0), product of:
             0.11667165 = queryWeight, product of:
               1.1275238 = boost
               7.0475073 = idf(docFreq=104, maxDocs=44421)
               0.014682638 = queryNorm
             1.3896174 = fieldWeight in 2360, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               7.0475073 = idf(docFreq=104, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2360)
         0.3343023 = weight(abstract_txt:ethical in 2360) [ClassicSimilarity], result of:
           0.3343023 = score(doc=2360,freq=5.0), product of:
             0.41256806 = queryWeight, product of:
               4.2405367 = boost
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.014682638 = queryNorm
             0.8102961 = fieldWeight in 2360, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2360)
       0.16 = coord(4/25)
   

4. Polat, H.; Du, W.: Privacy-preserving top-N recommendation on distributed data (2008) 0.09

   0.088353954 = sum of:
     0.088353954 = product of:
       0.55221224 = sum of:
         0.053094737 = weight(abstract_txt:conduct in 2864) [ClassicSimilarity], result of:
           0.053094737 = score(doc=2864,freq=1.0), product of:
             0.10275565 = queryWeight, product of:
               1.0581467 = boost
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.014682638 = queryNorm
             0.5167087 = fieldWeight in 2864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.078125 = fieldNorm(doc=2864)
         0.1267581 = weight(abstract_txt:privacy in 2864) [ClassicSimilarity], result of:
           0.1267581 = score(doc=2864,freq=5.0), product of:
             0.107340895 = queryWeight, product of:
               1.0814978 = boost
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.014682638 = queryNorm
             1.180893 = fieldWeight in 2864, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.078125 = fieldNorm(doc=2864)
         0.015169168 = weight(abstract_txt:between in 2864) [ClassicSimilarity], result of:
           0.015169168 = score(doc=2864,freq=1.0), product of:
             0.056159414 = queryWeight, product of:
               1.1062908 = boost
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.014682638 = queryNorm
             0.2701091 = fieldWeight in 2864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.078125 = fieldNorm(doc=2864)
         0.35719025 = weight(abstract_txt:owners in 2864) [ClassicSimilarity], result of:
           0.35719025 = score(doc=2864,freq=1.0), product of:
             0.52813494 = queryWeight, product of:
               4.1550484 = boost
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.014682638 = queryNorm
             0.67632383 = fieldWeight in 2864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.078125 = fieldNorm(doc=2864)
       0.16 = coord(4/25)
   

5. Van der Walt, M.S.: Normative ethics in knowledge organisation (2008) 0.09

   0.08573741 = sum of:
     0.08573741 = product of:
       0.71447843 = sum of:
         0.08495158 = weight(abstract_txt:conduct in 2696) [ClassicSimilarity], result of:
           0.08495158 = score(doc=2696,freq=4.0), product of:
             0.10275565 = queryWeight, product of:
               1.0581467 = boost
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.014682638 = queryNorm
             0.8267339 = fieldWeight in 2696, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.0625 = fieldNorm(doc=2696)
         0.24746712 = weight(abstract_txt:unethical in 2696) [ClassicSimilarity], result of:
           0.24746712 = score(doc=2696,freq=3.0), product of:
             0.23068321 = queryWeight, product of:
               1.5854445 = boost
               9.909708 = idf(docFreq=5, maxDocs=44421)
               0.014682638 = queryNorm
             1.0727574 = fieldWeight in 2696, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               9.909708 = idf(docFreq=5, maxDocs=44421)
               0.0625 = fieldNorm(doc=2696)
         0.38205975 = weight(abstract_txt:ethical in 2696) [ClassicSimilarity], result of:
           0.38205975 = score(doc=2696,freq=5.0), product of:
             0.41256806 = queryWeight, product of:
               4.2405367 = boost
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.014682638 = queryNorm
             0.9260527 = fieldWeight in 2696, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.6262937 = idf(docFreq=159, maxDocs=44421)
               0.0625 = fieldNorm(doc=2696)
       0.12 = coord(3/25)