Document (#38628)

Author

Alqaraleh, S.
Ramadan, O.
Salamah, M.

Title

Efficient watcher based web crawler design

Source

Aslib journal of information management. 67(2015) no.6, S.663-686

Year

2015

Abstract

Purpose The purpose of this paper is to design a watcher-based crawler (WBC) that has the ability of crawling static and dynamic web sites, and can download only the updated and newly added web pages. Design/methodology/approach In the proposed WBC crawler, a watcher file, which can be uploaded to the web sites servers, prepares a report that contains the addresses of the updated and the newly added web pages. In addition, the WBC is split into five units, where each unit is responsible for performing a specific crawling process. Findings Several experiments have been conducted and it has been observed that the proposed WBC increases the number of uniquely visited static and dynamic web sites as compared with the existing crawling techniques. In addition, the proposed watcher file not only allows the crawlers to visit the updated and newly web pages, but also solves the crawlers overlapping and communication problems. Originality/value The proposed WBC performs all crawling processes in the sense that it detects all updated and newly added pages automatically without any human explicit intervention or downloading the entire web sites.

Content

Vgl.: http://dx.doi.org/10.1108/AJIM-02-2015-0019.

Theme

Suchmaschinen

Similar documents (content)

1. Thelwall, M.; Stuart, D.: Web crawling ethics revisited : cost, privacy, and denial of service (2006) 0.17

   0.17100506 = sum of:
     0.17100506 = product of:
       0.8550253 = sum of:
         0.0097198775 = weight(abstract_txt:that in 98) [ClassicSimilarity], result of:
           0.0097198775 = score(doc=98,freq=2.0), product of:
             0.03719945 = queryWeight, product of:
               1.1486217 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013694312 = queryNorm
             0.2612909 = fieldWeight in 98, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=98)
         0.19337004 = weight(abstract_txt:crawlers in 98) [ClassicSimilarity], result of:
           0.19337004 = score(doc=98,freq=1.0), product of:
             0.27312124 = queryWeight, product of:
               2.2007542 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.013694312 = queryNorm
             0.7080008 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.078125 = fieldNorm(doc=98)
         0.082020134 = weight(abstract_txt:sites in 98) [ClassicSimilarity], result of:
           0.082020134 = score(doc=98,freq=1.0), product of:
             0.19426131 = queryWeight, product of:
               2.6248345 = boost
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.013694312 = queryNorm
             0.4222155 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.078125 = fieldNorm(doc=98)
         0.25713766 = weight(abstract_txt:crawler in 98) [ClassicSimilarity], result of:
           0.25713766 = score(doc=98,freq=1.0), product of:
             0.37806824 = queryWeight, product of:
               3.1712065 = boost
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.013694312 = queryNorm
             0.68013555 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.078125 = fieldNorm(doc=98)
         0.31277758 = weight(abstract_txt:crawling in 98) [ClassicSimilarity], result of:
           0.31277758 = score(doc=98,freq=1.0), product of:
             0.47416526 = queryWeight, product of:
               4.1008477 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.013694312 = queryNorm
             0.65963835 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.078125 = fieldNorm(doc=98)
       0.2 = coord(5/25)
   

2. Lee, L.-H.; Luh, C.-J.: Generation of pornographic blacklist and its incremental update using an inverse chi-square based method (2008) 0.13

   0.13150653 = sum of:
     0.13150653 = product of:
       0.5479439 = sum of:
         0.0068729906 = weight(abstract_txt:that in 2340) [ClassicSimilarity], result of:
           0.0068729906 = score(doc=2340,freq=1.0), product of:
             0.03719945 = queryWeight, product of:
               1.1486217 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013694312 = queryNorm
             0.18476056 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=2340)
         0.08671071 = weight(abstract_txt:added in 2340) [ClassicSimilarity], result of:
           0.08671071 = score(doc=2340,freq=1.0), product of:
             0.1831646 = queryWeight, product of:
               2.207294 = boost
               6.059561 = idf(docFreq=281, maxDocs=44421)
               0.013694312 = queryNorm
             0.4734032 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.059561 = idf(docFreq=281, maxDocs=44421)
               0.078125 = fieldNorm(doc=2340)
         0.071904875 = weight(abstract_txt:proposed in 2340) [ClassicSimilarity], result of:
           0.071904875 = score(doc=2340,freq=2.0), product of:
             0.14123258 = queryWeight, product of:
               2.2380831 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.013694312 = queryNorm
             0.50912386 = fieldWeight in 2340, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.078125 = fieldNorm(doc=2340)
         0.11599399 = weight(abstract_txt:sites in 2340) [ClassicSimilarity], result of:
           0.11599399 = score(doc=2340,freq=2.0), product of:
             0.19426131 = queryWeight, product of:
               2.6248345 = boost
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.013694312 = queryNorm
             0.5971029 = fieldWeight in 2340, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.078125 = fieldNorm(doc=2340)
         0.0915302 = weight(abstract_txt:pages in 2340) [ClassicSimilarity], result of:
           0.0915302 = score(doc=2340,freq=1.0), product of:
             0.20900126 = queryWeight, product of:
               2.7225962 = boost
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.013694312 = queryNorm
             0.4379409 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.078125 = fieldNorm(doc=2340)
         0.1749311 = weight(abstract_txt:newly in 2340) [ClassicSimilarity], result of:
           0.1749311 = score(doc=2340,freq=1.0), product of:
             0.3218726 = queryWeight, product of:
               3.3787088 = boost
               6.9565353 = idf(docFreq=114, maxDocs=44421)
               0.013694312 = queryNorm
             0.5434793 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9565353 = idf(docFreq=114, maxDocs=44421)
               0.078125 = fieldNorm(doc=2340)
       0.24 = coord(6/25)
   

3. Vidmar, D.; Anderson, C.: History of Internet search tools (2002) 0.13

   0.12527983 = sum of:
     0.12527983 = product of:
       0.39149946 = sum of:
         0.0138218505 = weight(abstract_txt:only in 5258) [ClassicSimilarity], result of:
           0.0138218505 = score(doc=5258,freq=1.0), product of:
             0.059667993 = queryWeight, product of:
               1.0286425 = boost
               4.235812 = idf(docFreq=1746, maxDocs=44421)
               0.013694312 = queryNorm
             0.23164597 = fieldWeight in 5258, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.235812 = idf(docFreq=1746, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.011784726 = weight(abstract_txt:that in 5258) [ClassicSimilarity], result of:
           0.011784726 = score(doc=5258,freq=6.0), product of:
             0.03719945 = queryWeight, product of:
               1.1486217 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013694312 = queryNorm
             0.3167984 = fieldWeight in 5258, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.03165522 = weight(abstract_txt:file in 5258) [ClassicSimilarity], result of:
           0.03165522 = score(doc=5258,freq=1.0), product of:
             0.10367185 = queryWeight, product of:
               1.3558892 = boost
               5.58337 = idf(docFreq=453, maxDocs=44421)
               0.013694312 = queryNorm
             0.30534056 = fieldWeight in 5258, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.58337 = idf(docFreq=453, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.03482671 = weight(abstract_txt:dynamic in 5258) [ClassicSimilarity], result of:
           0.03482671 = score(doc=5258,freq=1.0), product of:
             0.11048558 = queryWeight, product of:
               1.3997375 = boost
               5.7639313 = idf(docFreq=378, maxDocs=44421)
               0.013694312 = queryNorm
             0.315215 = fieldWeight in 5258, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7639313 = idf(docFreq=378, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.06554739 = weight(abstract_txt:static in 5258) [ClassicSimilarity], result of:
           0.06554739 = score(doc=5258,freq=1.0), product of:
             0.1684228 = queryWeight, product of:
               1.7282009 = boost
               7.1165 = idf(docFreq=97, maxDocs=44421)
               0.013694312 = queryNorm
             0.38918358 = fieldWeight in 5258, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.1165 = idf(docFreq=97, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.08583922 = weight(abstract_txt:added in 5258) [ClassicSimilarity], result of:
           0.08583922 = score(doc=5258,freq=2.0), product of:
             0.1831646 = queryWeight, product of:
               2.207294 = boost
               6.059561 = idf(docFreq=281, maxDocs=44421)
               0.013694312 = queryNorm
             0.46864527 = fieldWeight in 5258, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.059561 = idf(docFreq=281, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.057414096 = weight(abstract_txt:sites in 5258) [ClassicSimilarity], result of:
           0.057414096 = score(doc=5258,freq=1.0), product of:
             0.19426131 = queryWeight, product of:
               2.6248345 = boost
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.013694312 = queryNorm
             0.29555085 = fieldWeight in 5258, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
         0.090610266 = weight(abstract_txt:pages in 5258) [ClassicSimilarity], result of:
           0.090610266 = score(doc=5258,freq=2.0), product of:
             0.20900126 = queryWeight, product of:
               2.7225962 = boost
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.013694312 = queryNorm
             0.43353933 = fieldWeight in 5258, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.0546875 = fieldNorm(doc=5258)
       0.32 = coord(8/25)
   

4. Choi, B.; Peng, X.: Dynamic and hierarchical classification of Web pages (2004) 0.11

   0.10596046 = sum of:
     0.10596046 = product of:
       0.44150192 = sum of:
         0.019745503 = weight(abstract_txt:only in 3555) [ClassicSimilarity], result of:
           0.019745503 = score(doc=3555,freq=1.0), product of:
             0.059667993 = queryWeight, product of:
               1.0286425 = boost
               4.235812 = idf(docFreq=1746, maxDocs=44421)
               0.013694312 = queryNorm
             0.33092284 = fieldWeight in 3555, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.235812 = idf(docFreq=1746, maxDocs=44421)
               0.078125 = fieldNorm(doc=3555)
         0.0097198775 = weight(abstract_txt:that in 3555) [ClassicSimilarity], result of:
           0.0097198775 = score(doc=3555,freq=2.0), product of:
             0.03719945 = queryWeight, product of:
               1.1486217 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013694312 = queryNorm
             0.2612909 = fieldWeight in 3555, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3555)
         0.07036058 = weight(abstract_txt:dynamic in 3555) [ClassicSimilarity], result of:
           0.07036058 = score(doc=3555,freq=2.0), product of:
             0.11048558 = queryWeight, product of:
               1.3997375 = boost
               5.7639313 = idf(docFreq=378, maxDocs=44421)
               0.013694312 = queryNorm
             0.6368304 = fieldWeight in 3555, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7639313 = idf(docFreq=378, maxDocs=44421)
               0.078125 = fieldNorm(doc=3555)
         0.08671071 = weight(abstract_txt:added in 3555) [ClassicSimilarity], result of:
           0.08671071 = score(doc=3555,freq=1.0), product of:
             0.1831646 = queryWeight, product of:
               2.207294 = boost
               6.059561 = idf(docFreq=281, maxDocs=44421)
               0.013694312 = queryNorm
             0.4734032 = fieldWeight in 3555, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.059561 = idf(docFreq=281, maxDocs=44421)
               0.078125 = fieldNorm(doc=3555)
         0.071904875 = weight(abstract_txt:proposed in 3555) [ClassicSimilarity], result of:
           0.071904875 = score(doc=3555,freq=2.0), product of:
             0.14123258 = queryWeight, product of:
               2.2380831 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.013694312 = queryNorm
             0.50912386 = fieldWeight in 3555, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.078125 = fieldNorm(doc=3555)
         0.1830604 = weight(abstract_txt:pages in 3555) [ClassicSimilarity], result of:
           0.1830604 = score(doc=3555,freq=4.0), product of:
             0.20900126 = queryWeight, product of:
               2.7225962 = boost
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.013694312 = queryNorm
             0.8758818 = fieldWeight in 3555, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.078125 = fieldNorm(doc=3555)
       0.24 = coord(6/25)
   

5. Bidoki, A.M.Z.; Yazdani, N.: an intelligent ranking algorithm for web pages : DistanceRank (2008) 0.11

   0.10582936 = sum of:
     0.10582936 = product of:
       0.6614335 = sum of:
         0.0097198775 = weight(abstract_txt:that in 3068) [ClassicSimilarity], result of:
           0.0097198775 = score(doc=3068,freq=2.0), product of:
             0.03719945 = queryWeight, product of:
               1.1486217 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013694312 = queryNorm
             0.2612909 = fieldWeight in 3068, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
         0.050844423 = weight(abstract_txt:proposed in 3068) [ClassicSimilarity], result of:
           0.050844423 = score(doc=3068,freq=1.0), product of:
             0.14123258 = queryWeight, product of:
               2.2380831 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.013694312 = queryNorm
             0.36000493 = fieldWeight in 3068, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
         0.15853494 = weight(abstract_txt:pages in 3068) [ClassicSimilarity], result of:
           0.15853494 = score(doc=3068,freq=3.0), product of:
             0.20900126 = queryWeight, product of:
               2.7225962 = boost
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.013694312 = queryNorm
             0.75853586 = fieldWeight in 3068, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.6056433 = idf(docFreq=443, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
         0.44233426 = weight(abstract_txt:crawling in 3068) [ClassicSimilarity], result of:
           0.44233426 = score(doc=3068,freq=2.0), product of:
             0.47416526 = queryWeight, product of:
               4.1008477 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.013694312 = queryNorm
             0.93286943 = fieldWeight in 3068, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
       0.16 = coord(4/25)