Document (#43012)

Author

Zou, J.
Thoma, G.
Antani, S.

Title

Unified deep neural network for segmentation and labeling of multipanel biomedical figures

Source

Journal of the Association for Information Science and Technology. 71(2020) no.11, S.1327-1340

Year

2020

Abstract

Recent efforts in biomedical visual question answering (VQA) research rely on combined information gathered from the image content and surrounding text supporting the figure. Biomedical journals are a rich source of information for such multimodal content indexing. For multipanel figures in these journals, it is critical to develop automatic figure panel splitting and label recognition algorithms to associate individual panels with text metadata in the figure caption and the body of the article. Challenges in this task include large variations in figure panel layout, label location, size, contrast to background, and so on. In this work, we propose a deep convolutional neural network, which splits the panels and recognizes the panel labels in a single step. Visual features are extracted from several layers at various depths of the backbone neural network and organized to form a feature pyramid. These features are fed into classification and regression networks to generate candidates of panels and their labels. These candidates are merged to create the final panel segmentation result through a beam search algorithm. We evaluated the proposed algorithm on the ImageCLEF data set and achieved better performance than the results reported in the literature. In order to thoroughly investigate the proposed algorithm, we also collected and annotated our own data set of 10,642 figures. The experiments, trained on 9,642 figures and evaluated on the remaining 1,000 figures, show that combining panel splitting and panel label recognition mutually benefit each other.

Content

https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24334.

Theme

Visualisierung

Field

Medizin

Similar documents (content)

1. Apostolova, E.; You, D.; Xue, Z.; Antani, S.; Demner-Fushman, D.; Thoma, G.R.: Image retrieval from scientific publications : text and image content processing to separate multipanel figures (2013) 0.40

   0.39840186 = sum of:
     0.39840186 = product of:
       1.4228637 = sum of:
         0.08144155 = weight(abstract_txt:caption in 1740) [ClassicSimilarity], result of:
           0.08144155 = score(doc=1740,freq=2.0), product of:
             0.10521902 = queryWeight, product of:
               1.0274086 = boost
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.01169484 = queryNorm
             0.7740193 = fieldWeight in 1740, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
         0.092769064 = weight(abstract_txt:labels in 1740) [ClassicSimilarity], result of:
           0.092769064 = score(doc=1740,freq=2.0), product of:
             0.14459136 = queryWeight, product of:
               1.7032651 = boost
               7.2588162 = idf(docFreq=84, maxDocs=44421)
               0.01169484 = queryNorm
             0.64159477 = fieldWeight in 1740, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.2588162 = idf(docFreq=84, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
         0.08586104 = weight(abstract_txt:segmentation in 1740) [ClassicSimilarity], result of:
           0.08586104 = score(doc=1740,freq=1.0), product of:
             0.1730139 = queryWeight, product of:
               1.8631661 = boost
               7.9402676 = idf(docFreq=42, maxDocs=44421)
               0.01169484 = queryNorm
             0.49626672 = fieldWeight in 1740, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.9402676 = idf(docFreq=42, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
         0.1831356 = weight(abstract_txt:panels in 1740) [ClassicSimilarity], result of:
           0.1831356 = score(doc=1740,freq=1.0), product of:
             0.3281677 = queryWeight, product of:
               3.142712 = boost
               8.928879 = idf(docFreq=15, maxDocs=44421)
               0.01169484 = queryNorm
             0.5580549 = fieldWeight in 1740, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.928879 = idf(docFreq=15, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
         0.16601554 = weight(abstract_txt:figure in 1740) [ClassicSimilarity], result of:
           0.16601554 = score(doc=1740,freq=1.0), product of:
             0.33831874 = queryWeight, product of:
               3.6845894 = boost
               7.85132 = idf(docFreq=46, maxDocs=44421)
               0.01169484 = queryNorm
             0.4907075 = fieldWeight in 1740, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.85132 = idf(docFreq=46, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
         0.39243734 = weight(abstract_txt:figures in 1740) [ClassicSimilarity], result of:
           0.39243734 = score(doc=1740,freq=5.0), product of:
             0.37819904 = queryWeight, product of:
               4.3555326 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.01169484 = queryNorm
             1.0376476 = fieldWeight in 1740, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
         0.4212036 = weight(abstract_txt:panel in 1740) [ClassicSimilarity], result of:
           0.4212036 = score(doc=1740,freq=3.0), product of:
             0.49951088 = queryWeight, product of:
               5.4833274 = boost
               7.7894444 = idf(docFreq=49, maxDocs=44421)
               0.01169484 = queryNorm
             0.8432321 = fieldWeight in 1740, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.7894444 = idf(docFreq=49, maxDocs=44421)
               0.0625 = fieldNorm(doc=1740)
       0.28 = coord(7/25)
   

2. Ru, C.; Tang, J.; Li, S.; Xie, S.; Wang, T.: Using semantic similarity to reduce wrong labels in distant supervision for relation extraction (2018) 0.13

   0.12780344 = sum of:
     0.12780344 = product of:
       0.45644087 = sum of:
         0.022706578 = weight(abstract_txt:features in 55) [ClassicSimilarity], result of:
           0.022706578 = score(doc=55,freq=2.0), product of:
             0.056577187 = queryWeight, product of:
               1.0654467 = boost
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.01169484 = queryNorm
             0.40133804 = fieldWeight in 55, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
         0.0654038 = weight(abstract_txt:convolutional in 55) [ClassicSimilarity], result of:
           0.0654038 = score(doc=55,freq=1.0), product of:
             0.11453605 = queryWeight, product of:
               1.0719318 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.01169484 = queryNorm
             0.5710324 = fieldWeight in 55, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
         0.01678209 = weight(abstract_txt:proposed in 55) [ClassicSimilarity], result of:
           0.01678209 = score(doc=55,freq=1.0), product of:
             0.058270346 = queryWeight, product of:
               1.0812718 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.01169484 = queryNorm
             0.28800395 = fieldWeight in 55, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
         0.035970118 = weight(abstract_txt:network in 55) [ClassicSimilarity], result of:
           0.035970118 = score(doc=55,freq=2.0), product of:
             0.088009976 = queryWeight, product of:
               1.6275061 = boost
               4.6239696 = idf(docFreq=1184, maxDocs=44421)
               0.01169484 = queryNorm
             0.40870503 = fieldWeight in 55, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6239696 = idf(docFreq=1184, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
         0.14668077 = weight(abstract_txt:labels in 55) [ClassicSimilarity], result of:
           0.14668077 = score(doc=55,freq=5.0), product of:
             0.14459136 = queryWeight, product of:
               1.7032651 = boost
               7.2588162 = idf(docFreq=84, maxDocs=44421)
               0.01169484 = queryNorm
             1.0144504 = fieldWeight in 55, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.2588162 = idf(docFreq=84, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
         0.047769703 = weight(abstract_txt:algorithm in 55) [ClassicSimilarity], result of:
           0.047769703 = score(doc=55,freq=1.0), product of:
             0.13397251 = queryWeight, product of:
               2.008004 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.01169484 = queryNorm
             0.35656348 = fieldWeight in 55, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
         0.12112781 = weight(abstract_txt:neural in 55) [ClassicSimilarity], result of:
           0.12112781 = score(doc=55,freq=2.0), product of:
             0.19772726 = queryWeight, product of:
               2.439439 = boost
               6.930783 = idf(docFreq=117, maxDocs=44421)
               0.01169484 = queryNorm
             0.61260045 = fieldWeight in 55, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.930783 = idf(docFreq=117, maxDocs=44421)
               0.0625 = fieldNorm(doc=55)
       0.28 = coord(7/25)
   

3. Seeber, M.; Vlegels, J.; Cattaneo, M.: Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluation (2022) 0.11

   0.11435867 = sum of:
     0.11435867 = product of:
       0.7147417 = sum of:
         0.008295468 = weight(abstract_txt:these in 1637) [ClassicSimilarity], result of:
           0.008295468 = score(doc=1637,freq=1.0), product of:
             0.041700497 = queryWeight, product of:
               1.1202819 = boost
               3.1828754 = idf(docFreq=5006, maxDocs=44421)
               0.01169484 = queryNorm
             0.19892971 = fieldWeight in 1637, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1828754 = idf(docFreq=5006, maxDocs=44421)
               0.0625 = fieldNorm(doc=1637)
         0.04533476 = weight(abstract_txt:evaluated in 1637) [ClassicSimilarity], result of:
           0.04533476 = score(doc=1637,freq=2.0), product of:
             0.0897073 = queryWeight, product of:
               1.3416059 = boost
               5.717531 = idf(docFreq=396, maxDocs=44421)
               0.01169484 = queryNorm
             0.5053631 = fieldWeight in 1637, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.717531 = idf(docFreq=396, maxDocs=44421)
               0.0625 = fieldNorm(doc=1637)
         0.31720015 = weight(abstract_txt:panels in 1637) [ClassicSimilarity], result of:
           0.31720015 = score(doc=1637,freq=3.0), product of:
             0.3281677 = queryWeight, product of:
               3.142712 = boost
               8.928879 = idf(docFreq=15, maxDocs=44421)
               0.01169484 = queryNorm
             0.96657944 = fieldWeight in 1637, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.928879 = idf(docFreq=15, maxDocs=44421)
               0.0625 = fieldNorm(doc=1637)
         0.3439113 = weight(abstract_txt:panel in 1637) [ClassicSimilarity], result of:
           0.3439113 = score(doc=1637,freq=2.0), product of:
             0.49951088 = queryWeight, product of:
               5.4833274 = boost
               7.7894444 = idf(docFreq=49, maxDocs=44421)
               0.01169484 = queryNorm
             0.6884961 = fieldWeight in 1637, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.7894444 = idf(docFreq=49, maxDocs=44421)
               0.0625 = fieldNorm(doc=1637)
       0.16 = coord(4/25)
   

4. Huang, H.-H.; Wang, J.-J.; Chen, H.-H.: Implicit opinion analysis : extraction and polarity labelling (2017) 0.11

   0.11066836 = sum of:
     0.11066836 = product of:
       0.5533418 = sum of:
         0.09810569 = weight(abstract_txt:convolutional in 4820) [ClassicSimilarity], result of:
           0.09810569 = score(doc=4820,freq=1.0), product of:
             0.11453605 = queryWeight, product of:
               1.0719318 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.01169484 = queryNorm
             0.8565486 = fieldWeight in 4820, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.09375 = fieldNorm(doc=4820)
         0.07340867 = weight(abstract_txt:deep in 4820) [ClassicSimilarity], result of:
           0.07340867 = score(doc=4820,freq=1.0), product of:
             0.11893809 = queryWeight, product of:
               1.5447974 = boost
               6.5834737 = idf(docFreq=166, maxDocs=44421)
               0.01169484 = queryNorm
             0.6172007 = fieldWeight in 4820, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.5834737 = idf(docFreq=166, maxDocs=44421)
               0.09375 = fieldNorm(doc=4820)
         0.05395518 = weight(abstract_txt:network in 4820) [ClassicSimilarity], result of:
           0.05395518 = score(doc=4820,freq=2.0), product of:
             0.088009976 = queryWeight, product of:
               1.6275061 = boost
               4.6239696 = idf(docFreq=1184, maxDocs=44421)
               0.01169484 = queryNorm
             0.61305755 = fieldWeight in 4820, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6239696 = idf(docFreq=1184, maxDocs=44421)
               0.09375 = fieldNorm(doc=4820)
         0.18169172 = weight(abstract_txt:neural in 4820) [ClassicSimilarity], result of:
           0.18169172 = score(doc=4820,freq=2.0), product of:
             0.19772726 = queryWeight, product of:
               2.439439 = boost
               6.930783 = idf(docFreq=117, maxDocs=44421)
               0.01169484 = queryNorm
             0.91890067 = fieldWeight in 4820, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.930783 = idf(docFreq=117, maxDocs=44421)
               0.09375 = fieldNorm(doc=4820)
         0.14618056 = weight(abstract_txt:label in 4820) [ClassicSimilarity], result of:
           0.14618056 = score(doc=4820,freq=1.0), product of:
             0.21549948 = queryWeight, product of:
               2.546712 = boost
               7.2355595 = idf(docFreq=86, maxDocs=44421)
               0.01169484 = queryNorm
             0.6783337 = fieldWeight in 4820, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2355595 = idf(docFreq=86, maxDocs=44421)
               0.09375 = fieldNorm(doc=4820)
       0.2 = coord(5/25)
   

5. Wang, P.; Li, X.: Assessing the quality of information on Wikipedia : a deep-learning approach (2020) 0.10

   0.10454816 = sum of:
     0.10454816 = product of:
       0.3733863 = sum of:
         0.022706578 = weight(abstract_txt:features in 505) [ClassicSimilarity], result of:
           0.022706578 = score(doc=505,freq=2.0), product of:
             0.056577187 = queryWeight, product of:
               1.0654467 = boost
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.01169484 = queryNorm
             0.40133804 = fieldWeight in 505, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
         0.0654038 = weight(abstract_txt:convolutional in 505) [ClassicSimilarity], result of:
           0.0654038 = score(doc=505,freq=1.0), product of:
             0.11453605 = queryWeight, product of:
               1.0719318 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.01169484 = queryNorm
             0.5710324 = fieldWeight in 505, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
         0.01678209 = weight(abstract_txt:proposed in 505) [ClassicSimilarity], result of:
           0.01678209 = score(doc=505,freq=1.0), product of:
             0.058270346 = queryWeight, product of:
               1.0812718 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.01169484 = queryNorm
             0.28800395 = fieldWeight in 505, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
         0.011731563 = weight(abstract_txt:these in 505) [ClassicSimilarity], result of:
           0.011731563 = score(doc=505,freq=2.0), product of:
             0.041700497 = queryWeight, product of:
               1.1202819 = boost
               3.1828754 = idf(docFreq=5006, maxDocs=44421)
               0.01169484 = queryNorm
             0.2813291 = fieldWeight in 505, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1828754 = idf(docFreq=5006, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
         0.084765024 = weight(abstract_txt:deep in 505) [ClassicSimilarity], result of:
           0.084765024 = score(doc=505,freq=3.0), product of:
             0.11893809 = queryWeight, product of:
               1.5447974 = boost
               6.5834737 = idf(docFreq=166, maxDocs=44421)
               0.01169484 = queryNorm
             0.7126819 = fieldWeight in 505, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.5834737 = idf(docFreq=166, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
         0.05086943 = weight(abstract_txt:network in 505) [ClassicSimilarity], result of:
           0.05086943 = score(doc=505,freq=4.0), product of:
             0.088009976 = queryWeight, product of:
               1.6275061 = boost
               4.6239696 = idf(docFreq=1184, maxDocs=44421)
               0.01169484 = queryNorm
             0.5779962 = fieldWeight in 505, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.6239696 = idf(docFreq=1184, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
         0.12112781 = weight(abstract_txt:neural in 505) [ClassicSimilarity], result of:
           0.12112781 = score(doc=505,freq=2.0), product of:
             0.19772726 = queryWeight, product of:
               2.439439 = boost
               6.930783 = idf(docFreq=117, maxDocs=44421)
               0.01169484 = queryNorm
             0.61260045 = fieldWeight in 505, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.930783 = idf(docFreq=117, maxDocs=44421)
               0.0625 = fieldNorm(doc=505)
       0.28 = coord(7/25)