Document (#43864)

Author

Noever, D.
Ciolino, M.

Title

¬The Turing deception

Source

https%3A%2F%2Farxiv.org%2Fabs%2F2212.06721&usg=AOvVaw3i_9pZm9y_dQWoHi6uv0EN

Year

2022

Abstract

This research revisits the classic Turing test and compares recent large language models such as ChatGPT for their abilities to reproduce human-level comprehension and compelling text generation. Two task challenges- summary and question answering- prompt ChatGPT to produce original content (98-99%) from a single text entry and sequential questions initially posed by Turing in 1950. We score the original and generated content against the OpenAI GPT-2 Output Detector from 2019, and establish multiple cases where the generated content proves original and undetectable (98%). The question of a machine fooling a human judge recedes in this work relative to the question of "how would one prove it?" The original contribution of the work presents a metric and simple grammatical set for understanding the writing mechanics of chatbots in evaluating their readability and statistical clarity, engagement, delivery, overall quality, and plagiarism risks. While Turing's original prose scores at least 14% below the machine-generated output, whether an algorithm displays hints of Turing's true initial thoughts (the "Lovelace 2.0" test) remains unanswerable.

Theme

Computerlinguistik

Object

ChatGPT
Turing-Test

Similar documents (content)

1. Aydin, Ö.; Karaarslan, E.: OpenAI ChatGPT generated literature review: : digital twin in healthcare (2022) 0.19

   0.1948051 = sum of:
     0.1948051 = product of:
       0.81168795 = sum of:
         0.027800057 = weight(abstract_txt:text in 1852) [ClassicSimilarity], result of:
           0.027800057 = score(doc=1852,freq=4.0), product of:
             0.073383465 = queryWeight, product of:
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.01816026 = queryNorm
             0.3788327 = fieldWeight in 1852, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.046875 = fieldNorm(doc=1852)
         0.07346349 = weight(abstract_txt:chatbots in 1852) [ClassicSimilarity], result of:
           0.07346349 = score(doc=1852,freq=1.0), product of:
             0.17672263 = queryWeight, product of:
               1.0973166 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.01816026 = queryNorm
             0.41569942 = fieldWeight in 1852, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.046875 = fieldNorm(doc=1852)
         0.24790676 = weight(abstract_txt:openai in 1852) [ClassicSimilarity], result of:
           0.24790676 = score(doc=1852,freq=10.0), product of:
             0.18454546 = queryWeight, product of:
               1.1213406 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.01816026 = queryNorm
             1.343337 = fieldWeight in 1852, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.046875 = fieldNorm(doc=1852)
         0.05293656 = weight(abstract_txt:human in 1852) [ClassicSimilarity], result of:
           0.05293656 = score(doc=1852,freq=6.0), product of:
             0.098486 = queryWeight, product of:
               1.158479 = boost
               4.681277 = idf(docFreq=1118, maxDocs=44421)
               0.01816026 = queryNorm
             0.5375034 = fieldWeight in 1852, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.681277 = idf(docFreq=1118, maxDocs=44421)
               0.046875 = fieldNorm(doc=1852)
         0.03262873 = weight(abstract_txt:content in 1852) [ClassicSimilarity], result of:
           0.03262873 = score(doc=1852,freq=2.0), product of:
             0.11776285 = queryWeight, product of:
               1.551496 = boost
               4.1796083 = idf(docFreq=1847, maxDocs=44421)
               0.01816026 = queryNorm
             0.2770715 = fieldWeight in 1852, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1796083 = idf(docFreq=1847, maxDocs=44421)
               0.046875 = fieldNorm(doc=1852)
         0.37695235 = weight(abstract_txt:chatgpt in 1852) [ClassicSimilarity], result of:
           0.37695235 = score(doc=1852,freq=13.0), product of:
             0.281707 = queryWeight, product of:
               1.9592944 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.01816026 = queryNorm
             1.3381008 = fieldWeight in 1852, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.046875 = fieldNorm(doc=1852)
       0.24 = coord(6/25)
   

2. Räwel, J.: Automatisierte Kommunikation (2023) 0.11

   0.11392721 = sum of:
     0.11392721 = product of:
       1.4240901 = sum of:
         0.58770794 = weight(abstract_txt:chatbots in 1910) [ClassicSimilarity], result of:
           0.58770794 = score(doc=1910,freq=1.0), product of:
             0.17672263 = queryWeight, product of:
               1.0973166 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.01816026 = queryNorm
             3.3255954 = fieldWeight in 1910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.375 = fieldNorm(doc=1910)
         0.83638215 = weight(abstract_txt:chatgpt in 1910) [ClassicSimilarity], result of:
           0.83638215 = score(doc=1910,freq=1.0), product of:
             0.281707 = queryWeight, product of:
               1.9592944 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.01816026 = queryNorm
             2.9689791 = fieldWeight in 1910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.375 = fieldNorm(doc=1910)
       0.08 = coord(2/25)
   

3. Jha, A.: Why GPT-4 isn't all it's cracked up to be (2023) 0.11

   0.10932816 = sum of:
     0.10932816 = product of:
       0.455534 = sum of:
         0.065329164 = weight(abstract_txt:openai in 1924) [ClassicSimilarity], result of:
           0.065329164 = score(doc=1924,freq=1.0), product of:
             0.18454546 = queryWeight, product of:
               1.1213406 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.01816026 = queryNorm
             0.3540004 = fieldWeight in 1924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.0390625 = fieldNorm(doc=1924)
         0.025469113 = weight(abstract_txt:human in 1924) [ClassicSimilarity], result of:
           0.025469113 = score(doc=1924,freq=2.0), product of:
             0.098486 = queryWeight, product of:
               1.158479 = boost
               4.681277 = idf(docFreq=1118, maxDocs=44421)
               0.01816026 = queryNorm
             0.25860643 = fieldWeight in 1924, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.681277 = idf(docFreq=1118, maxDocs=44421)
               0.0390625 = fieldNorm(doc=1924)
         0.022555616 = weight(abstract_txt:test in 1924) [ClassicSimilarity], result of:
           0.022555616 = score(doc=1924,freq=1.0), product of:
             0.11443136 = queryWeight, product of:
               1.248744 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.01816026 = queryNorm
             0.19711044 = fieldWeight in 1924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.0390625 = fieldNorm(doc=1924)
         0.025767254 = weight(abstract_txt:machine in 1924) [ClassicSimilarity], result of:
           0.025767254 = score(doc=1924,freq=1.0), product of:
             0.12505105 = queryWeight, product of:
               1.3054029 = boost
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.01816026 = queryNorm
             0.20605387 = fieldWeight in 1924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.0390625 = fieldNorm(doc=1924)
         0.1509017 = weight(abstract_txt:chatgpt in 1924) [ClassicSimilarity], result of:
           0.1509017 = score(doc=1924,freq=3.0), product of:
             0.281707 = queryWeight, product of:
               1.9592944 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.01816026 = queryNorm
             0.535669 = fieldWeight in 1924, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.0390625 = fieldNorm(doc=1924)
         0.16551116 = weight(abstract_txt:turing in 1924) [ClassicSimilarity], result of:
           0.16551116 = score(doc=1924,freq=1.0), product of:
             0.4946415 = queryWeight, product of:
               3.1797414 = boost
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.01816026 = queryNorm
             0.33460832 = fieldWeight in 1924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.0390625 = fieldNorm(doc=1924)
       0.24 = coord(6/25)
   

4. Lund, B.D.: ¬A brief review of ChatGPT : its value and the underlying GPT technology (2023) 0.09

   0.09271525 = sum of:
     0.09271525 = product of:
       0.57947034 = sum of:
         0.023166714 = weight(abstract_txt:text in 1874) [ClassicSimilarity], result of:
           0.023166714 = score(doc=1874,freq=1.0), product of:
             0.073383465 = queryWeight, product of:
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.01816026 = queryNorm
             0.3156939 = fieldWeight in 1874, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.078125 = fieldNorm(doc=1874)
         0.13065833 = weight(abstract_txt:openai in 1874) [ClassicSimilarity], result of:
           0.13065833 = score(doc=1874,freq=1.0), product of:
             0.18454546 = queryWeight, product of:
               1.1213406 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.01816026 = queryNorm
             0.7080008 = fieldWeight in 1874, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.078125 = fieldNorm(doc=1874)
         0.036018766 = weight(abstract_txt:human in 1874) [ClassicSimilarity], result of:
           0.036018766 = score(doc=1874,freq=1.0), product of:
             0.098486 = queryWeight, product of:
               1.158479 = boost
               4.681277 = idf(docFreq=1118, maxDocs=44421)
               0.01816026 = queryNorm
             0.36572474 = fieldWeight in 1874, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.681277 = idf(docFreq=1118, maxDocs=44421)
               0.078125 = fieldNorm(doc=1874)
         0.38962653 = weight(abstract_txt:chatgpt in 1874) [ClassicSimilarity], result of:
           0.38962653 = score(doc=1874,freq=5.0), product of:
             0.281707 = queryWeight, product of:
               1.9592944 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.01816026 = queryNorm
             1.3830914 = fieldWeight in 1874, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.078125 = fieldNorm(doc=1874)
       0.16 = coord(4/25)
   

5. Dampz, N.: ChatGPT interpretiert jetzt auch Bilder : Neue Version (2023) 0.08

   0.07676452 = sum of:
     0.07676452 = product of:
       0.63970435 = sum of:
         0.04633343 = weight(abstract_txt:text in 1875) [ClassicSimilarity], result of:
           0.04633343 = score(doc=1875,freq=1.0), product of:
             0.073383465 = queryWeight, product of:
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.01816026 = queryNorm
             0.6313878 = fieldWeight in 1875, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.15625 = fieldNorm(doc=1875)
         0.24487834 = weight(abstract_txt:chatbots in 1875) [ClassicSimilarity], result of:
           0.24487834 = score(doc=1875,freq=1.0), product of:
             0.17672263 = queryWeight, product of:
               1.0973166 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.01816026 = queryNorm
             1.3856648 = fieldWeight in 1875, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.15625 = fieldNorm(doc=1875)
         0.34849256 = weight(abstract_txt:chatgpt in 1875) [ClassicSimilarity], result of:
           0.34849256 = score(doc=1875,freq=1.0), product of:
             0.281707 = queryWeight, product of:
               1.9592944 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.01816026 = queryNorm
             1.2370746 = fieldWeight in 1875, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.15625 = fieldNorm(doc=1875)
       0.12 = coord(3/25)