Until that hypothesis arrives in which the limits between the creations of one and the other will blur, what we can do now is try to find out if a text has been written by a human or by an AI.
Detect text typed by AI
OpenAI has released an AI text classifier that tries to detect if input content was generated using artificial intelligence tools like ChatGPT. “The AI text classifier is a fitted GPT model that predicts the probability that an AI has generated a text from a variety of sources, such as ChatGPT,” explains a new post on the OpenAI blog.
OpenAI launched the tool after numerous universities and school districts banned the company’s popular AI-powered chatbot, ChatGPT, due to its ability to complete student homeworksuch as writing book reviews, essays, and even finishing programming assignments.
The operation is quite simple. You simply have to go to the website of this verification tool and log in or register (your account will do if you use ChatGPT). After this, you will have to paste the text that you want to verify and the tool will return a result saying if it was probably written by an AI or a human.
“Not 100% reliable,” warns OpenAI
As educators are likely to use this new AI text verification tool to check whether students cheated on their assigned assignments, OpenAI cautions that it should not be used as the “only test” to determine academic dishonesty.
“Our classifier is not totally reliable. In our assessments of a ‘challenge set’ of English texts, our classifier correctly identifies 26% of text typed by AI (true positives) as “probably written by AI”, while incorrectly labeling human-written text as 9% written by AI (false positives). The reliability of our classifier generally improves as the length of the input text increases. Compared to our previously released classifier, this new classifier is significantly more reliable in the text of the latest AI systems.”
In addition, they have shared a number of limitations so that the use of the text classifier can be done in the most effective way possible.
The classifier is very unreliable on short texts (less than 1,000 characters). Even the longest texts are sometimes mislabeled by the classifier. Sometimes the classifier will incorrectly label human-written text as AI-written. It is recommended to use the classifier for English text only. It works significantly worse in other languages and is unreliable in code. Text that is highly predictable cannot be reliably identified. For example, it is impossible to predict whether a list of the first 1,000 prime numbers was written by AI or by humans, because the correct answer is always the same. AI typed text can be edited to bypass the classifier. Classifiers can be updated and retrained based on successful attacks, but it’s unclear if detection has a long-term advantage. Neural network-based classifiers are known to be poorly calibrated outside of their training data. For inputs that are very different from the text in the training set, the classifier is sometimes very confident in an incorrect prediction.
