3 Approaches to Detect AI Writing

As we discussed earlier this week, artificial intelligence was by far the biggest and most important plagiarism story of 2022 and likely will be in 2023.

The issue is simple. AI writing has evolved to a point where it can be difficult to distinguish between text written by a machine and text written by a human. 

That has been further compounded by the release of ChatGPT in November 2022. ChatGPT is a high-end AI writing tool created by the team at OpenAI that has been made free to use and available to the public. 

While the lion’s share of the focus has understandably been on the academic implications of this, it can be an issue for any kind of writing. For example, The Washington Post has used an AI reporter for over half a decade. 

Still, the academic world is where the concern the strongest and, as such, it’s worth focusing on.

However, that concern may be somewhat overstated, at least at this time. While it is true that Ai writing does pose serious challenges for academia, there are currently several ways to detect AI writing.

To that end, we’re going to look at three of the main ones available right now.

1: Direct Detection

Sometimes the best way to detect and AI is with another AI. 

Several tools are available that use AI to examine the patterns in the writing itself and determine if it was written by a fellow bot. A Princeton student recently made headlines for developing an anti-ChatGPT tool (the tool itself is not working reliably due to high traffic). However, such tools have existed for a long time, including the RoBERTa Base OpenAI Detector, which uses OpenAI’s own models to detect AI writing. 

This is technology that will likely grow and evolve alongside AI writing. Whether it keeps pace with it or not is impossible to say, but in my limited testing, it does a good job of detecting works that are written by AI, even if a human edits and rewrites it.

Advantages: 

• Current tools are free and easy to use. 
• They are generally very reliable at detecting AI writing.
• The technology is evolving alongside AI writing.

Disadvantages: 

• The tools exist outside of current assessment workflows
• Taking any action based upon information provided by them may be difficult as the results are not transparent
• Limited to just AI detection and often targeted at specific AIs

2: Authorship Detection

Authorship detection is the same concept as AI detection, but from the opposite direction. 

Here, you feed AI a large amount of text that is known to be written by the student. It then compares new writing submitted by them to look for drastic changes in language and flag anything that is suspicious.

This is the approach that has been adopted by both Turnitin and Unicheck, who have similar systems designed to not to detect AI writing, but whether a different author (human or AI), wrote the paper than the student who is claiming it.

Advantages: 

• Already baked into current assessment systems
• Can detect essay mill and other forms of contract cheating
• Simple to use and already familiar to many instructors

Disadvantages:

• Access to such systems can be expensive and difficult
• Requires a large amount of writing known to be by the student
• Taking any action based upon information provided by them may be difficult as the results are not transparent

3: Creating AI-Resistant Assignments

The final approach doesn’t involve examining the work, but in creating assignments that an AI would struggle with. 

AI can only write about things that it has access to as part of its data set. This, in general, means things that are available on the public internet. An AI won’t and can’t know things that are specific to your classroom, unique to the student or books and information that is not available online.

Creating “Google Resistant” assignments has been a good practice for some time, as it reduces copy/paste plagiarism and makes it more difficult for students to use essay mills. Many of the same techniques work on AI, making it difficult for students to use AI and making it more obvious when they do by filling the paper with inaccurate information.

Best of all, this approach can be used in conjunction with other tools, providing an extra layer of both deterrence and detection.

Advantages: 

• Relatively easy to apply in class with little change to assessment
• Can be combined with other tools to provide an additional layer of protection
• Often creates more engaging and more interesting assignments for students

Disadvantages:

• Can’t guarantee detection of AI writing, especially if students are struggling with writing
• May require more work on assessment, a difficult proposition in many classrooms
• May be less helpful as AI writing advances

Why I Didn’t Mention Watermarking

One approach that has generated headlines has been the idea of “watermarking” AI writing to make it easier to detect.

This approach has promise and advantages. The biggest would be that it would remove the uncertainty we find in current detection methods. The watermark is either present or it is not.

However, there are many questions about this approach that are unanswered and won’t be until it’s developed and released. For example, how well does it hold up to rewriting? How much text is needed to read the watermark? What information will be in that watermark? How easy is the watermark to remove? And so forth.

But the biggest problem is that it only applies to AIs that choose to implement it. All it takes is one AI system to decide they don’t care, and we’re back to square one. 

While there’s a some interesting potential here, there are simply too many unknowns, and it appears to be a short-term answer for a long-term problem. 

Bottom Line

The situation around AI writing is evolving quickly. However, there’s no call for complete panic at this time. The essay is not dead and, while there are good reasons to be worried and expect changes, a sense of doom is not warranted.

AI writing is a problem we were always going to face. Though ChatGPT brought it to the forefront faster than most expected, the truth is it doesn’t matter whether it happened in November 2022 or November 2032, it was always going to happen and academia was always going to be under prepared.

Fortunately, the fight against essay mills has given academia some preparation and, in that context, AI just represents a shift in that ongoing fight.

So, while academia has much to be worried about, it also has reasons to hope and access to some good tools to combat the problem, at least for right now.