Generative AI is the tool that launched a million thought-pieces in the education world and beyond. This makes sense, as despite its current limitations and endless cliches and somewhat unsettling voice, it is still a remarkable, society-altering new invention that our students will encounter, use, and have to contend with throughout their lives. 

In the nearly eighteen months since it burst onto the scene, I’ve been wary of joining the cacophony of here’s-how-AI-will change-education pieces because there is still so much we don’t know, but as I watched a student of mine last week just blindly accept some rather massive changes to her writing being offered to her by Grammarly, I realized it was time to weigh-in on a question that I’ve gotten a lot from students, parents, and districts/schools I’ve worked with over the last year*:

Should we encourage students to use AI to get more feedback on their writing beyond what we can offer?

The backstory to this question is that you can ask ChatGPT or any other AI bot to give you feedback on a piece of writing, and it will do it in mere seconds. And in recent months, AI-supported programs like Grammarly and Google have grown more brazen and automatic when it comes to their “suggestions” for what and how writers should write as they compose. They (at least Grammarly, but I assume there are others) have even begun to act in a way that reminds me of video-game-upselling: The free grammar-check version will underline things and tell you that if you upgrade to Premium, it will offer more comprehensive whole-sentence and tonal fixes. 

Many of my students report that the pushiness of these programs has gotten so bad that they have uninstalled them or turned off suggestions so they can compose in peace.  

All year, I’ve found myself torn about the use of these tools for offering feedback. On the one hand, one of the most frustrating parts of being a composition teacher is that no matter how efficient one gets with feedback, there is still a hard cap on how much one-to-one feedback we can provide. A tool to provide more quality feedback to supplement our feedback would be really impactful. At the same time though, I fear that AI might give bad feedback, which can often be worse than no feedback at all. 

I finally came to some answers last month when I read a study from the June 2024 volume of the Journal of Learning and Instruction that sought to answer those exact questions. In it, the researchers (who include writing instruction titans like Steve Graham and Carol Booth Olson) compared the feedback of sixteen experienced educators with the feedback given by a ChatGPT 3.5, and the results were pretty clear: The human feedback was meaningfully better in four of five categories that the researchers selected as being markers of strong feedback (clarity, accuracy, tone, and the ability to prioritize). The only category where AI scored higher was on the ability to stay focused on the criteria, which makes sense because computer programs generally don’t stray far from the original task.

Here is a chart of the results (and you can read the study here courtesy of Science Direct):

These results—that humans are still being better than AI at higher-order language—aren’t exactly paradigm-shifting, but there were a few things that stood out to me as I read the study:

• While Chat GPT wasn’t as strong as humans, it wasn’t terribly worse. According to the study authors, in many cases it still gave pretty decent feedback. 
• The authors also noted that a huge advantage was that ChatGPT gave the feedback in seconds while it took the humans, who averaged 20-25 minutes per paper, several weeks to provide all the feedback.    

These findings raise the question of whether AI, while clearly not ready to replace humans as the sole purveyor of feedback, might indeed be a useful digital assistant that can offer additional feedback alongside teachers.

I was pondering this concept as I read the study until I stumbled across this line that didn’t just throw my whole maybe-AI-could-assist-in-feedback train of thought into question, it crashed the train into a brick wall: 

We did, however, see differences in the quality of feedback given across high-, average-, and low-scoring essays…ChatGPT struggled with accuracy when providing feedback on the high-quality papers and with maintaining a supportive tone on the low-quality papers. –Steiss et al., 2024

Here the study authors offer a deeply important clarification to the data: While AI was surprisingly decent at offering suggestions to papers they considered average, it fell apart when it came to offering feedback to very high or very low scoring essays.  

This struggle with responding to outliers is exactly what I found in my own experiments with AI. I have found that it invariably tries to make really strong and interesting pieces of writing—writing with lots of voice and style and perspective—boring. This tracks with how AI works. It is essentially a prediction machine, so by design it seeks to make writing more predictable. And its issues with tone (for example type in something like “heartfelt breakup letter” into a Generative AI and see what happens) are well documented. 

In the end, the researchers ultimately don’t weigh in with a clear yes or no on whether we should encourage students to use AI for feedback. Instead, they recommend, “developing students’ and teachers’ AI literacy.” 

This notion of teaching about the tool and its strengths, problems, potential applications, and potential pitfalls makes a lot of sense to me, as AI is only going to grow more embedded in the world. And understanding something better allows one to drive it instead of being driven by it. 

For my part though, as a teacher of high school freshmen, I don’t think I will be encouraging them to use it for feedback just yet because I see two fatal flaws in how it currently works:

1. One of my core goals is for students to value the unique power of their voices and to use those unique voices to connect, move, and change the world. Anything that pushes students away from their voices and towards the paths more trodden gives me great pause. 
2. Many young writers come into my class with fraught relationships with writing. Often, they need to be seen and heard to grow to their full potential, and AI can’t actually see or hear them. And so, while AI might offer them accurate suggestions about comma splices and formatting quotes, I fear it won’t offer them the right responses about how to best move forward into better relationships with writing.

---

This is new territory for us all, so if you have had your own experiments with AI-feedback, I’d love to hear about them and maybe even feature you here!

Thanks for reading, and yours in teaching,
Matt

If you liked this…

Join my mailing list and you will receive a thoughtful post about finding balance and success as a writing teacher each week along with exciting subscriber-only content. Also, as an additional thank you for signing up, you will also receive a short ebook on how to cut feedback time without cutting feedback quality that is adapted from my book Flash Feedback: Responding to Student Writing Better and Faster – Without Burning Out from Corwin Literacy.