TL;DR
Turn the school run and nap time into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
A randomized trial involving 1,179 students in 10 U.K. public schools found that AI tutoring with human oversight performed about as well as expert human tutors on similar follow-up math questions. After 10 weeks, neither tutoring approach improved scores on a broad standardized math test compared with standard practice, and tentative gender differences, accuracy issues and uneven student participation remain concerns.
Ed-tech startup Eedi and Google DeepMind say a randomized trial of 1,179 students across 10 U.K. public schools found that an AI tutor with human oversight matched expert human tutors on short-term follow-up math questions, but neither approach improved scores on a broad standardized test after 10 weeks. The findings complicate claims that AI tutoring can produce lasting academic gains, even as they suggest it can provide some of the immediate help offered by a human tutor.
The study ran from April through July, with secondary school students spending one class period a week — about 30 to 35 minutes — on Eedi’s math platform. A teacher remained in the room. When a student answered incorrectly, the platform randomly assigned one of four responses: standard in-app remediation, text chat with an expert human tutor, AI-generated tutoring messages approved by a human, or the same AI system supplied with a summary of the student’s past misconceptions and work habits. Each student stayed in the same condition throughout the trial.
Students receiving any of the three tutoring approaches answered similar follow-up questions correctly 56% to 58% of the time, compared with 43% after standard remediation. The figures describe performance on similar questions during the platform’s practice, not scores on a broad exam. Bibi Groot, Eedi’s chief impact officer, said the AI and human approaches were “basically on par.”
That immediate result did not translate into higher scores on Star Maths, a standardized test students took before and after the trial. Among the 916 students who sat for both tests, AI-tutored students scored about the same as those receiving static practice without tutoring. Students assigned human tutors scored slightly lower than both groups. Groot said the trial found no Star Maths improvement from interactive tutoring beyond standard remediation.
The study also reported that supplying the AI with personalized summaries of students’ prior work did not meaningfully improve results. Researchers suggested the extra information may have added little because the system already received information about the misconception behind each wrong answer. That explanation is a possible interpretation, not a demonstrated cause.
Immediate Help, No Test-Score Gain
The distinction between answering a related question correctly and improving performance on a wider test is central to the results. The tutoring systems helped students respond to similar follow-up problems more often than standard remediation did, but the trial did not find that this translated into higher Star Maths scores over the study period. Schools weighing tutoring technology may therefore need to distinguish short-term practice support from evidence of broader learning gains.
The comparison also speaks to the potential cost and reach of tutoring. Groot said human tutors are expensive and often beyond school budgets, so an AI system that performs comparably on immediate follow-up questions could be useful. But this trial tested an AI tutor with a human approving every draft message; that arrangement also requires staff time and is not necessarily a scalable substitute for human tutoring. The study does not establish how the AI would perform without that oversight.
Subgroup findings add a further caution. Girls receiving AI or human tutoring tended to do slightly better on Star Maths than girls receiving static practice, while tutored boys tended to do worse, particularly those assigned human tutors. The researchers described these patterns as tentative. They do not establish that tutoring harms boys, but they raise questions about whether the format — including being prompted into a text exchange after each mistake — works equally well for different students.
Products Worth Considering
Garmin Forerunner 55 GPS Running Smartwatch, Black
Levoit Core 600S-P Smart Air Purifier for Large Rooms
Bowflex SelectTech 840 Adjustable Kettlebell
As an affiliate, we earn on qualifying purchases.
A Larger Test of Eedi’s Tutor
The new trial follows a smaller study reported the previous December, in which Eedi said a human-supervised AI tutor slightly outperformed human tutors on new problems in later topics: 66.2% versus 60.7%. The newer trial was seven times larger, according to the report, and included a broad standardized test to examine whether results extended beyond performance on practice questions.
The two studies are not direct proof that the system’s performance rose or fell over time: they were separate trials, and their reported results cover different study conditions and measures. In the new trial, the AI’s short-term results were similar to those of expert human tutors, while the standardized-test comparison was flat for AI tutoring relative to standard practice.
The current study tested human review of AI messages, not fully autonomous tutoring. Groot said Eedi started a separate trial in September involving more than 15,000 students, where the AI interacts directly with students and human help is used if conversations go off track. Results from that separate trial were not provided in the report.
“The AI and human tutor are basically on par.”
— Bibi Groot, Eedi’s chief impact officer
Accuracy, Participation and Gender
The reported subgroup patterns are tentative, and the available account does not establish why girls and boys appeared to respond differently. Researchers suggested that being drawn into a conversation after every mistake could feel like a “signal of failure” to some students, while text chat may favor students more confident in reading and writing. Those are possible explanations, not confirmed causes. The report found no clear differences by family income or age.
The system was not error-free. Human tutors approved 94.7% of AI draft messages without changes and rewrote about one in 20. Researchers found 11 AI messages containing false math statements, seven of which reached students. Human tutors’ messages also contained seven math errors, along with more typing slips. The report says no harmful content reached students; tutors blocked two drafts containing a winking emoji they considered potentially too familiar.
Engagement was also incomplete: the source report says one in five students never replied, but the supplied excerpt ends before giving further details about participation or its effects on results. It is not clear from the available material how non-response varied by tutoring group, what caused it, or whether students who did not reply completed the same amount of work.
The report does not provide enough detail here to judge the size or statistical certainty of the subgroup effects, or how well the findings would generalize beyond these 10 schools. It also does not establish whether the short-term practice advantage would persist, fade, or lead to later gains.
Larger Direct-Use Trial Ahead
Eedi has begun a separate trial with more than 15,000 students, according to Groot. In that study, the AI is expected to interact directly with students, with human support brought in if a conversation goes off track. The reported results are not yet available, so it remains unknown whether that setup can retain the short-term tutoring performance seen with human review while reducing the need for constant approval.
Eedi said it would release the current study’s data, including subgroup results and conversation-level analysis. Those materials may let researchers examine student participation, message accuracy and the tentative differences by gender in more detail. The key next evidence will be whether direct AI tutoring can support learning beyond immediate follow-up questions, and how its outcomes compare with standard practice on broader assessments.
Key Questions
How many students took part in the AI tutor study?
The randomized trial included 1,179 students at 10 U.K. public schools. A total of 916 students sat for both the Star Maths test before and after the trial.
Did the AI tutor improve students’ math test scores?
The study did not find that AI tutoring improved Star Maths scores compared with static practice without tutoring. The AI group did better than standard remediation on similar follow-up questions during practice, which is a different measure.
How did the AI tutor compare with a human tutor?
Students in the AI and human tutoring groups answered similar follow-up questions correctly at comparable rates: 56% to 58% across tutoring approaches. The AI’s draft messages were reviewed by a human before being sent.
Was the AI tutor always accurate?
No. Researchers found 11 AI messages with false math statements, seven of which reached students. Human tutors also made math errors in their messages, according to the report.
What happens next?
Eedi has started a separate trial involving more than 15,000 students, where the AI interacts directly and humans intervene if conversations go off track. Results from that study have not yet been reported.
Source: the74million.org
