Yes, the UK government put Microsoft CoPilot on trial in one department for a 3 month period. And the bureaucrats produced an interesting assessment on the results: apps most used with CoPilot, types of jobs used with AI, time savings, quality assurance but they missed an important assessment.
UK puts CoPilot on trial
The UK department for Business and Trade obtained 1,000 licenses between October and December 2024. Around 300 of the participants agreed to their data being analyzed to produce the report.
So what did the employees use CoPilot for successfully? The top three tasks:
- transcribing/summarizing meetings
- writing an email
- summarizing written communications
More details on CoPilot useage in the chart below.
The apps that CoPilot made more difficult or time consuming? PowerPoint and Excel. Also users found CoPilot did not help with scheduling and picture creation.
What about CoPilot hallucinations?
The study found that while most users were aware of the risk of CoPilot hallucinations or errors the users did not always check the answers that CoPilot provided.
Users were more likely to check for CoPilot if they shared the output with others. If the output was only for their own work, users were unlikely to check for CoPilot errors.
One thing that is missing from this study is how would the end users know if CoPilot hallucinated? What safeguards did the department put in place to make sure hallucinations didn’t end up in government documents?
Not all CoPilot users are equal
One part of study I found fascinating is that CoPilot definitely helped certain distinct groups of employees. People with English as a second language, neuro-divergent folks and employees with visual or hearing disabilities found CoPilot “levelled the playing field”.
One user for whom English is not their first language reported that M365 CoPilot “allowed them to communicate in a more effective and professional manner while enhancing the quality of their written outputs.”
Training required
The study also looked at what amount of training provided the best results. I summarize a complicated table but basically one hour of group training and one of hour of self-led personal training provided best results for employees.
This leads me to the one shocking omission from the study.
Where is the Return on Investment?
Where is the ROI in this study? Nowhere. The study authors admit they did not compare the time savings from CoPilot versus the cost of setting up CoPilot as the money for value assessment is being conducted separately.
What numbers should be shown here? Take the average hourly wage of employees, let’s say $25 an hour, add the time savings reported by users. Let’s say 1.5 hours x$25 a month and subtract the cost of monthly license for CoPilot $48 (rounded off). Then add the cost of training, the hours lost from the employee for training and cost of deploying and supporting CoPilot.
Not sure how you would figure this out, but one could add the monetary bonus of English not first language folks, neurodivergent employees and workers with visual/ hearing disabilities getting higher job performance and satisfaction.
While we await the money for value assessment, this study provided a lot of valuable information about how employees use CoPilot, the amount of training costs required and where CoPilot is most useful. The authors put great effort in avoiding biases, discounting the AI promoters to generate information on software useage.
You can access the UK puts CoPilot on trial PDF here.
