Generative AI Can Enhance the Quality of Professional Writing by 18%

A study conducted by MIT researchers explored the productivity effects of generative AI (ChatGPT) on professional writing tasks. Findings support the potential of generative AI to augment individual productivity and address broader issues of inequality in workplace performance.

Methodology of the experiment

To assess the impact of ChatGPT on productivity and the quality of professional writing tasks, 444 college-educated professionals were recruited to participate in this study. The participants came from diverse backgrounds including marketing, consulting, and management.

They were assigned two writing tasks related to their occupation, ensuring that professional scenarios from their work were replicated. Each task was for 20–30 minutes and included — writing cover letters for grant applications, emails about organizational restructuring, and plans for customer engagement strategies.

How was the experiment conducted and evaluated?

Participants were randomly divided into two groups. The control group completed their tasks using traditional methods, while the treatment group used ChatGPT only for their second task. The treatment group signed up for ChatGPT and was instructed on how to use it.

To ensure unbiased and rigorous evaluation, the assessment was done by blind evaluation, i.e., the evaluators were not aware of which submissions were completed with AI assistance. The tasks were rated based on various quality metrics, including clarity, innovation, and overall effectiveness. Also, to measure any gains in efficiency, the time spent on each task was meticulously recorded.

In addition to objective outcomes, the experiment also assessed participants’ satisfaction and how they perceived AI impacting their work through follow-up surveys. This well-rounded approach provided a deeper understanding of how AI integration affects professional environments.

Some questions asked to participants were:

1. The treatment group was asked how much they would willingly pay for ChatGPT per month.
2. To rank themselves between 1 to 3 on their skills in communication (writing and speaking), problem-solving, and creativity.

What we can say for sure is generative AI is going to have a big effect on white-collar work. — Shakked Noy, co-author of the study
- Source

Results‍

Here are the key findings from the experiment:

• Time efficiency: The treatment group was able to complete their assignments 10 minutes faster, reducing the average time spent on tasks by about 37% — demonstrating a substantial efficiency gain.
• Quality improvement: An 18% improvement in the quality of work produced by the treatment group suggested that ChatGPT can speed up the writing process as well as enhance the quality of responses. ‍
• Reduction in productivity inequality: Lower-performing individuals from the first task showed greater improvements when using ChatGPT for the second task, indicating that ChatGPT may serve as a leveling tool in professional settings.‍
• Sustained use and perceptions: A follow-up survey revealed continued use of ChatGPT among participants, with 33% of those in the treatment group incorporating ChatGPT into their regular professional tasks. Furthermore, participants reported an increase in job satisfaction and a positive shift in their attitudes toward AI and automation technologies.‍

Takeaway- 1: Effect of ChatGPT on task structure

ChatGPT allows to reduce the time in drafting by about 50% and spend more time editing indicating a shift of focus from creation to modification, source

The integration of ChatGPT notably altered the structure of writing tasks and time allocated:

• Before treatment: Initially, participants allocated approximately 25% of their time to brainstorming, 50% to drafting, and the remaining 25% to editing their work.
• After treatment: With ChatGPT, the time spent on drafting was more than halved, while editing time more than doubled.

This shift indicates that while the drafting phase became significantly less time-consuming due to AI assistance, participants chose to allocate more time to refining and editing the response — suggesting a shift in task focus from creation to modification.

The experiment demonstrates that it does bring significant speed benefits, even if those speed benefits are lesser in the real world because you need to spend time fact-checking and writing the prompts. — Noy says.

Takeaway — 2: Benefits of ChatGPT and skill demand

Professionals are willing to pay 0.5% of their salary for a ChatGPT subscription, however, the statistics don’t show any substantial improvement in writing, source

ChatGPT has the potential to help those with poor writing skills but strong idea-generation skills, expand their occupational choices.

Findings:

• ChatGPT subscription: Regardless of the participants’ written skills, they were willing to pay 0.5% of their monthly salary for the ChatGPT subscription.
• Grade gains: Improvement in written skill was not significant. This means people with poor writing skills do not experience exceptionally high levels of grade gains with ChatGPT.

Explore how to use ChatGPT to create a marketing strategy.‍

Takeaway — 3: Perception of automation after using ChatGPT

On using ChatGPT, the treatment group is more worried than the control group about being replaced as well as optimistic about the changes with AI, source

Amongst the treatment group, before the study, 30% of them had never heard of ChatGPT and 70% had never used it, making this experiment their first encounter with the AI tool. After using ChatGPT for writing tasks, we surveyed their views on how AI impacts their professions on a scale of 1 to 10. The results showed:

• Concerns about AI and jobs: A slight increase in worry about AI replacing human jobs, with concern rising by 0.26 standard deviations (p = 0.006).
• Optimism about AI and productivity: Participants showed increased optimism about AI’s ability to enhance productivity, with excitement rising by 0.39 standard deviations (p < 0.001).
• General outlook on AI: Overall optimism about the future of AI increased moderately, by about 0.20 standard deviations (p = 0.037).

Key learnings

Overall, there is no doubt that ChatGPT and other generative AI technologies can improve productivity and work quality in professional writing tasks. It helps low-skilled workers reduce the time spent on completing tasks. This allows high-skilled employees maintain their work quality and finish work faster.

Even business leaders, hiring managers, and candidates are increasingly accepting of AI-generated content.

ChatGPT can aid in brainstorming and drafting, yet it necessitates more time for editing, indicating a shift towards enhancing writers’ editing skills within organizations. Furthermore, the content produced by language models requires factual verification, pointing to the need for roles specializing in fact-checking and research.

There are also concerns about job security with AI, however, optimism about AI improving productivity and future advancements remains strong. This nuanced perspective on AI in the workplace suggests a thoughtful integration of AI technologies, balancing efficiency gains with the broader implications for employment and workplace dynamics.

As AI continues to evolve, ongoing research will be essential to maximize its benefits while addressing potential challenges.

Do you agree with the findings of this study? Let us know your thoughts.

‍