Technical Writing: Complement or compete with Generative AI?

ChatGPT, Bard, Bing, Midjourney, and DALL-E — these are buzzwords now. With every flashy new Generative AI tool that hits the scene, it makes you think: are computers about to take our jobs? Some see Gen AI as revolutionary, a technological spark. Others view it as Pandora’s box, a looming crisis, a whirlwind where everything moves too fast. Moreover, a growing number of companies are banning Gen AI tools from being used internally, including Apple, Samsung, and Amazon, indicating cybersecurity risks, employee ethical standards, and regulatory compliance challenges.

But should we really be afraid? How can AI impact IT professionals? And what about us, Technical Writers? We often pride ourselves on our ability to convey complex technical concepts in clear and comprehensible language. With the advent of AI-driven tools capable of creating new content, there’s an underlying concern. Could these tools replicate the nuances and understanding that a human writer brings to the table? Or might they simply augment the writing process, assisting in research, content organization, or even basic draft creation? Let’s trace back to the origins to understand better.

Our own research

Recognizing the importance of keeping up with the times, we at SoftServe have embarked on research to explore whether and how Gen AI can enhance the productivity and quality of work of IT professionals engaged in SDLC projects. This included Software and Quality Assurance Engineers, Business Analysts, Project and Product Managers, and, of course, Technical Writers 😊

The methodology of this research revolves around exploring specific tasks and how and to what extent Gen AI can help with their execution.

In our research, we decided to concentrate on separate tasks that Technical Writers perform on a daily basis rather than focusing on large and time-consuming tasks. It allowed us to make it easier to compare results and assess the impact of Gen AI.

Out of the many tasks that were considered, we chose to go with:

1. Creating release notes
2. Creating a topic based on a user story
3. Creating a news post
4. Reviewing a piece of text

To easily evaluate the success of each task, we defined a set of precise and measurable requirements. These requirements served as a shared understanding, ensuring that everyone involved in the process knows what is expected of them and what constitutes a successful result.

For example, here are some of the requirements for creating a topic based on a user story:

• Microsoft Writing Style Guide is followed.
• A descriptive heading is used to create a clear picture of what this topic is about.
• 1–2 intro sentences about new functionality are added.
• Step-by-step instruction (7–10 steps) with the needed level of detail is added.

Among a variety of Gen AI tools available on the market, we used the latest version of ChatGPT — GPT-4 as it is easy to use and quick to learn.

Specific tasks were assigned then to pairs of individuals with similar maturity levels. One individual in each pair performed the task using AI, while the other performed the task without AI. Participants performing the tasks using AI were asked to create a conversation with ChatGPT to discuss the task-related questions where AI can provide support.

For every individual task, we assigned an expert whose responsibility was to ensure that the tasks met all the requirements. They then evaluated the task execution and the quality of outcomes from both groups, those with and without Gen AI. Results not adhering to the established requirements were omitted from the final calculation. Experts also looked at how participants talked with AI and pointed out which prompts worked well, and which didn’t.

Results

Our study gave us a clear picture of how Gen AI performs in different technical writing tasks. While there were moments of increased speed, there were clear challenges too. Let’s dive in.

• Release notes. We observed a 35% boost in productivity when it comes to crafting release notes. ChatGPT works better if creating release notes in a more marketing style. When generating standard release notes that detail fixes and improvements for a software product or service, ChatGPT benefits from having clear guidelines or sample sentences you want it to follow. Prompts matter here a lot.
• Topic. Here we could see a productivity increase of 20%. As ChatGPT’s last training cut-off is September 2021, it is not aware of changes and updates that have been made to the documentation guidelines, namely the Microsoft Writing Style Guide we follow. A lot of time went into fixing what ChatGPT wrote. That’s why, to receive a good result, the prompt should be super detailed and informative, with a list of all the rules you want ChatGPT to follow. In this way, you’ll receive a very good draft of your future topic.
• Post. With a productivity increase of 28%, ChatGPT showed its capability to generate compelling social media content. However, a key observation was that AI performs best when equipped with thorough background details. So, although it has the capacity to create relevant posts, the output’s quality directly depends on the depth of information provided. Otherwise, a post will be too general. Have I already said that prompts matter? 😊
• Document review. The document review process witnessed a productivity increase of 33% with the assistance of ChatGPT. Yet, there were challenges. ChatGPT doesn’t highlight all the edits it makes. Moreover, as it was already said, it lacks the latest information on technical writing guidelines. Also, ChatGPT cannot review a big piece of content, for example, a user guide, as it has a prompt limit of 1000 characters. Thus, while it accelerates the review, a meticulous Technical Writer’s touch is still needed to ensure adherence to specific style guides.

And conclusions on whether AI will replace writers or not, you can make yourself 😉

Dive deeper into our research and its results by reading the Redefining the Economics of Software Development whitepaper.

Prompting matters

In the hands of those who know how to use Gen AI, it can automate mundane tasks, releasing time to focus on more creative and complex problems. In this case, effective prompting is essential. This saves time, makes the interaction smoother, and ensures you get the information you’re looking for.

Simply put, a good question leads to a good answer.

Below you can find an example of the effective prompt for creating a topic based on a user story. It’s super detailed and lists the rules for ChatGPT to follow.

Hi ChatGPT!

You are a Technical Writer and I want you to create a topic with step-by-step instruction describing a new functionality of the application based on a user story I’ll provide to you. The topic should have an intro sentence describing the benefits of new functionality and 7–10 steps on how to use it.

Follow the Microsoft Writing Style Guide principles in your writing:

• Use simple English as the topic should be written for a general audience familiar with the application who doesn’t know technical terms.
• Use the second person “you” instead of referring to “the user” to directly address and guide the reader, making instructions clearer and more actionable.
• Use imperative mood and active voice.
• Avoid the verb “click”, which is specific to using a mouse. Instead, use “select”.
• Use bold formatting for UI elements.
• Use “dialog” instead of “dialog box” and “pop-up window”.
• Avoid using the words “choose” and “press”.
• Use ”box” instead of “field” to refer to a text-entry box.

Here’s an example of an ineffective prompt that may lead to unsatisfactory outcomes. The prompt lacks the specificity and details required for creating a good topic. It does not provide clear instructions and rules to follow.

Hi ChatGPT!

You are a Technical Writer and I want you to create a topic with step-by-step instruction (7–10 steps) describing a new functionality of the application based on a user story I’ll provide. Follow the Microsoft Writing Style Guide principles, be clear, and consistent.

Here are some resources you may find useful while dealing with prompts:

• 10 tech comm use cases for AI
• Awesome ChatGPT prompts
• The Art of ChatGPT Prompting

Conclusions

Technical writing, at its core, is more nuanced than just crafting words on paper or a screen. It’s not merely the act of writing that defines it; it’s the balance of several tasks. First and foremost, it revolves around effective communication. This involves not just passive documentation but also active interactions with various stakeholders like developers, quality assurance professionals, product owners, and more.

Moreover, understanding the users’ needs is a crucial aspect of this role. This requires a deep level of empathy and a user-centric mindset. Interpreting feedback, asking the right questions, and putting oneself in the shoes of the user are skills that cannot be underestimated in this field. While AI might assist in some aspects of the technical writing process, the human element remains irreplaceable.

But still, there’s a growing belief that professionals skilled in using AI effectively will be highly sought after. They can get their tasks done more efficiently, which makes them more beneficial for employers. However, concerns about client confidentiality and information security still persist, so it’s important to be cautious and follow generic security rules. Check my colleagues’ article ChatGPT in Technical Writing: risks and dangers for more details.