Are AI tools advanced enough for product documentation?
Can AI increase productivity and save PMs valuable time?
In a poll on the product management subreddit, more than 1000 product managers voted on whether or not AI tools are advanced enough to be used to write product documents. The consensus? AI tools can be useful for structuring and outlining documents but are not yet accurate enough to create meaningful product documentation without significant input and effort.
In the poll, more than 400 voted “Not yet but soon,” while more than 500 users voted that they could be used “only for the first draft.” In the minority, only just over 50 users voted “Yes 100%” and 175 voted “Never.”
Many PMs extolled the virtues of using tools like ChatGPT and Quillbot to write fluid texts that would have otherwise taken hours–but almost all agree that in order for AI to write a meaningful product document, the quality of input is crucial. Without first inputting collected data, research, notes, and product information, AI is likely to produce factual errors such as inaccurate or erratic numbers and fabricated product information.
But is it still worthwhile to use AI to write product documents? Let’s take a closer look.
Write product documents faster
Collato’s free and powerful templates save you time by helping you create effective docs to share with stakeholders. Discover free templates
How to use AI to write product documents
Technically speaking, there are a few different ways that AI can be used to write product documents:
Natural language generation (NLG) systems can be used to automatically generate documents such as user stories or technical documents. However, structured data is needed as input in order for the machine learning algorithms to generate text that is coherent and follows a particular style or format.
AI-powered writing assistants can help PMs generate ideas, organize their thoughts, and improve the overall quality of their writing. These tools also use machine learning algorithms to suggest alternative phrases or rephrasings, and typically also offer grammar and spelling corrections.
AI-powered templates can be used to streamline the document writing process. These templates use machine learning algorithms to suggest the most relevant sections to include in a particular type of document, such as a PRD (product requirements document).
AI-powered summarization tools can be used to automatically generate summaries of longer documents. These tools can be particularly helpful for generating executive summaries or summarizing technical documents for a non-technical audience.
Whichever route you decide to take, be sure to input correct and relevant data and thoroughly review the output before sharing or publishing your product document. AI is not without limitations:
Limitations of AI-written product docs
Whenever you use AI to produce content, it’s important to remember that it’s always “learning” from available data sources, which means it’s also limited to what’s currently out there. Open AI specifically mentions its limitations, including potentially generating incorrect information, harmful instructions, or biased content. It’s also limited to knowledge of the world and events after 2021.
Moreover, in order for product documentation to be valuable, it needs to include intricate knowledge of your software’s behavior, the problems your product is trying to solve, and the information you learned in user interviews and research. If you want to use AI to write your docs, you’ll need to manually input these details, and you’ll likely spend time editing and tweaking the output.
If you want to see if AI really saves you time, track and see how long it takes you to write a draft vs inputting information into ChatGPT.
Collato puts Open AI to the test
We decided to put our money where our mouth is and test out ChatGPT ourselves to see if it could write a valuable PRD. For our first test, we used Open AI’s dialogue feature and asked it to “please write a PRD for Collato” with no additional input.
The output was lackluster at best-it clearly didn’t fully grasp what Collato was as a tool, using generic terms like “project management software” and “team collaboration tool” (FYI-Collato is actually a platform for connective thinking, which is like a spider web of information connecting documents, tasks, and company goals).
Moreover, the text the tool generated was less a comprehensive document and more a simplistic overview of basic requirements of any product, such as “intuitive and easy to use” and “responsive and works well on a variety of devices, including desktop computers, laptops, and mobile phones.”
Basically, it reads like a product requirement document written by someone very new to product management and very unfamiliar with the product itself.
However, once we gave Open AI more context and information by using our own PRD template, we got something a little more interesting and useful:
Example of PRD written by ChatGPT
Collato is a connective thinking platform that helps teams streamline their work by bringing all their information and tasks into a single, visual interface. It allows users to create and connect documents, tasks, and company goals, making it easy to find and understand the relationships between different pieces of information.
Purpose: The purpose of Collato is to help teams be more productive and fulfilled at work by eliminating the need to search for missing pieces of information across different tools. It allows users to see the bigger picture and understand how their work fits into the overall goals of the company.
Necessary Features:
Visual interface for creating and connecting documents, tasks, and company goals.
Search functionality for quickly finding and accessing relevant information
Collaboration tools for team communication and real-time editing
Integration with other productivity tools and services
Robust security and privacy measures
Functionality: Collato should be intuitive and easy to use, with a clean and organized interface. Users should be able to create and customize their own workspaces, and easily access and edit the information and tasks within them.
Desired Behavior: Users should be able to quickly find and understand the relationships between different pieces of information, and easily communicate and collaborate with their team members within the platform.
Project Specifics:
Participants: Development team, product manager, design team, stakeholders
Status: In development
Target release: Q2 2024
Team Goals: Create a comprehensive connective thinking platform that helps teams be more productive and fulfilled at work. Eliminate the need for users to search for information across different tools. Improve team communication and collaboration.
Business Objectives: Increase user productivity and satisfaction. Differentiate from competitors by offering a unique and comprehensive solution.
Assumptions: Users will primarily access Collato through desktop computers and laptops. Users will have a variety of experience levels with productivity tools.
User Stories: As a project manager, I want to be able to see all my team’s tasks and progress in one place, so I can track the overall progress of the project.
As a team member, I want to be able to easily communicate with my team and collaborate on tasks in real-time, so we can work more efficiently.
As a company leader, I want to be able to see how our team’s work fits into the bigger picture of our company’s goals, so I can make informed decisions about our direction.
User Interaction: Users will primarily interact with Collato through the visual interface and search functionality. Users can access and edit information and tasks within their own workspaces or those shared with their team. Users can communicate and collaborate with their team through chat and real-time editing
Design Wireframes:
[Insert design wireframes here]
Questions for Further Research: How can we ensure that the user interface is intuitive and easy to use for users of all experience levels? What types of integrations would be most valuable to our users? How can we balance the need for robust security measures with the need for seamless user experience?
Clearly, by providing more contextual input, you’ll get something more meaningful out of AI. The more examples you provide, the more you can train ChatGPT to streamline the production process of writing documents like user stories, PRD, technical docs, API docs, etc.
Tl;dr
AI tools often output generic content without new or interesting information. However, it can also help you write clearer, well-phrased text. The key is to feed it quality information about your product, users, and data and to specify exactly how you want your document formatted. So it might save you time in structuring and drafting your text, but in the end, you will still need to spend time preparing, editing, and reviewing your product documents.
Your role as a PM is to envision the future of the product, which means you need to be able to come up with strong ideas on your own. AI tools like ChatGPT can give you a starting point for structuring your documents, but refining and curating the output into something valuable for your product will always require context and talent on your side.
Originally published at https://collato.com.
