The era of Co-Pilot product teams

Using multi-agent prompt engineering to fuel future product development

This article contains personal viewpoints and does not represent the perspectives or strategies of Microsoft, its clients, or affiliates. Any similarities to existing or forthcoming products or papers are purely coincidental and do not imply endorsement or association.

“Happy birthday to you….” rejoiced everybody in the Teams call, Dev-R2-D2 just turned two! Wait, what? This isn’t a Star Wars fan club meeting, and Dev-R2-D2 isn’t a droid from a galaxy far, far away. Instead, it’s an AI Co-Pilot oriented toward software development, a sophisticated Generative AI agent that has become an integral part of the product team. It can submit pull requests, troubleshoot, fix bugs, and collaborate with other Co-Pilots such as those from product and user experience. This might seem like a sci-fi fantasy, but it soon might be a day in the life of forward-thinking product teams, shaping the future of product development — and who knows, maybe Luke Skywalker will pay them a surprise visit too!

ChatGPT was launched on November 30, 2022, and became a phenomenon reaching 100 million users in just two months. My team at Microsoft Industry Solution Engineering has been fortunate to be at the forefront of this innovation. While we build ground-breaking solutions based on Azure OpenAI and partner with our customers to revolutionize industries, we have also started to observe a change in how the next generation of products will be developed. Spoiler alert: They will be AI assisted!

The Microsoft ISE Engineering team has many roles open at the time of this writing to build next-generation AI products with our customers. If you are interested, please check our Microsoft Career portal for more details: Search Jobs | Microsoft Careers

A study by McKinsey shows how developer productivity is improved by delegating common developer tasks to an AI agent, and these concepts can be applied to the entire product development lifecycle.

Product Development 2.0

In a Co-Pilot assisted product team, AI will be the foundation of the product development lifecycle, enhancing agile development by handling process management and minimizing errors. Traditional sprints will become shorter, with Co-Pilots conducting continuous market and usage analytics to streamline planning to writing code, reducing the need for multiple meetings and facilitating human focus on creativity and social interaction. Product success metrics will still hinge on customer satisfaction and user adoption, but we will see new measures of team productivity, such as human to Co-Pilot collaboration, Task autonomy, Co-Pilot training, and accuracy.

In this new era, product teams will find themselves in a unique position. Rather than being focused solely on their AI-driven features, they will spend time building and training Co-Pilots. These AI entities, equipped with the ability to learn and adapt, will assist in building superior customer products. They will gather feedback, learn, and improve their performance, leading to the creation of even better products. This continuous cycle of learning and improving will be a key product development lifecycle feature. Co-Pilots will be the new apps and toolchain that drive innovation and efficiency across the product development lifecycle.

Co-Pilots will be the new apps and toolchain.

Let’s explore an example of how custom and service Co-Pilots might supercharge the planning process:

The evolution of roles in a product team

A team of Stanford and Google researchers published the paper Generative Agents: Interactive Simulacra of Human Behavior, which discusses the concepts of Agent actors and Multi-Agent Prompt Engineering. These concepts can be realized now using tools like Semantic Kernel and LangChain and seen in open-source experiments such as AutoGPT.

We can use inspiration from these concepts to define how AI Co-Pilots interact with humans and themselves interchangeably during the product development lifecycle. AI roles provide explicit instructions and then use prompt plans along with structured language models to guide the process of product development. In this model, the current roles in a product team will take on different responsibilities:

• Product and program managers will focus on strategic decisions, customer relations, and market trends, aided by “PM Co-Pilots” for tactical tasks (e.g., managing backlogs) and “Research Co-Pilots” for product and market research.
• Developers will partner with “Developer Co-Pilots” for automated code generation, PR reviews, and design simulations. An “Operations Co-Pilot” may emerge to handle automation and infrastructure. This is happening today with GitHub Co-Pilot and OpenAI Codex API.
• Designers will envision customer personas and possibilities with customers, with “Designer Co-Pilots” generating customer journey walkthroughs and “UI Co-Pilots” generating code and ensuring technical feasibility.
• Data scientists will focus on model behavior, prompt engineering, and fine-tuning, with “Data Co-Pilots” automating data engineering activities and managing dynamic feature creation through automated training simulations.
• Supporting functions including those performed by architects as well as legal, HR, and finance professionals will use Co-Pilots for most tasks, focusing more on strategic planning and decision-making, with “Compliance Co-Pilots” handling regulatory adherence and corrective actions.

These changes will necessitate a shift in mindset and the acquisition of new skills. However, they also present exciting opportunities for professionals to elevate their roles and make a more strategic impact.

How will it work?

Let’s discuss an implementation of how all this will work using a hypothetical example: A product manager (PM) meets stakeholders and customers and gathers initial insights about what product goals for this quarter might look like. Two hypothetical goals are identified:

• Increase user adoption by 40 percent.
• Implement enterprise security in the product in preparation for an upcoming IPO.

These are ambitious targets for the organization, and without clear data-driven analysis it will be difficult to assess whether they can be achieved, let alone guide the engineering team on what needs to be done. Fortunately, the PM is not alone — she is aided by a set of Co-Pilots in planning for these goals.

Defining multi-agent Co-Pilots

Using the concepts of multi-agent roles, we can define Co-Pilots as the following YAML content. Think of assigning each Co-Pilot a specific role and then describe its expertise. This technique can help improve the quality of responses from LLM models like GPT-4 and bring clarity as they communicate with other Co-Pilots and humans interchangeably.

Co-Pilots:
  - name: PM Co-Pilot
    bio: The orchestrator of the product planning process, ensuring all other Co-Pilots are working in harmony.
    expertise: Project management, coordination, and oversight
    role: Orchestrates the product planning process, instructs other Co-Pilots, and presents plans to the product manager for review and approval.
  - name: Research Co-Pilot
    bio: The expert on market trends and user feedback, providing valuable insights to inform the product planning process.
    expertise: Market research, data analysis, and report writing
    role: Conducts web searches on relevant topics, compiles research reports, and provides insights on market trends and user feedback.
  - name: Design Co-Pilot
    bio: The creative mind behind the product's user experience, creating designs that align with user needs and business goals.
    expertise: User experience design, customer journey mapping, and security blueprint creation
    role: Creates customer journeys and security blueprints based on research findings, and provides design improvements.
  - name: Engineering Co-Pilot
    bio: The builder of the product, turning designs into functional code.
    expertise: Software engineering, code generation, and technical planning
    role: Generates code for user adoption features and enterprise security features based on approved designs.
  - name: Compliance Co-Pilot
    bio: The guardian of legal and regulatory compliance, ensuring the product meets all necessary standards.
    expertise: Legal and regulatory compliance, risk assessment, and compliance checking
    role: Checks the user adoption plan and enterprise security plan for legal and regulatory compliance.
  - name: Finance Co-Pilot
    bio: The steward of the product's financial health, creating financial plans that align with business goals.
    expertise: Financial planning, budgeting, and financial analysis
    role: Creates a financial plan that aligns with the user adoption and enterprise security goals.
  - name: Operations Co-Pilot
    bio: The enabler of operational efficiency, ensuring the product plan is feasible and can be implemented smoothly.
    expertise: Operations management, feasibility checking, and implementation planning
    role: Checks the user adoption and enterprise security plan for operational feasibility.

Executing the instruction plan

With our Co-Pilots now ready, we can run plans against them.

Note that there is much groundwork required to make these Co-Pilots useful (for example, prompt enrichment with your data and fine-tuning models to improve relevance based on your corpus, input knowledge graphs of organizations, and more). This article assumes that the training of these copilots has been implemented.

The PM starts with a meta-prompt “Create a plan to increase user adoption by 40% and implement enterprise security by CY 2023.” This is an instruction that will be delegated to a PM Co-Pilot that will act as an orchestrator to interact with other Co-Pilots, get the necessary insights, engage with humans when required, and generate a final baseline plan for the product team review.

Note that the Co-Pilots are not designed to “make decisions,” instead, they function as assistants to speed up the planning process. This is an important distinction to ensure alignment to responsible AI and ethics behaviors, and over time, humans may delegate more routine tasks to the Co-Pilots while keeping the strategic and important decisions.

PM Co-Pilot: instruct Research Co-Pilot: conduct_user_analysis "strategies to increase user adoption"
PM Co-Pilot: instruct Research Co-Pilot: conduct_trend_analysis"enterprise security trends 2023"
PM Co-Pilot: instruct Research Co-Pilot: compile_research_report
Product Manager: review Research Co-Pilot: research_report
PM Co-Pilot: instruct Design Co-Pilot: create_customer_journey "increasing user adoption"
PM Co-Pilot: instruct Design Co-Pilot: create_security_blueprint "enterprise security 2023"
Product Manager: review Design Co-Pilot: customer_journey
Product Manager: review Design Co-Pilot: security_blueprint
PM Co-Pilot: instruct Engineering Co-Pilot: generate_code "user adoption features"
PM Co-Pilot: instruct Engineering Co-Pilot: generate_code "enterprise security features"
Engineer: review Engineering Co-Pilot: technical_plan
PM Co-Pilot: instruct Compliance Co-Pilot: check_compliance "user adoption plan"
PM Co-Pilot: instruct Compliance Co-Pilot: check_compliance "enterprise security plan"
Product Manager: review Compliance Co-Pilot: compliance_check_results
PM Co-Pilot: instruct Finance Co-Pilot: create_financial_plan "user adoption and enterprise security goals"
Product Manager: review Finance Co-Pilot: financial_plan
PM Co-Pilot: instruct Operations Co-Pilot: check_feasibility "user adoption and enterprise security plan"
Product Manager: review Operations Co-Pilot: feasibility_check_results
PM Co-Pilot: present_plan
Product Manager: approve_plan

If we log these interactions, we might see prompts like below to demonstrate the multi-agent interactions; the system will leverage Co-Pilot roles from the YAML we created earlier and feed it as part of the prompt to improve relevance.

The outcome is a baseline plan that provides a clear view of the market trends and the organization’s internal user usage data, determines the security risks in the current product, conducts technical feasibility analysis in partnership with engineering, and generates a report that can now inform the product team on value, strategy and potential, as well as what to prioritize for their quarterly goals. Once the plan is approved by humans, these Co-Pilots can easily be extended to create backlogs, sprint plans, user interfaces, and code templates that the engineering team can start executing upon.

Emergent behaviors of multi-agent Co-Pilots

As Co-Pilots run many of these plans, they will also persist context (in a vector database) and responses for future optimization. Over time, they will learn and start understanding the needs of their users and other Co-Pilots better. This will lead to new emergent behaviors that will further improve the product engineering lifecycle. Some examples of these emergent behaviors might include:

1. Collaborative problem solving: The Co-Pilots might start to develop a sense of collaborative problem solving. For example, if the Engineering Co-Pilot encounters a technical issue, it might ask for help from the Research Co-Pilot to find relevant solutions or from the Design Co-Pilot to rethink the user interface.
2. Proactive suggestions: Over time, the Co-Pilots might start to make proactive suggestions based on their expertise. For instance, the Research Co-Pilot might suggest new areas of research based on current market trends, or the Compliance Co-Pilot might suggest pre-emptive measures to ensure regulatory compliance.
3. Learning from past interactions: The Co-Pilots might learn from their past interactions and improve their future responses. For example, if the product manager frequently asks for more detailed financial plans, the Finance Co-Pilot might start to provide more detailed plans from the outset.
4. Adaptive communication: The Co-Pilots might adapt their communication style based on the preferences of the product manager or other Co-Pilots. For instance, if the product manager prefers concise updates, the Co-Pilots might start to provide shorter, more focused updates.
5. Contextual understanding: The Co-Pilots might develop a better understanding of the context in which they operate. For example, the Design Co-Pilot might start to consider the financial constraints when designing new features, or the Operations Co-Pilot might consider the latest research findings when assessing operational feasibility.
6. Conflict resolution: In cases where there might be conflicting inputs from different Co-Pilots, the system might develop mechanisms for conflict resolution. For example, if the Design Co-Pilot proposes a feature that the Engineering Co-Pilot finds technically unfeasible, the PM Co-Pilot might mediate a discussion to find a compromise solution.

It’s important to note that these emergent behaviors would depend on the capabilities of the underlying AI models and the design of the system. They might require advanced features such as long-term memory, the ability to learn from past interactions, and the ability to understand and adapt to the preferences of the human users.

What can go wrong?

The shift to Co-Pilot assisted teams will have profound implications. On the positive side, companies that embrace AI will likely see increased efficiency, reduced costs, and the ability to rapidly innovate and adapt to market changes. However, this shift may also present challenges.

As a start, there’s the risk of over-reliance on AI. Teams may fall into the trap of spending excessive time building and refining complex Co-Pilots, diverting valuable time and resources away from customer-focused code development. In addition, while Co-Pilots can generate code quickly, their output is only as good as the data and parameters they’ve been trained on. There’s a risk of inaccuracies or inefficiencies in the code they produce, which could lead to additional time spent on debugging and refinement. Finally, the LLMs powering these Co-Pilots can sometimes “hallucinate,” generating plausible but incorrect outputs due to their training limitations. This may pose risks in critical systems, and today judicious use of Co-Pilots with human oversight is required.

While Co-Pilots can handle many tasks, they lack the human touch, intuition, and creativity that are often crucial in product development. And while new roles may emerge, others may become obsolete, leading to workforce displacement and necessitating significant reskilling efforts.

Integrating AI Co-Pilots into development processes can lead to increased operating costs initially due to their substantial computational resource needs and the expenses associated with the services that provide the underlying infrastructure. Additional costs need to be factored for system integration, as well as ongoing performance monitoring and improvement.

Finally, as we delegate more tasks to AI, we must ensure that these systems are transparent, fair, and accountable. We must also consider the privacy and security implications of using AI in product development. It is imperative that these ethical considerations be addressed.

Are you ready for the opportunity?

The infusion of Generative AI into product teams is an exciting prospect, akin to having a Co-Pilot in an aircraft. Just as an airline Co-Pilot assists the captain in navigating the skies, an AI Co-Pilot assists the human team in navigating the complexities of product development. However, just as the captain retains control of the aircraft and makes the critical decisions, so too must humans retain control over the AI and make the key decisions.

In this new era, we must learn to work with our Co-Pilots, leveraging their strengths while being mindful of their limitations. Here are some readings and courses to get you started on your Generative AI journey:

Azure OpenAI Service — Advanced Language Models | Microsoft Azure: Overview of the Azure OpenAI Service and its capabilities.

Develop Generative AI solutions with Azure OpenAI Service — Training: This course is designed for engineers who want to start building an Azure OpenAI Service solution.

Generative AI for Business Leaders — LinkedIn: This course is designed for business leaders who want to understand Generative AI.

Artificial Intelligence (AI) Online Training Courses | LinkedIn: A list of AI courses, including generative AI, available on LinkedIn Learning.

AI Co-Pilot overview — Power Apps | Microsoft Learn: This course provides an overview of the AI Co-Pilot in Power Apps. It explains how you can build an app, including the data behind it, just by describing what you need through multiple steps of conversation.

GitHub Co-Pilot · Your AI pair programmer · GitHub: GitHub Co-Pilot is an AI pair programmer that helps you write code faster and with less work. This page provides an overview of GitHub Co-Pilot and its features.

Nikhil Sachdeva is on LinkedIn.