Featured
Emerging product considerations for LLM systems: Reasoning Architectures
LLMs and AI systems are reshaping the way we think about product development. As LLMs introduce a level of reasoning into previously workflow-only systems, we see a rise of “Reasoning Architecture” that lies on top of traditional underlying “Technical Architecture.
What Has Changed?
The Reasoning Architecture
Unlike traditional product design, AI-driven systems must explicitly manage a new conceptual layer: the Reasoning Architecture. This layer captures expert judgment, domain-specific logic, and nuanced decision-making criteria, especially critical in subjective scenarios and edge cases where ambiguity reigns.
Reasoning Architecture is distinctly different from Technical Architecture, which remains responsible for data storage, retrieval systems (like vector databases), infrastructure, APIs, and overall system performance. While engineers maintain the Technical Architecture, domain experts — often non-technical — directly shape the Reasoning Architecture. This is because we now have an element in AI systems that directly looks at and reasons across natural language. These LLM systems are design to mimic how humans and experts in specific workflows think and automate the way the system thinks about unstructured information.
The Reasoning Planning infrastructure is essentially advanced Chain of Thought reasoning that reflects the ability for specific human experts to, in a no-code way, easily load and edit their reasoning about information in different situations.
An example of Reasoning Architecture
Reasoning architecture is essentially the mapping of how the system reasons through specific information and comes to specific types of conclusions, categorizations, or decisions based on the information at hand. It essentially acts almost like a checklist or an SOP of how to think about specific information.
This answers questions and assessments like ‘What should I spot and consider in the information given?’ or ‘I should give this category a score unless there is this other fact.’
This is different from technical questions like ‘Where is the information going to come from?’ or ‘How is this information stored?’
Reasoning Planning is different from Technical Planning as the former is focused on how to think about and categorize information versus taking actions and steps (i.e. to retrieve information, to access a website, to send an email, etc).
An example of Technical Architecture
You can see major differences in the Technical and Reasoning Architecture simply by looking at the type of nodes being represented in these graphs. Technical Architecture focuses a lot on performing actions (Doing things) versus Reasoning Architecture focuses on evaluating information (Reasoning through things).
All types of technical processes include some assumption of a Reasoning Process, but there are certain workflows, especially for highly technical white-collar knowledge-intensive fields (law, healthcare, manufacturing) where there are specific steps which are pure information reasoning, without any new external data or action being taken throughout the entire reasoning process (i.e. no Technical Workflow).
We believe reasoning architectures and reasoning maps will become more popular as people begin to focus on encoding more complex context and custom expert workflows into systems. As AI is increasingly used to map out custom workflows that are highly specialized, we believe platforms specifically designed for mapping expert reasoning processes will become increasingly useful. Most tools out there are more focused on workflows because that is the vast majority of what is a known process to developers (i.e. retrieve customer information, retrieve name and search LinkedIn for their CV, extract title & interests).
The Role of Domain Experts
To effectively manage Reasoning Architecture, new user interfaces and workflows are required. Domain experts need specialized UX tools allowing them to audit, adjust, and refine the logic and decisions made by LLMs. This approach transforms previously implicit judgments by experts (“you know it when you see it”) into clearly documented and operational guidelines, improving transparency and accountability.
There are a few ways to ensure your AI system is reasoning the way you want it to — Fine-tuning a model with reasoning capabilities or a Multi-Agent System with Chained Prompts.
In many scenarios, where you want auditability of systems, frequent changes to the reasoning flow, and most importantly, some level of deterministic reasoning that is not a black box model, without having to use millions of training examples, a Multi-Agent System is what people prefer to turn to.
Understanding Product Development
The emergence of a distinct Reasoning Architecture shifts our understanding of product development. At their core, successful products continue to revolve around clear user outcomes. Engineers still need to prioritize a deep understanding of what Users want and their pain points. Users still prioritize the ease and intuitiveness of the final product rather than understanding the complex processes running behind the scenes. Continuous iteration and user feedback loops remain essential to refining and improving products, ensuring they meet evolving user needs. Product development is obviously still centered on ideas of engineering efficient technical solutions, and delivering intuitive and seamless user experiences.
However, product development teams must explicitly manage and operationalize nuanced human judgment and expert reasoning as core elements of the product itself. How the LLM thinks about information is now as important as to how the system gets information. This requires teams to be conscious of how the LLM is navigating and reasoning through information where domain experts are no longer passive advisors but active collaborators who shape product logic directly.
Further, the explicit separation of reasoning and technical architectures elevates the importance of explainability, transparency, and accountability. Products must now clearly document and justify the logic behind their decisions, changing how teams approach both product design and iteration. Instead of purely technical feedback loops, there are parallel processes:
- Engineers optimizing performance
- Experts refining logical decision-making
Such audit-ability of reasoning steps may also go beyond essential Reasoning Development work, but also for legal explainability purposes.
Why This Distinction Matters for Product Developers & Managers:
Clearer Explainability & Accountability
- Simplifies understanding and communicating how AI-driven decisions are made, essential for compliance and user trust.
Efficient Auditing Workflows:
- Enables faster tracing and validation of the reasoning logic, speeding up troubleshooting and reducing downtime.
New Expert-Centric Workflow:
- Requires adopting specialized interfaces and iterative workflows that actively involve domain experts in continuously defining, refining, and auditing reasoning logic separately from technical iterations.
A point I find interesting is that just like how traditional product managers would have a deep understanding of the workflows of their users, to the point that they can act as consultants on best practice workflows, AI product managers are likely going to have to essentially upskill on the reasoning of their users, to the point that they can act as consultants on the specific reasoning employed by their users. From this perspective, you can see how you are likely going to have more lawyers as legal-tech AI PMs, or accountants as accounting-tech AI PMs, etc.
The Future of Product Teams
This shift necessitates evolving roles within product teams. Domain experts become integral participants, actively shaping product logic alongside engineers and product designers. Ultimately, clearly distinguishing Reasoning Architecture from Technical Architecture ensures AI-driven products remain transparent, accountable, and reliably aligned with human expertise and ethical standards.
At WhyHow.AI, we focus on giving structure to enterprise data using Knowledge Graphs to improve LLM retrieval, particularly in legal applications.
If you want to understand more about the law, expert reasoning flows, and Knowledge Graphs:
📩 Follow us on Medium
📬 Sign up for our newsletter
📅 Find a time to chat with me
