Part I — Beyond the Buzz: Highlighting the Impact of AI in Modernizing Applications
In today’s world, there’s no shortage of news about AI, from the recent OpenAI controversies to discussions about Ethical AI and fears of AI taking over everything. It seems like AI is everywhere. On a positive note, OpenAI’s success in making Generative AI accessible to many has caused a major stir and might even challenge big players in the IT industry (Hello Search Engine!). Personally, after traveling to different cities and talking to tech leaders, I’ve noticed a common theme — everyone sees AI, especially Generative AI, as the future. They talk about building new capabilities to enhance customer experiences and create fresh revenue streams. However, having focused on Application Modernization and Digital Transformation for the better part of the last decade, I’ve started pondering how AI will transform existing business capabilities and the applications that drive them in the modernization journey. This line of thought led me to write this blog, where I take an imaginary journey and make predictions about the future of AI’s role in reshaping businesses. So, buckle up and enjoy the ride! Almost slipped my mind — I’ll be focusing more on the far-reaching impacts of AI rather than delving into the nitty-gritty details of building your AI capabilities or models. Given the widespread interest, community and the constant influx of AI announcements from various vendors, you shouldn’t have any trouble finding resources if that’s what you’re seeking.
Where are we on the AI Journey:
Various frameworks exist to assess technology maturity and impact, with one noteworthy model being Gartner’s Hype Cycle, which effectively charts a technology’s progression and estimates the time it will take to reach its peak business value.
Notably, familiar names like Generative AI are still a few years away from achieving the “Plateau of Productivity.” Personally, I prefer to view technology maturity and impact through a three-phase categorization
- Experimentation
- Democratization
- Realization.
In my assessment, AI, especially Generative AI, currently resides in the Democratization phase, while it is very popular, only a subset of organizations are actively exploring or have implemented early versions of it. It is yet to enter the Realization phase, where the broader population can directly witness its day-to-day benefits. While some argue that AI’s impact is already evident in sectors like healthcare, finance, and autonomous vehicles, the transformative effects are not yet universally felt by consumers. Drawing parallels, the complete life cycle achievement is similar to the adoption of smartphones, where the technology progressed from Experimentation to Realization, profoundly altering our approach to technology within a relatively short span of five years. While AI hasn’t reached that stage yet, it’s on the horizon — the days when AI will fundamentally reshape our technology access are not too distant.
If you’re expecting me to predict a doomsday scenario where AI takes over the world and spells doom for humanity, you’re on the wrong track. I believe such a catastrophic event is a long way off in the future, and we might be ancient history by the time it becomes a real possibility. Let me break it down for you: We’re currently in the early stages of the AI journey, and if we look at AI capabilities as the measuring stick, we can categorize AI into three types
- Weak AI (also known as Narrow AI)
- Strong AI
- Super AI
Now, don’t be surprised to learn that the only AI currently in existence is the Weak AI. It excels at performing specific tasks it’s trained for sometimes even outperforming humans and showcasing limited learning capabilities for improvement. However, it’s crucial to note that Weak AI operates within predefined scopes and requires training to execute actions. It lacks the inherent ability to comprehend human emotions or understand the rationale behind interactions, making it incapable of manipulating us on a deeper emotional or cognitive level. The idea of AI manipulating human emotions falls under Super AI, which is still just a theoretical concept and may not become a reality for a long time (we are talking about decades here). While it’s true that Weak AI could generate incorrect or biased suggestions that might lead to human tragedies, let’s trust our fellow beings not to make such a mistake.
Where do I see us heading with AI?
As illustrated in the Gartner diagram above, the realm of AI is expansive, encompassing various iterations, primarily falling under Weak AI. Certain variations, such as computer vision, may be tailored to specific industries, adopted for niche use cases not widely accessible to the general population. However, there has been a significant shift with the recent democratization of Generative AI. Now, organizations across the board are actively exploring ways to integrate Generative AI capabilities into their service delivery and establish new business functionalities. According to the MIT Technology Review, before the prominence of Generative AI, a 2022 survey revealed limited ambitions among CIOs, while 94% of organizations were utilizing AI in some capacity, only 14% aimed for “enterprise-wide” AI adoption by 2025. Yet, with the empowering capabilities of Generative AI tools democratizing access, there is an anticipation that it will permeate every facet of the enterprise, supporting every employee and engaging every customer. This transformation is poised to redefine the modern enterprise comprehensively. The sheer success and popularity of Generative AI have transitioned AI initiatives from being confined to niche exploratory side projects to becoming pivotal enterprise-wide capabilities, influencing every facet of the organization. It’s crucial to highlight that Generative AI isn’t just a game-changer in processes but is also making significant strides in software development, facilitating accelerated innovation and cost reduction. Examples of these groundbreaking code generators include IBM Watson Code Assistance and Microsoft’s Copilot, underscoring how Generative AI is reshaping and enhancing various aspects of organizational functionality.
With this, it’s reasonable to assume a high probability, if not certainty, that organizations will consider incorporating Generative AI, especially Large Language Models (LLMs) a subset of Generative AI systems that are designed to process and analyze vast amounts of natural language data and then use that information to generate responses to user prompts, as a core component in their IT portfolio. However, this raises a significant question: What’s the best approach? Should one build a model from scratch, use an existing model as is, or take a domain-specific model and fine-tune it with proprietary data? Eric Lamarre from McKinsey Digital classifies it succinctly:
- Take: Utilize an existing model as is.
- Shape: Fine-tune an existing model with your data (typically an ongoing process).
- Make: Construct your own model.
So, what’s the verdict? As always, the answer depends on various factors, but there’s a strong inclination toward the “Shape” option. Here’s why: Unless building and selling LLMs is at the core of your intellectual property, there might be little justification for constructing your model from scratch. Delving into the complexities involved in building and maintaining a proprietary model, OpenAI reportedly spends $40 million monthly on processing user queries, and Microsoft’s Bing chatbot demands a staggering $4 billion worth of infrastructure to serve its potential user base. Environmental concerns add another layer of consideration, the training of OpenAI’s GPT-3 consumed a substantial amount of electricity (1,287 MWh) and emitted over 550 tons of carbon dioxide. While the impact of building an enterprise model may be less severe, it is not negligible. On the flip side, merely using an existing model, whether foundational or domain-specific, may not yield the distinctive business value that most organizations seek. Shaping an existing model to integrate your organization’s unique attributes falls in between and aligns more sensibly with the business goals of most organizations.
Another crucial decision to consider is whether to opt for proprietary models built and managed by a vendor or utilize open-source foundational models through a community-driven platform like Hugging Face. As open source has become the widely accepted standard for innovation in the past decade, it’s challenging to argue against using open-source, domain-specific models when available, unless there is a very specific business reason not to do so. In essence, I believe that most organizations will lean towards utilizing domain-specific open source foundational models and fine-tune them to align with their business needs. This approach also provides them with the flexibility to deploy the model where it makes the most business sense, whether it’s on-premises, in the public cloud, or, in some cases, at the edge.
Transformative Business Advantages delivered by Large Language Models:
Now, let’s pivot our discussion to explore the substantial advantages that a proficiently implemented and integrated Large Language Model (LLM) can bring to an enterprise within their digital landscape. While acknowledging the internal productivity gains, such as code generation or serving as a go-to email assistant or internal search engine, which are widely recognized, I aim to emphasize the critical business impact. The key benefits that deserve our attention are elaborated below:
- Simplified and Inclusive User Interface: A well-integrated LLM can revolutionize the user interface, making it more intuitive and accessible. By understanding and responding to natural language, it streamlines user interactions, fostering a simplified and inclusive interface that caters to a broader audience. The primary advantage introduced by Large Language Models (LLMs) lies in the creation of a simplified conversational user interface. Diverging from the conventional user interfaces we’re accustomed to, where users must learn the application and follow a sequence of predefined actions to accomplish a task (e.g., transferring money through a mobile banking app), the LLM paradigm allows for the same task to be executed effortlessly through a natural language prompt, as illustrated in the accompanying image. The displayed image exemplifies the potential capabilities of AI enabled applications with LLMs within enterprises. We will delve into the architectural aspects of achieving this later in the discussion. However, the profound impact lies in the ability to condense user interactions, typically involving multiple actions and screens, into a single input box triggered by a simple command (such as a submit button). Stretching the imagination a bit, it is not farfetched to dream of a future where single screen applications are the norm. The simplicity of this approach also enhances inclusivity, as it facilitates the easy integration of voice interfaces, catering to visually challenged users and making the technology more accessible to a broader userbase.
- Improved Customer Experience and Participation: As shown above, the impact of LLM on customer experience can be significant. By leveraging natural language processing, enterprises can enhance customer interactions, offering personalized and contextually relevant experiences. This not only elevates customer satisfaction but also encourages increased participation and engagement. Empowered with the capacity to deliver personalized, insightful value-added facts and offers, AI enabled applications leveraging Large Language Models (LLMs) have the potential to elevate customer engagement in additional revenue streams that they might not currently be part of. This not only contributes to revenue growth but also plays a pivotal role in enhancing customer retention.
- Unleashing the Power of Data to Disrupt the Disruption: Recognizing the disruptive potential of data, an adeptly implemented LLM can act as a catalyst for innovation. By extracting meaningful insights and patterns from vast datasets, it empowers enterprises to stay ahead of disruptions, leveraging their own data to proactively disrupt industry norms. I firmly believe that the capability to unlock the long-untapped potential of data within enterprises, providing meaningful insights and enhanced value for customers, enables organizations to respond more efficiently to disruption threats. In other words, this empowerment means that the once disrupted can now become the disruptors. This becomes particularly crucial, especially in the face of challenges posed by digital startups that leverage innovative technology and new approaches to disrupt established organizations.
In essence, these benefits extend beyond internal operational efficiency, focusing on pivotal outcomes that reshape how enterprises interact with users, enhance customer satisfaction, and strategically leverage data to navigate and disrupt the evolving digital landscape.
Aligning AI and App Modernization:
With the value of AI-enabled applications clearly outlined, it’s crucial to shift our focus to how organizations are integrating AI into their operations. As highlighted in the initial section of this article, many organizations tend to compartmentalize AI initiatives from their broader digital transformation efforts, where they’re concurrently modernizing applications to thrive in the cloud-centric landscape, fostering quicker innovation and heightened competitiveness. While it might be worth exploring the change of the name from “digital transformation” to “digital standardization” considering the broader industry maturity and adoption a topic for another day, I firmly believe that maintaining a separate track for AI and modernization is a misstep. I advocate for aligning these streams, as the synergy between modernization and AI efforts yields superior results at an accelerated pace. Let’s delve into what this integrated approach might entail. Oh, and before we proceed, let’s not forget about the significance of edge computing; its growing prominence suggests it should be seamlessly integrated into our strategies as deemed appropriate.
Starting with the AI layer:
So, let’s begin by focusing on the AI layer, specifically, with the “Shape” option that I’ve identified as my preferred approach. The implementation strategy I propose involves starting with a simple, cost-effective model that allows for evolution and enhancement as per your organization’s needs. The initial step in this approach is to leverage a pre-trained open-source model, as mentioned earlier. This sequence unfolds as follows:
Select a Pre-trained Open Source Model:
Choose a suitable pre-trained open-source model as the foundation for your AI initiatives. I love what Hugging Face is doing in this area and I hope you will also leverage it for this step. At this point Llama 2 seems to be a great LLM to choose based on the results and popularity. If you would like to start with a domain specific LLM there are few great options available to choose from as well. I believe open source models are the way to go since they benefit from the thriving community of contributors. It also provides greater transparency and safety that seems to be an on-going challenge with properitory models, In summary, if your model is hidden behind an API wall and you have little or no visibilty or control on how it is built and where it is heading, it may not be a good fit for you.
Fine-tune the Model ONCE with Enterprise Data:
Fine-tuning is a process that trains a model often periodically on additional data to make it perform better on the specific tasks and domains that the data set details. For example, fine-tuning is how code-specific LLMs are created. Google’s Codey is fine-tuned on a curated dataset of code examples in a variety of languages. This makes Codey perform substantially better at coding tasks than Google’s Duet or PaLM 2 models, but at the expense of general chat performance.
In our case, we will start with fine-tuning the selected model. Notice the emphasis on ONCE In the title, I think with the right model and right architecture you will be able to reduce the number of fine tuning required significantly or to just once in most case, I will elaborate on it further in the next section. Having said that, I do believe fine-tuning the selected model once, incorporating your enterprise-specific data to enhance its fit and derive maximum value.
While asserting this, I acknowledge the importance of considering business requirements and the maturity level of your existing AI system in place. Periodic fine-tuning might indeed be necessary. For instance, in the context of the XBC bank output mentioned earlier, the ability of the Large Language Model (LLM) to communicate information about reducing late payment fees and recovering historically paid overdue charges might be more effectively addressed by the LLM model than the RAG retrieval process. Thus, depending on your current status and specific needs, you may find yourself fine-tuning the process regularly or opting for a one-time adjustment, leveraging RAG retrieval as the primary contextual engine for delivering real-time information.
Customize Existing APIs for Real-time Business Data:
This is the core component of the architecture. The triumph or setback of your AI success is fundamentally shaped by this. The fundamental concept here is to expand the capabilities of the Large Language Model (LLM) by integrating it with your business processes and data, offering valuable insights to your customers. An effective starting point could be your existing business processes and data, often exposed through APIs. In many enterprises, minimal modifications suffice to bring this vision to life. This integration can be achieved through a synergistic blend of the Retrieval Augmented Generation (RAG) framework and real-time data sourced from existing or upgraded APIs. Let’s delve deeper into this approach.
Retrieval-augmented generation (RAG)
RAG is an AI framework for improving the quality and relevance of LLM-generated responses by anchoring LLMs on precise, up-to-date, and pertinent information retrieved from an external knowledge store. Prompting LLMs with this contextual knowledge makes it possible to create domain-specific applications that require a deep and evolving understanding of facts, despite LLM training data remaining static. Implementing RAG in an LLM-based question answering system ensures that the model has access to the most current, reliable facts, and that users have access to the model’s sources, ensuring that its claims can be checked for accuracy and ultimately trusted. RAG can drastically improve the accuracy of an LLM’s responses especially in a domain specific context.
The following conceptual architecture diagram shows how RAG fits in the LLM model.
When the inference flow is triggered, the orchestration layer knits together the necessary tools and LLMs to gather context from your RAG retrieval tools and generate contextually relevant, informed responses. The orchestration layer handles all your API calls and RAG-specific prompting strategies. Orchestration layers are typically composed of tools like LangChain, Semantic Kernel, and others with some native code (often Python) knitting it all together. The retrieval process of RAG can be implemented either with knowledge base or APIs. Knowledge base typically involves usage of vector database and might be the preferred approach for may orgnaizations. I do think this option should be kept on the table as you go deeper in your AI journey.
But to start with, I suggest you to explore API-based retrieval, since most organizations have all key data sources with programmatic access (customer records databases, an internal ticketing system, etc.). It is easy to make them accessible to the AI orchestration layer as APIs with minimal changes to convert the output as text for easy processing. At run time, your orchestration layer can query the API-based retrieval systems to provide additional context pertinent to the user request. The following diagram shows a populated prompt template before the RAG retrieval tool invoke the backend API in our example
The retrieval results in context being populated that is then sent to the LLM for further processing as shown below
The LLM uses the context information provided in the prompt template as part of the inference to refine the response. However, before that happens it may be important to check the RAG retrieval output to ensure no sensitive data is included in the generated context. With this process you will be able to derive an output as shown in the Diagram 1 above. This is how with the usage of RAG to provide the real-time business context, the requirement for fine-tuning will be significantly reduced if not completely eliminated after performing once during the initial setup as explained in this section.
The ability to leverage existing APIs to build integrated smart AI systems with minimal changes is one of the main reasons I beleive we need to visit Application modernization and AI together. This approach ensures faster business outcomes, reduced architecture complexity and improved ROI. It will also provide a new lease of life for legacy business processes and APIs buying you more time and budget to modernize/ migrate them to the future AI landscape. Another note to highlight is this approach may be an anti microservice pattern where preference may be given to retaining the legacy backend APIs which are typically bound better to the supported business processes as is than breaking them into multiple smaller microservices.
Modernize Web and Mobile User Interface:
As previously explored, the integration of AI into conversational user interfaces brings forth a notable advantage — simplicity. When harmonizing Application Modernization and AI initiatives, there’s a strong likelihood of cultivating a more seamless user interface and navigation. This convergence offers the potential to minimize the amount of user interface work typically associated with independent Application Modernization endeavors. The following diagram shows the simulated user interace for the XBC bank output we discussed above
Deploy the Model for Consumption:
I firmly believe that the hybrid deployment model has already solidified its position as the preferred choice for various reasons like data sovereignty, architectural compatibility, edge and AI deployments, and cost reduction. I’m not introducing a new perspective here; rather, I strongly advocate for the hybrid deployment model. With containers firmly established as the preferred deployment unit, I foresee an enterprise Kubernetes platform on a hybrid cloud architecture becoming the go-to approach. I refer to this as the application platform for the next decade. This approach not only caters to AI requirements but also serves as the foundational framework for all other application and workload deployments, even those not yet falling under the AI umbrella.
Summary:
Having endured the brunt of digital disruption, it’s time for businesses to fight back. By using their data smartly and implementing a strong AI solution, enterprises can take charge and make a powerful comeback and build a resilient business for the decades to come.
As we embark on an imaginary journey into the future, the convergence of AI and application modernization takes center stage. In this article, I have tried to underscore the significance of AI, especially Generative AI and as an extention LLMs, as not just the future but the catalyst for building new capabilities and revenue streams. I emphasize the need for organizations to align their modernization and AI initiatives for optimal results, debunking the myth and practice of keeping them separate. Delving into the practicalities, I have outlined a detailed approach for incorporating AI, particularly LLMs, into the fabric of existing business capabilities.
I believe integrating the Application Modernization and AI especially LLM adoption together, organizations can significantly improve the probability of success while drastically reducing the upfront investment and archiecture complexity along the way. The broader theme revolves around how the integration of AI transforms not just specific functionalities but the overall user experience.
In conclusion, I have to tried encapsulate a comprehensive exploration of AI’s multifaceted impact on businesses, technology, and user experiences by traversing the realms of application modernization, Generative AI, Large Language Models, deployment models, and conversational interfaces, presenting a holistic view of the AI landscape and its transformative potential across diverse industries.
Hope you enjoyed the ride and looking forward to hearing from you!
References
- MIT Technology Review : The great acceleration: CIO perspectives on generative AI
- https://stackoverflow.blog/2023/10/18/retrieval-augmented-generation-keeping-llms-relevant-and-current/
