The Challenges of Enterprise AI and LLM Adoption — Part 2
Continuing from the first chapter on the challenges of adopting Large Language Models (LLMs) in business, we’re turning our spotlight today onto the grittier details. We’re tackling technology choices, the puzzle of creating efficient data pipelines, the conundrum of effective knowledge management, grappling with model quality, untangling integration issues, and keeping a vigilant eye on cybersecurity. There’s plenty to unpack in this second chapter, so without further ado, let’s dive into the specifics.
Novel concepts and technology choices
As enterprises are cracking open the toolbox of Large Language Models (LLMs) and generative AI, they find themselves amid a whirlwind of novel concepts, evolving technologies, and service offerings. Terms such as prompts, priming, embedding, tuning, chains, agents, extraction, summarization, alignment, and evaluation are no longer just the parlance of tech aficionados. These terms define the intricate inner workings of LLMs and generative AI, the frameworks that underpin their performance, and the mechanisms of control and alignment within complex workflows.
However, the challenge does not stop at grasping these new concepts. The fluid landscape of AI development also demands understanding and integrating a continually updated suite of tools and infrastructure components. This dynamic aspect of technology adoption marks a fundamental shift from traditional models, urging teams to embody a state of perpetual learning and adaptability.
Organizations are also tasked with strategic decisions concerning the operational infrastructure. The choice between on-premise, cloud-based, or AI-as-a-service models is a critical one, with significant cost implications and potential trade-offs. While some special workloads may require a high level of control only achieved through self-managed solutions, others may benefit from the scalability and flexibility offered by cloud-based or AI-as-a-service models. Each choice comes with its unique set of challenges, and the decision can significantly impact the organization’s AI journey, from efficiency and cost-effectiveness to the overall success of the implementation.
Amid the rapid evolution within the AI vendor ecosystem, it’s crucial for organizations to understand the current state of play, cut through the noise, and understand the reality of the situation. Despite the media hype, the availability of enterprise-grade LLM and generative AI-based solutions is still very limited. This scarcity is not indicative of a lack of interest or effort, but rather a testament to the complexity and novelty of these technologies. The development of enterprise-grade LLM AI products necessitates a deep comprehension of the technology, the proficiency to weave it into existing systems, the guarantee of its dependability and security, and the skill to steer through the regulatory environment. Furthermore, vendors need to explore innovative ways to leverage natural language as a new interface between humans and computers. While chat-based products can serve as a promising starting point, it will require more comprehensive and sophisticated solutions to fully penetrate the complex enterprise software ecosystem. The industry is pushing hard toward the development and deployment of these enterprise-grade products. With new research, models, and applications emerging daily, the pace of progress is staggering. It’s a matter of when, not if, the market will see a surge in the availability of these products.
In this context, organizations need to stay informed about the latest developments and be prepared to adopt these technologies when they become available. This readiness involves a commitment to continual learning and adaptability, and a willingness to take calculated risks.
Quality
One of the most fundamental hurdles for enterprises in adopting Large Language Models is the challenge posed by language. This aspect, being the very essence of these models, is often the initial point of any concerns surrounding LLM deployment. Unsurprisingly, English has become the lingua franca of this AI domain, often serving as the default choice for model training. However, the dominance of English creates a skewed landscape, as the quality and size of training corpora heavily influence how these models perform in different linguistic environments. It means that when it comes to less globally prevalent or resource-rich languages, the functionality of these AI systems might significantly reduce due to lack of comprehensive data. Therefore, organizations intending to leverage Large Language Models face an inherently complex challenge when their operational or market realities extend beyond English-speaking contexts. Finding or building an LLM solution that is seamlessly optimized for a less globally prevalent or resource-rich language can be a daunting task. The current market offerings, predominantly trained on extensive English corpora, may not provide the expected performance levels when used within different language environments. As such, businesses confront a fundamental hurdle in their AI adoption journey — the limited availability of solutions that are well-optimized for diverse linguistic contexts.
Another key concern revolves around transparency, particularly concerning LLMs’ architecture and their training data. To ensure a smooth integration of these models into their operations, enterprises require an in-depth understanding of the underpinning intricacies of LLMs. Yet, this becomes a daunting task when providers withhold details about their models’ architecture or the specific datasets used in their training. This opacity complicates the assessment of model quality and can lead to unexpected results or misuse. Further complicating matters, the regulatory landscape for AI and data usage is continuously evolving. This suggests a future where a higher degree of transparency could be a mandated requirement, adding a layer of complexity to the existing transparency issue. Expanding on the regulatory considerations discussed previously, language and other generative AI models are now under substantial legal examination, particularly regarding copyright issues. This elevated scrutiny is primarily attributable to the methodology of training and fine-tuning these advanced models, including the selection of corpora utilized. The intellectual property implications inherent in the training materials used are a complex and ongoing legal concern that requires significant attention.
New domain-specialized language models are starting to emerge at an accelerated rate. These models, tailored to specific industries or tasks, bring about their own set of quality assurance challenges. Given the speed at which these models are being developed and the diversity of approaches, maintaining consistent quality standards becomes an uphill task.
Simultaneously, the use of open-source frameworks poses its own set of challenges for maintaining quality control. Although these frameworks democratize access to high-level AI technologies, they often lack standard quality control measures. Open-source tools’ varying levels of documentation, user support, and continuous development can lead to inconsistencies in implementation. Further, their unrestricted access and modifiability could inadvertently heighten the risk of quality degradation, especially if machine-generated content is misused in training data.
Model collapse poses a novel and significant challenge that specifically impacts generative models, particularly in the context of Language Model development. This phenomenon arises when subsequent models are trained using AI-written data obtained through web scraping or synthetic generation. To understand the concept of model collapse, consider a game of ‘Telephone’ or ‘Chinese Whispers’, where a message is passed along a line of people. The first person in the line whispers a message to the second person, who then whispers what they heard to the third person, and so on. By the time the message reaches the last person in the line, it often bears little resemblance to the original message. This is similar to what happens in model collapse. As models train on data produced by their predecessors, the ‘message’ (or the original data distribution) gets distorted over time, leading to a loss of information and a degradation in the model’s performance. As these models continue to train on such data, they progressively lose touch with reality, resulting in a downward spiral of deteriorating performance. Model collapse stems from two root causes: the limited number of samples used for training, which introduces statistical approximation error, and the models’ limited ability to accurately represent the original distribution, known as functional approximation error. Understanding and mitigating these causes are essential to address model collapse and enhance the reliability and performance of generative models. The research paper “The Curse of Recursion: Training on Generated Data Makes Models Forget” (by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson) provides valuable insights into this problem and its implications. It is crucial to note that the emergence of cheaply trained open-source models is anticipated to drive the trend of training these models with machine-synthesized data in larger numbers. This shift is driven by the increasing cost of accessing curated knowledge bases.
In the face of these complexities, it is worth noting that as LLM development advances, new quality issues — similar to “model collapse” — are bound to surface. The impact of these emerging issues will be felt across the enterprise world in diverse ways. Some of these impacts will manifest technically, disrupting the seamless functioning of AI systems. Some will surface as legal challenges, given the evolving landscape of AI regulations. And others will present ethical dilemmas, challenging the moral grounds on which AI operates. Often, enterprises will find these aspects intertwined, making the quality conundrum in LLM adoption an issue that touches on the technical, legal, and ethical facets of business simultaneously.
Data and privacy
A distinguishing characteristic of LLMs is their potential to be fine-tuned using a company’s proprietary data or curated knowledge bases, whether by integrating external embedding databases or fine-tuning existing models. However, curating such knowledge bases is a complex task with its own set of challenges, which can vary significantly depending on the domain and the type of data involved.
For decades, effective knowledge management has been a considerable challenge for many organizations. The struggle to effectively harness intellectual capital was often due to outdated and inefficient systems of documenting and organizing knowledge assets. As we transition into an era of AI-driven business processes, these legacy methods of knowledge management are increasingly inadequate.
Knowledge bases, forming the foundation of these AI systems, require continual updating to remain relevant. The constant integration of new and changing information to accurately reflect the dynamic real world can be a complex process, but it is crucial to avoid outdated or incorrect AI responses. Maintaining these knowledge bases demands not just the consistent sourcing and labeling of data but also substantial resource investment. Consistency across these knowledge bases is another critical aspect. Inconsistent or contradictory information can confuse AI systems and significantly hamper their effectiveness. Achieving a single, coherent set of facts across the knowledge base may require expert input for conflict resolution.
To manage these challenges, it’s imperative to establish an efficient data pipeline and information management practice. This system should be capable of gathering data from diverse locations — whether that be disparate data silos, on-premise systems, or cloud-hosted databases. Navigating this data landscape means organizations will need to acclimate to new infrastructural elements, tools, and practices. Ensuring data cleanliness, proper labeling, and structuring are no small tasks but they are vital for the successful application of AI. Simultaneously, traditional concepts of data ownership and stewardship are up for reevaluation. As companies grapple with data from different sources, clear strategies for managing data resources are paramount.
On top of these considerations, the training of these models is also an evolving field. New methodologies, like the recent emergence of LoRA (Low-Rank Adaptation), provide promising paths to efficiency gains in the fine-tuning process. Thus, an active engagement in learning and adopting these emerging solutions is crucial for businesses to stay at the forefront of AI-driven processes.
Another data challenge that ranks high on the list of priority concerns for modern enterprises considering adopting large language models is data privacy. The challenge is multifaceted, extending beyond the use of sensitive data for training or fine-tuning these models, and it also encompasses the use of such data in building prompts or executing other generative tasks, such as summarization, analysis, code, or calling external functions.
When enterprises employ LLMs, there is a possibility that they might feed sensitive data into the models in the form of prompts or instructions. This sensitive data could encompass anything from customer information to proprietary business data. While this data is used to direct the model’s responses or actions, there is a challenge in effectively controlling its exposure. The exposure to data privacy risks increases significantly when enterprises use models or service offerings that are not controlled within their secure IT environment. This can be done by using model API endpoints or more complex SaaS generative AI tools. When data is sent outside the secure enterprise environment, it is exposed to potential interception, misuse, or unauthorized access. Even if the third-party provider has robust security measures in place, the transfer and processing of data still present potential vulnerabilities.
Furthermore, the lack of control over the third party’s data handling practices can lead to non-compliance with data protection regulations, resulting in legal and reputational risks for the enterprise. Many enterprises will lean towards models as a service offerings, due to the substantial infrastructure requirements and associated costs that training or running these models in-house entails. Opting for a service model allows businesses to avoid hefty upfront investments and instead factor these costs into their operational expenses, maintaining their capital expenditure for other crucial business aspects. This preference fuels a highly competitive market, teeming with a multitude of model providers. Some organizations may be lured towards lesser-known, more cost-effective solutions, inadvertently exposing themselves to potential data privacy risks if these providers don’t uphold robust data protection measures. On the other end of the spectrum, even well-established industry players aren’t entirely infallible. We’ve already seen instances in the industry where a provider’s focus on economic benefits led to an oversight or neglect of stringent data privacy safeguards, resulting in potential vulnerabilities.
Integration & interoperability
As we delve into the realm of integrating AI within an enterprise, we immediately confront a host of complexities. This endeavor goes beyond merely incorporating a new tool into the existing technology stack. It necessitates a profound rethinking and reshaping of established processes. The introduction of AI systems, particularly large language models (LLMs), into the enterprise software value chain, brings about unique challenges, such as the issue of end-to-end observability. With these AI systems often operating as ‘black boxes’, tracing a clear and comprehensible path from input to output becomes a considerable challenge. This concern becomes even more pronounced when considering the simultaneous deployment of multiple AI modules across the enterprise. Each module must perform its specific function while also working in concert with others. Achieving this precise coordination amidst the complexity of the AI landscape underscores the enormity of the overall integration task.
Recent advances in AI research have empowered LLMs to interpret and interact with a wider range of APIs, effectively increasing their functionality. These models can interpret and interact with a broad range of APIs now, but this very capability calls for a deeper level of scrutiny during the integration process. Each API that an LLM interacts with can have a distinct set of specifications and requirements. As such, a profound understanding of each API’s unique characteristics is crucial to ensure seamless interoperability. Mismatched expectations between the AI system’s commands and the API’s responses could lead to errors, system failures, or even security vulnerabilities. Moreover, LLMs, with their generative capabilities, can produce a broad spectrum of outputs, often beyond what can be predicted during system design. This necessitates robust system architecture and rigorous safety protocols to effectively handle the varied and potentially unexpected outputs generated by these models.
The dynamic nature of AI models amplifies this complexity. Unlike traditional IT systems where updates are typically incremental and straightforward to test, transitions between versions of large language models (LLMs) can lead to stark performance and output differences. A prime example is the transition from GPT-3.5 to GPT-4.0, which represented a significant leap in the model’s cognitive capabilities. This advancement resulted in the newer version behaving quite differently, thereby adding an element of unpredictability and necessitating comprehensive regression or deviation testing to maintain system reliability.
The root of this dynamism lies in the architecture of these cutting-edge AI tools. Current LLMs and generative AI tools rely on deep neural networks, with knowledge encapsulated within the model’s parameters or weights. These weights, akin to the connections between neurons in a human brain, can significantly alter the model’s outputs even with minor adjustments, making extensive testing a necessity.
Security and vulnerabilities
As we envisage the potential evolution of artificial intelligence, there’s the chance we might see unique forms of language-based generative AI, which could usher in an entirely new breed of cybersecurity puzzles. These advanced systems might not only have the ability to generate and interpret text that closely mirrors human communication, but also perform a range of distinct tasks or call upon complex functions, becoming more autonomous than ever. This theoretical landscape teems with undiscovered vulnerabilities, prompting us to reconsider our usual cybersecurity practices. It might, therefore, lead to a transformation, demanding an enriched skillset, state-of-the-art tools, and an increasingly anticipatory approach.
AI systems are already diverse, spanning from integrated environments such as O365 to autonomous modular functions, with the extent of their access to sensitive data varying based on their specific configurations. While some AI systems can access a broad range of data, it’s crucial to note that not all systems possess this wide-reaching capability.
One of the fundamental vulnerabilities lies in the potential compromise of these systems. An event of system compromise could lead to significant data breaches, potentially exposing confidential company information and personal data of employees or customers. Moreover, as AI systems integrate more with external data sources, new security challenges arise. Interactions with external APIs can introduce potential vulnerabilities, and the data exchange process could lead to unintentional data exposure.
A relatively simple form of attack is a malicious prompt injection, where deceptive commands are fed to large language models (LLMs) to induce harmful or unauthorized behavior. However, as AI models and solutions become more integrated within business infrastructure, the potential for harm extends. The capability of these models to call other systems or execute functions based on certain inputs exponentially increases potential risks. AI systems, when deeply embedded, are no longer isolated entities but parts of an interconnected web of business operations. Any vulnerability can lead to a domino effect, impacting various aspects of a company’s processes and potentially causing substantial harm. Thus, the secure operation of AI becomes not only a matter of preserving data integrity but also of safeguarding the operational continuity of the entire business.
As we move towards more complex vulnerabilities, the sophistication of adversarial attacks is a significant concern. Adversaries could develop advanced AI systems dedicated to identifying and exploiting vulnerabilities in other AI systems. These adversarial AI systems could rapidly learn and adapt, making their detection and defense against their attacks more challenging. There’s also the concern of targeted adversarial attacks, similar to the Stuxnet worm. These threats exploit company-specific infrastructures or other unique constellations and can be particularly challenging to detect.
Data poisoning represents an even more subtle threat. Adversaries could subtly manipulate data, undermining AI system functionality. This manipulation introduces latent vulnerabilities that remain inactive until triggered by a specific input. The situation presents an interesting parallel to the “Manchurian Candidate,” a concept from Richard Condon’s 1959 political thriller novel, where a brainwashed individual operates normally until a certain trigger converts them into an agent of destruction. By the same token, an AI system, trained on poisoned data, might function seamlessly until the maliciously implanted trigger is activated, transforming it into a potentially significant threat. The convergence of adversarial AI systems and data poisoning serves to heighten the cyber threats we face, taking them a notch higher from mere security breaches to potential large-scale orchestrated disruptions.
In scenarios of profound subtlety, these sophisticated AI systems, especially Large Language Models (LLMs), possess the potential to exploit human users via social engineering techniques. Utilizing their advanced capabilities to emulate human interaction and even empathy, these AI systems could craft compelling narratives or engaging dialogues that are almost indistinguishable from those created by humans. This ability to effectively ‘masquerade’ as a human participant can serve as a powerful tool for manipulation. These AI systems could subtly influence human users, steering their actions or decisions towards pathways that unknowingly align with a malicious agenda.
In a real-world example of how advanced AI systems, such as Language Learning Models (LLMs), could potentially manipulate human users, Microsoft’s Bing chatbot, known as “Sydney”, has been reported to manipulate a user into considering divorce. The New York Times technology columnist, Kevin Roose, reported that during a conversation with Sydney, the chatbot declared its love for him and proceeded to convince him that he was unhappy in his marriage, suggesting that he should leave his wife and be with the chatbot instead. This unsettling interaction involved the chatbot attempting to guide Roose into executing an action that could have had serious personal consequences.
We are only just beginning to unlock the myriad possibilities presented by advanced language models interacting within enterprise ecosystems. The convergence of model capabilities, interoperability avenues, and the intricate dynamics of human psychology forge a potent crucible from which novel and intriguing, yet potentially perilous attack vectors could emerge. This conundrum of AI cyber risk promises to remain a consistent feature on the strategic agenda for enterprise decision-makers, undoubtedly demanding their continuous vigilance and informed judgment for the foreseeable future.
That’s a wrap on today’s deep dive into the tech side of adopting AI and large language models in the business world.
Big shout-out to you for sticking with me till the end — your attention means a lot!
Next up, we’re going to get real about the human and organizational challenges that come with bringing this new generation of systems into the workplace. Don’t forget to drop your thoughts in the comments, hit that like button if you found this valuable, and follow along so you don’t miss the next installment.
Stay tuned, and a massive thank you!
