Large Language Models: Open Source vs Proprietary

Photo by Milad Fakurian on Unsplash

Open Source Software environments are known to be highly innovative and collaborative. The same can be implemented for the improvement of LLMs, but there are few challenges ahead, due to which proprietary LLMs have better usability today.

This article will include information on:

• Large Language model
• Open source LLM
• Proprietary LLM
• How OSS will create innovation and growth in AI
• Constraints with open source LLMs
• AI ethics with open source LLMs

Large Language Model

A Large Language model (LLM) is a deep learning model that can understand and generate natural language text (OpenAI, 2023). They usually contain the transformer architecture (Vaswani et. al, 2017) that are trained on very large datasets. Natural language text is understood and generated by such LLMs through sequential ordering (OpenAI, 2019).

Photo by Justin Campbell on Unsplash

Open Source models

Open source large language models are LLMs that are made available to the public free-of-cost for anyone to use, examine, alter and redistribute however they like or through a licensing agreement (IBM, 2024). It allows developers to share models, resources, and research (Lutkevich, 2023) on the development of LLMs.

Open source LLMs have greater transparency of training architecture and data which allows enterprises to have more trust. They offer greater flexibility in using these LLMs in their infrastructure. They can be cost-saving compared to proprietary LLMs in the long-term (IBM Data and AI Team, 2023). Other than these advantages open source systems typically lead to greater innovation (Dong et. al, 2019).

Some open source LLMs include Llama-2 by MetaAI, and GPT-2 by OpenAI. Various open source LLMs can be found at huggingface.

Proprietary Models

Proprietary large language models are LLMs that are sold to the public as a service by the creator, that cannot edited, enhanced or redistributed except as specified by the creator. For example, GPT-4 by OpenAI, and Bard by Google. Proprietary LLMs have better usability and stability.

My view on the Open-source vs Proprietary LMM debate

Ecosystems developed as Open Source Software (OSS) are considered to be highly innovative and reactive to new market trends due to their openness and wide-ranging contributor base (Linåker, 2016). Hence, in my opinion making LLMs open sourced and allowing people to train on their own would foster innovation at an exponential level. I see a future where LLMs can be trained conveniently and effortlessly by everyone.

The key constraint why proprietary LLMs have a better usability is the limitation of processing power. To train an open-source llm or generate responses from these LLMs requires a system with large storage capacity and GPU is a must, even with these today it is quite slow.

Making LLMs open sourced would require technological advancement in processing power and architecture of LLMs, but I think it can be done.

There are various AI ethics and risks with open source LLMs that needs to be addressed such as hallucinations, biases, privacy, malicious attacks, security threats, and accountability, but I am more excited about the innovation ahead in the AI sphere.

References

1. “Better Language Models and Their Implications”. OpenAI. 2019–02–14. Retrieved 2024–01–06.
2. OpenAI (2023–03–27). “GPT-4 Technical Report”. arXiv:2303.08774
3. Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). “Attention is All you Need” (PDF). Advances in Neural Information Processing Systems. Curran Associates, Inc. 30.
4. IBM. (2024). What is open source software? https://www.ibm.com/topics/open-source
5. IBM Data and AI Team. (2023, September 27). Open source large language models: Benefits, risks and types. IBM Blog. https://www.ibm.com/blog/open-source-large-language-models-benefits-risks-and-types/
6. Lutkevich, B. (2023, September 13). What is huggingface?. TechTarget. https://www.techtarget.com/whatis/definition/Hugging-Face
7. John Qi Dong, Weifang Wu, & Yixin (Sarah) Zhang. (2019). The faster the better? Innovation speed and user interest in open source software. Information & Management, 56(5), 669–680. https://doi.org/10.1016/j.im.2018.11.002
8. Linåker, J., Rempel, P., Regnell, B., & Mäder, P. (2016). How firms adapt and interact in open source ecosystems: Analyzing stakeholder influence and collaboration patterns. Lecture Notes in Computer Science (pp. 63–81). Springer International Publishing. doi:10.1007/978–3–319–30282–9_5