Types of Open Source & Closed Source LLMs(Large Language Models)

Introduction

Large language models (LLMs) have revolutionized natural language processing and AI. Open source & Closed Source LLMs provide researchers, developers and the general public with access to this powerful technology.

Open Source LLMs

There are many different types of open source LLMs, each with their own strengths, limitations and use cases. Here are some of the major types:

1. General-purpose LLMs

These models are trained on large amounts of general web text and aim to be useful for a wide range of tasks. Examples include:

• GPT-3 — Developed by OpenAI, GPT-3 has 175 billion parameters and can generate text, answer questions and perform other tasks.
• BERT — Developed by Google, BERT uses Transformer architecture and is widely used for downstream NLP tasks.

2. Domain-specific LLMs

These models are trained on text from a specific domain like science, medicine or law. They tend to perform better for domain-specific tasks. Examples are:

• BioBERT — Trained on biomedical literature and used for biomedical NLP tasks.
• SciBERT — Trained on scientific text and effective for scientific information extraction and question answering.

3. Multilingual LLMs

These models support multiple languages and are trained on text data from different languages. Examples are:

• XLM — Developed by Facebook, XLM supports 100 languages and aims to be useful for cross-lingual tasks.
• Multilingual BERT — Supports 104 languages and can perform tasks like sentiment analysis and named entity recognition in any of those languages.

4. Few-shot LLMs

These models are designed to perform well even when fine-tuned with small amounts of labeled data. Examples are:

• GPT-3
• T5 — Developed by Google, T5 can achieve high performance with as little as 10 examples for training.

5. Task-specific LLMs

These models are tailored for a specific NLP task like summarization, question answering, translation, etc. Examples are:

• BART — Facebook’s model for text generation tasks like summarization and question generation.
• ALBERT — Google’s model for question answering and sentence classification.

In summary, there are many types of open source large language models to choose from based on your specific needs — general purpose or domain-specific, multilingual, few-shot, and task-specific. The diversity of LLMs available allows for a high degree of flexibility and customization.

Note

These are some open-source LLMs, we can also see some closed source LLMs.

What is the Difference between Open Source LLMs & Closed Source LLMs?

Open Source LLMs (Language Models) and Closed Source LLMs refer to two different types of language models based on their availability and accessibility to the public. Let’s explore the differences between them:

Open Source LLMs

Open Source LLMs are language models whose source code is publicly available and can be freely accessed, used, modified, and distributed by anyone. Open source models encourage collaboration, transparency, and community involvement. Developers, researchers, and enthusiasts can contribute to the development, improvement, and customization of these models. The open-source nature allows for greater innovation, knowledge sharing, and collective development efforts.

Closed Source LLMs

Closed Source LLMs, on the other hand, are language models whose source code is not publicly available. They are developed and maintained by organizations or companies that typically keep the underlying code proprietary and closed to the public. These models are often developed as commercial products and may require licenses or subscriptions for their use. The specific details of their architecture, training data, and algorithms are generally not accessible to the public.

Key differences between Open Source and Closed Source LLMs include:

1. Accessibility: Open Source LLMs are accessible to anyone, allowing users to inspect the code, modify it, and use it for various purposes without restrictions. Closed Source LLMs, however, are not publicly accessible, and their usage is subject to the terms and conditions set by the organization or company that owns them.

2. Customization: Open Source LLMs offer flexibility for customization and modification according to specific needs or use cases. Developers can adapt the models, fine-tune them for specific tasks, and experiment with novel techniques. Closed Source LLMs typically have limited customization options, as the underlying code is not accessible for modification.

3. Community Contributions: Open Source LLMs often have a thriving community of developers and researchers who contribute to their improvement, bug fixes, and feature enhancements. The collaborative nature of open-source projects allows for collective intelligence and diverse perspectives. Closed Source LLMs rely on internal development teams for updates and enhancements, limiting external contributions.

4. Licensing and Costs: Open Source LLMs are generally available under permissive licenses that allow free usage, modification, and distribution. This can significantly reduce costs for users. Closed Source LLMs may require licensing or subscription fees, as they are often commercial products developed and maintained by organizations.

It’s worth noting that the availability of LLMs, both open source and closed source, is subject to the policies and decisions of the organizations or companies behind them. The specific features, performance, and limitations of each LLM can vary regardless of their open or closed source nature.

Closed Source LLMs

Here are some closed source LLMs which the researchers refuse to hand over to the public.

HyperCLOVA

Naver Corp’s HyperCLOVA, the South Korean-language AI model, was released in May 2021. The company all set to launch this July to launch an upgraded version called HyperCLOVA X, which can understand images and speech in a multimodal format. Trained on a massive corpus of 560B tokens, the Korean GPT-3, as it is called, can be a game-changer in the world of natural language processing, according to Kim Yu-won, CEO of Naver Cloud Corp.

Gopher

DeepMind’s Gopher is a 280 billion parameter transformer language model. The researchers claimed that the model almost halves the accuracy gap from GPT-3 to human expert performance, exceeding forecaster expectations and lifting performance over current state-of-the-art language models across roughly 81% of tasks.

Chinchilla

Deepmind’s another addition to their animal-inspired lineup of models is Chinchilla — a 70B parameters model is designed to be compute-optimal. With 1.4 trillion tokens in its training data, Chinchilla was found to be optimally trained by equally scaling both model size and training tokens. Despite using the same compute budget as Gopher, Chinchilla boasts 4x more training data, making it a formidable contender in the language model space.

BloombergGPT

Last month, Bloomberg unveiled BloombergGPT, a new large-scale generative AI model, specifically designed to tackle the complex landscape of the financial industry. This highly trained language model, optimised to parse and process vast quantities of financial data, seems promising in the NLP domain.

Overview of Some Open Source & Closed Source LLMs one by one

GPT-3

A large autoregressive language model. Good for generating text.

GPT-3 is a large language model developed by OpenAI that exhibits a wide range of capabilities including natural language generation, text summarization, question answering, and translation. GPT-3 uses a Transformer-based architecture and was trained on an immense dataset of over 750GB of Internet text. It has a staggering 175 billion parameters, making it the largest language model to date. GPT-3’s main strengths lie in its ability to produce human-like text and responses, due to being trained on massive amounts of real-world data. However, like all large language models, GPT-3 also suffers from issues like bias, lack of context and inability to understand complex ideas. Despite its limitations, GPT-3 has proven to be a groundbreaking model that has the potential to change the way we interact with and build AI systems.

Grover

A large bidirectional Transformer for text generation and question answering.

Grover is an open source large language model developed by the Hugging Face organization. It uses a Transformer-based architecture and has 2.6 billion parameters, making it one of the largest openly available models. Grover was trained on a very large dataset consisting of over 750 GB of web text. Unlike many other large language models that focus only on language generation, Grover was trained for both language generation and question answering, making it well suited for tasks like summarization, text completion and fact-checking. In evaluations, Grover has demonstrated state-of-the-art or near state-of-the-art performance on a wide range of NLP tasks including question answering, reading comprehension and text generation. Grover’s multi-task training approach and ability to perform both language understanding and language generation tasks make it a versatile open source alternative to commercial large language models.

BERT

A Transformer-based model for language understanding tasks like question answering and sentiment analysis.

BERT stands for Bidirectional Encoder Representations from Transformers. It is a large language model created by Google in 2018 that pioneered the technique of pre-training deep learning models on large text corpora. BERT uses a multi-layer Transformer encoder and is trained on two tasks: masked language modeling and next sentence prediction. This pre-training approach allows BERT to learn contextual relationships between words that can be used for a wide range of downstream natural language processing tasks. After pre-training, BERT can be fine-tuned with a small amount of task-specific labeled data for various applications like question answering, text classification, sentiment analysis, named entity recognition and more. BERT has significantly improved state-of-the-art results on many NLP tasks and has led to the development of many BERT-based models. It has become an important building block for many natural language processing applications.

OpenChatKit

OpenChatKit is an open source large language model developed by Anthropic. Some key points about OpenChatKit:

• It is a 1.3 billion parameter model trained on web data and books.

• The goal of OpenChatKit is to provide an ethical and unbiased alternative to closed large language models like GPT-3.

• The developers claim OpenChatKit avoids issues like toxicity, repetition and factual inaccuracies seen in other LLMs.

• OpenChatKit can be used for text generation, question answering, summarization and classification tasks.

• The model and code are openly available under an MIT license.

• OpenChatKit can be accessed through an API or used locally.

The main advantages of OpenChatKit are:

1. It is fully open source and freely available.

2. The developers claim it has been trained for ethical alignment and reduced biases.

3. It provides an alternative to commercial LLMs, especially for privacy-focused use cases.

4. It can be easily run locally.

Some limitations of OpenChatKit are:

1. The model is significantly smaller than commercial LLMs like GPT-3.

2. The claims of reduced bias and ethical alignment have not been independently verified.

3. It is still an early stage model and likely lags the capabilities of much larger closed LLMs.

In summary, OpenChatKit shows potential as an ethical alternative for open source and privacy-focused use cases. But more research and development is needed to improve its capabilities and match the scale of commercial LLMs.

The main differences between OpenChatKit and other open source LLMs like ChatRWKV and Alpaca are:

• OpenChatKit is significantly smaller at 1.3B parameters vs 1.5B for ChatRWKV and 10–13B for Alpaca

• OpenChatKit was trained only on web data and books while the others used a wider range of sources

• The licensing and development teams also differ between the models

But in general, all of these open source LLMs aim to provide alternatives to commercial models like GPT-3 that are fully open, ethical and capable of a wide range of natural language tasks.

Vicuna

Vicuana is an open source large language model developed by Vicarious AI. Some key points about Vicuana:

• It is a 10 billion parameter model, making it one of the largest publicly available LLMs.

• Vicuana was trained on a mixture of web data, books, scientific papers and other sources.

• The developers claim Vicuana generates coherent, accurate and unbiased text.

• Vicuana can be used for text generation, question answering, summarization and other natural language tasks.

• The model and code are openly available under an MIT license.

• Vicuana can be accessed through an API or used locally.

The main advantages of Vicuana are:

1) Its extremely large size of 10 billion parameters. This likely gives it more capabilities compared to most other open source LLMs.

2) The developers claim Vicuana avoids issues like repetition, contradiction and toxicity seen in other large LLMs.

3) It is fully open source and freely available for any use.

4) It can be self-hosted for privacy-focused use cases.

Some limitations of Vicuana are:

1) Independent evaluations of the model’s capabilities and ethical alignment have been limited so far.

2) Using a model of this size requires powerful hardware and computational resources.

3) The model is still in relatively early stages of development and refinement.

In summary, Vicuana shows potential as an alternative to commercial LLMs like GPT-3, especially for use cases requiring an extremely large openly available model. But more independent research is needed to fully evaluate its capabilities and limitations.

The main differences between Vicuana and other large open source LLMs like Alpaca are:

• Vicuana is slightly larger at 10B parameters vs 13B for Alpaca

• They were trained on different datasets and may have different capabilities and limitations

��� The licensing is different — Vicuana uses an MIT license while Alpaca uses Apache 2.0

• The claims about reduced bias, toxicity and accuracy also differ between the two models

But in general, both Vicuana and Alpaca represent some of the largest and most capable openly available LLMs currently in development.

Alpaca

Alpaca is an open source large language model developed by Anthropic. Some key points about Alpaca:

• It is a 13 billion parameter model, making it one of the largest publicly available LLMs.

• Alpaca was trained on a mixture of web data, books, Wikipedia and other sources.

• The developers claim Alpaca generates more factually correct and coherent text compared to other large LLMs.

• Alpaca can be used for text generation, question answering, summarization and other natural language tasks.

• The model and code are openly available under an Apache 2.0 license.

• Alpaca can be accessed through an API or used locally.

The main advantages of Alpaca are:

1) Its extremely large size of 13 billion parameters. This likely gives it more capabilities compared to most other open source LLMs.

2) The developers claim Alpaca avoids many of the issues seen in other large LLMs like repetition, contradiction and toxicity.

3) It is fully open source and freely available for any use.

4) It can be self-hosted for privacy-focused use cases.

Some limitations of Alpaca are:

1) Independent evaluations of the model’s capabilities and ethical alignment have been limited so far.

2) Using a model of this size requires powerful hardware and computational resources.

3) The model is still in relatively early stages of development and refinement.

In summary, Alpaca shows potential as an alternative to commercial LLMs like GPT-3, especially for use cases requiring an extremely large openly available model. But more independent research is needed to fully evaluate its capabilities and limitations.

GPTall

GPT4all is a community-driven project trained on a massive curated collection of written texts of assistant interactions, including code, stories, depictions, and multi-turn dialogue. The team has provided datasets, model weights, data curation processes, and training code to promote the open-source model. There is also a release of a quantized 4-bit version of the model that is able to run on your laptop as the memory and computation power required is less. A Python client is also available that you can use to interact with the model.

ChatRWKV

ChatRWKV is an open source large language model developed by researchers at the Indian Institute of Technology Madras. Some key points about ChatRWKV:

• It is a 1.5 billion parameter model trained on web data and books.

• It aims to generate unbiased and ethically aligned text.

• The researchers claim that ChatRWKV avoids many of the issues seen in models like GPT-3, including toxicity, repetition and inconsistency.

• The model can be used for text generation, question answering, summarization and classification tasks.

• ChatRWKV can be accessed through an API or used locally through a Jupyter Notebook interface.

• The model and code are openly available under a permissive license.

The key advantages of ChatRWKV compared to other large LLMs are:

1) It is fully open source and freely available.

2) The researchers claim it has been trained for ethical alignment and reduced biases.

3) It is a large scale model comparable to GPT-2 and other 1B+ parameter models.

4) It can be easily run locally for privacy-focused use cases.

Some limitations of ChatRWKV are:

1) The model is still in relatively early stages of development.

2) It likely lags behind the capabilities of models like GPT-3 and ChatGPT due to its smaller size.

3) The claims of reduced bias and ethical alignment have not been independently verified.

In summary, ChatRWKV shows potential as an alternative to commercial LLMs, especially for open source and privacy-focused use cases. But more research and development is needed to improve its capabilities and match models from large tech companies.

Bloom

Bloom LLMs are a family of large language models developed by Bloom Intelligence. Some key points about Bloom LLMs:

• They are open source and available under the Apache 2.0 license.

• They cover a range of sizes from 600M parameters to 3.8B parameters.

• They are trained on a large corpus of web data and books.

• They aim to be unbiased and avoid toxic output.

• They can generate text, answer questions, summarize text, and perform other natural language tasks.

• They can be fine-tuned for specific domains and tasks.

The main Bloom LLMs are:

• Bloom QA — A question answering model with 3B parameters.

• Bloom Text Generation — A text generation model with 2.6B parameters.

• Bloom Embeddings — Word and sentence embeddings extracted from the Bloom models.

• Bloom Foundation — A 600M parameter LLM intended as a starting point for fine-tuning.

The main advantages of Bloom LLMs are their open source nature, large sizes, and claimed reduced biases compared to other large LLMs. However, they are still in early stages of development and lag behind models like GPT-3 in terms of capabilities.

The Bloom models can be accessed through an API, downloaded for self-hosting, or used through Jupyter Notebooks. This makes them a potentially useful alternative to ChatGPT for some natural language tasks.

HyperCLOVA

HyperCLOVA is a closed source language model developed by Naver Corporation. With its advanced natural language processing capabilities, HyperCLOVA enables users to interact with various applications and services through voice commands and text input. Its closed source nature ensures that the underlying code and algorithms are not accessible to the public. HyperCLOVA’s proprietary design allows Naver Corporation to maintain control over its development, updates, and distribution, offering a tailored user experience while protecting their intellectual property rights.

Chinchilla

Chinchilla is a closed source large language model created by Anthropic. It is trained on a massive dataset of over 500 billion parameters, making it one of the largest language models available. Chinchilla uses a Transformer-based architecture and is capable of generating human-like text, answering complex questions and performing a variety of language tasks with high accuracy. Due to its closed source nature, not much is known about the exact techniques used to train and optimize Chinchilla. However, Anthropic claims that it outperforms other publicly available language models in several tasks.

BloombergGPT

BloombergGPT is Bloomberg’s own closed source large language model. Not much is known about the size, architecture and techniques used to train BloombergGPT due to its closed source nature. However, Bloomberg claims that BloombergGPT outperforms other publicly available language models in generating text relevant for the financial domain. It is aimed at assisting Bloomberg’s journalists, analysts and traders with tasks like summarization of financial documents, generating reports from financial data, and answering queries related to financial information. BloombergGPT was likely trained on Bloomberg’s huge corpus of financial data comprising news articles, company reports, transcripts, and market data. It indicates Bloomberg’s aim to leverage advanced AI technologies like large language models to gain a competitive edge in the financial information and analytics space.

Conclusion

In conclusion, there are many types of large language models available today — both open source and closed source.

Open source LLMs offer several advantages like accessibility, customizability, community contributions and lower costs. They promote transparency, collaboration and innovation. However, they are typically smaller in size and lag behind closed source LLMs in capabilities.

Closed source LLMs tend to be larger, more optimized and capable of higher performance. But their proprietary nature limits accessibility, customization and contributions from the community. They are often developed as commercial products.

Both open source and closed source LLMs have their pros and cons. The choice depends on factors like requirements, use cases, resources and ethics. Open source LLMs provide a good starting point for research and experimentation, while closed source models offer higher performance for production applications. An ideal approach may be to start with open source LLMs and gradually move to closed source ones as needs scale up.

Overall, the availability of different types of large language models — from GPT-3 and BERT to HyperCLOVA and BloombergGPT — represents significant progress in natural language processing and AI. Both open source and closed source LLMs will likely continue to evolve and improve in the coming years, shaping how we interact with and build intelligent systems.

Author: Ayush Kumar