[Web3 not in the Books]Transformers in AI — From Tokenomics to Tokenology
Transformers are a key part of generative language AI architecture, so much so that without them large language models like ChatGPT wouldn’t function nearly as well as they do. Similarly, tokenomics are a key element of any economic model and incentive structure surrounding tokens, including blockchain technology. Further to tokenomics, tokenology encompasses the broader ecosystem and the interactions between various token-based systems, essentially giving tokens use and meaning. In this article we’ll explore how transformers play a crucial role in interpreting the world through the lens of language as tokens, and how this interpretation connects to tokenomics and tokenology.
Transformer Architecture in AI
Transformer technology has become essential for generative AI, LLMs in particular. It’s a deep learning model that has revolutionized natural language processing (NLP), and was set forth in the 2017 paper “Attention Is All You Need” by Vaswani et al. in 2017 (and interestingly, Ilia Polosukhin, co-founder of Near protocol, co-wrote the paper).
Transformers are designed to process and understand sequential data. The key innovation of the transformer model is its use of self-attention mechanisms that allows the model to weigh the importance of different words within a sequence, determining how each word should attend to all other words. This process enables it to capture contextual relationships between words & sentences in a way that is more dynamic and context-sensitive than previous approaches.
Early architectures for handling sequential data, like RNNs (recurring neural networks) and LSTM (long short term memory) were limited in that they could not process data in parallel and had diminishing processing capabilities with longer strings of data (longer sentences). Transformers address these limitations with parallel processing and much higher scaling capabilities.
Transformers are so integral to AI language models, in fact, that they feature in their titles. The ‘T’ in ChatGPT stands for transformer, as does the ‘T’ in Google’s BERT AI language model.
GPT, or ‘Generative Pre-trained Transformer’, is designed to understand and generate human-like text, which it does through its extensive pre-training, self-learning and transformer architecture.
BERT, or ‘Bidirectional Encoder Representations from Transformers’, is Google’s AI language model that the company now applies to search results, helping Google to better understand the context around searches using AI in the form of natural language processing and sentiment analysis, processing every word in a search query in relation to all the other words.
Transformers have become integral to language-based generative AI, but they have even more potential to fine-tune context in language generation.
The Power of Transformers: Enabling Language Tokenization
Transformers can enable the tokenization of language (both human and AI generated language) by breaking text down into smaller units which can be regarded as tokens. Each token represents a meaningful unit of language, such as a word, phrase, or subword. By tokenizing language, transformers can analyze and understand the contextual relationships between these tokens, allowing for more accurate and nuanced processing of language, along with an understanding of potential monetary values of language units.
In our first article on tokens we talked about token velocity, which is the speed of token issuance and the rate of token use, essentially meaning the speed of exchange of a token or currency. We saw that token issuance must reflect meaningful token use — that we need to know the speed and velocity of the creation and consumption of the product we are offering, so we can make useful tokens at a rate that users can derive tangible benefit from, accurately reflecting the speed of their use. In our second article we spoke about funnel optimization, where we discussed how to capture users from the awareness stage down to the purchase and token-use stages of the funnel.
Both articles discussed the value of a single sentence within an LLM (large language model), and we saw that the process of sentence generation in an LLM starts and ends with a token.
Tokenomics underlies the economic structure of this process, and tokenology underlies the entire process itself from a broad and semantic perspective..
The Milk Example: Tokenization and Granular Value
Transformers can assign probabilities to the occurrence of specific language-based tokens based on the context in which they appear. We’ll use an example here — let’s say you have a dog named “Milk”. If you were to ask the AI Network LLM a question about your dog, specifying your dog’s name, in the context of the conversation the transformer may assign a probability of around 10% to the phrase “is cute” following the word “Milk” in its subsequent sentence generation, ie “Your dog Milk is cute”. In contrast, it may assign the probability of the word “flows” as extremely low, perhaps around 0.0001%. This is because the possibility of “flows” following “Milk” still exists (because milk can flow), but because the transformer has been given the context that Milk is the name of a dog, the probability of “flows” becomes highly unlikely. This demonstrates how transformers analyze the contextual relationships between tokens and assign probabilities based on the likelihood of their occurrence.
All word and associated token data in the AI Network token economy is recorded on the blockchain, and “Milk”, and any other language based token are considered as non-fungible tokens (NFTs). This means every single language-based token in the LLM represents a unique entity. The value of the “Milk” token would be determined by various factors, such as the frequency of its use, the context in which it appears, and the overall demand for dog-related content within the network.
The same language-based token coming into the LLM can have different values based on the context of the sentence. The probabilities of word usage in the LLM change every time words are used in new sentences, and the more often specific words are used, the more value they will gain as tokens and, generally speaking, the higher the value of the $AIN token will climb. The $AIN token is AI Network’s utility token and is separate from language tokens (the $AIN token is the driving factor behind the LLM and is what allows the model to generate sentences).
This is essentially a form of language mining. If a user has meaningful conversations with an LLM, it increases the value of the LLM and therefore the underlying tokens. The accurate assignment of correct probabilities of words in an LLM will make the AI model more value (e.g, in our dog “Milk” example, the LLM that assigns higher probability to “cute” after milk than “flows”) will hold more value due to a more successful contextual understanding.
By applying this granular approach to token valuation, the AI Network can assign a precise value to each token, reflecting its unique characteristics and potential within the token economy.
Tokenomics to Tokenology
While tokenomics focuses on the economic models of individual tokens, tokenology takes a broader view, encompassing the entire ecosystem of token-based interactions and all within it. Tokenomics is a role of the economy, describing token supplies, caps, burn & mint mechanisms etc, whereas tokenology describes meanings of tokens; what they’re used for and what value they bring to users. There is no tokenomics without tokenology, because tokenomics alone cannot answer why a product is valuable. Tokenology essentially gives tokens their meaning and defines models & structures for token use within an economy and ecosystem.
In keeping with tokenology and its encompassing of token-based interaction ecosystems, global market capitalization of language represents the total value of all language-based content and interactions. Within this global language market cap, various entities, such as ChatGPT, human-generated content, and AI Network’s language models, compete for a share. The token economy facilitates the exchange of value and incentivizes participation and contribution within the entire ecosystem.
Connecting Transformers, Tokenology, and Tokenomics
Transformers enable the tokenization of language, breaking it down into meaningful units that can be analyzed and valued. These tokens can be recorded on the blockchain as unique entities and used in LLMs for language generation, possessing dynamic values as they’re utilized in the LLM by users.
Tokenology provides the economic models and incentive structures that govern the creation, distribution and use of these language-based tokens. The tokenomics, in turn, facilitates the exchange of value and incentivizes participation and contribution within the ecosystem. The granular valuation of language-based tokens, as demonstrated by the “Milk” example, highlights the potential for tokenomics to capture the nuanced and contextual value of language.
In conclusion, transformers have revolutionized generative language AI through deep learning, parallel processing and context-finding dynamics. When connected to the concept of tokenization, transformers enable the interpretation of the world through the lens of language as tokens, and how these interpretations connect to the concepts of tokenology and the token economy. Transformers play a crucial role in tokenizing language, breaking it down into meaningful units that can be analyzed, valued and utilized in AI models. AI Network can unlock the full potential of language-based tokens through its ecosystem and LLM, creating new opportunities for value creation, exchange and collaboration for all users.
AI Network is a decentralized AI development ecosystem based on blockchain technology. Within its ecosystem, resource providers can earn $AIN tokens for their GPUs, developers can gain access to GPUs for open source AI programs, and creators can transform their AI creations into AINFTs. The ultimate goal of AI Network is to bring AI to Web3, where everyone can easily develop and utilize artificial intelligence.
If you want to know more about us,
- Homepage: https://www.ainetwork.ai/
- AI Network DAO Discord: https://discord.com/invite/aindao/
- AI Network YouTube: https://www.youtube.com/@ainetwork_ai/
- AI Network Telegram: https://t.me/ainetwork_en
- AI Network Twitter: https://twitter.com/ainetwork_ai
