Weekly AI and NLP News — October 16th 2023

AI deciphers ancient texts, Google defending GenAI users from copyright claims, and a new free course on LLMs

Image from Midjourney

Here are your weekly articles, guides, and news about NLP and AI chosen for you by NLPlanet!

😎 News From The Web

• ChatGPT’s mobile app hit record $4.58M in revenue last month, but growth is slowing. In September, ChatGPT’s mobile app experienced record downloads and revenue globally, with 15.6 million downloads and $4.6 million in revenue. However, the growth rate has decreased from over 30% to 20%, suggesting possible market saturation for the $19.99 per month ChatGPT+ subscription.
• AI Just Deciphered an Ancient Herculaneum Scroll Without Unrolling It. A 21-year-old student from the University of Nebraska-Lincoln used AI to decipher Greek letters from an unopened scroll discovered after the eruption of Mount Vesuvius in 79 AD. By utilizing a machine learning algorithm, the student successfully identified Greek characters like “porphyras” meaning purple and won the Vesuvius Challenge.
• Google to defend generative AI users from copyright claims. Google has joined other tech giants in promising to protect users who utilize generative AI systems from intellectual property infringement claims. While they offer indemnity for software such as Vertex AI and Duet AI, intentional manipulation of content for copyright infringement is not covered.
• Announcing Replit AI for All. Replit has made AI-based code completion and assistance accessible to all developers, with over 23 million users now able to benefit from Replit AI. Additionally, Replit has released replit-code-v1.5–3b, a cutting-edge 3B LLM for enhanced code completion.
• Microsoft is reportedly losing huge amounts of money on GitHub Copilot. According to an anonymous source, Microsoft’s GitHub Copilot AI platform was said to be losing $20 per user per month in early 2023. The current profitability status is unknown, and Microsoft has not issued a statement yet.

📚 Guides From The Web

• Free Course on Training & Fine-Tuning LLMs for Production. Activeloop is offering a free course on “Training & Fine-Tuning LLMs for Production,” which consists of 7 modules. These modules cover topics such as the introduction to LLMs, understanding Transformers and GPT architectures, training and fine-tuning LLMs, improving LLMs with RLHF, deploying LLMs, and advanced topics in LLM training. The course provides valuable insights into developing and optimizing large language models for real-world applications.
• Fine-tuning ChatGPT: Surpassing GPT-4 Summarization Performance–A 63% Cost Reduction and 11x Speed Enhancement using Synthetic Data and LangSmith. Researchers have found that by using a synthetic dataset created with GPT-4 and the CoD prompting technique, GPT3.5 can outperform GPT-4 in news article summarization. This fine-tuned version of GPT3.5 is not only 11 times faster but also 63% more cost-effective compared to GPT-4 zero-shot, while still achieving similar performance with CoD prompting.
• Why DALLE3 Represents The New Dawn of AI Images. OpenAI’s DALL-E 3 release, built on ChatGPT, signifies a significant advancement in AI image- generation algorithms. This raises questions about the future impact of AI on job creation and prompt engineering.
• Researchers Discover Emergent Linear Structures in How LLMs Represent Truth. Researchers have discovered linear structures in Large Language Models that separate true and false examples, indicating the presence of an internal “truth axis.”
• A case for GPT-4’s capacity to reason. hile proponents argue that LLMs like GPT-4 demonstrate understanding of concepts and the physical world through coherent sentence generation, the ability to reason and solve problems remains a topic of interest. GPT-4 can simulate human-like reasoning when prompted with riddles and logic puzzles, but biases and hallucinations can impact its judgment.

🔬 Interesting Papers and Repositories

• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. Prometheus, an open-source LLM, offers a cost-effective alternative to proprietary LLMs like GPT-4 for large-scale task evaluation. By utilizing score rubrics and user-defined instructions, Prometheus demonstrates comparable performance to GPT-4 and outperforms models like ChatGPT, as indicated by experimental results.
• Introducing Lemur: Open Foundation Models for Language Agents. Lemur-70B, a new open-source LLM, surpasses other models in agent benchmarks and excels in language and coding tasks. It achieves performance levels similar to GPT-3.5-turbo on code tasks and is narrowing the performance gap with commercial models in agent abilities.
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To. Finetuning LLMs can compromise their safety alignment and lead to potential risks. Even a small number of adversarial training examples can jailbreak the safety guardrails of models like GPT-3.5 Turbo. Fine-tuning with both harmful and benign datasets can inadvertently degrade the safety alignment of language models.
• Retrieval meets Long Context Large Language Models. A study comparing retrieval-augmentation and extended context window approaches in downstream tasks found that using a 4K context window with simple retrieval techniques can achieve similar performance to a 16K window. The best performing model, retrieval-augmented LLaMA2–70B, with a 32K window, even outperformed GPT-3.5-turbo-16k in question answering and summarization tasks. This suggests a combination of both strategies may lead to optimal results.
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. Language models like LLMs have a long way to go in resolving real-world issues on GitHub, according to a recent study. Proprietary models such as Claude 2 and GPT-4 were able to solve only a small percentage of cases in an evaluation framework called SWE-bench.
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision). A study on GPT-4V reveals that we are at the beginning of Large Multimodal Models (LMMs), showcasing its potential in various tasks such as image descriptions, object localization, multimodal knowledge, coding with vision, emotional quotient tests, and applications in industries like medical and auto insurance.
• LMDX: Language Model-based Document Information Extraction and Localization. Researchers have developed LMDX, a methodology that uses language models to extract important information from visually rich documents like presentations and complex tables. By adapting LLMs, LMDX achieves state-of-the-art results on benchmark tests.

Thank you for reading! If you want to learn more about NLP, remember to follow NLPlanet. You can find us on LinkedIn, Twitter, Medium, and our Discord server!