Machine Learns — Newsletter #14
🤖 AI: Latest News, Research, and Open-Source
Hi everyone,
I trust you are all doing well. Here is the new issue of Machine Learns which covers the past two weeks.
During this time, there have been some exciting career developments, which took this issue one day off.
Also, if you haven’t had the chance to read my post “Goods & Bads of Being an Open-Source AI Company” I encourage you to give it a read.
So let’s dive in …
Bookmarks
📰 Apple is enhancing Siri — 📎Link
and Messages in iOS 17.4 by incorporating AI and large language models, including internal testing with OpenAI’s ChatGPT API. They plan to replace ChatGPT with their own model in iOS 18.
📰 A Leaked Memo from Google CEO Sundar Pichai Comes Amidst Employee Discontent — 📎Link
👩💻 RWKV-v5 is out — 📎Link
.Eagle 7B is a 7.52B parameter model built on the RWKV-v5 architecture that excels in multi-lingual benchmarks and approaches the performance of larger models while being attention-free and available under the Apache 2.0 license.
👩💻 Google’s AMIE AI — 📎Link
is being developed to assist in diagnosing patients, aiming to enhance healthcare through advanced understanding and diagnostic capabilities.
📰 The FTC is investigating Microsoft, Amazon, and Google’s investments — 📎Link
in AI companies OpenAI and Anthropic to assess their impact on competition and innovation in the generative AI sector.
📌 Canny’s hiring guide — 📎Link
to growing teams at bootstrapped startups. Great read!
📌 2024 forecasts a year of resilience for startups and VCs — 📎Link
with challenges and strategic opportunities in funding, IPOs, M&As, and the evolving role of generative AI in enterprise applications.
📌 AI today and trends for an AI future — by David Song — 📎Link
📰 China’s Jinping Underground Laboratory becomes the world’s deepest and largest, advancing the global quest for dark matter detection. — 📎Link
👩💻 Lumiere — Google’s latest AI video generator — 📎Link
📌 Should The Future Be Human? — by Scott Alexander — 📎Link
📌 How Users Read on the Web — 📎Link
We don’t read the web, we scan. When we can’t scan, we skip. Good tactics to make a content more scannable.
📚 UX Myths — 📎Link
Debunking the most common UX myths. Great resource!
👩💼 How to have “the pricing talk” with your customers — 📎Link
“Just don’t literally ask them “how much are you willing to pay?” — they’ll low-ball you.”
👩💼 Tech’s Blind Spots: The Startups Building for Underserved Markets — 📎Link
👩💻 PyTorch 2 Internals — Talk — 📎Link
👩💼 I regret selling my startup — 📎Link
Great read!
👩💼 Measuring Developer Productivity — 📎Link
Examples from different big-tech companies.
Papers
- Good old papers — Averaging Weights Leads to Wider Optima and Better Generalization
📎 Paper
This paper introduces Stochastic Weight Averaging (SWA), a training method for deep neural networks that averages multiple points along the trajectory of SGD with a cyclical or constant learning rate, leading to better generalization and flatter solutions than conventional SGD. SWA achieves notable improvements in test accuracy on various state-of-the-art networks and datasets with minimal computational overhead. Its importance lies in its simplicity, ease of implementation, and ability to improve model generalization without significant additional computational cost.
Lumiere: A Space-Time Diffusion Model for Video Generation — Google
📎 Paper
“Lumiere” is a text-to-video diffusion model for synthesizing videos that depict realistic, diverse, and coherent motion, addressing a key challenge in video synthesis. It employs a Space-Time U-Net architecture for generating the full temporal duration of videos in one pass, avoiding the limitations of previous models that synthesize keyframes and then apply temporal super-resolution. This approach ensures global temporal consistency and leverages spatial and temporal down- and up-sampling, significantly improving text-to-video generation results. “Lumiere” is pivotal for advancing video synthesis, offering enhanced content creation and editing capabilities.
Levels of AGI: Operationalizing Progress on the Path to AGI
📎 Paper
Although AGI is a very hot topic right now, there is no consensus on what it is, how to measure it, or how to measure progress towards it. This paper argues different definitions of AGI. It gives 5 levels of AGI.
- No AI: Calculator
- Lvl1 — Emerging Narrow AI: Better than unskilled humans
- Lvl2 — Competent: at least 50th percentile of skilled adults
- Lvl3 — Expert: at least 90th percentile of skilled adults
- Lvl4 — Virtuoso: at least 99th percentile of skilled adults
- Lvl5 — Superhuman: at least 100 h percentile of skilled adults
It discusses current AI systems are lvl2 systems and require more work to fulfill the requirements of a Competent AI.
It also proposes a framework for evaluating Artificial General Intelligence (AGI) progress by defining levels of AGI based on performance, generality, and autonomy. It aims to provide a shared language for comparing models, assessing risks, and tracking AGI development, similar to autonomous driving levels. By analyzing various AGI definitions, the authors distill six principles for a useful AGI ontology, focusing on capabilities over mechanisms and emphasizing the importance of metacognitive tasks.
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
📎 Paper
SliceGPT relies on computational invariance within transformer networks by orthogonal transformation to reduce the size of large language models by deleting rows and columns from weight matrices. It applies PCA to the signals between layers to identify less important dimensions. Deletes rows/columns of weight matrices corresponding to minor principal components. This reduces the embedding dimension/size of weight matrices while maintaining accuracy. It can compress models like OPT 66B and LLAMA-2 70B by up to 25% while maintaining 99% and 90% of the original performance.
Open-Source
Plock
Use an LLM (or anything else that can stream to stdout) directly from literally anywhere you can type. Outputs in real-time.
UpTrain
UpTrain is an open-source evaluation tool for LLM applications, providing metrics for analyzing response quality, context relevance, and language quality, as well as options for custom evaluations. It is for detailed assessments, including features for customization, real-time dashboards, and embedding similarity checks, aimed at enhancing LLM application performance and mitigating issues like hallucinations.
Semantic Router
Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow LLM generations to make tool-use decisions, we use the magic of semantic vector space to make those decisions — routing our requests using semantic meaning.
This is especially useful for function calls or executing specific actions, where the LLM is not needed to generate a response.
TaiPy
Taipy is an open-source Python library for easy, end-to-end application development,
featuring what-if analyses, smart pipeline execution, built-in scheduling, and deployment tools.
