AI Top-of-Mind for 3.11.24 — Speculative Decoding
Top-of-Mind is a good optimization technique known as ‘speculative decoding.’ Basically, as Benjamin Marie in ‘Towards Data Science’ explains:
Speculative decoding uses a small LLM to generate the tokens which are then validated, or corrected if needed, by a much better and larger LLM. If the small LLM is accurate enough, speculative decoding can dramatically speed up inference.
It requires model alignment, as depicted in the table below. Looking forward, one could see future availability of other models like Mixtral that would also support this approach.
Four additional items on models. The first by Fabio Matricardi looks at the new (MIT) licensing for Microsoft’s Phi-2 that permits commercial use. Next is Iulia Brezeanu in ‘Towards Data Science’ on prompt compression to reduce Retrieval-Augmented Generation (RAG) costs with a goal of speeding up LLMs while reducing resource requirements. Also on the edge front, Alvaro Fernandez in the ‘Deep Hub’ describes model compression approaches. And then Karl Ostroski in ‘Slalom Data & AI’ taking another look at biases based on data, algorithms, and testing.
Turning to practicality, a good follow-up by Anthony Alcaraz in ‘The Modern Scientist’ on structure-aware AI for documents including DocLLM and DocGraphLLM.
And time for another update on OpenAI’s ‘Sora.’ Frank Lee in ‘Generative AI’ offers up a non-technical explanation as to how it works. Also on the creative front, Ianacio de Gregorio reports on an interesting new tool — Genie — that takes static images and turns them into video game-like motion. If you want to know what a ‘spatiotemporal transformer’ is, read on, and a link to the original paper is here. Further down the rabbit hole, what is the meaning of conceptual art? Cezary Gesikowski in ‘Algography Art’ offers some thoughts.
Parallel to any Apple Gen AI rumors, as penned by Ignacio de Gregorio in ‘Towards AI,’ there are still many features available today on the iPhone that leverage AI as reported by ‘CNET.’ Capabilities include digital voice cloning, ‘Live Text’ to copy text from images, and Portrait Mode. But looking forward, Ignacio looks at MLLM-Guided Image Editing (MGIE) that combines multimodal LLMs and diffusion models.
Thinking back to International Women’s Day, we still have a long way to go, if you remember earlier reporting on Taylor Swift deepfakes. Megan Rashid writing in ‘Women in Technology’ covers some history, new threats due to AI, and most importantly, the lack of an adequate response by technology companies.
Lastly, an innovative use of AI by the Washington state Lottery. ‘AdAge’ offers details. As part of the campaign, the site https://testdriveawin.com/ places your photo in an exotic destination.
