Market Digest — SIGGRAPH, AI, AR/VR news

WEEK OF JULY 29 — Meta announces Llama 3.1 and AI Agents that mimic creators’ personalities, OpenAI Launches SearchGPT, Microsoft files AR glass patent with Copilot GenAI integration.

By Joana Wong and Tejaswini Kumar — Edition 75 (Edition 2 for public consumption)

💥 Welcome to the Market Digest, where we bring you the hottest weekly headlines in AI and AR/VR, covering everything from news to patents.

We’re Joana Wong and Tejaswini Kumar. Joana is a Product Marketing Manager and Product Planner, while Tejaswini is a Product Manager with a background in Data Science. We are currently seeking our next opportunities in the tech industry.

The Headlines

🧠 AI News

1. NVIDIA x Meta’s Session Highlights at SIGGRAPH: Emphasis on open-source, AI breakthroughs and the next computing platforms.
2. Meta announces SAM 2 for “real-time, promptable object segmentation in images & videos.”
3. OpenAI Launches SearchGPT, an AI-Powered Search Engine Prototype.
4. Introducing Meta Llama 3.1 — expands context length to 128K and includes Llama 3.1 405B.
5. “Faulty NVIDIA H100 GPUs and HBM3 memory caused half of the failures” during Llama 3 training.
6. NVIDIA developers can build “highly accurate, AI enabled virtual worlds to build the next wave of physical AI and robots.” Ad agencies will also be able to create generative 3D worlds.

👓 Headset News

1. While HoloLens 3 is uncertain, Microsoft files patents for AR glasses with Copilot GenAI integration.
2. Vivo plans to release its first MR headset in 2025 and has partnered with Rokid to make the X100 Ultra capable of 3D photography.

🤖 AR Tech News

1. Optical waveguides can be mass produced for AR display applications by using a modified 3D printing approach
2. Meta files patent for smartglass eye tracking and gaze direction and gaze prediction.
3. Google files three patents to (1) predict the fit of a wearable device (2) Address display and gaze tracking system misalignment and (3) Improve color uniformity and reduce rainbow artifacts.

The Deep Dive

A summary of each headline

AI News 🧠

1. NVIDIA x Meta’s Session Highlights at SIGGRAPH: Emphasis on open-source, AI breakthroughs and the next computing platforms. (Meta)

• Open-Source Philosophy: Zuckerberg emphasized repeatedly that open-source is the way to go. Meta continues its open-source contributions, including the PyTorch framework, the Llama LLM, and advancements in AR/VR tech, like hand-tracking.
• Meta’s AI Studio can help you build a personalized “clone-type” AI agent: Creators can train AI agents and build personalized AI on their material, allowing the AI to communicate authentically, emulating how the creator would respond, imitating their personality and speech patterns. AI studio offers: (1) AI characters: Ideal for entertainment, providing practical uses such as cooking tips and daily affirmations. (2) Creator AIs — Assist creators in scaling their interactions by responding to common DMs and story replies.
• Computing Platforms According to Zuckerberg: An ecosystem of wearable XR devices with two main categories: (1) VR/MR (passthrough) Headsets: workstations with significant compute power and (2) AI Glasses Strategy: Meta is focusing on different Product Lines with a range of price points — → (2.1) Without Displays: Affordable models priced under $300, such as Meta’s Ray-Ban glasses developed with EssilorLuxottica, featuring cameras, microphones, and multimedia capabilities. — → (2.2) With High-End Holographic Displays: Premium glasses offering advanced AI and display technologies.

2. Meta Introduces the “Segment Anything Model 2” (SAM 2) for real-time, promptable object segmentation in images & videos. The outputs of SAM 2 can be used to “create new video effects and unlock new creative applications” or to “aid in faster annotation tools for visual data to build better computer vision systems.” Furthermore, SAM can be used in STEM research — for example, to track and segment “moving cells in videos captured from a microscope.” (Meta)

Meta SAM 2 can be used to build better computer vision. Image from Meta.

3. Developers can build AI enabled “virtual worlds to build the next wave of physical AI and robots” on NVIDIA Omniverse. Also, the WPP ad conglomerate will test and utilize the new NVIDIA NIM microservices to build virtual environments and landscapes from prompts like, “build me a table with tacos.” (WPP, NVIDIA).

Ask NVIDIA, “build me a table with tacos.” And now we’re all hungry.

4. OpenAI Launches SearchGPT, an AI-Powered Search Engine Prototype. (FastCompany)

• OpenAI is alpha testing an AI search tool that provides direct answers from web sources and plans to integrate it with the ChatGPT app.
• The company has partnered with publishers like News Corp. and The Atlantic, and the AI search tool will include citations and links to original sources.
• Despite innovative features and potential to challenge Google’s dominance, OpenAI faces challenges in maintaining a world-class search platform.

5. Introducing Meta Llama 3.1 Meta’s latest Llama 3 Herd of Models expands context length to 128K and includes Llama 3.1 405B — “the first frontier-level open source AI model.” Llama 3.1 405B has “Our new model will …unlock new workflows, such as synthetic data generation and model distillation.” In the announcement, Meta again made a distinction of them as being open source and competition being closed source. Meta also boasted its ecosystem of 25+ partners (AWS, NVIDIA, Groq, Azure, Google Cloud, and more). (Meta)

6. “Faulty NVIDIA H100 GPUs and HBM3 memory caused half of failures” during LLama 3 training — one failure every three hours for Meta’s 16,384 GPU training cluster.
Over the 54-day training run, the cluster experienced 419 unexpected component failures. GPUs or their onboard HBM3 memory were responsible for approximately half of these failures. (Tom’s Hardware)

Image of NVIDIA GPU

7. On Monday, Condé Nast (publisher of the New Yorker, Vogue, Wired) Sent a Cease-and-Desist Letter to Perplexity. The letter demanded that Perplexity stop allegedly scraping its content for its genAI responses. (engadget, The Information). This is amidst senators introduced legislation (The COPIED Act) which aims to “Increase Transparency, Combat AI Deepfakes & Put Journalists, Artists & Songwriters Back in Control of Their Content” (Senate.gov)

Headset News 👓

1. While HoloLens 3 is uncertain, Microsoft files patents for AR glasses with Copilot GenAI integration. The “Composite Pose Estimate For Wearable Computing Device” patent describes a device with spatial and kinetic awareness via IMU sensors, without relying on GPS or visual data. The “Resolution Enhancement in Spatial-Frequency Space” patent describes an advanced camera system that improves image resolution using a lamp for lighting, a lenslet array, and an image engine for high-resolution images. (XR Today)

Image from the patent, from Windows Latest

2. Vivo plans to release its first mixed reality headset in 2025. Their next gen MR device is expected to be “more autonomous and offer a more immersive experience similar to Apple Vision Pro.” Vivo is a large player in China, owning 19% share of its smartphone market. (Gagadget). Vivo announced a partnership with Rokid to make the X100 Ultra capable of 3D photography. (PR Newswire)

Image of Vivo at the event of the announcement (Imaging Festival in China), from Gagadget.

AR Tech News 🤖

1. Using a modified 3D printing approach, optical waveguides can be mass produced for AR display applications (AIP.org)
2. Meta files patent for smartglass eye tracking and gaze direction and gaze prediction via fringe projection or time-of-flight analysis. (Patently Apple)

3. Google files three patents:

• Predicting the fit of a wearable device from image data obtained by a computing device together with position and orientation of the computing device. (WIPO)
• Addressing display and gaze tracking system misalignment: a processor and sensor are used to determine a change to the frame based on temperature cycles. It adjusts a parameter of the device in response to exceeding a threshold. (WIPO)
• Improving the uniformity of colors and reduce rainbow artifacts: A view control layer optically coupled to a waveguide having one or more diffraction structures. (WIPO)

📥If you enjoyed this newsletter, please follow this newsletter and subscribe by emailing joana.and.tejaswini@gmail.com to receive it directly in your inbox.

🔈Sharing is caring. Share this article with a friend to help us keep producing newsletters for you.

💬 We’d love to hear from you! Send us your feedback or suggestions to improve our content.