UIST’24 Preprint Collection

Looking for current research on HCI + AI? Here’s a list.

Here’s a collection of UIST’24 preprints, in no particular order. Found via arXiv, Twitter and Mastodon. Did we miss a paper? Have other interesting work you’d like to share? Please reach out to me! (mastodon, twitter)

Textual: Text, Documents, Code

• Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition (arxiv)
• NotePlayer: Engaging Jupyter Notebooks for Dynamic Presentation of Analytical Processes (arxiv)
• Clarify: Improving Model Robustness With Natural Language Corrections (arxiv)
• Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualization (arxiv)
• SQLucid: Grounding Natural Language Database Queries with Interactive Explanations (arxiv)
• Qlarify: Recursively Expandable Abstracts for Directed Information Retrieval over Scientific Papers (arxiv)
• Story-Driven: Real-time Context-Synchronized Storytelling in Mobile Environments (pdf)
• “The Data Says Otherwise”-Towards Automated Fact-checking and Communication of Data Claims (arxiv)

Visual: Images, Videos, VR, AR

• JetUnit: Rendering Diverse Force Feedback in Virtual Reality Using Water Jets (arxiv)
• Feminist Interaction Techniques: Deterring Non-Consensual Screenshots with Interaction Techniques (arxiv)
• DrawTalking: Building Interactive Worlds by Sketching and Speaking (arxiv)
• VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas (arxiv)
• VRCopilot: Authoring 3D Layouts with Generative AI Models in VR (arxiv)
• Auptimize: Optimal Placement of Spatial Audio Cues for Extended Reality (arxiv)
• WorldScribe: Towards Context-Aware Live Visual Descriptions (arxiv)
• Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams (arxiv)
• Desk2Desk: Optimization-based Mixed Reality Workspace Integration for Remote Side-by-side Collaboration (arxiv)
• SonifyAR: Context-Aware Sound Generation in Augmented Reality (arxiv)
• SonoHaptics: An Audio-Haptic Cursor for Gaze-Based Object Selection in XR (arxiv)
• Augmented Object Intelligence with XR-Objects (arxiv)

Physical: Hardware, Sensors, Fabrication

• IRIS: Wireless Ring for Vision-based Smart Home Interaction (arxiv)
• Degrade to Function: Towards Eco-friendly Morphing Devices that Function Through Programmed Sequential Degradation (arxiv)
• Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interaction (arxiv)
• Demonstration of Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interaction (arxiv)
• SHAPE-IT: Exploring Text-to-Shape-Display for Generative Shape-Changing Behaviors with LLMs (arxiv)

Agentic: Chatbots, Robots, Agents

• ComPeer: A Generative Conversational Agent for Proactive Peer Support (arxiv)
• PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch (arxiv)
• EVE: Enabling Anyone to Train Robots using Augmented Reality (arxiv)
• WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization (arxiv)
• CARDinality: Interactive Card-shaped Robots with Locomotion and Haptics using Vibration (arxiv)
• EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning (arxiv)

Authoring & End-User Development

• LessonPlanner: Assisting Novice Teachers to Prepare Pedagogy-Driven Lesson Plans with Large Language Models (pdf)
• ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Programming (arxiv)
• GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning (arxiv)

UI Design and Evaluation

• DesignChecker: Visual Design Support for Blind and Low Vision Web Developers (arxiv)
• LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation (arxiv)
• UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset (arxiv)