Course Overview
Interface Lab
Course Description
This production course will survey alternative interfaces with an emphasis towards embodied interactions. Incorporating aspects of physical and tangible computing, students will be exposed to the internal machinations of systems, networks and sensors that underlie these interfaces. Areas covered include microcontrollers, connected devices, computer vision, virtual reality and augmented reality amongst others.
Course Objectives
At the completion of this course students will be able to demonstrate basic understanding of:
- Design concepts that incorporate software and hardware elements in interfaces
- Unity 2D and 3D scene design and asset management
- C# scripting to connect variables among different objects
- Physical computing elements such as sensors and actuators
- Arduino basic syntax to manipulate variables
- Communication concepts
- Electrical diagrams to connect components in a breadboard
Course Outline
Week 1: The Basics
Prereqs:
- Download and join the [Discord] and say hello.
- Download and Install Unity Hub and the latest release of Unity (2019.4 as of the date of writing).
- Download and install the latest release of Arduino IDE (Link).
- Get familiar with IOT nano 33 (Link).
Self-Paced Activities:
- Unity Tutorial Series [videos]: The Very Basics.
- Arduino Tutorial Series: Building a circuit.
- Arduino Tutorial Series: Blinking an LED.
Live Session:
- ‘Game’ Engines as the Future of Interactive Media Interfaces.
- Imagine: A Digital Interface to Future Things.
- Hello PComp & Paper Prototyping.
Readings & Videos [optional]:
- [Video] Zach Lieberman: Interactive Art
- [Video] The secret to the incredible landscapes of “The Mandalorian”
- [Video] Massimo Banzi: How Arduino is open-sourcing imagination
- ITP Electronics Lab
Project: Virtual Sculpture
Using the freeform example in the video as a guide, create a virtual sculpture — you can choose to have the sculpture dynamic (e.g., when pressing play things happen) or static. Make use of at least 2 different primitive 3D shapes, 2 materials and 2 lights.
Take a screenshot of your scene, or if you have a dynamic scene, record a video (using OBS or Quicktime) and upload it to the class Discord.
Project: A Physical Interface
Using basic prototyping tools, including the Arduino board, build a physical device that is a representation of the imaginary interface we designed in class. The interface should have LEDs, potentiometers, and push-buttons. Don’t worry about making it actually function! Instead, focus on the affordances that the interface provides. Record and upload a short video to the class Discord demonstrating your creation by the next live session.
Week 2: Sensing and Interaction
Self-Paced Activities:
- Unity Tutorial Series [videos]: Input and Scripting
- Arduino Tutorial Series: Sensors, Variables & Conditionals
- Arduino Tutorial Series: Connecting to Unity
Live Session:
- Demo: Future Interfaces
- Serial Communication in Unity: Ardity
- Unity + Arduino Mashups
Readings & Videos [optional]:
- Art+Science NOW. Chapter 6 “New Interfaces” by Steven Wilson
- Söke Dinkla, “The History of the Interface in Interactive Art” (1994)
- Crawford, Chris. The art of interactive design (Chapter 13) — NYU Library code QA76.76.I59 C73 2003
- Programming Terms and Programming Environments
Project: One Button Interface
Using the work of Week 1 as a source of inspiration, connect a button to an Arduino as the sole input you have to control one or more object(s) in your scene (this can be a shape, a light, a material). Your goal can be as simple as changing the position of a 3D shape, or as complex as changing the sky from day to night. What’s important is that you are able to receive input from the physical world (Arduino) and do something with that in the virtual world (Unity)
Think about creative ways to respond to different ways the button can be used (pressed, released, held, etc…) as well as what constitutes a button!
Week 3: Feedback
Self-Paced Activities:
- Unity Tutorial Series [videos]: A simple game
- Arduino Tutorial Series: Working with many sensors
Live Session:
- Let’s play: One Button Interfaces
- Sensor fusion from Arduino to Unity
Readings:
- Tom Igoe & Dan O’Sullivan, Physical Computing: Introduction
- A Brief Rant on the Future of Interaction Design by Bret Victor
- Responses: A Brief Rant on the Future of Interaction Design by Bret Victor
- Making Interactive Art: Set the Stage, Then Shut Up and Listen by Tom Igoe
- The Art of Interactivity by Margaret Morse
Project: Timely feedback
Build upon the simple game we created in the video series, only this time, incorporate feedback to let the player know something has happened. This should be non-screen-based feedback using one or more LEDs. For example, you might want to light up a red LED if the player hits a wall, or perhaps you want to blink the LED each time the player hits a brick. You can also incorporate other sensors if you’d like!
Week 4: Audio and More Feedback
Self-Paced Activities:
- Unity Tutorial Series [videos]: Audio
- Arduino Tutorial Series: Working with Servos
- Arduino-Unity Tutorial Series: Bidirectional Communication
Live Session:
- Let’s play: Timely Feedback
- Working with Servos
- Introduction to Actuators
Readings & Videos [optional]:
- Arduino’s Servos Library: Optional Commands — Makezine
- How to Make Almost Anything: The Digital Fabrication Revolution by Neil Gershenfeld
- Haptic Techniques for Media Control
Project: Mixed Reality or the Virtual-Physical Continuum
In this week’s project, you’ll be challenged to incorporate the servo motor as an additional feedback device for the player. For example, you might design and build a small cardboard box, which opens whenever the player hits a special brick, or perhaps a fan that speeds up as the player hits more bricks. The goal is for you to exercise your imagination and discover more of the potential to marry real and digital feedback.
Week 5: Optimization
Self-Paced Activities:
- Unity Tutorial Series [videos]: Rendering and Making it Look Awesome
- Arduino Tutorial Series: Optimizing execution times
Readings & Videos [optional]:
Live Session:
- Imagine: Digital Physical Wonderland
Project: Digital Physical Wonderland
For your final project, you have two options:
- Return to your digital sculpture that you created in the first two weeks, and, using the video tutorial as a guide, upgrade your project to incorporate more rendering concepts. Include at least one input from the Arduino.
- Use the new rendering concepts from this week to continue with your game. Add an additional feedback loop for the player that incorporates an additional sensor. This sensor should control some aspect of the game experience for the player, making it easier or harder to accomplish the goal of breaking a brick. For example, you might choose to have the light sensor underneath the buttons that move the paddle control and when covered the game world becomes dark.
