Augment Your Reality: Sketching with Machine Learning to Storytell
Experimental Camera | AR Decentralized World Building Camera | 2023Experimental Camera | AR Decentralized World Building Camera V.1 | Machine Learning on Web2023
⭐️ This is an AR world-building camera based on users’ drawings. The goal is to -
reverse the existing trend of bringing the physical world to the digital.
This camera brings the digital world’s abilities to the physical world in creating a third realm, where the existing can be altered and reimagined through the unlimited information provided by the digital world.
🔬 This Experimental Camera is an AR world-building tool that allows users to draw objects in their physical space using gesture recognition, which are then identified by a Teachable Machine. Once recognized, the camera pulls related sentences from the Google Books API, enabling users to generate a personalized story based on their drawings. This project blends physical and digital spaces, empowering users to interactively build narratives in an augmented reality environment.
📚 This is “Experimental Camera” assignment from Foundation Critical Computation Class during Sophomore year at Parsons School of Design under Design and Technology major.
Final Product
The project has 3 parts:
- Concept Animation
- Experimental Camera Version 1 (2D)
- Experimental Camera Version 2 (3D Point cloud)
One is to draw in 2D with stroke and paint, and another one is to use body to draw in 3D volumetrically. To learn more about the concept animation:
#0 Concept Animation
This page focuses on version 1, which is sketch 1 below.**Sketches might take up to 30 to one minute to load in the beginning for the handpose, object detection, and 3D camera libraries.**
#1 SKETCH (P5.JS Link)
Painting Effect + Teachable Machine + Google Books API
#2 Sketch: (GitHub & Final Website)
Painting Effect + Teachable Machine + Google Books API + WebGL
Function
How to use this camera:
1. once the sketch is running, click “Start Webcam” to activate the painting function
2. Draw in the camera what you want in the place, right now, only “apple”, “banana”, “strawberry”, and “stars” work, with your fingers what you desire to have in the space. 👌🏻 do this gesture with your hand to create the paint brush. The paint brush color are shown on the tip of your fingers. To erase, use your left pinky finger.
3. While you are drawing the teachable machine will try to identify what you draw and show the label on the bottom of the top camera. Once you see the thing you want on the label, click “genereate storyline”. It will go into the google books api to find sentences that include the object you draw.
4. To slect the desire storyline, click on the storyline buttons. It will print the line on the bottom of the camera. — this part is still work in progress.
5. In the end, you will have at least a paragraph of a story about the things you draw in your environment.
Description
This is an AR world-building camera based on users’ drawings. The goal is to reverse the existing trend of bringing the physical world to the digital. This camera brings the digital world’s abilities to the physical world in creating a third realm, where the existing can be altered and reimagined through the unlimited information provided by the digital world.
Basically, it works this way:
1. The user will draw things on the camera as what they wish to have in the existing space.
2. The camera will use a teachable machine to read the drawings and define what is drawn.
3. Once it knows what is drawn, it will search in the google books API and find sentences that include the object’s name. Then the camera will display two sentences as options of what the user wants to use in their story.
4. In the end, you will have your drawings placed as AR objects in your existing space in the camera and a story generated and built upon your decisions of picking the relating sentences of your drawn imagined object of the space. You will have a new world and story that goes along with it.
Design Process
In this project, I used a gesture painting open source called “handsfree.js” created by Oz Ramos He started this project in order to help a friend while he was homeless. It was an absolutely amazing project considering that he has nothing and was struggling to sustain himself with food, a place to live, and money. All he has was a computer that some engineers from Google visited him at the shelter and bought for him. He recently achieved the project on Github because he became homeless again due to many incidents in his life. This was really sad and heavy to me. Therefore, I wanted to make good use of the amazing handsfree technology he has created and left on the internet.
In my studio class, we are learning about decentralized storytelling, where the reader contributes to the creation of the story. In order to push the story forward, the reader will be constantly asked to make decisions for the main character or the world in the story. Every choice affects other factors in the world; therefore, they all lead to different endings at the end. Expanding from the decentralized storytelling concept, I integrated it with the world-building skills I learned from my lab class. In my lab class, we learned a lot about 3D modeling and interaction between users. One of my biggest takeaways is the detailed, personalized, and randomized generative quality of the user services that make it the product good. With these in mind, I came up with this camera. Like the following diagram:
This was the original idea and was changed in the end.
Challenges
As usual, I faced a lot of challenges with my project. But this time, I went for help. I had three DT study appointments in total to make this camera. I find it extremely helpful, but the appointments are just too short for me to completely troubleshoot all the issues I have. During the process, I was mostly stuck on two things:
**I solved this problem on 3/2/2022**
1. My teachable machine is not accurate in reading the drawing. It is probably reading the camera instead of the drawing. In the drawing I provided, I was thinking to have another canvas with a blank background and mirroring the drawing the user did, so it can read the drawing separately from the camera.
**I solved this problem on 3/2/2022**
2. I couldn’t pull single sentences out from the google books api. I was trying to do google books search like this example but in order to only fetch single sentences out from the description of the book information from API, I have to use “substring” to look for periods to define the beginning of the sentence and the end of the sentences. Below is the working file for pulling sentence data out from google books api.
3. Since the camera is constantly drawing over the canvas, I couldn’t find a place to store the story lines the user picked at the end.
//to search for sentenes contain keywords, change the keyword variable on line 34.
let titles = []; //global array of titles from NYT Books API
let done = false;
let descriptions = []; //global array of descriptions
let dict = {}; //dictionary of title + desc
let matchingSentences;
let start;
let redo;
let option1;
function apiRequest(url) {
// reference: https://developers.google.com/books/docs/v1/using#PerformingSearch
let http = new XMLHttpRequest();
http.open("GET", url, false);
http.send(null);
return http.responseText;
}
function gotJSON(data) {
//parse GOOGLE BOOKS API for description
let array = JSON.parse(data).items[0];
let info = array.volumeInfo;
// let title = info.title;
let sentence = info.description;
return sentence;
}
function setup() {
createCanvas(320, 260);
let keyword = "allison";
matchingSentences = lookForWord(keyword); //an array of sentences that contains keyword
start = createButton("start");
start.position(0, 0);
start.mousePressed(optionButton);
}
function draw() {
// background(220);
textSize(10);
// text(matchingSentences[0], 10, 50, width - 10, height);
// text(matchingSentences[1], 10, 80, width - 10, height);
redo = createButton("redo");
redo.position(300, 300);
redo.mousePressed(setup);
}
//write the story down accroding to which sentence the user chose
function story1() {
let sentence1 = matchingSentences[0];
background("white");
let initalHeight = height-100;
text(sentence1, 0, initalHeight , width - 10, height);
}
function story2() {
let sentence2 = matchingSentences[1];
background("white");
let initalHeight = height-100;
text(sentence2, 0, initalHeight , width - 10, height);
}
//create sentnce option buttons
function optionButton() {
let sentence1 = matchingSentences[0];
option1 = createButton(sentence1);
option1.position(0, 50);
option1.mousePressed(story1);
let sentence2 = matchingSentences[1];
sentence2 = createButton(sentence2);
sentence2.position(0, 100);
sentence2.mousePressed(story2);
}
function searchBooks(keyword) {
let url =
"https://www.googleapis.com/books/v1/volumes?q=" + "intitle:" + keyword; //google books api, look for books with the keyword in title
return gotJSON(apiRequest(url)); // searchBooks(keyword) = gotJSON(apiRequest(url)), i.e book description
}
function lookForWord(keyword) {
let sentences = searchBooks(keyword).match(/\(?[^\.\?\!]+[\.!\?]\)?/g); //get an array of sentences from the book description
//the cunk of punctuations above is called regular expression, reference: https://stackoverflow.com/questions/11761563/javascript-regexp-for-splitting-text-into-sentences-and-keeping-the-delimiter;
let matches = sentences.filter((s) => {
return s.toLowerCase().includes(keyword); // filter the array with only sentences with the keyword
});
if (matches === null) {
//if no sentence has the keyword
console.log("No results");
return;
}
console.log(matches); // print matching sentences
return matches;
}4. The camera right now, only recognize four things: “apple”, “banana”, “strawberry”, and “stars” because I don’t want to overwork the teachable machine with too many samples. However the initial intention was to have the teachable machine to have as many samples to recognize from, so it has a response/guess for whatever you draw. Since it works on these four fruits, it will work on more than four sample/class, if I add more.
Reflection
I am a little disappointed that I wasn’t able to complete everything because I thought I should have most of the skills I need to make this happen so far. However, there are still many blind spots that I didn’t see. Due to the workload, I have this semester, this might be the best I can do for this week. BUT! I am still passionate about making this happen. I wish I can implement this concept and technology that I have figured out in my future projects.
Credits
Handsfree.js
2D Hand Tracking https://handsfree.js.org/ref/model/hands.html#usage by Oz Ramos
