Building Scribble: Our 1st Prototype
For our first functional prototype, our idea was to have a mobile phone act as a sort of “Drawpad” where the user could Scribble letters. This phone would then connect to an app running on our Gear VR. As a user Scribble’d, the Gear VR app would display the letters in real-time.
In order to make this work, we decided to split it up into 3 components:
On the mobile side, we decided to use React Native so it could work cross-platform across iOS and Android. We used the undocumented API that Google uses for their Handwriting App to parse the user input to actual characters. We also used a web socket, so that every time a user scribbles a letter, it is interpreted by the Google API and then relayed to our web socket, which then relays it to our Unity app.
Our Oculus app receives the text input from our web socket, and transforms it into text right in front of the users’ own eyes.
