Curious Resources: Volume 1
Interesting finds in interactive technologies — covering the Web, Unity, Mixed Reality, Deep Learning, JavaScript, C#, 3D, Audio, Art & more
This week we’re covering:
- Spatial Audio (Ambisonics) for 360/ VR / AR / XR
- GraphQL For Faster Front-End Iterations
- Live Streaming 360 Video to YouTube From Inside of Unity
- Assorted Deep Learning Finds: Poetry, Audio, Hi-Res Images, Unity Agents
Hi! I’m starting a series of posts that will collect some of the interesting repositories and articles that I’ve come across lately. I’ll also dig up references from my ever growing toolbelt of Github stars. I love research, exploring patterns, making connections, and creating new possibilities.
I own a company that specializes in experiences — get in touch with me on LinkedIn!
Spatial Audio (Ambisonics) for 360 / VR / AR / XR
I’ve been thinking of applying to Google’s “Jump Start” program to get my hands on their new YI HALO 360 camera rigs for a series of creative shoots involving magical creatures in abandoned places. One of the questions on the application asks about spatial audio aka ambisonics, which remains somewhat unexplored compared to reactive visual systems. I went back through some of my finds:
Google has a spatial audio SDK for ambisonics called Resonance
https://developers.google.com/resonance-audio/
The Unity package is on the Releases page here
https://github.com/resonance-audio/resonance-audio-unity-sdk
Web devs can use this library
https://github.com/resonance-audio/resonance-audio-web-sdk
The engine behind the Web SDK
https://github.com/GoogleChrome/omnitone
For advanced users, you can edit in your favorite DAW as a VST
https://github.com/resonance-audio/resonance-audio-daw-tools
Google’s encoding specs for audio and video
https://github.com/google/spatial-media
Facebook has some slick software, but its source is nowhere to be found
https://facebook360.fb.com/spatial-workstation/
An Ableton-specific ambisonics solution
https://github.com/EnvelopSound/EnvelopForLive
GraphQL For Faster Front-End Iterations
Front-end developers building client applications can sometimes get blocked waiting for back-end API changes to keep up with design.
What if you could just go ahead change that API from the front end? What if you didn’t have to worry about whether a particular endpoint exists — if the data is there, you can construct one yourself. Even better if you could keep that app trim by only getting the data you want when you make a request, instead of a ton of bloat.
To that end, I’ve been considering whether to recommend a GraphQL API instead of a REST API to an e-commerce client. It’s important that they be able to move quickly. While doing my research, I came across some interesting resources:
Starting from this excellent article
https://time2hack.com/2018/02/introduction-quick-guide-to-graphql-for-backend-frontend/
GraphQL query visualizer
https://github.com/graphql/graphiql
GraphQL server for Express
https://github.com/apollographql/apollo-server
Interesting how GraphQL is just an interface on top of whatever database you want to use. For example, you can use it with MongoDB https://www.compose.com/articles/using-graphql-with-mongodb/
Or PostgreSQL
https://github.com/postgraphql/postgraphql
The spec itself, although I’d start with articles and tutorials instead
https://github.com/facebook/graphql
Some front end tools for interacting with GraphQL
https://github.com/apollographql/apollo-client
Specifically for React
https://github.com/apollographql/react-apollo
A whole framework from Facebook that couples React with GraphQL
https://github.com/facebook/relay
Modifying control flow of queries and results
https://github.com/apollographql/apollo-link
Shorthand for writing queries
https://github.com/apollographql/graphql-tag
Live Streaming 360 Video to YouTube From Inside of Unity
I’ve been trying to sort out a pipeline to live stream equirectangular video captured from a camera inside of a Unity scene to YouTube 360. I want viewers to be able to see inside of live game worlds as they’re happening. I’ve found a couple of useful leads:
Here’s one to output using FFMPEG (which could then stream to YouTube)
via the amazing Keijiro
https://github.com/keijiro/FFmpegOut
But at what speed? The new scriptable render pipeline in Unity 2018 might enable a faster solution
https://github.com/keijiro/AsyncCaptureTest
I also evaluated a paid option
https://github.com/SystemFriend/GPUVideoEncoder-Trial
The paid option makes use of this very interesting concurrency library
https://assetstore.unity.com/packages/tools/spicy-pixel-concurrency-kit-3586
Assorted Deep Learning Finds
Generate poetry from images using CNNS and RNNs
https://github.com/rossgoodwin/neuralsnap
Get audio into a Pytorch tensor — more audio Deep Learning projects, please!
https://github.com/pytorch/audio
Speech at super-resolution could lead to clearer communications.
https://github.com/jhetherly/EnglishSpeechUpsampler
Make music alongside a Google AI — a new generation of instruments and live performances may be collaborations with a computer
https://github.com/tensorflow/magenta
Future Alert: with the right kind of sensors, deep learning should enable subvocal speech recognition — the ability to communicate without making sound, which would come in handy in smart gear for divers or firefighters
https://github.com/Mohamed-Elbially/ThesisCodes
Many state-of the art networks have been trained on extremely small images (300px by 300px) due to GPU memory size constraints— however, they have been very accurate despite this limitation. I’m excited about anything trained on high resolution images, since it will help us understand whether smaller training images are sufficient or only scratching the surface of what’s possible. More pixels!
https://github.com/NVIDIA/pix2pixHD
Unity is a great environment to train deep learning agents — especially multi-agent environments.
https://github.com/Unity-Technologies/ml-agents
Here’s a hide/escape example
https://github.com/HappySlice/hide-escape
Here’s one with pirates!
https://github.com/HugoCMU/pirateAI
