Maybe NYC: The alternative fates of New York City in Mixed Reality

Project by Andrew Ippoliti and Anna Nicanorova

The iconic New York City skyline involuntarily springs up feelings of fatalism. However, it probably can not be a better showcase for indeterminism — a perfect playground of all the possible futures, unrealized ambitions, abandoned aspirations and stark disillusionments. This summer we wanted to explore alternative futures of the city and showcase them in a more intuitive and interactive environment of Mixed Reality. Our goal was to allow users to dynamically explore the parallel possibilities of the city that is and the city that might have been.

The Summer Project

First of all we need to point out that the work was inspired and could not have been executed without “Never Build New York” (Greg and Sam, thank you for your inspirational work!). This summer we got the book, selected a list of buildings and slowly started modeling the buildings that have never been executed. We envisioned a perfectly logical plan with aspirations to finish in just 2 weekends:

* create current rendering of New York City
* remake all unbuild buildings in Blender
* place them onto existing New York city (adding styling, text)
* create voice interaction in Unity and HoloLens (user can say “Maybe” and render New York that could have been or “Now” and render the current version of New York)

Since the most crucial ingredient was the actual model of New York city we naively hoped we could find it online. Our efforts bore no fruit. However, knowing that New York City published an OPEN GML format we thought it would a piece of cake. It took us a whole two days of wrestling with files and we were almost at the point of tears when we finally created the full model of the city. We published it for free public use on Free3D.

After we got our New York City, we enthusiastically started recreating alternative buildings in Blender. We chose a “divide and conquer” approach, meaning that each of us had to design an equal number of buildings. The first building was the most challenging, because we had to re-learn shortcuts in Blender, but we quickly got a hold of our drawing and modeling skills after a few hours of practice and Youtube tutorials.

We then hit yet another roadblock. We wanted the user to interact with the city via gaze and gesture commands. It took several hours of struggling before we fixed the problem. “Let’s try voice commands instead!” It turns out that adding voice commands was easy. We added two voice commands: “now” and “maybe.” They would make the existing buildings collapse and show new buildings. We wanted our user to truly experience an “alternative future” and see how that compares directly to what is there now. Since adding voice commands and animating the buildings was the easiest part of the project, we decided to add some background city-sound ambience and play sounds of the buildings being torn down and rebuilt as the building changed.

The initial reception of the project raised questions why Mixed Reality. We did not want to isolate the viewer from the current reality. We wanted the city to be readily available as a regular object in your environment, like a lamp or an alarm clock. We also did not want “to immerse” — we wanted the feeling of indeterminism projected in the current environment, and hence Mixed Reality was the perfect choice.

All project workflow is located on the GitHub. While the Unity Files are too heavy to host in GitHub, we have published the open-source version of New York. The live experience demo is hosted on YouTube to showcase sound effects and animations.


All projects go through states of inspiration, disillusionment, frustration, but this it’s usually the inspiration and persistence that carries us forward. After 6 weekends of co-working, our present and “imaginary” New York City is can be rendered on a table in a coffee shop or in the hallway of an apartment, gleaming as a visual confirmation that nothing is set in stone.