The evolution towards creative computing

from creating solutions to a problem … to solving a creative problem

Computing is the attempt to solve real life problems by designing a set of instructions following an algorithm that can be executed by an electronic device capable of processing the instructions set by a programmer. Computing has been used mainly to develop a solution to a problem, and computer programmers are highly skilled in finding creative ways on how to solve everyday problems.

The Electronic Numerical Integrator and Calculator (ENIAC) was commissioned at the University of Pennsylvania in 1945 and one hour of processing was displacing 2,400 human hours.

As computers developed in processing power and speed, the aspirations of mankind grew exponentially, and computing is evolving towards finding creative solutions to old problems with new algorithms that make use of the new era of computing, such as artificial intelligence and evolutionary algorithms. Furthermore, humans are using computing as a solution to a creative problem, in other words using innovative algorithms to tackle challenges offered by new technologies such as smart phones and virtual reality in the creative industry.

Dylan Seychell is an academic at Saint Martin’s Institute lecturing in User Experience and with other members of academic staff and students researches the use of creative computing in cultural heritage.

The Creative Industry

Beauty lies in the eye of the beholder

The creative industry, comprising the economic activities centered around entertainment and education, is focused on the human user. It is all about the user experience through perception: sensory information interpreted through the consumer’s past experiences.

As part of this industry, Creative Computing allows designers and developers to create interactive artifacts targeting the expectations of the human consumer.

Delivering Realism to Satisfy the Senses

The senses provide humans a sense of perception

To meet these expectations, we must interface with the human mind through its senses : sight, sound, touch, smell and taste. Creative computing engineers are occupied in building systems that attempt to feed each of these senses with information that provides the right perception to make a realistic experience

The Sistine Chapel, painted mostly by Michelangelo takes the visitor through the old and new testaments culminating in the last judgement. More than 500 years ago, humans already made use of the technology available to develop an environment that touches the senses of the visitor and provides surrealism of what cannot be experienced in reality.

Challenges for Creative Computing

Can we make a representation look as real as the represented?

Can movement be realistic?

Can lighting be realistic?

Can the aural experience be realistic?

Can we fake reality believably?

Once we believably simulate reality we can build Virtual Reality that provides a totally immersive experience and we can modify perception of Reality through Augmented Reality…


How Creative Computing is resolving some of the challenges by targeting the Senses

Vision

Targeting vision with 3-dimensional images, Augmented Reality and Virtual Reality

Image Analysis — Image analysis, such as recognizing 3-dimensional shapes from 2-dimensional images, is needed by technology that aims to automate reactionary systems, such as robotics, as well as those aiming to represent live real scenarios in virtual space such as remote surgery systems.

An infinite number of potential reconstructions are possible from a simple sketch Image Source: Johnson, Chang and Lipson (2010)

Rendering of Light and Shadows — The rendering of illumination in virtual scenes based on virtual light sources has been the subject of much research in computer imaging. Challenges in this area include realistic modelling of partially transparent surfaces andhistorically correct illumination of buried ruins and efficient realtime representation of a multiple sources of light.

Animation of Objects and Environments-Realistic representation of any 3D landscape needs to consider the non-static nature of all objects within field of view: wind effect on light fabric, twitching in old people, fauna, weak architecture bending under heavy load… animation gives life to static objects, in this case virtual objects represented by a lot of coloured pixels. Between key-framed flat animation and motion capture techniques, what new ways can we devise for efficient, realistic animation?


Audio

Targeting hearing for Situational Realism

Sound Effect Synthesis — Recording real world sounds has its challenges such as the lack of control on the sound’s properties, the acoustics of its environment, and other background sounds during capture. Audio Engineers thus design parameterised sound effect generators that gives them such control — and here is where the challenge shifts onto making the synthesised effect as realistic as possible. When synthesising sound effects of fictional objects — and thus no one really know what it sounds like — the emphasis there is to be believable: that such a sound would be justifiable given the sound source’s nature and the properties of its environment. This is a field which is becoming even more in demand due to its role in Virtual Reality as it accompanies the head mounted visual displays.

Music had a role to play even in silent films: musical scripts were delivered to cinema houses for the pianist to play in accompaniment to the movie.

Music Analysis and Composition — Music has a very important role in building a user experience: either by setting the mood when it takes a background role, or a narrative device when used to voice the non-verbal dialogue between characters. Computaitonal power allows for efficient analysis of large musical corpora and, based on such analysis, assist composers in the generation of, compositions that are similar in style or genre. These would be necessary in immersing the player into the virtual world.

Speech Synthesis and Recognition- The ubiquitous nature of the mobile devices demands an interface that favours hands-free communication. Speech is used incessantly between humans in both face-to-face and mediated forms. Speech synthesis has been used in situations where visual attention is not affordable or limited — such as GPS navigation — or as assistive technology for visually or vocally impaired persons such as the late Prof Stephen Hawking. What remains just out of grasp are aspects of speech that makes virtual characters believable: the vocal inflections, the expression, and the emotional tinge that gives meaning to speech.

Acoustics Modelling- It is well and good to have 3D laser scans of real buildings used to model the virtual representation of that space such that any illumination and reflections are true to the real space — but what about aural fidelity? The effect of the environment on the sound’s property should reflect the process that such a sound would undergo in that space in reality. As the head-mounted visual devices occupy the wearer’s full field of vision, immersing them totally within the virtual space, attention is now drawn to the aural dimension that reflects the tele-presence of the wearer in the virtual world by providing sound that mimics the acoustic properties of the modeled environment.

Top-end research in the field considers real-time acoustic modelling in complex environments at par with raytracing in visual systems.

Haptics and skin-sense

Targeting touch to make the virtual, tangible

The sense of touch has so far manifested itself in wearable gloves that provide sensory information to the fingers allowing the virtual reality visitor to “touch” the intangible objects depicted in the head-mounted screen and heard through the earphones. But the skin-sense allows us to sense the environment around us beyond touch: feeling wind, sensing temperature, burning sensations from abrasions and wounds etc. Making the virtual experience more realistic will mean stimulating these sensations whilst keeping them harmless and safe.

Haptics allow sense of touch in virtual reality

Smell and Taste

Targeting smell for a heady experience
Targeting taste for a savory experience

Since 1906 several attempts were made in various cinema theatres to augment movies with smells, but all involved broadcast: scent is emitted into an enclosed space for all to smell, but lack of control on air flow and lingering odours, they never were a success. A solution was to go for personalized scent and this was minimally achieved using scratch-cards which, however, interrupted the film experience and broke immersions. Now, as VR is a personalized activity, complete with headphones and gloves, an opportunity arises to present personalized smell experiences to heighten the immersion — and we are literally not far away from extending that to taste!


Motivations for Creative Computing

What are the objectives that motivate Creative Computing scientists to face and surmount these challenges?

Communication

Situating remote speakers in a co-located virtual environment can help immerse the communication into a non-distractive scenario that can actually support the discussion with situational imagery and allow full-body non-verbal communication especially in mixed-reality scenarios where the real person is projected into the virtual environment. Kind of Second Life but with real people representations.

This concept has been taken up by news stations, especially sports rooms. What if this became the norm when calling your colleague at work with workflow charts, live data, and website mock-ups surrounding you or telling your mum about your family holiday while photos of the places you visited occupy the space around you?

Commercial Benefit

Fast food chains are slowly integrating technology into their customer’s experience: from child-friendly tablets offering the menu and entertainment to self-ordering kiosks to save you time at the ordering queue. But eating out is also a social activity: meeting friends at the local eatery is a commonplace and healthy activity, but limited to friends within your vicinity. What if augmented reality would allow your faraway friends to meet you for dinner by going down to their local outlet and sitting at the table “next” to you (actually a screen projecting their table in their outlet while they’re seeing your projected table at the screen beside their table). Thus, mixed reality can provide a commercial advantage in selling a realistic experience, rather than just a product.

Entertainment

Entertainment is at the forefront of taking advantage of technology. Games like Ingress use a handheld screen device to portray the virtual world onto the real world.

Ingress takes the game into the real world

Other approaches include projecting the virtual world onto your living room walls (works best if there’s no furniture of course!), an idea that was proposed by Microsoft but taken to new heights by an Australian company called Euclideon who are re-inventing computer graphics. Will casual daily entertainment cross the screen into our world?

Euclideon’s wall-projected holograms for immersive entertainment

Education

Mixing realities can provide a show don’t tell approach to education that takes the history student into the thick of the action, the geography student in the midst of the earthquake, the language student with a personal virtual tutor immersed in the country’s culture, the entrepreneurial student with a virtual company of employees to manage to success.

Embedding visitors into the exhibits via projected self into mixed reality
Holoverse: the most futuristic theme park in Australia from Euclideon using atoms rather than polygon graphics

Training

Situating the trainee in realistic situations to provide hands-on training in a controlled environment allows room for failure with no danger to the trainee or trainer or target. From skills, to performance, to timed operations, virtual reality can provide a safe environment within which real-life scenarios are presented to train the user in learning new skills, how to handle situation, the correct order for a procedure and how to prioritise tasks.

Training Engineers through VR

A binaural recording of a live orchestra can place you in the midst of a real performing orchestra, but what if you are a budding musician and want to rehearse with an orchestra? Will you pay for a real orchestra to accompany you? Can you fit them all in your room or do you need to rent a hall? And where will you fit that large double bass? What if you immerse yourself in a virtual orchestra, that stops and repeats for you as you make mistakes and re-attempt the music?

A virtual orchestra performing… but what if it could let you rehearse with it?

Emotional Closure

Virtual reality can provide an alternative ending to emotionally charged situations that may have traumatised its participants who may need closure in order to move on. Such alternative endings may explain why such things happened, and whether they could have done anything to change the situation and run of events. Sometimes talking to other survivors also helps, but these survivors only live for so long, so projects are trying to capture their experiences and share it with today’s and future generations, such as the New Dimensions in Testimony project (See video below).

The New Dimensions in Testimony project captures the experience of a Holocaust Survivor and presents it in an interactive dialogue format

Therapy

Phobias are common phenomena where people fear objects or situations which inhibit their normal day-to-day routines. Treating phobic patients with exposure to their fears has been found to help them face and overcome their fears: be it fear of heights, fear of spiders or snakes, fear of public speaking and many others. Exposure to such phobias in a safe environment is possible by a gradual exposure to virtual reality followed up by augmented reality to situate the object of fear within familiar environments.

Using VR headsets to cure phobias

VR can also help recovering patients in re-engaging with their daily life with virtual environments while still in hospital care, such as in recovering from a stroke. Thus, from within the limited environment of a hospital, patients can start practising facing their normal routine under supervision and with help ready at hand.

Using VR for stroke rehabilitation

Join the Global Creative Computing Community

Creative Computing is the effort to provide society a better future by taking advantage of modern and future technology to provide means of solving problems of today and tomorrow using algorithmic skills leveraged by mobile intelligent systems.

If you are interested in following a career in creative computing, then you are at the right place. Saint Martin’s Institute of Higher Education offers a wide range of MQF level 5 diplomas that are open to students who are 16 years and over and can present at least four ordinary levels, including Mathematics and English.

The MQF level 5 diploma programmes, read as a one year full time or two year evening study, that lead to this field of creative computing are;

Diploma in Computing with Digital Marketing & eBusiness;

Diploma in Computing with Games Design & Development;

Diploma in Computing with User Experience;

Diploma in Computing with Web Design & Development.

These MQF level 5 diploma lead either to an MQF level 6 BSc (Hons) degree in the same specialisations awarded by Saint Martin’s Institute of Higher Education or may be used as an entry to the BSc (Hons) Creative Computing conferred by the University of London under the academic direction of Goldsmiths College.

Saint Martin’s Institute of Higher Education, established in 1985, is licensed by the NCFHE with license #196 ● Postal Address: Saint Martin’s Institute Foundation Building, 2, Schembri Street, Hamrun HMR 1541 ● Telephone: +356 2123 5451 ● eMail: infodesk@stmartins.edu

Learn More

Join the Games Development Challenge — Click Here

The Successful Games Studio for 2020 — Click Here

Computing @ SMI #NotJustANumber — Click Here

#StartMyInspiration — Click Here