The evolution towards creative computing
from creating solutions to a problem … to solving a creative problem
Computing is the attempt to solve real life problems by designing a set of instructions following an algorithm that can be executed by an electronic device capable of processing the instructions set by a programmer. Computing has been used mainly to develop a solution to a problem, and computer programmers are highly skilled in finding creative ways on how to solve everyday problems.
As computers developed in processing power and speed, the aspirations of mankind grew exponentially, and computing is evolving towards finding creative solutions to old problems with new algorithms that make use of the new era of computing, such as artificial intelligence and evolutionary algorithms. Furthermore, humans are using computing as a solution to a creative problem, in other words using innovative algorithms to tackle challenges offered by new technologies such as smart phones and virtual reality in the creative industry.
The Creative Industry
Beauty lies in the eye of the beholder
The creative industry, comprising the economic activities centered around entertainment and education, is focused on the human user. It is all about the user experience through perception: sensory information interpreted through the consumer’s past experiences.
As part of this industry, Creative Computing allows designers and developers to create interactive artifacts targeting the expectations of the human consumer.
Delivering Realism to Satisfy the Senses
To meet these expectations, we must interface with the human mind through its senses : sight, sound, touch, smell and taste. Creative computing engineers are occupied in building systems that attempt to feed each of these senses with information that provides the right perception to make a realistic experience
Challenges for Creative Computing
Can we make a representation look as real as the represented?
Can movement be realistic?
Can lighting be realistic?
Can the aural experience be realistic?
Can we fake reality believably?
Once we believably simulate reality we can build Virtual Reality that provides a totally immersive experience and we can modify perception of Reality through Augmented Reality…
How Creative Computing is resolving some of the challenges by targeting the Senses
Image Analysis — Image analysis, such as recognizing 3-dimensional shapes from 2-dimensional images, is needed by technology that aims to automate reactionary systems, such as robotics, as well as those aiming to represent live real scenarios in virtual space such as remote surgery systems.
Rendering of Light and Shadows — The rendering of illumination in virtual scenes based on virtual light sources has been the subject of much research in computer imaging. Challenges in this area include realistic modelling of partially transparent surfaces andhistorically correct illumination of buried ruins and efficient realtime representation of a multiple sources of light.
Animation of Objects and Environments-Realistic representation of any 3D landscape needs to consider the non-static nature of all objects within field of view: wind effect on light fabric, twitching in old people, fauna, weak architecture bending under heavy load… animation gives life to static objects, in this case virtual objects represented by a lot of coloured pixels. Between key-framed flat animation and motion capture techniques, what new ways can we devise for efficient, realistic animation?
Sound Effect Synthesis — Recording real world sounds has its challenges such as the lack of control on the sound’s properties, the acoustics of its environment, and other background sounds during capture. Audio Engineers thus design parameterised sound effect generators that gives them such control — and here is where the challenge shifts onto making the synthesised effect as realistic as possible. When synthesising sound effects of fictional objects — and thus no one really know what it sounds like — the emphasis there is to be believable: that such a sound would be justifiable given the sound source’s nature and the properties of its environment. This is a field which is becoming even more in demand due to its role in Virtual Reality as it accompanies the head mounted visual displays.
Music Analysis and Composition — Music has a very important role in building a user experience: either by setting the mood when it takes a background role, or a narrative device when used to voice the non-verbal dialogue between characters. Computaitonal power allows for efficient analysis of large musical corpora and, based on such analysis, assist composers in the generation of, compositions that are similar in style or genre. These would be necessary in immersing the player into the virtual world.
Speech Synthesis and Recognition- The ubiquitous nature of the mobile devices demands an interface that favours hands-free communication. Speech is used incessantly between humans in both face-to-face and mediated forms. Speech synthesis has been used in situations where visual attention is not affordable or limited — such as GPS navigation — or as assistive technology for visually or vocally impaired persons such as the late Prof Stephen Hawking. What remains just out of grasp are aspects of speech that makes virtual characters believable: the vocal inflections, the expression, and the emotional tinge that gives meaning to speech.
Acoustics Modelling- It is well and good to have 3D laser scans of real buildings used to model the virtual representation of that space such that any illumination and reflections are true to the real space — but what about aural fidelity? The effect of the environment on the sound’s property should reflect the process that such a sound would undergo in that space in reality. As the head-mounted visual devices occupy the wearer’s full field of vision, immersing them totally within the virtual space, attention is now drawn to the aural dimension that reflects the tele-presence of the wearer in the virtual world by providing sound that mimics the acoustic properties of the modeled environment.
Haptics and skin-sense
The sense of touch has so far manifested itself in wearable gloves that provide sensory information to the fingers allowing the virtual reality visitor to “touch” the intangible objects depicted in the head-mounted screen and heard through the earphones. But the skin-sense allows us to sense the environment around us beyond touch: feeling wind, sensing temperature, burning sensations from abrasions and wounds etc. Making the virtual experience more realistic will mean stimulating these sensations whilst keeping them harmless and safe.
Smell and Taste
Since 1906 several attempts were made in various cinema theatres to augment movies with smells, but all involved broadcast: scent is emitted into an enclosed space for all to smell, but lack of control on air flow and lingering odours, they never were a success. A solution was to go for personalized scent and this was minimally achieved using scratch-cards which, however, interrupted the film experience and broke immersions. Now, as VR is a personalized activity, complete with headphones and gloves, an opportunity arises to present personalized smell experiences to heighten the immersion — and we are literally not far away from extending that to taste!
Motivations for Creative Computing
What are the objectives that motivate Creative Computing scientists to face and surmount these challenges?
Situating remote speakers in a co-located virtual environment can help immerse the communication into a non-distractive scenario that can actually support the discussion with situational imagery and allow full-body non-verbal communication especially in mixed-reality scenarios where the real person is projected into the virtual environment. Kind of Second Life but with real people representations.
Second Life - Holodeck at Portobello Beach Sandbox
The Introduction to Digital game-Based Learning class met Frank Lassard at the Virtual University of Edinburgh's…
This concept has been taken up by news stations, especially sports rooms. What if this became the norm when calling your colleague at work with workflow charts, live data, and website mock-ups surrounding you or telling your mum about your family holiday while photos of the places you visited occupy the space around you?
Singtel makes exclusive partnership with augmented reality studio
Singtel's advertising arm Singtel Advertising has tied up exclusively with a virtual reality and augmented reality…
Fast food chains are slowly integrating technology into their customer’s experience: from child-friendly tablets offering the menu and entertainment to self-ordering kiosks to save you time at the ordering queue. But eating out is also a social activity: meeting friends at the local eatery is a commonplace and healthy activity, but limited to friends within your vicinity. What if augmented reality would allow your faraway friends to meet you for dinner by going down to their local outlet and sitting at the table “next” to you (actually a screen projecting their table in their outlet while they’re seeing your projected table at the screen beside their table). Thus, mixed reality can provide a commercial advantage in selling a realistic experience, rather than just a product.
Entertainment is at the forefront of taking advantage of technology. Games like Ingress use a handheld screen device to portray the virtual world onto the real world.
Other approaches include projecting the virtual world onto your living room walls (works best if there’s no furniture of course!), an idea that was proposed by Microsoft but taken to new heights by an Australian company called Euclideon who are re-inventing computer graphics. Will casual daily entertainment cross the screen into our world?
Mixing realities can provide a show don’t tell approach to education that takes the history student into the thick of the action, the geography student in the midst of the earthquake, the language student with a personal virtual tutor immersed in the country’s culture, the entrepreneurial student with a virtual company of employees to manage to success.
Situating the trainee in realistic situations to provide hands-on training in a controlled environment allows room for failure with no danger to the trainee or trainer or target. From skills, to performance, to timed operations, virtual reality can provide a safe environment within which real-life scenarios are presented to train the user in learning new skills, how to handle situation, the correct order for a procedure and how to prioritise tasks.
A binaural recording of a live orchestra can place you in the midst of a real performing orchestra, but what if you are a budding musician and want to rehearse with an orchestra? Will you pay for a real orchestra to accompany you? Can you fit them all in your room or do you need to rent a hall? And where will you fit that large double bass? What if you immerse yourself in a virtual orchestra, that stops and repeats for you as you make mistakes and re-attempt the music?
Virtual reality can provide an alternative ending to emotionally charged situations that may have traumatised its participants who may need closure in order to move on. Such alternative endings may explain why such things happened, and whether they could have done anything to change the situation and run of events. Sometimes talking to other survivors also helps, but these survivors only live for so long, so projects are trying to capture their experiences and share it with today’s and future generations, such as the New Dimensions in Testimony project (See video below).
Phobias are common phenomena where people fear objects or situations which inhibit their normal day-to-day routines. Treating phobic patients with exposure to their fears has been found to help them face and overcome their fears: be it fear of heights, fear of spiders or snakes, fear of public speaking and many others. Exposure to such phobias in a safe environment is possible by a gradual exposure to virtual reality followed up by augmented reality to situate the object of fear within familiar environments.
VR can also help recovering patients in re-engaging with their daily life with virtual environments while still in hospital care, such as in recovering from a stroke. Thus, from within the limited environment of a hospital, patients can start practising facing their normal routine under supervision and with help ready at hand.
Join the Global Creative Computing Community
Creative Computing is the effort to provide society a better future by taking advantage of modern and future technology to provide means of solving problems of today and tomorrow using algorithmic skills leveraged by mobile intelligent systems.
If you are interested in following a career in creative computing, then you are at the right place. Saint Martin’s Institute of Higher Education offers a wide range of MQF level 5 diplomas that are open to students who are 16 years and over and can present at least four ordinary levels, including Mathematics and English.
The MQF level 5 diploma programmes, read as a one year full time or two year evening study, that lead to this field of creative computing are;
These MQF level 5 diploma lead either to an MQF level 6 BSc (Hons) degree in the same specialisations awarded by Saint Martin’s Institute of Higher Education and the University of London, under the academic direction of Goldsmiths College.
Contact us with whatever medium you feel comfortable. Join the future — Read for a Computing Degree at Saint Martin’s Institute of Higher Education #StudiedLocallyValuedGlobally
Join the Games Development Challenge — Click Here
The Successful Games Studio for 2020 — Click Here
Computing @ SMI #NotJustANumber — Click Here
#StartMyInspiration — Click Here