Snow Queen

I've neglected my blog slightly the for the last month, been pretty busy with uni work and lots of other projects! 

Just finished my second collaboration with fusion belly dancer Ariana Faethe so thought I would share it here, wrote some ethereal electronic nastiness for her performance as the 'Snow Queen'.

https://www.youtube.com/watch?v=pxkkA8Aih0k#t=7

Procedural Audio

In this post I will explore the relatively underused method of implementing a procedural audio approach to drive elements of game audio rather than the more widely used ‘data-driven’ approach of using many individual sound files. There seems to be much uncertainty and scepticism surrounding PA in the game audio community, I will attempt to explore whether this method could be a viable option to replace or augment audio systems in the current production paradigm.

In the video games world procedural audio (PA) refers to the computational process of generating audio from nothing, or almost nothing [1]. In an interview, legendary sound designer and computer scientist Andy Farnell describes PA as “a philosophy about sound being a process and not data.” [2].

The main principle behind this system is rather than focussing on creating audio assets, the sound designer focuses on creating a system by which these assets are modelled.  In a game context, rather than storing many wav files in RAM or on the game disk, audio assets are created at run-time and are modulated and controlled in real time using game parameters. The end goal is to create a convincing system which models equivalent audio data to pre-recorded files but is constructed using little to no wav data.

Methodologies

According to Nicolas Fournel, Principal Audio Programmer at Sony Computing Europe, there are two main paradigms when designing a procedural audio system [3].

Teleological Modelling (Bottom-up approach)

This is the process of creating an asset model from the ground up based on real world laws of physics. Of the two methodologies it is the most challenging to create because it requires an in-depth knowledge of synthesis and sound production mechanisms (physics, mechanics, anatomy etc.).

With a system such as this, Andy Farnell suggests that many of the tedious object interactions that sound designer would have to create (environmental sounds such as wind, object impact noises etc.) would be automatically created allowing the sound designer to focus on much more ‘emotionally significant’ sounds such as character weaponry [2].

Nikunj Raghuvanshi, a researcher in field of physically-based modelling, argues that this type of methodology would result in a loss of artistic control for the sound designer [4]. Farnell, however proposes that a slight structural re-shuffling of game audio departments would mean that one main sound designer would preside over a team of programmers, guiding the process from a more aesthetic perspective.

Although this method can create realistic results, it is computationally very demanding and when implemented into a game situation, where CPU cycles are shared between various other systems, does not yet provide an efficient enough replacement for a data-driven approach. With further funding and research however, it is not unreasonable to predict that a much more efficient system could be constructed.

Ontogenetic Modelling (Top-down approach)

Using this approach, a base-sound for the required asset is provided by the sound designer which is analysed and deconstructed in order to attain the characteristics of the sound. This data is then used as a template to partially or fully re-create the sound using various synthesis techniques [5]. This method allows the sound designer to work in a very similar work flow to existing techniques and it would therefore require very little re-structuring to implement this into existing game design methodologies.

Of the two methods, the top-down approach is slightly more CPU friendly and because a model is provided to form the base for sound construction, less specialised knowledge is required to create audio assets.

Advantages

Procedural design offers an alternative to the widely adopted method ‘data driven’ system of using many different wav files to sonify the game environment.  At a glance, it has many advantages over using pre-recorded material [3]:

It saves memory by using code instead of wav data

Using wav files for the means of sound reproduction means either streaming these from disk (usually reserved for longer, looping sounds and music), or loading them into RAM for playback at run-time. With either of these methods the sound designer will be competing for space with the graphics components and game engine and will thus have to make a compromise in the sounds they produce [4]. When using a PA approach all audio assets are synthesised at run-time and thus require a fraction of the disk and memory space and when compared to using wav data.

Better response to game physics

The nature of PA to be created in real-time allows the system to respond in a much more realistic way to the physics system of the game. Requirements for certain sounds can be calculated and synthesised according to game parameters allowing the two components to be more synchronised. This is particularly useful for creating sound in response to rolling, sliding or scraping within the game environment where a ‘data driven’ approach may be too clumsy to recreate the interaction realistically.

Reduces repetition

Due to the creation of an ‘asset model’ rather than just an asset, PA is perfect for creating game sounds and textures that more closely mimic real-life situations. The use of a more granular, low-level structure means that sound interactions could be modelled to ensure that the same sound is never heard more than once. This is something which is highly sought after by sound designers but is almost impossible (with current technology) using the ‘data driven’ approach. One-shot sound effects such as footsteps, gunshots and impacts can benefit hugely from a procedural approach as these types of sound are constantly heard by the player and any noticeable repetition can break the immersion of the game.

Obstacles

It may seem like a no-brainer to adopt PA for many audio systems in interactive games but in reality the solution is not quite that simple. There are many issues that developers of PA systems have to overcome before it will become a viable option for wide-scale integration into Interactive games. A few of the main obstacles include:

Variable CPU cost

One of the main advantages of PA, this is also the cause of one of its main problems. The nature of dynamically created audio to be variable means that it can be incredibly hard to predict the ‘cost’ (in terms of system resources) of producing a certain sound prior to execution. A dynamic method requires that the cost of an operation must be predicted in advance and system resources allocated accordingly. Farnell describes that a system in which the sound ‘gracefully degrades’ depending on available resources could overcome such obstacles [6].

Lack of Skills and tools of existing Sound Designers

Currently in the games industry there is a huge investment of skills and knowledge with existing tools and Farnell suggests that “even sound designers who are comfortable with progressive technology feel threatened by the need to adapt their skills and learn new tools.” [6]. Before a new, potentially disruptive technology such as this could be widely adopted, suitable tool-chains would have to be developed and training given to existing sound designers.

Fear factor

As mentioned in the opening section of the post, at present there is much aversion to Procedural Audio methods in the game audio community. There are fears among many sound designers that a method such as this will replace them and that large corporate companies will see it as an opportunity to ‘streamline’ their production process by disbanding the audio department. Many of these fears are unfounded and as mentioned previously the integration of PA systems could potentially allow the sound designer to focus on more important and relevant tasks.

Conclusion

There is no doubt that using a procedural approach to generate game audio has many benefits. It can model a closer relationship between game objects and potentially free the sound designer from having to produce many tedious game-world interactions. It seems that it may require a small re-structuring of existing production methodologies but it would still allow the sound designer to have creative control over content produced.

Unfortunately it seems that before PA can reach an acceptable level for widespread integration into games, much research still needs to be carried out to create more efficient and better sounding models. Andy Farnell equates it to the 3D graphics of early first person shooters. “Once games did not have super 3D graphics, early titles like Wolfenstein and Quake were basically box walled mazes covered in low resolution textures Synthetic sound is stuck at an equivalent stage of development, mainly because it has been excluded and neglected for 15 years.”.

I believe that with real investment and development procedural audio offers an excellent addition to the game sound designer’s tool box. If it is viewed as a tool to augment current systems of audio reproduction and not as a direct replacement the acceptance of this new technology does not seem so daunting. It is a technique which has already been used to great effect in several popular titles such as Spore [7] and Crackdown 1 and 2 [4]. If current scepticism and opposition can be overcome I believe there is definitely a place for procedural audio among current and next generation games. 

References

1. http://www.develop-online.net/tools-and-tech/procedural-audio-with-unity/011743
2. Stevens, Richard, and Dave Raybould. The Game Audio Tutorial: A Practical Guide to Sound and Music for Interactive Games / Richard Stevens, Dave Raybould. Amsterdam ; Boston : Focal Press/Elsevier, c2011.
3. http://www.procedural-audio.com/papers/GDC%202011%20-%20Audio%20Boot%20Camp.pdf
4.  Nikunj Raghuvanshi. (2011). Sound Synthesis in CRACKDOWN 2 and Wave Acoustics for Games. Available:http://www.gdcvault.com/play/1014416/Sound-Synthesis-in-CRACKDOWN-2. 
5.  Rutherford, S., 2008. Procedural Methods for Audio Generation in Interactive Games. Available at: http://medcontent.metapress.com/index/A65RM03P4874243N.pdf [Accessed November 4, 2013].
6. Farnell, A., 2007. An introduction to procedural audio and its application in computer games . , (September), pp.1–31.
7.  http://spaceoddity.sgsgames.com/?p=799

Interactive Music

From the early days of orchestral accompaniments to silent films, music has played an important role in enhancing media. In early cinema, music was played as a method of masking unwanted noise from the street outside and from the projector [1]. As film became more popular however, practitioners in the field soon noticed the impact of including music as a way of enhancing the psychological drama of a film [2] and thus music became a much more integral part of the performance.

In the context of computer games music is a slightly more complex subject. Due to their interactive nature, games present the unique challenge that the actions of the player in the game cannot be exactly anticipated. This creates interesting problems when considering musical accompaniment which, if used in a similar way to film, must reflect the emotional state of the player and the game environment. In a game situation a linear score can quickly become repetitive. If a player is stuck in a certain place in the game for example, the monotonous looping of music will become increasingly maddening and will only serve to infuriate the player as they struggle to progress. 

The answer to this problem is therefore to create a system which allows the music to flow and change with the game dynamics, reflecting the mood of the game in real time. There have been many attempts to create interactive music systems in games and many have been unsuccessful!

This is a good example of an interactive music score failing to react to a change in emotional state. As you can see, the high-energy action music, which is perfectly suited to fighting gorillas does not work so well when all the gorillas are dead and you’re standing still. This demonstrates some of the problems that need to be overcome when considering an interactive music score.

One of the key elements with interactive music is to address the balance of ‘Musical Structure vs Player Autonomy’. The music must be structured enough that it can complement the emotional state of the player but also allow and be able to adapt to the unpredictable nature of the game environment. This presents a constant challenge for the audio development team and especially in recent years have led to many novel applications of technology to address this critical balance.

A great example of modern day interactive music is demonstrated by the highly stylised, side-scroller puzzle game Limbo by Playdead studios [3]. Limbo takes a refreshingly different attitude to in-game music, merging the boundary between the diegetic sounds of the game and the otherworldly textures that form the music. Arnt Jensen, the creator and game director said that he wanted to ‘give prominence to the boy’s Foley sounds, to emphasize silence and subtlety in the ambiences, and to avoid music that would manipulate the emotions of the player.’

The music below is taken from the title screen of Limbo.

Immediately the tone of the game becomes apparent. Bleak sonic textures and unidentifiable acousmatic sound combine to create an eerie soundscape which is strangely organic. A subtle drone is layered with what sounds like a combination of waves and bee and traffic or possibly rain, a waterfall, maybe wind. This highlights the beauty of the sound design for this game. All the elements are suggestive of real-world sounds but are expertly manipulated and combined to become one flowing sonic journey that remains strangely ambiguous. The sound designer, Martin Stig Anderson, has a background in ‘electroacoustic’ music and has developed techniques that perfectly complement the artistic style of the game.

This next video shows an example of how the music responds to mirror and emphasise in-game events. At the start of the clip a high pitch drone can be heard, layered with what sounds like birdsong and forest ambience. This continues until the boulder crashes into the ground whereby a different tonality can be observed. Notice how after the sounds of debris from the impact settle, the music adopts a more granular character almost mimicking and continuing the sounds heard from the previous event.

This second gameplay video shows another example of the adaptive, almost narrative nature of the music. Before the encounter with the infamous spider, the music takes the form of what appears to be a wind-like texture. The spider comes into view and unfurls its legs, smashing the ground as the boy approaches. As before, in synchronicity to the impact of the spider’s leg to the ground the music reacts, evolving into much more ominous and menacing tone.

As you can see, this demonstrates an extremely effective method of implementing an interactive music score into a game which perfectly fits the aesthetic and tone the developer is trying to achieve. The music enhances and complements the emotional state of the player, seamlessly adapting to the changing game world without breaking immersion.

Achieving such an immersive example of interactive music was a long process and many paradigms about existing techniques had to be rethought. Usually with game sounds all audio artefacts will be presented in mono however, in an interview with Martin Stig Anderson he explains about how when he put everything into mono he couldn’t engage with the sounds, stating that it was just not immersive enough [4].

Anderson continues to explain that ambiences and textures were broken down into individual ‘grains’. Sound effects such as rain and wind might only consist of a two second sample which was then cut into ten to fifteen different slices and stitched back together at run time. This allowed huge varied textures to be created from very small chicks of audio. This technique also ensures that the probability of hearing any repetition within the music is incredibly low.

It must be stated that the music in Limbo does not directly confront many of the issues presented with interactive music, it merely sidesteps them. None of the music has a notable pulse or rhythm meaning that sections can easily transitioned between without breaking the immersion of the game. Nevertheless it is still a great example of how effective sound design and clever use of technology can be used to create a powerful sense of identity and style within a game.

In an article on Kotaku [5], Stephen Totilo suggests that video game music is neither essential nor necessary for complete functioning of a computer game, implying that the same experience could be had if the music is replaced by a podcast or other such media. Limbo is a shining example of why (properly implemented) music is essential in modern games not only as an accompaniment to the graphic but also as a key game mechanic. The way the music functions in Limbo to highlight certain objects in the landscape or emphasise certain actions is just as integral to the experience as the graphics. The game is presented as a whole, with the graphics and sound so tightly integrated and essential to the feel of the game that they cannot be separated. Simply put, to play Limbo without music would be to play half a game.

References:

1.      Calvalcanti, A. (1985). Sounds in films. In Film Sound:Theory and practive (eds. E. Weis & J. Belton), pp 98-111. New York: Columbia University Press.

2.      Lipscomb, S.D. & Tolchinsky, D.E. (in press). The role of music communication in cinema. In Music Communication (Eds., D. Miell, R. MacDonald, & D. Hargreaves), Oxford, UK: Oxford University Press.

3.      http://limbogame.org/

4.      http://www.nxtbook.com/nxtbooks/newbay/audiomedia_201107/#/32

5.      http://kotaku.com/5730637/the-year-i-gained-the-courage-to-ignore- video-game-music      

Non repetitive sound design

One of the biggest challenges faced by the Sound Designer and Audio Programmer when implementing sounds into a game is the issue of sound repetition. In the real world (excluding artificial systems like a fire alarm which convey information) you would not expect to hear the exact same sound twice. For example hearing the exact same dog bark three times in a row would probably be most unsettling and you would start to question whether you were actually in real life or in-fact part of some artificially created world that was experiencing a minor glitch.

Provided we are talking about non-symbolic sounds (which carry meaning or convey information), the same is true for computer games. If a player experiences something that is particularly unnatural or breaks the ‘flow’ ¹ of the game then they will pulled from the game environment as their attention is drawn to this event, breaking any immersion built by the game so far ². Immersion is seen as the Holy Grail of the gaming experience and therefore it is the challenge of the game audio team to overcome these obstacles within the technical limitations posed by the game platform.

With the increase in system resources that have come from improvements in technology, various systems have been developed to deal with the issue of sound repetition in games, two of the most commonly used systems are explored below:

·         Pitch Shifting and volume modulation can be used to increase variation in one-shot sounds and layered textures. When applied subtly and randomly, using modest values, a large range of variation can be achieved with only a few sound files giving savings on both time and memory, which is always a good thing! This technique typically works best on repetitive sounds like footsteps or gunshots but can equally be applied to layers of atmospheric textures such as bird song.

·         Horizontal Concatenation, which involves combining several small sound files to create a longer sound, is frequently used to provide variation in ‘one-shot’ sounds. When this process is randomised, this allows for a huge number of variations from only a few original sounds (the actual number of permutations is '(2^x) - 1' where x is the number of sound files used.). A simple concatenation system from UDK is shown below: 

The boxes to the right side of the of the system are the sound files which will be combined to create the overall longer sound. These are combined at random using the 'Random' object and then passed to the 'Concatenator' object where they will be 'stitched' back together in a top down order.

The video below demonstrates an example of non-repetitive sound design in the latest of the ‘Elder Scrolls’ series of games, Skyrim.

As you can hear, each time the bow is fired, a different set of sounds is heard. This is relatively close to what you would expect to happen in real life and thus does not draw attention to the action and break the 'flow' of the game. Obviously I can only speculate about the methods actually used to implement this system but for the purposes of this analysis I will assume that they have used the methods I have covered above. Sonically it appears that the firing mechanic of the bow is split into four distinct stages.








Once these stages are established the various recorded or premade bow sounds can be edited and divided among the four groups. A system much like the basic concatenation system shown above could then be implemented to randomly splice the samples together in the correct order. Pitch and volume modulation could be applied to each of the stages to provide further variation.

References:

1. Chen, Jenova. “Flow in Games (and Everything Else).” Communications of the ACM 50, no. 4 (April 2007): 31–34. - Great paper on the 'flow' theory in games.
2. Stevens, Richard, and Dave Raybould. The Game Audio Tutorial: A Practical Guide to Sound and Music for Interactive Games / Richard Stevens, Dave Raybould. Amsterdam ; Boston : Focal Press/Elsevier, c2011., 2011. - Excellent book on all things game sound related

For some further reading on non-repetitive design check out this website: http://designingsound.org/2010/03/audio-implementation-greats-5-ambient-the-hills-are-alive/

Functions of Game Audio

From the earliest conception of computer games, audio has been used to perform many functions. In early games such as Pong very simple ‘beeps’ were used to give players feedback about events happening in the game. In modern games, as technology has improved, audio has become a multi-faceted tool used to perform a growing number of tasks.

The Taxonomy of Audio for Games (TAG) project aims to highlight the key areas where audio is used for specific effects in modern games. One of the areas outlined is the use of audio to promote ‘Mastery and Autonomy’ for the player within the game environment. These are split into six main functions:

•  Instruction
• Notification
•  Feedback
• Orientation
• Rhythm-Action
• Mechanic

In this post I will use an example within a modern game to demonstrate one of these functions further.

The video below shows a short clip from Deus Ex: Human Revolution. It demonstrates an example of Notification for the player. One of the key gameplay techniques of this game is stealth and completing many of the main missions rely on the player’s mastery of this method.

There are a few devices in the game to help a player remain unseen by the enemy. Vision cones can be added to the mini map and augmented vision allows the player to see through walls, highlighting any nearby enemies. However, the device I found most useful when playing was the verbosity of the enemy guards. The video shows an example of what happens when the player is seen in an area where enemies are overtly hostile, in this example, an enemy base.

Notice as I run into the field of view of the guard, he gives notification that he has seen me and begins to fire. As I hide behind the wall out of sight he stops firing for a moment until I reappear and he verbally confirms that he has seen me again. This is a clear example of audio being used to notify the player that the guard has changed state and is now hostile towards the player.

Throughout the game verbal cues are given by enemies to notify the player about their state. If the player were to sneak into a position where the guard could not see them and make enough noise to cause a disturbance, the guard would verbally notify them that he had heard the sound and would enter an increased state of alertness to future noise and movement. This prompts the player to either leave the area until the guard returns to his neutral state or be extra careful moving around the area.

This type of notification is incredibly useful for the player, especially when they are still learning the fundamental game mechanics. It allows the player to quickly develop skills that will aid them in mastery of the game and will decrease frustration at higher difficulty levels. 

Example of Good Sound Design

The short video below shows some footage recorded from the original Bioshock, this demonstrates what I think is an example of particularly good sound design. The video features an encounter with one of the main NPC characters in the game, the ‘Big Daddy’.  The ‘Big Daddy’ is a hulking, super-powered behemoth of a creature clad in a sturdy, steampunk styled deep-sea diving suit. If left alone, the ‘Big Daddy’ will calmly pursue their own agenda, but if antagonised they can quickly become a deadly force which can bring about a swift death for the player if they are caught off-guard.

The ‘Big Daddy’ can be found in levels by following it’s characteristic bellow, as heard in the video. This low, guttural moan is the first indicator to a player that they are in close proximity to one of these beasts and can either act as a warning or as a beckoning call to the kind of player that casually swigs Buckfast as they play.

The moan of the ‘Big Daddy’ serves many purposes and as a sound design device is tremendously effective. For a player new to the game hearing the moan of the ‘Big Daddy’ doesn't immediately conjure an image of this beast but rather implies that something large and powerful could possibly be lurking around the next corner. It creates suspense and adds to the feeling that the player is part of one big ecosystem which is constantly changing and adapting around them. For the more experienced player the moan is an effective way of tracking these giants. One bellow can give the player enough positional information to track these brutes down and, if necessary, get a bit shooty to reap the rewards of killing one.

This is a great example of how a simple device can be used to fulfil a number of different roles within a game environment, using acousmatic audio in order to pique a player’s curiosity or warn of a possible threat.