Unpacking foley design in video games

It's often easy to pay more attention to the visuals of a game than its audio, even though the latter is arguably what truly immerses you -- especially if you have a surround sound set-up or a good pair of headphones. But it's notable that sound has been touted as an important element of the new generation of consoles, from the PlayStation 5's 3D Audio to Dolby Atmos support on Xbox.

What's even harder to appreciate about audio is it's often at its most effective when you don't notice it, which is paradoxically the mark of great sound design. While crunchy combat and big explosions are likely to be bigger talking points, it's the little details that make game worlds truly come alive.

Much of this comes from foley, which is a familiar term in sound post-production for film where it originates -- named after pioneering sound effect artist Jack Foley -- but works similarly in games: sound designers record and recreate the everyday sounds you hear, such as footsteps or the creaking of a door being opened.

Yet while a film production might have foley artists specialising in just this area, it's not uncommon for games, especially smaller indie ones, where the sound designer is also the composer, as is the case with Jeff van Dyck, audio director of Unpacking. More uniquely, his wife Angela is credited with foley production, with further assistance from their daughter Ella, making this cozy meditative indie game a family affair.

The authentic sounds for each of the hundreds of objects you unpack and place perhaps isn't what you might expect from a game with a pixel art aesthetic that recalls the 16-bit era, back when audio was still mostly generated from the sound chip. But if anything, realistic foley just adds to the game's quality.

"Pixel art is impressionistic in a way with a lot of detail missing, but there's nothing to say that the audio couldn't have all that detail," says Jeff. "Ultimately, there's not much else going on in the game -- all you do is pick up stuff and put it down -- so why not have a lot of detail there and make that really satisfying?"

As players would discover and share over social media, that attention to detail extends to the same object having different sounds based on the different surfaces it's placed down on such as a wooden table or ceramic floor, which accounts for why Unpacking has around 14,000 foley files.

"The biggest part of the foley was the data management," Angela explains, who was able to leverage her past experience with managing masses of play-by-play commentary dialogue in EA Sports titles for the game. "The planning and scripting, and then the recording and editing -- everything had to have a really solid plan and process to make because otherwise it just would have become this huge data nightmare."

While the game's premise of unpacking a person's belongings in a new living space -- with many personal but also familiar household items -- might have you assume the Van Dycks just recorded everything in their home, there's actually a lot more to it than that.

"We found that obviously if we'd recorded the sound from the exact same object for everything, Ange would still be working on it," Jeff explains. "So we deemed some items as 'hero items,' and they had the exact sound that they had but then we realised there's a lot of things that actually make the same sound, like plastic bottles. Or we'd group it into small, medium or large plastic bottles. So we made what we called a generic library based on the material that things are made of."

"If we'd recorded the sound from the exact same object for everything, we would still be working on it"

Jeff van Dyck, Witch Beam

This generic approach still allowed for finer details, which Angela refers to as sweeteners. "For example, the piggy bank is a ceramic sound, so it's ceramic medium, which is also used by other ceramic things like a cookie jar," she explains. "But then we had the coins recorded separately, and they get combined as that item gets used in the game that triggers both of them."

While Unpacking's accessibility allows you to click objects just once to pick up and drop, holding them also yields little touches. Shaking a water bottle, although not necessary to the game, was still an anticipated interaction so the Van Dycks edited the sounds twice to get both the push and return difference when shaking an object.

Capturing cars

As an indie project with a smaller budget, you might expect the audio designers needing to get more creative with limitations, but even for a large AAA production like Forza Horizon 5, budgeting is still an important factor. And that's not just the cost but also the budget for memory and computational power that you're sharing alongside art, AI, and all other facets of running a game smoothly.

As the series' lead audio designer Fraser Strachan says, "I have to take a bigger global overview of how [audio] is all coming together to really strip out some unnecessary things and focus the stuff that's really going to sell the gameplay at that moment in time, all within budget."

With around 500 cars in the game at launch, getting the authentic sound of each car is also more complex than simply sticking a microphone next to the engine or exhaust of every vehicle (although that is still the case to some extent).

"Some cars, such as a Toyota GT 86 and a Subaru BRZ, actually share the same engine manufacturer," Strachan continues. "Potentially, if there was one car that's rare to access in the UK, then we might try and find a counterpart that has the same manufacturer or engine. It's actually becoming more common, especially with electric cars that share the same motors."

Getting to the crux of the foley, however, the focus is on the chassis of the car, as the team captures the metallic body of the car creaking, wobbling and interacting with the environment it's moving through. "It's also made up of the four tires on the car, each doing different things depending on which surface that they're driving on. So you could be driving on snow or you could have one tire on gravel kicking up stones. At the same time, that car also has lots of other different dynamic sounds attached to it that make up the character of the vehicle."

The approach to Forza Horizon 5's foley was two-fold. First, there's the authentic capture of how a car sounds in real life, taking these vehicles out on a real track and getting them to do all kinds of wild tricks. But there's also the process of getting bits and pieces together in a room and smashing things up. The latter more accurately describes the point of foley, because ultimately the technique isn't so much capturing the natural sound of objects or actions but enhancing that for an audience.

"Part of a sound designer's job is that sometimes what you record in real life doesn't translate to what you expect," says Strachan. "I'd say when it comes to the foley of the chassis sounds and the tires, a lot of that is very enhanced sound design, whereas we like to be very authentic when it comes to the actual engine and exhaust notes of the cars."

Fantastic beasts and how to foley them

Of course, while sound designers will pride themselves on sourcing and creating their own sounds, there's always limitations that can come up. For VR prehistoric survival game Song in the Smoke, sound designer Ally Mobbs was able to faithfully produce the moment-to-moment in-game actions your caveman avatar would do, like banging stones or rubbings wooden sticks together, by doing the same actions in real life.

"I was lucky enough to get access to Kyoto Zoo - and not a single animal made a useful sound. I was there for a whole day"

Ally Mobbs, 17-Bit

Meanwhile, capturing the combat against the wild animals or gutting them for various parts used the classic approach of smashing and cracking vegetables -- notably cabbages -- for good gorey sounds. What wasn't so easy was capturing the wild beasts themselves; for a start, they are fictional variants in the evolutionary chain, but also real life isn't always on your side.

"I was lucky enough to get access to Kyoto Zoo, and I was really excited about doing this recording session -- and not a single animal made a useful sound," Mobbs says. "I was there for a whole day -- they had lions and tigers, in my mind it was going to be amazing, but they were just like sleeping. Then I'd hear the elephants and I'd run over to record those and by the time I got there, they stopped."

Fortunately, there are also plenty of external sound libraries that sound designers can turn to.

Another important aspect to Song in the Smoke's immersive audio is, for a game where you're essentially outside in the wild, the natural ambience. This was comparatively easier for Mobbs to capture, who has an interest in field recording -- audio produced outside a studio environment -- and lives near the remote mountains in Kyoto (where developer 17-Bit is also based).

"I'd go in the summer to get insects and then, because the project was quite long, I'd go again in different seasons and try and get wind in trees and evening ambiences to use in the game. The tricky part was getting far enough away from the city to not have the wrong kind of ambient noise."

Indeed, ambience and foley go together, and while audio purists might consider the former as unnecessary noise in games, it's a subtle but crucial aspect for being immersed in the world. Its implementation in Unpacking is perhaps a very subtle example of this.

"Sometimes what you record in real life doesn't translate to what you expect"

Fraser Strachan, Playground Games

"Think of the sound of each room in your house," Jeff van Dyck explains. "Your kitchen has a certain sound, your bedroom has a sound, and your bathroom will have another sound because of the shape of the room or things that are in there. So in Unpacking, each room has its room tone."

Even more subtle are the faint background sounds that denote where the house or apartment you're in is located. The first room is in the suburbs near a school, while the second room is in the city with traffic outside and the noise of roommates and tenants nearby and upstairs.

"I actually got Ange to walk around upstairs and I had a microphone downstairs to get the actual sound of somebody walking above you," Jeff adds.

This also extends to objects in the space, so that a music player turned on in one room can still be heard in another (albeit muffled), providing another dimension to a seemingly 2D game.

Capturing the sound of Mexico's natural environments was equally important in Forza Horizon 5, especially in terms of weather effects as the region's extreme conditions like tropical storms or dust storms allow for a more distinctive audio soundscape -- you can really differentiate between hot, wet, dry, or cold seasons by ear alone. But Strachan also stresses the importance of simply capturing the natural wildlife or the hustle and bustle of the city, even if that might be lost in the mix when a player is speeding through at 200 miles per hour.

"More and more, we have players who love to just explore the world, who love to drive to a location and stop their car and enjoy the scenery, and you can't enjoy the scenery without enjoying how good that sounds," he explains. "So when you've got something like the waterfalls of Agua Azeul, we've got flamingos, waterfalls, this tropical paradise feeling, and doing justice with an audio soundscape as well as a visual landscape is something we really wanted to achieve."

Foley for games

There are certainly different approaches to audio design between AAA and indie productions, and having begun in the former at EA and Creative Assembly, Jeff finds the latter more creatively satisfying -- even if that means working with smaller budgets.

"It's more of a personal experience with the people I'm working with, and there's the corporate thing -- there's marketing trying to influence things, execs are trying to influence things, and this thing that you're working on becomes designed by committee in a way. There's lots of politics, and I don't have time for that stuff."

But where they have something in common is how they differ from audio in other media like film or animation, which Strachan and Mobbs had previously worked in respectively. In these fields, the work is very much post-production and amounting to deliverables.

"You just create the sounds and pass them over to somebody else who does all the implementation for you," Strachan says. "So you kind of shove them into a black box and hopefully it comes out sounding as you wished."

Starting out just being contracted to deliver sound effects, Mobbs ended up getting hired full-time by 17-Bit, where he also learned to use Unreal and Wwise middleware in order to implement his sounds into the game himself, allowing for greater control over the final quality.

Game audio can also come across as more 'alive,' as it's common to record multiple variations of the same sound and then randomise their output, which is especially noticeable in Unpacking as it would sound unnatural for an object to sound exactly the same every time you pick it up and put it down, even on the same surface.

Given the interactive nature of games, having the freedom to play with sound, or rather having the sound respond to how the player wants to express themselves, whether they're driving slowly on the road or wanting to hurtle through a jungle smashing past bushes, is what makes game foley unique.

"We actually work with a lot of the same people that work on film and TV to create and record sounds but we won't strictly mix that down to a final master track," says Strachan. "We'll keep the individual sounds so that they can play back in multiple layers in multiple different ways.

"Something I had to learn quite quickly when I moved from post production into game audio is that you really have to compartmentalise your sound design a lot more, so you're working with a lot smaller pieces of audio played back simultaneously. It's a very different world."