The Class Of Quality Assurance

Quality Assurance, or game testing, has a bad reputation in the games industry. In some ways it's justified: the working conditions can be grim and the pay incredulously low. But, more damaging and dangerous, the bad reputation is often promoted and propagated by in-house attitudes towards QA that brands staff "minions"; low-level, disposable work-horses who'll do the most mundane of tasks for hours on end. As a microcosm for the way that society at large treats its lowest paid - and often hardest working - members, it's apt. And it's unfair.

"Doing the same things over and over again... mentally I think you have to be really strong to get through it and make sure that you're paying attention"

Rob Karp, Sony X-Dev

During my own time in Quality Assurance, across two years at Studio Liverpool from 2008-2010 I had the chance to see the truth and the lies of the QA myth, and found it a vital stepping stone for my subsequent role as a game journalist for Edge magazine and beyond.

First, the truth: It's low paid. It's not glamorous (you may have to spend an entire evening, or week, scrolling through menus or rebooting discs, for example, and make no mistake: your eyes and thumbs will take a battering).

The lies? QA staff aren't disposable minions; they're quite often the most passionate game players and consumers you'll come across, hence their ability to endure painfully repetitive shifts, and the right studios should acknowledge and capitalise on this. Another falsehood is that QA staff are largely uneducated, disengaged drones. Many QA staff are highly educated, and most (if not all) aspire to progress through the industry from QA and many, like David Deeble, now a designer at Sumo Digital, achieve that dream.

Deeble got his start as a tester following a stint in video game retail, and found it a mostly encouraging experience. "I was super-excited at landing my first job in the industry and had so much damn fun in the process," he gushes. "I still have that same excitement and enthusiasm today, I mean there aren't many jobs out there where you literally can't wait to get into the office every morning."

Deeble moved to the ill-fated Bizarre Creations to continue his testing career before making the leap to game design at Sumo, but it was his time at Studio Liverpool that seems to have left its biggest mark, building a solid foundation for his career. "I was at SCEE back in 2007, so this was when Studio Liverpool was still around," he says. "[There were] loads of departments, from First Party QA, Format QA, Third-Party QA, a localisation team, a video department... not to mention an entire dev studio. My department alone had at least 40-50 staff and that was just the night-shift."

Another evangelist for QA is Thomas Smith. Now working as a tech artist at Ubisoft's Massive studio in Sweden he, like Deeble, made a horizontal move from Studio Liverpool to Bizarre Creations, finding the more connected experience of working alongside a dev team ("embedded testing" as its widely known) an important experience. "Internal QA departments are critical to a studio," he says. "They create work as well as verify incoming work from external departments for the rest of the teams, they stabilise those important press and trade show builds as well as the end product. Localisation, user feedback, tools, distributing builds to external studios... the list goes on. If you didn't have great people in control of all that the project would come off the rails quickly."

"When I first joined QA, there were two other women on my shift and 50-plus men. At times I did find it difficult"

Michelle Tilley, Studio Liverpool

For people like Deeble and Smith, QA acted as a launch-pad into the nitty-gritty of game development, and it clearly primed them with a knowledge of the processes they'd later be working with and around.

Others, like Rob Karp - now a producer at SCE's XDev Studio Europe - have found room to manoeuvre within the very building where they enlisted as testers. Karp's current role has him jet-setting around the world, hand-picking the next big things for Sony to take under its wing as publisher, but his entry-point into this high-powered, high-level role was anything but glamorous: "One task I had [as a tester] on Formula One 2005 on the PS2," he reveals, "was to check the collision detection on the barriers for each track. I would boot up each track and drive along pushing into the barriers on the left side the whole time, to make sure the car didn't get stuck or fall through the world, I'd then repeat this down the right hand side of the track... It was as thrilling as it sounds."

While Deeble and many like him host a seemingly bottomless well of enthusiasm for simply being around games, Karp's view is a sharp contrast, with no illusions that it can be a soul-crushing role. "It's not easy to play the game for three, six or nine months," Karp adds. "Doing the same things over and over again... mentally I think you have to be really strong to get through it and make sure that you're paying attention, because the last thing you want is to miss a bug."

The mundane nature of some of the tasks you're handed in QA is something I know all too well. I've sang my way through Singstar, played infinite rounds of the same general knowledge quiz in Buzz! (I'm none the wiser for it) and completed inhuman amounts of foreign language playthroughs on a range of titles good and bad. One of the benefits of such grind-work is that it familiarises you with the evolution of builds and the general workflow of a development team as you watch a title mutate towards master.

Karp agrees: "It did give me a great grounding in part of the development process, it let me see and understand the transition from versions of the game that were sent for familiarity, to alpha, to beta to master. In turn this helped to give me a good overview on scheduling, particularly when it comes to the QA phase, but also before that, because I've seen the number of issues that implementing certain features late in the development cycle can cause; which areas are the highest risk."

Michelle Tilley, Planning Release Coordinator at Studio Liverpool, is another example of a tester graduating to greater things within the same four walls she started out in. "When I was a tester, I sometimes got frustrated by the lack of direction - I didn't know exactly where I could progress from QA," she explains. "But this became clear when I had the chance to shadow the Planning and Release team and when a job became available, I applied and was lucky enough to be given a place on the team."

Tilley's presence here, as the lone woman out of my interviewees, reveals another truth of QA: the disproportion in male and female employment figures. Tilley is open about the intimidation of such a skew in gender representation: "When I first joined QA, there were two other women on my shift and 50-plus men. At times I did find it difficult, I worried that people didn't take me as seriously as I thought they should."

"There was definitely a feeling of isolation and disconnect, not just from the company itself but with the developers you were supposed to be working with"

David Deeble, Sumo Digital

Karp identified the same issue but is adamant there's no discrimination at play; it's all about the applications. "Standards for getting into QA aren't the highest," he says, "but very few women applied and they were held to exactly the same standards as the men that applied."

"I think this pushed me to try harder," Tilley chips in. "To take on more responsibility and to constantly think of ways we could improve and change our processes for the better. It also gave me thicker skin and the motivation to strive for another job."

Encouragingly, Tilley and Karp both saw the tide changing even in the short time since they started out in QA. "It is still an industry dominated by men," says Tilley, "though, I do feel this is changing. I think this may be in part because of marketing heavily focusing on men or the simple fact that a game development career isn't something advertised to young men or women. I don't think it should be a simple thing of 'hiring more women' but getting more young women interested in a career in the games industry [in general]."

"I think the proportions are starting to get better, albeit slowly," adds Karp. "Both in QA and the broader industry. I think it's still a male dominated industry, across all sectors from gamers to  developers, and I think it is super important to hire the best person for the role irrelevant of their sex, whether their sex is disproportionately represented in the industry or not. "

These four examples already demonstrate QA to be more than a dead-end role for minions, work-horses, drones. But QA itself can (and with a little more encouragement from studios in terms of pay and inclusion should) be an attractive career in itself. Take Dan O'Shaughnessy, who came fresh from degree-level study of games into QA and worked his way through a range of exclusively QA-focused roles. "My first position at SCEE Liverpool was in the Format QA department as a summer contractor", he explains. "From there I moved to First Party QA as a Functionality Tester, and eventually became a TRC (Technical Requirements Checklist) Tester. I had the opportunity on a number of occasions to go on-site to work with development teams, like Evolution Studios in Runcorn and Guerrilla in Amsterdam, which allowed me to gain an insight to how things differ from studio to studio." O'Shaughnessy's experience speaks for itself, and clearly spoke to Ubisoft, who has hired him to lead testing procedures, on-site, for its Abu Dhabi team. O'Shaughnessy's progress is also a valuable lesson for videogame degree graduates disheartened by the prospect of an entry-level role after full-time study.

It's interesting, and perhaps points out a flaw in today's QA structure, that none of my interviewees managed to move into the creative development field at the very studio that was their home for so long; each either stuck to the more formal procedure-based roles of QA or jumped ship altogether. I certainly didn't find a creative outlet or opportunity during my stint in QA, but times and attitudes change and, hopefully, a heightening awareness of QA's importance and the quality of many of its staff, will catalyse a shift from within.

Deeble identifies a reason that may be to blame for the blockade often facing those with more creative ambitions - the size of the workforce: "It was easy to feel very small in a place like [Studio Liverpool]. We tested games from studios all over the world, so direct contact with developers was incredibly rare and even then was usually handled by a Lead Tester. Contact with the studio upstairs from us was practically non-existent, even when working on their games. I think there was definitely a feeling of isolation and disconnect, not just from the company itself but with the developers you were supposed to be working with."

"Nobody wants to be told there are bugs in their work. There is a tendency to forget just how crucial a good QA team is in shipping a solid product"

David Deeble, Sumo Digital

It wasn't until Deeble made the move to Bizarre and its more close-knit environment that he "learned a tonne" about actual design, interacting with staff from all major disciplines, from programmers to artists. The experience at smaller studios with more intimate headcounts likely equates to a more personalised experience and therefore a chance to make more of a mark on the development and QA process. It's an industry-wide phenomenon we've all heard before: when a studio grows too large many of its voices and faces get lost in the crowd and the sense of community, of belonging, can be lost irretrievably.

The stories of Deeble, Smith, Karp, Tilley and O'Shaughnessy are all unique and exciting in their own right but what they share in common, besides getting their start in the same QA workforce, is an insider's view that QA deserves more kudos. As Karp points out: "There's a big problem with the whole of the QA environment and that is that it's their job to find problems, which in turn causes a negative attitude. For me, the closer that QA works with the dev team the better; there's nothing worse than testing sections that get cut from the title or just aren't ready to be tested... it's such a waste of time."

Deeble backs up the sense that, because QA deals in problems, it's staff are often viewed as problematic: "Nobody wants to be told there are bugs in their work, right? Especially at 2am the night before submitting a build. I think there is a tendency to forget just how crucial a good QA team is in shipping a solid product. I can't speak for other companies, but here at Sumo we focus a lot on talent and ability. A good example is during development of Sonic & All-Stars Racing Transformed we hired staff to form an embedded QA and balancing team. Some of those guys showed real promise so we hired them full-time in Junior Design and Production roles."

QA clearly has its problems, both in terms of reputation and its working environment, particularly at larger studios, but if the voices in this feature - and the closing words below - point to one thing, it's that if you're aspiring to enter the games industry, it's not a bad place to start. And QA staff are often much more than "minions".

Famous last words:

David Deeble, designer, Sumo Digital: "Don't assume that you can waltz into a company and use QA as a stepping-stone to fulfil your dream role. Never lose sight that you're there to do a job, be professional, be the best you can and take the opportunity to talk to people and learn about the game development process. If you decide you want to move into other roles, hone your craft in your spare time and most of all be patient."

Thomas Smith, tech artist, Massive: "I would say if you do know what it is you want to do in the industry then go and study for it, don't join a QA department. If you don't know what you want to do but feel you like the idea of working with computer games I would say don't do it. You will have a bad experience getting stuck working on something you don't like and end up leaving in six months. If you have a plan and you need industry experience to make it happen then QA is the right place to cut your teeth but there is going to be a lot of hard work." 

Rob Karp, producer, XDev Studio Europe: "It can definitely be a thankless role, I try to make sure that the QA teams I work with now don't feel that way, but the problem is that QA is often just a small portion of the total time you're working on a game and when the QA testing is occurring it's more than likely that the team and producer are going to be very, very busy, with the QA often slipping down the priorities.... It's only a dead end job if you let it be one, it's so important to show enthusiasm and initiative, once you're ready to move on, it's up to you to make it happen. But be warned: don't just expect opportunities to come your way for simply doing your job. Over 50 per cent of the producers/senior producers from X-Dev Studio Europe started in QA. Be prepared to work long hours, don't expect for the team and producer to be happy when you find a must-fix issue the day before submission... "

Dan O'Shaughnessy, First Party Quality Assurance Tester, SCEE: "It should definitely be considered a gateway into the gaming industry. Think about where you want to go in the industry and never stop working towards the career you want to pursue... I have definitely benefited from starting my career in QA, but when I started at SCEE, I was hoping to work my way up the ranks and eventually become a SCEE programmer. Unfortunately it didn't pan out that way for me, but that's not to say that's always the case."

Michelle Tilley, Planning Release Coordinator, SCEE: "I would encourage anyone to join QA as a way of joining the industry. As long as you're passionate about games and have a willingness to learn, you can move into any job position you want. A lot of new starters seem to automatically want a job in production, but there are so many different roles at varying entry-levels the opportunities are endless. For my role it gives me a unique perspective, as I understand both the importance of scheduling release to hit a target date and creating a good user experience. Funnily enough, basic skills embedded into any software tester such as attention to detail, problem solving and working as a team can be applied to most jobs.