Testing the game of play
March 5, 2010 4:43 PM Subscribe
Penny Arcade TV did an episode on game testing recently. It followed their thought process on coming up with a comic about Sony's reality show The Tester.
You may know that Sony is putting together a reality show where human beings caper for our amusement. That's regular stuff. But the prize they're competing for - a testing position - is, to be clear, not the sort of thing one competes for.
True enough, it's not hard to find stories about how testing is not that groovy.
You may know that Sony is putting together a reality show where human beings caper for our amusement. That's regular stuff. But the prize they're competing for - a testing position - is, to be clear, not the sort of thing one competes for.
True enough, it's not hard to find stories about how testing is not that groovy.
Hasn't anyone in the industry ever heard of Brook's Law?
Far too few believe it. And that's not just the game industry.
The real problem comes when "crunch time" goes on for months or years. I've worked on (non-game) projects like that, and it's even more common in the game industry (plus they generally get paid worse --- there's more people willing to work long hours for less money if they get to work on games than something like enterprise software).
Projects where everyone is expected to work 10+ hour days, 7 days a week, for months... yeah. Luckily I'm at a point where I don't have to put up with that sort of thing anymore.
posted by wildcrdj at 5:24 PM on March 5, 2010 [1 favorite]
Far too few believe it. And that's not just the game industry.
The real problem comes when "crunch time" goes on for months or years. I've worked on (non-game) projects like that, and it's even more common in the game industry (plus they generally get paid worse --- there's more people willing to work long hours for less money if they get to work on games than something like enterprise software).
Projects where everyone is expected to work 10+ hour days, 7 days a week, for months... yeah. Luckily I'm at a point where I don't have to put up with that sort of thing anymore.
posted by wildcrdj at 5:24 PM on March 5, 2010 [1 favorite]
That was very enjoyable. Thank you for posting!
posted by Pope Guilty at 5:28 PM on March 5, 2010 [1 favorite]
posted by Pope Guilty at 5:28 PM on March 5, 2010 [1 favorite]
DEATH MARCH. Original edition published by Ed Yourdon, 1997. Also, the Philippines in WWII.
posted by GuyZero at 5:30 PM on March 5, 2010
posted by GuyZero at 5:30 PM on March 5, 2010
Every time I hear about how game developers work 60-100 hours a week during crunch time, I laugh. Hasn't anyone in the industry ever heard of Brook's Law?
You either don't understand what crunch mode is, or you don't understand what Brook's saying. Do you work in the industry?
The real problem comes when "crunch time" goes on for months or years.
That's a death march, not crunch time.
posted by effbot at 6:02 PM on March 5, 2010
You either don't understand what crunch mode is, or you don't understand what Brook's saying. Do you work in the industry?
The real problem comes when "crunch time" goes on for months or years.
That's a death march, not crunch time.
posted by effbot at 6:02 PM on March 5, 2010
You either don't understand what crunch mode is, or you don't understand what Brook's saying.
How about instead of dropping empty snark you enlighten us?
posted by adamdschneider at 6:40 PM on March 5, 2010
How about instead of dropping empty snark you enlighten us?
posted by adamdschneider at 6:40 PM on March 5, 2010
How about instead of dropping empty snark you enlighten us?
If you read the wikipedia link, you'll see that it says that the two primary reasons adding more people results in projects taking longer are 1) it takes time to train the new people on the project and 2) communications-related efficiency loss increases as the number of people on the project does.
Clearly, neither of these reasons apply if you keep the same number of people working on the project and just have them work more hours each. There certainly are studies showing that the effectiveness of extra hours drops off dramatically the more you do, but Brooks's Law simply doesn't apply here.
posted by inkyz at 6:58 PM on March 5, 2010 [2 favorites]
If you read the wikipedia link, you'll see that it says that the two primary reasons adding more people results in projects taking longer are 1) it takes time to train the new people on the project and 2) communications-related efficiency loss increases as the number of people on the project does.
Clearly, neither of these reasons apply if you keep the same number of people working on the project and just have them work more hours each. There certainly are studies showing that the effectiveness of extra hours drops off dramatically the more you do, but Brooks's Law simply doesn't apply here.
posted by inkyz at 6:58 PM on March 5, 2010 [2 favorites]
I'm completely unfamiliar with how games are developed, but don't they use test driven development techniques? Are people using higher level languages that reduce bugs or is it still cpp/c for everything? I guess you're probably tied to whatever development platform the consoles push out, but I'm sort of surprised that this is such a tedious process.
posted by geoff. at 7:17 PM on March 5, 2010 [1 favorite]
posted by geoff. at 7:17 PM on March 5, 2010 [1 favorite]
I've always been under the impression that game testing is more akin to acceptance testing, than unit testing. That is, game testers are looking for bugs at a much higher level than what unit tests would (generally) catch.
posted by timelord at 7:33 PM on March 5, 2010
posted by timelord at 7:33 PM on March 5, 2010
I thought video game testers got to play games and work on tightening up the graphics. (yes that was a real commercial.)
posted by ALongDecember at 8:25 PM on March 5, 2010 [1 favorite]
posted by ALongDecember at 8:25 PM on March 5, 2010 [1 favorite]
If anything, "crunch time" is an attempt to avoid the consequences of Brooks' law. If adding people to your late project will make it later, maybe making the people who are already on the project work harder will fix everything!
There's probably another law somewhere about the point at which making a person work an extra hour a week makes them less productive overall. For me that point is at about 30 hours, which is one reason I don't work in game development (or IT at all, for that matter).
posted by A Thousand Baited Hooks at 8:50 PM on March 5, 2010 [1 favorite]
There's probably another law somewhere about the point at which making a person work an extra hour a week makes them less productive overall. For me that point is at about 30 hours, which is one reason I don't work in game development (or IT at all, for that matter).
posted by A Thousand Baited Hooks at 8:50 PM on March 5, 2010 [1 favorite]
I would never, ever take a job as a tester. The whole idea of this reality show seems like a cruel joke. I haven't been able to bring myself to watch it. I've been out of the industry for a few years now, but I could still rant for hours and hours about how poorly it was run. To keep it short(ish), I think these are the three main reasons game development works terrible hours:
1. It needs to be an iterative process, but it's scheduled as a linear one. Games need to be balanced, and (hopefully) fun. You can't design that perfectly on paper. You have to go back and make changes, try it again, and repeat. But there's no time in the schedule for that! The schedule is only new features. So when do the changes get made? After hours. Can't let the ship date slip, or the publisher won't pay us.
2. Quantity is valued over quality. How many bullet points can we get on the back of the box? How many features did you add today? How many bugs did you find/fix? Good design, preventing bugs, unit testing, even code reviews are not rewarded (and sometimes even punished!), because you could have been adding more features.
3. Developers are not treated with respect. 99% of the people I worked with sincerely wanted to make a good game, and would have gladly put in extra hours to do so. But they're only human. When they say a feature is going to take 40 hours, management scheduling 20 isn't going to make it so. When it's 2:30 in the morning and they tell you they'll be able to concentrate better after a good night's sleep, it's not an act. When they say they can't add that great new feature you came up with last night, it's because there aren't any more hours left in the day to do it, not because they're lazy.
Of course, it's not like this everywhere, and it's gotten better since I left. But it takes such a toll on you that many people decide it's just not worth it.
posted by Sibrax at 9:31 PM on March 5, 2010 [2 favorites]
1. It needs to be an iterative process, but it's scheduled as a linear one. Games need to be balanced, and (hopefully) fun. You can't design that perfectly on paper. You have to go back and make changes, try it again, and repeat. But there's no time in the schedule for that! The schedule is only new features. So when do the changes get made? After hours. Can't let the ship date slip, or the publisher won't pay us.
2. Quantity is valued over quality. How many bullet points can we get on the back of the box? How many features did you add today? How many bugs did you find/fix? Good design, preventing bugs, unit testing, even code reviews are not rewarded (and sometimes even punished!), because you could have been adding more features.
3. Developers are not treated with respect. 99% of the people I worked with sincerely wanted to make a good game, and would have gladly put in extra hours to do so. But they're only human. When they say a feature is going to take 40 hours, management scheduling 20 isn't going to make it so. When it's 2:30 in the morning and they tell you they'll be able to concentrate better after a good night's sleep, it's not an act. When they say they can't add that great new feature you came up with last night, it's because there aren't any more hours left in the day to do it, not because they're lazy.
Of course, it's not like this everywhere, and it's gotten better since I left. But it takes such a toll on you that many people decide it's just not worth it.
posted by Sibrax at 9:31 PM on March 5, 2010 [2 favorites]
Maybe it's partially the editing, but the intensity with which Jerry seems to approach ping pong really weirds me out for someone who I know only through humour.
posted by ODiV at 11:36 PM on March 5, 2010
posted by ODiV at 11:36 PM on March 5, 2010
Yeah, yeah, testing sucks as a job, but you shouldn't understimate how good some of that nutrient rich sludge tastes.
posted by slimepuppy at 12:08 AM on March 6, 2010 [1 favorite]
posted by slimepuppy at 12:08 AM on March 6, 2010 [1 favorite]
geoff.: "I'm completely unfamiliar with how games are developed, but don't they use test driven development techniques? Are people using higher level languages that reduce bugs or is it still cpp/c for everything?"
To understand the game development process you must understand two things 1) the dramatic willingness of 'talent' to bid down salaries just to break into this field 2) the dramatic ineptitude of game studio management. The end result is that the practice of human testing scales far cheaper than one would presume, and developers are focused on adding more polygons per second rather than fixing bugs.
Software wise, it is common to see C/C++/C# for games, owing to the strange platforms and demand to put parallel hardware (GPUs, multicores, etc) to efficient use. It's become far more common to license rendering and physics engines from third parties who actually do care about bugs. But that won't eliminate the need for manual testing. You're in competition with a flood of other studios, and need to push the edge, even if it means substantially modifying the engine you got.
In other areas, we can automate testing with unit tests or acceptance testing, but 3D games dramatically increase the computer vision techniques required to try that kind of trick. I recall reading a story of a guy who's job was to run through Metal Gear Solid 2 and shoot every polygon to make sure it made the right sound when shot. Writing a good automated test for this would probably extend development time by a half year or longer.
posted by pwnguin at 1:17 AM on March 6, 2010
To understand the game development process you must understand two things 1) the dramatic willingness of 'talent' to bid down salaries just to break into this field 2) the dramatic ineptitude of game studio management. The end result is that the practice of human testing scales far cheaper than one would presume, and developers are focused on adding more polygons per second rather than fixing bugs.
Software wise, it is common to see C/C++/C# for games, owing to the strange platforms and demand to put parallel hardware (GPUs, multicores, etc) to efficient use. It's become far more common to license rendering and physics engines from third parties who actually do care about bugs. But that won't eliminate the need for manual testing. You're in competition with a flood of other studios, and need to push the edge, even if it means substantially modifying the engine you got.
In other areas, we can automate testing with unit tests or acceptance testing, but 3D games dramatically increase the computer vision techniques required to try that kind of trick. I recall reading a story of a guy who's job was to run through Metal Gear Solid 2 and shoot every polygon to make sure it made the right sound when shot. Writing a good automated test for this would probably extend development time by a half year or longer.
posted by pwnguin at 1:17 AM on March 6, 2010
I'm working as a game tester now, and actually, it's fine -- our company has their shit together, and we get a lot of support when we say "X would make our job easier" and "Y is counterproductive." We use a combination of automation and human-testing, depending on what the particular problem is and how feasible it is to automate a particular task.
The bulk of my work is this:
-Play a level. Do various things to try and break the level (run into corners and jump a lot), and keep an eye out for things that are broken or poorly implemented (the menu says X should do Y. Is X actually doing Y?).
-If I find anything that looks wrong, I write it up and submit it to our massive database. That bug will get passed around, eventually making its way to someone who can fix it.
- In an ideal scenario, they'll fix the bug and send it back, then I or somebody else will make sure it's actually fixed. Sometimes, however, they won't be able to reproduce the bug, or they'll disagree as to whether it actually is a bug, or whatever. We have a bunch of procedures in place for all of these kinds of things, and generally it all works out. Occasionally, we'll get little game of bug ping-pong going, where someone insists something is fixed, but it keeps breaking on our end. In this case, something's wrong in the communication chain, and the producer might get involved to sort it out.
The downside of the game testing industry is that it turns playing games into a job. The upside is that you learn a crapload about how games are actually put together - often by trial and error, but sometimes developers will educate you on certain things in order to help hone in on a particular bug ("X is breaking for me but I don't know why, it's hooked into systems A and B, can you take a look? A and B basically work like this..."). It also can build communication and team skills in a somewhat "safer" environment than live production.
posted by ®@ at 6:52 AM on March 6, 2010
The bulk of my work is this:
-Play a level. Do various things to try and break the level (run into corners and jump a lot), and keep an eye out for things that are broken or poorly implemented (the menu says X should do Y. Is X actually doing Y?).
-If I find anything that looks wrong, I write it up and submit it to our massive database. That bug will get passed around, eventually making its way to someone who can fix it.
- In an ideal scenario, they'll fix the bug and send it back, then I or somebody else will make sure it's actually fixed. Sometimes, however, they won't be able to reproduce the bug, or they'll disagree as to whether it actually is a bug, or whatever. We have a bunch of procedures in place for all of these kinds of things, and generally it all works out. Occasionally, we'll get little game of bug ping-pong going, where someone insists something is fixed, but it keeps breaking on our end. In this case, something's wrong in the communication chain, and the producer might get involved to sort it out.
The downside of the game testing industry is that it turns playing games into a job. The upside is that you learn a crapload about how games are actually put together - often by trial and error, but sometimes developers will educate you on certain things in order to help hone in on a particular bug ("X is breaking for me but I don't know why, it's hooked into systems A and B, can you take a look? A and B basically work like this..."). It also can build communication and team skills in a somewhat "safer" environment than live production.
posted by ®@ at 6:52 AM on March 6, 2010
I'm working as a game tester now, and actually, it's fine -- our company has their shit together, and we get a lot of support when we say "X would make our job easier" and "Y is counterproductive."
I don't think testing is that bad of a job, and when I did it my overall synopsis would be that it was alright for the most part, BUT I will say you must be at a really great company if anyone outside of the testing dept. actually listens to you. Because I know first hand that is NOT the way the larger companies work.
The upside is that you learn a crapload about how games are actually put together - often by trial and error, but sometimes developers will educate you on certain things in order to help hone in on a particular bug
Again, I will have to say that this is by and large not typical.
So what it comes down to is if you land a job at the right company, testing decent games. Otherwise, you'll find out about testing soon enough when you have something like a Disney game demo in your hands. Because after an expericence of having two weeks, of eight hour days, to listen to all of two songs about how tough it is to be a high school singing sensation you'll find that it can get a little...unnerving.
posted by P.o.B. at 3:40 PM on March 6, 2010
I don't think testing is that bad of a job, and when I did it my overall synopsis would be that it was alright for the most part, BUT I will say you must be at a really great company if anyone outside of the testing dept. actually listens to you. Because I know first hand that is NOT the way the larger companies work.
The upside is that you learn a crapload about how games are actually put together - often by trial and error, but sometimes developers will educate you on certain things in order to help hone in on a particular bug
Again, I will have to say that this is by and large not typical.
So what it comes down to is if you land a job at the right company, testing decent games. Otherwise, you'll find out about testing soon enough when you have something like a Disney game demo in your hands. Because after an expericence of having two weeks, of eight hour days, to listen to all of two songs about how tough it is to be a high school singing sensation you'll find that it can get a little...unnerving.
posted by P.o.B. at 3:40 PM on March 6, 2010
There already was a documentary on game testers. Grandma's Boy
posted by inthe80s at 4:04 PM on March 6, 2010 [1 favorite]
posted by inthe80s at 4:04 PM on March 6, 2010 [1 favorite]
the dramatic willingness of 'talent' to bid down salaries just to break into this field
This is key. I predict that video games are going to be like film/TV is today. It won't be long before testers are like runners: unpaid staff who are lured in with promises of a chance of 'making it' in the industry. Very few actually will but there'll be plenty of new people to replace the ones that leave.
There's already hints of this happening now; I've worked with a few "zero hour" agencies who have a pool testers who are on-call at a days notice. Sounds pretty good in theory (big pool of testers, don't have to pay them for days there is no work in and you can increase the amount of testers temporarily if needed) but in practice they bullied people into coming into work and didn't have very stringent requirements for the employees. Though we were able to make sure none of the bad testers ever came back, it was still a days testing that was wasted. The best people ended up working for 30 days straight because they wanted to make it in the industry and didn't want to risk saying 'no' to the agency that penalized people for declining work by blacklisting them for a week or so. I think this was in part to keep the pool as big as possible which is going to be less and less of a problem in the future as the popularity of the industry grows.
tl;dr version: I'm glad I got my testing days done with because in a few year's time, you'll be lucky to get paid for it.
posted by slimepuppy at 11:25 PM on March 6, 2010
This is key. I predict that video games are going to be like film/TV is today. It won't be long before testers are like runners: unpaid staff who are lured in with promises of a chance of 'making it' in the industry. Very few actually will but there'll be plenty of new people to replace the ones that leave.
There's already hints of this happening now; I've worked with a few "zero hour" agencies who have a pool testers who are on-call at a days notice. Sounds pretty good in theory (big pool of testers, don't have to pay them for days there is no work in and you can increase the amount of testers temporarily if needed) but in practice they bullied people into coming into work and didn't have very stringent requirements for the employees. Though we were able to make sure none of the bad testers ever came back, it was still a days testing that was wasted. The best people ended up working for 30 days straight because they wanted to make it in the industry and didn't want to risk saying 'no' to the agency that penalized people for declining work by blacklisting them for a week or so. I think this was in part to keep the pool as big as possible which is going to be less and less of a problem in the future as the popularity of the industry grows.
tl;dr version: I'm glad I got my testing days done with because in a few year's time, you'll be lucky to get paid for it.
posted by slimepuppy at 11:25 PM on March 6, 2010
« Older Goodnight Forest Moon | David Lynch's A Goofy Movie Newer »
This thread has been archived and is closed to new comments
posted by rebent at 5:21 PM on March 5, 2010 [1 favorite]