How I do playtests

February 5, 2018

A quickly drawn picture of a notepad and someone playing a game in front of a laptop.

I play a lot of indie games, as well as a lot of stuff– small­er, short­er and sim­pler games. What I often notice, though, is that some of these games have real­ly tiny but frus­trat­ing issues that could’ve been pre­vent­ed, or moments where I just get stuck think­ing the game glitched out.

One tool you could use to help to avoid these frus­trat­ing moments is playtest­ing, which will help you to spot the most obvi­ous issues in your game that could pre­vent enjoy­ment. How­ev­er, playtest­ing will face you with the ugly faces of your game– espe­cial­ly the first few tests, where you can see all of your game’s short­com­ings. It can be depress­ing, and might make you reluc­tant to do tests at all. This post urges you to press on, because playtests intro­duce the ele­ment that com­pletes your game: the play­ers.

You will need to move the game out of the bub­ble it’s being devel­oped in. When you give it to some­one else, you will learn new things about your own game. These can be either good or bad, infor­ma­tive or not real­ly help­ful. You’ll need to learn to deal with that. Don’t make it the player’s fault (‘You’re play­ing it wrong!’), instead, look at where it is going wrong on the design lev­el, as objec­tive­ly as pos­si­ble. It is human ten­den­cy to do this— I find watch­ing playtests embar­rass­ing, per­son­al­ly. It can be scary, but in the end, it’s for the good of the game.

In this post I explain which playtest­ing progress works for me. I playtest my games from time to time, so hav­ing a struc­ture for orga­niz­ing it is pret­ty help­ful. It was designed to be effi­cient: get as much use­ful info from tests while orga­niz­ing as few of them as pos­si­ble. I’m intend­ing this post for peo­ple who want to get start­ed with playtest­ing, so if you already have a pro­ce­dure that works for you, please stick to it!

Setting up the test

I attempt to begin test­ing as ear­ly in devel­op­ment as pos­si­ble. The soon­er you know the game is hot or not, the more informed you can make deci­sions on things like scope, risk areas, or per­haps even to deem the project a fail­ure and can­cel it. (Also known as “Fail faster.”)

I have a note­book to make anno­ta­tions while spec­tat­ing the playtesters. With this you can quick­ly make notes, lit­tle draw­ings and dia­grams. I would only use a key­board if you’re an adept blind typer.

I also use note­books for sketch­ing ideas. Pro tip: date your entries to you can track your progress.

Make sure you have a sta­ble build of your game to avoid a crash in the mid­dle of the test. Note that a playtest and a bug test are two sep­a­rate things– although you should note found bugs down to fix lat­er. I do think you should compile/build your game before­hand, to make test­ing faster to set up. (You can still use the in-engine play­er as a Plan B.)

I record a video of the playtest to review it lat­er if need­ed. You can either set up a cam­era to record the room and game­play, or you can make your com­put­er record its screen (see my tuto­r­i­al for help with set­ting this up.)

During the test

Invite a friend, fam­i­ly mem­ber, or some­one else you know (or a group if you have a mul­ti­play­er game), prefer­ably peo­ple who haven’t played the game yet. (If you have trou­ble per­suad­ing some­one to test, offer­ing food could help.) If your game has a tar­get audi­ence defined, the per­fect tester would be a part of that audi­ence. I live in a small stu­dent house, so usu­al­ly I sit in the liv­ing room to invite some­one to test. Anoth­er thing I do is send­ing playtests per email, sup­ply­ing brows­er-playable builds and request­ing them to record footage. (I should note that friends and fam­i­ly are like­ly bad testers: they want to stay on your good side, so they won’t be hon­est if they think the game is bad. This is why I don’t just record lit­er­al feed­back, but also pay atten­tion to what they do and how they react to the game. More on this lat­er.)

Gen­er­al­ly, I try to avoid telling the testers any­thing in advance! No sto­ry, no con­trols, no rules, maybe a bit about the theme and genre to set the tone, or a logo or screen­shot for email tests. So I inter­fere as lit­tle as pos­si­ble (and prefer­ably, not at all) dur­ing the test. (so-called ‘blind test­ing’.) I tell them the game isn’t done yet, so they can fill in the blanks I haven’t drawn in yet, and look past the more obvi­ous short­com­ings.

Then, I start the game, sit down, watch silent­ly, and take a ton of notes. Dis­tin­guish your obser­va­tions (things of inter­est you see hap­pen­ing) from the feed­back you get (com­ments and sug­ges­tions giv­en by testers). After test­ing, I ask them some ques­tions. I rarely pre­pare those up front, but some­times I’m test­ing spe­cif­ic new changes that require me to poke the tester a lit­tle to mea­sure its effec­tive­ness. Avoid closed ques­tions (those answered with yes or no) for bet­ter feed­back, and ask fol­low up ques­tions if nec­es­sary. Always thank the tester, even if the feed­back was neg­a­tive, and I tell what I plan to do with the results to improve the game.

The more testers you can get, the bet­ter, though you should stop when tests give you less new infor­ma­tion. I usu­al­ly make one real­ly pol­ished build of the game and make sure two or three peo­ple play that, but more would be even bet­ter if you have the time.

Footage from a playtester who active­ly avoid­ed check­points because of their sim­i­lar­i­ty to the saws, mak­ing the game a lot hard­er than it need­ed to be.

Compiling feedback afterwards

Now I turn bits of info I obtained from the playtest into some­thing use­ful that I can improve my game with!

First off, sug­ges­tions. Peo­ple giv­ing sug­ges­tions don’t con­scious­ly real­ize what the under­ly­ing prob­lem is, so lit­er­al­ly imple­ment­ing their sug­ges­tion might cause new prob­lems on top of exist­ing ones. Sounds harsh, but I find this to be true. For exam­ple, if a play­er com­plains an ene­my is too pow­er­ful, the obvi­ous sug­ges­tion would be to low­er its attack pow­er. Upon fur­ther test­ing, you could dis­cov­er that the prob­lem lies else­where entire­ly– like a well-hid­den bug that caus­es incor­rect hit­box­es, or ene­mies behav­ing unpre­dictably. Look beyond the sur­face lev­el of these sug­ges­tions.

Then, the remain­ing feed­back. The same goes as for sug­ges­tions: they know what emo­tions they have (which results in their feed­back), but miss the exact cause(s) that trig­gered them. Again, it is up to design­er to uncov­er this. For exam­ple, in Fort­nite from Epic Games, design­ers stum­bled upon an issue where play­ers had dif­fi­cul­ty aim­ing. If you’d take that at face val­ue, you’d make the hit­box­es big­ger, right? After some more dig­ging, it appeared that ene­mies took too sharp turns around obsta­cles, mak­ing it hard­er to pre­dict where to shoot. Every prob­lem might be deep­er than how it seems ini­tial­ly, and it’s up to you to dig to the core of the issue.

Final­ly, my obser­va­tions. These might be small things I noticed (play­ers bump their heads into one par­tic­u­lar block) that know can fix eas­i­ly, or big­ger issues through­out the entire test (play­ers keep bump­ing their head con­stant­ly) that you don’t real­ly know the solu­tion to yet. Put both on a task list to fix lat­er.

Now I have an exten­sive list of tasks to fix! For me with two/three testers, these often range between the 30–40 items, vary­ing in sever­i­ty. Nor­mal­ly, I start a new test once all issues from the pre­vi­ous test have been fixed, to see if the issues actu­al­ly dis­ap­peared, but you can do one any­time.

A typ­i­cal check­list I would end up with after a playtest. I made this one in Trel­lo.

Alternative approaches

Further reading

Celia Hodent (Most­ly user expe­ri­ence relat­ed, but has some good info on how play­ers think about games and some snip­pets about playtest­ing. Espe­cial­ly the Gamer’s Brain series of talks is very good and a nice entry point into user expe­ri­ence, which is close­ly relat­ed to playtest­ing.)
Avoid­ing Evil Data (The design­er of Hid­den Folks has a more rig­or­ous playtest­ing dis­ci­pline, where playtests are held at high fre­quen­cy. Very inter­est­ing GDC talk.)
Ide­al amount of testers (This arti­cle argues that, if you test with five users, you’ll find 85% of all pos­si­ble issues, which I think is a pret­ty good per­cent­age. Debat­able, but you could start off with this amount of testers.)