Wednesday, August 12, 2009

Beta Testers: Getting Them. Keeping Them.

I've been running Spiderweb Software for about fifteen years now. I owe my ability to make a living writing my cute, little Indie RPGs to a lot of things. Tenacity. A moderate amount of designing skill. Dumb luck.

But, as much as anything else, I owe my ability to stay in business to a group of skilled, dedicated, hard-working volunteer beta testers.

There is a huge chasm between the rough, quivering lump of code you just wrote and a solid product you can distribute to actual people. To make your game/mod/level/adventure publicly available without melting someone's computer, you will need testers. And, if you are a lone individual or running a little company like mine, you won't be able to afford to pay QA people. Thus, you will need to go out and find testers.

I've been doing this for a long time. Here is some advice for how to find the people you need to test your game (or other sort of application) and how to deal with them when you have them.

1. How To Get Testing Volunteers

I am established and have a fan base, so this part is easy for me. I write that I need testers on my web site, take the beta testing application page live, and winnow through a huge pile of applications. If you're just getting started, you will have a harder time.

Make a web page and put up some info and screenshots, so that people can drop by and see what you are making. Then go to online forums where gamers hang out. Write a short, respectful post saying "Hey, I'm making this and I need testers. Drop by my web site and take a look. Send such and such information here." While forums justifiably hate advertisements, offering an opportunity like this is generally tolerated. (And even sometimes appreciated.)

Ask potential testers to send you information about themselves. What sort of computer do you have? What sort of hardware? What sort of games do you like? What have you tested before?

Hopefully, you will get applicants. However, I'll warn you now. You will get very few decent testers. People who can play a game, analyze it, spot bugs, report them in a coherent manner, and come back for more are very rare. They are precious jewels. You'll have to sort through a lot of, well, people who aren't jewels, to find them.

Also, don't be afraid to use your friends, but don't get your hopes up. Very few people have the mental skills and dedication it takes to be a good tester. The odds of any given friend having those characteristics are low. But give it a shot.

2. How To Screen Your Testers

When you get a pile of testers, you'll need to pick out the ones who are potentially useful. One advantage of asking them to write about themselves is that you will be able to scratch out a bunch of incoherent, confused, and crazy folks right away. If they can't answer a few simple questions, they will not be able to report bugs well. We all know how dangerous the incoherent can be in positions of importance.

Once I've picked out a few applications, I make them provide a signed Non-Disclosure Agreement (I let them fax the form, mail it, or e-mail me a .pdf signature). Google "non-disclosure agreement template" to find a few boilerplate versions.

Now, to be honest, if someone leaks your beta, there isn't much you can do about it. But make people sign an NDA anyway. It's legal protection. It stresses to them how important secrecy is. And making them send an NDA in is great for weeding out flakes. If they aren't together enough to send a signature, they probably won't be able to test well.

OK. Hopefully you have a few testers. Some of them might even be decent. If not, you'll be back to begging on the forums again.

3. How To Instruct Your Testers

You will need to give your testers clear instructions. How should they report bugs? Where should they report bugs to? Any particular format you want bug reports in? The exact details will vary depending on the sort of product you are trying to test. However, you will need to tell the testers something.

This is also a good chance to explain to them important basics. Things like how to take a screenshot or how to compress large files before sending them. And make sure to stress that, in any given report, they need to say the exact version of the beta they found the problem in. (So you don't go chasing down bugs you have already fixed.)

4. How To Interact With Your Testers

First off, always be nice. Always take constructive criticism well. These people are volunteers. They are spending their priceless and irreplaceable time helping you for free. Be polite. Be professional. Never lose your temper. If you find yourself writing a snide e-mail, get up. Take a walk. Count to ten. Then delete it. I have lost my temper with a tester only two or three times, and I always felt terrible about it afterwards.

This is a good chance to work on developing a thick skin. Nothing the testers subject you to will compare to the abuse you'll take online after your project has been released.

Don't hesitate to ask for more information about a report when you need it. Ask for exact steps to reproduce the problem. Screenshots. Saved games. Stress to testers that they need to provide lots of details. You never know what tiny scrap of information will help you to find a bug. Vague reports are useless reports. And, if a tester never answers your follow-up questions, that tester should probably be fired.

Yes, you will sometimes need to fire your volunteers. If they don't use the product. If they don't send helpful reports. If they don't treat you in a polite and professional manner. (The way you are treating them. Right?) Just because they aren't being paid doesn't mean that they should always be kept around. They are taking up your time and they are eating up a precious beta tester slot. Also, being able to beta test is valuable, even if no cash is paid. It is professional experience (more on that later) and a chance to actually take part in the creative process. These are valuable things.

5. How To Reward Your Testers

You must treat your testers with the respect and gratitude they deserve. Thank them. List them in the credits. Give them a free copy of your game.

Some testers will help you in order to get experience, in the hopes of getting a real QA job. If a helpful tester asks you for a reference, give one, and make it good. I have had people ask me for references before, and my success rate for getting people their desired positions is 100%.

And Good Luck.

Don't be ashamed to ask for help. For a certain, rare, marvelous breed of person, the ability to take part in the creative process is a great opportunity. Find those people and let them help you. Be nice. Be professional. Figure out what works for your particular product and go with it. After all, without testers, all software would suck.

31 comments:

  1. Thanks so much for this post, Jeff. I know that this advice will come in handy when I'm at this point with my project. Especially the part about giving the testers a few hoops to jump through to check their dedication. I'm always happy to see what you have to say :)

    ReplyDelete
  2. I was a beta tester for you way back in the day (on Exile 2 and 3). At the same time I was working in the QA Dept. at LucasArts. So I must have been a real masochist to want to come home and do more beta testing after work! But that's how psyched I was about your games. :) Also, I was interested to see how testing RPGs compared to testing other types of games.

    Unfortunately, I don't have the time or patience to look at betas (anything) anymore. But I did want to thank you for the opportunity. And I think I still list it at the bottom of my resume.

    ReplyDelete
  3. Whoa. I didn't know indie game companies require beta testers.

    I am currently working as a tester. I like games. How and where can I do some freelance beta testing for indie companies? :D

    ReplyDelete
  4. Like adampasz, I was honored to be a member of Spiderweb's beta tester stable for a good number of years (though I never worked in the field professionally). And it's on my resume as well.

    "Re-playability" gets thrown around a lot with RPGs, but while the QA work could get tedious, I never got bored working through the various Avernum or Geneforge game builds over and over. They are re-playable games.

    I always felt like my contributions were acknowledged and appreciated at Spiderweb, even when I insisted on testing out paths through the game that no sane person would take (like not summoning creatures ever in Geneforge!)

    Helping to solve a pernicious bug is a great feeling, and I'm still pleased that in one of the Geneforge games, there's a NPC with my last name on it. I always made a point of killing him for some reason . . .

    ReplyDelete
  5. I once beta tested for you (Geneforge 4 for Windows) and it was a lot of fun. My husband and I worked together on it, since he has an uncanny ability to "break" games. I had never tested before, but I always felt like I was doing something important and useful. Even though we really only ran into one minor bug that I think had already been found when we saw it, we still felt like we were contributing in some way. I think that's the real key for me, since it kept us going. It made it more fun. You had constant contact with us, and that contact made us feel needed. I'd test again for you in a heartbeat.

    ReplyDelete
  6. Like a surprising number of your former beta testers, I beta tested Avernum II (ten years ago) while I working as a QA tester for ADP. I know you read my bug reports, both because things I found got fixed and at least once you had to ask me to write a better report (my fault!).

    I'm always pleased to recommend your games to people, in part because I know how much care you put into them. Also, because as a college classmate of HWRNMNBSOL, I appreciate seeing him as a cranky old wizard.

    ReplyDelete
  7. Great post. Fun for me since I've worked in real QA for generally dull government software for a decade, and am generally well respected by my peers as a testing 'expert', and even reasonably paid (!), yet if I moved into games testing, which I would probably be much better at, I certainly wouldn't be valued as a "precious jewel"; I'd take a huge pay hit and barely be valued at all.

    ReplyDelete
  8. hey Jeff =) i'm interested in how to become a beta tester for the Spidweb game series, i would have loved to be a tester for the Avernum and especially Geneforge series!

    ReplyDelete
  9. Nice work. I've bought (if I'm counting right) 10 Spiderweb games. I'm going to pirating Avernum 6 to see the end of it, and then I'm done with your company because of the Sarah Palin shot.

    ReplyDelete
  10. Dear Jeff,

    I love your games, especially the Geneforge series, they kept me company during all my life (I'm 20). I'd love to be a tester of one of your next games.
    How can I become part of it?

    Antonio

    ReplyDelete
  11. No matter the ending is perfect or not, you can't disappear from my world eq2 platinum, In you life, there will at least one time that you forget yourself for someone, asking for no result eq2 plat, no company, no ownership nor love.Just ask for meeting you in my most beautiful years eq2 gold.

    I know someone in the world is waiting for me, although I've no idea of who he is. But I feel happy every day for this 4story Gold, If a woman is not sexy, she needs emotion; if she is not emotional, she needs reason 4story Luna, in this world, only those men who really feel happy can give women happiness 4story Gold.

    ReplyDelete
  12. After scan some of the articles on your web site these few days, and that i really like your form of blogging. I tag it to my favorites information processing system list and can be checking back shortly. Please cross-check my internet site additionally and let American state recognize what you think that. Agen Bola

    ReplyDelete
  13. This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. Italy Travel Guide Asia Travel Guide

    ReplyDelete
  14. About getting them and keeping them, it shows that you created this page with a lot of professionalism, keenness and preparedness. I just love the way you keep explaining issues for a deeper understanding. You have done well. For more information about project management, the lin provided is the best to visit.

    ReplyDelete
  15. friv games 2018 is new flash games which are expected to be the best Friv 4 games to play. You can find all game Friv 4 school free in Friv 2018 and Friv 4 School .Friv 2018 is one of the number one websites that offers thousands of exciting games around the world with high quality. Here, you can search for games in any category. We guarantee that your satisfaction is our pleasure.

    tủ nấu cơm
    máy làm Ðá

    ReplyDelete

  16. Free slots and free slots online games selected by professional! Find the best payout casino, free bonus online casino and best real money slots games!

    ReplyDelete
  17. I am often to blogging and i really appreciate your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information.

    Click Here
    Edublogs.org

    ReplyDelete
  18. The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought you have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention.

    Hub.docker.com
    Website
    Information

    ReplyDelete
  19. Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from youjogos friv gratis 2019
    Jogos 2019
    jogos friv
    jogos io net

    ReplyDelete
  20. Thank for this brilliant post.
    Quickbooks get easily integrated with hardware equipment such as receipt printers, barcode while square offers its offline mode square vs quickbooks

    ReplyDelete
  21. What an interesting article! I'm glad i finally found what i was looking for.
    토토사이트
    온라인경마

    ReplyDelete
  22. Yes i am totally agreed with this article and i just want to say that this article is very nice and vey informative. More blogs please.
    majortotositepro
    racesiteinfo
    oncasinositeinfo
    totopickpro

    ReplyDelete
  23. Howdy! Would you mind if I share your blog with my facebook group? There’s a lot of folks that I think would really enjoy your content. Please let me know. 카지노사이트
    바카라사이트
    카지노사이트

    ReplyDelete