Getting a map or mod tested is very important and it’s something I’ve written about many times in the past and I still feel passionate about.
I even went as far as creating a SP beta testing team [now dead] that contained members with varying quality PCs, different levels of playing skill and even some modders. The idea was to create a committed group of individuals who took testing seriously and offer that service to other mods.
I am sad to say that the idea never really became successful but I still have faith it.
The problem is that testing a map is more than just playing it and while I could write about the various things that need to be done, this article looks at what the modders can do to help players test their mod.
I’ve seen a lot of demos/betas released recently and I doubt that the authors receive many feedback reports and that’s generally their fault. I can only guess but I bet the actual response rate is less than 2 percent. Why? I believe it’s because most players can’t be bothered to write a report and even if they can they might not know how.
Proper testing takes time and is quite hard work. That work should be not only acknowledged but also valued. There’s a modder that comes to mind that never credits their testers. I have little doubt if that person used models, sounds or textures from somebody else they would credit them, so why not the tester? I believe it’s because it’s not considered an important task.
Everybody thinks they can be a tester but that’s not true. There are skills required other than just being able to play something and look for mistakes; you must be able to communicate that information. In addition, the modder needs to find people they work well with. It’s no different in other skill sets within modding, so why would it be different here? There might be great testers but if they don’t gel with the modder, that won’t work.
But this is where things often fall down – the communication between the modder and tester. The modder must make it as easy as possible for the tester to communicate their findings. And that doesn’t simply mean giving them your email address or having a page on ModDB to post comments on.
You only need to take a look at the download section of ModDB to see that beta are rarely commented on, and when they are, the comments are not exactly helpful.
So, what is the problem?
At its core is the fact that the modder doesn’t really know what kind of feedback they want or need. They just release something because they have got a little bored or need some inspiration, and that’s the wrong time to release betas.
To get the right answers, you must ask the right questions.
What that means is that when you release a beta, either privately or publicly, you need to tell the testers EXACTLY what feedback you want AND make providing that feedback as simple as possible.
The first thing a modder needs to do is ask themselves what kind of feedback they require.
If they want bug reports then they must produce some simple form that can be completed:
– The nature of the bug.
– Where it occurred.
– What the player was doing what it occurred.
– Is it reproducible?
– The system specs of the tester’s PC.
– An easy way to include screenshots.
Of course they are lots of other questions that could be asked but the above is just an example. Then the modder can contact other testers and ask if they can reproduce the bug.
Some modders are looking for a more general view of the mod. Perhaps they feel that the bugs and technical issues have been corrected and want feedback about how it plays, the new HUD features, the weapons etc. Well, the same process can be used. Ask questions that can be easily answered. Preferably with open questions rather than closed questions.
A closed question is one that can be answered “Yes” or “No”. For example: “Did you enjoy the mod?” is a closed question. Whatever answer is given, it doesn’t really help the modder. “What was your favourite part of the mod and why?” is more open. It forces the tester to explain themselves. Now, if the modder can’t express themselves very well, are they really suitable as a tester?
My recommendation for modders looking for feedback is to use an online survey system. I use PollDaddy and the free service is easy to set up and use and provides a quick way to view the results. I have no doubt there are other services and modders should look for one that suits them.
You could use a simple text file template, an Excel or Word file, there are many options. Whilst not the point of this article, it’s worth noting that modders should try and encourage the use of demo recordings from their testers, especially for the first play through.
Next, comes the question of writing those questions and not every modder will have the skills to do that. Wouldn’t it be great if we could create a huge list of questions for modder to choose from, like a pick and mix shop? Is this even possible? I don’t know, but somebody could try.
What a modder needs to do is think very carefully about what he wants to know. “Is it easy to understand where the player needs to go and what they need to do?”, “If not, how can that be improved?” These are the kind of questions that can make a difference to the final product.
Lastly, I’d like to mention Interlopers.net. The feedback given on their forums to WIP posts is often excellent and a modder looking for help should spend more time on there than ModDB.
Well, I hope this article has given modders something to think about and in its small way helped them.
Whilst I can’t write the questions for you I am prepared to help with writing any questions you have, especially if your native language isn’t English. I won’t guarantee I can help everybody all the time but I will do my best.
Perhaps other PP readers with experience in Q&A or beta testing would offer to help with the questions?
This is s good point, PP. I wonder if the PP “team” could pull together the questions and actually post online here as a FAQ of some sort for any author to download and use. I don’t know how many authors might actually use it, but at any rate, it might at least strike the right nerve so they know what to look for when working with testers. But maybe a tester form already exists on some of the other modding fora out there???
If you ever create an SP beta testing team count me in.
I thoroughly agree with the whole article.
We’ve all seen mods that could have been so much better with proper testing.
‘Q&A” Yes that’s needed as a guideline and we all need those.
However a fixed Q& A will never, of it’s self only, be able to cover all eventualities as different mods will have different requirements
The good playtester must also go beyond guidelines.
The good playtester will test in as many ways of playing as can be imagined and this includes playing as you would not normally do (and might consider completely bonkers).
It’s for sure that I can come up with questions for testing.
I will return to this as I hope others do.
It’s up to some of us to help the artists release a polished product. This will enhance the enjoyment and pleasure for everyone, including the playtesters.
But modders, you need us to do this, just ask. The worst possible answer is no but then there’s nothing lost, just ask others.
The biggest hurdle I see is to get modders and totally trustworthy testers together. The prime concern being trustworthiness.
I don’t see an easy answer to this.
Ironically, this site is the best place to get feedback, reviews, etc. If we could set up a system to work around that, it would do wonders. If you suggested that in the article, my bad. Tl;dr.
I think it would be great if there was a centralised list of people that could be contacted who are willing to beta test mods.
I believe that simplifying the process of finding a suitable tester will increase the number of tested mods and improve the overall quality of new mods.
I’m not sure if this has been done already; If it has, could someone post a link. It would certainly be of benefit to new modders.
Otherwise, something should be set-up. It could even be part of a mod team recruitment site. I know I keep hearing on Podcast 17 that there is a critical shortage of source coders.
great idea Phillip!
as you know was I one of those (sp-)mappers in the beginning that didn’t want to give testers (outside the modteam) a change to beta test my sp-maps. I had very bad experiences with beta testers, one tried to steel my work and another one started to sent it around to his friends.
and this made that I didn’t want to work with beta testers for at least 5 years!!
so, good (read trustworthy) beta testers are not that easy to find.
still, after releasing several very large sp-mods without the help of good (external, meaning not there when the maps where actualy made) testers I had to agree that a good mappack or mod is only made with the help of good testers.
personally I never understand why people release a beta version of a mod, (other then like mentioned above, to get feedback on it). because I for one never play the final version of a mod ones I played the beta version of it, or it should be a really good mod. but in that case I probably wait for the final release. and I guess most players are like that.
it’s funny because the last few months we are in “beta-stage”, we did start out with 2 beta testers. it’s a shame that one of them only tested the first just to never hear of him again, without giving any reason. but, I am glad to say that the other beta tester, Jasper (see comment 2) is a very valuable member of our mod team by now.
should you need a good beta testers that takes his task seriously then I can recommend Jasper. although I hope that not to many request are made to him for testing because I hope to be able to use his testers skill for a long time, lol. not only does he test really good, beside that he really does his best to give input outside the testing.
about the testing itself, personally I asked Jasper to make his findings available to me through text and one screen (at least) of each item that is wrong. and there where needed to make a small video of it, because some issues can only be shown well through a recording.
I really want to become a beta tester. I’ve played too many mods that I thought were either too buggy or too boring. Ever since I first played Half-Life 2 back in “06, it changed the way I play games for the worse. I put valve games on such a high pedestal, that now when I play through games, I have to critique & criticize every little detail. Whenever I hear about a “great mod” I find myself going into it with too high expectations. There are a lot of mods & maps that just make me wish I had been there to say what was good, what needed work, and what could have been expanded on.
I’ve come up with plenty of ideas that I hate to see go unused. I’ve wanted to make mods of my own, so I’ve tried to learn how to model, to map & to program but I just hate all of it. The only other thing I can think of to do is playtest games. That way I might not see all of my ideas die.