Post Mortem on the Elemental Beta

Stardock is smart to do the beta the way they did. If they had of released a full version near the end of programming developing stage it would have been out there on the Pirate sites before the 24th of Aug got here thus cutting into their revenue and profits even more. I've been beta testing for 25 years and I've seen it all. Most beta testers just want a free game and hardly ever send in a bug report. In this case we all paid to be in the beta so really you weren't required to do as a normal beta tester would do. Back in my day we always got a free copy of the finished product for beta testing. I'm finding there aren't many that do that anymore. Today is you pay to be a beta tester either before the game comes out or after it comes out like Paradox (EU and HOI series) and Creative Assembly games (Total War series).

Definitely don't do a more closed application based beta. I pre-ordered Elemental JUST to have access to the beta and to try out this amazing game. I'd hate for them to suddenly say, "You can pre-order and just wait until the game comes out to try it". It's something unique with Stardock beta's that I love and I go into it KNOWING that this is a beta and not a demo. More people just need to understand what that means and be less ignorant of that fact.

I believe that the beta went well. I was only in two phases 3-C and 4 but it seems like they had steady progress, constant communication, and were responding to problems, bugs, and concerns. I don't think you can ask for much more then that.

Lastly, I will slightly agree that I wish we had more testing based around gameplay, HOWEVER, I don't think that is necessary for a good launch, beta, or game. It's just a personal request coming from a customer.

Stardock is doing fine, as is. Good work Stardock

Depends upon how you look at it, are you sure you didn't buy an experience and get a game in return? See an explanation I gave in some of the pre-order? threads that cropped up from time to time:

I agree that putting the buy-in up front is important as a first filter for folks looking for a demo.  You can say it as many times as you want, but the thought that there's an actual pre-build of a game out there gets folks (self-included sometimes) into a Demo mindset.  That was a good choice.  I wouldn't have called it a beta, necessarily, perhaps Alpha, or even just called it something completely different - perhaps "vested customer".

Build turnarounds were good, team communication with customers (through frogboy) was also good.  In other betas or more hierarchic designs, I think it'd be really cool to be able to talk to team-members beyond Frogboy, though I suppose that would expose them to the harshness and timesink that is the forum.

I think the Beta process was a great idea and I'd continue to do similar Betas in the future. 

I have mixed feelings about the beta process.

I was expecting the game-play to be more fun from the start and I was expecting to be testing the balance more while still in cloth map mode. At the end of the day if Stardock got what they needed from the process then it has to be called a success - even if the process became a bit frustrating from the testing side.

If what we did was called 'Alpha' instead of 'Beta' then I wouldn't have expected anything more than what we got. It certainly felt more of an alpha test than a beta test because we were mainly focussing on specific mechanics rather than the game itself. Thinking about it, that distinction would have saved me a fair amount of doubt about the final product during testing.

Another element I found frustrating was the lack of specific feedback about ideas in the beta forum. It seemed more feedback was given from the devs in the non-beta forums than in the beta forums and in the end I was posting hoping for some sort of response in other forums rather than providing direct feedback in the beta forum. I'm not sure how the devs processed the suggestions and bugs keeping it separate and controllable. From a beta tester perspective, doing justice to the beta process took an enormous amount of structured time and it felt at times that this time wasn't all that valued. Not that the devs could have done much more than they did - but that was a perception that I had during the process.

Also, there were many suggestions that testers were very passionate about that never got a response back despite obvious frustration and desperation. Even a comment like 'We're unsure at this stage but are discussing it.' would have sufficed at the time.

Having said all that about specific feedback, I was very impressed with the amount of developer information that kept coming through. Unheard of really when you consider their workloads.

As a conclusion I don't know if I would want to go through the process again because of the emotional involvement you get with the project that is hamstrung through the lack of power to change things.

I'd say that it was overall very positive. I'll say that again, as I'm going to focus on the improvement areas: a very positive experience.

So my main concern is on the 'demo' vs 'beta' expectations thing. Having been on the other side of this thing in the past (i.e. people beta testing my work), I know just what a large proportion of 'beta testers' don't post a single report. In fact in my experience, I could count on about 5% to do so, and only a handful of individuals to do it consistently and well. I think therefore guiding beta testers more is a must, in three ways:

1. Ensure a beta tester undersands what is wanted of him. There's a couple of posts on this in the forums, but this should also be a splash screen when the game loads, or in the front menu. With this, a beta tester knows what their going to see, but also what they're meant to do with it (i.e. not just hope for an early play of the game!)
2. Make it clear what your testing - each version should have some focus areas of things to look out for. Again, there were a couple of forum posts with each release, but again, not the right place for it.
3. Make uploading a beta test report even easier, with in game instructions. It was actually quite easy using dropbox to post a report, but it would have been even better if it could have been done in game (at least for non-crash bugs). Even taking a screenshot to report a UI bug involved alt-tabbing out and causing a crash. Since reload times were so long this discouraged reporting.

I think that the biggest thing they need to add in the next beta (or alpha, which what it really was) is a way to auto-report bugs. The way to provide dumps was very nonsensical and took a lot of time and effort and required Dropbox. 

If I have a little pop-up that lets me upload the debug.log file and lets me add a little comment, I'd file every bug and crash I could. 

I really like how beta started, and continued, but not how it ended. It was great, that stardock took so many ideas from players, and later on implemented them into the game. However there were some great ones, which were left behind. Overall, this building part of the beta was great success (I think). 

What I did not like, that beta ended before game was even complete, and all the balancing issues and polish is being done by stardock only. Other then that. I was really impressed. Not with the game itself, because gameplay experience was horrible, but with amount of changes and feedback gathered and implemented into the game. And that is what beta should be about.

I haven't been in Stardock betas before so I didn't know what was coming. But I agree with other posters, what we tested was an Alpha rather than Beta. The Beta will be the release version, and six months after that the game will be complete.

If that is the way Stardock do things I suppose it is ok. It was fun to be part of the process.

However. For testing to be more successful the builds need to be more stable. If you can't run the game you can't test anything, simple as that. more focus on stability and performance on an early stage is my suggestion for future betas.

And as mentioned above, an automatic bug reporting tool is needed. Or at least a forum with free image and debug-file uploads.

wouldn't mind if they gave out updates more often even if they are small.

I have been following the beta process from the outside, as when this all began I didn't have the money to plop down a preorder, and by the time I did they had cancelled it (you know, because it was a month from release and all that). So I missed it this time, and I am sad, but after reading over the Elemental BETA forums and the Elemental General forums, I can only say I wish I'd had the money sooner so I could have offered input.

I agree with the fellow who said that a lot of beta testers just sign in for a free demo, never comment, and therefore miss the point entirely. Alternatively, there are a number of people that end up beta testing that should just never be beta testers, because their understanding of things like "balance" and "practicality" and so on are.... negligible.

It's a tough situation, but I suggest the only way to get around that is either to hire professional beta testers or increase the pool of beta testers as best as possible -- the latter being something that the method of "everyone who preorders by X date is a beta tester" does very well.

i think that the experience overall was a good one.  i learned alot about the game design process.  i think that stardock showed a good balance of fairness to us and them.  i think sometimes that we the consumer get into this mentality that we are always right because we are the consumer, but we should also defer to the artists as well.  to be honest, what they did in giving us the beta took alot of courage.  it would be very hard for me to put something i worked so hard on out into the public to be so scrutinized(especially by the internet).

what i would change:  the only thing i would change would be an easier way to report bugs, like another had said.

They were up front with how the beta would be from the start...cloth map only and then would add in features.  Heck it even said on the opening screen about how painful this would be.  I appreciated the honesty and the chance to try it out.  I really loved when they added that old homage to dos games and thought that was great.  They listened to ideas and made changes and have done a great job!

Really like the Beta.

Stardock warned folks it was "Work in Progress" but many either did not read it, or more likely thought they knew what it meant and then were upset.

Betas used to be less a market tool and more a way to test and get feed back.

Most betas today are not beta but Pre-releases to push sales, and most feedback is ignored as the game is 99% finished.

This Beta was Bug Filled Wonderland, we got to see the concept taken shaped, tweaked, parts added, parts removed, new ideas tried, and old ones returned.

This should have helped players understand more of how a game is made.

All the hard work, watching things grow change, SNAP break, out of Mem, CTD, and the eager awaited Update.

At this point we never saw the "Beta" most games release, when we get the game we find all sorts off ohhs, ahhs, and a few WTF happened.

and for that I bought in many moons ago, and I am really wishing my travel vacation did not start on Sat.

Lee