With regards to testing, I think the industry overall is to blame, more than Stardock 'individually'
these days, "beta testing" pretty much means - publicity demo. We give the player base a near complete version to play with, catch the most egregious errors, and triage everything else.
And you can't blame them for that. On the one hand, it's cheap, basically free advertising of the best sort - hands on. Customers don't have to hold that box in their hand and wonder, "is this what i want?" And potentially put it back, go home, read some reviews (or forget about it entirely). And word of mouth - do you trust any other source as much? Think about, say, gamespot reviews about how awesome the combat is in dragon age. Seriously? The combat in dragon age... is pathetic. gory, but pathetic. It's like the most expensive, shiny, sexy and sleek car that is missing an engine, transmission, steering wheel, and doors. Not surprisingly, DA had spent quite a lot of money on advertising on that website. it's practically a given and understood now that there are, indeed, back curtain transactions going on between large companies and large review websites.
So what is a smaller, more ethically inclined company going to do? Especially if it doesn't have deep pockets to sink into more traditional forms of advertising? Short interviews, mag articles, previews, etc. only get you so far... especially if those outlets are, on the one hand 1) not getting paid by you to update info on your product, and thus may not generate the advertising revenue needed to stave off cup-o-noodle, and on the other hand 2) ARE getting paid by others to do just that. Who is going to get top billing and faster service?
So open, and even closed betas, whatever the genre and company (because even the big sleezy can get in on the action) become advertising outlets - if timed properly, it generates the right amount of release day hype and demand, and even works to 'guarantee' a certain amount of sales - kind of like a pre-order.
As for error catching in betas, again, you can't blame a company, any company, for triage-everything-but-the-most-egregious. Have you looked in the support forums lately? 30 pages. I'm certain there are NOT 30 pages of bugs. Some of it is suggestion. Some of it is people reporting a "bug" simply because they disagree with how a feature should be implemented, damn what anybody else thinks.. Some of it is duplicate posting. Some of it is people not sure if something is a bug. of the legitimate bug reports, some don't adequately describe the error. Some are not reproducible. Some are unintelligible to educated persons because it was written by an overexcited 11-27 year old who can't be bothered to use the proper syntax and spelling of their language, whichever it may be. All of them are tagged differently, and written in different formats.
And then there are the trolls. And the preponderance of conspiracy theorists. And the endless flood of people with a false sense of entitlement and self-righteousness. People who just can't say anything nice. You really want to do anything to help anyone out after reading page after page of how you are a crooked individual? And let's not forget the ubiquitous "i used to be a game developer" post (yes, the operative is "used to").
So basically, you end up with a clusterfuck. that's about the nicest way you can put it. You pay someone to go through it and consolidate the information as best they can, but it's not as simple as a job as you might think. Plus that person ends up with a seething hatred for humanity. And everyday, pages and pages are added, so it's a never ending job (30 pages, and the game has been out, what, a week?).
The key is the internal beta, and even then, there are problems. The generally involve fewer people (especially vs. open beta), and they are given a checklist of things to beta - to test.
From here on it, this is speculation: As i understand it, it's a tedious job of "do this, and then do it 50 more times." And you aren't really asked for your opinion (in the event that you even care anymore, after days and days of hours of "research this tech and then find this quest and complete it" over and over again). It's "does this work yes/no - what was the problem __________" It's not "how did this make you feel, what are some improvements that can be made, would you like some cake?"
Did combat work? yes/no - well let's see, I was able to engage in a fight, units were basically on opposite ends, and I was able to make attacks that killed the other side. that's a yes. Not a "Yes, see comments" and then a page about how the tactical map makes it hard to see the smaller units, sometimes it's hard to click specifically on a unit, wtf is this morale system, can we make hits and misses more clear on how those work, and so on and so forth."
because that doesn't help the coding team FINISH the product. because that gets right back up into "clusterfuck." In pretty much all of the real world, it's not about perfection - it's about screwing up the least. it's not about acing your exam - it's about not failing it.
I would even suspect, as the internal betas are frequently actually in-house, that they are all run on clone-pcs... identical rigs that will run the game. Because the point is not to see if it will run on EVERY possible hardware and software configuration. the point is to see if it runs AT ALL - if there is such a grievous coding error somewhere in the program that it will not even run on a PC meant to run the program. Because that NEEDS to be caught - that affects everyone. Whether your IRQ tubes having a hate-conflicting with your over-graphics heated super neon lighted whatever it is does-not-matter. Because it is impossible to anticipate for every possible hardware configuration.
Officer Calhoun, why didn't you shoot the fleeing armed robber?
"Well, I didn't have any bullets you see.."
Why not? Records indicate that the station is well stocked on 9mm rounds and that you were issued two clips before duty.
"Yes, that's true, but you see, I use a Frabrique Nationale Five seveN instead of a 9mm pistol, and it wont use 9mm bullets; those are the wrong caliber for an FN."
Dot dot dot.
Officer Calhoun, why didn't you shoot the fleeing armed robber?
"Well, I tried, but the bullets exploded in my gun."
"I used my standard issue sidearm and the standard issue bullets given to me, and when I tried to fire the pistol, it exploded. As you can see, my right hand is now a bloody stump."
Yes that is a problem, we'll have to thoroughly investigate this matter.
So it's not surprising that we are now seeing, almost as a surprise, that the AI is a bit... special. It rides a very short front-side bus. or whatever. And that all sorts of suggestions are now being made. What we, the consumer, think of when we hear "beta test" is actually more like "focus group." Where a select few people of the appropriate target market segment play around with something, and then are asked for their opinions, and so forth.
But even there, you got problems. 1) money. People don't spend that kind of time and effort for free, no matter what they say (and the people that DO, aren't necessarily the ones you want). and 2) you can't make everyone happy.
For example, there is at least one person who wants real-time action in elemental. Personally, i DON'T want that. So what's the result? Between the two of us, stalemate. whichever way stardock goes, in this closed universe of two-players there are 2 guarantees:
1) at most, 50% of the base will be happy. Potentially 0% will be happy.
2) it is 100% certain someone will not be happy.
...did you really need to spend money to figure that out? So you do what you do, and you generally figure people will self-select: once the details get out in the "open beta" (haha, there it is again!) people will either figure out "yes i want this" or "no i don't want this".
Predictably, there is always a significant proportion of the consumer base that decides, "No, I don't want this, but Yes, I do want to bitch about it." this isn't necessarily bad for the company - because income (let alone profit) is determined by how much you can sell. So from a "god, not cup o noodle again" standpoint, it's a good thing to capture that part of the market as well. It's just too bad that part of the market tends to piss everyone off, and occasionally muck up an otherwise great product.
TL;DR - beta testing isn't what you think it is, and stop bitching and let stardock do it's thing. Much love.