Quality Assurance Team

[Z-Man]

It seems we could use one. The jobs of a QA Team are:

Find and report bugs in a developer friendly way. "Developer friendly" means that a developer should need as little of his time to reproduce the bug on his machine; a thirty seconds recording would be perfect.

Take bug reports from the community and turn them into developer friendly reports.

Have an ear for the needs of the community and suggest those features that would benefit the big picture best.

Occasionaly get special versions where security flaws or cheats have been fixed with instructions how to exploit the flaw in affected versions to verify that it was indeed fixed.

Possibly maintain the bug/feature request database. Assigning priorities/severities to bugs definitely is a QA job.

There is other stuff as well. Some members of the QA Team usually know a bit of scripting (shell scripts for us mostly, right now) and can directly get involved in the development here.

The power of the QA Team should be the timing of the releases. IMHO, it should go roughly like this: We're first doing alpha builds before a release, those get published on AABeta on the usual "try at your own risk" warning. The QA Team tests them a bit and decides when the alphas are good enough to begin the RC (Release Candidate) phase. RCs are also first published on AABeta by the developers at their pace, then the QA Team tests them thouroughly, compiles a list of known bugs, and only if they give the green light, the RC gets pushed to SourceForge and gets announced on the webpage, together with the bug list. After more testing, the QA Team then decides whether the RC is good enough to be called the final release; then it gets rebuild with the final version tag and treated again like a RC: published on AABeta, tested again, and published on SourceForge.

In short: The QA is the authority that turns mere builds into Releases. Nobody, not even the admins, can override the QA decision not to release. Within reasonable bounds. If no bugs in a build have been found for four weeks and QA still blocks the release, that would be stretching it. Or if the only problem left is a non-fatal bug that, according to the developer in charge, can't be fixed reasonably within the current framework.

Of course, giving this authority to QA is an admin decision. I think it should be settled and QA's task list should be completed (bet I forgot something) before we recruit anyone. Potential volunteer testers and those already on the Beta Testers group are welcome to join the discussion, of course, you should have a say in what shall be your job

[2020]

would love to take part
though i may have to be given strict instructions
since i know very little programming
and have a mac

but do consider me an enthusiastic play-tester...

[Lucifer]

I can go with it, with the minor caveat that the QA team doesn't get to be overridden by admins in the stable release branch. I'd like to keep as many blocks from the development series releases as possible, but I'd still like the QA team to at least sanity check the development series. Of course, if they're taking too long to do so (like a week or so), then we can and maybe should easily override them, but that's only in the development series.

I'd say at a minimum a QA guy needs to be able to build the game on his system. Unless we come up with some way to do something like automated builds for all platforms with nightly snapshots. I'd like the QA guys to be able to build on request to confirm whether or not bugs have actually been fixed. OTOH, I don't want to load them down with so many requirements we can't put together a good QA team.

As for their tasklist, why don't we just get a couple of guys who want to be on the QA team and ask them to work up their tasklist on the wiki. We developers will watch over it and add stuff from the depth of our own experience.

I guess obviously we need to try to get them off anonymous scm mirrors to avoid bottlenecks caused by those, but we can provide that ourselves. And access to the beta testers only forum.

[Lucifer]

Started the government page:

http://wiki.armagetronad.net/index.php/Government

[Z-Man]

Right, the QA Team's job is to make sure we meet our set quality standards for releases. Since those will be generally lower on experimental phases, possibly almost as low as for our alpha builds, QA won't have that much to say about them. I'd vote for making branches each 0.3.X release as well, making an alpha at the beginning of the branch, and giving QA one or two weeks to test that before 0.3.X final is build from the branch. Our quality standards should be defined somewhere on the wiki.

Yes, the list in the post basically is a personal wishlist of what work the QA Team should take away from me If the real team wants do do other things instead that help, that's fine by me.

What I personally REALLY hate is when I go to a server and play, and people try to make feature requests. I've got an impolite instant chat for these cases However, ingame chat is for most players the only way to communicate; they're not here on the forum, they don't come to IRC, and email is sooo last millennium. Maybe the QA Team members could, if they're in the mood to chat and take feature requests, wear a [QA] tag in their name as a sign for that? Of course, I'd do the same (if I'm in the mood). The critical part here is to make this convention known, I guess, but I bet people will ask what the QA Clan is and whether they can join and then it can be explained

It definitely helps if you can build your own version, but there is enough work for testers if they stick to alphas. Building those for Linux and Windows is almost automated, it takes me maybe ten minutes of real work and and hour waiting for the builds and the uploads to complete to do one. I could commit to making one almost every week.

2020: Unfortunately, Mac builds aren't automated for me, I have to ask Nemo to do them I don't know how much work or expertise is needed. But any help is appreciated; even if you have to stick to the builds that get done, that would be OK.

About a silly acronym for the team: How about Armagetronad QUality Assurers, AQUA?

So, who else wants to be a QAer?

[Tank Program]

It's a freaky idea, but OK I guess...

How about Armagetron Quality Und Assurance Team?

AQUAT?

[gnorty]

Sounds like the kind of thing I could help with, depending on how much time it takes up. I would be OK with up to about 6 hours a week.

Basic shell scripting is not a problem for me, and I have a basic knowledge of C++, certainly enough to be able to understand how bugs can happen. What's more, I am an engineer by trade, and am heavily involved in defect analysis and product improvement, so quite likely the skills I use day to day in my work would be migratable across to this.

[Lucifer]

So, gnorty and 2020. Can we get self destructo to jump in?

How about SQUAT? S(whatever) QUality Assurance Team.

"Why does this release keep crashing? What have the developers been doing all these years?"
"They didn't do SQUAT."

And of course S would have some official meaning, but would periodically mean "shitty" or "sucky" or "superb" or, mmm, another complimentary word that starts with S...

[Z-Man]

The usual procedure to give a random first letter a meaning is to make the acronym recursive. Silly, but it works.

[Lucifer]

I was thinking about what n54 said in irc, that the qa team would have too much power. I mentioned to him that there would have to be some synergy between the two for the thing to work anyway.

so what happens when the synergy isn't there? Then the qa team might have too much power, if you're inclined to rate stuff like that.

Aha, so the obvious check then is that the development team can fire the qa team.

Obviously, we wouldn't want it to come to that. That would be like invading-a-sovereign-country bad, so we'd want to reestablish the synergy thing if at all possible.

[wrtlprnft]

meaning of SQUAT:
Squat: QUaliaty Assurance Team

long live recursive acronymns

[belenus]

If someone has to "assure" the quality he needs a detailed changelog, for me thats especially for all the scripts and for others who understand C(++) a little better the sourcecode. If you change something, write it down because I would wanna see what was changed and not run into it blindfolded.

[wrtlprnft]

Well, there's the CVS/SVN logs that should have all important information on changes... There's a site with our CVS logs that also has a newsfeed.

Btw, will that CIA thing still work with SVN?

[Z-Man]

Yes, after the import, the CIA report script can be activated for SVN. I suspect if we do it too early, the whole import would get reported.

Belenus points to another task and power of SQUAT: they should read the ChangeLog (generated from commit logs) and be allowed to give telewedgies to developers who produce unreadable log entries. And of course they should guard over the documentation. That it gets written, that it's useful, that kind of stuff.

[wrtlprnft]

Then do activate it early to give us a boost in the activity stats

/me wonders if his log entries fall into the unreadable category